kythe.io@v0.0.68-0.20240422202219-7225dbc01741/kythe/docs/rfc/2909.md

# Background

The current Kythe serving data format is just a simple key-value store from string keys (usually
[Kythe tickets](https://kythe.io/docs/kythe-uri-spec.html)) to ProtocolBuffer message values.  This
format requires the Kythe PostProcessor to perform a complex reduction over a potentially massive
number of decorations/references/edges/etc. to produce a single lookup value per node.  A set of
CrossReferences can be so large that it already requires manual paging, another complex
transformation in the PostProcessor.  These operations tend to create hot shards and stragglers,
leading to long post-processing times and potentially failures due to memory exhaustion.  Likewise,
the server is forced to slowly decode large ProtocolBuffer messages even when only a subset of the
data is required and the large value sizes can cause issues with some backend stores.

# Proposal

Split serving table ProtocolBuffer values into their component fields, making each an independent
key-value entry.  Single lookup API calls become scans over a key prefix; paging becomes trivial.
Due the characteristics of LevelDB/SSTables, the split key-value entries will most likely be read
together (see SSTable block encoding), keys will be prefix-encoded, and there should be no
significant performance hit for reads.  On the contrary, since the Kythe API gives users the ability
to request partial data, the split key-value entries allows the server to possibly skip over large
segments of data while scanning and save the cost of decoding for a performance improvement.
Examples include a user requesting only small span of a file's decorations or a user requesting only
a single kind of cross-references.

More importantly, keeping serving data separate lets the Kythe PostProcessor do fewer large joins.
There will no longer need to be a join of all file decorations to create the single
`FileDecorations` proto (likewise for CrossReferences).  Fewer large joins avoids stragglers in the
PostProcessor and removes the need for many "limiters" (i.e. cutoff values for the size of outputs).

The columnar format is designed primarily to accommodate NoSQL/LevelDB constraints (e.g. unique,
ordered keys with arbitrary values), but it could be general enough to fit other storage backends.
For instance, the keys could be split and put into a relational database alongside their values.

# Implementation

Essentially each field of an existing ProtocolBuffer value in the serving data becomes zero or more
key-value entries depending on whether it's unset, set, or a repeated field.  The bulk of the field
data will be represented in the key and the keys will be encoded as
[orderedcodes](https://godoc.org/github.com/google/orderedcode).  It's important that each key be
unique to fit storage models like LevelDB which are not multimaps (unlike the underlying SSTables).
Tags within keys will indicate how to decode the values themselves and help order the key-value
entries.  The ordering of the keys will be chosen as to best fit a performant server, allowing for
easy, single-pass scans to retrieve all possible data.

For message-valued fields, further deconstruction of their component fields is necessary to produce
key-value entries.  Much of their component field values are moved into an entry's key so that it is
guaranteed unique.  The leftover data are made into the entry's value.  Sometimes a single field may
be transformed into multiple entries to remove data redundancy and to allow entries to be
independently processed.  A good example is the `FileDecorations.decoration` field.  A single
`Decoration` is actually decomposed into two separate entries, one storing what can be thought of as
the "actual" reference within the file (i.e. `start-end-kind-target` with no value) and one to store
the `target_definition`.

As most Kythe ProtocolMessages are relatively flat and consist of basic types (strings, integers,
and floats), most of the below format is trivial.  More complex types can also be handled as long as
a finite, unique key can be constructed of the aforementioned basic values for each value.  For
instance, enums may be converted to integers and arbitrary bytes may be base64-encoded.  Even
recursive types such as `MarkedSource` may be stored as values since they are associated with
`VName` keys.

The scope of this proposal only includes FileDecorations, CrossReferences, and Edges.  Other
services could be made to use similar formats, but the most improvement will be gained from these
three APIs.

## FileDecorations

### Existing ProtocolBuffer definition:

```protobuf
// Simplified FileDecorations
message FileDecorations {
  File file = 1;

  repeated Decoration decoration = 2;
  repeated ExpandedAnchor target_definitions = 3;
  repeated Node target = 4;
  repeated Override target_override = 5;
  repeated kythe.proto.common.Diagnostic diagnostic = 6;

  message Decoration {
    RawAnchor anchor = 1;
    string kind = 2;
    string target = 5;
    string target_definition = 4;
  }
  message Override {
    enum Kind {
      OVERRIDES  = 0;
      EXTENDS    = 1;
    }
    string overriding = 1;
    string overridden = 2;
    string overridden_definition = 5;
    Kind kind = 3;
    kythe.proto.common.MarkedSource marked_source = 4;
  }
}
```

### Key-value format:

| key | value | kind |
|---|---|---|
| `"fd"-file` | `FileDecorationsIndex` | file metadata |
| `"fd"-file-00-start-end` | bytes | file contents for span |
| `"fd"-file-10-start-end-kind-target` | `<empty>` | decoration targets |
| `"fd"-file-20-target-kind-override` | `<empty>` | override per target |
| `"fd"-file-30-target` | `srvpb.Node` | target node facts |
| `"fd"-file-40-target` | `spb.VName` | definition for each target |
| `"fd"-file-50-def` | `srvpb.ExpandedAnchor` | each definition's location |
| `"fd"-file-60-override` | `MarkedSource` | `MarkedSource` per override |
| `"fd"-file-70-start-end-hash` | `Diagnostic` | `Diagnostic` per span (use `-1:-1` span for entire file) |


Notes:

* Each value will be contained within a wrapper message to support evolution of the table format
* All file decoration entries are prefixed by "fd" and the file's `VName`
* `VNames` will be stored instead of [Kythe tickets](https://kythe.io/docs/kythe-uri-spec.html)
* The `FileDecorationsIndex` is used to store metadata for the rest of the entries and acts as an existence check
* File text is chunked to support very large files (most files should be a single entry)
* Decorations are ordered by span to support a narrow `FileDecorationsRequest` (group 10)
* Node facts/definitions (groups 30 and 40) are stored after decorations and overrides to allow for the reader to skip over unneeded nodes (due to a narrow request)
* Each definition is stored uniquely (group 50) and are related to nodes separately (group 40)

## CrossReferences

### Existing ProtocolBuffer definition:

```protobuf
// Simplified PagedCrossReference (no paging, removed deprecated features)
message PagedCrossReferences {
  string source_ticket = 1;
  kythe.proto.common.MarkedSource marked_source = 6;
  repeated string merge_with = 7;

  repeated Group group = 2;
  Node source_node = 8;

  message Group {
    string kind = 1;
    repeated ExpandedAnchor anchor = 2;
    repeated RelatedNode related_node = 3;
    repeated Caller caller = 4;
  }
  message RelatedNode {
    Node node = 1;
    int32 ordinal = 2;
  }
  message Caller {
    ExpandedAnchor caller = 1;
    string semantic_caller = 2;
    kythe.proto.common.MarkedSource marked_source = 3;
    repeated ExpandedAnchor callsite = 4;
  }
}
```

### Key-value format:

| key                                    | value                                 | kind                     |
|----------------------------------------|---------------------------------------|--------------------------|
| `"xr"-source`                          | `CrossReferencesIndex`                |                          |
| `"xr"-source-00-kind-file-start-end`   | `srvpb.ExpandedAnchor`                | Regular cross-reference  |
| `"xr"-source-10-kind-ordinal-node`     |                                       | Non-ref relations        |
| `"xr"-source-20-caller`                | `srvpb.ExpandedAnchor`+`MarkedSource` | Definition for Caller    |
| `"xr"-source-20-caller-file-start-end` | `srvpb.ExpandedAnchor`                | Callsite                 |
| `"xr"-source-30-node`                  | `srvpb.Node`                          | Related nodes            |
| `"xr"-source-40-node`                  | `srvpb.ExpandedAnchor`                | Related node definitions |


Notes:

* Each value will be contained within a wrapper message to support evolution of the table format
* All cross-reference entries are prefixed by "xr" and the node's `VName`
* `VNames` will be stored instead of [Kythe tickets](https://kythe.io/docs/kythe-uri-spec.html)
* The `CrossReferencesIndex` is used to store metadata for the rest of the entries (e.g. counts, node facts, `merge_with`) and acts as an existence check
* `Group`s are not used (each `Group` message is spread across multiple KV entries with a shared prefix)
* `kind` fields in keys include a "priority" to achieve desired sorting order (this could alternatively be included in the group number)
* After their `kind`, cross-references are ordered by file location

## Edges

### Existing ProtocolBuffer definition:

```protobuf
// Simplified PagedEdgeSet (no paging, removed deprecated features)
message PagedEdgeSet {
  message EdgeGroup {
    message Edge {
      Node target = 1;
      int32 ordinal = 2;
    }
    string kind = 1;
    repeated Edge edge = 2;
  }

  Node source = 1;
  repeated EdgeGroup group = 2;
}
```

### Key-value format:

| key                                  | value                  | kind                          |
|--------------------------------------|------------------------|-------------------------------|
| `"eg"-source`                        | `EdgesIndex`           |                               |
| `"eg"-source-10-kind-ordinal-target` | `srvpb.EdgeGroup.Edge` | Single edge                   |
| `"eg"-source-20-target`              | `srvpb.Node`           | Node facts for an edge target |


Notes:

* Each value will be contained within a wrapper message to support evolution of the table format
* All edge entries are prefixed by "eg" and the node's `VName`
* `VNames` will be stored instead of [Kythe tickets](https://kythe.io/docs/kythe-uri-spec.html)
* Each `EdgesIndex` will include the source node's facts (for the `Nodes` API)
* Node facts for edge targets are separated from the edge relations to remove duplication

## Beam Pipeline

The changes to the Beam workflow will be relatively straight-forward.  The current
[combineDecorPieces](https://github.com/kythe/kythe/blob/7f5ba6a/kythe/go/serving/pipeline/beam.go#L178)
and
[groupCrossRefs](https://github.com/kythe/kythe/blob/7f5ba6a/kythe/go/serving/pipeline/beam.go#L116)
`beam.CombinePerKey` operations will each be replaced by a single `beam.ParDo` that converts the
`*ppb.Reference`/`*ppb.DecorationPiece` messages to `([]byte, []byte)` key-value entries as per the
formats above.

## Serving Table Reader

As opposed to the post-processor changes, the changes to
[serving/xrefs/xrefs.go](https://github.com/kythe/kythe/blob/7f5ba6a28370d6e3e2530b4750ec56e07888ea41/kythe/go/serving/xrefs/xrefs.go)
will be necessarily expansive.  The current implementation is intimately tied to the current format.
Transitionally, the columnar format will be added alongside the current `CombinedTable`
implementation so that either it or the new table format will be accepted by the `http_server` until
the legacy format is deprecated and removed.