github.com/cockroachdb/pebble@v1.1.2/docs/RFCS/20221122_virtual_sstable.md

- Feature Name: Virtual sstables
- Status: in-progress
- Start Date: 2022-10-27
- Authors: Arjun Nair
- RFC PR: https://github.com/cockroachdb/pebble/pull/2116
- Pebble Issues:
  https://github.com/cockroachdb/pebble/issues/1683


** Design Draft**

# Summary

The RFC outlines the design to enable virtualizing of physical sstables
in Pebble.

A virtual sstable has no associated physical data on disk, and is instead backed
by an existing physical sstable. Each physical sstable may be shared by one, or
more than one virtual sstable.

Initially, the design will be used to lower the read-amp and the write-amp
caused by certain ingestions. Sometimes, ingestions are unable to place incoming
files, which have no data overlap with other files in the lsm, lower in the lsm
because of file boundary overlap with files in the lsm. In this case, we are
forced to place files higher in the lsm, sometimes in L0, which can cause higher
read-amp and unnecessary write-amp as the file is moved lower down the lsm. See
https://github.com/cockroachdb/cockroach/issues/80589 for the problem occurring
in practice.

Eventually, the design will also be used for the disaggregated storage masking
use-case: https://github.com/cockroachdb/cockroach/pull/70419/files.

This document describes the design of virtual sstables in Pebble with enough
detail to aid the implementation and code review.

# Design

### Ingestion

When an sstable is ingested into Pebble, we try to place it in the lowest level
without any data overlap, or any file boundary overlap. We can make use of
virtual sstables in the cases where we're forced to place the ingested sstable
at a higher level due to file boundary overlap, but no data overlap.

```
                                  s2
ingest:                     [i-j-------n]
                                  s1
L6:                 [e---g-----------------p---r]
             a b c d e f g h i j k l m n o p q r s t u v w x y z
```

Consider the sstable s1 in L6 and the ingesting sstable s2. It is clear that
the file boundaries of s1 and s2 overlap, but there is no data overlap as shown
in the diagram. Currently, we will be forced to ingest the sstable s2 into a
level higher than L6. With virtual sstables, we can split the existing sstable
s1 into two sstables s3 and s4 as shown in the following diagram.

```
                       s3         s2        s4
L6:                 [e---g]-[i-j-------n]-[p---r]
             a b c d e f g h i j k l m n o p q r s t u v w x y z
```

The sstable s1 will be deleted from the lsm. If s1 was a physical sstable, then
we will keep the file on disk as long as we need to so that it can back the
virtual sstables.

There are cases where the ingesting sstables have no data overlap with existing
sstables, but we can't make use of virtual sstables. Consider:
```
                                  s2
ingest:               [f-----i-j-------n]
                                  s1
L6:                 [e---g-----------------p---r]
             a b c d e f g h i j k l m n o p q r s t u v w x y z
```
We cannot use virtual sstables in the above scenario for two reasons:
1. We don't have a quick method of detecting no data overlap.
2. We will be forced to split the sstable in L6 into more than two virtual
   sstables, but we want to avoid many small virtual sstables in the lsm.

Note that in Cockroach, the easier-to-solve case happens very regularly when an
sstable spans a range boundary (which pebble has no knowledge of), and we ingest
a snapshot of a range in between the two already-present ranges.

slide in between two existing sstables is more likely to happen. It occurs when
we ingest a snapshot of a range in between two already present ranges.

`ingestFindTargetLevel` changes:
- The `ingestFindTargetLevel` function is used to determine the target level
  of the file which is being ingested. Currently, this function returns an `int`
  which is the target level for the ingesting file. Two additional return
  parameters, `[]manifest.NewFileEntry` and `*manifest.DeletedFileEntry`, will be
  added to the function.
- If `ingestFindTargetLevel` decides to split an existing sstable into virtual
  sstables, then it will return new and deleted entries. Otherwise, it will only
  return the target level of the ingesting file.
- Within the `ingestFindTargetLevel` function, the `overlapWithIterator`
  function is used to quickly detect data overlap. In the case with file
  boundary overlap, but no data overlap, in the lowest possible level, we will
  split the existing sstable into virtual sstables and generate the
  `NewFileEntry`s and the `DeletedFileEntry`. The `FilemetaData` section
  describes how the various fields in the `FilemetaData` will be computed for
  the newly created virtual sstables.

- Note that we will not split physical sstables into virtual sstables in L0 for
  the use case described in this RFC. The benefit of doing so would be to reduce
  the number of L0 sublevels, but the cost would be additional implementation
  complexity(see the `FilemetaData` section). We also want to avoid too many
  virtual sstables in the lsm as they can lead to space amp(see `Compaction`
  section). However, in the future, for the disaggregated storage masking case,
  we would need to support ingestion and use of virtual sstables in L0.

- Note that we may need an upper bound on the number of times an sstable is
  split into smaller virtual sstables. We can further reduce the risk of many
  small sstables:
  1. For CockroachDB's snapshot ingestion, there is one large sst (up to 512MB)
     and many tiny ones. We can choose the apply this splitting logic only for
     the large sst. It is ok for the tiny ssts to be ingested into L0.
  2. Split only if the ingested sst is at least half the size of the sst being
     split. So if we have a smaller ingested sst, we will pick a higher level to
     split at (where the ssts are smaller). The lifetime of virtual ssts at a
     higher level is smaller, so there is lower risk of littering the LSM with
     long-lived small virtual ssts.
  3. For disaggregated storage implementation, we can avoid masking for tiny
     sstables being ingested and instead write a range delete like we currently
     do. Precise details on the masking use case are out of the scope of this
     RFC.

`ingestApply` changes:
- The new and deleted file entries returned by the `ingestFindTargetLevel`
  function will be added to the version edit in `ingestApply`.
- We will appropriately update the `levelMetrics` based on the new information
  returned by `ingestFindTargetLevel`.


### `FilemetaData` changes

Each virtual sstables will have a unique file metadata value associated with it.
The metadata may be borrowed from the backing physical sstable, or it may be
unique to the virtual sstable.

This rfc lists out the fields in the `FileMetadata` struct with information on
how each field will be populated.

`Atomic.AllowedSeeks`: Field is used for read triggered compactions, and we can
populate this field for each virtual sstable since virtual sstables can be
picked for compactions.

`Atomic.statsValid`: We can set this to true(`1`) when the virtual sstable is
created. On virtual sstable creation we will estimate the table stats of the
virtual sstable based on the table stats of the physical sstable. We can also
set this to `0` and let the table stats job asynchronously compute the stats.

`refs`: The will be turned into a pointer which will be shared by the
virtual/physical sstables. See the deletion section of the RFC to learn how the
`refs` count will be used.

`FileNum`: We could give each virtual sstable its own file number or share
the file number between all the virtual sstables. In the former case, the virtual
sstables will be distinguished by the file number, and will have an additional
metadata field to indicate the file number of the parent sstable. In the latter
case, we can use a few of the most significant bits of the 64 bit file number to
distinguish the virtual sstables.

The benefit of using a single file number for each virtual sstable, is that we
don't need to use additional space to store the file number of the backing
physical sstable.

It might make sense to give each virtual sstable its own file number. Virtual
sstables are picked for compactions, and compactions and compaction picking
expect a unique file number for each of the files which it is compacting.
For example, read compactions will use the file number of the file to determine
if a file picked for compaction has already been compacted, the version edit
will expect a different file number for each virtual sstable, etc.

There are direct references to the `FilemetaData.FileNum` throughout Pebble. For
example, the file number is accessed when the the `DB.Checkpoint` function is
called. This function iterates through the files in each level of the lsm,
constructs the filepath using the file number, and reads the file from disk. In
such cases, it is important to exclude virtual sstables.

`Size`: We compute this using linear interpolation on the number of blocks in
the parent sstable and the number of blocks in the newly created virtual sstable.

`SmallestSeqNum/LargestSeqNum`: These fields depend on the parent sstable,
but we would need to perform a scan of the physical sstable to compute these
accurately for the virtual sstable upon creation. Instead, we could convert
these fields into lower and upper bounds of the sequence numbers in a file.

These fields are used for l0 sublevels, pebble tooling, delete compaction hints,
and a lot of plumbing. We don't need to worry about the L0 sublevels use case
because we won't have virtual sstables in L0 for the use case in this RFC. For
the rest of the use cases we can use lower bound for the smallest seq number,
and an upper bound for the largest seq number work.

TODO(bananabrick): Add more detail for any delete compaction hint changes if
necessary.

`Smallest/Largest`: These, along with the smallest/largest ranges for the range
and point keys can be computed upon virtual sstable creation. Precisely, these
can be computed when we try and detect data overlap in the `overlapWithIterator`
function during ingestion.

`Stats`: `TableStats` will either be computed upon virtual sstable creation
using linear interpolation on the block counts of the virtual/physical sstables
or asynchronously using the file bounds of the virtual sstable.

`PhysicalState`: We can add an additional struct with state associated with
physical ssts which have been virtualized.

```
type PhysicalState struct {
  // Total refs across all virtual ssts * versions. That is, if the same virtual
  // sst is present in multiple versions, it may have multiple refs, if the
  // btree node is not the same.
  totalRefs int32

  // Number of virtual ssts in the latest version that refer to this physical
  // SST. Will be 1 if there is only a physical sst, or there is only 1 virtual
  // sst referencing this physical sst.
  // INVARIANT: refsInLatestVersion <= totalRefs
  // refsInLatestVersion == 0 is a zombie sstable.
  refsInLatestVersion int32

  fileSize uint64

  // If sst is not virtualized and in latest version
  // virtualSizeSumInLatestVersion == fileSize. If
  // virtualSizeSumInLatestVersion > 0 and
  // virtualSizeSumInLatestVersion/fileSize is very small, the corresponding
  // virtual sst(s) should be candidates for compaction. These candidates can be
  // tracked via btree annotations. Incrementlly updated in
  // BulkVersionEdit.Apply, when updating refsInLatestVersion.
  virtualSizeSumInLatestVersion uint64
}
```

The `Deletion` section and the `Compactions` section describe why we need to
store the `PhysicalState`.

### Deletion of physical and virtual sstables

We want to ensure that the physical sstable is only deleted from disk when no
version references it, and when there are no virtual sstables which are backed
by the physical sstable.

Since `FilemetaData.refs` is a pointer which is shared by the physical and
virtual sstables, the physical sstable won't be deleted when it is removed
from the latest version as the `FilemetaData.refs` will have been increased
when the virtual sstable is added to a version. Therefore, we only need to
ensure that the physical sstable is eventually deleted when there are no
versions which reference it.

Sstables are deleted from disk by the `DB.doDeleteObsoleteFiles` function which
looks for files to delete in the the `DB.mu.versions.obsoleteTables` slice.
So we need to ensure that any physical sstable which was virtualized is added to
the obsolete tables list iff `FilemetaData.refs` is 0.

Sstable are added to the obsolete file list when a `Version` is unrefed and
when `DB.scanObsoleteFiles` is called when Pebble is opened.

When a `Version` is unrefed, sstables referenced by it are only added to the
obsolete table list if the `FilemetaData.refs` hits 0 for the sstable. With
virtual sstables, we can have a case where the last version which directly
references a physical sstable is unrefed, but the physical sstable is not added
to the obsolete table list because its `FilemetaData.refs` count is not 0
because of indirect references through virtual sstables. Since the last Version
which directly references the physical sstable is deleted, the physical sstable
will never get added to the obsolete table list. Since virtual sstables keep
track of their parent physical sstable, we can just add the physical sstable to
the obsolete table list when the last virtual sstable which references it is
deleted.

`DB.scanObsoleteFiles` will delete any file which isn't referenced by the
`VersionSet.versions` list. So, it's possible that a physical sstable associated
with a virtual sstable will be deleted. This problem can be fixed by a small
tweak in the `d.mu.versions.addLiveFileNums` to treat the parent sstable of
a virtual sstable as a live file.

Deleted files still referenced by older versions are considered zombie sstables.
We can extend the definition of zombie sstables to be any sstable which is not
directly, or indirectly through virtual sstables, referenced by the latest
version. See the `PhysicalState` subsection of the `FilemetaData` section
where we describe how the references in the latest version will be tracked.


### Reading from virtual sstables

Since virtual sstables do not exist on disk, we will have to redirect reads
to the physical sstable which backs the virtual sstable.

All reads to the physical files go through the table cache which opens the file
on disk and creates a `Reader` for the reads. The table cache currently creates
a `FileNum` -> `Reader` mapping for the physical sstables.

Most of the functions in table cache API take the file metadata of the file as
a parameter. Examples include `newIters`, `newRangeKeyIter`, `withReader`, etc.
Each of these functions then calls a subsequent function on the sstable
`Reader`.

In the `Reader` API, some functions only really need to be called on physical
sstables, whereas some functions need to be called on both physical and virtual
sstables. For example, the `Reader.EstimateDiskUsage` usage function, or the
`Reader.Layout` function only need to be called on physical sstables, whereas,
some function like, `Reader.NewIter`, and `Reader.NewCompactionIter` need to
work with virtual sstables.

We could either have an abstraction over the physical sstable `Reader` per
virtual sstable, or update the `Reader` API to accept file bounds of the
sstable. In the latter case, we would create one `Reader` on the physical
sstable for all of the virtual sstables, and update the `Reader` API to accept
the file bounds of the sstable.

Changes required to share a `Reader` on the physical sstable among the virtual
sstable:
- If the file metadata of the virtual sstable is passed into the table cache, on
  a table cache miss, the table cache will load the Reader for the physical
  sstable. This step can be performed in the `tableCacheValue.load` function. On
  a table cache hit, the file number of the parent sstable will be used to fetch
  the appropriate sstable `Reader`.
- The `Reader` api will be updated to support reads from virtual sstables. For
  example, the `NewCompactionIter` function will take additional
  `lower,upper []byte` parameters.

Updates to iterators:
- `Reader.NewIter` already has `lower,upper []byte` parameters so this requires
   no change.
- Add `lower,upper` fields to the `Reader.NewCompactionIter`. The function
  initializes single level and two level iterators, and we can pass in the
  `lower,upper` values to those. TODO(bananabrick): Make sure that the value
  of `bytesIterated` in the compaction iterator is still accurate.
- `Reader.NewRawRangeKeyIter/NewRawRangeDelIter`: We need to add `lower/upper`
   fields to the functions. Both iterators make use of a `fragmentBlockIter`. We
   could filter keys above the `fragmentBlockIter` or add filtering within the
   `fragmentBlockIter`. To add filtering within the `fragmentBlockIter` we will
   initialize it with two additional `lower/upper []byte` fields.
- We would need to update the `SetBounds` logic for the sstable iterators to
  never set bounds for the iterators outside the virtual sstable bounds. This
  could lead to keys outside the virtual sstable bounds, but inside the physical
  sstable bounds, to be surfaced.

TODO(bananabrick): Add a section about sstable properties, if necessary.

### Compactions

Virtual sstables can be picked for compactions. If the `FilemetaData` and the
iterator stack changes work, then compaction shouldn't require much, if any,
additional work.

Virtual sstables which are picked for compactions may cause space amplification.
For example, if we have two virtual sstables `a` and `b` in L5, backed by a
physical sstable `c`, and the sstable `a` is picked for a compaction. We will
write some additional data into L6, but we won't delete sstable `c` because
sstable `b` still refers to it. In the worst case, sstable `b` will never be
picked for compaction and will never be compacted into and we'll have permanent
space amplification. We should try prioritize compaction of sstable `b` to
prevent such a scenario.

See the `PhysicalState` subsection in the `FilemetaData` section to see how
we'll store compaction picking metrics to reduce virtual sstable space-amp.

### `VersionEdit` decode/encode
Any additional fields added to the `FilemetaData` need to be supported in the
version edit `decode/encode` functions.