github.com/swiftstack/ProxyFS@v0.0.0-20210203235616-4017c267d62f/docs/source/architecture/log-structured-intro.rst (about)

     1  Log-Structured Filesystem
     2  =========================
     3  
     4  The ProxyFS filesystem takes advantage of the properties of its
     5  underlying object storage platform.
     6  
     7  Eventual Consistency and Strong Read-your-Writes
     8  ------------------------------------------------
     9  
    10  Object storage systems utilize eventual consistency properties that help
    11  enable its durability and availability properties. However, any new data
    12  that is written into the system isn’t subject to eventual consistency.
    13  This provides a strong read-your writes guarantee for new objects.
    14  
    15  Log-Structured Files
    16  --------------------
    17  
    18  This strong read-your writes property enables a log-structured object
    19  format. With this log-structured format, any data written is a new,
    20  uniquely-named object in the back-end storage. A 64-bit nonce number
    21  sequence that never repeats provides uniqueness.
    22  
    23  When a file is modified through the filesystem, a new log segment is
    24  written into the back-end storage that contains only the changes. Then,
    25  the log-structure is updated with the new segment.
    26  
    27  .. image:: /_static/ProxyFS-log-segments.png
    28   :width: 275pt
    29  
    30  
    31  Log-structured objects store data in such a way that is great for
    32  filesystem access and at the same time supports object API access. This
    33  is very similar to the strategies used for large object support where a
    34  manifest file is created for a multi-part object.
    35  
    36  Log-Structured Filesystem
    37  -------------------------
    38  
    39  The filesystem tree is stored in a similar way as a log-structured data
    40  itself. This filesystem tree can be snapshotted and stored into the
    41  back-end storage, just like the file data.
    42  
    43  Where filesystems and object APIs differ is that a filesystem is a
    44  hierarchy, whereas an object API is a flat namespace.
    45  
    46  When looking at the data through the filesystem, you will see folders,
    47  directories and files. When using an object API, those folders become
    48  slashes in the URL. The top-level directories simply map to
    49  containers/buckets in the object API.