github.com/cs3org/reva/v2@v2.27.7/changelog/2.3.0_2022-05-02/change-ocis-s3ng-fs-blob-layout.md (about)

     1  Change: Change the oCIS and S3NG  storage driver blob store layout
     2  
     3  We've optimized the oCIS and S3NG storage driver blob store layout.
     4  
     5  For the oCIS storage driver, blobs will now be stored inside the folder
     6  of a space, next to the nodes. This allows admins to easily archive, backup and restore
     7  spaces as a whole with UNIX tooling. We also moved from a single folder for blobs to
     8  multiple folders for blobs, to make the filesystem interactions more performant for
     9  large numbers of files.
    10  
    11  The previous layout on disk looked like this:
    12  
    13  ```markdown
    14  |-- spaces
    15  | |-- ..
    16  | |  |-- ..
    17  | |-- xx
    18  |   |-- xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned space id
    19  |     |-- nodes
    20  |       |-- ..
    21  |       |-- xx
    22  |         |-- xx
    23  |           |-- xx
    24  |             |-- xx
    25  |               |-- -xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned node id
    26  |-- blobs
    27    |-- ..
    28    |-- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <- blob id
    29  ```
    30  
    31  Now it looks like this:
    32  
    33  ```markdown
    34  |-- spaces
    35  | |-- ..
    36  | |  |-- ..
    37    |-- xx
    38      |-- xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned space id
    39        |-- nodes
    40        |  |-- ..
    41        |  |-- xx
    42        |    |-- xx
    43        |      |-- xx
    44        |        |-- xx
    45        |          |-- -xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned node id
    46        |-- blobs
    47          |-- ..
    48          |-- xx
    49            |-- xx
    50              |-- xx
    51                |-- xx
    52                  |-- -xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned blob id
    53  ```
    54  
    55  For the S3NG storage driver, blobs will now be prefixed with the space id and
    56  also a part of the blob id will be used as prefix. This creates a better prefix partitioning
    57  and mitigates S3 api performance drops for large buckets (https://aws.amazon.com/de/premiumsupport/knowledge-center/s3-prefix-nested-folders-difference/).
    58  
    59  The previous S3 bucket (blobs only looked like this):
    60  
    61  ```markdown
    62  |-- ..
    63  |-- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <- blob id
    64  ```
    65  
    66  Now it looks like this:
    67  
    68  ```markdown
    69  |-- ..
    70  |-- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <- space id
    71     |-- ..
    72     |-- xx
    73       |-- xx
    74         |-- xx
    75           |-- xx
    76             |-- -xxxx-xxxx-xxxx-xxxxxxxxxxxx <- partitioned blob id
    77  ```
    78  
    79  https://github.com/cs3org/reva/pull/2763
    80  https://github.com/owncloud/ocis/issues/3557