github.com/minio/minio@v0.0.0-20240328213742-3f72439b8a27/docs/bucket/versioning/DESIGN.md (about)

     1  # Bucket Versioning Design Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
     2  
     3  ## Description of `xl.meta`
     4  
     5  `xl.meta` is a new self describing backend format used by MinIO to support AWS S3 compatible versioning.
     6  This file is the source of truth for each `version` at rest. `xl.meta` is a msgpack file serialized from a
     7  well defined data structure. To understand `xl.meta` here are the few things to start with
     8  
     9  `xl.meta` carries first 8 bytes an XL header which describes the current format and the format version,
    10  allowing the unmarshaller's to automatically use the right data structures to parse the subsequent content in the stream.
    11  
    12  ### v1.0
    13  
    14  | Entry     | Encoding    | Content
    15  | ----------|-------------|----------------------------------------
    16  | xlHeader  | [4]byte     | `'X', 'L', '2', ' '`
    17  | xlVersion | [4]byte     | `'1', ' ', ' ', ' '`
    18  | xlMetaV2  | msgp object | All versions as single messagepack object
    19  | [EOF] | |
    20  
    21  ### v1.1+
    22  
    23  Version 1.1 added inline data, which will be placed after the metadata.
    24  
    25  Therefore, the metadata is wrapped as a binary array for easy skipping.
    26  
    27  | Entry          | Encoding       | Content
    28  | ---------------|----------------|----------------------------------------
    29  | xlHeader       | [4]byte        | `'X', 'L', '2', ' '`
    30  | xlVersionMajor | uint16         | Major xl-meta version.
    31  | xlVersionMinor | uint16         | Minor xl-meta version.
    32  | xlMetaV2       | msgp bin array | Bin array with serialized metadata
    33  | crc            | msgp uint      | Lower 32 bits of 64 bit xxhash of previous array contents (v1.2+ only)
    34  | inline data    | binary         | Inline data if any, see Inline Data section for encoding.  
    35  | [EOF] | |
    36  
    37  ## v1.0-v1.2 Versions
    38  
    39  `xl.meta` carries three types of object entries which designate the type of version object stored.
    40  
    41  - ObjectType (default)
    42  - LegacyObjectType (preserves existing deployments and older xl.json format)
    43  - DeleteMarker (a versionId to capture the DELETE sequences implemented primarily for AWS spec compatibility)
    44  
    45  A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/minio/minio/tree/master/docs/debugging#decoding-metadata) program.
    46  
    47  ```json
    48  {
    49    "Versions": [
    50      {
    51        "Type": 1,
    52        "V2Obj": {
    53          "ID": "KWUs8S+8RZq4Vp5TWy6KFg==",
    54          "DDir": "X3pDAFu8Rjyft7QD6t7W5g==",
    55          "EcAlgo": 1,
    56          "EcM": 2,
    57          "EcN": 2,
    58          "EcBSize": 10485760,
    59          "EcIndex": 3,
    60          "EcDist": [3, 4, 1, 2],
    61          "CSumAlgo": 1,
    62          "PartNums": [1],
    63          "PartETags": [""],
    64          "PartSizes": [314],
    65          "PartASizes": [282],
    66          "Size": 314,
    67          "MTime": 1591820730,
    68          "MetaSys": {
    69            "X-Minio-Internal-Server-Side-Encryption-S3-Kms-Key-Id": "bXktbWluaW8ta2V5",
    70            "X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key": "ZXlKaFpXRmtJam9pUVVWVExUSTFOaTFIUTAwdFNFMUJReTFUU0VFdE1qVTJJaXdpYVhZaU9pSkJMMVZzZFVnelZYVjZSR2N6UkhGWUwycEViRmRCUFQwaUxDSnViMjVqWlNJNklpdE9lbkJXVWtseFlWSlNVa2t2UVhNaUxDSmllWFJsY3lJNklrNDBabVZsZG5WU1NWVnRLMFoyUWpBMVlYTk9aMU41YVhoU1RrNUpkMDlhTkdKa2RuaGpLMjFuVDNnMFFYbFJhbE15V0hkU1pEZzNRMk54ZUN0SFFuSWlmUT09",
    71            "X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "REFSRXYyLUhNQUMtU0hBMjU2",
    72            "X-Minio-Internal-Server-Side-Encryption-Iv": "bW5YRDhRUGczMVhkc2pJT1V1UVlnbWJBcndIQVhpTUN1dnVBS0QwNUVpaz0=",
    73            "X-Minio-Internal-Server-Side-Encryption-S3-Sealed-Key": "SUFBZkFPeUo5ZHVVSEkxYXFLU0NSRkJTTnM0QkVJNk9JWU1QcFVTSXFhK2dHVThXeE9oSHJCZWwwdnRvTldUNE8zS1BtcWluR0cydmlNNFRWa0N0Mmc9PQ=="
    74          },
    75          "MetaUsr": {
    76            "content-type": "application/octet-stream",
    77            "etag": "20000f00f58c508b40720270929bd90e9f07b9bd78fb605e5432a67635fc34722e4fc53b1d5fab9ff8400eb9ded4fba2"
    78          }
    79        }
    80      }
    81    ]
    82  }
    83  ```
    84  
    85  ### v1.3+ versions
    86  
    87  Version 1.3 introduces changes to help with [faster metadata reads and updates](https://blog.min.io/minio-versioning-metadata-deep-dive/)
    88  
    89  | Entry           | Encoding                    | Content
    90  | ----------------|-----------------------------|----------------------------------------
    91  | xlHeaderVersion | msgp uint                   | header version identifier
    92  | xlMetaVersion   | msgp uint                   | metadata version identifier
    93  | versions        | msgp int                    | Number of versions following
    94  | header_1        | msgp bin array              | Header of version 1
    95  | metadata_1      | msgp bin array              | Metadata of version 1
    96  | ...header_n     | msgp bin array              | Header of last version
    97  | ...metadata_n   | msgp bin array              | Metadata of last version
    98  
    99  Each header contains a mspg array (tuple) encoded object:
   100  
   101  xlHeaderVersion version == 1:
   102  
   103  ```
   104  //msgp:tuple xlMetaV2VersionHeader
   105  type xlMetaV2VersionHeader struct {
   106   VersionID [16]byte  // Version UUID, raw.
   107   ModTime   int64     // Unix nanoseconds.
   108   Signature [4]byte   // Signature of metadata.
   109   Type      uint8     // Type if the version
   110   Flags     uint8
   111  }
   112  ```
   113  
   114  The following flags are defined:
   115  
   116  ```
   117  const (
   118   FreeVersion = 1 << 0
   119   UsesDataDir = 1 << 1
   120   InlineData  = 1 << 2
   121  )
   122  ```
   123  
   124  The "Metadata" section contains a single version, encoded in similar fashion as each version in the `Versions` array
   125  of the previous version.
   126  
   127  ## Inline Data
   128  
   129  Inline data is optional. If no inline data is present, it is encoded as 0 bytes.
   130  
   131  | Entry               | Encoding                    | Content
   132  | --------------------|-----------------------------|----------------------------------------
   133  | xlMetaInlineDataVer | byte                        | version identifier
   134  | id -> data          | msgp `map[string][]byte`      | Map of string id -> byte content
   135  
   136  Currently only xlMetaInlineDataVer == 1 exists.
   137  
   138  The ID is the string encoded Version ID of which the data corresponds.