github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/design/accepted/openapi-presign-multipart-upload.md (about)

     1  # OpenAPI for S3 Like Multipart Upload Proposal using Presigned URLs
     2  
     3  ## Introduction
     4  
     5  This proposal outlines the development of Open API endpoints to enable multipart upload capabilities similar to Amazon S3 multipart using presigned URLs to upload parts. The intention is to enhance the data uploading process for large files, ensuring efficient and reliable data transfer.
     6  
     7  ## Goal
     8  
     9  - Implementing multi-part upload APIs with support for parallel upload optimizes data transfer for large files over varying network conditions using lakectl.
    10  
    11  ## Non-goals
    12  
    13  - Support all underlying storage types as we can experiment with the one that currently support this feature.
    14  - SDK to wrap the new API and support this capability.
    15  
    16  
    17  ## Features and Functionalities
    18  
    19  To minimize the scope of this feature and understand the benefits before supporting all blockstores, this feature will be scoped to S3 blockstore and only if presigned capability is enabled.
    20  
    21  Initiation of Upload: The API will allow clients to initiate a multipart upload session, assigning a unique upload ID for subsequent operations. The client will require passing the number of parts it requires and the call will provide a set of URLs to upload each part.
    22  
    23  Uploading Parts: Clients can upload individual parts of the file in parallel or in sequence. Each part will be based on the presigned URL provided by the initial call.
    24  
    25  Support for Large Files: The API will handle files of substantial sizes, ensuring that large datasets can be uploaded without issues. Minimum part size will be 5M, except for the last block.
    26  
    27  Data Integrity Checks: Each uploaded part will require the client to store the ETag as done today for presigned URL upload. The value of the ETag will be provided when the upload of all the parts is completed.
    28  
    29  Pause and Resume: Clients can pause the upload and resume later, leveraging the upload ID to reinitiate the process. (TDB need to specify how long we will keep and if cleanup will be done)
    30  
    31  Completion of Upload: Once all parts are uploaded, the client will send a request to complete the upload, address the file in lakeFS as a single file. Upload compete request will include the ETag for check part, in case of mismatch or fail to provide bad request code will return.
    32  
    33  Cancelation: In order to cancel partial upload of multipart request, the client will require to call cancellation with the upload ID provided.
    34  
    35  ### Multipart upload capability using OpenAPI specification
    36  
    37  **Paths for presign multipart upload operations**
    38  
    39  ```yaml
    40    /repositories/{repository}/branches/{branch}/staging/pmpu:
    41      parameters:
    42        - in: path
    43          name: repository
    44          required: true
    45          schema:
    46            type: string
    47        - in: path
    48          name: branch
    49          required: true
    50          schema:
    51            type: string
    52        - in: query
    53          name: path
    54          description: relative to the branch
    55          required: true
    56          schema:
    57            type: string
    58        - in: query
    59          name: presigned_parts
    60          description: number of presigned URL parts required to upload
    61          schema:
    62            type: integer
    63      post:
    64        tags:
    65          - experimental
    66        operationId: createPresignMultipartUpload
    67        summary: Initiate a presign multipart upload
    68        description: Initiates a presign multipart upload and returns an upload ID with presigned URLs for each part. Part numbers starts with 1. Each part except the last one has minimum size depends on the underlying blockstore implementation. For example working with S3 blockstore, minimum size is 5MB (excluding the last part).
    69        responses:
    70          200:
    71            description: Presign multipart upload initiated
    72            content:
    73              application/json:
    74                schema:
    75                  $ref: "#/components/schemas/PresignMultipartUpload"
    76          # ... other error responses
    77  
    78    /repositories/{repository}/branches/{branch}/staging/pmpu/{uploadId}:
    79        parameters:
    80        - in: path
    81          name: repository
    82          required: true
    83          schema:
    84            type: string
    85        - in: path
    86          name: branch
    87          required: true
    88          schema:
    89            type: string
    90        - in: path
    91          name: uploadId
    92          required: true
    93          schema:
    94            type: string
    95        - in: query
    96          name: path
    97          description: relative to the branch
    98          required: true
    99          schema:
   100            type: string
   101      put:
   102        tags:
   103          - experimental
   104        operationId: completePresignMultipartUpload
   105        summary: Complete a presign multipart upload
   106        description: Completes a presign multipart upload by assembling the uploaded parts.
   107        requestBody:
   108          content:
   109            application/json:
   110              schema:
   111                $ref: "#/components/schemas/CompleteMultipartUpload"
   112        responses:
   113          200:
   114            description: Presign multipart upload completed
   115            content:
   116              application/json:
   117                schema:
   118                  $ref: "#/components/schemas/ObjectStats"
   119          # ... other error responses
   120  
   121      delete:
   122        tags:
   123          - experimental
   124        operationId: abortPresignMultipartUpload
   125        summary: Abort a presign multipart upload
   126        description: Aborts a presign multipart upload.
   127        responses:
   128          204:
   129            description: Presign multipart upload aborted
   130  
   131  ```
   132  
   133  
   134  **Schemas used by operations**
   135  
   136  ```yaml
   137  components:
   138    schemas:
   139      PresignMultipartUpload:
   140        type: object
   141        properties:
   142          upload_id:
   143            type: string
   144          physical_address:
   145            type: string
   146          presigned_urls:
   147            type: array
   148            items:
   149              type: string
   150        required:
   151          - upload_id
   152          - physical_address
   153          
   154      UploadPart:
   155        type: object
   156        properties:
   157          part_number:
   158            type: integer
   159          etag:
   160            type: string
   161        required:
   162          - part_number
   163          - etag
   164          
   165      CompletePresignMultipartUpload:
   166        type: object
   167        properties:
   168          physical_address:
   169            type: string
   170          parts:
   171            type: array
   172            description: "List of uploaded parts, should be ordered by ascending part number"
   173            items:
   174              $ref: "#/components/schemas/UploadPart"
   175          user_metadata:
   176            type: object
   177            additionalProperties:
   178              type: string
   179          content_type:
   180            type: string
   181            description: Object media type
   182        required:
   183          - physical_address
   184          - parts
   185  ```
   186  
   187  ### Support and discover
   188  
   189  Presign support is a capability lakectl discover before switching to use presign for upload or download from lakeFS.
   190  The presign multipart upload support will be part of the storage capability add `pre_sign_multipart_upload` optional field that when set to `true` the user can perform multipart upload using the new API.
   191  lakeFS will return presign multipart support only on S3 with presign support enabled. There will be a configurable parameter `disable_pre_signed_multipart` under s3 block adapter that can be used to disable the capability if needed.
   192  
   193  ```yaml
   194      StorageConfig:
   195        type: object
   196        required:
   197          - blockstore_type
   198          - blockstore_namespace_example
   199          - blockstore_namespace_ValidityRegex
   200          - pre_sign_support
   201          - pre_sign_support_ui
   202          - import_support
   203          - import_validity_regex
   204        properties:
   205          blockstore_type:
   206            type: string
   207          blockstore_namespace_example:
   208            type: string
   209          blockstore_namespace_ValidityRegex:
   210            type: string
   211          default_namespace_prefix:
   212            type: string
   213          pre_sign_support:
   214            type: boolean
   215          pre_sign_support_ui:
   216            type: boolean
   217          import_support:
   218            type: boolean
   219          import_validity_regex:
   220            type: string
   221          pre_sign_multipart_upload:
   222            type: boolean
   223  ```
   224  
   225  ## Limitations
   226  
   227  - **S3 block adapter exclusive:** Uploading files in multiple parts (multipart upload) is only available when using the S3 block adapter. This feature isn't currently supported with other storage options. This exclude the multipart upload support we provide by our S3 gateway where we provide implementation above each storage.
   228  - **Part size and count restrictions:** There are limits on how you can split your file for upload:
   229      - **Maximum parts:** You can split your file into a maximum of 10000 parts.
   230      - **Minimum part size:** Each part must be at least 5MB in size. This is a temporary constraint and isn't currently configurable or discoverable. It will become an option when additional storage options are supported.
   231  - **Initiating the upload:** When starting a multipart upload, you'll need to specify the total number of parts in your request. It will reduce the requests for each part's presigned URL when the client already knows the size.
   232  - **Presigning part:** Request for a presigned URL of a specific part number will not be supported. This will block unknown size upload using this API.
   233  - **Limited support:** Request for list upload parts will not be provided at this point.
   234  
   235  ## Next steps
   236  
   237  None of the returned URLs has to be used, it is fine to ask for more than are needed.
   238  In future we may add an _additional_ API call to URLs for uploading more parts.
   239  This will allow more "streaming" uses, for instance as parallels to how Hadoop S3A uses the S3 MPU API and how the AWS SDKs upload manager handle streaming.