github.com/git-lfs/git-lfs@v2.5.2+incompatible/docs/proposals/transfer_adapters.md (about)

     1  # Transfer adapters for resumable upload / download
     2  
     3  ## Concept
     4  
     5  To allow the uploading and downloading of LFS content to be implemented in more
     6  ways than the current simple HTTP GET/PUT approach. Features that could be
     7  supported by opening this up to other protocols might include:
     8  
     9    - Resumable transfers
    10    - Block-level de-duplication
    11    - Delegation to 3rd party services like Dropbox / Google Drive / OneDrive
    12    - Non-HTTP services
    13  
    14  ## API extensions
    15  
    16  See the [API documentation](../http-v1-batch.md) for specifics. All changes
    17  are optional extras so there are no breaking changes to the API.
    18  
    19  The current HTTP GET/PUT system will remain the default. When a version of the
    20  git-lfs client supports alternative transfer mechanisms, it notifies the server
    21  in the API request using the `accept-transfers` field. 
    22  
    23  If the server also supports one of the mechanisms the client advertised, it may 
    24  select one and alter the upload / download URLs to point at resources 
    25  compatible with this transfer mechanism. It must also indicate the chosen 
    26  transfer mechanism in the response using the `transfer` field. 
    27  
    28  The URLs provided in this case may not be HTTP, they may be custom protocols.
    29  It is up to each individual transfer mechanism to define how URLs are used.
    30  
    31  ## Client extensions
    32  
    33  ### Phase 1: refactoring & abstraction
    34  
    35  1. Introduce a new concept of 'transfer adapter'. 
    36  2. Adapters can provide either upload or download support, or both. This is 
    37     necessary because some mechanisms are unidirectional, e.g. HTTP Content-Range
    38     is download only, tus.io is upload only.
    39  3. Refactor our current HTTP GET/PUT mechanism to be the default implementation 
    40     for both upload & download
    41  4. The LFS core will pass oids to transfer to this adapter in bulk, and receive 
    42     events back from the adapter for transfer progress, and file completion.
    43  5. Each adapter is responsible for its own parallelism, but should respect the
    44     `lfs.concurrenttransfers` setting. For example the default (current) approach
    45     will parallelise on files (oids), but others may parallelise in other ways
    46     e.g. downloading multiple parts of the same file at once
    47  6. Each adapter should store its own temporary files. On file completion it must
    48     notify the core which in the case of a download is then responsible for 
    49     moving a completed file into permanent storage.
    50  7. Update the core to have a registry of available transfer mechanisms which it
    51     passes to the API, and can recognise a chosen one in the response. Default
    52     to our refactored original.
    53  
    54  ### Phase 2: basic resumable downloads
    55  
    56  1. Add a client transfer adapter for [HTTP Range headers](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35)
    57  2. Add a range request reference implementation to our integration test server
    58  
    59  ### Phase 3: basic resumable uploads
    60  
    61  1. Add a client transfer adapter for [tus.io](http://tus.io) (upload only)
    62  2. Add a tus.io reference implementation to our integration test server
    63  
    64  ### Phase 4: external transfer adapters
    65  
    66  Ideally we should allow people to add other transfer implementations so that
    67  we don't have to implement everything, or bloat the git-lfs binary with every
    68  custom system possible.
    69  
    70  Because Go is statically linked it's not possible to extend client functionality
    71  at runtime through loading libaries, so instead I propose allowing an external
    72  process to be invoked, and communicated with via a defined stream protocol. This
    73  protocol will be logically identical to the internal adapters; the core passing
    74  oids and receiving back progress and completion notifications; just that the 
    75  implementation will be in an external process and the messages will be 
    76  serialised over streams.
    77  
    78  Only one process will be launched and will remain for the entire period of all
    79  transfers. Like internal adapters, the external process will be responsible for
    80  its own parallelism and temporary storage, so internally they can (should) do
    81  multiple transfers at once.
    82  
    83  1. Build a generic 'external' adapter which can invoke a named process and
    84     communicate with it using the standard stream protocol (probably just over
    85     stdout / stdin)
    86  2. Establish a configuration for external adapters; minimum is an identifier 
    87     (client and server must agree on what that is) and a path to invoke
    88  3. Implement a small test process in Go which simply wraps the default HTTP
    89     mechanism in an external process, to prove the approach (not in release)
    90  
    91