github.com/saracen/git-lfs@v2.5.2+incompatible/docs/proposals/transfer_adapters.md (about) 1 # Transfer adapters for resumable upload / download 2 3 ## Concept 4 5 To allow the uploading and downloading of LFS content to be implemented in more 6 ways than the current simple HTTP GET/PUT approach. Features that could be 7 supported by opening this up to other protocols might include: 8 9 - Resumable transfers 10 - Block-level de-duplication 11 - Delegation to 3rd party services like Dropbox / Google Drive / OneDrive 12 - Non-HTTP services 13 14 ## API extensions 15 16 See the [API documentation](../http-v1-batch.md) for specifics. All changes 17 are optional extras so there are no breaking changes to the API. 18 19 The current HTTP GET/PUT system will remain the default. When a version of the 20 git-lfs client supports alternative transfer mechanisms, it notifies the server 21 in the API request using the `accept-transfers` field. 22 23 If the server also supports one of the mechanisms the client advertised, it may 24 select one and alter the upload / download URLs to point at resources 25 compatible with this transfer mechanism. It must also indicate the chosen 26 transfer mechanism in the response using the `transfer` field. 27 28 The URLs provided in this case may not be HTTP, they may be custom protocols. 29 It is up to each individual transfer mechanism to define how URLs are used. 30 31 ## Client extensions 32 33 ### Phase 1: refactoring & abstraction 34 35 1. Introduce a new concept of 'transfer adapter'. 36 2. Adapters can provide either upload or download support, or both. This is 37 necessary because some mechanisms are unidirectional, e.g. HTTP Content-Range 38 is download only, tus.io is upload only. 39 3. Refactor our current HTTP GET/PUT mechanism to be the default implementation 40 for both upload & download 41 4. The LFS core will pass oids to transfer to this adapter in bulk, and receive 42 events back from the adapter for transfer progress, and file completion. 43 5. Each adapter is responsible for its own parallelism, but should respect the 44 `lfs.concurrenttransfers` setting. For example the default (current) approach 45 will parallelise on files (oids), but others may parallelise in other ways 46 e.g. downloading multiple parts of the same file at once 47 6. Each adapter should store its own temporary files. On file completion it must 48 notify the core which in the case of a download is then responsible for 49 moving a completed file into permanent storage. 50 7. Update the core to have a registry of available transfer mechanisms which it 51 passes to the API, and can recognise a chosen one in the response. Default 52 to our refactored original. 53 54 ### Phase 2: basic resumable downloads 55 56 1. Add a client transfer adapter for [HTTP Range headers](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35) 57 2. Add a range request reference implementation to our integration test server 58 59 ### Phase 3: basic resumable uploads 60 61 1. Add a client transfer adapter for [tus.io](http://tus.io) (upload only) 62 2. Add a tus.io reference implementation to our integration test server 63 64 ### Phase 4: external transfer adapters 65 66 Ideally we should allow people to add other transfer implementations so that 67 we don't have to implement everything, or bloat the git-lfs binary with every 68 custom system possible. 69 70 Because Go is statically linked it's not possible to extend client functionality 71 at runtime through loading libaries, so instead I propose allowing an external 72 process to be invoked, and communicated with via a defined stream protocol. This 73 protocol will be logically identical to the internal adapters; the core passing 74 oids and receiving back progress and completion notifications; just that the 75 implementation will be in an external process and the messages will be 76 serialised over streams. 77 78 Only one process will be launched and will remain for the entire period of all 79 transfers. Like internal adapters, the external process will be responsible for 80 its own parallelism and temporary storage, so internally they can (should) do 81 multiple transfers at once. 82 83 1. Build a generic 'external' adapter which can invoke a named process and 84 communicate with it using the standard stream protocol (probably just over 85 stdout / stdin) 86 2. Establish a configuration for external adapters; minimum is an identifier 87 (client and server must agree on what that is) and a path to invoke 88 3. Implement a small test process in Go which simply wraps the default HTTP 89 mechanism in an external process, to prove the approach (not in release) 90 91