github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/docs/content/cache.md (about) 1 --- 2 title: "Cache" 3 description: "Rclone docs for cache remote" 4 date: "2017-09-03" 5 --- 6 7 <i class="fa fa-archive"></i> Cache (BETA) 8 ----------------------------------------- 9 10 The `cache` remote wraps another existing remote and stores file structure 11 and its data for long running tasks like `rclone mount`. 12 13 To get started you just need to have an existing remote which can be configured 14 with `cache`. 15 16 Here is an example of how to make a remote called `test-cache`. First run: 17 18 rclone config 19 20 This will guide you through an interactive setup process: 21 22 ``` 23 No remotes found - make a new one 24 n) New remote 25 r) Rename remote 26 c) Copy remote 27 s) Set configuration password 28 q) Quit config 29 n/r/c/s/q> n 30 name> test-cache 31 Type of storage to configure. 32 Choose a number from below, or type in your own value 33 ... 34 5 / Cache a remote 35 \ "cache" 36 ... 37 Storage> 5 38 Remote to cache. 39 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 40 "myremote:bucket" or maybe "myremote:" (not recommended). 41 remote> local:/test 42 Optional: The URL of the Plex server 43 plex_url> http://127.0.0.1:32400 44 Optional: The username of the Plex user 45 plex_username> dummyusername 46 Optional: The password of the Plex user 47 y) Yes type in my own password 48 g) Generate random password 49 n) No leave this optional password blank 50 y/g/n> y 51 Enter the password: 52 password: 53 Confirm the password: 54 password: 55 The size of a chunk. Lower value good for slow connections but can affect seamless reading. 56 Default: 5M 57 Choose a number from below, or type in your own value 58 1 / 1MB 59 \ "1m" 60 2 / 5 MB 61 \ "5M" 62 3 / 10 MB 63 \ "10M" 64 chunk_size> 2 65 How much time should object info (file size, file hashes etc) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. 66 Accepted units are: "s", "m", "h". 67 Default: 5m 68 Choose a number from below, or type in your own value 69 1 / 1 hour 70 \ "1h" 71 2 / 24 hours 72 \ "24h" 73 3 / 24 hours 74 \ "48h" 75 info_age> 2 76 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. 77 Default: 10G 78 Choose a number from below, or type in your own value 79 1 / 500 MB 80 \ "500M" 81 2 / 1 GB 82 \ "1G" 83 3 / 10 GB 84 \ "10G" 85 chunk_total_size> 3 86 Remote config 87 -------------------- 88 [test-cache] 89 remote = local:/test 90 plex_url = http://127.0.0.1:32400 91 plex_username = dummyusername 92 plex_password = *** ENCRYPTED *** 93 chunk_size = 5M 94 info_age = 48h 95 chunk_total_size = 10G 96 ``` 97 98 You can then use it like this, 99 100 List directories in top level of your drive 101 102 rclone lsd test-cache: 103 104 List all the files in your drive 105 106 rclone ls test-cache: 107 108 To start a cached mount 109 110 rclone mount --allow-other test-cache: /var/tmp/test-cache 111 112 ### Write Features ### 113 114 ### Offline uploading ### 115 116 In an effort to make writing through cache more reliable, the backend 117 now supports this feature which can be activated by specifying a 118 `cache-tmp-upload-path`. 119 120 A files goes through these states when using this feature: 121 122 1. An upload is started (usually by copying a file on the cache remote) 123 2. When the copy to the temporary location is complete the file is part 124 of the cached remote and looks and behaves like any other file (reading included) 125 3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move` 126 is used to move the file to the cloud provider 127 4. Reading the file still works during the upload but most modifications on it will be prohibited 128 5. Once the move is complete the file is unlocked for modifications as it 129 becomes as any other regular file 130 6. If the file is being read through `cache` when it's actually 131 deleted from the temporary path then `cache` will simply swap the source 132 to the cloud provider without interrupting the reading (small blip can happen though) 133 134 Files are uploaded in sequence and only one file is uploaded at a time. 135 Uploads will be stored in a queue and be processed based on the order they were added. 136 The queue and the temporary storage is persistent across restarts but 137 can be cleared on startup with the `--cache-db-purge` flag. 138 139 ### Write Support ### 140 141 Writes are supported through `cache`. 142 One caveat is that a mounted cache remote does not add any retry or fallback 143 mechanism to the upload operation. This will depend on the implementation 144 of the wrapped remote. Consider using `Offline uploading` for reliable writes. 145 146 One special case is covered with `cache-writes` which will cache the file 147 data at the same time as the upload when it is enabled making it available 148 from the cache store immediately once the upload is finished. 149 150 ### Read Features ### 151 152 #### Multiple connections #### 153 154 To counter the high latency between a local PC where rclone is running 155 and cloud providers, the cache remote can split multiple requests to the 156 cloud provider for smaller file chunks and combines them together locally 157 where they can be available almost immediately before the reader usually 158 needs them. 159 160 This is similar to buffering when media files are played online. Rclone 161 will stay around the current marker but always try its best to stay ahead 162 and prepare the data before. 163 164 #### Plex Integration #### 165 166 There is a direct integration with Plex which allows cache to detect during reading 167 if the file is in playback or not. This helps cache to adapt how it queries 168 the cloud provider depending on what is needed for. 169 170 Scans will have a minimum amount of workers (1) while in a confirmed playback cache 171 will deploy the configured number of workers. 172 173 This integration opens the doorway to additional performance improvements 174 which will be explored in the near future. 175 176 **Note:** If Plex options are not configured, `cache` will function with its 177 configured options without adapting any of its settings. 178 179 How to enable? Run `rclone config` and add all the Plex options (endpoint, username 180 and password) in your remote and it will be automatically enabled. 181 182 Affected settings: 183 - `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times 184 185 ##### Certificate Validation ##### 186 187 When the Plex server is configured to only accept secure connections, it is 188 possible to use `.plex.direct` URL's to ensure certificate validation succeeds. 189 These URL's are used by Plex internally to connect to the Plex server securely. 190 191 The format for this URL's is the following: 192 193 https://ip-with-dots-replaced.server-hash.plex.direct:32400/ 194 195 The `ip-with-dots-replaced` part can be any IPv4 address, where the dots 196 have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`. 197 198 To get the `server-hash` part, the easiest way is to visit 199 200 https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token 201 202 This page will list all the available Plex servers for your account 203 with at least one `.plex.direct` link for each. Copy one URL and replace 204 the IP address with the desired address. This can be used as the 205 `plex_url` value. 206 207 ### Known issues ### 208 209 #### Mount and --dir-cache-time #### 210 211 --dir-cache-time controls the first layer of directory caching which works at the mount layer. 212 Being an independent caching mechanism from the `cache` backend, it will manage its own entries 213 based on the configured time. 214 215 To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct 216 one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are 217 already configured in this way. 218 219 #### Windows support - Experimental #### 220 221 There are a couple of issues with Windows `mount` functionality that still require some investigations. 222 It should be considered as experimental thus far as fixes come in for this OS. 223 224 Most of the issues seem to be related to the difference between filesystems 225 on Linux flavors and Windows as cache is heavily dependant on them. 226 227 Any reports or feedback on how cache behaves on this OS is greatly appreciated. 228 229 - https://github.com/ncw/rclone/issues/1935 230 - https://github.com/ncw/rclone/issues/1907 231 - https://github.com/ncw/rclone/issues/1834 232 233 #### Risk of throttling #### 234 235 Future iterations of the cache backend will make use of the pooling functionality 236 of the cloud provider to synchronize and at the same time make writing through it 237 more tolerant to failures. 238 239 There are a couple of enhancements in track to add these but in the meantime 240 there is a valid concern that the expiring cache listings can lead to cloud provider 241 throttles or bans due to repeated queries on it for very large mounts. 242 243 Some recommendations: 244 - don't use a very small interval for entry informations (`--cache-info-age`) 245 - while writes aren't yet optimised, you can still write through `cache` which gives you the advantage 246 of adding the file in the cache at the same time if configured to do so. 247 248 Future enhancements: 249 250 - https://github.com/ncw/rclone/issues/1937 251 - https://github.com/ncw/rclone/issues/1936 252 253 #### cache and crypt #### 254 255 One common scenario is to keep your data encrypted in the cloud provider 256 using the `crypt` remote. `crypt` uses a similar technique to wrap around 257 an existing remote and handles this translation in a seamless way. 258 259 There is an issue with wrapping the remotes in this order: 260 <span style="color:red">**cloud remote** -> **crypt** -> **cache**</span> 261 262 During testing, I experienced a lot of bans with the remotes in this order. 263 I suspect it might be related to how crypt opens files on the cloud provider 264 which makes it think we're downloading the full file instead of small chunks. 265 Organizing the remotes in this order yields better results: 266 <span style="color:green">**cloud remote** -> **cache** -> **crypt**</span> 267 268 #### absolute remote paths #### 269 270 `cache` can not differentiate between relative and absolute paths for the wrapped remote. 271 Any path given in the `remote` config setting and on the command line will be passed to 272 the wrapped remote as is, but for storing the chunks on disk the path will be made 273 relative by removing any leading `/` character. 274 275 This behavior is irrelevant for most backend types, but there are backends where a leading `/` 276 changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are 277 relative to the root of the SSH server and paths without are relative to the user home directory. 278 As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent 279 a different directory on the SSH server. 280 281 ### Cache and Remote Control (--rc) ### 282 Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points: 283 By default, the listener is disabled if you do not add the flag. 284 285 ### rc cache/expire 286 Purge a remote from the cache backend. Supports either a directory or a file. 287 It supports both encrypted and unencrypted file names if cache is wrapped by crypt. 288 289 Params: 290 - **remote** = path to remote **(required)** 291 - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_ 292 293 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/cache/cache.go then run make backenddocs --> 294 ### Standard Options 295 296 Here are the standard options specific to cache (Cache a remote). 297 298 #### --cache-remote 299 300 Remote to cache. 301 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 302 "myremote:bucket" or maybe "myremote:" (not recommended). 303 304 - Config: remote 305 - Env Var: RCLONE_CACHE_REMOTE 306 - Type: string 307 - Default: "" 308 309 #### --cache-plex-url 310 311 The URL of the Plex server 312 313 - Config: plex_url 314 - Env Var: RCLONE_CACHE_PLEX_URL 315 - Type: string 316 - Default: "" 317 318 #### --cache-plex-username 319 320 The username of the Plex user 321 322 - Config: plex_username 323 - Env Var: RCLONE_CACHE_PLEX_USERNAME 324 - Type: string 325 - Default: "" 326 327 #### --cache-plex-password 328 329 The password of the Plex user 330 331 - Config: plex_password 332 - Env Var: RCLONE_CACHE_PLEX_PASSWORD 333 - Type: string 334 - Default: "" 335 336 #### --cache-chunk-size 337 338 The size of a chunk (partial file data). 339 340 Use lower numbers for slower connections. If the chunk size is 341 changed, any downloaded chunks will be invalid and cache-chunk-path 342 will need to be cleared or unexpected EOF errors will occur. 343 344 - Config: chunk_size 345 - Env Var: RCLONE_CACHE_CHUNK_SIZE 346 - Type: SizeSuffix 347 - Default: 5M 348 - Examples: 349 - "1m" 350 - 1MB 351 - "5M" 352 - 5 MB 353 - "10M" 354 - 10 MB 355 356 #### --cache-info-age 357 358 How long to cache file structure information (directory listings, file size, times etc). 359 If all write operations are done through the cache then you can safely make 360 this value very large as the cache store will also be updated in real time. 361 362 - Config: info_age 363 - Env Var: RCLONE_CACHE_INFO_AGE 364 - Type: Duration 365 - Default: 6h0m0s 366 - Examples: 367 - "1h" 368 - 1 hour 369 - "24h" 370 - 24 hours 371 - "48h" 372 - 48 hours 373 374 #### --cache-chunk-total-size 375 376 The total size that the chunks can take up on the local disk. 377 378 If the cache exceeds this value then it will start to delete the 379 oldest chunks until it goes under this value. 380 381 - Config: chunk_total_size 382 - Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE 383 - Type: SizeSuffix 384 - Default: 10G 385 - Examples: 386 - "500M" 387 - 500 MB 388 - "1G" 389 - 1 GB 390 - "10G" 391 - 10 GB 392 393 ### Advanced Options 394 395 Here are the advanced options specific to cache (Cache a remote). 396 397 #### --cache-plex-token 398 399 The plex token for authentication - auto set normally 400 401 - Config: plex_token 402 - Env Var: RCLONE_CACHE_PLEX_TOKEN 403 - Type: string 404 - Default: "" 405 406 #### --cache-plex-insecure 407 408 Skip all certificate verifications when connecting to the Plex server 409 410 - Config: plex_insecure 411 - Env Var: RCLONE_CACHE_PLEX_INSECURE 412 - Type: string 413 - Default: "" 414 415 #### --cache-db-path 416 417 Directory to store file structure metadata DB. 418 The remote name is used as the DB file name. 419 420 - Config: db_path 421 - Env Var: RCLONE_CACHE_DB_PATH 422 - Type: string 423 - Default: "$HOME/.cache/rclone/cache-backend" 424 425 #### --cache-chunk-path 426 427 Directory to cache chunk files. 428 429 Path to where partial file data (chunks) are stored locally. The remote 430 name is appended to the final path. 431 432 This config follows the "--cache-db-path". If you specify a custom 433 location for "--cache-db-path" and don't specify one for "--cache-chunk-path" 434 then "--cache-chunk-path" will use the same path as "--cache-db-path". 435 436 - Config: chunk_path 437 - Env Var: RCLONE_CACHE_CHUNK_PATH 438 - Type: string 439 - Default: "$HOME/.cache/rclone/cache-backend" 440 441 #### --cache-db-purge 442 443 Clear all the cached data for this remote on start. 444 445 - Config: db_purge 446 - Env Var: RCLONE_CACHE_DB_PURGE 447 - Type: bool 448 - Default: false 449 450 #### --cache-chunk-clean-interval 451 452 How often should the cache perform cleanups of the chunk storage. 453 The default value should be ok for most people. If you find that the 454 cache goes over "cache-chunk-total-size" too often then try to lower 455 this value to force it to perform cleanups more often. 456 457 - Config: chunk_clean_interval 458 - Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL 459 - Type: Duration 460 - Default: 1m0s 461 462 #### --cache-read-retries 463 464 How many times to retry a read from a cache storage. 465 466 Since reading from a cache stream is independent from downloading file 467 data, readers can get to a point where there's no more data in the 468 cache. Most of the times this can indicate a connectivity issue if 469 cache isn't able to provide file data anymore. 470 471 For really slow connections, increase this to a point where the stream is 472 able to provide data but your experience will be very stuttering. 473 474 - Config: read_retries 475 - Env Var: RCLONE_CACHE_READ_RETRIES 476 - Type: int 477 - Default: 10 478 479 #### --cache-workers 480 481 How many workers should run in parallel to download chunks. 482 483 Higher values will mean more parallel processing (better CPU needed) 484 and more concurrent requests on the cloud provider. This impacts 485 several aspects like the cloud provider API limits, more stress on the 486 hardware that rclone runs on but it also means that streams will be 487 more fluid and data will be available much more faster to readers. 488 489 **Note**: If the optional Plex integration is enabled then this 490 setting will adapt to the type of reading performed and the value 491 specified here will be used as a maximum number of workers to use. 492 493 - Config: workers 494 - Env Var: RCLONE_CACHE_WORKERS 495 - Type: int 496 - Default: 4 497 498 #### --cache-chunk-no-memory 499 500 Disable the in-memory cache for storing chunks during streaming. 501 502 By default, cache will keep file data during streaming in RAM as well 503 to provide it to readers as fast as possible. 504 505 This transient data is evicted as soon as it is read and the number of 506 chunks stored doesn't exceed the number of workers. However, depending 507 on other settings like "cache-chunk-size" and "cache-workers" this footprint 508 can increase if there are parallel streams too (multiple files being read 509 at the same time). 510 511 If the hardware permits it, use this feature to provide an overall better 512 performance during streaming but it can also be disabled if RAM is not 513 available on the local machine. 514 515 - Config: chunk_no_memory 516 - Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY 517 - Type: bool 518 - Default: false 519 520 #### --cache-rps 521 522 Limits the number of requests per second to the source FS (-1 to disable) 523 524 This setting places a hard limit on the number of requests per second 525 that cache will be doing to the cloud provider remote and try to 526 respect that value by setting waits between reads. 527 528 If you find that you're getting banned or limited on the cloud 529 provider through cache and know that a smaller number of requests per 530 second will allow you to work with it then you can use this setting 531 for that. 532 533 A good balance of all the other settings should make this setting 534 useless but it is available to set for more special cases. 535 536 **NOTE**: This will limit the number of requests during streams but 537 other API calls to the cloud provider like directory listings will 538 still pass. 539 540 - Config: rps 541 - Env Var: RCLONE_CACHE_RPS 542 - Type: int 543 - Default: -1 544 545 #### --cache-writes 546 547 Cache file data on writes through the FS 548 549 If you need to read files immediately after you upload them through 550 cache you can enable this flag to have their data stored in the 551 cache store at the same time during upload. 552 553 - Config: writes 554 - Env Var: RCLONE_CACHE_WRITES 555 - Type: bool 556 - Default: false 557 558 #### --cache-tmp-upload-path 559 560 Directory to keep temporary files until they are uploaded. 561 562 This is the path where cache will use as a temporary storage for new 563 files that need to be uploaded to the cloud provider. 564 565 Specifying a value will enable this feature. Without it, it is 566 completely disabled and files will be uploaded directly to the cloud 567 provider 568 569 - Config: tmp_upload_path 570 - Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH 571 - Type: string 572 - Default: "" 573 574 #### --cache-tmp-wait-time 575 576 How long should files be stored in local cache before being uploaded 577 578 This is the duration that a file must wait in the temporary location 579 _cache-tmp-upload-path_ before it is selected for upload. 580 581 Note that only one file is uploaded at a time and it can take longer 582 to start the upload if a queue formed for this purpose. 583 584 - Config: tmp_wait_time 585 - Env Var: RCLONE_CACHE_TMP_WAIT_TIME 586 - Type: Duration 587 - Default: 15s 588 589 #### --cache-db-wait-time 590 591 How long to wait for the DB to be available - 0 is unlimited 592 593 Only one process can have the DB open at any one time, so rclone waits 594 for this duration for the DB to become available before it gives an 595 error. 596 597 If you set it to 0 then it will wait forever. 598 599 - Config: db_wait_time 600 - Env Var: RCLONE_CACHE_DB_WAIT_TIME 601 - Type: Duration 602 - Default: 1s 603 604 <!--- autogenerated options stop -->