github.com/xhghs/rclone@v1.51.1-0.20200430155106-e186a28cced8/MANUAL.md (about) 1 % rclone(1) User Manual 2 % Nick Craig-Wood 3 % Feb 01, 2020 4 5 # Rclone - rsync for cloud storage 6 7 Rclone is a command line program to sync files and directories to and from: 8 9 * 1Fichier 10 * Alibaba Cloud (Aliyun) Object Storage System (OSS) 11 * Amazon Drive ([See note](/amazonclouddrive/#status)) 12 * Amazon S3 13 * Backblaze B2 14 * Box 15 * Ceph 16 * Citrix ShareFile 17 * C14 18 * DigitalOcean Spaces 19 * Dreamhost 20 * Dropbox 21 * FTP 22 * Google Cloud Storage 23 * Google Drive 24 * Google Photos 25 * HTTP 26 * Hubic 27 * Jottacloud 28 * IBM COS S3 29 * Koofr 30 * Mail.ru Cloud 31 * Memset Memstore 32 * Mega 33 * Memory 34 * Microsoft Azure Blob Storage 35 * Microsoft OneDrive 36 * Minio 37 * Nextcloud 38 * OVH 39 * OpenDrive 40 * Openstack Swift 41 * Oracle Cloud Storage 42 * ownCloud 43 * pCloud 44 * premiumize.me 45 * put.io 46 * QingStor 47 * Rackspace Cloud Files 48 * rsync.net 49 * Scaleway 50 * SFTP 51 * SugarSync 52 * Wasabi 53 * WebDAV 54 * Yandex Disk 55 * The local filesystem 56 57 Features 58 59 * MD5/SHA1 hashes checked at all times for file integrity 60 * Timestamps preserved on files 61 * Partial syncs supported on a whole file basis 62 * [Copy](https://rclone.org/commands/rclone_copy/) mode to just copy new/changed files 63 * [Sync](https://rclone.org/commands/rclone_sync/) (one way) mode to make a directory identical 64 * [Check](https://rclone.org/commands/rclone_check/) mode to check for file hash equality 65 * Can sync to and from network, eg two different cloud accounts 66 * [Encryption](https://rclone.org/crypt/) backend 67 * [Cache](https://rclone.org/cache/) backend 68 * [Chunking](https://rclone.org/chunker/) backend 69 * [Union](https://rclone.org/union/) backend 70 * Optional FUSE mount ([rclone mount](https://rclone.org/commands/rclone_mount/)) 71 * Multi-threaded downloads to local disk 72 * Can [serve](https://rclone.org/commands/rclone_serve/) local or remote files over [HTTP](https://rclone.org/commands/rclone_serve_http/)/[WebDav](https://rclone.org/commands/rclone_serve_webdav/)/[FTP](https://rclone.org/commands/rclone_serve_ftp/)/[SFTP](https://rclone.org/commands/rclone_serve_sftp/)/[dlna](https://rclone.org/commands/rclone_serve_dlna/) 73 * Experimental [Web based GUI](https://rclone.org/gui/) 74 75 Links 76 77 * [Home page](https://rclone.org/) 78 * [GitHub project page for source and bug tracker](https://github.com/rclone/rclone) 79 * [Rclone Forum](https://forum.rclone.org) 80 * [Downloads](https://rclone.org/downloads/) 81 82 # Install # 83 84 Rclone is a Go program and comes as a single binary file. 85 86 ## Quickstart ## 87 88 * [Download](https://rclone.org/downloads/) the relevant binary. 89 * Extract the `rclone` or `rclone.exe` binary from the archive 90 * Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 91 92 See below for some expanded Linux / macOS instructions. 93 94 See the [Usage section](https://rclone.org/docs/) of the docs for how to use rclone, or 95 run `rclone -h`. 96 97 ## Script installation ## 98 99 To install rclone on Linux/macOS/BSD systems, run: 100 101 curl https://rclone.org/install.sh | sudo bash 102 103 For beta installation, run: 104 105 curl https://rclone.org/install.sh | sudo bash -s beta 106 107 Note that this script checks the version of rclone installed first and 108 won't re-download if not needed. 109 110 ## Linux installation from precompiled binary ## 111 112 Fetch and unpack 113 114 curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip 115 unzip rclone-current-linux-amd64.zip 116 cd rclone-*-linux-amd64 117 118 Copy binary file 119 120 sudo cp rclone /usr/bin/ 121 sudo chown root:root /usr/bin/rclone 122 sudo chmod 755 /usr/bin/rclone 123 124 Install manpage 125 126 sudo mkdir -p /usr/local/share/man/man1 127 sudo cp rclone.1 /usr/local/share/man/man1/ 128 sudo mandb 129 130 Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 131 132 rclone config 133 134 ## macOS installation with brew ## 135 136 brew install rclone 137 138 ## macOS installation from precompiled binary, using curl ## 139 140 To avoid problems with macOS gatekeeper enforcing the binary to be signed and 141 notarized it is enough to download with `curl`. 142 143 Download the latest version of rclone. 144 145 cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip 146 147 Unzip the download and cd to the extracted folder. 148 149 unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64 150 151 Move rclone to your $PATH. You will be prompted for your password. 152 153 sudo mkdir -p /usr/local/bin 154 sudo mv rclone /usr/local/bin/ 155 156 (the `mkdir` command is safe to run, even if the directory already exists). 157 158 Remove the leftover files. 159 160 cd .. && rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip 161 162 Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 163 164 rclone config 165 166 ## macOS installation from precompiled binary, using a web browser ## 167 168 When downloading a binary with a web browser, the browser will set the macOS 169 gatekeeper quarantine attribute. Starting from Catalina, when attempting to run 170 `rclone`, a pop-up will appear saying: 171 172 “rclone” cannot be opened because the developer cannot be verified. 173 macOS cannot verify that this app is free from malware. 174 175 The simplest fix is to run 176 177 xattr -d com.apple.quarantine rclone 178 179 ## Install with docker ## 180 181 The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone). 182 These images are autobuilt by docker hub from the rclone source based 183 on a minimal Alpine linux image. 184 185 The `:latest` tag will always point to the latest stable release. You 186 can use the `:beta` tag to get the latest build from master. You can 187 also use version tags, eg `:1.49.1`, `:1.49` or `:1`. 188 189 ``` 190 $ docker pull rclone/rclone:latest 191 latest: Pulling from rclone/rclone 192 Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11 193 ... 194 $ docker run --rm rclone/rclone:latest version 195 rclone v1.49.1 196 - os/arch: linux/amd64 197 - go version: go1.12.9 198 ``` 199 200 There are a few command line options to consider when starting an rclone Docker container 201 from the rclone image. 202 203 - You need to mount the host rclone config dir at `/config/rclone` into the Docker 204 container. Due to the fact that rclone updates tokens inside its config file, and that 205 the update process involves a file rename, you need to mount the whole host rclone 206 config dir, not just the single host rclone config file. 207 208 - You need to mount a host data dir at `/data` into the Docker container. 209 210 - By default, the rclone binary inside a Docker container runs with UID=0 (root). 211 As a result, all files created in a run will have UID=0. If your config and data files 212 reside on the host with a non-root UID:GID, you need to pass these on the container 213 start command line. 214 215 - It is possible to use `rclone mount` inside a userspace Docker container, and expose 216 the resulting fuse mount to the host. The exact `docker run` options to do that might 217 vary slightly between hosts. See, e.g. the discussion in this 218 [thread](https://github.com/moby/moby/issues/9448). 219 220 You also need to mount the host `/etc/passwd` and `/etc/group` for fuse to work inside 221 the container. 222 223 Here are some commands tested on an Ubuntu 18.04.3 host: 224 225 ``` 226 # config on host at ~/.config/rclone/rclone.conf 227 # data on host at ~/data 228 229 # make sure the config is ok by listing the remotes 230 docker run --rm \ 231 --volume ~/.config/rclone:/config/rclone \ 232 --volume ~/data:/data:shared \ 233 --user $(id -u):$(id -g) \ 234 rclone/rclone \ 235 listremotes 236 237 # perform mount inside Docker container, expose result to host 238 mkdir -p ~/data/mount 239 docker run --rm \ 240 --volume ~/.config/rclone:/config/rclone \ 241 --volume ~/data:/data:shared \ 242 --user $(id -u):$(id -g) \ 243 --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \ 244 --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \ 245 rclone/rclone \ 246 mount dropbox:Photos /data/mount & 247 ls ~/data/mount 248 kill %1 249 ``` 250 251 ## Install from source ## 252 253 Make sure you have at least [Go](https://golang.org/) 1.7 254 installed. [Download go](https://golang.org/dl/) if necessary. The 255 latest release is recommended. Then 256 257 git clone https://github.com/rclone/rclone.git 258 cd rclone 259 go build 260 ./rclone version 261 262 You can also build and install rclone in the 263 [GOPATH](https://github.com/golang/go/wiki/GOPATH) (which defaults to 264 `~/go`) with: 265 266 go get -u -v github.com/rclone/rclone 267 268 and this will build the binary in `$GOPATH/bin` (`~/go/bin/rclone` by 269 default) after downloading the source to 270 `$GOPATH/src/github.com/rclone/rclone` (`~/go/src/github.com/rclone/rclone` 271 by default). 272 273 ## Installation with Ansible ## 274 275 This can be done with [Stefan Weichinger's ansible 276 role](https://github.com/stefangweichinger/ansible-rclone). 277 278 Instructions 279 280 1. `git clone https://github.com/stefangweichinger/ansible-rclone.git` into your local roles-directory 281 2. add the role to the hosts you want rclone installed to: 282 283 ``` 284 - hosts: rclone-hosts 285 roles: 286 - rclone 287 ``` 288 289 Configure 290 --------- 291 292 First, you'll need to configure rclone. As the object storage systems 293 have quite complicated authentication these are kept in a config file. 294 (See the `--config` entry for how to find the config file and choose 295 its location.) 296 297 The easiest way to make the config is to run rclone with the config 298 option: 299 300 rclone config 301 302 See the following for detailed instructions for 303 304 * [1Fichier](https://rclone.org/fichier/) 305 * [Alias](https://rclone.org/alias/) 306 * [Amazon Drive](https://rclone.org/amazonclouddrive/) 307 * [Amazon S3](https://rclone.org/s3/) 308 * [Backblaze B2](https://rclone.org/b2/) 309 * [Box](https://rclone.org/box/) 310 * [Cache](https://rclone.org/cache/) 311 * [Chunker](https://rclone.org/chunker/) - transparently splits large files for other remotes 312 * [Citrix ShareFile](https://rclone.org/sharefile/) 313 * [Crypt](https://rclone.org/crypt/) - to encrypt other remotes 314 * [DigitalOcean Spaces](/s3/#digitalocean-spaces) 315 * [Dropbox](https://rclone.org/dropbox/) 316 * [FTP](https://rclone.org/ftp/) 317 * [Google Cloud Storage](https://rclone.org/googlecloudstorage/) 318 * [Google Drive](https://rclone.org/drive/) 319 * [Google Photos](https://rclone.org/googlephotos/) 320 * [HTTP](https://rclone.org/http/) 321 * [Hubic](https://rclone.org/hubic/) 322 * [Jottacloud / GetSky.no](https://rclone.org/jottacloud/) 323 * [Koofr](https://rclone.org/koofr/) 324 * [Mail.ru Cloud](https://rclone.org/mailru/) 325 * [Mega](https://rclone.org/mega/) 326 * [Memory](https://rclone.org/memory/) 327 * [Microsoft Azure Blob Storage](https://rclone.org/azureblob/) 328 * [Microsoft OneDrive](https://rclone.org/onedrive/) 329 * [Openstack Swift / Rackspace Cloudfiles / Memset Memstore](https://rclone.org/swift/) 330 * [OpenDrive](https://rclone.org/opendrive/) 331 * [Pcloud](https://rclone.org/pcloud/) 332 * [premiumize.me](https://rclone.org/premiumizeme/) 333 * [put.io](https://rclone.org/putio/) 334 * [QingStor](https://rclone.org/qingstor/) 335 * [SFTP](https://rclone.org/sftp/) 336 * [SugarSync](https://rclone.org/sugarsync/) 337 * [Union](https://rclone.org/union/) 338 * [WebDAV](https://rclone.org/webdav/) 339 * [Yandex Disk](https://rclone.org/yandex/) 340 * [The local filesystem](https://rclone.org/local/) 341 342 Usage 343 ----- 344 345 Rclone syncs a directory tree from one storage system to another. 346 347 Its syntax is like this 348 349 Syntax: [options] subcommand <parameters> <parameters...> 350 351 Source and destination paths are specified by the name you gave the 352 storage system in the config file then the sub path, eg 353 "drive:myfolder" to look at "myfolder" in Google drive. 354 355 You can define as many storage paths as you like in the config file. 356 357 Subcommands 358 ----------- 359 360 rclone uses a system of subcommands. For example 361 362 rclone ls remote:path # lists a remote 363 rclone copy /local/path remote:path # copies /local/path to the remote 364 rclone sync /local/path remote:path # syncs /local/path to the remote 365 366 ## rclone config 367 368 Enter an interactive configuration session. 369 370 ### Synopsis 371 372 Enter an interactive configuration session where you can setup new 373 remotes and manage existing ones. You may also set or remove a 374 password to protect your configuration. 375 376 377 ``` 378 rclone config [flags] 379 ``` 380 381 ### Options 382 383 ``` 384 -h, --help help for config 385 ``` 386 387 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 388 389 ### SEE ALSO 390 391 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 392 * [rclone config create](https://rclone.org/commands/rclone_config_create/) - Create a new remote with name, type and options. 393 * [rclone config delete](https://rclone.org/commands/rclone_config_delete/) - Delete an existing remote <name>. 394 * [rclone config disconnect](https://rclone.org/commands/rclone_config_disconnect/) - Disconnects user from remote 395 * [rclone config dump](https://rclone.org/commands/rclone_config_dump/) - Dump the config file as JSON. 396 * [rclone config edit](https://rclone.org/commands/rclone_config_edit/) - Enter an interactive configuration session. 397 * [rclone config file](https://rclone.org/commands/rclone_config_file/) - Show path of configuration file in use. 398 * [rclone config password](https://rclone.org/commands/rclone_config_password/) - Update password in an existing remote. 399 * [rclone config providers](https://rclone.org/commands/rclone_config_providers/) - List in JSON format all the providers and options. 400 * [rclone config reconnect](https://rclone.org/commands/rclone_config_reconnect/) - Re-authenticates user with remote. 401 * [rclone config show](https://rclone.org/commands/rclone_config_show/) - Print (decrypted) config file, or the config for a single remote. 402 * [rclone config update](https://rclone.org/commands/rclone_config_update/) - Update options in an existing remote. 403 * [rclone config userinfo](https://rclone.org/commands/rclone_config_userinfo/) - Prints info about logged in user of remote. 404 405 ## rclone copy 406 407 Copy files from source to dest, skipping already copied 408 409 ### Synopsis 410 411 412 Copy the source to the destination. Doesn't transfer 413 unchanged files, testing by size and modification time or 414 MD5SUM. Doesn't delete files from the destination. 415 416 Note that it is always the contents of the directory that is synced, 417 not the directory so when source:path is a directory, it's the 418 contents of source:path that are copied, not the directory name and 419 contents. 420 421 If dest:path doesn't exist, it is created and the source:path contents 422 go there. 423 424 For example 425 426 rclone copy source:sourcepath dest:destpath 427 428 Let's say there are two files in sourcepath 429 430 sourcepath/one.txt 431 sourcepath/two.txt 432 433 This copies them to 434 435 destpath/one.txt 436 destpath/two.txt 437 438 Not to 439 440 destpath/sourcepath/one.txt 441 destpath/sourcepath/two.txt 442 443 If you are familiar with `rsync`, rclone always works as if you had 444 written a trailing / - meaning "copy the contents of this directory". 445 This applies to all commands and whether you are talking about the 446 source or destination. 447 448 See the [--no-traverse](/docs/#no-traverse) option for controlling 449 whether rclone lists the destination directory or not. Supplying this 450 option when copying a small number of files into a large destination 451 can speed transfers up greatly. 452 453 For example, if you have many files in /path/to/src but only a few of 454 them change every day, you can to copy all the files which have 455 changed recently very efficiently like this: 456 457 rclone copy --max-age 24h --no-traverse /path/to/src remote: 458 459 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 460 461 462 ``` 463 rclone copy source:path dest:path [flags] 464 ``` 465 466 ### Options 467 468 ``` 469 --create-empty-src-dirs Create empty source dirs on destination after copy 470 -h, --help help for copy 471 ``` 472 473 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 474 475 ### SEE ALSO 476 477 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 478 479 ## rclone sync 480 481 Make source and dest identical, modifying destination only. 482 483 ### Synopsis 484 485 486 Sync the source to the destination, changing the destination 487 only. Doesn't transfer unchanged files, testing by size and 488 modification time or MD5SUM. Destination is updated to match 489 source, including deleting files if necessary. 490 491 **Important**: Since this can cause data loss, test first with the 492 `--dry-run` flag to see exactly what would be copied and deleted. 493 494 Note that files in the destination won't be deleted if there were any 495 errors at any point. 496 497 It is always the contents of the directory that is synced, not the 498 directory so when source:path is a directory, it's the contents of 499 source:path that are copied, not the directory name and contents. See 500 extended explanation in the `copy` command above if unsure. 501 502 If dest:path doesn't exist, it is created and the source:path contents 503 go there. 504 505 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 506 507 508 ``` 509 rclone sync source:path dest:path [flags] 510 ``` 511 512 ### Options 513 514 ``` 515 --create-empty-src-dirs Create empty source dirs on destination after sync 516 -h, --help help for sync 517 ``` 518 519 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 520 521 ### SEE ALSO 522 523 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 524 525 ## rclone move 526 527 Move files from source to dest. 528 529 ### Synopsis 530 531 532 Moves the contents of the source directory to the destination 533 directory. Rclone will error if the source and destination overlap and 534 the remote does not support a server side directory move operation. 535 536 If no filters are in use and if possible this will server side move 537 `source:path` into `dest:path`. After this `source:path` will no 538 longer longer exist. 539 540 Otherwise for each file in `source:path` selected by the filters (if 541 any) this will move it into `dest:path`. If possible a server side 542 move will be used, otherwise it will copy it (server side if possible) 543 into `dest:path` then delete the original (if no errors on copy) in 544 `source:path`. 545 546 If you want to delete empty source directories after move, use the --delete-empty-src-dirs flag. 547 548 See the [--no-traverse](/docs/#no-traverse) option for controlling 549 whether rclone lists the destination directory or not. Supplying this 550 option when moving a small number of files into a large destination 551 can speed transfers up greatly. 552 553 **Important**: Since this can cause data loss, test first with the 554 --dry-run flag. 555 556 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics. 557 558 559 ``` 560 rclone move source:path dest:path [flags] 561 ``` 562 563 ### Options 564 565 ``` 566 --create-empty-src-dirs Create empty source dirs on destination after move 567 --delete-empty-src-dirs Delete empty source dirs after move 568 -h, --help help for move 569 ``` 570 571 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 572 573 ### SEE ALSO 574 575 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 576 577 ## rclone delete 578 579 Remove the contents of path. 580 581 ### Synopsis 582 583 584 Remove the files in path. Unlike `purge` it obeys include/exclude 585 filters so can be used to selectively delete files. 586 587 `rclone delete` only deletes objects but leaves the directory structure 588 alone. If you want to delete a directory and all of its contents use 589 `rclone purge` 590 591 Eg delete all files bigger than 100MBytes 592 593 Check what would be deleted first (use either) 594 595 rclone --min-size 100M lsl remote:path 596 rclone --dry-run --min-size 100M delete remote:path 597 598 Then delete 599 600 rclone --min-size 100M delete remote:path 601 602 That reads "delete everything with a minimum size of 100 MB", hence 603 delete all files bigger than 100MBytes. 604 605 606 ``` 607 rclone delete remote:path [flags] 608 ``` 609 610 ### Options 611 612 ``` 613 -h, --help help for delete 614 ``` 615 616 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 617 618 ### SEE ALSO 619 620 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 621 622 ## rclone purge 623 624 Remove the path and all of its contents. 625 626 ### Synopsis 627 628 629 Remove the path and all of its contents. Note that this does not obey 630 include/exclude filters - everything will be removed. Use `delete` if 631 you want to selectively delete files. 632 633 634 ``` 635 rclone purge remote:path [flags] 636 ``` 637 638 ### Options 639 640 ``` 641 -h, --help help for purge 642 ``` 643 644 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 645 646 ### SEE ALSO 647 648 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 649 650 ## rclone mkdir 651 652 Make the path if it doesn't already exist. 653 654 ### Synopsis 655 656 Make the path if it doesn't already exist. 657 658 ``` 659 rclone mkdir remote:path [flags] 660 ``` 661 662 ### Options 663 664 ``` 665 -h, --help help for mkdir 666 ``` 667 668 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 669 670 ### SEE ALSO 671 672 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 673 674 ## rclone rmdir 675 676 Remove the path if empty. 677 678 ### Synopsis 679 680 681 Remove the path. Note that you can't remove a path with 682 objects in it, use purge for that. 683 684 ``` 685 rclone rmdir remote:path [flags] 686 ``` 687 688 ### Options 689 690 ``` 691 -h, --help help for rmdir 692 ``` 693 694 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 695 696 ### SEE ALSO 697 698 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 699 700 ## rclone check 701 702 Checks the files in the source and destination match. 703 704 ### Synopsis 705 706 707 Checks the files in the source and destination match. It compares 708 sizes and hashes (MD5 or SHA1) and logs a report of files which don't 709 match. It doesn't alter the source or destination. 710 711 If you supply the --size-only flag, it will only compare the sizes not 712 the hashes as well. Use this for a quick check. 713 714 If you supply the --download flag, it will download the data from 715 both remotes and check them against each other on the fly. This can 716 be useful for remotes that don't support hashes or if you really want 717 to check all the data. 718 719 If you supply the --one-way flag, it will only check that files in source 720 match the files in destination, not the other way around. Meaning extra files in 721 destination that are not in the source will not trigger an error. 722 723 724 ``` 725 rclone check source:path dest:path [flags] 726 ``` 727 728 ### Options 729 730 ``` 731 --download Check by downloading rather than with hash. 732 -h, --help help for check 733 --one-way Check one way only, source files must exist on remote 734 ``` 735 736 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 737 738 ### SEE ALSO 739 740 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 741 742 ## rclone ls 743 744 List the objects in the path with size and path. 745 746 ### Synopsis 747 748 749 Lists the objects in the source path to standard output in a human 750 readable format with size and path. Recurses by default. 751 752 Eg 753 754 $ rclone ls swift:bucket 755 60295 bevajer5jef 756 90613 canole 757 94467 diwogej7 758 37600 fubuwic 759 760 761 Any of the filtering options can be applied to this command. 762 763 There are several related list commands 764 765 * `ls` to list size and path of objects only 766 * `lsl` to list modification time, size and path of objects only 767 * `lsd` to list directories only 768 * `lsf` to list objects and directories in easy to parse format 769 * `lsjson` to list objects and directories in JSON format 770 771 `ls`,`lsl`,`lsd` are designed to be human readable. 772 `lsf` is designed to be human and machine readable. 773 `lsjson` is designed to be machine readable. 774 775 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 776 777 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 778 779 Listing a non existent directory will produce an error except for 780 remotes which can't have empty directories (eg s3, swift, gcs, etc - 781 the bucket based remotes). 782 783 784 ``` 785 rclone ls remote:path [flags] 786 ``` 787 788 ### Options 789 790 ``` 791 -h, --help help for ls 792 ``` 793 794 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 795 796 ### SEE ALSO 797 798 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 799 800 ## rclone lsd 801 802 List all directories/containers/buckets in the path. 803 804 ### Synopsis 805 806 807 Lists the directories in the source path to standard output. Does not 808 recurse by default. Use the -R flag to recurse. 809 810 This command lists the total size of the directory (if known, -1 if 811 not), the modification time (if known, the current time if not), the 812 number of objects in the directory (if known, -1 if not) and the name 813 of the directory, Eg 814 815 $ rclone lsd swift: 816 494000 2018-04-26 08:43:20 10000 10000files 817 65 2018-04-26 08:43:20 1 1File 818 819 Or 820 821 $ rclone lsd drive:test 822 -1 2016-10-17 17:41:53 -1 1000files 823 -1 2017-01-03 14:40:54 -1 2500files 824 -1 2017-07-08 14:39:28 -1 4000files 825 826 If you just want the directory names use "rclone lsf --dirs-only". 827 828 829 Any of the filtering options can be applied to this command. 830 831 There are several related list commands 832 833 * `ls` to list size and path of objects only 834 * `lsl` to list modification time, size and path of objects only 835 * `lsd` to list directories only 836 * `lsf` to list objects and directories in easy to parse format 837 * `lsjson` to list objects and directories in JSON format 838 839 `ls`,`lsl`,`lsd` are designed to be human readable. 840 `lsf` is designed to be human and machine readable. 841 `lsjson` is designed to be machine readable. 842 843 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 844 845 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 846 847 Listing a non existent directory will produce an error except for 848 remotes which can't have empty directories (eg s3, swift, gcs, etc - 849 the bucket based remotes). 850 851 852 ``` 853 rclone lsd remote:path [flags] 854 ``` 855 856 ### Options 857 858 ``` 859 -h, --help help for lsd 860 -R, --recursive Recurse into the listing. 861 ``` 862 863 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 864 865 ### SEE ALSO 866 867 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 868 869 ## rclone lsl 870 871 List the objects in path with modification time, size and path. 872 873 ### Synopsis 874 875 876 Lists the objects in the source path to standard output in a human 877 readable format with modification time, size and path. Recurses by default. 878 879 Eg 880 881 $ rclone lsl swift:bucket 882 60295 2016-06-25 18:55:41.062626927 bevajer5jef 883 90613 2016-06-25 18:55:43.302607074 canole 884 94467 2016-06-25 18:55:43.046609333 diwogej7 885 37600 2016-06-25 18:55:40.814629136 fubuwic 886 887 888 Any of the filtering options can be applied to this command. 889 890 There are several related list commands 891 892 * `ls` to list size and path of objects only 893 * `lsl` to list modification time, size and path of objects only 894 * `lsd` to list directories only 895 * `lsf` to list objects and directories in easy to parse format 896 * `lsjson` to list objects and directories in JSON format 897 898 `ls`,`lsl`,`lsd` are designed to be human readable. 899 `lsf` is designed to be human and machine readable. 900 `lsjson` is designed to be machine readable. 901 902 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 903 904 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 905 906 Listing a non existent directory will produce an error except for 907 remotes which can't have empty directories (eg s3, swift, gcs, etc - 908 the bucket based remotes). 909 910 911 ``` 912 rclone lsl remote:path [flags] 913 ``` 914 915 ### Options 916 917 ``` 918 -h, --help help for lsl 919 ``` 920 921 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 922 923 ### SEE ALSO 924 925 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 926 927 ## rclone md5sum 928 929 Produces an md5sum file for all the objects in the path. 930 931 ### Synopsis 932 933 934 Produces an md5sum file for all the objects in the path. This 935 is in the same format as the standard md5sum tool produces. 936 937 938 ``` 939 rclone md5sum remote:path [flags] 940 ``` 941 942 ### Options 943 944 ``` 945 -h, --help help for md5sum 946 ``` 947 948 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 949 950 ### SEE ALSO 951 952 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 953 954 ## rclone sha1sum 955 956 Produces an sha1sum file for all the objects in the path. 957 958 ### Synopsis 959 960 961 Produces an sha1sum file for all the objects in the path. This 962 is in the same format as the standard sha1sum tool produces. 963 964 965 ``` 966 rclone sha1sum remote:path [flags] 967 ``` 968 969 ### Options 970 971 ``` 972 -h, --help help for sha1sum 973 ``` 974 975 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 976 977 ### SEE ALSO 978 979 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 980 981 ## rclone size 982 983 Prints the total size and number of objects in remote:path. 984 985 ### Synopsis 986 987 Prints the total size and number of objects in remote:path. 988 989 ``` 990 rclone size remote:path [flags] 991 ``` 992 993 ### Options 994 995 ``` 996 -h, --help help for size 997 --json format output as JSON 998 ``` 999 1000 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1001 1002 ### SEE ALSO 1003 1004 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1005 1006 ## rclone version 1007 1008 Show the version number. 1009 1010 ### Synopsis 1011 1012 1013 Show the version number, the go version and the architecture. 1014 1015 Eg 1016 1017 $ rclone version 1018 rclone v1.41 1019 - os/arch: linux/amd64 1020 - go version: go1.10 1021 1022 If you supply the --check flag, then it will do an online check to 1023 compare your version with the latest release and the latest beta. 1024 1025 $ rclone version --check 1026 yours: 1.42.0.6 1027 latest: 1.42 (released 2018-06-16) 1028 beta: 1.42.0.5 (released 2018-06-17) 1029 1030 Or 1031 1032 $ rclone version --check 1033 yours: 1.41 1034 latest: 1.42 (released 2018-06-16) 1035 upgrade: https://downloads.rclone.org/v1.42 1036 beta: 1.42.0.5 (released 2018-06-17) 1037 upgrade: https://beta.rclone.org/v1.42-005-g56e1e820 1038 1039 1040 1041 ``` 1042 rclone version [flags] 1043 ``` 1044 1045 ### Options 1046 1047 ``` 1048 --check Check for new version. 1049 -h, --help help for version 1050 ``` 1051 1052 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1053 1054 ### SEE ALSO 1055 1056 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1057 1058 ## rclone cleanup 1059 1060 Clean up the remote if possible 1061 1062 ### Synopsis 1063 1064 1065 Clean up the remote if possible. Empty the trash or delete old file 1066 versions. Not supported by all remotes. 1067 1068 1069 ``` 1070 rclone cleanup remote:path [flags] 1071 ``` 1072 1073 ### Options 1074 1075 ``` 1076 -h, --help help for cleanup 1077 ``` 1078 1079 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1080 1081 ### SEE ALSO 1082 1083 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1084 1085 ## rclone dedupe 1086 1087 Interactively find duplicate files and delete/rename them. 1088 1089 ### Synopsis 1090 1091 1092 By default `dedupe` interactively finds duplicate files and offers to 1093 delete all but one or rename them to be different. Only useful with 1094 Google Drive which can have duplicate file names. 1095 1096 In the first pass it will merge directories with the same name. It 1097 will do this iteratively until all the identical directories have been 1098 merged. 1099 1100 The `dedupe` command will delete all but one of any identical (same 1101 md5sum) files it finds without confirmation. This means that for most 1102 duplicated files the `dedupe` command will not be interactive. You 1103 can use `--dry-run` to see what would happen without doing anything. 1104 1105 Here is an example run. 1106 1107 Before - with duplicates 1108 1109 $ rclone lsl drive:dupes 1110 6048320 2016-03-05 16:23:16.798000000 one.txt 1111 6048320 2016-03-05 16:23:11.775000000 one.txt 1112 564374 2016-03-05 16:23:06.731000000 one.txt 1113 6048320 2016-03-05 16:18:26.092000000 one.txt 1114 6048320 2016-03-05 16:22:46.185000000 two.txt 1115 1744073 2016-03-05 16:22:38.104000000 two.txt 1116 564374 2016-03-05 16:22:52.118000000 two.txt 1117 1118 Now the `dedupe` session 1119 1120 $ rclone dedupe drive:dupes 1121 2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode. 1122 one.txt: Found 4 duplicates - deleting identical copies 1123 one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36") 1124 one.txt: 2 duplicates remain 1125 1: 6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36 1126 2: 564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81 1127 s) Skip and do nothing 1128 k) Keep just one (choose which in next step) 1129 r) Rename all to be different (by changing file.jpg to file-1.jpg) 1130 s/k/r> k 1131 Enter the number of the file to keep> 1 1132 one.txt: Deleted 1 extra copies 1133 two.txt: Found 3 duplicates - deleting identical copies 1134 two.txt: 3 duplicates remain 1135 1: 564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81 1136 2: 6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36 1137 3: 1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802 1138 s) Skip and do nothing 1139 k) Keep just one (choose which in next step) 1140 r) Rename all to be different (by changing file.jpg to file-1.jpg) 1141 s/k/r> r 1142 two-1.txt: renamed from: two.txt 1143 two-2.txt: renamed from: two.txt 1144 two-3.txt: renamed from: two.txt 1145 1146 The result being 1147 1148 $ rclone lsl drive:dupes 1149 6048320 2016-03-05 16:23:16.798000000 one.txt 1150 564374 2016-03-05 16:22:52.118000000 two-1.txt 1151 6048320 2016-03-05 16:22:46.185000000 two-2.txt 1152 1744073 2016-03-05 16:22:38.104000000 two-3.txt 1153 1154 Dedupe can be run non interactively using the `--dedupe-mode` flag or by using an extra parameter with the same value 1155 1156 * `--dedupe-mode interactive` - interactive as above. 1157 * `--dedupe-mode skip` - removes identical files then skips anything left. 1158 * `--dedupe-mode first` - removes identical files then keeps the first one. 1159 * `--dedupe-mode newest` - removes identical files then keeps the newest one. 1160 * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. 1161 * `--dedupe-mode largest` - removes identical files then keeps the largest one. 1162 * `--dedupe-mode smallest` - removes identical files then keeps the smallest one. 1163 * `--dedupe-mode rename` - removes identical files then renames the rest to be different. 1164 1165 For example to rename all the identically named photos in your Google Photos directory, do 1166 1167 rclone dedupe --dedupe-mode rename "drive:Google Photos" 1168 1169 Or 1170 1171 rclone dedupe rename "drive:Google Photos" 1172 1173 1174 ``` 1175 rclone dedupe [mode] remote:path [flags] 1176 ``` 1177 1178 ### Options 1179 1180 ``` 1181 --dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename. (default "interactive") 1182 -h, --help help for dedupe 1183 ``` 1184 1185 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1186 1187 ### SEE ALSO 1188 1189 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1190 1191 ## rclone about 1192 1193 Get quota information from the remote. 1194 1195 ### Synopsis 1196 1197 1198 Get quota information from the remote, like bytes used/free/quota and bytes 1199 used in the trash. Not supported by all remotes. 1200 1201 This will print to stdout something like this: 1202 1203 Total: 17G 1204 Used: 7.444G 1205 Free: 1.315G 1206 Trashed: 100.000M 1207 Other: 8.241G 1208 1209 Where the fields are: 1210 1211 * Total: total size available. 1212 * Used: total size used 1213 * Free: total amount this user could upload. 1214 * Trashed: total amount in the trash 1215 * Other: total amount in other storage (eg Gmail, Google Photos) 1216 * Objects: total number of objects in the storage 1217 1218 Note that not all the backends provide all the fields - they will be 1219 missing if they are not known for that backend. Where it is known 1220 that the value is unlimited the value will also be omitted. 1221 1222 Use the --full flag to see the numbers written out in full, eg 1223 1224 Total: 18253611008 1225 Used: 7993453766 1226 Free: 1411001220 1227 Trashed: 104857602 1228 Other: 8849156022 1229 1230 Use the --json flag for a computer readable output, eg 1231 1232 { 1233 "total": 18253611008, 1234 "used": 7993453766, 1235 "trashed": 104857602, 1236 "other": 8849156022, 1237 "free": 1411001220 1238 } 1239 1240 1241 ``` 1242 rclone about remote: [flags] 1243 ``` 1244 1245 ### Options 1246 1247 ``` 1248 --full Full numbers instead of SI units 1249 -h, --help help for about 1250 --json Format output as JSON 1251 ``` 1252 1253 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1254 1255 ### SEE ALSO 1256 1257 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1258 1259 ## rclone authorize 1260 1261 Remote authorization. 1262 1263 ### Synopsis 1264 1265 1266 Remote authorization. Used to authorize a remote or headless 1267 rclone from a machine with a browser - use as instructed by 1268 rclone config. 1269 1270 Use the --auth-no-open-browser to prevent rclone to open auth 1271 link in default browser automatically. 1272 1273 ``` 1274 rclone authorize [flags] 1275 ``` 1276 1277 ### Options 1278 1279 ``` 1280 --auth-no-open-browser Do not automatically open auth link in default browser 1281 -h, --help help for authorize 1282 ``` 1283 1284 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1285 1286 ### SEE ALSO 1287 1288 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1289 1290 ## rclone cachestats 1291 1292 Print cache stats for a remote 1293 1294 ### Synopsis 1295 1296 1297 Print cache stats for a remote in JSON format 1298 1299 1300 ``` 1301 rclone cachestats source: [flags] 1302 ``` 1303 1304 ### Options 1305 1306 ``` 1307 -h, --help help for cachestats 1308 ``` 1309 1310 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1311 1312 ### SEE ALSO 1313 1314 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1315 1316 ## rclone cat 1317 1318 Concatenates any files and sends them to stdout. 1319 1320 ### Synopsis 1321 1322 1323 rclone cat sends any files to standard output. 1324 1325 You can use it like this to output a single file 1326 1327 rclone cat remote:path/to/file 1328 1329 Or like this to output any file in dir or subdirectories. 1330 1331 rclone cat remote:path/to/dir 1332 1333 Or like this to output any .txt files in dir or subdirectories. 1334 1335 rclone --include "*.txt" cat remote:path/to/dir 1336 1337 Use the --head flag to print characters only at the start, --tail for 1338 the end and --offset and --count to print a section in the middle. 1339 Note that if offset is negative it will count from the end, so 1340 --offset -1 --count 1 is equivalent to --tail 1. 1341 1342 1343 ``` 1344 rclone cat remote:path [flags] 1345 ``` 1346 1347 ### Options 1348 1349 ``` 1350 --count int Only print N characters. (default -1) 1351 --discard Discard the output instead of printing. 1352 --head int Only print the first N characters. 1353 -h, --help help for cat 1354 --offset int Start printing at offset N (or from end if -ve). 1355 --tail int Only print the last N characters. 1356 ``` 1357 1358 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1359 1360 ### SEE ALSO 1361 1362 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1363 1364 ## rclone config create 1365 1366 Create a new remote with name, type and options. 1367 1368 ### Synopsis 1369 1370 1371 Create a new remote of <name> with <type> and options. The options 1372 should be passed in in pairs of <key> <value>. 1373 1374 For example to make a swift remote of name myremote using auto config 1375 you would do: 1376 1377 rclone config create myremote swift env_auth true 1378 1379 Note that if the config process would normally ask a question the 1380 default is taken. Each time that happens rclone will print a message 1381 saying how to affect the value taken. 1382 1383 If any of the parameters passed is a password field, then rclone will 1384 automatically obscure them before putting them in the config file. 1385 1386 So for example if you wanted to configure a Google Drive remote but 1387 using remote authorization you would do this: 1388 1389 rclone config create mydrive drive config_is_local false 1390 1391 1392 ``` 1393 rclone config create <name> <type> [<key> <value>]* [flags] 1394 ``` 1395 1396 ### Options 1397 1398 ``` 1399 -h, --help help for create 1400 ``` 1401 1402 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1403 1404 ### SEE ALSO 1405 1406 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1407 1408 ## rclone config delete 1409 1410 Delete an existing remote <name>. 1411 1412 ### Synopsis 1413 1414 Delete an existing remote <name>. 1415 1416 ``` 1417 rclone config delete <name> [flags] 1418 ``` 1419 1420 ### Options 1421 1422 ``` 1423 -h, --help help for delete 1424 ``` 1425 1426 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1427 1428 ### SEE ALSO 1429 1430 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1431 1432 ## rclone config disconnect 1433 1434 Disconnects user from remote 1435 1436 ### Synopsis 1437 1438 1439 This disconnects the remote: passed in to the cloud storage system. 1440 1441 This normally means revoking the oauth token. 1442 1443 To reconnect use "rclone config reconnect". 1444 1445 1446 ``` 1447 rclone config disconnect remote: [flags] 1448 ``` 1449 1450 ### Options 1451 1452 ``` 1453 -h, --help help for disconnect 1454 ``` 1455 1456 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1457 1458 ### SEE ALSO 1459 1460 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1461 1462 ## rclone config dump 1463 1464 Dump the config file as JSON. 1465 1466 ### Synopsis 1467 1468 Dump the config file as JSON. 1469 1470 ``` 1471 rclone config dump [flags] 1472 ``` 1473 1474 ### Options 1475 1476 ``` 1477 -h, --help help for dump 1478 ``` 1479 1480 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1481 1482 ### SEE ALSO 1483 1484 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1485 1486 ## rclone config edit 1487 1488 Enter an interactive configuration session. 1489 1490 ### Synopsis 1491 1492 Enter an interactive configuration session where you can setup new 1493 remotes and manage existing ones. You may also set or remove a 1494 password to protect your configuration. 1495 1496 1497 ``` 1498 rclone config edit [flags] 1499 ``` 1500 1501 ### Options 1502 1503 ``` 1504 -h, --help help for edit 1505 ``` 1506 1507 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1508 1509 ### SEE ALSO 1510 1511 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1512 1513 ## rclone config file 1514 1515 Show path of configuration file in use. 1516 1517 ### Synopsis 1518 1519 Show path of configuration file in use. 1520 1521 ``` 1522 rclone config file [flags] 1523 ``` 1524 1525 ### Options 1526 1527 ``` 1528 -h, --help help for file 1529 ``` 1530 1531 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1532 1533 ### SEE ALSO 1534 1535 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1536 1537 ## rclone config password 1538 1539 Update password in an existing remote. 1540 1541 ### Synopsis 1542 1543 1544 Update an existing remote's password. The password 1545 should be passed in in pairs of <key> <value>. 1546 1547 For example to set password of a remote of name myremote you would do: 1548 1549 rclone config password myremote fieldname mypassword 1550 1551 This command is obsolete now that "config update" and "config create" 1552 both support obscuring passwords directly. 1553 1554 1555 ``` 1556 rclone config password <name> [<key> <value>]+ [flags] 1557 ``` 1558 1559 ### Options 1560 1561 ``` 1562 -h, --help help for password 1563 ``` 1564 1565 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1566 1567 ### SEE ALSO 1568 1569 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1570 1571 ## rclone config providers 1572 1573 List in JSON format all the providers and options. 1574 1575 ### Synopsis 1576 1577 List in JSON format all the providers and options. 1578 1579 ``` 1580 rclone config providers [flags] 1581 ``` 1582 1583 ### Options 1584 1585 ``` 1586 -h, --help help for providers 1587 ``` 1588 1589 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1590 1591 ### SEE ALSO 1592 1593 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1594 1595 ## rclone config reconnect 1596 1597 Re-authenticates user with remote. 1598 1599 ### Synopsis 1600 1601 1602 This reconnects remote: passed in to the cloud storage system. 1603 1604 To disconnect the remote use "rclone config disconnect". 1605 1606 This normally means going through the interactive oauth flow again. 1607 1608 1609 ``` 1610 rclone config reconnect remote: [flags] 1611 ``` 1612 1613 ### Options 1614 1615 ``` 1616 -h, --help help for reconnect 1617 ``` 1618 1619 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1620 1621 ### SEE ALSO 1622 1623 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1624 1625 ## rclone config show 1626 1627 Print (decrypted) config file, or the config for a single remote. 1628 1629 ### Synopsis 1630 1631 Print (decrypted) config file, or the config for a single remote. 1632 1633 ``` 1634 rclone config show [<remote>] [flags] 1635 ``` 1636 1637 ### Options 1638 1639 ``` 1640 -h, --help help for show 1641 ``` 1642 1643 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1644 1645 ### SEE ALSO 1646 1647 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1648 1649 ## rclone config update 1650 1651 Update options in an existing remote. 1652 1653 ### Synopsis 1654 1655 1656 Update an existing remote's options. The options should be passed in 1657 in pairs of <key> <value>. 1658 1659 For example to update the env_auth field of a remote of name myremote 1660 you would do: 1661 1662 rclone config update myremote swift env_auth true 1663 1664 If any of the parameters passed is a password field, then rclone will 1665 automatically obscure them before putting them in the config file. 1666 1667 If the remote uses oauth the token will be updated, if you don't 1668 require this add an extra parameter thus: 1669 1670 rclone config update myremote swift env_auth true config_refresh_token false 1671 1672 1673 ``` 1674 rclone config update <name> [<key> <value>]+ [flags] 1675 ``` 1676 1677 ### Options 1678 1679 ``` 1680 -h, --help help for update 1681 ``` 1682 1683 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1684 1685 ### SEE ALSO 1686 1687 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1688 1689 ## rclone config userinfo 1690 1691 Prints info about logged in user of remote. 1692 1693 ### Synopsis 1694 1695 1696 This prints the details of the person logged in to the cloud storage 1697 system. 1698 1699 1700 ``` 1701 rclone config userinfo remote: [flags] 1702 ``` 1703 1704 ### Options 1705 1706 ``` 1707 -h, --help help for userinfo 1708 --json Format output as JSON 1709 ``` 1710 1711 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1712 1713 ### SEE ALSO 1714 1715 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1716 1717 ## rclone copyto 1718 1719 Copy files from source to dest, skipping already copied 1720 1721 ### Synopsis 1722 1723 1724 If source:path is a file or directory then it copies it to a file or 1725 directory named dest:path. 1726 1727 This can be used to upload single files to other than their current 1728 name. If the source is a directory then it acts exactly like the copy 1729 command. 1730 1731 So 1732 1733 rclone copyto src dst 1734 1735 where src and dst are rclone paths, either remote:path or 1736 /path/to/local or C:\windows\path\if\on\windows. 1737 1738 This will: 1739 1740 if src is file 1741 copy it to dst, overwriting an existing file if it exists 1742 if src is directory 1743 copy it to dst, overwriting existing files if they exist 1744 see copy command for full details 1745 1746 This doesn't transfer unchanged files, testing by size and 1747 modification time or MD5SUM. It doesn't delete files from the 1748 destination. 1749 1750 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 1751 1752 1753 ``` 1754 rclone copyto source:path dest:path [flags] 1755 ``` 1756 1757 ### Options 1758 1759 ``` 1760 -h, --help help for copyto 1761 ``` 1762 1763 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1764 1765 ### SEE ALSO 1766 1767 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1768 1769 ## rclone copyurl 1770 1771 Copy url content to dest. 1772 1773 ### Synopsis 1774 1775 1776 Download a URL's content and copy it to the destination without saving 1777 it in temporary storage. 1778 1779 Setting --auto-filename will cause the file name to be retreived from 1780 the from URL (after any redirections) and used in the destination 1781 path. 1782 1783 Setting --stdout or making the output file name "-" will cause the 1784 output to be written to standard output. 1785 1786 1787 ``` 1788 rclone copyurl https://example.com dest:path [flags] 1789 ``` 1790 1791 ### Options 1792 1793 ``` 1794 -a, --auto-filename Get the file name from the URL and use it for destination file path 1795 -h, --help help for copyurl 1796 --stdout Write the output to stdout rather than a file 1797 ``` 1798 1799 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1800 1801 ### SEE ALSO 1802 1803 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1804 1805 ## rclone cryptcheck 1806 1807 Cryptcheck checks the integrity of a crypted remote. 1808 1809 ### Synopsis 1810 1811 1812 rclone cryptcheck checks a remote against a crypted remote. This is 1813 the equivalent of running rclone check, but able to check the 1814 checksums of the crypted remote. 1815 1816 For it to work the underlying remote of the cryptedremote must support 1817 some kind of checksum. 1818 1819 It works by reading the nonce from each file on the cryptedremote: and 1820 using that to encrypt each file on the remote:. It then checks the 1821 checksum of the underlying file on the cryptedremote: against the 1822 checksum of the file it has just encrypted. 1823 1824 Use it like this 1825 1826 rclone cryptcheck /path/to/files encryptedremote:path 1827 1828 You can use it like this also, but that will involve downloading all 1829 the files in remote:path. 1830 1831 rclone cryptcheck remote:path encryptedremote:path 1832 1833 After it has run it will log the status of the encryptedremote:. 1834 1835 If you supply the --one-way flag, it will only check that files in source 1836 match the files in destination, not the other way around. Meaning extra files in 1837 destination that are not in the source will not trigger an error. 1838 1839 1840 ``` 1841 rclone cryptcheck remote:path cryptedremote:path [flags] 1842 ``` 1843 1844 ### Options 1845 1846 ``` 1847 -h, --help help for cryptcheck 1848 --one-way Check one way only, source files must exist on destination 1849 ``` 1850 1851 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1852 1853 ### SEE ALSO 1854 1855 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1856 1857 ## rclone cryptdecode 1858 1859 Cryptdecode returns unencrypted file names. 1860 1861 ### Synopsis 1862 1863 1864 rclone cryptdecode returns unencrypted file names when provided with 1865 a list of encrypted file names. List limit is 10 items. 1866 1867 If you supply the --reverse flag, it will return encrypted file names. 1868 1869 use it like this 1870 1871 rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2 1872 1873 rclone cryptdecode --reverse encryptedremote: filename1 filename2 1874 1875 1876 ``` 1877 rclone cryptdecode encryptedremote: encryptedfilename [flags] 1878 ``` 1879 1880 ### Options 1881 1882 ``` 1883 -h, --help help for cryptdecode 1884 --reverse Reverse cryptdecode, encrypts filenames 1885 ``` 1886 1887 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1888 1889 ### SEE ALSO 1890 1891 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1892 1893 ## rclone dbhashsum 1894 1895 Produces a Dropbox hash file for all the objects in the path. 1896 1897 ### Synopsis 1898 1899 1900 Produces a Dropbox hash file for all the objects in the path. The 1901 hashes are calculated according to [Dropbox content hash 1902 rules](https://www.dropbox.com/developers/reference/content-hash). 1903 The output is in the same format as md5sum and sha1sum. 1904 1905 1906 ``` 1907 rclone dbhashsum remote:path [flags] 1908 ``` 1909 1910 ### Options 1911 1912 ``` 1913 -h, --help help for dbhashsum 1914 ``` 1915 1916 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1917 1918 ### SEE ALSO 1919 1920 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1921 1922 ## rclone deletefile 1923 1924 Remove a single file from remote. 1925 1926 ### Synopsis 1927 1928 1929 Remove a single file from remote. Unlike `delete` it cannot be used to 1930 remove a directory and it doesn't obey include/exclude filters - if the specified file exists, 1931 it will always be removed. 1932 1933 1934 ``` 1935 rclone deletefile remote:path [flags] 1936 ``` 1937 1938 ### Options 1939 1940 ``` 1941 -h, --help help for deletefile 1942 ``` 1943 1944 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1945 1946 ### SEE ALSO 1947 1948 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1949 1950 ## rclone genautocomplete 1951 1952 Output completion script for a given shell. 1953 1954 ### Synopsis 1955 1956 1957 Generates a shell completion script for rclone. 1958 Run with --help to list the supported shells. 1959 1960 1961 ### Options 1962 1963 ``` 1964 -h, --help help for genautocomplete 1965 ``` 1966 1967 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1968 1969 ### SEE ALSO 1970 1971 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1972 * [rclone genautocomplete bash](https://rclone.org/commands/rclone_genautocomplete_bash/) - Output bash completion script for rclone. 1973 * [rclone genautocomplete zsh](https://rclone.org/commands/rclone_genautocomplete_zsh/) - Output zsh completion script for rclone. 1974 1975 ## rclone genautocomplete bash 1976 1977 Output bash completion script for rclone. 1978 1979 ### Synopsis 1980 1981 1982 Generates a bash shell autocompletion script for rclone. 1983 1984 This writes to /etc/bash_completion.d/rclone by default so will 1985 probably need to be run with sudo or as root, eg 1986 1987 sudo rclone genautocomplete bash 1988 1989 Logout and login again to use the autocompletion scripts, or source 1990 them directly 1991 1992 . /etc/bash_completion 1993 1994 If you supply a command line argument the script will be written 1995 there. 1996 1997 1998 ``` 1999 rclone genautocomplete bash [output_file] [flags] 2000 ``` 2001 2002 ### Options 2003 2004 ``` 2005 -h, --help help for bash 2006 ``` 2007 2008 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2009 2010 ### SEE ALSO 2011 2012 * [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. 2013 2014 ## rclone genautocomplete zsh 2015 2016 Output zsh completion script for rclone. 2017 2018 ### Synopsis 2019 2020 2021 Generates a zsh autocompletion script for rclone. 2022 2023 This writes to /usr/share/zsh/vendor-completions/_rclone by default so will 2024 probably need to be run with sudo or as root, eg 2025 2026 sudo rclone genautocomplete zsh 2027 2028 Logout and login again to use the autocompletion scripts, or source 2029 them directly 2030 2031 autoload -U compinit && compinit 2032 2033 If you supply a command line argument the script will be written 2034 there. 2035 2036 2037 ``` 2038 rclone genautocomplete zsh [output_file] [flags] 2039 ``` 2040 2041 ### Options 2042 2043 ``` 2044 -h, --help help for zsh 2045 ``` 2046 2047 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2048 2049 ### SEE ALSO 2050 2051 * [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. 2052 2053 ## rclone gendocs 2054 2055 Output markdown docs for rclone to the directory supplied. 2056 2057 ### Synopsis 2058 2059 2060 This produces markdown docs for the rclone commands to the directory 2061 supplied. These are in a format suitable for hugo to render into the 2062 rclone.org website. 2063 2064 ``` 2065 rclone gendocs output_directory [flags] 2066 ``` 2067 2068 ### Options 2069 2070 ``` 2071 -h, --help help for gendocs 2072 ``` 2073 2074 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2075 2076 ### SEE ALSO 2077 2078 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2079 2080 ## rclone hashsum 2081 2082 Produces an hashsum file for all the objects in the path. 2083 2084 ### Synopsis 2085 2086 2087 Produces a hash file for all the objects in the path using the hash 2088 named. The output is in the same format as the standard 2089 md5sum/sha1sum tool. 2090 2091 Run without a hash to see the list of supported hashes, eg 2092 2093 $ rclone hashsum 2094 Supported hashes are: 2095 * MD5 2096 * SHA-1 2097 * DropboxHash 2098 * QuickXorHash 2099 2100 Then 2101 2102 $ rclone hashsum MD5 remote:path 2103 2104 2105 ``` 2106 rclone hashsum <hash> remote:path [flags] 2107 ``` 2108 2109 ### Options 2110 2111 ``` 2112 --base64 Output base64 encoded hashsum 2113 -h, --help help for hashsum 2114 ``` 2115 2116 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2117 2118 ### SEE ALSO 2119 2120 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2121 2122 ## rclone link 2123 2124 Generate public link to file/folder. 2125 2126 ### Synopsis 2127 2128 2129 rclone link will create or retrieve a public link to the given file or folder. 2130 2131 rclone link remote:path/to/file 2132 rclone link remote:path/to/folder/ 2133 2134 If successful, the last line of the output will contain the link. Exact 2135 capabilities depend on the remote, but the link will always be created with 2136 the least constraints – e.g. no expiry, no password protection, accessible 2137 without account. 2138 2139 2140 ``` 2141 rclone link remote:path [flags] 2142 ``` 2143 2144 ### Options 2145 2146 ``` 2147 -h, --help help for link 2148 ``` 2149 2150 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2151 2152 ### SEE ALSO 2153 2154 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2155 2156 ## rclone listremotes 2157 2158 List all the remotes in the config file. 2159 2160 ### Synopsis 2161 2162 2163 rclone listremotes lists all the available remotes from the config file. 2164 2165 When uses with the -l flag it lists the types too. 2166 2167 2168 ``` 2169 rclone listremotes [flags] 2170 ``` 2171 2172 ### Options 2173 2174 ``` 2175 -h, --help help for listremotes 2176 --long Show the type as well as names. 2177 ``` 2178 2179 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2180 2181 ### SEE ALSO 2182 2183 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2184 2185 ## rclone lsf 2186 2187 List directories and objects in remote:path formatted for parsing 2188 2189 ### Synopsis 2190 2191 2192 List the contents of the source path (directories and objects) to 2193 standard output in a form which is easy to parse by scripts. By 2194 default this will just be the names of the objects and directories, 2195 one per line. The directories will have a / suffix. 2196 2197 Eg 2198 2199 $ rclone lsf swift:bucket 2200 bevajer5jef 2201 canole 2202 diwogej7 2203 ferejej3gux/ 2204 fubuwic 2205 2206 Use the --format option to control what gets listed. By default this 2207 is just the path, but you can use these parameters to control the 2208 output: 2209 2210 p - path 2211 s - size 2212 t - modification time 2213 h - hash 2214 i - ID of object 2215 o - Original ID of underlying object 2216 m - MimeType of object if known 2217 e - encrypted name 2218 T - tier of storage if known, eg "Hot" or "Cool" 2219 2220 So if you wanted the path, size and modification time, you would use 2221 --format "pst", or maybe --format "tsp" to put the path last. 2222 2223 Eg 2224 2225 $ rclone lsf --format "tsp" swift:bucket 2226 2016-06-25 18:55:41;60295;bevajer5jef 2227 2016-06-25 18:55:43;90613;canole 2228 2016-06-25 18:55:43;94467;diwogej7 2229 2018-04-26 08:50:45;0;ferejej3gux/ 2230 2016-06-25 18:55:40;37600;fubuwic 2231 2232 If you specify "h" in the format you will get the MD5 hash by default, 2233 use the "--hash" flag to change which hash you want. Note that this 2234 can be returned as an empty string if it isn't available on the object 2235 (and for directories), "ERROR" if there was an error reading it from 2236 the object and "UNSUPPORTED" if that object does not support that hash 2237 type. 2238 2239 For example to emulate the md5sum command you can use 2240 2241 rclone lsf -R --hash MD5 --format hp --separator " " --files-only . 2242 2243 Eg 2244 2245 $ rclone lsf -R --hash MD5 --format hp --separator " " --files-only swift:bucket 2246 7908e352297f0f530b84a756f188baa3 bevajer5jef 2247 cd65ac234e6fea5925974a51cdd865cc canole 2248 03b5341b4f234b9d984d03ad076bae91 diwogej7 2249 8fd37c3810dd660778137ac3a66cc06d fubuwic 2250 99713e14a4c4ff553acaf1930fad985b gixacuh7ku 2251 2252 (Though "rclone md5sum ." is an easier way of typing this.) 2253 2254 By default the separator is ";" this can be changed with the 2255 --separator flag. Note that separators aren't escaped in the path so 2256 putting it last is a good strategy. 2257 2258 Eg 2259 2260 $ rclone lsf --separator "," --format "tshp" swift:bucket 2261 2016-06-25 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef 2262 2016-06-25 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole 2263 2016-06-25 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7 2264 2018-04-26 08:52:53,0,,ferejej3gux/ 2265 2016-06-25 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic 2266 2267 You can output in CSV standard format. This will escape things in " 2268 if they contain , 2269 2270 Eg 2271 2272 $ rclone lsf --csv --files-only --format ps remote:path 2273 test.log,22355 2274 test.sh,449 2275 "this file contains a comma, in the file name.txt",6 2276 2277 Note that the --absolute parameter is useful for making lists of files 2278 to pass to an rclone copy with the --files-from flag. 2279 2280 For example to find all the files modified within one day and copy 2281 those only (without traversing the whole directory structure): 2282 2283 rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files 2284 rclone copy --files-from new_files /path/to/local remote:path 2285 2286 2287 Any of the filtering options can be applied to this command. 2288 2289 There are several related list commands 2290 2291 * `ls` to list size and path of objects only 2292 * `lsl` to list modification time, size and path of objects only 2293 * `lsd` to list directories only 2294 * `lsf` to list objects and directories in easy to parse format 2295 * `lsjson` to list objects and directories in JSON format 2296 2297 `ls`,`lsl`,`lsd` are designed to be human readable. 2298 `lsf` is designed to be human and machine readable. 2299 `lsjson` is designed to be machine readable. 2300 2301 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 2302 2303 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 2304 2305 Listing a non existent directory will produce an error except for 2306 remotes which can't have empty directories (eg s3, swift, gcs, etc - 2307 the bucket based remotes). 2308 2309 2310 ``` 2311 rclone lsf remote:path [flags] 2312 ``` 2313 2314 ### Options 2315 2316 ``` 2317 --absolute Put a leading / in front of path names. 2318 --csv Output in CSV format. 2319 -d, --dir-slash Append a slash to directory names. (default true) 2320 --dirs-only Only list directories. 2321 --files-only Only list files. 2322 -F, --format string Output format - see help for details (default "p") 2323 --hash h Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "MD5") 2324 -h, --help help for lsf 2325 -R, --recursive Recurse into the listing. 2326 -s, --separator string Separator for the items in the format. (default ";") 2327 ``` 2328 2329 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2330 2331 ### SEE ALSO 2332 2333 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2334 2335 ## rclone lsjson 2336 2337 List directories and objects in the path in JSON format. 2338 2339 ### Synopsis 2340 2341 List directories and objects in the path in JSON format. 2342 2343 The output is an array of Items, where each Item looks like this 2344 2345 { 2346 "Hashes" : { 2347 "SHA-1" : "f572d396fae9206628714fb2ce00f72e94f2258f", 2348 "MD5" : "b1946ac92492d2347c6235b4d2611184", 2349 "DropboxHash" : "ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc" 2350 }, 2351 "ID": "y2djkhiujf83u33", 2352 "OrigID": "UYOJVTUW00Q1RzTDA", 2353 "IsBucket" : false, 2354 "IsDir" : false, 2355 "MimeType" : "application/octet-stream", 2356 "ModTime" : "2017-05-31T16:15:57.034468261+01:00", 2357 "Name" : "file.txt", 2358 "Encrypted" : "v0qpsdq8anpci8n929v3uu9338", 2359 "EncryptedPath" : "kja9098349023498/v0qpsdq8anpci8n929v3uu9338", 2360 "Path" : "full/path/goes/here/file.txt", 2361 "Size" : 6, 2362 "Tier" : "hot", 2363 } 2364 2365 If --hash is not specified the Hashes property won't be emitted. 2366 2367 If --no-modtime is specified then ModTime will be blank. This can speed things up on remotes where reading the ModTime takes an extra request (eg s3, swift). 2368 2369 If --no-mimetype is specified then MimeType will be blank. This can speed things up on remotes where reading the MimeType takes an extra request (eg s3, swift). 2370 2371 If --encrypted is not specified the Encrypted won't be emitted. 2372 2373 If --dirs-only is not specified files in addition to directories are returned 2374 2375 If --files-only is not specified directories in addition to the files will be returned. 2376 2377 The Path field will only show folders below the remote path being listed. 2378 If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" 2379 will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". 2380 When used without --recursive the Path will always be the same as Name. 2381 2382 If the directory is a bucket in a bucket based backend, then 2383 "IsBucket" will be set to true. This key won't be present unless it is 2384 "true". 2385 2386 The time is in RFC3339 format with up to nanosecond precision. The 2387 number of decimal digits in the seconds will depend on the precision 2388 that the remote can hold the times, so if times are accurate to the 2389 nearest millisecond (eg Google Drive) then 3 digits will always be 2390 shown ("2017-05-31T16:15:57.034+01:00") whereas if the times are 2391 accurate to the nearest second (Dropbox, Box, WebDav etc) no digits 2392 will be shown ("2017-05-31T16:15:57+01:00"). 2393 2394 The whole output can be processed as a JSON blob, or alternatively it 2395 can be processed line by line as each item is written one to a line. 2396 2397 Any of the filtering options can be applied to this command. 2398 2399 There are several related list commands 2400 2401 * `ls` to list size and path of objects only 2402 * `lsl` to list modification time, size and path of objects only 2403 * `lsd` to list directories only 2404 * `lsf` to list objects and directories in easy to parse format 2405 * `lsjson` to list objects and directories in JSON format 2406 2407 `ls`,`lsl`,`lsd` are designed to be human readable. 2408 `lsf` is designed to be human and machine readable. 2409 `lsjson` is designed to be machine readable. 2410 2411 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 2412 2413 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 2414 2415 Listing a non existent directory will produce an error except for 2416 remotes which can't have empty directories (eg s3, swift, gcs, etc - 2417 the bucket based remotes). 2418 2419 2420 ``` 2421 rclone lsjson remote:path [flags] 2422 ``` 2423 2424 ### Options 2425 2426 ``` 2427 --dirs-only Show only directories in the listing. 2428 -M, --encrypted Show the encrypted names. 2429 --files-only Show only files in the listing. 2430 --hash Include hashes in the output (may take longer). 2431 -h, --help help for lsjson 2432 --no-mimetype Don't read the mime type (can speed things up). 2433 --no-modtime Don't read the modification time (can speed things up). 2434 --original Show the ID of the underlying Object. 2435 -R, --recursive Recurse into the listing. 2436 ``` 2437 2438 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2439 2440 ### SEE ALSO 2441 2442 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2443 2444 ## rclone mount 2445 2446 Mount the remote as file system on a mountpoint. 2447 2448 ### Synopsis 2449 2450 2451 rclone mount allows Linux, FreeBSD, macOS and Windows to 2452 mount any of Rclone's cloud storage systems as a file system with 2453 FUSE. 2454 2455 First set up your remote using `rclone config`. Check it works with `rclone ls` etc. 2456 2457 Start the mount like this 2458 2459 rclone mount remote:path/to/files /path/to/local/mount 2460 2461 Or on Windows like this where X: is an unused drive letter 2462 2463 rclone mount remote:path/to/files X: 2464 2465 When the program ends, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, 2466 the mount is automatically stopped. 2467 2468 The umount operation can fail, for example when the mountpoint is busy. 2469 When that happens, it is the user's responsibility to stop the mount manually with 2470 2471 # Linux 2472 fusermount -u /path/to/local/mount 2473 # OS X 2474 umount /path/to/local/mount 2475 2476 ### Installing on Windows 2477 2478 To run rclone mount on Windows, you will need to 2479 download and install [WinFsp](http://www.secfs.net/winfsp/). 2480 2481 WinFsp is an [open source](https://github.com/billziss-gh/winfsp) 2482 Windows File System Proxy which makes it easy to write user space file 2483 systems for Windows. It provides a FUSE emulation layer which rclone 2484 uses combination with 2485 [cgofuse](https://github.com/billziss-gh/cgofuse). Both of these 2486 packages are by Bill Zissimopoulos who was very helpful during the 2487 implementation of rclone mount for Windows. 2488 2489 #### Windows caveats 2490 2491 Note that drives created as Administrator are not visible by other 2492 accounts (including the account that was elevated as 2493 Administrator). So if you start a Windows drive from an Administrative 2494 Command Prompt and then try to access the same drive from Explorer 2495 (which does not run as Administrator), you will not be able to see the 2496 new drive. 2497 2498 The easiest way around this is to start the drive from a normal 2499 command prompt. It is also possible to start a drive from the SYSTEM 2500 account (using [the WinFsp.Launcher 2501 infrastructure](https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture)) 2502 which creates drives accessible for everyone on the system or 2503 alternatively using [the nssm service manager](https://nssm.cc/usage). 2504 2505 ### Limitations 2506 2507 Without the use of "--vfs-cache-mode" this can only write files 2508 sequentially, it can only seek when reading. This means that many 2509 applications won't work with their files on an rclone mount without 2510 "--vfs-cache-mode writes" or "--vfs-cache-mode full". See the [File 2511 Caching](#file-caching) section for more info. 2512 2513 The bucket based remotes (eg Swift, S3, Google Compute Storage, B2, 2514 Hubic) do not support the concept of empty directories, so empty 2515 directories will have a tendency to disappear once they fall out of 2516 the directory cache. 2517 2518 Only supported on Linux, FreeBSD, OS X and Windows at the moment. 2519 2520 ### rclone mount vs rclone sync/copy 2521 2522 File systems expect things to be 100% reliable, whereas cloud storage 2523 systems are a long way from 100% reliable. The rclone sync/copy 2524 commands cope with this with lots of retries. However rclone mount 2525 can't use retries in the same way without making local copies of the 2526 uploads. Look at the [file caching](#file-caching) 2527 for solutions to make mount more reliable. 2528 2529 ### Attribute caching 2530 2531 You can use the flag --attr-timeout to set the time the kernel caches 2532 the attributes (size, modification time etc) for directory entries. 2533 2534 The default is "1s" which caches files just long enough to avoid 2535 too many callbacks to rclone from the kernel. 2536 2537 In theory 0s should be the correct value for filesystems which can 2538 change outside the control of the kernel. However this causes quite a 2539 few problems such as 2540 [rclone using too much memory](https://github.com/rclone/rclone/issues/2157), 2541 [rclone not serving files to samba](https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) 2542 and [excessive time listing directories](https://github.com/rclone/rclone/issues/2095#issuecomment-371141147). 2543 2544 The kernel can cache the info about a file for the time given by 2545 "--attr-timeout". You may see corruption if the remote file changes 2546 length during this window. It will show up as either a truncated file 2547 or a file with garbage on the end. With "--attr-timeout 1s" this is 2548 very unlikely but not impossible. The higher you set "--attr-timeout" 2549 the more likely it is. The default setting of "1s" is the lowest 2550 setting which mitigates the problems above. 2551 2552 If you set it higher ('10s' or '1m' say) then the kernel will call 2553 back to rclone less often making it more efficient, however there is 2554 more chance of the corruption issue above. 2555 2556 If files don't change on the remote outside of the control of rclone 2557 then there is no chance of corruption. 2558 2559 This is the same as setting the attr_timeout option in mount.fuse. 2560 2561 ### Filters 2562 2563 Note that all the rclone filters can be used to select a subset of the 2564 files to be visible in the mount. 2565 2566 ### systemd 2567 2568 When running rclone mount as a systemd service, it is possible 2569 to use Type=notify. In this case the service will enter the started state 2570 after the mountpoint has been successfully set up. 2571 Units having the rclone mount service specified as a requirement 2572 will see all files and folders immediately in this mode. 2573 2574 ### chunked reading ### 2575 2576 --vfs-read-chunk-size will enable reading the source objects in parts. 2577 This can reduce the used download quota for some remotes by requesting only chunks 2578 from the remote that are actually read at the cost of an increased number of requests. 2579 2580 When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, 2581 the chunk size for each open file will get doubled for each chunk read, until the 2582 specified value is reached. A value of -1 will disable the limit and the chunk size will 2583 grow indefinitely. 2584 2585 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following 2586 parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. 2587 When --vfs-read-chunk-size-limit 500M is specified, the result would be 2588 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. 2589 2590 Chunked reading will only work with --vfs-cache-mode < full, as the file will always 2591 be copied to the vfs cache before opening with --vfs-cache-mode full. 2592 2593 ### Directory Cache 2594 2595 Using the `--dir-cache-time` flag, you can set how long a 2596 directory should be considered up to date and not refreshed from the 2597 backend. Changes made locally in the mount may appear immediately or 2598 invalidate the cache. However, changes done on the remote will only 2599 be picked up once the cache expires. 2600 2601 Alternatively, you can send a `SIGHUP` signal to rclone for 2602 it to flush all directory caches, regardless of how old they are. 2603 Assuming only one rclone instance is running, you can reset the cache 2604 like this: 2605 2606 kill -SIGHUP $(pidof rclone) 2607 2608 If you configure rclone with a [remote control](/rc) then you can use 2609 rclone rc to flush the whole directory cache: 2610 2611 rclone rc vfs/forget 2612 2613 Or individual files or directories: 2614 2615 rclone rc vfs/forget file=path/to/file dir=path/to/dir 2616 2617 ### File Buffering 2618 2619 The `--buffer-size` flag determines the amount of memory, 2620 that will be used to buffer data in advance. 2621 2622 Each open file descriptor will try to keep the specified amount of 2623 data in memory at all times. The buffered data is bound to one file 2624 descriptor and won't be shared between multiple open file descriptors 2625 of the same file. 2626 2627 This flag is a upper limit for the used memory per file descriptor. 2628 The buffer will only use memory for data that is downloaded but not 2629 not yet read. If the buffer is empty, only a small amount of memory 2630 will be used. 2631 The maximum memory used by rclone for buffering can be up to 2632 `--buffer-size * open files`. 2633 2634 ### File Caching 2635 2636 These flags control the VFS file caching options. The VFS layer is 2637 used by rclone mount to make a cloud storage system work more like a 2638 normal file system. 2639 2640 You'll need to enable VFS caching if you want, for example, to read 2641 and write simultaneously to a file. See below for more details. 2642 2643 Note that the VFS cache works in addition to the cache backend and you 2644 may find that you need one or the other or both. 2645 2646 --cache-dir string Directory rclone will use for caching. 2647 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 2648 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 2649 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 2650 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 2651 2652 If run with `-vv` rclone will print the location of the file cache. The 2653 files are stored in the user cache file area which is OS dependent but 2654 can be controlled with `--cache-dir` or setting the appropriate 2655 environment variable. 2656 2657 The cache has 4 different modes selected by `--vfs-cache-mode`. 2658 The higher the cache mode the more compatible rclone becomes at the 2659 cost of using disk space. 2660 2661 Note that files are written back to the remote only when they are 2662 closed so if rclone is quit or dies with open files then these won't 2663 get written back to the remote. However they will still be in the on 2664 disk cache. 2665 2666 If using --vfs-cache-max-size note that the cache may exceed this size 2667 for two reasons. Firstly because it is only checked every 2668 --vfs-cache-poll-interval. Secondly because open files cannot be 2669 evicted from the cache. 2670 2671 #### --vfs-cache-mode off 2672 2673 In this mode the cache will read directly from the remote and write 2674 directly to the remote without caching anything on disk. 2675 2676 This will mean some operations are not possible 2677 2678 * Files can't be opened for both read AND write 2679 * Files opened for write can't be seeked 2680 * Existing files opened for write must have O_TRUNC set 2681 * Files open for read with O_TRUNC will be opened write only 2682 * Files open for write only will behave as if O_TRUNC was supplied 2683 * Open modes O_APPEND, O_TRUNC are ignored 2684 * If an upload fails it can't be retried 2685 2686 #### --vfs-cache-mode minimal 2687 2688 This is very similar to "off" except that files opened for read AND 2689 write will be buffered to disks. This means that files opened for 2690 write will be a lot more compatible, but uses the minimal disk space. 2691 2692 These operations are not possible 2693 2694 * Files opened for write only can't be seeked 2695 * Existing files opened for write must have O_TRUNC set 2696 * Files opened for write only will ignore O_APPEND, O_TRUNC 2697 * If an upload fails it can't be retried 2698 2699 #### --vfs-cache-mode writes 2700 2701 In this mode files opened for read only are still read directly from 2702 the remote, write only and read/write files are buffered to disk 2703 first. 2704 2705 This mode should support all normal file system operations. 2706 2707 If an upload fails it will be retried up to --low-level-retries times. 2708 2709 #### --vfs-cache-mode full 2710 2711 In this mode all reads and writes are buffered to and from disk. When 2712 a file is opened for read it will be downloaded in its entirety first. 2713 2714 This may be appropriate for your needs, or you may prefer to look at 2715 the cache backend which does a much more sophisticated job of caching, 2716 including caching directory hierarchies and chunks of files. 2717 2718 In this mode, unlike the others, when a file is written to the disk, 2719 it will be kept on the disk after it is written to the remote. It 2720 will be purged on a schedule according to `--vfs-cache-max-age`. 2721 2722 This mode should support all normal file system operations. 2723 2724 If an upload or download fails it will be retried up to 2725 --low-level-retries times. 2726 2727 2728 ``` 2729 rclone mount remote:path /path/to/mountpoint [flags] 2730 ``` 2731 2732 ### Options 2733 2734 ``` 2735 --allow-non-empty Allow mounting over a non-empty directory. 2736 --allow-other Allow access to other users. 2737 --allow-root Allow access to root user. 2738 --attr-timeout duration Time for which file/directory attributes are cached. (default 1s) 2739 --daemon Run mount as a daemon (background mode). 2740 --daemon-timeout duration Time limit for rclone to respond to kernel (not supported by all OSes). 2741 --debug-fuse Debug the FUSE internals - needs -v. 2742 --default-permissions Makes kernel enforce access control based on the file mode. 2743 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 2744 --dir-perms FileMode Directory permissions (default 0777) 2745 --file-perms FileMode File permissions (default 0666) 2746 --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required. 2747 --gid uint32 Override the gid field set by the filesystem. (default 1000) 2748 -h, --help help for mount 2749 --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads. (default 128k) 2750 --no-checksum Don't compare checksums on up/download. 2751 --no-modtime Don't read/write the modification time (can speed things up). 2752 --no-seek Don't allow seeking in files. 2753 -o, --option stringArray Option for libfuse/WinFsp. Repeat if required. 2754 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 2755 --read-only Mount read-only. 2756 --uid uint32 Override the uid field set by the filesystem. (default 1000) 2757 --umask int Override the permission bits set by the filesystem. 2758 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 2759 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 2760 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 2761 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 2762 --vfs-case-insensitive If a file name not found, find a case insensitive match. 2763 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 2764 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 2765 --volname string Set the volume name (not supported by all OSes). 2766 --write-back-cache Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used. 2767 ``` 2768 2769 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2770 2771 ### SEE ALSO 2772 2773 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2774 2775 ## rclone moveto 2776 2777 Move file or directory from source to dest. 2778 2779 ### Synopsis 2780 2781 2782 If source:path is a file or directory then it moves it to a file or 2783 directory named dest:path. 2784 2785 This can be used to rename files or upload single files to other than 2786 their existing name. If the source is a directory then it acts exactly 2787 like the move command. 2788 2789 So 2790 2791 rclone moveto src dst 2792 2793 where src and dst are rclone paths, either remote:path or 2794 /path/to/local or C:\windows\path\if\on\windows. 2795 2796 This will: 2797 2798 if src is file 2799 move it to dst, overwriting an existing file if it exists 2800 if src is directory 2801 move it to dst, overwriting existing files if they exist 2802 see move command for full details 2803 2804 This doesn't transfer unchanged files, testing by size and 2805 modification time or MD5SUM. src will be deleted on successful 2806 transfer. 2807 2808 **Important**: Since this can cause data loss, test first with the 2809 --dry-run flag. 2810 2811 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics. 2812 2813 2814 ``` 2815 rclone moveto source:path dest:path [flags] 2816 ``` 2817 2818 ### Options 2819 2820 ``` 2821 -h, --help help for moveto 2822 ``` 2823 2824 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2825 2826 ### SEE ALSO 2827 2828 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2829 2830 ## rclone ncdu 2831 2832 Explore a remote with a text based user interface. 2833 2834 ### Synopsis 2835 2836 2837 This displays a text based user interface allowing the navigation of a 2838 remote. It is most useful for answering the question - "What is using 2839 all my disk space?". 2840 2841 <script src="https://asciinema.org/a/157793.js" id="asciicast-157793" async></script> 2842 2843 To make the user interface it first scans the entire remote given and 2844 builds an in memory representation. rclone ncdu can be used during 2845 this scanning phase and you will see it building up the directory 2846 structure as it goes along. 2847 2848 Here are the keys - press '?' to toggle the help on and off 2849 2850 ↑,↓ or k,j to Move 2851 →,l to enter 2852 ←,h to return 2853 c toggle counts 2854 g toggle graph 2855 n,s,C sort by name,size,count 2856 d delete file/directory 2857 y copy current path to clipbard 2858 Y display current path 2859 ^L refresh screen 2860 ? to toggle help on and off 2861 q/ESC/c-C to quit 2862 2863 This an homage to the [ncdu tool](https://dev.yorhel.nl/ncdu) but for 2864 rclone remotes. It is missing lots of features at the moment 2865 but is useful as it stands. 2866 2867 Note that it might take some time to delete big files/folders. The 2868 UI won't respond in the meantime since the deletion is done synchronously. 2869 2870 2871 ``` 2872 rclone ncdu remote:path [flags] 2873 ``` 2874 2875 ### Options 2876 2877 ``` 2878 -h, --help help for ncdu 2879 ``` 2880 2881 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2882 2883 ### SEE ALSO 2884 2885 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2886 2887 ## rclone obscure 2888 2889 Obscure password for use in the rclone.conf 2890 2891 ### Synopsis 2892 2893 Obscure password for use in the rclone.conf 2894 2895 ``` 2896 rclone obscure password [flags] 2897 ``` 2898 2899 ### Options 2900 2901 ``` 2902 -h, --help help for obscure 2903 ``` 2904 2905 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2906 2907 ### SEE ALSO 2908 2909 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2910 2911 ## rclone rc 2912 2913 Run a command against a running rclone. 2914 2915 ### Synopsis 2916 2917 2918 2919 This runs a command against a running rclone. Use the --url flag to 2920 specify an non default URL to connect on. This can be either a 2921 ":port" which is taken to mean "http://localhost:port" or a 2922 "host:port" which is taken to mean "http://host:port" 2923 2924 A username and password can be passed in with --user and --pass. 2925 2926 Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, 2927 --user, --pass. 2928 2929 Arguments should be passed in as parameter=value. 2930 2931 The result will be returned as a JSON object by default. 2932 2933 The --json parameter can be used to pass in a JSON blob as an input 2934 instead of key=value arguments. This is the only way of passing in 2935 more complicated values. 2936 2937 Use --loopback to connect to the rclone instance running "rclone rc". 2938 This is very useful for testing commands without having to run an 2939 rclone rc server, eg: 2940 2941 rclone rc --loopback operations/about fs=/ 2942 2943 Use "rclone rc" to see a list of all possible commands. 2944 2945 ``` 2946 rclone rc commands parameter [flags] 2947 ``` 2948 2949 ### Options 2950 2951 ``` 2952 -h, --help help for rc 2953 --json string Input JSON - use instead of key=value args. 2954 --loopback If set connect to this rclone instance not via HTTP. 2955 --no-output If set don't output the JSON result. 2956 --pass string Password to use to connect to rclone remote control. 2957 --url string URL to connect to rclone remote control. (default "http://localhost:5572/") 2958 --user string Username to use to rclone remote control. 2959 ``` 2960 2961 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2962 2963 ### SEE ALSO 2964 2965 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2966 2967 ## rclone rcat 2968 2969 Copies standard input to file on remote. 2970 2971 ### Synopsis 2972 2973 2974 rclone rcat reads from standard input (stdin) and copies it to a 2975 single remote file. 2976 2977 echo "hello world" | rclone rcat remote:path/to/file 2978 ffmpeg - | rclone rcat remote:path/to/file 2979 2980 If the remote file already exists, it will be overwritten. 2981 2982 rcat will try to upload small files in a single request, which is 2983 usually more efficient than the streaming/chunked upload endpoints, 2984 which use multiple requests. Exact behaviour depends on the remote. 2985 What is considered a small file may be set through 2986 `--streaming-upload-cutoff`. Uploading only starts after 2987 the cutoff is reached or if the file ends before that. The data 2988 must fit into RAM. The cutoff needs to be small enough to adhere 2989 the limits of your remote, please see there. Generally speaking, 2990 setting this cutoff too high will decrease your performance. 2991 2992 Note that the upload can also not be retried because the data is 2993 not kept around until the upload succeeds. If you need to transfer 2994 a lot of data, you're better off caching locally and then 2995 `rclone move` it to the destination. 2996 2997 ``` 2998 rclone rcat remote:path [flags] 2999 ``` 3000 3001 ### Options 3002 3003 ``` 3004 -h, --help help for rcat 3005 ``` 3006 3007 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3008 3009 ### SEE ALSO 3010 3011 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3012 3013 ## rclone rcd 3014 3015 Run rclone listening to remote control commands only. 3016 3017 ### Synopsis 3018 3019 3020 This runs rclone so that it only listens to remote control commands. 3021 3022 This is useful if you are controlling rclone via the rc API. 3023 3024 If you pass in a path to a directory, rclone will serve that directory 3025 for GET requests on the URL passed in. It will also open the URL in 3026 the browser when rclone is run. 3027 3028 See the [rc documentation](https://rclone.org/rc/) for more info on the rc flags. 3029 3030 3031 ``` 3032 rclone rcd <path to files to serve>* [flags] 3033 ``` 3034 3035 ### Options 3036 3037 ``` 3038 -h, --help help for rcd 3039 ``` 3040 3041 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3042 3043 ### SEE ALSO 3044 3045 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3046 3047 ## rclone rmdirs 3048 3049 Remove empty directories under the path. 3050 3051 ### Synopsis 3052 3053 This removes any empty directories (or directories that only contain 3054 empty directories) under the path that it finds, including the path if 3055 it has nothing in. 3056 3057 If you supply the --leave-root flag, it will not remove the root directory. 3058 3059 This is useful for tidying up remotes that rclone has left a lot of 3060 empty directories in. 3061 3062 3063 3064 ``` 3065 rclone rmdirs remote:path [flags] 3066 ``` 3067 3068 ### Options 3069 3070 ``` 3071 -h, --help help for rmdirs 3072 --leave-root Do not remove root directory if empty 3073 ``` 3074 3075 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3076 3077 ### SEE ALSO 3078 3079 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3080 3081 ## rclone serve 3082 3083 Serve a remote over a protocol. 3084 3085 ### Synopsis 3086 3087 rclone serve is used to serve a remote over a given protocol. This 3088 command requires the use of a subcommand to specify the protocol, eg 3089 3090 rclone serve http remote: 3091 3092 Each subcommand has its own options which you can see in their help. 3093 3094 3095 ``` 3096 rclone serve <protocol> [opts] <remote> [flags] 3097 ``` 3098 3099 ### Options 3100 3101 ``` 3102 -h, --help help for serve 3103 ``` 3104 3105 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3106 3107 ### SEE ALSO 3108 3109 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3110 * [rclone serve dlna](https://rclone.org/commands/rclone_serve_dlna/) - Serve remote:path over DLNA 3111 * [rclone serve ftp](https://rclone.org/commands/rclone_serve_ftp/) - Serve remote:path over FTP. 3112 * [rclone serve http](https://rclone.org/commands/rclone_serve_http/) - Serve the remote over HTTP. 3113 * [rclone serve restic](https://rclone.org/commands/rclone_serve_restic/) - Serve the remote for restic's REST API. 3114 * [rclone serve sftp](https://rclone.org/commands/rclone_serve_sftp/) - Serve the remote over SFTP. 3115 * [rclone serve webdav](https://rclone.org/commands/rclone_serve_webdav/) - Serve remote:path over webdav. 3116 3117 ## rclone serve dlna 3118 3119 Serve remote:path over DLNA 3120 3121 ### Synopsis 3122 3123 rclone serve dlna is a DLNA media server for media stored in a rclone remote. Many 3124 devices, such as the Xbox and PlayStation, can automatically discover this server in the LAN 3125 and play audio/video from it. VLC is also supported. Service discovery uses UDP multicast 3126 packets (SSDP) and will thus only work on LANs. 3127 3128 Rclone will list all files present in the remote, without filtering based on media formats or 3129 file extensions. Additionally, there is no media transcoding support. This means that some 3130 players might show files that they are not able to play back correctly. 3131 3132 3133 ### Server options 3134 3135 Use --addr to specify which IP address and port the server should 3136 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3137 IPs. 3138 3139 Use --name to choose the friendly server name, which is by 3140 default "rclone (hostname)". 3141 3142 Use --log-trace in conjunction with -vv to enable additional debug 3143 logging of all UPNP traffic. 3144 3145 ### Directory Cache 3146 3147 Using the `--dir-cache-time` flag, you can set how long a 3148 directory should be considered up to date and not refreshed from the 3149 backend. Changes made locally in the mount may appear immediately or 3150 invalidate the cache. However, changes done on the remote will only 3151 be picked up once the cache expires. 3152 3153 Alternatively, you can send a `SIGHUP` signal to rclone for 3154 it to flush all directory caches, regardless of how old they are. 3155 Assuming only one rclone instance is running, you can reset the cache 3156 like this: 3157 3158 kill -SIGHUP $(pidof rclone) 3159 3160 If you configure rclone with a [remote control](/rc) then you can use 3161 rclone rc to flush the whole directory cache: 3162 3163 rclone rc vfs/forget 3164 3165 Or individual files or directories: 3166 3167 rclone rc vfs/forget file=path/to/file dir=path/to/dir 3168 3169 ### File Buffering 3170 3171 The `--buffer-size` flag determines the amount of memory, 3172 that will be used to buffer data in advance. 3173 3174 Each open file descriptor will try to keep the specified amount of 3175 data in memory at all times. The buffered data is bound to one file 3176 descriptor and won't be shared between multiple open file descriptors 3177 of the same file. 3178 3179 This flag is a upper limit for the used memory per file descriptor. 3180 The buffer will only use memory for data that is downloaded but not 3181 not yet read. If the buffer is empty, only a small amount of memory 3182 will be used. 3183 The maximum memory used by rclone for buffering can be up to 3184 `--buffer-size * open files`. 3185 3186 ### File Caching 3187 3188 These flags control the VFS file caching options. The VFS layer is 3189 used by rclone mount to make a cloud storage system work more like a 3190 normal file system. 3191 3192 You'll need to enable VFS caching if you want, for example, to read 3193 and write simultaneously to a file. See below for more details. 3194 3195 Note that the VFS cache works in addition to the cache backend and you 3196 may find that you need one or the other or both. 3197 3198 --cache-dir string Directory rclone will use for caching. 3199 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3200 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 3201 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3202 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 3203 3204 If run with `-vv` rclone will print the location of the file cache. The 3205 files are stored in the user cache file area which is OS dependent but 3206 can be controlled with `--cache-dir` or setting the appropriate 3207 environment variable. 3208 3209 The cache has 4 different modes selected by `--vfs-cache-mode`. 3210 The higher the cache mode the more compatible rclone becomes at the 3211 cost of using disk space. 3212 3213 Note that files are written back to the remote only when they are 3214 closed so if rclone is quit or dies with open files then these won't 3215 get written back to the remote. However they will still be in the on 3216 disk cache. 3217 3218 If using --vfs-cache-max-size note that the cache may exceed this size 3219 for two reasons. Firstly because it is only checked every 3220 --vfs-cache-poll-interval. Secondly because open files cannot be 3221 evicted from the cache. 3222 3223 #### --vfs-cache-mode off 3224 3225 In this mode the cache will read directly from the remote and write 3226 directly to the remote without caching anything on disk. 3227 3228 This will mean some operations are not possible 3229 3230 * Files can't be opened for both read AND write 3231 * Files opened for write can't be seeked 3232 * Existing files opened for write must have O_TRUNC set 3233 * Files open for read with O_TRUNC will be opened write only 3234 * Files open for write only will behave as if O_TRUNC was supplied 3235 * Open modes O_APPEND, O_TRUNC are ignored 3236 * If an upload fails it can't be retried 3237 3238 #### --vfs-cache-mode minimal 3239 3240 This is very similar to "off" except that files opened for read AND 3241 write will be buffered to disks. This means that files opened for 3242 write will be a lot more compatible, but uses the minimal disk space. 3243 3244 These operations are not possible 3245 3246 * Files opened for write only can't be seeked 3247 * Existing files opened for write must have O_TRUNC set 3248 * Files opened for write only will ignore O_APPEND, O_TRUNC 3249 * If an upload fails it can't be retried 3250 3251 #### --vfs-cache-mode writes 3252 3253 In this mode files opened for read only are still read directly from 3254 the remote, write only and read/write files are buffered to disk 3255 first. 3256 3257 This mode should support all normal file system operations. 3258 3259 If an upload fails it will be retried up to --low-level-retries times. 3260 3261 #### --vfs-cache-mode full 3262 3263 In this mode all reads and writes are buffered to and from disk. When 3264 a file is opened for read it will be downloaded in its entirety first. 3265 3266 This may be appropriate for your needs, or you may prefer to look at 3267 the cache backend which does a much more sophisticated job of caching, 3268 including caching directory hierarchies and chunks of files. 3269 3270 In this mode, unlike the others, when a file is written to the disk, 3271 it will be kept on the disk after it is written to the remote. It 3272 will be purged on a schedule according to `--vfs-cache-max-age`. 3273 3274 This mode should support all normal file system operations. 3275 3276 If an upload or download fails it will be retried up to 3277 --low-level-retries times. 3278 3279 3280 ``` 3281 rclone serve dlna remote:path [flags] 3282 ``` 3283 3284 ### Options 3285 3286 ``` 3287 --addr string ip:port or :port to bind the DLNA http server to. (default ":7879") 3288 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 3289 --dir-perms FileMode Directory permissions (default 0777) 3290 --file-perms FileMode File permissions (default 0666) 3291 --gid uint32 Override the gid field set by the filesystem. (default 1000) 3292 -h, --help help for dlna 3293 --log-trace enable trace logging of SOAP traffic 3294 --name string name of DLNA server 3295 --no-checksum Don't compare checksums on up/download. 3296 --no-modtime Don't read/write the modification time (can speed things up). 3297 --no-seek Don't allow seeking in files. 3298 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 3299 --read-only Mount read-only. 3300 --uid uint32 Override the uid field set by the filesystem. (default 1000) 3301 --umask int Override the permission bits set by the filesystem. (default 2) 3302 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3303 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 3304 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 3305 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3306 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3307 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3308 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3309 ``` 3310 3311 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3312 3313 ### SEE ALSO 3314 3315 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 3316 3317 ## rclone serve ftp 3318 3319 Serve remote:path over FTP. 3320 3321 ### Synopsis 3322 3323 3324 rclone serve ftp implements a basic ftp server to serve the 3325 remote over FTP protocol. This can be viewed with a ftp client 3326 or you can make a remote of type ftp to read and write it. 3327 3328 ### Server options 3329 3330 Use --addr to specify which IP address and port the server should 3331 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3332 IPs. By default it only listens on localhost. You can use port 3333 :0 to let the OS choose an available port. 3334 3335 If you set --addr to listen on a public or LAN accessible IP address 3336 then using Authentication is advised - see the next section for info. 3337 3338 #### Authentication 3339 3340 By default this will serve files without needing a login. 3341 3342 You can set a single username and password with the --user and --pass flags. 3343 3344 ### Directory Cache 3345 3346 Using the `--dir-cache-time` flag, you can set how long a 3347 directory should be considered up to date and not refreshed from the 3348 backend. Changes made locally in the mount may appear immediately or 3349 invalidate the cache. However, changes done on the remote will only 3350 be picked up once the cache expires. 3351 3352 Alternatively, you can send a `SIGHUP` signal to rclone for 3353 it to flush all directory caches, regardless of how old they are. 3354 Assuming only one rclone instance is running, you can reset the cache 3355 like this: 3356 3357 kill -SIGHUP $(pidof rclone) 3358 3359 If you configure rclone with a [remote control](/rc) then you can use 3360 rclone rc to flush the whole directory cache: 3361 3362 rclone rc vfs/forget 3363 3364 Or individual files or directories: 3365 3366 rclone rc vfs/forget file=path/to/file dir=path/to/dir 3367 3368 ### File Buffering 3369 3370 The `--buffer-size` flag determines the amount of memory, 3371 that will be used to buffer data in advance. 3372 3373 Each open file descriptor will try to keep the specified amount of 3374 data in memory at all times. The buffered data is bound to one file 3375 descriptor and won't be shared between multiple open file descriptors 3376 of the same file. 3377 3378 This flag is a upper limit for the used memory per file descriptor. 3379 The buffer will only use memory for data that is downloaded but not 3380 not yet read. If the buffer is empty, only a small amount of memory 3381 will be used. 3382 The maximum memory used by rclone for buffering can be up to 3383 `--buffer-size * open files`. 3384 3385 ### File Caching 3386 3387 These flags control the VFS file caching options. The VFS layer is 3388 used by rclone mount to make a cloud storage system work more like a 3389 normal file system. 3390 3391 You'll need to enable VFS caching if you want, for example, to read 3392 and write simultaneously to a file. See below for more details. 3393 3394 Note that the VFS cache works in addition to the cache backend and you 3395 may find that you need one or the other or both. 3396 3397 --cache-dir string Directory rclone will use for caching. 3398 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3399 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 3400 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3401 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 3402 3403 If run with `-vv` rclone will print the location of the file cache. The 3404 files are stored in the user cache file area which is OS dependent but 3405 can be controlled with `--cache-dir` or setting the appropriate 3406 environment variable. 3407 3408 The cache has 4 different modes selected by `--vfs-cache-mode`. 3409 The higher the cache mode the more compatible rclone becomes at the 3410 cost of using disk space. 3411 3412 Note that files are written back to the remote only when they are 3413 closed so if rclone is quit or dies with open files then these won't 3414 get written back to the remote. However they will still be in the on 3415 disk cache. 3416 3417 If using --vfs-cache-max-size note that the cache may exceed this size 3418 for two reasons. Firstly because it is only checked every 3419 --vfs-cache-poll-interval. Secondly because open files cannot be 3420 evicted from the cache. 3421 3422 #### --vfs-cache-mode off 3423 3424 In this mode the cache will read directly from the remote and write 3425 directly to the remote without caching anything on disk. 3426 3427 This will mean some operations are not possible 3428 3429 * Files can't be opened for both read AND write 3430 * Files opened for write can't be seeked 3431 * Existing files opened for write must have O_TRUNC set 3432 * Files open for read with O_TRUNC will be opened write only 3433 * Files open for write only will behave as if O_TRUNC was supplied 3434 * Open modes O_APPEND, O_TRUNC are ignored 3435 * If an upload fails it can't be retried 3436 3437 #### --vfs-cache-mode minimal 3438 3439 This is very similar to "off" except that files opened for read AND 3440 write will be buffered to disks. This means that files opened for 3441 write will be a lot more compatible, but uses the minimal disk space. 3442 3443 These operations are not possible 3444 3445 * Files opened for write only can't be seeked 3446 * Existing files opened for write must have O_TRUNC set 3447 * Files opened for write only will ignore O_APPEND, O_TRUNC 3448 * If an upload fails it can't be retried 3449 3450 #### --vfs-cache-mode writes 3451 3452 In this mode files opened for read only are still read directly from 3453 the remote, write only and read/write files are buffered to disk 3454 first. 3455 3456 This mode should support all normal file system operations. 3457 3458 If an upload fails it will be retried up to --low-level-retries times. 3459 3460 #### --vfs-cache-mode full 3461 3462 In this mode all reads and writes are buffered to and from disk. When 3463 a file is opened for read it will be downloaded in its entirety first. 3464 3465 This may be appropriate for your needs, or you may prefer to look at 3466 the cache backend which does a much more sophisticated job of caching, 3467 including caching directory hierarchies and chunks of files. 3468 3469 In this mode, unlike the others, when a file is written to the disk, 3470 it will be kept on the disk after it is written to the remote. It 3471 will be purged on a schedule according to `--vfs-cache-max-age`. 3472 3473 This mode should support all normal file system operations. 3474 3475 If an upload or download fails it will be retried up to 3476 --low-level-retries times. 3477 3478 ### Auth Proxy 3479 3480 If you supply the parameter `--auth-proxy /path/to/program` then 3481 rclone will use that program to generate backends on the fly which 3482 then are used to authenticate incoming requests. This uses a simple 3483 JSON based protocl with input on STDIN and output on STDOUT. 3484 3485 There is an example program 3486 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 3487 in the rclone source code. 3488 3489 The program's job is to take a `user` and `pass` on the input and turn 3490 those into the config for a backend on STDOUT in JSON format. This 3491 config will have any default parameters for the backend added, but it 3492 won't use configuration from environment variables or command line 3493 options - it is the job of the proxy program to make a complete 3494 config. 3495 3496 This config generated must have this extra parameter 3497 - `_root` - root to use for the backend 3498 3499 And it may have this parameter 3500 - `_obscure` - comma separated strings for parameters to obscure 3501 3502 For example the program might take this on STDIN 3503 3504 ``` 3505 { 3506 "user": "me", 3507 "pass": "mypassword" 3508 } 3509 ``` 3510 3511 And return this on STDOUT 3512 3513 ``` 3514 { 3515 "type": "sftp", 3516 "_root": "", 3517 "_obscure": "pass", 3518 "user": "me", 3519 "pass": "mypassword", 3520 "host": "sftp.example.com" 3521 } 3522 ``` 3523 3524 This would mean that an SFTP backend would be created on the fly for 3525 the `user` and `pass` returned in the output to the host given. Note 3526 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 3527 parameter before creating the backend (which is required for sftp 3528 backends). 3529 3530 The progam can manipulate the supplied `user` in any way, for example 3531 to make proxy to many different sftp backends, you could make the 3532 `user` be `user@example.com` and then set the `host` to `example.com` 3533 in the output and the user to `user`. For security you'd probably want 3534 to restrict the `host` to a limited list. 3535 3536 Note that an internal cache is keyed on `user` so only use that for 3537 configuration, don't use `pass`. This also means that if a user's 3538 password is changed the cache will need to expire (which takes 5 mins) 3539 before it takes effect. 3540 3541 This can be used to build general purpose proxies to any kind of 3542 backend that rclone supports. 3543 3544 3545 ``` 3546 rclone serve ftp remote:path [flags] 3547 ``` 3548 3549 ### Options 3550 3551 ``` 3552 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:2121") 3553 --auth-proxy string A program to use to create the backend from the auth. 3554 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 3555 --dir-perms FileMode Directory permissions (default 0777) 3556 --file-perms FileMode File permissions (default 0666) 3557 --gid uint32 Override the gid field set by the filesystem. (default 1000) 3558 -h, --help help for ftp 3559 --no-checksum Don't compare checksums on up/download. 3560 --no-modtime Don't read/write the modification time (can speed things up). 3561 --no-seek Don't allow seeking in files. 3562 --pass string Password for authentication. (empty value allow every password) 3563 --passive-port string Passive port range to use. (default "30000-32000") 3564 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 3565 --public-ip string Public IP address to advertise for passive connections. 3566 --read-only Mount read-only. 3567 --uid uint32 Override the uid field set by the filesystem. (default 1000) 3568 --umask int Override the permission bits set by the filesystem. (default 2) 3569 --user string User name for authentication. (default "anonymous") 3570 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3571 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 3572 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 3573 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3574 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3575 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3576 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3577 ``` 3578 3579 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3580 3581 ### SEE ALSO 3582 3583 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 3584 3585 ## rclone serve http 3586 3587 Serve the remote over HTTP. 3588 3589 ### Synopsis 3590 3591 rclone serve http implements a basic web server to serve the remote 3592 over HTTP. This can be viewed in a web browser or you can make a 3593 remote of type http read from it. 3594 3595 You can use the filter flags (eg --include, --exclude) to control what 3596 is served. 3597 3598 The server will log errors. Use -v to see access logs. 3599 3600 --bwlimit will be respected for file transfers. Use --stats to 3601 control the stats printing. 3602 3603 ### Server options 3604 3605 Use --addr to specify which IP address and port the server should 3606 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3607 IPs. By default it only listens on localhost. You can use port 3608 :0 to let the OS choose an available port. 3609 3610 If you set --addr to listen on a public or LAN accessible IP address 3611 then using Authentication is advised - see the next section for info. 3612 3613 --server-read-timeout and --server-write-timeout can be used to 3614 control the timeouts on the server. Note that this is the total time 3615 for a transfer. 3616 3617 --max-header-bytes controls the maximum number of bytes the server will 3618 accept in the HTTP header. 3619 3620 --baseurl controls the URL prefix that rclone serves from. By default 3621 rclone will serve from the root. If you used --baseurl "/rclone" then 3622 rclone would serve from a URL starting with "/rclone/". This is 3623 useful if you wish to proxy rclone serve. Rclone automatically 3624 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 3625 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 3626 identically. 3627 3628 #### Authentication 3629 3630 By default this will serve files without needing a login. 3631 3632 You can either use an htpasswd file which can take lots of users, or 3633 set a single username and password with the --user and --pass flags. 3634 3635 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 3636 in standard apache format and supports MD5, SHA1 and BCrypt for basic 3637 authentication. Bcrypt is recommended. 3638 3639 To create an htpasswd file: 3640 3641 touch htpasswd 3642 htpasswd -B htpasswd user 3643 htpasswd -B htpasswd anotherUser 3644 3645 The password file can be updated while rclone is running. 3646 3647 Use --realm to set the authentication realm. 3648 3649 #### SSL/TLS 3650 3651 By default this will serve over http. If you want you can serve over 3652 https. You will need to supply the --cert and --key flags. If you 3653 wish to do client side certificate validation then you will need to 3654 supply --client-ca also. 3655 3656 --cert should be a either a PEM encoded certificate or a concatenation 3657 of that with the CA certificate. --key should be the PEM encoded 3658 private key and --client-ca should be the PEM encoded client 3659 certificate authority certificate. 3660 3661 ### Directory Cache 3662 3663 Using the `--dir-cache-time` flag, you can set how long a 3664 directory should be considered up to date and not refreshed from the 3665 backend. Changes made locally in the mount may appear immediately or 3666 invalidate the cache. However, changes done on the remote will only 3667 be picked up once the cache expires. 3668 3669 Alternatively, you can send a `SIGHUP` signal to rclone for 3670 it to flush all directory caches, regardless of how old they are. 3671 Assuming only one rclone instance is running, you can reset the cache 3672 like this: 3673 3674 kill -SIGHUP $(pidof rclone) 3675 3676 If you configure rclone with a [remote control](/rc) then you can use 3677 rclone rc to flush the whole directory cache: 3678 3679 rclone rc vfs/forget 3680 3681 Or individual files or directories: 3682 3683 rclone rc vfs/forget file=path/to/file dir=path/to/dir 3684 3685 ### File Buffering 3686 3687 The `--buffer-size` flag determines the amount of memory, 3688 that will be used to buffer data in advance. 3689 3690 Each open file descriptor will try to keep the specified amount of 3691 data in memory at all times. The buffered data is bound to one file 3692 descriptor and won't be shared between multiple open file descriptors 3693 of the same file. 3694 3695 This flag is a upper limit for the used memory per file descriptor. 3696 The buffer will only use memory for data that is downloaded but not 3697 not yet read. If the buffer is empty, only a small amount of memory 3698 will be used. 3699 The maximum memory used by rclone for buffering can be up to 3700 `--buffer-size * open files`. 3701 3702 ### File Caching 3703 3704 These flags control the VFS file caching options. The VFS layer is 3705 used by rclone mount to make a cloud storage system work more like a 3706 normal file system. 3707 3708 You'll need to enable VFS caching if you want, for example, to read 3709 and write simultaneously to a file. See below for more details. 3710 3711 Note that the VFS cache works in addition to the cache backend and you 3712 may find that you need one or the other or both. 3713 3714 --cache-dir string Directory rclone will use for caching. 3715 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3716 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 3717 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3718 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 3719 3720 If run with `-vv` rclone will print the location of the file cache. The 3721 files are stored in the user cache file area which is OS dependent but 3722 can be controlled with `--cache-dir` or setting the appropriate 3723 environment variable. 3724 3725 The cache has 4 different modes selected by `--vfs-cache-mode`. 3726 The higher the cache mode the more compatible rclone becomes at the 3727 cost of using disk space. 3728 3729 Note that files are written back to the remote only when they are 3730 closed so if rclone is quit or dies with open files then these won't 3731 get written back to the remote. However they will still be in the on 3732 disk cache. 3733 3734 If using --vfs-cache-max-size note that the cache may exceed this size 3735 for two reasons. Firstly because it is only checked every 3736 --vfs-cache-poll-interval. Secondly because open files cannot be 3737 evicted from the cache. 3738 3739 #### --vfs-cache-mode off 3740 3741 In this mode the cache will read directly from the remote and write 3742 directly to the remote without caching anything on disk. 3743 3744 This will mean some operations are not possible 3745 3746 * Files can't be opened for both read AND write 3747 * Files opened for write can't be seeked 3748 * Existing files opened for write must have O_TRUNC set 3749 * Files open for read with O_TRUNC will be opened write only 3750 * Files open for write only will behave as if O_TRUNC was supplied 3751 * Open modes O_APPEND, O_TRUNC are ignored 3752 * If an upload fails it can't be retried 3753 3754 #### --vfs-cache-mode minimal 3755 3756 This is very similar to "off" except that files opened for read AND 3757 write will be buffered to disks. This means that files opened for 3758 write will be a lot more compatible, but uses the minimal disk space. 3759 3760 These operations are not possible 3761 3762 * Files opened for write only can't be seeked 3763 * Existing files opened for write must have O_TRUNC set 3764 * Files opened for write only will ignore O_APPEND, O_TRUNC 3765 * If an upload fails it can't be retried 3766 3767 #### --vfs-cache-mode writes 3768 3769 In this mode files opened for read only are still read directly from 3770 the remote, write only and read/write files are buffered to disk 3771 first. 3772 3773 This mode should support all normal file system operations. 3774 3775 If an upload fails it will be retried up to --low-level-retries times. 3776 3777 #### --vfs-cache-mode full 3778 3779 In this mode all reads and writes are buffered to and from disk. When 3780 a file is opened for read it will be downloaded in its entirety first. 3781 3782 This may be appropriate for your needs, or you may prefer to look at 3783 the cache backend which does a much more sophisticated job of caching, 3784 including caching directory hierarchies and chunks of files. 3785 3786 In this mode, unlike the others, when a file is written to the disk, 3787 it will be kept on the disk after it is written to the remote. It 3788 will be purged on a schedule according to `--vfs-cache-max-age`. 3789 3790 This mode should support all normal file system operations. 3791 3792 If an upload or download fails it will be retried up to 3793 --low-level-retries times. 3794 3795 3796 ``` 3797 rclone serve http remote:path [flags] 3798 ``` 3799 3800 ### Options 3801 3802 ``` 3803 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 3804 --baseurl string Prefix for URLs - leave blank for root. 3805 --cert string SSL PEM key (concatenation of certificate and CA certificate) 3806 --client-ca string Client certificate authority to verify clients with 3807 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 3808 --dir-perms FileMode Directory permissions (default 0777) 3809 --file-perms FileMode File permissions (default 0666) 3810 --gid uint32 Override the gid field set by the filesystem. (default 1000) 3811 -h, --help help for http 3812 --htpasswd string htpasswd file - if not provided no authentication is done 3813 --key string SSL PEM Private key 3814 --max-header-bytes int Maximum size of request header (default 4096) 3815 --no-checksum Don't compare checksums on up/download. 3816 --no-modtime Don't read/write the modification time (can speed things up). 3817 --no-seek Don't allow seeking in files. 3818 --pass string Password for authentication. 3819 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 3820 --read-only Mount read-only. 3821 --realm string realm for authentication (default "rclone") 3822 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 3823 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 3824 --uid uint32 Override the uid field set by the filesystem. (default 1000) 3825 --umask int Override the permission bits set by the filesystem. (default 2) 3826 --user string User name for authentication. 3827 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3828 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 3829 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 3830 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3831 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3832 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3833 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3834 ``` 3835 3836 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3837 3838 ### SEE ALSO 3839 3840 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 3841 3842 ## rclone serve restic 3843 3844 Serve the remote for restic's REST API. 3845 3846 ### Synopsis 3847 3848 rclone serve restic implements restic's REST backend API 3849 over HTTP. This allows restic to use rclone as a data storage 3850 mechanism for cloud providers that restic does not support directly. 3851 3852 [Restic](https://restic.net/) is a command line program for doing 3853 backups. 3854 3855 The server will log errors. Use -v to see access logs. 3856 3857 --bwlimit will be respected for file transfers. Use --stats to 3858 control the stats printing. 3859 3860 ### Setting up rclone for use by restic ### 3861 3862 First [set up a remote for your chosen cloud provider](/docs/#configure). 3863 3864 Once you have set up the remote, check it is working with, for example 3865 "rclone lsd remote:". You may have called the remote something other 3866 than "remote:" - just substitute whatever you called it in the 3867 following instructions. 3868 3869 Now start the rclone restic server 3870 3871 rclone serve restic -v remote:backup 3872 3873 Where you can replace "backup" in the above by whatever path in the 3874 remote you wish to use. 3875 3876 By default this will serve on "localhost:8080" you can change this 3877 with use of the "--addr" flag. 3878 3879 You might wish to start this server on boot. 3880 3881 ### Setting up restic to use rclone ### 3882 3883 Now you can [follow the restic 3884 instructions](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) 3885 on setting up restic. 3886 3887 Note that you will need restic 0.8.2 or later to interoperate with 3888 rclone. 3889 3890 For the example above you will want to use "http://localhost:8080/" as 3891 the URL for the REST server. 3892 3893 For example: 3894 3895 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/ 3896 $ export RESTIC_PASSWORD=yourpassword 3897 $ restic init 3898 created restic backend 8b1a4b56ae at rest:http://localhost:8080/ 3899 3900 Please note that knowledge of your password is required to access 3901 the repository. Losing your password means that your data is 3902 irrecoverably lost. 3903 $ restic backup /path/to/files/to/backup 3904 scan [/path/to/files/to/backup] 3905 scanned 189 directories, 312 files in 0:00 3906 [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00 3907 duration: 0:00 3908 snapshot 45c8fdd8 saved 3909 3910 #### Multiple repositories #### 3911 3912 Note that you can use the endpoint to host multiple repositories. Do 3913 this by adding a directory name or path after the URL. Note that 3914 these **must** end with /. Eg 3915 3916 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/ 3917 # backup user1 stuff 3918 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/ 3919 # backup user2 stuff 3920 3921 #### Private repositories #### 3922 3923 The "--private-repos" flag can be used to limit users to repositories starting 3924 with a path of "/<username>/". 3925 3926 ### Server options 3927 3928 Use --addr to specify which IP address and port the server should 3929 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3930 IPs. By default it only listens on localhost. You can use port 3931 :0 to let the OS choose an available port. 3932 3933 If you set --addr to listen on a public or LAN accessible IP address 3934 then using Authentication is advised - see the next section for info. 3935 3936 --server-read-timeout and --server-write-timeout can be used to 3937 control the timeouts on the server. Note that this is the total time 3938 for a transfer. 3939 3940 --max-header-bytes controls the maximum number of bytes the server will 3941 accept in the HTTP header. 3942 3943 --baseurl controls the URL prefix that rclone serves from. By default 3944 rclone will serve from the root. If you used --baseurl "/rclone" then 3945 rclone would serve from a URL starting with "/rclone/". This is 3946 useful if you wish to proxy rclone serve. Rclone automatically 3947 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 3948 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 3949 identically. 3950 3951 #### Authentication 3952 3953 By default this will serve files without needing a login. 3954 3955 You can either use an htpasswd file which can take lots of users, or 3956 set a single username and password with the --user and --pass flags. 3957 3958 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 3959 in standard apache format and supports MD5, SHA1 and BCrypt for basic 3960 authentication. Bcrypt is recommended. 3961 3962 To create an htpasswd file: 3963 3964 touch htpasswd 3965 htpasswd -B htpasswd user 3966 htpasswd -B htpasswd anotherUser 3967 3968 The password file can be updated while rclone is running. 3969 3970 Use --realm to set the authentication realm. 3971 3972 #### SSL/TLS 3973 3974 By default this will serve over http. If you want you can serve over 3975 https. You will need to supply the --cert and --key flags. If you 3976 wish to do client side certificate validation then you will need to 3977 supply --client-ca also. 3978 3979 --cert should be a either a PEM encoded certificate or a concatenation 3980 of that with the CA certificate. --key should be the PEM encoded 3981 private key and --client-ca should be the PEM encoded client 3982 certificate authority certificate. 3983 3984 3985 ``` 3986 rclone serve restic remote:path [flags] 3987 ``` 3988 3989 ### Options 3990 3991 ``` 3992 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 3993 --append-only disallow deletion of repository data 3994 --baseurl string Prefix for URLs - leave blank for root. 3995 --cert string SSL PEM key (concatenation of certificate and CA certificate) 3996 --client-ca string Client certificate authority to verify clients with 3997 -h, --help help for restic 3998 --htpasswd string htpasswd file - if not provided no authentication is done 3999 --key string SSL PEM Private key 4000 --max-header-bytes int Maximum size of request header (default 4096) 4001 --pass string Password for authentication. 4002 --private-repos users can only access their private repo 4003 --realm string realm for authentication (default "rclone") 4004 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 4005 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 4006 --stdio run an HTTP2 server on stdin/stdout 4007 --user string User name for authentication. 4008 ``` 4009 4010 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4011 4012 ### SEE ALSO 4013 4014 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4015 4016 ## rclone serve sftp 4017 4018 Serve the remote over SFTP. 4019 4020 ### Synopsis 4021 4022 rclone serve sftp implements an SFTP server to serve the remote 4023 over SFTP. This can be used with an SFTP client or you can make a 4024 remote of type sftp to use with it. 4025 4026 You can use the filter flags (eg --include, --exclude) to control what 4027 is served. 4028 4029 The server will log errors. Use -v to see access logs. 4030 4031 --bwlimit will be respected for file transfers. Use --stats to 4032 control the stats printing. 4033 4034 You must provide some means of authentication, either with --user/--pass, 4035 an authorized keys file (specify location with --authorized-keys - the 4036 default is the same as ssh) or set the --no-auth flag for no 4037 authentication when logging in. 4038 4039 Note that this also implements a small number of shell commands so 4040 that it can provide md5sum/sha1sum/df information for the rclone sftp 4041 backend. This means that is can support SHA1SUMs, MD5SUMs and the 4042 about command when paired with the rclone sftp backend. 4043 4044 If you don't supply a --key then rclone will generate one and cache it 4045 for later use. 4046 4047 By default the server binds to localhost:2022 - if you want it to be 4048 reachable externally then supply "--addr :2022" for example. 4049 4050 Note that the default of "--vfs-cache-mode off" is fine for the rclone 4051 sftp backend, but it may not be with other SFTP clients. 4052 4053 4054 ### Directory Cache 4055 4056 Using the `--dir-cache-time` flag, you can set how long a 4057 directory should be considered up to date and not refreshed from the 4058 backend. Changes made locally in the mount may appear immediately or 4059 invalidate the cache. However, changes done on the remote will only 4060 be picked up once the cache expires. 4061 4062 Alternatively, you can send a `SIGHUP` signal to rclone for 4063 it to flush all directory caches, regardless of how old they are. 4064 Assuming only one rclone instance is running, you can reset the cache 4065 like this: 4066 4067 kill -SIGHUP $(pidof rclone) 4068 4069 If you configure rclone with a [remote control](/rc) then you can use 4070 rclone rc to flush the whole directory cache: 4071 4072 rclone rc vfs/forget 4073 4074 Or individual files or directories: 4075 4076 rclone rc vfs/forget file=path/to/file dir=path/to/dir 4077 4078 ### File Buffering 4079 4080 The `--buffer-size` flag determines the amount of memory, 4081 that will be used to buffer data in advance. 4082 4083 Each open file descriptor will try to keep the specified amount of 4084 data in memory at all times. The buffered data is bound to one file 4085 descriptor and won't be shared between multiple open file descriptors 4086 of the same file. 4087 4088 This flag is a upper limit for the used memory per file descriptor. 4089 The buffer will only use memory for data that is downloaded but not 4090 not yet read. If the buffer is empty, only a small amount of memory 4091 will be used. 4092 The maximum memory used by rclone for buffering can be up to 4093 `--buffer-size * open files`. 4094 4095 ### File Caching 4096 4097 These flags control the VFS file caching options. The VFS layer is 4098 used by rclone mount to make a cloud storage system work more like a 4099 normal file system. 4100 4101 You'll need to enable VFS caching if you want, for example, to read 4102 and write simultaneously to a file. See below for more details. 4103 4104 Note that the VFS cache works in addition to the cache backend and you 4105 may find that you need one or the other or both. 4106 4107 --cache-dir string Directory rclone will use for caching. 4108 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4109 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 4110 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4111 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 4112 4113 If run with `-vv` rclone will print the location of the file cache. The 4114 files are stored in the user cache file area which is OS dependent but 4115 can be controlled with `--cache-dir` or setting the appropriate 4116 environment variable. 4117 4118 The cache has 4 different modes selected by `--vfs-cache-mode`. 4119 The higher the cache mode the more compatible rclone becomes at the 4120 cost of using disk space. 4121 4122 Note that files are written back to the remote only when they are 4123 closed so if rclone is quit or dies with open files then these won't 4124 get written back to the remote. However they will still be in the on 4125 disk cache. 4126 4127 If using --vfs-cache-max-size note that the cache may exceed this size 4128 for two reasons. Firstly because it is only checked every 4129 --vfs-cache-poll-interval. Secondly because open files cannot be 4130 evicted from the cache. 4131 4132 #### --vfs-cache-mode off 4133 4134 In this mode the cache will read directly from the remote and write 4135 directly to the remote without caching anything on disk. 4136 4137 This will mean some operations are not possible 4138 4139 * Files can't be opened for both read AND write 4140 * Files opened for write can't be seeked 4141 * Existing files opened for write must have O_TRUNC set 4142 * Files open for read with O_TRUNC will be opened write only 4143 * Files open for write only will behave as if O_TRUNC was supplied 4144 * Open modes O_APPEND, O_TRUNC are ignored 4145 * If an upload fails it can't be retried 4146 4147 #### --vfs-cache-mode minimal 4148 4149 This is very similar to "off" except that files opened for read AND 4150 write will be buffered to disks. This means that files opened for 4151 write will be a lot more compatible, but uses the minimal disk space. 4152 4153 These operations are not possible 4154 4155 * Files opened for write only can't be seeked 4156 * Existing files opened for write must have O_TRUNC set 4157 * Files opened for write only will ignore O_APPEND, O_TRUNC 4158 * If an upload fails it can't be retried 4159 4160 #### --vfs-cache-mode writes 4161 4162 In this mode files opened for read only are still read directly from 4163 the remote, write only and read/write files are buffered to disk 4164 first. 4165 4166 This mode should support all normal file system operations. 4167 4168 If an upload fails it will be retried up to --low-level-retries times. 4169 4170 #### --vfs-cache-mode full 4171 4172 In this mode all reads and writes are buffered to and from disk. When 4173 a file is opened for read it will be downloaded in its entirety first. 4174 4175 This may be appropriate for your needs, or you may prefer to look at 4176 the cache backend which does a much more sophisticated job of caching, 4177 including caching directory hierarchies and chunks of files. 4178 4179 In this mode, unlike the others, when a file is written to the disk, 4180 it will be kept on the disk after it is written to the remote. It 4181 will be purged on a schedule according to `--vfs-cache-max-age`. 4182 4183 This mode should support all normal file system operations. 4184 4185 If an upload or download fails it will be retried up to 4186 --low-level-retries times. 4187 4188 ### Auth Proxy 4189 4190 If you supply the parameter `--auth-proxy /path/to/program` then 4191 rclone will use that program to generate backends on the fly which 4192 then are used to authenticate incoming requests. This uses a simple 4193 JSON based protocl with input on STDIN and output on STDOUT. 4194 4195 There is an example program 4196 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 4197 in the rclone source code. 4198 4199 The program's job is to take a `user` and `pass` on the input and turn 4200 those into the config for a backend on STDOUT in JSON format. This 4201 config will have any default parameters for the backend added, but it 4202 won't use configuration from environment variables or command line 4203 options - it is the job of the proxy program to make a complete 4204 config. 4205 4206 This config generated must have this extra parameter 4207 - `_root` - root to use for the backend 4208 4209 And it may have this parameter 4210 - `_obscure` - comma separated strings for parameters to obscure 4211 4212 For example the program might take this on STDIN 4213 4214 ``` 4215 { 4216 "user": "me", 4217 "pass": "mypassword" 4218 } 4219 ``` 4220 4221 And return this on STDOUT 4222 4223 ``` 4224 { 4225 "type": "sftp", 4226 "_root": "", 4227 "_obscure": "pass", 4228 "user": "me", 4229 "pass": "mypassword", 4230 "host": "sftp.example.com" 4231 } 4232 ``` 4233 4234 This would mean that an SFTP backend would be created on the fly for 4235 the `user` and `pass` returned in the output to the host given. Note 4236 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 4237 parameter before creating the backend (which is required for sftp 4238 backends). 4239 4240 The progam can manipulate the supplied `user` in any way, for example 4241 to make proxy to many different sftp backends, you could make the 4242 `user` be `user@example.com` and then set the `host` to `example.com` 4243 in the output and the user to `user`. For security you'd probably want 4244 to restrict the `host` to a limited list. 4245 4246 Note that an internal cache is keyed on `user` so only use that for 4247 configuration, don't use `pass`. This also means that if a user's 4248 password is changed the cache will need to expire (which takes 5 mins) 4249 before it takes effect. 4250 4251 This can be used to build general purpose proxies to any kind of 4252 backend that rclone supports. 4253 4254 4255 ``` 4256 rclone serve sftp remote:path [flags] 4257 ``` 4258 4259 ### Options 4260 4261 ``` 4262 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:2022") 4263 --auth-proxy string A program to use to create the backend from the auth. 4264 --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") 4265 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 4266 --dir-perms FileMode Directory permissions (default 0777) 4267 --file-perms FileMode File permissions (default 0666) 4268 --gid uint32 Override the gid field set by the filesystem. (default 1000) 4269 -h, --help help for sftp 4270 --key string SSH private key file (leave blank to auto generate) 4271 --no-auth Allow connections with no authentication if set. 4272 --no-checksum Don't compare checksums on up/download. 4273 --no-modtime Don't read/write the modification time (can speed things up). 4274 --no-seek Don't allow seeking in files. 4275 --pass string Password for authentication. 4276 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 4277 --read-only Mount read-only. 4278 --uid uint32 Override the uid field set by the filesystem. (default 1000) 4279 --umask int Override the permission bits set by the filesystem. (default 2) 4280 --user string User name for authentication. 4281 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4282 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 4283 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 4284 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4285 --vfs-case-insensitive If a file name not found, find a case insensitive match. 4286 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 4287 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 4288 ``` 4289 4290 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4291 4292 ### SEE ALSO 4293 4294 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4295 4296 ## rclone serve webdav 4297 4298 Serve remote:path over webdav. 4299 4300 ### Synopsis 4301 4302 4303 rclone serve webdav implements a basic webdav server to serve the 4304 remote over HTTP via the webdav protocol. This can be viewed with a 4305 webdav client, through a web browser, or you can make a remote of 4306 type webdav to read and write it. 4307 4308 ### Webdav options 4309 4310 #### --etag-hash 4311 4312 This controls the ETag header. Without this flag the ETag will be 4313 based on the ModTime and Size of the object. 4314 4315 If this flag is set to "auto" then rclone will choose the first 4316 supported hash on the backend or you can use a named hash such as 4317 "MD5" or "SHA-1". 4318 4319 Use "rclone hashsum" to see the full list. 4320 4321 4322 ### Server options 4323 4324 Use --addr to specify which IP address and port the server should 4325 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 4326 IPs. By default it only listens on localhost. You can use port 4327 :0 to let the OS choose an available port. 4328 4329 If you set --addr to listen on a public or LAN accessible IP address 4330 then using Authentication is advised - see the next section for info. 4331 4332 --server-read-timeout and --server-write-timeout can be used to 4333 control the timeouts on the server. Note that this is the total time 4334 for a transfer. 4335 4336 --max-header-bytes controls the maximum number of bytes the server will 4337 accept in the HTTP header. 4338 4339 --baseurl controls the URL prefix that rclone serves from. By default 4340 rclone will serve from the root. If you used --baseurl "/rclone" then 4341 rclone would serve from a URL starting with "/rclone/". This is 4342 useful if you wish to proxy rclone serve. Rclone automatically 4343 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 4344 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 4345 identically. 4346 4347 #### Authentication 4348 4349 By default this will serve files without needing a login. 4350 4351 You can either use an htpasswd file which can take lots of users, or 4352 set a single username and password with the --user and --pass flags. 4353 4354 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 4355 in standard apache format and supports MD5, SHA1 and BCrypt for basic 4356 authentication. Bcrypt is recommended. 4357 4358 To create an htpasswd file: 4359 4360 touch htpasswd 4361 htpasswd -B htpasswd user 4362 htpasswd -B htpasswd anotherUser 4363 4364 The password file can be updated while rclone is running. 4365 4366 Use --realm to set the authentication realm. 4367 4368 #### SSL/TLS 4369 4370 By default this will serve over http. If you want you can serve over 4371 https. You will need to supply the --cert and --key flags. If you 4372 wish to do client side certificate validation then you will need to 4373 supply --client-ca also. 4374 4375 --cert should be a either a PEM encoded certificate or a concatenation 4376 of that with the CA certificate. --key should be the PEM encoded 4377 private key and --client-ca should be the PEM encoded client 4378 certificate authority certificate. 4379 4380 ### Directory Cache 4381 4382 Using the `--dir-cache-time` flag, you can set how long a 4383 directory should be considered up to date and not refreshed from the 4384 backend. Changes made locally in the mount may appear immediately or 4385 invalidate the cache. However, changes done on the remote will only 4386 be picked up once the cache expires. 4387 4388 Alternatively, you can send a `SIGHUP` signal to rclone for 4389 it to flush all directory caches, regardless of how old they are. 4390 Assuming only one rclone instance is running, you can reset the cache 4391 like this: 4392 4393 kill -SIGHUP $(pidof rclone) 4394 4395 If you configure rclone with a [remote control](/rc) then you can use 4396 rclone rc to flush the whole directory cache: 4397 4398 rclone rc vfs/forget 4399 4400 Or individual files or directories: 4401 4402 rclone rc vfs/forget file=path/to/file dir=path/to/dir 4403 4404 ### File Buffering 4405 4406 The `--buffer-size` flag determines the amount of memory, 4407 that will be used to buffer data in advance. 4408 4409 Each open file descriptor will try to keep the specified amount of 4410 data in memory at all times. The buffered data is bound to one file 4411 descriptor and won't be shared between multiple open file descriptors 4412 of the same file. 4413 4414 This flag is a upper limit for the used memory per file descriptor. 4415 The buffer will only use memory for data that is downloaded but not 4416 not yet read. If the buffer is empty, only a small amount of memory 4417 will be used. 4418 The maximum memory used by rclone for buffering can be up to 4419 `--buffer-size * open files`. 4420 4421 ### File Caching 4422 4423 These flags control the VFS file caching options. The VFS layer is 4424 used by rclone mount to make a cloud storage system work more like a 4425 normal file system. 4426 4427 You'll need to enable VFS caching if you want, for example, to read 4428 and write simultaneously to a file. See below for more details. 4429 4430 Note that the VFS cache works in addition to the cache backend and you 4431 may find that you need one or the other or both. 4432 4433 --cache-dir string Directory rclone will use for caching. 4434 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4435 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 4436 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4437 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 4438 4439 If run with `-vv` rclone will print the location of the file cache. The 4440 files are stored in the user cache file area which is OS dependent but 4441 can be controlled with `--cache-dir` or setting the appropriate 4442 environment variable. 4443 4444 The cache has 4 different modes selected by `--vfs-cache-mode`. 4445 The higher the cache mode the more compatible rclone becomes at the 4446 cost of using disk space. 4447 4448 Note that files are written back to the remote only when they are 4449 closed so if rclone is quit or dies with open files then these won't 4450 get written back to the remote. However they will still be in the on 4451 disk cache. 4452 4453 If using --vfs-cache-max-size note that the cache may exceed this size 4454 for two reasons. Firstly because it is only checked every 4455 --vfs-cache-poll-interval. Secondly because open files cannot be 4456 evicted from the cache. 4457 4458 #### --vfs-cache-mode off 4459 4460 In this mode the cache will read directly from the remote and write 4461 directly to the remote without caching anything on disk. 4462 4463 This will mean some operations are not possible 4464 4465 * Files can't be opened for both read AND write 4466 * Files opened for write can't be seeked 4467 * Existing files opened for write must have O_TRUNC set 4468 * Files open for read with O_TRUNC will be opened write only 4469 * Files open for write only will behave as if O_TRUNC was supplied 4470 * Open modes O_APPEND, O_TRUNC are ignored 4471 * If an upload fails it can't be retried 4472 4473 #### --vfs-cache-mode minimal 4474 4475 This is very similar to "off" except that files opened for read AND 4476 write will be buffered to disks. This means that files opened for 4477 write will be a lot more compatible, but uses the minimal disk space. 4478 4479 These operations are not possible 4480 4481 * Files opened for write only can't be seeked 4482 * Existing files opened for write must have O_TRUNC set 4483 * Files opened for write only will ignore O_APPEND, O_TRUNC 4484 * If an upload fails it can't be retried 4485 4486 #### --vfs-cache-mode writes 4487 4488 In this mode files opened for read only are still read directly from 4489 the remote, write only and read/write files are buffered to disk 4490 first. 4491 4492 This mode should support all normal file system operations. 4493 4494 If an upload fails it will be retried up to --low-level-retries times. 4495 4496 #### --vfs-cache-mode full 4497 4498 In this mode all reads and writes are buffered to and from disk. When 4499 a file is opened for read it will be downloaded in its entirety first. 4500 4501 This may be appropriate for your needs, or you may prefer to look at 4502 the cache backend which does a much more sophisticated job of caching, 4503 including caching directory hierarchies and chunks of files. 4504 4505 In this mode, unlike the others, when a file is written to the disk, 4506 it will be kept on the disk after it is written to the remote. It 4507 will be purged on a schedule according to `--vfs-cache-max-age`. 4508 4509 This mode should support all normal file system operations. 4510 4511 If an upload or download fails it will be retried up to 4512 --low-level-retries times. 4513 4514 ### Auth Proxy 4515 4516 If you supply the parameter `--auth-proxy /path/to/program` then 4517 rclone will use that program to generate backends on the fly which 4518 then are used to authenticate incoming requests. This uses a simple 4519 JSON based protocl with input on STDIN and output on STDOUT. 4520 4521 There is an example program 4522 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 4523 in the rclone source code. 4524 4525 The program's job is to take a `user` and `pass` on the input and turn 4526 those into the config for a backend on STDOUT in JSON format. This 4527 config will have any default parameters for the backend added, but it 4528 won't use configuration from environment variables or command line 4529 options - it is the job of the proxy program to make a complete 4530 config. 4531 4532 This config generated must have this extra parameter 4533 - `_root` - root to use for the backend 4534 4535 And it may have this parameter 4536 - `_obscure` - comma separated strings for parameters to obscure 4537 4538 For example the program might take this on STDIN 4539 4540 ``` 4541 { 4542 "user": "me", 4543 "pass": "mypassword" 4544 } 4545 ``` 4546 4547 And return this on STDOUT 4548 4549 ``` 4550 { 4551 "type": "sftp", 4552 "_root": "", 4553 "_obscure": "pass", 4554 "user": "me", 4555 "pass": "mypassword", 4556 "host": "sftp.example.com" 4557 } 4558 ``` 4559 4560 This would mean that an SFTP backend would be created on the fly for 4561 the `user` and `pass` returned in the output to the host given. Note 4562 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 4563 parameter before creating the backend (which is required for sftp 4564 backends). 4565 4566 The progam can manipulate the supplied `user` in any way, for example 4567 to make proxy to many different sftp backends, you could make the 4568 `user` be `user@example.com` and then set the `host` to `example.com` 4569 in the output and the user to `user`. For security you'd probably want 4570 to restrict the `host` to a limited list. 4571 4572 Note that an internal cache is keyed on `user` so only use that for 4573 configuration, don't use `pass`. This also means that if a user's 4574 password is changed the cache will need to expire (which takes 5 mins) 4575 before it takes effect. 4576 4577 This can be used to build general purpose proxies to any kind of 4578 backend that rclone supports. 4579 4580 4581 ``` 4582 rclone serve webdav remote:path [flags] 4583 ``` 4584 4585 ### Options 4586 4587 ``` 4588 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 4589 --auth-proxy string A program to use to create the backend from the auth. 4590 --baseurl string Prefix for URLs - leave blank for root. 4591 --cert string SSL PEM key (concatenation of certificate and CA certificate) 4592 --client-ca string Client certificate authority to verify clients with 4593 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 4594 --dir-perms FileMode Directory permissions (default 0777) 4595 --disable-dir-list Disable HTML directory list on GET request for a directory 4596 --etag-hash string Which hash to use for the ETag, or auto or blank for off 4597 --file-perms FileMode File permissions (default 0666) 4598 --gid uint32 Override the gid field set by the filesystem. (default 1000) 4599 -h, --help help for webdav 4600 --htpasswd string htpasswd file - if not provided no authentication is done 4601 --key string SSL PEM Private key 4602 --max-header-bytes int Maximum size of request header (default 4096) 4603 --no-checksum Don't compare checksums on up/download. 4604 --no-modtime Don't read/write the modification time (can speed things up). 4605 --no-seek Don't allow seeking in files. 4606 --pass string Password for authentication. 4607 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 4608 --read-only Mount read-only. 4609 --realm string realm for authentication (default "rclone") 4610 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 4611 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 4612 --uid uint32 Override the uid field set by the filesystem. (default 1000) 4613 --umask int Override the permission bits set by the filesystem. (default 2) 4614 --user string User name for authentication. 4615 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4616 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 4617 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 4618 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4619 --vfs-case-insensitive If a file name not found, find a case insensitive match. 4620 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 4621 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 4622 ``` 4623 4624 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4625 4626 ### SEE ALSO 4627 4628 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4629 4630 ## rclone settier 4631 4632 Changes storage class/tier of objects in remote. 4633 4634 ### Synopsis 4635 4636 4637 rclone settier changes storage tier or class at remote if supported. 4638 Few cloud storage services provides different storage classes on objects, 4639 for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, 4640 Google Cloud Storage, Regional Storage, Nearline, Coldline etc. 4641 4642 Note that, certain tier changes make objects not available to access immediately. 4643 For example tiering to archive in azure blob storage makes objects in frozen state, 4644 user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object 4645 inaccessible.true 4646 4647 You can use it to tier single object 4648 4649 rclone settier Cool remote:path/file 4650 4651 Or use rclone filters to set tier on only specific files 4652 4653 rclone --include "*.txt" settier Hot remote:path/dir 4654 4655 Or just provide remote directory and all files in directory will be tiered 4656 4657 rclone settier tier remote:path/dir 4658 4659 4660 ``` 4661 rclone settier tier remote:path [flags] 4662 ``` 4663 4664 ### Options 4665 4666 ``` 4667 -h, --help help for settier 4668 ``` 4669 4670 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4671 4672 ### SEE ALSO 4673 4674 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 4675 4676 ## rclone touch 4677 4678 Create new file or change file modification time. 4679 4680 ### Synopsis 4681 4682 Create new file or change file modification time. 4683 4684 ``` 4685 rclone touch remote:path [flags] 4686 ``` 4687 4688 ### Options 4689 4690 ``` 4691 -h, --help help for touch 4692 -C, --no-create Do not create the file if it does not exist. 4693 -t, --timestamp string Change the modification times to the specified time instead of the current time of day. The argument is of the form 'YYMMDD' (ex. 17.10.30) or 'YYYY-MM-DDTHH:MM:SS' (ex. 2006-01-02T15:04:05) 4694 ``` 4695 4696 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4697 4698 ### SEE ALSO 4699 4700 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 4701 4702 ## rclone tree 4703 4704 List the contents of the remote in a tree like fashion. 4705 4706 ### Synopsis 4707 4708 4709 rclone tree lists the contents of a remote in a similar way to the 4710 unix tree command. 4711 4712 For example 4713 4714 $ rclone tree remote:path 4715 / 4716 ├── file1 4717 ├── file2 4718 ├── file3 4719 └── subdir 4720 ├── file4 4721 └── file5 4722 4723 1 directories, 5 files 4724 4725 You can use any of the filtering options with the tree command (eg 4726 --include and --exclude). You can also use --fast-list. 4727 4728 The tree command has many options for controlling the listing which 4729 are compatible with the tree command. Note that not all of them have 4730 short options as they conflict with rclone's short options. 4731 4732 4733 ``` 4734 rclone tree remote:path [flags] 4735 ``` 4736 4737 ### Options 4738 4739 ``` 4740 -a, --all All files are listed (list . files too). 4741 -C, --color Turn colorization on always. 4742 -d, --dirs-only List directories only. 4743 --dirsfirst List directories before files (-U disables). 4744 --full-path Print the full path prefix for each file. 4745 -h, --help help for tree 4746 --human Print the size in a more human readable way. 4747 --level int Descend only level directories deep. 4748 -D, --modtime Print the date of last modification. 4749 -i, --noindent Don't print indentation lines. 4750 --noreport Turn off file/directory count at end of tree listing. 4751 -o, --output string Output to file instead of stdout. 4752 -p, --protections Print the protections for each file. 4753 -Q, --quote Quote filenames with double quotes. 4754 -s, --size Print the size in bytes of each file. 4755 --sort string Select sort: name,version,size,mtime,ctime. 4756 --sort-ctime Sort files by last status change time. 4757 -t, --sort-modtime Sort files by last modification time. 4758 -r, --sort-reverse Reverse the order of the sort. 4759 -U, --unsorted Leave files unsorted. 4760 --version Sort files alphanumerically by version. 4761 ``` 4762 4763 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4764 4765 ### SEE ALSO 4766 4767 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 4768 4769 4770 Copying single files 4771 -------------------- 4772 4773 rclone normally syncs or copies directories. However, if the source 4774 remote points to a file, rclone will just copy that file. The 4775 destination remote must point to a directory - rclone will give the 4776 error `Failed to create file system for "remote:file": is a file not a 4777 directory` if it isn't. 4778 4779 For example, suppose you have a remote with a file in called 4780 `test.jpg`, then you could copy just that file like this 4781 4782 rclone copy remote:test.jpg /tmp/download 4783 4784 The file `test.jpg` will be placed inside `/tmp/download`. 4785 4786 This is equivalent to specifying 4787 4788 rclone copy --files-from /tmp/files remote: /tmp/download 4789 4790 Where `/tmp/files` contains the single line 4791 4792 test.jpg 4793 4794 It is recommended to use `copy` when copying individual files, not `sync`. 4795 They have pretty much the same effect but `copy` will use a lot less 4796 memory. 4797 4798 Syntax of remote paths 4799 ---------------------- 4800 4801 The syntax of the paths passed to the rclone command are as follows. 4802 4803 ### /path/to/dir 4804 4805 This refers to the local file system. 4806 4807 On Windows only `\` may be used instead of `/` in local paths 4808 **only**, non local paths must use `/`. 4809 4810 These paths needn't start with a leading `/` - if they don't then they 4811 will be relative to the current directory. 4812 4813 ### remote:path/to/dir 4814 4815 This refers to a directory `path/to/dir` on `remote:` as defined in 4816 the config file (configured with `rclone config`). 4817 4818 ### remote:/path/to/dir 4819 4820 On most backends this is refers to the same directory as 4821 `remote:path/to/dir` and that format should be preferred. On a very 4822 small number of remotes (FTP, SFTP, Dropbox for business) this will 4823 refer to a different directory. On these, paths without a leading `/` 4824 will refer to your "home" directory and paths with a leading `/` will 4825 refer to the root. 4826 4827 ### :backend:path/to/dir 4828 4829 This is an advanced form for creating remotes on the fly. `backend` 4830 should be the name or prefix of a backend (the `type` in the config 4831 file) and all the configuration for the backend should be provided on 4832 the command line (or in environment variables). 4833 4834 Here are some examples: 4835 4836 rclone lsd --http-url https://pub.rclone.org :http: 4837 4838 To list all the directories in the root of `https://pub.rclone.org/`. 4839 4840 rclone lsf --http-url https://example.com :http:path/to/dir 4841 4842 To list files and directories in `https://example.com/path/to/dir/` 4843 4844 rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir 4845 4846 To copy files and directories in `https://example.com/path/to/dir` to `/tmp/dir`. 4847 4848 rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir 4849 4850 To copy files and directories from `example.com` in the relative 4851 directory `path/to/dir` to `/tmp/dir` using sftp. 4852 4853 Quoting and the shell 4854 --------------------- 4855 4856 When you are typing commands to your computer you are using something 4857 called the command line shell. This interprets various characters in 4858 an OS specific way. 4859 4860 Here are some gotchas which may help users unfamiliar with the shell rules 4861 4862 ### Linux / OSX ### 4863 4864 If your names have spaces or shell metacharacters (eg `*`, `?`, `$`, 4865 `'`, `"` etc) then you must quote them. Use single quotes `'` by default. 4866 4867 rclone copy 'Important files?' remote:backup 4868 4869 If you want to send a `'` you will need to use `"`, eg 4870 4871 rclone copy "O'Reilly Reviews" remote:backup 4872 4873 The rules for quoting metacharacters are complicated and if you want 4874 the full details you'll have to consult the manual page for your 4875 shell. 4876 4877 ### Windows ### 4878 4879 If your names have spaces in you need to put them in `"`, eg 4880 4881 rclone copy "E:\folder name\folder name\folder name" remote:backup 4882 4883 If you are using the root directory on its own then don't quote it 4884 (see [#464](https://github.com/rclone/rclone/issues/464) for why), eg 4885 4886 rclone copy E:\ remote:backup 4887 4888 Copying files or directories with `:` in the names 4889 -------------------------------------------------- 4890 4891 rclone uses `:` to mark a remote name. This is, however, a valid 4892 filename component in non-Windows OSes. The remote name parser will 4893 only search for a `:` up to the first `/` so if you need to act on a 4894 file or directory like this then use the full path starting with a 4895 `/`, or use `./` as a current directory prefix. 4896 4897 So to sync a directory called `sync:me` to a remote called `remote:` use 4898 4899 rclone sync ./sync:me remote:path 4900 4901 or 4902 4903 rclone sync /full/path/to/sync:me remote:path 4904 4905 Server Side Copy 4906 ---------------- 4907 4908 Most remotes (but not all - see [the 4909 overview](/overview/#optional-features)) support server side copy. 4910 4911 This means if you want to copy one folder to another then rclone won't 4912 download all the files and re-upload them; it will instruct the server 4913 to copy them in place. 4914 4915 Eg 4916 4917 rclone copy s3:oldbucket s3:newbucket 4918 4919 Will copy the contents of `oldbucket` to `newbucket` without 4920 downloading and re-uploading. 4921 4922 Remotes which don't support server side copy **will** download and 4923 re-upload in this case. 4924 4925 Server side copies are used with `sync` and `copy` and will be 4926 identified in the log when using the `-v` flag. The `move` command 4927 may also use them if remote doesn't support server side move directly. 4928 This is done by issuing a server side copy then a delete which is much 4929 quicker than a download and re-upload. 4930 4931 Server side copies will only be attempted if the remote names are the 4932 same. 4933 4934 This can be used when scripting to make aged backups efficiently, eg 4935 4936 rclone sync remote:current-backup remote:previous-backup 4937 rclone sync /path/to/files remote:current-backup 4938 4939 Options 4940 ------- 4941 4942 Rclone has a number of options to control its behaviour. 4943 4944 Options that take parameters can have the values passed in two ways, 4945 `--option=value` or `--option value`. However boolean (true/false) 4946 options behave slightly differently to the other options in that 4947 `--boolean` sets the option to `true` and the absence of the flag sets 4948 it to `false`. It is also possible to specify `--boolean=false` or 4949 `--boolean=true`. Note that `--boolean false` is not valid - this is 4950 parsed as `--boolean` and the `false` is parsed as an extra command 4951 line argument for rclone. 4952 4953 Options which use TIME use the go time parser. A duration string is a 4954 possibly signed sequence of decimal numbers, each with optional 4955 fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid 4956 time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". 4957 4958 Options which use SIZE use kByte by default. However, a suffix of `b` 4959 for bytes, `k` for kBytes, `M` for MBytes, `G` for GBytes, `T` for 4960 TBytes and `P` for PBytes may be used. These are the binary units, eg 4961 1, 2\*\*10, 2\*\*20, 2\*\*30 respectively. 4962 4963 ### --backup-dir=DIR ### 4964 4965 When using `sync`, `copy` or `move` any files which would have been 4966 overwritten or deleted are moved in their original hierarchy into this 4967 directory. 4968 4969 If `--suffix` is set, then the moved files will have the suffix added 4970 to them. If there is a file with the same path (after the suffix has 4971 been added) in DIR, then it will be overwritten. 4972 4973 The remote in use must support server side move or copy and you must 4974 use the same remote as the destination of the sync. The backup 4975 directory must not overlap the destination directory. 4976 4977 For example 4978 4979 rclone sync /path/to/local remote:current --backup-dir remote:old 4980 4981 will sync `/path/to/local` to `remote:current`, but for any files 4982 which would have been updated or deleted will be stored in 4983 `remote:old`. 4984 4985 If running rclone from a script you might want to use today's date as 4986 the directory name passed to `--backup-dir` to store the old files, or 4987 you might want to pass `--suffix` with today's date. 4988 4989 See `--compare-dest` and `--copy-dest`. 4990 4991 ### --bind string ### 4992 4993 Local address to bind to for outgoing connections. This can be an 4994 IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or host name. If 4995 the host name doesn't resolve or resolves to more than one IP address 4996 it will give an error. 4997 4998 ### --bwlimit=BANDWIDTH_SPEC ### 4999 5000 This option controls the bandwidth limit. Limits can be specified 5001 in two ways: As a single limit, or as a timetable. 5002 5003 Single limits last for the duration of the session. To use a single limit, 5004 specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The 5005 default is `0` which means to not limit bandwidth. 5006 5007 For example, to limit bandwidth usage to 10 MBytes/s use `--bwlimit 10M` 5008 5009 It is also possible to specify a "timetable" of limits, which will cause 5010 certain limits to be applied at certain times. To specify a timetable, format your 5011 entries as "WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH..." where: 5012 WEEKDAY is optional element. 5013 It could be written as whole world or only using 3 first characters. 5014 HH:MM is an hour from 00:00 to 23:59. 5015 5016 An example of a typical timetable to avoid link saturation during daytime 5017 working hours could be: 5018 5019 `--bwlimit "08:00,512 12:00,10M 13:00,512 18:00,30M 23:00,off"` 5020 5021 In this example, the transfer bandwidth will be every day set to 512kBytes/sec at 8am. 5022 At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. 5023 At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be 5024 completely disabled (full speed). Anything between 11pm and 8am will remain 5025 unlimited. 5026 5027 An example of timetable with WEEKDAY could be: 5028 5029 `--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"` 5030 5031 It mean that, the transfer bandwidth will be set to 512kBytes/sec on Monday. 5032 It will raise to 10Mbytes/s before the end of Friday. 5033 At 10:00 on Sunday it will be set to 1Mbyte/s. 5034 From 20:00 at Sunday will be unlimited. 5035 5036 Timeslots without weekday are extended to whole week. 5037 So this one example: 5038 5039 `--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"` 5040 5041 Is equal to this: 5042 5043 `--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"` 5044 5045 Bandwidth limits only apply to the data transfer. They don't apply to the 5046 bandwidth of the directory listings etc. 5047 5048 Note that the units are Bytes/s, not Bits/s. Typically connections are 5049 measured in Bits/s - to convert divide by 8. For example, let's say 5050 you have a 10 Mbit/s connection and you wish rclone to use half of it 5051 - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a `--bwlimit 5052 0.625M` parameter for rclone. 5053 5054 On Unix systems (Linux, MacOS, …) the bandwidth limiter can be toggled by 5055 sending a `SIGUSR2` signal to rclone. This allows to remove the limitations 5056 of a long running rclone transfer and to restore it back to the value specified 5057 with `--bwlimit` quickly when needed. Assuming there is only one rclone instance 5058 running, you can toggle the limiter like this: 5059 5060 kill -SIGUSR2 $(pidof rclone) 5061 5062 If you configure rclone with a [remote control](/rc) then you can use 5063 change the bwlimit dynamically: 5064 5065 rclone rc core/bwlimit rate=1M 5066 5067 ### --buffer-size=SIZE ### 5068 5069 Use this sized buffer to speed up file transfers. Each `--transfer` 5070 will use this much memory for buffering. 5071 5072 When using `mount` or `cmount` each open file descriptor will use this much 5073 memory for buffering. 5074 See the [mount](/commands/rclone_mount/#file-buffering) documentation for more details. 5075 5076 Set to 0 to disable the buffering for the minimum memory usage. 5077 5078 Note that the memory allocation of the buffers is influenced by the 5079 [--use-mmap](#use-mmap) flag. 5080 5081 ### --checkers=N ### 5082 5083 The number of checkers to run in parallel. Checkers do the equality 5084 checking of files during a sync. For some storage systems (eg S3, 5085 Swift, Dropbox) this can take a significant amount of time so they are 5086 run in parallel. 5087 5088 The default is to run 8 checkers in parallel. 5089 5090 ### -c, --checksum ### 5091 5092 Normally rclone will look at modification time and size of files to 5093 see if they are equal. If you set this flag then rclone will check 5094 the file hash and size to determine if files are equal. 5095 5096 This is useful when the remote doesn't support setting modified time 5097 and a more accurate sync is desired than just checking the file size. 5098 5099 This is very useful when transferring between remotes which store the 5100 same hash type on the object, eg Drive and Swift. For details of which 5101 remotes support which hash type see the table in the [overview 5102 section](https://rclone.org/overview/). 5103 5104 Eg `rclone --checksum sync s3:/bucket swift:/bucket` would run much 5105 quicker than without the `--checksum` flag. 5106 5107 When using this flag, rclone won't update mtimes of remote files if 5108 they are incorrect as it would normally. 5109 5110 ### --compare-dest=DIR ### 5111 5112 When using `sync`, `copy` or `move` DIR is checked in addition to the 5113 destination for files. If a file identical to the source is found that 5114 file is NOT copied from source. This is useful to copy just files that 5115 have changed since the last backup. 5116 5117 You must use the same remote as the destination of the sync. The 5118 compare directory must not overlap the destination directory. 5119 5120 See `--copy-dest` and `--backup-dir`. 5121 5122 ### --config=CONFIG_FILE ### 5123 5124 Specify the location of the rclone config file. 5125 5126 Normally the config file is in your home directory as a file called 5127 `.config/rclone/rclone.conf` (or `.rclone.conf` if created with an 5128 older version). If `$XDG_CONFIG_HOME` is set it will be at 5129 `$XDG_CONFIG_HOME/rclone/rclone.conf`. 5130 5131 If there is a file `rclone.conf` in the same directory as the rclone 5132 executable it will be preferred. This file must be created manually 5133 for Rclone to use it, it will never be created automatically. 5134 5135 If you run `rclone config file` you will see where the default 5136 location is for you. 5137 5138 Use this flag to override the config location, eg `rclone 5139 --config=".myconfig" .config`. 5140 5141 ### --contimeout=TIME ### 5142 5143 Set the connection timeout. This should be in go time format which 5144 looks like `5s` for 5 seconds, `10m` for 10 minutes, or `3h30m`. 5145 5146 The connection timeout is the amount of time rclone will wait for a 5147 connection to go through to a remote object storage system. It is 5148 `1m` by default. 5149 5150 ### --copy-dest=DIR ### 5151 5152 When using `sync`, `copy` or `move` DIR is checked in addition to the 5153 destination for files. If a file identical to the source is found that 5154 file is server side copied from DIR to the destination. This is useful 5155 for incremental backup. 5156 5157 The remote in use must support server side copy and you must 5158 use the same remote as the destination of the sync. The compare 5159 directory must not overlap the destination directory. 5160 5161 See `--compare-dest` and `--backup-dir`. 5162 5163 ### --dedupe-mode MODE ### 5164 5165 Mode to run dedupe command in. One of `interactive`, `skip`, `first`, `newest`, `oldest`, `rename`. The default is `interactive`. See the dedupe command for more information as to what these options mean. 5166 5167 ### --disable FEATURE,FEATURE,... ### 5168 5169 This disables a comma separated list of optional features. For example 5170 to disable server side move and server side copy use: 5171 5172 --disable move,copy 5173 5174 The features can be put in in any case. 5175 5176 To see a list of which features can be disabled use: 5177 5178 --disable help 5179 5180 See the overview [features](/overview/#features) and 5181 [optional features](/overview/#optional-features) to get an idea of 5182 which feature does what. 5183 5184 This flag can be useful for debugging and in exceptional circumstances 5185 (eg Google Drive limiting the total volume of Server Side Copies to 5186 100GB/day). 5187 5188 ### -n, --dry-run ### 5189 5190 Do a trial run with no permanent changes. Use this to see what rclone 5191 would do without actually doing it. Useful when setting up the `sync` 5192 command which deletes files in the destination. 5193 5194 ### --expect-continue-timeout=TIME ### 5195 5196 This specifies the amount of time to wait for a server's first 5197 response headers after fully writing the request headers if the 5198 request has an "Expect: 100-continue" header. Not all backends support 5199 using this. 5200 5201 Zero means no timeout and causes the body to be sent immediately, 5202 without waiting for the server to approve. This time does not include 5203 the time to send the request header. 5204 5205 The default is `1s`. Set to 0 to disable. 5206 5207 ### --ignore-case-sync ### 5208 5209 Using this option will cause rclone to ignore the case of the files 5210 when synchronizing so files will not be copied/synced when the 5211 existing filenames are the same, even if the casing is different. 5212 5213 ### --ignore-checksum ### 5214 5215 Normally rclone will check that the checksums of transferred files 5216 match, and give an error "corrupted on transfer" if they don't. 5217 5218 You can use this option to skip that check. You should only use it if 5219 you have had the "corrupted on transfer" error message and you are 5220 sure you might want to transfer potentially corrupted data. 5221 5222 ### --ignore-existing ### 5223 5224 Using this option will make rclone unconditionally skip all files 5225 that exist on the destination, no matter the content of these files. 5226 5227 While this isn't a generally recommended option, it can be useful 5228 in cases where your files change due to encryption. However, it cannot 5229 correct partial transfers in case a transfer was interrupted. 5230 5231 ### --ignore-size ### 5232 5233 Normally rclone will look at modification time and size of files to 5234 see if they are equal. If you set this flag then rclone will check 5235 only the modification time. If `--checksum` is set then it only 5236 checks the checksum. 5237 5238 It will also cause rclone to skip verifying the sizes are the same 5239 after transfer. 5240 5241 This can be useful for transferring files to and from OneDrive which 5242 occasionally misreports the size of image files (see 5243 [#399](https://github.com/rclone/rclone/issues/399) for more info). 5244 5245 ### -I, --ignore-times ### 5246 5247 Using this option will cause rclone to unconditionally upload all 5248 files regardless of the state of files on the destination. 5249 5250 Normally rclone would skip any files that have the same 5251 modification time and are the same size (or have the same checksum if 5252 using `--checksum`). 5253 5254 ### --immutable ### 5255 5256 Treat source and destination files as immutable and disallow 5257 modification. 5258 5259 With this option set, files will be created and deleted as requested, 5260 but existing files will never be updated. If an existing file does 5261 not match between the source and destination, rclone will give the error 5262 `Source and destination exist but do not match: immutable file modified`. 5263 5264 Note that only commands which transfer files (e.g. `sync`, `copy`, 5265 `move`) are affected by this behavior, and only modification is 5266 disallowed. Files may still be deleted explicitly (e.g. `delete`, 5267 `purge`) or implicitly (e.g. `sync`, `move`). Use `copy --immutable` 5268 if it is desired to avoid deletion as well as modification. 5269 5270 This can be useful as an additional layer of protection for immutable 5271 or append-only data sets (notably backup archives), where modification 5272 implies corruption and should not be propagated. 5273 5274 ## --leave-root ### 5275 5276 During rmdirs it will not remove root directory, even if it's empty. 5277 5278 ### --log-file=FILE ### 5279 5280 Log all of rclone's output to FILE. This is not active by default. 5281 This can be useful for tracking down problems with syncs in 5282 combination with the `-v` flag. See the [Logging section](#logging) 5283 for more info. 5284 5285 Note that if you are using the `logrotate` program to manage rclone's 5286 logs, then you should use the `copytruncate` option as rclone doesn't 5287 have a signal to rotate logs. 5288 5289 ### --log-format LIST ### 5290 5291 Comma separated list of log format options. `date`, `time`, `microseconds`, `longfile`, `shortfile`, `UTC`. The default is "`date`,`time`". 5292 5293 ### --log-level LEVEL ### 5294 5295 This sets the log level for rclone. The default log level is `NOTICE`. 5296 5297 `DEBUG` is equivalent to `-vv`. It outputs lots of debug info - useful 5298 for bug reports and really finding out what rclone is doing. 5299 5300 `INFO` is equivalent to `-v`. It outputs information about each transfer 5301 and prints stats once a minute by default. 5302 5303 `NOTICE` is the default log level if no logging flags are supplied. It 5304 outputs very little when things are working normally. It outputs 5305 warnings and significant events. 5306 5307 `ERROR` is equivalent to `-q`. It only outputs error messages. 5308 5309 ### --use-json-log ### 5310 5311 This switches the log format to JSON for rclone. The fields of json log 5312 are level, msg, source, time. 5313 5314 ### --low-level-retries NUMBER ### 5315 5316 This controls the number of low level retries rclone does. 5317 5318 A low level retry is used to retry a failing operation - typically one 5319 HTTP request. This might be uploading a chunk of a big file for 5320 example. You will see low level retries in the log with the `-v` 5321 flag. 5322 5323 This shouldn't need to be changed from the default in normal operations. 5324 However, if you get a lot of low level retries you may wish 5325 to reduce the value so rclone moves on to a high level retry (see the 5326 `--retries` flag) quicker. 5327 5328 Disable low level retries with `--low-level-retries 1`. 5329 5330 ### --max-backlog=N ### 5331 5332 This is the maximum allowable backlog of files in a sync/copy/move 5333 queued for being checked or transferred. 5334 5335 This can be set arbitrarily large. It will only use memory when the 5336 queue is in use. Note that it will use in the order of N kB of memory 5337 when the backlog is in use. 5338 5339 Setting this large allows rclone to calculate how many files are 5340 pending more accurately, give a more accurate estimated finish 5341 time and make `--order-by` work more accurately. 5342 5343 Setting this small will make rclone more synchronous to the listings 5344 of the remote which may be desirable. 5345 5346 ### --max-delete=N ### 5347 5348 This tells rclone not to delete more than N files. If that limit is 5349 exceeded then a fatal error will be generated and rclone will stop the 5350 operation in progress. 5351 5352 ### --max-depth=N ### 5353 5354 This modifies the recursion depth for all the commands except purge. 5355 5356 So if you do `rclone --max-depth 1 ls remote:path` you will see only 5357 the files in the top level directory. Using `--max-depth 2` means you 5358 will see all the files in first two directory levels and so on. 5359 5360 For historical reasons the `lsd` command defaults to using a 5361 `--max-depth` of 1 - you can override this with the command line flag. 5362 5363 You can use this command to disable recursion (with `--max-depth 1`). 5364 5365 Note that if you use this with `sync` and `--delete-excluded` the 5366 files not recursed through are considered excluded and will be deleted 5367 on the destination. Test first with `--dry-run` if you are not sure 5368 what will happen. 5369 5370 ### --max-duration=TIME ### 5371 5372 Rclone will stop scheduling new transfers when it has run for the 5373 duration specified. 5374 5375 Defaults to off. 5376 5377 When the limit is reached any existing transfers will complete. 5378 5379 Rclone won't exit with an error if the transfer limit is reached. 5380 5381 ### --max-transfer=SIZE ### 5382 5383 Rclone will stop transferring when it has reached the size specified. 5384 Defaults to off. 5385 5386 When the limit is reached all transfers will stop immediately. 5387 5388 Rclone will exit with exit code 8 if the transfer limit is reached. 5389 5390 ### --modify-window=TIME ### 5391 5392 When checking whether a file has been modified, this is the maximum 5393 allowed time difference that a file can have and still be considered 5394 equivalent. 5395 5396 The default is `1ns` unless this is overridden by a remote. For 5397 example OS X only stores modification times to the nearest second so 5398 if you are reading and writing to an OS X filing system this will be 5399 `1s` by default. 5400 5401 This command line flag allows you to override that computed default. 5402 5403 ### --multi-thread-cutoff=SIZE ### 5404 5405 When downloading files to the local backend above this size, rclone 5406 will use multiple threads to download the file. (default 250M) 5407 5408 Rclone preallocates the file (using `fallocate(FALLOC_FL_KEEP_SIZE)` 5409 on unix or `NTSetInformationFile` on Windows both of which takes no 5410 time) then each thread writes directly into the file at the correct 5411 place. This means that rclone won't create fragmented or sparse files 5412 and there won't be any assembly time at the end of the transfer. 5413 5414 The number of threads used to dowload is controlled by 5415 `--multi-thread-streams`. 5416 5417 Use `-vv` if you wish to see info about the threads. 5418 5419 This will work with the `sync`/`copy`/`move` commands and friends 5420 `copyto`/`moveto`. Multi thread downloads will be used with `rclone 5421 mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or 5422 above. 5423 5424 **NB** that this **only** works for a local destination but will work 5425 with any source. 5426 5427 **NB** that multi thread copies are disabled for local to local copies 5428 as they are faster without unless `--multi-thread-streams` is set 5429 explicitly. 5430 5431 ### --multi-thread-streams=N ### 5432 5433 When using multi thread downloads (see above `--multi-thread-cutoff`) 5434 this sets the maximum number of streams to use. Set to `0` to disable 5435 multi thread downloads. (Default 4) 5436 5437 Exactly how many streams rclone uses for the download depends on the 5438 size of the file. To calculate the number of download streams Rclone 5439 divides the size of the file by the `--multi-thread-cutoff` and rounds 5440 up, up to the maximum set with `--multi-thread-streams`. 5441 5442 So if `--multi-thread-cutoff 250MB` and `--multi-thread-streams 4` are 5443 in effect (the defaults): 5444 5445 - 0MB.250MB files will be downloaded with 1 stream 5446 - 250MB..500MB files will be downloaded with 2 streams 5447 - 500MB..750MB files will be downloaded with 3 streams 5448 - 750MB+ files will be downloaded with 4 streams 5449 5450 ### --no-check-dest ### 5451 5452 The `--no-check-dest` can be used with `move` or `copy` and it causes 5453 rclone not to check the destination at all when copying files. 5454 5455 This means that: 5456 5457 - the destination is not listed minimising the API calls 5458 - files are always transferred 5459 - this can cause duplicates on remotes which allow it (eg Google Drive) 5460 - `--retries 1` is recommended otherwise you'll transfer everything again on a retry 5461 5462 This flag is useful to minimise the transactions if you know that none 5463 of the files are on the destination. 5464 5465 This is a specialized flag which should be ignored by most users! 5466 5467 ### --no-gzip-encoding ### 5468 5469 Don't set `Accept-Encoding: gzip`. This means that rclone won't ask 5470 the server for compressed files automatically. Useful if you've set 5471 the server to return files with `Content-Encoding: gzip` but you 5472 uploaded compressed files. 5473 5474 There is no need to set this in normal operation, and doing so will 5475 decrease the network transfer efficiency of rclone. 5476 5477 ### --no-traverse ### 5478 5479 The `--no-traverse` flag controls whether the destination file system 5480 is traversed when using the `copy` or `move` commands. 5481 `--no-traverse` is not compatible with `sync` and will be ignored if 5482 you supply it with `sync`. 5483 5484 If you are only copying a small number of files (or are filtering most 5485 of the files) and/or have a large number of files on the destination 5486 then `--no-traverse` will stop rclone listing the destination and save 5487 time. 5488 5489 However, if you are copying a large number of files, especially if you 5490 are doing a copy where lots of the files under consideration haven't 5491 changed and won't need copying then you shouldn't use `--no-traverse`. 5492 5493 See [rclone copy](https://rclone.org/commands/rclone_copy/) for an example of how to use it. 5494 5495 ### --no-update-modtime ### 5496 5497 When using this flag, rclone won't update modification times of remote 5498 files if they are incorrect as it would normally. 5499 5500 This can be used if the remote is being synced with another tool also 5501 (eg the Google Drive client). 5502 5503 ### --order-by string ### 5504 5505 The `--order-by` flag controls the order in which files in the backlog 5506 are processed in `rclone sync`, `rclone copy` and `rclone move`. 5507 5508 The order by string is constructed like this. The first part 5509 describes what aspect is being measured: 5510 5511 - `size` - order by the size of the files 5512 - `name` - order by the full path of the files 5513 - `modtime` - order by the modification date of the files 5514 5515 This can have a modifier appended with a comma: 5516 5517 - `ascending` or `asc` - order so that the smallest (or oldest) is processed first 5518 - `descending` or `desc` - order so that the largest (or newest) is processed first 5519 5520 If no modifier is supplied then the order is `ascending`. 5521 5522 For example 5523 5524 - `--order-by size,desc` - send the largest files first 5525 - `--order-by modtime,ascending` - send the oldest files first 5526 - `--order-by name` - send the files with alphabetically by path first 5527 5528 If the `--order-by` flag is not supplied or it is supplied with an 5529 empty string then the default ordering will be used which is as 5530 scanned. With `--checkers 1` this is mostly alphabetical, however 5531 with the default `--checkers 8` it is somewhat random. 5532 5533 #### Limitations 5534 5535 The `--order-by` flag does not do a separate pass over the data. This 5536 means that it may transfer some files out of the order specified if 5537 5538 - there are no files in the backlog or the source has not been fully scanned yet 5539 - there are more than [--max-backlog](#max-backlog-n) files in the backlog 5540 5541 Rclone will do its best to transfer the best file it has so in 5542 practice this should not cause a problem. Think of `--order-by` as 5543 being more of a best efforts flag rather than a perfect ordering. 5544 5545 ### --password-command SpaceSepList ### 5546 5547 This flag supplies a program which should supply the config password 5548 when run. This is an alternative to rclone prompting for the password 5549 or setting the `RCLONE_CONFIG_PASS` variable. 5550 5551 The argument to this should be a command with a space separated list 5552 of arguments. If one of the arguments has a space in then enclose it 5553 in `"`, if you want a literal `"` in an argument then enclose the 5554 argument in `"` and double the `"`. See [CSV encoding](https://godoc.org/encoding/csv) 5555 for more info. 5556 5557 Eg 5558 5559 --password-command echo hello 5560 --password-command echo "hello with space" 5561 --password-command echo "hello with ""quotes"" and space" 5562 5563 See the [Configuration Encryption](#configuration-encryption) for more info. 5564 5565 ### -P, --progress ### 5566 5567 This flag makes rclone update the stats in a static block in the 5568 terminal providing a realtime overview of the transfer. 5569 5570 Any log messages will scroll above the static block. Log messages 5571 will push the static block down to the bottom of the terminal where it 5572 will stay. 5573 5574 Normally this is updated every 500mS but this period can be overridden 5575 with the `--stats` flag. 5576 5577 This can be used with the `--stats-one-line` flag for a simpler 5578 display. 5579 5580 Note: On Windows until [this bug](https://github.com/Azure/go-ansiterm/issues/26) 5581 is fixed all non-ASCII characters will be replaced with `.` when 5582 `--progress` is in use. 5583 5584 ### -q, --quiet ### 5585 5586 Normally rclone outputs stats and a completion message. If you set 5587 this flag it will make as little output as possible. 5588 5589 ### --retries int ### 5590 5591 Retry the entire sync if it fails this many times it fails (default 3). 5592 5593 Some remotes can be unreliable and a few retries help pick up the 5594 files which didn't get transferred because of errors. 5595 5596 Disable retries with `--retries 1`. 5597 5598 ### --retries-sleep=TIME ### 5599 5600 This sets the interval between each retry specified by `--retries` 5601 5602 The default is 0. Use 0 to disable. 5603 5604 ### --size-only ### 5605 5606 Normally rclone will look at modification time and size of files to 5607 see if they are equal. If you set this flag then rclone will check 5608 only the size. 5609 5610 This can be useful transferring files from Dropbox which have been 5611 modified by the desktop sync client which doesn't set checksums of 5612 modification times in the same way as rclone. 5613 5614 ### --stats=TIME ### 5615 5616 Commands which transfer data (`sync`, `copy`, `copyto`, `move`, 5617 `moveto`) will print data transfer stats at regular intervals to show 5618 their progress. 5619 5620 This sets the interval. 5621 5622 The default is `1m`. Use 0 to disable. 5623 5624 If you set the stats interval then all commands can show stats. This 5625 can be useful when running other commands, `check` or `mount` for 5626 example. 5627 5628 Stats are logged at `INFO` level by default which means they won't 5629 show at default log level `NOTICE`. Use `--stats-log-level NOTICE` or 5630 `-v` to make them show. See the [Logging section](#logging) for more 5631 info on log levels. 5632 5633 Note that on macOS you can send a SIGINFO (which is normally ctrl-T in 5634 the terminal) to make the stats print immediately. 5635 5636 ### --stats-file-name-length integer ### 5637 By default, the `--stats` output will truncate file names and paths longer 5638 than 40 characters. This is equivalent to providing 5639 `--stats-file-name-length 40`. Use `--stats-file-name-length 0` to disable 5640 any truncation of file names printed by stats. 5641 5642 ### --stats-log-level string ### 5643 5644 Log level to show `--stats` output at. This can be `DEBUG`, `INFO`, 5645 `NOTICE`, or `ERROR`. The default is `INFO`. This means at the 5646 default level of logging which is `NOTICE` the stats won't show - if 5647 you want them to then use `--stats-log-level NOTICE`. See the [Logging 5648 section](#logging) for more info on log levels. 5649 5650 ### --stats-one-line ### 5651 5652 When this is specified, rclone condenses the stats into a single line 5653 showing the most important stats only. 5654 5655 ### --stats-one-line-date ### 5656 5657 When this is specified, rclone enables the single-line stats and prepends 5658 the display with a date string. The default is `2006/01/02 15:04:05 - ` 5659 5660 ### --stats-one-line-date-format ### 5661 5662 When this is specified, rclone enables the single-line stats and prepends 5663 the display with a user-supplied date string. The date string MUST be 5664 enclosed in quotes. Follow [golang specs](https://golang.org/pkg/time/#Time.Format) for 5665 date formatting syntax. 5666 5667 ### --stats-unit=bits|bytes ### 5668 5669 By default, data transfer rates will be printed in bytes/second. 5670 5671 This option allows the data rate to be printed in bits/second. 5672 5673 Data transfer volume will still be reported in bytes. 5674 5675 The rate is reported as a binary unit, not SI unit. So 1 Mbit/s 5676 equals 1,048,576 bits/s and not 1,000,000 bits/s. 5677 5678 The default is `bytes`. 5679 5680 ### --suffix=SUFFIX ### 5681 5682 When using `sync`, `copy` or `move` any files which would have been 5683 overwritten or deleted will have the suffix added to them. If there 5684 is a file with the same path (after the suffix has been added), then 5685 it will be overwritten. 5686 5687 The remote in use must support server side move or copy and you must 5688 use the same remote as the destination of the sync. 5689 5690 This is for use with files to add the suffix in the current directory 5691 or with `--backup-dir`. See `--backup-dir` for more info. 5692 5693 For example 5694 5695 rclone sync /path/to/local/file remote:current --suffix .bak 5696 5697 will sync `/path/to/local` to `remote:current`, but for any files 5698 which would have been updated or deleted have .bak added. 5699 5700 ### --suffix-keep-extension ### 5701 5702 When using `--suffix`, setting this causes rclone put the SUFFIX 5703 before the extension of the files that it backs up rather than after. 5704 5705 So let's say we had `--suffix -2019-01-01`, without the flag `file.txt` 5706 would be backed up to `file.txt-2019-01-01` and with the flag it would 5707 be backed up to `file-2019-01-01.txt`. This can be helpful to make 5708 sure the suffixed files can still be opened. 5709 5710 ### --syslog ### 5711 5712 On capable OSes (not Windows or Plan9) send all log output to syslog. 5713 5714 This can be useful for running rclone in a script or `rclone mount`. 5715 5716 ### --syslog-facility string ### 5717 5718 If using `--syslog` this sets the syslog facility (eg `KERN`, `USER`). 5719 See `man syslog` for a list of possible facilities. The default 5720 facility is `DAEMON`. 5721 5722 ### --tpslimit float ### 5723 5724 Limit HTTP transactions per second to this. Default is 0 which is used 5725 to mean unlimited transactions per second. 5726 5727 For example to limit rclone to 10 HTTP transactions per second use 5728 `--tpslimit 10`, or to 1 transaction every 2 seconds use `--tpslimit 5729 0.5`. 5730 5731 Use this when the number of transactions per second from rclone is 5732 causing a problem with the cloud storage provider (eg getting you 5733 banned or rate limited). 5734 5735 This can be very useful for `rclone mount` to control the behaviour of 5736 applications using it. 5737 5738 See also `--tpslimit-burst`. 5739 5740 ### --tpslimit-burst int ### 5741 5742 Max burst of transactions for `--tpslimit`. (default 1) 5743 5744 Normally `--tpslimit` will do exactly the number of transaction per 5745 second specified. However if you supply `--tps-burst` then rclone can 5746 save up some transactions from when it was idle giving a burst of up 5747 to the parameter supplied. 5748 5749 For example if you provide `--tpslimit-burst 10` then if rclone has 5750 been idle for more than 10*`--tpslimit` then it can do 10 transactions 5751 very quickly before they are limited again. 5752 5753 This may be used to increase performance of `--tpslimit` without 5754 changing the long term average number of transactions per second. 5755 5756 ### --track-renames ### 5757 5758 By default, rclone doesn't keep track of renamed files, so if you 5759 rename a file locally then sync it to a remote, rclone will delete the 5760 old file on the remote and upload a new copy. 5761 5762 If you use this flag, and the remote supports server side copy or 5763 server side move, and the source and destination have a compatible 5764 hash, then this will track renames during `sync` 5765 operations and perform renaming server-side. 5766 5767 Files will be matched by size and hash - if both match then a rename 5768 will be considered. 5769 5770 If the destination does not support server-side copy or move, rclone 5771 will fall back to the default behaviour and log an error level message 5772 to the console. Note: Encrypted destinations are not supported 5773 by `--track-renames`. 5774 5775 Note that `--track-renames` is incompatible with `--no-traverse` and 5776 that it uses extra memory to keep track of all the rename candidates. 5777 5778 Note also that `--track-renames` is incompatible with 5779 `--delete-before` and will select `--delete-after` instead of 5780 `--delete-during`. 5781 5782 ### --delete-(before,during,after) ### 5783 5784 This option allows you to specify when files on your destination are 5785 deleted when you sync folders. 5786 5787 Specifying the value `--delete-before` will delete all files present 5788 on the destination, but not on the source *before* starting the 5789 transfer of any new or updated files. This uses two passes through the 5790 file systems, one for the deletions and one for the copies. 5791 5792 Specifying `--delete-during` will delete files while checking and 5793 uploading files. This is the fastest option and uses the least memory. 5794 5795 Specifying `--delete-after` (the default value) will delay deletion of 5796 files until all new/updated files have been successfully transferred. 5797 The files to be deleted are collected in the copy pass then deleted 5798 after the copy pass has completed successfully. The files to be 5799 deleted are held in memory so this mode may use more memory. This is 5800 the safest mode as it will only delete files if there have been no 5801 errors subsequent to that. If there have been errors before the 5802 deletions start then you will get the message `not deleting files as 5803 there were IO errors`. 5804 5805 ### --fast-list ### 5806 5807 When doing anything which involves a directory listing (eg `sync`, 5808 `copy`, `ls` - in fact nearly every command), rclone normally lists a 5809 directory and processes it before using more directory lists to 5810 process any subdirectories. This can be parallelised and works very 5811 quickly using the least amount of memory. 5812 5813 However, some remotes have a way of listing all files beneath a 5814 directory in one (or a small number) of transactions. These tend to 5815 be the bucket based remotes (eg S3, B2, GCS, Swift, Hubic). 5816 5817 If you use the `--fast-list` flag then rclone will use this method for 5818 listing directories. This will have the following consequences for 5819 the listing: 5820 5821 * It **will** use fewer transactions (important if you pay for them) 5822 * It **will** use more memory. Rclone has to load the whole listing into memory. 5823 * It *may* be faster because it uses fewer transactions 5824 * It *may* be slower because it can't be parallelized 5825 5826 rclone should always give identical results with and without 5827 `--fast-list`. 5828 5829 If you pay for transactions and can fit your entire sync listing into 5830 memory then `--fast-list` is recommended. If you have a very big sync 5831 to do then don't use `--fast-list` otherwise you will run out of 5832 memory. 5833 5834 If you use `--fast-list` on a remote which doesn't support it, then 5835 rclone will just ignore it. 5836 5837 ### --timeout=TIME ### 5838 5839 This sets the IO idle timeout. If a transfer has started but then 5840 becomes idle for this long it is considered broken and disconnected. 5841 5842 The default is `5m`. Set to 0 to disable. 5843 5844 ### --transfers=N ### 5845 5846 The number of file transfers to run in parallel. It can sometimes be 5847 useful to set this to a smaller number if the remote is giving a lot 5848 of timeouts or bigger if you have lots of bandwidth and a fast remote. 5849 5850 The default is to run 4 file transfers in parallel. 5851 5852 ### -u, --update ### 5853 5854 This forces rclone to skip any files which exist on the destination 5855 and have a modified time that is newer than the source file. 5856 5857 This can be useful when transferring to a remote which doesn't support 5858 mod times directly (or when using `--use-server-modtime` to avoid extra 5859 API calls) as it is more accurate than a `--size-only` check and faster 5860 than using `--checksum`. 5861 5862 If an existing destination file has a modification time equal (within 5863 the computed modify window precision) to the source file's, it will be 5864 updated if the sizes are different. If `--checksum` is set then 5865 rclone will update the destination if the checksums differ too. 5866 5867 If an existing destination file is older than the source file then 5868 it will be updated if the size or checksum differs from the source file. 5869 5870 On remotes which don't support mod time directly (or when using 5871 `--use-server-modtime`) the time checked will be the uploaded time. 5872 This means that if uploading to one of these remotes, rclone will skip 5873 any files which exist on the destination and have an uploaded time that 5874 is newer than the modification time of the source file. 5875 5876 ### --use-mmap ### 5877 5878 If this flag is set then rclone will use anonymous memory allocated by 5879 mmap on Unix based platforms and VirtualAlloc on Windows for its 5880 transfer buffers (size controlled by `--buffer-size`). Memory 5881 allocated like this does not go on the Go heap and can be returned to 5882 the OS immediately when it is finished with. 5883 5884 If this flag is not set then rclone will allocate and free the buffers 5885 using the Go memory allocator which may use more memory as memory 5886 pages are returned less aggressively to the OS. 5887 5888 It is possible this does not work well on all platforms so it is 5889 disabled by default; in the future it may be enabled by default. 5890 5891 ### --use-server-modtime ### 5892 5893 Some object-store backends (e.g, Swift, S3) do not preserve file modification 5894 times (modtime). On these backends, rclone stores the original modtime as 5895 additional metadata on the object. By default it will make an API call to 5896 retrieve the metadata when the modtime is needed by an operation. 5897 5898 Use this flag to disable the extra API call and rely instead on the server's 5899 modified time. In cases such as a local to remote sync using `--update`, 5900 knowing the local file is newer than the time it was last uploaded to the 5901 remote is sufficient. In those cases, this flag can speed up the process and 5902 reduce the number of API calls necessary. 5903 5904 Using this flag on a sync operation without also using `--update` would cause 5905 all files modified at any time other than the last upload time to be uploaded 5906 again, which is probably not what you want. 5907 5908 ### -v, -vv, --verbose ### 5909 5910 With `-v` rclone will tell you about each file that is transferred and 5911 a small number of significant events. 5912 5913 With `-vv` rclone will become very verbose telling you about every 5914 file it considers and transfers. Please send bug reports with a log 5915 with this setting. 5916 5917 ### -V, --version ### 5918 5919 Prints the version number 5920 5921 SSL/TLS options 5922 --------------- 5923 5924 The outoing SSL/TLS connections rclone makes can be controlled with 5925 these options. For example this can be very useful with the HTTP or 5926 WebDAV backends. Rclone HTTP servers have their own set of 5927 configuration for SSL/TLS which you can find in their documentation. 5928 5929 ### --ca-cert string 5930 5931 This loads the PEM encoded certificate authority certificate and uses 5932 it to verify the certificates of the servers rclone connects to. 5933 5934 If you have generated certificates signed with a local CA then you 5935 will need this flag to connect to servers using those certificates. 5936 5937 ### --client-cert string 5938 5939 This loads the PEM encoded client side certificate. 5940 5941 This is used for [mutual TLS authentication](https://en.wikipedia.org/wiki/Mutual_authentication). 5942 5943 The `--client-key` flag is required too when using this. 5944 5945 ### --client-key string 5946 5947 This loads the PEM encoded client side private key used for mutual TLS 5948 authentication. Used in conjunction with `--client-cert`. 5949 5950 ### --no-check-certificate=true/false ### 5951 5952 `--no-check-certificate` controls whether a client verifies the 5953 server's certificate chain and host name. 5954 If `--no-check-certificate` is true, TLS accepts any certificate 5955 presented by the server and any host name in that certificate. 5956 In this mode, TLS is susceptible to man-in-the-middle attacks. 5957 5958 This option defaults to `false`. 5959 5960 **This should be used only for testing.** 5961 5962 Configuration Encryption 5963 ------------------------ 5964 Your configuration file contains information for logging in to 5965 your cloud services. This means that you should keep your 5966 `.rclone.conf` file in a secure location. 5967 5968 If you are in an environment where that isn't possible, you can 5969 add a password to your configuration. This means that you will 5970 have to supply the password every time you start rclone. 5971 5972 To add a password to your rclone configuration, execute `rclone config`. 5973 5974 ``` 5975 >rclone config 5976 Current remotes: 5977 5978 e) Edit existing remote 5979 n) New remote 5980 d) Delete remote 5981 s) Set configuration password 5982 q) Quit config 5983 e/n/d/s/q> 5984 ``` 5985 5986 Go into `s`, Set configuration password: 5987 ``` 5988 e/n/d/s/q> s 5989 Your configuration is not encrypted. 5990 If you add a password, you will protect your login information to cloud services. 5991 a) Add Password 5992 q) Quit to main menu 5993 a/q> a 5994 Enter NEW configuration password: 5995 password: 5996 Confirm NEW password: 5997 password: 5998 Password set 5999 Your configuration is encrypted. 6000 c) Change Password 6001 u) Unencrypt configuration 6002 q) Quit to main menu 6003 c/u/q> 6004 ``` 6005 6006 Your configuration is now encrypted, and every time you start rclone 6007 you will have to supply the password. See below for details. 6008 In the same menu, you can change the password or completely remove 6009 encryption from your configuration. 6010 6011 There is no way to recover the configuration if you lose your password. 6012 6013 rclone uses [nacl secretbox](https://godoc.org/golang.org/x/crypto/nacl/secretbox) 6014 which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate 6015 your configuration with secret-key cryptography. 6016 The password is SHA-256 hashed, which produces the key for secretbox. 6017 The hashed password is not stored. 6018 6019 While this provides very good security, we do not recommend storing 6020 your encrypted rclone configuration in public if it contains sensitive 6021 information, maybe except if you use a very strong password. 6022 6023 If it is safe in your environment, you can set the `RCLONE_CONFIG_PASS` 6024 environment variable to contain your password, in which case it will be 6025 used for decrypting the configuration. 6026 6027 You can set this for a session from a script. For unix like systems 6028 save this to a file called `set-rclone-password`: 6029 6030 ``` 6031 #!/bin/echo Source this file don't run it 6032 6033 read -s RCLONE_CONFIG_PASS 6034 export RCLONE_CONFIG_PASS 6035 ``` 6036 6037 Then source the file when you want to use it. From the shell you 6038 would do `source set-rclone-password`. It will then ask you for the 6039 password and set it in the environment variable. 6040 6041 An alternate means of supplying the password is to provide a script 6042 which will retrieve the password and print on standard output. This 6043 script should have a fully specified path name and not rely on any 6044 environment variables. The script is supplied either via 6045 `--password-command="..."` command line argument or via the 6046 `RCLONE_PASSWORD_COMMAND` environment variable. 6047 6048 One useful example of this is using the `passwordstore` application 6049 to retrieve the password: 6050 6051 ``` 6052 export RCLONE_PASSWORD_COMMAND="pass rclone/config" 6053 ``` 6054 6055 If the `passwordstore` password manager holds the password for the 6056 rclone configuration, using the script method means the password 6057 is primarily protected by the `passwordstore` system, and is never 6058 embedded in the clear in scripts, nor available for examination 6059 using the standard commands available. It is quite possible with 6060 long running rclone sessions for copies of passwords to be innocently 6061 captured in log files or terminal scroll buffers, etc. Using the 6062 script method of supplying the password enhances the security of 6063 the config password considerably. 6064 6065 If you are running rclone inside a script, unless you are using the 6066 `--password-command` method, you might want to disable 6067 password prompts. To do that, pass the parameter 6068 `--ask-password=false` to rclone. This will make rclone fail instead 6069 of asking for a password if `RCLONE_CONFIG_PASS` doesn't contain 6070 a valid password, and `--password-command` has not been supplied. 6071 6072 6073 Developer options 6074 ----------------- 6075 6076 These options are useful when developing or debugging rclone. There 6077 are also some more remote specific options which aren't documented 6078 here which are used for testing. These start with remote name eg 6079 `--drive-test-option` - see the docs for the remote in question. 6080 6081 ### --cpuprofile=FILE ### 6082 6083 Write CPU profile to file. This can be analysed with `go tool pprof`. 6084 6085 #### --dump flag,flag,flag #### 6086 6087 The `--dump` flag takes a comma separated list of flags to dump info 6088 about. 6089 6090 Note that some headers including `Accept-Encoding` as shown may not 6091 be correct in the request and the response may not show `Content-Encoding` 6092 if the go standard libraries auto gzip encoding was in effect. In this case 6093 the body of the request will be gunzipped before showing it. 6094 6095 The available flags are: 6096 6097 #### --dump headers #### 6098 6099 Dump HTTP headers with `Authorization:` lines removed. May still 6100 contain sensitive info. Can be very verbose. Useful for debugging 6101 only. 6102 6103 Use `--dump auth` if you do want the `Authorization:` headers. 6104 6105 #### --dump bodies #### 6106 6107 Dump HTTP headers and bodies - may contain sensitive info. Can be 6108 very verbose. Useful for debugging only. 6109 6110 Note that the bodies are buffered in memory so don't use this for 6111 enormous files. 6112 6113 #### --dump requests #### 6114 6115 Like `--dump bodies` but dumps the request bodies and the response 6116 headers. Useful for debugging download problems. 6117 6118 #### --dump responses #### 6119 6120 Like `--dump bodies` but dumps the response bodies and the request 6121 headers. Useful for debugging upload problems. 6122 6123 #### --dump auth #### 6124 6125 Dump HTTP headers - will contain sensitive info such as 6126 `Authorization:` headers - use `--dump headers` to dump without 6127 `Authorization:` headers. Can be very verbose. Useful for debugging 6128 only. 6129 6130 #### --dump filters #### 6131 6132 Dump the filters to the output. Useful to see exactly what include 6133 and exclude options are filtering on. 6134 6135 #### --dump goroutines #### 6136 6137 This dumps a list of the running go-routines at the end of the command 6138 to standard output. 6139 6140 #### --dump openfiles #### 6141 6142 This dumps a list of the open files at the end of the command. It 6143 uses the `lsof` command to do that so you'll need that installed to 6144 use it. 6145 6146 ### --memprofile=FILE ### 6147 6148 Write memory profile to file. This can be analysed with `go tool pprof`. 6149 6150 Filtering 6151 --------- 6152 6153 For the filtering options 6154 6155 * `--delete-excluded` 6156 * `--filter` 6157 * `--filter-from` 6158 * `--exclude` 6159 * `--exclude-from` 6160 * `--include` 6161 * `--include-from` 6162 * `--files-from` 6163 * `--min-size` 6164 * `--max-size` 6165 * `--min-age` 6166 * `--max-age` 6167 * `--dump filters` 6168 6169 See the [filtering section](https://rclone.org/filtering/). 6170 6171 Remote control 6172 -------------- 6173 6174 For the remote control options and for instructions on how to remote control rclone 6175 6176 * `--rc` 6177 * and anything starting with `--rc-` 6178 6179 See [the remote control section](https://rclone.org/rc/). 6180 6181 Logging 6182 ------- 6183 6184 rclone has 4 levels of logging, `ERROR`, `NOTICE`, `INFO` and `DEBUG`. 6185 6186 By default, rclone logs to standard error. This means you can redirect 6187 standard error and still see the normal output of rclone commands (eg 6188 `rclone ls`). 6189 6190 By default, rclone will produce `Error` and `Notice` level messages. 6191 6192 If you use the `-q` flag, rclone will only produce `Error` messages. 6193 6194 If you use the `-v` flag, rclone will produce `Error`, `Notice` and 6195 `Info` messages. 6196 6197 If you use the `-vv` flag, rclone will produce `Error`, `Notice`, 6198 `Info` and `Debug` messages. 6199 6200 You can also control the log levels with the `--log-level` flag. 6201 6202 If you use the `--log-file=FILE` option, rclone will redirect `Error`, 6203 `Info` and `Debug` messages along with standard error to FILE. 6204 6205 If you use the `--syslog` flag then rclone will log to syslog and the 6206 `--syslog-facility` control which facility it uses. 6207 6208 Rclone prefixes all log messages with their level in capitals, eg INFO 6209 which makes it easy to grep the log file for different kinds of 6210 information. 6211 6212 Exit Code 6213 --------- 6214 6215 If any errors occur during the command execution, rclone will exit with a 6216 non-zero exit code. This allows scripts to detect when rclone 6217 operations have failed. 6218 6219 During the startup phase, rclone will exit immediately if an error is 6220 detected in the configuration. There will always be a log message 6221 immediately before exiting. 6222 6223 When rclone is running it will accumulate errors as it goes along, and 6224 only exit with a non-zero exit code if (after retries) there were 6225 still failed transfers. For every error counted there will be a high 6226 priority log message (visible with `-q`) showing the message and 6227 which file caused the problem. A high priority message is also shown 6228 when starting a retry so the user can see that any previous error 6229 messages may not be valid after the retry. If rclone has done a retry 6230 it will log a high priority message if the retry was successful. 6231 6232 ### List of exit codes ### 6233 * `0` - success 6234 * `1` - Syntax or usage error 6235 * `2` - Error not otherwise categorised 6236 * `3` - Directory not found 6237 * `4` - File not found 6238 * `5` - Temporary error (one that more retries might fix) (Retry errors) 6239 * `6` - Less serious errors (like 461 errors from dropbox) (NoRetry errors) 6240 * `7` - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors) 6241 * `8` - Transfer exceeded - limit set by --max-transfer reached 6242 6243 Environment Variables 6244 --------------------- 6245 6246 Rclone can be configured entirely using environment variables. These 6247 can be used to set defaults for options or config file entries. 6248 6249 ### Options ### 6250 6251 Every option in rclone can have its default set by environment 6252 variable. 6253 6254 To find the name of the environment variable, first, take the long 6255 option name, strip the leading `--`, change `-` to `_`, make 6256 upper case and prepend `RCLONE_`. 6257 6258 For example, to always set `--stats 5s`, set the environment variable 6259 `RCLONE_STATS=5s`. If you set stats on the command line this will 6260 override the environment variable setting. 6261 6262 Or to always use the trash in drive `--drive-use-trash`, set 6263 `RCLONE_DRIVE_USE_TRASH=true`. 6264 6265 The same parser is used for the options and the environment variables 6266 so they take exactly the same form. 6267 6268 ### Config file ### 6269 6270 You can set defaults for values in the config file on an individual 6271 remote basis. If you want to use this feature, you will need to 6272 discover the name of the config items that you want. The easiest way 6273 is to run through `rclone config` by hand, then look in the config 6274 file to see what the values are (the config file can be found by 6275 looking at the help for `--config` in `rclone help`). 6276 6277 To find the name of the environment variable, you need to set, take 6278 `RCLONE_CONFIG_` + name of remote + `_` + name of config file option 6279 and make it all uppercase. 6280 6281 For example, to configure an S3 remote named `mys3:` without a config 6282 file (using unix ways of setting environment variables): 6283 6284 ``` 6285 $ export RCLONE_CONFIG_MYS3_TYPE=s3 6286 $ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX 6287 $ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX 6288 $ rclone lsd MYS3: 6289 -1 2016-09-21 12:54:21 -1 my-bucket 6290 $ rclone listremotes | grep mys3 6291 mys3: 6292 ``` 6293 6294 Note that if you want to create a remote using environment variables 6295 you must create the `..._TYPE` variable as above. 6296 6297 ### Other environment variables ### 6298 6299 * RCLONE_CONFIG_PASS` set to contain your config file password (see [Configuration Encryption](#configuration-encryption) section) 6300 * HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions thereof). 6301 * HTTPS_PROXY takes precedence over HTTP_PROXY for https requests. 6302 * The environment values may be either a complete URL or a "host[:port]" for, in which case the "http" scheme is assumed. 6303 6304 # Configuring rclone on a remote / headless machine # 6305 6306 Some of the configurations (those involving oauth2) require an 6307 Internet connected web browser. 6308 6309 If you are trying to set rclone up on a remote or headless box with no 6310 browser available on it (eg a NAS or a server in a datacenter) then 6311 you will need to use an alternative means of configuration. There are 6312 two ways of doing it, described below. 6313 6314 ## Configuring using rclone authorize ## 6315 6316 On the headless box 6317 6318 ``` 6319 ... 6320 Remote config 6321 Use auto config? 6322 * Say Y if not sure 6323 * Say N if you are working on a remote or headless machine 6324 y) Yes 6325 n) No 6326 y/n> n 6327 For this to work, you will need rclone available on a machine that has a web browser available. 6328 Execute the following on your machine: 6329 rclone authorize "amazon cloud drive" 6330 Then paste the result below: 6331 result> 6332 ``` 6333 6334 Then on your main desktop machine 6335 6336 ``` 6337 rclone authorize "amazon cloud drive" 6338 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 6339 Log in and authorize rclone for access 6340 Waiting for code... 6341 Got code 6342 Paste the following into your remote machine ---> 6343 SECRET_TOKEN 6344 <---End paste 6345 ``` 6346 6347 Then back to the headless box, paste in the code 6348 6349 ``` 6350 result> SECRET_TOKEN 6351 -------------------- 6352 [acd12] 6353 client_id = 6354 client_secret = 6355 token = SECRET_TOKEN 6356 -------------------- 6357 y) Yes this is OK 6358 e) Edit this remote 6359 d) Delete this remote 6360 y/e/d> 6361 ``` 6362 6363 ## Configuring by copying the config file ## 6364 6365 Rclone stores all of its config in a single configuration file. This 6366 can easily be copied to configure a remote rclone. 6367 6368 So first configure rclone on your desktop machine 6369 6370 rclone config 6371 6372 to set up the config file. 6373 6374 Find the config file by running `rclone config file`, for example 6375 6376 ``` 6377 $ rclone config file 6378 Configuration file is stored at: 6379 /home/user/.rclone.conf 6380 ``` 6381 6382 Now transfer it to the remote box (scp, cut paste, ftp, sftp etc) and 6383 place it in the correct place (use `rclone config file` on the remote 6384 box to find out where). 6385 6386 # Filtering, includes and excludes # 6387 6388 Rclone has a sophisticated set of include and exclude rules. Some of 6389 these are based on patterns and some on other things like file size. 6390 6391 The filters are applied for the `copy`, `sync`, `move`, `ls`, `lsl`, 6392 `md5sum`, `sha1sum`, `size`, `delete` and `check` operations. 6393 Note that `purge` does not obey the filters. 6394 6395 Each path as it passes through rclone is matched against the include 6396 and exclude rules like `--include`, `--exclude`, `--include-from`, 6397 `--exclude-from`, `--filter`, or `--filter-from`. The simplest way to 6398 try them out is using the `ls` command, or `--dry-run` together with 6399 `-v`. 6400 6401 ## Patterns ## 6402 6403 The patterns used to match files for inclusion or exclusion are based 6404 on "file globs" as used by the unix shell. 6405 6406 If the pattern starts with a `/` then it only matches at the top level 6407 of the directory tree, **relative to the root of the remote** (not 6408 necessarily the root of the local drive). If it doesn't start with `/` 6409 then it is matched starting at the **end of the path**, but it will 6410 only match a complete path element: 6411 6412 file.jpg - matches "file.jpg" 6413 - matches "directory/file.jpg" 6414 - doesn't match "afile.jpg" 6415 - doesn't match "directory/afile.jpg" 6416 /file.jpg - matches "file.jpg" in the root directory of the remote 6417 - doesn't match "afile.jpg" 6418 - doesn't match "directory/file.jpg" 6419 6420 **Important** Note that you must use `/` in patterns and not `\` even 6421 if running on Windows. 6422 6423 A `*` matches anything but not a `/`. 6424 6425 *.jpg - matches "file.jpg" 6426 - matches "directory/file.jpg" 6427 - doesn't match "file.jpg/something" 6428 6429 Use `**` to match anything, including slashes (`/`). 6430 6431 dir/** - matches "dir/file.jpg" 6432 - matches "dir/dir1/dir2/file.jpg" 6433 - doesn't match "directory/file.jpg" 6434 - doesn't match "adir/file.jpg" 6435 6436 A `?` matches any character except a slash `/`. 6437 6438 l?ss - matches "less" 6439 - matches "lass" 6440 - doesn't match "floss" 6441 6442 A `[` and `]` together make a character class, such as `[a-z]` or 6443 `[aeiou]` or `[[:alpha:]]`. See the [go regexp 6444 docs](https://golang.org/pkg/regexp/syntax/) for more info on these. 6445 6446 h[ae]llo - matches "hello" 6447 - matches "hallo" 6448 - doesn't match "hullo" 6449 6450 A `{` and `}` define a choice between elements. It should contain a 6451 comma separated list of patterns, any of which might match. These 6452 patterns can contain wildcards. 6453 6454 {one,two}_potato - matches "one_potato" 6455 - matches "two_potato" 6456 - doesn't match "three_potato" 6457 - doesn't match "_potato" 6458 6459 Special characters can be escaped with a `\` before them. 6460 6461 \*.jpg - matches "*.jpg" 6462 \\.jpg - matches "\.jpg" 6463 \[one\].jpg - matches "[one].jpg" 6464 6465 Patterns are case sensitive unless the `--ignore-case` flag is used. 6466 6467 Without `--ignore-case` (default) 6468 6469 potato - matches "potato" 6470 - doesn't match "POTATO" 6471 6472 With `--ignore-case` 6473 6474 potato - matches "potato" 6475 - matches "POTATO" 6476 6477 Note also that rclone filter globs can only be used in one of the 6478 filter command line flags, not in the specification of the remote, so 6479 `rclone copy "remote:dir*.jpg" /path/to/dir` won't work - what is 6480 required is `rclone --include "*.jpg" copy remote:dir /path/to/dir` 6481 6482 ### Directories ### 6483 6484 Rclone keeps track of directories that could match any file patterns. 6485 6486 Eg if you add the include rule 6487 6488 /a/*.jpg 6489 6490 Rclone will synthesize the directory include rule 6491 6492 /a/ 6493 6494 If you put any rules which end in `/` then it will only match 6495 directories. 6496 6497 Directory matches are **only** used to optimise directory access 6498 patterns - you must still match the files that you want to match. 6499 Directory matches won't optimise anything on bucket based remotes (eg 6500 s3, swift, google compute storage, b2) which don't have a concept of 6501 directory. 6502 6503 ### Differences between rsync and rclone patterns ### 6504 6505 Rclone implements bash style `{a,b,c}` glob matching which rsync doesn't. 6506 6507 Rclone always does a wildcard match so `\` must always escape a `\`. 6508 6509 ## How the rules are used ## 6510 6511 Rclone maintains a combined list of include rules and exclude rules. 6512 6513 Each file is matched in order, starting from the top, against the rule 6514 in the list until it finds a match. The file is then included or 6515 excluded according to the rule type. 6516 6517 If the matcher fails to find a match after testing against all the 6518 entries in the list then the path is included. 6519 6520 For example given the following rules, `+` being include, `-` being 6521 exclude, 6522 6523 - secret*.jpg 6524 + *.jpg 6525 + *.png 6526 + file2.avi 6527 - * 6528 6529 This would include 6530 6531 * `file1.jpg` 6532 * `file3.png` 6533 * `file2.avi` 6534 6535 This would exclude 6536 6537 * `secret17.jpg` 6538 * non `*.jpg` and `*.png` 6539 6540 A similar process is done on directory entries before recursing into 6541 them. This only works on remotes which have a concept of directory 6542 (Eg local, google drive, onedrive, amazon drive) and not on bucket 6543 based remotes (eg s3, swift, google compute storage, b2). 6544 6545 ## Adding filtering rules ## 6546 6547 Filtering rules are added with the following command line flags. 6548 6549 ### Repeating options ## 6550 6551 You can repeat the following options to add more than one rule of that 6552 type. 6553 6554 * `--include` 6555 * `--include-from` 6556 * `--exclude` 6557 * `--exclude-from` 6558 * `--filter` 6559 * `--filter-from` 6560 6561 **Important** You should not use `--include*` together with `--exclude*`. 6562 It may produce different results than you expected. In that case try to use: `--filter*`. 6563 6564 Note that all the options of the same type are processed together in 6565 the order above, regardless of what order they were placed on the 6566 command line. 6567 6568 So all `--include` options are processed first in the order they 6569 appeared on the command line, then all `--include-from` options etc. 6570 6571 To mix up the order includes and excludes, the `--filter` flag can be 6572 used. 6573 6574 ### `--exclude` - Exclude files matching pattern ### 6575 6576 Add a single exclude rule with `--exclude`. 6577 6578 This flag can be repeated. See above for the order the flags are 6579 processed in. 6580 6581 Eg `--exclude *.bak` to exclude all bak files from the sync. 6582 6583 ### `--exclude-from` - Read exclude patterns from file ### 6584 6585 Add exclude rules from a file. 6586 6587 This flag can be repeated. See above for the order the flags are 6588 processed in. 6589 6590 Prepare a file like this `exclude-file.txt` 6591 6592 # a sample exclude rule file 6593 *.bak 6594 file2.jpg 6595 6596 Then use as `--exclude-from exclude-file.txt`. This will sync all 6597 files except those ending in `bak` and `file2.jpg`. 6598 6599 This is useful if you have a lot of rules. 6600 6601 ### `--include` - Include files matching pattern ### 6602 6603 Add a single include rule with `--include`. 6604 6605 This flag can be repeated. See above for the order the flags are 6606 processed in. 6607 6608 Eg `--include *.{png,jpg}` to include all `png` and `jpg` files in the 6609 backup and no others. 6610 6611 This adds an implicit `--exclude *` at the very end of the filter 6612 list. This means you can mix `--include` and `--include-from` with the 6613 other filters (eg `--exclude`) but you must include all the files you 6614 want in the include statement. If this doesn't provide enough 6615 flexibility then you must use `--filter-from`. 6616 6617 ### `--include-from` - Read include patterns from file ### 6618 6619 Add include rules from a file. 6620 6621 This flag can be repeated. See above for the order the flags are 6622 processed in. 6623 6624 Prepare a file like this `include-file.txt` 6625 6626 # a sample include rule file 6627 *.jpg 6628 *.png 6629 file2.avi 6630 6631 Then use as `--include-from include-file.txt`. This will sync all 6632 `jpg`, `png` files and `file2.avi`. 6633 6634 This is useful if you have a lot of rules. 6635 6636 This adds an implicit `--exclude *` at the very end of the filter 6637 list. This means you can mix `--include` and `--include-from` with the 6638 other filters (eg `--exclude`) but you must include all the files you 6639 want in the include statement. If this doesn't provide enough 6640 flexibility then you must use `--filter-from`. 6641 6642 ### `--filter` - Add a file-filtering rule ### 6643 6644 This can be used to add a single include or exclude rule. Include 6645 rules start with `+ ` and exclude rules start with `- `. A special 6646 rule called `!` can be used to clear the existing rules. 6647 6648 This flag can be repeated. See above for the order the flags are 6649 processed in. 6650 6651 Eg `--filter "- *.bak"` to exclude all bak files from the sync. 6652 6653 ### `--filter-from` - Read filtering patterns from a file ### 6654 6655 Add include/exclude rules from a file. 6656 6657 This flag can be repeated. See above for the order the flags are 6658 processed in. 6659 6660 Prepare a file like this `filter-file.txt` 6661 6662 # a sample filter rule file 6663 - secret*.jpg 6664 + *.jpg 6665 + *.png 6666 + file2.avi 6667 - /dir/Trash/** 6668 + /dir/** 6669 # exclude everything else 6670 - * 6671 6672 Then use as `--filter-from filter-file.txt`. The rules are processed 6673 in the order that they are defined. 6674 6675 This example will include all `jpg` and `png` files, exclude any files 6676 matching `secret*.jpg` and include `file2.avi`. It will also include 6677 everything in the directory `dir` at the root of the sync, except 6678 `dir/Trash` which it will exclude. Everything else will be excluded 6679 from the sync. 6680 6681 ### `--files-from` - Read list of source-file names ### 6682 6683 This reads a list of file names from the file passed in and **only** 6684 these files are transferred. The **filtering rules are ignored** 6685 completely if you use this option. 6686 6687 `--files-from` expects a list of files as it's input. [rclone lsf](https://rclone.org/commands/rclone_lsf/) 6688 has a compatible format that can be used to export file lists from 6689 remotes. 6690 6691 Rclone will traverse the file system if you use `--files-from`, 6692 effectively using the files in `--files-from` as a set of filters. 6693 Rclone will not error if any of the files are missing. 6694 6695 If you use `--no-traverse` as well as `--files-from` then rclone will 6696 not traverse the destination file system, it will find each file 6697 individually using approximately 1 API call. This can be more 6698 efficient for small lists of files. 6699 6700 This option can be repeated to read from more than one file. These 6701 are read in the order that they are placed on the command line. 6702 6703 Paths within the `--files-from` file will be interpreted as starting 6704 with the root specified in the command. Leading `/` characters are 6705 ignored. 6706 6707 For example, suppose you had `files-from.txt` with this content: 6708 6709 # comment 6710 file1.jpg 6711 subdir/file2.jpg 6712 6713 You could then use it like this: 6714 6715 rclone copy --files-from files-from.txt /home/me/pics remote:pics 6716 6717 This will transfer these files only (if they exist) 6718 6719 /home/me/pics/file1.jpg → remote:pics/file1.jpg 6720 /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg 6721 6722 To take a more complicated example, let's say you had a few files you 6723 want to back up regularly with these absolute paths: 6724 6725 /home/user1/important 6726 /home/user1/dir/file 6727 /home/user2/stuff 6728 6729 To copy these you'd find a common subdirectory - in this case `/home` 6730 and put the remaining files in `files-from.txt` with or without 6731 leading `/`, eg 6732 6733 user1/important 6734 user1/dir/file 6735 user2/stuff 6736 6737 You could then copy these to a remote like this 6738 6739 rclone copy --files-from files-from.txt /home remote:backup 6740 6741 The 3 files will arrive in `remote:backup` with the paths as in the 6742 `files-from.txt` like this: 6743 6744 /home/user1/important → remote:backup/user1/important 6745 /home/user1/dir/file → remote:backup/user1/dir/file 6746 /home/user2/stuff → remote:backup/user2/stuff 6747 6748 You could of course choose `/` as the root too in which case your 6749 `files-from.txt` might look like this. 6750 6751 /home/user1/important 6752 /home/user1/dir/file 6753 /home/user2/stuff 6754 6755 And you would transfer it like this 6756 6757 rclone copy --files-from files-from.txt / remote:backup 6758 6759 In this case there will be an extra `home` directory on the remote: 6760 6761 /home/user1/important → remote:backup/home/user1/important 6762 /home/user1/dir/file → remote:backup/home/user1/dir/file 6763 /home/user2/stuff → remote:backup/home/user2/stuff 6764 6765 ### `--min-size` - Don't transfer any file smaller than this ### 6766 6767 This option controls the minimum size file which will be transferred. 6768 This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be 6769 used. 6770 6771 For example `--min-size 50k` means no files smaller than 50kByte will be 6772 transferred. 6773 6774 ### `--max-size` - Don't transfer any file larger than this ### 6775 6776 This option controls the maximum size file which will be transferred. 6777 This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be 6778 used. 6779 6780 For example `--max-size 1G` means no files larger than 1GByte will be 6781 transferred. 6782 6783 ### `--max-age` - Don't transfer any file older than this ### 6784 6785 This option controls the maximum age of files to transfer. Give in 6786 seconds or with a suffix of: 6787 6788 * `ms` - Milliseconds 6789 * `s` - Seconds 6790 * `m` - Minutes 6791 * `h` - Hours 6792 * `d` - Days 6793 * `w` - Weeks 6794 * `M` - Months 6795 * `y` - Years 6796 6797 For example `--max-age 2d` means no files older than 2 days will be 6798 transferred. 6799 6800 ### `--min-age` - Don't transfer any file younger than this ### 6801 6802 This option controls the minimum age of files to transfer. Give in 6803 seconds or with a suffix (see `--max-age` for list of suffixes) 6804 6805 For example `--min-age 2d` means no files younger than 2 days will be 6806 transferred. 6807 6808 ### `--delete-excluded` - Delete files on dest excluded from sync ### 6809 6810 **Important** this flag is dangerous - use with `--dry-run` and `-v` first. 6811 6812 When doing `rclone sync` this will delete any files which are excluded 6813 from the sync on the destination. 6814 6815 If for example you did a sync from `A` to `B` without the `--min-size 50k` flag 6816 6817 rclone sync A: B: 6818 6819 Then you repeated it like this with the `--delete-excluded` 6820 6821 rclone --min-size 50k --delete-excluded sync A: B: 6822 6823 This would delete all files on `B` which are less than 50 kBytes as 6824 these are now excluded from the sync. 6825 6826 Always test first with `--dry-run` and `-v` before using this flag. 6827 6828 ### `--dump filters` - dump the filters to the output ### 6829 6830 This dumps the defined filters to the output as regular expressions. 6831 6832 Useful for debugging. 6833 6834 ### `--ignore-case` - make searches case insensitive ### 6835 6836 Normally filter patterns are case sensitive. If this flag is supplied 6837 then filter patterns become case insensitive. 6838 6839 Normally a `--include "file.txt"` will not match a file called 6840 `FILE.txt`. However if you use the `--ignore-case` flag then 6841 `--include "file.txt"` this will match a file called `FILE.txt`. 6842 6843 ## Quoting shell metacharacters ## 6844 6845 The examples above may not work verbatim in your shell as they have 6846 shell metacharacters in them (eg `*`), and may require quoting. 6847 6848 Eg linux, OSX 6849 6850 * `--include \*.jpg` 6851 * `--include '*.jpg'` 6852 * `--include='*.jpg'` 6853 6854 In Windows the expansion is done by the command not the shell so this 6855 should work fine 6856 6857 * `--include *.jpg` 6858 6859 ## Exclude directory based on a file ## 6860 6861 It is possible to exclude a directory based on a file, which is 6862 present in this directory. Filename should be specified using the 6863 `--exclude-if-present` flag. This flag has a priority over the other 6864 filtering flags. 6865 6866 Imagine, you have the following directory structure: 6867 6868 dir1/file1 6869 dir1/dir2/file2 6870 dir1/dir2/dir3/file3 6871 dir1/dir2/dir3/.ignore 6872 6873 You can exclude `dir3` from sync by running the following command: 6874 6875 rclone sync --exclude-if-present .ignore dir1 remote:backup 6876 6877 Currently only one filename is supported, i.e. `--exclude-if-present` 6878 should not be used multiple times. 6879 6880 # GUI (Experimental) 6881 6882 Rclone can serve a web based GUI (graphical user interface). This is 6883 somewhat experimental at the moment so things may be subject to 6884 change. 6885 6886 Run this command in a terminal and rclone will download and then 6887 display the GUI in a web browser. 6888 6889 ``` 6890 rclone rcd --rc-web-gui 6891 ``` 6892 6893 This will produce logs like this and rclone needs to continue to run to serve the GUI: 6894 6895 ``` 6896 2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip 6897 2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip] 6898 2019/08/25 11:40:16 NOTICE: Unzipping 6899 2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/ 6900 ``` 6901 6902 This assumes you are running rclone locally on your machine. It is 6903 possible to separate the rclone and the GUI - see below for details. 6904 6905 If you wish to check for updates then you can add `--rc-web-gui-update` 6906 to the command line. 6907 6908 If you find your GUI broken, you may force it to update by add `--rc-web-gui-force-update`. 6909 6910 By default, rclone will open your browser. Add `--rc-web-gui-no-open-browser` 6911 to disable this feature. 6912 6913 ## Using the GUI 6914 6915 Once the GUI opens, you will be looking at the dashboard which has an overall overview. 6916 6917 On the left hand side you will see a series of view buttons you can click on: 6918 6919 - Dashboard - main overview 6920 - Configs - examine and create new configurations 6921 - Explorer - view, download and upload files to the cloud storage systems 6922 - Backend - view or alter the backend config 6923 - Log out 6924 6925 (More docs and walkthrough video to come!) 6926 6927 ## How it works 6928 6929 When you run the `rclone rcd --rc-web-gui` this is what happens 6930 6931 - Rclone starts but only runs the remote control API ("rc"). 6932 - The API is bound to localhost with an auto generated username and password. 6933 - If the API bundle is missing then rclone will download it. 6934 - rclone will start serving the files from the API bundle over the same port as the API 6935 - rclone will open the browser with a `login_token` so it can log straight in. 6936 6937 ## Advanced use 6938 6939 The `rclone rcd` may use any of the [flags documented on the rc page](https://rclone.org/rc/#supported-parameters). 6940 6941 The flag `--rc-web-gui` is shorthand for 6942 6943 - Download the web GUI if necessary 6944 - Check we are using some authentication 6945 - `--rc-user gui` 6946 - `--rc-pass <random password>` 6947 - `--rc-serve` 6948 6949 These flags can be overidden as desired. 6950 6951 See also the [rclone rcd documentation](https://rclone.org/commands/rclone_rcd/). 6952 6953 ### Example: Running a public GUI 6954 6955 For example the GUI could be served on a public port over SSL using an htpasswd file using the following flags: 6956 6957 - `--rc-web-gui` 6958 - `--rc-addr :443` 6959 - `--rc-htpasswd /path/to/htpasswd` 6960 - `--rc-cert /path/to/ssl.crt` 6961 - `--rc-key /path/to/ssl.key` 6962 6963 ### Example: Running a GUI behind a proxy 6964 6965 If you want to run the GUI behind a proxy at `/rclone` you could use these flags: 6966 6967 - `--rc-web-gui` 6968 - `--rc-baseurl rclone` 6969 - `--rc-htpasswd /path/to/htpasswd` 6970 6971 Or instead of htpassword if you just want a single user and password: 6972 6973 - `--rc-user me` 6974 - `--rc-pass mypassword` 6975 6976 ## Project 6977 6978 The GUI is being developed in the: [rclone/rclone-webui-react respository](https://github.com/rclone/rclone-webui-react). 6979 6980 Bug reports and contributions are very welcome :-) 6981 6982 If you have questions then please ask them on the [rclone forum](https://forum.rclone.org/). 6983 6984 # Remote controlling rclone # 6985 6986 If rclone is run with the `--rc` flag then it starts an http server 6987 which can be used to remote control rclone. 6988 6989 If you just want to run a remote control then see the [rcd command](https://rclone.org/commands/rclone_rcd/). 6990 6991 **NB** this is experimental and everything here is subject to change! 6992 6993 ## Supported parameters 6994 6995 ### --rc 6996 6997 Flag to start the http server listen on remote requests 6998 6999 ### --rc-addr=IP 7000 7001 IPaddress:Port or :Port to bind server to. (default "localhost:5572") 7002 7003 ### --rc-cert=KEY 7004 SSL PEM key (concatenation of certificate and CA certificate) 7005 7006 ### --rc-client-ca=PATH 7007 Client certificate authority to verify clients with 7008 7009 ### --rc-htpasswd=PATH 7010 7011 htpasswd file - if not provided no authentication is done 7012 7013 ### --rc-key=PATH 7014 7015 SSL PEM Private key 7016 7017 ### --rc-max-header-bytes=VALUE 7018 7019 Maximum size of request header (default 4096) 7020 7021 ### --rc-user=VALUE 7022 7023 User name for authentication. 7024 7025 ### --rc-pass=VALUE 7026 7027 Password for authentication. 7028 7029 ### --rc-realm=VALUE 7030 7031 Realm for authentication (default "rclone") 7032 7033 ### --rc-server-read-timeout=DURATION 7034 7035 Timeout for server reading data (default 1h0m0s) 7036 7037 ### --rc-server-write-timeout=DURATION 7038 7039 Timeout for server writing data (default 1h0m0s) 7040 7041 ### --rc-serve 7042 7043 Enable the serving of remote objects via the HTTP interface. This 7044 means objects will be accessible at http://127.0.0.1:5572/ by default, 7045 so you can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/* 7046 to see a listing of the remotes. Objects may be requested from 7047 remotes using this syntax http://127.0.0.1:5572/[remote:path]/path/to/object 7048 7049 Default Off. 7050 7051 ### --rc-files /path/to/directory 7052 7053 Path to local files to serve on the HTTP server. 7054 7055 If this is set then rclone will serve the files in that directory. It 7056 will also open the root in the web browser if specified. This is for 7057 implementing browser based GUIs for rclone functions. 7058 7059 If `--rc-user` or `--rc-pass` is set then the URL that is opened will 7060 have the authorization in the URL in the `http://user:pass@localhost/` 7061 style. 7062 7063 Default Off. 7064 7065 ### --rc-web-gui 7066 7067 Set this flag to serve the default web gui on the same port as rclone. 7068 7069 Default Off. 7070 7071 ### --rc-allow-origin 7072 7073 Set the allowed Access-Control-Allow-Origin for rc requests. 7074 7075 Can be used with --rc-web-gui if the rclone is running on different IP than the web-gui. 7076 7077 Default is IP address on which rc is running. 7078 7079 ### --rc-web-fetch-url 7080 7081 Set the URL to fetch the rclone-web-gui files from. 7082 7083 Default https://api.github.com/repos/rclone/rclone-webui-react/releases/latest. 7084 7085 ### --rc-web-gui-update 7086 7087 Set this flag to check and update rclone-webui-react from the rc-web-fetch-url. 7088 7089 Default Off. 7090 7091 ### --rc-web-gui-force-update 7092 7093 Set this flag to force update rclone-webui-react from the rc-web-fetch-url. 7094 7095 Default Off. 7096 7097 ### --rc-web-gui-no-open-browser 7098 7099 Set this flag to disable opening browser automatically when using web-gui. 7100 7101 Default Off. 7102 7103 ### --rc-job-expire-duration=DURATION 7104 7105 Expire finished async jobs older than DURATION (default 60s). 7106 7107 ### --rc-job-expire-interval=DURATION 7108 7109 Interval duration to check for expired async jobs (default 10s). 7110 7111 ### --rc-no-auth 7112 7113 By default rclone will require authorisation to have been set up on 7114 the rc interface in order to use any methods which access any rclone 7115 remotes. Eg `operations/list` is denied as it involved creating a 7116 remote as is `sync/copy`. 7117 7118 If this is set then no authorisation will be required on the server to 7119 use these methods. The alternative is to use `--rc-user` and 7120 `--rc-pass` and use these credentials in the request. 7121 7122 Default Off. 7123 7124 ## Accessing the remote control via the rclone rc command 7125 7126 Rclone itself implements the remote control protocol in its `rclone 7127 rc` command. 7128 7129 You can use it like this 7130 7131 ``` 7132 $ rclone rc rc/noop param1=one param2=two 7133 { 7134 "param1": "one", 7135 "param2": "two" 7136 } 7137 ``` 7138 7139 Run `rclone rc` on its own to see the help for the installed remote 7140 control commands. 7141 7142 `rclone rc` also supports a `--json` flag which can be used to send 7143 more complicated input parameters. 7144 7145 ``` 7146 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop 7147 { 7148 "p1": [ 7149 1, 7150 "2", 7151 null, 7152 4 7153 ], 7154 "p2": { 7155 "a": 1, 7156 "b": 2 7157 } 7158 } 7159 ``` 7160 7161 ## Special parameters 7162 7163 The rc interface supports some special parameters which apply to 7164 **all** commands. These start with `_` to show they are different. 7165 7166 ### Running asynchronous jobs with _async = true 7167 7168 Each rc call is classified as a job and it is assigned its own id. By default 7169 jobs are executed immediately as they are created or synchronously. 7170 7171 If `_async` has a true value when supplied to an rc call then it will 7172 return immediately with a job id and the task will be run in the 7173 background. The `job/status` call can be used to get information of 7174 the background job. The job can be queried for up to 1 minute after 7175 it has finished. 7176 7177 It is recommended that potentially long running jobs, eg `sync/sync`, 7178 `sync/copy`, `sync/move`, `operations/purge` are run with the `_async` 7179 flag to avoid any potential problems with the HTTP request and 7180 response timing out. 7181 7182 Starting a job with the `_async` flag: 7183 7184 ``` 7185 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop 7186 { 7187 "jobid": 2 7188 } 7189 ``` 7190 7191 Query the status to see if the job has finished. For more information 7192 on the meaning of these return parameters see the `job/status` call. 7193 7194 ``` 7195 $ rclone rc --json '{ "jobid":2 }' job/status 7196 { 7197 "duration": 0.000124163, 7198 "endTime": "2018-10-27T11:38:07.911245881+01:00", 7199 "error": "", 7200 "finished": true, 7201 "id": 2, 7202 "output": { 7203 "_async": true, 7204 "p1": [ 7205 1, 7206 "2", 7207 null, 7208 4 7209 ], 7210 "p2": { 7211 "a": 1, 7212 "b": 2 7213 } 7214 }, 7215 "startTime": "2018-10-27T11:38:07.911121728+01:00", 7216 "success": true 7217 } 7218 ``` 7219 7220 `job/list` can be used to show the running or recently completed jobs 7221 7222 ``` 7223 $ rclone rc job/list 7224 { 7225 "jobids": [ 7226 2 7227 ] 7228 } 7229 ``` 7230 7231 ### Assigning operations to groups with _group = <value> 7232 7233 Each rc call has it's own stats group for tracking it's metrics. By default 7234 grouping is done by the composite group name from prefix `job/` and id of the 7235 job like so `job/1`. 7236 7237 If `_group` has a value then stats for that request will be grouped under that 7238 value. This allows caller to group stats under their own name. 7239 7240 Stats for specific group can be accessed by passing `group` to `core/stats`: 7241 7242 ``` 7243 $ rclone rc --json '{ "group": "job/1" }' core/stats 7244 { 7245 "speed": 12345 7246 ... 7247 } 7248 ``` 7249 7250 ## Supported commands 7251 <!--- autogenerated start - run make rcdocs - don't edit here --> 7252 ### cache/expire: Purge a remote from cache {#cache/expire} 7253 7254 Purge a remote from the cache backend. Supports either a directory or a file. 7255 Params: 7256 - remote = path to remote (required) 7257 - withData = true/false to delete cached data (chunks) as well (optional) 7258 7259 Eg 7260 7261 rclone rc cache/expire remote=path/to/sub/folder/ 7262 rclone rc cache/expire remote=/ withData=true 7263 7264 ### cache/fetch: Fetch file chunks {#cache/fetch} 7265 7266 Ensure the specified file chunks are cached on disk. 7267 7268 The chunks= parameter specifies the file chunks to check. 7269 It takes a comma separated list of array slice indices. 7270 The slice indices are similar to Python slices: start[:end] 7271 7272 start is the 0 based chunk number from the beginning of the file 7273 to fetch inclusive. end is 0 based chunk number from the beginning 7274 of the file to fetch exclusive. 7275 Both values can be negative, in which case they count from the back 7276 of the file. The value "-5:" represents the last 5 chunks of a file. 7277 7278 Some valid examples are: 7279 ":5,-5:" -> the first and last five chunks 7280 "0,-2" -> the first and the second last chunk 7281 "0:10" -> the first ten chunks 7282 7283 Any parameter with a key that starts with "file" can be used to 7284 specify files to fetch, eg 7285 7286 rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye 7287 7288 File names will automatically be encrypted when the a crypt remote 7289 is used on top of the cache. 7290 7291 ### cache/stats: Get cache stats {#cache/stats} 7292 7293 Show statistics for the cache remote. 7294 7295 ### config/create: create the config for a remote. {#config/create} 7296 7297 This takes the following parameters 7298 7299 - name - name of remote 7300 - parameters - a map of \{ "key": "value" \} pairs 7301 - type - type of the new remote 7302 7303 7304 See the [config create command](https://rclone.org/commands/rclone_config_create/) command for more information on the above. 7305 7306 Authentication is required for this call. 7307 7308 ### config/delete: Delete a remote in the config file. {#config/delete} 7309 7310 Parameters: 7311 7312 - name - name of remote to delete 7313 7314 See the [config delete command](https://rclone.org/commands/rclone_config_delete/) command for more information on the above. 7315 7316 Authentication is required for this call. 7317 7318 ### config/dump: Dumps the config file. {#config/dump} 7319 7320 Returns a JSON object: 7321 - key: value 7322 7323 Where keys are remote names and values are the config parameters. 7324 7325 See the [config dump command](https://rclone.org/commands/rclone_config_dump/) command for more information on the above. 7326 7327 Authentication is required for this call. 7328 7329 ### config/get: Get a remote in the config file. {#config/get} 7330 7331 Parameters: 7332 7333 - name - name of remote to get 7334 7335 See the [config dump command](https://rclone.org/commands/rclone_config_dump/) command for more information on the above. 7336 7337 Authentication is required for this call. 7338 7339 ### config/listremotes: Lists the remotes in the config file. {#config/listremotes} 7340 7341 Returns 7342 - remotes - array of remote names 7343 7344 See the [listremotes command](https://rclone.org/commands/rclone_listremotes/) command for more information on the above. 7345 7346 Authentication is required for this call. 7347 7348 ### config/password: password the config for a remote. {#config/password} 7349 7350 This takes the following parameters 7351 7352 - name - name of remote 7353 - parameters - a map of \{ "key": "value" \} pairs 7354 7355 7356 See the [config password command](https://rclone.org/commands/rclone_config_password/) command for more information on the above. 7357 7358 Authentication is required for this call. 7359 7360 ### config/providers: Shows how providers are configured in the config file. {#config/providers} 7361 7362 Returns a JSON object: 7363 - providers - array of objects 7364 7365 See the [config providers command](https://rclone.org/commands/rclone_config_providers/) command for more information on the above. 7366 7367 Authentication is required for this call. 7368 7369 ### config/update: update the config for a remote. {#config/update} 7370 7371 This takes the following parameters 7372 7373 - name - name of remote 7374 - parameters - a map of \{ "key": "value" \} pairs 7375 7376 7377 See the [config update command](https://rclone.org/commands/rclone_config_update/) command for more information on the above. 7378 7379 Authentication is required for this call. 7380 7381 ### core/bwlimit: Set the bandwidth limit. {#core/bwlimit} 7382 7383 This sets the bandwidth limit to that passed in. 7384 7385 Eg 7386 7387 rclone rc core/bwlimit rate=off 7388 { 7389 "bytesPerSecond": -1, 7390 "rate": "off" 7391 } 7392 rclone rc core/bwlimit rate=1M 7393 { 7394 "bytesPerSecond": 1048576, 7395 "rate": "1M" 7396 } 7397 7398 7399 If the rate parameter is not suppied then the bandwidth is queried 7400 7401 rclone rc core/bwlimit 7402 { 7403 "bytesPerSecond": 1048576, 7404 "rate": "1M" 7405 } 7406 7407 The format of the parameter is exactly the same as passed to --bwlimit 7408 except only one bandwidth may be specified. 7409 7410 In either case "rate" is returned as a human readable string, and 7411 "bytesPerSecond" is returned as a number. 7412 7413 ### core/gc: Runs a garbage collection. {#core/gc} 7414 7415 This tells the go runtime to do a garbage collection run. It isn't 7416 necessary to call this normally, but it can be useful for debugging 7417 memory problems. 7418 7419 ### core/group-list: Returns list of stats. {#core/group-list} 7420 7421 This returns list of stats groups currently in memory. 7422 7423 Returns the following values: 7424 ``` 7425 { 7426 "groups": an array of group names: 7427 [ 7428 "group1", 7429 "group2", 7430 ... 7431 ] 7432 } 7433 ``` 7434 7435 ### core/memstats: Returns the memory statistics {#core/memstats} 7436 7437 This returns the memory statistics of the running program. What the values mean 7438 are explained in the go docs: https://golang.org/pkg/runtime/#MemStats 7439 7440 The most interesting values for most people are: 7441 7442 * HeapAlloc: This is the amount of memory rclone is actually using 7443 * HeapSys: This is the amount of memory rclone has obtained from the OS 7444 * Sys: this is the total amount of memory requested from the OS 7445 * It is virtual memory so may include unused memory 7446 7447 ### core/obscure: Obscures a string passed in. {#core/obscure} 7448 7449 Pass a clear string and rclone will obscure it for the config file: 7450 - clear - string 7451 7452 Returns 7453 - obscured - string 7454 7455 ### core/pid: Return PID of current process {#core/pid} 7456 7457 This returns PID of current process. 7458 Useful for stopping rclone process. 7459 7460 ### core/quit: Terminates the app. {#core/quit} 7461 7462 (optional) Pass an exit code to be used for terminating the app: 7463 - exitCode - int 7464 7465 ### core/stats: Returns stats about current transfers. {#core/stats} 7466 7467 This returns all available stats: 7468 7469 rclone rc core/stats 7470 7471 If group is not provided then summed up stats for all groups will be 7472 returned. 7473 7474 Parameters 7475 7476 - group - name of the stats group (string) 7477 7478 Returns the following values: 7479 7480 ``` 7481 { 7482 "speed": average speed in bytes/sec since start of the process, 7483 "bytes": total transferred bytes since the start of the process, 7484 "errors": number of errors, 7485 "fatalError": whether there has been at least one FatalError, 7486 "retryError": whether there has been at least one non-NoRetryError, 7487 "checks": number of checked files, 7488 "transfers": number of transferred files, 7489 "deletes" : number of deleted files, 7490 "elapsedTime": time in seconds since the start of the process, 7491 "lastError": last occurred error, 7492 "transferring": an array of currently active file transfers: 7493 [ 7494 { 7495 "bytes": total transferred bytes for this file, 7496 "eta": estimated time in seconds until file transfer completion 7497 "name": name of the file, 7498 "percentage": progress of the file transfer in percent, 7499 "speed": speed in bytes/sec, 7500 "speedAvg": speed in bytes/sec as an exponentially weighted moving average, 7501 "size": size of the file in bytes 7502 } 7503 ], 7504 "checking": an array of names of currently active file checks 7505 [] 7506 } 7507 ``` 7508 Values for "transferring", "checking" and "lastError" are only assigned if data is available. 7509 The value for "eta" is null if an eta cannot be determined. 7510 7511 ### core/stats-delete: Delete stats group. {#core/stats-delete} 7512 7513 This deletes entire stats group 7514 7515 Parameters 7516 7517 - group - name of the stats group (string) 7518 7519 ### core/stats-reset: Reset stats. {#core/stats-reset} 7520 7521 This clears counters, errors and finished transfers for all stats or specific 7522 stats group if group is provided. 7523 7524 Parameters 7525 7526 - group - name of the stats group (string) 7527 7528 ### core/transferred: Returns stats about completed transfers. {#core/transferred} 7529 7530 This returns stats about completed transfers: 7531 7532 rclone rc core/transferred 7533 7534 If group is not provided then completed transfers for all groups will be 7535 returned. 7536 7537 Note only the last 100 completed transfers are returned. 7538 7539 Parameters 7540 7541 - group - name of the stats group (string) 7542 7543 Returns the following values: 7544 ``` 7545 { 7546 "transferred": an array of completed transfers (including failed ones): 7547 [ 7548 { 7549 "name": name of the file, 7550 "size": size of the file in bytes, 7551 "bytes": total transferred bytes for this file, 7552 "checked": if the transfer is only checked (skipped, deleted), 7553 "timestamp": integer representing millisecond unix epoch, 7554 "error": string description of the error (empty if successfull), 7555 "jobid": id of the job that this transfer belongs to 7556 } 7557 ] 7558 } 7559 ``` 7560 7561 ### core/version: Shows the current version of rclone and the go runtime. {#core/version} 7562 7563 This shows the current version of go and the go runtime 7564 7565 - version - rclone version, eg "v1.44" 7566 - decomposed - version number as [major, minor, patch, subpatch] 7567 - note patch and subpatch will be 999 for a git compiled version 7568 - isGit - boolean - true if this was compiled from the git version 7569 - os - OS in use as according to Go 7570 - arch - cpu architecture in use according to Go 7571 - goVersion - version of Go runtime in use 7572 7573 ### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug/set-block-profile-rate} 7574 7575 SetBlockProfileRate controls the fraction of goroutine blocking events 7576 that are reported in the blocking profile. The profiler aims to sample 7577 an average of one blocking event per rate nanoseconds spent blocked. 7578 7579 To include every blocking event in the profile, pass rate = 1. To turn 7580 off profiling entirely, pass rate <= 0. 7581 7582 After calling this you can use this to see the blocking profile: 7583 7584 go tool pprof http://localhost:5572/debug/pprof/block 7585 7586 Parameters 7587 7588 - rate - int 7589 7590 ### debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling. {#debug/set-mutex-profile-fraction} 7591 7592 SetMutexProfileFraction controls the fraction of mutex contention 7593 events that are reported in the mutex profile. On average 1/rate 7594 events are reported. The previous rate is returned. 7595 7596 To turn off profiling entirely, pass rate 0. To just read the current 7597 rate, pass rate < 0. (For n>1 the details of sampling may change.) 7598 7599 Once this is set you can look use this to profile the mutex contention: 7600 7601 go tool pprof http://localhost:5572/debug/pprof/mutex 7602 7603 Parameters 7604 7605 - rate - int 7606 7607 Results 7608 7609 - previousRate - int 7610 7611 ### job/list: Lists the IDs of the running jobs {#job/list} 7612 7613 Parameters - None 7614 7615 Results 7616 7617 - jobids - array of integer job ids 7618 7619 ### job/status: Reads the status of the job ID {#job/status} 7620 7621 Parameters 7622 7623 - jobid - id of the job (integer) 7624 7625 Results 7626 7627 - finished - boolean 7628 - duration - time in seconds that the job ran for 7629 - endTime - time the job finished (eg "2018-10-26T18:50:20.528746884+01:00") 7630 - error - error from the job or empty string for no error 7631 - finished - boolean whether the job has finished or not 7632 - id - as passed in above 7633 - startTime - time the job started (eg "2018-10-26T18:50:20.528336039+01:00") 7634 - success - boolean - true for success false otherwise 7635 - output - output of the job as would have been returned if called synchronously 7636 - progress - output of the progress related to the underlying job 7637 7638 ### job/stop: Stop the running job {#job/stop} 7639 7640 Parameters 7641 7642 - jobid - id of the job (integer) 7643 7644 ### operations/about: Return the space used on the remote {#operations/about} 7645 7646 This takes the following parameters 7647 7648 - fs - a remote name string eg "drive:" 7649 7650 The result is as returned from rclone about --json 7651 7652 See the [about command](https://rclone.org/commands/rclone_size/) command for more information on the above. 7653 7654 Authentication is required for this call. 7655 7656 ### operations/cleanup: Remove trashed files in the remote or path {#operations/cleanup} 7657 7658 This takes the following parameters 7659 7660 - fs - a remote name string eg "drive:" 7661 7662 See the [cleanup command](https://rclone.org/commands/rclone_cleanup/) command for more information on the above. 7663 7664 Authentication is required for this call. 7665 7666 ### operations/copyfile: Copy a file from source remote to destination remote {#operations/copyfile} 7667 7668 This takes the following parameters 7669 7670 - srcFs - a remote name string eg "drive:" for the source 7671 - srcRemote - a path within that remote eg "file.txt" for the source 7672 - dstFs - a remote name string eg "drive2:" for the destination 7673 - dstRemote - a path within that remote eg "file2.txt" for the destination 7674 7675 Authentication is required for this call. 7676 7677 ### operations/copyurl: Copy the URL to the object {#operations/copyurl} 7678 7679 This takes the following parameters 7680 7681 - fs - a remote name string eg "drive:" 7682 - remote - a path within that remote eg "dir" 7683 - url - string, URL to read from 7684 - autoFilename - boolean, set to true to retrieve destination file name from url 7685 See the [copyurl command](https://rclone.org/commands/rclone_copyurl/) command for more information on the above. 7686 7687 Authentication is required for this call. 7688 7689 ### operations/delete: Remove files in the path {#operations/delete} 7690 7691 This takes the following parameters 7692 7693 - fs - a remote name string eg "drive:" 7694 7695 See the [delete command](https://rclone.org/commands/rclone_delete/) command for more information on the above. 7696 7697 Authentication is required for this call. 7698 7699 ### operations/deletefile: Remove the single file pointed to {#operations/deletefile} 7700 7701 This takes the following parameters 7702 7703 - fs - a remote name string eg "drive:" 7704 - remote - a path within that remote eg "dir" 7705 7706 See the [deletefile command](https://rclone.org/commands/rclone_deletefile/) command for more information on the above. 7707 7708 Authentication is required for this call. 7709 7710 ### operations/fsinfo: Return information about the remote {#operations/fsinfo} 7711 7712 This takes the following parameters 7713 7714 - fs - a remote name string eg "drive:" 7715 7716 This returns info about the remote passed in; 7717 7718 ``` 7719 { 7720 // optional features and whether they are available or not 7721 "Features": { 7722 "About": true, 7723 "BucketBased": false, 7724 "CanHaveEmptyDirectories": true, 7725 "CaseInsensitive": false, 7726 "ChangeNotify": false, 7727 "CleanUp": false, 7728 "Copy": false, 7729 "DirCacheFlush": false, 7730 "DirMove": true, 7731 "DuplicateFiles": false, 7732 "GetTier": false, 7733 "ListR": false, 7734 "MergeDirs": false, 7735 "Move": true, 7736 "OpenWriterAt": true, 7737 "PublicLink": false, 7738 "Purge": true, 7739 "PutStream": true, 7740 "PutUnchecked": false, 7741 "ReadMimeType": false, 7742 "ServerSideAcrossConfigs": false, 7743 "SetTier": false, 7744 "SetWrapper": false, 7745 "UnWrap": false, 7746 "WrapFs": false, 7747 "WriteMimeType": false 7748 }, 7749 // Names of hashes available 7750 "Hashes": [ 7751 "MD5", 7752 "SHA-1", 7753 "DropboxHash", 7754 "QuickXorHash" 7755 ], 7756 "Name": "local", // Name as created 7757 "Precision": 1, // Precision of timestamps in ns 7758 "Root": "/", // Path as created 7759 "String": "Local file system at /" // how the remote will appear in logs 7760 } 7761 ``` 7762 7763 This command does not have a command line equivalent so use this instead: 7764 7765 rclone rc --loopback operations/fsinfo fs=remote: 7766 7767 ### operations/list: List the given remote and path in JSON format {#operations/list} 7768 7769 This takes the following parameters 7770 7771 - fs - a remote name string eg "drive:" 7772 - remote - a path within that remote eg "dir" 7773 - opt - a dictionary of options to control the listing (optional) 7774 - recurse - If set recurse directories 7775 - noModTime - If set return modification time 7776 - showEncrypted - If set show decrypted names 7777 - showOrigIDs - If set show the IDs for each item if known 7778 - showHash - If set return a dictionary of hashes 7779 7780 The result is 7781 7782 - list 7783 - This is an array of objects as described in the lsjson command 7784 7785 See the [lsjson command](https://rclone.org/commands/rclone_lsjson/) for more information on the above and examples. 7786 7787 Authentication is required for this call. 7788 7789 ### operations/mkdir: Make a destination directory or container {#operations/mkdir} 7790 7791 This takes the following parameters 7792 7793 - fs - a remote name string eg "drive:" 7794 - remote - a path within that remote eg "dir" 7795 7796 See the [mkdir command](https://rclone.org/commands/rclone_mkdir/) command for more information on the above. 7797 7798 Authentication is required for this call. 7799 7800 ### operations/movefile: Move a file from source remote to destination remote {#operations/movefile} 7801 7802 This takes the following parameters 7803 7804 - srcFs - a remote name string eg "drive:" for the source 7805 - srcRemote - a path within that remote eg "file.txt" for the source 7806 - dstFs - a remote name string eg "drive2:" for the destination 7807 - dstRemote - a path within that remote eg "file2.txt" for the destination 7808 7809 Authentication is required for this call. 7810 7811 ### operations/publiclink: Create or retrieve a public link to the given file or folder. {#operations/publiclink} 7812 7813 This takes the following parameters 7814 7815 - fs - a remote name string eg "drive:" 7816 - remote - a path within that remote eg "dir" 7817 7818 Returns 7819 7820 - url - URL of the resource 7821 7822 See the [link command](https://rclone.org/commands/rclone_link/) command for more information on the above. 7823 7824 Authentication is required for this call. 7825 7826 ### operations/purge: Remove a directory or container and all of its contents {#operations/purge} 7827 7828 This takes the following parameters 7829 7830 - fs - a remote name string eg "drive:" 7831 - remote - a path within that remote eg "dir" 7832 7833 See the [purge command](https://rclone.org/commands/rclone_purge/) command for more information on the above. 7834 7835 Authentication is required for this call. 7836 7837 ### operations/rmdir: Remove an empty directory or container {#operations/rmdir} 7838 7839 This takes the following parameters 7840 7841 - fs - a remote name string eg "drive:" 7842 - remote - a path within that remote eg "dir" 7843 7844 See the [rmdir command](https://rclone.org/commands/rclone_rmdir/) command for more information on the above. 7845 7846 Authentication is required for this call. 7847 7848 ### operations/rmdirs: Remove all the empty directories in the path {#operations/rmdirs} 7849 7850 This takes the following parameters 7851 7852 - fs - a remote name string eg "drive:" 7853 - remote - a path within that remote eg "dir" 7854 - leaveRoot - boolean, set to true not to delete the root 7855 7856 See the [rmdirs command](https://rclone.org/commands/rclone_rmdirs/) command for more information on the above. 7857 7858 Authentication is required for this call. 7859 7860 ### operations/size: Count the number of bytes and files in remote {#operations/size} 7861 7862 This takes the following parameters 7863 7864 - fs - a remote name string eg "drive:path/to/dir" 7865 7866 Returns 7867 7868 - count - number of files 7869 - bytes - number of bytes in those files 7870 7871 See the [size command](https://rclone.org/commands/rclone_size/) command for more information on the above. 7872 7873 Authentication is required for this call. 7874 7875 ### options/blocks: List all the option blocks {#options/blocks} 7876 7877 Returns 7878 - options - a list of the options block names 7879 7880 ### options/get: Get all the options {#options/get} 7881 7882 Returns an object where keys are option block names and values are an 7883 object with the current option values in. 7884 7885 This shows the internal names of the option within rclone which should 7886 map to the external options very easily with a few exceptions. 7887 7888 ### options/set: Set an option {#options/set} 7889 7890 Parameters 7891 7892 - option block name containing an object with 7893 - key: value 7894 7895 Repeated as often as required. 7896 7897 Only supply the options you wish to change. If an option is unknown 7898 it will be silently ignored. Not all options will have an effect when 7899 changed like this. 7900 7901 For example: 7902 7903 This sets DEBUG level logs (-vv) 7904 7905 rclone rc options/set --json '{"main": {"LogLevel": 8}}' 7906 7907 And this sets INFO level logs (-v) 7908 7909 rclone rc options/set --json '{"main": {"LogLevel": 7}}' 7910 7911 And this sets NOTICE level logs (normal without -v) 7912 7913 rclone rc options/set --json '{"main": {"LogLevel": 6}}' 7914 7915 ### rc/error: This returns an error {#rc/error} 7916 7917 This returns an error with the input as part of its error string. 7918 Useful for testing error handling. 7919 7920 ### rc/list: List all the registered remote control commands {#rc/list} 7921 7922 This lists all the registered remote control commands as a JSON map in 7923 the commands response. 7924 7925 ### rc/noop: Echo the input to the output parameters {#rc/noop} 7926 7927 This echoes the input parameters to the output parameters for testing 7928 purposes. It can be used to check that rclone is still alive and to 7929 check that parameter passing is working properly. 7930 7931 ### rc/noopauth: Echo the input to the output parameters requiring auth {#rc/noopauth} 7932 7933 This echoes the input parameters to the output parameters for testing 7934 purposes. It can be used to check that rclone is still alive and to 7935 check that parameter passing is working properly. 7936 7937 Authentication is required for this call. 7938 7939 ### sync/copy: copy a directory from source remote to destination remote {#sync/copy} 7940 7941 This takes the following parameters 7942 7943 - srcFs - a remote name string eg "drive:src" for the source 7944 - dstFs - a remote name string eg "drive:dst" for the destination 7945 7946 7947 See the [copy command](https://rclone.org/commands/rclone_copy/) command for more information on the above. 7948 7949 Authentication is required for this call. 7950 7951 ### sync/move: move a directory from source remote to destination remote {#sync/move} 7952 7953 This takes the following parameters 7954 7955 - srcFs - a remote name string eg "drive:src" for the source 7956 - dstFs - a remote name string eg "drive:dst" for the destination 7957 - deleteEmptySrcDirs - delete empty src directories if set 7958 7959 7960 See the [move command](https://rclone.org/commands/rclone_move/) command for more information on the above. 7961 7962 Authentication is required for this call. 7963 7964 ### sync/sync: sync a directory from source remote to destination remote {#sync/sync} 7965 7966 This takes the following parameters 7967 7968 - srcFs - a remote name string eg "drive:src" for the source 7969 - dstFs - a remote name string eg "drive:dst" for the destination 7970 7971 7972 See the [sync command](https://rclone.org/commands/rclone_sync/) command for more information on the above. 7973 7974 Authentication is required for this call. 7975 7976 ### vfs/forget: Forget files or directories in the directory cache. {#vfs/forget} 7977 7978 This forgets the paths in the directory cache causing them to be 7979 re-read from the remote when needed. 7980 7981 If no paths are passed in then it will forget all the paths in the 7982 directory cache. 7983 7984 rclone rc vfs/forget 7985 7986 Otherwise pass files or dirs in as file=path or dir=path. Any 7987 parameter key starting with file will forget that file and any 7988 starting with dir will forget that dir, eg 7989 7990 rclone rc vfs/forget file=hello file2=goodbye dir=home/junk 7991 7992 ### vfs/poll-interval: Get the status or update the value of the poll-interval option. {#vfs/poll-interval} 7993 7994 Without any parameter given this returns the current status of the 7995 poll-interval setting. 7996 7997 When the interval=duration parameter is set, the poll-interval value 7998 is updated and the polling function is notified. 7999 Setting interval=0 disables poll-interval. 8000 8001 rclone rc vfs/poll-interval interval=5m 8002 8003 The timeout=duration parameter can be used to specify a time to wait 8004 for the current poll function to apply the new value. 8005 If timeout is less or equal 0, which is the default, wait indefinitely. 8006 8007 The new poll-interval value will only be active when the timeout is 8008 not reached. 8009 8010 If poll-interval is updated or disabled temporarily, some changes 8011 might not get picked up by the polling function, depending on the 8012 used remote. 8013 8014 ### vfs/refresh: Refresh the directory cache. {#vfs/refresh} 8015 8016 This reads the directories for the specified paths and freshens the 8017 directory cache. 8018 8019 If no paths are passed in then it will refresh the root directory. 8020 8021 rclone rc vfs/refresh 8022 8023 Otherwise pass directories in as dir=path. Any parameter key 8024 starting with dir will refresh that directory, eg 8025 8026 rclone rc vfs/refresh dir=home/junk dir2=data/misc 8027 8028 If the parameter recursive=true is given the whole directory tree 8029 will get refreshed. This refresh will use --fast-list if enabled. 8030 8031 <!--- autogenerated stop --> 8032 8033 ## Accessing the remote control via HTTP 8034 8035 Rclone implements a simple HTTP based protocol. 8036 8037 Each endpoint takes an JSON object and returns a JSON object or an 8038 error. The JSON objects are essentially a map of string names to 8039 values. 8040 8041 All calls must made using POST. 8042 8043 The input objects can be supplied using URL parameters, POST 8044 parameters or by supplying "Content-Type: application/json" and a JSON 8045 blob in the body. There are examples of these below using `curl`. 8046 8047 The response will be a JSON blob in the body of the response. This is 8048 formatted to be reasonably human readable. 8049 8050 ### Error returns 8051 8052 If an error occurs then there will be an HTTP error status (eg 500) 8053 and the body of the response will contain a JSON encoded error object, 8054 eg 8055 8056 ``` 8057 { 8058 "error": "Expecting string value for key \"remote\" (was float64)", 8059 "input": { 8060 "fs": "/tmp", 8061 "remote": 3 8062 }, 8063 "status": 400 8064 "path": "operations/rmdir", 8065 } 8066 ``` 8067 8068 The keys in the error response are 8069 - error - error string 8070 - input - the input parameters to the call 8071 - status - the HTTP status code 8072 - path - the path of the call 8073 8074 ### CORS 8075 8076 The sever implements basic CORS support and allows all origins for that. 8077 The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back. 8078 8079 ### Using POST with URL parameters only 8080 8081 ``` 8082 curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2' 8083 ``` 8084 8085 Response 8086 8087 ``` 8088 { 8089 "potato": "1", 8090 "sausage": "2" 8091 } 8092 ``` 8093 8094 Here is what an error response looks like: 8095 8096 ``` 8097 curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 8098 ``` 8099 8100 ``` 8101 { 8102 "error": "arbitrary error on input map[potato:1 sausage:2]", 8103 "input": { 8104 "potato": "1", 8105 "sausage": "2" 8106 } 8107 } 8108 ``` 8109 8110 Note that curl doesn't return errors to the shell unless you use the `-f` option 8111 8112 ``` 8113 $ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 8114 curl: (22) The requested URL returned error: 400 Bad Request 8115 $ echo $? 8116 22 8117 ``` 8118 8119 ### Using POST with a form 8120 8121 ``` 8122 curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop 8123 ``` 8124 8125 Response 8126 8127 ``` 8128 { 8129 "potato": "1", 8130 "sausage": "2" 8131 } 8132 ``` 8133 8134 Note that you can combine these with URL parameters too with the POST 8135 parameters taking precedence. 8136 8137 ``` 8138 curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4" 8139 ``` 8140 8141 Response 8142 8143 ``` 8144 { 8145 "potato": "1", 8146 "rutabaga": "3", 8147 "sausage": "4" 8148 } 8149 8150 ``` 8151 8152 ### Using POST with a JSON blob 8153 8154 ``` 8155 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop 8156 ``` 8157 8158 response 8159 8160 ``` 8161 { 8162 "password": "xyz", 8163 "username": "xyz" 8164 } 8165 ``` 8166 8167 This can be combined with URL parameters too if required. The JSON 8168 blob takes precedence. 8169 8170 ``` 8171 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4' 8172 ``` 8173 8174 ``` 8175 { 8176 "potato": 2, 8177 "rutabaga": "3", 8178 "sausage": 1 8179 } 8180 ``` 8181 8182 ## Debugging rclone with pprof ## 8183 8184 If you use the `--rc` flag this will also enable the use of the go 8185 profiling tools on the same port. 8186 8187 To use these, first [install go](https://golang.org/doc/install). 8188 8189 ### Debugging memory use 8190 8191 To profile rclone's memory use you can run: 8192 8193 go tool pprof -web http://localhost:5572/debug/pprof/heap 8194 8195 This should open a page in your browser showing what is using what 8196 memory. 8197 8198 You can also use the `-text` flag to produce a textual summary 8199 8200 ``` 8201 $ go tool pprof -text http://localhost:5572/debug/pprof/heap 8202 Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total 8203 flat flat% sum% cum cum% 8204 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode 8205 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize 8206 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init 8207 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init 8208 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init 8209 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init 8210 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init 8211 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0 8212 0 0% 100% 1024.03kB 66.62% main.init 8213 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest 8214 0 0% 100% 513kB 33.38% net/http.(*conn).serve 8215 0 0% 100% 1024.03kB 66.62% runtime.main 8216 ``` 8217 8218 ### Debugging go routine leaks 8219 8220 Memory leaks are most often caused by go routine leaks keeping memory 8221 alive which should have been garbage collected. 8222 8223 See all active go routines using 8224 8225 curl http://localhost:5572/debug/pprof/goroutine?debug=1 8226 8227 Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your browser. 8228 8229 ### Other profiles to look at 8230 8231 You can see a summary of profiles available at http://localhost:5572/debug/pprof/ 8232 8233 Here is how to use some of them: 8234 8235 - Memory: `go tool pprof http://localhost:5572/debug/pprof/heap` 8236 - Go routines: `curl http://localhost:5572/debug/pprof/goroutine?debug=1` 8237 - 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile` 8238 - 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5` 8239 - Goroutine blocking profile 8240 - Enable first with: `rclone rc debug/set-block-profile-rate rate=1` ([docs](#debug/set-block-profile-rate)) 8241 - `go tool pprof http://localhost:5572/debug/pprof/block` 8242 - Contended mutexes: 8243 - Enable first with: `rclone rc debug/set-mutex-profile-fraction rate=1` ([docs](#debug/set-mutex-profile-fraction)) 8244 - `go tool pprof http://localhost:5572/debug/pprof/mutex` 8245 8246 See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/) 8247 for more info on how to use the profiling and for a general overview 8248 see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs). 8249 8250 The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234). 8251 8252 # Overview of cloud storage systems # 8253 8254 Each cloud storage system is slightly different. Rclone attempts to 8255 provide a unified interface to them, but some underlying differences 8256 show through. 8257 8258 ## Features ## 8259 8260 Here is an overview of the major features of each cloud storage system. 8261 8262 | Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | 8263 | ---------------------------- |:-----------:|:-------:|:----------------:|:---------------:|:---------:| 8264 | 1Fichier | Whirlpool | No | No | Yes | R | 8265 | Amazon Drive | MD5 | No | Yes | No | R | 8266 | Amazon S3 | MD5 | Yes | No | No | R/W | 8267 | Backblaze B2 | SHA1 | Yes | No | No | R/W | 8268 | Box | SHA1 | Yes | Yes | No | - | 8269 | Citrix ShareFile | MD5 | Yes | Yes | No | - | 8270 | Dropbox | DBHASH † | Yes | Yes | No | - | 8271 | FTP | - | No | No | No | - | 8272 | Google Cloud Storage | MD5 | Yes | No | No | R/W | 8273 | Google Drive | MD5 | Yes | No | Yes | R/W | 8274 | Google Photos | - | No | No | Yes | R | 8275 | HTTP | - | No | No | No | R | 8276 | Hubic | MD5 | Yes | No | No | R/W | 8277 | Jottacloud | MD5 | Yes | Yes | No | R/W | 8278 | Koofr | MD5 | No | Yes | No | - | 8279 | Mail.ru Cloud | Mailru ‡‡‡ | Yes | Yes | No | - | 8280 | Mega | - | No | No | Yes | - | 8281 | Memory | MD5 | Yes | No | No | - | 8282 | Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W | 8283 | Microsoft OneDrive | SHA1 ‡‡ | Yes | Yes | No | R | 8284 | OpenDrive | MD5 | Yes | Yes | No | - | 8285 | Openstack Swift | MD5 | Yes | No | No | R/W | 8286 | pCloud | MD5, SHA1 | Yes | No | No | W | 8287 | premiumize.me | - | No | Yes | No | R | 8288 | put.io | CRC-32 | Yes | No | Yes | R | 8289 | QingStor | MD5 | No | No | No | R/W | 8290 | SFTP | MD5, SHA1 ‡ | Yes | Depends | No | - | 8291 | SugarSync | - | No | No | No | - | 8292 | WebDAV | MD5, SHA1 ††| Yes ††† | Depends | No | - | 8293 | Yandex Disk | MD5 | Yes | No | No | R/W | 8294 | The local filesystem | All | Yes | Depends | No | - | 8295 8296 ### Hash ### 8297 8298 The cloud storage system supports various hash types of the objects. 8299 The hashes are used when transferring data as an integrity check and 8300 can be specifically used with the `--checksum` flag in syncs and in 8301 the `check` command. 8302 8303 To use the verify checksums when transferring between cloud storage 8304 systems they must support a common hash type. 8305 8306 † Note that Dropbox supports [its own custom 8307 hash](https://www.dropbox.com/developers/reference/content-hash). 8308 This is an SHA256 sum of all the 4MB block SHA256s. 8309 8310 ‡ SFTP supports checksums if the same login has shell access and `md5sum` 8311 or `sha1sum` as well as `echo` are in the remote's PATH. 8312 8313 †† WebDAV supports hashes when used with Owncloud and Nextcloud only. 8314 8315 ††† WebDAV supports modtimes when used with Owncloud and Nextcloud only. 8316 8317 ‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive 8318 for business and SharePoint server support Microsoft's own 8319 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 8320 8321 ‡‡‡ Mail.ru uses its own modified SHA1 hash 8322 8323 ### ModTime ### 8324 8325 The cloud storage system supports setting modification times on 8326 objects. If it does then this enables a using the modification times 8327 as part of the sync. If not then only the size will be checked by 8328 default, though the MD5SUM can be checked with the `--checksum` flag. 8329 8330 All cloud storage systems support some kind of date on the object and 8331 these will be set when transferring from the cloud storage system. 8332 8333 ### Case Insensitive ### 8334 8335 If a cloud storage systems is case sensitive then it is possible to 8336 have two files which differ only in case, eg `file.txt` and 8337 `FILE.txt`. If a cloud storage system is case insensitive then that 8338 isn't possible. 8339 8340 This can cause problems when syncing between a case insensitive 8341 system and a case sensitive system. The symptom of this is that no 8342 matter how many times you run the sync it never completes fully. 8343 8344 The local filesystem and SFTP may or may not be case sensitive 8345 depending on OS. 8346 8347 * Windows - usually case insensitive, though case is preserved 8348 * OSX - usually case insensitive, though it is possible to format case sensitive 8349 * Linux - usually case sensitive, but there are case insensitive file systems (eg FAT formatted USB keys) 8350 8351 Most of the time this doesn't cause any problems as people tend to 8352 avoid files whose name differs only by case even on case sensitive 8353 systems. 8354 8355 ### Duplicate files ### 8356 8357 If a cloud storage system allows duplicate files then it can have two 8358 objects with the same name. 8359 8360 This confuses rclone greatly when syncing - use the `rclone dedupe` 8361 command to rename or remove duplicates. 8362 8363 ### Restricted filenames ### 8364 8365 Some cloud storage systems might have restrictions on the characters 8366 that are usable in file or directory names. 8367 When `rclone` detects such a name during a file upload, it will 8368 transparently replace the restricted characters with similar looking 8369 Unicode characters. 8370 8371 This process is designed to avoid ambiguous file names as much as 8372 possible and allow to move files between many cloud storage systems 8373 transparently. 8374 8375 The name shown by `rclone` to the user or during log output will only 8376 contain a minimal set of [replaced characters](#restricted-characters) 8377 to ensure correct formatting and not necessarily the actual name used 8378 on the cloud storage. 8379 8380 This transformation is reversed when downloading a file or parsing 8381 `rclone` arguments. 8382 For example, when uploading a file named `my file?.txt` to Onedrive 8383 will be displayed as `my file?.txt` on the console, but stored as 8384 `my file?.txt` (the `?` gets replaced by the similar looking `?` 8385 character) to Onedrive. 8386 The reverse transformation allows to read a file`unusual/name.txt` 8387 from Google Drive, by passing the name `unusual/name.txt` (the `/` needs 8388 to be replaced by the similar looking `/` character) on the command line. 8389 8390 #### Default restricted characters {#restricted-characters} 8391 8392 The table below shows the characters that are replaced by default. 8393 8394 When a replacement character is found in a filename, this character 8395 will be escaped with the `‛` character to avoid ambiguous file names. 8396 (e.g. a file named `␀.txt` would shown as `‛␀.txt`) 8397 8398 Each cloud storage backend can use a different set of characters, 8399 which will be specified in the documentation for each backend. 8400 8401 | Character | Value | Replacement | 8402 | --------- |:-----:|:-----------:| 8403 | NUL | 0x00 | ␀ | 8404 | SOH | 0x01 | ␁ | 8405 | STX | 0x02 | ␂ | 8406 | ETX | 0x03 | ␃ | 8407 | EOT | 0x04 | ␄ | 8408 | ENQ | 0x05 | ␅ | 8409 | ACK | 0x06 | ␆ | 8410 | BEL | 0x07 | ␇ | 8411 | BS | 0x08 | ␈ | 8412 | HT | 0x09 | ␉ | 8413 | LF | 0x0A | ␊ | 8414 | VT | 0x0B | ␋ | 8415 | FF | 0x0C | ␌ | 8416 | CR | 0x0D | ␍ | 8417 | SO | 0x0E | ␎ | 8418 | SI | 0x0F | ␏ | 8419 | DLE | 0x10 | ␐ | 8420 | DC1 | 0x11 | ␑ | 8421 | DC2 | 0x12 | ␒ | 8422 | DC3 | 0x13 | ␓ | 8423 | DC4 | 0x14 | ␔ | 8424 | NAK | 0x15 | ␕ | 8425 | SYN | 0x16 | ␖ | 8426 | ETB | 0x17 | ␗ | 8427 | CAN | 0x18 | ␘ | 8428 | EM | 0x19 | ␙ | 8429 | SUB | 0x1A | ␚ | 8430 | ESC | 0x1B | ␛ | 8431 | FS | 0x1C | ␜ | 8432 | GS | 0x1D | ␝ | 8433 | RS | 0x1E | ␞ | 8434 | US | 0x1F | ␟ | 8435 | / | 0x2F | / | 8436 | DEL | 0x7F | ␡ | 8437 8438 The default encoding will also encode these file names as they are 8439 problematic with many cloud storage systems. 8440 8441 | File name | Replacement | 8442 | --------- |:-----------:| 8443 | . | . | 8444 | .. | .. | 8445 8446 #### Invalid UTF-8 bytes {#invalid-utf8} 8447 8448 Some backends only support a sequence of well formed UTF-8 bytes 8449 as file or directory names. 8450 8451 In this case all invalid UTF-8 bytes will be replaced with a quoted 8452 representation of the byte value to allow uploading a file to such a 8453 backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`. 8454 8455 A common source of invalid UTF-8 bytes are local filesystems, that store 8456 names in a different encoding than UTF-8 or UTF-16, like latin1. See the 8457 [local filenames](/local/#filenames) section for details. 8458 8459 #### Encoding option {#encoding} 8460 8461 Most backends have an encoding options, specified as a flag 8462 `--backend-encoding` where `backend` is the name of the backend, or as 8463 a config parameter `encoding` (you'll need to select the Advanced 8464 config in `rclone config` to see it). 8465 8466 This will have default value which encodes and decodes characters in 8467 such a way as to preserve the maximum number of characters (see 8468 above). 8469 8470 However this can be incorrect in some scenarios, for example if you 8471 have a Windows file system with characters such as `*` and `?` that 8472 you want to remain as those characters on the remote rather than being 8473 translated to `*` and `?`. 8474 8475 The `--backend-encoding` flags allow you to change that. You can 8476 disable the encoding completely with `--backend-encoding None` or set 8477 `encoding = None` in the config file. 8478 8479 Encoding takes a comma separated list of encodings. You can see the 8480 list of all available characters by passing an invalid value to this 8481 flag, eg `--local-encoding "help"` and `rclone help flags encoding` 8482 will show you the defaults for the backends. 8483 8484 | Encoding | Characters | 8485 | --------- | ---------- | 8486 | Asterisk | `*` | 8487 | BackQuote | `` ` `` | 8488 | BackSlash | `\` | 8489 | Colon | `:` | 8490 | CrLf | CR 0x0D, LF 0x0A | 8491 | Ctl | All control characters 0x00-0x1F | 8492 | Del | DEL 0x7F | 8493 | Dollar | `$` | 8494 | Dot | `.` | 8495 | DoubleQuote | `"` | 8496 | Hash | `#` | 8497 | InvalidUtf8 | An invalid UTF-8 character (eg latin1) | 8498 | LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string | 8499 | LeftPeriod | `.` on the left of a string | 8500 | LeftSpace | SPACE on the left of a string | 8501 | LeftTilde | `~` on the left of a string | 8502 | LtGt | `<`, `>` | 8503 | None | No characters are encoded | 8504 | Percent | `%` | 8505 | Pipe | <code>\|</code> | 8506 | Question | `?` | 8507 | RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | 8508 | RightPeriod | `.` on the right of a string | 8509 | RightSpace | SPACE on the right of a string | 8510 | SingleQuote | `'` | 8511 | Slash | `/` | 8512 8513 To take a specific example, the FTP backend's default encoding is 8514 8515 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 8516 8517 However, let's say the FTP server is running on Windows and can't have 8518 any of the invalid Windows characters in file names. You are backing 8519 up Linux servers to this FTP server which do have those characters in 8520 file names. So you would add the Windows set which are 8521 8522 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 8523 8524 to the existing ones, giving: 8525 8526 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 8527 8528 This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file. 8529 8530 Or let's say you have a Windows server but you want to preserve `*` 8531 and `?`, you would then have this as the encoding (the Windows 8532 encoding minus `Asterisk` and `Question`). 8533 8534 Slash,LtGt,DoubleQuote,Colon,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 8535 8536 This can be specified using the `--local-encoding` flag or using an 8537 `encoding` parameter in the config file. 8538 8539 ### MIME Type ### 8540 8541 MIME types (also known as media types) classify types of documents 8542 using a simple text classification, eg `text/html` or 8543 `application/pdf`. 8544 8545 Some cloud storage systems support reading (`R`) the MIME type of 8546 objects and some support writing (`W`) the MIME type of objects. 8547 8548 The MIME type can be important if you are serving files directly to 8549 HTTP from the storage system. 8550 8551 If you are copying from a remote which supports reading (`R`) to a 8552 remote which supports writing (`W`) then rclone will preserve the MIME 8553 types. Otherwise they will be guessed from the extension, or the 8554 remote itself may assign the MIME type. 8555 8556 ## Optional Features ## 8557 8558 All the remotes support a basic set of features, but there are some 8559 optional features supported by some remotes used to make some 8560 operations more efficient. 8561 8562 | Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir | 8563 | ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:| :------: | 8564 | 1Fichier | No | No | No | No | No | No | No | No | No | Yes | 8565 | Amazon Drive | Yes | No | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 8566 | Amazon S3 | No | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 8567 | Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 8568 | Box | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | No | Yes | 8569 | Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | 8570 | Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | Yes | Yes | 8571 | FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 8572 | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 8573 | Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 8574 | Google Photos | No | No | No | No | No | No | No | No | No | No | 8575 | HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 8576 | Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 8577 | Jottacloud | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | 8578 | Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 8579 | Mega | Yes | No | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 8580 | Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | 8581 | Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 8582 | Microsoft OneDrive | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | Yes | Yes | Yes | 8583 | OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | 8584 | Openstack Swift | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 8585 | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 8586 | premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes | 8587 | put.io | Yes | No | Yes | Yes | Yes | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 8588 | QingStor | No | Yes | No | No | No | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 8589 | SFTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 8590 | SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | 8591 | WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 8592 | Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | 8593 | The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 8594 8595 ### Purge ### 8596 8597 This deletes a directory quicker than just deleting all the files in 8598 the directory. 8599 8600 † Note Swift and Hubic implement this in order to delete directory 8601 markers but they don't actually have a quicker way of deleting files 8602 other than deleting them individually. 8603 8604 ‡ StreamUpload is not supported with Nextcloud 8605 8606 ### Copy ### 8607 8608 Used when copying an object to and from the same remote. This known 8609 as a server side copy so you can copy a file without downloading it 8610 and uploading it again. It is used if you use `rclone copy` or 8611 `rclone move` if the remote doesn't support `Move` directly. 8612 8613 If the server doesn't support `Copy` directly then for copy operations 8614 the file is downloaded then re-uploaded. 8615 8616 ### Move ### 8617 8618 Used when moving/renaming an object on the same remote. This is known 8619 as a server side move of a file. This is used in `rclone move` if the 8620 server doesn't support `DirMove`. 8621 8622 If the server isn't capable of `Move` then rclone simulates it with 8623 `Copy` then delete. If the server doesn't support `Copy` then rclone 8624 will download the file and re-upload it. 8625 8626 ### DirMove ### 8627 8628 This is used to implement `rclone move` to move a directory if 8629 possible. If it isn't then it will use `Move` on each file (which 8630 falls back to `Copy` then download and upload - see `Move` section). 8631 8632 ### CleanUp ### 8633 8634 This is used for emptying the trash for a remote by `rclone cleanup`. 8635 8636 If the server can't do `CleanUp` then `rclone cleanup` will return an 8637 error. 8638 8639 ### ListR ### 8640 8641 The remote supports a recursive list to list all the contents beneath 8642 a directory quickly. This enables the `--fast-list` flag to work. 8643 See the [rclone docs](/docs/#fast-list) for more details. 8644 8645 ### StreamUpload ### 8646 8647 Some remotes allow files to be uploaded without knowing the file size 8648 in advance. This allows certain operations to work without spooling the 8649 file to local disk first, e.g. `rclone rcat`. 8650 8651 ### LinkSharing ### 8652 8653 Sets the necessary permissions on a file or folder and prints a link 8654 that allows others to access them, even if they don't have an account 8655 on the particular cloud provider. 8656 8657 ### About ### 8658 8659 This is used to fetch quota information from the remote, like bytes 8660 used/free/quota and bytes used in the trash. 8661 8662 This is also used to return the space used, available for `rclone mount`. 8663 8664 If the server can't do `About` then `rclone about` will return an 8665 error. 8666 8667 ### EmptyDir ### 8668 8669 The remote supports empty directories. See [Limitations](/bugs/#limitations) 8670 for details. Most Object/Bucket based remotes do not support this. 8671 8672 # Global Flags 8673 8674 This describes the global flags available to every rclone command 8675 split into two groups, non backend and backend flags. 8676 8677 ## Non Backend Flags 8678 8679 These flags are available for every command. 8680 8681 ``` 8682 --ask-password Allow prompt for password for encrypted configuration. (default true) 8683 --auto-confirm If enabled, do not request console confirmation. 8684 --backup-dir string Make backups into hierarchy based in DIR. 8685 --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name. 8686 --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer. (default 16M) 8687 --bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable. 8688 --ca-cert string CA certificate used to verify servers 8689 --cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone") 8690 --checkers int Number of checkers to run in parallel. (default 8) 8691 -c, --checksum Skip based on checksum (if available) & size, not mod-time & size 8692 --client-cert string Client SSL certificate (PEM) for mutual TLS auth 8693 --client-key string Client SSL private key (PEM) for mutual TLS auth 8694 --compare-dest string Include additional server-side path during comparison. 8695 --config string Config file. (default "$HOME/.config/rclone/rclone.conf") 8696 --contimeout duration Connect timeout (default 1m0s) 8697 --copy-dest string Implies --compare-dest but also copies files from path into destination. 8698 --cpuprofile string Write cpu profile to file 8699 --delete-after When synchronizing, delete files on destination after transferring (default) 8700 --delete-before When synchronizing, delete files on destination before transferring 8701 --delete-during When synchronizing, delete files during transfer 8702 --delete-excluded Delete files on dest excluded from sync 8703 --disable string Disable a comma separated list of features. Use help to see a list. 8704 -n, --dry-run Do a trial run with no permanent changes 8705 --dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles 8706 --dump-bodies Dump HTTP headers and bodies - may contain sensitive info 8707 --dump-headers Dump HTTP headers - may contain sensitive info 8708 --exclude stringArray Exclude files matching pattern 8709 --exclude-from stringArray Read exclude patterns from file 8710 --exclude-if-present string Exclude directories if filename is present 8711 --expect-continue-timeout duration Timeout when using expect / 100-continue in HTTP (default 1s) 8712 --fast-list Use recursive list if available. Uses more memory but fewer transactions. 8713 --files-from stringArray Read list of source-file names from file 8714 -f, --filter stringArray Add a file-filtering rule 8715 --filter-from stringArray Read filtering patterns from a file 8716 --ignore-case Ignore case in filters (case insensitive) 8717 --ignore-case-sync Ignore case when synchronizing 8718 --ignore-checksum Skip post copy check of checksums. 8719 --ignore-errors delete even if there are I/O errors 8720 --ignore-existing Skip all files that exist on destination 8721 --ignore-size Ignore size when skipping use mod-time or checksum. 8722 -I, --ignore-times Don't skip files that match size and time - transfer all files 8723 --immutable Do not modify files. Fail if existing files have been modified. 8724 --include stringArray Include files matching pattern 8725 --include-from stringArray Read include patterns from file 8726 --log-file string Log everything to this file 8727 --log-format string Comma separated list of log format options (default "date,time") 8728 --log-level string Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE") 8729 --low-level-retries int Number of low level retries to do. (default 10) 8730 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8731 --max-backlog int Maximum number of objects in sync or check backlog. (default 10000) 8732 --max-delete int When synchronizing, limit the number of deletes (default -1) 8733 --max-depth int If set limits the recursion depth to this. (default -1) 8734 --max-duration duration Maximum duration rclone will transfer data for. 8735 --max-size SizeSuffix Only transfer files smaller than this in k or suffix b|k|M|G (default off) 8736 --max-stats-groups int Maximum number of stats groups to keep in memory. On max oldest is discarded. (default 1000) 8737 --max-transfer SizeSuffix Maximum size of data to transfer. (default off) 8738 --memprofile string Write memory profile to file 8739 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8740 --min-size SizeSuffix Only transfer files bigger than this in k or suffix b|k|M|G (default off) 8741 --modify-window duration Max time diff to be considered the same (default 1ns) 8742 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size. (default 250M) 8743 --multi-thread-streams int Max number of streams to use for multi-thread downloads. (default 4) 8744 --no-check-certificate Do not verify the server SSL certificate. Insecure. 8745 --no-check-dest Don't check the destination, copy regardless. 8746 --no-gzip-encoding Don't set Accept-Encoding: gzip. 8747 --no-traverse Don't traverse destination file system on copy. 8748 --no-update-modtime Don't update destination mod-time if files identical. 8749 --order-by string Instructions on how to order the transfers, eg 'size,descending' 8750 --password-command SpaceSepList Command for supplying password for encrypted configuration. 8751 -P, --progress Show progress during transfer. 8752 -q, --quiet Print as little stuff as possible 8753 --rc Enable the remote control server. 8754 --rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572") 8755 --rc-allow-origin string Set the allowed origin for CORS. 8756 --rc-baseurl string Prefix for URLs - leave blank for root. 8757 --rc-cert string SSL PEM key (concatenation of certificate and CA certificate) 8758 --rc-client-ca string Client certificate authority to verify clients with 8759 --rc-files string Path to local files to serve on the HTTP server. 8760 --rc-htpasswd string htpasswd file - if not provided no authentication is done 8761 --rc-job-expire-duration duration expire finished async jobs older than this value (default 1m0s) 8762 --rc-job-expire-interval duration interval to check for expired async jobs (default 10s) 8763 --rc-key string SSL PEM Private key 8764 --rc-max-header-bytes int Maximum size of request header (default 4096) 8765 --rc-no-auth Don't require auth for certain methods. 8766 --rc-pass string Password for authentication. 8767 --rc-realm string realm for authentication (default "rclone") 8768 --rc-serve Enable the serving of remote objects. 8769 --rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s) 8770 --rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s) 8771 --rc-user string User name for authentication. 8772 --rc-web-fetch-url string URL to fetch the releases for webgui. (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest") 8773 --rc-web-gui Launch WebGUI on localhost 8774 --rc-web-gui-force-update Force update to latest version of web gui 8775 --rc-web-gui-no-open-browser Don't open the browser automatically 8776 --rc-web-gui-update Check and update to latest version of web gui 8777 --retries int Retry operations this many times if they fail (default 3) 8778 --retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable) 8779 --size-only Skip based on size only, not mod-time or checksum 8780 --stats duration Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable) (default 1m0s) 8781 --stats-file-name-length int Max file name length in stats. 0 for no limit (default 45) 8782 --stats-log-level string Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO") 8783 --stats-one-line Make the stats fit on one line. 8784 --stats-one-line-date Enables --stats-one-line and add current date/time prefix. 8785 --stats-one-line-date-format string Enables --stats-one-line-date and uses custom formatted date. Enclose date string in double quotes ("). See https://golang.org/pkg/time/#Time.Format 8786 --stats-unit string Show data rate in stats as either 'bits' or 'bytes'/s (default "bytes") 8787 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown. Upload starts after reaching cutoff or when file ends. (default 100k) 8788 --suffix string Suffix to add to changed files. 8789 --suffix-keep-extension Preserve the extension when using --suffix. 8790 --syslog Use Syslog for logging 8791 --syslog-facility string Facility for syslog, eg KERN,USER,... (default "DAEMON") 8792 --timeout duration IO idle timeout (default 5m0s) 8793 --tpslimit float Limit HTTP transactions per second to this. 8794 --tpslimit-burst int Max burst of transactions for --tpslimit. (default 1) 8795 --track-renames When synchronizing, track file renames and do a server side move if possible 8796 --transfers int Number of file transfers to run in parallel. (default 4) 8797 -u, --update Skip files that are newer on the destination. 8798 --use-cookies Enable session cookiejar. 8799 --use-json-log Use json log format. 8800 --use-mmap Use mmap allocator (see docs). 8801 --use-server-modtime Use server modified time instead of object metadata 8802 --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.51.0") 8803 -v, --verbose count Print lots more stuff (repeat for more) 8804 ``` 8805 8806 ## Backend Flags 8807 8808 These flags are available for every command. They control the backends 8809 and may be set in the config file. 8810 8811 ``` 8812 --acd-auth-url string Auth server URL. 8813 --acd-client-id string Amazon Application Client ID. 8814 --acd-client-secret string Amazon Application Client Secret. 8815 --acd-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 8816 --acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink. (default 9G) 8817 --acd-token-url string Token server url. 8818 --acd-upload-wait-per-gb Duration Additional time per GB to wait after a failed complete upload to see if it appears. (default 3m0s) 8819 --alias-remote string Remote or path to alias. 8820 --azureblob-access-tier string Access tier of blob: hot, cool or archive. 8821 --azureblob-account string Storage Account Name (leave blank to use SAS URL or Emulator) 8822 --azureblob-chunk-size SizeSuffix Upload chunk size (<= 100MB). (default 4M) 8823 --azureblob-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8) 8824 --azureblob-endpoint string Endpoint for the service 8825 --azureblob-key string Storage Account Key (leave blank to use SAS URL or Emulator) 8826 --azureblob-list-chunk int Size of blob list. (default 5000) 8827 --azureblob-sas-url string SAS URL for container level access only 8828 --azureblob-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (<= 256MB). (default 256M) 8829 --azureblob-use-emulator Uses local storage emulator if provided as 'true' (leave blank if using real azure storage endpoint) 8830 --b2-account string Account ID or Application Key ID 8831 --b2-chunk-size SizeSuffix Upload chunk size. Must fit in memory. (default 96M) 8832 --b2-disable-checksum Disable checksums for large (> upload cutoff) files 8833 --b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d. (default 1w) 8834 --b2-download-url string Custom endpoint for downloads. 8835 --b2-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 8836 --b2-endpoint string Endpoint for the service. 8837 --b2-hard-delete Permanently delete files on remote removal, otherwise hide files. 8838 --b2-key string Application Key 8839 --b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging. 8840 --b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload. (default 200M) 8841 --b2-versions Include old versions in directory listings. 8842 --box-box-config-file string Box App config.json location 8843 --box-box-sub-type string (default "user") 8844 --box-client-id string Box App Client Id. 8845 --box-client-secret string Box App Client Secret 8846 --box-commit-retries int Max number of times to try committing a multipart file. (default 100) 8847 --box-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot) 8848 --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50MB). (default 50M) 8849 --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s) 8850 --cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming. 8851 --cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend") 8852 --cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M) 8853 --cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G) 8854 --cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend") 8855 --cache-db-purge Clear all the cached data for this remote on start. 8856 --cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s) 8857 --cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s) 8858 --cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server 8859 --cache-plex-password string The password of the Plex user 8860 --cache-plex-url string The URL of the Plex server 8861 --cache-plex-username string The username of the Plex user 8862 --cache-read-retries int How many times to retry a read from a cache storage. (default 10) 8863 --cache-remote string Remote to cache. 8864 --cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1) 8865 --cache-tmp-upload-path string Directory to keep temporary files until they are uploaded. 8866 --cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s) 8867 --cache-workers int How many workers should run in parallel to download chunks. (default 4) 8868 --cache-writes Cache file data on writes through the FS 8869 --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks. (default 2G) 8870 --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks. 8871 --chunker-hash-type string Choose how chunker handles hash sums. All modes but "none" require metadata. (default "md5") 8872 --chunker-meta-format string Format of the metadata object or "none". By default "simplejson". (default "simplejson") 8873 --chunker-name-format string String format of chunk file names. (default "*.rclone_chunk.###") 8874 --chunker-remote string Remote to chunk/unchunk. 8875 --chunker-start-from int Minimum valid chunk number. Usually 0 or 1. (default 1) 8876 -L, --copy-links Follow symlinks and copy the pointed to item. 8877 --crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true) 8878 --crypt-filename-encryption string How to encrypt the filenames. (default "standard") 8879 --crypt-password string Password or pass phrase for encryption. 8880 --crypt-password2 string Password or pass phrase for salt. Optional but recommended. 8881 --crypt-remote string Remote to encrypt/decrypt. 8882 --crypt-show-mapping For all files listed show how the names encrypt. 8883 --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded. 8884 --drive-allow-import-name-change Allow the filetype to change when uploading Google docs (e.g. file.doc to file.docx). This will confuse sync and reupload every time. 8885 --drive-alternate-export Use alternate export URLs for google documents export., 8886 --drive-auth-owner-only Only consider files owned by the authenticated user. 8887 --drive-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 8M) 8888 --drive-client-id string Google Application Client Id 8889 --drive-client-secret string Google Application Client Secret 8890 --drive-disable-http2 Disable drive using http2 (default true) 8891 --drive-encoding MultiEncoder This sets the encoding for the backend. (default InvalidUtf8) 8892 --drive-export-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg") 8893 --drive-formats string Deprecated: see export_formats 8894 --drive-impersonate string Impersonate this user when using a service account. 8895 --drive-import-formats string Comma separated list of preferred formats for uploading Google docs. 8896 --drive-keep-revision-forever Keep new head revision of each file forever. 8897 --drive-list-chunk int Size of listing chunk 100-1000. 0 to disable. (default 1000) 8898 --drive-pacer-burst int Number of API calls to allow without sleeping. (default 100) 8899 --drive-pacer-min-sleep Duration Minimum time to sleep between API calls. (default 100ms) 8900 --drive-root-folder-id string ID of the root folder 8901 --drive-scope string Scope that rclone should use when requesting access from drive. 8902 --drive-server-side-across-configs Allow server side operations (eg copy) to work across different drive configs. 8903 --drive-service-account-credentials string Service Account Credentials JSON blob 8904 --drive-service-account-file string Service Account Credentials JSON file path 8905 --drive-shared-with-me Only show files that are shared with me. 8906 --drive-size-as-quota Show sizes as storage quota usage, not actual size. 8907 --drive-skip-checksum-gphotos Skip MD5 checksum on Google photos and videos only. 8908 --drive-skip-gdocs Skip google documents in all listings. 8909 --drive-stop-on-upload-limit Make upload limit errors be fatal 8910 --drive-team-drive string ID of the Team Drive 8911 --drive-trashed-only Only show files that are in the trash. 8912 --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8M) 8913 --drive-use-created-date Use file created date instead of modified date., 8914 --drive-use-shared-date Use date file was shared instead of modified date. 8915 --drive-use-trash Send files to the trash instead of deleting permanently. (default true) 8916 --drive-v2-download-min-size SizeSuffix If Object's are greater, use drive v2 API to download. (default off) 8917 --dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M) 8918 --dropbox-client-id string Dropbox App Client Id 8919 --dropbox-client-secret string Dropbox App Client Secret 8920 --dropbox-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot) 8921 --dropbox-impersonate string Impersonate this user when using a business account. 8922 --fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl 8923 --fichier-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot) 8924 --fichier-shared-folder string If you want to download a shared folder, add this parameter 8925 --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited 8926 --ftp-disable-epsv Disable using EPSV even if server advertises support 8927 --ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot) 8928 --ftp-host string FTP host to connect to 8929 --ftp-no-check-certificate Do not verify the TLS certificate of the server 8930 --ftp-pass string FTP password 8931 --ftp-port string FTP port, leave blank to use default (21) 8932 --ftp-tls Use FTP over TLS (Implicit) 8933 --ftp-user string FTP username, leave blank for current username, $USER 8934 --gcs-bucket-acl string Access Control List for new buckets. 8935 --gcs-bucket-policy-only Access checks should use bucket-level IAM policies. 8936 --gcs-client-id string Google Application Client Id 8937 --gcs-client-secret string Google Application Client Secret 8938 --gcs-encoding MultiEncoder This sets the encoding for the backend. (default Slash,CrLf,InvalidUtf8,Dot) 8939 --gcs-location string Location for the newly created buckets. 8940 --gcs-object-acl string Access Control List for new objects. 8941 --gcs-project-number string Project number. 8942 --gcs-service-account-file string Service Account Credentials JSON file path 8943 --gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage. 8944 --gphotos-client-id string Google Application Client Id 8945 --gphotos-client-secret string Google Application Client Secret 8946 --gphotos-read-only Set to make the Google Photos backend read only. 8947 --gphotos-read-size Set to read the size of media items. 8948 --http-headers CommaSepList Set HTTP headers for all transactions 8949 --http-no-head Don't use HEAD requests to find file sizes in dir listing 8950 --http-no-slash Set this if the site doesn't end directories with / 8951 --http-url string URL of http host to connect to 8952 --hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G) 8953 --hubic-client-id string Hubic Client Id 8954 --hubic-client-secret string Hubic Client Secret 8955 --hubic-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8) 8956 --hubic-no-chunk Don't chunk files during streaming upload. 8957 --jottacloud-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot) 8958 --jottacloud-hard-delete Delete files permanently rather than putting them into the trash. 8959 --jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required. (default 10M) 8960 --jottacloud-unlink Remove existing public link to file/folder with link command rather than creating. 8961 --jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's. (default 10M) 8962 --koofr-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 8963 --koofr-endpoint string The Koofr API endpoint to use (default "https://app.koofr.net") 8964 --koofr-mountid string Mount ID of the mount to use. If omitted, the primary mount is used. 8965 --koofr-password string Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) 8966 --koofr-setmtime Does the backend support setting modification time. Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend. (default true) 8967 --koofr-user string Your Koofr user name 8968 -l, --links Translate symlinks to/from regular files with a '.rclonelink' extension 8969 --local-case-insensitive Force the filesystem to report itself as case insensitive 8970 --local-case-sensitive Force the filesystem to report itself as case sensitive. 8971 --local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot) 8972 --local-no-check-updated Don't check to see if the files change during upload 8973 --local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated) 8974 --local-nounc string Disable UNC (long path names) conversion on Windows 8975 --mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true) 8976 --mailru-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot) 8977 --mailru-pass string Password 8978 --mailru-speedup-enable Skip full upload if there is another file with same data hash. (default true) 8979 --mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash). (default "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf") 8980 --mailru-speedup-max-disk SizeSuffix This option allows you to disable speedup (put by hash) for large files (default 3G) 8981 --mailru-speedup-max-memory SizeSuffix Files larger than the size given below will always be hashed on disk. (default 32M) 8982 --mailru-user string User name (usually email) 8983 --mega-debug Output more debug from Mega. 8984 --mega-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 8985 --mega-hard-delete Delete files permanently rather than putting them into the trash. 8986 --mega-pass string Password. 8987 --mega-user string User name 8988 -x, --one-file-system Don't cross filesystem boundaries (unix/macOS only). 8989 --onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes). (default 10M) 8990 --onedrive-client-id string Microsoft App Client Id 8991 --onedrive-client-secret string Microsoft App Client Secret 8992 --onedrive-drive-id string The ID of the drive to use 8993 --onedrive-drive-type string The type of the drive ( personal | business | documentLibrary ) 8994 --onedrive-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot) 8995 --onedrive-expose-onenote-files Set to make OneNote files show up in directory listings. 8996 --opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size. (default 10M) 8997 --opendrive-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) 8998 --opendrive-password string Password. 8999 --opendrive-username string Username 9000 --pcloud-client-id string Pcloud App Client Id 9001 --pcloud-client-secret string Pcloud App Client Secret 9002 --pcloud-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9003 --premiumizeme-encoding MultiEncoder This sets the encoding for the backend. (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9004 --putio-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9005 --qingstor-access-key-id string QingStor Access Key ID 9006 --qingstor-chunk-size SizeSuffix Chunk size to use for uploading. (default 4M) 9007 --qingstor-connection-retries int Number of connection retries. (default 3) 9008 --qingstor-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Ctl,InvalidUtf8) 9009 --qingstor-endpoint string Enter a endpoint URL to connection QingStor API. 9010 --qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 9011 --qingstor-secret-access-key string QingStor Secret Access Key (password) 9012 --qingstor-upload-concurrency int Concurrency for multipart uploads. (default 1) 9013 --qingstor-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200M) 9014 --qingstor-zone string Zone to connect to. 9015 --s3-access-key-id string AWS Access Key ID. 9016 --s3-acl string Canned ACL used when creating buckets and storing or copying objects. 9017 --s3-bucket-acl string Canned ACL used when creating buckets. 9018 --s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M) 9019 --s3-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 5G) 9020 --s3-disable-checksum Don't store MD5 checksum with object metadata 9021 --s3-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 9022 --s3-endpoint string Endpoint for S3 API. 9023 --s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 9024 --s3-force-path-style If true use path style access if false use virtual hosted style. (default true) 9025 --s3-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. 9026 --s3-list-chunk int Size of listing chunk (response list for each ListObject S3 request). (default 1000) 9027 --s3-location-constraint string Location constraint - must be set to match the Region. 9028 --s3-provider string Choose your S3 provider. 9029 --s3-region string Region to connect to. 9030 --s3-secret-access-key string AWS Secret Access Key (password) 9031 --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3. 9032 --s3-session-token string An AWS session token 9033 --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key. 9034 --s3-storage-class string The storage class to use when storing new objects in S3. 9035 --s3-upload-concurrency int Concurrency for multipart uploads. (default 4) 9036 --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200M) 9037 --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint. 9038 --s3-v2-auth If true use v2 authentication. 9039 --sftp-ask-password Allow asking for SFTP password when needed. 9040 --sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available. 9041 --sftp-host string SSH host to connect to 9042 --sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent. 9043 --sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. 9044 --sftp-key-use-agent When set forces the usage of the ssh-agent. 9045 --sftp-md5sum-command string The command used to read md5 hashes. Leave blank for autodetect. 9046 --sftp-pass string SSH password, leave blank to use ssh-agent. 9047 --sftp-path-override string Override path used by SSH connection. 9048 --sftp-port string SSH port, leave blank to use default (22) 9049 --sftp-set-modtime Set the modified time on the remote if set. (default true) 9050 --sftp-sha1sum-command string The command used to read sha1 hashes. Leave blank for autodetect. 9051 --sftp-skip-links Set to skip any symlinks and any other non regular files. 9052 --sftp-use-insecure-cipher Enable the use of insecure ciphers and key exchange methods. 9053 --sftp-user string SSH username, leave blank for current username, ncw 9054 --sharefile-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 64M) 9055 --sharefile-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot) 9056 --sharefile-endpoint string Endpoint for API calls. 9057 --sharefile-root-folder-id string ID of the root folder 9058 --sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload. (default 128M) 9059 --skip-links Don't warn about skipped symlinks. 9060 --sugarsync-access-key-id string Sugarsync Access Key ID. 9061 --sugarsync-app-id string Sugarsync App ID. 9062 --sugarsync-authorization string Sugarsync authorization 9063 --sugarsync-authorization-expiry string Sugarsync authorization expiry 9064 --sugarsync-deleted-id string Sugarsync deleted folder id 9065 --sugarsync-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Ctl,InvalidUtf8,Dot) 9066 --sugarsync-hard-delete Permanently delete files if true 9067 --sugarsync-private-access-key string Sugarsync Private Access Key 9068 --sugarsync-refresh-token string Sugarsync refresh token 9069 --sugarsync-root-id string Sugarsync root id 9070 --sugarsync-user string Sugarsync user 9071 --swift-application-credential-id string Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 9072 --swift-application-credential-name string Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 9073 --swift-application-credential-secret string Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 9074 --swift-auth string Authentication URL for server (OS_AUTH_URL). 9075 --swift-auth-token string Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 9076 --swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 9077 --swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G) 9078 --swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 9079 --swift-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8) 9080 --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") 9081 --swift-env-auth Get swift credentials from environment variables in standard OpenStack form. 9082 --swift-key string API key or password (OS_PASSWORD). 9083 --swift-no-chunk Don't chunk files during streaming upload. 9084 --swift-region string Region name - optional (OS_REGION_NAME) 9085 --swift-storage-policy string The storage policy to use when creating a new container 9086 --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) 9087 --swift-tenant string Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 9088 --swift-tenant-domain string Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 9089 --swift-tenant-id string Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 9090 --swift-user string User name to log in (OS_USERNAME). 9091 --swift-user-id string User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 9092 --union-remotes string List of space separated remotes. 9093 --webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon) 9094 --webdav-bearer-token-command string Command to run to get a bearer token 9095 --webdav-pass string Password. 9096 --webdav-url string URL of http host to connect to 9097 --webdav-user string User name 9098 --webdav-vendor string Name of the Webdav site/service/software you are using 9099 --yandex-client-id string Yandex Client Id 9100 --yandex-client-secret string Yandex Client Secret 9101 --yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot) 9102 --yandex-unlink Remove existing public link to file/folder with link command rather than creating. 9103 ``` 9104 9105 1Fichier 9106 ----------------------------------------- 9107 9108 This is a backend for the [1ficher](https://1fichier.com) cloud 9109 storage service. Note that a Premium subscription is required to use 9110 the API. 9111 9112 Paths are specified as `remote:path` 9113 9114 Paths may be as deep as required, eg `remote:directory/subdirectory`. 9115 9116 The initial setup for 1Fichier involves getting the API key from the website which you 9117 need to do in your browser. 9118 9119 Here is an example of how to make a remote called `remote`. First run: 9120 9121 rclone config 9122 9123 This will guide you through an interactive setup process: 9124 9125 ``` 9126 No remotes found - make a new one 9127 n) New remote 9128 s) Set configuration password 9129 q) Quit config 9130 n/s/q> n 9131 name> remote 9132 Type of storage to configure. 9133 Enter a string value. Press Enter for the default (""). 9134 Choose a number from below, or type in your own value 9135 [snip] 9136 XX / 1Fichier 9137 \ "fichier" 9138 [snip] 9139 Storage> fichier 9140 ** See help for fichier backend at: https://rclone.org/fichier/ ** 9141 9142 Your API Key, get it from https://1fichier.com/console/params.pl 9143 Enter a string value. Press Enter for the default (""). 9144 api_key> example_key 9145 9146 Edit advanced config? (y/n) 9147 y) Yes 9148 n) No 9149 y/n> 9150 Remote config 9151 -------------------- 9152 [remote] 9153 type = fichier 9154 api_key = example_key 9155 -------------------- 9156 y) Yes this is OK 9157 e) Edit this remote 9158 d) Delete this remote 9159 y/e/d> y 9160 ``` 9161 9162 Once configured you can then use `rclone` like this, 9163 9164 List directories in top level of your 1Fichier account 9165 9166 rclone lsd remote: 9167 9168 List all the files in your 1Fichier account 9169 9170 rclone ls remote: 9171 9172 To copy a local directory to a 1Fichier directory called backup 9173 9174 rclone copy /home/source remote:backup 9175 9176 ### Modified time and hashes ### 9177 9178 1Fichier does not support modification times. It supports the Whirlpool hash algorithm. 9179 9180 ### Duplicated files ### 9181 9182 1Fichier can have two files with exactly the same name and path (unlike a 9183 normal file system). 9184 9185 Duplicated files cause problems with the syncing and you will see 9186 messages in the log about duplicates. 9187 9188 #### Restricted filename characters 9189 9190 In addition to the [default restricted characters set](/overview/#restricted-characters) 9191 the following characters are also replaced: 9192 9193 | Character | Value | Replacement | 9194 | --------- |:-----:|:-----------:| 9195 | \ | 0x5C | \ | 9196 | < | 0x3C | < | 9197 | > | 0x3E | > | 9198 | " | 0x22 | " | 9199 | $ | 0x24 | $ | 9200 | ` | 0x60 | ` | 9201 | ' | 0x27 | ' | 9202 9203 File names can also not start or end with the following characters. 9204 These only get replaced if they are first or last character in the 9205 name: 9206 9207 | Character | Value | Replacement | 9208 | --------- |:-----:|:-----------:| 9209 | SP | 0x20 | ␠ | 9210 9211 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 9212 as they can't be used in JSON strings. 9213 9214 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/fichier/fichier.go then run make backenddocs --> 9215 ### Standard Options 9216 9217 Here are the standard options specific to fichier (1Fichier). 9218 9219 #### --fichier-api-key 9220 9221 Your API Key, get it from https://1fichier.com/console/params.pl 9222 9223 - Config: api_key 9224 - Env Var: RCLONE_FICHIER_API_KEY 9225 - Type: string 9226 - Default: "" 9227 9228 ### Advanced Options 9229 9230 Here are the advanced options specific to fichier (1Fichier). 9231 9232 #### --fichier-shared-folder 9233 9234 If you want to download a shared folder, add this parameter 9235 9236 - Config: shared_folder 9237 - Env Var: RCLONE_FICHIER_SHARED_FOLDER 9238 - Type: string 9239 - Default: "" 9240 9241 #### --fichier-encoding 9242 9243 This sets the encoding for the backend. 9244 9245 See: the [encoding section in the overview](/overview/#encoding) for more info. 9246 9247 - Config: encoding 9248 - Env Var: RCLONE_FICHIER_ENCODING 9249 - Type: MultiEncoder 9250 - Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot 9251 9252 <!--- autogenerated options stop --> 9253 9254 Alias 9255 ----------------------------------------- 9256 9257 The `alias` remote provides a new name for another remote. 9258 9259 Paths may be as deep as required or a local path, 9260 eg `remote:directory/subdirectory` or `/directory/subdirectory`. 9261 9262 During the initial setup with `rclone config` you will specify the target 9263 remote. The target remote can either be a local path or another remote. 9264 9265 Subfolders can be used in target remote. Assume a alias remote named `backup` 9266 with the target `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop` 9267 is exactly the same as invoking `rclone mkdir mydrive:private/backup/desktop`. 9268 9269 There will be no special handling of paths containing `..` segments. 9270 Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking 9271 `rclone mkdir mydrive:private/backup/../desktop`. 9272 The empty path is not allowed as a remote. To alias the current directory 9273 use `.` instead. 9274 9275 Here is an example of how to make a alias called `remote` for local folder. 9276 First run: 9277 9278 rclone config 9279 9280 This will guide you through an interactive setup process: 9281 9282 ``` 9283 No remotes found - make a new one 9284 n) New remote 9285 s) Set configuration password 9286 q) Quit config 9287 n/s/q> n 9288 name> remote 9289 Type of storage to configure. 9290 Choose a number from below, or type in your own value 9291 [snip] 9292 XX / Alias for an existing remote 9293 \ "alias" 9294 [snip] 9295 Storage> alias 9296 Remote or path to alias. 9297 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". 9298 remote> /mnt/storage/backup 9299 Remote config 9300 -------------------- 9301 [remote] 9302 remote = /mnt/storage/backup 9303 -------------------- 9304 y) Yes this is OK 9305 e) Edit this remote 9306 d) Delete this remote 9307 y/e/d> y 9308 Current remotes: 9309 9310 Name Type 9311 ==== ==== 9312 remote alias 9313 9314 e) Edit existing remote 9315 n) New remote 9316 d) Delete remote 9317 r) Rename remote 9318 c) Copy remote 9319 s) Set configuration password 9320 q) Quit config 9321 e/n/d/r/c/s/q> q 9322 ``` 9323 9324 Once configured you can then use `rclone` like this, 9325 9326 List directories in top level in `/mnt/storage/backup` 9327 9328 rclone lsd remote: 9329 9330 List all the files in `/mnt/storage/backup` 9331 9332 rclone ls remote: 9333 9334 Copy another local directory to the alias directory called source 9335 9336 rclone copy /home/source remote:source 9337 9338 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/alias/alias.go then run make backenddocs --> 9339 ### Standard Options 9340 9341 Here are the standard options specific to alias (Alias for an existing remote). 9342 9343 #### --alias-remote 9344 9345 Remote or path to alias. 9346 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". 9347 9348 - Config: remote 9349 - Env Var: RCLONE_ALIAS_REMOTE 9350 - Type: string 9351 - Default: "" 9352 9353 <!--- autogenerated options stop --> 9354 9355 Amazon Drive 9356 ----------------------------------------- 9357 9358 Amazon Drive, formerly known as Amazon Cloud Drive, is a cloud storage 9359 service run by Amazon for consumers. 9360 9361 ## Status 9362 9363 **Important:** rclone supports Amazon Drive only if you have your own 9364 set of API keys. Unfortunately the [Amazon Drive developer 9365 program](https://developer.amazon.com/amazon-drive) is now closed to 9366 new entries so if you don't already have your own set of keys you will 9367 not be able to use rclone with Amazon Drive. 9368 9369 For the history on why rclone no longer has a set of Amazon Drive API 9370 keys see [the forum](https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314). 9371 9372 If you happen to know anyone who works at Amazon then please ask them 9373 to re-instate rclone into the Amazon Drive developer program - thanks! 9374 9375 ## Setup 9376 9377 The initial setup for Amazon Drive involves getting a token from 9378 Amazon which you need to do in your browser. `rclone config` walks 9379 you through it. 9380 9381 The configuration process for Amazon Drive may involve using an [oauth 9382 proxy](https://github.com/ncw/oauthproxy). This is used to keep the 9383 Amazon credentials out of the source code. The proxy runs in Google's 9384 very secure App Engine environment and doesn't store any credentials 9385 which pass through it. 9386 9387 Since rclone doesn't currently have its own Amazon Drive credentials 9388 so you will either need to have your own `client_id` and 9389 `client_secret` with Amazon Drive, or use a a third party ouath proxy 9390 in which case you will need to enter `client_id`, `client_secret`, 9391 `auth_url` and `token_url`. 9392 9393 Note also if you are not using Amazon's `auth_url` and `token_url`, 9394 (ie you filled in something for those) then if setting up on a remote 9395 machine you can only use the [copying the config method of 9396 configuration](https://rclone.org/remote_setup/#configuring-by-copying-the-config-file) 9397 - `rclone authorize` will not work. 9398 9399 Here is an example of how to make a remote called `remote`. First run: 9400 9401 rclone config 9402 9403 This will guide you through an interactive setup process: 9404 9405 ``` 9406 No remotes found - make a new one 9407 n) New remote 9408 r) Rename remote 9409 c) Copy remote 9410 s) Set configuration password 9411 q) Quit config 9412 n/r/c/s/q> n 9413 name> remote 9414 Type of storage to configure. 9415 Choose a number from below, or type in your own value 9416 [snip] 9417 XX / Amazon Drive 9418 \ "amazon cloud drive" 9419 [snip] 9420 Storage> amazon cloud drive 9421 Amazon Application Client Id - required. 9422 client_id> your client ID goes here 9423 Amazon Application Client Secret - required. 9424 client_secret> your client secret goes here 9425 Auth server URL - leave blank to use Amazon's. 9426 auth_url> Optional auth URL 9427 Token server url - leave blank to use Amazon's. 9428 token_url> Optional token URL 9429 Remote config 9430 Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config. 9431 Use auto config? 9432 * Say Y if not sure 9433 * Say N if you are working on a remote or headless machine 9434 y) Yes 9435 n) No 9436 y/n> y 9437 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 9438 Log in and authorize rclone for access 9439 Waiting for code... 9440 Got code 9441 -------------------- 9442 [remote] 9443 client_id = your client ID goes here 9444 client_secret = your client secret goes here 9445 auth_url = Optional auth URL 9446 token_url = Optional token URL 9447 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"} 9448 -------------------- 9449 y) Yes this is OK 9450 e) Edit this remote 9451 d) Delete this remote 9452 y/e/d> y 9453 ``` 9454 9455 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 9456 machine with no Internet browser available. 9457 9458 Note that rclone runs a webserver on your local machine to collect the 9459 token as returned from Amazon. This only runs from the moment it 9460 opens your browser to the moment you get back the verification 9461 code. This is on `http://127.0.0.1:53682/` and this it may require 9462 you to unblock it temporarily if you are running a host firewall. 9463 9464 Once configured you can then use `rclone` like this, 9465 9466 List directories in top level of your Amazon Drive 9467 9468 rclone lsd remote: 9469 9470 List all the files in your Amazon Drive 9471 9472 rclone ls remote: 9473 9474 To copy a local directory to an Amazon Drive directory called backup 9475 9476 rclone copy /home/source remote:backup 9477 9478 ### Modified time and MD5SUMs ### 9479 9480 Amazon Drive doesn't allow modification times to be changed via 9481 the API so these won't be accurate or used for syncing. 9482 9483 It does store MD5SUMs so for a more accurate sync, you can use the 9484 `--checksum` flag. 9485 9486 #### Restricted filename characters 9487 9488 | Character | Value | Replacement | 9489 | --------- |:-----:|:-----------:| 9490 | NUL | 0x00 | ␀ | 9491 | / | 0x2F | / | 9492 9493 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 9494 as they can't be used in JSON strings. 9495 9496 ### Deleting files ### 9497 9498 Any files you delete with rclone will end up in the trash. Amazon 9499 don't provide an API to permanently delete files, nor to empty the 9500 trash, so you will have to do that with one of Amazon's apps or via 9501 the Amazon Drive website. As of November 17, 2016, files are 9502 automatically deleted by Amazon from the trash after 30 days. 9503 9504 ### Using with non `.com` Amazon accounts ### 9505 9506 Let's say you usually use `amazon.co.uk`. When you authenticate with 9507 rclone it will take you to an `amazon.com` page to log in. Your 9508 `amazon.co.uk` email and password should work here just fine. 9509 9510 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/amazonclouddrive/amazonclouddrive.go then run make backenddocs --> 9511 ### Standard Options 9512 9513 Here are the standard options specific to amazon cloud drive (Amazon Drive). 9514 9515 #### --acd-client-id 9516 9517 Amazon Application Client ID. 9518 9519 - Config: client_id 9520 - Env Var: RCLONE_ACD_CLIENT_ID 9521 - Type: string 9522 - Default: "" 9523 9524 #### --acd-client-secret 9525 9526 Amazon Application Client Secret. 9527 9528 - Config: client_secret 9529 - Env Var: RCLONE_ACD_CLIENT_SECRET 9530 - Type: string 9531 - Default: "" 9532 9533 ### Advanced Options 9534 9535 Here are the advanced options specific to amazon cloud drive (Amazon Drive). 9536 9537 #### --acd-auth-url 9538 9539 Auth server URL. 9540 Leave blank to use Amazon's. 9541 9542 - Config: auth_url 9543 - Env Var: RCLONE_ACD_AUTH_URL 9544 - Type: string 9545 - Default: "" 9546 9547 #### --acd-token-url 9548 9549 Token server url. 9550 leave blank to use Amazon's. 9551 9552 - Config: token_url 9553 - Env Var: RCLONE_ACD_TOKEN_URL 9554 - Type: string 9555 - Default: "" 9556 9557 #### --acd-checkpoint 9558 9559 Checkpoint for internal polling (debug). 9560 9561 - Config: checkpoint 9562 - Env Var: RCLONE_ACD_CHECKPOINT 9563 - Type: string 9564 - Default: "" 9565 9566 #### --acd-upload-wait-per-gb 9567 9568 Additional time per GB to wait after a failed complete upload to see if it appears. 9569 9570 Sometimes Amazon Drive gives an error when a file has been fully 9571 uploaded but the file appears anyway after a little while. This 9572 happens sometimes for files over 1GB in size and nearly every time for 9573 files bigger than 10GB. This parameter controls the time rclone waits 9574 for the file to appear. 9575 9576 The default value for this parameter is 3 minutes per GB, so by 9577 default it will wait 3 minutes for every GB uploaded to see if the 9578 file appears. 9579 9580 You can disable this feature by setting it to 0. This may cause 9581 conflict errors as rclone retries the failed upload but the file will 9582 most likely appear correctly eventually. 9583 9584 These values were determined empirically by observing lots of uploads 9585 of big files for a range of file sizes. 9586 9587 Upload with the "-v" flag to see more info about what rclone is doing 9588 in this situation. 9589 9590 - Config: upload_wait_per_gb 9591 - Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB 9592 - Type: Duration 9593 - Default: 3m0s 9594 9595 #### --acd-templink-threshold 9596 9597 Files >= this size will be downloaded via their tempLink. 9598 9599 Files this size or more will be downloaded via their "tempLink". This 9600 is to work around a problem with Amazon Drive which blocks downloads 9601 of files bigger than about 10GB. The default for this is 9GB which 9602 shouldn't need to be changed. 9603 9604 To download files above this threshold, rclone requests a "tempLink" 9605 which downloads the file through a temporary URL directly from the 9606 underlying S3 storage. 9607 9608 - Config: templink_threshold 9609 - Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD 9610 - Type: SizeSuffix 9611 - Default: 9G 9612 9613 #### --acd-encoding 9614 9615 This sets the encoding for the backend. 9616 9617 See: the [encoding section in the overview](/overview/#encoding) for more info. 9618 9619 - Config: encoding 9620 - Env Var: RCLONE_ACD_ENCODING 9621 - Type: MultiEncoder 9622 - Default: Slash,InvalidUtf8,Dot 9623 9624 <!--- autogenerated options stop --> 9625 9626 ### Limitations ### 9627 9628 Note that Amazon Drive is case insensitive so you can't have a 9629 file called "Hello.doc" and one called "hello.doc". 9630 9631 Amazon Drive has rate limiting so you may notice errors in the 9632 sync (429 errors). rclone will automatically retry the sync up to 3 9633 times by default (see `--retries` flag) which should hopefully work 9634 around this problem. 9635 9636 Amazon Drive has an internal limit of file sizes that can be uploaded 9637 to the service. This limit is not officially published, but all files 9638 larger than this will fail. 9639 9640 At the time of writing (Jan 2016) is in the area of 50GB per file. 9641 This means that larger files are likely to fail. 9642 9643 Unfortunately there is no way for rclone to see that this failure is 9644 because of file size, so it will retry the operation, as any other 9645 failure. To avoid this problem, use `--max-size 50000M` option to limit 9646 the maximum size of uploaded files. Note that `--max-size` does not split 9647 files into segments, it only ignores files over this size. 9648 9649 Amazon S3 Storage Providers 9650 -------------------------------------------------------- 9651 9652 The S3 backend can be used with a number of different providers: 9653 9654 * AWS S3 9655 * Alibaba Cloud (Aliyun) Object Storage System (OSS) 9656 * Ceph 9657 * DigitalOcean Spaces 9658 * Dreamhost 9659 * IBM COS S3 9660 * Minio 9661 * Scaleway 9662 * Wasabi 9663 9664 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 9665 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 9666 9667 Once you have made a remote (see the provider specific section above) 9668 you can use it like this: 9669 9670 See all buckets 9671 9672 rclone lsd remote: 9673 9674 Make a new bucket 9675 9676 rclone mkdir remote:bucket 9677 9678 List the contents of a bucket 9679 9680 rclone ls remote:bucket 9681 9682 Sync `/home/local/directory` to the remote bucket, deleting any excess 9683 files in the bucket. 9684 9685 rclone sync /home/local/directory remote:bucket 9686 9687 ## AWS S3 {#amazon-s3} 9688 9689 Here is an example of making an s3 configuration. First run 9690 9691 rclone config 9692 9693 This will guide you through an interactive setup process. 9694 9695 ``` 9696 No remotes found - make a new one 9697 n) New remote 9698 s) Set configuration password 9699 q) Quit config 9700 n/s/q> n 9701 name> remote 9702 Type of storage to configure. 9703 Choose a number from below, or type in your own value 9704 [snip] 9705 XX / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio) 9706 \ "s3" 9707 [snip] 9708 Storage> s3 9709 Choose your S3 provider. 9710 Choose a number from below, or type in your own value 9711 1 / Amazon Web Services (AWS) S3 9712 \ "AWS" 9713 2 / Ceph Object Storage 9714 \ "Ceph" 9715 3 / Digital Ocean Spaces 9716 \ "DigitalOcean" 9717 4 / Dreamhost DreamObjects 9718 \ "Dreamhost" 9719 5 / IBM COS S3 9720 \ "IBMCOS" 9721 6 / Minio Object Storage 9722 \ "Minio" 9723 7 / Wasabi Object Storage 9724 \ "Wasabi" 9725 8 / Any other S3 compatible provider 9726 \ "Other" 9727 provider> 1 9728 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 9729 Choose a number from below, or type in your own value 9730 1 / Enter AWS credentials in the next step 9731 \ "false" 9732 2 / Get AWS credentials from the environment (env vars or IAM) 9733 \ "true" 9734 env_auth> 1 9735 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 9736 access_key_id> XXX 9737 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 9738 secret_access_key> YYY 9739 Region to connect to. 9740 Choose a number from below, or type in your own value 9741 / The default endpoint - a good choice if you are unsure. 9742 1 | US Region, Northern Virginia or Pacific Northwest. 9743 | Leave location constraint empty. 9744 \ "us-east-1" 9745 / US East (Ohio) Region 9746 2 | Needs location constraint us-east-2. 9747 \ "us-east-2" 9748 / US West (Oregon) Region 9749 3 | Needs location constraint us-west-2. 9750 \ "us-west-2" 9751 / US West (Northern California) Region 9752 4 | Needs location constraint us-west-1. 9753 \ "us-west-1" 9754 / Canada (Central) Region 9755 5 | Needs location constraint ca-central-1. 9756 \ "ca-central-1" 9757 / EU (Ireland) Region 9758 6 | Needs location constraint EU or eu-west-1. 9759 \ "eu-west-1" 9760 / EU (London) Region 9761 7 | Needs location constraint eu-west-2. 9762 \ "eu-west-2" 9763 / EU (Frankfurt) Region 9764 8 | Needs location constraint eu-central-1. 9765 \ "eu-central-1" 9766 / Asia Pacific (Singapore) Region 9767 9 | Needs location constraint ap-southeast-1. 9768 \ "ap-southeast-1" 9769 / Asia Pacific (Sydney) Region 9770 10 | Needs location constraint ap-southeast-2. 9771 \ "ap-southeast-2" 9772 / Asia Pacific (Tokyo) Region 9773 11 | Needs location constraint ap-northeast-1. 9774 \ "ap-northeast-1" 9775 / Asia Pacific (Seoul) 9776 12 | Needs location constraint ap-northeast-2. 9777 \ "ap-northeast-2" 9778 / Asia Pacific (Mumbai) 9779 13 | Needs location constraint ap-south-1. 9780 \ "ap-south-1" 9781 / Asia Patific (Hong Kong) Region 9782 14 | Needs location constraint ap-east-1. 9783 \ "ap-east-1" 9784 / South America (Sao Paulo) Region 9785 15 | Needs location constraint sa-east-1. 9786 \ "sa-east-1" 9787 region> 1 9788 Endpoint for S3 API. 9789 Leave blank if using AWS to use the default endpoint for the region. 9790 endpoint> 9791 Location constraint - must be set to match the Region. Used when creating buckets only. 9792 Choose a number from below, or type in your own value 9793 1 / Empty for US Region, Northern Virginia or Pacific Northwest. 9794 \ "" 9795 2 / US East (Ohio) Region. 9796 \ "us-east-2" 9797 3 / US West (Oregon) Region. 9798 \ "us-west-2" 9799 4 / US West (Northern California) Region. 9800 \ "us-west-1" 9801 5 / Canada (Central) Region. 9802 \ "ca-central-1" 9803 6 / EU (Ireland) Region. 9804 \ "eu-west-1" 9805 7 / EU (London) Region. 9806 \ "eu-west-2" 9807 8 / EU Region. 9808 \ "EU" 9809 9 / Asia Pacific (Singapore) Region. 9810 \ "ap-southeast-1" 9811 10 / Asia Pacific (Sydney) Region. 9812 \ "ap-southeast-2" 9813 11 / Asia Pacific (Tokyo) Region. 9814 \ "ap-northeast-1" 9815 12 / Asia Pacific (Seoul) 9816 \ "ap-northeast-2" 9817 13 / Asia Pacific (Mumbai) 9818 \ "ap-south-1" 9819 14 / Asia Pacific (Hong Kong) 9820 \ "ap-east-1" 9821 15 / South America (Sao Paulo) Region. 9822 \ "sa-east-1" 9823 location_constraint> 1 9824 Canned ACL used when creating buckets and/or storing objects in S3. 9825 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 9826 Choose a number from below, or type in your own value 9827 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 9828 \ "private" 9829 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 9830 \ "public-read" 9831 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 9832 3 | Granting this on a bucket is generally not recommended. 9833 \ "public-read-write" 9834 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 9835 \ "authenticated-read" 9836 / Object owner gets FULL_CONTROL. Bucket owner gets READ access. 9837 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 9838 \ "bucket-owner-read" 9839 / Both the object owner and the bucket owner get FULL_CONTROL over the object. 9840 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 9841 \ "bucket-owner-full-control" 9842 acl> 1 9843 The server-side encryption algorithm used when storing this object in S3. 9844 Choose a number from below, or type in your own value 9845 1 / None 9846 \ "" 9847 2 / AES256 9848 \ "AES256" 9849 server_side_encryption> 1 9850 The storage class to use when storing objects in S3. 9851 Choose a number from below, or type in your own value 9852 1 / Default 9853 \ "" 9854 2 / Standard storage class 9855 \ "STANDARD" 9856 3 / Reduced redundancy storage class 9857 \ "REDUCED_REDUNDANCY" 9858 4 / Standard Infrequent Access storage class 9859 \ "STANDARD_IA" 9860 5 / One Zone Infrequent Access storage class 9861 \ "ONEZONE_IA" 9862 6 / Glacier storage class 9863 \ "GLACIER" 9864 7 / Glacier Deep Archive storage class 9865 \ "DEEP_ARCHIVE" 9866 8 / Intelligent-Tiering storage class 9867 \ "INTELLIGENT_TIERING" 9868 storage_class> 1 9869 Remote config 9870 -------------------- 9871 [remote] 9872 type = s3 9873 provider = AWS 9874 env_auth = false 9875 access_key_id = XXX 9876 secret_access_key = YYY 9877 region = us-east-1 9878 endpoint = 9879 location_constraint = 9880 acl = private 9881 server_side_encryption = 9882 storage_class = 9883 -------------------- 9884 y) Yes this is OK 9885 e) Edit this remote 9886 d) Delete this remote 9887 y/e/d> 9888 ``` 9889 9890 ### --fast-list ### 9891 9892 This remote supports `--fast-list` which allows you to use fewer 9893 transactions in exchange for more memory. See the [rclone 9894 docs](/docs/#fast-list) for more details. 9895 9896 ### --update and --use-server-modtime ### 9897 9898 As noted below, the modified time is stored on metadata on the object. It is 9899 used by default for all operations that require checking the time a file was 9900 last updated. It allows rclone to treat the remote more like a true filesystem, 9901 but it is inefficient because it requires an extra API call to retrieve the 9902 metadata. 9903 9904 For many operations, the time the object was last uploaded to the remote is 9905 sufficient to determine if it is "dirty". By using `--update` along with 9906 `--use-server-modtime`, you can avoid the extra API call and simply upload 9907 files whose local modtime is newer than the time it was last uploaded. 9908 9909 ### Modified time ### 9910 9911 The modified time is stored as metadata on the object as 9912 `X-Amz-Meta-Mtime` as floating point since the epoch accurate to 1 ns. 9913 9914 If the modification time needs to be updated rclone will attempt to perform a server 9915 side copy to update the modification if the object can be copied in a single part. 9916 In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive 9917 storage the object will be uploaded rather than copied. 9918 9919 #### Restricted filename characters 9920 9921 S3 allows any valid UTF-8 string as a key. 9922 9923 Invalid UTF-8 bytes will be [replaced](/overview/#invalid-utf8), as 9924 they can't be used in XML. 9925 9926 The following characters are replaced since these are problematic when 9927 dealing with the REST API: 9928 9929 | Character | Value | Replacement | 9930 | --------- |:-----:|:-----------:| 9931 | NUL | 0x00 | ␀ | 9932 | / | 0x2F | / | 9933 9934 The encoding will also encode these file names as they don't seem to 9935 work with the SDK properly: 9936 9937 | File name | Replacement | 9938 | --------- |:-----------:| 9939 | . | . | 9940 | .. | .. | 9941 9942 ### Multipart uploads ### 9943 9944 rclone supports multipart uploads with S3 which means that it can 9945 upload files bigger than 5GB. 9946 9947 Note that files uploaded *both* with multipart upload *and* through 9948 crypt remotes do not have MD5 sums. 9949 9950 rclone switches from single part uploads to multipart uploads at the 9951 point specified by `--s3-upload-cutoff`. This can be a maximum of 5GB 9952 and a minimum of 0 (ie always upload multipart files). 9953 9954 The chunk sizes used in the multipart upload are specified by 9955 `--s3-chunk-size` and the number of chunks uploaded concurrently is 9956 specified by `--s3-upload-concurrency`. 9957 9958 Multipart uploads will use `--transfers` * `--s3-upload-concurrency` * 9959 `--s3-chunk-size` extra memory. Single part uploads to not use extra 9960 memory. 9961 9962 Single part transfers can be faster than multipart transfers or slower 9963 depending on your latency from S3 - the more latency, the more likely 9964 single part transfers will be faster. 9965 9966 Increasing `--s3-upload-concurrency` will increase throughput (8 would 9967 be a sensible value) and increasing `--s3-chunk-size` also increases 9968 throughput (16M would be sensible). Increasing either of these will 9969 use more memory. The default values are high enough to gain most of 9970 the possible performance without using too much memory. 9971 9972 9973 ### Buckets and Regions ### 9974 9975 With Amazon S3 you can list buckets (`rclone lsd`) using any region, 9976 but you can only access the content of a bucket from the region it was 9977 created in. If you attempt to access a bucket from the wrong region, 9978 you will get an error, `incorrect region, the bucket is not in 'XXX' 9979 region`. 9980 9981 ### Authentication ### 9982 9983 There are a number of ways to supply `rclone` with a set of AWS 9984 credentials, with and without using the environment. 9985 9986 The different authentication methods are tried in this order: 9987 9988 - Directly in the rclone configuration file (`env_auth = false` in the config file): 9989 - `access_key_id` and `secret_access_key` are required. 9990 - `session_token` can be optionally set when using AWS STS. 9991 - Runtime configuration (`env_auth = true` in the config file): 9992 - Export the following environment variables before running `rclone`: 9993 - Access Key ID: `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY` 9994 - Secret Access Key: `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY` 9995 - Session Token: `AWS_SESSION_TOKEN` (optional) 9996 - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html): 9997 - Profile files are standard files used by AWS CLI tools 9998 - By default it will use the profile in your home directory (eg `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables: 9999 - `AWS_SHARED_CREDENTIALS_FILE` to control which file. 10000 - `AWS_PROFILE` to control which profile to use. 10001 - Or, run `rclone` in an ECS task with an IAM role (AWS only). 10002 - Or, run `rclone` on an EC2 instance with an IAM role (AWS only). 10003 - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only). 10004 10005 If none of these option actually end up providing `rclone` with AWS 10006 credentials then S3 interaction will be non-authenticated (see below). 10007 10008 ### S3 Permissions ### 10009 10010 When using the `sync` subcommand of `rclone` the following minimum 10011 permissions are required to be available on the bucket being written to: 10012 10013 * `ListBucket` 10014 * `DeleteObject` 10015 * `GetObject` 10016 * `PutObject` 10017 * `PutObjectACL` 10018 10019 When using the `lsd` subcommand, the `ListAllMyBuckets` permission is required. 10020 10021 Example policy: 10022 10023 ``` 10024 { 10025 "Version": "2012-10-17", 10026 "Statement": [ 10027 { 10028 "Effect": "Allow", 10029 "Principal": { 10030 "AWS": "arn:aws:iam::USER_SID:user/USER_NAME" 10031 }, 10032 "Action": [ 10033 "s3:ListBucket", 10034 "s3:DeleteObject", 10035 "s3:GetObject", 10036 "s3:PutObject", 10037 "s3:PutObjectAcl" 10038 ], 10039 "Resource": [ 10040 "arn:aws:s3:::BUCKET_NAME/*", 10041 "arn:aws:s3:::BUCKET_NAME" 10042 ] 10043 }, 10044 { 10045 "Effect": "Allow", 10046 "Action": "s3:ListAllMyBuckets", 10047 "Resource": "arn:aws:s3:::*" 10048 } 10049 ] 10050 } 10051 ``` 10052 10053 Notes on above: 10054 10055 1. This is a policy that can be used when creating bucket. It assumes 10056 that `USER_NAME` has been created. 10057 2. The Resource entry must include both resource ARNs, as one implies 10058 the bucket and the other implies the bucket's objects. 10059 10060 For reference, [here's an Ansible script](https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b) 10061 that will generate one or more buckets that will work with `rclone sync`. 10062 10063 ### Key Management System (KMS) ### 10064 10065 If you are using server side encryption with KMS then you will find 10066 you can't transfer small objects. As a work-around you can use the 10067 `--ignore-checksum` flag. 10068 10069 A proper fix is being worked on in [issue #1824](https://github.com/rclone/rclone/issues/1824). 10070 10071 ### Glacier and Glacier Deep Archive ### 10072 10073 You can upload objects using the glacier storage class or transition them to glacier using a [lifecycle policy](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html). 10074 The bucket can still be synced or copied into normally, but if rclone 10075 tries to access data from the glacier storage class you will see an error like below. 10076 10077 2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file 10078 10079 In this case you need to [restore](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html) 10080 the object(s) in question before using rclone. 10081 10082 Note that rclone only speaks the S3 API it does not speak the Glacier 10083 Vault API, so rclone cannot directly access Glacier Vaults. 10084 10085 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/s3/s3.go then run make backenddocs --> 10086 ### Standard Options 10087 10088 Here are the standard options specific to s3 (Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)). 10089 10090 #### --s3-provider 10091 10092 Choose your S3 provider. 10093 10094 - Config: provider 10095 - Env Var: RCLONE_S3_PROVIDER 10096 - Type: string 10097 - Default: "" 10098 - Examples: 10099 - "AWS" 10100 - Amazon Web Services (AWS) S3 10101 - "Alibaba" 10102 - Alibaba Cloud Object Storage System (OSS) formerly Aliyun 10103 - "Ceph" 10104 - Ceph Object Storage 10105 - "DigitalOcean" 10106 - Digital Ocean Spaces 10107 - "Dreamhost" 10108 - Dreamhost DreamObjects 10109 - "IBMCOS" 10110 - IBM COS S3 10111 - "Minio" 10112 - Minio Object Storage 10113 - "Netease" 10114 - Netease Object Storage (NOS) 10115 - "Wasabi" 10116 - Wasabi Object Storage 10117 - "Other" 10118 - Any other S3 compatible provider 10119 10120 #### --s3-env-auth 10121 10122 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 10123 Only applies if access_key_id and secret_access_key is blank. 10124 10125 - Config: env_auth 10126 - Env Var: RCLONE_S3_ENV_AUTH 10127 - Type: bool 10128 - Default: false 10129 - Examples: 10130 - "false" 10131 - Enter AWS credentials in the next step 10132 - "true" 10133 - Get AWS credentials from the environment (env vars or IAM) 10134 10135 #### --s3-access-key-id 10136 10137 AWS Access Key ID. 10138 Leave blank for anonymous access or runtime credentials. 10139 10140 - Config: access_key_id 10141 - Env Var: RCLONE_S3_ACCESS_KEY_ID 10142 - Type: string 10143 - Default: "" 10144 10145 #### --s3-secret-access-key 10146 10147 AWS Secret Access Key (password) 10148 Leave blank for anonymous access or runtime credentials. 10149 10150 - Config: secret_access_key 10151 - Env Var: RCLONE_S3_SECRET_ACCESS_KEY 10152 - Type: string 10153 - Default: "" 10154 10155 #### --s3-region 10156 10157 Region to connect to. 10158 10159 - Config: region 10160 - Env Var: RCLONE_S3_REGION 10161 - Type: string 10162 - Default: "" 10163 - Examples: 10164 - "us-east-1" 10165 - The default endpoint - a good choice if you are unsure. 10166 - US Region, Northern Virginia or Pacific Northwest. 10167 - Leave location constraint empty. 10168 - "us-east-2" 10169 - US East (Ohio) Region 10170 - Needs location constraint us-east-2. 10171 - "us-west-2" 10172 - US West (Oregon) Region 10173 - Needs location constraint us-west-2. 10174 - "us-west-1" 10175 - US West (Northern California) Region 10176 - Needs location constraint us-west-1. 10177 - "ca-central-1" 10178 - Canada (Central) Region 10179 - Needs location constraint ca-central-1. 10180 - "eu-west-1" 10181 - EU (Ireland) Region 10182 - Needs location constraint EU or eu-west-1. 10183 - "eu-west-2" 10184 - EU (London) Region 10185 - Needs location constraint eu-west-2. 10186 - "eu-north-1" 10187 - EU (Stockholm) Region 10188 - Needs location constraint eu-north-1. 10189 - "eu-central-1" 10190 - EU (Frankfurt) Region 10191 - Needs location constraint eu-central-1. 10192 - "ap-southeast-1" 10193 - Asia Pacific (Singapore) Region 10194 - Needs location constraint ap-southeast-1. 10195 - "ap-southeast-2" 10196 - Asia Pacific (Sydney) Region 10197 - Needs location constraint ap-southeast-2. 10198 - "ap-northeast-1" 10199 - Asia Pacific (Tokyo) Region 10200 - Needs location constraint ap-northeast-1. 10201 - "ap-northeast-2" 10202 - Asia Pacific (Seoul) 10203 - Needs location constraint ap-northeast-2. 10204 - "ap-south-1" 10205 - Asia Pacific (Mumbai) 10206 - Needs location constraint ap-south-1. 10207 - "ap-east-1" 10208 - Asia Patific (Hong Kong) Region 10209 - Needs location constraint ap-east-1. 10210 - "sa-east-1" 10211 - South America (Sao Paulo) Region 10212 - Needs location constraint sa-east-1. 10213 10214 #### --s3-region 10215 10216 Region to connect to. 10217 Leave blank if you are using an S3 clone and you don't have a region. 10218 10219 - Config: region 10220 - Env Var: RCLONE_S3_REGION 10221 - Type: string 10222 - Default: "" 10223 - Examples: 10224 - "" 10225 - Use this if unsure. Will use v4 signatures and an empty region. 10226 - "other-v2-signature" 10227 - Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH. 10228 10229 #### --s3-endpoint 10230 10231 Endpoint for S3 API. 10232 Leave blank if using AWS to use the default endpoint for the region. 10233 10234 - Config: endpoint 10235 - Env Var: RCLONE_S3_ENDPOINT 10236 - Type: string 10237 - Default: "" 10238 10239 #### --s3-endpoint 10240 10241 Endpoint for IBM COS S3 API. 10242 Specify if using an IBM COS On Premise. 10243 10244 - Config: endpoint 10245 - Env Var: RCLONE_S3_ENDPOINT 10246 - Type: string 10247 - Default: "" 10248 - Examples: 10249 - "s3-api.us-geo.objectstorage.softlayer.net" 10250 - US Cross Region Endpoint 10251 - "s3-api.dal.us-geo.objectstorage.softlayer.net" 10252 - US Cross Region Dallas Endpoint 10253 - "s3-api.wdc-us-geo.objectstorage.softlayer.net" 10254 - US Cross Region Washington DC Endpoint 10255 - "s3-api.sjc-us-geo.objectstorage.softlayer.net" 10256 - US Cross Region San Jose Endpoint 10257 - "s3-api.us-geo.objectstorage.service.networklayer.com" 10258 - US Cross Region Private Endpoint 10259 - "s3-api.dal-us-geo.objectstorage.service.networklayer.com" 10260 - US Cross Region Dallas Private Endpoint 10261 - "s3-api.wdc-us-geo.objectstorage.service.networklayer.com" 10262 - US Cross Region Washington DC Private Endpoint 10263 - "s3-api.sjc-us-geo.objectstorage.service.networklayer.com" 10264 - US Cross Region San Jose Private Endpoint 10265 - "s3.us-east.objectstorage.softlayer.net" 10266 - US Region East Endpoint 10267 - "s3.us-east.objectstorage.service.networklayer.com" 10268 - US Region East Private Endpoint 10269 - "s3.us-south.objectstorage.softlayer.net" 10270 - US Region South Endpoint 10271 - "s3.us-south.objectstorage.service.networklayer.com" 10272 - US Region South Private Endpoint 10273 - "s3.eu-geo.objectstorage.softlayer.net" 10274 - EU Cross Region Endpoint 10275 - "s3.fra-eu-geo.objectstorage.softlayer.net" 10276 - EU Cross Region Frankfurt Endpoint 10277 - "s3.mil-eu-geo.objectstorage.softlayer.net" 10278 - EU Cross Region Milan Endpoint 10279 - "s3.ams-eu-geo.objectstorage.softlayer.net" 10280 - EU Cross Region Amsterdam Endpoint 10281 - "s3.eu-geo.objectstorage.service.networklayer.com" 10282 - EU Cross Region Private Endpoint 10283 - "s3.fra-eu-geo.objectstorage.service.networklayer.com" 10284 - EU Cross Region Frankfurt Private Endpoint 10285 - "s3.mil-eu-geo.objectstorage.service.networklayer.com" 10286 - EU Cross Region Milan Private Endpoint 10287 - "s3.ams-eu-geo.objectstorage.service.networklayer.com" 10288 - EU Cross Region Amsterdam Private Endpoint 10289 - "s3.eu-gb.objectstorage.softlayer.net" 10290 - Great Britain Endpoint 10291 - "s3.eu-gb.objectstorage.service.networklayer.com" 10292 - Great Britain Private Endpoint 10293 - "s3.ap-geo.objectstorage.softlayer.net" 10294 - APAC Cross Regional Endpoint 10295 - "s3.tok-ap-geo.objectstorage.softlayer.net" 10296 - APAC Cross Regional Tokyo Endpoint 10297 - "s3.hkg-ap-geo.objectstorage.softlayer.net" 10298 - APAC Cross Regional HongKong Endpoint 10299 - "s3.seo-ap-geo.objectstorage.softlayer.net" 10300 - APAC Cross Regional Seoul Endpoint 10301 - "s3.ap-geo.objectstorage.service.networklayer.com" 10302 - APAC Cross Regional Private Endpoint 10303 - "s3.tok-ap-geo.objectstorage.service.networklayer.com" 10304 - APAC Cross Regional Tokyo Private Endpoint 10305 - "s3.hkg-ap-geo.objectstorage.service.networklayer.com" 10306 - APAC Cross Regional HongKong Private Endpoint 10307 - "s3.seo-ap-geo.objectstorage.service.networklayer.com" 10308 - APAC Cross Regional Seoul Private Endpoint 10309 - "s3.mel01.objectstorage.softlayer.net" 10310 - Melbourne Single Site Endpoint 10311 - "s3.mel01.objectstorage.service.networklayer.com" 10312 - Melbourne Single Site Private Endpoint 10313 - "s3.tor01.objectstorage.softlayer.net" 10314 - Toronto Single Site Endpoint 10315 - "s3.tor01.objectstorage.service.networklayer.com" 10316 - Toronto Single Site Private Endpoint 10317 10318 #### --s3-endpoint 10319 10320 Endpoint for OSS API. 10321 10322 - Config: endpoint 10323 - Env Var: RCLONE_S3_ENDPOINT 10324 - Type: string 10325 - Default: "" 10326 - Examples: 10327 - "oss-cn-hangzhou.aliyuncs.com" 10328 - East China 1 (Hangzhou) 10329 - "oss-cn-shanghai.aliyuncs.com" 10330 - East China 2 (Shanghai) 10331 - "oss-cn-qingdao.aliyuncs.com" 10332 - North China 1 (Qingdao) 10333 - "oss-cn-beijing.aliyuncs.com" 10334 - North China 2 (Beijing) 10335 - "oss-cn-zhangjiakou.aliyuncs.com" 10336 - North China 3 (Zhangjiakou) 10337 - "oss-cn-huhehaote.aliyuncs.com" 10338 - North China 5 (Huhehaote) 10339 - "oss-cn-shenzhen.aliyuncs.com" 10340 - South China 1 (Shenzhen) 10341 - "oss-cn-hongkong.aliyuncs.com" 10342 - Hong Kong (Hong Kong) 10343 - "oss-us-west-1.aliyuncs.com" 10344 - US West 1 (Silicon Valley) 10345 - "oss-us-east-1.aliyuncs.com" 10346 - US East 1 (Virginia) 10347 - "oss-ap-southeast-1.aliyuncs.com" 10348 - Southeast Asia Southeast 1 (Singapore) 10349 - "oss-ap-southeast-2.aliyuncs.com" 10350 - Asia Pacific Southeast 2 (Sydney) 10351 - "oss-ap-southeast-3.aliyuncs.com" 10352 - Southeast Asia Southeast 3 (Kuala Lumpur) 10353 - "oss-ap-southeast-5.aliyuncs.com" 10354 - Asia Pacific Southeast 5 (Jakarta) 10355 - "oss-ap-northeast-1.aliyuncs.com" 10356 - Asia Pacific Northeast 1 (Japan) 10357 - "oss-ap-south-1.aliyuncs.com" 10358 - Asia Pacific South 1 (Mumbai) 10359 - "oss-eu-central-1.aliyuncs.com" 10360 - Central Europe 1 (Frankfurt) 10361 - "oss-eu-west-1.aliyuncs.com" 10362 - West Europe (London) 10363 - "oss-me-east-1.aliyuncs.com" 10364 - Middle East 1 (Dubai) 10365 10366 #### --s3-endpoint 10367 10368 Endpoint for S3 API. 10369 Required when using an S3 clone. 10370 10371 - Config: endpoint 10372 - Env Var: RCLONE_S3_ENDPOINT 10373 - Type: string 10374 - Default: "" 10375 - Examples: 10376 - "objects-us-east-1.dream.io" 10377 - Dream Objects endpoint 10378 - "nyc3.digitaloceanspaces.com" 10379 - Digital Ocean Spaces New York 3 10380 - "ams3.digitaloceanspaces.com" 10381 - Digital Ocean Spaces Amsterdam 3 10382 - "sgp1.digitaloceanspaces.com" 10383 - Digital Ocean Spaces Singapore 1 10384 - "s3.wasabisys.com" 10385 - Wasabi US East endpoint 10386 - "s3.us-west-1.wasabisys.com" 10387 - Wasabi US West endpoint 10388 - "s3.eu-central-1.wasabisys.com" 10389 - Wasabi EU Central endpoint 10390 10391 #### --s3-location-constraint 10392 10393 Location constraint - must be set to match the Region. 10394 Used when creating buckets only. 10395 10396 - Config: location_constraint 10397 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 10398 - Type: string 10399 - Default: "" 10400 - Examples: 10401 - "" 10402 - Empty for US Region, Northern Virginia or Pacific Northwest. 10403 - "us-east-2" 10404 - US East (Ohio) Region. 10405 - "us-west-2" 10406 - US West (Oregon) Region. 10407 - "us-west-1" 10408 - US West (Northern California) Region. 10409 - "ca-central-1" 10410 - Canada (Central) Region. 10411 - "eu-west-1" 10412 - EU (Ireland) Region. 10413 - "eu-west-2" 10414 - EU (London) Region. 10415 - "eu-north-1" 10416 - EU (Stockholm) Region. 10417 - "EU" 10418 - EU Region. 10419 - "ap-southeast-1" 10420 - Asia Pacific (Singapore) Region. 10421 - "ap-southeast-2" 10422 - Asia Pacific (Sydney) Region. 10423 - "ap-northeast-1" 10424 - Asia Pacific (Tokyo) Region. 10425 - "ap-northeast-2" 10426 - Asia Pacific (Seoul) 10427 - "ap-south-1" 10428 - Asia Pacific (Mumbai) 10429 - "ap-east-1" 10430 - Asia Pacific (Hong Kong) 10431 - "sa-east-1" 10432 - South America (Sao Paulo) Region. 10433 10434 #### --s3-location-constraint 10435 10436 Location constraint - must match endpoint when using IBM Cloud Public. 10437 For on-prem COS, do not make a selection from this list, hit enter 10438 10439 - Config: location_constraint 10440 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 10441 - Type: string 10442 - Default: "" 10443 - Examples: 10444 - "us-standard" 10445 - US Cross Region Standard 10446 - "us-vault" 10447 - US Cross Region Vault 10448 - "us-cold" 10449 - US Cross Region Cold 10450 - "us-flex" 10451 - US Cross Region Flex 10452 - "us-east-standard" 10453 - US East Region Standard 10454 - "us-east-vault" 10455 - US East Region Vault 10456 - "us-east-cold" 10457 - US East Region Cold 10458 - "us-east-flex" 10459 - US East Region Flex 10460 - "us-south-standard" 10461 - US South Region Standard 10462 - "us-south-vault" 10463 - US South Region Vault 10464 - "us-south-cold" 10465 - US South Region Cold 10466 - "us-south-flex" 10467 - US South Region Flex 10468 - "eu-standard" 10469 - EU Cross Region Standard 10470 - "eu-vault" 10471 - EU Cross Region Vault 10472 - "eu-cold" 10473 - EU Cross Region Cold 10474 - "eu-flex" 10475 - EU Cross Region Flex 10476 - "eu-gb-standard" 10477 - Great Britain Standard 10478 - "eu-gb-vault" 10479 - Great Britain Vault 10480 - "eu-gb-cold" 10481 - Great Britain Cold 10482 - "eu-gb-flex" 10483 - Great Britain Flex 10484 - "ap-standard" 10485 - APAC Standard 10486 - "ap-vault" 10487 - APAC Vault 10488 - "ap-cold" 10489 - APAC Cold 10490 - "ap-flex" 10491 - APAC Flex 10492 - "mel01-standard" 10493 - Melbourne Standard 10494 - "mel01-vault" 10495 - Melbourne Vault 10496 - "mel01-cold" 10497 - Melbourne Cold 10498 - "mel01-flex" 10499 - Melbourne Flex 10500 - "tor01-standard" 10501 - Toronto Standard 10502 - "tor01-vault" 10503 - Toronto Vault 10504 - "tor01-cold" 10505 - Toronto Cold 10506 - "tor01-flex" 10507 - Toronto Flex 10508 10509 #### --s3-location-constraint 10510 10511 Location constraint - must be set to match the Region. 10512 Leave blank if not sure. Used when creating buckets only. 10513 10514 - Config: location_constraint 10515 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 10516 - Type: string 10517 - Default: "" 10518 10519 #### --s3-acl 10520 10521 Canned ACL used when creating buckets and storing or copying objects. 10522 10523 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 10524 10525 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 10526 10527 Note that this ACL is applied when server side copying objects as S3 10528 doesn't copy the ACL from the source but rather writes a fresh one. 10529 10530 - Config: acl 10531 - Env Var: RCLONE_S3_ACL 10532 - Type: string 10533 - Default: "" 10534 - Examples: 10535 - "private" 10536 - Owner gets FULL_CONTROL. No one else has access rights (default). 10537 - "public-read" 10538 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. 10539 - "public-read-write" 10540 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 10541 - Granting this on a bucket is generally not recommended. 10542 - "authenticated-read" 10543 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 10544 - "bucket-owner-read" 10545 - Object owner gets FULL_CONTROL. Bucket owner gets READ access. 10546 - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 10547 - "bucket-owner-full-control" 10548 - Both the object owner and the bucket owner get FULL_CONTROL over the object. 10549 - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 10550 - "private" 10551 - Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS 10552 - "public-read" 10553 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS 10554 - "public-read-write" 10555 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS 10556 - "authenticated-read" 10557 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS 10558 10559 #### --s3-server-side-encryption 10560 10561 The server-side encryption algorithm used when storing this object in S3. 10562 10563 - Config: server_side_encryption 10564 - Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION 10565 - Type: string 10566 - Default: "" 10567 - Examples: 10568 - "" 10569 - None 10570 - "AES256" 10571 - AES256 10572 - "aws:kms" 10573 - aws:kms 10574 10575 #### --s3-sse-kms-key-id 10576 10577 If using KMS ID you must provide the ARN of Key. 10578 10579 - Config: sse_kms_key_id 10580 - Env Var: RCLONE_S3_SSE_KMS_KEY_ID 10581 - Type: string 10582 - Default: "" 10583 - Examples: 10584 - "" 10585 - None 10586 - "arn:aws:kms:us-east-1:*" 10587 - arn:aws:kms:* 10588 10589 #### --s3-storage-class 10590 10591 The storage class to use when storing new objects in S3. 10592 10593 - Config: storage_class 10594 - Env Var: RCLONE_S3_STORAGE_CLASS 10595 - Type: string 10596 - Default: "" 10597 - Examples: 10598 - "" 10599 - Default 10600 - "STANDARD" 10601 - Standard storage class 10602 - "REDUCED_REDUNDANCY" 10603 - Reduced redundancy storage class 10604 - "STANDARD_IA" 10605 - Standard Infrequent Access storage class 10606 - "ONEZONE_IA" 10607 - One Zone Infrequent Access storage class 10608 - "GLACIER" 10609 - Glacier storage class 10610 - "DEEP_ARCHIVE" 10611 - Glacier Deep Archive storage class 10612 - "INTELLIGENT_TIERING" 10613 - Intelligent-Tiering storage class 10614 10615 #### --s3-storage-class 10616 10617 The storage class to use when storing new objects in OSS. 10618 10619 - Config: storage_class 10620 - Env Var: RCLONE_S3_STORAGE_CLASS 10621 - Type: string 10622 - Default: "" 10623 - Examples: 10624 - "" 10625 - Default 10626 - "STANDARD" 10627 - Standard storage class 10628 - "GLACIER" 10629 - Archive storage mode. 10630 - "STANDARD_IA" 10631 - Infrequent access storage mode. 10632 10633 ### Advanced Options 10634 10635 Here are the advanced options specific to s3 (Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)). 10636 10637 #### --s3-bucket-acl 10638 10639 Canned ACL used when creating buckets. 10640 10641 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 10642 10643 Note that this ACL is applied when only when creating buckets. If it 10644 isn't set then "acl" is used instead. 10645 10646 - Config: bucket_acl 10647 - Env Var: RCLONE_S3_BUCKET_ACL 10648 - Type: string 10649 - Default: "" 10650 - Examples: 10651 - "private" 10652 - Owner gets FULL_CONTROL. No one else has access rights (default). 10653 - "public-read" 10654 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. 10655 - "public-read-write" 10656 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 10657 - Granting this on a bucket is generally not recommended. 10658 - "authenticated-read" 10659 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 10660 10661 #### --s3-upload-cutoff 10662 10663 Cutoff for switching to chunked upload 10664 10665 Any files larger than this will be uploaded in chunks of chunk_size. 10666 The minimum is 0 and the maximum is 5GB. 10667 10668 - Config: upload_cutoff 10669 - Env Var: RCLONE_S3_UPLOAD_CUTOFF 10670 - Type: SizeSuffix 10671 - Default: 200M 10672 10673 #### --s3-chunk-size 10674 10675 Chunk size to use for uploading. 10676 10677 When uploading files larger than upload_cutoff or files with unknown 10678 size (eg from "rclone rcat" or uploaded with "rclone mount" or google 10679 photos or google docs) they will be uploaded as multipart uploads 10680 using this chunk size. 10681 10682 Note that "--s3-upload-concurrency" chunks of this size are buffered 10683 in memory per transfer. 10684 10685 If you are transferring large files over high speed links and you have 10686 enough memory, then increasing this will speed up the transfers. 10687 10688 Rclone will automatically increase the chunk size when uploading a 10689 large file of known size to stay below the 10,000 chunks limit. 10690 10691 Files of unknown size are uploaded with the configured 10692 chunk_size. Since the default chunk size is 5MB and there can be at 10693 most 10,000 chunks, this means that by default the maximum size of 10694 file you can stream upload is 48GB. If you wish to stream upload 10695 larger files then you will need to increase chunk_size. 10696 10697 - Config: chunk_size 10698 - Env Var: RCLONE_S3_CHUNK_SIZE 10699 - Type: SizeSuffix 10700 - Default: 5M 10701 10702 #### --s3-copy-cutoff 10703 10704 Cutoff for switching to multipart copy 10705 10706 Any files larger than this that need to be server side copied will be 10707 copied in chunks of this size. 10708 10709 The minimum is 0 and the maximum is 5GB. 10710 10711 - Config: copy_cutoff 10712 - Env Var: RCLONE_S3_COPY_CUTOFF 10713 - Type: SizeSuffix 10714 - Default: 5G 10715 10716 #### --s3-disable-checksum 10717 10718 Don't store MD5 checksum with object metadata 10719 10720 - Config: disable_checksum 10721 - Env Var: RCLONE_S3_DISABLE_CHECKSUM 10722 - Type: bool 10723 - Default: false 10724 10725 #### --s3-session-token 10726 10727 An AWS session token 10728 10729 - Config: session_token 10730 - Env Var: RCLONE_S3_SESSION_TOKEN 10731 - Type: string 10732 - Default: "" 10733 10734 #### --s3-upload-concurrency 10735 10736 Concurrency for multipart uploads. 10737 10738 This is the number of chunks of the same file that are uploaded 10739 concurrently. 10740 10741 If you are uploading small numbers of large file over high speed link 10742 and these uploads do not fully utilize your bandwidth, then increasing 10743 this may help to speed up the transfers. 10744 10745 - Config: upload_concurrency 10746 - Env Var: RCLONE_S3_UPLOAD_CONCURRENCY 10747 - Type: int 10748 - Default: 4 10749 10750 #### --s3-force-path-style 10751 10752 If true use path style access if false use virtual hosted style. 10753 10754 If this is true (the default) then rclone will use path style access, 10755 if false then rclone will use virtual path style. See [the AWS S3 10756 docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro) 10757 for more info. 10758 10759 Some providers (eg AWS, Aliyun OSS or Netease COS) require this set to 10760 false - rclone will do this automatically based on the provider 10761 setting. 10762 10763 - Config: force_path_style 10764 - Env Var: RCLONE_S3_FORCE_PATH_STYLE 10765 - Type: bool 10766 - Default: true 10767 10768 #### --s3-v2-auth 10769 10770 If true use v2 authentication. 10771 10772 If this is false (the default) then rclone will use v4 authentication. 10773 If it is set then rclone will use v2 authentication. 10774 10775 Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH. 10776 10777 - Config: v2_auth 10778 - Env Var: RCLONE_S3_V2_AUTH 10779 - Type: bool 10780 - Default: false 10781 10782 #### --s3-use-accelerate-endpoint 10783 10784 If true use the AWS S3 accelerated endpoint. 10785 10786 See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html) 10787 10788 - Config: use_accelerate_endpoint 10789 - Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT 10790 - Type: bool 10791 - Default: false 10792 10793 #### --s3-leave-parts-on-error 10794 10795 If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. 10796 10797 It should be set to true for resuming uploads across different sessions. 10798 10799 WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up. 10800 10801 10802 - Config: leave_parts_on_error 10803 - Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR 10804 - Type: bool 10805 - Default: false 10806 10807 #### --s3-list-chunk 10808 10809 Size of listing chunk (response list for each ListObject S3 request). 10810 10811 This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification. 10812 Most services truncate the response list to 1000 objects even if requested more than that. 10813 In AWS S3 this is a global maximum and cannot be changed, see [AWS S3](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html). 10814 In Ceph, this can be increased with the "rgw list buckets max chunk" option. 10815 10816 10817 - Config: list_chunk 10818 - Env Var: RCLONE_S3_LIST_CHUNK 10819 - Type: int 10820 - Default: 1000 10821 10822 #### --s3-encoding 10823 10824 This sets the encoding for the backend. 10825 10826 See: the [encoding section in the overview](/overview/#encoding) for more info. 10827 10828 - Config: encoding 10829 - Env Var: RCLONE_S3_ENCODING 10830 - Type: MultiEncoder 10831 - Default: Slash,InvalidUtf8,Dot 10832 10833 <!--- autogenerated options stop --> 10834 10835 ### Anonymous access to public buckets ### 10836 10837 If you want to use rclone to access a public bucket, configure with a 10838 blank `access_key_id` and `secret_access_key`. Your config should end 10839 up looking like this: 10840 10841 ``` 10842 [anons3] 10843 type = s3 10844 provider = AWS 10845 env_auth = false 10846 access_key_id = 10847 secret_access_key = 10848 region = us-east-1 10849 endpoint = 10850 location_constraint = 10851 acl = private 10852 server_side_encryption = 10853 storage_class = 10854 ``` 10855 10856 Then use it as normal with the name of the public bucket, eg 10857 10858 rclone lsd anons3:1000genomes 10859 10860 You will be able to list and copy data but not upload it. 10861 10862 ### Ceph ### 10863 10864 [Ceph](https://ceph.com/) is an open source unified, distributed 10865 storage system designed for excellent performance, reliability and 10866 scalability. It has an S3 compatible object storage interface. 10867 10868 To use rclone with Ceph, configure as above but leave the region blank 10869 and set the endpoint. You should end up with something like this in 10870 your config: 10871 10872 10873 ``` 10874 [ceph] 10875 type = s3 10876 provider = Ceph 10877 env_auth = false 10878 access_key_id = XXX 10879 secret_access_key = YYY 10880 region = 10881 endpoint = https://ceph.endpoint.example.com 10882 location_constraint = 10883 acl = 10884 server_side_encryption = 10885 storage_class = 10886 ``` 10887 10888 If you are using an older version of CEPH, eg 10.2.x Jewel, then you 10889 may need to supply the parameter `--s3-upload-cutoff 0` or put this in 10890 the config file as `upload_cutoff 0` to work around a bug which causes 10891 uploading of small files to fail. 10892 10893 Note also that Ceph sometimes puts `/` in the passwords it gives 10894 users. If you read the secret access key using the command line tools 10895 you will get a JSON blob with the `/` escaped as `\/`. Make sure you 10896 only write `/` in the secret access key. 10897 10898 Eg the dump from Ceph looks something like this (irrelevant keys 10899 removed). 10900 10901 ``` 10902 { 10903 "user_id": "xxx", 10904 "display_name": "xxxx", 10905 "keys": [ 10906 { 10907 "user": "xxx", 10908 "access_key": "xxxxxx", 10909 "secret_key": "xxxxxx\/xxxx" 10910 } 10911 ], 10912 } 10913 ``` 10914 10915 Because this is a json dump, it is encoding the `/` as `\/`, so if you 10916 use the secret key as `xxxxxx/xxxx` it will work fine. 10917 10918 ### Dreamhost ### 10919 10920 Dreamhost [DreamObjects](https://www.dreamhost.com/cloud/storage/) is 10921 an object storage system based on CEPH. 10922 10923 To use rclone with Dreamhost, configure as above but leave the region blank 10924 and set the endpoint. You should end up with something like this in 10925 your config: 10926 10927 ``` 10928 [dreamobjects] 10929 type = s3 10930 provider = DreamHost 10931 env_auth = false 10932 access_key_id = your_access_key 10933 secret_access_key = your_secret_key 10934 region = 10935 endpoint = objects-us-west-1.dream.io 10936 location_constraint = 10937 acl = private 10938 server_side_encryption = 10939 storage_class = 10940 ``` 10941 10942 ### DigitalOcean Spaces ### 10943 10944 [Spaces](https://www.digitalocean.com/products/object-storage/) is an [S3-interoperable](https://developers.digitalocean.com/documentation/spaces/) object storage service from cloud provider DigitalOcean. 10945 10946 To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "[Applications & API](https://cloud.digitalocean.com/settings/api/tokens)" page of the DigitalOcean control panel. They will be needed when promted by `rclone config` for your `access_key_id` and `secret_access_key`. 10947 10948 When prompted for a `region` or `location_constraint`, press enter to use the default value. The region must be included in the `endpoint` setting (e.g. `nyc3.digitaloceanspaces.com`). The default values can be used for other settings. 10949 10950 Going through the whole process of creating a new remote by running `rclone config`, each prompt should be answered as shown below: 10951 10952 ``` 10953 Storage> s3 10954 env_auth> 1 10955 access_key_id> YOUR_ACCESS_KEY 10956 secret_access_key> YOUR_SECRET_KEY 10957 region> 10958 endpoint> nyc3.digitaloceanspaces.com 10959 location_constraint> 10960 acl> 10961 storage_class> 10962 ``` 10963 10964 The resulting configuration file should look like: 10965 10966 ``` 10967 [spaces] 10968 type = s3 10969 provider = DigitalOcean 10970 env_auth = false 10971 access_key_id = YOUR_ACCESS_KEY 10972 secret_access_key = YOUR_SECRET_KEY 10973 region = 10974 endpoint = nyc3.digitaloceanspaces.com 10975 location_constraint = 10976 acl = 10977 server_side_encryption = 10978 storage_class = 10979 ``` 10980 10981 Once configured, you can create a new Space and begin copying files. For example: 10982 10983 ``` 10984 rclone mkdir spaces:my-new-space 10985 rclone copy /path/to/files spaces:my-new-space 10986 ``` 10987 10988 ### IBM COS (S3) ### 10989 10990 Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage) 10991 10992 To configure access to IBM COS S3, follow the steps below: 10993 10994 1. Run rclone config and select n for a new remote. 10995 ``` 10996 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults 10997 No remotes found - make a new one 10998 n) New remote 10999 s) Set configuration password 11000 q) Quit config 11001 n/s/q> n 11002 ``` 11003 11004 2. Enter the name for the configuration 11005 ``` 11006 name> <YOUR NAME> 11007 ``` 11008 11009 3. Select "s3" storage. 11010 ``` 11011 Choose a number from below, or type in your own value 11012 1 / Alias for an existing remote 11013 \ "alias" 11014 2 / Amazon Drive 11015 \ "amazon cloud drive" 11016 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, Minio, IBM COS) 11017 \ "s3" 11018 4 / Backblaze B2 11019 \ "b2" 11020 [snip] 11021 23 / http Connection 11022 \ "http" 11023 Storage> 3 11024 ``` 11025 11026 4. Select IBM COS as the S3 Storage Provider. 11027 ``` 11028 Choose the S3 provider. 11029 Choose a number from below, or type in your own value 11030 1 / Choose this option to configure Storage to AWS S3 11031 \ "AWS" 11032 2 / Choose this option to configure Storage to Ceph Systems 11033 \ "Ceph" 11034 3 / Choose this option to configure Storage to Dreamhost 11035 \ "Dreamhost" 11036 4 / Choose this option to the configure Storage to IBM COS S3 11037 \ "IBMCOS" 11038 5 / Choose this option to the configure Storage to Minio 11039 \ "Minio" 11040 Provider>4 11041 ``` 11042 11043 5. Enter the Access Key and Secret. 11044 ``` 11045 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 11046 access_key_id> <> 11047 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 11048 secret_access_key> <> 11049 ``` 11050 11051 6. Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an enpoint address. 11052 ``` 11053 Endpoint for IBM COS S3 API. 11054 Specify if using an IBM COS On Premise. 11055 Choose a number from below, or type in your own value 11056 1 / US Cross Region Endpoint 11057 \ "s3-api.us-geo.objectstorage.softlayer.net" 11058 2 / US Cross Region Dallas Endpoint 11059 \ "s3-api.dal.us-geo.objectstorage.softlayer.net" 11060 3 / US Cross Region Washington DC Endpoint 11061 \ "s3-api.wdc-us-geo.objectstorage.softlayer.net" 11062 4 / US Cross Region San Jose Endpoint 11063 \ "s3-api.sjc-us-geo.objectstorage.softlayer.net" 11064 5 / US Cross Region Private Endpoint 11065 \ "s3-api.us-geo.objectstorage.service.networklayer.com" 11066 6 / US Cross Region Dallas Private Endpoint 11067 \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com" 11068 7 / US Cross Region Washington DC Private Endpoint 11069 \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com" 11070 8 / US Cross Region San Jose Private Endpoint 11071 \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com" 11072 9 / US Region East Endpoint 11073 \ "s3.us-east.objectstorage.softlayer.net" 11074 10 / US Region East Private Endpoint 11075 \ "s3.us-east.objectstorage.service.networklayer.com" 11076 11 / US Region South Endpoint 11077 [snip] 11078 34 / Toronto Single Site Private Endpoint 11079 \ "s3.tor01.objectstorage.service.networklayer.com" 11080 endpoint>1 11081 ``` 11082 11083 11084 7. Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter 11085 ``` 11086 1 / US Cross Region Standard 11087 \ "us-standard" 11088 2 / US Cross Region Vault 11089 \ "us-vault" 11090 3 / US Cross Region Cold 11091 \ "us-cold" 11092 4 / US Cross Region Flex 11093 \ "us-flex" 11094 5 / US East Region Standard 11095 \ "us-east-standard" 11096 6 / US East Region Vault 11097 \ "us-east-vault" 11098 7 / US East Region Cold 11099 \ "us-east-cold" 11100 8 / US East Region Flex 11101 \ "us-east-flex" 11102 9 / US South Region Standard 11103 \ "us-south-standard" 11104 10 / US South Region Vault 11105 \ "us-south-vault" 11106 [snip] 11107 32 / Toronto Flex 11108 \ "tor01-flex" 11109 location_constraint>1 11110 ``` 11111 11112 9. Specify a canned ACL. IBM Cloud (Strorage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs. 11113 ``` 11114 Canned ACL used when creating buckets and/or storing objects in S3. 11115 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 11116 Choose a number from below, or type in your own value 11117 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS 11118 \ "private" 11119 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS 11120 \ "public-read" 11121 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS 11122 \ "public-read-write" 11123 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS 11124 \ "authenticated-read" 11125 acl> 1 11126 ``` 11127 11128 11129 12. Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this 11130 ``` 11131 [xxx] 11132 type = s3 11133 Provider = IBMCOS 11134 access_key_id = xxx 11135 secret_access_key = yyy 11136 endpoint = s3-api.us-geo.objectstorage.softlayer.net 11137 location_constraint = us-standard 11138 acl = private 11139 ``` 11140 11141 13. Execute rclone commands 11142 ``` 11143 1) Create a bucket. 11144 rclone mkdir IBM-COS-XREGION:newbucket 11145 2) List available buckets. 11146 rclone lsd IBM-COS-XREGION: 11147 -1 2017-11-08 21:16:22 -1 test 11148 -1 2018-02-14 20:16:39 -1 newbucket 11149 3) List contents of a bucket. 11150 rclone ls IBM-COS-XREGION:newbucket 11151 18685952 test.exe 11152 4) Copy a file from local to remote. 11153 rclone copy /Users/file.txt IBM-COS-XREGION:newbucket 11154 5) Copy a file from remote to local. 11155 rclone copy IBM-COS-XREGION:newbucket/file.txt . 11156 6) Delete a file on remote. 11157 rclone delete IBM-COS-XREGION:newbucket/file.txt 11158 ``` 11159 11160 ### Minio ### 11161 11162 [Minio](https://minio.io/) is an object storage server built for cloud application developers and devops. 11163 11164 It is very easy to install and provides an S3 compatible server which can be used by rclone. 11165 11166 To use it, install Minio following the instructions [here](https://docs.minio.io/docs/minio-quickstart-guide). 11167 11168 When it configures itself Minio will print something like this 11169 11170 ``` 11171 Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000 11172 AccessKey: USWUXHGYZQYFYFFIT3RE 11173 SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 11174 Region: us-east-1 11175 SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis 11176 11177 Browser Access: 11178 http://192.168.1.106:9000 http://172.23.0.1:9000 11179 11180 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide 11181 $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 11182 11183 Object API (Amazon S3 compatible): 11184 Go: https://docs.minio.io/docs/golang-client-quickstart-guide 11185 Java: https://docs.minio.io/docs/java-client-quickstart-guide 11186 Python: https://docs.minio.io/docs/python-client-quickstart-guide 11187 JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide 11188 .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide 11189 11190 Drive Capacity: 26 GiB Free, 165 GiB Total 11191 ``` 11192 11193 These details need to go into `rclone config` like this. Note that it 11194 is important to put the region in as stated above. 11195 11196 ``` 11197 env_auth> 1 11198 access_key_id> USWUXHGYZQYFYFFIT3RE 11199 secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 11200 region> us-east-1 11201 endpoint> http://192.168.1.106:9000 11202 location_constraint> 11203 server_side_encryption> 11204 ``` 11205 11206 Which makes the config file look like this 11207 11208 ``` 11209 [minio] 11210 type = s3 11211 provider = Minio 11212 env_auth = false 11213 access_key_id = USWUXHGYZQYFYFFIT3RE 11214 secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 11215 region = us-east-1 11216 endpoint = http://192.168.1.106:9000 11217 location_constraint = 11218 server_side_encryption = 11219 ``` 11220 11221 So once set up, for example to copy files into a bucket 11222 11223 ``` 11224 rclone copy /path/to/files minio:bucket 11225 ``` 11226 11227 ### Scaleway {#scaleway} 11228 11229 [Scaleway](https://www.scaleway.com/object-storage/) The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos. 11230 Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool. 11231 11232 Scaleway provides an S3 interface which can be configured for use with rclone like this: 11233 11234 ``` 11235 [scaleway] 11236 type = s3 11237 env_auth = false 11238 endpoint = s3.nl-ams.scw.cloud 11239 access_key_id = SCWXXXXXXXXXXXXXX 11240 secret_access_key = 1111111-2222-3333-44444-55555555555555 11241 region = nl-ams 11242 location_constraint = 11243 acl = private 11244 force_path_style = false 11245 server_side_encryption = 11246 storage_class = 11247 ``` 11248 11249 ### Wasabi ### 11250 11251 [Wasabi](https://wasabi.com) is a cloud-based object storage service for a 11252 broad range of applications and use cases. Wasabi is designed for 11253 individuals and organizations that require a high-performance, 11254 reliable, and secure data storage infrastructure at minimal cost. 11255 11256 Wasabi provides an S3 interface which can be configured for use with 11257 rclone like this. 11258 11259 ``` 11260 No remotes found - make a new one 11261 n) New remote 11262 s) Set configuration password 11263 n/s> n 11264 name> wasabi 11265 Type of storage to configure. 11266 Choose a number from below, or type in your own value 11267 [snip] 11268 XX / Amazon S3 (also Dreamhost, Ceph, Minio) 11269 \ "s3" 11270 [snip] 11271 Storage> s3 11272 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 11273 Choose a number from below, or type in your own value 11274 1 / Enter AWS credentials in the next step 11275 \ "false" 11276 2 / Get AWS credentials from the environment (env vars or IAM) 11277 \ "true" 11278 env_auth> 1 11279 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 11280 access_key_id> YOURACCESSKEY 11281 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 11282 secret_access_key> YOURSECRETACCESSKEY 11283 Region to connect to. 11284 Choose a number from below, or type in your own value 11285 / The default endpoint - a good choice if you are unsure. 11286 1 | US Region, Northern Virginia or Pacific Northwest. 11287 | Leave location constraint empty. 11288 \ "us-east-1" 11289 [snip] 11290 region> us-east-1 11291 Endpoint for S3 API. 11292 Leave blank if using AWS to use the default endpoint for the region. 11293 Specify if using an S3 clone such as Ceph. 11294 endpoint> s3.wasabisys.com 11295 Location constraint - must be set to match the Region. Used when creating buckets only. 11296 Choose a number from below, or type in your own value 11297 1 / Empty for US Region, Northern Virginia or Pacific Northwest. 11298 \ "" 11299 [snip] 11300 location_constraint> 11301 Canned ACL used when creating buckets and/or storing objects in S3. 11302 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 11303 Choose a number from below, or type in your own value 11304 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 11305 \ "private" 11306 [snip] 11307 acl> 11308 The server-side encryption algorithm used when storing this object in S3. 11309 Choose a number from below, or type in your own value 11310 1 / None 11311 \ "" 11312 2 / AES256 11313 \ "AES256" 11314 server_side_encryption> 11315 The storage class to use when storing objects in S3. 11316 Choose a number from below, or type in your own value 11317 1 / Default 11318 \ "" 11319 2 / Standard storage class 11320 \ "STANDARD" 11321 3 / Reduced redundancy storage class 11322 \ "REDUCED_REDUNDANCY" 11323 4 / Standard Infrequent Access storage class 11324 \ "STANDARD_IA" 11325 storage_class> 11326 Remote config 11327 -------------------- 11328 [wasabi] 11329 env_auth = false 11330 access_key_id = YOURACCESSKEY 11331 secret_access_key = YOURSECRETACCESSKEY 11332 region = us-east-1 11333 endpoint = s3.wasabisys.com 11334 location_constraint = 11335 acl = 11336 server_side_encryption = 11337 storage_class = 11338 -------------------- 11339 y) Yes this is OK 11340 e) Edit this remote 11341 d) Delete this remote 11342 y/e/d> y 11343 ``` 11344 11345 This will leave the config file looking like this. 11346 11347 ``` 11348 [wasabi] 11349 type = s3 11350 provider = Wasabi 11351 env_auth = false 11352 access_key_id = YOURACCESSKEY 11353 secret_access_key = YOURSECRETACCESSKEY 11354 region = 11355 endpoint = s3.wasabisys.com 11356 location_constraint = 11357 acl = 11358 server_side_encryption = 11359 storage_class = 11360 ``` 11361 11362 ### Alibaba OSS {#alibaba-oss} 11363 11364 Here is an example of making an [Alibaba Cloud (Aliyun) OSS](https://www.alibabacloud.com/product/oss/) 11365 configuration. First run: 11366 11367 rclone config 11368 11369 This will guide you through an interactive setup process. 11370 11371 ``` 11372 No remotes found - make a new one 11373 n) New remote 11374 s) Set configuration password 11375 q) Quit config 11376 n/s/q> n 11377 name> oss 11378 Type of storage to configure. 11379 Enter a string value. Press Enter for the default (""). 11380 Choose a number from below, or type in your own value 11381 [snip] 11382 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) 11383 \ "s3" 11384 [snip] 11385 Storage> s3 11386 Choose your S3 provider. 11387 Enter a string value. Press Enter for the default (""). 11388 Choose a number from below, or type in your own value 11389 1 / Amazon Web Services (AWS) S3 11390 \ "AWS" 11391 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun 11392 \ "Alibaba" 11393 3 / Ceph Object Storage 11394 \ "Ceph" 11395 [snip] 11396 provider> Alibaba 11397 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 11398 Only applies if access_key_id and secret_access_key is blank. 11399 Enter a boolean value (true or false). Press Enter for the default ("false"). 11400 Choose a number from below, or type in your own value 11401 1 / Enter AWS credentials in the next step 11402 \ "false" 11403 2 / Get AWS credentials from the environment (env vars or IAM) 11404 \ "true" 11405 env_auth> 1 11406 AWS Access Key ID. 11407 Leave blank for anonymous access or runtime credentials. 11408 Enter a string value. Press Enter for the default (""). 11409 access_key_id> accesskeyid 11410 AWS Secret Access Key (password) 11411 Leave blank for anonymous access or runtime credentials. 11412 Enter a string value. Press Enter for the default (""). 11413 secret_access_key> secretaccesskey 11414 Endpoint for OSS API. 11415 Enter a string value. Press Enter for the default (""). 11416 Choose a number from below, or type in your own value 11417 1 / East China 1 (Hangzhou) 11418 \ "oss-cn-hangzhou.aliyuncs.com" 11419 2 / East China 2 (Shanghai) 11420 \ "oss-cn-shanghai.aliyuncs.com" 11421 3 / North China 1 (Qingdao) 11422 \ "oss-cn-qingdao.aliyuncs.com" 11423 [snip] 11424 endpoint> 1 11425 Canned ACL used when creating buckets and storing or copying objects. 11426 11427 Note that this ACL is applied when server side copying objects as S3 11428 doesn't copy the ACL from the source but rather writes a fresh one. 11429 Enter a string value. Press Enter for the default (""). 11430 Choose a number from below, or type in your own value 11431 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 11432 \ "private" 11433 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 11434 \ "public-read" 11435 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 11436 [snip] 11437 acl> 1 11438 The storage class to use when storing new objects in OSS. 11439 Enter a string value. Press Enter for the default (""). 11440 Choose a number from below, or type in your own value 11441 1 / Default 11442 \ "" 11443 2 / Standard storage class 11444 \ "STANDARD" 11445 3 / Archive storage mode. 11446 \ "GLACIER" 11447 4 / Infrequent access storage mode. 11448 \ "STANDARD_IA" 11449 storage_class> 1 11450 Edit advanced config? (y/n) 11451 y) Yes 11452 n) No 11453 y/n> n 11454 Remote config 11455 -------------------- 11456 [oss] 11457 type = s3 11458 provider = Alibaba 11459 env_auth = false 11460 access_key_id = accesskeyid 11461 secret_access_key = secretaccesskey 11462 endpoint = oss-cn-hangzhou.aliyuncs.com 11463 acl = private 11464 storage_class = Standard 11465 -------------------- 11466 y) Yes this is OK 11467 e) Edit this remote 11468 d) Delete this remote 11469 y/e/d> y 11470 ``` 11471 11472 ### Netease NOS ### 11473 11474 For Netease NOS configure as per the configurator `rclone config` 11475 setting the provider `Netease`. This will automatically set 11476 `force_path_style = false` which is necessary for it to run properly. 11477 11478 Backblaze B2 11479 ---------------------------------------- 11480 11481 B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/). 11482 11483 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 11484 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 11485 11486 Here is an example of making a b2 configuration. First run 11487 11488 rclone config 11489 11490 This will guide you through an interactive setup process. To authenticate 11491 you will either need your Account ID (a short hex number) and Master 11492 Application Key (a long hex number) OR an Application Key, which is the 11493 recommended method. See below for further details on generating and using 11494 an Application Key. 11495 11496 ``` 11497 No remotes found - make a new one 11498 n) New remote 11499 q) Quit config 11500 n/q> n 11501 name> remote 11502 Type of storage to configure. 11503 Choose a number from below, or type in your own value 11504 [snip] 11505 XX / Backblaze B2 11506 \ "b2" 11507 [snip] 11508 Storage> b2 11509 Account ID or Application Key ID 11510 account> 123456789abc 11511 Application Key 11512 key> 0123456789abcdef0123456789abcdef0123456789 11513 Endpoint for the service - leave blank normally. 11514 endpoint> 11515 Remote config 11516 -------------------- 11517 [remote] 11518 account = 123456789abc 11519 key = 0123456789abcdef0123456789abcdef0123456789 11520 endpoint = 11521 -------------------- 11522 y) Yes this is OK 11523 e) Edit this remote 11524 d) Delete this remote 11525 y/e/d> y 11526 ``` 11527 11528 This remote is called `remote` and can now be used like this 11529 11530 See all buckets 11531 11532 rclone lsd remote: 11533 11534 Create a new bucket 11535 11536 rclone mkdir remote:bucket 11537 11538 List the contents of a bucket 11539 11540 rclone ls remote:bucket 11541 11542 Sync `/home/local/directory` to the remote bucket, deleting any 11543 excess files in the bucket. 11544 11545 rclone sync /home/local/directory remote:bucket 11546 11547 ### Application Keys ### 11548 11549 B2 supports multiple [Application Keys for different access permission 11550 to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html). 11551 11552 You can use these with rclone too; you will need to use rclone version 1.43 11553 or later. 11554 11555 Follow Backblaze's docs to create an Application Key with the required 11556 permission and add the `applicationKeyId` as the `account` and the 11557 `Application Key` itself as the `key`. 11558 11559 Note that you must put the _applicationKeyId_ as the `account` – you 11560 can't use the master Account ID. If you try then B2 will return 401 11561 errors. 11562 11563 ### --fast-list ### 11564 11565 This remote supports `--fast-list` which allows you to use fewer 11566 transactions in exchange for more memory. See the [rclone 11567 docs](/docs/#fast-list) for more details. 11568 11569 ### Modified time ### 11570 11571 The modified time is stored as metadata on the object as 11572 `X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01 11573 in the Backblaze standard. Other tools should be able to use this as 11574 a modified time. 11575 11576 Modified times are used in syncing and are fully supported. Note that 11577 if a modification time needs to be updated on an object then it will 11578 create a new version of the object. 11579 11580 #### Restricted filename characters 11581 11582 In addition to the [default restricted characters set](/overview/#restricted-characters) 11583 the following characters are also replaced: 11584 11585 | Character | Value | Replacement | 11586 | --------- |:-----:|:-----------:| 11587 | \ | 0x5C | \ | 11588 11589 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 11590 as they can't be used in JSON strings. 11591 11592 ### SHA1 checksums ### 11593 11594 The SHA1 checksums of the files are checked on upload and download and 11595 will be used in the syncing process. 11596 11597 Large files (bigger than the limit in `--b2-upload-cutoff`) which are 11598 uploaded in chunks will store their SHA1 on the object as 11599 `X-Bz-Info-large_file_sha1` as recommended by Backblaze. 11600 11601 For a large file to be uploaded with an SHA1 checksum, the source 11602 needs to support SHA1 checksums. The local disk supports SHA1 11603 checksums so large file transfers from local disk will have an SHA1. 11604 See [the overview](/overview/#features) for exactly which remotes 11605 support SHA1. 11606 11607 Sources which don't support SHA1, in particular `crypt` will upload 11608 large files without SHA1 checksums. This may be fixed in the future 11609 (see [#1767](https://github.com/rclone/rclone/issues/1767)). 11610 11611 Files sizes below `--b2-upload-cutoff` will always have an SHA1 11612 regardless of the source. 11613 11614 ### Transfers ### 11615 11616 Backblaze recommends that you do lots of transfers simultaneously for 11617 maximum speed. In tests from my SSD equipped laptop the optimum 11618 setting is about `--transfers 32` though higher numbers may be used 11619 for a slight speed improvement. The optimum number for you may vary 11620 depending on your hardware, how big the files are, how much you want 11621 to load your computer, etc. The default of `--transfers 4` is 11622 definitely too low for Backblaze B2 though. 11623 11624 Note that uploading big files (bigger than 200 MB by default) will use 11625 a 96 MB RAM buffer by default. There can be at most `--transfers` of 11626 these in use at any moment, so this sets the upper limit on the memory 11627 used. 11628 11629 ### Versions ### 11630 11631 When rclone uploads a new version of a file it creates a [new version 11632 of it](https://www.backblaze.com/b2/docs/file_versions.html). 11633 Likewise when you delete a file, the old version will be marked hidden 11634 and still be available. Conversely, you may opt in to a "hard delete" 11635 of files with the `--b2-hard-delete` flag which would permanently remove 11636 the file instead of hiding it. 11637 11638 Old versions of files, where available, are visible using the 11639 `--b2-versions` flag. 11640 11641 **NB** Note that `--b2-versions` does not work with crypt at the 11642 moment [#1627](https://github.com/rclone/rclone/issues/1627). Using 11643 [--backup-dir](/docs/#backup-dir-dir) with rclone is the recommended 11644 way of working around this. 11645 11646 If you wish to remove all the old versions then you can use the 11647 `rclone cleanup remote:bucket` command which will delete all the old 11648 versions of files, leaving the current ones intact. You can also 11649 supply a path and only old versions under that path will be deleted, 11650 eg `rclone cleanup remote:bucket/path/to/stuff`. 11651 11652 Note that `cleanup` will remove partially uploaded files from the bucket 11653 if they are more than a day old. 11654 11655 When you `purge` a bucket, the current and the old versions will be 11656 deleted then the bucket will be deleted. 11657 11658 However `delete` will cause the current versions of the files to 11659 become hidden old versions. 11660 11661 Here is a session showing the listing and retrieval of an old 11662 version followed by a `cleanup` of the old versions. 11663 11664 Show current version and all the versions with `--b2-versions` flag. 11665 11666 ``` 11667 $ rclone -q ls b2:cleanup-test 11668 9 one.txt 11669 11670 $ rclone -q --b2-versions ls b2:cleanup-test 11671 9 one.txt 11672 8 one-v2016-07-04-141032-000.txt 11673 16 one-v2016-07-04-141003-000.txt 11674 15 one-v2016-07-02-155621-000.txt 11675 ``` 11676 11677 Retrieve an old version 11678 11679 ``` 11680 $ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp 11681 11682 $ ls -l /tmp/one-v2016-07-04-141003-000.txt 11683 -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt 11684 ``` 11685 11686 Clean up all the old versions and show that they've gone. 11687 11688 ``` 11689 $ rclone -q cleanup b2:cleanup-test 11690 11691 $ rclone -q ls b2:cleanup-test 11692 9 one.txt 11693 11694 $ rclone -q --b2-versions ls b2:cleanup-test 11695 9 one.txt 11696 ``` 11697 11698 ### Data usage ### 11699 11700 It is useful to know how many requests are sent to the server in different scenarios. 11701 11702 All copy commands send the following 4 requests: 11703 11704 ``` 11705 /b2api/v1/b2_authorize_account 11706 /b2api/v1/b2_create_bucket 11707 /b2api/v1/b2_list_buckets 11708 /b2api/v1/b2_list_file_names 11709 ``` 11710 11711 The `b2_list_file_names` request will be sent once for every 1k files 11712 in the remote path, providing the checksum and modification time of 11713 the listed files. As of version 1.33 issue 11714 [#818](https://github.com/rclone/rclone/issues/818) causes extra requests 11715 to be sent when using B2 with Crypt. When a copy operation does not 11716 require any files to be uploaded, no more requests will be sent. 11717 11718 Uploading files that do not require chunking, will send 2 requests per 11719 file upload: 11720 11721 ``` 11722 /b2api/v1/b2_get_upload_url 11723 /b2api/v1/b2_upload_file/ 11724 ``` 11725 11726 Uploading files requiring chunking, will send 2 requests (one each to 11727 start and finish the upload) and another 2 requests for each chunk: 11728 11729 ``` 11730 /b2api/v1/b2_start_large_file 11731 /b2api/v1/b2_get_upload_part_url 11732 /b2api/v1/b2_upload_part/ 11733 /b2api/v1/b2_finish_large_file 11734 ``` 11735 11736 #### Versions #### 11737 11738 Versions can be viewed with the `--b2-versions` flag. When it is set 11739 rclone will show and act on older versions of files. For example 11740 11741 Listing without `--b2-versions` 11742 11743 ``` 11744 $ rclone -q ls b2:cleanup-test 11745 9 one.txt 11746 ``` 11747 11748 And with 11749 11750 ``` 11751 $ rclone -q --b2-versions ls b2:cleanup-test 11752 9 one.txt 11753 8 one-v2016-07-04-141032-000.txt 11754 16 one-v2016-07-04-141003-000.txt 11755 15 one-v2016-07-02-155621-000.txt 11756 ``` 11757 11758 Showing that the current version is unchanged but older versions can 11759 be seen. These have the UTC date that they were uploaded to the 11760 server to the nearest millisecond appended to them. 11761 11762 Note that when using `--b2-versions` no file write operations are 11763 permitted, so you can't upload files or delete them. 11764 11765 ### B2 and rclone link ### 11766 11767 Rclone supports generating file share links for private B2 buckets. 11768 They can either be for a file for example: 11769 11770 ``` 11771 ./rclone link B2:bucket/path/to/file.txt 11772 https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx 11773 11774 ``` 11775 11776 or if run on a directory you will get: 11777 11778 ``` 11779 ./rclone link B2:bucket/path 11780 https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx 11781 ``` 11782 11783 you can then use the authorization token (the part of the url from the 11784 `?Authorization=` on) on any file path under that directory. For example: 11785 11786 ``` 11787 https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx 11788 https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx 11789 https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx 11790 11791 ``` 11792 11793 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/b2/b2.go then run make backenddocs --> 11794 ### Standard Options 11795 11796 Here are the standard options specific to b2 (Backblaze B2). 11797 11798 #### --b2-account 11799 11800 Account ID or Application Key ID 11801 11802 - Config: account 11803 - Env Var: RCLONE_B2_ACCOUNT 11804 - Type: string 11805 - Default: "" 11806 11807 #### --b2-key 11808 11809 Application Key 11810 11811 - Config: key 11812 - Env Var: RCLONE_B2_KEY 11813 - Type: string 11814 - Default: "" 11815 11816 #### --b2-hard-delete 11817 11818 Permanently delete files on remote removal, otherwise hide files. 11819 11820 - Config: hard_delete 11821 - Env Var: RCLONE_B2_HARD_DELETE 11822 - Type: bool 11823 - Default: false 11824 11825 ### Advanced Options 11826 11827 Here are the advanced options specific to b2 (Backblaze B2). 11828 11829 #### --b2-endpoint 11830 11831 Endpoint for the service. 11832 Leave blank normally. 11833 11834 - Config: endpoint 11835 - Env Var: RCLONE_B2_ENDPOINT 11836 - Type: string 11837 - Default: "" 11838 11839 #### --b2-test-mode 11840 11841 A flag string for X-Bz-Test-Mode header for debugging. 11842 11843 This is for debugging purposes only. Setting it to one of the strings 11844 below will cause b2 to return specific errors: 11845 11846 * "fail_some_uploads" 11847 * "expire_some_account_authorization_tokens" 11848 * "force_cap_exceeded" 11849 11850 These will be set in the "X-Bz-Test-Mode" header which is documented 11851 in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html). 11852 11853 - Config: test_mode 11854 - Env Var: RCLONE_B2_TEST_MODE 11855 - Type: string 11856 - Default: "" 11857 11858 #### --b2-versions 11859 11860 Include old versions in directory listings. 11861 Note that when using this no file write operations are permitted, 11862 so you can't upload files or delete them. 11863 11864 - Config: versions 11865 - Env Var: RCLONE_B2_VERSIONS 11866 - Type: bool 11867 - Default: false 11868 11869 #### --b2-upload-cutoff 11870 11871 Cutoff for switching to chunked upload. 11872 11873 Files above this size will be uploaded in chunks of "--b2-chunk-size". 11874 11875 This value should be set no larger than 4.657GiB (== 5GB). 11876 11877 - Config: upload_cutoff 11878 - Env Var: RCLONE_B2_UPLOAD_CUTOFF 11879 - Type: SizeSuffix 11880 - Default: 200M 11881 11882 #### --b2-chunk-size 11883 11884 Upload chunk size. Must fit in memory. 11885 11886 When uploading large files, chunk the file into this size. Note that 11887 these chunks are buffered in memory and there might a maximum of 11888 "--transfers" chunks in progress at once. 5,000,000 Bytes is the 11889 minimum size. 11890 11891 - Config: chunk_size 11892 - Env Var: RCLONE_B2_CHUNK_SIZE 11893 - Type: SizeSuffix 11894 - Default: 96M 11895 11896 #### --b2-disable-checksum 11897 11898 Disable checksums for large (> upload cutoff) files 11899 11900 - Config: disable_checksum 11901 - Env Var: RCLONE_B2_DISABLE_CHECKSUM 11902 - Type: bool 11903 - Default: false 11904 11905 #### --b2-download-url 11906 11907 Custom endpoint for downloads. 11908 11909 This is usually set to a Cloudflare CDN URL as Backblaze offers 11910 free egress for data downloaded through the Cloudflare network. 11911 This is probably only useful for a public bucket. 11912 Leave blank if you want to use the endpoint provided by Backblaze. 11913 11914 - Config: download_url 11915 - Env Var: RCLONE_B2_DOWNLOAD_URL 11916 - Type: string 11917 - Default: "" 11918 11919 #### --b2-download-auth-duration 11920 11921 Time before the authorization token will expire in s or suffix ms|s|m|h|d. 11922 11923 The duration before the download authorization token will expire. 11924 The minimum value is 1 second. The maximum value is one week. 11925 11926 - Config: download_auth_duration 11927 - Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION 11928 - Type: Duration 11929 - Default: 1w 11930 11931 #### --b2-encoding 11932 11933 This sets the encoding for the backend. 11934 11935 See: the [encoding section in the overview](/overview/#encoding) for more info. 11936 11937 - Config: encoding 11938 - Env Var: RCLONE_B2_ENCODING 11939 - Type: MultiEncoder 11940 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 11941 11942 <!--- autogenerated options stop --> 11943 11944 Box 11945 ----------------------------------------- 11946 11947 Paths are specified as `remote:path` 11948 11949 Paths may be as deep as required, eg `remote:directory/subdirectory`. 11950 11951 The initial setup for Box involves getting a token from Box which you 11952 can do either in your browser, or with a config.json downloaded from Box 11953 to use JWT authentication. `rclone config` walks you through it. 11954 11955 Here is an example of how to make a remote called `remote`. First run: 11956 11957 rclone config 11958 11959 This will guide you through an interactive setup process: 11960 11961 ``` 11962 No remotes found - make a new one 11963 n) New remote 11964 s) Set configuration password 11965 q) Quit config 11966 n/s/q> n 11967 name> remote 11968 Type of storage to configure. 11969 Choose a number from below, or type in your own value 11970 [snip] 11971 XX / Box 11972 \ "box" 11973 [snip] 11974 Storage> box 11975 Box App Client Id - leave blank normally. 11976 client_id> 11977 Box App Client Secret - leave blank normally. 11978 client_secret> 11979 Box App config.json location 11980 Leave blank normally. 11981 Enter a string value. Press Enter for the default (""). 11982 config_json> 11983 'enterprise' or 'user' depending on the type of token being requested. 11984 Enter a string value. Press Enter for the default ("user"). 11985 box_sub_type> 11986 Remote config 11987 Use auto config? 11988 * Say Y if not sure 11989 * Say N if you are working on a remote or headless machine 11990 y) Yes 11991 n) No 11992 y/n> y 11993 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 11994 Log in and authorize rclone for access 11995 Waiting for code... 11996 Got code 11997 -------------------- 11998 [remote] 11999 client_id = 12000 client_secret = 12001 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} 12002 -------------------- 12003 y) Yes this is OK 12004 e) Edit this remote 12005 d) Delete this remote 12006 y/e/d> y 12007 ``` 12008 12009 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 12010 machine with no Internet browser available. 12011 12012 Note that rclone runs a webserver on your local machine to collect the 12013 token as returned from Box. This only runs from the moment it opens 12014 your browser to the moment you get back the verification code. This 12015 is on `http://127.0.0.1:53682/` and this it may require you to unblock 12016 it temporarily if you are running a host firewall. 12017 12018 Once configured you can then use `rclone` like this, 12019 12020 List directories in top level of your Box 12021 12022 rclone lsd remote: 12023 12024 List all the files in your Box 12025 12026 rclone ls remote: 12027 12028 To copy a local directory to an Box directory called backup 12029 12030 rclone copy /home/source remote:backup 12031 12032 ### Using rclone with an Enterprise account with SSO ### 12033 12034 If you have an "Enterprise" account type with Box with single sign on 12035 (SSO), you need to create a password to use Box with rclone. This can 12036 be done at your Enterprise Box account by going to Settings, "Account" 12037 Tab, and then set the password in the "Authentication" field. 12038 12039 Once you have done this, you can setup your Enterprise Box account 12040 using the same procedure detailed above in the, using the password you 12041 have just set. 12042 12043 ### Invalid refresh token ### 12044 12045 According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens): 12046 12047 > Each refresh_token is valid for one use in 60 days. 12048 12049 This means that if you 12050 12051 * Don't use the box remote for 60 days 12052 * Copy the config file with a box refresh token in and use it in two places 12053 * Get an error on a token refresh 12054 12055 then rclone will return an error which includes the text `Invalid 12056 refresh token`. 12057 12058 To fix this you will need to use oauth2 again to update the refresh 12059 token. You can use the methods in [the remote setup 12060 docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the 12061 config file method, you should not use that remote on the computer you 12062 did the authentication on. 12063 12064 Here is how to do it. 12065 12066 ``` 12067 $ rclone config 12068 Current remotes: 12069 12070 Name Type 12071 ==== ==== 12072 remote box 12073 12074 e) Edit existing remote 12075 n) New remote 12076 d) Delete remote 12077 r) Rename remote 12078 c) Copy remote 12079 s) Set configuration password 12080 q) Quit config 12081 e/n/d/r/c/s/q> e 12082 Choose a number from below, or type in an existing value 12083 1 > remote 12084 remote> remote 12085 -------------------- 12086 [remote] 12087 type = box 12088 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} 12089 -------------------- 12090 Edit remote 12091 Value "client_id" = "" 12092 Edit? (y/n)> 12093 y) Yes 12094 n) No 12095 y/n> n 12096 Value "client_secret" = "" 12097 Edit? (y/n)> 12098 y) Yes 12099 n) No 12100 y/n> n 12101 Remote config 12102 Already have a token - refresh? 12103 y) Yes 12104 n) No 12105 y/n> y 12106 Use auto config? 12107 * Say Y if not sure 12108 * Say N if you are working on a remote or headless machine 12109 y) Yes 12110 n) No 12111 y/n> y 12112 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 12113 Log in and authorize rclone for access 12114 Waiting for code... 12115 Got code 12116 -------------------- 12117 [remote] 12118 type = box 12119 token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} 12120 -------------------- 12121 y) Yes this is OK 12122 e) Edit this remote 12123 d) Delete this remote 12124 y/e/d> y 12125 ``` 12126 12127 ### Modified time and hashes ### 12128 12129 Box allows modification times to be set on objects accurate to 1 12130 second. These will be used to detect whether objects need syncing or 12131 not. 12132 12133 Box supports SHA1 type hashes, so you can use the `--checksum` 12134 flag. 12135 12136 #### Restricted filename characters 12137 12138 In addition to the [default restricted characters set](/overview/#restricted-characters) 12139 the following characters are also replaced: 12140 12141 | Character | Value | Replacement | 12142 | --------- |:-----:|:-----------:| 12143 | \ | 0x5C | \ | 12144 12145 File names can also not end with the following characters. 12146 These only get replaced if they are last character in the name: 12147 12148 | Character | Value | Replacement | 12149 | --------- |:-----:|:-----------:| 12150 | SP | 0x20 | ␠ | 12151 12152 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 12153 as they can't be used in JSON strings. 12154 12155 ### Transfers ### 12156 12157 For files above 50MB rclone will use a chunked transfer. Rclone will 12158 upload up to `--transfers` chunks at the same time (shared among all 12159 the multipart uploads). Chunks are buffered in memory and are 12160 normally 8MB so increasing `--transfers` will increase memory use. 12161 12162 ### Deleting files ### 12163 12164 Depending on the enterprise settings for your user, the item will 12165 either be actually deleted from Box or moved to the trash. 12166 12167 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/box/box.go then run make backenddocs --> 12168 ### Standard Options 12169 12170 Here are the standard options specific to box (Box). 12171 12172 #### --box-client-id 12173 12174 Box App Client Id. 12175 Leave blank normally. 12176 12177 - Config: client_id 12178 - Env Var: RCLONE_BOX_CLIENT_ID 12179 - Type: string 12180 - Default: "" 12181 12182 #### --box-client-secret 12183 12184 Box App Client Secret 12185 Leave blank normally. 12186 12187 - Config: client_secret 12188 - Env Var: RCLONE_BOX_CLIENT_SECRET 12189 - Type: string 12190 - Default: "" 12191 12192 #### --box-box-config-file 12193 12194 Box App config.json location 12195 Leave blank normally. 12196 12197 - Config: box_config_file 12198 - Env Var: RCLONE_BOX_BOX_CONFIG_FILE 12199 - Type: string 12200 - Default: "" 12201 12202 #### --box-box-sub-type 12203 12204 12205 12206 - Config: box_sub_type 12207 - Env Var: RCLONE_BOX_BOX_SUB_TYPE 12208 - Type: string 12209 - Default: "user" 12210 - Examples: 12211 - "user" 12212 - Rclone should act on behalf of a user 12213 - "enterprise" 12214 - Rclone should act on behalf of a service account 12215 12216 ### Advanced Options 12217 12218 Here are the advanced options specific to box (Box). 12219 12220 #### --box-upload-cutoff 12221 12222 Cutoff for switching to multipart upload (>= 50MB). 12223 12224 - Config: upload_cutoff 12225 - Env Var: RCLONE_BOX_UPLOAD_CUTOFF 12226 - Type: SizeSuffix 12227 - Default: 50M 12228 12229 #### --box-commit-retries 12230 12231 Max number of times to try committing a multipart file. 12232 12233 - Config: commit_retries 12234 - Env Var: RCLONE_BOX_COMMIT_RETRIES 12235 - Type: int 12236 - Default: 100 12237 12238 #### --box-encoding 12239 12240 This sets the encoding for the backend. 12241 12242 See: the [encoding section in the overview](/overview/#encoding) for more info. 12243 12244 - Config: encoding 12245 - Env Var: RCLONE_BOX_ENCODING 12246 - Type: MultiEncoder 12247 - Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot 12248 12249 <!--- autogenerated options stop --> 12250 12251 ### Limitations ### 12252 12253 Note that Box is case insensitive so you can't have a file called 12254 "Hello.doc" and one called "hello.doc". 12255 12256 Box file names can't have the `\` character in. rclone maps this to 12257 and from an identical looking unicode equivalent `\`. 12258 12259 Box only supports filenames up to 255 characters in length. 12260 12261 Cache (BETA) 12262 ----------------------------------------- 12263 12264 The `cache` remote wraps another existing remote and stores file structure 12265 and its data for long running tasks like `rclone mount`. 12266 12267 To get started you just need to have an existing remote which can be configured 12268 with `cache`. 12269 12270 Here is an example of how to make a remote called `test-cache`. First run: 12271 12272 rclone config 12273 12274 This will guide you through an interactive setup process: 12275 12276 ``` 12277 No remotes found - make a new one 12278 n) New remote 12279 r) Rename remote 12280 c) Copy remote 12281 s) Set configuration password 12282 q) Quit config 12283 n/r/c/s/q> n 12284 name> test-cache 12285 Type of storage to configure. 12286 Choose a number from below, or type in your own value 12287 [snip] 12288 XX / Cache a remote 12289 \ "cache" 12290 [snip] 12291 Storage> cache 12292 Remote to cache. 12293 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 12294 "myremote:bucket" or maybe "myremote:" (not recommended). 12295 remote> local:/test 12296 Optional: The URL of the Plex server 12297 plex_url> http://127.0.0.1:32400 12298 Optional: The username of the Plex user 12299 plex_username> dummyusername 12300 Optional: The password of the Plex user 12301 y) Yes type in my own password 12302 g) Generate random password 12303 n) No leave this optional password blank 12304 y/g/n> y 12305 Enter the password: 12306 password: 12307 Confirm the password: 12308 password: 12309 The size of a chunk. Lower value good for slow connections but can affect seamless reading. 12310 Default: 5M 12311 Choose a number from below, or type in your own value 12312 1 / 1MB 12313 \ "1m" 12314 2 / 5 MB 12315 \ "5M" 12316 3 / 10 MB 12317 \ "10M" 12318 chunk_size> 2 12319 How much time should object info (file size, file hashes etc) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. 12320 Accepted units are: "s", "m", "h". 12321 Default: 5m 12322 Choose a number from below, or type in your own value 12323 1 / 1 hour 12324 \ "1h" 12325 2 / 24 hours 12326 \ "24h" 12327 3 / 24 hours 12328 \ "48h" 12329 info_age> 2 12330 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. 12331 Default: 10G 12332 Choose a number from below, or type in your own value 12333 1 / 500 MB 12334 \ "500M" 12335 2 / 1 GB 12336 \ "1G" 12337 3 / 10 GB 12338 \ "10G" 12339 chunk_total_size> 3 12340 Remote config 12341 -------------------- 12342 [test-cache] 12343 remote = local:/test 12344 plex_url = http://127.0.0.1:32400 12345 plex_username = dummyusername 12346 plex_password = *** ENCRYPTED *** 12347 chunk_size = 5M 12348 info_age = 48h 12349 chunk_total_size = 10G 12350 ``` 12351 12352 You can then use it like this, 12353 12354 List directories in top level of your drive 12355 12356 rclone lsd test-cache: 12357 12358 List all the files in your drive 12359 12360 rclone ls test-cache: 12361 12362 To start a cached mount 12363 12364 rclone mount --allow-other test-cache: /var/tmp/test-cache 12365 12366 ### Write Features ### 12367 12368 ### Offline uploading ### 12369 12370 In an effort to make writing through cache more reliable, the backend 12371 now supports this feature which can be activated by specifying a 12372 `cache-tmp-upload-path`. 12373 12374 A files goes through these states when using this feature: 12375 12376 1. An upload is started (usually by copying a file on the cache remote) 12377 2. When the copy to the temporary location is complete the file is part 12378 of the cached remote and looks and behaves like any other file (reading included) 12379 3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move` 12380 is used to move the file to the cloud provider 12381 4. Reading the file still works during the upload but most modifications on it will be prohibited 12382 5. Once the move is complete the file is unlocked for modifications as it 12383 becomes as any other regular file 12384 6. If the file is being read through `cache` when it's actually 12385 deleted from the temporary path then `cache` will simply swap the source 12386 to the cloud provider without interrupting the reading (small blip can happen though) 12387 12388 Files are uploaded in sequence and only one file is uploaded at a time. 12389 Uploads will be stored in a queue and be processed based on the order they were added. 12390 The queue and the temporary storage is persistent across restarts but 12391 can be cleared on startup with the `--cache-db-purge` flag. 12392 12393 ### Write Support ### 12394 12395 Writes are supported through `cache`. 12396 One caveat is that a mounted cache remote does not add any retry or fallback 12397 mechanism to the upload operation. This will depend on the implementation 12398 of the wrapped remote. Consider using `Offline uploading` for reliable writes. 12399 12400 One special case is covered with `cache-writes` which will cache the file 12401 data at the same time as the upload when it is enabled making it available 12402 from the cache store immediately once the upload is finished. 12403 12404 ### Read Features ### 12405 12406 #### Multiple connections #### 12407 12408 To counter the high latency between a local PC where rclone is running 12409 and cloud providers, the cache remote can split multiple requests to the 12410 cloud provider for smaller file chunks and combines them together locally 12411 where they can be available almost immediately before the reader usually 12412 needs them. 12413 12414 This is similar to buffering when media files are played online. Rclone 12415 will stay around the current marker but always try its best to stay ahead 12416 and prepare the data before. 12417 12418 #### Plex Integration #### 12419 12420 There is a direct integration with Plex which allows cache to detect during reading 12421 if the file is in playback or not. This helps cache to adapt how it queries 12422 the cloud provider depending on what is needed for. 12423 12424 Scans will have a minimum amount of workers (1) while in a confirmed playback cache 12425 will deploy the configured number of workers. 12426 12427 This integration opens the doorway to additional performance improvements 12428 which will be explored in the near future. 12429 12430 **Note:** If Plex options are not configured, `cache` will function with its 12431 configured options without adapting any of its settings. 12432 12433 How to enable? Run `rclone config` and add all the Plex options (endpoint, username 12434 and password) in your remote and it will be automatically enabled. 12435 12436 Affected settings: 12437 - `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times 12438 12439 ##### Certificate Validation ##### 12440 12441 When the Plex server is configured to only accept secure connections, it is 12442 possible to use `.plex.direct` URL's to ensure certificate validation succeeds. 12443 These URL's are used by Plex internally to connect to the Plex server securely. 12444 12445 The format for this URL's is the following: 12446 12447 https://ip-with-dots-replaced.server-hash.plex.direct:32400/ 12448 12449 The `ip-with-dots-replaced` part can be any IPv4 address, where the dots 12450 have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`. 12451 12452 To get the `server-hash` part, the easiest way is to visit 12453 12454 https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token 12455 12456 This page will list all the available Plex servers for your account 12457 with at least one `.plex.direct` link for each. Copy one URL and replace 12458 the IP address with the desired address. This can be used as the 12459 `plex_url` value. 12460 12461 ### Known issues ### 12462 12463 #### Mount and --dir-cache-time #### 12464 12465 --dir-cache-time controls the first layer of directory caching which works at the mount layer. 12466 Being an independent caching mechanism from the `cache` backend, it will manage its own entries 12467 based on the configured time. 12468 12469 To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct 12470 one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are 12471 already configured in this way. 12472 12473 #### Windows support - Experimental #### 12474 12475 There are a couple of issues with Windows `mount` functionality that still require some investigations. 12476 It should be considered as experimental thus far as fixes come in for this OS. 12477 12478 Most of the issues seem to be related to the difference between filesystems 12479 on Linux flavors and Windows as cache is heavily dependant on them. 12480 12481 Any reports or feedback on how cache behaves on this OS is greatly appreciated. 12482 12483 - https://github.com/rclone/rclone/issues/1935 12484 - https://github.com/rclone/rclone/issues/1907 12485 - https://github.com/rclone/rclone/issues/1834 12486 12487 #### Risk of throttling #### 12488 12489 Future iterations of the cache backend will make use of the pooling functionality 12490 of the cloud provider to synchronize and at the same time make writing through it 12491 more tolerant to failures. 12492 12493 There are a couple of enhancements in track to add these but in the meantime 12494 there is a valid concern that the expiring cache listings can lead to cloud provider 12495 throttles or bans due to repeated queries on it for very large mounts. 12496 12497 Some recommendations: 12498 - don't use a very small interval for entry informations (`--cache-info-age`) 12499 - while writes aren't yet optimised, you can still write through `cache` which gives you the advantage 12500 of adding the file in the cache at the same time if configured to do so. 12501 12502 Future enhancements: 12503 12504 - https://github.com/rclone/rclone/issues/1937 12505 - https://github.com/rclone/rclone/issues/1936 12506 12507 #### cache and crypt #### 12508 12509 One common scenario is to keep your data encrypted in the cloud provider 12510 using the `crypt` remote. `crypt` uses a similar technique to wrap around 12511 an existing remote and handles this translation in a seamless way. 12512 12513 There is an issue with wrapping the remotes in this order: 12514 <span style="color:red">**cloud remote** -> **crypt** -> **cache**</span> 12515 12516 During testing, I experienced a lot of bans with the remotes in this order. 12517 I suspect it might be related to how crypt opens files on the cloud provider 12518 which makes it think we're downloading the full file instead of small chunks. 12519 Organizing the remotes in this order yields better results: 12520 <span style="color:green">**cloud remote** -> **cache** -> **crypt**</span> 12521 12522 #### absolute remote paths #### 12523 12524 `cache` can not differentiate between relative and absolute paths for the wrapped remote. 12525 Any path given in the `remote` config setting and on the command line will be passed to 12526 the wrapped remote as is, but for storing the chunks on disk the path will be made 12527 relative by removing any leading `/` character. 12528 12529 This behavior is irrelevant for most backend types, but there are backends where a leading `/` 12530 changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are 12531 relative to the root of the SSH server and paths without are relative to the user home directory. 12532 As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent 12533 a different directory on the SSH server. 12534 12535 ### Cache and Remote Control (--rc) ### 12536 Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points: 12537 By default, the listener is disabled if you do not add the flag. 12538 12539 ### rc cache/expire 12540 Purge a remote from the cache backend. Supports either a directory or a file. 12541 It supports both encrypted and unencrypted file names if cache is wrapped by crypt. 12542 12543 Params: 12544 - **remote** = path to remote **(required)** 12545 - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_ 12546 12547 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/cache/cache.go then run make backenddocs --> 12548 ### Standard Options 12549 12550 Here are the standard options specific to cache (Cache a remote). 12551 12552 #### --cache-remote 12553 12554 Remote to cache. 12555 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 12556 "myremote:bucket" or maybe "myremote:" (not recommended). 12557 12558 - Config: remote 12559 - Env Var: RCLONE_CACHE_REMOTE 12560 - Type: string 12561 - Default: "" 12562 12563 #### --cache-plex-url 12564 12565 The URL of the Plex server 12566 12567 - Config: plex_url 12568 - Env Var: RCLONE_CACHE_PLEX_URL 12569 - Type: string 12570 - Default: "" 12571 12572 #### --cache-plex-username 12573 12574 The username of the Plex user 12575 12576 - Config: plex_username 12577 - Env Var: RCLONE_CACHE_PLEX_USERNAME 12578 - Type: string 12579 - Default: "" 12580 12581 #### --cache-plex-password 12582 12583 The password of the Plex user 12584 12585 - Config: plex_password 12586 - Env Var: RCLONE_CACHE_PLEX_PASSWORD 12587 - Type: string 12588 - Default: "" 12589 12590 #### --cache-chunk-size 12591 12592 The size of a chunk (partial file data). 12593 12594 Use lower numbers for slower connections. If the chunk size is 12595 changed, any downloaded chunks will be invalid and cache-chunk-path 12596 will need to be cleared or unexpected EOF errors will occur. 12597 12598 - Config: chunk_size 12599 - Env Var: RCLONE_CACHE_CHUNK_SIZE 12600 - Type: SizeSuffix 12601 - Default: 5M 12602 - Examples: 12603 - "1m" 12604 - 1MB 12605 - "5M" 12606 - 5 MB 12607 - "10M" 12608 - 10 MB 12609 12610 #### --cache-info-age 12611 12612 How long to cache file structure information (directory listings, file size, times etc). 12613 If all write operations are done through the cache then you can safely make 12614 this value very large as the cache store will also be updated in real time. 12615 12616 - Config: info_age 12617 - Env Var: RCLONE_CACHE_INFO_AGE 12618 - Type: Duration 12619 - Default: 6h0m0s 12620 - Examples: 12621 - "1h" 12622 - 1 hour 12623 - "24h" 12624 - 24 hours 12625 - "48h" 12626 - 48 hours 12627 12628 #### --cache-chunk-total-size 12629 12630 The total size that the chunks can take up on the local disk. 12631 12632 If the cache exceeds this value then it will start to delete the 12633 oldest chunks until it goes under this value. 12634 12635 - Config: chunk_total_size 12636 - Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE 12637 - Type: SizeSuffix 12638 - Default: 10G 12639 - Examples: 12640 - "500M" 12641 - 500 MB 12642 - "1G" 12643 - 1 GB 12644 - "10G" 12645 - 10 GB 12646 12647 ### Advanced Options 12648 12649 Here are the advanced options specific to cache (Cache a remote). 12650 12651 #### --cache-plex-token 12652 12653 The plex token for authentication - auto set normally 12654 12655 - Config: plex_token 12656 - Env Var: RCLONE_CACHE_PLEX_TOKEN 12657 - Type: string 12658 - Default: "" 12659 12660 #### --cache-plex-insecure 12661 12662 Skip all certificate verifications when connecting to the Plex server 12663 12664 - Config: plex_insecure 12665 - Env Var: RCLONE_CACHE_PLEX_INSECURE 12666 - Type: string 12667 - Default: "" 12668 12669 #### --cache-db-path 12670 12671 Directory to store file structure metadata DB. 12672 The remote name is used as the DB file name. 12673 12674 - Config: db_path 12675 - Env Var: RCLONE_CACHE_DB_PATH 12676 - Type: string 12677 - Default: "$HOME/.cache/rclone/cache-backend" 12678 12679 #### --cache-chunk-path 12680 12681 Directory to cache chunk files. 12682 12683 Path to where partial file data (chunks) are stored locally. The remote 12684 name is appended to the final path. 12685 12686 This config follows the "--cache-db-path". If you specify a custom 12687 location for "--cache-db-path" and don't specify one for "--cache-chunk-path" 12688 then "--cache-chunk-path" will use the same path as "--cache-db-path". 12689 12690 - Config: chunk_path 12691 - Env Var: RCLONE_CACHE_CHUNK_PATH 12692 - Type: string 12693 - Default: "$HOME/.cache/rclone/cache-backend" 12694 12695 #### --cache-db-purge 12696 12697 Clear all the cached data for this remote on start. 12698 12699 - Config: db_purge 12700 - Env Var: RCLONE_CACHE_DB_PURGE 12701 - Type: bool 12702 - Default: false 12703 12704 #### --cache-chunk-clean-interval 12705 12706 How often should the cache perform cleanups of the chunk storage. 12707 The default value should be ok for most people. If you find that the 12708 cache goes over "cache-chunk-total-size" too often then try to lower 12709 this value to force it to perform cleanups more often. 12710 12711 - Config: chunk_clean_interval 12712 - Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL 12713 - Type: Duration 12714 - Default: 1m0s 12715 12716 #### --cache-read-retries 12717 12718 How many times to retry a read from a cache storage. 12719 12720 Since reading from a cache stream is independent from downloading file 12721 data, readers can get to a point where there's no more data in the 12722 cache. Most of the times this can indicate a connectivity issue if 12723 cache isn't able to provide file data anymore. 12724 12725 For really slow connections, increase this to a point where the stream is 12726 able to provide data but your experience will be very stuttering. 12727 12728 - Config: read_retries 12729 - Env Var: RCLONE_CACHE_READ_RETRIES 12730 - Type: int 12731 - Default: 10 12732 12733 #### --cache-workers 12734 12735 How many workers should run in parallel to download chunks. 12736 12737 Higher values will mean more parallel processing (better CPU needed) 12738 and more concurrent requests on the cloud provider. This impacts 12739 several aspects like the cloud provider API limits, more stress on the 12740 hardware that rclone runs on but it also means that streams will be 12741 more fluid and data will be available much more faster to readers. 12742 12743 **Note**: If the optional Plex integration is enabled then this 12744 setting will adapt to the type of reading performed and the value 12745 specified here will be used as a maximum number of workers to use. 12746 12747 - Config: workers 12748 - Env Var: RCLONE_CACHE_WORKERS 12749 - Type: int 12750 - Default: 4 12751 12752 #### --cache-chunk-no-memory 12753 12754 Disable the in-memory cache for storing chunks during streaming. 12755 12756 By default, cache will keep file data during streaming in RAM as well 12757 to provide it to readers as fast as possible. 12758 12759 This transient data is evicted as soon as it is read and the number of 12760 chunks stored doesn't exceed the number of workers. However, depending 12761 on other settings like "cache-chunk-size" and "cache-workers" this footprint 12762 can increase if there are parallel streams too (multiple files being read 12763 at the same time). 12764 12765 If the hardware permits it, use this feature to provide an overall better 12766 performance during streaming but it can also be disabled if RAM is not 12767 available on the local machine. 12768 12769 - Config: chunk_no_memory 12770 - Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY 12771 - Type: bool 12772 - Default: false 12773 12774 #### --cache-rps 12775 12776 Limits the number of requests per second to the source FS (-1 to disable) 12777 12778 This setting places a hard limit on the number of requests per second 12779 that cache will be doing to the cloud provider remote and try to 12780 respect that value by setting waits between reads. 12781 12782 If you find that you're getting banned or limited on the cloud 12783 provider through cache and know that a smaller number of requests per 12784 second will allow you to work with it then you can use this setting 12785 for that. 12786 12787 A good balance of all the other settings should make this setting 12788 useless but it is available to set for more special cases. 12789 12790 **NOTE**: This will limit the number of requests during streams but 12791 other API calls to the cloud provider like directory listings will 12792 still pass. 12793 12794 - Config: rps 12795 - Env Var: RCLONE_CACHE_RPS 12796 - Type: int 12797 - Default: -1 12798 12799 #### --cache-writes 12800 12801 Cache file data on writes through the FS 12802 12803 If you need to read files immediately after you upload them through 12804 cache you can enable this flag to have their data stored in the 12805 cache store at the same time during upload. 12806 12807 - Config: writes 12808 - Env Var: RCLONE_CACHE_WRITES 12809 - Type: bool 12810 - Default: false 12811 12812 #### --cache-tmp-upload-path 12813 12814 Directory to keep temporary files until they are uploaded. 12815 12816 This is the path where cache will use as a temporary storage for new 12817 files that need to be uploaded to the cloud provider. 12818 12819 Specifying a value will enable this feature. Without it, it is 12820 completely disabled and files will be uploaded directly to the cloud 12821 provider 12822 12823 - Config: tmp_upload_path 12824 - Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH 12825 - Type: string 12826 - Default: "" 12827 12828 #### --cache-tmp-wait-time 12829 12830 How long should files be stored in local cache before being uploaded 12831 12832 This is the duration that a file must wait in the temporary location 12833 _cache-tmp-upload-path_ before it is selected for upload. 12834 12835 Note that only one file is uploaded at a time and it can take longer 12836 to start the upload if a queue formed for this purpose. 12837 12838 - Config: tmp_wait_time 12839 - Env Var: RCLONE_CACHE_TMP_WAIT_TIME 12840 - Type: Duration 12841 - Default: 15s 12842 12843 #### --cache-db-wait-time 12844 12845 How long to wait for the DB to be available - 0 is unlimited 12846 12847 Only one process can have the DB open at any one time, so rclone waits 12848 for this duration for the DB to become available before it gives an 12849 error. 12850 12851 If you set it to 0 then it will wait forever. 12852 12853 - Config: db_wait_time 12854 - Env Var: RCLONE_CACHE_DB_WAIT_TIME 12855 - Type: Duration 12856 - Default: 1s 12857 12858 <!--- autogenerated options stop --> 12859 12860 Chunker (BETA) 12861 ---------------------------------------- 12862 12863 The `chunker` overlay transparently splits large files into smaller chunks 12864 during upload to wrapped remote and transparently assembles them back 12865 when the file is downloaded. This allows to effectively overcome size limits 12866 imposed by storage providers. 12867 12868 To use it, first set up the underlying remote following the configuration 12869 instructions for that remote. You can also use a local pathname instead of 12870 a remote. 12871 12872 First check your chosen remote is working - we'll call it `remote:path` here. 12873 Note that anything inside `remote:path` will be chunked and anything outside 12874 won't. This means that if you are using a bucket based remote (eg S3, B2, swift) 12875 then you should probably put the bucket in the remote `s3:bucket`. 12876 12877 Now configure `chunker` using `rclone config`. We will call this one `overlay` 12878 to separate it from the `remote` itself. 12879 12880 ``` 12881 No remotes found - make a new one 12882 n) New remote 12883 s) Set configuration password 12884 q) Quit config 12885 n/s/q> n 12886 name> overlay 12887 Type of storage to configure. 12888 Choose a number from below, or type in your own value 12889 [snip] 12890 XX / Transparently chunk/split large files 12891 \ "chunker" 12892 [snip] 12893 Storage> chunker 12894 Remote to chunk/unchunk. 12895 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 12896 "myremote:bucket" or maybe "myremote:" (not recommended). 12897 Enter a string value. Press Enter for the default (""). 12898 remote> remote:path 12899 Files larger than chunk size will be split in chunks. 12900 Enter a size with suffix k,M,G,T. Press Enter for the default ("2G"). 12901 chunk_size> 100M 12902 Choose how chunker handles hash sums. All modes but "none" require metadata. 12903 Enter a string value. Press Enter for the default ("md5"). 12904 Choose a number from below, or type in your own value 12905 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise 12906 \ "none" 12907 2 / MD5 for composite files 12908 \ "md5" 12909 3 / SHA1 for composite files 12910 \ "sha1" 12911 4 / MD5 for all files 12912 \ "md5all" 12913 5 / SHA1 for all files 12914 \ "sha1all" 12915 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported 12916 \ "md5quick" 12917 7 / Similar to "md5quick" but prefers SHA1 over MD5 12918 \ "sha1quick" 12919 hash_type> md5 12920 Edit advanced config? (y/n) 12921 y) Yes 12922 n) No 12923 y/n> n 12924 Remote config 12925 -------------------- 12926 [overlay] 12927 type = chunker 12928 remote = remote:bucket 12929 chunk_size = 100M 12930 hash_type = md5 12931 -------------------- 12932 y) Yes this is OK 12933 e) Edit this remote 12934 d) Delete this remote 12935 y/e/d> y 12936 ``` 12937 12938 ### Specifying the remote 12939 12940 In normal use, make sure the remote has a `:` in. If you specify the remote 12941 without a `:` then rclone will use a local directory of that name. 12942 So if you use a remote of `/path/to/secret/files` then rclone will 12943 chunk stuff in that directory. If you use a remote of `name` then rclone 12944 will put files in a directory called `name` in the current directory. 12945 12946 12947 ### Chunking 12948 12949 When rclone starts a file upload, chunker checks the file size. If it 12950 doesn't exceed the configured chunk size, chunker will just pass the file 12951 to the wrapped remote. If a file is large, chunker will transparently cut 12952 data in pieces with temporary names and stream them one by one, on the fly. 12953 Each data chunk will contain the specified number of bytes, except for the 12954 last one which may have less data. If file size is unknown in advance 12955 (this is called a streaming upload), chunker will internally create 12956 a temporary copy, record its size and repeat the above process. 12957 12958 When upload completes, temporary chunk files are finally renamed. 12959 This scheme guarantees that operations can be run in parallel and look 12960 from outside as atomic. 12961 A similar method with hidden temporary chunks is used for other operations 12962 (copy/move/rename etc). If an operation fails, hidden chunks are normally 12963 destroyed, and the target composite file stays intact. 12964 12965 When a composite file download is requested, chunker transparently 12966 assembles it by concatenating data chunks in order. As the split is trivial 12967 one could even manually concatenate data chunks together to obtain the 12968 original content. 12969 12970 When the `list` rclone command scans a directory on wrapped remote, 12971 the potential chunk files are accounted for, grouped and assembled into 12972 composite directory entries. Any temporary chunks are hidden. 12973 12974 List and other commands can sometimes come across composite files with 12975 missing or invalid chunks, eg. shadowed by like-named directory or 12976 another file. This usually means that wrapped file system has been directly 12977 tampered with or damaged. If chunker detects a missing chunk it will 12978 by default print warning, skip the whole incomplete group of chunks but 12979 proceed with current command. 12980 You can set the `--chunker-fail-hard` flag to have commands abort with 12981 error message in such cases. 12982 12983 12984 #### Chunk names 12985 12986 The default chunk name format is `*.rclone_chunk.###`, hence by default 12987 chunk names are `BIG_FILE_NAME.rclone_chunk.001`, 12988 `BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format 12989 using the `name_format` configuration file option. The format uses asterisk 12990 `*` as a placeholder for the base file name and one or more consecutive 12991 hash characters `#` as a placeholder for sequential chunk number. 12992 There must be one and only one asterisk. The number of consecutive hash 12993 characters defines the minimum length of a string representing a chunk number. 12994 If decimal chunk number has less digits than the number of hashes, it is 12995 left-padded by zeros. If the decimal string is longer, it is left intact. 12996 By default numbering starts from 1 but there is another option that allows 12997 user to start from 0, eg. for compatibility with legacy software. 12998 12999 For example, if name format is `big_*-##.part` and original file name is 13000 `data.txt` and numbering starts from 0, then the first chunk will be named 13001 `big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part` 13002 and the 302nd chunk will become `big_data.txt-301.part`. 13003 13004 Note that `list` assembles composite directory entries only when chunk names 13005 match the configured format and treats non-conforming file names as normal 13006 non-chunked files. 13007 13008 13009 ### Metadata 13010 13011 Besides data chunks chunker will by default create metadata object for 13012 a composite file. The object is named after the original file. 13013 Chunker allows user to disable metadata completely (the `none` format). 13014 Note that metadata is normally not created for files smaller than the 13015 configured chunk size. This may change in future rclone releases. 13016 13017 #### Simple JSON metadata format 13018 13019 This is the default format. It supports hash sums and chunk validation 13020 for composite files. Meta objects carry the following fields: 13021 13022 - `ver` - version of format, currently `1` 13023 - `size` - total size of composite file 13024 - `nchunks` - number of data chunks in file 13025 - `md5` - MD5 hashsum of composite file (if present) 13026 - `sha1` - SHA1 hashsum (if present) 13027 13028 There is no field for composite file name as it's simply equal to the name 13029 of meta object on the wrapped remote. Please refer to respective sections 13030 for details on hashsums and modified time handling. 13031 13032 #### No metadata 13033 13034 You can disable meta objects by setting the meta format option to `none`. 13035 In this mode chunker will scan directory for all files that follow 13036 configured chunk name format, group them by detecting chunks with the same 13037 base name and show group names as virtual composite files. 13038 This method is more prone to missing chunk errors (especially missing 13039 last chunk) than format with metadata enabled. 13040 13041 13042 ### Hashsums 13043 13044 Chunker supports hashsums only when a compatible metadata is present. 13045 Hence, if you choose metadata format of `none`, chunker will report hashsum 13046 as `UNSUPPORTED`. 13047 13048 Please note that by default metadata is stored only for composite files. 13049 If a file is smaller than configured chunk size, chunker will transparently 13050 redirect hash requests to wrapped remote, so support depends on that. 13051 You will see the empty string as a hashsum of requested type for small 13052 files if the wrapped remote doesn't support it. 13053 13054 Many storage backends support MD5 and SHA1 hash types, so does chunker. 13055 With chunker you can choose one or another but not both. 13056 MD5 is set by default as the most supported type. 13057 Since chunker keeps hashes for composite files and falls back to the 13058 wrapped remote hash for non-chunked ones, we advise you to choose the same 13059 hash type as supported by wrapped remote so that your file listings 13060 look coherent. 13061 13062 If your storage backend does not support MD5 or SHA1 but you need consistent 13063 file hashing, configure chunker with `md5all` or `sha1all`. These two modes 13064 guarantee given hash for all files. If wrapped remote doesn't support it, 13065 chunker will then add metadata to all files, even small. However, this can 13066 double the amount of small files in storage and incur additional service charges. 13067 You can even use chunker to force md5/sha1 support in any other remote 13068 at expence of sidecar meta objects by setting eg. `chunk_type=sha1all` 13069 to force hashsums and `chunk_size=1P` to effectively disable chunking. 13070 13071 Normally, when a file is copied to chunker controlled remote, chunker 13072 will ask the file source for compatible file hash and revert to on-the-fly 13073 calculation if none is found. This involves some CPU overhead but provides 13074 a guarantee that given hashsum is available. Also, chunker will reject 13075 a server-side copy or move operation if source and destination hashsum 13076 types are different resulting in the extra network bandwidth, too. 13077 In some rare cases this may be undesired, so chunker provides two optional 13078 choices: `sha1quick` and `md5quick`. If the source does not support primary 13079 hash type and the quick mode is enabled, chunker will try to fall back to 13080 the secondary type. This will save CPU and bandwidth but can result in empty 13081 hashsums at destination. Beware of consequences: the `sync` command will 13082 revert (sometimes silently) to time/size comparison if compatible hashsums 13083 between source and target are not found. 13084 13085 13086 ### Modified time 13087 13088 Chunker stores modification times using the wrapped remote so support 13089 depends on that. For a small non-chunked file the chunker overlay simply 13090 manipulates modification time of the wrapped remote file. 13091 For a composite file with metadata chunker will get and set 13092 modification time of the metadata object on the wrapped remote. 13093 If file is chunked but metadata format is `none` then chunker will 13094 use modification time of the first data chunk. 13095 13096 13097 ### Migrations 13098 13099 The idiomatic way to migrate to a different chunk size, hash type or 13100 chunk naming scheme is to: 13101 13102 - Collect all your chunked files under a directory and have your 13103 chunker remote point to it. 13104 - Create another directory (most probably on the same cloud storage) 13105 and configure a new remote with desired metadata format, 13106 hash type, chunk naming etc. 13107 - Now run `rclone sync oldchunks: newchunks:` and all your data 13108 will be transparently converted in transfer. 13109 This may take some time, yet chunker will try server-side 13110 copy if possible. 13111 - After checking data integrity you may remove configuration section 13112 of the old remote. 13113 13114 If rclone gets killed during a long operation on a big composite file, 13115 hidden temporary chunks may stay in the directory. They will not be 13116 shown by the `list` command but will eat up your account quota. 13117 Please note that the `deletefile` command deletes only active 13118 chunks of a file. As a workaround, you can use remote of the wrapped 13119 file system to see them. 13120 An easy way to get rid of hidden garbage is to copy littered directory 13121 somewhere using the chunker remote and purge the original directory. 13122 The `copy` command will copy only active chunks while the `purge` will 13123 remove everything including garbage. 13124 13125 13126 ### Caveats and Limitations 13127 13128 Chunker requires wrapped remote to support server side `move` (or `copy` + 13129 `delete`) operations, otherwise it will explicitly refuse to start. 13130 This is because it internally renames temporary chunk files to their final 13131 names when an operation completes successfully. 13132 13133 Chunker encodes chunk number in file name, so with default `name_format` 13134 setting it adds 17 characters. Also chunker adds 7 characters of temporary 13135 suffix during operations. Many file systems limit base file name without path 13136 by 255 characters. Using rclone's crypt remote as a base file system limits 13137 file name by 143 characters. Thus, maximum name length is 231 for most files 13138 and 119 for chunker-over-crypt. A user in need can change name format to 13139 eg. `*.rcc##` and save 10 characters (provided at most 99 chunks per file). 13140 13141 Note that a move implemented using the copy-and-delete method may incur 13142 double charging with some cloud storage providers. 13143 13144 Chunker will not automatically rename existing chunks when you run 13145 `rclone config` on a live remote and change the chunk name format. 13146 Beware that in result of this some files which have been treated as chunks 13147 before the change can pop up in directory listings as normal files 13148 and vice versa. The same warning holds for the chunk size. 13149 If you desperately need to change critical chunking setings, you should 13150 run data migration as described above. 13151 13152 If wrapped remote is case insensitive, the chunker overlay will inherit 13153 that property (so you can't have a file called "Hello.doc" and "hello.doc" 13154 in the same directory). 13155 13156 13157 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/chunker/chunker.go then run make backenddocs --> 13158 ### Standard Options 13159 13160 Here are the standard options specific to chunker (Transparently chunk/split large files). 13161 13162 #### --chunker-remote 13163 13164 Remote to chunk/unchunk. 13165 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13166 "myremote:bucket" or maybe "myremote:" (not recommended). 13167 13168 - Config: remote 13169 - Env Var: RCLONE_CHUNKER_REMOTE 13170 - Type: string 13171 - Default: "" 13172 13173 #### --chunker-chunk-size 13174 13175 Files larger than chunk size will be split in chunks. 13176 13177 - Config: chunk_size 13178 - Env Var: RCLONE_CHUNKER_CHUNK_SIZE 13179 - Type: SizeSuffix 13180 - Default: 2G 13181 13182 #### --chunker-hash-type 13183 13184 Choose how chunker handles hash sums. All modes but "none" require metadata. 13185 13186 - Config: hash_type 13187 - Env Var: RCLONE_CHUNKER_HASH_TYPE 13188 - Type: string 13189 - Default: "md5" 13190 - Examples: 13191 - "none" 13192 - Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise 13193 - "md5" 13194 - MD5 for composite files 13195 - "sha1" 13196 - SHA1 for composite files 13197 - "md5all" 13198 - MD5 for all files 13199 - "sha1all" 13200 - SHA1 for all files 13201 - "md5quick" 13202 - Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported 13203 - "sha1quick" 13204 - Similar to "md5quick" but prefers SHA1 over MD5 13205 13206 ### Advanced Options 13207 13208 Here are the advanced options specific to chunker (Transparently chunk/split large files). 13209 13210 #### --chunker-name-format 13211 13212 String format of chunk file names. 13213 The two placeholders are: base file name (*) and chunk number (#...). 13214 There must be one and only one asterisk and one or more consecutive hash characters. 13215 If chunk number has less digits than the number of hashes, it is left-padded by zeros. 13216 If there are more digits in the number, they are left as is. 13217 Possible chunk files are ignored if their name does not match given format. 13218 13219 - Config: name_format 13220 - Env Var: RCLONE_CHUNKER_NAME_FORMAT 13221 - Type: string 13222 - Default: "*.rclone_chunk.###" 13223 13224 #### --chunker-start-from 13225 13226 Minimum valid chunk number. Usually 0 or 1. 13227 By default chunk numbers start from 1. 13228 13229 - Config: start_from 13230 - Env Var: RCLONE_CHUNKER_START_FROM 13231 - Type: int 13232 - Default: 1 13233 13234 #### --chunker-meta-format 13235 13236 Format of the metadata object or "none". By default "simplejson". 13237 Metadata is a small JSON file named after the composite file. 13238 13239 - Config: meta_format 13240 - Env Var: RCLONE_CHUNKER_META_FORMAT 13241 - Type: string 13242 - Default: "simplejson" 13243 - Examples: 13244 - "none" 13245 - Do not use metadata files at all. Requires hash type "none". 13246 - "simplejson" 13247 - Simple JSON supports hash sums and chunk validation. 13248 - It has the following fields: ver, size, nchunks, md5, sha1. 13249 13250 #### --chunker-fail-hard 13251 13252 Choose how chunker should handle files with missing or invalid chunks. 13253 13254 - Config: fail_hard 13255 - Env Var: RCLONE_CHUNKER_FAIL_HARD 13256 - Type: bool 13257 - Default: false 13258 - Examples: 13259 - "true" 13260 - Report errors and abort current command. 13261 - "false" 13262 - Warn user, skip incomplete file and proceed. 13263 13264 <!--- autogenerated options stop --> 13265 13266 ## Citrix ShareFile 13267 13268 [Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business. 13269 13270 The initial setup for Citrix ShareFile involves getting a token from 13271 Citrix ShareFile which you can in your browser. `rclone config` walks you 13272 through it. 13273 13274 Here is an example of how to make a remote called `remote`. First run: 13275 13276 rclone config 13277 13278 This will guide you through an interactive setup process: 13279 13280 ``` 13281 No remotes found - make a new one 13282 n) New remote 13283 s) Set configuration password 13284 q) Quit config 13285 n/s/q> n 13286 name> remote 13287 Type of storage to configure. 13288 Enter a string value. Press Enter for the default (""). 13289 Choose a number from below, or type in your own value 13290 XX / Citrix Sharefile 13291 \ "sharefile" 13292 Storage> sharefile 13293 ** See help for sharefile backend at: https://rclone.org/sharefile/ ** 13294 13295 ID of the root folder 13296 13297 Leave blank to access "Personal Folders". You can use one of the 13298 standard values here or any folder ID (long hex number ID). 13299 Enter a string value. Press Enter for the default (""). 13300 Choose a number from below, or type in your own value 13301 1 / Access the Personal Folders. (Default) 13302 \ "" 13303 2 / Access the Favorites folder. 13304 \ "favorites" 13305 3 / Access all the shared folders. 13306 \ "allshared" 13307 4 / Access all the individual connectors. 13308 \ "connectors" 13309 5 / Access the home, favorites, and shared folders as well as the connectors. 13310 \ "top" 13311 root_folder_id> 13312 Edit advanced config? (y/n) 13313 y) Yes 13314 n) No 13315 y/n> n 13316 Remote config 13317 Use auto config? 13318 * Say Y if not sure 13319 * Say N if you are working on a remote or headless machine 13320 y) Yes 13321 n) No 13322 y/n> y 13323 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX 13324 Log in and authorize rclone for access 13325 Waiting for code... 13326 Got code 13327 -------------------- 13328 [remote] 13329 type = sharefile 13330 endpoint = https://XXX.sharefile.com 13331 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} 13332 -------------------- 13333 y) Yes this is OK 13334 e) Edit this remote 13335 d) Delete this remote 13336 y/e/d> y 13337 ``` 13338 13339 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 13340 machine with no Internet browser available. 13341 13342 Note that rclone runs a webserver on your local machine to collect the 13343 token as returned from Citrix ShareFile. This only runs from the moment it opens 13344 your browser to the moment you get back the verification code. This 13345 is on `http://127.0.0.1:53682/` and this it may require you to unblock 13346 it temporarily if you are running a host firewall. 13347 13348 Once configured you can then use `rclone` like this, 13349 13350 List directories in top level of your ShareFile 13351 13352 rclone lsd remote: 13353 13354 List all the files in your ShareFile 13355 13356 rclone ls remote: 13357 13358 To copy a local directory to an ShareFile directory called backup 13359 13360 rclone copy /home/source remote:backup 13361 13362 Paths may be as deep as required, eg `remote:directory/subdirectory`. 13363 13364 ### Modified time and hashes ### 13365 13366 ShareFile allows modification times to be set on objects accurate to 1 13367 second. These will be used to detect whether objects need syncing or 13368 not. 13369 13370 ShareFile supports MD5 type hashes, so you can use the `--checksum` 13371 flag. 13372 13373 ### Transfers ### 13374 13375 For files above 128MB rclone will use a chunked transfer. Rclone will 13376 upload up to `--transfers` chunks at the same time (shared among all 13377 the multipart uploads). Chunks are buffered in memory and are 13378 normally 64MB so increasing `--transfers` will increase memory use. 13379 13380 ### Limitations ### 13381 13382 Note that ShareFile is case insensitive so you can't have a file called 13383 "Hello.doc" and one called "hello.doc". 13384 13385 ShareFile only supports filenames up to 256 characters in length. 13386 13387 #### Restricted filename characters 13388 13389 In addition to the [default restricted characters set](/overview/#restricted-characters) 13390 the following characters are also replaced: 13391 13392 | Character | Value | Replacement | 13393 | --------- |:-----:|:-----------:| 13394 | \\ | 0x5C | \ | 13395 | * | 0x2A | * | 13396 | < | 0x3C | < | 13397 | > | 0x3E | > | 13398 | ? | 0x3F | ? | 13399 | : | 0x3A | : | 13400 | \| | 0x7C | | | 13401 | " | 0x22 | " | 13402 13403 File names can also not start or end with the following characters. 13404 These only get replaced if they are first or last character in the 13405 name: 13406 13407 | Character | Value | Replacement | 13408 | --------- |:-----:|:-----------:| 13409 | SP | 0x20 | ␠ | 13410 | . | 0x2E | . | 13411 13412 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 13413 as they can't be used in JSON strings. 13414 13415 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/sharefile/sharefile.go then run make backenddocs --> 13416 ### Standard Options 13417 13418 Here are the standard options specific to sharefile (Citrix Sharefile). 13419 13420 #### --sharefile-root-folder-id 13421 13422 ID of the root folder 13423 13424 Leave blank to access "Personal Folders". You can use one of the 13425 standard values here or any folder ID (long hex number ID). 13426 13427 - Config: root_folder_id 13428 - Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID 13429 - Type: string 13430 - Default: "" 13431 - Examples: 13432 - "" 13433 - Access the Personal Folders. (Default) 13434 - "favorites" 13435 - Access the Favorites folder. 13436 - "allshared" 13437 - Access all the shared folders. 13438 - "connectors" 13439 - Access all the individual connectors. 13440 - "top" 13441 - Access the home, favorites, and shared folders as well as the connectors. 13442 13443 ### Advanced Options 13444 13445 Here are the advanced options specific to sharefile (Citrix Sharefile). 13446 13447 #### --sharefile-upload-cutoff 13448 13449 Cutoff for switching to multipart upload. 13450 13451 - Config: upload_cutoff 13452 - Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF 13453 - Type: SizeSuffix 13454 - Default: 128M 13455 13456 #### --sharefile-chunk-size 13457 13458 Upload chunk size. Must a power of 2 >= 256k. 13459 13460 Making this larger will improve performance, but note that each chunk 13461 is buffered in memory one per transfer. 13462 13463 Reducing this will reduce memory usage but decrease performance. 13464 13465 - Config: chunk_size 13466 - Env Var: RCLONE_SHAREFILE_CHUNK_SIZE 13467 - Type: SizeSuffix 13468 - Default: 64M 13469 13470 #### --sharefile-endpoint 13471 13472 Endpoint for API calls. 13473 13474 This is usually auto discovered as part of the oauth process, but can 13475 be set manually to something like: https://XXX.sharefile.com 13476 13477 13478 - Config: endpoint 13479 - Env Var: RCLONE_SHAREFILE_ENDPOINT 13480 - Type: string 13481 - Default: "" 13482 13483 #### --sharefile-encoding 13484 13485 This sets the encoding for the backend. 13486 13487 See: the [encoding section in the overview](/overview/#encoding) for more info. 13488 13489 - Config: encoding 13490 - Env Var: RCLONE_SHAREFILE_ENCODING 13491 - Type: MultiEncoder 13492 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot 13493 13494 <!--- autogenerated options stop --> 13495 13496 Crypt 13497 ---------------------------------------- 13498 13499 The `crypt` remote encrypts and decrypts another remote. 13500 13501 To use it first set up the underlying remote following the config 13502 instructions for that remote. You can also use a local pathname 13503 instead of a remote which will encrypt and decrypt from that directory 13504 which might be useful for encrypting onto a USB stick for example. 13505 13506 First check your chosen remote is working - we'll call it 13507 `remote:path` in these docs. Note that anything inside `remote:path` 13508 will be encrypted and anything outside won't. This means that if you 13509 are using a bucket based remote (eg S3, B2, swift) then you should 13510 probably put the bucket in the remote `s3:bucket`. If you just use 13511 `s3:` then rclone will make encrypted bucket names too (if using file 13512 name encryption) which may or may not be what you want. 13513 13514 Now configure `crypt` using `rclone config`. We will call this one 13515 `secret` to differentiate it from the `remote`. 13516 13517 ``` 13518 No remotes found - make a new one 13519 n) New remote 13520 s) Set configuration password 13521 q) Quit config 13522 n/s/q> n 13523 name> secret 13524 Type of storage to configure. 13525 Choose a number from below, or type in your own value 13526 [snip] 13527 XX / Encrypt/Decrypt a remote 13528 \ "crypt" 13529 [snip] 13530 Storage> crypt 13531 Remote to encrypt/decrypt. 13532 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13533 "myremote:bucket" or maybe "myremote:" (not recommended). 13534 remote> remote:path 13535 How to encrypt the filenames. 13536 Choose a number from below, or type in your own value 13537 1 / Don't encrypt the file names. Adds a ".bin" extension only. 13538 \ "off" 13539 2 / Encrypt the filenames see the docs for the details. 13540 \ "standard" 13541 3 / Very simple filename obfuscation. 13542 \ "obfuscate" 13543 filename_encryption> 2 13544 Option to either encrypt directory names or leave them intact. 13545 Choose a number from below, or type in your own value 13546 1 / Encrypt directory names. 13547 \ "true" 13548 2 / Don't encrypt directory names, leave them intact. 13549 \ "false" 13550 filename_encryption> 1 13551 Password or pass phrase for encryption. 13552 y) Yes type in my own password 13553 g) Generate random password 13554 y/g> y 13555 Enter the password: 13556 password: 13557 Confirm the password: 13558 password: 13559 Password or pass phrase for salt. Optional but recommended. 13560 Should be different to the previous password. 13561 y) Yes type in my own password 13562 g) Generate random password 13563 n) No leave this optional password blank 13564 y/g/n> g 13565 Password strength in bits. 13566 64 is just about memorable 13567 128 is secure 13568 1024 is the maximum 13569 Bits> 128 13570 Your password is: JAsJvRcgR-_veXNfy_sGmQ 13571 Use this password? 13572 y) Yes 13573 n) No 13574 y/n> y 13575 Remote config 13576 -------------------- 13577 [secret] 13578 remote = remote:path 13579 filename_encryption = standard 13580 password = *** ENCRYPTED *** 13581 password2 = *** ENCRYPTED *** 13582 -------------------- 13583 y) Yes this is OK 13584 e) Edit this remote 13585 d) Delete this remote 13586 y/e/d> y 13587 ``` 13588 13589 **Important** The password is stored in the config file is lightly 13590 obscured so it isn't immediately obvious what it is. It is in no way 13591 secure unless you use config file encryption. 13592 13593 A long passphrase is recommended, or you can use a random one. Note 13594 that if you reconfigure rclone with the same passwords/passphrases 13595 elsewhere it will be compatible - all the secrets used are derived 13596 from those two passwords/passphrases. 13597 13598 Note that rclone does not encrypt 13599 13600 * file length - this can be calcuated within 16 bytes 13601 * modification time - used for syncing 13602 13603 ## Specifying the remote ## 13604 13605 In normal use, make sure the remote has a `:` in. If you specify the 13606 remote without a `:` then rclone will use a local directory of that 13607 name. So if you use a remote of `/path/to/secret/files` then rclone 13608 will encrypt stuff to that directory. If you use a remote of `name` 13609 then rclone will put files in a directory called `name` in the current 13610 directory. 13611 13612 If you specify the remote as `remote:path/to/dir` then rclone will 13613 store encrypted files in `path/to/dir` on the remote. If you are using 13614 file name encryption, then when you save files to 13615 `secret:subdir/subfile` this will store them in the unencrypted path 13616 `path/to/dir` but the `subdir/subpath` bit will be encrypted. 13617 13618 Note that unless you want encrypted bucket names (which are difficult 13619 to manage because you won't know what directory they represent in web 13620 interfaces etc), you should probably specify a bucket, eg 13621 `remote:secretbucket` when using bucket based remotes such as S3, 13622 Swift, Hubic, B2, GCS. 13623 13624 ## Example ## 13625 13626 To test I made a little directory of files using "standard" file name 13627 encryption. 13628 13629 ``` 13630 plaintext/ 13631 ├── file0.txt 13632 ├── file1.txt 13633 └── subdir 13634 ├── file2.txt 13635 ├── file3.txt 13636 └── subsubdir 13637 └── file4.txt 13638 ``` 13639 13640 Copy these to the remote and list them back 13641 13642 ``` 13643 $ rclone -q copy plaintext secret: 13644 $ rclone -q ls secret: 13645 7 file1.txt 13646 6 file0.txt 13647 8 subdir/file2.txt 13648 10 subdir/subsubdir/file4.txt 13649 9 subdir/file3.txt 13650 ``` 13651 13652 Now see what that looked like when encrypted 13653 13654 ``` 13655 $ rclone -q ls remote:path 13656 55 hagjclgavj2mbiqm6u6cnjjqcg 13657 54 v05749mltvv1tf4onltun46gls 13658 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 13659 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 13660 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 13661 ``` 13662 13663 Note that this retains the directory structure which means you can do this 13664 13665 ``` 13666 $ rclone -q ls secret:subdir 13667 8 file2.txt 13668 9 file3.txt 13669 10 subsubdir/file4.txt 13670 ``` 13671 13672 If don't use file name encryption then the remote will look like this 13673 - note the `.bin` extensions added to prevent the cloud provider 13674 attempting to interpret the data. 13675 13676 ``` 13677 $ rclone -q ls remote:path 13678 54 file0.txt.bin 13679 57 subdir/file3.txt.bin 13680 56 subdir/file2.txt.bin 13681 58 subdir/subsubdir/file4.txt.bin 13682 55 file1.txt.bin 13683 ``` 13684 13685 ### File name encryption modes ### 13686 13687 Here are some of the features of the file name encryption modes 13688 13689 Off 13690 13691 * doesn't hide file names or directory structure 13692 * allows for longer file names (~246 characters) 13693 * can use sub paths and copy single files 13694 13695 Standard 13696 13697 * file names encrypted 13698 * file names can't be as long (~143 characters) 13699 * can use sub paths and copy single files 13700 * directory structure visible 13701 * identical files names will have identical uploaded names 13702 * can use shortcuts to shorten the directory recursion 13703 13704 Obfuscation 13705 13706 This is a simple "rotate" of the filename, with each file having a rot 13707 distance based on the filename. We store the distance at the beginning 13708 of the filename. So a file called "hello" may become "53.jgnnq" 13709 13710 This is not a strong encryption of filenames, but it may stop automated 13711 scanning tools from picking up on filename patterns. As such it's an 13712 intermediate between "off" and "standard". The advantage is that it 13713 allows for longer path segment names. 13714 13715 There is a possibility with some unicode based filenames that the 13716 obfuscation is weak and may map lower case characters to upper case 13717 equivalents. You can not rely on this for strong protection. 13718 13719 * file names very lightly obfuscated 13720 * file names can be longer than standard encryption 13721 * can use sub paths and copy single files 13722 * directory structure visible 13723 * identical files names will have identical uploaded names 13724 13725 Cloud storage systems have various limits on file name length and 13726 total path length which you are more likely to hit using "Standard" 13727 file name encryption. If you keep your file names to below 156 13728 characters in length then you should be OK on all providers. 13729 13730 There may be an even more secure file name encryption mode in the 13731 future which will address the long file name problem. 13732 13733 ### Directory name encryption ### 13734 Crypt offers the option of encrypting dir names or leaving them intact. 13735 There are two options: 13736 13737 True 13738 13739 Encrypts the whole file path including directory names 13740 Example: 13741 `1/12/123.txt` is encrypted to 13742 `p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0` 13743 13744 False 13745 13746 Only encrypts file names, skips directory names 13747 Example: 13748 `1/12/123.txt` is encrypted to 13749 `1/12/qgm4avr35m5loi1th53ato71v0` 13750 13751 13752 ### Modified time and hashes ### 13753 13754 Crypt stores modification times using the underlying remote so support 13755 depends on that. 13756 13757 Hashes are not stored for crypt. However the data integrity is 13758 protected by an extremely strong crypto authenticator. 13759 13760 Note that you should use the `rclone cryptcheck` command to check the 13761 integrity of a crypted remote instead of `rclone check` which can't 13762 check the checksums properly. 13763 13764 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/crypt/crypt.go then run make backenddocs --> 13765 ### Standard Options 13766 13767 Here are the standard options specific to crypt (Encrypt/Decrypt a remote). 13768 13769 #### --crypt-remote 13770 13771 Remote to encrypt/decrypt. 13772 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13773 "myremote:bucket" or maybe "myremote:" (not recommended). 13774 13775 - Config: remote 13776 - Env Var: RCLONE_CRYPT_REMOTE 13777 - Type: string 13778 - Default: "" 13779 13780 #### --crypt-filename-encryption 13781 13782 How to encrypt the filenames. 13783 13784 - Config: filename_encryption 13785 - Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION 13786 - Type: string 13787 - Default: "standard" 13788 - Examples: 13789 - "standard" 13790 - Encrypt the filenames see the docs for the details. 13791 - "obfuscate" 13792 - Very simple filename obfuscation. 13793 - "off" 13794 - Don't encrypt the file names. Adds a ".bin" extension only. 13795 13796 #### --crypt-directory-name-encryption 13797 13798 Option to either encrypt directory names or leave them intact. 13799 13800 - Config: directory_name_encryption 13801 - Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION 13802 - Type: bool 13803 - Default: true 13804 - Examples: 13805 - "true" 13806 - Encrypt directory names. 13807 - "false" 13808 - Don't encrypt directory names, leave them intact. 13809 13810 #### --crypt-password 13811 13812 Password or pass phrase for encryption. 13813 13814 - Config: password 13815 - Env Var: RCLONE_CRYPT_PASSWORD 13816 - Type: string 13817 - Default: "" 13818 13819 #### --crypt-password2 13820 13821 Password or pass phrase for salt. Optional but recommended. 13822 Should be different to the previous password. 13823 13824 - Config: password2 13825 - Env Var: RCLONE_CRYPT_PASSWORD2 13826 - Type: string 13827 - Default: "" 13828 13829 ### Advanced Options 13830 13831 Here are the advanced options specific to crypt (Encrypt/Decrypt a remote). 13832 13833 #### --crypt-show-mapping 13834 13835 For all files listed show how the names encrypt. 13836 13837 If this flag is set then for each file that the remote is asked to 13838 list, it will log (at level INFO) a line stating the decrypted file 13839 name and the encrypted file name. 13840 13841 This is so you can work out which encrypted names are which decrypted 13842 names just in case you need to do something with the encrypted file 13843 names, or for debugging purposes. 13844 13845 - Config: show_mapping 13846 - Env Var: RCLONE_CRYPT_SHOW_MAPPING 13847 - Type: bool 13848 - Default: false 13849 13850 <!--- autogenerated options stop --> 13851 13852 ## Backing up a crypted remote ## 13853 13854 If you wish to backup a crypted remote, it it recommended that you use 13855 `rclone sync` on the encrypted files, and make sure the passwords are 13856 the same in the new encrypted remote. 13857 13858 This will have the following advantages 13859 13860 * `rclone sync` will check the checksums while copying 13861 * you can use `rclone check` between the encrypted remotes 13862 * you don't decrypt and encrypt unnecessarily 13863 13864 For example, let's say you have your original remote at `remote:` with 13865 the encrypted version at `eremote:` with path `remote:crypt`. You 13866 would then set up the new remote `remote2:` and then the encrypted 13867 version `eremote2:` with path `remote2:crypt` using the same passwords 13868 as `eremote:`. 13869 13870 To sync the two remotes you would do 13871 13872 rclone sync remote:crypt remote2:crypt 13873 13874 And to check the integrity you would do 13875 13876 rclone check remote:crypt remote2:crypt 13877 13878 ## File formats ## 13879 13880 ### File encryption ### 13881 13882 Files are encrypted 1:1 source file to destination object. The file 13883 has a header and is divided into chunks. 13884 13885 #### Header #### 13886 13887 * 8 bytes magic string `RCLONE\x00\x00` 13888 * 24 bytes Nonce (IV) 13889 13890 The initial nonce is generated from the operating systems crypto 13891 strong random number generator. The nonce is incremented for each 13892 chunk read making sure each nonce is unique for each block written. 13893 The chance of a nonce being re-used is minuscule. If you wrote an 13894 exabyte of data (10¹⁸ bytes) you would have a probability of 13895 approximately 2×10⁻³² of re-using a nonce. 13896 13897 #### Chunk #### 13898 13899 Each chunk will contain 64kB of data, except for the last one which 13900 may have less data. The data chunk is in standard NACL secretbox 13901 format. Secretbox uses XSalsa20 and Poly1305 to encrypt and 13902 authenticate messages. 13903 13904 Each chunk contains: 13905 13906 * 16 Bytes of Poly1305 authenticator 13907 * 1 - 65536 bytes XSalsa20 encrypted data 13908 13909 64k chunk size was chosen as the best performing chunk size (the 13910 authenticator takes too much time below this and the performance drops 13911 off due to cache effects above this). Note that these chunks are 13912 buffered in memory so they can't be too big. 13913 13914 This uses a 32 byte (256 bit key) key derived from the user password. 13915 13916 #### Examples #### 13917 13918 1 byte file will encrypt to 13919 13920 * 32 bytes header 13921 * 17 bytes data chunk 13922 13923 49 bytes total 13924 13925 1MB (1048576 bytes) file will encrypt to 13926 13927 * 32 bytes header 13928 * 16 chunks of 65568 bytes 13929 13930 1049120 bytes total (a 0.05% overhead). This is the overhead for big 13931 files. 13932 13933 ### Name encryption ### 13934 13935 File names are encrypted segment by segment - the path is broken up 13936 into `/` separated strings and these are encrypted individually. 13937 13938 File segments are padded using using PKCS#7 to a multiple of 16 bytes 13939 before encryption. 13940 13941 They are then encrypted with EME using AES with 256 bit key. EME 13942 (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003 13943 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway. 13944 13945 This makes for deterministic encryption which is what we want - the 13946 same filename must encrypt to the same thing otherwise we can't find 13947 it on the cloud storage system. 13948 13949 This means that 13950 13951 * filenames with the same name will encrypt the same 13952 * filenames which start the same won't have a common prefix 13953 13954 This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of 13955 which are derived from the user password. 13956 13957 After encryption they are written out using a modified version of 13958 standard `base32` encoding as described in RFC4648. The standard 13959 encoding is modified in two ways: 13960 13961 * it becomes lower case (no-one likes upper case filenames!) 13962 * we strip the padding character `=` 13963 13964 `base32` is used rather than the more efficient `base64` so rclone can be 13965 used on case insensitive remotes (eg Windows, Amazon Drive). 13966 13967 ### Key derivation ### 13968 13969 Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an 13970 optional user supplied salt (password2) to derive the 32+32+16 = 80 13971 bytes of key material required. If the user doesn't supply a salt 13972 then rclone uses an internal one. 13973 13974 `scrypt` makes it impractical to mount a dictionary attack on rclone 13975 encrypted data. For full protection against this you should always use 13976 a salt. 13977 13978 Dropbox 13979 --------------------------------- 13980 13981 Paths are specified as `remote:path` 13982 13983 Dropbox paths may be as deep as required, eg 13984 `remote:directory/subdirectory`. 13985 13986 The initial setup for dropbox involves getting a token from Dropbox 13987 which you need to do in your browser. `rclone config` walks you 13988 through it. 13989 13990 Here is an example of how to make a remote called `remote`. First run: 13991 13992 rclone config 13993 13994 This will guide you through an interactive setup process: 13995 13996 ``` 13997 n) New remote 13998 d) Delete remote 13999 q) Quit config 14000 e/n/d/q> n 14001 name> remote 14002 Type of storage to configure. 14003 Choose a number from below, or type in your own value 14004 [snip] 14005 XX / Dropbox 14006 \ "dropbox" 14007 [snip] 14008 Storage> dropbox 14009 Dropbox App Key - leave blank normally. 14010 app_key> 14011 Dropbox App Secret - leave blank normally. 14012 app_secret> 14013 Remote config 14014 Please visit: 14015 https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code 14016 Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX 14017 -------------------- 14018 [remote] 14019 app_key = 14020 app_secret = 14021 token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 14022 -------------------- 14023 y) Yes this is OK 14024 e) Edit this remote 14025 d) Delete this remote 14026 y/e/d> y 14027 ``` 14028 14029 You can then use it like this, 14030 14031 List directories in top level of your dropbox 14032 14033 rclone lsd remote: 14034 14035 List all the files in your dropbox 14036 14037 rclone ls remote: 14038 14039 To copy a local directory to a dropbox directory called backup 14040 14041 rclone copy /home/source remote:backup 14042 14043 ### Dropbox for business ### 14044 14045 Rclone supports Dropbox for business and Team Folders. 14046 14047 When using Dropbox for business `remote:` and `remote:path/to/file` 14048 will refer to your personal folder. 14049 14050 If you wish to see Team Folders you must use a leading `/` in the 14051 path, so `rclone lsd remote:/` will refer to the root and show you all 14052 Team Folders and your User Folder. 14053 14054 You can then use team folders like this `remote:/TeamFolder` and 14055 `remote:/TeamFolder/path/to/file`. 14056 14057 A leading `/` for a Dropbox personal account will do nothing, but it 14058 will take an extra HTTP transaction so it should be avoided. 14059 14060 ### Modified time and Hashes ### 14061 14062 Dropbox supports modified times, but the only way to set a 14063 modification time is to re-upload the file. 14064 14065 This means that if you uploaded your data with an older version of 14066 rclone which didn't support the v2 API and modified times, rclone will 14067 decide to upload all your old data to fix the modification times. If 14068 you don't want this to happen use `--size-only` or `--checksum` flag 14069 to stop it. 14070 14071 Dropbox supports [its own hash 14072 type](https://www.dropbox.com/developers/reference/content-hash) which 14073 is checked for all transfers. 14074 14075 #### Restricted filename characters 14076 14077 | Character | Value | Replacement | 14078 | --------- |:-----:|:-----------:| 14079 | NUL | 0x00 | ␀ | 14080 | / | 0x2F | / | 14081 | DEL | 0x7F | ␡ | 14082 | \ | 0x5C | \ | 14083 14084 File names can also not end with the following characters. 14085 These only get replaced if they are last character in the name: 14086 14087 | Character | Value | Replacement | 14088 | --------- |:-----:|:-----------:| 14089 | SP | 0x20 | ␠ | 14090 14091 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 14092 as they can't be used in JSON strings. 14093 14094 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/dropbox/dropbox.go then run make backenddocs --> 14095 ### Standard Options 14096 14097 Here are the standard options specific to dropbox (Dropbox). 14098 14099 #### --dropbox-client-id 14100 14101 Dropbox App Client Id 14102 Leave blank normally. 14103 14104 - Config: client_id 14105 - Env Var: RCLONE_DROPBOX_CLIENT_ID 14106 - Type: string 14107 - Default: "" 14108 14109 #### --dropbox-client-secret 14110 14111 Dropbox App Client Secret 14112 Leave blank normally. 14113 14114 - Config: client_secret 14115 - Env Var: RCLONE_DROPBOX_CLIENT_SECRET 14116 - Type: string 14117 - Default: "" 14118 14119 ### Advanced Options 14120 14121 Here are the advanced options specific to dropbox (Dropbox). 14122 14123 #### --dropbox-chunk-size 14124 14125 Upload chunk size. (< 150M). 14126 14127 Any files larger than this will be uploaded in chunks of this size. 14128 14129 Note that chunks are buffered in memory (one at a time) so rclone can 14130 deal with retries. Setting this larger will increase the speed 14131 slightly (at most 10% for 128MB in tests) at the cost of using more 14132 memory. It can be set smaller if you are tight on memory. 14133 14134 - Config: chunk_size 14135 - Env Var: RCLONE_DROPBOX_CHUNK_SIZE 14136 - Type: SizeSuffix 14137 - Default: 48M 14138 14139 #### --dropbox-impersonate 14140 14141 Impersonate this user when using a business account. 14142 14143 - Config: impersonate 14144 - Env Var: RCLONE_DROPBOX_IMPERSONATE 14145 - Type: string 14146 - Default: "" 14147 14148 #### --dropbox-encoding 14149 14150 This sets the encoding for the backend. 14151 14152 See: the [encoding section in the overview](/overview/#encoding) for more info. 14153 14154 - Config: encoding 14155 - Env Var: RCLONE_DROPBOX_ENCODING 14156 - Type: MultiEncoder 14157 - Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot 14158 14159 <!--- autogenerated options stop --> 14160 14161 ### Limitations ### 14162 14163 Note that Dropbox is case insensitive so you can't have a file called 14164 "Hello.doc" and one called "hello.doc". 14165 14166 There are some file names such as `thumbs.db` which Dropbox can't 14167 store. There is a full list of them in the ["Ignored Files" section 14168 of this document](https://www.dropbox.com/en/help/145). Rclone will 14169 issue an error message `File name disallowed - not uploading` if it 14170 attempts to upload one of those file names, but the sync won't fail. 14171 14172 If you have more than 10,000 files in a directory then `rclone purge 14173 dropbox:dir` will return the error `Failed to purge: There are too 14174 many files involved in this operation`. As a work-around do an 14175 `rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`. 14176 14177 ### Get your own Dropbox App ID ### 14178 14179 When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users. 14180 14181 Here is how to create your own Dropbox App ID for rclone: 14182 14183 1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not 14184 to be the same account as the Dropbox you want to access) 14185 14186 2. Choose an API => Usually this should be `Dropbox API` 14187 14188 3. Choose the type of access you want to use => Full Dropbox or App Folder 14189 14190 4. Name your App 14191 14192 5. Click the button `Create App` 14193 14194 6. Find the `App key` and `App secret` Use these values in rclone config to add a new remote or edit an existing remote. 14195 14196 FTP 14197 ------------------------------ 14198 14199 FTP is the File Transfer Protocol. FTP support is provided using the 14200 [github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp) 14201 package. 14202 14203 Here is an example of making an FTP configuration. First run 14204 14205 rclone config 14206 14207 This will guide you through an interactive setup process. An FTP remote only 14208 needs a host together with and a username and a password. With anonymous FTP 14209 server, you will need to use `anonymous` as username and your email address as 14210 the password. 14211 14212 ``` 14213 No remotes found - make a new one 14214 n) New remote 14215 r) Rename remote 14216 c) Copy remote 14217 s) Set configuration password 14218 q) Quit config 14219 n/r/c/s/q> n 14220 name> remote 14221 Type of storage to configure. 14222 Enter a string value. Press Enter for the default (""). 14223 Choose a number from below, or type in your own value 14224 [snip] 14225 XX / FTP Connection 14226 \ "ftp" 14227 [snip] 14228 Storage> ftp 14229 ** See help for ftp backend at: https://rclone.org/ftp/ ** 14230 14231 FTP host to connect to 14232 Enter a string value. Press Enter for the default (""). 14233 Choose a number from below, or type in your own value 14234 1 / Connect to ftp.example.com 14235 \ "ftp.example.com" 14236 host> ftp.example.com 14237 FTP username, leave blank for current username, ncw 14238 Enter a string value. Press Enter for the default (""). 14239 user> 14240 FTP port, leave blank to use default (21) 14241 Enter a string value. Press Enter for the default (""). 14242 port> 14243 FTP password 14244 y) Yes type in my own password 14245 g) Generate random password 14246 y/g> y 14247 Enter the password: 14248 password: 14249 Confirm the password: 14250 password: 14251 Use FTP over TLS (Implicit) 14252 Enter a boolean value (true or false). Press Enter for the default ("false"). 14253 tls> 14254 Remote config 14255 -------------------- 14256 [remote] 14257 type = ftp 14258 host = ftp.example.com 14259 pass = *** ENCRYPTED *** 14260 -------------------- 14261 y) Yes this is OK 14262 e) Edit this remote 14263 d) Delete this remote 14264 y/e/d> y 14265 ``` 14266 14267 This remote is called `remote` and can now be used like this 14268 14269 See all directories in the home directory 14270 14271 rclone lsd remote: 14272 14273 Make a new directory 14274 14275 rclone mkdir remote:path/to/directory 14276 14277 List the contents of a directory 14278 14279 rclone ls remote:path/to/directory 14280 14281 Sync `/home/local/directory` to the remote directory, deleting any 14282 excess files in the directory. 14283 14284 rclone sync /home/local/directory remote:directory 14285 14286 ### Modified time ### 14287 14288 FTP does not support modified times. Any times you see on the server 14289 will be time of upload. 14290 14291 ### Checksums ### 14292 14293 FTP does not support any checksums. 14294 14295 #### Restricted filename characters 14296 14297 In addition to the [default restricted characters set](/overview/#restricted-characters) 14298 the following characters are also replaced: 14299 14300 File names can also not end with the following characters. 14301 These only get replaced if they are last character in the name: 14302 14303 | Character | Value | Replacement | 14304 | --------- |:-----:|:-----------:| 14305 | SP | 0x20 | ␠ | 14306 14307 Note that not all FTP servers can have all characters in file names, for example: 14308 14309 | FTP Server| Forbidden characters | 14310 | --------- |:--------------------:| 14311 | proftpd | `*` | 14312 | pureftpd | `\ [ ]` | 14313 14314 ### Implicit TLS ### 14315 14316 FTP supports implicit FTP over TLS servers (FTPS). This has to be enabled 14317 in the config for the remote. The default FTPS port is `990` so the 14318 port will likely have to be explictly set in the config for the remote. 14319 14320 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/ftp/ftp.go then run make backenddocs --> 14321 ### Standard Options 14322 14323 Here are the standard options specific to ftp (FTP Connection). 14324 14325 #### --ftp-host 14326 14327 FTP host to connect to 14328 14329 - Config: host 14330 - Env Var: RCLONE_FTP_HOST 14331 - Type: string 14332 - Default: "" 14333 - Examples: 14334 - "ftp.example.com" 14335 - Connect to ftp.example.com 14336 14337 #### --ftp-user 14338 14339 FTP username, leave blank for current username, $USER 14340 14341 - Config: user 14342 - Env Var: RCLONE_FTP_USER 14343 - Type: string 14344 - Default: "" 14345 14346 #### --ftp-port 14347 14348 FTP port, leave blank to use default (21) 14349 14350 - Config: port 14351 - Env Var: RCLONE_FTP_PORT 14352 - Type: string 14353 - Default: "" 14354 14355 #### --ftp-pass 14356 14357 FTP password 14358 14359 - Config: pass 14360 - Env Var: RCLONE_FTP_PASS 14361 - Type: string 14362 - Default: "" 14363 14364 #### --ftp-tls 14365 14366 Use FTP over TLS (Implicit) 14367 14368 - Config: tls 14369 - Env Var: RCLONE_FTP_TLS 14370 - Type: bool 14371 - Default: false 14372 14373 ### Advanced Options 14374 14375 Here are the advanced options specific to ftp (FTP Connection). 14376 14377 #### --ftp-concurrency 14378 14379 Maximum number of FTP simultaneous connections, 0 for unlimited 14380 14381 - Config: concurrency 14382 - Env Var: RCLONE_FTP_CONCURRENCY 14383 - Type: int 14384 - Default: 0 14385 14386 #### --ftp-no-check-certificate 14387 14388 Do not verify the TLS certificate of the server 14389 14390 - Config: no_check_certificate 14391 - Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE 14392 - Type: bool 14393 - Default: false 14394 14395 #### --ftp-disable-epsv 14396 14397 Disable using EPSV even if server advertises support 14398 14399 - Config: disable_epsv 14400 - Env Var: RCLONE_FTP_DISABLE_EPSV 14401 - Type: bool 14402 - Default: false 14403 14404 #### --ftp-encoding 14405 14406 This sets the encoding for the backend. 14407 14408 See: the [encoding section in the overview](/overview/#encoding) for more info. 14409 14410 - Config: encoding 14411 - Env Var: RCLONE_FTP_ENCODING 14412 - Type: MultiEncoder 14413 - Default: Slash,Del,Ctl,RightSpace,Dot 14414 14415 <!--- autogenerated options stop --> 14416 14417 ### Limitations ### 14418 14419 Note that since FTP isn't HTTP based the following flags don't work 14420 with it: `--dump-headers`, `--dump-bodies`, `--dump-auth` 14421 14422 Note that `--timeout` isn't supported (but `--contimeout` is). 14423 14424 Note that `--bind` isn't supported. 14425 14426 FTP could support server side move but doesn't yet. 14427 14428 Note that the ftp backend does not support the `ftp_proxy` environment 14429 variable yet. 14430 14431 Note that while implicit FTP over TLS is supported, 14432 explicit FTP over TLS is not. 14433 14434 Google Cloud Storage 14435 ------------------------------------------------- 14436 14437 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 14438 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 14439 14440 The initial setup for google cloud storage involves getting a token from Google Cloud Storage 14441 which you need to do in your browser. `rclone config` walks you 14442 through it. 14443 14444 Here is an example of how to make a remote called `remote`. First run: 14445 14446 rclone config 14447 14448 This will guide you through an interactive setup process: 14449 14450 ``` 14451 n) New remote 14452 d) Delete remote 14453 q) Quit config 14454 e/n/d/q> n 14455 name> remote 14456 Type of storage to configure. 14457 Choose a number from below, or type in your own value 14458 [snip] 14459 XX / Google Cloud Storage (this is not Google Drive) 14460 \ "google cloud storage" 14461 [snip] 14462 Storage> google cloud storage 14463 Google Application Client Id - leave blank normally. 14464 client_id> 14465 Google Application Client Secret - leave blank normally. 14466 client_secret> 14467 Project number optional - needed only for list/create/delete buckets - see your developer console. 14468 project_number> 12345678 14469 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. 14470 service_account_file> 14471 Access Control List for new objects. 14472 Choose a number from below, or type in your own value 14473 1 / Object owner gets OWNER access, and all Authenticated Users get READER access. 14474 \ "authenticatedRead" 14475 2 / Object owner gets OWNER access, and project team owners get OWNER access. 14476 \ "bucketOwnerFullControl" 14477 3 / Object owner gets OWNER access, and project team owners get READER access. 14478 \ "bucketOwnerRead" 14479 4 / Object owner gets OWNER access [default if left blank]. 14480 \ "private" 14481 5 / Object owner gets OWNER access, and project team members get access according to their roles. 14482 \ "projectPrivate" 14483 6 / Object owner gets OWNER access, and all Users get READER access. 14484 \ "publicRead" 14485 object_acl> 4 14486 Access Control List for new buckets. 14487 Choose a number from below, or type in your own value 14488 1 / Project team owners get OWNER access, and all Authenticated Users get READER access. 14489 \ "authenticatedRead" 14490 2 / Project team owners get OWNER access [default if left blank]. 14491 \ "private" 14492 3 / Project team members get access according to their roles. 14493 \ "projectPrivate" 14494 4 / Project team owners get OWNER access, and all Users get READER access. 14495 \ "publicRead" 14496 5 / Project team owners get OWNER access, and all Users get WRITER access. 14497 \ "publicReadWrite" 14498 bucket_acl> 2 14499 Location for the newly created buckets. 14500 Choose a number from below, or type in your own value 14501 1 / Empty for default location (US). 14502 \ "" 14503 2 / Multi-regional location for Asia. 14504 \ "asia" 14505 3 / Multi-regional location for Europe. 14506 \ "eu" 14507 4 / Multi-regional location for United States. 14508 \ "us" 14509 5 / Taiwan. 14510 \ "asia-east1" 14511 6 / Tokyo. 14512 \ "asia-northeast1" 14513 7 / Singapore. 14514 \ "asia-southeast1" 14515 8 / Sydney. 14516 \ "australia-southeast1" 14517 9 / Belgium. 14518 \ "europe-west1" 14519 10 / London. 14520 \ "europe-west2" 14521 11 / Iowa. 14522 \ "us-central1" 14523 12 / South Carolina. 14524 \ "us-east1" 14525 13 / Northern Virginia. 14526 \ "us-east4" 14527 14 / Oregon. 14528 \ "us-west1" 14529 location> 12 14530 The storage class to use when storing objects in Google Cloud Storage. 14531 Choose a number from below, or type in your own value 14532 1 / Default 14533 \ "" 14534 2 / Multi-regional storage class 14535 \ "MULTI_REGIONAL" 14536 3 / Regional storage class 14537 \ "REGIONAL" 14538 4 / Nearline storage class 14539 \ "NEARLINE" 14540 5 / Coldline storage class 14541 \ "COLDLINE" 14542 6 / Durable reduced availability storage class 14543 \ "DURABLE_REDUCED_AVAILABILITY" 14544 storage_class> 5 14545 Remote config 14546 Use auto config? 14547 * Say Y if not sure 14548 * Say N if you are working on a remote or headless machine or Y didn't work 14549 y) Yes 14550 n) No 14551 y/n> y 14552 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 14553 Log in and authorize rclone for access 14554 Waiting for code... 14555 Got code 14556 -------------------- 14557 [remote] 14558 type = google cloud storage 14559 client_id = 14560 client_secret = 14561 token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} 14562 project_number = 12345678 14563 object_acl = private 14564 bucket_acl = private 14565 -------------------- 14566 y) Yes this is OK 14567 e) Edit this remote 14568 d) Delete this remote 14569 y/e/d> y 14570 ``` 14571 14572 Note that rclone runs a webserver on your local machine to collect the 14573 token as returned from Google if you use auto config mode. This only 14574 runs from the moment it opens your browser to the moment you get back 14575 the verification code. This is on `http://127.0.0.1:53682/` and this 14576 it may require you to unblock it temporarily if you are running a host 14577 firewall, or use manual mode. 14578 14579 This remote is called `remote` and can now be used like this 14580 14581 See all the buckets in your project 14582 14583 rclone lsd remote: 14584 14585 Make a new bucket 14586 14587 rclone mkdir remote:bucket 14588 14589 List the contents of a bucket 14590 14591 rclone ls remote:bucket 14592 14593 Sync `/home/local/directory` to the remote bucket, deleting any excess 14594 files in the bucket. 14595 14596 rclone sync /home/local/directory remote:bucket 14597 14598 ### Service Account support ### 14599 14600 You can set up rclone with Google Cloud Storage in an unattended mode, 14601 i.e. not tied to a specific end-user Google account. This is useful 14602 when you want to synchronise files onto machines that don't have 14603 actively logged-in users, for example build machines. 14604 14605 To get credentials for Google Cloud Platform 14606 [IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts), 14607 please head to the 14608 [Service Account](https://console.cloud.google.com/permissions/serviceaccounts) 14609 section of the Google Developer Console. Service Accounts behave just 14610 like normal `User` permissions in 14611 [Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control), 14612 so you can limit their access (e.g. make them read only). After 14613 creating an account, a JSON file containing the Service Account's 14614 credentials will be downloaded onto your machines. These credentials 14615 are what rclone will use for authentication. 14616 14617 To use a Service Account instead of OAuth2 token flow, enter the path 14618 to your Service Account credentials at the `service_account_file` 14619 prompt and rclone won't use the browser based authentication 14620 flow. If you'd rather stuff the contents of the credentials file into 14621 the rclone config file, you can set `service_account_credentials` with 14622 the actual contents of the file instead, or set the equivalent 14623 environment variable. 14624 14625 ### Application Default Credentials ### 14626 14627 If no other source of credentials is provided, rclone will fall back 14628 to 14629 [Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials) 14630 this is useful both when you already have configured authentication 14631 for your developer account, or in production when running on a google 14632 compute host. Note that if running in docker, you may need to run 14633 additional commands on your google compute machine - 14634 [see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper). 14635 14636 Note that in the case application default credentials are used, there 14637 is no need to explicitly configure a project number. 14638 14639 ### --fast-list ### 14640 14641 This remote supports `--fast-list` which allows you to use fewer 14642 transactions in exchange for more memory. See the [rclone 14643 docs](/docs/#fast-list) for more details. 14644 14645 ### Modified time ### 14646 14647 Google google cloud storage stores md5sums natively and rclone stores 14648 modification times as metadata on the object, under the "mtime" key in 14649 RFC3339 format accurate to 1ns. 14650 14651 #### Restricted filename characters 14652 14653 | Character | Value | Replacement | 14654 | --------- |:-----:|:-----------:| 14655 | NUL | 0x00 | ␀ | 14656 | LF | 0x0A | ␊ | 14657 | CR | 0x0D | ␍ | 14658 | / | 0x2F | / | 14659 14660 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 14661 as they can't be used in JSON strings. 14662 14663 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/googlecloudstorage/googlecloudstorage.go then run make backenddocs --> 14664 ### Standard Options 14665 14666 Here are the standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 14667 14668 #### --gcs-client-id 14669 14670 Google Application Client Id 14671 Leave blank normally. 14672 14673 - Config: client_id 14674 - Env Var: RCLONE_GCS_CLIENT_ID 14675 - Type: string 14676 - Default: "" 14677 14678 #### --gcs-client-secret 14679 14680 Google Application Client Secret 14681 Leave blank normally. 14682 14683 - Config: client_secret 14684 - Env Var: RCLONE_GCS_CLIENT_SECRET 14685 - Type: string 14686 - Default: "" 14687 14688 #### --gcs-project-number 14689 14690 Project number. 14691 Optional - needed only for list/create/delete buckets - see your developer console. 14692 14693 - Config: project_number 14694 - Env Var: RCLONE_GCS_PROJECT_NUMBER 14695 - Type: string 14696 - Default: "" 14697 14698 #### --gcs-service-account-file 14699 14700 Service Account Credentials JSON file path 14701 Leave blank normally. 14702 Needed only if you want use SA instead of interactive login. 14703 14704 - Config: service_account_file 14705 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE 14706 - Type: string 14707 - Default: "" 14708 14709 #### --gcs-service-account-credentials 14710 14711 Service Account Credentials JSON blob 14712 Leave blank normally. 14713 Needed only if you want use SA instead of interactive login. 14714 14715 - Config: service_account_credentials 14716 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS 14717 - Type: string 14718 - Default: "" 14719 14720 #### --gcs-object-acl 14721 14722 Access Control List for new objects. 14723 14724 - Config: object_acl 14725 - Env Var: RCLONE_GCS_OBJECT_ACL 14726 - Type: string 14727 - Default: "" 14728 - Examples: 14729 - "authenticatedRead" 14730 - Object owner gets OWNER access, and all Authenticated Users get READER access. 14731 - "bucketOwnerFullControl" 14732 - Object owner gets OWNER access, and project team owners get OWNER access. 14733 - "bucketOwnerRead" 14734 - Object owner gets OWNER access, and project team owners get READER access. 14735 - "private" 14736 - Object owner gets OWNER access [default if left blank]. 14737 - "projectPrivate" 14738 - Object owner gets OWNER access, and project team members get access according to their roles. 14739 - "publicRead" 14740 - Object owner gets OWNER access, and all Users get READER access. 14741 14742 #### --gcs-bucket-acl 14743 14744 Access Control List for new buckets. 14745 14746 - Config: bucket_acl 14747 - Env Var: RCLONE_GCS_BUCKET_ACL 14748 - Type: string 14749 - Default: "" 14750 - Examples: 14751 - "authenticatedRead" 14752 - Project team owners get OWNER access, and all Authenticated Users get READER access. 14753 - "private" 14754 - Project team owners get OWNER access [default if left blank]. 14755 - "projectPrivate" 14756 - Project team members get access according to their roles. 14757 - "publicRead" 14758 - Project team owners get OWNER access, and all Users get READER access. 14759 - "publicReadWrite" 14760 - Project team owners get OWNER access, and all Users get WRITER access. 14761 14762 #### --gcs-bucket-policy-only 14763 14764 Access checks should use bucket-level IAM policies. 14765 14766 If you want to upload objects to a bucket with Bucket Policy Only set 14767 then you will need to set this. 14768 14769 When it is set, rclone: 14770 14771 - ignores ACLs set on buckets 14772 - ignores ACLs set on objects 14773 - creates buckets with Bucket Policy Only set 14774 14775 Docs: https://cloud.google.com/storage/docs/bucket-policy-only 14776 14777 14778 - Config: bucket_policy_only 14779 - Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY 14780 - Type: bool 14781 - Default: false 14782 14783 #### --gcs-location 14784 14785 Location for the newly created buckets. 14786 14787 - Config: location 14788 - Env Var: RCLONE_GCS_LOCATION 14789 - Type: string 14790 - Default: "" 14791 - Examples: 14792 - "" 14793 - Empty for default location (US). 14794 - "asia" 14795 - Multi-regional location for Asia. 14796 - "eu" 14797 - Multi-regional location for Europe. 14798 - "us" 14799 - Multi-regional location for United States. 14800 - "asia-east1" 14801 - Taiwan. 14802 - "asia-east2" 14803 - Hong Kong. 14804 - "asia-northeast1" 14805 - Tokyo. 14806 - "asia-south1" 14807 - Mumbai. 14808 - "asia-southeast1" 14809 - Singapore. 14810 - "australia-southeast1" 14811 - Sydney. 14812 - "europe-north1" 14813 - Finland. 14814 - "europe-west1" 14815 - Belgium. 14816 - "europe-west2" 14817 - London. 14818 - "europe-west3" 14819 - Frankfurt. 14820 - "europe-west4" 14821 - Netherlands. 14822 - "us-central1" 14823 - Iowa. 14824 - "us-east1" 14825 - South Carolina. 14826 - "us-east4" 14827 - Northern Virginia. 14828 - "us-west1" 14829 - Oregon. 14830 - "us-west2" 14831 - California. 14832 14833 #### --gcs-storage-class 14834 14835 The storage class to use when storing objects in Google Cloud Storage. 14836 14837 - Config: storage_class 14838 - Env Var: RCLONE_GCS_STORAGE_CLASS 14839 - Type: string 14840 - Default: "" 14841 - Examples: 14842 - "" 14843 - Default 14844 - "MULTI_REGIONAL" 14845 - Multi-regional storage class 14846 - "REGIONAL" 14847 - Regional storage class 14848 - "NEARLINE" 14849 - Nearline storage class 14850 - "COLDLINE" 14851 - Coldline storage class 14852 - "DURABLE_REDUCED_AVAILABILITY" 14853 - Durable reduced availability storage class 14854 14855 ### Advanced Options 14856 14857 Here are the advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 14858 14859 #### --gcs-encoding 14860 14861 This sets the encoding for the backend. 14862 14863 See: the [encoding section in the overview](/overview/#encoding) for more info. 14864 14865 - Config: encoding 14866 - Env Var: RCLONE_GCS_ENCODING 14867 - Type: MultiEncoder 14868 - Default: Slash,CrLf,InvalidUtf8,Dot 14869 14870 <!--- autogenerated options stop --> 14871 14872 Google Drive 14873 ----------------------------------------- 14874 14875 Paths are specified as `drive:path` 14876 14877 Drive paths may be as deep as required, eg `drive:directory/subdirectory`. 14878 14879 The initial setup for drive involves getting a token from Google drive 14880 which you need to do in your browser. `rclone config` walks you 14881 through it. 14882 14883 Here is an example of how to make a remote called `remote`. First run: 14884 14885 rclone config 14886 14887 This will guide you through an interactive setup process: 14888 14889 ``` 14890 No remotes found - make a new one 14891 n) New remote 14892 r) Rename remote 14893 c) Copy remote 14894 s) Set configuration password 14895 q) Quit config 14896 n/r/c/s/q> n 14897 name> remote 14898 Type of storage to configure. 14899 Choose a number from below, or type in your own value 14900 [snip] 14901 XX / Google Drive 14902 \ "drive" 14903 [snip] 14904 Storage> drive 14905 Google Application Client Id - leave blank normally. 14906 client_id> 14907 Google Application Client Secret - leave blank normally. 14908 client_secret> 14909 Scope that rclone should use when requesting access from drive. 14910 Choose a number from below, or type in your own value 14911 1 / Full access all files, excluding Application Data Folder. 14912 \ "drive" 14913 2 / Read-only access to file metadata and file contents. 14914 \ "drive.readonly" 14915 / Access to files created by rclone only. 14916 3 | These are visible in the drive website. 14917 | File authorization is revoked when the user deauthorizes the app. 14918 \ "drive.file" 14919 / Allows read and write access to the Application Data folder. 14920 4 | This is not visible in the drive website. 14921 \ "drive.appfolder" 14922 / Allows read-only access to file metadata but 14923 5 | does not allow any access to read or download file content. 14924 \ "drive.metadata.readonly" 14925 scope> 1 14926 ID of the root folder - leave blank normally. Fill in to access "Computers" folders. (see docs). 14927 root_folder_id> 14928 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. 14929 service_account_file> 14930 Remote config 14931 Use auto config? 14932 * Say Y if not sure 14933 * Say N if you are working on a remote or headless machine or Y didn't work 14934 y) Yes 14935 n) No 14936 y/n> y 14937 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 14938 Log in and authorize rclone for access 14939 Waiting for code... 14940 Got code 14941 Configure this as a team drive? 14942 y) Yes 14943 n) No 14944 y/n> n 14945 -------------------- 14946 [remote] 14947 client_id = 14948 client_secret = 14949 scope = drive 14950 root_folder_id = 14951 service_account_file = 14952 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} 14953 -------------------- 14954 y) Yes this is OK 14955 e) Edit this remote 14956 d) Delete this remote 14957 y/e/d> y 14958 ``` 14959 14960 Note that rclone runs a webserver on your local machine to collect the 14961 token as returned from Google if you use auto config mode. This only 14962 runs from the moment it opens your browser to the moment you get back 14963 the verification code. This is on `http://127.0.0.1:53682/` and this 14964 it may require you to unblock it temporarily if you are running a host 14965 firewall, or use manual mode. 14966 14967 You can then use it like this, 14968 14969 List directories in top level of your drive 14970 14971 rclone lsd remote: 14972 14973 List all the files in your drive 14974 14975 rclone ls remote: 14976 14977 To copy a local directory to a drive directory called backup 14978 14979 rclone copy /home/source remote:backup 14980 14981 ### Scopes ### 14982 14983 Rclone allows you to select which scope you would like for rclone to 14984 use. This changes what type of token is granted to rclone. [The 14985 scopes are defined 14986 here.](https://developers.google.com/drive/v3/web/about-auth). 14987 14988 The scope are 14989 14990 #### drive #### 14991 14992 This is the default scope and allows full access to all files, except 14993 for the Application Data Folder (see below). 14994 14995 Choose this one if you aren't sure. 14996 14997 #### drive.readonly #### 14998 14999 This allows read only access to all files. Files may be listed and 15000 downloaded but not uploaded, renamed or deleted. 15001 15002 #### drive.file #### 15003 15004 With this scope rclone can read/view/modify only those files and 15005 folders it creates. 15006 15007 So if you uploaded files to drive via the web interface (or any other 15008 means) they will not be visible to rclone. 15009 15010 This can be useful if you are using rclone to backup data and you want 15011 to be sure confidential data on your drive is not visible to rclone. 15012 15013 Files created with this scope are visible in the web interface. 15014 15015 #### drive.appfolder #### 15016 15017 This gives rclone its own private area to store files. Rclone will 15018 not be able to see any other files on your drive and you won't be able 15019 to see rclone's files from the web interface either. 15020 15021 #### drive.metadata.readonly #### 15022 15023 This allows read only access to file names only. It does not allow 15024 rclone to download or upload data, or rename or delete files or 15025 directories. 15026 15027 ### Root folder ID ### 15028 15029 You can set the `root_folder_id` for rclone. This is the directory 15030 (identified by its `Folder ID`) that rclone considers to be the root 15031 of your drive. 15032 15033 Normally you will leave this blank and rclone will determine the 15034 correct root to use itself. 15035 15036 However you can set this to restrict rclone to a specific folder 15037 hierarchy or to access data within the "Computers" tab on the drive 15038 web interface (where files from Google's Backup and Sync desktop 15039 program go). 15040 15041 In order to do this you will have to find the `Folder ID` of the 15042 directory you wish rclone to display. This will be the last segment 15043 of the URL when you open the relevant folder in the drive web 15044 interface. 15045 15046 So if the folder you want rclone to use has a URL which looks like 15047 `https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` 15048 in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as 15049 the `root_folder_id` in the config. 15050 15051 **NB** folders under the "Computers" tab seem to be read only (drive 15052 gives a 500 error) when using rclone. 15053 15054 There doesn't appear to be an API to discover the folder IDs of the 15055 "Computers" tab - please contact us if you know otherwise! 15056 15057 Note also that rclone can't access any data under the "Backups" tab on 15058 the google drive web interface yet. 15059 15060 ### Service Account support ### 15061 15062 You can set up rclone with Google Drive in an unattended mode, 15063 i.e. not tied to a specific end-user Google account. This is useful 15064 when you want to synchronise files onto machines that don't have 15065 actively logged-in users, for example build machines. 15066 15067 To use a Service Account instead of OAuth2 token flow, enter the path 15068 to your Service Account credentials at the `service_account_file` 15069 prompt during `rclone config` and rclone won't use the browser based 15070 authentication flow. If you'd rather stuff the contents of the 15071 credentials file into the rclone config file, you can set 15072 `service_account_credentials` with the actual contents of the file 15073 instead, or set the equivalent environment variable. 15074 15075 #### Use case - Google Apps/G-suite account and individual Drive #### 15076 15077 Let's say that you are the administrator of a Google Apps (old) or 15078 G-suite account. 15079 The goal is to store data on an individual's Drive account, who IS 15080 a member of the domain. 15081 We'll call the domain **example.com**, and the user 15082 **foo@example.com**. 15083 15084 There's a few steps we need to go through to accomplish this: 15085 15086 ##### 1. Create a service account for example.com ##### 15087 - To create a service account and obtain its credentials, go to the 15088 [Google Developer Console](https://console.developers.google.com). 15089 - You must have a project - create one if you don't. 15090 - Then go to "IAM & admin" -> "Service Accounts". 15091 - Use the "Create Credentials" button. Fill in "Service account name" 15092 with something that identifies your client. "Role" can be empty. 15093 - Tick "Furnish a new private key" - select "Key type JSON". 15094 - Tick "Enable G Suite Domain-wide Delegation". This option makes 15095 "impersonation" possible, as documented here: 15096 [Delegating domain-wide authority to the service account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority) 15097 - These credentials are what rclone will use for authentication. 15098 If you ever need to remove access, press the "Delete service 15099 account key" button. 15100 15101 ##### 2. Allowing API access to example.com Google Drive ##### 15102 - Go to example.com's admin console 15103 - Go into "Security" (or use the search bar) 15104 - Select "Show more" and then "Advanced settings" 15105 - Select "Manage API client access" in the "Authentication" section 15106 - In the "Client Name" field enter the service account's 15107 "Client ID" - this can be found in the Developer Console under 15108 "IAM & Admin" -> "Service Accounts", then "View Client ID" for 15109 the newly created service account. 15110 It is a ~21 character numerical string. 15111 - In the next field, "One or More API Scopes", enter 15112 `https://www.googleapis.com/auth/drive` 15113 to grant access to Google Drive specifically. 15114 15115 ##### 3. Configure rclone, assuming a new install ##### 15116 15117 ``` 15118 rclone config 15119 15120 n/s/q> n # New 15121 name>gdrive # Gdrive is an example name 15122 Storage> # Select the number shown for Google Drive 15123 client_id> # Can be left blank 15124 client_secret> # Can be left blank 15125 scope> # Select your scope, 1 for example 15126 root_folder_id> # Can be left blank 15127 service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! 15128 y/n> # Auto config, y 15129 15130 ``` 15131 15132 ##### 4. Verify that it's working ##### 15133 - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup` 15134 - The arguments do: 15135 - `-v` - verbose logging 15136 - `--drive-impersonate foo@example.com` - this is what does 15137 the magic, pretending to be user foo. 15138 - `lsf` - list files in a parsing friendly way 15139 - `gdrive:backup` - use the remote called gdrive, work in 15140 the folder named backup. 15141 15142 ### Team drives ### 15143 15144 If you want to configure the remote to point to a Google Team Drive 15145 then answer `y` to the question `Configure this as a team drive?`. 15146 15147 This will fetch the list of Team Drives from google and allow you to 15148 configure which one you want to use. You can also type in a team 15149 drive ID if you prefer. 15150 15151 For example: 15152 15153 ``` 15154 Configure this as a team drive? 15155 y) Yes 15156 n) No 15157 y/n> y 15158 Fetching team drive list... 15159 Choose a number from below, or type in your own value 15160 1 / Rclone Test 15161 \ "xxxxxxxxxxxxxxxxxxxx" 15162 2 / Rclone Test 2 15163 \ "yyyyyyyyyyyyyyyyyyyy" 15164 3 / Rclone Test 3 15165 \ "zzzzzzzzzzzzzzzzzzzz" 15166 Enter a Team Drive ID> 1 15167 -------------------- 15168 [remote] 15169 client_id = 15170 client_secret = 15171 token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} 15172 team_drive = xxxxxxxxxxxxxxxxxxxx 15173 -------------------- 15174 y) Yes this is OK 15175 e) Edit this remote 15176 d) Delete this remote 15177 y/e/d> y 15178 ``` 15179 15180 ### --fast-list ### 15181 15182 This remote supports `--fast-list` which allows you to use fewer 15183 transactions in exchange for more memory. See the [rclone 15184 docs](/docs/#fast-list) for more details. 15185 15186 It does this by combining multiple `list` calls into a single API request. 15187 15188 This works by combining many `'%s' in parents` filters into one expression. 15189 To list the contents of directories a, b and c, the following requests will be send by the regular `List` function: 15190 ``` 15191 trashed=false and 'a' in parents 15192 trashed=false and 'b' in parents 15193 trashed=false and 'c' in parents 15194 ``` 15195 These can now be combined into a single request: 15196 ``` 15197 trashed=false and ('a' in parents or 'b' in parents or 'c' in parents) 15198 ``` 15199 15200 The implementation of `ListR` will put up to 50 `parents` filters into one request. 15201 It will use the `--checkers` value to specify the number of requests to run in parallel. 15202 15203 In tests, these batch requests were up to 20x faster than the regular method. 15204 Running the following command against different sized folders gives: 15205 ``` 15206 rclone lsjson -vv -R --checkers=6 gdrive:folder 15207 ``` 15208 15209 small folder (220 directories, 700 files): 15210 15211 - without `--fast-list`: 38s 15212 - with `--fast-list`: 10s 15213 15214 large folder (10600 directories, 39000 files): 15215 15216 - without `--fast-list`: 22:05 min 15217 - with `--fast-list`: 58s 15218 15219 ### Modified time ### 15220 15221 Google drive stores modification times accurate to 1 ms. 15222 15223 #### Restricted filename characters 15224 15225 Only Invalid UTF-8 bytes will be [replaced](/overview/#invalid-utf8), 15226 as they can't be used in JSON strings. 15227 15228 In contrast to other backends, `/` can also be used in names and `.` 15229 or `..` are valid names. 15230 15231 ### Revisions ### 15232 15233 Google drive stores revisions of files. When you upload a change to 15234 an existing file to google drive using rclone it will create a new 15235 revision of that file. 15236 15237 Revisions follow the standard google policy which at time of writing 15238 was 15239 15240 * They are deleted after 30 days or 100 revisions (whatever comes first). 15241 * They do not count towards a user storage quota. 15242 15243 ### Deleting files ### 15244 15245 By default rclone will send all files to the trash when deleting 15246 files. If deleting them permanently is required then use the 15247 `--drive-use-trash=false` flag, or set the equivalent environment 15248 variable. 15249 15250 ### Emptying trash ### 15251 15252 If you wish to empty your trash you can use the `rclone cleanup remote:` 15253 command which will permanently delete all your trashed files. This command 15254 does not take any path arguments. 15255 15256 Note that Google Drive takes some time (minutes to days) to empty the 15257 trash even though the command returns within a few seconds. No output 15258 is echoed, so there will be no confirmation even using -v or -vv. 15259 15260 ### Quota information ### 15261 15262 To view your current quota you can use the `rclone about remote:` 15263 command which will display your usage limit (quota), the usage in Google 15264 Drive, the size of all files in the Trash and the space used by other 15265 Google services such as Gmail. This command does not take any path 15266 arguments. 15267 15268 #### Import/Export of google documents #### 15269 15270 Google documents can be exported from and uploaded to Google Drive. 15271 15272 When rclone downloads a Google doc it chooses a format to download 15273 depending upon the `--drive-export-formats` setting. 15274 By default the export formats are `docx,xlsx,pptx,svg` which are a 15275 sensible default for an editable document. 15276 15277 When choosing a format, rclone runs down the list provided in order 15278 and chooses the first file format the doc can be exported as from the 15279 list. If the file can't be exported to a format on the formats list, 15280 then rclone will choose a format from the default list. 15281 15282 If you prefer an archive copy then you might use `--drive-export-formats 15283 pdf`, or if you prefer openoffice/libreoffice formats you might use 15284 `--drive-export-formats ods,odt,odp`. 15285 15286 Note that rclone adds the extension to the google doc, so if it is 15287 called `My Spreadsheet` on google docs, it will be exported as `My 15288 Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc. 15289 15290 When importing files into Google Drive, rclone will convert all 15291 files with an extension in `--drive-import-formats` to their 15292 associated document type. 15293 rclone will not convert any files by default, since the conversion 15294 is lossy process. 15295 15296 The conversion must result in a file with the same extension when 15297 the `--drive-export-formats` rules are applied to the uploaded document. 15298 15299 Here are some examples for allowed and prohibited conversions. 15300 15301 | export-formats | import-formats | Upload Ext | Document Ext | Allowed | 15302 | -------------- | -------------- | ---------- | ------------ | ------- | 15303 | odt | odt | odt | odt | Yes | 15304 | odt | docx,odt | odt | odt | Yes | 15305 | | docx | docx | docx | Yes | 15306 | | odt | odt | docx | No | 15307 | odt,docx | docx,odt | docx | odt | No | 15308 | docx,odt | docx,odt | docx | docx | Yes | 15309 | docx,odt | docx,odt | odt | docx | No | 15310 15311 This limitation can be disabled by specifying `--drive-allow-import-name-change`. 15312 When using this flag, rclone can convert multiple files types resulting 15313 in the same document type at once, eg with `--drive-import-formats docx,odt,txt`, 15314 all files having these extension would result in a document represented as a docx file. 15315 This brings the additional risk of overwriting a document, if multiple files 15316 have the same stem. Many rclone operations will not handle this name change 15317 in any way. They assume an equal name when copying files and might copy the 15318 file again or delete them when the name changes. 15319 15320 Here are the possible export extensions with their corresponding mime types. 15321 Most of these can also be used for importing, but there more that are not 15322 listed here. Some of these additional ones might only be available when 15323 the operating system provides the correct MIME type entries. 15324 15325 This list can be changed by Google Drive at any time and might not 15326 represent the currently available conversions. 15327 15328 | Extension | Mime Type | Description | 15329 | --------- |-----------| ------------| 15330 | csv | text/csv | Standard CSV format for Spreadsheets | 15331 | docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document | 15332 | epub | application/epub+zip | E-book format | 15333 | html | text/html | An HTML Document | 15334 | jpg | image/jpeg | A JPEG Image File | 15335 | json | application/vnd.google-apps.script+json | JSON Text Format | 15336 | odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation | 15337 | ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 15338 | ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 15339 | odt | application/vnd.oasis.opendocument.text | Openoffice Document | 15340 | pdf | application/pdf | Adobe PDF Format | 15341 | png | image/png | PNG Image Format| 15342 | pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint | 15343 | rtf | application/rtf | Rich Text Format | 15344 | svg | image/svg+xml | Scalable Vector Graphics Format | 15345 | tsv | text/tab-separated-values | Standard TSV format for spreadsheets | 15346 | txt | text/plain | Plain Text | 15347 | xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet | 15348 | zip | application/zip | A ZIP file of HTML, Images CSS | 15349 15350 Google documents can also be exported as link files. These files will 15351 open a browser window for the Google Docs website of that document 15352 when opened. The link file extension has to be specified as a 15353 `--drive-export-formats` parameter. They will match all available 15354 Google Documents. 15355 15356 | Extension | Description | OS Support | 15357 | --------- | ----------- | ---------- | 15358 | desktop | freedesktop.org specified desktop entry | Linux | 15359 | link.html | An HTML Document with a redirect | All | 15360 | url | INI style link file | macOS, Windows | 15361 | webloc | macOS specific XML format | macOS | 15362 15363 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/drive/drive.go then run make backenddocs --> 15364 ### Standard Options 15365 15366 Here are the standard options specific to drive (Google Drive). 15367 15368 #### --drive-client-id 15369 15370 Google Application Client Id 15371 Setting your own is recommended. 15372 See https://rclone.org/drive/#making-your-own-client-id for how to create your own. 15373 If you leave this blank, it will use an internal key which is low performance. 15374 15375 - Config: client_id 15376 - Env Var: RCLONE_DRIVE_CLIENT_ID 15377 - Type: string 15378 - Default: "" 15379 15380 #### --drive-client-secret 15381 15382 Google Application Client Secret 15383 Setting your own is recommended. 15384 15385 - Config: client_secret 15386 - Env Var: RCLONE_DRIVE_CLIENT_SECRET 15387 - Type: string 15388 - Default: "" 15389 15390 #### --drive-scope 15391 15392 Scope that rclone should use when requesting access from drive. 15393 15394 - Config: scope 15395 - Env Var: RCLONE_DRIVE_SCOPE 15396 - Type: string 15397 - Default: "" 15398 - Examples: 15399 - "drive" 15400 - Full access all files, excluding Application Data Folder. 15401 - "drive.readonly" 15402 - Read-only access to file metadata and file contents. 15403 - "drive.file" 15404 - Access to files created by rclone only. 15405 - These are visible in the drive website. 15406 - File authorization is revoked when the user deauthorizes the app. 15407 - "drive.appfolder" 15408 - Allows read and write access to the Application Data folder. 15409 - This is not visible in the drive website. 15410 - "drive.metadata.readonly" 15411 - Allows read-only access to file metadata but 15412 - does not allow any access to read or download file content. 15413 15414 #### --drive-root-folder-id 15415 15416 ID of the root folder 15417 Leave blank normally. 15418 15419 Fill in to access "Computers" folders (see docs), or for rclone to use 15420 a non root folder as its starting point. 15421 15422 Note that if this is blank, the first time rclone runs it will fill it 15423 in with the ID of the root folder. 15424 15425 15426 - Config: root_folder_id 15427 - Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID 15428 - Type: string 15429 - Default: "" 15430 15431 #### --drive-service-account-file 15432 15433 Service Account Credentials JSON file path 15434 Leave blank normally. 15435 Needed only if you want use SA instead of interactive login. 15436 15437 - Config: service_account_file 15438 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE 15439 - Type: string 15440 - Default: "" 15441 15442 ### Advanced Options 15443 15444 Here are the advanced options specific to drive (Google Drive). 15445 15446 #### --drive-service-account-credentials 15447 15448 Service Account Credentials JSON blob 15449 Leave blank normally. 15450 Needed only if you want use SA instead of interactive login. 15451 15452 - Config: service_account_credentials 15453 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS 15454 - Type: string 15455 - Default: "" 15456 15457 #### --drive-team-drive 15458 15459 ID of the Team Drive 15460 15461 - Config: team_drive 15462 - Env Var: RCLONE_DRIVE_TEAM_DRIVE 15463 - Type: string 15464 - Default: "" 15465 15466 #### --drive-auth-owner-only 15467 15468 Only consider files owned by the authenticated user. 15469 15470 - Config: auth_owner_only 15471 - Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY 15472 - Type: bool 15473 - Default: false 15474 15475 #### --drive-use-trash 15476 15477 Send files to the trash instead of deleting permanently. 15478 Defaults to true, namely sending files to the trash. 15479 Use `--drive-use-trash=false` to delete files permanently instead. 15480 15481 - Config: use_trash 15482 - Env Var: RCLONE_DRIVE_USE_TRASH 15483 - Type: bool 15484 - Default: true 15485 15486 #### --drive-skip-gdocs 15487 15488 Skip google documents in all listings. 15489 If given, gdocs practically become invisible to rclone. 15490 15491 - Config: skip_gdocs 15492 - Env Var: RCLONE_DRIVE_SKIP_GDOCS 15493 - Type: bool 15494 - Default: false 15495 15496 #### --drive-skip-checksum-gphotos 15497 15498 Skip MD5 checksum on Google photos and videos only. 15499 15500 Use this if you get checksum errors when transferring Google photos or 15501 videos. 15502 15503 Setting this flag will cause Google photos and videos to return a 15504 blank MD5 checksum. 15505 15506 Google photos are identifed by being in the "photos" space. 15507 15508 Corrupted checksums are caused by Google modifying the image/video but 15509 not updating the checksum. 15510 15511 - Config: skip_checksum_gphotos 15512 - Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS 15513 - Type: bool 15514 - Default: false 15515 15516 #### --drive-shared-with-me 15517 15518 Only show files that are shared with me. 15519 15520 Instructs rclone to operate on your "Shared with me" folder (where 15521 Google Drive lets you access the files and folders others have shared 15522 with you). 15523 15524 This works both with the "list" (lsd, lsl, etc) and the "copy" 15525 commands (copy, sync, etc), and with all other commands too. 15526 15527 - Config: shared_with_me 15528 - Env Var: RCLONE_DRIVE_SHARED_WITH_ME 15529 - Type: bool 15530 - Default: false 15531 15532 #### --drive-trashed-only 15533 15534 Only show files that are in the trash. 15535 This will show trashed files in their original directory structure. 15536 15537 - Config: trashed_only 15538 - Env Var: RCLONE_DRIVE_TRASHED_ONLY 15539 - Type: bool 15540 - Default: false 15541 15542 #### --drive-formats 15543 15544 Deprecated: see export_formats 15545 15546 - Config: formats 15547 - Env Var: RCLONE_DRIVE_FORMATS 15548 - Type: string 15549 - Default: "" 15550 15551 #### --drive-export-formats 15552 15553 Comma separated list of preferred formats for downloading Google docs. 15554 15555 - Config: export_formats 15556 - Env Var: RCLONE_DRIVE_EXPORT_FORMATS 15557 - Type: string 15558 - Default: "docx,xlsx,pptx,svg" 15559 15560 #### --drive-import-formats 15561 15562 Comma separated list of preferred formats for uploading Google docs. 15563 15564 - Config: import_formats 15565 - Env Var: RCLONE_DRIVE_IMPORT_FORMATS 15566 - Type: string 15567 - Default: "" 15568 15569 #### --drive-allow-import-name-change 15570 15571 Allow the filetype to change when uploading Google docs (e.g. file.doc to file.docx). This will confuse sync and reupload every time. 15572 15573 - Config: allow_import_name_change 15574 - Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE 15575 - Type: bool 15576 - Default: false 15577 15578 #### --drive-use-created-date 15579 15580 Use file created date instead of modified date., 15581 15582 Useful when downloading data and you want the creation date used in 15583 place of the last modified date. 15584 15585 **WARNING**: This flag may have some unexpected consequences. 15586 15587 When uploading to your drive all files will be overwritten unless they 15588 haven't been modified since their creation. And the inverse will occur 15589 while downloading. This side effect can be avoided by using the 15590 "--checksum" flag. 15591 15592 This feature was implemented to retain photos capture date as recorded 15593 by google photos. You will first need to check the "Create a Google 15594 Photos folder" option in your google drive settings. You can then copy 15595 or move the photos locally and use the date the image was taken 15596 (created) set as the modification date. 15597 15598 - Config: use_created_date 15599 - Env Var: RCLONE_DRIVE_USE_CREATED_DATE 15600 - Type: bool 15601 - Default: false 15602 15603 #### --drive-use-shared-date 15604 15605 Use date file was shared instead of modified date. 15606 15607 Note that, as with "--drive-use-created-date", this flag may have 15608 unexpected consequences when uploading/downloading files. 15609 15610 If both this flag and "--drive-use-created-date" are set, the created 15611 date is used. 15612 15613 - Config: use_shared_date 15614 - Env Var: RCLONE_DRIVE_USE_SHARED_DATE 15615 - Type: bool 15616 - Default: false 15617 15618 #### --drive-list-chunk 15619 15620 Size of listing chunk 100-1000. 0 to disable. 15621 15622 - Config: list_chunk 15623 - Env Var: RCLONE_DRIVE_LIST_CHUNK 15624 - Type: int 15625 - Default: 1000 15626 15627 #### --drive-impersonate 15628 15629 Impersonate this user when using a service account. 15630 15631 - Config: impersonate 15632 - Env Var: RCLONE_DRIVE_IMPERSONATE 15633 - Type: string 15634 - Default: "" 15635 15636 #### --drive-alternate-export 15637 15638 Use alternate export URLs for google documents export., 15639 15640 If this option is set this instructs rclone to use an alternate set of 15641 export URLs for drive documents. Users have reported that the 15642 official export URLs can't export large documents, whereas these 15643 unofficial ones can. 15644 15645 See rclone issue [#2243](https://github.com/rclone/rclone/issues/2243) for background, 15646 [this google drive issue](https://issuetracker.google.com/issues/36761333) and 15647 [this helpful post](https://www.labnol.org/internet/direct-links-for-google-drive/28356/). 15648 15649 - Config: alternate_export 15650 - Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT 15651 - Type: bool 15652 - Default: false 15653 15654 #### --drive-upload-cutoff 15655 15656 Cutoff for switching to chunked upload 15657 15658 - Config: upload_cutoff 15659 - Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF 15660 - Type: SizeSuffix 15661 - Default: 8M 15662 15663 #### --drive-chunk-size 15664 15665 Upload chunk size. Must a power of 2 >= 256k. 15666 15667 Making this larger will improve performance, but note that each chunk 15668 is buffered in memory one per transfer. 15669 15670 Reducing this will reduce memory usage but decrease performance. 15671 15672 - Config: chunk_size 15673 - Env Var: RCLONE_DRIVE_CHUNK_SIZE 15674 - Type: SizeSuffix 15675 - Default: 8M 15676 15677 #### --drive-acknowledge-abuse 15678 15679 Set to allow files which return cannotDownloadAbusiveFile to be downloaded. 15680 15681 If downloading a file returns the error "This file has been identified 15682 as malware or spam and cannot be downloaded" with the error code 15683 "cannotDownloadAbusiveFile" then supply this flag to rclone to 15684 indicate you acknowledge the risks of downloading the file and rclone 15685 will download it anyway. 15686 15687 - Config: acknowledge_abuse 15688 - Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE 15689 - Type: bool 15690 - Default: false 15691 15692 #### --drive-keep-revision-forever 15693 15694 Keep new head revision of each file forever. 15695 15696 - Config: keep_revision_forever 15697 - Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER 15698 - Type: bool 15699 - Default: false 15700 15701 #### --drive-size-as-quota 15702 15703 Show sizes as storage quota usage, not actual size. 15704 15705 Show the size of a file as the the storage quota used. This is the 15706 current version plus any older versions that have been set to keep 15707 forever. 15708 15709 **WARNING**: This flag may have some unexpected consequences. 15710 15711 It is not recommended to set this flag in your config - the 15712 recommended usage is using the flag form --drive-size-as-quota when 15713 doing rclone ls/lsl/lsf/lsjson/etc only. 15714 15715 If you do use this flag for syncing (not recommended) then you will 15716 need to use --ignore size also. 15717 15718 - Config: size_as_quota 15719 - Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA 15720 - Type: bool 15721 - Default: false 15722 15723 #### --drive-v2-download-min-size 15724 15725 If Object's are greater, use drive v2 API to download. 15726 15727 - Config: v2_download_min_size 15728 - Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE 15729 - Type: SizeSuffix 15730 - Default: off 15731 15732 #### --drive-pacer-min-sleep 15733 15734 Minimum time to sleep between API calls. 15735 15736 - Config: pacer_min_sleep 15737 - Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP 15738 - Type: Duration 15739 - Default: 100ms 15740 15741 #### --drive-pacer-burst 15742 15743 Number of API calls to allow without sleeping. 15744 15745 - Config: pacer_burst 15746 - Env Var: RCLONE_DRIVE_PACER_BURST 15747 - Type: int 15748 - Default: 100 15749 15750 #### --drive-server-side-across-configs 15751 15752 Allow server side operations (eg copy) to work across different drive configs. 15753 15754 This can be useful if you wish to do a server side copy between two 15755 different Google drives. Note that this isn't enabled by default 15756 because it isn't easy to tell if it will work between any two 15757 configurations. 15758 15759 - Config: server_side_across_configs 15760 - Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS 15761 - Type: bool 15762 - Default: false 15763 15764 #### --drive-disable-http2 15765 15766 Disable drive using http2 15767 15768 There is currently an unsolved issue with the google drive backend and 15769 HTTP/2. HTTP/2 is therefore disabled by default for the drive backend 15770 but can be re-enabled here. When the issue is solved this flag will 15771 be removed. 15772 15773 See: https://github.com/rclone/rclone/issues/3631 15774 15775 15776 15777 - Config: disable_http2 15778 - Env Var: RCLONE_DRIVE_DISABLE_HTTP2 15779 - Type: bool 15780 - Default: true 15781 15782 #### --drive-stop-on-upload-limit 15783 15784 Make upload limit errors be fatal 15785 15786 At the time of writing it is only possible to upload 750GB of data to 15787 Google Drive a day (this is an undocumented limit). When this limit is 15788 reached Google Drive produces a slightly different error message. When 15789 this flag is set it causes these errors to be fatal. These will stop 15790 the in-progress sync. 15791 15792 Note that this detection is relying on error message strings which 15793 Google don't document so it may break in the future. 15794 15795 See: https://github.com/rclone/rclone/issues/3857 15796 15797 15798 - Config: stop_on_upload_limit 15799 - Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT 15800 - Type: bool 15801 - Default: false 15802 15803 #### --drive-encoding 15804 15805 This sets the encoding for the backend. 15806 15807 See: the [encoding section in the overview](/overview/#encoding) for more info. 15808 15809 - Config: encoding 15810 - Env Var: RCLONE_DRIVE_ENCODING 15811 - Type: MultiEncoder 15812 - Default: InvalidUtf8 15813 15814 <!--- autogenerated options stop --> 15815 15816 ### Limitations ### 15817 15818 Drive has quite a lot of rate limiting. This causes rclone to be 15819 limited to transferring about 2 files per second only. Individual 15820 files may be transferred much faster at 100s of MBytes/s but lots of 15821 small files can take a long time. 15822 15823 Server side copies are also subject to a separate rate limit. If you 15824 see User rate limit exceeded errors, wait at least 24 hours and retry. 15825 You can disable server side copies with `--disable copy` to download 15826 and upload the files if you prefer. 15827 15828 #### Limitations of Google Docs #### 15829 15830 Google docs will appear as size -1 in `rclone ls` and as size 0 in 15831 anything which uses the VFS layer, eg `rclone mount`, `rclone serve`. 15832 15833 This is because rclone can't find out the size of the Google docs 15834 without downloading them. 15835 15836 Google docs will transfer correctly with `rclone sync`, `rclone copy` 15837 etc as rclone knows to ignore the size when doing the transfer. 15838 15839 However an unfortunate consequence of this is that you may not be able 15840 to download Google docs using `rclone mount`. If it doesn't work you 15841 will get a 0 sized file. If you try again the doc may gain its 15842 correct size and be downloadable. Whther it will work on not depends 15843 on the application accessing the mount and the OS you are running - 15844 experiment to find out if it does work for you! 15845 15846 ### Duplicated files ### 15847 15848 Sometimes, for no reason I've been able to track down, drive will 15849 duplicate a file that rclone uploads. Drive unlike all the other 15850 remotes can have duplicated files. 15851 15852 Duplicated files cause problems with the syncing and you will see 15853 messages in the log about duplicates. 15854 15855 Use `rclone dedupe` to fix duplicated files. 15856 15857 Note that this isn't just a problem with rclone, even Google Photos on 15858 Android duplicates files on drive sometimes. 15859 15860 ### Rclone appears to be re-copying files it shouldn't ### 15861 15862 The most likely cause of this is the duplicated file issue above - run 15863 `rclone dedupe` and check your logs for duplicate object or directory 15864 messages. 15865 15866 This can also be caused by a delay/caching on google drive's end when 15867 comparing directory listings. Specifically with team drives used in 15868 combination with --fast-list. Files that were uploaded recently may 15869 not appear on the directory list sent to rclone when using --fast-list. 15870 15871 Waiting a moderate period of time between attempts (estimated to be 15872 approximately 1 hour) and/or not using --fast-list both seem to be 15873 effective in preventing the problem. 15874 15875 ### Making your own client_id ### 15876 15877 When you use rclone with Google drive in its default configuration you 15878 are using rclone's client_id. This is shared between all the rclone 15879 users. There is a global rate limit on the number of queries per 15880 second that each client_id can do set by Google. rclone already has a 15881 high quota and I will continue to make sure it is high enough by 15882 contacting Google. 15883 15884 It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower. 15885 15886 Here is how to create your own Google Drive client ID for rclone: 15887 15888 1. Log into the [Google API 15889 Console](https://console.developers.google.com/) with your Google 15890 account. It doesn't matter what Google account you use. (It need not 15891 be the same account as the Google Drive you want to access) 15892 15893 2. Select a project or create a new project. 15894 15895 3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the 15896 "Google Drive API". 15897 15898 4. Click "Credentials" in the left-side panel (not "Create 15899 credentials", which opens the wizard), then "Create credentials", then 15900 "OAuth client ID". It will prompt you to set the OAuth consent screen 15901 product name, if you haven't set one already. 15902 15903 5. Choose an application type of "other", and click "Create". (the 15904 default name is fine) 15905 15906 6. It will show you a client ID and client secret. Use these values 15907 in rclone config to add a new remote or edit an existing remote. 15908 15909 (Thanks to @balazer on github for these instructions.) 15910 15911 Google Photos 15912 ------------------------------------------------- 15913 15914 The rclone backend for [Google Photos](https://www.google.com/photos/about/) is 15915 a specialized backend for transferring photos and videos to and from 15916 Google Photos. 15917 15918 **NB** The Google Photos API which rclone uses has quite a few 15919 limitations, so please read the [limitations section](#limitations) 15920 carefully to make sure it is suitable for your use. 15921 15922 ## Configuring Google Photos 15923 15924 The initial setup for google cloud storage involves getting a token from Google Photos 15925 which you need to do in your browser. `rclone config` walks you 15926 through it. 15927 15928 Here is an example of how to make a remote called `remote`. First run: 15929 15930 rclone config 15931 15932 This will guide you through an interactive setup process: 15933 15934 ``` 15935 No remotes found - make a new one 15936 n) New remote 15937 s) Set configuration password 15938 q) Quit config 15939 n/s/q> n 15940 name> remote 15941 Type of storage to configure. 15942 Enter a string value. Press Enter for the default (""). 15943 Choose a number from below, or type in your own value 15944 [snip] 15945 XX / Google Photos 15946 \ "google photos" 15947 [snip] 15948 Storage> google photos 15949 ** See help for google photos backend at: https://rclone.org/googlephotos/ ** 15950 15951 Google Application Client Id 15952 Leave blank normally. 15953 Enter a string value. Press Enter for the default (""). 15954 client_id> 15955 Google Application Client Secret 15956 Leave blank normally. 15957 Enter a string value. Press Enter for the default (""). 15958 client_secret> 15959 Set to make the Google Photos backend read only. 15960 15961 If you choose read only then rclone will only request read only access 15962 to your photos, otherwise rclone will request full access. 15963 Enter a boolean value (true or false). Press Enter for the default ("false"). 15964 read_only> 15965 Edit advanced config? (y/n) 15966 y) Yes 15967 n) No 15968 y/n> n 15969 Remote config 15970 Use auto config? 15971 * Say Y if not sure 15972 * Say N if you are working on a remote or headless machine 15973 y) Yes 15974 n) No 15975 y/n> y 15976 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 15977 Log in and authorize rclone for access 15978 Waiting for code... 15979 Got code 15980 15981 *** IMPORTANT: All media items uploaded to Google Photos with rclone 15982 *** are stored in full resolution at original quality. These uploads 15983 *** will count towards storage in your Google Account. 15984 15985 -------------------- 15986 [remote] 15987 type = google photos 15988 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} 15989 -------------------- 15990 y) Yes this is OK 15991 e) Edit this remote 15992 d) Delete this remote 15993 y/e/d> y 15994 ``` 15995 15996 Note that rclone runs a webserver on your local machine to collect the 15997 token as returned from Google if you use auto config mode. This only 15998 runs from the moment it opens your browser to the moment you get back 15999 the verification code. This is on `http://127.0.0.1:53682/` and this 16000 may require you to unblock it temporarily if you are running a host 16001 firewall, or use manual mode. 16002 16003 This remote is called `remote` and can now be used like this 16004 16005 See all the albums in your photos 16006 16007 rclone lsd remote:album 16008 16009 Make a new album 16010 16011 rclone mkdir remote:album/newAlbum 16012 16013 List the contents of an album 16014 16015 rclone ls remote:album/newAlbum 16016 16017 Sync `/home/local/images` to the Google Photos, removing any excess 16018 files in the album. 16019 16020 rclone sync /home/local/image remote:album/newAlbum 16021 16022 ## Layout 16023 16024 As Google Photos is not a general purpose cloud storage system the 16025 backend is laid out to help you navigate it. 16026 16027 The directories under `media` show different ways of categorizing the 16028 media. Each file will appear multiple times. So if you want to make 16029 a backup of your google photos you might choose to backup 16030 `remote:media/by-month`. (**NB** `remote:media/by-day` is rather slow 16031 at the moment so avoid for syncing.) 16032 16033 Note that all your photos and videos will appear somewhere under 16034 `media`, but they may not appear under `album` unless you've put them 16035 into albums. 16036 16037 ``` 16038 / 16039 - upload 16040 - file1.jpg 16041 - file2.jpg 16042 - ... 16043 - media 16044 - all 16045 - file1.jpg 16046 - file2.jpg 16047 - ... 16048 - by-year 16049 - 2000 16050 - file1.jpg 16051 - ... 16052 - 2001 16053 - file2.jpg 16054 - ... 16055 - ... 16056 - by-month 16057 - 2000 16058 - 2000-01 16059 - file1.jpg 16060 - ... 16061 - 2000-02 16062 - file2.jpg 16063 - ... 16064 - ... 16065 - by-day 16066 - 2000 16067 - 2000-01-01 16068 - file1.jpg 16069 - ... 16070 - 2000-01-02 16071 - file2.jpg 16072 - ... 16073 - ... 16074 - album 16075 - album name 16076 - album name/sub 16077 - shared-album 16078 - album name 16079 - album name/sub 16080 ``` 16081 16082 There are two writable parts of the tree, the `upload` directory and 16083 sub directories of the the `album` directory. 16084 16085 The `upload` directory is for uploading files you don't want to put 16086 into albums. This will be empty to start with and will contain the 16087 files you've uploaded for one rclone session only, becoming empty 16088 again when you restart rclone. The use case for this would be if you 16089 have a load of files you just want to once off dump into Google 16090 Photos. For repeated syncing, uploading to `album` will work better. 16091 16092 Directories within the `album` directory are also writeable and you 16093 may create new directories (albums) under `album`. If you copy files 16094 with a directory hierarchy in there then rclone will create albums 16095 with the `/` character in them. For example if you do 16096 16097 rclone copy /path/to/images remote:album/images 16098 16099 and the images directory contains 16100 16101 ``` 16102 images 16103 - file1.jpg 16104 dir 16105 file2.jpg 16106 dir2 16107 dir3 16108 file3.jpg 16109 ``` 16110 16111 Then rclone will create the following albums with the following files in 16112 16113 - images 16114 - file1.jpg 16115 - images/dir 16116 - file2.jpg 16117 - images/dir2/dir3 16118 - file3.jpg 16119 16120 This means that you can use the `album` path pretty much like a normal 16121 filesystem and it is a good target for repeated syncing. 16122 16123 The `shared-album` directory shows albums shared with you or by you. 16124 This is similar to the Sharing tab in the Google Photos web interface. 16125 16126 ## Limitations 16127 16128 Only images and videos can be uploaded. If you attempt to upload non 16129 videos or images or formats that Google Photos doesn't understand, 16130 rclone will upload the file, then Google Photos will give an error 16131 when it is put turned into a media item. 16132 16133 Note that all media items uploaded to Google Photos through the API 16134 are stored in full resolution at "original quality" and **will** count 16135 towards your storage quota in your Google Account. The API does 16136 **not** offer a way to upload in "high quality" mode.. 16137 16138 ### Downloading Images 16139 16140 When Images are downloaded this strips EXIF location (according to the 16141 docs and my tests). This is a limitation of the Google Photos API and 16142 is covered by [bug #112096115](https://issuetracker.google.com/issues/112096115). 16143 16144 **The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort** 16145 16146 ### Downloading Videos 16147 16148 When videos are downloaded they are downloaded in a really compressed 16149 version of the video compared to downloading it via the Google Photos 16150 web interface. This is covered by [bug #113672044](https://issuetracker.google.com/issues/113672044). 16151 16152 ### Duplicates 16153 16154 If a file name is duplicated in a directory then rclone will add the 16155 file ID into its name. So two files called `file.jpg` would then 16156 appear as `file {123456}.jpg` and `file {ABCDEF}.jpg` (the actual IDs 16157 are a lot longer alas!). 16158 16159 If you upload the same image (with the same binary data) twice then 16160 Google Photos will deduplicate it. However it will retain the 16161 filename from the first upload which may confuse rclone. For example 16162 if you uploaded an image to `upload` then uploaded the same image to 16163 `album/my_album` the filename of the image in `album/my_album` will be 16164 what it was uploaded with initially, not what you uploaded it with to 16165 `album`. In practise this shouldn't cause too many problems. 16166 16167 ### Modified time 16168 16169 The date shown of media in Google Photos is the creation date as 16170 determined by the EXIF information, or the upload date if that is not 16171 known. 16172 16173 This is not changeable by rclone and is not the modification date of 16174 the media on local disk. This means that rclone cannot use the dates 16175 from Google Photos for syncing purposes. 16176 16177 ### Size 16178 16179 The Google Photos API does not return the size of media. This means 16180 that when syncing to Google Photos, rclone can only do a file 16181 existence check. 16182 16183 It is possible to read the size of the media, but this needs an extra 16184 HTTP HEAD request per media item so is **very slow** and uses up a lot of 16185 transactions. This can be enabled with the `--gphotos-read-size` 16186 option or the `read_size = true` config parameter. 16187 16188 If you want to use the backend with `rclone mount` you may need to 16189 enable this flag (depending on your OS and application using the 16190 photos) otherwise you may not be able to read media off the mount. 16191 You'll need to experiment to see if it works for you without the flag. 16192 16193 ### Albums 16194 16195 Rclone can only upload files to albums it created. This is a 16196 [limitation of the Google Photos API](https://developers.google.com/photos/library/guides/manage-albums). 16197 16198 Rclone can remove files it uploaded from albums it created only. 16199 16200 ### Deleting files 16201 16202 Rclone can remove files from albums it created, but note that the 16203 Google Photos API does not allow media to be deleted permanently so 16204 this media will still remain. See [bug #109759781](https://issuetracker.google.com/issues/109759781). 16205 16206 Rclone cannot delete files anywhere except under `album`. 16207 16208 ### Deleting albums 16209 16210 The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733). 16211 16212 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/googlephotos/googlephotos.go then run make backenddocs --> 16213 ### Standard Options 16214 16215 Here are the standard options specific to google photos (Google Photos). 16216 16217 #### --gphotos-client-id 16218 16219 Google Application Client Id 16220 Leave blank normally. 16221 16222 - Config: client_id 16223 - Env Var: RCLONE_GPHOTOS_CLIENT_ID 16224 - Type: string 16225 - Default: "" 16226 16227 #### --gphotos-client-secret 16228 16229 Google Application Client Secret 16230 Leave blank normally. 16231 16232 - Config: client_secret 16233 - Env Var: RCLONE_GPHOTOS_CLIENT_SECRET 16234 - Type: string 16235 - Default: "" 16236 16237 #### --gphotos-read-only 16238 16239 Set to make the Google Photos backend read only. 16240 16241 If you choose read only then rclone will only request read only access 16242 to your photos, otherwise rclone will request full access. 16243 16244 - Config: read_only 16245 - Env Var: RCLONE_GPHOTOS_READ_ONLY 16246 - Type: bool 16247 - Default: false 16248 16249 ### Advanced Options 16250 16251 Here are the advanced options specific to google photos (Google Photos). 16252 16253 #### --gphotos-read-size 16254 16255 Set to read the size of media items. 16256 16257 Normally rclone does not read the size of media items since this takes 16258 another transaction. This isn't necessary for syncing. However 16259 rclone mount needs to know the size of files in advance of reading 16260 them, so setting this flag when using rclone mount is recommended if 16261 you want to read the media. 16262 16263 - Config: read_size 16264 - Env Var: RCLONE_GPHOTOS_READ_SIZE 16265 - Type: bool 16266 - Default: false 16267 16268 <!--- autogenerated options stop --> 16269 16270 HTTP 16271 ------------------------------------------------- 16272 16273 The HTTP remote is a read only remote for reading files of a 16274 webserver. The webserver should provide file listings which rclone 16275 will read and turn into a remote. This has been tested with common 16276 webservers such as Apache/Nginx/Caddy and will likely work with file 16277 listings from most web servers. (If it doesn't then please file an 16278 issue, or send a pull request!) 16279 16280 Paths are specified as `remote:` or `remote:path/to/dir`. 16281 16282 Here is an example of how to make a remote called `remote`. First 16283 run: 16284 16285 rclone config 16286 16287 This will guide you through an interactive setup process: 16288 16289 ``` 16290 No remotes found - make a new one 16291 n) New remote 16292 s) Set configuration password 16293 q) Quit config 16294 n/s/q> n 16295 name> remote 16296 Type of storage to configure. 16297 Choose a number from below, or type in your own value 16298 [snip] 16299 XX / http Connection 16300 \ "http" 16301 [snip] 16302 Storage> http 16303 URL of http host to connect to 16304 Choose a number from below, or type in your own value 16305 1 / Connect to example.com 16306 \ "https://example.com" 16307 url> https://beta.rclone.org 16308 Remote config 16309 -------------------- 16310 [remote] 16311 url = https://beta.rclone.org 16312 -------------------- 16313 y) Yes this is OK 16314 e) Edit this remote 16315 d) Delete this remote 16316 y/e/d> y 16317 Current remotes: 16318 16319 Name Type 16320 ==== ==== 16321 remote http 16322 16323 e) Edit existing remote 16324 n) New remote 16325 d) Delete remote 16326 r) Rename remote 16327 c) Copy remote 16328 s) Set configuration password 16329 q) Quit config 16330 e/n/d/r/c/s/q> q 16331 ``` 16332 16333 This remote is called `remote` and can now be used like this 16334 16335 See all the top level directories 16336 16337 rclone lsd remote: 16338 16339 List the contents of a directory 16340 16341 rclone ls remote:directory 16342 16343 Sync the remote `directory` to `/home/local/directory`, deleting any excess files. 16344 16345 rclone sync remote:directory /home/local/directory 16346 16347 ### Read only ### 16348 16349 This remote is read only - you can't upload files to an HTTP server. 16350 16351 ### Modified time ### 16352 16353 Most HTTP servers store time accurate to 1 second. 16354 16355 ### Checksum ### 16356 16357 No checksums are stored. 16358 16359 ### Usage without a config file ### 16360 16361 Since the http remote only has one config parameter it is easy to use 16362 without a config file: 16363 16364 rclone lsd --http-url https://beta.rclone.org :http: 16365 16366 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/http/http.go then run make backenddocs --> 16367 ### Standard Options 16368 16369 Here are the standard options specific to http (http Connection). 16370 16371 #### --http-url 16372 16373 URL of http host to connect to 16374 16375 - Config: url 16376 - Env Var: RCLONE_HTTP_URL 16377 - Type: string 16378 - Default: "" 16379 - Examples: 16380 - "https://example.com" 16381 - Connect to example.com 16382 - "https://user:pass@example.com" 16383 - Connect to example.com using a username and password 16384 16385 ### Advanced Options 16386 16387 Here are the advanced options specific to http (http Connection). 16388 16389 #### --http-headers 16390 16391 Set HTTP headers for all transactions 16392 16393 Use this to set additional HTTP headers for all transactions 16394 16395 The input format is comma separated list of key,value pairs. Standard 16396 [CSV encoding](https://godoc.org/encoding/csv) may be used. 16397 16398 For example to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'. 16399 16400 You can set multiple headers, eg '"Cookie","name=value","Authorization","xxx"'. 16401 16402 16403 - Config: headers 16404 - Env Var: RCLONE_HTTP_HEADERS 16405 - Type: CommaSepList 16406 - Default: 16407 16408 #### --http-no-slash 16409 16410 Set this if the site doesn't end directories with / 16411 16412 Use this if your target website does not use / on the end of 16413 directories. 16414 16415 A / on the end of a path is how rclone normally tells the difference 16416 between files and directories. If this flag is set, then rclone will 16417 treat all files with Content-Type: text/html as directories and read 16418 URLs from them rather than downloading them. 16419 16420 Note that this may cause rclone to confuse genuine HTML files with 16421 directories. 16422 16423 - Config: no_slash 16424 - Env Var: RCLONE_HTTP_NO_SLASH 16425 - Type: bool 16426 - Default: false 16427 16428 #### --http-no-head 16429 16430 Don't use HEAD requests to find file sizes in dir listing 16431 16432 If your site is being very slow to load then you can try this option. 16433 Normally rclone does a HEAD request for each potential file in a 16434 directory listing to: 16435 16436 - find its size 16437 - check it really exists 16438 - check to see if it is a directory 16439 16440 If you set this option, rclone will not do the HEAD request. This will mean 16441 16442 - directory listings are much quicker 16443 - rclone won't have the times or sizes of any files 16444 - some files that don't exist may be in the listing 16445 16446 16447 - Config: no_head 16448 - Env Var: RCLONE_HTTP_NO_HEAD 16449 - Type: bool 16450 - Default: false 16451 16452 <!--- autogenerated options stop --> 16453 16454 Hubic 16455 ----------------------------------------- 16456 16457 Paths are specified as `remote:path` 16458 16459 Paths are specified as `remote:container` (or `remote:` for the `lsd` 16460 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 16461 16462 The initial setup for Hubic involves getting a token from Hubic which 16463 you need to do in your browser. `rclone config` walks you through it. 16464 16465 Here is an example of how to make a remote called `remote`. First run: 16466 16467 rclone config 16468 16469 This will guide you through an interactive setup process: 16470 16471 ``` 16472 n) New remote 16473 s) Set configuration password 16474 n/s> n 16475 name> remote 16476 Type of storage to configure. 16477 Choose a number from below, or type in your own value 16478 [snip] 16479 XX / Hubic 16480 \ "hubic" 16481 [snip] 16482 Storage> hubic 16483 Hubic Client Id - leave blank normally. 16484 client_id> 16485 Hubic Client Secret - leave blank normally. 16486 client_secret> 16487 Remote config 16488 Use auto config? 16489 * Say Y if not sure 16490 * Say N if you are working on a remote or headless machine 16491 y) Yes 16492 n) No 16493 y/n> y 16494 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 16495 Log in and authorize rclone for access 16496 Waiting for code... 16497 Got code 16498 -------------------- 16499 [remote] 16500 client_id = 16501 client_secret = 16502 token = {"access_token":"XXXXXX"} 16503 -------------------- 16504 y) Yes this is OK 16505 e) Edit this remote 16506 d) Delete this remote 16507 y/e/d> y 16508 ``` 16509 16510 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 16511 machine with no Internet browser available. 16512 16513 Note that rclone runs a webserver on your local machine to collect the 16514 token as returned from Hubic. This only runs from the moment it opens 16515 your browser to the moment you get back the verification code. This 16516 is on `http://127.0.0.1:53682/` and this it may require you to unblock 16517 it temporarily if you are running a host firewall. 16518 16519 Once configured you can then use `rclone` like this, 16520 16521 List containers in the top level of your Hubic 16522 16523 rclone lsd remote: 16524 16525 List all the files in your Hubic 16526 16527 rclone ls remote: 16528 16529 To copy a local directory to an Hubic directory called backup 16530 16531 rclone copy /home/source remote:backup 16532 16533 If you want the directory to be visible in the official *Hubic 16534 browser*, you need to copy your files to the `default` directory 16535 16536 rclone copy /home/source remote:default/backup 16537 16538 ### --fast-list ### 16539 16540 This remote supports `--fast-list` which allows you to use fewer 16541 transactions in exchange for more memory. See the [rclone 16542 docs](/docs/#fast-list) for more details. 16543 16544 ### Modified time ### 16545 16546 The modified time is stored as metadata on the object as 16547 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 16548 ns. 16549 16550 This is a de facto standard (used in the official python-swiftclient 16551 amongst others) for storing the modification time for an object. 16552 16553 Note that Hubic wraps the Swift backend, so most of the properties of 16554 are the same. 16555 16556 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/hubic/hubic.go then run make backenddocs --> 16557 ### Standard Options 16558 16559 Here are the standard options specific to hubic (Hubic). 16560 16561 #### --hubic-client-id 16562 16563 Hubic Client Id 16564 Leave blank normally. 16565 16566 - Config: client_id 16567 - Env Var: RCLONE_HUBIC_CLIENT_ID 16568 - Type: string 16569 - Default: "" 16570 16571 #### --hubic-client-secret 16572 16573 Hubic Client Secret 16574 Leave blank normally. 16575 16576 - Config: client_secret 16577 - Env Var: RCLONE_HUBIC_CLIENT_SECRET 16578 - Type: string 16579 - Default: "" 16580 16581 ### Advanced Options 16582 16583 Here are the advanced options specific to hubic (Hubic). 16584 16585 #### --hubic-chunk-size 16586 16587 Above this size files will be chunked into a _segments container. 16588 16589 Above this size files will be chunked into a _segments container. The 16590 default for this is 5GB which is its maximum value. 16591 16592 - Config: chunk_size 16593 - Env Var: RCLONE_HUBIC_CHUNK_SIZE 16594 - Type: SizeSuffix 16595 - Default: 5G 16596 16597 #### --hubic-no-chunk 16598 16599 Don't chunk files during streaming upload. 16600 16601 When doing streaming uploads (eg using rcat or mount) setting this 16602 flag will cause the swift backend to not upload chunked files. 16603 16604 This will limit the maximum upload size to 5GB. However non chunked 16605 files are easier to deal with and have an MD5SUM. 16606 16607 Rclone will still chunk files bigger than chunk_size when doing normal 16608 copy operations. 16609 16610 - Config: no_chunk 16611 - Env Var: RCLONE_HUBIC_NO_CHUNK 16612 - Type: bool 16613 - Default: false 16614 16615 #### --hubic-encoding 16616 16617 This sets the encoding for the backend. 16618 16619 See: the [encoding section in the overview](/overview/#encoding) for more info. 16620 16621 - Config: encoding 16622 - Env Var: RCLONE_HUBIC_ENCODING 16623 - Type: MultiEncoder 16624 - Default: Slash,InvalidUtf8 16625 16626 <!--- autogenerated options stop --> 16627 16628 ### Limitations ### 16629 16630 This uses the normal OpenStack Swift mechanism to refresh the Swift 16631 API credentials and ignores the expires field returned by the Hubic 16632 API. 16633 16634 The Swift API doesn't return a correct MD5SUM for segmented files 16635 (Dynamic or Static Large Objects) so rclone won't check or use the 16636 MD5SUM for these. 16637 16638 Jottacloud 16639 ----------------------------------------- 16640 16641 Paths are specified as `remote:path` 16642 16643 Paths may be as deep as required, eg `remote:directory/subdirectory`. 16644 16645 To configure Jottacloud you will need to generate a personal security token in the Jottacloud web inteface. You will the option to do in your [account security settings](https://www.jottacloud.com/web/secure). Note that the web inteface may refer to this token as a JottaCli token. 16646 16647 Here is an example of how to make a remote called `remote`. First run: 16648 16649 rclone config 16650 16651 This will guide you through an interactive setup process: 16652 16653 ``` 16654 No remotes found - make a new one 16655 n) New remote 16656 s) Set configuration password 16657 q) Quit config 16658 n/s/q> n 16659 name> jotta 16660 Type of storage to configure. 16661 Enter a string value. Press Enter for the default (""). 16662 Choose a number from below, or type in your own value 16663 [snip] 16664 XX / JottaCloud 16665 \ "jottacloud" 16666 [snip] 16667 Storage> jottacloud 16668 ** See help for jottacloud backend at: https://rclone.org/jottacloud/ ** 16669 16670 Edit advanced config? (y/n) 16671 y) Yes 16672 n) No 16673 y/n> n 16674 Remote config 16675 16676 Generate a personal login token here: https://www.jottacloud.com/web/secure 16677 Login Token> <your token here> 16678 16679 Do you want to use a non standard device/mountpoint e.g. for accessing files uploaded using the official Jottacloud client? 16680 16681 y) Yes 16682 n) No 16683 y/n> y 16684 Please select the device to use. Normally this will be Jotta 16685 Choose a number from below, or type in an existing value 16686 1 > DESKTOP-3H31129 16687 2 > fla1 16688 3 > Jotta 16689 Devices> 3 16690 Please select the mountpoint to user. Normally this will be Archive 16691 Choose a number from below, or type in an existing value 16692 1 > Archive 16693 2 > Shared 16694 3 > Sync 16695 Mountpoints> 1 16696 -------------------- 16697 [jotta] 16698 type = jottacloud 16699 user = 0xC4KE@gmail.com 16700 token = {........} 16701 device = Jotta 16702 mountpoint = Archive 16703 configVersion = 1 16704 -------------------- 16705 y) Yes this is OK 16706 e) Edit this remote 16707 d) Delete this remote 16708 y/e/d> y 16709 ``` 16710 Once configured you can then use `rclone` like this, 16711 16712 List directories in top level of your Jottacloud 16713 16714 rclone lsd remote: 16715 16716 List all the files in your Jottacloud 16717 16718 rclone ls remote: 16719 16720 To copy a local directory to an Jottacloud directory called backup 16721 16722 rclone copy /home/source remote:backup 16723 16724 ### Devices and Mountpoints ### 16725 16726 The official Jottacloud client registers a device for each computer you install it on and then creates a mountpoint for each folder you select for Backup. 16727 The web interface uses a special device called Jotta for the Archive, Sync and Shared mountpoints. In most cases you'll want to use the Jotta/Archive device/mounpoint however if you want to access files uploaded by any of the official clients rclone provides the option to select other devices and mountpoints during config. 16728 16729 ### --fast-list ### 16730 16731 This remote supports `--fast-list` which allows you to use fewer 16732 transactions in exchange for more memory. See the [rclone 16733 docs](/docs/#fast-list) for more details. 16734 16735 Note that the implementation in Jottacloud always uses only a single 16736 API request to get the entire list, so for large folders this could 16737 lead to long wait time before the first results are shown. 16738 16739 ### Modified time and hashes ### 16740 16741 Jottacloud allows modification times to be set on objects accurate to 1 16742 second. These will be used to detect whether objects need syncing or 16743 not. 16744 16745 Jottacloud supports MD5 type hashes, so you can use the `--checksum` 16746 flag. 16747 16748 Note that Jottacloud requires the MD5 hash before upload so if the 16749 source does not have an MD5 checksum then the file will be cached 16750 temporarily on disk (wherever the `TMPDIR` environment variable points 16751 to) before it is uploaded. Small files will be cached in memory - see 16752 the `--jottacloud-md5-memory-limit` flag. 16753 16754 #### Restricted filename characters 16755 16756 In addition to the [default restricted characters set](/overview/#restricted-characters) 16757 the following characters are also replaced: 16758 16759 | Character | Value | Replacement | 16760 | --------- |:-----:|:-----------:| 16761 | " | 0x22 | " | 16762 | * | 0x2A | * | 16763 | : | 0x3A | : | 16764 | < | 0x3C | < | 16765 | > | 0x3E | > | 16766 | ? | 0x3F | ? | 16767 | \| | 0x7C | | | 16768 16769 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 16770 as they can't be used in XML strings. 16771 16772 ### Deleting files ### 16773 16774 By default rclone will send all files to the trash when deleting files. 16775 Due to a lack of API documentation emptying the trash is currently 16776 only possible via the Jottacloud website. If deleting permanently 16777 is required then use the `--jottacloud-hard-delete` flag, 16778 or set the equivalent environment variable. 16779 16780 ### Versions ### 16781 16782 Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it. Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website. 16783 16784 ### Quota information ### 16785 16786 To view your current quota you can use the `rclone about remote:` 16787 command which will display your usage limit (unless it is unlimited) 16788 and the current usage. 16789 16790 ### Device IDs ### 16791 16792 Jottacloud requires each 'device' to be registered. Rclone brings such a registration to easily access your account but if you want to use Jottacloud together with rclone on multiple machines you NEED to create a seperate deviceID/deviceSecrect on each machine. You will asked during setting up the remote. Please be aware that this also means that copying the rclone config from one machine to another does NOT work with Jottacloud accounts. You have to create it on each machine. 16793 16794 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/jottacloud/jottacloud.go then run make backenddocs --> 16795 ### Advanced Options 16796 16797 Here are the advanced options specific to jottacloud (JottaCloud). 16798 16799 #### --jottacloud-md5-memory-limit 16800 16801 Files bigger than this will be cached on disk to calculate the MD5 if required. 16802 16803 - Config: md5_memory_limit 16804 - Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT 16805 - Type: SizeSuffix 16806 - Default: 10M 16807 16808 #### --jottacloud-hard-delete 16809 16810 Delete files permanently rather than putting them into the trash. 16811 16812 - Config: hard_delete 16813 - Env Var: RCLONE_JOTTACLOUD_HARD_DELETE 16814 - Type: bool 16815 - Default: false 16816 16817 #### --jottacloud-unlink 16818 16819 Remove existing public link to file/folder with link command rather than creating. 16820 Default is false, meaning link command will create or retrieve public link. 16821 16822 - Config: unlink 16823 - Env Var: RCLONE_JOTTACLOUD_UNLINK 16824 - Type: bool 16825 - Default: false 16826 16827 #### --jottacloud-upload-resume-limit 16828 16829 Files bigger than this can be resumed if the upload fail's. 16830 16831 - Config: upload_resume_limit 16832 - Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT 16833 - Type: SizeSuffix 16834 - Default: 10M 16835 16836 #### --jottacloud-encoding 16837 16838 This sets the encoding for the backend. 16839 16840 See: the [encoding section in the overview](/overview/#encoding) for more info. 16841 16842 - Config: encoding 16843 - Env Var: RCLONE_JOTTACLOUD_ENCODING 16844 - Type: MultiEncoder 16845 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot 16846 16847 <!--- autogenerated options stop --> 16848 16849 ### Limitations ### 16850 16851 Note that Jottacloud is case insensitive so you can't have a file called 16852 "Hello.doc" and one called "hello.doc". 16853 16854 There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical looking unicode equivalent. For example if a file has a ? in it will be mapped to ? instead. 16855 16856 Jottacloud only supports filenames up to 255 characters in length. 16857 16858 ### Troubleshooting ### 16859 16860 Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove operations to previously deleted paths to fail. Emptying the trash should help in such cases. 16861 16862 Koofr 16863 ----------------------------------------- 16864 16865 Paths are specified as `remote:path` 16866 16867 Paths may be as deep as required, eg `remote:directory/subdirectory`. 16868 16869 The initial setup for Koofr involves creating an application password for 16870 rclone. You can do that by opening the Koofr 16871 [web application](https://app.koofr.net/app/admin/preferences/password), 16872 giving the password a nice name like `rclone` and clicking on generate. 16873 16874 Here is an example of how to make a remote called `koofr`. First run: 16875 16876 rclone config 16877 16878 This will guide you through an interactive setup process: 16879 16880 ``` 16881 No remotes found - make a new one 16882 n) New remote 16883 s) Set configuration password 16884 q) Quit config 16885 n/s/q> n 16886 name> koofr 16887 Type of storage to configure. 16888 Enter a string value. Press Enter for the default (""). 16889 Choose a number from below, or type in your own value 16890 [snip] 16891 XX / Koofr 16892 \ "koofr" 16893 [snip] 16894 Storage> koofr 16895 ** See help for koofr backend at: https://rclone.org/koofr/ ** 16896 16897 Your Koofr user name 16898 Enter a string value. Press Enter for the default (""). 16899 user> USER@NAME 16900 Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) 16901 y) Yes type in my own password 16902 g) Generate random password 16903 y/g> y 16904 Enter the password: 16905 password: 16906 Confirm the password: 16907 password: 16908 Edit advanced config? (y/n) 16909 y) Yes 16910 n) No 16911 y/n> n 16912 Remote config 16913 -------------------- 16914 [koofr] 16915 type = koofr 16916 baseurl = https://app.koofr.net 16917 user = USER@NAME 16918 password = *** ENCRYPTED *** 16919 -------------------- 16920 y) Yes this is OK 16921 e) Edit this remote 16922 d) Delete this remote 16923 y/e/d> y 16924 ``` 16925 16926 You can choose to edit advanced config in order to enter your own service URL 16927 if you use an on-premise or white label Koofr instance, or choose an alternative 16928 mount instead of your primary storage. 16929 16930 Once configured you can then use `rclone` like this, 16931 16932 List directories in top level of your Koofr 16933 16934 rclone lsd koofr: 16935 16936 List all the files in your Koofr 16937 16938 rclone ls koofr: 16939 16940 To copy a local directory to an Koofr directory called backup 16941 16942 rclone copy /home/source remote:backup 16943 16944 #### Restricted filename characters 16945 16946 In addition to the [default restricted characters set](/overview/#restricted-characters) 16947 the following characters are also replaced: 16948 16949 | Character | Value | Replacement | 16950 | --------- |:-----:|:-----------:| 16951 | \ | 0x5C | \ | 16952 16953 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 16954 as they can't be used in XML strings. 16955 16956 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/koofr/koofr.go then run make backenddocs --> 16957 ### Standard Options 16958 16959 Here are the standard options specific to koofr (Koofr). 16960 16961 #### --koofr-user 16962 16963 Your Koofr user name 16964 16965 - Config: user 16966 - Env Var: RCLONE_KOOFR_USER 16967 - Type: string 16968 - Default: "" 16969 16970 #### --koofr-password 16971 16972 Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) 16973 16974 - Config: password 16975 - Env Var: RCLONE_KOOFR_PASSWORD 16976 - Type: string 16977 - Default: "" 16978 16979 ### Advanced Options 16980 16981 Here are the advanced options specific to koofr (Koofr). 16982 16983 #### --koofr-endpoint 16984 16985 The Koofr API endpoint to use 16986 16987 - Config: endpoint 16988 - Env Var: RCLONE_KOOFR_ENDPOINT 16989 - Type: string 16990 - Default: "https://app.koofr.net" 16991 16992 #### --koofr-mountid 16993 16994 Mount ID of the mount to use. If omitted, the primary mount is used. 16995 16996 - Config: mountid 16997 - Env Var: RCLONE_KOOFR_MOUNTID 16998 - Type: string 16999 - Default: "" 17000 17001 #### --koofr-setmtime 17002 17003 Does the backend support setting modification time. Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend. 17004 17005 - Config: setmtime 17006 - Env Var: RCLONE_KOOFR_SETMTIME 17007 - Type: bool 17008 - Default: true 17009 17010 #### --koofr-encoding 17011 17012 This sets the encoding for the backend. 17013 17014 See: the [encoding section in the overview](/overview/#encoding) for more info. 17015 17016 - Config: encoding 17017 - Env Var: RCLONE_KOOFR_ENCODING 17018 - Type: MultiEncoder 17019 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 17020 17021 <!--- autogenerated options stop --> 17022 17023 ### Limitations ### 17024 17025 Note that Koofr is case insensitive so you can't have a file called 17026 "Hello.doc" and one called "hello.doc". 17027 17028 Mail.ru Cloud 17029 ---------------------------------------- 17030 17031 [Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/), available only on Windows. (Please note that official sites are in Russian) 17032 17033 ### Features highlights ### 17034 17035 - Paths may be as deep as required, eg `remote:directory/subdirectory` 17036 - Files have a `last modified time` property, directories don't 17037 - Deleted files are by default moved to the trash 17038 - Files and directories can be shared via public links 17039 - Partial uploads or streaming are not supported, file size must be known before upload 17040 - Maximum file size is limited to 2G for a free acount, unlimited for paid accounts 17041 - Storage keeps hash for all files and performs transparent deduplication, 17042 the hash algorithm is a modified SHA1 17043 - If a particular file is already present in storage, one can quickly submit file hash 17044 instead of long file upload (this optimization is supported by rclone) 17045 17046 ### Configuration ### 17047 17048 Here is an example of making a mailru configuration. First create a Mail.ru Cloud 17049 account and choose a tariff, then run 17050 17051 rclone config 17052 17053 This will guide you through an interactive setup process: 17054 17055 ``` 17056 No remotes found - make a new one 17057 n) New remote 17058 s) Set configuration password 17059 q) Quit config 17060 n/s/q> n 17061 name> remote 17062 Type of storage to configure. 17063 Type of storage to configure. 17064 Enter a string value. Press Enter for the default (""). 17065 Choose a number from below, or type in your own value 17066 [snip] 17067 XX / Mail.ru Cloud 17068 \ "mailru" 17069 [snip] 17070 Storage> mailru 17071 User name (usually email) 17072 Enter a string value. Press Enter for the default (""). 17073 user> username@mail.ru 17074 Password 17075 y) Yes type in my own password 17076 g) Generate random password 17077 y/g> y 17078 Enter the password: 17079 password: 17080 Confirm the password: 17081 password: 17082 Skip full upload if there is another file with same data hash. 17083 This feature is called "speedup" or "put by hash". It is especially efficient 17084 in case of generally available files like popular books, video or audio clips 17085 [snip] 17086 Enter a boolean value (true or false). Press Enter for the default ("true"). 17087 Choose a number from below, or type in your own value 17088 1 / Enable 17089 \ "true" 17090 2 / Disable 17091 \ "false" 17092 speedup_enable> 1 17093 Edit advanced config? (y/n) 17094 y) Yes 17095 n) No 17096 y/n> n 17097 Remote config 17098 -------------------- 17099 [remote] 17100 type = mailru 17101 user = username@mail.ru 17102 pass = *** ENCRYPTED *** 17103 speedup_enable = true 17104 -------------------- 17105 y) Yes this is OK 17106 e) Edit this remote 17107 d) Delete this remote 17108 y/e/d> y 17109 ``` 17110 17111 Configuration of this backend does not require a local web browser. 17112 You can use the configured backend as shown below: 17113 17114 See top level directories 17115 17116 rclone lsd remote: 17117 17118 Make a new directory 17119 17120 rclone mkdir remote:directory 17121 17122 List the contents of a directory 17123 17124 rclone ls remote:directory 17125 17126 Sync `/home/local/directory` to the remote path, deleting any 17127 excess files in the path. 17128 17129 rclone sync /home/local/directory remote:directory 17130 17131 ### Modified time ### 17132 17133 Files support a modification time attribute with up to 1 second precision. 17134 Directories do not have a modification time, which is shown as "Jan 1 1970". 17135 17136 ### Hash checksums ### 17137 17138 Hash sums use a custom Mail.ru algorithm based on SHA1. 17139 If file size is less than or equal to the SHA1 block size (20 bytes), 17140 its hash is simply its data right-padded with zero bytes. 17141 Hash sum of a larger file is computed as a SHA1 sum of the file data 17142 bytes concatenated with a decimal representation of the data length. 17143 17144 ### Emptying Trash ### 17145 17146 Removing a file or directory actually moves it to the trash, which is not 17147 visible to rclone but can be seen in a web browser. The trashed file 17148 still occupies part of total quota. If you wish to empty your trash 17149 and free some quota, you can use the `rclone cleanup remote:` command, 17150 which will permanently delete all your trashed files. 17151 This command does not take any path arguments. 17152 17153 ### Quota information ### 17154 17155 To view your current quota you can use the `rclone about remote:` 17156 command which will display your usage limit (quota) and the current usage. 17157 17158 #### Restricted filename characters 17159 17160 In addition to the [default restricted characters set](/overview/#restricted-characters) 17161 the following characters are also replaced: 17162 17163 | Character | Value | Replacement | 17164 | --------- |:-----:|:-----------:| 17165 | " | 0x22 | " | 17166 | * | 0x2A | * | 17167 | : | 0x3A | : | 17168 | < | 0x3C | < | 17169 | > | 0x3E | > | 17170 | ? | 0x3F | ? | 17171 | \ | 0x5C | \ | 17172 | \| | 0x7C | | | 17173 17174 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 17175 as they can't be used in JSON strings. 17176 17177 ### Limitations ### 17178 17179 File size limits depend on your account. A single file size is limited by 2G 17180 for a free account and unlimited for paid tariffs. Please refer to the Mail.ru 17181 site for the total uploaded size limits. 17182 17183 Note that Mailru is case insensitive so you can't have a file called 17184 "Hello.doc" and one called "hello.doc". 17185 17186 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/mailru/mailru.go then run make backenddocs --> 17187 ### Standard Options 17188 17189 Here are the standard options specific to mailru (Mail.ru Cloud). 17190 17191 #### --mailru-user 17192 17193 User name (usually email) 17194 17195 - Config: user 17196 - Env Var: RCLONE_MAILRU_USER 17197 - Type: string 17198 - Default: "" 17199 17200 #### --mailru-pass 17201 17202 Password 17203 17204 - Config: pass 17205 - Env Var: RCLONE_MAILRU_PASS 17206 - Type: string 17207 - Default: "" 17208 17209 #### --mailru-speedup-enable 17210 17211 Skip full upload if there is another file with same data hash. 17212 This feature is called "speedup" or "put by hash". It is especially efficient 17213 in case of generally available files like popular books, video or audio clips, 17214 because files are searched by hash in all accounts of all mailru users. 17215 Please note that rclone may need local memory and disk space to calculate 17216 content hash in advance and decide whether full upload is required. 17217 Also, if rclone does not know file size in advance (e.g. in case of 17218 streaming or partial uploads), it will not even try this optimization. 17219 17220 - Config: speedup_enable 17221 - Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE 17222 - Type: bool 17223 - Default: true 17224 - Examples: 17225 - "true" 17226 - Enable 17227 - "false" 17228 - Disable 17229 17230 ### Advanced Options 17231 17232 Here are the advanced options specific to mailru (Mail.ru Cloud). 17233 17234 #### --mailru-speedup-file-patterns 17235 17236 Comma separated list of file name patterns eligible for speedup (put by hash). 17237 Patterns are case insensitive and can contain '*' or '?' meta characters. 17238 17239 - Config: speedup_file_patterns 17240 - Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS 17241 - Type: string 17242 - Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf" 17243 - Examples: 17244 - "" 17245 - Empty list completely disables speedup (put by hash). 17246 - "*" 17247 - All files will be attempted for speedup. 17248 - "*.mkv,*.avi,*.mp4,*.mp3" 17249 - Only common audio/video files will be tried for put by hash. 17250 - "*.zip,*.gz,*.rar,*.pdf" 17251 - Only common archives or PDF books will be tried for speedup. 17252 17253 #### --mailru-speedup-max-disk 17254 17255 This option allows you to disable speedup (put by hash) for large files 17256 (because preliminary hashing can exhaust you RAM or disk space) 17257 17258 - Config: speedup_max_disk 17259 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK 17260 - Type: SizeSuffix 17261 - Default: 3G 17262 - Examples: 17263 - "0" 17264 - Completely disable speedup (put by hash). 17265 - "1G" 17266 - Files larger than 1Gb will be uploaded directly. 17267 - "3G" 17268 - Choose this option if you have less than 3Gb free on local disk. 17269 17270 #### --mailru-speedup-max-memory 17271 17272 Files larger than the size given below will always be hashed on disk. 17273 17274 - Config: speedup_max_memory 17275 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY 17276 - Type: SizeSuffix 17277 - Default: 32M 17278 - Examples: 17279 - "0" 17280 - Preliminary hashing will always be done in a temporary disk location. 17281 - "32M" 17282 - Do not dedicate more than 32Mb RAM for preliminary hashing. 17283 - "256M" 17284 - You have at most 256Mb RAM free for hash calculations. 17285 17286 #### --mailru-check-hash 17287 17288 What should copy do if file checksum is mismatched or invalid 17289 17290 - Config: check_hash 17291 - Env Var: RCLONE_MAILRU_CHECK_HASH 17292 - Type: bool 17293 - Default: true 17294 - Examples: 17295 - "true" 17296 - Fail with error. 17297 - "false" 17298 - Ignore and continue. 17299 17300 #### --mailru-user-agent 17301 17302 HTTP user agent used internally by client. 17303 Defaults to "rclone/VERSION" or "--user-agent" provided on command line. 17304 17305 - Config: user_agent 17306 - Env Var: RCLONE_MAILRU_USER_AGENT 17307 - Type: string 17308 - Default: "" 17309 17310 #### --mailru-quirks 17311 17312 Comma separated list of internal maintenance flags. 17313 This option must not be used by an ordinary user. It is intended only to 17314 facilitate remote troubleshooting of backend issues. Strict meaning of 17315 flags is not documented and not guaranteed to persist between releases. 17316 Quirks will be removed when the backend grows stable. 17317 Supported quirks: atomicmkdir binlist gzip insecure retry400 17318 17319 - Config: quirks 17320 - Env Var: RCLONE_MAILRU_QUIRKS 17321 - Type: string 17322 - Default: "" 17323 17324 #### --mailru-encoding 17325 17326 This sets the encoding for the backend. 17327 17328 See: the [encoding section in the overview](/overview/#encoding) for more info. 17329 17330 - Config: encoding 17331 - Env Var: RCLONE_MAILRU_ENCODING 17332 - Type: MultiEncoder 17333 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot 17334 17335 <!--- autogenerated options stop --> 17336 17337 Mega 17338 ----------------------------------------- 17339 17340 [Mega](https://mega.nz/) is a cloud storage and file hosting service 17341 known for its security feature where all files are encrypted locally 17342 before they are uploaded. This prevents anyone (including employees of 17343 Mega) from accessing the files without knowledge of the key used for 17344 encryption. 17345 17346 This is an rclone backend for Mega which supports the file transfer 17347 features of Mega using the same client side encryption. 17348 17349 Paths are specified as `remote:path` 17350 17351 Paths may be as deep as required, eg `remote:directory/subdirectory`. 17352 17353 Here is an example of how to make a remote called `remote`. First run: 17354 17355 rclone config 17356 17357 This will guide you through an interactive setup process: 17358 17359 ``` 17360 No remotes found - make a new one 17361 n) New remote 17362 s) Set configuration password 17363 q) Quit config 17364 n/s/q> n 17365 name> remote 17366 Type of storage to configure. 17367 Choose a number from below, or type in your own value 17368 [snip] 17369 XX / Mega 17370 \ "mega" 17371 [snip] 17372 Storage> mega 17373 User name 17374 user> you@example.com 17375 Password. 17376 y) Yes type in my own password 17377 g) Generate random password 17378 n) No leave this optional password blank 17379 y/g/n> y 17380 Enter the password: 17381 password: 17382 Confirm the password: 17383 password: 17384 Remote config 17385 -------------------- 17386 [remote] 17387 type = mega 17388 user = you@example.com 17389 pass = *** ENCRYPTED *** 17390 -------------------- 17391 y) Yes this is OK 17392 e) Edit this remote 17393 d) Delete this remote 17394 y/e/d> y 17395 ``` 17396 17397 **NOTE:** The encryption keys need to have been already generated after a regular login 17398 via the browser, otherwise attempting to use the credentials in `rclone` will fail. 17399 17400 Once configured you can then use `rclone` like this, 17401 17402 List directories in top level of your Mega 17403 17404 rclone lsd remote: 17405 17406 List all the files in your Mega 17407 17408 rclone ls remote: 17409 17410 To copy a local directory to an Mega directory called backup 17411 17412 rclone copy /home/source remote:backup 17413 17414 ### Modified time and hashes ### 17415 17416 Mega does not support modification times or hashes yet. 17417 17418 #### Restricted filename characters 17419 17420 | Character | Value | Replacement | 17421 | --------- |:-----:|:-----------:| 17422 | NUL | 0x00 | ␀ | 17423 | / | 0x2F | / | 17424 17425 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 17426 as they can't be used in JSON strings. 17427 17428 ### Duplicated files ### 17429 17430 Mega can have two files with exactly the same name and path (unlike a 17431 normal file system). 17432 17433 Duplicated files cause problems with the syncing and you will see 17434 messages in the log about duplicates. 17435 17436 Use `rclone dedupe` to fix duplicated files. 17437 17438 ### Failure to log-in ### 17439 17440 Mega remotes seem to get blocked (reject logins) under "heavy use". 17441 We haven't worked out the exact blocking rules but it seems to be 17442 related to fast paced, sucessive rclone commands. 17443 17444 For example, executing this command 90 times in a row `rclone link 17445 remote:file` will cause the remote to become "blocked". This is not an 17446 abnormal situation, for example if you wish to get the public links of 17447 a directory with hundred of files... After more or less a week, the 17448 remote will remote accept rclone logins normally again. 17449 17450 You can mitigate this issue by mounting the remote it with `rclone 17451 mount`. This will log-in when mounting and a log-out when unmounting 17452 only. You can also run `rclone rcd` and then use `rclone rc` to run 17453 the commands over the API to avoid logging in each time. 17454 17455 Rclone does not currently close mega sessions (you can see them in the 17456 web interface), however closing the sessions does not solve the issue. 17457 17458 If you space rclone commands by 3 seconds it will avoid blocking the 17459 remote. We haven't identified the exact blocking rules, so perhaps one 17460 could execute the command 80 times without waiting and avoid blocking 17461 by waiting 3 seconds, then continuing... 17462 17463 Note that this has been observed by trial and error and might not be 17464 set in stone. 17465 17466 Other tools seem not to produce this blocking effect, as they use a 17467 different working approach (state-based, using sessionIDs instead of 17468 log-in) which isn't compatible with the current stateless rclone 17469 approach. 17470 17471 Note that once blocked, the use of other tools (such as megacmd) is 17472 not a sure workaround: following megacmd login times have been 17473 observed in sucession for blocked remote: 7 minutes, 20 min, 30min, 30 17474 min, 30min. Web access looks unaffected though. 17475 17476 Investigation is continuing in relation to workarounds based on 17477 timeouts, pacers, retrials and tpslimits - if you discover something 17478 relevant, please post on the forum. 17479 17480 So, if rclone was working nicely and suddenly you are unable to log-in 17481 and you are sure the user and the password are correct, likely you 17482 have got the remote blocked for a while. 17483 17484 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/mega/mega.go then run make backenddocs --> 17485 ### Standard Options 17486 17487 Here are the standard options specific to mega (Mega). 17488 17489 #### --mega-user 17490 17491 User name 17492 17493 - Config: user 17494 - Env Var: RCLONE_MEGA_USER 17495 - Type: string 17496 - Default: "" 17497 17498 #### --mega-pass 17499 17500 Password. 17501 17502 - Config: pass 17503 - Env Var: RCLONE_MEGA_PASS 17504 - Type: string 17505 - Default: "" 17506 17507 ### Advanced Options 17508 17509 Here are the advanced options specific to mega (Mega). 17510 17511 #### --mega-debug 17512 17513 Output more debug from Mega. 17514 17515 If this flag is set (along with -vv) it will print further debugging 17516 information from the mega backend. 17517 17518 - Config: debug 17519 - Env Var: RCLONE_MEGA_DEBUG 17520 - Type: bool 17521 - Default: false 17522 17523 #### --mega-hard-delete 17524 17525 Delete files permanently rather than putting them into the trash. 17526 17527 Normally the mega backend will put all deletions into the trash rather 17528 than permanently deleting them. If you specify this then rclone will 17529 permanently delete objects instead. 17530 17531 - Config: hard_delete 17532 - Env Var: RCLONE_MEGA_HARD_DELETE 17533 - Type: bool 17534 - Default: false 17535 17536 #### --mega-encoding 17537 17538 This sets the encoding for the backend. 17539 17540 See: the [encoding section in the overview](/overview/#encoding) for more info. 17541 17542 - Config: encoding 17543 - Env Var: RCLONE_MEGA_ENCODING 17544 - Type: MultiEncoder 17545 - Default: Slash,InvalidUtf8,Dot 17546 17547 <!--- autogenerated options stop --> 17548 17549 ### Limitations ### 17550 17551 This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource 17552 go library implementing the Mega API. There doesn't appear to be any 17553 documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code 17554 so there are likely quite a few errors still remaining in this library. 17555 17556 Mega allows duplicate files which may confuse rclone. 17557 17558 Memory 17559 ----------------------------------------- 17560 17561 The memory backend is an in RAM backend. It does not persist its 17562 data - use the local backend for that. 17563 17564 The memory backend behaves like a bucket based remote (eg like 17565 s3). Because it has no parameters you can just use it with the 17566 `:memory:` remote name. 17567 17568 You can configure it as a remote like this with `rclone config` too if 17569 you want to: 17570 17571 ``` 17572 No remotes found - make a new one 17573 n) New remote 17574 s) Set configuration password 17575 q) Quit config 17576 n/s/q> n 17577 name> remote 17578 Type of storage to configure. 17579 Enter a string value. Press Enter for the default (""). 17580 Choose a number from below, or type in your own value 17581 [snip] 17582 XX / Memory 17583 \ "memory" 17584 [snip] 17585 Storage> memory 17586 ** See help for memory backend at: https://rclone.org/memory/ ** 17587 17588 Remote config 17589 17590 -------------------- 17591 [remote] 17592 type = memory 17593 -------------------- 17594 y) Yes this is OK (default) 17595 e) Edit this remote 17596 d) Delete this remote 17597 y/e/d> y 17598 ``` 17599 17600 Because the memory backend isn't persistent it is most useful for 17601 testing or with an rclone server or rclone mount, eg 17602 17603 rclone mount :memory: /mnt/tmp 17604 rclone serve webdav :memory: 17605 rclone serve sftp :memory: 17606 17607 ### Modified time and hashes ### 17608 17609 The memory backend supports MD5 hashes and modification times accurate to 1 nS. 17610 17611 #### Restricted filename characters 17612 17613 The memory backend replaces the [default restricted characters 17614 set](/overview/#restricted-characters). 17615 17616 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/memory/memory.go then run make backenddocs --> 17617 <!--- autogenerated options stop --> 17618 17619 Microsoft Azure Blob Storage 17620 ----------------------------------------- 17621 17622 Paths are specified as `remote:container` (or `remote:` for the `lsd` 17623 command.) You may put subdirectories in too, eg 17624 `remote:container/path/to/dir`. 17625 17626 Here is an example of making a Microsoft Azure Blob Storage 17627 configuration. For a remote called `remote`. First run: 17628 17629 rclone config 17630 17631 This will guide you through an interactive setup process: 17632 17633 ``` 17634 No remotes found - make a new one 17635 n) New remote 17636 s) Set configuration password 17637 q) Quit config 17638 n/s/q> n 17639 name> remote 17640 Type of storage to configure. 17641 Choose a number from below, or type in your own value 17642 [snip] 17643 XX / Microsoft Azure Blob Storage 17644 \ "azureblob" 17645 [snip] 17646 Storage> azureblob 17647 Storage Account Name 17648 account> account_name 17649 Storage Account Key 17650 key> base64encodedkey== 17651 Endpoint for the service - leave blank normally. 17652 endpoint> 17653 Remote config 17654 -------------------- 17655 [remote] 17656 account = account_name 17657 key = base64encodedkey== 17658 endpoint = 17659 -------------------- 17660 y) Yes this is OK 17661 e) Edit this remote 17662 d) Delete this remote 17663 y/e/d> y 17664 ``` 17665 17666 See all containers 17667 17668 rclone lsd remote: 17669 17670 Make a new container 17671 17672 rclone mkdir remote:container 17673 17674 List the contents of a container 17675 17676 rclone ls remote:container 17677 17678 Sync `/home/local/directory` to the remote container, deleting any excess 17679 files in the container. 17680 17681 rclone sync /home/local/directory remote:container 17682 17683 ### --fast-list ### 17684 17685 This remote supports `--fast-list` which allows you to use fewer 17686 transactions in exchange for more memory. See the [rclone 17687 docs](/docs/#fast-list) for more details. 17688 17689 ### Modified time ### 17690 17691 The modified time is stored as metadata on the object with the `mtime` 17692 key. It is stored using RFC3339 Format time with nanosecond 17693 precision. The metadata is supplied during directory listings so 17694 there is no overhead to using it. 17695 17696 ### Restricted filename characters 17697 17698 In addition to the [default restricted characters set](/overview/#restricted-characters) 17699 the following characters are also replaced: 17700 17701 | Character | Value | Replacement | 17702 | --------- |:-----:|:-----------:| 17703 | / | 0x2F | / | 17704 | \ | 0x5C | \ | 17705 17706 File names can also not end with the following characters. 17707 These only get replaced if they are last character in the name: 17708 17709 | Character | Value | Replacement | 17710 | --------- |:-----:|:-----------:| 17711 | . | 0x2E | . | 17712 17713 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 17714 as they can't be used in JSON strings. 17715 17716 ### Hashes ### 17717 17718 MD5 hashes are stored with blobs. However blobs that were uploaded in 17719 chunks only have an MD5 if the source remote was capable of MD5 17720 hashes, eg the local disk. 17721 17722 ### Authenticating with Azure Blob Storage 17723 17724 Rclone has 3 ways of authenticating with Azure Blob Storage: 17725 17726 #### Account and Key 17727 17728 This is the most straight forward and least flexible way. Just fill in the `account` and `key` lines and leave the rest blank. 17729 17730 #### SAS URL 17731 17732 This can be an account level SAS URL or container level SAS URL 17733 17734 To use it leave `account`, `key` blank and fill in `sas_url`. 17735 17736 Account level SAS URL or container level SAS URL can be obtained from Azure portal or Azure Storage Explorer. 17737 To get a container level SAS URL right click on a container in the Azure Blob explorer in the Azure portal. 17738 17739 If You use container level SAS URL, rclone operations are permitted only on particular container, eg 17740 17741 rclone ls azureblob:container or rclone ls azureblob: 17742 17743 Since container name already exists in SAS URL, you can leave it empty as well. 17744 17745 However these will not work 17746 17747 rclone lsd azureblob: 17748 rclone ls azureblob:othercontainer 17749 17750 This would be useful for temporarily allowing third parties access to a single container or putting credentials into an untrusted environment. 17751 17752 ### Multipart uploads ### 17753 17754 Rclone supports multipart uploads with Azure Blob storage. Files 17755 bigger than 256MB will be uploaded using chunked upload by default. 17756 17757 The files will be uploaded in parallel in 4MB chunks (by default). 17758 Note that these chunks are buffered in memory and there may be up to 17759 `--transfers` of them being uploaded at once. 17760 17761 Files can't be split into more than 50,000 chunks so by default, so 17762 the largest file that can be uploaded with 4MB chunk size is 195GB. 17763 Above this rclone will double the chunk size until it creates less 17764 than 50,000 chunks. By default this will mean a maximum file size of 17765 3.2TB can be uploaded. This can be raised to 5TB using 17766 `--azureblob-chunk-size 100M`. 17767 17768 Note that rclone doesn't commit the block list until the end of the 17769 upload which means that there is a limit of 9.5TB of multipart uploads 17770 in progress as Azure won't allow more than that amount of uncommitted 17771 blocks. 17772 17773 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/azureblob/azureblob.go then run make backenddocs --> 17774 ### Standard Options 17775 17776 Here are the standard options specific to azureblob (Microsoft Azure Blob Storage). 17777 17778 #### --azureblob-account 17779 17780 Storage Account Name (leave blank to use SAS URL or Emulator) 17781 17782 - Config: account 17783 - Env Var: RCLONE_AZUREBLOB_ACCOUNT 17784 - Type: string 17785 - Default: "" 17786 17787 #### --azureblob-key 17788 17789 Storage Account Key (leave blank to use SAS URL or Emulator) 17790 17791 - Config: key 17792 - Env Var: RCLONE_AZUREBLOB_KEY 17793 - Type: string 17794 - Default: "" 17795 17796 #### --azureblob-sas-url 17797 17798 SAS URL for container level access only 17799 (leave blank if using account/key or Emulator) 17800 17801 - Config: sas_url 17802 - Env Var: RCLONE_AZUREBLOB_SAS_URL 17803 - Type: string 17804 - Default: "" 17805 17806 #### --azureblob-use-emulator 17807 17808 Uses local storage emulator if provided as 'true' (leave blank if using real azure storage endpoint) 17809 17810 - Config: use_emulator 17811 - Env Var: RCLONE_AZUREBLOB_USE_EMULATOR 17812 - Type: bool 17813 - Default: false 17814 17815 ### Advanced Options 17816 17817 Here are the advanced options specific to azureblob (Microsoft Azure Blob Storage). 17818 17819 #### --azureblob-endpoint 17820 17821 Endpoint for the service 17822 Leave blank normally. 17823 17824 - Config: endpoint 17825 - Env Var: RCLONE_AZUREBLOB_ENDPOINT 17826 - Type: string 17827 - Default: "" 17828 17829 #### --azureblob-upload-cutoff 17830 17831 Cutoff for switching to chunked upload (<= 256MB). 17832 17833 - Config: upload_cutoff 17834 - Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF 17835 - Type: SizeSuffix 17836 - Default: 256M 17837 17838 #### --azureblob-chunk-size 17839 17840 Upload chunk size (<= 100MB). 17841 17842 Note that this is stored in memory and there may be up to 17843 "--transfers" chunks stored at once in memory. 17844 17845 - Config: chunk_size 17846 - Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE 17847 - Type: SizeSuffix 17848 - Default: 4M 17849 17850 #### --azureblob-list-chunk 17851 17852 Size of blob list. 17853 17854 This sets the number of blobs requested in each listing chunk. Default 17855 is the maximum, 5000. "List blobs" requests are permitted 2 minutes 17856 per megabyte to complete. If an operation is taking longer than 2 17857 minutes per megabyte on average, it will time out ( 17858 [source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval) 17859 ). This can be used to limit the number of blobs items to return, to 17860 avoid the time out. 17861 17862 - Config: list_chunk 17863 - Env Var: RCLONE_AZUREBLOB_LIST_CHUNK 17864 - Type: int 17865 - Default: 5000 17866 17867 #### --azureblob-access-tier 17868 17869 Access tier of blob: hot, cool or archive. 17870 17871 Archived blobs can be restored by setting access tier to hot or 17872 cool. Leave blank if you intend to use default access tier, which is 17873 set at account level 17874 17875 If there is no "access tier" specified, rclone doesn't apply any tier. 17876 rclone performs "Set Tier" operation on blobs while uploading, if objects 17877 are not modified, specifying "access tier" to new one will have no effect. 17878 If blobs are in "archive tier" at remote, trying to perform data transfer 17879 operations from remote will not be allowed. User should first restore by 17880 tiering blob to "Hot" or "Cool". 17881 17882 - Config: access_tier 17883 - Env Var: RCLONE_AZUREBLOB_ACCESS_TIER 17884 - Type: string 17885 - Default: "" 17886 17887 #### --azureblob-encoding 17888 17889 This sets the encoding for the backend. 17890 17891 See: the [encoding section in the overview](/overview/#encoding) for more info. 17892 17893 - Config: encoding 17894 - Env Var: RCLONE_AZUREBLOB_ENCODING 17895 - Type: MultiEncoder 17896 - Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8 17897 17898 <!--- autogenerated options stop --> 17899 17900 ### Limitations ### 17901 17902 MD5 sums are only uploaded with chunked files if the source has an MD5 17903 sum. This will always be the case for a local to azure copy. 17904 17905 ### Azure Storage Emulator Support ### 17906 You can test rlcone with storage emulator locally, to do this make sure azure storage emulator 17907 installed locally and set up a new remote with `rclone config` follow instructions described in 17908 introduction, set `use_emulator` config as `true`, you do not need to provide default account name 17909 or key if using emulator. 17910 17911 Microsoft OneDrive 17912 ----------------------------------------- 17913 17914 Paths are specified as `remote:path` 17915 17916 Paths may be as deep as required, eg `remote:directory/subdirectory`. 17917 17918 The initial setup for OneDrive involves getting a token from 17919 Microsoft which you need to do in your browser. `rclone config` walks 17920 you through it. 17921 17922 Here is an example of how to make a remote called `remote`. First run: 17923 17924 rclone config 17925 17926 This will guide you through an interactive setup process: 17927 17928 ``` 17929 e) Edit existing remote 17930 n) New remote 17931 d) Delete remote 17932 r) Rename remote 17933 c) Copy remote 17934 s) Set configuration password 17935 q) Quit config 17936 e/n/d/r/c/s/q> n 17937 name> remote 17938 Type of storage to configure. 17939 Enter a string value. Press Enter for the default (""). 17940 Choose a number from below, or type in your own value 17941 [snip] 17942 XX / Microsoft OneDrive 17943 \ "onedrive" 17944 [snip] 17945 Storage> onedrive 17946 Microsoft App Client Id 17947 Leave blank normally. 17948 Enter a string value. Press Enter for the default (""). 17949 client_id> 17950 Microsoft App Client Secret 17951 Leave blank normally. 17952 Enter a string value. Press Enter for the default (""). 17953 client_secret> 17954 Edit advanced config? (y/n) 17955 y) Yes 17956 n) No 17957 y/n> n 17958 Remote config 17959 Use auto config? 17960 * Say Y if not sure 17961 * Say N if you are working on a remote or headless machine 17962 y) Yes 17963 n) No 17964 y/n> y 17965 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 17966 Log in and authorize rclone for access 17967 Waiting for code... 17968 Got code 17969 Choose a number from below, or type in an existing value 17970 1 / OneDrive Personal or Business 17971 \ "onedrive" 17972 2 / Sharepoint site 17973 \ "sharepoint" 17974 3 / Type in driveID 17975 \ "driveid" 17976 4 / Type in SiteID 17977 \ "siteid" 17978 5 / Search a Sharepoint site 17979 \ "search" 17980 Your choice> 1 17981 Found 1 drives, please select the one you want to use: 17982 0: OneDrive (business) id=b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk 17983 Chose drive to use:> 0 17984 Found drive 'root' of type 'business', URL: https://org-my.sharepoint.com/personal/you/Documents 17985 Is that okay? 17986 y) Yes 17987 n) No 17988 y/n> y 17989 -------------------- 17990 [remote] 17991 type = onedrive 17992 token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} 17993 drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk 17994 drive_type = business 17995 -------------------- 17996 y) Yes this is OK 17997 e) Edit this remote 17998 d) Delete this remote 17999 y/e/d> y 18000 ``` 18001 18002 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 18003 machine with no Internet browser available. 18004 18005 Note that rclone runs a webserver on your local machine to collect the 18006 token as returned from Microsoft. This only runs from the moment it 18007 opens your browser to the moment you get back the verification 18008 code. This is on `http://127.0.0.1:53682/` and this it may require 18009 you to unblock it temporarily if you are running a host firewall. 18010 18011 Once configured you can then use `rclone` like this, 18012 18013 List directories in top level of your OneDrive 18014 18015 rclone lsd remote: 18016 18017 List all the files in your OneDrive 18018 18019 rclone ls remote: 18020 18021 To copy a local directory to an OneDrive directory called backup 18022 18023 rclone copy /home/source remote:backup 18024 18025 ### Getting your own Client ID and Key ### 18026 18027 You can use your own Client ID if the default (`client_id` left blank) 18028 one doesn't work for you or you see lots of throttling. The default 18029 Client ID and Key is shared by all rclone users when performing 18030 requests. 18031 18032 If you are having problems with them (E.g., seeing a lot of throttling), you can get your own 18033 Client ID and Key by following the steps below: 18034 18035 1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade, then click `New registration`. 18036 2. Enter a name for your app, choose account type `Any Azure AD directory - Multitenant`, select `Web` in `Redirect URI` Enter `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use. 18037 3. Under `manage` select `Certificates & secrets`, click `New client secret`. Copy and keep that secret for later use. 18038 4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`. 18039 5. Search and select the follwing permssions: `Files.Read`, `Files.ReadWrite`, `Files.Read.All`, `Files.ReadWrite.All`, `offline_access`, `User.Read`. Once selected click `Add permissions` at the bottom. 18040 18041 Now the application is complete. Run `rclone config` to create or edit a OneDrive remote. 18042 Supply the app ID and password as Client ID and Secret, respectively. rclone will walk you through the remaining steps. 18043 18044 ### Modification time and hashes ### 18045 18046 OneDrive allows modification times to be set on objects accurate to 1 18047 second. These will be used to detect whether objects need syncing or 18048 not. 18049 18050 OneDrive personal supports SHA1 type hashes. OneDrive for business and 18051 Sharepoint Server support 18052 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 18053 18054 For all types of OneDrive you can use the `--checksum` flag. 18055 18056 ### Restricted filename characters ### 18057 18058 In addition to the [default restricted characters set](/overview/#restricted-characters) 18059 the following characters are also replaced: 18060 18061 | Character | Value | Replacement | 18062 | --------- |:-----:|:-----------:| 18063 | " | 0x22 | " | 18064 | * | 0x2A | * | 18065 | : | 0x3A | : | 18066 | < | 0x3C | < | 18067 | > | 0x3E | > | 18068 | ? | 0x3F | ? | 18069 | \ | 0x5C | \ | 18070 | \| | 0x7C | | | 18071 | # | 0x23 | # | 18072 | % | 0x25 | % | 18073 18074 File names can also not end with the following characters. 18075 These only get replaced if they are last character in the name: 18076 18077 | Character | Value | Replacement | 18078 | --------- |:-----:|:-----------:| 18079 | SP | 0x20 | ␠ | 18080 | . | 0x2E | . | 18081 18082 File names can also not begin with the following characters. 18083 These only get replaced if they are first character in the name: 18084 18085 | Character | Value | Replacement | 18086 | --------- |:-----:|:-----------:| 18087 | SP | 0x20 | ␠ | 18088 | ~ | 0x7E | ~ | 18089 18090 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 18091 as they can't be used in JSON strings. 18092 18093 ### Deleting files ### 18094 18095 Any files you delete with rclone will end up in the trash. Microsoft 18096 doesn't provide an API to permanently delete files, nor to empty the 18097 trash, so you will have to do that with one of Microsoft's apps or via 18098 the OneDrive website. 18099 18100 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/onedrive/onedrive.go then run make backenddocs --> 18101 ### Standard Options 18102 18103 Here are the standard options specific to onedrive (Microsoft OneDrive). 18104 18105 #### --onedrive-client-id 18106 18107 Microsoft App Client Id 18108 Leave blank normally. 18109 18110 - Config: client_id 18111 - Env Var: RCLONE_ONEDRIVE_CLIENT_ID 18112 - Type: string 18113 - Default: "" 18114 18115 #### --onedrive-client-secret 18116 18117 Microsoft App Client Secret 18118 Leave blank normally. 18119 18120 - Config: client_secret 18121 - Env Var: RCLONE_ONEDRIVE_CLIENT_SECRET 18122 - Type: string 18123 - Default: "" 18124 18125 ### Advanced Options 18126 18127 Here are the advanced options specific to onedrive (Microsoft OneDrive). 18128 18129 #### --onedrive-chunk-size 18130 18131 Chunk size to upload files with - must be multiple of 320k (327,680 bytes). 18132 18133 Above this size files will be chunked - must be multiple of 320k (327,680 bytes). Note 18134 that the chunks will be buffered into memory. 18135 18136 - Config: chunk_size 18137 - Env Var: RCLONE_ONEDRIVE_CHUNK_SIZE 18138 - Type: SizeSuffix 18139 - Default: 10M 18140 18141 #### --onedrive-drive-id 18142 18143 The ID of the drive to use 18144 18145 - Config: drive_id 18146 - Env Var: RCLONE_ONEDRIVE_DRIVE_ID 18147 - Type: string 18148 - Default: "" 18149 18150 #### --onedrive-drive-type 18151 18152 The type of the drive ( personal | business | documentLibrary ) 18153 18154 - Config: drive_type 18155 - Env Var: RCLONE_ONEDRIVE_DRIVE_TYPE 18156 - Type: string 18157 - Default: "" 18158 18159 #### --onedrive-expose-onenote-files 18160 18161 Set to make OneNote files show up in directory listings. 18162 18163 By default rclone will hide OneNote files in directory listings because 18164 operations like "Open" and "Update" won't work on them. But this 18165 behaviour may also prevent you from deleting them. If you want to 18166 delete OneNote files or otherwise want them to show up in directory 18167 listing, set this option. 18168 18169 - Config: expose_onenote_files 18170 - Env Var: RCLONE_ONEDRIVE_EXPOSE_ONENOTE_FILES 18171 - Type: bool 18172 - Default: false 18173 18174 #### --onedrive-encoding 18175 18176 This sets the encoding for the backend. 18177 18178 See: the [encoding section in the overview](/overview/#encoding) for more info. 18179 18180 - Config: encoding 18181 - Env Var: RCLONE_ONEDRIVE_ENCODING 18182 - Type: MultiEncoder 18183 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot 18184 18185 <!--- autogenerated options stop --> 18186 18187 ### Limitations ### 18188 18189 #### Naming #### 18190 18191 Note that OneDrive is case insensitive so you can't have a 18192 file called "Hello.doc" and one called "hello.doc". 18193 18194 There are quite a few characters that can't be in OneDrive file 18195 names. These can't occur on Windows platforms, but on non-Windows 18196 platforms they are common. Rclone will map these names to and from an 18197 identical looking unicode equivalent. For example if a file has a `?` 18198 in it will be mapped to `?` instead. 18199 18200 #### File sizes #### 18201 18202 The largest allowed file sizes are 15GB for OneDrive for Business and 35GB for OneDrive Personal (Updated 4 Jan 2019). 18203 18204 #### Path length #### 18205 18206 The entire path, including the file name, must contain fewer than 400 characters for OneDrive, OneDrive for Business and SharePoint Online. If you are encrypting file and folder names with rclone, you may want to pay attention to this limitation because the encrypted names are typically longer than the original ones. 18207 18208 #### Number of files #### 18209 18210 OneDrive seems to be OK with at least 50,000 files in a folder, but at 18211 100,000 rclone will get errors listing the directory like `couldn’t 18212 list files: UnknownError:`. See 18213 [#2707](https://github.com/rclone/rclone/issues/2707) for more info. 18214 18215 An official document about the limitations for different types of OneDrive can be found [here](https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa). 18216 18217 ### Versioning issue ### 18218 18219 Every change in OneDrive causes the service to create a new version. 18220 This counts against a users quota. 18221 For example changing the modification time of a file creates a second 18222 version, so the file is using twice the space. 18223 18224 The `copy` is the only rclone command affected by this as we copy 18225 the file and then afterwards set the modification time to match the 18226 source file. 18227 18228 **Note**: Starting October 2018, users will no longer be able to disable versioning by default. This is because Microsoft has brought an [update](https://techcommunity.microsoft.com/t5/Microsoft-OneDrive-Blog/New-Updates-to-OneDrive-and-SharePoint-Team-Site-Versioning/ba-p/204390) to the mechanism. To change this new default setting, a PowerShell command is required to be run by a SharePoint admin. If you are an admin, you can run these commands in PowerShell to change that setting: 18229 18230 1. `Install-Module -Name Microsoft.Online.SharePoint.PowerShell` (in case you haven't installed this already) 18231 1. `Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking` 18232 1. `Connect-SPOService -Url https://YOURSITE-admin.sharepoint.com -Credential YOU@YOURSITE.COM` (replacing `YOURSITE`, `YOU`, `YOURSITE.COM` with the actual values; this will prompt for your credentials) 18233 1. `Set-SPOTenant -EnableMinimumVersionRequirement $False` 18234 1. `Disconnect-SPOService` (to disconnect from the server) 18235 18236 *Below are the steps for normal users to disable versioning. If you don't see the "No Versioning" option, make sure the above requirements are met.* 18237 18238 User [Weropol](https://github.com/Weropol) has found a method to disable 18239 versioning on OneDrive 18240 18241 1. Open the settings menu by clicking on the gear symbol at the top of the OneDrive Business page. 18242 2. Click Site settings. 18243 3. Once on the Site settings page, navigate to Site Administration > Site libraries and lists. 18244 4. Click Customize "Documents". 18245 5. Click General Settings > Versioning Settings. 18246 6. Under Document Version History select the option No versioning. 18247 Note: This will disable the creation of new file versions, but will not remove any previous versions. Your documents are safe. 18248 7. Apply the changes by clicking OK. 18249 8. Use rclone to upload or modify files. (I also use the --no-update-modtime flag) 18250 9. Restore the versioning settings after using rclone. (Optional) 18251 18252 ### Troubleshooting ### 18253 18254 #### Unexpected file size/hash differences on Sharepoint #### 18255 18256 It is a 18257 [known](https://github.com/OneDrive/onedrive-api-docs/issues/935#issuecomment-441741631) 18258 issue that Sharepoint (not OneDrive or OneDrive for Business) silently modifies 18259 uploaded files, mainly Office files (.docx, .xlsx, etc.), causing file size and 18260 hash checks to fail. To use rclone with such affected files on Sharepoint, you 18261 may disable these checks with the following command line arguments: 18262 18263 ``` 18264 --ignore-checksum --ignore-size 18265 ``` 18266 18267 #### Replacing/deleting existing files on Sharepoint gets "item not found" #### 18268 18269 It is a [known](https://github.com/OneDrive/onedrive-api-docs/issues/1068) issue 18270 that Sharepoint (not OneDrive or OneDrive for Business) may return "item not 18271 found" errors when users try to replace or delete uploaded files; this seems to 18272 mainly affect Office files (.docx, .xlsx, etc.). As a workaround, you may use 18273 the `--backup-dir <BACKUP_DIR>` command line argument so rclone moves the 18274 files to be replaced/deleted into a given backup directory (instead of directly 18275 replacing/deleting them). For example, to instruct rclone to move the files into 18276 the directory `rclone-backup-dir` on backend `mysharepoint`, you may use: 18277 18278 ``` 18279 --backup-dir mysharepoint:rclone-backup-dir 18280 ``` 18281 18282 #### access\_denied (AADSTS65005) #### 18283 18284 ``` 18285 Error: access_denied 18286 Code: AADSTS65005 18287 Description: Using application 'rclone' is currently not supported for your organization [YOUR_ORGANIZATION] because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of [YOUR_ORGANIZATION] before the application rclone can be provisioned. 18288 ``` 18289 18290 This means that rclone can't use the OneDrive for Business API with your account. You can't do much about it, maybe write an email to your admins. 18291 18292 However, there are other ways to interact with your OneDrive account. Have a look at the webdav backend: https://rclone.org/webdav/#sharepoint 18293 18294 #### invalid\_grant (AADSTS50076) #### 18295 18296 ``` 18297 Error: invalid_grant 18298 Code: AADSTS50076 18299 Description: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '...'. 18300 ``` 18301 18302 If you see the error above after enabling multi-factor authentication for your account, you can fix it by refreshing your OAuth refresh token. To do that, run `rclone config`, and choose to edit your OneDrive backend. Then, you don't need to actually make any changes until you reach this question: `Already have a token - refresh?`. For this question, answer `y` and go through the process to refresh your token, just like the first time the backend is configured. After this, rclone should work again for this backend. 18303 18304 OpenDrive 18305 ------------------------------------ 18306 18307 Paths are specified as `remote:path` 18308 18309 Paths may be as deep as required, eg `remote:directory/subdirectory`. 18310 18311 Here is an example of how to make a remote called `remote`. First run: 18312 18313 rclone config 18314 18315 This will guide you through an interactive setup process: 18316 18317 ``` 18318 n) New remote 18319 d) Delete remote 18320 q) Quit config 18321 e/n/d/q> n 18322 name> remote 18323 Type of storage to configure. 18324 Choose a number from below, or type in your own value 18325 [snip] 18326 XX / OpenDrive 18327 \ "opendrive" 18328 [snip] 18329 Storage> opendrive 18330 Username 18331 username> 18332 Password 18333 y) Yes type in my own password 18334 g) Generate random password 18335 y/g> y 18336 Enter the password: 18337 password: 18338 Confirm the password: 18339 password: 18340 -------------------- 18341 [remote] 18342 username = 18343 password = *** ENCRYPTED *** 18344 -------------------- 18345 y) Yes this is OK 18346 e) Edit this remote 18347 d) Delete this remote 18348 y/e/d> y 18349 ``` 18350 18351 List directories in top level of your OpenDrive 18352 18353 rclone lsd remote: 18354 18355 List all the files in your OpenDrive 18356 18357 rclone ls remote: 18358 18359 To copy a local directory to an OpenDrive directory called backup 18360 18361 rclone copy /home/source remote:backup 18362 18363 ### Modified time and MD5SUMs ### 18364 18365 OpenDrive allows modification times to be set on objects accurate to 1 18366 second. These will be used to detect whether objects need syncing or 18367 not. 18368 18369 #### Restricted filename characters 18370 18371 | Character | Value | Replacement | 18372 | --------- |:-----:|:-----------:| 18373 | NUL | 0x00 | ␀ | 18374 | / | 0x2F | / | 18375 | " | 0x22 | " | 18376 | * | 0x2A | * | 18377 | : | 0x3A | : | 18378 | < | 0x3C | < | 18379 | > | 0x3E | > | 18380 | ? | 0x3F | ? | 18381 | \ | 0x5C | \ | 18382 | \| | 0x7C | | | 18383 18384 File names can also not begin or end with the following characters. 18385 These only get replaced if they are the first or last character in the name: 18386 18387 | Character | Value | Replacement | 18388 | --------- |:-----:|:-----------:| 18389 | SP | 0x20 | ␠ | 18390 | HT | 0x09 | ␉ | 18391 | LF | 0x0A | ␊ | 18392 | VT | 0x0B | ␋ | 18393 | CR | 0x0D | ␍ | 18394 18395 18396 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 18397 as they can't be used in JSON strings. 18398 18399 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/opendrive/opendrive.go then run make backenddocs --> 18400 ### Standard Options 18401 18402 Here are the standard options specific to opendrive (OpenDrive). 18403 18404 #### --opendrive-username 18405 18406 Username 18407 18408 - Config: username 18409 - Env Var: RCLONE_OPENDRIVE_USERNAME 18410 - Type: string 18411 - Default: "" 18412 18413 #### --opendrive-password 18414 18415 Password. 18416 18417 - Config: password 18418 - Env Var: RCLONE_OPENDRIVE_PASSWORD 18419 - Type: string 18420 - Default: "" 18421 18422 ### Advanced Options 18423 18424 Here are the advanced options specific to opendrive (OpenDrive). 18425 18426 #### --opendrive-encoding 18427 18428 This sets the encoding for the backend. 18429 18430 See: the [encoding section in the overview](/overview/#encoding) for more info. 18431 18432 - Config: encoding 18433 - Env Var: RCLONE_OPENDRIVE_ENCODING 18434 - Type: MultiEncoder 18435 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot 18436 18437 #### --opendrive-chunk-size 18438 18439 Files will be uploaded in chunks this size. 18440 18441 Note that these chunks are buffered in memory so increasing them will 18442 increase memory use. 18443 18444 - Config: chunk_size 18445 - Env Var: RCLONE_OPENDRIVE_CHUNK_SIZE 18446 - Type: SizeSuffix 18447 - Default: 10M 18448 18449 <!--- autogenerated options stop --> 18450 18451 ### Limitations ### 18452 18453 Note that OpenDrive is case insensitive so you can't have a 18454 file called "Hello.doc" and one called "hello.doc". 18455 18456 There are quite a few characters that can't be in OpenDrive file 18457 names. These can't occur on Windows platforms, but on non-Windows 18458 platforms they are common. Rclone will map these names to and from an 18459 identical looking unicode equivalent. For example if a file has a `?` 18460 in it will be mapped to `?` instead. 18461 18462 QingStor 18463 --------------------------------------- 18464 18465 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 18466 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 18467 18468 Here is an example of making an QingStor configuration. First run 18469 18470 rclone config 18471 18472 This will guide you through an interactive setup process. 18473 18474 ``` 18475 No remotes found - make a new one 18476 n) New remote 18477 r) Rename remote 18478 c) Copy remote 18479 s) Set configuration password 18480 q) Quit config 18481 n/r/c/s/q> n 18482 name> remote 18483 Type of storage to configure. 18484 Choose a number from below, or type in your own value 18485 [snip] 18486 XX / QingStor Object Storage 18487 \ "qingstor" 18488 [snip] 18489 Storage> qingstor 18490 Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 18491 Choose a number from below, or type in your own value 18492 1 / Enter QingStor credentials in the next step 18493 \ "false" 18494 2 / Get QingStor credentials from the environment (env vars or IAM) 18495 \ "true" 18496 env_auth> 1 18497 QingStor Access Key ID - leave blank for anonymous access or runtime credentials. 18498 access_key_id> access_key 18499 QingStor Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 18500 secret_access_key> secret_key 18501 Enter a endpoint URL to connection QingStor API. 18502 Leave blank will use the default value "https://qingstor.com:443" 18503 endpoint> 18504 Zone connect to. Default is "pek3a". 18505 Choose a number from below, or type in your own value 18506 / The Beijing (China) Three Zone 18507 1 | Needs location constraint pek3a. 18508 \ "pek3a" 18509 / The Shanghai (China) First Zone 18510 2 | Needs location constraint sh1a. 18511 \ "sh1a" 18512 zone> 1 18513 Number of connnection retry. 18514 Leave blank will use the default value "3". 18515 connection_retries> 18516 Remote config 18517 -------------------- 18518 [remote] 18519 env_auth = false 18520 access_key_id = access_key 18521 secret_access_key = secret_key 18522 endpoint = 18523 zone = pek3a 18524 connection_retries = 18525 -------------------- 18526 y) Yes this is OK 18527 e) Edit this remote 18528 d) Delete this remote 18529 y/e/d> y 18530 ``` 18531 18532 This remote is called `remote` and can now be used like this 18533 18534 See all buckets 18535 18536 rclone lsd remote: 18537 18538 Make a new bucket 18539 18540 rclone mkdir remote:bucket 18541 18542 List the contents of a bucket 18543 18544 rclone ls remote:bucket 18545 18546 Sync `/home/local/directory` to the remote bucket, deleting any excess 18547 files in the bucket. 18548 18549 rclone sync /home/local/directory remote:bucket 18550 18551 ### --fast-list ### 18552 18553 This remote supports `--fast-list` which allows you to use fewer 18554 transactions in exchange for more memory. See the [rclone 18555 docs](/docs/#fast-list) for more details. 18556 18557 ### Multipart uploads ### 18558 18559 rclone supports multipart uploads with QingStor which means that it can 18560 upload files bigger than 5GB. Note that files uploaded with multipart 18561 upload don't have an MD5SUM. 18562 18563 ### Buckets and Zone ### 18564 18565 With QingStor you can list buckets (`rclone lsd`) using any zone, 18566 but you can only access the content of a bucket from the zone it was 18567 created in. If you attempt to access a bucket from the wrong zone, 18568 you will get an error, `incorrect zone, the bucket is not in 'XXX' 18569 zone`. 18570 18571 ### Authentication ### 18572 18573 There are two ways to supply `rclone` with a set of QingStor 18574 credentials. In order of precedence: 18575 18576 - Directly in the rclone configuration file (as configured by `rclone config`) 18577 - set `access_key_id` and `secret_access_key` 18578 - Runtime configuration: 18579 - set `env_auth` to `true` in the config file 18580 - Exporting the following environment variables before running `rclone` 18581 - Access Key ID: `QS_ACCESS_KEY_ID` or `QS_ACCESS_KEY` 18582 - Secret Access Key: `QS_SECRET_ACCESS_KEY` or `QS_SECRET_KEY` 18583 18584 ### Restricted filename characters 18585 18586 The control characters 0x00-0x1F and / are replaced as in the [default 18587 restricted characters set](/overview/#restricted-characters). Note 18588 that 0x7F is not replaced. 18589 18590 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 18591 as they can't be used in JSON strings. 18592 18593 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/qingstor/qingstor.go then run make backenddocs --> 18594 ### Standard Options 18595 18596 Here are the standard options specific to qingstor (QingCloud Object Storage). 18597 18598 #### --qingstor-env-auth 18599 18600 Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 18601 18602 - Config: env_auth 18603 - Env Var: RCLONE_QINGSTOR_ENV_AUTH 18604 - Type: bool 18605 - Default: false 18606 - Examples: 18607 - "false" 18608 - Enter QingStor credentials in the next step 18609 - "true" 18610 - Get QingStor credentials from the environment (env vars or IAM) 18611 18612 #### --qingstor-access-key-id 18613 18614 QingStor Access Key ID 18615 Leave blank for anonymous access or runtime credentials. 18616 18617 - Config: access_key_id 18618 - Env Var: RCLONE_QINGSTOR_ACCESS_KEY_ID 18619 - Type: string 18620 - Default: "" 18621 18622 #### --qingstor-secret-access-key 18623 18624 QingStor Secret Access Key (password) 18625 Leave blank for anonymous access or runtime credentials. 18626 18627 - Config: secret_access_key 18628 - Env Var: RCLONE_QINGSTOR_SECRET_ACCESS_KEY 18629 - Type: string 18630 - Default: "" 18631 18632 #### --qingstor-endpoint 18633 18634 Enter a endpoint URL to connection QingStor API. 18635 Leave blank will use the default value "https://qingstor.com:443" 18636 18637 - Config: endpoint 18638 - Env Var: RCLONE_QINGSTOR_ENDPOINT 18639 - Type: string 18640 - Default: "" 18641 18642 #### --qingstor-zone 18643 18644 Zone to connect to. 18645 Default is "pek3a". 18646 18647 - Config: zone 18648 - Env Var: RCLONE_QINGSTOR_ZONE 18649 - Type: string 18650 - Default: "" 18651 - Examples: 18652 - "pek3a" 18653 - The Beijing (China) Three Zone 18654 - Needs location constraint pek3a. 18655 - "sh1a" 18656 - The Shanghai (China) First Zone 18657 - Needs location constraint sh1a. 18658 - "gd2a" 18659 - The Guangdong (China) Second Zone 18660 - Needs location constraint gd2a. 18661 18662 ### Advanced Options 18663 18664 Here are the advanced options specific to qingstor (QingCloud Object Storage). 18665 18666 #### --qingstor-connection-retries 18667 18668 Number of connection retries. 18669 18670 - Config: connection_retries 18671 - Env Var: RCLONE_QINGSTOR_CONNECTION_RETRIES 18672 - Type: int 18673 - Default: 3 18674 18675 #### --qingstor-upload-cutoff 18676 18677 Cutoff for switching to chunked upload 18678 18679 Any files larger than this will be uploaded in chunks of chunk_size. 18680 The minimum is 0 and the maximum is 5GB. 18681 18682 - Config: upload_cutoff 18683 - Env Var: RCLONE_QINGSTOR_UPLOAD_CUTOFF 18684 - Type: SizeSuffix 18685 - Default: 200M 18686 18687 #### --qingstor-chunk-size 18688 18689 Chunk size to use for uploading. 18690 18691 When uploading files larger than upload_cutoff they will be uploaded 18692 as multipart uploads using this chunk size. 18693 18694 Note that "--qingstor-upload-concurrency" chunks of this size are buffered 18695 in memory per transfer. 18696 18697 If you are transferring large files over high speed links and you have 18698 enough memory, then increasing this will speed up the transfers. 18699 18700 - Config: chunk_size 18701 - Env Var: RCLONE_QINGSTOR_CHUNK_SIZE 18702 - Type: SizeSuffix 18703 - Default: 4M 18704 18705 #### --qingstor-upload-concurrency 18706 18707 Concurrency for multipart uploads. 18708 18709 This is the number of chunks of the same file that are uploaded 18710 concurrently. 18711 18712 NB if you set this to > 1 then the checksums of multpart uploads 18713 become corrupted (the uploads themselves are not corrupted though). 18714 18715 If you are uploading small numbers of large file over high speed link 18716 and these uploads do not fully utilize your bandwidth, then increasing 18717 this may help to speed up the transfers. 18718 18719 - Config: upload_concurrency 18720 - Env Var: RCLONE_QINGSTOR_UPLOAD_CONCURRENCY 18721 - Type: int 18722 - Default: 1 18723 18724 #### --qingstor-encoding 18725 18726 This sets the encoding for the backend. 18727 18728 See: the [encoding section in the overview](/overview/#encoding) for more info. 18729 18730 - Config: encoding 18731 - Env Var: RCLONE_QINGSTOR_ENCODING 18732 - Type: MultiEncoder 18733 - Default: Slash,Ctl,InvalidUtf8 18734 18735 <!--- autogenerated options stop --> 18736 18737 Swift 18738 ---------------------------------------- 18739 18740 Swift refers to [Openstack Object Storage](https://docs.openstack.org/swift/latest/). 18741 Commercial implementations of that being: 18742 18743 * [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/) 18744 * [Memset Memstore](https://www.memset.com/cloud/storage/) 18745 * [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/) 18746 * [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc) 18747 * [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html) 18748 18749 Paths are specified as `remote:container` (or `remote:` for the `lsd` 18750 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 18751 18752 Here is an example of making a swift configuration. First run 18753 18754 rclone config 18755 18756 This will guide you through an interactive setup process. 18757 18758 ``` 18759 No remotes found - make a new one 18760 n) New remote 18761 s) Set configuration password 18762 q) Quit config 18763 n/s/q> n 18764 name> remote 18765 Type of storage to configure. 18766 Choose a number from below, or type in your own value 18767 [snip] 18768 XX / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 18769 \ "swift" 18770 [snip] 18771 Storage> swift 18772 Get swift credentials from environment variables in standard OpenStack form. 18773 Choose a number from below, or type in your own value 18774 1 / Enter swift credentials in the next step 18775 \ "false" 18776 2 / Get swift credentials from environment vars. Leave other fields blank if using this. 18777 \ "true" 18778 env_auth> true 18779 User name to log in (OS_USERNAME). 18780 user> 18781 API key or password (OS_PASSWORD). 18782 key> 18783 Authentication URL for server (OS_AUTH_URL). 18784 Choose a number from below, or type in your own value 18785 1 / Rackspace US 18786 \ "https://auth.api.rackspacecloud.com/v1.0" 18787 2 / Rackspace UK 18788 \ "https://lon.auth.api.rackspacecloud.com/v1.0" 18789 3 / Rackspace v2 18790 \ "https://identity.api.rackspacecloud.com/v2.0" 18791 4 / Memset Memstore UK 18792 \ "https://auth.storage.memset.com/v1.0" 18793 5 / Memset Memstore UK v2 18794 \ "https://auth.storage.memset.com/v2.0" 18795 6 / OVH 18796 \ "https://auth.cloud.ovh.net/v3" 18797 auth> 18798 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 18799 user_id> 18800 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 18801 domain> 18802 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 18803 tenant> 18804 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 18805 tenant_id> 18806 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 18807 tenant_domain> 18808 Region name - optional (OS_REGION_NAME) 18809 region> 18810 Storage URL - optional (OS_STORAGE_URL) 18811 storage_url> 18812 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 18813 auth_token> 18814 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 18815 auth_version> 18816 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 18817 Choose a number from below, or type in your own value 18818 1 / Public (default, choose this if not sure) 18819 \ "public" 18820 2 / Internal (use internal service net) 18821 \ "internal" 18822 3 / Admin 18823 \ "admin" 18824 endpoint_type> 18825 Remote config 18826 -------------------- 18827 [test] 18828 env_auth = true 18829 user = 18830 key = 18831 auth = 18832 user_id = 18833 domain = 18834 tenant = 18835 tenant_id = 18836 tenant_domain = 18837 region = 18838 storage_url = 18839 auth_token = 18840 auth_version = 18841 endpoint_type = 18842 -------------------- 18843 y) Yes this is OK 18844 e) Edit this remote 18845 d) Delete this remote 18846 y/e/d> y 18847 ``` 18848 18849 This remote is called `remote` and can now be used like this 18850 18851 See all containers 18852 18853 rclone lsd remote: 18854 18855 Make a new container 18856 18857 rclone mkdir remote:container 18858 18859 List the contents of a container 18860 18861 rclone ls remote:container 18862 18863 Sync `/home/local/directory` to the remote container, deleting any 18864 excess files in the container. 18865 18866 rclone sync /home/local/directory remote:container 18867 18868 ### Configuration from an OpenStack credentials file ### 18869 18870 An OpenStack credentials file typically looks something something 18871 like this (without the comments) 18872 18873 ``` 18874 export OS_AUTH_URL=https://a.provider.net/v2.0 18875 export OS_TENANT_ID=ffffffffffffffffffffffffffffffff 18876 export OS_TENANT_NAME="1234567890123456" 18877 export OS_USERNAME="123abc567xy" 18878 echo "Please enter your OpenStack Password: " 18879 read -sr OS_PASSWORD_INPUT 18880 export OS_PASSWORD=$OS_PASSWORD_INPUT 18881 export OS_REGION_NAME="SBG1" 18882 if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi 18883 ``` 18884 18885 The config file needs to look something like this where `$OS_USERNAME` 18886 represents the value of the `OS_USERNAME` variable - `123abc567xy` in 18887 the example above. 18888 18889 ``` 18890 [remote] 18891 type = swift 18892 user = $OS_USERNAME 18893 key = $OS_PASSWORD 18894 auth = $OS_AUTH_URL 18895 tenant = $OS_TENANT_NAME 18896 ``` 18897 18898 Note that you may (or may not) need to set `region` too - try without first. 18899 18900 ### Configuration from the environment ### 18901 18902 If you prefer you can configure rclone to use swift using a standard 18903 set of OpenStack environment variables. 18904 18905 When you run through the config, make sure you choose `true` for 18906 `env_auth` and leave everything else blank. 18907 18908 rclone will then set any empty config parameters from the environment 18909 using standard OpenStack environment variables. There is [a list of 18910 the 18911 variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment) 18912 in the docs for the swift library. 18913 18914 ### Using an alternate authentication method ### 18915 18916 If your OpenStack installation uses a non-standard authentication method 18917 that might not be yet supported by rclone or the underlying swift library, 18918 you can authenticate externally (e.g. calling manually the `openstack` 18919 commands to get a token). Then, you just need to pass the two 18920 configuration variables ``auth_token`` and ``storage_url``. 18921 If they are both provided, the other variables are ignored. rclone will 18922 not try to authenticate but instead assume it is already authenticated 18923 and use these two variables to access the OpenStack installation. 18924 18925 #### Using rclone without a config file #### 18926 18927 You can use rclone with swift without a config file, if desired, like 18928 this: 18929 18930 ``` 18931 source openstack-credentials-file 18932 export RCLONE_CONFIG_MYREMOTE_TYPE=swift 18933 export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true 18934 rclone lsd myremote: 18935 ``` 18936 18937 ### --fast-list ### 18938 18939 This remote supports `--fast-list` which allows you to use fewer 18940 transactions in exchange for more memory. See the [rclone 18941 docs](/docs/#fast-list) for more details. 18942 18943 ### --update and --use-server-modtime ### 18944 18945 As noted below, the modified time is stored on metadata on the object. It is 18946 used by default for all operations that require checking the time a file was 18947 last updated. It allows rclone to treat the remote more like a true filesystem, 18948 but it is inefficient because it requires an extra API call to retrieve the 18949 metadata. 18950 18951 For many operations, the time the object was last uploaded to the remote is 18952 sufficient to determine if it is "dirty". By using `--update` along with 18953 `--use-server-modtime`, you can avoid the extra API call and simply upload 18954 files whose local modtime is newer than the time it was last uploaded. 18955 18956 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/swift/swift.go then run make backenddocs --> 18957 ### Standard Options 18958 18959 Here are the standard options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 18960 18961 #### --swift-env-auth 18962 18963 Get swift credentials from environment variables in standard OpenStack form. 18964 18965 - Config: env_auth 18966 - Env Var: RCLONE_SWIFT_ENV_AUTH 18967 - Type: bool 18968 - Default: false 18969 - Examples: 18970 - "false" 18971 - Enter swift credentials in the next step 18972 - "true" 18973 - Get swift credentials from environment vars. Leave other fields blank if using this. 18974 18975 #### --swift-user 18976 18977 User name to log in (OS_USERNAME). 18978 18979 - Config: user 18980 - Env Var: RCLONE_SWIFT_USER 18981 - Type: string 18982 - Default: "" 18983 18984 #### --swift-key 18985 18986 API key or password (OS_PASSWORD). 18987 18988 - Config: key 18989 - Env Var: RCLONE_SWIFT_KEY 18990 - Type: string 18991 - Default: "" 18992 18993 #### --swift-auth 18994 18995 Authentication URL for server (OS_AUTH_URL). 18996 18997 - Config: auth 18998 - Env Var: RCLONE_SWIFT_AUTH 18999 - Type: string 19000 - Default: "" 19001 - Examples: 19002 - "https://auth.api.rackspacecloud.com/v1.0" 19003 - Rackspace US 19004 - "https://lon.auth.api.rackspacecloud.com/v1.0" 19005 - Rackspace UK 19006 - "https://identity.api.rackspacecloud.com/v2.0" 19007 - Rackspace v2 19008 - "https://auth.storage.memset.com/v1.0" 19009 - Memset Memstore UK 19010 - "https://auth.storage.memset.com/v2.0" 19011 - Memset Memstore UK v2 19012 - "https://auth.cloud.ovh.net/v3" 19013 - OVH 19014 19015 #### --swift-user-id 19016 19017 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 19018 19019 - Config: user_id 19020 - Env Var: RCLONE_SWIFT_USER_ID 19021 - Type: string 19022 - Default: "" 19023 19024 #### --swift-domain 19025 19026 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 19027 19028 - Config: domain 19029 - Env Var: RCLONE_SWIFT_DOMAIN 19030 - Type: string 19031 - Default: "" 19032 19033 #### --swift-tenant 19034 19035 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 19036 19037 - Config: tenant 19038 - Env Var: RCLONE_SWIFT_TENANT 19039 - Type: string 19040 - Default: "" 19041 19042 #### --swift-tenant-id 19043 19044 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 19045 19046 - Config: tenant_id 19047 - Env Var: RCLONE_SWIFT_TENANT_ID 19048 - Type: string 19049 - Default: "" 19050 19051 #### --swift-tenant-domain 19052 19053 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 19054 19055 - Config: tenant_domain 19056 - Env Var: RCLONE_SWIFT_TENANT_DOMAIN 19057 - Type: string 19058 - Default: "" 19059 19060 #### --swift-region 19061 19062 Region name - optional (OS_REGION_NAME) 19063 19064 - Config: region 19065 - Env Var: RCLONE_SWIFT_REGION 19066 - Type: string 19067 - Default: "" 19068 19069 #### --swift-storage-url 19070 19071 Storage URL - optional (OS_STORAGE_URL) 19072 19073 - Config: storage_url 19074 - Env Var: RCLONE_SWIFT_STORAGE_URL 19075 - Type: string 19076 - Default: "" 19077 19078 #### --swift-auth-token 19079 19080 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 19081 19082 - Config: auth_token 19083 - Env Var: RCLONE_SWIFT_AUTH_TOKEN 19084 - Type: string 19085 - Default: "" 19086 19087 #### --swift-application-credential-id 19088 19089 Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 19090 19091 - Config: application_credential_id 19092 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID 19093 - Type: string 19094 - Default: "" 19095 19096 #### --swift-application-credential-name 19097 19098 Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 19099 19100 - Config: application_credential_name 19101 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME 19102 - Type: string 19103 - Default: "" 19104 19105 #### --swift-application-credential-secret 19106 19107 Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 19108 19109 - Config: application_credential_secret 19110 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET 19111 - Type: string 19112 - Default: "" 19113 19114 #### --swift-auth-version 19115 19116 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 19117 19118 - Config: auth_version 19119 - Env Var: RCLONE_SWIFT_AUTH_VERSION 19120 - Type: int 19121 - Default: 0 19122 19123 #### --swift-endpoint-type 19124 19125 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 19126 19127 - Config: endpoint_type 19128 - Env Var: RCLONE_SWIFT_ENDPOINT_TYPE 19129 - Type: string 19130 - Default: "public" 19131 - Examples: 19132 - "public" 19133 - Public (default, choose this if not sure) 19134 - "internal" 19135 - Internal (use internal service net) 19136 - "admin" 19137 - Admin 19138 19139 #### --swift-storage-policy 19140 19141 The storage policy to use when creating a new container 19142 19143 This applies the specified storage policy when creating a new 19144 container. The policy cannot be changed afterwards. The allowed 19145 configuration values and their meaning depend on your Swift storage 19146 provider. 19147 19148 - Config: storage_policy 19149 - Env Var: RCLONE_SWIFT_STORAGE_POLICY 19150 - Type: string 19151 - Default: "" 19152 - Examples: 19153 - "" 19154 - Default 19155 - "pcs" 19156 - OVH Public Cloud Storage 19157 - "pca" 19158 - OVH Public Cloud Archive 19159 19160 ### Advanced Options 19161 19162 Here are the advanced options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 19163 19164 #### --swift-chunk-size 19165 19166 Above this size files will be chunked into a _segments container. 19167 19168 Above this size files will be chunked into a _segments container. The 19169 default for this is 5GB which is its maximum value. 19170 19171 - Config: chunk_size 19172 - Env Var: RCLONE_SWIFT_CHUNK_SIZE 19173 - Type: SizeSuffix 19174 - Default: 5G 19175 19176 #### --swift-no-chunk 19177 19178 Don't chunk files during streaming upload. 19179 19180 When doing streaming uploads (eg using rcat or mount) setting this 19181 flag will cause the swift backend to not upload chunked files. 19182 19183 This will limit the maximum upload size to 5GB. However non chunked 19184 files are easier to deal with and have an MD5SUM. 19185 19186 Rclone will still chunk files bigger than chunk_size when doing normal 19187 copy operations. 19188 19189 - Config: no_chunk 19190 - Env Var: RCLONE_SWIFT_NO_CHUNK 19191 - Type: bool 19192 - Default: false 19193 19194 #### --swift-encoding 19195 19196 This sets the encoding for the backend. 19197 19198 See: the [encoding section in the overview](/overview/#encoding) for more info. 19199 19200 - Config: encoding 19201 - Env Var: RCLONE_SWIFT_ENCODING 19202 - Type: MultiEncoder 19203 - Default: Slash,InvalidUtf8 19204 19205 <!--- autogenerated options stop --> 19206 19207 ### Modified time ### 19208 19209 The modified time is stored as metadata on the object as 19210 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 19211 ns. 19212 19213 This is a defacto standard (used in the official python-swiftclient 19214 amongst others) for storing the modification time for an object. 19215 19216 ### Restricted filename characters 19217 19218 | Character | Value | Replacement | 19219 | --------- |:-----:|:-----------:| 19220 | NUL | 0x00 | ␀ | 19221 | / | 0x2F | / | 19222 19223 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 19224 as they can't be used in JSON strings. 19225 19226 ### Limitations ### 19227 19228 The Swift API doesn't return a correct MD5SUM for segmented files 19229 (Dynamic or Static Large Objects) so rclone won't check or use the 19230 MD5SUM for these. 19231 19232 ### Troubleshooting ### 19233 19234 #### Rclone gives Failed to create file system for "remote:": Bad Request #### 19235 19236 Due to an oddity of the underlying swift library, it gives a "Bad 19237 Request" error rather than a more sensible error when the 19238 authentication fails for Swift. 19239 19240 So this most likely means your username / password is wrong. You can 19241 investigate further with the `--dump-bodies` flag. 19242 19243 This may also be caused by specifying the region when you shouldn't 19244 have (eg OVH). 19245 19246 #### Rclone gives Failed to create file system: Response didn't have storage storage url and auth token #### 19247 19248 This is most likely caused by forgetting to specify your tenant when 19249 setting up a swift remote. 19250 19251 pCloud 19252 ----------------------------------------- 19253 19254 Paths are specified as `remote:path` 19255 19256 Paths may be as deep as required, eg `remote:directory/subdirectory`. 19257 19258 The initial setup for pCloud involves getting a token from pCloud which you 19259 need to do in your browser. `rclone config` walks you through it. 19260 19261 Here is an example of how to make a remote called `remote`. First run: 19262 19263 rclone config 19264 19265 This will guide you through an interactive setup process: 19266 19267 ``` 19268 No remotes found - make a new one 19269 n) New remote 19270 s) Set configuration password 19271 q) Quit config 19272 n/s/q> n 19273 name> remote 19274 Type of storage to configure. 19275 Choose a number from below, or type in your own value 19276 [snip] 19277 XX / Pcloud 19278 \ "pcloud" 19279 [snip] 19280 Storage> pcloud 19281 Pcloud App Client Id - leave blank normally. 19282 client_id> 19283 Pcloud App Client Secret - leave blank normally. 19284 client_secret> 19285 Remote config 19286 Use auto config? 19287 * Say Y if not sure 19288 * Say N if you are working on a remote or headless machine 19289 y) Yes 19290 n) No 19291 y/n> y 19292 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 19293 Log in and authorize rclone for access 19294 Waiting for code... 19295 Got code 19296 -------------------- 19297 [remote] 19298 client_id = 19299 client_secret = 19300 token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} 19301 -------------------- 19302 y) Yes this is OK 19303 e) Edit this remote 19304 d) Delete this remote 19305 y/e/d> y 19306 ``` 19307 19308 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 19309 machine with no Internet browser available. 19310 19311 Note that rclone runs a webserver on your local machine to collect the 19312 token as returned from pCloud. This only runs from the moment it opens 19313 your browser to the moment you get back the verification code. This 19314 is on `http://127.0.0.1:53682/` and this it may require you to unblock 19315 it temporarily if you are running a host firewall. 19316 19317 Once configured you can then use `rclone` like this, 19318 19319 List directories in top level of your pCloud 19320 19321 rclone lsd remote: 19322 19323 List all the files in your pCloud 19324 19325 rclone ls remote: 19326 19327 To copy a local directory to an pCloud directory called backup 19328 19329 rclone copy /home/source remote:backup 19330 19331 ### Modified time and hashes ### 19332 19333 pCloud allows modification times to be set on objects accurate to 1 19334 second. These will be used to detect whether objects need syncing or 19335 not. In order to set a Modification time pCloud requires the object 19336 be re-uploaded. 19337 19338 pCloud supports MD5 and SHA1 type hashes, so you can use the 19339 `--checksum` flag. 19340 19341 #### Restricted filename characters 19342 19343 In addition to the [default restricted characters set](/overview/#restricted-characters) 19344 the following characters are also replaced: 19345 19346 | Character | Value | Replacement | 19347 | --------- |:-----:|:-----------:| 19348 | \ | 0x5C | \ | 19349 19350 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 19351 as they can't be used in JSON strings. 19352 19353 ### Deleting files ### 19354 19355 Deleted files will be moved to the trash. Your subscription level 19356 will determine how long items stay in the trash. `rclone cleanup` can 19357 be used to empty the trash. 19358 19359 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/pcloud/pcloud.go then run make backenddocs --> 19360 ### Standard Options 19361 19362 Here are the standard options specific to pcloud (Pcloud). 19363 19364 #### --pcloud-client-id 19365 19366 Pcloud App Client Id 19367 Leave blank normally. 19368 19369 - Config: client_id 19370 - Env Var: RCLONE_PCLOUD_CLIENT_ID 19371 - Type: string 19372 - Default: "" 19373 19374 #### --pcloud-client-secret 19375 19376 Pcloud App Client Secret 19377 Leave blank normally. 19378 19379 - Config: client_secret 19380 - Env Var: RCLONE_PCLOUD_CLIENT_SECRET 19381 - Type: string 19382 - Default: "" 19383 19384 ### Advanced Options 19385 19386 Here are the advanced options specific to pcloud (Pcloud). 19387 19388 #### --pcloud-encoding 19389 19390 This sets the encoding for the backend. 19391 19392 See: the [encoding section in the overview](/overview/#encoding) for more info. 19393 19394 - Config: encoding 19395 - Env Var: RCLONE_PCLOUD_ENCODING 19396 - Type: MultiEncoder 19397 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 19398 19399 <!--- autogenerated options stop --> 19400 19401 premiumize.me 19402 ----------------------------------------- 19403 19404 Paths are specified as `remote:path` 19405 19406 Paths may be as deep as required, eg `remote:directory/subdirectory`. 19407 19408 The initial setup for [premiumize.me](https://premiumize.me/) involves getting a token from premiumize.me which you 19409 need to do in your browser. `rclone config` walks you through it. 19410 19411 Here is an example of how to make a remote called `remote`. First run: 19412 19413 rclone config 19414 19415 This will guide you through an interactive setup process: 19416 19417 ``` 19418 No remotes found - make a new one 19419 n) New remote 19420 s) Set configuration password 19421 q) Quit config 19422 n/s/q> n 19423 name> remote 19424 Type of storage to configure. 19425 Enter a string value. Press Enter for the default (""). 19426 Choose a number from below, or type in your own value 19427 [snip] 19428 XX / premiumize.me 19429 \ "premiumizeme" 19430 [snip] 19431 Storage> premiumizeme 19432 ** See help for premiumizeme backend at: https://rclone.org/premiumizeme/ ** 19433 19434 Remote config 19435 Use auto config? 19436 * Say Y if not sure 19437 * Say N if you are working on a remote or headless machine 19438 y) Yes 19439 n) No 19440 y/n> y 19441 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 19442 Log in and authorize rclone for access 19443 Waiting for code... 19444 Got code 19445 -------------------- 19446 [remote] 19447 type = premiumizeme 19448 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} 19449 -------------------- 19450 y) Yes this is OK 19451 e) Edit this remote 19452 d) Delete this remote 19453 y/e/d> 19454 ``` 19455 19456 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 19457 machine with no Internet browser available. 19458 19459 Note that rclone runs a webserver on your local machine to collect the 19460 token as returned from premiumize.me. This only runs from the moment it opens 19461 your browser to the moment you get back the verification code. This 19462 is on `http://127.0.0.1:53682/` and this it may require you to unblock 19463 it temporarily if you are running a host firewall. 19464 19465 Once configured you can then use `rclone` like this, 19466 19467 List directories in top level of your premiumize.me 19468 19469 rclone lsd remote: 19470 19471 List all the files in your premiumize.me 19472 19473 rclone ls remote: 19474 19475 To copy a local directory to an premiumize.me directory called backup 19476 19477 rclone copy /home/source remote:backup 19478 19479 ### Modified time and hashes ### 19480 19481 premiumize.me does not support modification times or hashes, therefore 19482 syncing will default to `--size-only` checking. Note that using 19483 `--update` will work. 19484 19485 #### Restricted filename characters 19486 19487 In addition to the [default restricted characters set](/overview/#restricted-characters) 19488 the following characters are also replaced: 19489 19490 | Character | Value | Replacement | 19491 | --------- |:-----:|:-----------:| 19492 | \ | 0x5C | \ | 19493 | " | 0x22 | " | 19494 19495 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 19496 as they can't be used in JSON strings. 19497 19498 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/premiumizeme/premiumizeme.go then run make backenddocs --> 19499 ### Standard Options 19500 19501 Here are the standard options specific to premiumizeme (premiumize.me). 19502 19503 #### --premiumizeme-api-key 19504 19505 API Key. 19506 19507 This is not normally used - use oauth instead. 19508 19509 19510 - Config: api_key 19511 - Env Var: RCLONE_PREMIUMIZEME_API_KEY 19512 - Type: string 19513 - Default: "" 19514 19515 ### Advanced Options 19516 19517 Here are the advanced options specific to premiumizeme (premiumize.me). 19518 19519 #### --premiumizeme-encoding 19520 19521 This sets the encoding for the backend. 19522 19523 See: the [encoding section in the overview](/overview/#encoding) for more info. 19524 19525 - Config: encoding 19526 - Env Var: RCLONE_PREMIUMIZEME_ENCODING 19527 - Type: MultiEncoder 19528 - Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot 19529 19530 <!--- autogenerated options stop --> 19531 19532 ### Limitations ### 19533 19534 Note that premiumize.me is case insensitive so you can't have a file called 19535 "Hello.doc" and one called "hello.doc". 19536 19537 premiumize.me file names can't have the `\` or `"` characters in. 19538 rclone maps these to and from an identical looking unicode equivalents 19539 `\` and `"` 19540 19541 premiumize.me only supports filenames up to 255 characters in length. 19542 19543 put.io 19544 --------------------------------- 19545 19546 Paths are specified as `remote:path` 19547 19548 put.io paths may be as deep as required, eg 19549 `remote:directory/subdirectory`. 19550 19551 The initial setup for put.io involves getting a token from put.io 19552 which you need to do in your browser. `rclone config` walks you 19553 through it. 19554 19555 Here is an example of how to make a remote called `remote`. First run: 19556 19557 rclone config 19558 19559 This will guide you through an interactive setup process: 19560 19561 ``` 19562 No remotes found - make a new one 19563 n) New remote 19564 s) Set configuration password 19565 q) Quit config 19566 n/s/q> n 19567 name> putio 19568 Type of storage to configure. 19569 Enter a string value. Press Enter for the default (""). 19570 Choose a number from below, or type in your own value 19571 [snip] 19572 XX / Put.io 19573 \ "putio" 19574 [snip] 19575 Storage> putio 19576 ** See help for putio backend at: https://rclone.org/putio/ ** 19577 19578 Remote config 19579 Use auto config? 19580 * Say Y if not sure 19581 * Say N if you are working on a remote or headless machine 19582 y) Yes 19583 n) No 19584 y/n> y 19585 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 19586 Log in and authorize rclone for access 19587 Waiting for code... 19588 Got code 19589 -------------------- 19590 [putio] 19591 type = putio 19592 token = {"access_token":"XXXXXXXX","expiry":"0001-01-01T00:00:00Z"} 19593 -------------------- 19594 y) Yes this is OK 19595 e) Edit this remote 19596 d) Delete this remote 19597 y/e/d> y 19598 Current remotes: 19599 19600 Name Type 19601 ==== ==== 19602 putio putio 19603 19604 e) Edit existing remote 19605 n) New remote 19606 d) Delete remote 19607 r) Rename remote 19608 c) Copy remote 19609 s) Set configuration password 19610 q) Quit config 19611 e/n/d/r/c/s/q> q 19612 ``` 19613 19614 Note that rclone runs a webserver on your local machine to collect the 19615 token as returned from Google if you use auto config mode. This only 19616 runs from the moment it opens your browser to the moment you get back 19617 the verification code. This is on `http://127.0.0.1:53682/` and this 19618 it may require you to unblock it temporarily if you are running a host 19619 firewall, or use manual mode. 19620 19621 You can then use it like this, 19622 19623 List directories in top level of your put.io 19624 19625 rclone lsd remote: 19626 19627 List all the files in your put.io 19628 19629 rclone ls remote: 19630 19631 To copy a local directory to a put.io directory called backup 19632 19633 rclone copy /home/source remote:backup 19634 19635 #### Restricted filename characters 19636 19637 In addition to the [default restricted characters set](/overview/#restricted-characters) 19638 the following characters are also replaced: 19639 19640 | Character | Value | Replacement | 19641 | --------- |:-----:|:-----------:| 19642 | \ | 0x5C | \ | 19643 19644 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 19645 as they can't be used in JSON strings. 19646 19647 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/putio/putio.go then run make backenddocs --> 19648 ### Advanced Options 19649 19650 Here are the advanced options specific to putio (Put.io). 19651 19652 #### --putio-encoding 19653 19654 This sets the encoding for the backend. 19655 19656 See: the [encoding section in the overview](/overview/#encoding) for more info. 19657 19658 - Config: encoding 19659 - Env Var: RCLONE_PUTIO_ENCODING 19660 - Type: MultiEncoder 19661 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 19662 19663 <!--- autogenerated options stop --> 19664 19665 SFTP 19666 ---------------------------------------- 19667 19668 SFTP is the [Secure (or SSH) File Transfer 19669 Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol). 19670 19671 The SFTP backend can be used with a number of different providers: 19672 19673 * C14 19674 * rsync.net 19675 19676 SFTP runs over SSH v2 and is installed as standard with most modern 19677 SSH installations. 19678 19679 Paths are specified as `remote:path`. If the path does not begin with 19680 a `/` it is relative to the home directory of the user. An empty path 19681 `remote:` refers to the user's home directory. 19682 19683 "Note that some SFTP servers will need the leading / - Synology is a 19684 good example of this. rsync.net, on the other hand, requires users to 19685 OMIT the leading /. 19686 19687 Here is an example of making an SFTP configuration. First run 19688 19689 rclone config 19690 19691 This will guide you through an interactive setup process. 19692 19693 ``` 19694 No remotes found - make a new one 19695 n) New remote 19696 s) Set configuration password 19697 q) Quit config 19698 n/s/q> n 19699 name> remote 19700 Type of storage to configure. 19701 Choose a number from below, or type in your own value 19702 [snip] 19703 XX / SSH/SFTP Connection 19704 \ "sftp" 19705 [snip] 19706 Storage> sftp 19707 SSH host to connect to 19708 Choose a number from below, or type in your own value 19709 1 / Connect to example.com 19710 \ "example.com" 19711 host> example.com 19712 SSH username, leave blank for current username, ncw 19713 user> sftpuser 19714 SSH port, leave blank to use default (22) 19715 port> 19716 SSH password, leave blank to use ssh-agent. 19717 y) Yes type in my own password 19718 g) Generate random password 19719 n) No leave this optional password blank 19720 y/g/n> n 19721 Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. 19722 key_file> 19723 Remote config 19724 -------------------- 19725 [remote] 19726 host = example.com 19727 user = sftpuser 19728 port = 19729 pass = 19730 key_file = 19731 -------------------- 19732 y) Yes this is OK 19733 e) Edit this remote 19734 d) Delete this remote 19735 y/e/d> y 19736 ``` 19737 19738 This remote is called `remote` and can now be used like this: 19739 19740 See all directories in the home directory 19741 19742 rclone lsd remote: 19743 19744 Make a new directory 19745 19746 rclone mkdir remote:path/to/directory 19747 19748 List the contents of a directory 19749 19750 rclone ls remote:path/to/directory 19751 19752 Sync `/home/local/directory` to the remote directory, deleting any 19753 excess files in the directory. 19754 19755 rclone sync /home/local/directory remote:directory 19756 19757 ### SSH Authentication ### 19758 19759 The SFTP remote supports three authentication methods: 19760 19761 * Password 19762 * Key file 19763 * ssh-agent 19764 19765 Key files should be PEM-encoded private key files. For instance `/home/$USER/.ssh/id_rsa`. 19766 Only unencrypted OpenSSH or PEM encrypted files are supported. 19767 19768 If you don't specify `pass` or `key_file` then rclone will attempt to contact an ssh-agent. 19769 19770 You can also specify `key_use_agent` to force the usage of an ssh-agent. In this case 19771 `key_file` can also be specified to force the usage of a specific key in the ssh-agent. 19772 19773 Using an ssh-agent is the only way to load encrypted OpenSSH keys at the moment. 19774 19775 If you set the `--sftp-ask-password` option, rclone will prompt for a 19776 password when needed and no password has been configured. 19777 19778 ### ssh-agent on macOS ### 19779 19780 Note that there seem to be various problems with using an ssh-agent on 19781 macOS due to recent changes in the OS. The most effective work-around 19782 seems to be to start an ssh-agent in each session, eg 19783 19784 eval `ssh-agent -s` && ssh-add -A 19785 19786 And then at the end of the session 19787 19788 eval `ssh-agent -k` 19789 19790 These commands can be used in scripts of course. 19791 19792 ### Modified time ### 19793 19794 Modified times are stored on the server to 1 second precision. 19795 19796 Modified times are used in syncing and are fully supported. 19797 19798 Some SFTP servers disable setting/modifying the file modification time after 19799 upload (for example, certain configurations of ProFTPd with mod_sftp). If you 19800 are using one of these servers, you can set the option `set_modtime = false` in 19801 your RClone backend configuration to disable this behaviour. 19802 19803 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/sftp/sftp.go then run make backenddocs --> 19804 ### Standard Options 19805 19806 Here are the standard options specific to sftp (SSH/SFTP Connection). 19807 19808 #### --sftp-host 19809 19810 SSH host to connect to 19811 19812 - Config: host 19813 - Env Var: RCLONE_SFTP_HOST 19814 - Type: string 19815 - Default: "" 19816 - Examples: 19817 - "example.com" 19818 - Connect to example.com 19819 19820 #### --sftp-user 19821 19822 SSH username, leave blank for current username, ncw 19823 19824 - Config: user 19825 - Env Var: RCLONE_SFTP_USER 19826 - Type: string 19827 - Default: "" 19828 19829 #### --sftp-port 19830 19831 SSH port, leave blank to use default (22) 19832 19833 - Config: port 19834 - Env Var: RCLONE_SFTP_PORT 19835 - Type: string 19836 - Default: "" 19837 19838 #### --sftp-pass 19839 19840 SSH password, leave blank to use ssh-agent. 19841 19842 - Config: pass 19843 - Env Var: RCLONE_SFTP_PASS 19844 - Type: string 19845 - Default: "" 19846 19847 #### --sftp-key-file 19848 19849 Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent. 19850 19851 - Config: key_file 19852 - Env Var: RCLONE_SFTP_KEY_FILE 19853 - Type: string 19854 - Default: "" 19855 19856 #### --sftp-key-file-pass 19857 19858 The passphrase to decrypt the PEM-encoded private key file. 19859 19860 Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys 19861 in the new OpenSSH format can't be used. 19862 19863 - Config: key_file_pass 19864 - Env Var: RCLONE_SFTP_KEY_FILE_PASS 19865 - Type: string 19866 - Default: "" 19867 19868 #### --sftp-key-use-agent 19869 19870 When set forces the usage of the ssh-agent. 19871 19872 When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key is 19873 requested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errors 19874 when the ssh-agent contains many keys. 19875 19876 - Config: key_use_agent 19877 - Env Var: RCLONE_SFTP_KEY_USE_AGENT 19878 - Type: bool 19879 - Default: false 19880 19881 #### --sftp-use-insecure-cipher 19882 19883 Enable the use of insecure ciphers and key exchange methods. 19884 19885 This enables the use of the the following insecure ciphers and key exchange methods: 19886 19887 - aes128-cbc 19888 - aes192-cbc 19889 - aes256-cbc 19890 - 3des-cbc 19891 - diffie-hellman-group-exchange-sha256 19892 - diffie-hellman-group-exchange-sha1 19893 19894 Those algorithms are insecure and may allow plaintext data to be recovered by an attacker. 19895 19896 - Config: use_insecure_cipher 19897 - Env Var: RCLONE_SFTP_USE_INSECURE_CIPHER 19898 - Type: bool 19899 - Default: false 19900 - Examples: 19901 - "false" 19902 - Use default Cipher list. 19903 - "true" 19904 - Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange. 19905 19906 #### --sftp-disable-hashcheck 19907 19908 Disable the execution of SSH commands to determine if remote file hashing is available. 19909 Leave blank or set to false to enable hashing (recommended), set to true to disable hashing. 19910 19911 - Config: disable_hashcheck 19912 - Env Var: RCLONE_SFTP_DISABLE_HASHCHECK 19913 - Type: bool 19914 - Default: false 19915 19916 ### Advanced Options 19917 19918 Here are the advanced options specific to sftp (SSH/SFTP Connection). 19919 19920 #### --sftp-ask-password 19921 19922 Allow asking for SFTP password when needed. 19923 19924 If this is set and no password is supplied then rclone will: 19925 - ask for a password 19926 - not contact the ssh agent 19927 19928 19929 - Config: ask_password 19930 - Env Var: RCLONE_SFTP_ASK_PASSWORD 19931 - Type: bool 19932 - Default: false 19933 19934 #### --sftp-path-override 19935 19936 Override path used by SSH connection. 19937 19938 This allows checksum calculation when SFTP and SSH paths are 19939 different. This issue affects among others Synology NAS boxes. 19940 19941 Shared folders can be found in directories representing volumes 19942 19943 rclone sync /home/local/directory remote:/directory --ssh-path-override /volume2/directory 19944 19945 Home directory can be found in a shared folder called "home" 19946 19947 rclone sync /home/local/directory remote:/home/directory --ssh-path-override /volume1/homes/USER/directory 19948 19949 - Config: path_override 19950 - Env Var: RCLONE_SFTP_PATH_OVERRIDE 19951 - Type: string 19952 - Default: "" 19953 19954 #### --sftp-set-modtime 19955 19956 Set the modified time on the remote if set. 19957 19958 - Config: set_modtime 19959 - Env Var: RCLONE_SFTP_SET_MODTIME 19960 - Type: bool 19961 - Default: true 19962 19963 #### --sftp-md5sum-command 19964 19965 The command used to read md5 hashes. Leave blank for autodetect. 19966 19967 - Config: md5sum_command 19968 - Env Var: RCLONE_SFTP_MD5SUM_COMMAND 19969 - Type: string 19970 - Default: "" 19971 19972 #### --sftp-sha1sum-command 19973 19974 The command used to read sha1 hashes. Leave blank for autodetect. 19975 19976 - Config: sha1sum_command 19977 - Env Var: RCLONE_SFTP_SHA1SUM_COMMAND 19978 - Type: string 19979 - Default: "" 19980 19981 #### --sftp-skip-links 19982 19983 Set to skip any symlinks and any other non regular files. 19984 19985 - Config: skip_links 19986 - Env Var: RCLONE_SFTP_SKIP_LINKS 19987 - Type: bool 19988 - Default: false 19989 19990 <!--- autogenerated options stop --> 19991 19992 ### Limitations ### 19993 19994 SFTP supports checksums if the same login has shell access and `md5sum` 19995 or `sha1sum` as well as `echo` are in the remote's PATH. 19996 This remote checksumming (file hashing) is recommended and enabled by default. 19997 Disabling the checksumming may be required if you are connecting to SFTP servers 19998 which are not under your control, and to which the execution of remote commands 19999 is prohibited. Set the configuration option `disable_hashcheck` to `true` to 20000 disable checksumming. 20001 20002 SFTP also supports `about` if the same login has shell 20003 access and `df` are in the remote's PATH. `about` will 20004 return the total space, free space, and used space on the remote 20005 for the disk of the specified path on the remote or, if not set, 20006 the disk of the root on the remote. 20007 `about` will fail if it does not have shell 20008 access or if `df` is not in the remote's PATH. 20009 20010 Note that some SFTP servers (eg Synology) the paths are different for 20011 SSH and SFTP so the hashes can't be calculated properly. For them 20012 using `disable_hashcheck` is a good idea. 20013 20014 The only ssh agent supported under Windows is Putty's pageant. 20015 20016 The Go SSH library disables the use of the aes128-cbc cipher by 20017 default, due to security concerns. This can be re-enabled on a 20018 per-connection basis by setting the `use_insecure_cipher` setting in 20019 the configuration file to `true`. Further details on the insecurity of 20020 this cipher can be found [in this paper] 20021 (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf). 20022 20023 SFTP isn't supported under plan9 until [this 20024 issue](https://github.com/pkg/sftp/issues/156) is fixed. 20025 20026 Note that since SFTP isn't HTTP based the following flags don't work 20027 with it: `--dump-headers`, `--dump-bodies`, `--dump-auth` 20028 20029 Note that `--timeout` isn't supported (but `--contimeout` is). 20030 20031 20032 ## C14 {#c14} 20033 20034 C14 is supported through the SFTP backend. 20035 20036 See [C14's documentation](https://www.online.net/en/storage/c14-cold-storage) 20037 20038 ## rsync.net {#rsync-net} 20039 20040 rsync.net is supported through the SFTP backend. 20041 20042 See [rsync.net's documentation of rclone examples](https://www.rsync.net/products/rclone.html). 20043 20044 SugarSync 20045 ----------------------------------------- 20046 20047 [SugarSync](https://sugarsync.com) is a cloud service that enables 20048 active synchronization of files across computers and other devices for 20049 file backup, access, syncing, and sharing. 20050 20051 The initial setup for SugarSync involves getting a token from SugarSync which you 20052 can do with rclone. `rclone config` walks you through it. 20053 20054 Here is an example of how to make a remote called `remote`. First run: 20055 20056 rclone config 20057 20058 This will guide you through an interactive setup process: 20059 20060 ``` 20061 No remotes found - make a new one 20062 n) New remote 20063 s) Set configuration password 20064 q) Quit config 20065 n/s/q> n 20066 name> remote 20067 Type of storage to configure. 20068 Enter a string value. Press Enter for the default (""). 20069 Choose a number from below, or type in your own value 20070 [snip] 20071 XX / Sugarsync 20072 \ "sugarsync" 20073 [snip] 20074 Storage> sugarsync 20075 ** See help for sugarsync backend at: https://rclone.org/sugarsync/ ** 20076 20077 Sugarsync App ID. 20078 Leave blank to use rclone's. 20079 Enter a string value. Press Enter for the default (""). 20080 app_id> 20081 Sugarsync Access Key ID. 20082 Leave blank to use rclone's. 20083 Enter a string value. Press Enter for the default (""). 20084 access_key_id> 20085 Sugarsync Private Access Key 20086 Leave blank to use rclone's. 20087 Enter a string value. Press Enter for the default (""). 20088 private_access_key> 20089 Permanently delete files if true 20090 otherwise put them in the deleted files. 20091 Enter a boolean value (true or false). Press Enter for the default ("false"). 20092 hard_delete> 20093 Edit advanced config? (y/n) 20094 y) Yes 20095 n) No (default) 20096 y/n> n 20097 Remote config 20098 Username (email address)> nick@craig-wood.com 20099 Your Sugarsync password is only required during setup and will not be stored. 20100 password: 20101 -------------------- 20102 [remote] 20103 type = sugarsync 20104 refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX 20105 -------------------- 20106 y) Yes this is OK (default) 20107 e) Edit this remote 20108 d) Delete this remote 20109 y/e/d> y 20110 ``` 20111 20112 Note that the config asks for your email and password but doesn't 20113 store them, it only uses them to get the initial token. 20114 20115 Once configured you can then use `rclone` like this, 20116 20117 List directories (sync folders) in top level of your SugarSync 20118 20119 rclone lsd remote: 20120 20121 List all the files in your SugarSync folder "Test" 20122 20123 rclone ls remote:Test 20124 20125 To copy a local directory to an SugarSync folder called backup 20126 20127 rclone copy /home/source remote:backup 20128 20129 Paths are specified as `remote:path` 20130 20131 Paths may be as deep as required, eg `remote:directory/subdirectory`. 20132 20133 **NB** you can't create files in the top level folder you have to 20134 create a folder, which rclone will create as a "Sync Folder" with 20135 SugarSync. 20136 20137 ### Modified time and hashes ### 20138 20139 SugarSync does not support modification times or hashes, therefore 20140 syncing will default to `--size-only` checking. Note that using 20141 `--update` will work as rclone can read the time files were uploaded. 20142 20143 #### Restricted filename characters 20144 20145 SugarSync replaces the [default restricted characters set](/overview/#restricted-characters) 20146 except for DEL. 20147 20148 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 20149 as they can't be used in XML strings. 20150 20151 ### Deleting files ### 20152 20153 Deleted files will be moved to the "Deleted items" folder by default. 20154 20155 However you can supply the flag `--sugarsync-hard-delete` or set the 20156 config parameter `hard_delete = true` if you would like files to be 20157 deleted straight away. 20158 20159 20160 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/sugarsync/sugarsync.go then run make backenddocs --> 20161 ### Standard Options 20162 20163 Here are the standard options specific to sugarsync (Sugarsync). 20164 20165 #### --sugarsync-app-id 20166 20167 Sugarsync App ID. 20168 20169 Leave blank to use rclone's. 20170 20171 - Config: app_id 20172 - Env Var: RCLONE_SUGARSYNC_APP_ID 20173 - Type: string 20174 - Default: "" 20175 20176 #### --sugarsync-access-key-id 20177 20178 Sugarsync Access Key ID. 20179 20180 Leave blank to use rclone's. 20181 20182 - Config: access_key_id 20183 - Env Var: RCLONE_SUGARSYNC_ACCESS_KEY_ID 20184 - Type: string 20185 - Default: "" 20186 20187 #### --sugarsync-private-access-key 20188 20189 Sugarsync Private Access Key 20190 20191 Leave blank to use rclone's. 20192 20193 - Config: private_access_key 20194 - Env Var: RCLONE_SUGARSYNC_PRIVATE_ACCESS_KEY 20195 - Type: string 20196 - Default: "" 20197 20198 #### --sugarsync-hard-delete 20199 20200 Permanently delete files if true 20201 otherwise put them in the deleted files. 20202 20203 - Config: hard_delete 20204 - Env Var: RCLONE_SUGARSYNC_HARD_DELETE 20205 - Type: bool 20206 - Default: false 20207 20208 ### Advanced Options 20209 20210 Here are the advanced options specific to sugarsync (Sugarsync). 20211 20212 #### --sugarsync-refresh-token 20213 20214 Sugarsync refresh token 20215 20216 Leave blank normally, will be auto configured by rclone. 20217 20218 - Config: refresh_token 20219 - Env Var: RCLONE_SUGARSYNC_REFRESH_TOKEN 20220 - Type: string 20221 - Default: "" 20222 20223 #### --sugarsync-authorization 20224 20225 Sugarsync authorization 20226 20227 Leave blank normally, will be auto configured by rclone. 20228 20229 - Config: authorization 20230 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION 20231 - Type: string 20232 - Default: "" 20233 20234 #### --sugarsync-authorization-expiry 20235 20236 Sugarsync authorization expiry 20237 20238 Leave blank normally, will be auto configured by rclone. 20239 20240 - Config: authorization_expiry 20241 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION_EXPIRY 20242 - Type: string 20243 - Default: "" 20244 20245 #### --sugarsync-user 20246 20247 Sugarsync user 20248 20249 Leave blank normally, will be auto configured by rclone. 20250 20251 - Config: user 20252 - Env Var: RCLONE_SUGARSYNC_USER 20253 - Type: string 20254 - Default: "" 20255 20256 #### --sugarsync-root-id 20257 20258 Sugarsync root id 20259 20260 Leave blank normally, will be auto configured by rclone. 20261 20262 - Config: root_id 20263 - Env Var: RCLONE_SUGARSYNC_ROOT_ID 20264 - Type: string 20265 - Default: "" 20266 20267 #### --sugarsync-deleted-id 20268 20269 Sugarsync deleted folder id 20270 20271 Leave blank normally, will be auto configured by rclone. 20272 20273 - Config: deleted_id 20274 - Env Var: RCLONE_SUGARSYNC_DELETED_ID 20275 - Type: string 20276 - Default: "" 20277 20278 #### --sugarsync-encoding 20279 20280 This sets the encoding for the backend. 20281 20282 See: the [encoding section in the overview](/overview/#encoding) for more info. 20283 20284 - Config: encoding 20285 - Env Var: RCLONE_SUGARSYNC_ENCODING 20286 - Type: MultiEncoder 20287 - Default: Slash,Ctl,InvalidUtf8,Dot 20288 20289 <!--- autogenerated options stop --> 20290 20291 Union 20292 ----------------------------------------- 20293 20294 The `union` remote provides a unification similar to UnionFS using other remotes. 20295 20296 Paths may be as deep as required or a local path, 20297 eg `remote:directory/subdirectory` or `/directory/subdirectory`. 20298 20299 During the initial setup with `rclone config` you will specify the target 20300 remotes as a space separated list. The target remotes can either be a local paths or other remotes. 20301 20302 The order of the remotes is important as it defines which remotes take precedence over others if there are files with the same name in the same logical path. 20303 The last remote is the topmost remote and replaces files with the same name from previous remotes. 20304 20305 Only the last remote is used to write to and delete from, all other remotes are read-only. 20306 20307 Subfolders can be used in target remote. Assume a union remote named `backup` 20308 with the remotes `mydrive:private/backup mydrive2:/backup`. Invoking `rclone mkdir backup:desktop` 20309 is exactly the same as invoking `rclone mkdir mydrive2:/backup/desktop`. 20310 20311 There will be no special handling of paths containing `..` segments. 20312 Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking 20313 `rclone mkdir mydrive2:/backup/../desktop`. 20314 20315 Here is an example of how to make a union called `remote` for local folders. 20316 First run: 20317 20318 rclone config 20319 20320 This will guide you through an interactive setup process: 20321 20322 ``` 20323 No remotes found - make a new one 20324 n) New remote 20325 s) Set configuration password 20326 q) Quit config 20327 n/s/q> n 20328 name> remote 20329 Type of storage to configure. 20330 Choose a number from below, or type in your own value 20331 [snip] 20332 XX / Union merges the contents of several remotes 20333 \ "union" 20334 [snip] 20335 Storage> union 20336 List of space separated remotes. 20337 Can be 'remotea:test/dir remoteb:', '"remotea:test/space dir" remoteb:', etc. 20338 The last remote is used to write to. 20339 Enter a string value. Press Enter for the default (""). 20340 remotes> 20341 Remote config 20342 -------------------- 20343 [remote] 20344 type = union 20345 remotes = C:\dir1 C:\dir2 C:\dir3 20346 -------------------- 20347 y) Yes this is OK 20348 e) Edit this remote 20349 d) Delete this remote 20350 y/e/d> y 20351 Current remotes: 20352 20353 Name Type 20354 ==== ==== 20355 remote union 20356 20357 e) Edit existing remote 20358 n) New remote 20359 d) Delete remote 20360 r) Rename remote 20361 c) Copy remote 20362 s) Set configuration password 20363 q) Quit config 20364 e/n/d/r/c/s/q> q 20365 ``` 20366 20367 Once configured you can then use `rclone` like this, 20368 20369 List directories in top level in `C:\dir1`, `C:\dir2` and `C:\dir3` 20370 20371 rclone lsd remote: 20372 20373 List all the files in `C:\dir1`, `C:\dir2` and `C:\dir3` 20374 20375 rclone ls remote: 20376 20377 Copy another local directory to the union directory called source, which will be placed into `C:\dir3` 20378 20379 rclone copy C:\source remote:source 20380 20381 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/union/union.go then run make backenddocs --> 20382 ### Standard Options 20383 20384 Here are the standard options specific to union (Union merges the contents of several remotes). 20385 20386 #### --union-remotes 20387 20388 List of space separated remotes. 20389 Can be 'remotea:test/dir remoteb:', '"remotea:test/space dir" remoteb:', etc. 20390 The last remote is used to write to. 20391 20392 - Config: remotes 20393 - Env Var: RCLONE_UNION_REMOTES 20394 - Type: string 20395 - Default: "" 20396 20397 <!--- autogenerated options stop --> 20398 20399 WebDAV 20400 ----------------------------------------- 20401 20402 Paths are specified as `remote:path` 20403 20404 Paths may be as deep as required, eg `remote:directory/subdirectory`. 20405 20406 To configure the WebDAV remote you will need to have a URL for it, and 20407 a username and password. If you know what kind of system you are 20408 connecting to then rclone can enable extra features. 20409 20410 Here is an example of how to make a remote called `remote`. First run: 20411 20412 rclone config 20413 20414 This will guide you through an interactive setup process: 20415 20416 ``` 20417 No remotes found - make a new one 20418 n) New remote 20419 s) Set configuration password 20420 q) Quit config 20421 n/s/q> n 20422 name> remote 20423 Type of storage to configure. 20424 Choose a number from below, or type in your own value 20425 [snip] 20426 XX / Webdav 20427 \ "webdav" 20428 [snip] 20429 Storage> webdav 20430 URL of http host to connect to 20431 Choose a number from below, or type in your own value 20432 1 / Connect to example.com 20433 \ "https://example.com" 20434 url> https://example.com/remote.php/webdav/ 20435 Name of the Webdav site/service/software you are using 20436 Choose a number from below, or type in your own value 20437 1 / Nextcloud 20438 \ "nextcloud" 20439 2 / Owncloud 20440 \ "owncloud" 20441 3 / Sharepoint 20442 \ "sharepoint" 20443 4 / Other site/service or software 20444 \ "other" 20445 vendor> 1 20446 User name 20447 user> user 20448 Password. 20449 y) Yes type in my own password 20450 g) Generate random password 20451 n) No leave this optional password blank 20452 y/g/n> y 20453 Enter the password: 20454 password: 20455 Confirm the password: 20456 password: 20457 Bearer token instead of user/pass (eg a Macaroon) 20458 bearer_token> 20459 Remote config 20460 -------------------- 20461 [remote] 20462 type = webdav 20463 url = https://example.com/remote.php/webdav/ 20464 vendor = nextcloud 20465 user = user 20466 pass = *** ENCRYPTED *** 20467 bearer_token = 20468 -------------------- 20469 y) Yes this is OK 20470 e) Edit this remote 20471 d) Delete this remote 20472 y/e/d> y 20473 ``` 20474 20475 Once configured you can then use `rclone` like this, 20476 20477 List directories in top level of your WebDAV 20478 20479 rclone lsd remote: 20480 20481 List all the files in your WebDAV 20482 20483 rclone ls remote: 20484 20485 To copy a local directory to an WebDAV directory called backup 20486 20487 rclone copy /home/source remote:backup 20488 20489 ### Modified time and hashes ### 20490 20491 Plain WebDAV does not support modified times. However when used with 20492 Owncloud or Nextcloud rclone will support modified times. 20493 20494 Likewise plain WebDAV does not support hashes, however when used with 20495 Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes. 20496 Depending on the exact version of Owncloud or Nextcloud hashes may 20497 appear on all objects, or only on objects which had a hash uploaded 20498 with them. 20499 20500 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/webdav/webdav.go then run make backenddocs --> 20501 ### Standard Options 20502 20503 Here are the standard options specific to webdav (Webdav). 20504 20505 #### --webdav-url 20506 20507 URL of http host to connect to 20508 20509 - Config: url 20510 - Env Var: RCLONE_WEBDAV_URL 20511 - Type: string 20512 - Default: "" 20513 - Examples: 20514 - "https://example.com" 20515 - Connect to example.com 20516 20517 #### --webdav-vendor 20518 20519 Name of the Webdav site/service/software you are using 20520 20521 - Config: vendor 20522 - Env Var: RCLONE_WEBDAV_VENDOR 20523 - Type: string 20524 - Default: "" 20525 - Examples: 20526 - "nextcloud" 20527 - Nextcloud 20528 - "owncloud" 20529 - Owncloud 20530 - "sharepoint" 20531 - Sharepoint 20532 - "other" 20533 - Other site/service or software 20534 20535 #### --webdav-user 20536 20537 User name 20538 20539 - Config: user 20540 - Env Var: RCLONE_WEBDAV_USER 20541 - Type: string 20542 - Default: "" 20543 20544 #### --webdav-pass 20545 20546 Password. 20547 20548 - Config: pass 20549 - Env Var: RCLONE_WEBDAV_PASS 20550 - Type: string 20551 - Default: "" 20552 20553 #### --webdav-bearer-token 20554 20555 Bearer token instead of user/pass (eg a Macaroon) 20556 20557 - Config: bearer_token 20558 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN 20559 - Type: string 20560 - Default: "" 20561 20562 ### Advanced Options 20563 20564 Here are the advanced options specific to webdav (Webdav). 20565 20566 #### --webdav-bearer-token-command 20567 20568 Command to run to get a bearer token 20569 20570 - Config: bearer_token_command 20571 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN_COMMAND 20572 - Type: string 20573 - Default: "" 20574 20575 <!--- autogenerated options stop --> 20576 20577 ## Provider notes ## 20578 20579 See below for notes on specific providers. 20580 20581 ### Owncloud ### 20582 20583 Click on the settings cog in the bottom right of the page and this 20584 will show the WebDAV URL that rclone needs in the config step. It 20585 will look something like `https://example.com/remote.php/webdav/`. 20586 20587 Owncloud supports modified times using the `X-OC-Mtime` header. 20588 20589 ### Nextcloud ### 20590 20591 This is configured in an identical way to Owncloud. Note that 20592 Nextcloud does not support streaming of files (`rcat`) whereas 20593 Owncloud does. This [may be 20594 fixed](https://github.com/nextcloud/nextcloud-snap/issues/365) in the 20595 future. 20596 20597 ### Sharepoint ### 20598 20599 Rclone can be used with Sharepoint provided by OneDrive for Business 20600 or Office365 Education Accounts. 20601 This feature is only needed for a few of these Accounts, 20602 mostly Office365 Education ones. These accounts are sometimes not 20603 verified by the domain owner [github#1975](https://github.com/rclone/rclone/issues/1975) 20604 20605 This means that these accounts can't be added using the official 20606 API (other Accounts should work with the "onedrive" option). However, 20607 it is possible to access them using webdav. 20608 20609 To use a sharepoint remote with rclone, add it like this: 20610 First, you need to get your remote's URL: 20611 20612 - Go [here](https://onedrive.live.com/about/en-us/signin/) 20613 to open your OneDrive or to sign in 20614 - Now take a look at your address bar, the URL should look like this: 20615 `https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/_layouts/15/onedrive.aspx` 20616 20617 You'll only need this URL upto the email address. After that, you'll 20618 most likely want to add "/Documents". That subdirectory contains 20619 the actual data stored on your OneDrive. 20620 20621 Add the remote to rclone like this: 20622 Configure the `url` as `https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents` 20623 and use your normal account email and password for `user` and `pass`. 20624 If you have 2FA enabled, you have to generate an app password. 20625 Set the `vendor` to `sharepoint`. 20626 20627 Your config file should look like this: 20628 20629 ``` 20630 [sharepoint] 20631 type = webdav 20632 url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents 20633 vendor = other 20634 user = YourEmailAddress 20635 pass = encryptedpassword 20636 ``` 20637 20638 #### Required Flags for SharePoint #### 20639 As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer. 20640 20641 For Rclone calls copying files (especially Office files such as .docx, .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents: 20642 20643 ``` 20644 --ignore-size --ignore-checksum --update 20645 ``` 20646 20647 ### dCache ### 20648 20649 dCache is a storage system that supports many protocols and 20650 authentication/authorisation schemes. For WebDAV clients, it allows 20651 users to authenticate with username and password (BASIC), X.509, 20652 Kerberos, and various bearer tokens, including 20653 [Macaroons](https://www.dcache.org/manuals/workshop-2017-05-29-Umea/000-Final/anupam_macaroons_v02.pdf) 20654 and [OpenID-Connect](https://en.wikipedia.org/wiki/OpenID_Connect) 20655 access tokens. 20656 20657 Configure as normal using the `other` type. Don't enter a username or 20658 password, instead enter your Macaroon as the `bearer_token`. 20659 20660 The config will end up looking something like this. 20661 20662 ``` 20663 [dcache] 20664 type = webdav 20665 url = https://dcache... 20666 vendor = other 20667 user = 20668 pass = 20669 bearer_token = your-macaroon 20670 ``` 20671 20672 There is a [script](https://github.com/sara-nl/GridScripts/blob/master/get-macaroon) that 20673 obtains a Macaroon from a dCache WebDAV endpoint, and creates an rclone config file. 20674 20675 Macaroons may also be obtained from the dCacheView 20676 web-browser/JavaScript client that comes with dCache. 20677 20678 ### OpenID-Connect ### 20679 20680 dCache also supports authenticating with OpenID-Connect access tokens. 20681 OpenID-Connect is a protocol (based on OAuth 2.0) that allows services 20682 to identify users who have authenticated with some central service. 20683 20684 Support for OpenID-Connect in rclone is currently achieved using 20685 another software package called 20686 [oidc-agent](https://github.com/indigo-dc/oidc-agent). This is a 20687 command-line tool that facilitates obtaining an access token. Once 20688 installed and configured, an access token is obtained by running the 20689 `oidc-token` command. The following example shows a (shortened) 20690 access token obtained from the *XDC* OIDC Provider. 20691 20692 ``` 20693 paul@celebrimbor:~$ oidc-token XDC 20694 eyJraWQ[...]QFXDt0 20695 paul@celebrimbor:~$ 20696 ``` 20697 20698 **Note** Before the `oidc-token` command will work, the refresh token 20699 must be loaded into the oidc agent. This is done with the `oidc-add` 20700 command (e.g., `oidc-add XDC`). This is typically done once per login 20701 session. Full details on this and how to register oidc-agent with 20702 your OIDC Provider are provided in the [oidc-agent 20703 documentation](https://indigo-dc.gitbooks.io/oidc-agent/). 20704 20705 The rclone `bearer_token_command` configuration option is used to 20706 fetch the access token from oidc-agent. 20707 20708 Configure as a normal WebDAV endpoint, using the 'other' vendor, 20709 leaving the username and password empty. When prompted, choose to 20710 edit the advanced config and enter the command to get a bearer token 20711 (e.g., `oidc-agent XDC`). 20712 20713 The following example config shows a WebDAV endpoint that uses 20714 oidc-agent to supply an access token from the *XDC* OIDC Provider. 20715 20716 ``` 20717 [dcache] 20718 type = webdav 20719 url = https://dcache.example.org/ 20720 vendor = other 20721 bearer_token_command = oidc-token XDC 20722 ``` 20723 20724 Yandex Disk 20725 ---------------------------------------- 20726 20727 [Yandex Disk](https://disk.yandex.com) is a cloud storage solution created by [Yandex](https://yandex.com). 20728 20729 Here is an example of making a yandex configuration. First run 20730 20731 rclone config 20732 20733 This will guide you through an interactive setup process: 20734 20735 ``` 20736 No remotes found - make a new one 20737 n) New remote 20738 s) Set configuration password 20739 n/s> n 20740 name> remote 20741 Type of storage to configure. 20742 Choose a number from below, or type in your own value 20743 [snip] 20744 XX / Yandex Disk 20745 \ "yandex" 20746 [snip] 20747 Storage> yandex 20748 Yandex Client Id - leave blank normally. 20749 client_id> 20750 Yandex Client Secret - leave blank normally. 20751 client_secret> 20752 Remote config 20753 Use auto config? 20754 * Say Y if not sure 20755 * Say N if you are working on a remote or headless machine 20756 y) Yes 20757 n) No 20758 y/n> y 20759 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 20760 Log in and authorize rclone for access 20761 Waiting for code... 20762 Got code 20763 -------------------- 20764 [remote] 20765 client_id = 20766 client_secret = 20767 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","expiry":"2016-12-29T12:27:11.362788025Z"} 20768 -------------------- 20769 y) Yes this is OK 20770 e) Edit this remote 20771 d) Delete this remote 20772 y/e/d> y 20773 ``` 20774 20775 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 20776 machine with no Internet browser available. 20777 20778 Note that rclone runs a webserver on your local machine to collect the 20779 token as returned from Yandex Disk. This only runs from the moment it 20780 opens your browser to the moment you get back the verification code. 20781 This is on `http://127.0.0.1:53682/` and this it may require you to 20782 unblock it temporarily if you are running a host firewall. 20783 20784 Once configured you can then use `rclone` like this, 20785 20786 See top level directories 20787 20788 rclone lsd remote: 20789 20790 Make a new directory 20791 20792 rclone mkdir remote:directory 20793 20794 List the contents of a directory 20795 20796 rclone ls remote:directory 20797 20798 Sync `/home/local/directory` to the remote path, deleting any 20799 excess files in the path. 20800 20801 rclone sync /home/local/directory remote:directory 20802 20803 Yandex paths may be as deep as required, eg `remote:directory/subdirectory`. 20804 20805 ### Modified time ### 20806 20807 Modified times are supported and are stored accurate to 1 ns in custom 20808 metadata called `rclone_modified` in RFC3339 with nanoseconds format. 20809 20810 ### MD5 checksums ### 20811 20812 MD5 checksums are natively supported by Yandex Disk. 20813 20814 ### Emptying Trash ### 20815 20816 If you wish to empty your trash you can use the `rclone cleanup remote:` 20817 command which will permanently delete all your trashed files. This command 20818 does not take any path arguments. 20819 20820 ### Quota information ### 20821 20822 To view your current quota you can use the `rclone about remote:` 20823 command which will display your usage limit (quota) and the current usage. 20824 20825 #### Restricted filename characters 20826 20827 The [default restricted characters set](/overview/#restricted-characters) 20828 are replaced. 20829 20830 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 20831 as they can't be used in JSON strings. 20832 20833 ### Limitations ### 20834 20835 When uploading very large files (bigger than about 5GB) you will need 20836 to increase the `--timeout` parameter. This is because Yandex pauses 20837 (perhaps to calculate the MD5SUM for the entire file) before returning 20838 confirmation that the file has been uploaded. The default handling of 20839 timeouts in rclone is to assume a 5 minute pause is an error and close 20840 the connection - you'll see `net/http: timeout awaiting response 20841 headers` errors in the logs if this is happening. Setting the timeout 20842 to twice the max size of file in GB should be enough, so if you want 20843 to upload a 30GB file set a timeout of `2 * 30 = 60m`, that is 20844 `--timeout 60m`. 20845 20846 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/yandex/yandex.go then run make backenddocs --> 20847 ### Standard Options 20848 20849 Here are the standard options specific to yandex (Yandex Disk). 20850 20851 #### --yandex-client-id 20852 20853 Yandex Client Id 20854 Leave blank normally. 20855 20856 - Config: client_id 20857 - Env Var: RCLONE_YANDEX_CLIENT_ID 20858 - Type: string 20859 - Default: "" 20860 20861 #### --yandex-client-secret 20862 20863 Yandex Client Secret 20864 Leave blank normally. 20865 20866 - Config: client_secret 20867 - Env Var: RCLONE_YANDEX_CLIENT_SECRET 20868 - Type: string 20869 - Default: "" 20870 20871 ### Advanced Options 20872 20873 Here are the advanced options specific to yandex (Yandex Disk). 20874 20875 #### --yandex-unlink 20876 20877 Remove existing public link to file/folder with link command rather than creating. 20878 Default is false, meaning link command will create or retrieve public link. 20879 20880 - Config: unlink 20881 - Env Var: RCLONE_YANDEX_UNLINK 20882 - Type: bool 20883 - Default: false 20884 20885 #### --yandex-encoding 20886 20887 This sets the encoding for the backend. 20888 20889 See: the [encoding section in the overview](/overview/#encoding) for more info. 20890 20891 - Config: encoding 20892 - Env Var: RCLONE_YANDEX_ENCODING 20893 - Type: MultiEncoder 20894 - Default: Slash,Del,Ctl,InvalidUtf8,Dot 20895 20896 <!--- autogenerated options stop --> 20897 20898 Local Filesystem 20899 ------------------------------------------- 20900 20901 Local paths are specified as normal filesystem paths, eg `/path/to/wherever`, so 20902 20903 rclone sync /home/source /tmp/destination 20904 20905 Will sync `/home/source` to `/tmp/destination` 20906 20907 These can be configured into the config file for consistencies sake, 20908 but it is probably easier not to. 20909 20910 ### Modified time ### 20911 20912 Rclone reads and writes the modified time using an accuracy determined by 20913 the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second 20914 on OS X. 20915 20916 ### Filenames ### 20917 20918 Filenames should be encoded in UTF-8 on disk. This is the normal case 20919 for Windows and OS X. 20920 20921 There is a bit more uncertainty in the Linux world, but new 20922 distributions will have UTF-8 encoded files names. If you are using an 20923 old Linux filesystem with non UTF-8 file names (eg latin1) then you 20924 can use the `convmv` tool to convert the filesystem to UTF-8. This 20925 tool is available in most distributions' package managers. 20926 20927 If an invalid (non-UTF8) filename is read, the invalid characters will 20928 be replaced with a quoted representation of the invalid bytes. The name 20929 `gro\xdf` will be transferred as `gro‛DF`. `rclone` will emit a debug 20930 message in this case (use `-v` to see), eg 20931 20932 ``` 20933 Local file system at .: Replacing invalid UTF-8 characters in "gro\xdf" 20934 ``` 20935 20936 #### Restricted characters 20937 20938 On non Windows platforms the following characters are replaced when 20939 handling file names. 20940 20941 | Character | Value | Replacement | 20942 | --------- |:-----:|:-----------:| 20943 | NUL | 0x00 | ␀ | 20944 | / | 0x2F | / | 20945 20946 When running on Windows the following characters are replaced. This 20947 list is based on the [Windows file naming conventions](https://docs.microsoft.com/de-de/windows/desktop/FileIO/naming-a-file#naming-conventions). 20948 20949 | Character | Value | Replacement | 20950 | --------- |:-----:|:-----------:| 20951 | NUL | 0x00 | ␀ | 20952 | SOH | 0x01 | ␁ | 20953 | STX | 0x02 | ␂ | 20954 | ETX | 0x03 | ␃ | 20955 | EOT | 0x04 | ␄ | 20956 | ENQ | 0x05 | ␅ | 20957 | ACK | 0x06 | ␆ | 20958 | BEL | 0x07 | ␇ | 20959 | BS | 0x08 | ␈ | 20960 | HT | 0x09 | ␉ | 20961 | LF | 0x0A | ␊ | 20962 | VT | 0x0B | ␋ | 20963 | FF | 0x0C | ␌ | 20964 | CR | 0x0D | ␍ | 20965 | SO | 0x0E | ␎ | 20966 | SI | 0x0F | ␏ | 20967 | DLE | 0x10 | ␐ | 20968 | DC1 | 0x11 | ␑ | 20969 | DC2 | 0x12 | ␒ | 20970 | DC3 | 0x13 | ␓ | 20971 | DC4 | 0x14 | ␔ | 20972 | NAK | 0x15 | ␕ | 20973 | SYN | 0x16 | ␖ | 20974 | ETB | 0x17 | ␗ | 20975 | CAN | 0x18 | ␘ | 20976 | EM | 0x19 | ␙ | 20977 | SUB | 0x1A | ␚ | 20978 | ESC | 0x1B | ␛ | 20979 | FS | 0x1C | ␜ | 20980 | GS | 0x1D | ␝ | 20981 | RS | 0x1E | ␞ | 20982 | US | 0x1F | ␟ | 20983 | / | 0x2F | / | 20984 | " | 0x22 | " | 20985 | * | 0x2A | * | 20986 | : | 0x3A | : | 20987 | < | 0x3C | < | 20988 | > | 0x3E | > | 20989 | ? | 0x3F | ? | 20990 | \ | 0x5C | \ | 20991 | \| | 0x7C | | | 20992 20993 File names on Windows can also not end with the following characters. 20994 These only get replaced if they are last character in the name: 20995 20996 | Character | Value | Replacement | 20997 | --------- |:-----:|:-----------:| 20998 | SP | 0x20 | ␠ | 20999 | . | 0x2E | . | 21000 21001 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 21002 as they can't be converted to UTF-16. 21003 21004 ### Long paths on Windows ### 21005 21006 Rclone handles long paths automatically, by converting all paths to long 21007 [UNC paths](https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath) 21008 which allows paths up to 32,767 characters. 21009 21010 This is why you will see that your paths, for instance `c:\files` is 21011 converted to the UNC path `\\?\c:\files` in the output, 21012 and `\\server\share` is converted to `\\?\UNC\server\share`. 21013 21014 However, in rare cases this may cause problems with buggy file 21015 system drivers like [EncFS](https://github.com/rclone/rclone/issues/261). 21016 To disable UNC conversion globally, add this to your `.rclone.conf` file: 21017 21018 ``` 21019 [local] 21020 nounc = true 21021 ``` 21022 21023 If you want to selectively disable UNC, you can add it to a separate entry like this: 21024 21025 ``` 21026 [nounc] 21027 type = local 21028 nounc = true 21029 ``` 21030 And use rclone like this: 21031 21032 `rclone copy c:\src nounc:z:\dst` 21033 21034 This will use UNC paths on `c:\src` but not on `z:\dst`. 21035 Of course this will cause problems if the absolute path length of a 21036 file exceeds 258 characters on z, so only use this option if you have to. 21037 21038 ### Symlinks / Junction points 21039 21040 Normally rclone will ignore symlinks or junction points (which behave 21041 like symlinks under Windows). 21042 21043 If you supply `--copy-links` or `-L` then rclone will follow the 21044 symlink and copy the pointed to file or directory. Note that this 21045 flag is incompatible with `-links` / `-l`. 21046 21047 This flag applies to all commands. 21048 21049 For example, supposing you have a directory structure like this 21050 21051 ``` 21052 $ tree /tmp/a 21053 /tmp/a 21054 ├── b -> ../b 21055 ├── expected -> ../expected 21056 ├── one 21057 └── two 21058 └── three 21059 ``` 21060 21061 Then you can see the difference with and without the flag like this 21062 21063 ``` 21064 $ rclone ls /tmp/a 21065 6 one 21066 6 two/three 21067 ``` 21068 21069 and 21070 21071 ``` 21072 $ rclone -L ls /tmp/a 21073 4174 expected 21074 6 one 21075 6 two/three 21076 6 b/two 21077 6 b/one 21078 ``` 21079 21080 #### --links, -l 21081 21082 Normally rclone will ignore symlinks or junction points (which behave 21083 like symlinks under Windows). 21084 21085 If you supply this flag then rclone will copy symbolic links from the local storage, 21086 and store them as text files, with a '.rclonelink' suffix in the remote storage. 21087 21088 The text file will contain the target of the symbolic link (see example). 21089 21090 This flag applies to all commands. 21091 21092 For example, supposing you have a directory structure like this 21093 21094 ``` 21095 $ tree /tmp/a 21096 /tmp/a 21097 ├── file1 -> ./file4 21098 └── file2 -> /home/user/file3 21099 ``` 21100 21101 Copying the entire directory with '-l' 21102 21103 ``` 21104 $ rclone copyto -l /tmp/a/file1 remote:/tmp/a/ 21105 ``` 21106 21107 The remote files are created with a '.rclonelink' suffix 21108 21109 ``` 21110 $ rclone ls remote:/tmp/a 21111 5 file1.rclonelink 21112 14 file2.rclonelink 21113 ``` 21114 21115 The remote files will contain the target of the symbolic links 21116 21117 ``` 21118 $ rclone cat remote:/tmp/a/file1.rclonelink 21119 ./file4 21120 21121 $ rclone cat remote:/tmp/a/file2.rclonelink 21122 /home/user/file3 21123 ``` 21124 21125 Copying them back with '-l' 21126 21127 ``` 21128 $ rclone copyto -l remote:/tmp/a/ /tmp/b/ 21129 21130 $ tree /tmp/b 21131 /tmp/b 21132 ├── file1 -> ./file4 21133 └── file2 -> /home/user/file3 21134 ``` 21135 21136 However, if copied back without '-l' 21137 21138 ``` 21139 $ rclone copyto remote:/tmp/a/ /tmp/b/ 21140 21141 $ tree /tmp/b 21142 /tmp/b 21143 ├── file1.rclonelink 21144 └── file2.rclonelink 21145 ```` 21146 21147 Note that this flag is incompatible with `-copy-links` / `-L`. 21148 21149 ### Restricting filesystems with --one-file-system 21150 21151 Normally rclone will recurse through filesystems as mounted. 21152 21153 However if you set `--one-file-system` or `-x` this tells rclone to 21154 stay in the filesystem specified by the root and not to recurse into 21155 different file systems. 21156 21157 For example if you have a directory hierarchy like this 21158 21159 ``` 21160 root 21161 ├── disk1 - disk1 mounted on the root 21162 │ └── file3 - stored on disk1 21163 ├── disk2 - disk2 mounted on the root 21164 │ └── file4 - stored on disk12 21165 ├── file1 - stored on the root disk 21166 └── file2 - stored on the root disk 21167 ``` 21168 21169 Using `rclone --one-file-system copy root remote:` will only copy `file1` and `file2`. Eg 21170 21171 ``` 21172 $ rclone -q --one-file-system ls root 21173 0 file1 21174 0 file2 21175 ``` 21176 21177 ``` 21178 $ rclone -q ls root 21179 0 disk1/file3 21180 0 disk2/file4 21181 0 file1 21182 0 file2 21183 ``` 21184 21185 **NB** Rclone (like most unix tools such as `du`, `rsync` and `tar`) 21186 treats a bind mount to the same device as being on the same 21187 filesystem. 21188 21189 **NB** This flag is only available on Unix based systems. On systems 21190 where it isn't supported (eg Windows) it will be ignored. 21191 21192 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/local/local.go then run make backenddocs --> 21193 ### Standard Options 21194 21195 Here are the standard options specific to local (Local Disk). 21196 21197 #### --local-nounc 21198 21199 Disable UNC (long path names) conversion on Windows 21200 21201 - Config: nounc 21202 - Env Var: RCLONE_LOCAL_NOUNC 21203 - Type: string 21204 - Default: "" 21205 - Examples: 21206 - "true" 21207 - Disables long file names 21208 21209 ### Advanced Options 21210 21211 Here are the advanced options specific to local (Local Disk). 21212 21213 #### --copy-links / -L 21214 21215 Follow symlinks and copy the pointed to item. 21216 21217 - Config: copy_links 21218 - Env Var: RCLONE_LOCAL_COPY_LINKS 21219 - Type: bool 21220 - Default: false 21221 21222 #### --links / -l 21223 21224 Translate symlinks to/from regular files with a '.rclonelink' extension 21225 21226 - Config: links 21227 - Env Var: RCLONE_LOCAL_LINKS 21228 - Type: bool 21229 - Default: false 21230 21231 #### --skip-links 21232 21233 Don't warn about skipped symlinks. 21234 This flag disables warning messages on skipped symlinks or junction 21235 points, as you explicitly acknowledge that they should be skipped. 21236 21237 - Config: skip_links 21238 - Env Var: RCLONE_LOCAL_SKIP_LINKS 21239 - Type: bool 21240 - Default: false 21241 21242 #### --local-no-unicode-normalization 21243 21244 Don't apply unicode normalization to paths and filenames (Deprecated) 21245 21246 This flag is deprecated now. Rclone no longer normalizes unicode file 21247 names, but it compares them with unicode normalization in the sync 21248 routine instead. 21249 21250 - Config: no_unicode_normalization 21251 - Env Var: RCLONE_LOCAL_NO_UNICODE_NORMALIZATION 21252 - Type: bool 21253 - Default: false 21254 21255 #### --local-no-check-updated 21256 21257 Don't check to see if the files change during upload 21258 21259 Normally rclone checks the size and modification time of files as they 21260 are being uploaded and aborts with a message which starts "can't copy 21261 - source file is being updated" if the file changes during upload. 21262 21263 However on some file systems this modification time check may fail (eg 21264 [Glusterfs #2206](https://github.com/rclone/rclone/issues/2206)) so this 21265 check can be disabled with this flag. 21266 21267 - Config: no_check_updated 21268 - Env Var: RCLONE_LOCAL_NO_CHECK_UPDATED 21269 - Type: bool 21270 - Default: false 21271 21272 #### --one-file-system / -x 21273 21274 Don't cross filesystem boundaries (unix/macOS only). 21275 21276 - Config: one_file_system 21277 - Env Var: RCLONE_LOCAL_ONE_FILE_SYSTEM 21278 - Type: bool 21279 - Default: false 21280 21281 #### --local-case-sensitive 21282 21283 Force the filesystem to report itself as case sensitive. 21284 21285 Normally the local backend declares itself as case insensitive on 21286 Windows/macOS and case sensitive for everything else. Use this flag 21287 to override the default choice. 21288 21289 - Config: case_sensitive 21290 - Env Var: RCLONE_LOCAL_CASE_SENSITIVE 21291 - Type: bool 21292 - Default: false 21293 21294 #### --local-case-insensitive 21295 21296 Force the filesystem to report itself as case insensitive 21297 21298 Normally the local backend declares itself as case insensitive on 21299 Windows/macOS and case sensitive for everything else. Use this flag 21300 to override the default choice. 21301 21302 - Config: case_insensitive 21303 - Env Var: RCLONE_LOCAL_CASE_INSENSITIVE 21304 - Type: bool 21305 - Default: false 21306 21307 #### --local-encoding 21308 21309 This sets the encoding for the backend. 21310 21311 See: the [encoding section in the overview](/overview/#encoding) for more info. 21312 21313 - Config: encoding 21314 - Env Var: RCLONE_LOCAL_ENCODING 21315 - Type: MultiEncoder 21316 - Default: Slash,Dot 21317 21318 <!--- autogenerated options stop --> 21319 21320 # Changelog 21321 21322 ## v1.51.0 - 2020-02-01 21323 21324 * New backends 21325 * [Memory](/memory) (Nick Craig-Wood) 21326 * [Sugarsync](/sugarsync) (Nick Craig-Wood) 21327 * New Features 21328 * Adjust all backends to have `--backend-encoding` parameter (Nick Craig-Wood) 21329 * this enables the encoding for special characters to be adjusted or disabled 21330 * Add `--max-duration` flag to control the maximum duration of a transfer session (boosh) 21331 * Add `--expect-continue-timeout` flag, default 1s (Nick Craig-Wood) 21332 * Add `--no-check-dest` flag for copying without testing the destination (Nick Craig-Wood) 21333 * Implement `--order-by` flag to order transfers (Nick Craig-Wood) 21334 * accounting 21335 * Don't show entries in both transferring and checking (Nick Craig-Wood) 21336 * Add option to delete stats (Aleksandar Jankovic) 21337 * build 21338 * Compress the test builds with gzip (Nick Craig-Wood) 21339 * Implement a framework for starting test servers during tests (Nick Craig-Wood) 21340 * cmd: Always print elapsed time to tenth place seconds in progress (Gary Kim) 21341 * config 21342 * Add `--password-command` to allow dynamic config password (Damon Permezel) 21343 * Give config questions default values (Nick Craig-Wood) 21344 * Check a remote exists when creating a new one (Nick Craig-Wood) 21345 * copyurl: Add `--stdout` flag to write to stdout (Nick Craig-Wood) 21346 * dedupe: Implement keep smallest too (Nick Craig-Wood) 21347 * hashsum: Add flag `--base64` flag (landall) 21348 * lsf: Speed up on s3/swift/etc by not reading mimetype by default (Nick Craig-Wood) 21349 * lsjson: Add `--no-mimetype` flag (Nick Craig-Wood) 21350 * rc: Add methods to turn on blocking and mutex profiling (Nick Craig-Wood) 21351 * rcd 21352 * Adding group parameter to stats (Chaitanya) 21353 * Move webgui apart; option to disable browser (Xiaoxing Ye) 21354 * serve sftp: Add support for public key with auth proxy (Paul Tinsley) 21355 * stats: Show deletes in stats and hide zero stats (anuar45) 21356 * Bug Fixes 21357 * accounting 21358 * Fix error counter counting multiple times (Ankur Gupta) 21359 * Fix error count shown as checks (Cnly) 21360 * Clear finished transfer in stats-reset (Maciej Zimnoch) 21361 * Added StatsInfo locking in statsGroups sum function (Michał Matczuk) 21362 * asyncreader: Fix EOF error (buengese) 21363 * check: Fix `--one-way` recursing more directories than it needs to (Nick Craig-Wood) 21364 * chunkedreader: Disable hash calculation for first segment (Nick Craig-Wood) 21365 * config 21366 * Do not open browser on headless on drive/gcs/google photos (Xiaoxing Ye) 21367 * SetValueAndSave ignore error if config section does not exist yet (buengese) 21368 * cmd: Fix completion with an encrypted config (Danil Semelenov) 21369 * dbhashsum: Stop it returning UNSUPPORTED on dropbox (Nick Craig-Wood) 21370 * dedupe: Add missing modes to help string (Nick Craig-Wood) 21371 * operations 21372 * Fix dedupe continuing on errors like insufficientFilePermisson (SezalAgrawal) 21373 * Clear accounting before low level retry (Maciej Zimnoch) 21374 * Write debug message when hashes could not be checked (Ole Schütt) 21375 * Move interface assertion to tests to remove pflag dependency (Nick Craig-Wood) 21376 * Make NewOverrideObjectInfo public and factor uses (Nick Craig-Wood) 21377 * proxy: Replace use of bcrypt with sha256 (Nick Craig-Wood) 21378 * vendor 21379 * Update bazil.org/fuse to fix FreeBSD 12.1 (Nick Craig-Wood) 21380 * Update github.com/t3rm1n4l/go-mega to fix mega "illegal base64 data at input byte 22" (Nick Craig-Wood) 21381 * Update termbox-go to fix ncdu command on FreeBSD (Kuang-che Wu) 21382 * Update t3rm1n4l/go-mega - fixes mega: couldn't login: crypto/aes: invalid key size 0 (Nick Craig-Wood) 21383 * Mount 21384 * Enable async reads for a 20% speedup (Nick Craig-Wood) 21385 * Replace use of WriteAt with Write for cache mode >= writes and O_APPEND (Brett Dutro) 21386 * Make sure we call unmount when exiting (Nick Craig-Wood) 21387 * Don't build on go1.10 as bazil/fuse no longer supports it (Nick Craig-Wood) 21388 * When setting dates discard out of range dates (Nick Craig-Wood) 21389 * VFS 21390 * Add a newly created file straight into the directory (Nick Craig-Wood) 21391 * Only calculate one hash for reads for a speedup (Nick Craig-Wood) 21392 * Make ReadAt for non cached files work better with non-sequential reads (Nick Craig-Wood) 21393 * Fix edge cases when reading ModTime from file (Nick Craig-Wood) 21394 * Make sure existing files opened for write show correct size (Nick Craig-Wood) 21395 * Don't cache the path in RW file objects to fix renaming (Nick Craig-Wood) 21396 * Fix rename of open files when using the VFS cache (Nick Craig-Wood) 21397 * When renaming files in the cache, rename the cache item in memory too (Nick Craig-Wood) 21398 * Fix open file renaming on drive when using `--vfs-cache-mode writes` (Nick Craig-Wood) 21399 * Fix incorrect modtime for mv into mount with `--vfs-cache-modes writes` (Nick Craig-Wood) 21400 * On rename, rename in cache too if the file exists (Anagh Kumar Baranwal) 21401 * Local 21402 * Make source file being updated errors be NoLowLevelRetry errors (Nick Craig-Wood) 21403 * Fix update of hidden files on Windows (Nick Craig-Wood) 21404 * Cache 21405 * Follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt (Nick Craig-Wood) 21406 * Fix `fatal error: concurrent map writes` (Nick Craig-Wood) 21407 * Crypt 21408 * Reorder the filename encryption options (Thomas Eales) 21409 * Correctly handle trailing dot (buengese) 21410 * Chunker 21411 * Reduce length of temporary suffix (Ivan Andreev) 21412 * Drive 21413 * Add `--drive-stop-on-upload-limit` flag to stop syncs when upload limit reached (Nick Craig-Wood) 21414 * Add `--drive-use-shared-date` to use date file was shared instead of modified date (Garry McNulty) 21415 * Make sure invalid auth for teamdrives always reports an error (Nick Craig-Wood) 21416 * Fix `--fast-list` when using appDataFolder (Nick Craig-Wood) 21417 * Use multipart resumable uploads for streaming and uploads in mount (Nick Craig-Wood) 21418 * Log an ERROR if an incomplete search is returned (Nick Craig-Wood) 21419 * Hide dangerous config from the configurator (Nick Craig-Wood) 21420 * Dropbox 21421 * Treat `insufficient_space` errors as non retriable errors (Nick Craig-Wood) 21422 * Jottacloud 21423 * Use new auth method used by official client (buengese) 21424 * Add URL to generate Login Token to config wizard (Nick Craig-Wood) 21425 * Add support whitelabel versions (buengese) 21426 * Koofr 21427 * Use rclone HTTP client. (jaKa) 21428 * Onedrive 21429 * Add Sites.Read.All permission (Benjamin Richter) 21430 * Add support "Retry-After" header (Motonori IWAMURO) 21431 * Opendrive 21432 * Implement `--opendrive-chunk-size` (Nick Craig-Wood) 21433 * S3 21434 * Re-implement multipart upload to fix memory issues (Nick Craig-Wood) 21435 * Add `--s3-copy-cutoff` for size to switch to multipart copy (Nick Craig-Wood) 21436 * Add new region Asia Patific (Hong Kong) (Outvi V) 21437 * Reduce memory usage streaming files by reducing max stream upload size (Nick Craig-Wood) 21438 * Add `--s3-list-chunk` option for bucket listing (Thomas Kriechbaumer) 21439 * Force path style bucket access to off for AWS deprecation (Nick Craig-Wood) 21440 * Use AWS web identity role provider if available (Tennix) 21441 * Fix ExpiryWindow value (Aleksandar Jankovic) 21442 * Fix DisableChecksum condition (Aleksandar Janković) 21443 * Fix URL decoding of NextMarker (Nick Craig-Wood) 21444 * SFTP 21445 * Add `--sftp-skip-links` to skip symlinks and non regular files (Nick Craig-Wood) 21446 * Retry Creation of Connection (Sebastian Brandt) 21447 * Fix "failed to parse private key file: ssh: not an encrypted key" error (Nick Craig-Wood) 21448 * Open files for update write only to fix AWS SFTP interop (Nick Craig-Wood) 21449 * Swift 21450 * Reserve segments of dynamic large object when delete objects in container what was enabled versioning. (Nguyễn Hữu Luân) 21451 * Fix parsing of X-Object-Manifest (Nick Craig-Wood) 21452 * Update OVH API endpoint (unbelauscht) 21453 * WebDAV 21454 * Make nextcloud only upload SHA1 checksums (Nick Craig-Wood) 21455 * Fix case of "Bearer" in Authorization: header to agree with RFC (Nick Craig-Wood) 21456 * Add Referer header to fix problems with WAFs (Nick Craig-Wood) 21457 21458 ## v1.50.2 - 2019-11-19 21459 21460 * Bug Fixes 21461 * accounting: Fix memory leak on retries operations (Nick Craig-Wood) 21462 * Drive 21463 * Fix listing of the root directory with drive.files scope (Nick Craig-Wood) 21464 * Fix --drive-root-folder-id with team/shared drives (Nick Craig-Wood) 21465 21466 ## v1.50.1 - 2019-11-02 21467 21468 * Bug Fixes 21469 * hash: Fix accidentally changed hash names for `DropboxHash` and `CRC-32` (Nick Craig-Wood) 21470 * fshttp: Fix error reporting on tpslimit token bucket errors (Nick Craig-Wood) 21471 * fshttp: Don't print token bucket errors on context cancelled (Nick Craig-Wood) 21472 * Local 21473 * Fix listings of . on Windows (Nick Craig-Wood) 21474 * Onedrive 21475 * Fix DirMove/Move after Onedrive change (Xiaoxing Ye) 21476 21477 ## v1.50.0 - 2019-10-26 21478 21479 * New backends 21480 * [Citrix Sharefile](/sharefile) (Nick Craig-Wood) 21481 * [Chunker](/chunker) - an overlay backend to split files into smaller parts (Ivan Andreev) 21482 * [Mail.ru Cloud](/mailru) (Ivan Andreev) 21483 * New Features 21484 * encodings (Fabian Möller & Nick Craig-Wood) 21485 * All backends now use file name encoding to ensure any file name can be written to any backend. 21486 * See the [restricted file name docs](/overview/#restricted-filenames) for more info and the [local backend docs](/local/#filenames). 21487 * Some file names may look different in rclone if you are using any control characters in names or [unicode FULLWIDTH symbols](https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)). 21488 * build 21489 * Update to use go1.13 for the build (Nick Craig-Wood) 21490 * Drop support for go1.9 (Nick Craig-Wood) 21491 * Build rclone with GitHub actions (Nick Craig-Wood) 21492 * Convert python scripts to python3 (Nick Craig-Wood) 21493 * Swap Azure/go-ansiterm for mattn/go-colorable (Nick Craig-Wood) 21494 * Dockerfile fixes (Matei David) 21495 * Add [plugin support](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-plugin) for backends and commands (Richard Patel) 21496 * config 21497 * Use alternating Red/Green in config to make more obvious (Nick Craig-Wood) 21498 * contrib 21499 * Add sample DLNA server Docker Compose manifest. (pataquets) 21500 * Add sample WebDAV server Docker Compose manifest. (pataquets) 21501 * copyurl 21502 * Add `--auto-filename` flag for using file name from URL in destination path (Denis) 21503 * serve dlna: 21504 * Many compatability improvements (Dan Walters) 21505 * Support for external srt subtitles (Dan Walters) 21506 * rc 21507 * Added command core/quit (Saksham Khanna) 21508 * Bug Fixes 21509 * sync 21510 * Make `--update`/`-u` not transfer files that haven't changed (Nick Craig-Wood) 21511 * Free objects after they come out of the transfer pipe to save memory (Nick Craig-Wood) 21512 * Fix `--files-from without --no-traverse` doing a recursive scan (Nick Craig-Wood) 21513 * operations 21514 * Fix accounting for server side copies (Nick Craig-Wood) 21515 * Display 'All duplicates removed' only if dedupe successful (Sezal Agrawal) 21516 * Display 'Deleted X extra copies' only if dedupe successful (Sezal Agrawal) 21517 * accounting 21518 * Only allow up to 100 completed transfers in the accounting list to save memory (Nick Craig-Wood) 21519 * Cull the old time ranges when possible to save memory (Nick Craig-Wood) 21520 * Fix panic due to server-side copy fallback (Ivan Andreev) 21521 * Fix memory leak noticeable for transfers of large numbers of objects (Nick Craig-Wood) 21522 * Fix total duration calculation (Nick Craig-Wood) 21523 * cmd 21524 * Fix environment variables not setting command line flags (Nick Craig-Wood) 21525 * Make autocomplete compatible with bash's posix mode for macOS (Danil Semelenov) 21526 * Make `--progress` work in git bash on Windows (Nick Craig-Wood) 21527 * Fix 'compopt: command not found' on autocomplete on macOS (Danil Semelenov) 21528 * config 21529 * Fix setting of non top level flags from environment variables (Nick Craig-Wood) 21530 * Check config names more carefully and report errors (Nick Craig-Wood) 21531 * Remove error: can't use `--size-only` and `--ignore-size` together. (Nick Craig-Wood) 21532 * filter: Prevent mixing options when `--files-from` is in use (Michele Caci) 21533 * serve sftp: Fix crash on unsupported operations (eg Readlink) (Nick Craig-Wood) 21534 * Mount 21535 * Allow files of unkown size to be read properly (Nick Craig-Wood) 21536 * Skip tests on <= 2 CPUs to avoid lockup (Nick Craig-Wood) 21537 * Fix panic on File.Open (Nick Craig-Wood) 21538 * Fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD (Nick Craig-Wood) 21539 * Don't pass huge filenames (>4k) to FUSE as it can't cope (Nick Craig-Wood) 21540 * VFS 21541 * Add flag `--vfs-case-insensitive` for windows/macOS mounts (Ivan Andreev) 21542 * Make objects of unknown size readable through the VFS (Nick Craig-Wood) 21543 * Move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush() (Brett Dutro) 21544 * Stop empty dirs disappearing when renamed on bucket based remotes (Nick Craig-Wood) 21545 * Stop change notify polling clearing so much of the directory cache (Nick Craig-Wood) 21546 * Azure Blob 21547 * Disable logging to the Windows event log (Nick Craig-Wood) 21548 * B2 21549 * Remove `unverified:` prefix on sha1 to improve interop (eg with CyberDuck) (Nick Craig-Wood) 21550 * Box 21551 * Add options to get access token via JWT auth (David) 21552 * Drive 21553 * Disable HTTP/2 by default to work around INTERNAL_ERROR problems (Nick Craig-Wood) 21554 * Make sure that drive root ID is always canonical (Nick Craig-Wood) 21555 * Fix `--drive-shared-with-me` from the root with lsand `--fast-list` (Nick Craig-Wood) 21556 * Fix ChangeNotify polling for shared drives (Nick Craig-Wood) 21557 * Fix change notify polling when using appDataFolder (Nick Craig-Wood) 21558 * Dropbox 21559 * Make disallowed filenames errors not retry (Nick Craig-Wood) 21560 * Fix nil pointer exception on restricted files (Nick Craig-Wood) 21561 * Fichier 21562 * Fix accessing files > 2GB on 32 bit systems (Nick Craig-Wood) 21563 * FTP 21564 * Allow disabling EPSV mode (Jon Fautley) 21565 * HTTP 21566 * HEAD directory entries in parallel to speedup (Nick Craig-Wood) 21567 * Add `--http-no-head` to stop rclone doing HEAD in listings (Nick Craig-Wood) 21568 * Putio 21569 * Add ability to resume uploads (Cenk Alti) 21570 * S3 21571 * Fix signature v2_auth headers (Anthony Rusdi) 21572 * Fix encoding for control characters (Nick Craig-Wood) 21573 * Only ask for URL encoded directory listings if we need them on Ceph (Nick Craig-Wood) 21574 * Add option for multipart failiure behaviour (Aleksandar Jankovic) 21575 * Support for multipart copy (庄天翼) 21576 * Fix nil pointer reference if no metadata returned for object (Nick Craig-Wood) 21577 * SFTP 21578 * Fix `--sftp-ask-password` trying to contact the ssh agent (Nick Craig-Wood) 21579 * Fix hashes of files with backslashes (Nick Craig-Wood) 21580 * Include more ciphers with `--sftp-use-insecure-cipher` (Carlos Ferreyra) 21581 * WebDAV 21582 * Parse and return Sharepoint error response (Henning Surmeier) 21583 21584 ## v1.49.5 - 2019-10-05 21585 21586 * Bug Fixes 21587 * Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing issues (Nick Craig-Wood) 21588 * Fix rpm packages by using master builds of nfpm (Nick Craig-Wood) 21589 * Fix macOS build after brew changes (Nick Craig-Wood) 21590 21591 ## v1.49.4 - 2019-09-29 21592 21593 * Bug Fixes 21594 * cmd/rcd: Address ZipSlip vulnerability (Richard Patel) 21595 * accounting: Fix file handle leak on errors (Nick Craig-Wood) 21596 * oauthutil: Fix security problem when running with two users on the same machine (Nick Craig-Wood) 21597 * FTP 21598 * Fix listing of an empty root returning: error dir not found (Nick Craig-Wood) 21599 * S3 21600 * Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier (Nick Craig-Wood) 21601 21602 ## v1.49.3 - 2019-09-15 21603 21604 * Bug Fixes 21605 * accounting 21606 * Fix total duration calculation (Aleksandar Jankovic) 21607 * Fix "file already closed" on transfer retries (Nick Craig-Wood) 21608 21609 ## v1.49.2 - 2019-09-08 21610 21611 * New Features 21612 * build: Add Docker workflow support (Alfonso Montero) 21613 * Bug Fixes 21614 * accounting: Fix locking in Transfer to avoid deadlock with `--progress` (Nick Craig-Wood) 21615 * docs: Fix template argument for mktemp in install.sh (Cnly) 21616 * operations: Fix `-u`/`--update` with google photos / files of unknown size (Nick Craig-Wood) 21617 * rc: Fix docs for config/create /update /password (Nick Craig-Wood) 21618 * Google Cloud Storage 21619 * Fix need for elevated permissions on SetModTime (Nick Craig-Wood) 21620 21621 ## v1.49.1 - 2019-08-28 21622 21623 * Bug Fixes 21624 * config: Fix generated passwords being stored as empty password (Nick Craig-Wood) 21625 * rcd: Added missing parameter for web-gui info logs. (Chaitanya) 21626 * Googlephotos 21627 * Fix crash on error response (Nick Craig-Wood) 21628 * Onedrive 21629 * Fix crash on error response (Nick Craig-Wood) 21630 21631 ## v1.49.0 - 2019-08-26 21632 21633 * New backends 21634 * [1fichier](https://rclone.org/fichier/) (Laura Hausmann) 21635 * [Google Photos](/googlephotos) (Nick Craig-Wood) 21636 * [Putio](https://rclone.org/putio/) (Cenk Alti) 21637 * [premiumize.me](https://rclone.org/premiumizeme/) (Nick Craig-Wood) 21638 * New Features 21639 * Experimental [web GUI](https://rclone.org/gui/) (Chaitanya Bankanhal) 21640 * Implement `--compare-dest` & `--copy-dest` (yparitcher) 21641 * Implement `--suffix` without `--backup-dir` for backup to current dir (yparitcher) 21642 * `config reconnect` to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood) 21643 * `config userinfo` to discover which user you are logged in as. (Nick Craig-Wood) 21644 * `config disconnect` to disconnect you (log out) from the backend. (Nick Craig-Wood) 21645 * Add `--use-json-log` for JSON logging (justinalin) 21646 * Add context propagation to rclone (Aleksandar Jankovic) 21647 * Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic) 21648 * Add Higher units for ETA (AbelThar) 21649 * Update rclone logos to new design (Andreas Chlupka) 21650 * hash: Add CRC-32 support (Cenk Alti) 21651 * help showbackend: Fixed advanced option category when there are no standard options (buengese) 21652 * ncdu: Display/Copy to Clipboard Current Path (Gary Kim) 21653 * operations: 21654 * Run hashing operations in parallel (Nick Craig-Wood) 21655 * Don't calculate checksums when using `--ignore-checksum` (Nick Craig-Wood) 21656 * Check transfer hashes when using `--size-only` mode (Nick Craig-Wood) 21657 * Disable multi thread copy for local to local copies (Nick Craig-Wood) 21658 * Debug successful hashes as well as failures (Nick Craig-Wood) 21659 * rc 21660 * Add ability to stop async jobs (Aleksandar Jankovic) 21661 * Return current settings if core/bwlimit called without parameters (Nick Craig-Wood) 21662 * Rclone-WebUI integration with rclone (Chaitanya Bankanhal) 21663 * Added command line parameter to control the cross origin resource sharing (CORS) in the rcd. (Security Improvement) (Chaitanya Bankanhal) 21664 * Add anchor tags to the docs so links are consistent (Nick Craig-Wood) 21665 * Remove _async key from input parameters after parsing so later operations won't get confused (buengese) 21666 * Add call to clear stats (Aleksandar Jankovic) 21667 * rcd 21668 * Auto-login for web-gui (Chaitanya Bankanhal) 21669 * Implement `--baseurl` for rcd and web-gui (Chaitanya Bankanhal) 21670 * serve dlna 21671 * Only select interfaces which can multicast for SSDP (Nick Craig-Wood) 21672 * Add more builtin mime types to cover standard audio/video (Nick Craig-Wood) 21673 * Fix missing mime types on Android causing missing videos (Nick Craig-Wood) 21674 * serve ftp 21675 * Refactor to bring into line with other serve commands (Nick Craig-Wood) 21676 * Implement `--auth-proxy` (Nick Craig-Wood) 21677 * serve http: Implement `--baseurl` (Nick Craig-Wood) 21678 * serve restic: Implement `--baseurl` (Nick Craig-Wood) 21679 * serve sftp 21680 * Implement auth proxy (Nick Craig-Wood) 21681 * Fix detection of whether server is authorized (Nick Craig-Wood) 21682 * serve webdav 21683 * Implement `--baseurl` (Nick Craig-Wood) 21684 * Support `--auth-proxy` (Nick Craig-Wood) 21685 * Bug Fixes 21686 * Make "bad record MAC" a retriable error (Nick Craig-Wood) 21687 * copyurl: Fix copying files that return HTTP errors (Nick Craig-Wood) 21688 * march: Fix checking sub-directories when using `--no-traverse` (buengese) 21689 * rc 21690 * Fix unmarshalable http.AuthFn in options and put in test for marshalability (Nick Craig-Wood) 21691 * Move job expire flags to rc to fix initalization problem (Nick Craig-Wood) 21692 * Fix `--loopback` with rc/list and others (Nick Craig-Wood) 21693 * rcat: Fix slowdown on systems with multiple hashes (Nick Craig-Wood) 21694 * rcd: Fix permissions problems on cache directory with web gui download (Nick Craig-Wood) 21695 * Mount 21696 * Default `--deamon-timout` to 15 minutes on macOS and FreeBSD (Nick Craig-Wood) 21697 * Update docs to show mounting from root OK for bucket based (Nick Craig-Wood) 21698 * Remove nonseekable flag from write files (Nick Craig-Wood) 21699 * VFS 21700 * Make write without cache more efficient (Nick Craig-Wood) 21701 * Fix `--vfs-cache-mode minimal` and `writes` ignoring cached files (Nick Craig-Wood) 21702 * Local 21703 * Add `--local-case-sensitive` and `--local-case-insensitive` (Nick Craig-Wood) 21704 * Avoid polluting page cache when uploading local files to remote backends (Michał Matczuk) 21705 * Don't calculate any hashes by default (Nick Craig-Wood) 21706 * Fadvise run syscall on a dedicated go routine (Michał Matczuk) 21707 * Azure Blob 21708 * Azure Storage Emulator support (Sandeep) 21709 * Updated config help details to remove connection string references (Sandeep) 21710 * Make all operations work from the root (Nick Craig-Wood) 21711 * B2 21712 * Implement link sharing (yparitcher) 21713 * Enable server side copy to copy between buckets (Nick Craig-Wood) 21714 * Make all operations work from the root (Nick Craig-Wood) 21715 * Drive 21716 * Fix server side copy of big files (Nick Craig-Wood) 21717 * Update API for teamdrive use (Nick Craig-Wood) 21718 * Add error for purge with `--drive-trashed-only` (ginvine) 21719 * Fichier 21720 * Make FolderID int and adjust related code (buengese) 21721 * Google Cloud Storage 21722 * Reduce oauth scope requested as suggested by Google (Nick Craig-Wood) 21723 * Make all operations work from the root (Nick Craig-Wood) 21724 * HTTP 21725 * Add `--http-headers` flag for setting arbitrary headers (Nick Craig-Wood) 21726 * Jottacloud 21727 * Use new api for retrieving internal username (buengese) 21728 * Refactor configuration and minor cleanup (buengese) 21729 * Koofr 21730 * Support setting modification times on Koofr backend. (jaKa) 21731 * Opendrive 21732 * Refactor to use existing lib/rest facilities for uploads (Nick Craig-Wood) 21733 * Qingstor 21734 * Upgrade to v3 SDK and fix listing loop (Nick Craig-Wood) 21735 * Make all operations work from the root (Nick Craig-Wood) 21736 * S3 21737 * Add INTELLIGENT_TIERING storage class (Matti Niemenmaa) 21738 * Make all operations work from the root (Nick Craig-Wood) 21739 * SFTP 21740 * Add missing interface check and fix About (Nick Craig-Wood) 21741 * Completely ignore all modtime checks if SetModTime=false (Jon Fautley) 21742 * Support md5/sha1 with rsync.net (Nick Craig-Wood) 21743 * Save the md5/sha1 command in use to the config file for efficiency (Nick Craig-Wood) 21744 * Opt-in support for diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 (Yi FU) 21745 * Swift 21746 * Use FixRangeOption to fix 0 length files via the VFS (Nick Craig-Wood) 21747 * Fix upload when using no_chunk to return the correct size (Nick Craig-Wood) 21748 * Make all operations work from the root (Nick Craig-Wood) 21749 * Fix segments leak during failed large file uploads. (nguyenhuuluan434) 21750 * WebDAV 21751 * Add `--webdav-bearer-token-command` (Nick Craig-Wood) 21752 * Refresh token when it expires with `--webdav-bearer-token-command` (Nick Craig-Wood) 21753 * Add docs for using bearer_token_command with oidc-agent (Paul Millar) 21754 21755 ## v1.48.0 - 2019-06-15 21756 21757 * New commands 21758 * serve sftp: Serve an rclone remote over SFTP (Nick Craig-Wood) 21759 * New Features 21760 * Multi threaded downloads to local storage (Nick Craig-Wood) 21761 * controlled with `--multi-thread-cutoff` and `--multi-thread-streams` 21762 * Use rclone.conf from rclone executable directory to enable portable use (albertony) 21763 * Allow sync of a file and a directory with the same name (forgems) 21764 * this is common on bucket based remotes, eg s3, gcs 21765 * Add `--ignore-case-sync` for forced case insensitivity (garry415) 21766 * Implement `--stats-one-line-date` and `--stats-one-line-date-format` (Peter Berbec) 21767 * Log an ERROR for all commands which exit with non-zero status (Nick Craig-Wood) 21768 * Use go-homedir to read the home directory more reliably (Nick Craig-Wood) 21769 * Enable creating encrypted config through external script invocation (Wojciech Smigielski) 21770 * build: Drop support for go1.8 (Nick Craig-Wood) 21771 * config: Make config create/update encrypt passwords where necessary (Nick Craig-Wood) 21772 * copyurl: Honor `--no-check-certificate` (Stefan Breunig) 21773 * install: Linux skip man pages if no mandb (didil) 21774 * lsf: Support showing the Tier of the object (Nick Craig-Wood) 21775 * lsjson 21776 * Added EncryptedPath to output (calisro) 21777 * Support showing the Tier of the object (Nick Craig-Wood) 21778 * Add IsBucket field for bucket based remote listing of the root (Nick Craig-Wood) 21779 * rc 21780 * Add `--loopback` flag to run commands directly without a server (Nick Craig-Wood) 21781 * Add operations/fsinfo: Return information about the remote (Nick Craig-Wood) 21782 * Skip auth for OPTIONS request (Nick Craig-Wood) 21783 * cmd/providers: Add DefaultStr, ValueStr and Type fields (Nick Craig-Wood) 21784 * jobs: Make job expiry timeouts configurable (Aleksandar Jankovic) 21785 * serve dlna reworked and improved (Dan Walters) 21786 * serve ftp: add `--ftp-public-ip` flag to specify public IP (calistri) 21787 * serve restic: Add support for `--private-repos` in `serve restic` (Florian Apolloner) 21788 * serve webdav: Combine serve webdav and serve http (Gary Kim) 21789 * size: Ignore negative sizes when calculating total (Garry McNulty) 21790 * Bug Fixes 21791 * Make move and copy individual files obey `--backup-dir` (Nick Craig-Wood) 21792 * If `--ignore-checksum` is in effect, don't calculate checksum (Nick Craig-Wood) 21793 * moveto: Fix case-insensitive same remote move (Gary Kim) 21794 * rc: Fix serving bucket based objects with `--rc-serve` (Nick Craig-Wood) 21795 * serve webdav: Fix serveDir not being updated with changes from webdav (Gary Kim) 21796 * Mount 21797 * Fix poll interval documentation (Animosity022) 21798 * VFS 21799 * Make WriteAt for non cached files work with non-sequential writes (Nick Craig-Wood) 21800 * Local 21801 * Only calculate the required hashes for big speedup (Nick Craig-Wood) 21802 * Log errors when listing instead of returning an error (Nick Craig-Wood) 21803 * Fix preallocate warning on Linux with ZFS (Nick Craig-Wood) 21804 * Crypt 21805 * Make rclone dedupe work through crypt (Nick Craig-Wood) 21806 * Fix wrapping of ChangeNotify to decrypt directories properly (Nick Craig-Wood) 21807 * Support PublicLink (rclone link) of underlying backend (Nick Craig-Wood) 21808 * Implement Optional methods SetTier, GetTier (Nick Craig-Wood) 21809 * B2 21810 * Implement server side copy (Nick Craig-Wood) 21811 * Implement SetModTime (Nick Craig-Wood) 21812 * Drive 21813 * Fix move and copy from TeamDrive to GDrive (Fionera) 21814 * Add notes that cleanup works in the background on drive (Nick Craig-Wood) 21815 * Add `--drive-server-side-across-configs` to default back to old server side copy semantics by default (Nick Craig-Wood) 21816 * Add `--drive-size-as-quota` to show storage quota usage for file size (Garry McNulty) 21817 * FTP 21818 * Add FTP List timeout (Jeff Quinn) 21819 * Add FTP over TLS support (Gary Kim) 21820 * Add `--ftp-no-check-certificate` option for FTPS (Gary Kim) 21821 * Google Cloud Storage 21822 * Fix upload errors when uploading pre 1970 files (Nick Craig-Wood) 21823 * Jottacloud 21824 * Add support for selecting device and mountpoint. (buengese) 21825 * Mega 21826 * Add cleanup support (Gary Kim) 21827 * Onedrive 21828 * More accurately check if root is found (Cnly) 21829 * S3 21830 * Suppport S3 Accelerated endpoints with `--s3-use-accelerate-endpoint` (Nick Craig-Wood) 21831 * Add config info for Wasabi's EU Central endpoint (Robert Marko) 21832 * Make SetModTime work for GLACIER while syncing (Philip Harvey) 21833 * SFTP 21834 * Add About support (Gary Kim) 21835 * Fix about parsing of `df` results so it can cope with -ve results (Nick Craig-Wood) 21836 * Send custom client version and debug server version (Nick Craig-Wood) 21837 * WebDAV 21838 * Retry on 423 Locked errors (Nick Craig-Wood) 21839 21840 ## v1.47.0 - 2019-04-13 21841 21842 * New backends 21843 * Backend for Koofr cloud storage service. (jaKa) 21844 * New Features 21845 * Resume downloads if the reader fails in copy (Nick Craig-Wood) 21846 * this means rclone will restart transfers if the source has an error 21847 * this is most useful for downloads or cloud to cloud copies 21848 * Use `--fast-list` for listing operations where it won't use more memory (Nick Craig-Wood) 21849 * this should speed up the following operations on remotes which support `ListR` 21850 * `dedupe`, `serve restic` `lsf`, `ls`, `lsl`, `lsjson`, `lsd`, `md5sum`, `sha1sum`, `hashsum`, `size`, `delete`, `cat`, `settier` 21851 * use `--disable ListR` to get old behaviour if required 21852 * Make `--files-from` traverse the destination unless `--no-traverse` is set (Nick Craig-Wood) 21853 * this fixes `--files-from` with Google drive and excessive API use in general. 21854 * Make server side copy account bytes and obey `--max-transfer` (Nick Craig-Wood) 21855 * Add `--create-empty-src-dirs` flag and default to not creating empty dirs (ishuah) 21856 * Add client side TLS/SSL flags `--ca-cert`/`--client-cert`/`--client-key` (Nick Craig-Wood) 21857 * Implement `--suffix-keep-extension` for use with `--suffix` (Nick Craig-Wood) 21858 * build: 21859 * Switch to semvar compliant version tags to be go modules compliant (Nick Craig-Wood) 21860 * Update to use go1.12.x for the build (Nick Craig-Wood) 21861 * serve dlna: Add connection manager service description to improve compatibility (Dan Walters) 21862 * lsf: Add 'e' format to show encrypted names and 'o' for original IDs (Nick Craig-Wood) 21863 * lsjson: Added `--files-only` and `--dirs-only` flags (calistri) 21864 * rc: Implement operations/publiclink the equivalent of `rclone link` (Nick Craig-Wood) 21865 * Bug Fixes 21866 * accounting: Fix total ETA when `--stats-unit bits` is in effect (Nick Craig-Wood) 21867 * Bash TAB completion 21868 * Use private custom func to fix clash between rclone and kubectl (Nick Craig-Wood) 21869 * Fix for remotes with underscores in their names (Six) 21870 * Fix completion of remotes (Florian Gamböck) 21871 * Fix autocompletion of remote paths with spaces (Danil Semelenov) 21872 * serve dlna: Fix root XML service descriptor (Dan Walters) 21873 * ncdu: Fix display corruption with Chinese characters (Nick Craig-Wood) 21874 * Add SIGTERM to signals which run the exit handlers on unix (Nick Craig-Wood) 21875 * rc: Reload filter when the options are set via the rc (Nick Craig-Wood) 21876 * VFS / Mount 21877 * Fix FreeBSD: Ignore Truncate if called with no readers and already the correct size (Nick Craig-Wood) 21878 * Read directory and check for a file before mkdir (Nick Craig-Wood) 21879 * Shorten the locking window for vfs/refresh (Nick Craig-Wood) 21880 * Azure Blob 21881 * Enable MD5 checksums when uploading files bigger than the "Cutoff" (Dr.Rx) 21882 * Fix SAS URL support (Nick Craig-Wood) 21883 * B2 21884 * Allow manual configuration of backblaze downloadUrl (Vince) 21885 * Ignore already_hidden error on remove (Nick Craig-Wood) 21886 * Ignore malformed `src_last_modified_millis` (Nick Craig-Wood) 21887 * Drive 21888 * Add `--skip-checksum-gphotos` to ignore incorrect checksums on Google Photos (Nick Craig-Wood) 21889 * Allow server side move/copy between different remotes. (Fionera) 21890 * Add docs on team drives and `--fast-list` eventual consistency (Nestar47) 21891 * Fix imports of text files (Nick Craig-Wood) 21892 * Fix range requests on 0 length files (Nick Craig-Wood) 21893 * Fix creation of duplicates with server side copy (Nick Craig-Wood) 21894 * Dropbox 21895 * Retry blank errors to fix long listings (Nick Craig-Wood) 21896 * FTP 21897 * Add `--ftp-concurrency` to limit maximum number of connections (Nick Craig-Wood) 21898 * Google Cloud Storage 21899 * Fall back to default application credentials (marcintustin) 21900 * Allow bucket policy only buckets (Nick Craig-Wood) 21901 * HTTP 21902 * Add `--http-no-slash` for websites with directories with no slashes (Nick Craig-Wood) 21903 * Remove duplicates from listings (Nick Craig-Wood) 21904 * Fix socket leak on 404 errors (Nick Craig-Wood) 21905 * Jottacloud 21906 * Fix token refresh (Sebastian Bünger) 21907 * Add device registration (Oliver Heyme) 21908 * Onedrive 21909 * Implement graceful cancel of multipart uploads if rclone is interrupted (Cnly) 21910 * Always add trailing colon to path when addressing items, (Cnly) 21911 * Return errors instead of panic for invalid uploads (Fabian Möller) 21912 * S3 21913 * Add support for "Glacier Deep Archive" storage class (Manu) 21914 * Update Dreamhost endpoint (Nick Craig-Wood) 21915 * Note incompatibility with CEPH Jewel (Nick Craig-Wood) 21916 * SFTP 21917 * Allow custom ssh client config (Alexandru Bumbacea) 21918 * Swift 21919 * Obey Retry-After to enable OVH restore from cold storage (Nick Craig-Wood) 21920 * Work around token expiry on CEPH (Nick Craig-Wood) 21921 * WebDAV 21922 * Allow IsCollection property to be integer or boolean (Nick Craig-Wood) 21923 * Fix race when creating directories (Nick Craig-Wood) 21924 * Fix About/df when reading the available/total returns 0 (Nick Craig-Wood) 21925 21926 ## v1.46 - 2019-02-09 21927 21928 * New backends 21929 * Support Alibaba Cloud (Aliyun) OSS via the s3 backend (Nick Craig-Wood) 21930 * New commands 21931 * serve dlna: serves a remove via DLNA for the local network (nicolov) 21932 * New Features 21933 * copy, move: Restore deprecated `--no-traverse` flag (Nick Craig-Wood) 21934 * This is useful for when transferring a small number of files into a large destination 21935 * genautocomplete: Add remote path completion for bash completion (Christopher Peterson & Danil Semelenov) 21936 * Buffer memory handling reworked to return memory to the OS better (Nick Craig-Wood) 21937 * Buffer recycling library to replace sync.Pool 21938 * Optionally use memory mapped memory for better memory shrinking 21939 * Enable with `--use-mmap` if having memory problems - not default yet 21940 * Parallelise reading of files specified by `--files-from` (Nick Craig-Wood) 21941 * check: Add stats showing total files matched. (Dario Guzik) 21942 * Allow rename/delete open files under Windows (Nick Craig-Wood) 21943 * lsjson: Use exactly the correct number of decimal places in the seconds (Nick Craig-Wood) 21944 * Add cookie support with cmdline switch `--use-cookies` for all HTTP based remotes (qip) 21945 * Warn if `--checksum` is set but there are no hashes available (Nick Craig-Wood) 21946 * Rework rate limiting (pacer) to be more accurate and allow bursting (Nick Craig-Wood) 21947 * Improve error reporting for too many/few arguments in commands (Nick Craig-Wood) 21948 * listremotes: Remove `-l` short flag as it conflicts with the new global flag (weetmuts) 21949 * Make http serving with auth generate INFO messages on auth fail (Nick Craig-Wood) 21950 * Bug Fixes 21951 * Fix layout of stats (Nick Craig-Wood) 21952 * Fix `--progress` crash under Windows Jenkins (Nick Craig-Wood) 21953 * Fix transfer of google/onedrive docs by calling Rcat in Copy when size is -1 (Cnly) 21954 * copyurl: Fix checking of `--dry-run` (Denis Skovpen) 21955 * Mount 21956 * Check that mountpoint and local directory to mount don't overlap (Nick Craig-Wood) 21957 * Fix mount size under 32 bit Windows (Nick Craig-Wood) 21958 * VFS 21959 * Implement renaming of directories for backends without DirMove (Nick Craig-Wood) 21960 * now all backends except b2 support renaming directories 21961 * Implement `--vfs-cache-max-size` to limit the total size of the cache (Nick Craig-Wood) 21962 * Add `--dir-perms` and `--file-perms` flags to set default permissions (Nick Craig-Wood) 21963 * Fix deadlock on concurrent operations on a directory (Nick Craig-Wood) 21964 * Fix deadlock between RWFileHandle.close and File.Remove (Nick Craig-Wood) 21965 * Fix renaming/deleting open files with cache mode "writes" under Windows (Nick Craig-Wood) 21966 * Fix panic on rename with `--dry-run` set (Nick Craig-Wood) 21967 * Fix vfs/refresh with recurse=true needing the `--fast-list` flag 21968 * Local 21969 * Add support for `-l`/`--links` (symbolic link translation) (yair@unicorn) 21970 * this works by showing links as `link.rclonelink` - see local backend docs for more info 21971 * this errors if used with `-L`/`--copy-links` 21972 * Fix renaming/deleting open files on Windows (Nick Craig-Wood) 21973 * Crypt 21974 * Check for maximum length before decrypting filename to fix panic (Garry McNulty) 21975 * Azure Blob 21976 * Allow building azureblob backend on *BSD (themylogin) 21977 * Use the rclone HTTP client to support `--dump headers`, `--tpslimit` etc (Nick Craig-Wood) 21978 * Use the s3 pacer for 0 delay in non error conditions (Nick Craig-Wood) 21979 * Ignore directory markers (Nick Craig-Wood) 21980 * Stop Mkdir attempting to create existing containers (Nick Craig-Wood) 21981 * B2 21982 * cleanup: will remove unfinished large files >24hrs old (Garry McNulty) 21983 * For a bucket limited application key check the bucket name (Nick Craig-Wood) 21984 * before this, rclone would use the authorised bucket regardless of what you put on the command line 21985 * Added `--b2-disable-checksum` flag (Wojciech Smigielski) 21986 * this enables large files to be uploaded without a SHA-1 hash for speed reasons 21987 * Drive 21988 * Set default pacer to 100ms for 10 tps (Nick Craig-Wood) 21989 * This fits the Google defaults much better and reduces the 403 errors massively 21990 * Add `--drive-pacer-min-sleep` and `--drive-pacer-burst` to control the pacer 21991 * Improve ChangeNotify support for items with multiple parents (Fabian Möller) 21992 * Fix ListR for items with multiple parents - this fixes oddities with `vfs/refresh` (Fabian Möller) 21993 * Fix using `--drive-impersonate` and appfolders (Nick Craig-Wood) 21994 * Fix google docs in rclone mount for some (not all) applications (Nick Craig-Wood) 21995 * Dropbox 21996 * Retry-After support for Dropbox backend (Mathieu Carbou) 21997 * FTP 21998 * Wait for 60 seconds for a connection to Close then declare it dead (Nick Craig-Wood) 21999 * helps with indefinite hangs on some FTP servers 22000 * Google Cloud Storage 22001 * Update google cloud storage endpoints (weetmuts) 22002 * HTTP 22003 * Add an example with username and password which is supported but wasn't documented (Nick Craig-Wood) 22004 * Fix backend with `--files-from` and non-existent files (Nick Craig-Wood) 22005 * Hubic 22006 * Make error message more informative if authentication fails (Nick Craig-Wood) 22007 * Jottacloud 22008 * Resume and deduplication support (Oliver Heyme) 22009 * Use token auth for all API requests Don't store password anymore (Sebastian Bünger) 22010 * Add support for 2-factor authentification (Sebastian Bünger) 22011 * Mega 22012 * Implement v2 account login which fixes logins for newer Mega accounts (Nick Craig-Wood) 22013 * Return error if an unknown length file is attempted to be uploaded (Nick Craig-Wood) 22014 * Add new error codes for better error reporting (Nick Craig-Wood) 22015 * Onedrive 22016 * Fix broken support for "shared with me" folders (Alex Chen) 22017 * Fix root ID not normalised (Cnly) 22018 * Return err instead of panic on unknown-sized uploads (Cnly) 22019 * Qingstor 22020 * Fix go routine leak on multipart upload errors (Nick Craig-Wood) 22021 * Add upload chunk size/concurrency/cutoff control (Nick Craig-Wood) 22022 * Default `--qingstor-upload-concurrency` to 1 to work around bug (Nick Craig-Wood) 22023 * S3 22024 * Implement `--s3-upload-cutoff` for single part uploads below this (Nick Craig-Wood) 22025 * Change `--s3-upload-concurrency` default to 4 to increase perfomance (Nick Craig-Wood) 22026 * Add `--s3-bucket-acl` to control bucket ACL (Nick Craig-Wood) 22027 * Auto detect region for buckets on operation failure (Nick Craig-Wood) 22028 * Add GLACIER storage class (William Cocker) 22029 * Add Scaleway to s3 documentation (Rémy Léone) 22030 * Add AWS endpoint eu-north-1 (weetmuts) 22031 * SFTP 22032 * Add support for PEM encrypted private keys (Fabian Möller) 22033 * Add option to force the usage of an ssh-agent (Fabian Möller) 22034 * Perform environment variable expansion on key-file (Fabian Möller) 22035 * Fix rmdir on Windows based servers (eg CrushFTP) (Nick Craig-Wood) 22036 * Fix rmdir deleting directory contents on some SFTP servers (Nick Craig-Wood) 22037 * Fix error on dangling symlinks (Nick Craig-Wood) 22038 * Swift 22039 * Add `--swift-no-chunk` to disable segmented uploads in rcat/mount (Nick Craig-Wood) 22040 * Introduce application credential auth support (kayrus) 22041 * Fix memory usage by slimming Object (Nick Craig-Wood) 22042 * Fix extra requests on upload (Nick Craig-Wood) 22043 * Fix reauth on big files (Nick Craig-Wood) 22044 * Union 22045 * Fix poll-interval not working (Nick Craig-Wood) 22046 * WebDAV 22047 * Support About which means rclone mount will show the correct disk size (Nick Craig-Wood) 22048 * Support MD5 and SHA1 hashes with Owncloud and Nextcloud (Nick Craig-Wood) 22049 * Fail soft on time parsing errors (Nick Craig-Wood) 22050 * Fix infinite loop on failed directory creation (Nick Craig-Wood) 22051 * Fix identification of directories for Bitrix Site Manager (Nick Craig-Wood) 22052 * Fix upload of 0 length files on some servers (Nick Craig-Wood) 22053 * Fix if MKCOL fails with 423 Locked assume the directory exists (Nick Craig-Wood) 22054 22055 ## v1.45 - 2018-11-24 22056 22057 * New backends 22058 * The Yandex backend was re-written - see below for details (Sebastian Bünger) 22059 * New commands 22060 * rcd: New command just to serve the remote control API (Nick Craig-Wood) 22061 * New Features 22062 * The remote control API (rc) was greatly expanded to allow full control over rclone (Nick Craig-Wood) 22063 * sensitive operations require authorization or the `--rc-no-auth` flag 22064 * config/* operations to configure rclone 22065 * options/* for reading/setting command line flags 22066 * operations/* for all low level operations, eg copy file, list directory 22067 * sync/* for sync, copy and move 22068 * `--rc-files` flag to serve files on the rc http server 22069 * this is for building web native GUIs for rclone 22070 * Optionally serving objects on the rc http server 22071 * Ensure rclone fails to start up if the `--rc` port is in use already 22072 * See [the rc docs](https://rclone.org/rc/) for more info 22073 * sync/copy/move 22074 * Make `--files-from` only read the objects specified and don't scan directories (Nick Craig-Wood) 22075 * This is a huge speed improvement for destinations with lots of files 22076 * filter: Add `--ignore-case` flag (Nick Craig-Wood) 22077 * ncdu: Add remove function ('d' key) (Henning Surmeier) 22078 * rc command 22079 * Add `--json` flag for structured JSON input (Nick Craig-Wood) 22080 * Add `--user` and `--pass` flags and interpret `--rc-user`, `--rc-pass`, `--rc-addr` (Nick Craig-Wood) 22081 * build 22082 * Require go1.8 or later for compilation (Nick Craig-Wood) 22083 * Enable softfloat on MIPS arch (Scott Edlund) 22084 * Integration test framework revamped with a better report and better retries (Nick Craig-Wood) 22085 * Bug Fixes 22086 * cmd: Make `--progress` update the stats correctly at the end (Nick Craig-Wood) 22087 * config: Create config directory on save if it is missing (Nick Craig-Wood) 22088 * dedupe: Check for existing filename before renaming a dupe file (ssaqua) 22089 * move: Don't create directories with `--dry-run` (Nick Craig-Wood) 22090 * operations: Fix Purge and Rmdirs when dir is not the root (Nick Craig-Wood) 22091 * serve http/webdav/restic: Ensure rclone exits if the port is in use (Nick Craig-Wood) 22092 * Mount 22093 * Make `--volname` work for Windows and macOS (Nick Craig-Wood) 22094 * Azure Blob 22095 * Avoid context deadline exceeded error by setting a large TryTimeout value (brused27) 22096 * Fix erroneous Rmdir error "directory not empty" (Nick Craig-Wood) 22097 * Wait for up to 60s to create a just deleted container (Nick Craig-Wood) 22098 * Dropbox 22099 * Add dropbox impersonate support (Jake Coggiano) 22100 * Jottacloud 22101 * Fix bug in `--fast-list` handing of empty folders (albertony) 22102 * Opendrive 22103 * Fix transfer of files with `+` and `&` in (Nick Craig-Wood) 22104 * Fix retries of upload chunks (Nick Craig-Wood) 22105 * S3 22106 * Set ACL for server side copies to that provided by the user (Nick Craig-Wood) 22107 * Fix role_arn, credential_source, ... (Erik Swanson) 22108 * Add config info for Wasabi's US-West endpoint (Henry Ptasinski) 22109 * SFTP 22110 * Ensure file hash checking is really disabled (Jon Fautley) 22111 * Swift 22112 * Add pacer for retries to make swift more reliable (Nick Craig-Wood) 22113 * WebDAV 22114 * Add Content-Type to PUT requests (Nick Craig-Wood) 22115 * Fix config parsing so `--webdav-user` and `--webdav-pass` flags work (Nick Craig-Wood) 22116 * Add RFC3339 date format (Ralf Hemberger) 22117 * Yandex 22118 * The yandex backend was re-written (Sebastian Bünger) 22119 * This implements low level retries (Sebastian Bünger) 22120 * Copy, Move, DirMove, PublicLink and About optional interfaces (Sebastian Bünger) 22121 * Improved general error handling (Sebastian Bünger) 22122 * Removed ListR for now due to inconsistent behaviour (Sebastian Bünger) 22123 22124 ## v1.44 - 2018-10-15 22125 22126 * New commands 22127 * serve ftp: Add ftp server (Antoine GIRARD) 22128 * settier: perform storage tier changes on supported remotes (sandeepkru) 22129 * New Features 22130 * Reworked command line help 22131 * Make default help less verbose (Nick Craig-Wood) 22132 * Split flags up into global and backend flags (Nick Craig-Wood) 22133 * Implement specialised help for flags and backends (Nick Craig-Wood) 22134 * Show URL of backend help page when starting config (Nick Craig-Wood) 22135 * stats: Long names now split in center (Joanna Marek) 22136 * Add `--log-format` flag for more control over log output (dcpu) 22137 * rc: Add support for OPTIONS and basic CORS (frenos) 22138 * stats: show FatalErrors and NoRetryErrors in stats (Cédric Connes) 22139 * Bug Fixes 22140 * Fix -P not ending with a new line (Nick Craig-Wood) 22141 * config: don't create default config dir when user supplies `--config` (albertony) 22142 * Don't print non-ASCII characters with `--progress` on windows (Nick Craig-Wood) 22143 * Correct logs for excluded items (ssaqua) 22144 * Mount 22145 * Remove EXPERIMENTAL tags (Nick Craig-Wood) 22146 * VFS 22147 * Fix race condition detected by serve ftp tests (Nick Craig-Wood) 22148 * Add vfs/poll-interval rc command (Fabian Möller) 22149 * Enable rename for nearly all remotes using server side Move or Copy (Nick Craig-Wood) 22150 * Reduce directory cache cleared by poll-interval (Fabian Möller) 22151 * Remove EXPERIMENTAL tags (Nick Craig-Wood) 22152 * Local 22153 * Skip bad symlinks in dir listing with -L enabled (Cédric Connes) 22154 * Preallocate files on Windows to reduce fragmentation (Nick Craig-Wood) 22155 * Preallocate files on linux with fallocate(2) (Nick Craig-Wood) 22156 * Cache 22157 * Add cache/fetch rc function (Fabian Möller) 22158 * Fix worker scale down (Fabian Möller) 22159 * Improve performance by not sending info requests for cached chunks (dcpu) 22160 * Fix error return value of cache/fetch rc method (Fabian Möller) 22161 * Documentation fix for cache-chunk-total-size (Anagh Kumar Baranwal) 22162 * Preserve leading / in wrapped remote path (Fabian Möller) 22163 * Add plex_insecure option to skip certificate validation (Fabian Möller) 22164 * Remove entries that no longer exist in the source (dcpu) 22165 * Crypt 22166 * Preserve leading / in wrapped remote path (Fabian Möller) 22167 * Alias 22168 * Fix handling of Windows network paths (Nick Craig-Wood) 22169 * Azure Blob 22170 * Add `--azureblob-list-chunk` parameter (Santiago Rodríguez) 22171 * Implemented settier command support on azureblob remote. (sandeepkru) 22172 * Work around SDK bug which causes errors for chunk-sized files (Nick Craig-Wood) 22173 * Box 22174 * Implement link sharing. (Sebastian Bünger) 22175 * Drive 22176 * Add `--drive-import-formats` - google docs can now be imported (Fabian Möller) 22177 * Rewrite mime type and extension handling (Fabian Möller) 22178 * Add document links (Fabian Möller) 22179 * Add support for multipart document extensions (Fabian Möller) 22180 * Add support for apps-script to json export (Fabian Möller) 22181 * Fix escaped chars in documents during list (Fabian Möller) 22182 * Add `--drive-v2-download-min-size` a workaround for slow downloads (Fabian Möller) 22183 * Improve directory notifications in ChangeNotify (Fabian Möller) 22184 * When listing team drives in config, continue on failure (Nick Craig-Wood) 22185 * FTP 22186 * Add a small pause after failed upload before deleting file (Nick Craig-Wood) 22187 * Google Cloud Storage 22188 * Fix service_account_file being ignored (Fabian Möller) 22189 * Jottacloud 22190 * Minor improvement in quota info (omit if unlimited) (albertony) 22191 * Add `--fast-list` support (albertony) 22192 * Add permanent delete support: `--jottacloud-hard-delete` (albertony) 22193 * Add link sharing support (albertony) 22194 * Fix handling of reserved characters. (Sebastian Bünger) 22195 * Fix socket leak on Object.Remove (Nick Craig-Wood) 22196 * Onedrive 22197 * Rework to support Microsoft Graph (Cnly) 22198 * **NB** this will require re-authenticating the remote 22199 * Removed upload cutoff and always do session uploads (Oliver Heyme) 22200 * Use single-part upload for empty files (Cnly) 22201 * Fix new fields not saved when editing old config (Alex Chen) 22202 * Fix sometimes special chars in filenames not replaced (Alex Chen) 22203 * Ignore OneNote files by default (Alex Chen) 22204 * Add link sharing support (jackyzy823) 22205 * S3 22206 * Use custom pacer, to retry operations when reasonable (Craig Miskell) 22207 * Use configured server-side-encryption and storace class options when calling CopyObject() (Paul Kohout) 22208 * Make `--s3-v2-auth` flag (Nick Craig-Wood) 22209 * Fix v2 auth on files with spaces (Nick Craig-Wood) 22210 * Union 22211 * Implement union backend which reads from multiple backends (Felix Brucker) 22212 * Implement optional interfaces (Move, DirMove, Copy etc) (Nick Craig-Wood) 22213 * Fix ChangeNotify to support multiple remotes (Fabian Möller) 22214 * Fix `--backup-dir` on union backend (Nick Craig-Wood) 22215 * WebDAV 22216 * Add another time format (Nick Craig-Wood) 22217 * Add a small pause after failed upload before deleting file (Nick Craig-Wood) 22218 * Add workaround for missing mtime (buergi) 22219 * Sharepoint: Renew cookies after 12hrs (Henning Surmeier) 22220 * Yandex 22221 * Remove redundant nil checks (teresy) 22222 22223 ## v1.43.1 - 2018-09-07 22224 22225 Point release to fix hubic and azureblob backends. 22226 22227 * Bug Fixes 22228 * ncdu: Return error instead of log.Fatal in Show (Fabian Möller) 22229 * cmd: Fix crash with `--progress` and `--stats 0` (Nick Craig-Wood) 22230 * docs: Tidy website display (Anagh Kumar Baranwal) 22231 * Azure Blob: 22232 * Fix multi-part uploads. (sandeepkru) 22233 * Hubic 22234 * Fix uploads (Nick Craig-Wood) 22235 * Retry auth fetching if it fails to make hubic more reliable (Nick Craig-Wood) 22236 22237 ## v1.43 - 2018-09-01 22238 22239 * New backends 22240 * Jottacloud (Sebastian Bünger) 22241 * New commands 22242 * copyurl: copies a URL to a remote (Denis) 22243 * New Features 22244 * Reworked config for backends (Nick Craig-Wood) 22245 * All backend config can now be supplied by command line, env var or config file 22246 * Advanced section in the config wizard for the optional items 22247 * A large step towards rclone backends being usable in other go software 22248 * Allow on the fly remotes with :backend: syntax 22249 * Stats revamp 22250 * Add `--progress`/`-P` flag to show interactive progress (Nick Craig-Wood) 22251 * Show the total progress of the sync in the stats (Nick Craig-Wood) 22252 * Add `--stats-one-line` flag for single line stats (Nick Craig-Wood) 22253 * Added weekday schedule into `--bwlimit` (Mateusz) 22254 * lsjson: Add option to show the original object IDs (Fabian Möller) 22255 * serve webdav: Make Content-Type without reading the file and add `--etag-hash` (Nick Craig-Wood) 22256 * build 22257 * Build macOS with native compiler (Nick Craig-Wood) 22258 * Update to use go1.11 for the build (Nick Craig-Wood) 22259 * rc 22260 * Added core/stats to return the stats (reddi1) 22261 * `version --check`: Prints the current release and beta versions (Nick Craig-Wood) 22262 * Bug Fixes 22263 * accounting 22264 * Fix time to completion estimates (Nick Craig-Wood) 22265 * Fix moving average speed for file stats (Nick Craig-Wood) 22266 * config: Fix error reading password from piped input (Nick Craig-Wood) 22267 * move: Fix `--delete-empty-src-dirs` flag to delete all empty dirs on move (ishuah) 22268 * Mount 22269 * Implement `--daemon-timeout` flag for OSXFUSE (Nick Craig-Wood) 22270 * Fix mount `--daemon` not working with encrypted config (Alex Chen) 22271 * Clip the number of blocks to 2^32-1 on macOS - fixes borg backup (Nick Craig-Wood) 22272 * VFS 22273 * Enable vfs-read-chunk-size by default (Fabian Möller) 22274 * Add the vfs/refresh rc command (Fabian Möller) 22275 * Add non recursive mode to vfs/refresh rc command (Fabian Möller) 22276 * Try to seek buffer on read only files (Fabian Möller) 22277 * Local 22278 * Fix crash when deprecated `--local-no-unicode-normalization` is supplied (Nick Craig-Wood) 22279 * Fix mkdir error when trying to copy files to the root of a drive on windows (Nick Craig-Wood) 22280 * Cache 22281 * Fix nil pointer deref when using lsjson on cached directory (Nick Craig-Wood) 22282 * Fix nil pointer deref for occasional crash on playback (Nick Craig-Wood) 22283 * Crypt 22284 * Fix accounting when checking hashes on upload (Nick Craig-Wood) 22285 * Amazon Cloud Drive 22286 * Make very clear in the docs that rclone has no ACD keys (Nick Craig-Wood) 22287 * Azure Blob 22288 * Add connection string and SAS URL auth (Nick Craig-Wood) 22289 * List the container to see if it exists (Nick Craig-Wood) 22290 * Port new Azure Blob Storage SDK (sandeepkru) 22291 * Added blob tier, tier between Hot, Cool and Archive. (sandeepkru) 22292 * Remove leading / from paths (Nick Craig-Wood) 22293 * B2 22294 * Support Application Keys (Nick Craig-Wood) 22295 * Remove leading / from paths (Nick Craig-Wood) 22296 * Box 22297 * Fix upload of > 2GB files on 32 bit platforms (Nick Craig-Wood) 22298 * Make `--box-commit-retries` flag defaulting to 100 to fix large uploads (Nick Craig-Wood) 22299 * Drive 22300 * Add `--drive-keep-revision-forever` flag (lewapm) 22301 * Handle gdocs when filtering file names in list (Fabian Möller) 22302 * Support using `--fast-list` for large speedups (Fabian Möller) 22303 * FTP 22304 * Fix Put mkParentDir failed: 521 for BunnyCDN (Nick Craig-Wood) 22305 * Google Cloud Storage 22306 * Fix index out of range error with `--fast-list` (Nick Craig-Wood) 22307 * Jottacloud 22308 * Fix MD5 error check (Oliver Heyme) 22309 * Handle empty time values (Martin Polden) 22310 * Calculate missing MD5s (Oliver Heyme) 22311 * Docs, fixes and tests for MD5 calculation (Nick Craig-Wood) 22312 * Add optional MimeTyper interface. (Sebastian Bünger) 22313 * Implement optional About interface (for `df` support). (Sebastian Bünger) 22314 * Mega 22315 * Wait for events instead of arbitrary sleeping (Nick Craig-Wood) 22316 * Add `--mega-hard-delete` flag (Nick Craig-Wood) 22317 * Fix failed logins with upper case chars in email (Nick Craig-Wood) 22318 * Onedrive 22319 * Shared folder support (Yoni Jah) 22320 * Implement DirMove (Cnly) 22321 * Fix rmdir sometimes deleting directories with contents (Nick Craig-Wood) 22322 * Pcloud 22323 * Delete half uploaded files on upload error (Nick Craig-Wood) 22324 * Qingstor 22325 * Remove leading / from paths (Nick Craig-Wood) 22326 * S3 22327 * Fix index out of range error with `--fast-list` (Nick Craig-Wood) 22328 * Add `--s3-force-path-style` (Nick Craig-Wood) 22329 * Add support for KMS Key ID (bsteiss) 22330 * Remove leading / from paths (Nick Craig-Wood) 22331 * Swift 22332 * Add `storage_policy` (Ruben Vandamme) 22333 * Make it so just `storage_url` or `auth_token` can be overidden (Nick Craig-Wood) 22334 * Fix server side copy bug for unusal file names (Nick Craig-Wood) 22335 * Remove leading / from paths (Nick Craig-Wood) 22336 * WebDAV 22337 * Ensure we call MKCOL with a URL with a trailing / for QNAP interop (Nick Craig-Wood) 22338 * If root ends with / then don't check if it is a file (Nick Craig-Wood) 22339 * Don't accept redirects when reading metadata (Nick Craig-Wood) 22340 * Add bearer token (Macaroon) support for dCache (Nick Craig-Wood) 22341 * Document dCache and Macaroons (Onno Zweers) 22342 * Sharepoint recursion with different depth (Henning) 22343 * Attempt to remove failed uploads (Nick Craig-Wood) 22344 * Yandex 22345 * Fix listing/deleting files in the root (Nick Craig-Wood) 22346 22347 ## v1.42 - 2018-06-16 22348 22349 * New backends 22350 * OpenDrive (Oliver Heyme, Jakub Karlicek, ncw) 22351 * New commands 22352 * deletefile command (Filip Bartodziej) 22353 * New Features 22354 * copy, move: Copy single files directly, don't use `--files-from` work-around 22355 * this makes them much more efficient 22356 * Implement `--max-transfer` flag to quit transferring at a limit 22357 * make exit code 8 for `--max-transfer` exceeded 22358 * copy: copy empty source directories to destination (Ishuah Kariuki) 22359 * check: Add `--one-way` flag (Kasper Byrdal Nielsen) 22360 * Add siginfo handler for macOS for ctrl-T stats (kubatasiemski) 22361 * rc 22362 * add core/gc to run a garbage collection on demand 22363 * enable go profiling by default on the `--rc` port 22364 * return error from remote on failure 22365 * lsf 22366 * Add `--absolute` flag to add a leading / onto path names 22367 * Add `--csv` flag for compliant CSV output 22368 * Add 'm' format specifier to show the MimeType 22369 * Implement 'i' format for showing object ID 22370 * lsjson 22371 * Add MimeType to the output 22372 * Add ID field to output to show Object ID 22373 * Add `--retries-sleep` flag (Benjamin Joseph Dag) 22374 * Oauth tidy up web page and error handling (Henning Surmeier) 22375 * Bug Fixes 22376 * Password prompt output with `--log-file` fixed for unix (Filip Bartodziej) 22377 * Calculate ModifyWindow each time on the fly to fix various problems (Stefan Breunig) 22378 * Mount 22379 * Only print "File.rename error" if there actually is an error (Stefan Breunig) 22380 * Delay rename if file has open writers instead of failing outright (Stefan Breunig) 22381 * Ensure atexit gets run on interrupt 22382 * macOS enhancements 22383 * Make `--noappledouble` `--noapplexattr` 22384 * Add `--volname` flag and remove special chars from it 22385 * Make Get/List/Set/Remove xattr return ENOSYS for efficiency 22386 * Make `--daemon` work for macOS without CGO 22387 * VFS 22388 * Add `--vfs-read-chunk-size` and `--vfs-read-chunk-size-limit` (Fabian Möller) 22389 * Fix ChangeNotify for new or changed folders (Fabian Möller) 22390 * Local 22391 * Fix symlink/junction point directory handling under Windows 22392 * **NB** you will need to add `-L` to your command line to copy files with reparse points 22393 * Cache 22394 * Add non cached dirs on notifications (Remus Bunduc) 22395 * Allow root to be expired from rc (Remus Bunduc) 22396 * Clean remaining empty folders from temp upload path (Remus Bunduc) 22397 * Cache lists using batch writes (Remus Bunduc) 22398 * Use secure websockets for HTTPS Plex addresses (John Clayton) 22399 * Reconnect plex websocket on failures (Remus Bunduc) 22400 * Fix panic when running without plex configs (Remus Bunduc) 22401 * Fix root folder caching (Remus Bunduc) 22402 * Crypt 22403 * Check the crypted hash of files when uploading for extra data security 22404 * Dropbox 22405 * Make Dropbox for business folders accessible using an initial `/` in the path 22406 * Google Cloud Storage 22407 * Low level retry all operations if necessary 22408 * Google Drive 22409 * Add `--drive-acknowledge-abuse` to download flagged files 22410 * Add `--drive-alternate-export` to fix large doc export 22411 * Don't attempt to choose Team Drives when using rclone config create 22412 * Fix change list polling with team drives 22413 * Fix ChangeNotify for folders (Fabian Möller) 22414 * Fix about (and df on a mount) for team drives 22415 * Onedrive 22416 * Errorhandler for onedrive for business requests (Henning Surmeier) 22417 * S3 22418 * Adjust upload concurrency with `--s3-upload-concurrency` (themylogin) 22419 * Fix `--s3-chunk-size` which was always using the minimum 22420 * SFTP 22421 * Add `--ssh-path-override` flag (Piotr Oleszczyk) 22422 * Fix slow downloads for long latency connections 22423 * Webdav 22424 * Add workarounds for biz.mail.ru 22425 * Ignore Reason-Phrase in status line to fix 4shared (Rodrigo) 22426 * Better error message generation 22427 22428 ## v1.41 - 2018-04-28 22429 22430 * New backends 22431 * Mega support added 22432 * Webdav now supports SharePoint cookie authentication (hensur) 22433 * New commands 22434 * link: create public link to files and folders (Stefan Breunig) 22435 * about: gets quota info from a remote (a-roussos, ncw) 22436 * hashsum: a generic tool for any hash to produce md5sum like output 22437 * New Features 22438 * lsd: Add -R flag and fix and update docs for all ls commands 22439 * ncdu: added a "refresh" key - CTRL-L (Keith Goldfarb) 22440 * serve restic: Add append-only mode (Steve Kriss) 22441 * serve restic: Disallow overwriting files in append-only mode (Alexander Neumann) 22442 * serve restic: Print actual listener address (Matt Holt) 22443 * size: Add --json flag (Matthew Holt) 22444 * sync: implement --ignore-errors (Mateusz Pabian) 22445 * dedupe: Add dedupe largest functionality (Richard Yang) 22446 * fs: Extend SizeSuffix to include TB and PB for rclone about 22447 * fs: add --dump goroutines and --dump openfiles for debugging 22448 * rc: implement core/memstats to print internal memory usage info 22449 * rc: new call rc/pid (Michael P. Dubner) 22450 * Compile 22451 * Drop support for go1.6 22452 * Release 22453 * Fix `make tarball` (Chih-Hsuan Yen) 22454 * Bug Fixes 22455 * filter: fix --min-age and --max-age together check 22456 * fs: limit MaxIdleConns and MaxIdleConnsPerHost in transport 22457 * lsd,lsf: make sure all times we output are in local time 22458 * rc: fix setting bwlimit to unlimited 22459 * rc: take note of the --rc-addr flag too as per the docs 22460 * Mount 22461 * Use About to return the correct disk total/used/free (eg in `df`) 22462 * Set `--attr-timeout default` to `1s` - fixes: 22463 * rclone using too much memory 22464 * rclone not serving files to samba 22465 * excessive time listing directories 22466 * Fix `df -i` (upstream fix) 22467 * VFS 22468 * Filter files `.` and `..` from directory listing 22469 * Only make the VFS cache if --vfs-cache-mode > Off 22470 * Local 22471 * Add --local-no-check-updated to disable updated file checks 22472 * Retry remove on Windows sharing violation error 22473 * Cache 22474 * Flush the memory cache after close 22475 * Purge file data on notification 22476 * Always forget parent dir for notifications 22477 * Integrate with Plex websocket 22478 * Add rc cache/stats (seuffert) 22479 * Add info log on notification 22480 * Box 22481 * Fix failure reading large directories - parse file/directory size as float 22482 * Dropbox 22483 * Fix crypt+obfuscate on dropbox 22484 * Fix repeatedly uploading the same files 22485 * FTP 22486 * Work around strange response from box FTP server 22487 * More workarounds for FTP servers to fix mkParentDir error 22488 * Fix no error on listing non-existent directory 22489 * Google Cloud Storage 22490 * Add service_account_credentials (Matt Holt) 22491 * Detect bucket presence by listing it - minimises permissions needed 22492 * Ignore zero length directory markers 22493 * Google Drive 22494 * Add service_account_credentials (Matt Holt) 22495 * Fix directory move leaving a hardlinked directory behind 22496 * Return proper google errors when Opening files 22497 * When initialized with a filepath, optional features used incorrect root path (Stefan Breunig) 22498 * HTTP 22499 * Fix sync for servers which don't return Content-Length in HEAD 22500 * Onedrive 22501 * Add QuickXorHash support for OneDrive for business 22502 * Fix socket leak in multipart session upload 22503 * S3 22504 * Look in S3 named profile files for credentials 22505 * Add `--s3-disable-checksum` to disable checksum uploading (Chris Redekop) 22506 * Hierarchical configuration support (Giri Badanahatti) 22507 * Add in config for all the supported S3 providers 22508 * Add One Zone Infrequent Access storage class (Craig Rachel) 22509 * Add --use-server-modtime support (Peter Baumgartner) 22510 * Add --s3-chunk-size option to control multipart uploads 22511 * Ignore zero length directory markers 22512 * SFTP 22513 * Update docs to match code, fix typos and clarify disable_hashcheck prompt (Michael G. Noll) 22514 * Update docs with Synology quirks 22515 * Fail soft with a debug on hash failure 22516 * Swift 22517 * Add --use-server-modtime support (Peter Baumgartner) 22518 * Webdav 22519 * Support SharePoint cookie authentication (hensur) 22520 * Strip leading and trailing / off root 22521 22522 ## v1.40 - 2018-03-19 22523 22524 * New backends 22525 * Alias backend to create aliases for existing remote names (Fabian Möller) 22526 * New commands 22527 * `lsf`: list for parsing purposes (Jakub Tasiemski) 22528 * by default this is a simple non recursive list of files and directories 22529 * it can be configured to add more info in an easy to parse way 22530 * `serve restic`: for serving a remote as a Restic REST endpoint 22531 * This enables restic to use any backends that rclone can access 22532 * Thanks Alexander Neumann for help, patches and review 22533 * `rc`: enable the remote control of a running rclone 22534 * The running rclone must be started with --rc and related flags. 22535 * Currently there is support for bwlimit, and flushing for mount and cache. 22536 * New Features 22537 * `--max-delete` flag to add a delete threshold (Bjørn Erik Pedersen) 22538 * All backends now support RangeOption for ranged Open 22539 * `cat`: Use RangeOption for limited fetches to make more efficient 22540 * `cryptcheck`: make reading of nonce more efficient with RangeOption 22541 * serve http/webdav/restic 22542 * support SSL/TLS 22543 * add `--user` `--pass` and `--htpasswd` for authentication 22544 * `copy`/`move`: detect file size change during copy/move and abort transfer (ishuah) 22545 * `cryptdecode`: added option to return encrypted file names. (ishuah) 22546 * `lsjson`: add `--encrypted` to show encrypted name (Jakub Tasiemski) 22547 * Add `--stats-file-name-length` to specify the printed file name length for stats (Will Gunn) 22548 * Compile 22549 * Code base was shuffled and factored 22550 * backends moved into a backend directory 22551 * large packages split up 22552 * See the CONTRIBUTING.md doc for info as to what lives where now 22553 * Update to using go1.10 as the default go version 22554 * Implement daily [full integration tests](https://pub.rclone.org/integration-tests/) 22555 * Release 22556 * Include a source tarball and sign it and the binaries 22557 * Sign the git tags as part of the release process 22558 * Add .deb and .rpm packages as part of the build 22559 * Make a beta release for all branches on the main repo (but not pull requests) 22560 * Bug Fixes 22561 * config: fixes errors on non existing config by loading config file only on first access 22562 * config: retry saving the config after failure (Mateusz) 22563 * sync: when using `--backup-dir` don't delete files if we can't set their modtime 22564 * this fixes odd behaviour with Dropbox and `--backup-dir` 22565 * fshttp: fix idle timeouts for HTTP connections 22566 * `serve http`: fix serving files with : in - fixes 22567 * Fix `--exclude-if-present` to ignore directories which it doesn't have permission for (Iakov Davydov) 22568 * Make accounting work properly with crypt and b2 22569 * remove `--no-traverse` flag because it is obsolete 22570 * Mount 22571 * Add `--attr-timeout` flag to control attribute caching in kernel 22572 * this now defaults to 0 which is correct but less efficient 22573 * see [the mount docs](/commands/rclone_mount/#attribute-caching) for more info 22574 * Add `--daemon` flag to allow mount to run in the background (ishuah) 22575 * Fix: Return ENOSYS rather than EIO on attempted link 22576 * This fixes FileZilla accessing an rclone mount served over sftp. 22577 * Fix setting modtime twice 22578 * Mount tests now run on CI for Linux (mount & cmount)/Mac/Windows 22579 * Many bugs fixed in the VFS layer - see below 22580 * VFS 22581 * Many fixes for `--vfs-cache-mode` writes and above 22582 * Update cached copy if we know it has changed (fixes stale data) 22583 * Clean path names before using them in the cache 22584 * Disable cache cleaner if `--vfs-cache-poll-interval=0` 22585 * Fill and clean the cache immediately on startup 22586 * Fix Windows opening every file when it stats the file 22587 * Fix applying modtime for an open Write Handle 22588 * Fix creation of files when truncating 22589 * Write 0 bytes when flushing unwritten handles to avoid race conditions in FUSE 22590 * Downgrade "poll-interval is not supported" message to Info 22591 * Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 22592 * Local 22593 * Downgrade "invalid cross-device link: trying copy" to debug 22594 * Make DirMove return fs.ErrorCantDirMove to allow fallback to Copy for cross device 22595 * Fix race conditions updating the hashes 22596 * Cache 22597 * Add support for polling - cache will update when remote changes on supported backends 22598 * Reduce log level for Plex api 22599 * Fix dir cache issue 22600 * Implement `--cache-db-wait-time` flag 22601 * Improve efficiency with RangeOption and RangeSeek 22602 * Fix dirmove with temp fs enabled 22603 * Notify vfs when using temp fs 22604 * Offline uploading 22605 * Remote control support for path flushing 22606 * Amazon cloud drive 22607 * Rclone no longer has any working keys - disable integration tests 22608 * Implement DirChangeNotify to notify cache/vfs/mount of changes 22609 * Azureblob 22610 * Don't check for bucket/container presense if listing was OK 22611 * this makes rclone do one less request per invocation 22612 * Improve accounting for chunked uploads 22613 * Backblaze B2 22614 * Don't check for bucket/container presense if listing was OK 22615 * this makes rclone do one less request per invocation 22616 * Box 22617 * Improve accounting for chunked uploads 22618 * Dropbox 22619 * Fix custom oauth client parameters 22620 * Google Cloud Storage 22621 * Don't check for bucket/container presense if listing was OK 22622 * this makes rclone do one less request per invocation 22623 * Google Drive 22624 * Migrate to api v3 (Fabian Möller) 22625 * Add scope configuration and root folder selection 22626 * Add `--drive-impersonate` for service accounts 22627 * thanks to everyone who tested, explored and contributed docs 22628 * Add `--drive-use-created-date` to use created date as modified date (nbuchanan) 22629 * Request the export formats only when required 22630 * This makes rclone quicker when there are no google docs 22631 * Fix finding paths with latin1 chars (a workaround for a drive bug) 22632 * Fix copying of a single Google doc file 22633 * Fix `--drive-auth-owner-only` to look in all directories 22634 * HTTP 22635 * Fix handling of directories with & in 22636 * Onedrive 22637 * Removed upload cutoff and always do session uploads 22638 * this stops the creation of multiple versions on business onedrive 22639 * Overwrite object size value with real size when reading file. (Victor) 22640 * this fixes oddities when onedrive misreports the size of images 22641 * Pcloud 22642 * Remove unused chunked upload flag and code 22643 * Qingstor 22644 * Don't check for bucket/container presense if listing was OK 22645 * this makes rclone do one less request per invocation 22646 * S3 22647 * Support hashes for multipart files (Chris Redekop) 22648 * Initial support for IBM COS (S3) (Giri Badanahatti) 22649 * Update docs to discourage use of v2 auth with CEPH and others 22650 * Don't check for bucket/container presense if listing was OK 22651 * this makes rclone do one less request per invocation 22652 * Fix server side copy and set modtime on files with + in 22653 * SFTP 22654 * Add option to disable remote hash check command execution (Jon Fautley) 22655 * Add `--sftp-ask-password` flag to prompt for password when needed (Leo R. Lundgren) 22656 * Add `set_modtime` configuration option 22657 * Fix following of symlinks 22658 * Fix reading config file outside of Fs setup 22659 * Fix reading $USER in username fallback not $HOME 22660 * Fix running under crontab - Use correct OS way of reading username 22661 * Swift 22662 * Fix refresh of authentication token 22663 * in v1.39 a bug was introduced which ignored new tokens - this fixes it 22664 * Fix extra HEAD transaction when uploading a new file 22665 * Don't check for bucket/container presense if listing was OK 22666 * this makes rclone do one less request per invocation 22667 * Webdav 22668 * Add new time formats to support mydrive.ch and others 22669 22670 ## v1.39 - 2017-12-23 22671 22672 * New backends 22673 * WebDAV 22674 * tested with nextcloud, owncloud, put.io and others! 22675 * Pcloud 22676 * cache - wraps a cache around other backends (Remus Bunduc) 22677 * useful in combination with mount 22678 * NB this feature is in beta so use with care 22679 * New commands 22680 * serve command with subcommands: 22681 * serve webdav: this implements a webdav server for any rclone remote. 22682 * serve http: command to serve a remote over HTTP 22683 * config: add sub commands for full config file management 22684 * create/delete/dump/edit/file/password/providers/show/update 22685 * touch: to create or update the timestamp of a file (Jakub Tasiemski) 22686 * New Features 22687 * curl install for rclone (Filip Bartodziej) 22688 * --stats now shows percentage, size, rate and ETA in condensed form (Ishuah Kariuki) 22689 * --exclude-if-present to exclude a directory if a file is present (Iakov Davydov) 22690 * rmdirs: add --leave-root flag (lewpam) 22691 * move: add --delete-empty-src-dirs flag to remove dirs after move (Ishuah Kariuki) 22692 * Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters 22693 * Obscure X-Auth-Token: from headers when dumping too 22694 * Document and implement exit codes for different failure modes (Ishuah Kariuki) 22695 * Compile 22696 * Bug Fixes 22697 * Retry lots more different types of errors to make multipart transfers more reliable 22698 * Save the config before asking for a token, fixes disappearing oauth config 22699 * Warn the user if --include and --exclude are used together (Ernest Borowski) 22700 * Fix duplicate files (eg on Google drive) causing spurious copies 22701 * Allow trailing and leading whitespace for passwords (Jason Rose) 22702 * ncdu: fix crashes on empty directories 22703 * rcat: fix goroutine leak 22704 * moveto/copyto: Fix to allow copying to the same name 22705 * Mount 22706 * --vfs-cache mode to make writes into mounts more reliable. 22707 * this requires caching files on the disk (see --cache-dir) 22708 * As this is a new feature, use with care 22709 * Use sdnotify to signal systemd the mount is ready (Fabian Möller) 22710 * Check if directory is not empty before mounting (Ernest Borowski) 22711 * Local 22712 * Add error message for cross file system moves 22713 * Fix equality check for times 22714 * Dropbox 22715 * Rework multipart upload 22716 * buffer the chunks when uploading large files so they can be retried 22717 * change default chunk size to 48MB now we are buffering them in memory 22718 * retry every error after the first chunk is done successfully 22719 * Fix error when renaming directories 22720 * Swift 22721 * Fix crash on bad authentication 22722 * Google Drive 22723 * Add service account support (Tim Cooijmans) 22724 * S3 22725 * Make it work properly with Digital Ocean Spaces (Andrew Starr-Bochicchio) 22726 * Fix crash if a bad listing is received 22727 * Add support for ECS task IAM roles (David Minor) 22728 * Backblaze B2 22729 * Fix multipart upload retries 22730 * Fix --hard-delete to make it work 100% of the time 22731 * Swift 22732 * Allow authentication with storage URL and auth key (Giovanni Pizzi) 22733 * Add new fields for swift configuration to support IBM Bluemix Swift (Pierre Carlson) 22734 * Add OS_TENANT_ID and OS_USER_ID to config 22735 * Allow configs with user id instead of user name 22736 * Check if swift segments container exists before creating (John Leach) 22737 * Fix memory leak in swift transfers (upstream fix) 22738 * SFTP 22739 * Add option to enable the use of aes128-cbc cipher (Jon Fautley) 22740 * Amazon cloud drive 22741 * Fix download of large files failing with "Only one auth mechanism allowed" 22742 * crypt 22743 * Option to encrypt directory names or leave them intact 22744 * Implement DirChangeNotify (Fabian Möller) 22745 * onedrive 22746 * Add option to choose resourceURL during setup of OneDrive Business account if more than one is available for user 22747 22748 ## v1.38 - 2017-09-30 22749 22750 * New backends 22751 * Azure Blob Storage (thanks Andrei Dragomir) 22752 * Box 22753 * Onedrive for Business (thanks Oliver Heyme) 22754 * QingStor from QingCloud (thanks wuyu) 22755 * New commands 22756 * `rcat` - read from standard input and stream upload 22757 * `tree` - shows a nicely formatted recursive listing 22758 * `cryptdecode` - decode crypted file names (thanks ishuah) 22759 * `config show` - print the config file 22760 * `config file` - print the config file location 22761 * New Features 22762 * Empty directories are deleted on `sync` 22763 * `dedupe` - implement merging of duplicate directories 22764 * `check` and `cryptcheck` made more consistent and use less memory 22765 * `cleanup` for remaining remotes (thanks ishuah) 22766 * `--immutable` for ensuring that files don't change (thanks Jacob McNamee) 22767 * `--user-agent` option (thanks Alex McGrath Kraak) 22768 * `--disable` flag to disable optional features 22769 * `--bind` flag for choosing the local addr on outgoing connections 22770 * Support for zsh auto-completion (thanks bpicode) 22771 * Stop normalizing file names but do a normalized compare in `sync` 22772 * Compile 22773 * Update to using go1.9 as the default go version 22774 * Remove snapd build due to maintenance problems 22775 * Bug Fixes 22776 * Improve retriable error detection which makes multipart uploads better 22777 * Make `check` obey `--ignore-size` 22778 * Fix bwlimit toggle in conjunction with schedules (thanks cbruegg) 22779 * `config` ensures newly written config is on the same mount 22780 * Local 22781 * Revert to copy when moving file across file system boundaries 22782 * `--skip-links` to suppress symlink warnings (thanks Zhiming Wang) 22783 * Mount 22784 * Re-use `rcat` internals to support uploads from all remotes 22785 * Dropbox 22786 * Fix "entry doesn't belong in directory" error 22787 * Stop using deprecated API methods 22788 * Swift 22789 * Fix server side copy to empty container with `--fast-list` 22790 * Google Drive 22791 * Change the default for `--drive-use-trash` to `true` 22792 * S3 22793 * Set session token when using STS (thanks Girish Ramakrishnan) 22794 * Glacier docs and error messages (thanks Jan Varho) 22795 * Read 1000 (not 1024) items in dir listings to fix Wasabi 22796 * Backblaze B2 22797 * Fix SHA1 mismatch when downloading files with no SHA1 22798 * Calculate missing hashes on the fly instead of spooling 22799 * `--b2-hard-delete` to permanently delete (not hide) files (thanks John Papandriopoulos) 22800 * Hubic 22801 * Fix creating containers - no longer have to use the `default` container 22802 * Swift 22803 * Optionally configure from a standard set of OpenStack environment vars 22804 * Add `endpoint_type` config 22805 * Google Cloud Storage 22806 * Fix bucket creation to work with limited permission users 22807 * SFTP 22808 * Implement connection pooling for multiple ssh connections 22809 * Limit new connections per second 22810 * Add support for MD5 and SHA1 hashes where available (thanks Christian Brüggemann) 22811 * HTTP 22812 * Fix URL encoding issues 22813 * Fix directories with `:` in 22814 * Fix panic with URL encoded content 22815 22816 ## v1.37 - 2017-07-22 22817 22818 * New backends 22819 * FTP - thanks to Antonio Messina 22820 * HTTP - thanks to Vasiliy Tolstov 22821 * New commands 22822 * rclone ncdu - for exploring a remote with a text based user interface. 22823 * rclone lsjson - for listing with a machine readable output 22824 * rclone dbhashsum - to show Dropbox style hashes of files (local or Dropbox) 22825 * New Features 22826 * Implement --fast-list flag 22827 * This allows remotes to list recursively if they can 22828 * This uses less transactions (important if you pay for them) 22829 * This may or may not be quicker 22830 * This will use more memory as it has to hold the listing in memory 22831 * --old-sync-method deprecated - the remaining uses are covered by --fast-list 22832 * This involved a major re-write of all the listing code 22833 * Add --tpslimit and --tpslimit-burst to limit transactions per second 22834 * this is useful in conjuction with `rclone mount` to limit external apps 22835 * Add --stats-log-level so can see --stats without -v 22836 * Print password prompts to stderr - Hraban Luyat 22837 * Warn about duplicate files when syncing 22838 * Oauth improvements 22839 * allow auth_url and token_url to be set in the config file 22840 * Print redirection URI if using own credentials. 22841 * Don't Mkdir at the start of sync to save transactions 22842 * Compile 22843 * Update build to go1.8.3 22844 * Require go1.6 for building rclone 22845 * Compile 386 builds with "GO386=387" for maximum compatibility 22846 * Bug Fixes 22847 * Fix menu selection when no remotes 22848 * Config saving reworked to not kill the file if disk gets full 22849 * Don't delete remote if name does not change while renaming 22850 * moveto, copyto: report transfers and checks as per move and copy 22851 * Local 22852 * Add --local-no-unicode-normalization flag - Bob Potter 22853 * Mount 22854 * Now supported on Windows using cgofuse and WinFsp - thanks to Bill Zissimopoulos for much help 22855 * Compare checksums on upload/download via FUSE 22856 * Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM - Jérôme Vizcaino 22857 * On read only open of file, make open pending until first read 22858 * Make --read-only reject modify operations 22859 * Implement ModTime via FUSE for remotes that support it 22860 * Allow modTime to be changed even before all writers are closed 22861 * Fix panic on renames 22862 * Fix hang on errored upload 22863 * Crypt 22864 * Report the name:root as specified by the user 22865 * Add an "obfuscate" option for filename encryption - Stephen Harris 22866 * Amazon Drive 22867 * Fix initialization order for token renewer 22868 * Remove revoked credentials, allow oauth proxy config and update docs 22869 * B2 22870 * Reduce minimum chunk size to 5MB 22871 * Drive 22872 * Add team drive support 22873 * Reduce bandwidth by adding fields for partial responses - Martin Kristensen 22874 * Implement --drive-shared-with-me flag to view shared with me files - Danny Tsai 22875 * Add --drive-trashed-only to read only the files in the trash 22876 * Remove obsolete --drive-full-list 22877 * Add missing seek to start on retries of chunked uploads 22878 * Fix stats accounting for upload 22879 * Convert / in names to a unicode equivalent (/) 22880 * Poll for Google Drive changes when mounted 22881 * OneDrive 22882 * Fix the uploading of files with spaces 22883 * Fix initialization order for token renewer 22884 * Display speeds accurately when uploading - Yoni Jah 22885 * Swap to using http://localhost:53682/ as redirect URL - Michael Ledin 22886 * Retry on token expired error, reset upload body on retry - Yoni Jah 22887 * Google Cloud Storage 22888 * Add ability to specify location and storage class via config and command line - thanks gdm85 22889 * Create container if necessary on server side copy 22890 * Increase directory listing chunk to 1000 to increase performance 22891 * Obtain a refresh token for GCS - Steven Lu 22892 * Yandex 22893 * Fix the name reported in log messages (was empty) 22894 * Correct error return for listing empty directory 22895 * Dropbox 22896 * Rewritten to use the v2 API 22897 * Now supports ModTime 22898 * Can only set by uploading the file again 22899 * If you uploaded with an old rclone, rclone may upload everything again 22900 * Use `--size-only` or `--checksum` to avoid this 22901 * Now supports the Dropbox content hashing scheme 22902 * Now supports low level retries 22903 * S3 22904 * Work around eventual consistency in bucket creation 22905 * Create container if necessary on server side copy 22906 * Add us-east-2 (Ohio) and eu-west-2 (London) S3 regions - Zahiar Ahmed 22907 * Swift, Hubic 22908 * Fix zero length directory markers showing in the subdirectory listing 22909 * this caused lots of duplicate transfers 22910 * Fix paged directory listings 22911 * this caused duplicate directory errors 22912 * Create container if necessary on server side copy 22913 * Increase directory listing chunk to 1000 to increase performance 22914 * Make sensible error if the user forgets the container 22915 * SFTP 22916 * Add support for using ssh key files 22917 * Fix under Windows 22918 * Fix ssh agent on Windows 22919 * Adapt to latest version of library - Igor Kharin 22920 22921 ## v1.36 - 2017-03-18 22922 22923 * New Features 22924 * SFTP remote (Jack Schmidt) 22925 * Re-implement sync routine to work a directory at a time reducing memory usage 22926 * Logging revamped to be more inline with rsync - now much quieter 22927 * -v only shows transfers 22928 * -vv is for full debug 22929 * --syslog to log to syslog on capable platforms 22930 * Implement --backup-dir and --suffix 22931 * Implement --track-renames (initial implementation by Bjørn Erik Pedersen) 22932 * Add time-based bandwidth limits (Lukas Loesche) 22933 * rclone cryptcheck: checks integrity of crypt remotes 22934 * Allow all config file variables and options to be set from environment variables 22935 * Add --buffer-size parameter to control buffer size for copy 22936 * Make --delete-after the default 22937 * Add --ignore-checksum flag (fixed by Hisham Zarka) 22938 * rclone check: Add --download flag to check all the data, not just hashes 22939 * rclone cat: add --head, --tail, --offset, --count and --discard 22940 * rclone config: when choosing from a list, allow the value to be entered too 22941 * rclone config: allow rename and copy of remotes 22942 * rclone obscure: for generating encrypted passwords for rclone's config (T.C. Ferguson) 22943 * Comply with XDG Base Directory specification (Dario Giovannetti) 22944 * this moves the default location of the config file in a backwards compatible way 22945 * Release changes 22946 * Ubuntu snap support (Dedsec1) 22947 * Compile with go 1.8 22948 * MIPS/Linux big and little endian support 22949 * Bug Fixes 22950 * Fix copyto copying things to the wrong place if the destination dir didn't exist 22951 * Fix parsing of remotes in moveto and copyto 22952 * Fix --delete-before deleting files on copy 22953 * Fix --files-from with an empty file copying everything 22954 * Fix sync: don't update mod times if --dry-run set 22955 * Fix MimeType propagation 22956 * Fix filters to add ** rules to directory rules 22957 * Local 22958 * Implement -L, --copy-links flag to allow rclone to follow symlinks 22959 * Open files in write only mode so rclone can write to an rclone mount 22960 * Fix unnormalised unicode causing problems reading directories 22961 * Fix interaction between -x flag and --max-depth 22962 * Mount 22963 * Implement proper directory handling (mkdir, rmdir, renaming) 22964 * Make include and exclude filters apply to mount 22965 * Implement read and write async buffers - control with --buffer-size 22966 * Fix fsync on for directories 22967 * Fix retry on network failure when reading off crypt 22968 * Crypt 22969 * Add --crypt-show-mapping to show encrypted file mapping 22970 * Fix crypt writer getting stuck in a loop 22971 * **IMPORTANT** this bug had the potential to cause data corruption when 22972 * reading data from a network based remote and 22973 * writing to a crypt on Google Drive 22974 * Use the cryptcheck command to validate your data if you are concerned 22975 * If syncing two crypt remotes, sync the unencrypted remote 22976 * Amazon Drive 22977 * Fix panics on Move (rename) 22978 * Fix panic on token expiry 22979 * B2 22980 * Fix inconsistent listings and rclone check 22981 * Fix uploading empty files with go1.8 22982 * Constrain memory usage when doing multipart uploads 22983 * Fix upload url not being refreshed properly 22984 * Drive 22985 * Fix Rmdir on directories with trashed files 22986 * Fix "Ignoring unknown object" when downloading 22987 * Add --drive-list-chunk 22988 * Add --drive-skip-gdocs (Károly Oláh) 22989 * OneDrive 22990 * Implement Move 22991 * Fix Copy 22992 * Fix overwrite detection in Copy 22993 * Fix waitForJob to parse errors correctly 22994 * Use token renewer to stop auth errors on long uploads 22995 * Fix uploading empty files with go1.8 22996 * Google Cloud Storage 22997 * Fix depth 1 directory listings 22998 * Yandex 22999 * Fix single level directory listing 23000 * Dropbox 23001 * Normalise the case for single level directory listings 23002 * Fix depth 1 listing 23003 * S3 23004 * Added ca-central-1 region (Jon Yergatian) 23005 23006 ## v1.35 - 2017-01-02 23007 23008 * New Features 23009 * moveto and copyto commands for choosing a destination name on copy/move 23010 * rmdirs command to recursively delete empty directories 23011 * Allow repeated --include/--exclude/--filter options 23012 * Only show transfer stats on commands which transfer stuff 23013 * show stats on any command using the `--stats` flag 23014 * Allow overlapping directories in move when server side dir move is supported 23015 * Add --stats-unit option - thanks Scott McGillivray 23016 * Bug Fixes 23017 * Fix the config file being overwritten when two rclones are running 23018 * Make rclone lsd obey the filters properly 23019 * Fix compilation on mips 23020 * Fix not transferring files that don't differ in size 23021 * Fix panic on nil retry/fatal error 23022 * Mount 23023 * Retry reads on error - should help with reliability a lot 23024 * Report the modification times for directories from the remote 23025 * Add bandwidth accounting and limiting (fixes --bwlimit) 23026 * If --stats provided will show stats and which files are transferring 23027 * Support R/W files if truncate is set. 23028 * Implement statfs interface so df works 23029 * Note that write is now supported on Amazon Drive 23030 * Report number of blocks in a file - thanks Stefan Breunig 23031 * Crypt 23032 * Prevent the user pointing crypt at itself 23033 * Fix failed to authenticate decrypted block errors 23034 * these will now return the underlying unexpected EOF instead 23035 * Amazon Drive 23036 * Add support for server side move and directory move - thanks Stefan Breunig 23037 * Fix nil pointer deref on size attribute 23038 * B2 23039 * Use new prefix and delimiter parameters in directory listings 23040 * This makes --max-depth 1 dir listings as used in mount much faster 23041 * Reauth the account while doing uploads too - should help with token expiry 23042 * Drive 23043 * Make DirMove more efficient and complain about moving the root 23044 * Create destination directory on Move() 23045 23046 ## v1.34 - 2016-11-06 23047 23048 * New Features 23049 * Stop single file and `--files-from` operations iterating through the source bucket. 23050 * Stop removing failed upload to cloud storage remotes 23051 * Make ContentType be preserved for cloud to cloud copies 23052 * Add support to toggle bandwidth limits via SIGUSR2 - thanks Marco Paganini 23053 * `rclone check` shows count of hashes that couldn't be checked 23054 * `rclone listremotes` command 23055 * Support linux/arm64 build - thanks Fredrik Fornwall 23056 * Remove `Authorization:` lines from `--dump-headers` output 23057 * Bug Fixes 23058 * Ignore files with control characters in the names 23059 * Fix `rclone move` command 23060 * Delete src files which already existed in dst 23061 * Fix deletion of src file when dst file older 23062 * Fix `rclone check` on crypted file systems 23063 * Make failed uploads not count as "Transferred" 23064 * Make sure high level retries show with `-q` 23065 * Use a vendor directory with godep for repeatable builds 23066 * `rclone mount` - FUSE 23067 * Implement FUSE mount options 23068 * `--no-modtime`, `--debug-fuse`, `--read-only`, `--allow-non-empty`, `--allow-root`, `--allow-other` 23069 * `--default-permissions`, `--write-back-cache`, `--max-read-ahead`, `--umask`, `--uid`, `--gid` 23070 * Add `--dir-cache-time` to control caching of directory entries 23071 * Implement seek for files opened for read (useful for video players) 23072 * with `-no-seek` flag to disable 23073 * Fix crash on 32 bit ARM (alignment of 64 bit counter) 23074 * ...and many more internal fixes and improvements! 23075 * Crypt 23076 * Don't show encrypted password in configurator to stop confusion 23077 * Amazon Drive 23078 * New wait for upload option `--acd-upload-wait-per-gb` 23079 * upload timeouts scale by file size and can be disabled 23080 * Add 502 Bad Gateway to list of errors we retry 23081 * Fix overwriting a file with a zero length file 23082 * Fix ACD file size warning limit - thanks Felix Bünemann 23083 * Local 23084 * Unix: implement `-x`/`--one-file-system` to stay on a single file system 23085 * thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana 23086 * Windows: ignore the symlink bit on files 23087 * Windows: Ignore directory based junction points 23088 * B2 23089 * Make sure each upload has at least one upload slot - fixes strange upload stats 23090 * Fix uploads when using crypt 23091 * Fix download of large files (sha1 mismatch) 23092 * Return error when we try to create a bucket which someone else owns 23093 * Update B2 docs with Data usage, and Crypt section - thanks Tomasz Mazur 23094 * S3 23095 * Command line and config file support for 23096 * Setting/overriding ACL - thanks Radek Senfeld 23097 * Setting storage class - thanks Asko Tamm 23098 * Drive 23099 * Make exponential backoff work exactly as per Google specification 23100 * add `.epub`, `.odp` and `.tsv` as export formats. 23101 * Swift 23102 * Don't read metadata for directory marker objects 23103 23104 ## v1.33 - 2016-08-24 23105 23106 * New Features 23107 * Implement encryption 23108 * data encrypted in NACL secretbox format 23109 * with optional file name encryption 23110 * New commands 23111 * rclone mount - implements FUSE mounting of remotes (EXPERIMENTAL) 23112 * works on Linux, FreeBSD and OS X (need testers for the last 2!) 23113 * rclone cat - outputs remote file or files to the terminal 23114 * rclone genautocomplete - command to make a bash completion script for rclone 23115 * Editing a remote using `rclone config` now goes through the wizard 23116 * Compile with go 1.7 - this fixes rclone on macOS Sierra and on 386 processors 23117 * Use cobra for sub commands and docs generation 23118 * drive 23119 * Document how to make your own client_id 23120 * s3 23121 * User-configurable Amazon S3 ACL (thanks Radek Šenfeld) 23122 * b2 23123 * Fix stats accounting for upload - no more jumping to 100% done 23124 * On cleanup delete hide marker if it is the current file 23125 * New B2 API endpoint (thanks Per Cederberg) 23126 * Set maximum backoff to 5 Minutes 23127 * onedrive 23128 * Fix URL escaping in file names - eg uploading files with `+` in them. 23129 * amazon cloud drive 23130 * Fix token expiry during large uploads 23131 * Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors 23132 * local 23133 * Fix filenames with invalid UTF-8 not being uploaded 23134 * Fix problem with some UTF-8 characters on OS X 23135 23136 ## v1.32 - 2016-07-13 23137 23138 * Backblaze B2 23139 * Fix upload of files large files not in root 23140 23141 ## v1.31 - 2016-07-13 23142 23143 * New Features 23144 * Reduce memory on sync by about 50% 23145 * Implement --no-traverse flag to stop copy traversing the destination remote. 23146 * This can be used to reduce memory usage down to the smallest possible. 23147 * Useful to copy a small number of files into a large destination folder. 23148 * Implement cleanup command for emptying trash / removing old versions of files 23149 * Currently B2 only 23150 * Single file handling improved 23151 * Now copied with --files-from 23152 * Automatically sets --no-traverse when copying a single file 23153 * Info on using installing with ansible - thanks Stefan Weichinger 23154 * Implement --no-update-modtime flag to stop rclone fixing the remote modified times. 23155 * Bug Fixes 23156 * Fix move command - stop it running for overlapping Fses - this was causing data loss. 23157 * Local 23158 * Fix incomplete hashes - this was causing problems for B2. 23159 * Amazon Drive 23160 * Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed. 23161 * Swift 23162 * Add support for non-default project domain - thanks Antonio Messina. 23163 * S3 23164 * Add instructions on how to use rclone with minio. 23165 * Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions. 23166 * Skip setting the modified time for objects > 5GB as it isn't possible. 23167 * Backblaze B2 23168 * Add --b2-versions flag so old versions can be listed and retreived. 23169 * Treat 403 errors (eg cap exceeded) as fatal. 23170 * Implement cleanup command for deleting old file versions. 23171 * Make error handling compliant with B2 integrations notes. 23172 * Fix handling of token expiry. 23173 * Implement --b2-test-mode to set `X-Bz-Test-Mode` header. 23174 * Set cutoff for chunked upload to 200MB as per B2 guidelines. 23175 * Make upload multi-threaded. 23176 * Dropbox 23177 * Don't retry 461 errors. 23178 23179 ## v1.30 - 2016-06-18 23180 23181 * New Features 23182 * Directory listing code reworked for more features and better error reporting (thanks to Klaus Post for help). This enables 23183 * Directory include filtering for efficiency 23184 * --max-depth parameter 23185 * Better error reporting 23186 * More to come 23187 * Retry more errors 23188 * Add --ignore-size flag - for uploading images to onedrive 23189 * Log -v output to stdout by default 23190 * Display the transfer stats in more human readable form 23191 * Make 0 size files specifiable with `--max-size 0b` 23192 * Add `b` suffix so we can specify bytes in --bwlimit, --min-size etc 23193 * Use "password:" instead of "password>" prompt - thanks Klaus Post and Leigh Klotz 23194 * Bug Fixes 23195 * Fix retry doing one too many retries 23196 * Local 23197 * Fix problems with OS X and UTF-8 characters 23198 * Amazon Drive 23199 * Check a file exists before uploading to help with 408 Conflict errors 23200 * Reauth on 401 errors - this has been causing a lot of problems 23201 * Work around spurious 403 errors 23202 * Restart directory listings on error 23203 * Google Drive 23204 * Check a file exists before uploading to help with duplicates 23205 * Fix retry of multipart uploads 23206 * Backblaze B2 23207 * Implement large file uploading 23208 * S3 23209 * Add AES256 server-side encryption for - thanks Justin R. Wilson 23210 * Google Cloud Storage 23211 * Make sure we don't use conflicting content types on upload 23212 * Add service account support - thanks Michal Witkowski 23213 * Swift 23214 * Add auth version parameter 23215 * Add domain option for openstack (v3 auth) - thanks Fabian Ruff 23216 23217 ## v1.29 - 2016-04-18 23218 23219 * New Features 23220 * Implement `-I, --ignore-times` for unconditional upload 23221 * Improve `dedupe`command 23222 * Now removes identical copies without asking 23223 * Now obeys `--dry-run` 23224 * Implement `--dedupe-mode` for non interactive running 23225 * `--dedupe-mode interactive` - interactive the default. 23226 * `--dedupe-mode skip` - removes identical files then skips anything left. 23227 * `--dedupe-mode first` - removes identical files then keeps the first one. 23228 * `--dedupe-mode newest` - removes identical files then keeps the newest one. 23229 * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. 23230 * `--dedupe-mode rename` - removes identical files then renames the rest to be different. 23231 * Bug fixes 23232 * Make rclone check obey the `--size-only` flag. 23233 * Use "application/octet-stream" if discovered mime type is invalid. 23234 * Fix missing "quit" option when there are no remotes. 23235 * Google Drive 23236 * Increase default chunk size to 8 MB - increases upload speed of big files 23237 * Speed up directory listings and make more reliable 23238 * Add missing retries for Move and DirMove - increases reliability 23239 * Preserve mime type on file update 23240 * Backblaze B2 23241 * Enable mod time syncing 23242 * This means that B2 will now check modification times 23243 * It will upload new files to update the modification times 23244 * (there isn't an API to just set the mod time.) 23245 * If you want the old behaviour use `--size-only`. 23246 * Update API to new version 23247 * Fix parsing of mod time when not in metadata 23248 * Swift/Hubic 23249 * Don't return an MD5SUM for static large objects 23250 * S3 23251 * Fix uploading files bigger than 50GB 23252 23253 ## v1.28 - 2016-03-01 23254 23255 * New Features 23256 * Configuration file encryption - thanks Klaus Post 23257 * Improve `rclone config` adding more help and making it easier to understand 23258 * Implement `-u`/`--update` so creation times can be used on all remotes 23259 * Implement `--low-level-retries` flag 23260 * Optionally disable gzip compression on downloads with `--no-gzip-encoding` 23261 * Bug fixes 23262 * Don't make directories if `--dry-run` set 23263 * Fix and document the `move` command 23264 * Fix redirecting stderr on unix-like OSes when using `--log-file` 23265 * Fix `delete` command to wait until all finished - fixes missing deletes. 23266 * Backblaze B2 23267 * Use one upload URL per go routine fixes `more than one upload using auth token` 23268 * Add pacing, retries and reauthentication - fixes token expiry problems 23269 * Upload without using a temporary file from local (and remotes which support SHA1) 23270 * Fix reading metadata for all files when it shouldn't have been 23271 * Drive 23272 * Fix listing drive documents at root 23273 * Disable copy and move for Google docs 23274 * Swift 23275 * Fix uploading of chunked files with non ASCII characters 23276 * Allow setting of `storage_url` in the config - thanks Xavier Lucas 23277 * S3 23278 * Allow IAM role and credentials from environment variables - thanks Brian Stengaard 23279 * Allow low privilege users to use S3 (check if directory exists during Mkdir) - thanks Jakub Gedeon 23280 * Amazon Drive 23281 * Retry on more things to make directory listings more reliable 23282 23283 ## v1.27 - 2016-01-31 23284 23285 * New Features 23286 * Easier headless configuration with `rclone authorize` 23287 * Add support for multiple hash types - we now check SHA1 as well as MD5 hashes. 23288 * `delete` command which does obey the filters (unlike `purge`) 23289 * `dedupe` command to deduplicate a remote. Useful with Google Drive. 23290 * Add `--ignore-existing` flag to skip all files that exist on destination. 23291 * Add `--delete-before`, `--delete-during`, `--delete-after` flags. 23292 * Add `--memprofile` flag to debug memory use. 23293 * Warn the user about files with same name but different case 23294 * Make `--include` rules add their implict exclude * at the end of the filter list 23295 * Deprecate compiling with go1.3 23296 * Amazon Drive 23297 * Fix download of files > 10 GB 23298 * Fix directory traversal ("Next token is expired") for large directory listings 23299 * Remove 409 conflict from error codes we will retry - stops very long pauses 23300 * Backblaze B2 23301 * SHA1 hashes now checked by rclone core 23302 * Drive 23303 * Add `--drive-auth-owner-only` to only consider files owned by the user - thanks Björn Harrtell 23304 * Export Google documents 23305 * Dropbox 23306 * Make file exclusion error controllable with -q 23307 * Swift 23308 * Fix upload from unprivileged user. 23309 * S3 23310 * Fix updating of mod times of files with `+` in. 23311 * Local 23312 * Add local file system option to disable UNC on Windows. 23313 23314 ## v1.26 - 2016-01-02 23315 23316 * New Features 23317 * Yandex storage backend - thank you Dmitry Burdeev ("dibu") 23318 * Implement Backblaze B2 storage backend 23319 * Add --min-age and --max-age flags - thank you Adriano Aurélio Meirelles 23320 * Make ls/lsl/md5sum/size/check obey includes and excludes 23321 * Fixes 23322 * Fix crash in http logging 23323 * Upload releases to github too 23324 * Swift 23325 * Fix sync for chunked files 23326 * OneDrive 23327 * Re-enable server side copy 23328 * Don't mask HTTP error codes with JSON decode error 23329 * S3 23330 * Fix corrupting Content-Type on mod time update (thanks Joseph Spurrier) 23331 23332 ## v1.25 - 2015-11-14 23333 23334 * New features 23335 * Implement Hubic storage system 23336 * Fixes 23337 * Fix deletion of some excluded files without --delete-excluded 23338 * This could have deleted files unexpectedly on sync 23339 * Always check first with `--dry-run`! 23340 * Swift 23341 * Stop SetModTime losing metadata (eg X-Object-Manifest) 23342 * This could have caused data loss for files > 5GB in size 23343 * Use ContentType from Object to avoid lookups in listings 23344 * OneDrive 23345 * disable server side copy as it seems to be broken at Microsoft 23346 23347 ## v1.24 - 2015-11-07 23348 23349 * New features 23350 * Add support for Microsoft OneDrive 23351 * Add `--no-check-certificate` option to disable server certificate verification 23352 * Add async readahead buffer for faster transfer of big files 23353 * Fixes 23354 * Allow spaces in remotes and check remote names for validity at creation time 23355 * Allow '&' and disallow ':' in Windows filenames. 23356 * Swift 23357 * Ignore directory marker objects where appropriate - allows working with Hubic 23358 * Don't delete the container if fs wasn't at root 23359 * S3 23360 * Don't delete the bucket if fs wasn't at root 23361 * Google Cloud Storage 23362 * Don't delete the bucket if fs wasn't at root 23363 23364 ## v1.23 - 2015-10-03 23365 23366 * New features 23367 * Implement `rclone size` for measuring remotes 23368 * Fixes 23369 * Fix headless config for drive and gcs 23370 * Tell the user they should try again if the webserver method failed 23371 * Improve output of `--dump-headers` 23372 * S3 23373 * Allow anonymous access to public buckets 23374 * Swift 23375 * Stop chunked operations logging "Failed to read info: Object Not Found" 23376 * Use Content-Length on uploads for extra reliability 23377 23378 ## v1.22 - 2015-09-28 23379 23380 * Implement rsync like include and exclude flags 23381 * swift 23382 * Support files > 5GB - thanks Sergey Tolmachev 23383 23384 ## v1.21 - 2015-09-22 23385 23386 * New features 23387 * Display individual transfer progress 23388 * Make lsl output times in localtime 23389 * Fixes 23390 * Fix allowing user to override credentials again in Drive, GCS and ACD 23391 * Amazon Drive 23392 * Implement compliant pacing scheme 23393 * Google Drive 23394 * Make directory reads concurrent for increased speed. 23395 23396 ## v1.20 - 2015-09-15 23397 23398 * New features 23399 * Amazon Drive support 23400 * Oauth support redone - fix many bugs and improve usability 23401 * Use "golang.org/x/oauth2" as oauth libary of choice 23402 * Improve oauth usability for smoother initial signup 23403 * drive, googlecloudstorage: optionally use auto config for the oauth token 23404 * Implement --dump-headers and --dump-bodies debug flags 23405 * Show multiple matched commands if abbreviation too short 23406 * Implement server side move where possible 23407 * local 23408 * Always use UNC paths internally on Windows - fixes a lot of bugs 23409 * dropbox 23410 * force use of our custom transport which makes timeouts work 23411 * Thanks to Klaus Post for lots of help with this release 23412 23413 ## v1.19 - 2015-08-28 23414 23415 * New features 23416 * Server side copies for s3/swift/drive/dropbox/gcs 23417 * Move command - uses server side copies if it can 23418 * Implement --retries flag - tries 3 times by default 23419 * Build for plan9/amd64 and solaris/amd64 too 23420 * Fixes 23421 * Make a current version download with a fixed URL for scripting 23422 * Ignore rmdir in limited fs rather than throwing error 23423 * dropbox 23424 * Increase chunk size to improve upload speeds massively 23425 * Issue an error message when trying to upload bad file name 23426 23427 ## v1.18 - 2015-08-17 23428 23429 * drive 23430 * Add `--drive-use-trash` flag so rclone trashes instead of deletes 23431 * Add "Forbidden to download" message for files with no downloadURL 23432 * dropbox 23433 * Remove datastore 23434 * This was deprecated and it caused a lot of problems 23435 * Modification times and MD5SUMs no longer stored 23436 * Fix uploading files > 2GB 23437 * s3 23438 * use official AWS SDK from github.com/aws/aws-sdk-go 23439 * **NB** will most likely require you to delete and recreate remote 23440 * enable multipart upload which enables files > 5GB 23441 * tested with Ceph / RadosGW / S3 emulation 23442 * many thanks to Sam Liston and Brian Haymore at the [Utah Center for High Performance Computing](https://www.chpc.utah.edu/) for a Ceph test account 23443 * misc 23444 * Show errors when reading the config file 23445 * Do not print stats in quiet mode - thanks Leonid Shalupov 23446 * Add FAQ 23447 * Fix created directories not obeying umask 23448 * Linux installation instructions - thanks Shimon Doodkin 23449 23450 ## v1.17 - 2015-06-14 23451 23452 * dropbox: fix case insensitivity issues - thanks Leonid Shalupov 23453 23454 ## v1.16 - 2015-06-09 23455 23456 * Fix uploading big files which was causing timeouts or panics 23457 * Don't check md5sum after download with --size-only 23458 23459 ## v1.15 - 2015-06-06 23460 23461 * Add --checksum flag to only discard transfers by MD5SUM - thanks Alex Couper 23462 * Implement --size-only flag to sync on size not checksum & modtime 23463 * Expand docs and remove duplicated information 23464 * Document rclone's limitations with directories 23465 * dropbox: update docs about case insensitivity 23466 23467 ## v1.14 - 2015-05-21 23468 23469 * local: fix encoding of non utf-8 file names - fixes a duplicate file problem 23470 * drive: docs about rate limiting 23471 * google cloud storage: Fix compile after API change in "google.golang.org/api/storage/v1" 23472 23473 ## v1.13 - 2015-05-10 23474 23475 * Revise documentation (especially sync) 23476 * Implement --timeout and --conntimeout 23477 * s3: ignore etags from multipart uploads which aren't md5sums 23478 23479 ## v1.12 - 2015-03-15 23480 23481 * drive: Use chunked upload for files above a certain size 23482 * drive: add --drive-chunk-size and --drive-upload-cutoff parameters 23483 * drive: switch to insert from update when a failed copy deletes the upload 23484 * core: Log duplicate files if they are detected 23485 23486 ## v1.11 - 2015-03-04 23487 23488 * swift: add region parameter 23489 * drive: fix crash on failed to update remote mtime 23490 * In remote paths, change native directory separators to / 23491 * Add synchronization to ls/lsl/lsd output to stop corruptions 23492 * Ensure all stats/log messages to go stderr 23493 * Add --log-file flag to log everything (including panics) to file 23494 * Make it possible to disable stats printing with --stats=0 23495 * Implement --bwlimit to limit data transfer bandwidth 23496 23497 ## v1.10 - 2015-02-12 23498 23499 * s3: list an unlimited number of items 23500 * Fix getting stuck in the configurator 23501 23502 ## v1.09 - 2015-02-07 23503 23504 * windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:) 23505 * local: Fix directory separators on Windows 23506 * drive: fix rate limit exceeded errors 23507 23508 ## v1.08 - 2015-02-04 23509 23510 * drive: fix subdirectory listing to not list entire drive 23511 * drive: Fix SetModTime 23512 * dropbox: adapt code to recent library changes 23513 23514 ## v1.07 - 2014-12-23 23515 23516 * google cloud storage: fix memory leak 23517 23518 ## v1.06 - 2014-12-12 23519 23520 * Fix "Couldn't find home directory" on OSX 23521 * swift: Add tenant parameter 23522 * Use new location of Google API packages 23523 23524 ## v1.05 - 2014-08-09 23525 23526 * Improved tests and consequently lots of minor fixes 23527 * core: Fix race detected by go race detector 23528 * core: Fixes after running errcheck 23529 * drive: reset root directory on Rmdir and Purge 23530 * fs: Document that Purger returns error on empty directory, test and fix 23531 * google cloud storage: fix ListDir on subdirectory 23532 * google cloud storage: re-read metadata in SetModTime 23533 * s3: make reading metadata more reliable to work around eventual consistency problems 23534 * s3: strip trailing / from ListDir() 23535 * swift: return directories without / in ListDir 23536 23537 ## v1.04 - 2014-07-21 23538 23539 * google cloud storage: Fix crash on Update 23540 23541 ## v1.03 - 2014-07-20 23542 23543 * swift, s3, dropbox: fix updated files being marked as corrupted 23544 * Make compile with go 1.1 again 23545 23546 ## v1.02 - 2014-07-19 23547 23548 * Implement Dropbox remote 23549 * Implement Google Cloud Storage remote 23550 * Verify Md5sums and Sizes after copies 23551 * Remove times from "ls" command - lists sizes only 23552 * Add add "lsl" - lists times and sizes 23553 * Add "md5sum" command 23554 23555 ## v1.01 - 2014-07-04 23556 23557 * drive: fix transfer of big files using up lots of memory 23558 23559 ## v1.00 - 2014-07-03 23560 23561 * drive: fix whole second dates 23562 23563 ## v0.99 - 2014-06-26 23564 23565 * Fix --dry-run not working 23566 * Make compatible with go 1.1 23567 23568 ## v0.98 - 2014-05-30 23569 23570 * s3: Treat missing Content-Length as 0 for some ceph installations 23571 * rclonetest: add file with a space in 23572 23573 ## v0.97 - 2014-05-05 23574 23575 * Implement copying of single files 23576 * s3 & swift: support paths inside containers/buckets 23577 23578 ## v0.96 - 2014-04-24 23579 23580 * drive: Fix multiple files of same name being created 23581 * drive: Use o.Update and fs.Put to optimise transfers 23582 * Add version number, -V and --version 23583 23584 ## v0.95 - 2014-03-28 23585 23586 * rclone.org: website, docs and graphics 23587 * drive: fix path parsing 23588 23589 ## v0.94 - 2014-03-27 23590 23591 * Change remote format one last time 23592 * GNU style flags 23593 23594 ## v0.93 - 2014-03-16 23595 23596 * drive: store token in config file 23597 * cross compile other versions 23598 * set strict permissions on config file 23599 23600 ## v0.92 - 2014-03-15 23601 23602 * Config fixes and --config option 23603 23604 ## v0.91 - 2014-03-15 23605 23606 * Make config file 23607 23608 ## v0.90 - 2013-06-27 23609 23610 * Project named rclone 23611 23612 ## v0.00 - 2012-11-18 23613 23614 * Project started 23615 23616 # Bugs and Limitations 23617 23618 ## Limitations 23619 23620 ### Directory timestamps aren't preserved 23621 23622 Rclone doesn't currently preserve the timestamps of directories. This 23623 is because rclone only really considers objects when syncing. 23624 23625 ### Rclone struggles with millions of files in a directory 23626 23627 Currently rclone loads each directory entirely into memory before 23628 using it. Since each Rclone object takes 0.5k-1k of memory this can 23629 take a very long time and use an extremely large amount of memory. 23630 23631 Millions of files in a directory tend caused by software writing cloud 23632 storage (eg S3 buckets). 23633 23634 ### Bucket based remotes and folders 23635 23636 Bucket based remotes (eg S3/GCS/Swift/B2) do not have a concept of 23637 directories. Rclone therefore cannot create directories in them which 23638 means that empty directories on a bucket based remote will tend to 23639 disappear. 23640 23641 Some software creates empty keys ending in `/` as directory markers. 23642 Rclone doesn't do this as it potentially creates more objects and 23643 costs more. It may do in future (probably with a flag). 23644 23645 ## Bugs 23646 23647 Bugs are stored in rclone's GitHub project: 23648 23649 * [Reported bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug) 23650 * [Known issues](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Known+Problem%22) 23651 23652 Frequently Asked Questions 23653 -------------------------- 23654 23655 ### Do all cloud storage systems support all rclone commands ### 23656 23657 Yes they do. All the rclone commands (eg `sync`, `copy` etc) will 23658 work on all the remote storage systems. 23659 23660 ### Can I copy the config from one machine to another ### 23661 23662 Sure! Rclone stores all of its config in a single file. If you want 23663 to find this file, run `rclone config file` which will tell you where 23664 it is. 23665 23666 See the [remote setup docs](https://rclone.org/remote_setup/) for more info. 23667 23668 ### How do I configure rclone on a remote / headless box with no browser? ### 23669 23670 This has now been documented in its own [remote setup page](https://rclone.org/remote_setup/). 23671 23672 ### Can rclone sync directly from drive to s3 ### 23673 23674 Rclone can sync between two remote cloud storage systems just fine. 23675 23676 Note that it effectively downloads the file and uploads it again, so 23677 the node running rclone would need to have lots of bandwidth. 23678 23679 The syncs would be incremental (on a file by file basis). 23680 23681 Eg 23682 23683 rclone sync drive:Folder s3:bucket 23684 23685 23686 ### Using rclone from multiple locations at the same time ### 23687 23688 You can use rclone from multiple places at the same time if you choose 23689 different subdirectory for the output, eg 23690 23691 ``` 23692 Server A> rclone sync /tmp/whatever remote:ServerA 23693 Server B> rclone sync /tmp/whatever remote:ServerB 23694 ``` 23695 23696 If you sync to the same directory then you should use rclone copy 23697 otherwise the two rclones may delete each others files, eg 23698 23699 ``` 23700 Server A> rclone copy /tmp/whatever remote:Backup 23701 Server B> rclone copy /tmp/whatever remote:Backup 23702 ``` 23703 23704 The file names you upload from Server A and Server B should be 23705 different in this case, otherwise some file systems (eg Drive) may 23706 make duplicates. 23707 23708 ### Why doesn't rclone support partial transfers / binary diffs like rsync? ### 23709 23710 Rclone stores each file you transfer as a native object on the remote 23711 cloud storage system. This means that you can see the files you 23712 upload as expected using alternative access methods (eg using the 23713 Google Drive web interface). There is a 1:1 mapping between files on 23714 your hard disk and objects created in the cloud storage system. 23715 23716 Cloud storage systems (at least none I've come across yet) don't 23717 support partially uploading an object. You can't take an existing 23718 object, and change some bytes in the middle of it. 23719 23720 It would be possible to make a sync system which stored binary diffs 23721 instead of whole objects like rclone does, but that would break the 23722 1:1 mapping of files on your hard disk to objects in the remote cloud 23723 storage system. 23724 23725 All the cloud storage systems support partial downloads of content, so 23726 it would be possible to make partial downloads work. However to make 23727 this work efficiently this would require storing a significant amount 23728 of metadata, which breaks the desired 1:1 mapping of files to objects. 23729 23730 ### Can rclone do bi-directional sync? ### 23731 23732 No, not at present. rclone only does uni-directional sync from A -> 23733 B. It may do in the future though since it has all the primitives - it 23734 just requires writing the algorithm to do it. 23735 23736 ### Can I use rclone with an HTTP proxy? ### 23737 23738 Yes. rclone will follow the standard environment variables for 23739 proxies, similar to cURL and other programs. 23740 23741 In general the variables are called `http_proxy` (for services reached 23742 over `http`) and `https_proxy` (for services reached over `https`). Most 23743 public services will be using `https`, but you may wish to set both. 23744 23745 The content of the variable is `protocol://server:port`. The protocol 23746 value is the one used to talk to the proxy server, itself, and is commonly 23747 either `http` or `socks5`. 23748 23749 Slightly annoyingly, there is no _standard_ for the name; some applications 23750 may use `http_proxy` but another one `HTTP_PROXY`. The `Go` libraries 23751 used by `rclone` will try both variations, but you may wish to set all 23752 possibilities. So, on Linux, you may end up with code similar to 23753 23754 export http_proxy=http://proxyserver:12345 23755 export https_proxy=$http_proxy 23756 export HTTP_PROXY=$http_proxy 23757 export HTTPS_PROXY=$http_proxy 23758 23759 The `NO_PROXY` allows you to disable the proxy for specific hosts. 23760 Hosts must be comma separated, and can contain domains or parts. 23761 For instance "foo.com" also matches "bar.foo.com". 23762 23763 e.g. 23764 23765 export no_proxy=localhost,127.0.0.0/8,my.host.name 23766 export NO_PROXY=$no_proxy 23767 23768 Note that the ftp backend does not support `ftp_proxy` yet. 23769 23770 ### Rclone gives x509: failed to load system roots and no roots provided error ### 23771 23772 This means that `rclone` can't file the SSL root certificates. Likely 23773 you are running `rclone` on a NAS with a cut-down Linux OS, or 23774 possibly on Solaris. 23775 23776 Rclone (via the Go runtime) tries to load the root certificates from 23777 these places on Linux. 23778 23779 "/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc. 23780 "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 23781 "/etc/ssl/ca-bundle.pem", // OpenSUSE 23782 "/etc/pki/tls/cacert.pem", // OpenELEC 23783 23784 So doing something like this should fix the problem. It also sets the 23785 time which is important for SSL to work properly. 23786 23787 ``` 23788 mkdir -p /etc/ssl/certs/ 23789 curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 23790 ntpclient -s -h pool.ntp.org 23791 ``` 23792 23793 The two environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR`, mentioned in the [x509 package](https://godoc.org/crypto/x509), 23794 provide an additional way to provide the SSL root certificates. 23795 23796 Note that you may need to add the `--insecure` option to the `curl` command line if it doesn't work without. 23797 23798 ``` 23799 curl --insecure -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 23800 ``` 23801 23802 ### Rclone gives Failed to load config file: function not implemented error ### 23803 23804 Likely this means that you are running rclone on Linux version not 23805 supported by the go runtime, ie earlier than version 2.6.23. 23806 23807 See the [system requirements section in the go install 23808 docs](https://golang.org/doc/install) for full details. 23809 23810 ### All my uploaded docx/xlsx/pptx files appear as archive/zip ### 23811 23812 This is caused by uploading these files from a Windows computer which 23813 hasn't got the Microsoft Office suite installed. The easiest way to 23814 fix is to install the Word viewer and the Microsoft Office 23815 Compatibility Pack for Word, Excel, and PowerPoint 2007 and later 23816 versions' file formats 23817 23818 ### tcp lookup some.domain.com no such host ### 23819 23820 This happens when rclone cannot resolve a domain. Please check that 23821 your DNS setup is generally working, e.g. 23822 23823 ``` 23824 # both should print a long list of possible IP addresses 23825 dig www.googleapis.com # resolve using your default DNS 23826 dig www.googleapis.com @8.8.8.8 # resolve with Google's DNS server 23827 ``` 23828 23829 If you are using `systemd-resolved` (default on Arch Linux), ensure it 23830 is at version 233 or higher. Previous releases contain a bug which 23831 causes not all domains to be resolved properly. 23832 23833 Additionally with the `GODEBUG=netdns=` environment variable the Go 23834 resolver decision can be influenced. This also allows to resolve certain 23835 issues with DNS resolution. See the [name resolution section in the go docs](https://golang.org/pkg/net/#hdr-Name_Resolution). 23836 23837 ### The total size reported in the stats for a sync is wrong and keeps changing 23838 23839 It is likely you have more than 10,000 files that need to be 23840 synced. By default rclone only gets 10,000 files ahead in a sync so as 23841 not to use up too much memory. You can change this default with the 23842 [--max-backlog](/docs/#max-backlog-n) flag. 23843 23844 ### Rclone is using too much memory or appears to have a memory leak 23845 23846 Rclone is written in Go which uses a garbage collector. The default 23847 settings for the garbage collector mean that it runs when the heap 23848 size has doubled. 23849 23850 However it is possible to tune the garbage collector to use less 23851 memory by [setting GOGC](https://dave.cheney.net/tag/gogc) to a lower 23852 value, say `export GOGC=20`. This will make the garbage collector 23853 work harder, reducing memory size at the expense of CPU usage. 23854 23855 The most common cause of rclone using lots of memory is a single 23856 directory with thousands or millions of files in. Rclone has to load 23857 this entirely into memory as rclone objects. Each rclone object takes 23858 0.5k-1k of memory. 23859 23860 License 23861 ------- 23862 23863 This is free software under the terms of MIT the license (check the 23864 COPYING file included with the source code). 23865 23866 ``` 23867 Copyright (C) 2019 by Nick Craig-Wood https://www.craig-wood.com/nick/ 23868 23869 Permission is hereby granted, free of charge, to any person obtaining a copy 23870 of this software and associated documentation files (the "Software"), to deal 23871 in the Software without restriction, including without limitation the rights 23872 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 23873 copies of the Software, and to permit persons to whom the Software is 23874 furnished to do so, subject to the following conditions: 23875 23876 The above copyright notice and this permission notice shall be included in 23877 all copies or substantial portions of the Software. 23878 23879 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 23880 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 23881 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 23882 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23883 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 23884 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 23885 THE SOFTWARE. 23886 ``` 23887 23888 Authors 23889 ------- 23890 23891 * Nick Craig-Wood <nick@craig-wood.com> 23892 23893 Contributors 23894 ------------ 23895 23896 * Alex Couper <amcouper@gmail.com> 23897 * Leonid Shalupov <leonid@shalupov.com> <shalupov@diverse.org.ru> 23898 * Shimon Doodkin <helpmepro1@gmail.com> 23899 * Colin Nicholson <colin@colinn.com> 23900 * Klaus Post <klauspost@gmail.com> 23901 * Sergey Tolmachev <tolsi.ru@gmail.com> 23902 * Adriano Aurélio Meirelles <adriano@atinge.com> 23903 * C. Bess <cbess@users.noreply.github.com> 23904 * Dmitry Burdeev <dibu28@gmail.com> 23905 * Joseph Spurrier <github@josephspurrier.com> 23906 * Björn Harrtell <bjorn@wololo.org> 23907 * Xavier Lucas <xavier.lucas@corp.ovh.com> 23908 * Werner Beroux <werner@beroux.com> 23909 * Brian Stengaard <brian@stengaard.eu> 23910 * Jakub Gedeon <jgedeon@sofi.com> 23911 * Jim Tittsler <jwt@onjapan.net> 23912 * Michal Witkowski <michal@improbable.io> 23913 * Fabian Ruff <fabian.ruff@sap.com> 23914 * Leigh Klotz <klotz@quixey.com> 23915 * Romain Lapray <lapray.romain@gmail.com> 23916 * Justin R. Wilson <jrw972@gmail.com> 23917 * Antonio Messina <antonio.s.messina@gmail.com> 23918 * Stefan G. Weichinger <office@oops.co.at> 23919 * Per Cederberg <cederberg@gmail.com> 23920 * Radek Šenfeld <rush@logic.cz> 23921 * Fredrik Fornwall <fredrik@fornwall.net> 23922 * Asko Tamm <asko@deekit.net> 23923 * xor-zz <xor@gstocco.com> 23924 * Tomasz Mazur <tmazur90@gmail.com> 23925 * Marco Paganini <paganini@paganini.net> 23926 * Felix Bünemann <buenemann@louis.info> 23927 * Durval Menezes <jmrclone@durval.com> 23928 * Luiz Carlos Rumbelsperger Viana <maxd13_luiz_carlos@hotmail.com> 23929 * Stefan Breunig <stefan-github@yrden.de> 23930 * Alishan Ladhani <ali-l@users.noreply.github.com> 23931 * 0xJAKE <0xJAKE@users.noreply.github.com> 23932 * Thibault Molleman <thibaultmol@users.noreply.github.com> 23933 * Scott McGillivray <scott.mcgillivray@gmail.com> 23934 * Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com> 23935 * Lukas Loesche <lukas@mesosphere.io> 23936 * emyarod <allllaboutyou@gmail.com> 23937 * T.C. Ferguson <tcf909@gmail.com> 23938 * Brandur <brandur@mutelight.org> 23939 * Dario Giovannetti <dev@dariogiovannetti.net> 23940 * Károly Oláh <okaresz@aol.com> 23941 * Jon Yergatian <jon@macfanatic.ca> 23942 * Jack Schmidt <github@mowsey.org> 23943 * Dedsec1 <Dedsec1@users.noreply.github.com> 23944 * Hisham Zarka <hzarka@gmail.com> 23945 * Jérôme Vizcaino <jerome.vizcaino@gmail.com> 23946 * Mike Tesch <mjt6129@rit.edu> 23947 * Marvin Watson <marvwatson@users.noreply.github.com> 23948 * Danny Tsai <danny8376@gmail.com> 23949 * Yoni Jah <yonjah+git@gmail.com> <yonjah+github@gmail.com> 23950 * Stephen Harris <github@spuddy.org> <sweharris@users.noreply.github.com> 23951 * Ihor Dvoretskyi <ihor.dvoretskyi@gmail.com> 23952 * Jon Craton <jncraton@gmail.com> 23953 * Hraban Luyat <hraban@0brg.net> 23954 * Michael Ledin <mledin89@gmail.com> 23955 * Martin Kristensen <me@azgul.com> 23956 * Too Much IO <toomuchio@users.noreply.github.com> 23957 * Anisse Astier <anisse@astier.eu> 23958 * Zahiar Ahmed <zahiar@live.com> 23959 * Igor Kharin <igorkharin@gmail.com> 23960 * Bill Zissimopoulos <billziss@navimatics.com> 23961 * Bob Potter <bobby.potter@gmail.com> 23962 * Steven Lu <tacticalazn@gmail.com> 23963 * Sjur Fredriksen <sjurtf@ifi.uio.no> 23964 * Ruwbin <hubus12345@gmail.com> 23965 * Fabian Möller <fabianm88@gmail.com> <f.moeller@nynex.de> 23966 * Edward Q. Bridges <github@eqbridges.com> 23967 * Vasiliy Tolstov <v.tolstov@selfip.ru> 23968 * Harshavardhana <harsha@minio.io> 23969 * sainaen <sainaen@gmail.com> 23970 * gdm85 <gdm85@users.noreply.github.com> 23971 * Yaroslav Halchenko <debian@onerussian.com> 23972 * John Papandriopoulos <jpap@users.noreply.github.com> 23973 * Zhiming Wang <zmwangx@gmail.com> 23974 * Andy Pilate <cubox@cubox.me> 23975 * Oliver Heyme <olihey@googlemail.com> <olihey@users.noreply.github.com> <de8olihe@lego.com> 23976 * wuyu <wuyu@yunify.com> 23977 * Andrei Dragomir <adragomi@adobe.com> 23978 * Christian Brüggemann <mail@cbruegg.com> 23979 * Alex McGrath Kraak <amkdude@gmail.com> 23980 * bpicode <bjoern.pirnay@googlemail.com> 23981 * Daniel Jagszent <daniel@jagszent.de> 23982 * Josiah White <thegenius2009@gmail.com> 23983 * Ishuah Kariuki <kariuki@ishuah.com> <ishuah91@gmail.com> 23984 * Jan Varho <jan@varho.org> 23985 * Girish Ramakrishnan <girish@cloudron.io> 23986 * LingMan <LingMan@users.noreply.github.com> 23987 * Jacob McNamee <jacobmcnamee@gmail.com> 23988 * jersou <jertux@gmail.com> 23989 * thierry <thierry@substantiel.fr> 23990 * Simon Leinen <simon.leinen@gmail.com> <ubuntu@s3-test.novalocal> 23991 * Dan Dascalescu <ddascalescu+github@gmail.com> 23992 * Jason Rose <jason@jro.io> 23993 * Andrew Starr-Bochicchio <a.starr.b@gmail.com> 23994 * John Leach <john@johnleach.co.uk> 23995 * Corban Raun <craun@instructure.com> 23996 * Pierre Carlson <mpcarl@us.ibm.com> 23997 * Ernest Borowski <er.borowski@gmail.com> 23998 * Remus Bunduc <remus.bunduc@gmail.com> 23999 * Iakov Davydov <iakov.davydov@unil.ch> <dav05.gith@myths.ru> 24000 * Jakub Tasiemski <tasiemski@gmail.com> 24001 * David Minor <dminor@saymedia.com> 24002 * Tim Cooijmans <cooijmans.tim@gmail.com> 24003 * Laurence <liuxy6@gmail.com> 24004 * Giovanni Pizzi <gio.piz@gmail.com> 24005 * Filip Bartodziej <filipbartodziej@gmail.com> 24006 * Jon Fautley <jon@dead.li> 24007 * lewapm <32110057+lewapm@users.noreply.github.com> 24008 * Yassine Imounachen <yassine256@gmail.com> 24009 * Chris Redekop <chris-redekop@users.noreply.github.com> <chris.redekop@gmail.com> 24010 * Jon Fautley <jon@adenoid.appstal.co.uk> 24011 * Will Gunn <WillGunn@users.noreply.github.com> 24012 * Lucas Bremgartner <lucas@bremis.ch> 24013 * Jody Frankowski <jody.frankowski@gmail.com> 24014 * Andreas Roussos <arouss1980@gmail.com> 24015 * nbuchanan <nbuchanan@utah.gov> 24016 * Durval Menezes <rclone@durval.com> 24017 * Victor <vb-github@viblo.se> 24018 * Mateusz <pabian.mateusz@gmail.com> 24019 * Daniel Loader <spicypixel@gmail.com> 24020 * David0rk <davidork@gmail.com> 24021 * Alexander Neumann <alexander@bumpern.de> 24022 * Giri Badanahatti <gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local> 24023 * Leo R. Lundgren <leo@finalresort.org> 24024 * wolfv <wolfv6@users.noreply.github.com> 24025 * Dave Pedu <dave@davepedu.com> 24026 * Stefan Lindblom <lindblom@spotify.com> 24027 * seuffert <oliver@seuffert.biz> 24028 * gbadanahatti <37121690+gbadanahatti@users.noreply.github.com> 24029 * Keith Goldfarb <barkofdelight@gmail.com> 24030 * Steve Kriss <steve@heptio.com> 24031 * Chih-Hsuan Yen <yan12125@gmail.com> 24032 * Alexander Neumann <fd0@users.noreply.github.com> 24033 * Matt Holt <mholt@users.noreply.github.com> 24034 * Eri Bastos <bastos.eri@gmail.com> 24035 * Michael P. Dubner <pywebmail@list.ru> 24036 * Antoine GIRARD <sapk@users.noreply.github.com> 24037 * Mateusz Piotrowski <mpp302@gmail.com> 24038 * Animosity022 <animosity22@users.noreply.github.com> <earl.texter@gmail.com> 24039 * Peter Baumgartner <pete@lincolnloop.com> 24040 * Craig Rachel <craig@craigrachel.com> 24041 * Michael G. Noll <miguno@users.noreply.github.com> 24042 * hensur <me@hensur.de> 24043 * Oliver Heyme <de8olihe@lego.com> 24044 * Richard Yang <richard@yenforyang.com> 24045 * Piotr Oleszczyk <piotr.oleszczyk@gmail.com> 24046 * Rodrigo <rodarima@gmail.com> 24047 * NoLooseEnds <NoLooseEnds@users.noreply.github.com> 24048 * Jakub Karlicek <jakub@karlicek.me> 24049 * John Clayton <john@codemonkeylabs.com> 24050 * Kasper Byrdal Nielsen <byrdal76@gmail.com> 24051 * Benjamin Joseph Dag <bjdag1234@users.noreply.github.com> 24052 * themylogin <themylogin@gmail.com> 24053 * Onno Zweers <onno.zweers@surfsara.nl> 24054 * Jasper Lievisse Adriaanse <jasper@humppa.nl> 24055 * sandeepkru <sandeep.ummadi@gmail.com> <sandeepkru@users.noreply.github.com> 24056 * HerrH <atomtigerzoo@users.noreply.github.com> 24057 * Andrew <4030760+sparkyman215@users.noreply.github.com> 24058 * dan smith <XX1011@gmail.com> 24059 * Oleg Kovalov <iamolegkovalov@gmail.com> 24060 * Ruben Vandamme <github-com-00ff86@vandamme.email> 24061 * Cnly <minecnly@gmail.com> 24062 * Andres Alvarez <1671935+kir4h@users.noreply.github.com> 24063 * reddi1 <xreddi@gmail.com> 24064 * Matt Tucker <matthewtckr@gmail.com> 24065 * Sebastian Bünger <buengese@gmail.com> 24066 * Martin Polden <mpolden@mpolden.no> 24067 * Alex Chen <Cnly@users.noreply.github.com> 24068 * Denis <deniskovpen@gmail.com> 24069 * bsteiss <35940619+bsteiss@users.noreply.github.com> 24070 * Cédric Connes <cedric.connes@gmail.com> 24071 * Dr. Tobias Quathamer <toddy15@users.noreply.github.com> 24072 * dcpu <42736967+dcpu@users.noreply.github.com> 24073 * Sheldon Rupp <me@shel.io> 24074 * albertony <12441419+albertony@users.noreply.github.com> 24075 * cron410 <cron410@gmail.com> 24076 * Anagh Kumar Baranwal <anaghk.dos@gmail.com> <6824881+darthShadow@users.noreply.github.com> 24077 * Felix Brucker <felix@felixbrucker.com> 24078 * Santiago Rodríguez <scollazo@users.noreply.github.com> 24079 * Craig Miskell <craig.miskell@fluxfederation.com> 24080 * Antoine GIRARD <sapk@sapk.fr> 24081 * Joanna Marek <joanna.marek@u2i.com> 24082 * frenos <frenos@users.noreply.github.com> 24083 * ssaqua <ssaqua@users.noreply.github.com> 24084 * xnaas <me@xnaas.info> 24085 * Frantisek Fuka <fuka@fuxoft.cz> 24086 * Paul Kohout <pauljkohout@yahoo.com> 24087 * dcpu <43330287+dcpu@users.noreply.github.com> 24088 * jackyzy823 <jackyzy823@gmail.com> 24089 * David Haguenauer <ml@kurokatta.org> 24090 * teresy <hi.teresy@gmail.com> 24091 * buergi <patbuergi@gmx.de> 24092 * Florian Gamboeck <mail@floga.de> 24093 * Ralf Hemberger <10364191+rhemberger@users.noreply.github.com> 24094 * Scott Edlund <sedlund@users.noreply.github.com> 24095 * Erik Swanson <erik@retailnext.net> 24096 * Jake Coggiano <jake@stripe.com> 24097 * brused27 <brused27@noemailaddress> 24098 * Peter Kaminski <kaminski@istori.com> 24099 * Henry Ptasinski <henry@logout.com> 24100 * Alexander <kharkovalexander@gmail.com> 24101 * Garry McNulty <garrmcnu@gmail.com> 24102 * Mathieu Carbou <mathieu.carbou@gmail.com> 24103 * Mark Otway <mark@otway.com> 24104 * William Cocker <37018962+WilliamCocker@users.noreply.github.com> 24105 * François Leurent <131.js@cloudyks.org> 24106 * Arkadius Stefanski <arkste@gmail.com> 24107 * Jay <dev@jaygoel.com> 24108 * andrea rota <a@xelera.eu> 24109 * nicolov <nicolov@users.noreply.github.com> 24110 * Dario Guzik <dario@guzik.com.ar> 24111 * qip <qip@users.noreply.github.com> 24112 * yair@unicorn <yair@unicorn> 24113 * Matt Robinson <brimstone@the.narro.ws> 24114 * kayrus <kay.diam@gmail.com> 24115 * Rémy Léone <remy.leone@gmail.com> 24116 * Wojciech Smigielski <wojciech.hieronim.smigielski@gmail.com> 24117 * weetmuts <oehrstroem@gmail.com> 24118 * Jonathan <vanillajonathan@users.noreply.github.com> 24119 * James Carpenter <orbsmiv@users.noreply.github.com> 24120 * Vince <vince0villamora@gmail.com> 24121 * Nestar47 <47841759+Nestar47@users.noreply.github.com> 24122 * Six <brbsix@gmail.com> 24123 * Alexandru Bumbacea <alexandru.bumbacea@booking.com> 24124 * calisro <robert.calistri@gmail.com> 24125 * Dr.Rx <david.rey@nventive.com> 24126 * marcintustin <marcintustin@users.noreply.github.com> 24127 * jaKa Močnik <jaka@koofr.net> 24128 * Fionera <fionera@fionera.de> 24129 * Dan Walters <dan@walters.io> 24130 * Danil Semelenov <sgtpep@users.noreply.github.com> 24131 * xopez <28950736+xopez@users.noreply.github.com> 24132 * Ben Boeckel <mathstuf@gmail.com> 24133 * Manu <manu@snapdragon.cc> 24134 * Kyle E. Mitchell <kyle@kemitchell.com> 24135 * Gary Kim <gary@garykim.dev> 24136 * Jon <jonathn@github.com> 24137 * Jeff Quinn <jeffrey.quinn@bluevoyant.com> 24138 * Peter Berbec <peter@berbec.com> 24139 * didil <1284255+didil@users.noreply.github.com> 24140 * id01 <gaviniboom@gmail.com> 24141 * Robert Marko <robimarko@gmail.com> 24142 * Philip Harvey <32467456+pharveybattelle@users.noreply.github.com> 24143 * JorisE <JorisE@users.noreply.github.com> 24144 * garry415 <garry.415@gmail.com> 24145 * forgems <forgems@gmail.com> 24146 * Florian Apolloner <florian@apolloner.eu> 24147 * Aleksandar Janković <office@ajankovic.com> <ajankovic@users.noreply.github.com> 24148 * Maran <maran@protonmail.com> 24149 * nguyenhuuluan434 <nguyenhuuluan434@gmail.com> 24150 * Laura Hausmann <zotan@zotan.pw> <laura@hausmann.dev> 24151 * yparitcher <y@paritcher.com> 24152 * AbelThar <abela.tharen@gmail.com> 24153 * Matti Niemenmaa <matti.niemenmaa+git@iki.fi> 24154 * Russell Davis <russelldavis@users.noreply.github.com> 24155 * Yi FU <yi.fu@tink.se> 24156 * Paul Millar <paul.millar@desy.de> 24157 * justinalin <justinalin@qnap.com> 24158 * EliEron <subanimehd@gmail.com> 24159 * justina777 <chiahuei.lin@gmail.com> 24160 * Chaitanya Bankanhal <bchaitanya15@gmail.com> 24161 * Michał Matczuk <michal@scylladb.com> 24162 * Macavirus <macavirus@zoho.com> 24163 * Abhinav Sharma <abhi18av@users.noreply.github.com> 24164 * ginvine <34869051+ginvine@users.noreply.github.com> 24165 * Patrick Wang <mail6543210@yahoo.com.tw> 24166 * Cenk Alti <cenkalti@gmail.com> 24167 * Andreas Chlupka <andy@chlupka.com> 24168 * Alfonso Montero <amontero@tinet.org> 24169 * Ivan Andreev <ivandeex@gmail.com> 24170 * David Baumgold <david@davidbaumgold.com> 24171 * Lars Lehtonen <lars.lehtonen@gmail.com> 24172 * Matei David <matei.david@gmail.com> 24173 * David <david.bramwell@endemolshine.com> 24174 * Anthony Rusdi <33247310+antrusd@users.noreply.github.com> 24175 * Richard Patel <me@terorie.dev> 24176 * 庄天翼 <zty0826@gmail.com> 24177 * SwitchJS <dev@switchjs.com> 24178 * Raphael <PowershellNinja@users.noreply.github.com> 24179 * Sezal Agrawal <sezalagrawal@gmail.com> 24180 * Tyler <TylerNakamura@users.noreply.github.com> 24181 * Brett Dutro <brett.dutro@gmail.com> 24182 * Vighnesh SK <booterror99@gmail.com> 24183 * Arijit Biswas <dibbyo456@gmail.com> 24184 * Michele Caci <michele.caci@gmail.com> 24185 * AlexandrBoltris <ua2fgb@gmail.com> 24186 * Bryce Larson <blarson@saltstack.com> 24187 * Carlos Ferreyra <crypticmind@gmail.com> 24188 * Saksham Khanna <sakshamkhanna@outlook.com> 24189 * dausruddin <5763466+dausruddin@users.noreply.github.com> 24190 * zero-24 <zero-24@users.noreply.github.com> 24191 * Xiaoxing Ye <ye@xiaoxing.us> 24192 * Barry Muldrey <barry@muldrey.net> 24193 * Sebastian Brandt <sebastian.brandt@friday.de> 24194 * Marco Molteni <marco.molteni@mailbox.org> 24195 * Ankur Gupta <ankur0493@gmail.com> 24196 * Maciej Zimnoch <maciej@scylladb.com> 24197 * anuar45 <serdaliyev.anuar@gmail.com> 24198 * Fernando <ferferga@users.noreply.github.com> 24199 * David Cole <david.cole@sohonet.com> 24200 * Wei He <git@weispot.com> 24201 * Outvi V <19144373+outloudvi@users.noreply.github.com> 24202 * Thomas Kriechbaumer <thomas@kriechbaumer.name> 24203 * Tennix <tennix@users.noreply.github.com> 24204 * Ole Schütt <ole@schuett.name> 24205 * Kuang-che Wu <kcwu@csie.org> 24206 * Thomas Eales <wingsuit@users.noreply.github.com> 24207 * Paul Tinsley <paul.tinsley@vitalsource.com> 24208 * Felix Hungenberg <git@shiftgeist.com> 24209 * Benjamin Richter <github@dev.telepath.de> 24210 * landall <cst_zf@qq.com> 24211 * thestigma <thestigma@gmail.com> 24212 * jtagcat <38327267+jtagcat@users.noreply.github.com> 24213 * Damon Permezel <permezel@me.com> 24214 * boosh <boosh@users.noreply.github.com> 24215 * unbelauscht <58393353+unbelauscht@users.noreply.github.com> 24216 * Motonori IWAMURO <vmi@nifty.com> 24217 * Benjapol Worakan <benwrk@live.com> 24218 24219 # Contact the rclone project # 24220 24221 ## Forum ## 24222 24223 Forum for questions and general discussion: 24224 24225 * https://forum.rclone.org 24226 24227 ## Gitub project ## 24228 24229 The project website is at: 24230 24231 * https://github.com/rclone/rclone 24232 24233 There you can file bug reports or contribute pull requests. 24234 24235 ## Twitter ## 24236 24237 You can also follow me on twitter for rclone announcements: 24238 24239 * [@njcw](https://twitter.com/njcw) 24240 24241 ## Email ## 24242 24243 Or if all else fails or you want to ask something private or 24244 confidential email [Nick Craig-Wood](mailto:nick@craig-wood.com). 24245 Please don't email me requests for help - those are better directed to 24246 the forum - thanks! 24247