github.com/10XDev/rclone@v1.52.3-0.20200626220027-16af9ab76b2a/MANUAL.md (about) 1 % rclone(1) User Manual 2 % Nick Craig-Wood 3 % Jun 24, 2020 4 5 # Rclone syncs your files to cloud storage 6 7 <img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" > 8 9 - [About rclone](#about) 10 - [What can rclone do for you?](#what) 11 - [What features does rclone have?](#features) 12 - [What providers does rclone support?](#providers) 13 - [Download](https://rclone.org/downloads/) 14 - [Install](https://rclone.org/install/) 15 - [Donate.](https://rclone.org/donate/) 16 17 ## About rclone {#about} 18 19 Rclone is a command line program to manage files on cloud storage. It 20 is a feature rich alternative to cloud vendors' web storage 21 interfaces. [Over 40 cloud storage products](#providers) support 22 rclone including S3 object stores, business & consumer file storage 23 services, as well as standard transfer protocols. 24 25 Rclone has powerful cloud equivalents to the unix commands rsync, cp, 26 mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax 27 includes shell pipeline support, and `--dry-run` protection. It is 28 used at the command line, in scripts or via its [API](/rc). 29 30 Users call rclone *"The Swiss army knife of cloud storage"*, and 31 *"Technology indistinguishable from magic"*. 32 33 Rclone really looks after your data. It preserves timestamps and 34 verifies checksums at all times. Transfers over limited bandwidth; 35 intermittent connections, or subject to quota can be restarted, from 36 the last good file transferred. You can 37 [check](https://rclone.org/commands/rclone_check/) the integrity of your files. Where 38 possible, rclone employs server side transfers to minimise local 39 bandwidth use and transfers from one provider to another without 40 using local disk. 41 42 Virtual backends wrap local and cloud file systems to apply 43 [encryption](https://rclone.org/crypt/), 44 [caching](https://rclone.org/cache/), 45 [chunking](https://rclone.org/chunker/) and 46 [joining](https://rclone.org/union/). 47 48 Rclone [mounts](https://rclone.org/commands/rclone_mount/) any local, cloud or 49 virtual filesystem as a disk on Windows, 50 macOS, linux and FreeBSD, and also serves these over 51 [SFTP](https://rclone.org/commands/rclone_serve_sftp/), 52 [HTTP](https://rclone.org/commands/rclone_serve_http/), 53 [WebDAV](https://rclone.org/commands/rclone_serve_webdav/), 54 [FTP](https://rclone.org/commands/rclone_serve_ftp/) and 55 [DLNA](https://rclone.org/commands/rclone_serve_dlna/). 56 57 Rclone is mature, open source software originally inspired by rsync 58 and written in [Go](https://golang.org). The friendly support 59 community are familiar with varied use cases. Official Ubuntu, Debian, 60 Fedora, Brew and Chocolatey repos. include rclone. For the latest 61 version [downloading from rclone.org](https://rclone.org/downloads/) is recommended. 62 63 Rclone is widely used on Linux, Windows and Mac. Third party 64 developers create innovative backup, restore, GUI and business 65 process solutions using the rclone command line or API. 66 67 Rclone does the heavy lifting of communicating with cloud storage. 68 69 ## What can rclone do for you? {#what} 70 71 Rclone helps you: 72 73 - Backup (and encrypt) files to cloud storage 74 - Restore (and decrypt) files from cloud storage 75 - Mirror cloud data to other cloud services or locally 76 - Migrate data to cloud, or between cloud storage vendors 77 - Mount multiple, encrypted, cached or diverse cloud storage as a disk 78 - Analyse and account for data held on cloud storage using [lsf](https://rclone.org/commands/rclone_lsf/), [ljson](https://rclone.org/commands/rclone_lsjson/), [size](https://rclone.org/commands/rclone_size/), [ncdu](https://rclone.org/commands/rclone_ncdu/) 79 - [Union](https://rclone.org/union/) file systems together to present multiple local and/or cloud file systems as one 80 81 ## Features {#features} 82 83 - Transfers 84 - MD5, SHA1 hashes are checked at all times for file integrity 85 - Timestamps are preserved on files 86 - Operations can be restarted at any time 87 - Can be to and from network, eg two different cloud providers 88 - Can use multi-threaded downloads to local disk 89 - [Copy](https://rclone.org/commands/rclone_copy/) new or changed files to cloud storage 90 - [Sync](https://rclone.org/commands/rclone_sync/) (one way) to make a directory identical 91 - [Move](https://rclone.org/commands/rclone_move/) files to cloud storage deleting the local after verification 92 - [Check](https://rclone.org/commands/rclone_check/) hashes and for missing/extra files 93 - [Mount](https://rclone.org/commands/rclone_mount/) your cloud storage as a network disk 94 - [Serve](https://rclone.org/commands/rclone_serve/) local or remote files over [HTTP](https://rclone.org/commands/rclone_serve_http/)/[WebDav](https://rclone.org/commands/rclone_serve_webdav/)/[FTP](https://rclone.org/commands/rclone_serve_ftp/)/[SFTP](https://rclone.org/commands/rclone_serve_sftp/)/[dlna](https://rclone.org/commands/rclone_serve_dlna/) 95 - Experimental [Web based GUI](https://rclone.org/gui/) 96 97 ## Supported providers {#providers} 98 99 (There are many others, built on standard protocols such as 100 WebDAV or S3, that work out of the box.) 101 102 103 - 1Fichier 104 - Alibaba Cloud (Aliyun) Object Storage System (OSS) 105 - Amazon Drive 106 - Amazon S3 107 - Backblaze B2 108 - Box 109 - Ceph 110 - Citrix ShareFile 111 - C14 112 - DigitalOcean Spaces 113 - Dreamhost 114 - Dropbox 115 - FTP 116 - Google Cloud Storage 117 - Google Drive 118 - Google Photos 119 - HTTP 120 - Hubic 121 - Jottacloud 122 - IBM COS S3 123 - Koofr 124 - Mail.ru Cloud 125 - Memset Memstore 126 - Mega 127 - Memory 128 - Microsoft Azure Blob Storage 129 - Microsoft OneDrive 130 - Minio 131 - Nextcloud 132 - OVH 133 - OpenDrive 134 - OpenStack Swift 135 - Oracle Cloud Storage 136 - ownCloud 137 - pCloud 138 - premiumize.me 139 - put.io 140 - QingStor 141 - Rackspace Cloud Files 142 - rsync.net 143 - Scaleway 144 - Seafile 145 - SFTP 146 - StackPath 147 - SugarSync 148 - Tardigrade 149 - Wasabi 150 - WebDAV 151 - Yandex Disk 152 - The local filesystem 153 154 155 Links 156 157 * [Home page](https://rclone.org/) 158 * [GitHub project page for source and bug tracker](https://github.com/rclone/rclone) 159 * [Rclone Forum](https://forum.rclone.org) 160 * [Downloads](https://rclone.org/downloads/) 161 162 # Install # 163 164 Rclone is a Go program and comes as a single binary file. 165 166 ## Quickstart ## 167 168 * [Download](https://rclone.org/downloads/) the relevant binary. 169 * Extract the `rclone` or `rclone.exe` binary from the archive 170 * Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 171 172 See below for some expanded Linux / macOS instructions. 173 174 See the [Usage section](https://rclone.org/docs/#usage) of the docs for how to use rclone, or 175 run `rclone -h`. 176 177 ## Script installation ## 178 179 To install rclone on Linux/macOS/BSD systems, run: 180 181 curl https://rclone.org/install.sh | sudo bash 182 183 For beta installation, run: 184 185 curl https://rclone.org/install.sh | sudo bash -s beta 186 187 Note that this script checks the version of rclone installed first and 188 won't re-download if not needed. 189 190 ## Linux installation from precompiled binary ## 191 192 Fetch and unpack 193 194 curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip 195 unzip rclone-current-linux-amd64.zip 196 cd rclone-*-linux-amd64 197 198 Copy binary file 199 200 sudo cp rclone /usr/bin/ 201 sudo chown root:root /usr/bin/rclone 202 sudo chmod 755 /usr/bin/rclone 203 204 Install manpage 205 206 sudo mkdir -p /usr/local/share/man/man1 207 sudo cp rclone.1 /usr/local/share/man/man1/ 208 sudo mandb 209 210 Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 211 212 rclone config 213 214 ## macOS installation with brew ## 215 216 brew install rclone 217 218 ## macOS installation from precompiled binary, using curl ## 219 220 To avoid problems with macOS gatekeeper enforcing the binary to be signed and 221 notarized it is enough to download with `curl`. 222 223 Download the latest version of rclone. 224 225 cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip 226 227 Unzip the download and cd to the extracted folder. 228 229 unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64 230 231 Move rclone to your $PATH. You will be prompted for your password. 232 233 sudo mkdir -p /usr/local/bin 234 sudo mv rclone /usr/local/bin/ 235 236 (the `mkdir` command is safe to run, even if the directory already exists). 237 238 Remove the leftover files. 239 240 cd .. && rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip 241 242 Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) for more details. 243 244 rclone config 245 246 ## macOS installation from precompiled binary, using a web browser ## 247 248 When downloading a binary with a web browser, the browser will set the macOS 249 gatekeeper quarantine attribute. Starting from Catalina, when attempting to run 250 `rclone`, a pop-up will appear saying: 251 252 “rclone” cannot be opened because the developer cannot be verified. 253 macOS cannot verify that this app is free from malware. 254 255 The simplest fix is to run 256 257 xattr -d com.apple.quarantine rclone 258 259 ## Install with docker ## 260 261 The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone). 262 These images are autobuilt by docker hub from the rclone source based 263 on a minimal Alpine linux image. 264 265 The `:latest` tag will always point to the latest stable release. You 266 can use the `:beta` tag to get the latest build from master. You can 267 also use version tags, eg `:1.49.1`, `:1.49` or `:1`. 268 269 ``` 270 $ docker pull rclone/rclone:latest 271 latest: Pulling from rclone/rclone 272 Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11 273 ... 274 $ docker run --rm rclone/rclone:latest version 275 rclone v1.49.1 276 - os/arch: linux/amd64 277 - go version: go1.12.9 278 ``` 279 280 There are a few command line options to consider when starting an rclone Docker container 281 from the rclone image. 282 283 - You need to mount the host rclone config dir at `/config/rclone` into the Docker 284 container. Due to the fact that rclone updates tokens inside its config file, and that 285 the update process involves a file rename, you need to mount the whole host rclone 286 config dir, not just the single host rclone config file. 287 288 - You need to mount a host data dir at `/data` into the Docker container. 289 290 - By default, the rclone binary inside a Docker container runs with UID=0 (root). 291 As a result, all files created in a run will have UID=0. If your config and data files 292 reside on the host with a non-root UID:GID, you need to pass these on the container 293 start command line. 294 295 - It is possible to use `rclone mount` inside a userspace Docker container, and expose 296 the resulting fuse mount to the host. The exact `docker run` options to do that might 297 vary slightly between hosts. See, e.g. the discussion in this 298 [thread](https://github.com/moby/moby/issues/9448). 299 300 You also need to mount the host `/etc/passwd` and `/etc/group` for fuse to work inside 301 the container. 302 303 Here are some commands tested on an Ubuntu 18.04.3 host: 304 305 ``` 306 # config on host at ~/.config/rclone/rclone.conf 307 # data on host at ~/data 308 309 # make sure the config is ok by listing the remotes 310 docker run --rm \ 311 --volume ~/.config/rclone:/config/rclone \ 312 --volume ~/data:/data:shared \ 313 --user $(id -u):$(id -g) \ 314 rclone/rclone \ 315 listremotes 316 317 # perform mount inside Docker container, expose result to host 318 mkdir -p ~/data/mount 319 docker run --rm \ 320 --volume ~/.config/rclone:/config/rclone \ 321 --volume ~/data:/data:shared \ 322 --user $(id -u):$(id -g) \ 323 --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \ 324 --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \ 325 rclone/rclone \ 326 mount dropbox:Photos /data/mount & 327 ls ~/data/mount 328 kill %1 329 ``` 330 331 ## Install from source ## 332 333 Make sure you have at least [Go](https://golang.org/) 1.7 334 installed. [Download go](https://golang.org/dl/) if necessary. The 335 latest release is recommended. Then 336 337 git clone https://github.com/rclone/rclone.git 338 cd rclone 339 go build 340 ./rclone version 341 342 You can also build and install rclone in the 343 [GOPATH](https://github.com/golang/go/wiki/GOPATH) (which defaults to 344 `~/go`) with: 345 346 go get -u -v github.com/rclone/rclone 347 348 and this will build the binary in `$GOPATH/bin` (`~/go/bin/rclone` by 349 default) after downloading the source to 350 `$GOPATH/src/github.com/rclone/rclone` (`~/go/src/github.com/rclone/rclone` 351 by default). 352 353 ## Installation with Ansible ## 354 355 This can be done with [Stefan Weichinger's ansible 356 role](https://github.com/stefangweichinger/ansible-rclone). 357 358 Instructions 359 360 1. `git clone https://github.com/stefangweichinger/ansible-rclone.git` into your local roles-directory 361 2. add the role to the hosts you want rclone installed to: 362 363 ``` 364 - hosts: rclone-hosts 365 roles: 366 - rclone 367 ``` 368 369 Configure 370 --------- 371 372 First, you'll need to configure rclone. As the object storage systems 373 have quite complicated authentication these are kept in a config file. 374 (See the `--config` entry for how to find the config file and choose 375 its location.) 376 377 The easiest way to make the config is to run rclone with the config 378 option: 379 380 rclone config 381 382 See the following for detailed instructions for 383 384 * [1Fichier](https://rclone.org/fichier/) 385 * [Alias](https://rclone.org/alias/) 386 * [Amazon Drive](https://rclone.org/amazonclouddrive/) 387 * [Amazon S3](https://rclone.org/s3/) 388 * [Backblaze B2](https://rclone.org/b2/) 389 * [Box](https://rclone.org/box/) 390 * [Cache](https://rclone.org/cache/) 391 * [Chunker](https://rclone.org/chunker/) - transparently splits large files for other remotes 392 * [Citrix ShareFile](https://rclone.org/sharefile/) 393 * [Crypt](https://rclone.org/crypt/) - to encrypt other remotes 394 * [DigitalOcean Spaces](https://rclone.org/s3/#digitalocean-spaces) 395 * [Dropbox](https://rclone.org/dropbox/) 396 * [FTP](https://rclone.org/ftp/) 397 * [Google Cloud Storage](https://rclone.org/googlecloudstorage/) 398 * [Google Drive](https://rclone.org/drive/) 399 * [Google Photos](https://rclone.org/googlephotos/) 400 * [HTTP](https://rclone.org/http/) 401 * [Hubic](https://rclone.org/hubic/) 402 * [Jottacloud / GetSky.no](https://rclone.org/jottacloud/) 403 * [Koofr](https://rclone.org/koofr/) 404 * [Mail.ru Cloud](https://rclone.org/mailru/) 405 * [Mega](https://rclone.org/mega/) 406 * [Memory](https://rclone.org/memory/) 407 * [Microsoft Azure Blob Storage](https://rclone.org/azureblob/) 408 * [Microsoft OneDrive](https://rclone.org/onedrive/) 409 * [OpenStack Swift / Rackspace Cloudfiles / Memset Memstore](https://rclone.org/swift/) 410 * [OpenDrive](https://rclone.org/opendrive/) 411 * [Pcloud](https://rclone.org/pcloud/) 412 * [premiumize.me](https://rclone.org/premiumizeme/) 413 * [put.io](https://rclone.org/putio/) 414 * [QingStor](https://rclone.org/qingstor/) 415 * [Seafile](https://rclone.org/seafile/) 416 * [SFTP](https://rclone.org/sftp/) 417 * [SugarSync](https://rclone.org/sugarsync/) 418 * [Tardigrade](https://rclone.org/tardigrade/) 419 * [Union](https://rclone.org/union/) 420 * [WebDAV](https://rclone.org/webdav/) 421 * [Yandex Disk](https://rclone.org/yandex/) 422 * [The local filesystem](https://rclone.org/local/) 423 424 Usage 425 ----- 426 427 Rclone syncs a directory tree from one storage system to another. 428 429 Its syntax is like this 430 431 Syntax: [options] subcommand <parameters> <parameters...> 432 433 Source and destination paths are specified by the name you gave the 434 storage system in the config file then the sub path, eg 435 "drive:myfolder" to look at "myfolder" in Google drive. 436 437 You can define as many storage paths as you like in the config file. 438 439 Subcommands 440 ----------- 441 442 rclone uses a system of subcommands. For example 443 444 rclone ls remote:path # lists a remote 445 rclone copy /local/path remote:path # copies /local/path to the remote 446 rclone sync /local/path remote:path # syncs /local/path to the remote 447 448 # rclone config 449 450 Enter an interactive configuration session. 451 452 ## Synopsis 453 454 Enter an interactive configuration session where you can setup new 455 remotes and manage existing ones. You may also set or remove a 456 password to protect your configuration. 457 458 459 ``` 460 rclone config [flags] 461 ``` 462 463 ## Options 464 465 ``` 466 -h, --help help for config 467 ``` 468 469 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 470 471 ## SEE ALSO 472 473 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 474 * [rclone config create](https://rclone.org/commands/rclone_config_create/) - Create a new remote with name, type and options. 475 * [rclone config delete](https://rclone.org/commands/rclone_config_delete/) - Delete an existing remote `name`. 476 * [rclone config disconnect](https://rclone.org/commands/rclone_config_disconnect/) - Disconnects user from remote 477 * [rclone config dump](https://rclone.org/commands/rclone_config_dump/) - Dump the config file as JSON. 478 * [rclone config edit](https://rclone.org/commands/rclone_config_edit/) - Enter an interactive configuration session. 479 * [rclone config file](https://rclone.org/commands/rclone_config_file/) - Show path of configuration file in use. 480 * [rclone config password](https://rclone.org/commands/rclone_config_password/) - Update password in an existing remote. 481 * [rclone config providers](https://rclone.org/commands/rclone_config_providers/) - List in JSON format all the providers and options. 482 * [rclone config reconnect](https://rclone.org/commands/rclone_config_reconnect/) - Re-authenticates user with remote. 483 * [rclone config show](https://rclone.org/commands/rclone_config_show/) - Print (decrypted) config file, or the config for a single remote. 484 * [rclone config update](https://rclone.org/commands/rclone_config_update/) - Update options in an existing remote. 485 * [rclone config userinfo](https://rclone.org/commands/rclone_config_userinfo/) - Prints info about logged in user of remote. 486 487 # rclone copy 488 489 Copy files from source to dest, skipping already copied 490 491 ## Synopsis 492 493 494 Copy the source to the destination. Doesn't transfer 495 unchanged files, testing by size and modification time or 496 MD5SUM. Doesn't delete files from the destination. 497 498 Note that it is always the contents of the directory that is synced, 499 not the directory so when source:path is a directory, it's the 500 contents of source:path that are copied, not the directory name and 501 contents. 502 503 If dest:path doesn't exist, it is created and the source:path contents 504 go there. 505 506 For example 507 508 rclone copy source:sourcepath dest:destpath 509 510 Let's say there are two files in sourcepath 511 512 sourcepath/one.txt 513 sourcepath/two.txt 514 515 This copies them to 516 517 destpath/one.txt 518 destpath/two.txt 519 520 Not to 521 522 destpath/sourcepath/one.txt 523 destpath/sourcepath/two.txt 524 525 If you are familiar with `rsync`, rclone always works as if you had 526 written a trailing / - meaning "copy the contents of this directory". 527 This applies to all commands and whether you are talking about the 528 source or destination. 529 530 See the [--no-traverse](https://rclone.org/docs/#no-traverse) option for controlling 531 whether rclone lists the destination directory or not. Supplying this 532 option when copying a small number of files into a large destination 533 can speed transfers up greatly. 534 535 For example, if you have many files in /path/to/src but only a few of 536 them change every day, you can copy all the files which have changed 537 recently very efficiently like this: 538 539 rclone copy --max-age 24h --no-traverse /path/to/src remote: 540 541 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 542 543 544 ``` 545 rclone copy source:path dest:path [flags] 546 ``` 547 548 ## Options 549 550 ``` 551 --create-empty-src-dirs Create empty source dirs on destination after copy 552 -h, --help help for copy 553 ``` 554 555 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 556 557 ## SEE ALSO 558 559 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 560 561 # rclone sync 562 563 Make source and dest identical, modifying destination only. 564 565 ## Synopsis 566 567 568 Sync the source to the destination, changing the destination 569 only. Doesn't transfer unchanged files, testing by size and 570 modification time or MD5SUM. Destination is updated to match 571 source, including deleting files if necessary. 572 573 **Important**: Since this can cause data loss, test first with the 574 `--dry-run` flag to see exactly what would be copied and deleted. 575 576 Note that files in the destination won't be deleted if there were any 577 errors at any point. 578 579 It is always the contents of the directory that is synced, not the 580 directory so when source:path is a directory, it's the contents of 581 source:path that are copied, not the directory name and contents. See 582 extended explanation in the `copy` command above if unsure. 583 584 If dest:path doesn't exist, it is created and the source:path contents 585 go there. 586 587 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 588 589 590 ``` 591 rclone sync source:path dest:path [flags] 592 ``` 593 594 ## Options 595 596 ``` 597 --create-empty-src-dirs Create empty source dirs on destination after sync 598 -h, --help help for sync 599 ``` 600 601 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 602 603 ## SEE ALSO 604 605 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 606 607 # rclone move 608 609 Move files from source to dest. 610 611 ## Synopsis 612 613 614 Moves the contents of the source directory to the destination 615 directory. Rclone will error if the source and destination overlap and 616 the remote does not support a server side directory move operation. 617 618 If no filters are in use and if possible this will server side move 619 `source:path` into `dest:path`. After this `source:path` will no 620 longer exist. 621 622 Otherwise for each file in `source:path` selected by the filters (if 623 any) this will move it into `dest:path`. If possible a server side 624 move will be used, otherwise it will copy it (server side if possible) 625 into `dest:path` then delete the original (if no errors on copy) in 626 `source:path`. 627 628 If you want to delete empty source directories after move, use the --delete-empty-src-dirs flag. 629 630 See the [--no-traverse](https://rclone.org/docs/#no-traverse) option for controlling 631 whether rclone lists the destination directory or not. Supplying this 632 option when moving a small number of files into a large destination 633 can speed transfers up greatly. 634 635 **Important**: Since this can cause data loss, test first with the 636 --dry-run flag. 637 638 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics. 639 640 641 ``` 642 rclone move source:path dest:path [flags] 643 ``` 644 645 ## Options 646 647 ``` 648 --create-empty-src-dirs Create empty source dirs on destination after move 649 --delete-empty-src-dirs Delete empty source dirs after move 650 -h, --help help for move 651 ``` 652 653 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 654 655 ## SEE ALSO 656 657 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 658 659 # rclone delete 660 661 Remove the contents of path. 662 663 ## Synopsis 664 665 666 Remove the files in path. Unlike `purge` it obeys include/exclude 667 filters so can be used to selectively delete files. 668 669 `rclone delete` only deletes objects but leaves the directory structure 670 alone. If you want to delete a directory and all of its contents use 671 `rclone purge` 672 673 If you supply the --rmdirs flag, it will remove all empty directories along with it. 674 675 Eg delete all files bigger than 100MBytes 676 677 Check what would be deleted first (use either) 678 679 rclone --min-size 100M lsl remote:path 680 rclone --dry-run --min-size 100M delete remote:path 681 682 Then delete 683 684 rclone --min-size 100M delete remote:path 685 686 That reads "delete everything with a minimum size of 100 MB", hence 687 delete all files bigger than 100MBytes. 688 689 690 ``` 691 rclone delete remote:path [flags] 692 ``` 693 694 ## Options 695 696 ``` 697 -h, --help help for delete 698 --rmdirs rmdirs removes empty directories but leaves root intact 699 ``` 700 701 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 702 703 ## SEE ALSO 704 705 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 706 707 # rclone purge 708 709 Remove the path and all of its contents. 710 711 ## Synopsis 712 713 714 Remove the path and all of its contents. Note that this does not obey 715 include/exclude filters - everything will be removed. Use `delete` if 716 you want to selectively delete files. 717 718 719 ``` 720 rclone purge remote:path [flags] 721 ``` 722 723 ## Options 724 725 ``` 726 -h, --help help for purge 727 ``` 728 729 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 730 731 ## SEE ALSO 732 733 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 734 735 # rclone mkdir 736 737 Make the path if it doesn't already exist. 738 739 ## Synopsis 740 741 Make the path if it doesn't already exist. 742 743 ``` 744 rclone mkdir remote:path [flags] 745 ``` 746 747 ## Options 748 749 ``` 750 -h, --help help for mkdir 751 ``` 752 753 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 754 755 ## SEE ALSO 756 757 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 758 759 # rclone rmdir 760 761 Remove the path if empty. 762 763 ## Synopsis 764 765 766 Remove the path. Note that you can't remove a path with 767 objects in it, use purge for that. 768 769 ``` 770 rclone rmdir remote:path [flags] 771 ``` 772 773 ## Options 774 775 ``` 776 -h, --help help for rmdir 777 ``` 778 779 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 780 781 ## SEE ALSO 782 783 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 784 785 # rclone check 786 787 Checks the files in the source and destination match. 788 789 ## Synopsis 790 791 792 Checks the files in the source and destination match. It compares 793 sizes and hashes (MD5 or SHA1) and logs a report of files which don't 794 match. It doesn't alter the source or destination. 795 796 If you supply the --size-only flag, it will only compare the sizes not 797 the hashes as well. Use this for a quick check. 798 799 If you supply the --download flag, it will download the data from 800 both remotes and check them against each other on the fly. This can 801 be useful for remotes that don't support hashes or if you really want 802 to check all the data. 803 804 If you supply the --one-way flag, it will only check that files in source 805 match the files in destination, not the other way around. Meaning extra files in 806 destination that are not in the source will not trigger an error. 807 808 809 ``` 810 rclone check source:path dest:path [flags] 811 ``` 812 813 ## Options 814 815 ``` 816 --download Check by downloading rather than with hash. 817 -h, --help help for check 818 --one-way Check one way only, source files must exist on remote 819 ``` 820 821 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 822 823 ## SEE ALSO 824 825 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 826 827 # rclone ls 828 829 List the objects in the path with size and path. 830 831 ## Synopsis 832 833 834 Lists the objects in the source path to standard output in a human 835 readable format with size and path. Recurses by default. 836 837 Eg 838 839 $ rclone ls swift:bucket 840 60295 bevajer5jef 841 90613 canole 842 94467 diwogej7 843 37600 fubuwic 844 845 846 Any of the filtering options can be applied to this command. 847 848 There are several related list commands 849 850 * `ls` to list size and path of objects only 851 * `lsl` to list modification time, size and path of objects only 852 * `lsd` to list directories only 853 * `lsf` to list objects and directories in easy to parse format 854 * `lsjson` to list objects and directories in JSON format 855 856 `ls`,`lsl`,`lsd` are designed to be human readable. 857 `lsf` is designed to be human and machine readable. 858 `lsjson` is designed to be machine readable. 859 860 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 861 862 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 863 864 Listing a non existent directory will produce an error except for 865 remotes which can't have empty directories (eg s3, swift, gcs, etc - 866 the bucket based remotes). 867 868 869 ``` 870 rclone ls remote:path [flags] 871 ``` 872 873 ## Options 874 875 ``` 876 -h, --help help for ls 877 ``` 878 879 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 880 881 ## SEE ALSO 882 883 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 884 885 # rclone lsd 886 887 List all directories/containers/buckets in the path. 888 889 ## Synopsis 890 891 892 Lists the directories in the source path to standard output. Does not 893 recurse by default. Use the -R flag to recurse. 894 895 This command lists the total size of the directory (if known, -1 if 896 not), the modification time (if known, the current time if not), the 897 number of objects in the directory (if known, -1 if not) and the name 898 of the directory, Eg 899 900 $ rclone lsd swift: 901 494000 2018-04-26 08:43:20 10000 10000files 902 65 2018-04-26 08:43:20 1 1File 903 904 Or 905 906 $ rclone lsd drive:test 907 -1 2016-10-17 17:41:53 -1 1000files 908 -1 2017-01-03 14:40:54 -1 2500files 909 -1 2017-07-08 14:39:28 -1 4000files 910 911 If you just want the directory names use "rclone lsf --dirs-only". 912 913 914 Any of the filtering options can be applied to this command. 915 916 There are several related list commands 917 918 * `ls` to list size and path of objects only 919 * `lsl` to list modification time, size and path of objects only 920 * `lsd` to list directories only 921 * `lsf` to list objects and directories in easy to parse format 922 * `lsjson` to list objects and directories in JSON format 923 924 `ls`,`lsl`,`lsd` are designed to be human readable. 925 `lsf` is designed to be human and machine readable. 926 `lsjson` is designed to be machine readable. 927 928 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 929 930 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 931 932 Listing a non existent directory will produce an error except for 933 remotes which can't have empty directories (eg s3, swift, gcs, etc - 934 the bucket based remotes). 935 936 937 ``` 938 rclone lsd remote:path [flags] 939 ``` 940 941 ## Options 942 943 ``` 944 -h, --help help for lsd 945 -R, --recursive Recurse into the listing. 946 ``` 947 948 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 949 950 ## SEE ALSO 951 952 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 953 954 # rclone lsl 955 956 List the objects in path with modification time, size and path. 957 958 ## Synopsis 959 960 961 Lists the objects in the source path to standard output in a human 962 readable format with modification time, size and path. Recurses by default. 963 964 Eg 965 966 $ rclone lsl swift:bucket 967 60295 2016-06-25 18:55:41.062626927 bevajer5jef 968 90613 2016-06-25 18:55:43.302607074 canole 969 94467 2016-06-25 18:55:43.046609333 diwogej7 970 37600 2016-06-25 18:55:40.814629136 fubuwic 971 972 973 Any of the filtering options can be applied to this command. 974 975 There are several related list commands 976 977 * `ls` to list size and path of objects only 978 * `lsl` to list modification time, size and path of objects only 979 * `lsd` to list directories only 980 * `lsf` to list objects and directories in easy to parse format 981 * `lsjson` to list objects and directories in JSON format 982 983 `ls`,`lsl`,`lsd` are designed to be human readable. 984 `lsf` is designed to be human and machine readable. 985 `lsjson` is designed to be machine readable. 986 987 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 988 989 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 990 991 Listing a non existent directory will produce an error except for 992 remotes which can't have empty directories (eg s3, swift, gcs, etc - 993 the bucket based remotes). 994 995 996 ``` 997 rclone lsl remote:path [flags] 998 ``` 999 1000 ## Options 1001 1002 ``` 1003 -h, --help help for lsl 1004 ``` 1005 1006 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1007 1008 ## SEE ALSO 1009 1010 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1011 1012 # rclone md5sum 1013 1014 Produces an md5sum file for all the objects in the path. 1015 1016 ## Synopsis 1017 1018 1019 Produces an md5sum file for all the objects in the path. This 1020 is in the same format as the standard md5sum tool produces. 1021 1022 1023 ``` 1024 rclone md5sum remote:path [flags] 1025 ``` 1026 1027 ## Options 1028 1029 ``` 1030 -h, --help help for md5sum 1031 ``` 1032 1033 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1034 1035 ## SEE ALSO 1036 1037 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1038 1039 # rclone sha1sum 1040 1041 Produces an sha1sum file for all the objects in the path. 1042 1043 ## Synopsis 1044 1045 1046 Produces an sha1sum file for all the objects in the path. This 1047 is in the same format as the standard sha1sum tool produces. 1048 1049 1050 ``` 1051 rclone sha1sum remote:path [flags] 1052 ``` 1053 1054 ## Options 1055 1056 ``` 1057 -h, --help help for sha1sum 1058 ``` 1059 1060 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1061 1062 ## SEE ALSO 1063 1064 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1065 1066 # rclone size 1067 1068 Prints the total size and number of objects in remote:path. 1069 1070 ## Synopsis 1071 1072 Prints the total size and number of objects in remote:path. 1073 1074 ``` 1075 rclone size remote:path [flags] 1076 ``` 1077 1078 ## Options 1079 1080 ``` 1081 -h, --help help for size 1082 --json format output as JSON 1083 ``` 1084 1085 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1086 1087 ## SEE ALSO 1088 1089 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1090 1091 # rclone version 1092 1093 Show the version number. 1094 1095 ## Synopsis 1096 1097 1098 Show the version number, the go version and the architecture. 1099 1100 Eg 1101 1102 $ rclone version 1103 rclone v1.41 1104 - os/arch: linux/amd64 1105 - go version: go1.10 1106 1107 If you supply the --check flag, then it will do an online check to 1108 compare your version with the latest release and the latest beta. 1109 1110 $ rclone version --check 1111 yours: 1.42.0.6 1112 latest: 1.42 (released 2018-06-16) 1113 beta: 1.42.0.5 (released 2018-06-17) 1114 1115 Or 1116 1117 $ rclone version --check 1118 yours: 1.41 1119 latest: 1.42 (released 2018-06-16) 1120 upgrade: https://downloads.rclone.org/v1.42 1121 beta: 1.42.0.5 (released 2018-06-17) 1122 upgrade: https://beta.rclone.org/v1.42-005-g56e1e820 1123 1124 1125 1126 ``` 1127 rclone version [flags] 1128 ``` 1129 1130 ## Options 1131 1132 ``` 1133 --check Check for new version. 1134 -h, --help help for version 1135 ``` 1136 1137 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1138 1139 ## SEE ALSO 1140 1141 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1142 1143 # rclone cleanup 1144 1145 Clean up the remote if possible 1146 1147 ## Synopsis 1148 1149 1150 Clean up the remote if possible. Empty the trash or delete old file 1151 versions. Not supported by all remotes. 1152 1153 1154 ``` 1155 rclone cleanup remote:path [flags] 1156 ``` 1157 1158 ## Options 1159 1160 ``` 1161 -h, --help help for cleanup 1162 ``` 1163 1164 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1165 1166 ## SEE ALSO 1167 1168 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1169 1170 # rclone dedupe 1171 1172 Interactively find duplicate files and delete/rename them. 1173 1174 ## Synopsis 1175 1176 1177 By default `dedupe` interactively finds duplicate files and offers to 1178 delete all but one or rename them to be different. Only useful with 1179 Google Drive which can have duplicate file names. 1180 1181 In the first pass it will merge directories with the same name. It 1182 will do this iteratively until all the identical directories have been 1183 merged. 1184 1185 The `dedupe` command will delete all but one of any identical (same 1186 md5sum) files it finds without confirmation. This means that for most 1187 duplicated files the `dedupe` command will not be interactive. You 1188 can use `--dry-run` to see what would happen without doing anything. 1189 1190 Here is an example run. 1191 1192 Before - with duplicates 1193 1194 $ rclone lsl drive:dupes 1195 6048320 2016-03-05 16:23:16.798000000 one.txt 1196 6048320 2016-03-05 16:23:11.775000000 one.txt 1197 564374 2016-03-05 16:23:06.731000000 one.txt 1198 6048320 2016-03-05 16:18:26.092000000 one.txt 1199 6048320 2016-03-05 16:22:46.185000000 two.txt 1200 1744073 2016-03-05 16:22:38.104000000 two.txt 1201 564374 2016-03-05 16:22:52.118000000 two.txt 1202 1203 Now the `dedupe` session 1204 1205 $ rclone dedupe drive:dupes 1206 2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode. 1207 one.txt: Found 4 duplicates - deleting identical copies 1208 one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36") 1209 one.txt: 2 duplicates remain 1210 1: 6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36 1211 2: 564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81 1212 s) Skip and do nothing 1213 k) Keep just one (choose which in next step) 1214 r) Rename all to be different (by changing file.jpg to file-1.jpg) 1215 s/k/r> k 1216 Enter the number of the file to keep> 1 1217 one.txt: Deleted 1 extra copies 1218 two.txt: Found 3 duplicates - deleting identical copies 1219 two.txt: 3 duplicates remain 1220 1: 564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81 1221 2: 6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36 1222 3: 1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802 1223 s) Skip and do nothing 1224 k) Keep just one (choose which in next step) 1225 r) Rename all to be different (by changing file.jpg to file-1.jpg) 1226 s/k/r> r 1227 two-1.txt: renamed from: two.txt 1228 two-2.txt: renamed from: two.txt 1229 two-3.txt: renamed from: two.txt 1230 1231 The result being 1232 1233 $ rclone lsl drive:dupes 1234 6048320 2016-03-05 16:23:16.798000000 one.txt 1235 564374 2016-03-05 16:22:52.118000000 two-1.txt 1236 6048320 2016-03-05 16:22:46.185000000 two-2.txt 1237 1744073 2016-03-05 16:22:38.104000000 two-3.txt 1238 1239 Dedupe can be run non interactively using the `--dedupe-mode` flag or by using an extra parameter with the same value 1240 1241 * `--dedupe-mode interactive` - interactive as above. 1242 * `--dedupe-mode skip` - removes identical files then skips anything left. 1243 * `--dedupe-mode first` - removes identical files then keeps the first one. 1244 * `--dedupe-mode newest` - removes identical files then keeps the newest one. 1245 * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. 1246 * `--dedupe-mode largest` - removes identical files then keeps the largest one. 1247 * `--dedupe-mode smallest` - removes identical files then keeps the smallest one. 1248 * `--dedupe-mode rename` - removes identical files then renames the rest to be different. 1249 1250 For example to rename all the identically named photos in your Google Photos directory, do 1251 1252 rclone dedupe --dedupe-mode rename "drive:Google Photos" 1253 1254 Or 1255 1256 rclone dedupe rename "drive:Google Photos" 1257 1258 1259 ``` 1260 rclone dedupe [mode] remote:path [flags] 1261 ``` 1262 1263 ## Options 1264 1265 ``` 1266 --dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename. (default "interactive") 1267 -h, --help help for dedupe 1268 ``` 1269 1270 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1271 1272 ## SEE ALSO 1273 1274 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1275 1276 # rclone about 1277 1278 Get quota information from the remote. 1279 1280 ## Synopsis 1281 1282 1283 Get quota information from the remote, like bytes used/free/quota and bytes 1284 used in the trash. Not supported by all remotes. 1285 1286 This will print to stdout something like this: 1287 1288 Total: 17G 1289 Used: 7.444G 1290 Free: 1.315G 1291 Trashed: 100.000M 1292 Other: 8.241G 1293 1294 Where the fields are: 1295 1296 * Total: total size available. 1297 * Used: total size used 1298 * Free: total amount this user could upload. 1299 * Trashed: total amount in the trash 1300 * Other: total amount in other storage (eg Gmail, Google Photos) 1301 * Objects: total number of objects in the storage 1302 1303 Note that not all the backends provide all the fields - they will be 1304 missing if they are not known for that backend. Where it is known 1305 that the value is unlimited the value will also be omitted. 1306 1307 Use the --full flag to see the numbers written out in full, eg 1308 1309 Total: 18253611008 1310 Used: 7993453766 1311 Free: 1411001220 1312 Trashed: 104857602 1313 Other: 8849156022 1314 1315 Use the --json flag for a computer readable output, eg 1316 1317 { 1318 "total": 18253611008, 1319 "used": 7993453766, 1320 "trashed": 104857602, 1321 "other": 8849156022, 1322 "free": 1411001220 1323 } 1324 1325 1326 ``` 1327 rclone about remote: [flags] 1328 ``` 1329 1330 ## Options 1331 1332 ``` 1333 --full Full numbers instead of SI units 1334 -h, --help help for about 1335 --json Format output as JSON 1336 ``` 1337 1338 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1339 1340 ## SEE ALSO 1341 1342 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1343 1344 # rclone authorize 1345 1346 Remote authorization. 1347 1348 ## Synopsis 1349 1350 1351 Remote authorization. Used to authorize a remote or headless 1352 rclone from a machine with a browser - use as instructed by 1353 rclone config. 1354 1355 Use the --auth-no-open-browser to prevent rclone to open auth 1356 link in default browser automatically. 1357 1358 ``` 1359 rclone authorize [flags] 1360 ``` 1361 1362 ## Options 1363 1364 ``` 1365 --auth-no-open-browser Do not automatically open auth link in default browser 1366 -h, --help help for authorize 1367 ``` 1368 1369 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1370 1371 ## SEE ALSO 1372 1373 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1374 1375 # rclone backend 1376 1377 Run a backend specific command. 1378 1379 ## Synopsis 1380 1381 1382 This runs a backend specific command. The commands themselves (except 1383 for "help" and "features") are defined by the backends and you should 1384 see the backend docs for definitions. 1385 1386 You can discover what commands a backend implements by using 1387 1388 rclone backend help remote: 1389 rclone backend help <backendname> 1390 1391 You can also discover information about the backend using (see 1392 [operations/fsinfo](https://rclone.org/rc/#operations/fsinfo) in the remote control docs 1393 for more info). 1394 1395 rclone backend features remote: 1396 1397 Pass options to the backend command with -o. This should be key=value or key, eg: 1398 1399 rclone backend stats remote:path stats -o format=json -o long 1400 1401 Pass arguments to the backend by placing them on the end of the line 1402 1403 rclone backend cleanup remote:path file1 file2 file3 1404 1405 Note to run these commands on a running backend then see 1406 [backend/command](https://rclone.org/rc/#backend/command) in the rc docs. 1407 1408 1409 ``` 1410 rclone backend <command> remote:path [opts] <args> [flags] 1411 ``` 1412 1413 ## Options 1414 1415 ``` 1416 -h, --help help for backend 1417 --json Always output in JSON format. 1418 -o, --option stringArray Option in the form name=value or name. 1419 ``` 1420 1421 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1422 1423 ## SEE ALSO 1424 1425 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1426 1427 # rclone cat 1428 1429 Concatenates any files and sends them to stdout. 1430 1431 ## Synopsis 1432 1433 1434 rclone cat sends any files to standard output. 1435 1436 You can use it like this to output a single file 1437 1438 rclone cat remote:path/to/file 1439 1440 Or like this to output any file in dir or its subdirectories. 1441 1442 rclone cat remote:path/to/dir 1443 1444 Or like this to output any .txt files in dir or its subdirectories. 1445 1446 rclone --include "*.txt" cat remote:path/to/dir 1447 1448 Use the --head flag to print characters only at the start, --tail for 1449 the end and --offset and --count to print a section in the middle. 1450 Note that if offset is negative it will count from the end, so 1451 --offset -1 --count 1 is equivalent to --tail 1. 1452 1453 1454 ``` 1455 rclone cat remote:path [flags] 1456 ``` 1457 1458 ## Options 1459 1460 ``` 1461 --count int Only print N characters. (default -1) 1462 --discard Discard the output instead of printing. 1463 --head int Only print the first N characters. 1464 -h, --help help for cat 1465 --offset int Start printing at offset N (or from end if -ve). 1466 --tail int Only print the last N characters. 1467 ``` 1468 1469 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1470 1471 ## SEE ALSO 1472 1473 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1474 1475 # rclone config create 1476 1477 Create a new remote with name, type and options. 1478 1479 ## Synopsis 1480 1481 1482 Create a new remote of `name` with `type` and options. The options 1483 should be passed in pairs of `key` `value`. 1484 1485 For example to make a swift remote of name myremote using auto config 1486 you would do: 1487 1488 rclone config create myremote swift env_auth true 1489 1490 Note that if the config process would normally ask a question the 1491 default is taken. Each time that happens rclone will print a message 1492 saying how to affect the value taken. 1493 1494 If any of the parameters passed is a password field, then rclone will 1495 automatically obscure them if they aren't already obscured before 1496 putting them in the config file. 1497 1498 **NB** If the password parameter is 22 characters or longer and 1499 consists only of base64 characters then rclone can get confused about 1500 whether the password is already obscured or not and put unobscured 1501 passwords into the config file. If you want to be 100% certain that 1502 the passwords get obscured then use the "--obscure" flag, or if you 1503 are 100% certain you are already passing obscured passwords then use 1504 "--no-obscure". You can also set osbscured passwords using the 1505 "rclone config password" command. 1506 1507 So for example if you wanted to configure a Google Drive remote but 1508 using remote authorization you would do this: 1509 1510 rclone config create mydrive drive config_is_local false 1511 1512 1513 ``` 1514 rclone config create `name` `type` [`key` `value`]* [flags] 1515 ``` 1516 1517 ## Options 1518 1519 ``` 1520 -h, --help help for create 1521 --no-obscure Force any passwords not to be obscured. 1522 --obscure Force any passwords to be obscured. 1523 ``` 1524 1525 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1526 1527 ## SEE ALSO 1528 1529 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1530 1531 # rclone config delete 1532 1533 Delete an existing remote `name`. 1534 1535 ## Synopsis 1536 1537 Delete an existing remote `name`. 1538 1539 ``` 1540 rclone config delete `name` [flags] 1541 ``` 1542 1543 ## Options 1544 1545 ``` 1546 -h, --help help for delete 1547 ``` 1548 1549 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1550 1551 ## SEE ALSO 1552 1553 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1554 1555 # rclone config disconnect 1556 1557 Disconnects user from remote 1558 1559 ## Synopsis 1560 1561 1562 This disconnects the remote: passed in to the cloud storage system. 1563 1564 This normally means revoking the oauth token. 1565 1566 To reconnect use "rclone config reconnect". 1567 1568 1569 ``` 1570 rclone config disconnect remote: [flags] 1571 ``` 1572 1573 ## Options 1574 1575 ``` 1576 -h, --help help for disconnect 1577 ``` 1578 1579 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1580 1581 ## SEE ALSO 1582 1583 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1584 1585 # rclone config dump 1586 1587 Dump the config file as JSON. 1588 1589 ## Synopsis 1590 1591 Dump the config file as JSON. 1592 1593 ``` 1594 rclone config dump [flags] 1595 ``` 1596 1597 ## Options 1598 1599 ``` 1600 -h, --help help for dump 1601 ``` 1602 1603 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1604 1605 ## SEE ALSO 1606 1607 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1608 1609 # rclone config edit 1610 1611 Enter an interactive configuration session. 1612 1613 ## Synopsis 1614 1615 Enter an interactive configuration session where you can setup new 1616 remotes and manage existing ones. You may also set or remove a 1617 password to protect your configuration. 1618 1619 1620 ``` 1621 rclone config edit [flags] 1622 ``` 1623 1624 ## Options 1625 1626 ``` 1627 -h, --help help for edit 1628 ``` 1629 1630 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1631 1632 ## SEE ALSO 1633 1634 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1635 1636 # rclone config file 1637 1638 Show path of configuration file in use. 1639 1640 ## Synopsis 1641 1642 Show path of configuration file in use. 1643 1644 ``` 1645 rclone config file [flags] 1646 ``` 1647 1648 ## Options 1649 1650 ``` 1651 -h, --help help for file 1652 ``` 1653 1654 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1655 1656 ## SEE ALSO 1657 1658 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1659 1660 # rclone config password 1661 1662 Update password in an existing remote. 1663 1664 ## Synopsis 1665 1666 1667 Update an existing remote's password. The password 1668 should be passed in pairs of `key` `value`. 1669 1670 For example to set password of a remote of name myremote you would do: 1671 1672 rclone config password myremote fieldname mypassword 1673 1674 This command is obsolete now that "config update" and "config create" 1675 both support obscuring passwords directly. 1676 1677 1678 ``` 1679 rclone config password `name` [`key` `value`]+ [flags] 1680 ``` 1681 1682 ## Options 1683 1684 ``` 1685 -h, --help help for password 1686 ``` 1687 1688 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1689 1690 ## SEE ALSO 1691 1692 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1693 1694 # rclone config providers 1695 1696 List in JSON format all the providers and options. 1697 1698 ## Synopsis 1699 1700 List in JSON format all the providers and options. 1701 1702 ``` 1703 rclone config providers [flags] 1704 ``` 1705 1706 ## Options 1707 1708 ``` 1709 -h, --help help for providers 1710 ``` 1711 1712 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1713 1714 ## SEE ALSO 1715 1716 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1717 1718 # rclone config reconnect 1719 1720 Re-authenticates user with remote. 1721 1722 ## Synopsis 1723 1724 1725 This reconnects remote: passed in to the cloud storage system. 1726 1727 To disconnect the remote use "rclone config disconnect". 1728 1729 This normally means going through the interactive oauth flow again. 1730 1731 1732 ``` 1733 rclone config reconnect remote: [flags] 1734 ``` 1735 1736 ## Options 1737 1738 ``` 1739 -h, --help help for reconnect 1740 ``` 1741 1742 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1743 1744 ## SEE ALSO 1745 1746 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1747 1748 # rclone config show 1749 1750 Print (decrypted) config file, or the config for a single remote. 1751 1752 ## Synopsis 1753 1754 Print (decrypted) config file, or the config for a single remote. 1755 1756 ``` 1757 rclone config show [<remote>] [flags] 1758 ``` 1759 1760 ## Options 1761 1762 ``` 1763 -h, --help help for show 1764 ``` 1765 1766 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1767 1768 ## SEE ALSO 1769 1770 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1771 1772 # rclone config update 1773 1774 Update options in an existing remote. 1775 1776 ## Synopsis 1777 1778 1779 Update an existing remote's options. The options should be passed in 1780 in pairs of `key` `value`. 1781 1782 For example to update the env_auth field of a remote of name myremote 1783 you would do: 1784 1785 rclone config update myremote swift env_auth true 1786 1787 If any of the parameters passed is a password field, then rclone will 1788 automatically obscure them if they aren't already obscured before 1789 putting them in the config file. 1790 1791 **NB** If the password parameter is 22 characters or longer and 1792 consists only of base64 characters then rclone can get confused about 1793 whether the password is already obscured or not and put unobscured 1794 passwords into the config file. If you want to be 100% certain that 1795 the passwords get obscured then use the "--obscure" flag, or if you 1796 are 100% certain you are already passing obscured passwords then use 1797 "--no-obscure". You can also set osbscured passwords using the 1798 "rclone config password" command. 1799 1800 If the remote uses OAuth the token will be updated, if you don't 1801 require this add an extra parameter thus: 1802 1803 rclone config update myremote swift env_auth true config_refresh_token false 1804 1805 1806 ``` 1807 rclone config update `name` [`key` `value`]+ [flags] 1808 ``` 1809 1810 ## Options 1811 1812 ``` 1813 -h, --help help for update 1814 --no-obscure Force any passwords not to be obscured. 1815 --obscure Force any passwords to be obscured. 1816 ``` 1817 1818 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1819 1820 ## SEE ALSO 1821 1822 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1823 1824 # rclone config userinfo 1825 1826 Prints info about logged in user of remote. 1827 1828 ## Synopsis 1829 1830 1831 This prints the details of the person logged in to the cloud storage 1832 system. 1833 1834 1835 ``` 1836 rclone config userinfo remote: [flags] 1837 ``` 1838 1839 ## Options 1840 1841 ``` 1842 -h, --help help for userinfo 1843 --json Format output as JSON 1844 ``` 1845 1846 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1847 1848 ## SEE ALSO 1849 1850 * [rclone config](https://rclone.org/commands/rclone_config/) - Enter an interactive configuration session. 1851 1852 # rclone copyto 1853 1854 Copy files from source to dest, skipping already copied 1855 1856 ## Synopsis 1857 1858 1859 If source:path is a file or directory then it copies it to a file or 1860 directory named dest:path. 1861 1862 This can be used to upload single files to other than their current 1863 name. If the source is a directory then it acts exactly like the copy 1864 command. 1865 1866 So 1867 1868 rclone copyto src dst 1869 1870 where src and dst are rclone paths, either remote:path or 1871 /path/to/local or C:\windows\path\if\on\windows. 1872 1873 This will: 1874 1875 if src is file 1876 copy it to dst, overwriting an existing file if it exists 1877 if src is directory 1878 copy it to dst, overwriting existing files if they exist 1879 see copy command for full details 1880 1881 This doesn't transfer unchanged files, testing by size and 1882 modification time or MD5SUM. It doesn't delete files from the 1883 destination. 1884 1885 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics 1886 1887 1888 ``` 1889 rclone copyto source:path dest:path [flags] 1890 ``` 1891 1892 ## Options 1893 1894 ``` 1895 -h, --help help for copyto 1896 ``` 1897 1898 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1899 1900 ## SEE ALSO 1901 1902 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1903 1904 # rclone copyurl 1905 1906 Copy url content to dest. 1907 1908 ## Synopsis 1909 1910 1911 Download a URL's content and copy it to the destination without saving 1912 it in temporary storage. 1913 1914 Setting --auto-filename will cause the file name to be retrieved from 1915 the from URL (after any redirections) and used in the destination 1916 path. 1917 1918 Setting --no-clobber will prevent overwriting file on the 1919 destination if there is one with the same name. 1920 1921 Setting --stdout or making the output file name "-" will cause the 1922 output to be written to standard output. 1923 1924 1925 ``` 1926 rclone copyurl https://example.com dest:path [flags] 1927 ``` 1928 1929 ## Options 1930 1931 ``` 1932 -a, --auto-filename Get the file name from the URL and use it for destination file path 1933 -h, --help help for copyurl 1934 --no-clobber Prevent overwriting file with same name 1935 --stdout Write the output to stdout rather than a file 1936 ``` 1937 1938 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1939 1940 ## SEE ALSO 1941 1942 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1943 1944 # rclone cryptcheck 1945 1946 Cryptcheck checks the integrity of a crypted remote. 1947 1948 ## Synopsis 1949 1950 1951 rclone cryptcheck checks a remote against a crypted remote. This is 1952 the equivalent of running rclone check, but able to check the 1953 checksums of the crypted remote. 1954 1955 For it to work the underlying remote of the cryptedremote must support 1956 some kind of checksum. 1957 1958 It works by reading the nonce from each file on the cryptedremote: and 1959 using that to encrypt each file on the remote:. It then checks the 1960 checksum of the underlying file on the cryptedremote: against the 1961 checksum of the file it has just encrypted. 1962 1963 Use it like this 1964 1965 rclone cryptcheck /path/to/files encryptedremote:path 1966 1967 You can use it like this also, but that will involve downloading all 1968 the files in remote:path. 1969 1970 rclone cryptcheck remote:path encryptedremote:path 1971 1972 After it has run it will log the status of the encryptedremote:. 1973 1974 If you supply the --one-way flag, it will only check that files in source 1975 match the files in destination, not the other way around. Meaning extra files in 1976 destination that are not in the source will not trigger an error. 1977 1978 1979 ``` 1980 rclone cryptcheck remote:path cryptedremote:path [flags] 1981 ``` 1982 1983 ## Options 1984 1985 ``` 1986 -h, --help help for cryptcheck 1987 --one-way Check one way only, source files must exist on destination 1988 ``` 1989 1990 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 1991 1992 ## SEE ALSO 1993 1994 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 1995 1996 # rclone cryptdecode 1997 1998 Cryptdecode returns unencrypted file names. 1999 2000 ## Synopsis 2001 2002 2003 rclone cryptdecode returns unencrypted file names when provided with 2004 a list of encrypted file names. List limit is 10 items. 2005 2006 If you supply the --reverse flag, it will return encrypted file names. 2007 2008 use it like this 2009 2010 rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2 2011 2012 rclone cryptdecode --reverse encryptedremote: filename1 filename2 2013 2014 2015 ``` 2016 rclone cryptdecode encryptedremote: encryptedfilename [flags] 2017 ``` 2018 2019 ## Options 2020 2021 ``` 2022 -h, --help help for cryptdecode 2023 --reverse Reverse cryptdecode, encrypts filenames 2024 ``` 2025 2026 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2027 2028 ## SEE ALSO 2029 2030 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2031 2032 # rclone deletefile 2033 2034 Remove a single file from remote. 2035 2036 ## Synopsis 2037 2038 2039 Remove a single file from remote. Unlike `delete` it cannot be used to 2040 remove a directory and it doesn't obey include/exclude filters - if the specified file exists, 2041 it will always be removed. 2042 2043 2044 ``` 2045 rclone deletefile remote:path [flags] 2046 ``` 2047 2048 ## Options 2049 2050 ``` 2051 -h, --help help for deletefile 2052 ``` 2053 2054 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2055 2056 ## SEE ALSO 2057 2058 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2059 2060 # rclone genautocomplete 2061 2062 Output completion script for a given shell. 2063 2064 ## Synopsis 2065 2066 2067 Generates a shell completion script for rclone. 2068 Run with --help to list the supported shells. 2069 2070 2071 ## Options 2072 2073 ``` 2074 -h, --help help for genautocomplete 2075 ``` 2076 2077 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2078 2079 ## SEE ALSO 2080 2081 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2082 * [rclone genautocomplete bash](https://rclone.org/commands/rclone_genautocomplete_bash/) - Output bash completion script for rclone. 2083 * [rclone genautocomplete fish](https://rclone.org/commands/rclone_genautocomplete_fish/) - Output fish completion script for rclone. 2084 * [rclone genautocomplete zsh](https://rclone.org/commands/rclone_genautocomplete_zsh/) - Output zsh completion script for rclone. 2085 2086 # rclone genautocomplete bash 2087 2088 Output bash completion script for rclone. 2089 2090 ## Synopsis 2091 2092 2093 Generates a bash shell autocompletion script for rclone. 2094 2095 This writes to /etc/bash_completion.d/rclone by default so will 2096 probably need to be run with sudo or as root, eg 2097 2098 sudo rclone genautocomplete bash 2099 2100 Logout and login again to use the autocompletion scripts, or source 2101 them directly 2102 2103 . /etc/bash_completion 2104 2105 If you supply a command line argument the script will be written 2106 there. 2107 2108 2109 ``` 2110 rclone genautocomplete bash [output_file] [flags] 2111 ``` 2112 2113 ## Options 2114 2115 ``` 2116 -h, --help help for bash 2117 ``` 2118 2119 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2120 2121 ## SEE ALSO 2122 2123 * [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. 2124 2125 # rclone genautocomplete fish 2126 2127 Output fish completion script for rclone. 2128 2129 ## Synopsis 2130 2131 2132 Generates a fish autocompletion script for rclone. 2133 2134 This writes to /etc/fish/completions/rclone.fish by default so will 2135 probably need to be run with sudo or as root, eg 2136 2137 sudo rclone genautocomplete fish 2138 2139 Logout and login again to use the autocompletion scripts, or source 2140 them directly 2141 2142 . /etc/fish/completions/rclone.fish 2143 2144 If you supply a command line argument the script will be written 2145 there. 2146 2147 2148 ``` 2149 rclone genautocomplete fish [output_file] [flags] 2150 ``` 2151 2152 ## Options 2153 2154 ``` 2155 -h, --help help for fish 2156 ``` 2157 2158 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2159 2160 ## SEE ALSO 2161 2162 * [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. 2163 2164 # rclone genautocomplete zsh 2165 2166 Output zsh completion script for rclone. 2167 2168 ## Synopsis 2169 2170 2171 Generates a zsh autocompletion script for rclone. 2172 2173 This writes to /usr/share/zsh/vendor-completions/_rclone by default so will 2174 probably need to be run with sudo or as root, eg 2175 2176 sudo rclone genautocomplete zsh 2177 2178 Logout and login again to use the autocompletion scripts, or source 2179 them directly 2180 2181 autoload -U compinit && compinit 2182 2183 If you supply a command line argument the script will be written 2184 there. 2185 2186 2187 ``` 2188 rclone genautocomplete zsh [output_file] [flags] 2189 ``` 2190 2191 ## Options 2192 2193 ``` 2194 -h, --help help for zsh 2195 ``` 2196 2197 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2198 2199 ## SEE ALSO 2200 2201 * [rclone genautocomplete](https://rclone.org/commands/rclone_genautocomplete/) - Output completion script for a given shell. 2202 2203 # rclone gendocs 2204 2205 Output markdown docs for rclone to the directory supplied. 2206 2207 ## Synopsis 2208 2209 2210 This produces markdown docs for the rclone commands to the directory 2211 supplied. These are in a format suitable for hugo to render into the 2212 rclone.org website. 2213 2214 ``` 2215 rclone gendocs output_directory [flags] 2216 ``` 2217 2218 ## Options 2219 2220 ``` 2221 -h, --help help for gendocs 2222 ``` 2223 2224 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2225 2226 ## SEE ALSO 2227 2228 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2229 2230 # rclone hashsum 2231 2232 Produces a hashsum file for all the objects in the path. 2233 2234 ## Synopsis 2235 2236 2237 Produces a hash file for all the objects in the path using the hash 2238 named. The output is in the same format as the standard 2239 md5sum/sha1sum tool. 2240 2241 Run without a hash to see the list of supported hashes, eg 2242 2243 $ rclone hashsum 2244 Supported hashes are: 2245 * MD5 2246 * SHA-1 2247 * DropboxHash 2248 * QuickXorHash 2249 2250 Then 2251 2252 $ rclone hashsum MD5 remote:path 2253 2254 2255 ``` 2256 rclone hashsum <hash> remote:path [flags] 2257 ``` 2258 2259 ## Options 2260 2261 ``` 2262 --base64 Output base64 encoded hashsum 2263 -h, --help help for hashsum 2264 ``` 2265 2266 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2267 2268 ## SEE ALSO 2269 2270 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2271 2272 # rclone link 2273 2274 Generate public link to file/folder. 2275 2276 ## Synopsis 2277 2278 2279 rclone link will create or retrieve a public link to the given file or folder. 2280 2281 rclone link remote:path/to/file 2282 rclone link remote:path/to/folder/ 2283 2284 If successful, the last line of the output will contain the link. Exact 2285 capabilities depend on the remote, but the link will always be created with 2286 the least constraints – e.g. no expiry, no password protection, accessible 2287 without account. 2288 2289 2290 ``` 2291 rclone link remote:path [flags] 2292 ``` 2293 2294 ## Options 2295 2296 ``` 2297 -h, --help help for link 2298 ``` 2299 2300 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2301 2302 ## SEE ALSO 2303 2304 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2305 2306 # rclone listremotes 2307 2308 List all the remotes in the config file. 2309 2310 ## Synopsis 2311 2312 2313 rclone listremotes lists all the available remotes from the config file. 2314 2315 When uses with the -l flag it lists the types too. 2316 2317 2318 ``` 2319 rclone listremotes [flags] 2320 ``` 2321 2322 ## Options 2323 2324 ``` 2325 -h, --help help for listremotes 2326 --long Show the type as well as names. 2327 ``` 2328 2329 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2330 2331 ## SEE ALSO 2332 2333 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2334 2335 # rclone lsf 2336 2337 List directories and objects in remote:path formatted for parsing 2338 2339 ## Synopsis 2340 2341 2342 List the contents of the source path (directories and objects) to 2343 standard output in a form which is easy to parse by scripts. By 2344 default this will just be the names of the objects and directories, 2345 one per line. The directories will have a / suffix. 2346 2347 Eg 2348 2349 $ rclone lsf swift:bucket 2350 bevajer5jef 2351 canole 2352 diwogej7 2353 ferejej3gux/ 2354 fubuwic 2355 2356 Use the --format option to control what gets listed. By default this 2357 is just the path, but you can use these parameters to control the 2358 output: 2359 2360 p - path 2361 s - size 2362 t - modification time 2363 h - hash 2364 i - ID of object 2365 o - Original ID of underlying object 2366 m - MimeType of object if known 2367 e - encrypted name 2368 T - tier of storage if known, eg "Hot" or "Cool" 2369 2370 So if you wanted the path, size and modification time, you would use 2371 --format "pst", or maybe --format "tsp" to put the path last. 2372 2373 Eg 2374 2375 $ rclone lsf --format "tsp" swift:bucket 2376 2016-06-25 18:55:41;60295;bevajer5jef 2377 2016-06-25 18:55:43;90613;canole 2378 2016-06-25 18:55:43;94467;diwogej7 2379 2018-04-26 08:50:45;0;ferejej3gux/ 2380 2016-06-25 18:55:40;37600;fubuwic 2381 2382 If you specify "h" in the format you will get the MD5 hash by default, 2383 use the "--hash" flag to change which hash you want. Note that this 2384 can be returned as an empty string if it isn't available on the object 2385 (and for directories), "ERROR" if there was an error reading it from 2386 the object and "UNSUPPORTED" if that object does not support that hash 2387 type. 2388 2389 For example to emulate the md5sum command you can use 2390 2391 rclone lsf -R --hash MD5 --format hp --separator " " --files-only . 2392 2393 Eg 2394 2395 $ rclone lsf -R --hash MD5 --format hp --separator " " --files-only swift:bucket 2396 7908e352297f0f530b84a756f188baa3 bevajer5jef 2397 cd65ac234e6fea5925974a51cdd865cc canole 2398 03b5341b4f234b9d984d03ad076bae91 diwogej7 2399 8fd37c3810dd660778137ac3a66cc06d fubuwic 2400 99713e14a4c4ff553acaf1930fad985b gixacuh7ku 2401 2402 (Though "rclone md5sum ." is an easier way of typing this.) 2403 2404 By default the separator is ";" this can be changed with the 2405 --separator flag. Note that separators aren't escaped in the path so 2406 putting it last is a good strategy. 2407 2408 Eg 2409 2410 $ rclone lsf --separator "," --format "tshp" swift:bucket 2411 2016-06-25 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef 2412 2016-06-25 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole 2413 2016-06-25 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7 2414 2018-04-26 08:52:53,0,,ferejej3gux/ 2415 2016-06-25 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic 2416 2417 You can output in CSV standard format. This will escape things in " 2418 if they contain , 2419 2420 Eg 2421 2422 $ rclone lsf --csv --files-only --format ps remote:path 2423 test.log,22355 2424 test.sh,449 2425 "this file contains a comma, in the file name.txt",6 2426 2427 Note that the --absolute parameter is useful for making lists of files 2428 to pass to an rclone copy with the --files-from-raw flag. 2429 2430 For example to find all the files modified within one day and copy 2431 those only (without traversing the whole directory structure): 2432 2433 rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files 2434 rclone copy --files-from-raw new_files /path/to/local remote:path 2435 2436 2437 Any of the filtering options can be applied to this command. 2438 2439 There are several related list commands 2440 2441 * `ls` to list size and path of objects only 2442 * `lsl` to list modification time, size and path of objects only 2443 * `lsd` to list directories only 2444 * `lsf` to list objects and directories in easy to parse format 2445 * `lsjson` to list objects and directories in JSON format 2446 2447 `ls`,`lsl`,`lsd` are designed to be human readable. 2448 `lsf` is designed to be human and machine readable. 2449 `lsjson` is designed to be machine readable. 2450 2451 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 2452 2453 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 2454 2455 Listing a non existent directory will produce an error except for 2456 remotes which can't have empty directories (eg s3, swift, gcs, etc - 2457 the bucket based remotes). 2458 2459 2460 ``` 2461 rclone lsf remote:path [flags] 2462 ``` 2463 2464 ## Options 2465 2466 ``` 2467 --absolute Put a leading / in front of path names. 2468 --csv Output in CSV format. 2469 -d, --dir-slash Append a slash to directory names. (default true) 2470 --dirs-only Only list directories. 2471 --files-only Only list files. 2472 -F, --format string Output format - see help for details (default "p") 2473 --hash h Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "MD5") 2474 -h, --help help for lsf 2475 -R, --recursive Recurse into the listing. 2476 -s, --separator string Separator for the items in the format. (default ";") 2477 ``` 2478 2479 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2480 2481 ## SEE ALSO 2482 2483 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2484 2485 # rclone lsjson 2486 2487 List directories and objects in the path in JSON format. 2488 2489 ## Synopsis 2490 2491 List directories and objects in the path in JSON format. 2492 2493 The output is an array of Items, where each Item looks like this 2494 2495 { 2496 "Hashes" : { 2497 "SHA-1" : "f572d396fae9206628714fb2ce00f72e94f2258f", 2498 "MD5" : "b1946ac92492d2347c6235b4d2611184", 2499 "DropboxHash" : "ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc" 2500 }, 2501 "ID": "y2djkhiujf83u33", 2502 "OrigID": "UYOJVTUW00Q1RzTDA", 2503 "IsBucket" : false, 2504 "IsDir" : false, 2505 "MimeType" : "application/octet-stream", 2506 "ModTime" : "2017-05-31T16:15:57.034468261+01:00", 2507 "Name" : "file.txt", 2508 "Encrypted" : "v0qpsdq8anpci8n929v3uu9338", 2509 "EncryptedPath" : "kja9098349023498/v0qpsdq8anpci8n929v3uu9338", 2510 "Path" : "full/path/goes/here/file.txt", 2511 "Size" : 6, 2512 "Tier" : "hot", 2513 } 2514 2515 If --hash is not specified the Hashes property won't be emitted. The 2516 types of hash can be specified with the --hash-type parameter (which 2517 may be repeated). If --hash-type is set then it implies --hash. 2518 2519 If --no-modtime is specified then ModTime will be blank. This can 2520 speed things up on remotes where reading the ModTime takes an extra 2521 request (eg s3, swift). 2522 2523 If --no-mimetype is specified then MimeType will be blank. This can 2524 speed things up on remotes where reading the MimeType takes an extra 2525 request (eg s3, swift). 2526 2527 If --encrypted is not specified the Encrypted won't be emitted. 2528 2529 If --dirs-only is not specified files in addition to directories are 2530 returned 2531 2532 If --files-only is not specified directories in addition to the files 2533 will be returned. 2534 2535 The Path field will only show folders below the remote path being listed. 2536 If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" 2537 will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". 2538 When used without --recursive the Path will always be the same as Name. 2539 2540 If the directory is a bucket in a bucket based backend, then 2541 "IsBucket" will be set to true. This key won't be present unless it is 2542 "true". 2543 2544 The time is in RFC3339 format with up to nanosecond precision. The 2545 number of decimal digits in the seconds will depend on the precision 2546 that the remote can hold the times, so if times are accurate to the 2547 nearest millisecond (eg Google Drive) then 3 digits will always be 2548 shown ("2017-05-31T16:15:57.034+01:00") whereas if the times are 2549 accurate to the nearest second (Dropbox, Box, WebDav etc) no digits 2550 will be shown ("2017-05-31T16:15:57+01:00"). 2551 2552 The whole output can be processed as a JSON blob, or alternatively it 2553 can be processed line by line as each item is written one to a line. 2554 2555 Any of the filtering options can be applied to this command. 2556 2557 There are several related list commands 2558 2559 * `ls` to list size and path of objects only 2560 * `lsl` to list modification time, size and path of objects only 2561 * `lsd` to list directories only 2562 * `lsf` to list objects and directories in easy to parse format 2563 * `lsjson` to list objects and directories in JSON format 2564 2565 `ls`,`lsl`,`lsd` are designed to be human readable. 2566 `lsf` is designed to be human and machine readable. 2567 `lsjson` is designed to be machine readable. 2568 2569 Note that `ls` and `lsl` recurse by default - use "--max-depth 1" to stop the recursion. 2570 2571 The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default - use "-R" to make them recurse. 2572 2573 Listing a non existent directory will produce an error except for 2574 remotes which can't have empty directories (eg s3, swift, gcs, etc - 2575 the bucket based remotes). 2576 2577 2578 ``` 2579 rclone lsjson remote:path [flags] 2580 ``` 2581 2582 ## Options 2583 2584 ``` 2585 --dirs-only Show only directories in the listing. 2586 -M, --encrypted Show the encrypted names. 2587 --files-only Show only files in the listing. 2588 --hash Include hashes in the output (may take longer). 2589 --hash-type stringArray Show only this hash type (may be repeated). 2590 -h, --help help for lsjson 2591 --no-mimetype Don't read the mime type (can speed things up). 2592 --no-modtime Don't read the modification time (can speed things up). 2593 --original Show the ID of the underlying Object. 2594 -R, --recursive Recurse into the listing. 2595 ``` 2596 2597 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 2598 2599 ## SEE ALSO 2600 2601 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 2602 2603 # rclone mount 2604 2605 Mount the remote as file system on a mountpoint. 2606 2607 ## Synopsis 2608 2609 2610 rclone mount allows Linux, FreeBSD, macOS and Windows to 2611 mount any of Rclone's cloud storage systems as a file system with 2612 FUSE. 2613 2614 First set up your remote using `rclone config`. Check it works with `rclone ls` etc. 2615 2616 You can either run mount in foreground mode or background (daemon) mode. Mount runs in 2617 foreground mode by default, use the --daemon flag to specify background mode mode. 2618 Background mode is only supported on Linux and OSX, you can only run mount in 2619 foreground mode on Windows. 2620 2621 On Linux/macOS/FreeBSD Start the mount like this where `/path/to/local/mount` 2622 is an **empty** **existing** directory. 2623 2624 rclone mount remote:path/to/files /path/to/local/mount 2625 2626 Or on Windows like this where `X:` is an unused drive letter 2627 or use a path to **non-existent** directory. 2628 2629 rclone mount remote:path/to/files X: 2630 rclone mount remote:path/to/files C:\path\to\nonexistent\directory 2631 2632 When running in background mode the user will have to stop the mount manually (specified below). 2633 2634 When the program ends while in foreground mode, either via Ctrl+C or receiving 2635 a SIGINT or SIGTERM signal, the mount is automatically stopped. 2636 2637 The umount operation can fail, for example when the mountpoint is busy. 2638 When that happens, it is the user's responsibility to stop the mount manually. 2639 2640 Stopping the mount manually: 2641 2642 # Linux 2643 fusermount -u /path/to/local/mount 2644 # OS X 2645 umount /path/to/local/mount 2646 2647 ## Installing on Windows 2648 2649 To run rclone mount on Windows, you will need to 2650 download and install [WinFsp](http://www.secfs.net/winfsp/). 2651 2652 [WinFsp](https://github.com/billziss-gh/winfsp) is an open source 2653 Windows File System Proxy which makes it easy to write user space file 2654 systems for Windows. It provides a FUSE emulation layer which rclone 2655 uses combination with 2656 [cgofuse](https://github.com/billziss-gh/cgofuse). Both of these 2657 packages are by Bill Zissimopoulos who was very helpful during the 2658 implementation of rclone mount for Windows. 2659 2660 ### Windows caveats 2661 2662 Note that drives created as Administrator are not visible by other 2663 accounts (including the account that was elevated as 2664 Administrator). So if you start a Windows drive from an Administrative 2665 Command Prompt and then try to access the same drive from Explorer 2666 (which does not run as Administrator), you will not be able to see the 2667 new drive. 2668 2669 The easiest way around this is to start the drive from a normal 2670 command prompt. It is also possible to start a drive from the SYSTEM 2671 account (using [the WinFsp.Launcher 2672 infrastructure](https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture)) 2673 which creates drives accessible for everyone on the system or 2674 alternatively using [the nssm service manager](https://nssm.cc/usage). 2675 2676 ### Mount as a network drive 2677 2678 By default, rclone will mount the remote as a normal drive. However, 2679 you can also mount it as a **Network Drive** (or **Network Share**, as 2680 mentioned in some places) 2681 2682 Unlike other systems, Windows provides a different filesystem type for 2683 network drives. Windows and other programs treat the network drives 2684 and fixed/removable drives differently: In network drives, many I/O 2685 operations are optimized, as the high latency and low reliability 2686 (compared to a normal drive) of a network is expected. 2687 2688 Although many people prefer network shares to be mounted as normal 2689 system drives, this might cause some issues, such as programs not 2690 working as expected or freezes and errors while operating with the 2691 mounted remote in Windows Explorer. If you experience any of those, 2692 consider mounting rclone remotes as network shares, as Windows expects 2693 normal drives to be fast and reliable, while cloud storage is far from 2694 that. See also [Limitations](#limitations) section below for more 2695 info 2696 2697 Add "--fuse-flag --VolumePrefix=\server\share" to your "mount" 2698 command, **replacing "share" with any other name of your choice if you 2699 are mounting more than one remote**. Otherwise, the mountpoints will 2700 conflict and your mounted filesystems will overlap. 2701 2702 [Read more about drive mapping](https://en.wikipedia.org/wiki/Drive_mapping) 2703 2704 ## Limitations 2705 2706 Without the use of "--vfs-cache-mode" this can only write files 2707 sequentially, it can only seek when reading. This means that many 2708 applications won't work with their files on an rclone mount without 2709 "--vfs-cache-mode writes" or "--vfs-cache-mode full". See the [File 2710 Caching](#file-caching) section for more info. 2711 2712 The bucket based remotes (eg Swift, S3, Google Compute Storage, B2, 2713 Hubic) do not support the concept of empty directories, so empty 2714 directories will have a tendency to disappear once they fall out of 2715 the directory cache. 2716 2717 Only supported on Linux, FreeBSD, OS X and Windows at the moment. 2718 2719 ## rclone mount vs rclone sync/copy 2720 2721 File systems expect things to be 100% reliable, whereas cloud storage 2722 systems are a long way from 100% reliable. The rclone sync/copy 2723 commands cope with this with lots of retries. However rclone mount 2724 can't use retries in the same way without making local copies of the 2725 uploads. Look at the [file caching](#file-caching) 2726 for solutions to make mount more reliable. 2727 2728 ## Attribute caching 2729 2730 You can use the flag --attr-timeout to set the time the kernel caches 2731 the attributes (size, modification time etc) for directory entries. 2732 2733 The default is "1s" which caches files just long enough to avoid 2734 too many callbacks to rclone from the kernel. 2735 2736 In theory 0s should be the correct value for filesystems which can 2737 change outside the control of the kernel. However this causes quite a 2738 few problems such as 2739 [rclone using too much memory](https://github.com/rclone/rclone/issues/2157), 2740 [rclone not serving files to samba](https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) 2741 and [excessive time listing directories](https://github.com/rclone/rclone/issues/2095#issuecomment-371141147). 2742 2743 The kernel can cache the info about a file for the time given by 2744 "--attr-timeout". You may see corruption if the remote file changes 2745 length during this window. It will show up as either a truncated file 2746 or a file with garbage on the end. With "--attr-timeout 1s" this is 2747 very unlikely but not impossible. The higher you set "--attr-timeout" 2748 the more likely it is. The default setting of "1s" is the lowest 2749 setting which mitigates the problems above. 2750 2751 If you set it higher ('10s' or '1m' say) then the kernel will call 2752 back to rclone less often making it more efficient, however there is 2753 more chance of the corruption issue above. 2754 2755 If files don't change on the remote outside of the control of rclone 2756 then there is no chance of corruption. 2757 2758 This is the same as setting the attr_timeout option in mount.fuse. 2759 2760 ## Filters 2761 2762 Note that all the rclone filters can be used to select a subset of the 2763 files to be visible in the mount. 2764 2765 ## systemd 2766 2767 When running rclone mount as a systemd service, it is possible 2768 to use Type=notify. In this case the service will enter the started state 2769 after the mountpoint has been successfully set up. 2770 Units having the rclone mount service specified as a requirement 2771 will see all files and folders immediately in this mode. 2772 2773 ## chunked reading ### 2774 2775 --vfs-read-chunk-size will enable reading the source objects in parts. 2776 This can reduce the used download quota for some remotes by requesting only chunks 2777 from the remote that are actually read at the cost of an increased number of requests. 2778 2779 When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, 2780 the chunk size for each open file will get doubled for each chunk read, until the 2781 specified value is reached. A value of -1 will disable the limit and the chunk size will 2782 grow indefinitely. 2783 2784 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following 2785 parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. 2786 When --vfs-read-chunk-size-limit 500M is specified, the result would be 2787 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. 2788 2789 Chunked reading will only work with --vfs-cache-mode < full, as the file will always 2790 be copied to the vfs cache before opening with --vfs-cache-mode full. 2791 2792 ## Directory Cache 2793 2794 Using the `--dir-cache-time` flag, you can set how long a 2795 directory should be considered up to date and not refreshed from the 2796 backend. Changes made locally in the mount may appear immediately or 2797 invalidate the cache. However, changes done on the remote will only 2798 be picked up once the cache expires if the backend configured does not 2799 support polling for changes. If the backend supports polling, changes 2800 will be picked up on within the polling interval. 2801 2802 Alternatively, you can send a `SIGHUP` signal to rclone for 2803 it to flush all directory caches, regardless of how old they are. 2804 Assuming only one rclone instance is running, you can reset the cache 2805 like this: 2806 2807 kill -SIGHUP $(pidof rclone) 2808 2809 If you configure rclone with a [remote control](/rc) then you can use 2810 rclone rc to flush the whole directory cache: 2811 2812 rclone rc vfs/forget 2813 2814 Or individual files or directories: 2815 2816 rclone rc vfs/forget file=path/to/file dir=path/to/dir 2817 2818 ## File Buffering 2819 2820 The `--buffer-size` flag determines the amount of memory, 2821 that will be used to buffer data in advance. 2822 2823 Each open file descriptor will try to keep the specified amount of 2824 data in memory at all times. The buffered data is bound to one file 2825 descriptor and won't be shared between multiple open file descriptors 2826 of the same file. 2827 2828 This flag is a upper limit for the used memory per file descriptor. 2829 The buffer will only use memory for data that is downloaded but not 2830 not yet read. If the buffer is empty, only a small amount of memory 2831 will be used. 2832 The maximum memory used by rclone for buffering can be up to 2833 `--buffer-size * open files`. 2834 2835 ## File Caching 2836 2837 These flags control the VFS file caching options. The VFS layer is 2838 used by rclone mount to make a cloud storage system work more like a 2839 normal file system. 2840 2841 You'll need to enable VFS caching if you want, for example, to read 2842 and write simultaneously to a file. See below for more details. 2843 2844 Note that the VFS cache works in addition to the cache backend and you 2845 may find that you need one or the other or both. 2846 2847 --cache-dir string Directory rclone will use for caching. 2848 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 2849 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 2850 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 2851 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 2852 2853 If run with `-vv` rclone will print the location of the file cache. The 2854 files are stored in the user cache file area which is OS dependent but 2855 can be controlled with `--cache-dir` or setting the appropriate 2856 environment variable. 2857 2858 The cache has 4 different modes selected by `--vfs-cache-mode`. 2859 The higher the cache mode the more compatible rclone becomes at the 2860 cost of using disk space. 2861 2862 Note that files are written back to the remote only when they are 2863 closed so if rclone is quit or dies with open files then these won't 2864 get written back to the remote. However they will still be in the on 2865 disk cache. 2866 2867 If using --vfs-cache-max-size note that the cache may exceed this size 2868 for two reasons. Firstly because it is only checked every 2869 --vfs-cache-poll-interval. Secondly because open files cannot be 2870 evicted from the cache. 2871 2872 ### --vfs-cache-mode off 2873 2874 In this mode the cache will read directly from the remote and write 2875 directly to the remote without caching anything on disk. 2876 2877 This will mean some operations are not possible 2878 2879 * Files can't be opened for both read AND write 2880 * Files opened for write can't be seeked 2881 * Existing files opened for write must have O_TRUNC set 2882 * Files open for read with O_TRUNC will be opened write only 2883 * Files open for write only will behave as if O_TRUNC was supplied 2884 * Open modes O_APPEND, O_TRUNC are ignored 2885 * If an upload fails it can't be retried 2886 2887 ### --vfs-cache-mode minimal 2888 2889 This is very similar to "off" except that files opened for read AND 2890 write will be buffered to disks. This means that files opened for 2891 write will be a lot more compatible, but uses the minimal disk space. 2892 2893 These operations are not possible 2894 2895 * Files opened for write only can't be seeked 2896 * Existing files opened for write must have O_TRUNC set 2897 * Files opened for write only will ignore O_APPEND, O_TRUNC 2898 * If an upload fails it can't be retried 2899 2900 ### --vfs-cache-mode writes 2901 2902 In this mode files opened for read only are still read directly from 2903 the remote, write only and read/write files are buffered to disk 2904 first. 2905 2906 This mode should support all normal file system operations. 2907 2908 If an upload fails it will be retried up to --low-level-retries times. 2909 2910 ### --vfs-cache-mode full 2911 2912 In this mode all reads and writes are buffered to and from disk. When 2913 a file is opened for read it will be downloaded in its entirety first. 2914 2915 This may be appropriate for your needs, or you may prefer to look at 2916 the cache backend which does a much more sophisticated job of caching, 2917 including caching directory hierarchies and chunks of files. 2918 2919 In this mode, unlike the others, when a file is written to the disk, 2920 it will be kept on the disk after it is written to the remote. It 2921 will be purged on a schedule according to `--vfs-cache-max-age`. 2922 2923 This mode should support all normal file system operations. 2924 2925 If an upload or download fails it will be retried up to 2926 --low-level-retries times. 2927 2928 ## Case Sensitivity 2929 2930 Linux file systems are case-sensitive: two files can differ only 2931 by case, and the exact case must be used when opening a file. 2932 2933 Windows is not like most other operating systems supported by rclone. 2934 File systems in modern Windows are case-insensitive but case-preserving: 2935 although existing files can be opened using any case, the exact case used 2936 to create the file is preserved and available for programs to query. 2937 It is not allowed for two files in the same directory to differ only by case. 2938 2939 Usually file systems on macOS are case-insensitive. It is possible to make macOS 2940 file systems case-sensitive but that is not the default 2941 2942 The "--vfs-case-insensitive" mount flag controls how rclone handles these 2943 two cases. If its value is "false", rclone passes file names to the mounted 2944 file system as is. If the flag is "true" (or appears without a value on 2945 command line), rclone may perform a "fixup" as explained below. 2946 2947 The user may specify a file name to open/delete/rename/etc with a case 2948 different than what is stored on mounted file system. If an argument refers 2949 to an existing file with exactly the same name, then the case of the existing 2950 file on the disk will be used. However, if a file name with exactly the same 2951 name is not found but a name differing only by case exists, rclone will 2952 transparently fixup the name. This fixup happens only when an existing file 2953 is requested. Case sensitivity of file names created anew by rclone is 2954 controlled by an underlying mounted file system. 2955 2956 Note that case sensitivity of the operating system running rclone (the target) 2957 may differ from case sensitivity of a file system mounted by rclone (the source). 2958 The flag controls whether "fixup" is performed to satisfy the target. 2959 2960 If the flag is not provided on command line, then its default value depends 2961 on the operating system where rclone runs: "true" on Windows and macOS, "false" 2962 otherwise. If the flag is provided without a value, then it is "true". 2963 2964 2965 ``` 2966 rclone mount remote:path /path/to/mountpoint [flags] 2967 ``` 2968 2969 ## Options 2970 2971 ``` 2972 --allow-non-empty Allow mounting over a non-empty directory (not Windows). 2973 --allow-other Allow access to other users. 2974 --allow-root Allow access to root user. 2975 --async-read Use asynchronous reads. (default true) 2976 --attr-timeout duration Time for which file/directory attributes are cached. (default 1s) 2977 --daemon Run mount as a daemon (background mode). 2978 --daemon-timeout duration Time limit for rclone to respond to kernel (not supported by all OSes). 2979 --debug-fuse Debug the FUSE internals - needs -v. 2980 --default-permissions Makes kernel enforce access control based on the file mode. 2981 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 2982 --dir-perms FileMode Directory permissions (default 0777) 2983 --file-perms FileMode File permissions (default 0666) 2984 --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required. 2985 --gid uint32 Override the gid field set by the filesystem. (default 1000) 2986 -h, --help help for mount 2987 --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads. (default 128k) 2988 --no-checksum Don't compare checksums on up/download. 2989 --no-modtime Don't read/write the modification time (can speed things up). 2990 --no-seek Don't allow seeking in files. 2991 -o, --option stringArray Option for libfuse/WinFsp. Repeat if required. 2992 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 2993 --read-only Mount read-only. 2994 --uid uint32 Override the uid field set by the filesystem. (default 1000) 2995 --umask int Override the permission bits set by the filesystem. 2996 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 2997 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 2998 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 2999 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3000 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3001 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3002 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3003 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 3004 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 3005 --volname string Set the volume name (not supported by all OSes). 3006 --write-back-cache Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used. 3007 ``` 3008 3009 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3010 3011 ## SEE ALSO 3012 3013 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3014 3015 # rclone moveto 3016 3017 Move file or directory from source to dest. 3018 3019 ## Synopsis 3020 3021 3022 If source:path is a file or directory then it moves it to a file or 3023 directory named dest:path. 3024 3025 This can be used to rename files or upload single files to other than 3026 their existing name. If the source is a directory then it acts exactly 3027 like the move command. 3028 3029 So 3030 3031 rclone moveto src dst 3032 3033 where src and dst are rclone paths, either remote:path or 3034 /path/to/local or C:\windows\path\if\on\windows. 3035 3036 This will: 3037 3038 if src is file 3039 move it to dst, overwriting an existing file if it exists 3040 if src is directory 3041 move it to dst, overwriting existing files if they exist 3042 see move command for full details 3043 3044 This doesn't transfer unchanged files, testing by size and 3045 modification time or MD5SUM. src will be deleted on successful 3046 transfer. 3047 3048 **Important**: Since this can cause data loss, test first with the 3049 --dry-run flag. 3050 3051 **Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics. 3052 3053 3054 ``` 3055 rclone moveto source:path dest:path [flags] 3056 ``` 3057 3058 ## Options 3059 3060 ``` 3061 -h, --help help for moveto 3062 ``` 3063 3064 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3065 3066 ## SEE ALSO 3067 3068 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3069 3070 # rclone ncdu 3071 3072 Explore a remote with a text based user interface. 3073 3074 ## Synopsis 3075 3076 3077 This displays a text based user interface allowing the navigation of a 3078 remote. It is most useful for answering the question - "What is using 3079 all my disk space?". 3080 3081 3082 3083 To make the user interface it first scans the entire remote given and 3084 builds an in memory representation. rclone ncdu can be used during 3085 this scanning phase and you will see it building up the directory 3086 structure as it goes along. 3087 3088 Here are the keys - press '?' to toggle the help on and off 3089 3090 ↑,↓ or k,j to Move 3091 →,l to enter 3092 ←,h to return 3093 c toggle counts 3094 g toggle graph 3095 n,s,C sort by name,size,count 3096 d delete file/directory 3097 y copy current path to clipbard 3098 Y display current path 3099 ^L refresh screen 3100 ? to toggle help on and off 3101 q/ESC/c-C to quit 3102 3103 This an homage to the [ncdu tool](https://dev.yorhel.nl/ncdu) but for 3104 rclone remotes. It is missing lots of features at the moment 3105 but is useful as it stands. 3106 3107 Note that it might take some time to delete big files/folders. The 3108 UI won't respond in the meantime since the deletion is done synchronously. 3109 3110 3111 ``` 3112 rclone ncdu remote:path [flags] 3113 ``` 3114 3115 ## Options 3116 3117 ``` 3118 -h, --help help for ncdu 3119 ``` 3120 3121 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3122 3123 ## SEE ALSO 3124 3125 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3126 3127 # rclone obscure 3128 3129 Obscure password for use in the rclone config file 3130 3131 ## Synopsis 3132 3133 In the rclone config file, human readable passwords are 3134 obscured. Obscuring them is done by encrypting them and writing them 3135 out in base64. This is **not** a secure way of encrypting these 3136 passwords as rclone can decrypt them - it is to prevent "eyedropping" 3137 - namely someone seeing a password in the rclone config file by 3138 accident. 3139 3140 Many equally important things (like access tokens) are not obscured in 3141 the config file. However it is very hard to shoulder surf a 64 3142 character hex token. 3143 3144 If you want to encrypt the config file then please use config file 3145 encryption - see [rclone config](https://rclone.org/commands/rclone_config/) for more 3146 info. 3147 3148 ``` 3149 rclone obscure password [flags] 3150 ``` 3151 3152 ## Options 3153 3154 ``` 3155 -h, --help help for obscure 3156 ``` 3157 3158 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3159 3160 ## SEE ALSO 3161 3162 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3163 3164 # rclone rc 3165 3166 Run a command against a running rclone. 3167 3168 ## Synopsis 3169 3170 3171 3172 This runs a command against a running rclone. Use the --url flag to 3173 specify an non default URL to connect on. This can be either a 3174 ":port" which is taken to mean "http://localhost:port" or a 3175 "host:port" which is taken to mean "http://host:port" 3176 3177 A username and password can be passed in with --user and --pass. 3178 3179 Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, 3180 --user, --pass. 3181 3182 Arguments should be passed in as parameter=value. 3183 3184 The result will be returned as a JSON object by default. 3185 3186 The --json parameter can be used to pass in a JSON blob as an input 3187 instead of key=value arguments. This is the only way of passing in 3188 more complicated values. 3189 3190 The -o/--opt option can be used to set a key "opt" with key, value 3191 options in the form "-o key=value" or "-o key". It can be repeated as 3192 many times as required. This is useful for rc commands which take the 3193 "opt" parameter which by convention is a dictionary of strings. 3194 3195 -o key=value -o key2 3196 3197 Will place this in the "opt" value 3198 3199 {"key":"value", "key2","") 3200 3201 3202 The -a/--arg option can be used to set strings in the "arg" value. It 3203 can be repeated as many times as required. This is useful for rc 3204 commands which take the "arg" parameter which by convention is a list 3205 of strings. 3206 3207 -a value -a value2 3208 3209 Will place this in the "arg" value 3210 3211 ["value", "value2"] 3212 3213 Use --loopback to connect to the rclone instance running "rclone rc". 3214 This is very useful for testing commands without having to run an 3215 rclone rc server, eg: 3216 3217 rclone rc --loopback operations/about fs=/ 3218 3219 Use "rclone rc" to see a list of all possible commands. 3220 3221 ``` 3222 rclone rc commands parameter [flags] 3223 ``` 3224 3225 ## Options 3226 3227 ``` 3228 -a, --arg stringArray Argument placed in the "arg" array. 3229 -h, --help help for rc 3230 --json string Input JSON - use instead of key=value args. 3231 --loopback If set connect to this rclone instance not via HTTP. 3232 --no-output If set don't output the JSON result. 3233 -o, --opt stringArray Option in the form name=value or name placed in the "opt" array. 3234 --pass string Password to use to connect to rclone remote control. 3235 --url string URL to connect to rclone remote control. (default "http://localhost:5572/") 3236 --user string Username to use to rclone remote control. 3237 ``` 3238 3239 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3240 3241 ## SEE ALSO 3242 3243 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3244 3245 # rclone rcat 3246 3247 Copies standard input to file on remote. 3248 3249 ## Synopsis 3250 3251 3252 rclone rcat reads from standard input (stdin) and copies it to a 3253 single remote file. 3254 3255 echo "hello world" | rclone rcat remote:path/to/file 3256 ffmpeg - | rclone rcat remote:path/to/file 3257 3258 If the remote file already exists, it will be overwritten. 3259 3260 rcat will try to upload small files in a single request, which is 3261 usually more efficient than the streaming/chunked upload endpoints, 3262 which use multiple requests. Exact behaviour depends on the remote. 3263 What is considered a small file may be set through 3264 `--streaming-upload-cutoff`. Uploading only starts after 3265 the cutoff is reached or if the file ends before that. The data 3266 must fit into RAM. The cutoff needs to be small enough to adhere 3267 the limits of your remote, please see there. Generally speaking, 3268 setting this cutoff too high will decrease your performance. 3269 3270 Note that the upload can also not be retried because the data is 3271 not kept around until the upload succeeds. If you need to transfer 3272 a lot of data, you're better off caching locally and then 3273 `rclone move` it to the destination. 3274 3275 ``` 3276 rclone rcat remote:path [flags] 3277 ``` 3278 3279 ## Options 3280 3281 ``` 3282 -h, --help help for rcat 3283 ``` 3284 3285 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3286 3287 ## SEE ALSO 3288 3289 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3290 3291 # rclone rcd 3292 3293 Run rclone listening to remote control commands only. 3294 3295 ## Synopsis 3296 3297 3298 This runs rclone so that it only listens to remote control commands. 3299 3300 This is useful if you are controlling rclone via the rc API. 3301 3302 If you pass in a path to a directory, rclone will serve that directory 3303 for GET requests on the URL passed in. It will also open the URL in 3304 the browser when rclone is run. 3305 3306 See the [rc documentation](https://rclone.org/rc/) for more info on the rc flags. 3307 3308 3309 ``` 3310 rclone rcd <path to files to serve>* [flags] 3311 ``` 3312 3313 ## Options 3314 3315 ``` 3316 -h, --help help for rcd 3317 ``` 3318 3319 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3320 3321 ## SEE ALSO 3322 3323 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3324 3325 # rclone rmdirs 3326 3327 Remove empty directories under the path. 3328 3329 ## Synopsis 3330 3331 This removes any empty directories (or directories that only contain 3332 empty directories) under the path that it finds, including the path if 3333 it has nothing in. 3334 3335 If you supply the --leave-root flag, it will not remove the root directory. 3336 3337 This is useful for tidying up remotes that rclone has left a lot of 3338 empty directories in. 3339 3340 3341 3342 ``` 3343 rclone rmdirs remote:path [flags] 3344 ``` 3345 3346 ## Options 3347 3348 ``` 3349 -h, --help help for rmdirs 3350 --leave-root Do not remove root directory if empty 3351 ``` 3352 3353 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3354 3355 ## SEE ALSO 3356 3357 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3358 3359 # rclone serve 3360 3361 Serve a remote over a protocol. 3362 3363 ## Synopsis 3364 3365 rclone serve is used to serve a remote over a given protocol. This 3366 command requires the use of a subcommand to specify the protocol, eg 3367 3368 rclone serve http remote: 3369 3370 Each subcommand has its own options which you can see in their help. 3371 3372 3373 ``` 3374 rclone serve <protocol> [opts] <remote> [flags] 3375 ``` 3376 3377 ## Options 3378 3379 ``` 3380 -h, --help help for serve 3381 ``` 3382 3383 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3384 3385 ## SEE ALSO 3386 3387 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 3388 * [rclone serve dlna](https://rclone.org/commands/rclone_serve_dlna/) - Serve remote:path over DLNA 3389 * [rclone serve ftp](https://rclone.org/commands/rclone_serve_ftp/) - Serve remote:path over FTP. 3390 * [rclone serve http](https://rclone.org/commands/rclone_serve_http/) - Serve the remote over HTTP. 3391 * [rclone serve restic](https://rclone.org/commands/rclone_serve_restic/) - Serve the remote for restic's REST API. 3392 * [rclone serve sftp](https://rclone.org/commands/rclone_serve_sftp/) - Serve the remote over SFTP. 3393 * [rclone serve webdav](https://rclone.org/commands/rclone_serve_webdav/) - Serve remote:path over webdav. 3394 3395 # rclone serve dlna 3396 3397 Serve remote:path over DLNA 3398 3399 ## Synopsis 3400 3401 rclone serve dlna is a DLNA media server for media stored in an rclone remote. Many 3402 devices, such as the Xbox and PlayStation, can automatically discover this server in the LAN 3403 and play audio/video from it. VLC is also supported. Service discovery uses UDP multicast 3404 packets (SSDP) and will thus only work on LANs. 3405 3406 Rclone will list all files present in the remote, without filtering based on media formats or 3407 file extensions. Additionally, there is no media transcoding support. This means that some 3408 players might show files that they are not able to play back correctly. 3409 3410 3411 ## Server options 3412 3413 Use --addr to specify which IP address and port the server should 3414 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3415 IPs. 3416 3417 Use --name to choose the friendly server name, which is by 3418 default "rclone (hostname)". 3419 3420 Use --log-trace in conjunction with -vv to enable additional debug 3421 logging of all UPNP traffic. 3422 3423 ## Directory Cache 3424 3425 Using the `--dir-cache-time` flag, you can set how long a 3426 directory should be considered up to date and not refreshed from the 3427 backend. Changes made locally in the mount may appear immediately or 3428 invalidate the cache. However, changes done on the remote will only 3429 be picked up once the cache expires if the backend configured does not 3430 support polling for changes. If the backend supports polling, changes 3431 will be picked up on within the polling interval. 3432 3433 Alternatively, you can send a `SIGHUP` signal to rclone for 3434 it to flush all directory caches, regardless of how old they are. 3435 Assuming only one rclone instance is running, you can reset the cache 3436 like this: 3437 3438 kill -SIGHUP $(pidof rclone) 3439 3440 If you configure rclone with a [remote control](/rc) then you can use 3441 rclone rc to flush the whole directory cache: 3442 3443 rclone rc vfs/forget 3444 3445 Or individual files or directories: 3446 3447 rclone rc vfs/forget file=path/to/file dir=path/to/dir 3448 3449 ## File Buffering 3450 3451 The `--buffer-size` flag determines the amount of memory, 3452 that will be used to buffer data in advance. 3453 3454 Each open file descriptor will try to keep the specified amount of 3455 data in memory at all times. The buffered data is bound to one file 3456 descriptor and won't be shared between multiple open file descriptors 3457 of the same file. 3458 3459 This flag is a upper limit for the used memory per file descriptor. 3460 The buffer will only use memory for data that is downloaded but not 3461 not yet read. If the buffer is empty, only a small amount of memory 3462 will be used. 3463 The maximum memory used by rclone for buffering can be up to 3464 `--buffer-size * open files`. 3465 3466 ## File Caching 3467 3468 These flags control the VFS file caching options. The VFS layer is 3469 used by rclone mount to make a cloud storage system work more like a 3470 normal file system. 3471 3472 You'll need to enable VFS caching if you want, for example, to read 3473 and write simultaneously to a file. See below for more details. 3474 3475 Note that the VFS cache works in addition to the cache backend and you 3476 may find that you need one or the other or both. 3477 3478 --cache-dir string Directory rclone will use for caching. 3479 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3480 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 3481 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3482 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 3483 3484 If run with `-vv` rclone will print the location of the file cache. The 3485 files are stored in the user cache file area which is OS dependent but 3486 can be controlled with `--cache-dir` or setting the appropriate 3487 environment variable. 3488 3489 The cache has 4 different modes selected by `--vfs-cache-mode`. 3490 The higher the cache mode the more compatible rclone becomes at the 3491 cost of using disk space. 3492 3493 Note that files are written back to the remote only when they are 3494 closed so if rclone is quit or dies with open files then these won't 3495 get written back to the remote. However they will still be in the on 3496 disk cache. 3497 3498 If using --vfs-cache-max-size note that the cache may exceed this size 3499 for two reasons. Firstly because it is only checked every 3500 --vfs-cache-poll-interval. Secondly because open files cannot be 3501 evicted from the cache. 3502 3503 ### --vfs-cache-mode off 3504 3505 In this mode the cache will read directly from the remote and write 3506 directly to the remote without caching anything on disk. 3507 3508 This will mean some operations are not possible 3509 3510 * Files can't be opened for both read AND write 3511 * Files opened for write can't be seeked 3512 * Existing files opened for write must have O_TRUNC set 3513 * Files open for read with O_TRUNC will be opened write only 3514 * Files open for write only will behave as if O_TRUNC was supplied 3515 * Open modes O_APPEND, O_TRUNC are ignored 3516 * If an upload fails it can't be retried 3517 3518 ### --vfs-cache-mode minimal 3519 3520 This is very similar to "off" except that files opened for read AND 3521 write will be buffered to disks. This means that files opened for 3522 write will be a lot more compatible, but uses the minimal disk space. 3523 3524 These operations are not possible 3525 3526 * Files opened for write only can't be seeked 3527 * Existing files opened for write must have O_TRUNC set 3528 * Files opened for write only will ignore O_APPEND, O_TRUNC 3529 * If an upload fails it can't be retried 3530 3531 ### --vfs-cache-mode writes 3532 3533 In this mode files opened for read only are still read directly from 3534 the remote, write only and read/write files are buffered to disk 3535 first. 3536 3537 This mode should support all normal file system operations. 3538 3539 If an upload fails it will be retried up to --low-level-retries times. 3540 3541 ### --vfs-cache-mode full 3542 3543 In this mode all reads and writes are buffered to and from disk. When 3544 a file is opened for read it will be downloaded in its entirety first. 3545 3546 This may be appropriate for your needs, or you may prefer to look at 3547 the cache backend which does a much more sophisticated job of caching, 3548 including caching directory hierarchies and chunks of files. 3549 3550 In this mode, unlike the others, when a file is written to the disk, 3551 it will be kept on the disk after it is written to the remote. It 3552 will be purged on a schedule according to `--vfs-cache-max-age`. 3553 3554 This mode should support all normal file system operations. 3555 3556 If an upload or download fails it will be retried up to 3557 --low-level-retries times. 3558 3559 ## Case Sensitivity 3560 3561 Linux file systems are case-sensitive: two files can differ only 3562 by case, and the exact case must be used when opening a file. 3563 3564 Windows is not like most other operating systems supported by rclone. 3565 File systems in modern Windows are case-insensitive but case-preserving: 3566 although existing files can be opened using any case, the exact case used 3567 to create the file is preserved and available for programs to query. 3568 It is not allowed for two files in the same directory to differ only by case. 3569 3570 Usually file systems on macOS are case-insensitive. It is possible to make macOS 3571 file systems case-sensitive but that is not the default 3572 3573 The "--vfs-case-insensitive" mount flag controls how rclone handles these 3574 two cases. If its value is "false", rclone passes file names to the mounted 3575 file system as is. If the flag is "true" (or appears without a value on 3576 command line), rclone may perform a "fixup" as explained below. 3577 3578 The user may specify a file name to open/delete/rename/etc with a case 3579 different than what is stored on mounted file system. If an argument refers 3580 to an existing file with exactly the same name, then the case of the existing 3581 file on the disk will be used. However, if a file name with exactly the same 3582 name is not found but a name differing only by case exists, rclone will 3583 transparently fixup the name. This fixup happens only when an existing file 3584 is requested. Case sensitivity of file names created anew by rclone is 3585 controlled by an underlying mounted file system. 3586 3587 Note that case sensitivity of the operating system running rclone (the target) 3588 may differ from case sensitivity of a file system mounted by rclone (the source). 3589 The flag controls whether "fixup" is performed to satisfy the target. 3590 3591 If the flag is not provided on command line, then its default value depends 3592 on the operating system where rclone runs: "true" on Windows and macOS, "false" 3593 otherwise. If the flag is provided without a value, then it is "true". 3594 3595 3596 ``` 3597 rclone serve dlna remote:path [flags] 3598 ``` 3599 3600 ## Options 3601 3602 ``` 3603 --addr string ip:port or :port to bind the DLNA http server to. (default ":7879") 3604 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 3605 --dir-perms FileMode Directory permissions (default 0777) 3606 --file-perms FileMode File permissions (default 0666) 3607 --gid uint32 Override the gid field set by the filesystem. (default 1000) 3608 -h, --help help for dlna 3609 --log-trace enable trace logging of SOAP traffic 3610 --name string name of DLNA server 3611 --no-checksum Don't compare checksums on up/download. 3612 --no-modtime Don't read/write the modification time (can speed things up). 3613 --no-seek Don't allow seeking in files. 3614 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 3615 --read-only Mount read-only. 3616 --uid uint32 Override the uid field set by the filesystem. (default 1000) 3617 --umask int Override the permission bits set by the filesystem. (default 2) 3618 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3619 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 3620 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 3621 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3622 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3623 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3624 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3625 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 3626 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 3627 ``` 3628 3629 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3630 3631 ## SEE ALSO 3632 3633 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 3634 3635 # rclone serve ftp 3636 3637 Serve remote:path over FTP. 3638 3639 ## Synopsis 3640 3641 3642 rclone serve ftp implements a basic ftp server to serve the 3643 remote over FTP protocol. This can be viewed with a ftp client 3644 or you can make a remote of type ftp to read and write it. 3645 3646 ## Server options 3647 3648 Use --addr to specify which IP address and port the server should 3649 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3650 IPs. By default it only listens on localhost. You can use port 3651 :0 to let the OS choose an available port. 3652 3653 If you set --addr to listen on a public or LAN accessible IP address 3654 then using Authentication is advised - see the next section for info. 3655 3656 ### Authentication 3657 3658 By default this will serve files without needing a login. 3659 3660 You can set a single username and password with the --user and --pass flags. 3661 3662 ## Directory Cache 3663 3664 Using the `--dir-cache-time` flag, you can set how long a 3665 directory should be considered up to date and not refreshed from the 3666 backend. Changes made locally in the mount may appear immediately or 3667 invalidate the cache. However, changes done on the remote will only 3668 be picked up once the cache expires if the backend configured does not 3669 support polling for changes. If the backend supports polling, changes 3670 will be picked up on within the polling interval. 3671 3672 Alternatively, you can send a `SIGHUP` signal to rclone for 3673 it to flush all directory caches, regardless of how old they are. 3674 Assuming only one rclone instance is running, you can reset the cache 3675 like this: 3676 3677 kill -SIGHUP $(pidof rclone) 3678 3679 If you configure rclone with a [remote control](/rc) then you can use 3680 rclone rc to flush the whole directory cache: 3681 3682 rclone rc vfs/forget 3683 3684 Or individual files or directories: 3685 3686 rclone rc vfs/forget file=path/to/file dir=path/to/dir 3687 3688 ## File Buffering 3689 3690 The `--buffer-size` flag determines the amount of memory, 3691 that will be used to buffer data in advance. 3692 3693 Each open file descriptor will try to keep the specified amount of 3694 data in memory at all times. The buffered data is bound to one file 3695 descriptor and won't be shared between multiple open file descriptors 3696 of the same file. 3697 3698 This flag is a upper limit for the used memory per file descriptor. 3699 The buffer will only use memory for data that is downloaded but not 3700 not yet read. If the buffer is empty, only a small amount of memory 3701 will be used. 3702 The maximum memory used by rclone for buffering can be up to 3703 `--buffer-size * open files`. 3704 3705 ## File Caching 3706 3707 These flags control the VFS file caching options. The VFS layer is 3708 used by rclone mount to make a cloud storage system work more like a 3709 normal file system. 3710 3711 You'll need to enable VFS caching if you want, for example, to read 3712 and write simultaneously to a file. See below for more details. 3713 3714 Note that the VFS cache works in addition to the cache backend and you 3715 may find that you need one or the other or both. 3716 3717 --cache-dir string Directory rclone will use for caching. 3718 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3719 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 3720 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3721 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 3722 3723 If run with `-vv` rclone will print the location of the file cache. The 3724 files are stored in the user cache file area which is OS dependent but 3725 can be controlled with `--cache-dir` or setting the appropriate 3726 environment variable. 3727 3728 The cache has 4 different modes selected by `--vfs-cache-mode`. 3729 The higher the cache mode the more compatible rclone becomes at the 3730 cost of using disk space. 3731 3732 Note that files are written back to the remote only when they are 3733 closed so if rclone is quit or dies with open files then these won't 3734 get written back to the remote. However they will still be in the on 3735 disk cache. 3736 3737 If using --vfs-cache-max-size note that the cache may exceed this size 3738 for two reasons. Firstly because it is only checked every 3739 --vfs-cache-poll-interval. Secondly because open files cannot be 3740 evicted from the cache. 3741 3742 ### --vfs-cache-mode off 3743 3744 In this mode the cache will read directly from the remote and write 3745 directly to the remote without caching anything on disk. 3746 3747 This will mean some operations are not possible 3748 3749 * Files can't be opened for both read AND write 3750 * Files opened for write can't be seeked 3751 * Existing files opened for write must have O_TRUNC set 3752 * Files open for read with O_TRUNC will be opened write only 3753 * Files open for write only will behave as if O_TRUNC was supplied 3754 * Open modes O_APPEND, O_TRUNC are ignored 3755 * If an upload fails it can't be retried 3756 3757 ### --vfs-cache-mode minimal 3758 3759 This is very similar to "off" except that files opened for read AND 3760 write will be buffered to disks. This means that files opened for 3761 write will be a lot more compatible, but uses the minimal disk space. 3762 3763 These operations are not possible 3764 3765 * Files opened for write only can't be seeked 3766 * Existing files opened for write must have O_TRUNC set 3767 * Files opened for write only will ignore O_APPEND, O_TRUNC 3768 * If an upload fails it can't be retried 3769 3770 ### --vfs-cache-mode writes 3771 3772 In this mode files opened for read only are still read directly from 3773 the remote, write only and read/write files are buffered to disk 3774 first. 3775 3776 This mode should support all normal file system operations. 3777 3778 If an upload fails it will be retried up to --low-level-retries times. 3779 3780 ### --vfs-cache-mode full 3781 3782 In this mode all reads and writes are buffered to and from disk. When 3783 a file is opened for read it will be downloaded in its entirety first. 3784 3785 This may be appropriate for your needs, or you may prefer to look at 3786 the cache backend which does a much more sophisticated job of caching, 3787 including caching directory hierarchies and chunks of files. 3788 3789 In this mode, unlike the others, when a file is written to the disk, 3790 it will be kept on the disk after it is written to the remote. It 3791 will be purged on a schedule according to `--vfs-cache-max-age`. 3792 3793 This mode should support all normal file system operations. 3794 3795 If an upload or download fails it will be retried up to 3796 --low-level-retries times. 3797 3798 ## Case Sensitivity 3799 3800 Linux file systems are case-sensitive: two files can differ only 3801 by case, and the exact case must be used when opening a file. 3802 3803 Windows is not like most other operating systems supported by rclone. 3804 File systems in modern Windows are case-insensitive but case-preserving: 3805 although existing files can be opened using any case, the exact case used 3806 to create the file is preserved and available for programs to query. 3807 It is not allowed for two files in the same directory to differ only by case. 3808 3809 Usually file systems on macOS are case-insensitive. It is possible to make macOS 3810 file systems case-sensitive but that is not the default 3811 3812 The "--vfs-case-insensitive" mount flag controls how rclone handles these 3813 two cases. If its value is "false", rclone passes file names to the mounted 3814 file system as is. If the flag is "true" (or appears without a value on 3815 command line), rclone may perform a "fixup" as explained below. 3816 3817 The user may specify a file name to open/delete/rename/etc with a case 3818 different than what is stored on mounted file system. If an argument refers 3819 to an existing file with exactly the same name, then the case of the existing 3820 file on the disk will be used. However, if a file name with exactly the same 3821 name is not found but a name differing only by case exists, rclone will 3822 transparently fixup the name. This fixup happens only when an existing file 3823 is requested. Case sensitivity of file names created anew by rclone is 3824 controlled by an underlying mounted file system. 3825 3826 Note that case sensitivity of the operating system running rclone (the target) 3827 may differ from case sensitivity of a file system mounted by rclone (the source). 3828 The flag controls whether "fixup" is performed to satisfy the target. 3829 3830 If the flag is not provided on command line, then its default value depends 3831 on the operating system where rclone runs: "true" on Windows and macOS, "false" 3832 otherwise. If the flag is provided without a value, then it is "true". 3833 3834 ## Auth Proxy 3835 3836 If you supply the parameter `--auth-proxy /path/to/program` then 3837 rclone will use that program to generate backends on the fly which 3838 then are used to authenticate incoming requests. This uses a simple 3839 JSON based protocl with input on STDIN and output on STDOUT. 3840 3841 **PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used 3842 together, if `--auth-proxy` is set the authorized keys option will be 3843 ignored. 3844 3845 There is an example program 3846 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 3847 in the rclone source code. 3848 3849 The program's job is to take a `user` and `pass` on the input and turn 3850 those into the config for a backend on STDOUT in JSON format. This 3851 config will have any default parameters for the backend added, but it 3852 won't use configuration from environment variables or command line 3853 options - it is the job of the proxy program to make a complete 3854 config. 3855 3856 This config generated must have this extra parameter 3857 - `_root` - root to use for the backend 3858 3859 And it may have this parameter 3860 - `_obscure` - comma separated strings for parameters to obscure 3861 3862 If password authentication was used by the client, input to the proxy 3863 process (on STDIN) would look similar to this: 3864 3865 ``` 3866 { 3867 "user": "me", 3868 "pass": "mypassword" 3869 } 3870 ``` 3871 3872 If public-key authentication was used by the client, input to the 3873 proxy process (on STDIN) would look similar to this: 3874 3875 ``` 3876 { 3877 "user": "me", 3878 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 3879 } 3880 ``` 3881 3882 And as an example return this on STDOUT 3883 3884 ``` 3885 { 3886 "type": "sftp", 3887 "_root": "", 3888 "_obscure": "pass", 3889 "user": "me", 3890 "pass": "mypassword", 3891 "host": "sftp.example.com" 3892 } 3893 ``` 3894 3895 This would mean that an SFTP backend would be created on the fly for 3896 the `user` and `pass`/`public_key` returned in the output to the host given. Note 3897 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 3898 parameter before creating the backend (which is required for sftp 3899 backends). 3900 3901 The program can manipulate the supplied `user` in any way, for example 3902 to make proxy to many different sftp backends, you could make the 3903 `user` be `user@example.com` and then set the `host` to `example.com` 3904 in the output and the user to `user`. For security you'd probably want 3905 to restrict the `host` to a limited list. 3906 3907 Note that an internal cache is keyed on `user` so only use that for 3908 configuration, don't use `pass` or `public_key`. This also means that if a user's 3909 password or public-key is changed the cache will need to expire (which takes 5 mins) 3910 before it takes effect. 3911 3912 This can be used to build general purpose proxies to any kind of 3913 backend that rclone supports. 3914 3915 3916 ``` 3917 rclone serve ftp remote:path [flags] 3918 ``` 3919 3920 ## Options 3921 3922 ``` 3923 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:2121") 3924 --auth-proxy string A program to use to create the backend from the auth. 3925 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 3926 --dir-perms FileMode Directory permissions (default 0777) 3927 --file-perms FileMode File permissions (default 0666) 3928 --gid uint32 Override the gid field set by the filesystem. (default 1000) 3929 -h, --help help for ftp 3930 --no-checksum Don't compare checksums on up/download. 3931 --no-modtime Don't read/write the modification time (can speed things up). 3932 --no-seek Don't allow seeking in files. 3933 --pass string Password for authentication. (empty value allow every password) 3934 --passive-port string Passive port range to use. (default "30000-32000") 3935 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 3936 --public-ip string Public IP address to advertise for passive connections. 3937 --read-only Mount read-only. 3938 --uid uint32 Override the uid field set by the filesystem. (default 1000) 3939 --umask int Override the permission bits set by the filesystem. (default 2) 3940 --user string User name for authentication. (default "anonymous") 3941 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 3942 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 3943 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 3944 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 3945 --vfs-case-insensitive If a file name not found, find a case insensitive match. 3946 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 3947 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 3948 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 3949 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 3950 ``` 3951 3952 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 3953 3954 ## SEE ALSO 3955 3956 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 3957 3958 # rclone serve http 3959 3960 Serve the remote over HTTP. 3961 3962 ## Synopsis 3963 3964 rclone serve http implements a basic web server to serve the remote 3965 over HTTP. This can be viewed in a web browser or you can make a 3966 remote of type http read from it. 3967 3968 You can use the filter flags (eg --include, --exclude) to control what 3969 is served. 3970 3971 The server will log errors. Use -v to see access logs. 3972 3973 --bwlimit will be respected for file transfers. Use --stats to 3974 control the stats printing. 3975 3976 ## Server options 3977 3978 Use --addr to specify which IP address and port the server should 3979 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 3980 IPs. By default it only listens on localhost. You can use port 3981 :0 to let the OS choose an available port. 3982 3983 If you set --addr to listen on a public or LAN accessible IP address 3984 then using Authentication is advised - see the next section for info. 3985 3986 --server-read-timeout and --server-write-timeout can be used to 3987 control the timeouts on the server. Note that this is the total time 3988 for a transfer. 3989 3990 --max-header-bytes controls the maximum number of bytes the server will 3991 accept in the HTTP header. 3992 3993 --baseurl controls the URL prefix that rclone serves from. By default 3994 rclone will serve from the root. If you used --baseurl "/rclone" then 3995 rclone would serve from a URL starting with "/rclone/". This is 3996 useful if you wish to proxy rclone serve. Rclone automatically 3997 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 3998 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 3999 identically. 4000 4001 --template allows a user to specify a custom markup template for http 4002 and webdav serve functions. The server exports the following markup 4003 to be used within the template to server pages: 4004 4005 | Parameter | Description | 4006 | :---------- | :---------- | 4007 | .Name | The full path of a file/directory. | 4008 | .Title | Directory listing of .Name | 4009 | .Sort | The current sort used. This is changeable via ?sort= parameter | 4010 | | Sort Options: namedirfist,name,size,time (default namedirfirst) | 4011 | .Order | The current ordering used. This is changeable via ?order= parameter | 4012 | | Order Options: asc,desc (default asc) | 4013 | .Query | Currently unused. | 4014 | .Breadcrumb | Allows for creating a relative navigation | 4015 |-- .Link | The relative to the root link of the Text. | 4016 |-- .Text | The Name of the directory. | 4017 | .Entries | Information about a specific file/directory. | 4018 |-- .URL | The 'url' of an entry. | 4019 |-- .Leaf | Currently same as 'URL' but intended to be 'just' the name. | 4020 |-- .IsDir | Boolean for if an entry is a directory or not. | 4021 |-- .Size | Size in Bytes of the entry. | 4022 |-- .ModTime | The UTC timestamp of an entry. | 4023 4024 ### Authentication 4025 4026 By default this will serve files without needing a login. 4027 4028 You can either use an htpasswd file which can take lots of users, or 4029 set a single username and password with the --user and --pass flags. 4030 4031 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 4032 in standard apache format and supports MD5, SHA1 and BCrypt for basic 4033 authentication. Bcrypt is recommended. 4034 4035 To create an htpasswd file: 4036 4037 touch htpasswd 4038 htpasswd -B htpasswd user 4039 htpasswd -B htpasswd anotherUser 4040 4041 The password file can be updated while rclone is running. 4042 4043 Use --realm to set the authentication realm. 4044 4045 ### SSL/TLS 4046 4047 By default this will serve over http. If you want you can serve over 4048 https. You will need to supply the --cert and --key flags. If you 4049 wish to do client side certificate validation then you will need to 4050 supply --client-ca also. 4051 4052 --cert should be either a PEM encoded certificate or a concatenation 4053 of that with the CA certificate. --key should be the PEM encoded 4054 private key and --client-ca should be the PEM encoded client 4055 certificate authority certificate. 4056 4057 ## Directory Cache 4058 4059 Using the `--dir-cache-time` flag, you can set how long a 4060 directory should be considered up to date and not refreshed from the 4061 backend. Changes made locally in the mount may appear immediately or 4062 invalidate the cache. However, changes done on the remote will only 4063 be picked up once the cache expires if the backend configured does not 4064 support polling for changes. If the backend supports polling, changes 4065 will be picked up on within the polling interval. 4066 4067 Alternatively, you can send a `SIGHUP` signal to rclone for 4068 it to flush all directory caches, regardless of how old they are. 4069 Assuming only one rclone instance is running, you can reset the cache 4070 like this: 4071 4072 kill -SIGHUP $(pidof rclone) 4073 4074 If you configure rclone with a [remote control](/rc) then you can use 4075 rclone rc to flush the whole directory cache: 4076 4077 rclone rc vfs/forget 4078 4079 Or individual files or directories: 4080 4081 rclone rc vfs/forget file=path/to/file dir=path/to/dir 4082 4083 ## File Buffering 4084 4085 The `--buffer-size` flag determines the amount of memory, 4086 that will be used to buffer data in advance. 4087 4088 Each open file descriptor will try to keep the specified amount of 4089 data in memory at all times. The buffered data is bound to one file 4090 descriptor and won't be shared between multiple open file descriptors 4091 of the same file. 4092 4093 This flag is a upper limit for the used memory per file descriptor. 4094 The buffer will only use memory for data that is downloaded but not 4095 not yet read. If the buffer is empty, only a small amount of memory 4096 will be used. 4097 The maximum memory used by rclone for buffering can be up to 4098 `--buffer-size * open files`. 4099 4100 ## File Caching 4101 4102 These flags control the VFS file caching options. The VFS layer is 4103 used by rclone mount to make a cloud storage system work more like a 4104 normal file system. 4105 4106 You'll need to enable VFS caching if you want, for example, to read 4107 and write simultaneously to a file. See below for more details. 4108 4109 Note that the VFS cache works in addition to the cache backend and you 4110 may find that you need one or the other or both. 4111 4112 --cache-dir string Directory rclone will use for caching. 4113 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4114 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 4115 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4116 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 4117 4118 If run with `-vv` rclone will print the location of the file cache. The 4119 files are stored in the user cache file area which is OS dependent but 4120 can be controlled with `--cache-dir` or setting the appropriate 4121 environment variable. 4122 4123 The cache has 4 different modes selected by `--vfs-cache-mode`. 4124 The higher the cache mode the more compatible rclone becomes at the 4125 cost of using disk space. 4126 4127 Note that files are written back to the remote only when they are 4128 closed so if rclone is quit or dies with open files then these won't 4129 get written back to the remote. However they will still be in the on 4130 disk cache. 4131 4132 If using --vfs-cache-max-size note that the cache may exceed this size 4133 for two reasons. Firstly because it is only checked every 4134 --vfs-cache-poll-interval. Secondly because open files cannot be 4135 evicted from the cache. 4136 4137 ### --vfs-cache-mode off 4138 4139 In this mode the cache will read directly from the remote and write 4140 directly to the remote without caching anything on disk. 4141 4142 This will mean some operations are not possible 4143 4144 * Files can't be opened for both read AND write 4145 * Files opened for write can't be seeked 4146 * Existing files opened for write must have O_TRUNC set 4147 * Files open for read with O_TRUNC will be opened write only 4148 * Files open for write only will behave as if O_TRUNC was supplied 4149 * Open modes O_APPEND, O_TRUNC are ignored 4150 * If an upload fails it can't be retried 4151 4152 ### --vfs-cache-mode minimal 4153 4154 This is very similar to "off" except that files opened for read AND 4155 write will be buffered to disks. This means that files opened for 4156 write will be a lot more compatible, but uses the minimal disk space. 4157 4158 These operations are not possible 4159 4160 * Files opened for write only can't be seeked 4161 * Existing files opened for write must have O_TRUNC set 4162 * Files opened for write only will ignore O_APPEND, O_TRUNC 4163 * If an upload fails it can't be retried 4164 4165 ### --vfs-cache-mode writes 4166 4167 In this mode files opened for read only are still read directly from 4168 the remote, write only and read/write files are buffered to disk 4169 first. 4170 4171 This mode should support all normal file system operations. 4172 4173 If an upload fails it will be retried up to --low-level-retries times. 4174 4175 ### --vfs-cache-mode full 4176 4177 In this mode all reads and writes are buffered to and from disk. When 4178 a file is opened for read it will be downloaded in its entirety first. 4179 4180 This may be appropriate for your needs, or you may prefer to look at 4181 the cache backend which does a much more sophisticated job of caching, 4182 including caching directory hierarchies and chunks of files. 4183 4184 In this mode, unlike the others, when a file is written to the disk, 4185 it will be kept on the disk after it is written to the remote. It 4186 will be purged on a schedule according to `--vfs-cache-max-age`. 4187 4188 This mode should support all normal file system operations. 4189 4190 If an upload or download fails it will be retried up to 4191 --low-level-retries times. 4192 4193 ## Case Sensitivity 4194 4195 Linux file systems are case-sensitive: two files can differ only 4196 by case, and the exact case must be used when opening a file. 4197 4198 Windows is not like most other operating systems supported by rclone. 4199 File systems in modern Windows are case-insensitive but case-preserving: 4200 although existing files can be opened using any case, the exact case used 4201 to create the file is preserved and available for programs to query. 4202 It is not allowed for two files in the same directory to differ only by case. 4203 4204 Usually file systems on macOS are case-insensitive. It is possible to make macOS 4205 file systems case-sensitive but that is not the default 4206 4207 The "--vfs-case-insensitive" mount flag controls how rclone handles these 4208 two cases. If its value is "false", rclone passes file names to the mounted 4209 file system as is. If the flag is "true" (or appears without a value on 4210 command line), rclone may perform a "fixup" as explained below. 4211 4212 The user may specify a file name to open/delete/rename/etc with a case 4213 different than what is stored on mounted file system. If an argument refers 4214 to an existing file with exactly the same name, then the case of the existing 4215 file on the disk will be used. However, if a file name with exactly the same 4216 name is not found but a name differing only by case exists, rclone will 4217 transparently fixup the name. This fixup happens only when an existing file 4218 is requested. Case sensitivity of file names created anew by rclone is 4219 controlled by an underlying mounted file system. 4220 4221 Note that case sensitivity of the operating system running rclone (the target) 4222 may differ from case sensitivity of a file system mounted by rclone (the source). 4223 The flag controls whether "fixup" is performed to satisfy the target. 4224 4225 If the flag is not provided on command line, then its default value depends 4226 on the operating system where rclone runs: "true" on Windows and macOS, "false" 4227 otherwise. If the flag is provided without a value, then it is "true". 4228 4229 4230 ``` 4231 rclone serve http remote:path [flags] 4232 ``` 4233 4234 ## Options 4235 4236 ``` 4237 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 4238 --baseurl string Prefix for URLs - leave blank for root. 4239 --cert string SSL PEM key (concatenation of certificate and CA certificate) 4240 --client-ca string Client certificate authority to verify clients with 4241 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 4242 --dir-perms FileMode Directory permissions (default 0777) 4243 --file-perms FileMode File permissions (default 0666) 4244 --gid uint32 Override the gid field set by the filesystem. (default 1000) 4245 -h, --help help for http 4246 --htpasswd string htpasswd file - if not provided no authentication is done 4247 --key string SSL PEM Private key 4248 --max-header-bytes int Maximum size of request header (default 4096) 4249 --no-checksum Don't compare checksums on up/download. 4250 --no-modtime Don't read/write the modification time (can speed things up). 4251 --no-seek Don't allow seeking in files. 4252 --pass string Password for authentication. 4253 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 4254 --read-only Mount read-only. 4255 --realm string realm for authentication (default "rclone") 4256 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 4257 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 4258 --template string User Specified Template. 4259 --uid uint32 Override the uid field set by the filesystem. (default 1000) 4260 --umask int Override the permission bits set by the filesystem. (default 2) 4261 --user string User name for authentication. 4262 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4263 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 4264 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 4265 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4266 --vfs-case-insensitive If a file name not found, find a case insensitive match. 4267 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 4268 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 4269 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 4270 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 4271 ``` 4272 4273 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4274 4275 ## SEE ALSO 4276 4277 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4278 4279 # rclone serve restic 4280 4281 Serve the remote for restic's REST API. 4282 4283 ## Synopsis 4284 4285 rclone serve restic implements restic's REST backend API 4286 over HTTP. This allows restic to use rclone as a data storage 4287 mechanism for cloud providers that restic does not support directly. 4288 4289 [Restic](https://restic.net/) is a command line program for doing 4290 backups. 4291 4292 The server will log errors. Use -v to see access logs. 4293 4294 --bwlimit will be respected for file transfers. Use --stats to 4295 control the stats printing. 4296 4297 ## Setting up rclone for use by restic ### 4298 4299 First [set up a remote for your chosen cloud provider](https://rclone.org/docs/#configure). 4300 4301 Once you have set up the remote, check it is working with, for example 4302 "rclone lsd remote:". You may have called the remote something other 4303 than "remote:" - just substitute whatever you called it in the 4304 following instructions. 4305 4306 Now start the rclone restic server 4307 4308 rclone serve restic -v remote:backup 4309 4310 Where you can replace "backup" in the above by whatever path in the 4311 remote you wish to use. 4312 4313 By default this will serve on "localhost:8080" you can change this 4314 with use of the "--addr" flag. 4315 4316 You might wish to start this server on boot. 4317 4318 ## Setting up restic to use rclone ### 4319 4320 Now you can [follow the restic 4321 instructions](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) 4322 on setting up restic. 4323 4324 Note that you will need restic 0.8.2 or later to interoperate with 4325 rclone. 4326 4327 For the example above you will want to use "http://localhost:8080/" as 4328 the URL for the REST server. 4329 4330 For example: 4331 4332 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/ 4333 $ export RESTIC_PASSWORD=yourpassword 4334 $ restic init 4335 created restic backend 8b1a4b56ae at rest:http://localhost:8080/ 4336 4337 Please note that knowledge of your password is required to access 4338 the repository. Losing your password means that your data is 4339 irrecoverably lost. 4340 $ restic backup /path/to/files/to/backup 4341 scan [/path/to/files/to/backup] 4342 scanned 189 directories, 312 files in 0:00 4343 [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00 4344 duration: 0:00 4345 snapshot 45c8fdd8 saved 4346 4347 ### Multiple repositories #### 4348 4349 Note that you can use the endpoint to host multiple repositories. Do 4350 this by adding a directory name or path after the URL. Note that 4351 these **must** end with /. Eg 4352 4353 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/ 4354 # backup user1 stuff 4355 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/ 4356 # backup user2 stuff 4357 4358 ### Private repositories #### 4359 4360 The "--private-repos" flag can be used to limit users to repositories starting 4361 with a path of `/<username>/`. 4362 4363 ## Server options 4364 4365 Use --addr to specify which IP address and port the server should 4366 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 4367 IPs. By default it only listens on localhost. You can use port 4368 :0 to let the OS choose an available port. 4369 4370 If you set --addr to listen on a public or LAN accessible IP address 4371 then using Authentication is advised - see the next section for info. 4372 4373 --server-read-timeout and --server-write-timeout can be used to 4374 control the timeouts on the server. Note that this is the total time 4375 for a transfer. 4376 4377 --max-header-bytes controls the maximum number of bytes the server will 4378 accept in the HTTP header. 4379 4380 --baseurl controls the URL prefix that rclone serves from. By default 4381 rclone will serve from the root. If you used --baseurl "/rclone" then 4382 rclone would serve from a URL starting with "/rclone/". This is 4383 useful if you wish to proxy rclone serve. Rclone automatically 4384 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 4385 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 4386 identically. 4387 4388 --template allows a user to specify a custom markup template for http 4389 and webdav serve functions. The server exports the following markup 4390 to be used within the template to server pages: 4391 4392 | Parameter | Description | 4393 | :---------- | :---------- | 4394 | .Name | The full path of a file/directory. | 4395 | .Title | Directory listing of .Name | 4396 | .Sort | The current sort used. This is changeable via ?sort= parameter | 4397 | | Sort Options: namedirfist,name,size,time (default namedirfirst) | 4398 | .Order | The current ordering used. This is changeable via ?order= parameter | 4399 | | Order Options: asc,desc (default asc) | 4400 | .Query | Currently unused. | 4401 | .Breadcrumb | Allows for creating a relative navigation | 4402 |-- .Link | The relative to the root link of the Text. | 4403 |-- .Text | The Name of the directory. | 4404 | .Entries | Information about a specific file/directory. | 4405 |-- .URL | The 'url' of an entry. | 4406 |-- .Leaf | Currently same as 'URL' but intended to be 'just' the name. | 4407 |-- .IsDir | Boolean for if an entry is a directory or not. | 4408 |-- .Size | Size in Bytes of the entry. | 4409 |-- .ModTime | The UTC timestamp of an entry. | 4410 4411 ### Authentication 4412 4413 By default this will serve files without needing a login. 4414 4415 You can either use an htpasswd file which can take lots of users, or 4416 set a single username and password with the --user and --pass flags. 4417 4418 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 4419 in standard apache format and supports MD5, SHA1 and BCrypt for basic 4420 authentication. Bcrypt is recommended. 4421 4422 To create an htpasswd file: 4423 4424 touch htpasswd 4425 htpasswd -B htpasswd user 4426 htpasswd -B htpasswd anotherUser 4427 4428 The password file can be updated while rclone is running. 4429 4430 Use --realm to set the authentication realm. 4431 4432 ### SSL/TLS 4433 4434 By default this will serve over http. If you want you can serve over 4435 https. You will need to supply the --cert and --key flags. If you 4436 wish to do client side certificate validation then you will need to 4437 supply --client-ca also. 4438 4439 --cert should be either a PEM encoded certificate or a concatenation 4440 of that with the CA certificate. --key should be the PEM encoded 4441 private key and --client-ca should be the PEM encoded client 4442 certificate authority certificate. 4443 4444 4445 ``` 4446 rclone serve restic remote:path [flags] 4447 ``` 4448 4449 ## Options 4450 4451 ``` 4452 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 4453 --append-only disallow deletion of repository data 4454 --baseurl string Prefix for URLs - leave blank for root. 4455 --cert string SSL PEM key (concatenation of certificate and CA certificate) 4456 --client-ca string Client certificate authority to verify clients with 4457 -h, --help help for restic 4458 --htpasswd string htpasswd file - if not provided no authentication is done 4459 --key string SSL PEM Private key 4460 --max-header-bytes int Maximum size of request header (default 4096) 4461 --pass string Password for authentication. 4462 --private-repos users can only access their private repo 4463 --realm string realm for authentication (default "rclone") 4464 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 4465 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 4466 --stdio run an HTTP2 server on stdin/stdout 4467 --template string User Specified Template. 4468 --user string User name for authentication. 4469 ``` 4470 4471 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4472 4473 ## SEE ALSO 4474 4475 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4476 4477 # rclone serve sftp 4478 4479 Serve the remote over SFTP. 4480 4481 ## Synopsis 4482 4483 rclone serve sftp implements an SFTP server to serve the remote 4484 over SFTP. This can be used with an SFTP client or you can make a 4485 remote of type sftp to use with it. 4486 4487 You can use the filter flags (eg --include, --exclude) to control what 4488 is served. 4489 4490 The server will log errors. Use -v to see access logs. 4491 4492 --bwlimit will be respected for file transfers. Use --stats to 4493 control the stats printing. 4494 4495 You must provide some means of authentication, either with --user/--pass, 4496 an authorized keys file (specify location with --authorized-keys - the 4497 default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no 4498 authentication when logging in. 4499 4500 Note that this also implements a small number of shell commands so 4501 that it can provide md5sum/sha1sum/df information for the rclone sftp 4502 backend. This means that is can support SHA1SUMs, MD5SUMs and the 4503 about command when paired with the rclone sftp backend. 4504 4505 If you don't supply a --key then rclone will generate one and cache it 4506 for later use. 4507 4508 By default the server binds to localhost:2022 - if you want it to be 4509 reachable externally then supply "--addr :2022" for example. 4510 4511 Note that the default of "--vfs-cache-mode off" is fine for the rclone 4512 sftp backend, but it may not be with other SFTP clients. 4513 4514 4515 ## Directory Cache 4516 4517 Using the `--dir-cache-time` flag, you can set how long a 4518 directory should be considered up to date and not refreshed from the 4519 backend. Changes made locally in the mount may appear immediately or 4520 invalidate the cache. However, changes done on the remote will only 4521 be picked up once the cache expires if the backend configured does not 4522 support polling for changes. If the backend supports polling, changes 4523 will be picked up on within the polling interval. 4524 4525 Alternatively, you can send a `SIGHUP` signal to rclone for 4526 it to flush all directory caches, regardless of how old they are. 4527 Assuming only one rclone instance is running, you can reset the cache 4528 like this: 4529 4530 kill -SIGHUP $(pidof rclone) 4531 4532 If you configure rclone with a [remote control](/rc) then you can use 4533 rclone rc to flush the whole directory cache: 4534 4535 rclone rc vfs/forget 4536 4537 Or individual files or directories: 4538 4539 rclone rc vfs/forget file=path/to/file dir=path/to/dir 4540 4541 ## File Buffering 4542 4543 The `--buffer-size` flag determines the amount of memory, 4544 that will be used to buffer data in advance. 4545 4546 Each open file descriptor will try to keep the specified amount of 4547 data in memory at all times. The buffered data is bound to one file 4548 descriptor and won't be shared between multiple open file descriptors 4549 of the same file. 4550 4551 This flag is a upper limit for the used memory per file descriptor. 4552 The buffer will only use memory for data that is downloaded but not 4553 not yet read. If the buffer is empty, only a small amount of memory 4554 will be used. 4555 The maximum memory used by rclone for buffering can be up to 4556 `--buffer-size * open files`. 4557 4558 ## File Caching 4559 4560 These flags control the VFS file caching options. The VFS layer is 4561 used by rclone mount to make a cloud storage system work more like a 4562 normal file system. 4563 4564 You'll need to enable VFS caching if you want, for example, to read 4565 and write simultaneously to a file. See below for more details. 4566 4567 Note that the VFS cache works in addition to the cache backend and you 4568 may find that you need one or the other or both. 4569 4570 --cache-dir string Directory rclone will use for caching. 4571 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4572 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 4573 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4574 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 4575 4576 If run with `-vv` rclone will print the location of the file cache. The 4577 files are stored in the user cache file area which is OS dependent but 4578 can be controlled with `--cache-dir` or setting the appropriate 4579 environment variable. 4580 4581 The cache has 4 different modes selected by `--vfs-cache-mode`. 4582 The higher the cache mode the more compatible rclone becomes at the 4583 cost of using disk space. 4584 4585 Note that files are written back to the remote only when they are 4586 closed so if rclone is quit or dies with open files then these won't 4587 get written back to the remote. However they will still be in the on 4588 disk cache. 4589 4590 If using --vfs-cache-max-size note that the cache may exceed this size 4591 for two reasons. Firstly because it is only checked every 4592 --vfs-cache-poll-interval. Secondly because open files cannot be 4593 evicted from the cache. 4594 4595 ### --vfs-cache-mode off 4596 4597 In this mode the cache will read directly from the remote and write 4598 directly to the remote without caching anything on disk. 4599 4600 This will mean some operations are not possible 4601 4602 * Files can't be opened for both read AND write 4603 * Files opened for write can't be seeked 4604 * Existing files opened for write must have O_TRUNC set 4605 * Files open for read with O_TRUNC will be opened write only 4606 * Files open for write only will behave as if O_TRUNC was supplied 4607 * Open modes O_APPEND, O_TRUNC are ignored 4608 * If an upload fails it can't be retried 4609 4610 ### --vfs-cache-mode minimal 4611 4612 This is very similar to "off" except that files opened for read AND 4613 write will be buffered to disks. This means that files opened for 4614 write will be a lot more compatible, but uses the minimal disk space. 4615 4616 These operations are not possible 4617 4618 * Files opened for write only can't be seeked 4619 * Existing files opened for write must have O_TRUNC set 4620 * Files opened for write only will ignore O_APPEND, O_TRUNC 4621 * If an upload fails it can't be retried 4622 4623 ### --vfs-cache-mode writes 4624 4625 In this mode files opened for read only are still read directly from 4626 the remote, write only and read/write files are buffered to disk 4627 first. 4628 4629 This mode should support all normal file system operations. 4630 4631 If an upload fails it will be retried up to --low-level-retries times. 4632 4633 ### --vfs-cache-mode full 4634 4635 In this mode all reads and writes are buffered to and from disk. When 4636 a file is opened for read it will be downloaded in its entirety first. 4637 4638 This may be appropriate for your needs, or you may prefer to look at 4639 the cache backend which does a much more sophisticated job of caching, 4640 including caching directory hierarchies and chunks of files. 4641 4642 In this mode, unlike the others, when a file is written to the disk, 4643 it will be kept on the disk after it is written to the remote. It 4644 will be purged on a schedule according to `--vfs-cache-max-age`. 4645 4646 This mode should support all normal file system operations. 4647 4648 If an upload or download fails it will be retried up to 4649 --low-level-retries times. 4650 4651 ## Case Sensitivity 4652 4653 Linux file systems are case-sensitive: two files can differ only 4654 by case, and the exact case must be used when opening a file. 4655 4656 Windows is not like most other operating systems supported by rclone. 4657 File systems in modern Windows are case-insensitive but case-preserving: 4658 although existing files can be opened using any case, the exact case used 4659 to create the file is preserved and available for programs to query. 4660 It is not allowed for two files in the same directory to differ only by case. 4661 4662 Usually file systems on macOS are case-insensitive. It is possible to make macOS 4663 file systems case-sensitive but that is not the default 4664 4665 The "--vfs-case-insensitive" mount flag controls how rclone handles these 4666 two cases. If its value is "false", rclone passes file names to the mounted 4667 file system as is. If the flag is "true" (or appears without a value on 4668 command line), rclone may perform a "fixup" as explained below. 4669 4670 The user may specify a file name to open/delete/rename/etc with a case 4671 different than what is stored on mounted file system. If an argument refers 4672 to an existing file with exactly the same name, then the case of the existing 4673 file on the disk will be used. However, if a file name with exactly the same 4674 name is not found but a name differing only by case exists, rclone will 4675 transparently fixup the name. This fixup happens only when an existing file 4676 is requested. Case sensitivity of file names created anew by rclone is 4677 controlled by an underlying mounted file system. 4678 4679 Note that case sensitivity of the operating system running rclone (the target) 4680 may differ from case sensitivity of a file system mounted by rclone (the source). 4681 The flag controls whether "fixup" is performed to satisfy the target. 4682 4683 If the flag is not provided on command line, then its default value depends 4684 on the operating system where rclone runs: "true" on Windows and macOS, "false" 4685 otherwise. If the flag is provided without a value, then it is "true". 4686 4687 ## Auth Proxy 4688 4689 If you supply the parameter `--auth-proxy /path/to/program` then 4690 rclone will use that program to generate backends on the fly which 4691 then are used to authenticate incoming requests. This uses a simple 4692 JSON based protocl with input on STDIN and output on STDOUT. 4693 4694 **PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used 4695 together, if `--auth-proxy` is set the authorized keys option will be 4696 ignored. 4697 4698 There is an example program 4699 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 4700 in the rclone source code. 4701 4702 The program's job is to take a `user` and `pass` on the input and turn 4703 those into the config for a backend on STDOUT in JSON format. This 4704 config will have any default parameters for the backend added, but it 4705 won't use configuration from environment variables or command line 4706 options - it is the job of the proxy program to make a complete 4707 config. 4708 4709 This config generated must have this extra parameter 4710 - `_root` - root to use for the backend 4711 4712 And it may have this parameter 4713 - `_obscure` - comma separated strings for parameters to obscure 4714 4715 If password authentication was used by the client, input to the proxy 4716 process (on STDIN) would look similar to this: 4717 4718 ``` 4719 { 4720 "user": "me", 4721 "pass": "mypassword" 4722 } 4723 ``` 4724 4725 If public-key authentication was used by the client, input to the 4726 proxy process (on STDIN) would look similar to this: 4727 4728 ``` 4729 { 4730 "user": "me", 4731 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 4732 } 4733 ``` 4734 4735 And as an example return this on STDOUT 4736 4737 ``` 4738 { 4739 "type": "sftp", 4740 "_root": "", 4741 "_obscure": "pass", 4742 "user": "me", 4743 "pass": "mypassword", 4744 "host": "sftp.example.com" 4745 } 4746 ``` 4747 4748 This would mean that an SFTP backend would be created on the fly for 4749 the `user` and `pass`/`public_key` returned in the output to the host given. Note 4750 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 4751 parameter before creating the backend (which is required for sftp 4752 backends). 4753 4754 The program can manipulate the supplied `user` in any way, for example 4755 to make proxy to many different sftp backends, you could make the 4756 `user` be `user@example.com` and then set the `host` to `example.com` 4757 in the output and the user to `user`. For security you'd probably want 4758 to restrict the `host` to a limited list. 4759 4760 Note that an internal cache is keyed on `user` so only use that for 4761 configuration, don't use `pass` or `public_key`. This also means that if a user's 4762 password or public-key is changed the cache will need to expire (which takes 5 mins) 4763 before it takes effect. 4764 4765 This can be used to build general purpose proxies to any kind of 4766 backend that rclone supports. 4767 4768 4769 ``` 4770 rclone serve sftp remote:path [flags] 4771 ``` 4772 4773 ## Options 4774 4775 ``` 4776 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:2022") 4777 --auth-proxy string A program to use to create the backend from the auth. 4778 --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") 4779 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 4780 --dir-perms FileMode Directory permissions (default 0777) 4781 --file-perms FileMode File permissions (default 0666) 4782 --gid uint32 Override the gid field set by the filesystem. (default 1000) 4783 -h, --help help for sftp 4784 --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) 4785 --no-auth Allow connections with no authentication if set. 4786 --no-checksum Don't compare checksums on up/download. 4787 --no-modtime Don't read/write the modification time (can speed things up). 4788 --no-seek Don't allow seeking in files. 4789 --pass string Password for authentication. 4790 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 4791 --read-only Mount read-only. 4792 --uid uint32 Override the uid field set by the filesystem. (default 1000) 4793 --umask int Override the permission bits set by the filesystem. (default 2) 4794 --user string User name for authentication. 4795 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4796 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 4797 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 4798 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4799 --vfs-case-insensitive If a file name not found, find a case insensitive match. 4800 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 4801 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 4802 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 4803 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 4804 ``` 4805 4806 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 4807 4808 ## SEE ALSO 4809 4810 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 4811 4812 # rclone serve webdav 4813 4814 Serve remote:path over webdav. 4815 4816 ## Synopsis 4817 4818 4819 rclone serve webdav implements a basic webdav server to serve the 4820 remote over HTTP via the webdav protocol. This can be viewed with a 4821 webdav client, through a web browser, or you can make a remote of 4822 type webdav to read and write it. 4823 4824 ## Webdav options 4825 4826 ### --etag-hash 4827 4828 This controls the ETag header. Without this flag the ETag will be 4829 based on the ModTime and Size of the object. 4830 4831 If this flag is set to "auto" then rclone will choose the first 4832 supported hash on the backend or you can use a named hash such as 4833 "MD5" or "SHA-1". 4834 4835 Use "rclone hashsum" to see the full list. 4836 4837 4838 ## Server options 4839 4840 Use --addr to specify which IP address and port the server should 4841 listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all 4842 IPs. By default it only listens on localhost. You can use port 4843 :0 to let the OS choose an available port. 4844 4845 If you set --addr to listen on a public or LAN accessible IP address 4846 then using Authentication is advised - see the next section for info. 4847 4848 --server-read-timeout and --server-write-timeout can be used to 4849 control the timeouts on the server. Note that this is the total time 4850 for a transfer. 4851 4852 --max-header-bytes controls the maximum number of bytes the server will 4853 accept in the HTTP header. 4854 4855 --baseurl controls the URL prefix that rclone serves from. By default 4856 rclone will serve from the root. If you used --baseurl "/rclone" then 4857 rclone would serve from a URL starting with "/rclone/". This is 4858 useful if you wish to proxy rclone serve. Rclone automatically 4859 inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", 4860 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 4861 identically. 4862 4863 --template allows a user to specify a custom markup template for http 4864 and webdav serve functions. The server exports the following markup 4865 to be used within the template to server pages: 4866 4867 | Parameter | Description | 4868 | :---------- | :---------- | 4869 | .Name | The full path of a file/directory. | 4870 | .Title | Directory listing of .Name | 4871 | .Sort | The current sort used. This is changeable via ?sort= parameter | 4872 | | Sort Options: namedirfist,name,size,time (default namedirfirst) | 4873 | .Order | The current ordering used. This is changeable via ?order= parameter | 4874 | | Order Options: asc,desc (default asc) | 4875 | .Query | Currently unused. | 4876 | .Breadcrumb | Allows for creating a relative navigation | 4877 |-- .Link | The relative to the root link of the Text. | 4878 |-- .Text | The Name of the directory. | 4879 | .Entries | Information about a specific file/directory. | 4880 |-- .URL | The 'url' of an entry. | 4881 |-- .Leaf | Currently same as 'URL' but intended to be 'just' the name. | 4882 |-- .IsDir | Boolean for if an entry is a directory or not. | 4883 |-- .Size | Size in Bytes of the entry. | 4884 |-- .ModTime | The UTC timestamp of an entry. | 4885 4886 ### Authentication 4887 4888 By default this will serve files without needing a login. 4889 4890 You can either use an htpasswd file which can take lots of users, or 4891 set a single username and password with the --user and --pass flags. 4892 4893 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is 4894 in standard apache format and supports MD5, SHA1 and BCrypt for basic 4895 authentication. Bcrypt is recommended. 4896 4897 To create an htpasswd file: 4898 4899 touch htpasswd 4900 htpasswd -B htpasswd user 4901 htpasswd -B htpasswd anotherUser 4902 4903 The password file can be updated while rclone is running. 4904 4905 Use --realm to set the authentication realm. 4906 4907 ### SSL/TLS 4908 4909 By default this will serve over http. If you want you can serve over 4910 https. You will need to supply the --cert and --key flags. If you 4911 wish to do client side certificate validation then you will need to 4912 supply --client-ca also. 4913 4914 --cert should be either a PEM encoded certificate or a concatenation 4915 of that with the CA certificate. --key should be the PEM encoded 4916 private key and --client-ca should be the PEM encoded client 4917 certificate authority certificate. 4918 4919 ## Directory Cache 4920 4921 Using the `--dir-cache-time` flag, you can set how long a 4922 directory should be considered up to date and not refreshed from the 4923 backend. Changes made locally in the mount may appear immediately or 4924 invalidate the cache. However, changes done on the remote will only 4925 be picked up once the cache expires if the backend configured does not 4926 support polling for changes. If the backend supports polling, changes 4927 will be picked up on within the polling interval. 4928 4929 Alternatively, you can send a `SIGHUP` signal to rclone for 4930 it to flush all directory caches, regardless of how old they are. 4931 Assuming only one rclone instance is running, you can reset the cache 4932 like this: 4933 4934 kill -SIGHUP $(pidof rclone) 4935 4936 If you configure rclone with a [remote control](/rc) then you can use 4937 rclone rc to flush the whole directory cache: 4938 4939 rclone rc vfs/forget 4940 4941 Or individual files or directories: 4942 4943 rclone rc vfs/forget file=path/to/file dir=path/to/dir 4944 4945 ## File Buffering 4946 4947 The `--buffer-size` flag determines the amount of memory, 4948 that will be used to buffer data in advance. 4949 4950 Each open file descriptor will try to keep the specified amount of 4951 data in memory at all times. The buffered data is bound to one file 4952 descriptor and won't be shared between multiple open file descriptors 4953 of the same file. 4954 4955 This flag is a upper limit for the used memory per file descriptor. 4956 The buffer will only use memory for data that is downloaded but not 4957 not yet read. If the buffer is empty, only a small amount of memory 4958 will be used. 4959 The maximum memory used by rclone for buffering can be up to 4960 `--buffer-size * open files`. 4961 4962 ## File Caching 4963 4964 These flags control the VFS file caching options. The VFS layer is 4965 used by rclone mount to make a cloud storage system work more like a 4966 normal file system. 4967 4968 You'll need to enable VFS caching if you want, for example, to read 4969 and write simultaneously to a file. See below for more details. 4970 4971 Note that the VFS cache works in addition to the cache backend and you 4972 may find that you need one or the other or both. 4973 4974 --cache-dir string Directory rclone will use for caching. 4975 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 4976 --vfs-cache-mode string Cache mode off|minimal|writes|full (default "off") 4977 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 4978 --vfs-cache-max-size int Max total size of objects in the cache. (default off) 4979 4980 If run with `-vv` rclone will print the location of the file cache. The 4981 files are stored in the user cache file area which is OS dependent but 4982 can be controlled with `--cache-dir` or setting the appropriate 4983 environment variable. 4984 4985 The cache has 4 different modes selected by `--vfs-cache-mode`. 4986 The higher the cache mode the more compatible rclone becomes at the 4987 cost of using disk space. 4988 4989 Note that files are written back to the remote only when they are 4990 closed so if rclone is quit or dies with open files then these won't 4991 get written back to the remote. However they will still be in the on 4992 disk cache. 4993 4994 If using --vfs-cache-max-size note that the cache may exceed this size 4995 for two reasons. Firstly because it is only checked every 4996 --vfs-cache-poll-interval. Secondly because open files cannot be 4997 evicted from the cache. 4998 4999 ### --vfs-cache-mode off 5000 5001 In this mode the cache will read directly from the remote and write 5002 directly to the remote without caching anything on disk. 5003 5004 This will mean some operations are not possible 5005 5006 * Files can't be opened for both read AND write 5007 * Files opened for write can't be seeked 5008 * Existing files opened for write must have O_TRUNC set 5009 * Files open for read with O_TRUNC will be opened write only 5010 * Files open for write only will behave as if O_TRUNC was supplied 5011 * Open modes O_APPEND, O_TRUNC are ignored 5012 * If an upload fails it can't be retried 5013 5014 ### --vfs-cache-mode minimal 5015 5016 This is very similar to "off" except that files opened for read AND 5017 write will be buffered to disks. This means that files opened for 5018 write will be a lot more compatible, but uses the minimal disk space. 5019 5020 These operations are not possible 5021 5022 * Files opened for write only can't be seeked 5023 * Existing files opened for write must have O_TRUNC set 5024 * Files opened for write only will ignore O_APPEND, O_TRUNC 5025 * If an upload fails it can't be retried 5026 5027 ### --vfs-cache-mode writes 5028 5029 In this mode files opened for read only are still read directly from 5030 the remote, write only and read/write files are buffered to disk 5031 first. 5032 5033 This mode should support all normal file system operations. 5034 5035 If an upload fails it will be retried up to --low-level-retries times. 5036 5037 ### --vfs-cache-mode full 5038 5039 In this mode all reads and writes are buffered to and from disk. When 5040 a file is opened for read it will be downloaded in its entirety first. 5041 5042 This may be appropriate for your needs, or you may prefer to look at 5043 the cache backend which does a much more sophisticated job of caching, 5044 including caching directory hierarchies and chunks of files. 5045 5046 In this mode, unlike the others, when a file is written to the disk, 5047 it will be kept on the disk after it is written to the remote. It 5048 will be purged on a schedule according to `--vfs-cache-max-age`. 5049 5050 This mode should support all normal file system operations. 5051 5052 If an upload or download fails it will be retried up to 5053 --low-level-retries times. 5054 5055 ## Case Sensitivity 5056 5057 Linux file systems are case-sensitive: two files can differ only 5058 by case, and the exact case must be used when opening a file. 5059 5060 Windows is not like most other operating systems supported by rclone. 5061 File systems in modern Windows are case-insensitive but case-preserving: 5062 although existing files can be opened using any case, the exact case used 5063 to create the file is preserved and available for programs to query. 5064 It is not allowed for two files in the same directory to differ only by case. 5065 5066 Usually file systems on macOS are case-insensitive. It is possible to make macOS 5067 file systems case-sensitive but that is not the default 5068 5069 The "--vfs-case-insensitive" mount flag controls how rclone handles these 5070 two cases. If its value is "false", rclone passes file names to the mounted 5071 file system as is. If the flag is "true" (or appears without a value on 5072 command line), rclone may perform a "fixup" as explained below. 5073 5074 The user may specify a file name to open/delete/rename/etc with a case 5075 different than what is stored on mounted file system. If an argument refers 5076 to an existing file with exactly the same name, then the case of the existing 5077 file on the disk will be used. However, if a file name with exactly the same 5078 name is not found but a name differing only by case exists, rclone will 5079 transparently fixup the name. This fixup happens only when an existing file 5080 is requested. Case sensitivity of file names created anew by rclone is 5081 controlled by an underlying mounted file system. 5082 5083 Note that case sensitivity of the operating system running rclone (the target) 5084 may differ from case sensitivity of a file system mounted by rclone (the source). 5085 The flag controls whether "fixup" is performed to satisfy the target. 5086 5087 If the flag is not provided on command line, then its default value depends 5088 on the operating system where rclone runs: "true" on Windows and macOS, "false" 5089 otherwise. If the flag is provided without a value, then it is "true". 5090 5091 ## Auth Proxy 5092 5093 If you supply the parameter `--auth-proxy /path/to/program` then 5094 rclone will use that program to generate backends on the fly which 5095 then are used to authenticate incoming requests. This uses a simple 5096 JSON based protocl with input on STDIN and output on STDOUT. 5097 5098 **PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used 5099 together, if `--auth-proxy` is set the authorized keys option will be 5100 ignored. 5101 5102 There is an example program 5103 [bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py) 5104 in the rclone source code. 5105 5106 The program's job is to take a `user` and `pass` on the input and turn 5107 those into the config for a backend on STDOUT in JSON format. This 5108 config will have any default parameters for the backend added, but it 5109 won't use configuration from environment variables or command line 5110 options - it is the job of the proxy program to make a complete 5111 config. 5112 5113 This config generated must have this extra parameter 5114 - `_root` - root to use for the backend 5115 5116 And it may have this parameter 5117 - `_obscure` - comma separated strings for parameters to obscure 5118 5119 If password authentication was used by the client, input to the proxy 5120 process (on STDIN) would look similar to this: 5121 5122 ``` 5123 { 5124 "user": "me", 5125 "pass": "mypassword" 5126 } 5127 ``` 5128 5129 If public-key authentication was used by the client, input to the 5130 proxy process (on STDIN) would look similar to this: 5131 5132 ``` 5133 { 5134 "user": "me", 5135 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 5136 } 5137 ``` 5138 5139 And as an example return this on STDOUT 5140 5141 ``` 5142 { 5143 "type": "sftp", 5144 "_root": "", 5145 "_obscure": "pass", 5146 "user": "me", 5147 "pass": "mypassword", 5148 "host": "sftp.example.com" 5149 } 5150 ``` 5151 5152 This would mean that an SFTP backend would be created on the fly for 5153 the `user` and `pass`/`public_key` returned in the output to the host given. Note 5154 that since `_obscure` is set to `pass`, rclone will obscure the `pass` 5155 parameter before creating the backend (which is required for sftp 5156 backends). 5157 5158 The program can manipulate the supplied `user` in any way, for example 5159 to make proxy to many different sftp backends, you could make the 5160 `user` be `user@example.com` and then set the `host` to `example.com` 5161 in the output and the user to `user`. For security you'd probably want 5162 to restrict the `host` to a limited list. 5163 5164 Note that an internal cache is keyed on `user` so only use that for 5165 configuration, don't use `pass` or `public_key`. This also means that if a user's 5166 password or public-key is changed the cache will need to expire (which takes 5 mins) 5167 before it takes effect. 5168 5169 This can be used to build general purpose proxies to any kind of 5170 backend that rclone supports. 5171 5172 5173 ``` 5174 rclone serve webdav remote:path [flags] 5175 ``` 5176 5177 ## Options 5178 5179 ``` 5180 --addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080") 5181 --auth-proxy string A program to use to create the backend from the auth. 5182 --baseurl string Prefix for URLs - leave blank for root. 5183 --cert string SSL PEM key (concatenation of certificate and CA certificate) 5184 --client-ca string Client certificate authority to verify clients with 5185 --dir-cache-time duration Time to cache directory entries for. (default 5m0s) 5186 --dir-perms FileMode Directory permissions (default 0777) 5187 --disable-dir-list Disable HTML directory list on GET request for a directory 5188 --etag-hash string Which hash to use for the ETag, or auto or blank for off 5189 --file-perms FileMode File permissions (default 0666) 5190 --gid uint32 Override the gid field set by the filesystem. (default 1000) 5191 -h, --help help for webdav 5192 --htpasswd string htpasswd file - if not provided no authentication is done 5193 --key string SSL PEM Private key 5194 --max-header-bytes int Maximum size of request header (default 4096) 5195 --no-checksum Don't compare checksums on up/download. 5196 --no-modtime Don't read/write the modification time (can speed things up). 5197 --no-seek Don't allow seeking in files. 5198 --pass string Password for authentication. 5199 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s) 5200 --read-only Mount read-only. 5201 --realm string realm for authentication (default "rclone") 5202 --server-read-timeout duration Timeout for server reading data (default 1h0m0s) 5203 --server-write-timeout duration Timeout for server writing data (default 1h0m0s) 5204 --template string User Specified Template. 5205 --uid uint32 Override the uid field set by the filesystem. (default 1000) 5206 --umask int Override the permission bits set by the filesystem. (default 2) 5207 --user string User name for authentication. 5208 --vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s) 5209 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off) 5210 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 5211 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s) 5212 --vfs-case-insensitive If a file name not found, find a case insensitive match. 5213 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M) 5214 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off) 5215 --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) 5216 --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) 5217 ``` 5218 5219 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 5220 5221 ## SEE ALSO 5222 5223 * [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol. 5224 5225 # rclone settier 5226 5227 Changes storage class/tier of objects in remote. 5228 5229 ## Synopsis 5230 5231 5232 rclone settier changes storage tier or class at remote if supported. 5233 Few cloud storage services provides different storage classes on objects, 5234 for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, 5235 Google Cloud Storage, Regional Storage, Nearline, Coldline etc. 5236 5237 Note that, certain tier changes make objects not available to access immediately. 5238 For example tiering to archive in azure blob storage makes objects in frozen state, 5239 user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object 5240 inaccessible.true 5241 5242 You can use it to tier single object 5243 5244 rclone settier Cool remote:path/file 5245 5246 Or use rclone filters to set tier on only specific files 5247 5248 rclone --include "*.txt" settier Hot remote:path/dir 5249 5250 Or just provide remote directory and all files in directory will be tiered 5251 5252 rclone settier tier remote:path/dir 5253 5254 5255 ``` 5256 rclone settier tier remote:path [flags] 5257 ``` 5258 5259 ## Options 5260 5261 ``` 5262 -h, --help help for settier 5263 ``` 5264 5265 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 5266 5267 ## SEE ALSO 5268 5269 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 5270 5271 # rclone touch 5272 5273 Create new file or change file modification time. 5274 5275 ## Synopsis 5276 5277 5278 Set the modification time on object(s) as specified by remote:path to 5279 have the current time. 5280 5281 If remote:path does not exist then a zero sized object will be created 5282 unless the --no-create flag is provided. 5283 5284 If --timestamp is used then it will set the modification time to that 5285 time instead of the current time. Times may be specified as one of: 5286 5287 - 'YYMMDD' - eg. 17.10.30 5288 - 'YYYY-MM-DDTHH:MM:SS' - eg. 2006-01-02T15:04:05 5289 5290 Note that --timestamp is in UTC if you want local time then add the 5291 --localtime flag. 5292 5293 5294 ``` 5295 rclone touch remote:path [flags] 5296 ``` 5297 5298 ## Options 5299 5300 ``` 5301 -h, --help help for touch 5302 --localtime Use localtime for timestamp, not UTC. 5303 -C, --no-create Do not create the file if it does not exist. 5304 -t, --timestamp string Use specified time instead of the current time of day. 5305 ``` 5306 5307 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 5308 5309 ## SEE ALSO 5310 5311 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 5312 5313 # rclone tree 5314 5315 List the contents of the remote in a tree like fashion. 5316 5317 ## Synopsis 5318 5319 5320 rclone tree lists the contents of a remote in a similar way to the 5321 unix tree command. 5322 5323 For example 5324 5325 $ rclone tree remote:path 5326 / 5327 ├── file1 5328 ├── file2 5329 ├── file3 5330 └── subdir 5331 ├── file4 5332 └── file5 5333 5334 1 directories, 5 files 5335 5336 You can use any of the filtering options with the tree command (eg 5337 --include and --exclude). You can also use --fast-list. 5338 5339 The tree command has many options for controlling the listing which 5340 are compatible with the tree command. Note that not all of them have 5341 short options as they conflict with rclone's short options. 5342 5343 5344 ``` 5345 rclone tree remote:path [flags] 5346 ``` 5347 5348 ## Options 5349 5350 ``` 5351 -a, --all All files are listed (list . files too). 5352 -C, --color Turn colorization on always. 5353 -d, --dirs-only List directories only. 5354 --dirsfirst List directories before files (-U disables). 5355 --full-path Print the full path prefix for each file. 5356 -h, --help help for tree 5357 --human Print the size in a more human readable way. 5358 --level int Descend only level directories deep. 5359 -D, --modtime Print the date of last modification. 5360 -i, --noindent Don't print indentation lines. 5361 --noreport Turn off file/directory count at end of tree listing. 5362 -o, --output string Output to file instead of stdout. 5363 -p, --protections Print the protections for each file. 5364 -Q, --quote Quote filenames with double quotes. 5365 -s, --size Print the size in bytes of each file. 5366 --sort string Select sort: name,version,size,mtime,ctime. 5367 --sort-ctime Sort files by last status change time. 5368 -t, --sort-modtime Sort files by last modification time. 5369 -r, --sort-reverse Reverse the order of the sort. 5370 -U, --unsorted Leave files unsorted. 5371 --version Sort files alphanumerically by version. 5372 ``` 5373 5374 See the [global flags page](https://rclone.org/flags/) for global options not listed here. 5375 5376 ## SEE ALSO 5377 5378 * [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends. 5379 5380 5381 Copying single files 5382 -------------------- 5383 5384 rclone normally syncs or copies directories. However, if the source 5385 remote points to a file, rclone will just copy that file. The 5386 destination remote must point to a directory - rclone will give the 5387 error `Failed to create file system for "remote:file": is a file not a 5388 directory` if it isn't. 5389 5390 For example, suppose you have a remote with a file in called 5391 `test.jpg`, then you could copy just that file like this 5392 5393 rclone copy remote:test.jpg /tmp/download 5394 5395 The file `test.jpg` will be placed inside `/tmp/download`. 5396 5397 This is equivalent to specifying 5398 5399 rclone copy --files-from /tmp/files remote: /tmp/download 5400 5401 Where `/tmp/files` contains the single line 5402 5403 test.jpg 5404 5405 It is recommended to use `copy` when copying individual files, not `sync`. 5406 They have pretty much the same effect but `copy` will use a lot less 5407 memory. 5408 5409 Syntax of remote paths 5410 ---------------------- 5411 5412 The syntax of the paths passed to the rclone command are as follows. 5413 5414 ### /path/to/dir 5415 5416 This refers to the local file system. 5417 5418 On Windows only `\` may be used instead of `/` in local paths 5419 **only**, non local paths must use `/`. 5420 5421 These paths needn't start with a leading `/` - if they don't then they 5422 will be relative to the current directory. 5423 5424 ### remote:path/to/dir 5425 5426 This refers to a directory `path/to/dir` on `remote:` as defined in 5427 the config file (configured with `rclone config`). 5428 5429 ### remote:/path/to/dir 5430 5431 On most backends this is refers to the same directory as 5432 `remote:path/to/dir` and that format should be preferred. On a very 5433 small number of remotes (FTP, SFTP, Dropbox for business) this will 5434 refer to a different directory. On these, paths without a leading `/` 5435 will refer to your "home" directory and paths with a leading `/` will 5436 refer to the root. 5437 5438 ### :backend:path/to/dir 5439 5440 This is an advanced form for creating remotes on the fly. `backend` 5441 should be the name or prefix of a backend (the `type` in the config 5442 file) and all the configuration for the backend should be provided on 5443 the command line (or in environment variables). 5444 5445 Here are some examples: 5446 5447 rclone lsd --http-url https://pub.rclone.org :http: 5448 5449 To list all the directories in the root of `https://pub.rclone.org/`. 5450 5451 rclone lsf --http-url https://example.com :http:path/to/dir 5452 5453 To list files and directories in `https://example.com/path/to/dir/` 5454 5455 rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir 5456 5457 To copy files and directories in `https://example.com/path/to/dir` to `/tmp/dir`. 5458 5459 rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir 5460 5461 To copy files and directories from `example.com` in the relative 5462 directory `path/to/dir` to `/tmp/dir` using sftp. 5463 5464 Quoting and the shell 5465 --------------------- 5466 5467 When you are typing commands to your computer you are using something 5468 called the command line shell. This interprets various characters in 5469 an OS specific way. 5470 5471 Here are some gotchas which may help users unfamiliar with the shell rules 5472 5473 ### Linux / OSX ### 5474 5475 If your names have spaces or shell metacharacters (eg `*`, `?`, `$`, 5476 `'`, `"` etc) then you must quote them. Use single quotes `'` by default. 5477 5478 rclone copy 'Important files?' remote:backup 5479 5480 If you want to send a `'` you will need to use `"`, eg 5481 5482 rclone copy "O'Reilly Reviews" remote:backup 5483 5484 The rules for quoting metacharacters are complicated and if you want 5485 the full details you'll have to consult the manual page for your 5486 shell. 5487 5488 ### Windows ### 5489 5490 If your names have spaces in you need to put them in `"`, eg 5491 5492 rclone copy "E:\folder name\folder name\folder name" remote:backup 5493 5494 If you are using the root directory on its own then don't quote it 5495 (see [#464](https://github.com/rclone/rclone/issues/464) for why), eg 5496 5497 rclone copy E:\ remote:backup 5498 5499 Copying files or directories with `:` in the names 5500 -------------------------------------------------- 5501 5502 rclone uses `:` to mark a remote name. This is, however, a valid 5503 filename component in non-Windows OSes. The remote name parser will 5504 only search for a `:` up to the first `/` so if you need to act on a 5505 file or directory like this then use the full path starting with a 5506 `/`, or use `./` as a current directory prefix. 5507 5508 So to sync a directory called `sync:me` to a remote called `remote:` use 5509 5510 rclone sync ./sync:me remote:path 5511 5512 or 5513 5514 rclone sync /full/path/to/sync:me remote:path 5515 5516 Server Side Copy 5517 ---------------- 5518 5519 Most remotes (but not all - see [the 5520 overview](https://rclone.org/overview/#optional-features)) support server side copy. 5521 5522 This means if you want to copy one folder to another then rclone won't 5523 download all the files and re-upload them; it will instruct the server 5524 to copy them in place. 5525 5526 Eg 5527 5528 rclone copy s3:oldbucket s3:newbucket 5529 5530 Will copy the contents of `oldbucket` to `newbucket` without 5531 downloading and re-uploading. 5532 5533 Remotes which don't support server side copy **will** download and 5534 re-upload in this case. 5535 5536 Server side copies are used with `sync` and `copy` and will be 5537 identified in the log when using the `-v` flag. The `move` command 5538 may also use them if remote doesn't support server side move directly. 5539 This is done by issuing a server side copy then a delete which is much 5540 quicker than a download and re-upload. 5541 5542 Server side copies will only be attempted if the remote names are the 5543 same. 5544 5545 This can be used when scripting to make aged backups efficiently, eg 5546 5547 rclone sync remote:current-backup remote:previous-backup 5548 rclone sync /path/to/files remote:current-backup 5549 5550 Options 5551 ------- 5552 5553 Rclone has a number of options to control its behaviour. 5554 5555 Options that take parameters can have the values passed in two ways, 5556 `--option=value` or `--option value`. However boolean (true/false) 5557 options behave slightly differently to the other options in that 5558 `--boolean` sets the option to `true` and the absence of the flag sets 5559 it to `false`. It is also possible to specify `--boolean=false` or 5560 `--boolean=true`. Note that `--boolean false` is not valid - this is 5561 parsed as `--boolean` and the `false` is parsed as an extra command 5562 line argument for rclone. 5563 5564 Options which use TIME use the go time parser. A duration string is a 5565 possibly signed sequence of decimal numbers, each with optional 5566 fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid 5567 time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". 5568 5569 Options which use SIZE use kByte by default. However, a suffix of `b` 5570 for bytes, `k` for kBytes, `M` for MBytes, `G` for GBytes, `T` for 5571 TBytes and `P` for PBytes may be used. These are the binary units, eg 5572 1, 2\*\*10, 2\*\*20, 2\*\*30 respectively. 5573 5574 ### --backup-dir=DIR ### 5575 5576 When using `sync`, `copy` or `move` any files which would have been 5577 overwritten or deleted are moved in their original hierarchy into this 5578 directory. 5579 5580 If `--suffix` is set, then the moved files will have the suffix added 5581 to them. If there is a file with the same path (after the suffix has 5582 been added) in DIR, then it will be overwritten. 5583 5584 The remote in use must support server side move or copy and you must 5585 use the same remote as the destination of the sync. The backup 5586 directory must not overlap the destination directory. 5587 5588 For example 5589 5590 rclone sync /path/to/local remote:current --backup-dir remote:old 5591 5592 will sync `/path/to/local` to `remote:current`, but for any files 5593 which would have been updated or deleted will be stored in 5594 `remote:old`. 5595 5596 If running rclone from a script you might want to use today's date as 5597 the directory name passed to `--backup-dir` to store the old files, or 5598 you might want to pass `--suffix` with today's date. 5599 5600 See `--compare-dest` and `--copy-dest`. 5601 5602 ### --bind string ### 5603 5604 Local address to bind to for outgoing connections. This can be an 5605 IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or host name. If 5606 the host name doesn't resolve or resolves to more than one IP address 5607 it will give an error. 5608 5609 ### --bwlimit=BANDWIDTH_SPEC ### 5610 5611 This option controls the bandwidth limit. Limits can be specified 5612 in two ways: As a single limit, or as a timetable. 5613 5614 Single limits last for the duration of the session. To use a single limit, 5615 specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The 5616 default is `0` which means to not limit bandwidth. 5617 5618 For example, to limit bandwidth usage to 10 MBytes/s use `--bwlimit 10M` 5619 5620 It is also possible to specify a "timetable" of limits, which will cause 5621 certain limits to be applied at certain times. To specify a timetable, format your 5622 entries as `WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH...` where: 5623 `WEEKDAY` is optional element. 5624 It could be written as whole world or only using 3 first characters. 5625 `HH:MM` is an hour from 00:00 to 23:59. 5626 5627 An example of a typical timetable to avoid link saturation during daytime 5628 working hours could be: 5629 5630 `--bwlimit "08:00,512 12:00,10M 13:00,512 18:00,30M 23:00,off"` 5631 5632 In this example, the transfer bandwidth will be every day set to 512kBytes/sec at 8am. 5633 At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. 5634 At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be 5635 completely disabled (full speed). Anything between 11pm and 8am will remain 5636 unlimited. 5637 5638 An example of timetable with `WEEKDAY` could be: 5639 5640 `--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"` 5641 5642 It mean that, the transfer bandwidth will be set to 512kBytes/sec on Monday. 5643 It will raise to 10Mbytes/s before the end of Friday. 5644 At 10:00 on Sunday it will be set to 1Mbyte/s. 5645 From 20:00 at Sunday will be unlimited. 5646 5647 Timeslots without weekday are extended to whole week. 5648 So this one example: 5649 5650 `--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"` 5651 5652 Is equal to this: 5653 5654 `--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"` 5655 5656 Bandwidth limits only apply to the data transfer. They don't apply to the 5657 bandwidth of the directory listings etc. 5658 5659 Note that the units are Bytes/s, not Bits/s. Typically connections are 5660 measured in Bits/s - to convert divide by 8. For example, let's say 5661 you have a 10 Mbit/s connection and you wish rclone to use half of it 5662 - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a `--bwlimit 5663 0.625M` parameter for rclone. 5664 5665 On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled by 5666 sending a `SIGUSR2` signal to rclone. This allows to remove the limitations 5667 of a long running rclone transfer and to restore it back to the value specified 5668 with `--bwlimit` quickly when needed. Assuming there is only one rclone instance 5669 running, you can toggle the limiter like this: 5670 5671 kill -SIGUSR2 $(pidof rclone) 5672 5673 If you configure rclone with a [remote control](/rc) then you can use 5674 change the bwlimit dynamically: 5675 5676 rclone rc core/bwlimit rate=1M 5677 5678 ### --buffer-size=SIZE ### 5679 5680 Use this sized buffer to speed up file transfers. Each `--transfer` 5681 will use this much memory for buffering. 5682 5683 When using `mount` or `cmount` each open file descriptor will use this much 5684 memory for buffering. 5685 See the [mount](https://rclone.org/commands/rclone_mount/#file-buffering) documentation for more details. 5686 5687 Set to `0` to disable the buffering for the minimum memory usage. 5688 5689 Note that the memory allocation of the buffers is influenced by the 5690 [--use-mmap](#use-mmap) flag. 5691 5692 ### --check-first ### 5693 5694 If this flag is set then in a `sync`, `copy` or `move`, rclone will do 5695 all the checks to see whether files need to be transferred before 5696 doing any of the transfers. Normally rclone would start running 5697 transfers as soon as possible. 5698 5699 This flag can be useful on IO limited systems where transfers 5700 interfere with checking. 5701 5702 Using this flag can use more memory as it effectively sets 5703 `--max-backlog` to infinite. This means that all the info on the 5704 objects to transfer is held in memory before the transfers start. 5705 5706 ### --checkers=N ### 5707 5708 The number of checkers to run in parallel. Checkers do the equality 5709 checking of files during a sync. For some storage systems (eg S3, 5710 Swift, Dropbox) this can take a significant amount of time so they are 5711 run in parallel. 5712 5713 The default is to run 8 checkers in parallel. 5714 5715 ### -c, --checksum ### 5716 5717 Normally rclone will look at modification time and size of files to 5718 see if they are equal. If you set this flag then rclone will check 5719 the file hash and size to determine if files are equal. 5720 5721 This is useful when the remote doesn't support setting modified time 5722 and a more accurate sync is desired than just checking the file size. 5723 5724 This is very useful when transferring between remotes which store the 5725 same hash type on the object, eg Drive and Swift. For details of which 5726 remotes support which hash type see the table in the [overview 5727 section](https://rclone.org/overview/). 5728 5729 Eg `rclone --checksum sync s3:/bucket swift:/bucket` would run much 5730 quicker than without the `--checksum` flag. 5731 5732 When using this flag, rclone won't update mtimes of remote files if 5733 they are incorrect as it would normally. 5734 5735 ### --compare-dest=DIR ### 5736 5737 When using `sync`, `copy` or `move` DIR is checked in addition to the 5738 destination for files. If a file identical to the source is found that 5739 file is NOT copied from source. This is useful to copy just files that 5740 have changed since the last backup. 5741 5742 You must use the same remote as the destination of the sync. The 5743 compare directory must not overlap the destination directory. 5744 5745 See `--copy-dest` and `--backup-dir`. 5746 5747 ### --config=CONFIG_FILE ### 5748 5749 Specify the location of the rclone config file. 5750 5751 Normally the config file is in your home directory as a file called 5752 `.config/rclone/rclone.conf` (or `.rclone.conf` if created with an 5753 older version). If `$XDG_CONFIG_HOME` is set it will be at 5754 `$XDG_CONFIG_HOME/rclone/rclone.conf`. 5755 5756 If there is a file `rclone.conf` in the same directory as the rclone 5757 executable it will be preferred. This file must be created manually 5758 for Rclone to use it, it will never be created automatically. 5759 5760 If you run `rclone config file` you will see where the default 5761 location is for you. 5762 5763 Use this flag to override the config location, eg `rclone 5764 --config=".myconfig" .config`. 5765 5766 ### --contimeout=TIME ### 5767 5768 Set the connection timeout. This should be in go time format which 5769 looks like `5s` for 5 seconds, `10m` for 10 minutes, or `3h30m`. 5770 5771 The connection timeout is the amount of time rclone will wait for a 5772 connection to go through to a remote object storage system. It is 5773 `1m` by default. 5774 5775 ### --copy-dest=DIR ### 5776 5777 When using `sync`, `copy` or `move` DIR is checked in addition to the 5778 destination for files. If a file identical to the source is found that 5779 file is server side copied from DIR to the destination. This is useful 5780 for incremental backup. 5781 5782 The remote in use must support server side copy and you must 5783 use the same remote as the destination of the sync. The compare 5784 directory must not overlap the destination directory. 5785 5786 See `--compare-dest` and `--backup-dir`. 5787 5788 ### --dedupe-mode MODE ### 5789 5790 Mode to run dedupe command in. One of `interactive`, `skip`, `first`, `newest`, `oldest`, `rename`. The default is `interactive`. See the dedupe command for more information as to what these options mean. 5791 5792 ### --disable FEATURE,FEATURE,... ### 5793 5794 This disables a comma separated list of optional features. For example 5795 to disable server side move and server side copy use: 5796 5797 --disable move,copy 5798 5799 The features can be put in any case. 5800 5801 To see a list of which features can be disabled use: 5802 5803 --disable help 5804 5805 See the overview [features](https://rclone.org/overview/#features) and 5806 [optional features](https://rclone.org/overview/#optional-features) to get an idea of 5807 which feature does what. 5808 5809 This flag can be useful for debugging and in exceptional circumstances 5810 (eg Google Drive limiting the total volume of Server Side Copies to 5811 100GB/day). 5812 5813 ### -n, --dry-run ### 5814 5815 Do a trial run with no permanent changes. Use this to see what rclone 5816 would do without actually doing it. Useful when setting up the `sync` 5817 command which deletes files in the destination. 5818 5819 ### --expect-continue-timeout=TIME ### 5820 5821 This specifies the amount of time to wait for a server's first 5822 response headers after fully writing the request headers if the 5823 request has an "Expect: 100-continue" header. Not all backends support 5824 using this. 5825 5826 Zero means no timeout and causes the body to be sent immediately, 5827 without waiting for the server to approve. This time does not include 5828 the time to send the request header. 5829 5830 The default is `1s`. Set to `0` to disable. 5831 5832 ### --error-on-no-transfer ### 5833 5834 By default, rclone will exit with return code 0 if there were no errors. 5835 5836 This option allows rclone to return exit code 9 if no files were transferred 5837 between the source and destination. This allows using rclone in scripts, and 5838 triggering follow-on actions if data was copied, or skipping if not. 5839 5840 NB: Enabling this option turns a usually non-fatal error into a potentially 5841 fatal one - please check and adjust your scripts accordingly! 5842 5843 ### --header ### 5844 5845 Add an HTTP header for all transactions. The flag can be repeated to 5846 add multiple headers. 5847 5848 If you want to add headers only for uploads use `--header-upload` and 5849 if you want to add headers only for downloads use `--header-download`. 5850 5851 This flag is supported for all HTTP based backends even those not 5852 supported by `--header-upload` and `--header-download` so may be used 5853 as a workaround for those with care. 5854 5855 ``` 5856 rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes" 5857 ``` 5858 5859 ### --header-download ### 5860 5861 Add an HTTP header for all download transactions. The flag can be repeated to 5862 add multiple headers. 5863 5864 ``` 5865 rclone sync s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar" 5866 ``` 5867 5868 See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for 5869 currently supported backends. 5870 5871 ### --header-upload ### 5872 5873 Add an HTTP header for all upload transactions. The flag can be repeated to add 5874 multiple headers. 5875 5876 ``` 5877 rclone sync ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar" 5878 ``` 5879 5880 See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for 5881 currently supported backends. 5882 5883 ### --ignore-case-sync ### 5884 5885 Using this option will cause rclone to ignore the case of the files 5886 when synchronizing so files will not be copied/synced when the 5887 existing filenames are the same, even if the casing is different. 5888 5889 ### --ignore-checksum ### 5890 5891 Normally rclone will check that the checksums of transferred files 5892 match, and give an error "corrupted on transfer" if they don't. 5893 5894 You can use this option to skip that check. You should only use it if 5895 you have had the "corrupted on transfer" error message and you are 5896 sure you might want to transfer potentially corrupted data. 5897 5898 ### --ignore-existing ### 5899 5900 Using this option will make rclone unconditionally skip all files 5901 that exist on the destination, no matter the content of these files. 5902 5903 While this isn't a generally recommended option, it can be useful 5904 in cases where your files change due to encryption. However, it cannot 5905 correct partial transfers in case a transfer was interrupted. 5906 5907 ### --ignore-size ### 5908 5909 Normally rclone will look at modification time and size of files to 5910 see if they are equal. If you set this flag then rclone will check 5911 only the modification time. If `--checksum` is set then it only 5912 checks the checksum. 5913 5914 It will also cause rclone to skip verifying the sizes are the same 5915 after transfer. 5916 5917 This can be useful for transferring files to and from OneDrive which 5918 occasionally misreports the size of image files (see 5919 [#399](https://github.com/rclone/rclone/issues/399) for more info). 5920 5921 ### -I, --ignore-times ### 5922 5923 Using this option will cause rclone to unconditionally upload all 5924 files regardless of the state of files on the destination. 5925 5926 Normally rclone would skip any files that have the same 5927 modification time and are the same size (or have the same checksum if 5928 using `--checksum`). 5929 5930 ### --immutable ### 5931 5932 Treat source and destination files as immutable and disallow 5933 modification. 5934 5935 With this option set, files will be created and deleted as requested, 5936 but existing files will never be updated. If an existing file does 5937 not match between the source and destination, rclone will give the error 5938 `Source and destination exist but do not match: immutable file modified`. 5939 5940 Note that only commands which transfer files (e.g. `sync`, `copy`, 5941 `move`) are affected by this behavior, and only modification is 5942 disallowed. Files may still be deleted explicitly (e.g. `delete`, 5943 `purge`) or implicitly (e.g. `sync`, `move`). Use `copy --immutable` 5944 if it is desired to avoid deletion as well as modification. 5945 5946 This can be useful as an additional layer of protection for immutable 5947 or append-only data sets (notably backup archives), where modification 5948 implies corruption and should not be propagated. 5949 5950 ## --leave-root ### 5951 5952 During rmdirs it will not remove root directory, even if it's empty. 5953 5954 ### --log-file=FILE ### 5955 5956 Log all of rclone's output to FILE. This is not active by default. 5957 This can be useful for tracking down problems with syncs in 5958 combination with the `-v` flag. See the [Logging section](#logging) 5959 for more info. 5960 5961 Note that if you are using the `logrotate` program to manage rclone's 5962 logs, then you should use the `copytruncate` option as rclone doesn't 5963 have a signal to rotate logs. 5964 5965 ### --log-format LIST ### 5966 5967 Comma separated list of log format options. `date`, `time`, `microseconds`, `longfile`, `shortfile`, `UTC`. The default is "`date`,`time`". 5968 5969 ### --log-level LEVEL ### 5970 5971 This sets the log level for rclone. The default log level is `NOTICE`. 5972 5973 `DEBUG` is equivalent to `-vv`. It outputs lots of debug info - useful 5974 for bug reports and really finding out what rclone is doing. 5975 5976 `INFO` is equivalent to `-v`. It outputs information about each transfer 5977 and prints stats once a minute by default. 5978 5979 `NOTICE` is the default log level if no logging flags are supplied. It 5980 outputs very little when things are working normally. It outputs 5981 warnings and significant events. 5982 5983 `ERROR` is equivalent to `-q`. It only outputs error messages. 5984 5985 ### --use-json-log ### 5986 5987 This switches the log format to JSON for rclone. The fields of json log 5988 are level, msg, source, time. 5989 5990 ### --low-level-retries NUMBER ### 5991 5992 This controls the number of low level retries rclone does. 5993 5994 A low level retry is used to retry a failing operation - typically one 5995 HTTP request. This might be uploading a chunk of a big file for 5996 example. You will see low level retries in the log with the `-v` 5997 flag. 5998 5999 This shouldn't need to be changed from the default in normal operations. 6000 However, if you get a lot of low level retries you may wish 6001 to reduce the value so rclone moves on to a high level retry (see the 6002 `--retries` flag) quicker. 6003 6004 Disable low level retries with `--low-level-retries 1`. 6005 6006 ### --max-backlog=N ### 6007 6008 This is the maximum allowable backlog of files in a sync/copy/move 6009 queued for being checked or transferred. 6010 6011 This can be set arbitrarily large. It will only use memory when the 6012 queue is in use. Note that it will use in the order of N kB of memory 6013 when the backlog is in use. 6014 6015 Setting this large allows rclone to calculate how many files are 6016 pending more accurately, give a more accurate estimated finish 6017 time and make `--order-by` work more accurately. 6018 6019 Setting this small will make rclone more synchronous to the listings 6020 of the remote which may be desirable. 6021 6022 Setting this to a negative number will make the backlog as large as 6023 possible. 6024 6025 ### --max-delete=N ### 6026 6027 This tells rclone not to delete more than N files. If that limit is 6028 exceeded then a fatal error will be generated and rclone will stop the 6029 operation in progress. 6030 6031 ### --max-depth=N ### 6032 6033 This modifies the recursion depth for all the commands except purge. 6034 6035 So if you do `rclone --max-depth 1 ls remote:path` you will see only 6036 the files in the top level directory. Using `--max-depth 2` means you 6037 will see all the files in first two directory levels and so on. 6038 6039 For historical reasons the `lsd` command defaults to using a 6040 `--max-depth` of 1 - you can override this with the command line flag. 6041 6042 You can use this command to disable recursion (with `--max-depth 1`). 6043 6044 Note that if you use this with `sync` and `--delete-excluded` the 6045 files not recursed through are considered excluded and will be deleted 6046 on the destination. Test first with `--dry-run` if you are not sure 6047 what will happen. 6048 6049 ### --max-duration=TIME ### 6050 6051 Rclone will stop scheduling new transfers when it has run for the 6052 duration specified. 6053 6054 Defaults to off. 6055 6056 When the limit is reached any existing transfers will complete. 6057 6058 Rclone won't exit with an error if the transfer limit is reached. 6059 6060 ### --max-transfer=SIZE ### 6061 6062 Rclone will stop transferring when it has reached the size specified. 6063 Defaults to off. 6064 6065 When the limit is reached all transfers will stop immediately. 6066 6067 Rclone will exit with exit code 8 if the transfer limit is reached. 6068 6069 ### --cutoff-mode=hard|soft|cautious ### 6070 6071 This modifies the behavior of `--max-transfer` 6072 Defaults to `--cutoff-mode=hard`. 6073 6074 Specifying `--cutoff-mode=hard` will stop transferring immediately 6075 when Rclone reaches the limit. 6076 6077 Specifying `--cutoff-mode=soft` will stop starting new transfers 6078 when Rclone reaches the limit. 6079 6080 Specifying `--cutoff-mode=cautious` will try to prevent Rclone 6081 from reaching the limit. 6082 6083 ### --modify-window=TIME ### 6084 6085 When checking whether a file has been modified, this is the maximum 6086 allowed time difference that a file can have and still be considered 6087 equivalent. 6088 6089 The default is `1ns` unless this is overridden by a remote. For 6090 example OS X only stores modification times to the nearest second so 6091 if you are reading and writing to an OS X filing system this will be 6092 `1s` by default. 6093 6094 This command line flag allows you to override that computed default. 6095 6096 ### --multi-thread-cutoff=SIZE ### 6097 6098 When downloading files to the local backend above this size, rclone 6099 will use multiple threads to download the file (default 250M). 6100 6101 Rclone preallocates the file (using `fallocate(FALLOC_FL_KEEP_SIZE)` 6102 on unix or `NTSetInformationFile` on Windows both of which takes no 6103 time) then each thread writes directly into the file at the correct 6104 place. This means that rclone won't create fragmented or sparse files 6105 and there won't be any assembly time at the end of the transfer. 6106 6107 The number of threads used to download is controlled by 6108 `--multi-thread-streams`. 6109 6110 Use `-vv` if you wish to see info about the threads. 6111 6112 This will work with the `sync`/`copy`/`move` commands and friends 6113 `copyto`/`moveto`. Multi thread downloads will be used with `rclone 6114 mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or 6115 above. 6116 6117 **NB** that this **only** works for a local destination but will work 6118 with any source. 6119 6120 **NB** that multi thread copies are disabled for local to local copies 6121 as they are faster without unless `--multi-thread-streams` is set 6122 explicitly. 6123 6124 **NB** on Windows using multi-thread downloads will cause the 6125 resulting files to be [sparse](https://en.wikipedia.org/wiki/Sparse_file). 6126 Use `--local-no-sparse` to disable sparse files (which may cause long 6127 delays at the start of downloads) or disable multi-thread downloads 6128 with `--multi-thread-streams 0` 6129 6130 ### --multi-thread-streams=N ### 6131 6132 When using multi thread downloads (see above `--multi-thread-cutoff`) 6133 this sets the maximum number of streams to use. Set to `0` to disable 6134 multi thread downloads (Default 4). 6135 6136 Exactly how many streams rclone uses for the download depends on the 6137 size of the file. To calculate the number of download streams Rclone 6138 divides the size of the file by the `--multi-thread-cutoff` and rounds 6139 up, up to the maximum set with `--multi-thread-streams`. 6140 6141 So if `--multi-thread-cutoff 250MB` and `--multi-thread-streams 4` are 6142 in effect (the defaults): 6143 6144 - 0MB..250MB files will be downloaded with 1 stream 6145 - 250MB..500MB files will be downloaded with 2 streams 6146 - 500MB..750MB files will be downloaded with 3 streams 6147 - 750MB+ files will be downloaded with 4 streams 6148 6149 ### --no-check-dest ### 6150 6151 The `--no-check-dest` can be used with `move` or `copy` and it causes 6152 rclone not to check the destination at all when copying files. 6153 6154 This means that: 6155 6156 - the destination is not listed minimising the API calls 6157 - files are always transferred 6158 - this can cause duplicates on remotes which allow it (eg Google Drive) 6159 - `--retries 1` is recommended otherwise you'll transfer everything again on a retry 6160 6161 This flag is useful to minimise the transactions if you know that none 6162 of the files are on the destination. 6163 6164 This is a specialized flag which should be ignored by most users! 6165 6166 ### --no-gzip-encoding ### 6167 6168 Don't set `Accept-Encoding: gzip`. This means that rclone won't ask 6169 the server for compressed files automatically. Useful if you've set 6170 the server to return files with `Content-Encoding: gzip` but you 6171 uploaded compressed files. 6172 6173 There is no need to set this in normal operation, and doing so will 6174 decrease the network transfer efficiency of rclone. 6175 6176 ### --no-traverse ### 6177 6178 The `--no-traverse` flag controls whether the destination file system 6179 is traversed when using the `copy` or `move` commands. 6180 `--no-traverse` is not compatible with `sync` and will be ignored if 6181 you supply it with `sync`. 6182 6183 If you are only copying a small number of files (or are filtering most 6184 of the files) and/or have a large number of files on the destination 6185 then `--no-traverse` will stop rclone listing the destination and save 6186 time. 6187 6188 However, if you are copying a large number of files, especially if you 6189 are doing a copy where lots of the files under consideration haven't 6190 changed and won't need copying then you shouldn't use `--no-traverse`. 6191 6192 See [rclone copy](https://rclone.org/commands/rclone_copy/) for an example of how to use it. 6193 6194 ### --no-unicode-normalization ### 6195 6196 Don't normalize unicode characters in filenames during the sync routine. 6197 6198 Sometimes, an operating system will store filenames containing unicode 6199 parts in their decomposed form (particularly macOS). Some cloud storage 6200 systems will then recompose the unicode, resulting in duplicate files if 6201 the data is ever copied back to a local filesystem. 6202 6203 Using this flag will disable that functionality, treating each unicode 6204 character as unique. For example, by default é and é will be normalized 6205 into the same character. With `--no-unicode-normalization` they will be 6206 treated as unique characters. 6207 6208 ### --no-update-modtime ### 6209 6210 When using this flag, rclone won't update modification times of remote 6211 files if they are incorrect as it would normally. 6212 6213 This can be used if the remote is being synced with another tool also 6214 (eg the Google Drive client). 6215 6216 ### --order-by string ### 6217 6218 The `--order-by` flag controls the order in which files in the backlog 6219 are processed in `rclone sync`, `rclone copy` and `rclone move`. 6220 6221 The order by string is constructed like this. The first part 6222 describes what aspect is being measured: 6223 6224 - `size` - order by the size of the files 6225 - `name` - order by the full path of the files 6226 - `modtime` - order by the modification date of the files 6227 6228 This can have a modifier appended with a comma: 6229 6230 - `ascending` or `asc` - order so that the smallest (or oldest) is processed first 6231 - `descending` or `desc` - order so that the largest (or newest) is processed first 6232 - `mixed` - order so that the smallest is processed first for some threads and the largest for others 6233 6234 If the modifier is `mixed` then it can have an optional percentage 6235 (which defaults to `50`), eg `size,mixed,25` which means that 25% of 6236 the threads should be taking the smallest items and 75% the 6237 largest. The threads which take the smallest first will always take 6238 the smallest first and likewise the largest first threads. The `mixed` 6239 mode can be useful to minimise the transfer time when you are 6240 transferring a mixture of large and small files - the large files are 6241 guaranteed upload threads and bandwidth and the small files will be 6242 processed continuously. 6243 6244 If no modifier is supplied then the order is `ascending`. 6245 6246 For example 6247 6248 - `--order-by size,desc` - send the largest files first 6249 - `--order-by modtime,ascending` - send the oldest files first 6250 - `--order-by name` - send the files with alphabetically by path first 6251 6252 If the `--order-by` flag is not supplied or it is supplied with an 6253 empty string then the default ordering will be used which is as 6254 scanned. With `--checkers 1` this is mostly alphabetical, however 6255 with the default `--checkers 8` it is somewhat random. 6256 6257 #### Limitations 6258 6259 The `--order-by` flag does not do a separate pass over the data. This 6260 means that it may transfer some files out of the order specified if 6261 6262 - there are no files in the backlog or the source has not been fully scanned yet 6263 - there are more than [--max-backlog](#max-backlog-n) files in the backlog 6264 6265 Rclone will do its best to transfer the best file it has so in 6266 practice this should not cause a problem. Think of `--order-by` as 6267 being more of a best efforts flag rather than a perfect ordering. 6268 6269 ### --password-command SpaceSepList ### 6270 6271 This flag supplies a program which should supply the config password 6272 when run. This is an alternative to rclone prompting for the password 6273 or setting the `RCLONE_CONFIG_PASS` variable. 6274 6275 The argument to this should be a command with a space separated list 6276 of arguments. If one of the arguments has a space in then enclose it 6277 in `"`, if you want a literal `"` in an argument then enclose the 6278 argument in `"` and double the `"`. See [CSV encoding](https://godoc.org/encoding/csv) 6279 for more info. 6280 6281 Eg 6282 6283 --password-command echo hello 6284 --password-command echo "hello with space" 6285 --password-command echo "hello with ""quotes"" and space" 6286 6287 See the [Configuration Encryption](#configuration-encryption) for more info. 6288 6289 See a [Windows PowerShell example on the Wiki](https://github.com/rclone/rclone/wiki/Windows-Powershell-use-rclone-password-command-for-Config-file-password). 6290 6291 ### -P, --progress ### 6292 6293 This flag makes rclone update the stats in a static block in the 6294 terminal providing a realtime overview of the transfer. 6295 6296 Any log messages will scroll above the static block. Log messages 6297 will push the static block down to the bottom of the terminal where it 6298 will stay. 6299 6300 Normally this is updated every 500mS but this period can be overridden 6301 with the `--stats` flag. 6302 6303 This can be used with the `--stats-one-line` flag for a simpler 6304 display. 6305 6306 Note: On Windows until [this bug](https://github.com/Azure/go-ansiterm/issues/26) 6307 is fixed all non-ASCII characters will be replaced with `.` when 6308 `--progress` is in use. 6309 6310 ### -q, --quiet ### 6311 6312 This flag will limit rclone's output to error messages only. 6313 6314 ### --retries int ### 6315 6316 Retry the entire sync if it fails this many times it fails (default 3). 6317 6318 Some remotes can be unreliable and a few retries help pick up the 6319 files which didn't get transferred because of errors. 6320 6321 Disable retries with `--retries 1`. 6322 6323 ### --retries-sleep=TIME ### 6324 6325 This sets the interval between each retry specified by `--retries` 6326 6327 The default is `0`. Use `0` to disable. 6328 6329 ### --size-only ### 6330 6331 Normally rclone will look at modification time and size of files to 6332 see if they are equal. If you set this flag then rclone will check 6333 only the size. 6334 6335 This can be useful transferring files from Dropbox which have been 6336 modified by the desktop sync client which doesn't set checksums of 6337 modification times in the same way as rclone. 6338 6339 ### --stats=TIME ### 6340 6341 Commands which transfer data (`sync`, `copy`, `copyto`, `move`, 6342 `moveto`) will print data transfer stats at regular intervals to show 6343 their progress. 6344 6345 This sets the interval. 6346 6347 The default is `1m`. Use `0` to disable. 6348 6349 If you set the stats interval then all commands can show stats. This 6350 can be useful when running other commands, `check` or `mount` for 6351 example. 6352 6353 Stats are logged at `INFO` level by default which means they won't 6354 show at default log level `NOTICE`. Use `--stats-log-level NOTICE` or 6355 `-v` to make them show. See the [Logging section](#logging) for more 6356 info on log levels. 6357 6358 Note that on macOS you can send a SIGINFO (which is normally ctrl-T in 6359 the terminal) to make the stats print immediately. 6360 6361 ### --stats-file-name-length integer ### 6362 By default, the `--stats` output will truncate file names and paths longer 6363 than 40 characters. This is equivalent to providing 6364 `--stats-file-name-length 40`. Use `--stats-file-name-length 0` to disable 6365 any truncation of file names printed by stats. 6366 6367 ### --stats-log-level string ### 6368 6369 Log level to show `--stats` output at. This can be `DEBUG`, `INFO`, 6370 `NOTICE`, or `ERROR`. The default is `INFO`. This means at the 6371 default level of logging which is `NOTICE` the stats won't show - if 6372 you want them to then use `--stats-log-level NOTICE`. See the [Logging 6373 section](#logging) for more info on log levels. 6374 6375 ### --stats-one-line ### 6376 6377 When this is specified, rclone condenses the stats into a single line 6378 showing the most important stats only. 6379 6380 ### --stats-one-line-date ### 6381 6382 When this is specified, rclone enables the single-line stats and prepends 6383 the display with a date string. The default is `2006/01/02 15:04:05 - ` 6384 6385 ### --stats-one-line-date-format ### 6386 6387 When this is specified, rclone enables the single-line stats and prepends 6388 the display with a user-supplied date string. The date string MUST be 6389 enclosed in quotes. Follow [golang specs](https://golang.org/pkg/time/#Time.Format) for 6390 date formatting syntax. 6391 6392 ### --stats-unit=bits|bytes ### 6393 6394 By default, data transfer rates will be printed in bytes/second. 6395 6396 This option allows the data rate to be printed in bits/second. 6397 6398 Data transfer volume will still be reported in bytes. 6399 6400 The rate is reported as a binary unit, not SI unit. So 1 Mbit/s 6401 equals 1,048,576 bits/s and not 1,000,000 bits/s. 6402 6403 The default is `bytes`. 6404 6405 ### --suffix=SUFFIX ### 6406 6407 When using `sync`, `copy` or `move` any files which would have been 6408 overwritten or deleted will have the suffix added to them. If there 6409 is a file with the same path (after the suffix has been added), then 6410 it will be overwritten. 6411 6412 The remote in use must support server side move or copy and you must 6413 use the same remote as the destination of the sync. 6414 6415 This is for use with files to add the suffix in the current directory 6416 or with `--backup-dir`. See `--backup-dir` for more info. 6417 6418 For example 6419 6420 rclone sync /path/to/local/file remote:current --suffix .bak 6421 6422 will sync `/path/to/local` to `remote:current`, but for any files 6423 which would have been updated or deleted have .bak added. 6424 6425 ### --suffix-keep-extension ### 6426 6427 When using `--suffix`, setting this causes rclone put the SUFFIX 6428 before the extension of the files that it backs up rather than after. 6429 6430 So let's say we had `--suffix -2019-01-01`, without the flag `file.txt` 6431 would be backed up to `file.txt-2019-01-01` and with the flag it would 6432 be backed up to `file-2019-01-01.txt`. This can be helpful to make 6433 sure the suffixed files can still be opened. 6434 6435 ### --syslog ### 6436 6437 On capable OSes (not Windows or Plan9) send all log output to syslog. 6438 6439 This can be useful for running rclone in a script or `rclone mount`. 6440 6441 ### --syslog-facility string ### 6442 6443 If using `--syslog` this sets the syslog facility (eg `KERN`, `USER`). 6444 See `man syslog` for a list of possible facilities. The default 6445 facility is `DAEMON`. 6446 6447 ### --tpslimit float ### 6448 6449 Limit HTTP transactions per second to this. Default is 0 which is used 6450 to mean unlimited transactions per second. 6451 6452 For example to limit rclone to 10 HTTP transactions per second use 6453 `--tpslimit 10`, or to 1 transaction every 2 seconds use `--tpslimit 6454 0.5`. 6455 6456 Use this when the number of transactions per second from rclone is 6457 causing a problem with the cloud storage provider (eg getting you 6458 banned or rate limited). 6459 6460 This can be very useful for `rclone mount` to control the behaviour of 6461 applications using it. 6462 6463 See also `--tpslimit-burst`. 6464 6465 ### --tpslimit-burst int ### 6466 6467 Max burst of transactions for `--tpslimit` (default `1`). 6468 6469 Normally `--tpslimit` will do exactly the number of transaction per 6470 second specified. However if you supply `--tps-burst` then rclone can 6471 save up some transactions from when it was idle giving a burst of up 6472 to the parameter supplied. 6473 6474 For example if you provide `--tpslimit-burst 10` then if rclone has 6475 been idle for more than 10*`--tpslimit` then it can do 10 transactions 6476 very quickly before they are limited again. 6477 6478 This may be used to increase performance of `--tpslimit` without 6479 changing the long term average number of transactions per second. 6480 6481 ### --track-renames ### 6482 6483 By default, rclone doesn't keep track of renamed files, so if you 6484 rename a file locally then sync it to a remote, rclone will delete the 6485 old file on the remote and upload a new copy. 6486 6487 If you use this flag, and the remote supports server side copy or 6488 server side move, and the source and destination have a compatible 6489 hash, then this will track renames during `sync` 6490 operations and perform renaming server-side. 6491 6492 Files will be matched by size and hash - if both match then a rename 6493 will be considered. 6494 6495 If the destination does not support server-side copy or move, rclone 6496 will fall back to the default behaviour and log an error level message 6497 to the console. Note: Encrypted destinations are not supported 6498 by `--track-renames`. 6499 6500 Note that `--track-renames` is incompatible with `--no-traverse` and 6501 that it uses extra memory to keep track of all the rename candidates. 6502 6503 Note also that `--track-renames` is incompatible with 6504 `--delete-before` and will select `--delete-after` instead of 6505 `--delete-during`. 6506 6507 ### --track-renames-strategy (hash,modtime) ### 6508 6509 This option changes the matching criteria for `--track-renames` to match 6510 by any combination of modtime, hash, size. Matching by size is always enabled 6511 no matter what option is selected here. This also means 6512 that it enables `--track-renames` support for encrypted destinations. 6513 If nothing is specified, the default option is matching by hashes. 6514 6515 ### --delete-(before,during,after) ### 6516 6517 This option allows you to specify when files on your destination are 6518 deleted when you sync folders. 6519 6520 Specifying the value `--delete-before` will delete all files present 6521 on the destination, but not on the source *before* starting the 6522 transfer of any new or updated files. This uses two passes through the 6523 file systems, one for the deletions and one for the copies. 6524 6525 Specifying `--delete-during` will delete files while checking and 6526 uploading files. This is the fastest option and uses the least memory. 6527 6528 Specifying `--delete-after` (the default value) will delay deletion of 6529 files until all new/updated files have been successfully transferred. 6530 The files to be deleted are collected in the copy pass then deleted 6531 after the copy pass has completed successfully. The files to be 6532 deleted are held in memory so this mode may use more memory. This is 6533 the safest mode as it will only delete files if there have been no 6534 errors subsequent to that. If there have been errors before the 6535 deletions start then you will get the message `not deleting files as 6536 there were IO errors`. 6537 6538 ### --fast-list ### 6539 6540 When doing anything which involves a directory listing (eg `sync`, 6541 `copy`, `ls` - in fact nearly every command), rclone normally lists a 6542 directory and processes it before using more directory lists to 6543 process any subdirectories. This can be parallelised and works very 6544 quickly using the least amount of memory. 6545 6546 However, some remotes have a way of listing all files beneath a 6547 directory in one (or a small number) of transactions. These tend to 6548 be the bucket based remotes (eg S3, B2, GCS, Swift, Hubic). 6549 6550 If you use the `--fast-list` flag then rclone will use this method for 6551 listing directories. This will have the following consequences for 6552 the listing: 6553 6554 * It **will** use fewer transactions (important if you pay for them) 6555 * It **will** use more memory. Rclone has to load the whole listing into memory. 6556 * It *may* be faster because it uses fewer transactions 6557 * It *may* be slower because it can't be parallelized 6558 6559 rclone should always give identical results with and without 6560 `--fast-list`. 6561 6562 If you pay for transactions and can fit your entire sync listing into 6563 memory then `--fast-list` is recommended. If you have a very big sync 6564 to do then don't use `--fast-list` otherwise you will run out of 6565 memory. 6566 6567 If you use `--fast-list` on a remote which doesn't support it, then 6568 rclone will just ignore it. 6569 6570 ### --timeout=TIME ### 6571 6572 This sets the IO idle timeout. If a transfer has started but then 6573 becomes idle for this long it is considered broken and disconnected. 6574 6575 The default is `5m`. Set to `0` to disable. 6576 6577 ### --transfers=N ### 6578 6579 The number of file transfers to run in parallel. It can sometimes be 6580 useful to set this to a smaller number if the remote is giving a lot 6581 of timeouts or bigger if you have lots of bandwidth and a fast remote. 6582 6583 The default is to run 4 file transfers in parallel. 6584 6585 ### -u, --update ### 6586 6587 This forces rclone to skip any files which exist on the destination 6588 and have a modified time that is newer than the source file. 6589 6590 This can be useful when transferring to a remote which doesn't support 6591 mod times directly (or when using `--use-server-modtime` to avoid extra 6592 API calls) as it is more accurate than a `--size-only` check and faster 6593 than using `--checksum`. 6594 6595 If an existing destination file has a modification time equal (within 6596 the computed modify window precision) to the source file's, it will be 6597 updated if the sizes are different. If `--checksum` is set then 6598 rclone will update the destination if the checksums differ too. 6599 6600 If an existing destination file is older than the source file then 6601 it will be updated if the size or checksum differs from the source file. 6602 6603 On remotes which don't support mod time directly (or when using 6604 `--use-server-modtime`) the time checked will be the uploaded time. 6605 This means that if uploading to one of these remotes, rclone will skip 6606 any files which exist on the destination and have an uploaded time that 6607 is newer than the modification time of the source file. 6608 6609 ### --use-mmap ### 6610 6611 If this flag is set then rclone will use anonymous memory allocated by 6612 mmap on Unix based platforms and VirtualAlloc on Windows for its 6613 transfer buffers (size controlled by `--buffer-size`). Memory 6614 allocated like this does not go on the Go heap and can be returned to 6615 the OS immediately when it is finished with. 6616 6617 If this flag is not set then rclone will allocate and free the buffers 6618 using the Go memory allocator which may use more memory as memory 6619 pages are returned less aggressively to the OS. 6620 6621 It is possible this does not work well on all platforms so it is 6622 disabled by default; in the future it may be enabled by default. 6623 6624 ### --use-server-modtime ### 6625 6626 Some object-store backends (e.g, Swift, S3) do not preserve file modification 6627 times (modtime). On these backends, rclone stores the original modtime as 6628 additional metadata on the object. By default it will make an API call to 6629 retrieve the metadata when the modtime is needed by an operation. 6630 6631 Use this flag to disable the extra API call and rely instead on the server's 6632 modified time. In cases such as a local to remote sync using `--update`, 6633 knowing the local file is newer than the time it was last uploaded to the 6634 remote is sufficient. In those cases, this flag can speed up the process and 6635 reduce the number of API calls necessary. 6636 6637 Using this flag on a sync operation without also using `--update` would cause 6638 all files modified at any time other than the last upload time to be uploaded 6639 again, which is probably not what you want. 6640 6641 ### -v, -vv, --verbose ### 6642 6643 With `-v` rclone will tell you about each file that is transferred and 6644 a small number of significant events. 6645 6646 With `-vv` rclone will become very verbose telling you about every 6647 file it considers and transfers. Please send bug reports with a log 6648 with this setting. 6649 6650 ### -V, --version ### 6651 6652 Prints the version number 6653 6654 SSL/TLS options 6655 --------------- 6656 6657 The outgoing SSL/TLS connections rclone makes can be controlled with 6658 these options. For example this can be very useful with the HTTP or 6659 WebDAV backends. Rclone HTTP servers have their own set of 6660 configuration for SSL/TLS which you can find in their documentation. 6661 6662 ### --ca-cert string 6663 6664 This loads the PEM encoded certificate authority certificate and uses 6665 it to verify the certificates of the servers rclone connects to. 6666 6667 If you have generated certificates signed with a local CA then you 6668 will need this flag to connect to servers using those certificates. 6669 6670 ### --client-cert string 6671 6672 This loads the PEM encoded client side certificate. 6673 6674 This is used for [mutual TLS authentication](https://en.wikipedia.org/wiki/Mutual_authentication). 6675 6676 The `--client-key` flag is required too when using this. 6677 6678 ### --client-key string 6679 6680 This loads the PEM encoded client side private key used for mutual TLS 6681 authentication. Used in conjunction with `--client-cert`. 6682 6683 ### --no-check-certificate=true/false ### 6684 6685 `--no-check-certificate` controls whether a client verifies the 6686 server's certificate chain and host name. 6687 If `--no-check-certificate` is true, TLS accepts any certificate 6688 presented by the server and any host name in that certificate. 6689 In this mode, TLS is susceptible to man-in-the-middle attacks. 6690 6691 This option defaults to `false`. 6692 6693 **This should be used only for testing.** 6694 6695 Configuration Encryption 6696 ------------------------ 6697 Your configuration file contains information for logging in to 6698 your cloud services. This means that you should keep your 6699 `.rclone.conf` file in a secure location. 6700 6701 If you are in an environment where that isn't possible, you can 6702 add a password to your configuration. This means that you will 6703 have to supply the password every time you start rclone. 6704 6705 To add a password to your rclone configuration, execute `rclone config`. 6706 6707 ``` 6708 >rclone config 6709 Current remotes: 6710 6711 e) Edit existing remote 6712 n) New remote 6713 d) Delete remote 6714 s) Set configuration password 6715 q) Quit config 6716 e/n/d/s/q> 6717 ``` 6718 6719 Go into `s`, Set configuration password: 6720 ``` 6721 e/n/d/s/q> s 6722 Your configuration is not encrypted. 6723 If you add a password, you will protect your login information to cloud services. 6724 a) Add Password 6725 q) Quit to main menu 6726 a/q> a 6727 Enter NEW configuration password: 6728 password: 6729 Confirm NEW password: 6730 password: 6731 Password set 6732 Your configuration is encrypted. 6733 c) Change Password 6734 u) Unencrypt configuration 6735 q) Quit to main menu 6736 c/u/q> 6737 ``` 6738 6739 Your configuration is now encrypted, and every time you start rclone 6740 you will have to supply the password. See below for details. 6741 In the same menu, you can change the password or completely remove 6742 encryption from your configuration. 6743 6744 There is no way to recover the configuration if you lose your password. 6745 6746 rclone uses [nacl secretbox](https://godoc.org/golang.org/x/crypto/nacl/secretbox) 6747 which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate 6748 your configuration with secret-key cryptography. 6749 The password is SHA-256 hashed, which produces the key for secretbox. 6750 The hashed password is not stored. 6751 6752 While this provides very good security, we do not recommend storing 6753 your encrypted rclone configuration in public if it contains sensitive 6754 information, maybe except if you use a very strong password. 6755 6756 If it is safe in your environment, you can set the `RCLONE_CONFIG_PASS` 6757 environment variable to contain your password, in which case it will be 6758 used for decrypting the configuration. 6759 6760 You can set this for a session from a script. For unix like systems 6761 save this to a file called `set-rclone-password`: 6762 6763 ``` 6764 #!/bin/echo Source this file don't run it 6765 6766 read -s RCLONE_CONFIG_PASS 6767 export RCLONE_CONFIG_PASS 6768 ``` 6769 6770 Then source the file when you want to use it. From the shell you 6771 would do `source set-rclone-password`. It will then ask you for the 6772 password and set it in the environment variable. 6773 6774 An alternate means of supplying the password is to provide a script 6775 which will retrieve the password and print on standard output. This 6776 script should have a fully specified path name and not rely on any 6777 environment variables. The script is supplied either via 6778 `--password-command="..."` command line argument or via the 6779 `RCLONE_PASSWORD_COMMAND` environment variable. 6780 6781 One useful example of this is using the `passwordstore` application 6782 to retrieve the password: 6783 6784 ``` 6785 export RCLONE_PASSWORD_COMMAND="pass rclone/config" 6786 ``` 6787 6788 If the `passwordstore` password manager holds the password for the 6789 rclone configuration, using the script method means the password 6790 is primarily protected by the `passwordstore` system, and is never 6791 embedded in the clear in scripts, nor available for examination 6792 using the standard commands available. It is quite possible with 6793 long running rclone sessions for copies of passwords to be innocently 6794 captured in log files or terminal scroll buffers, etc. Using the 6795 script method of supplying the password enhances the security of 6796 the config password considerably. 6797 6798 If you are running rclone inside a script, unless you are using the 6799 `--password-command` method, you might want to disable 6800 password prompts. To do that, pass the parameter 6801 `--ask-password=false` to rclone. This will make rclone fail instead 6802 of asking for a password if `RCLONE_CONFIG_PASS` doesn't contain 6803 a valid password, and `--password-command` has not been supplied. 6804 6805 6806 Developer options 6807 ----------------- 6808 6809 These options are useful when developing or debugging rclone. There 6810 are also some more remote specific options which aren't documented 6811 here which are used for testing. These start with remote name eg 6812 `--drive-test-option` - see the docs for the remote in question. 6813 6814 ### --cpuprofile=FILE ### 6815 6816 Write CPU profile to file. This can be analysed with `go tool pprof`. 6817 6818 #### --dump flag,flag,flag #### 6819 6820 The `--dump` flag takes a comma separated list of flags to dump info 6821 about. 6822 6823 Note that some headers including `Accept-Encoding` as shown may not 6824 be correct in the request and the response may not show `Content-Encoding` 6825 if the go standard libraries auto gzip encoding was in effect. In this case 6826 the body of the request will be gunzipped before showing it. 6827 6828 The available flags are: 6829 6830 #### --dump headers #### 6831 6832 Dump HTTP headers with `Authorization:` lines removed. May still 6833 contain sensitive info. Can be very verbose. Useful for debugging 6834 only. 6835 6836 Use `--dump auth` if you do want the `Authorization:` headers. 6837 6838 #### --dump bodies #### 6839 6840 Dump HTTP headers and bodies - may contain sensitive info. Can be 6841 very verbose. Useful for debugging only. 6842 6843 Note that the bodies are buffered in memory so don't use this for 6844 enormous files. 6845 6846 #### --dump requests #### 6847 6848 Like `--dump bodies` but dumps the request bodies and the response 6849 headers. Useful for debugging download problems. 6850 6851 #### --dump responses #### 6852 6853 Like `--dump bodies` but dumps the response bodies and the request 6854 headers. Useful for debugging upload problems. 6855 6856 #### --dump auth #### 6857 6858 Dump HTTP headers - will contain sensitive info such as 6859 `Authorization:` headers - use `--dump headers` to dump without 6860 `Authorization:` headers. Can be very verbose. Useful for debugging 6861 only. 6862 6863 #### --dump filters #### 6864 6865 Dump the filters to the output. Useful to see exactly what include 6866 and exclude options are filtering on. 6867 6868 #### --dump goroutines #### 6869 6870 This dumps a list of the running go-routines at the end of the command 6871 to standard output. 6872 6873 #### --dump openfiles #### 6874 6875 This dumps a list of the open files at the end of the command. It 6876 uses the `lsof` command to do that so you'll need that installed to 6877 use it. 6878 6879 ### --memprofile=FILE ### 6880 6881 Write memory profile to file. This can be analysed with `go tool pprof`. 6882 6883 Filtering 6884 --------- 6885 6886 For the filtering options 6887 6888 * `--delete-excluded` 6889 * `--filter` 6890 * `--filter-from` 6891 * `--exclude` 6892 * `--exclude-from` 6893 * `--include` 6894 * `--include-from` 6895 * `--files-from` 6896 * `--files-from-raw` 6897 * `--min-size` 6898 * `--max-size` 6899 * `--min-age` 6900 * `--max-age` 6901 * `--dump filters` 6902 6903 See the [filtering section](https://rclone.org/filtering/). 6904 6905 Remote control 6906 -------------- 6907 6908 For the remote control options and for instructions on how to remote control rclone 6909 6910 * `--rc` 6911 * and anything starting with `--rc-` 6912 6913 See [the remote control section](https://rclone.org/rc/). 6914 6915 Logging 6916 ------- 6917 6918 rclone has 4 levels of logging, `ERROR`, `NOTICE`, `INFO` and `DEBUG`. 6919 6920 By default, rclone logs to standard error. This means you can redirect 6921 standard error and still see the normal output of rclone commands (eg 6922 `rclone ls`). 6923 6924 By default, rclone will produce `Error` and `Notice` level messages. 6925 6926 If you use the `-q` flag, rclone will only produce `Error` messages. 6927 6928 If you use the `-v` flag, rclone will produce `Error`, `Notice` and 6929 `Info` messages. 6930 6931 If you use the `-vv` flag, rclone will produce `Error`, `Notice`, 6932 `Info` and `Debug` messages. 6933 6934 You can also control the log levels with the `--log-level` flag. 6935 6936 If you use the `--log-file=FILE` option, rclone will redirect `Error`, 6937 `Info` and `Debug` messages along with standard error to FILE. 6938 6939 If you use the `--syslog` flag then rclone will log to syslog and the 6940 `--syslog-facility` control which facility it uses. 6941 6942 Rclone prefixes all log messages with their level in capitals, eg INFO 6943 which makes it easy to grep the log file for different kinds of 6944 information. 6945 6946 Exit Code 6947 --------- 6948 6949 If any errors occur during the command execution, rclone will exit with a 6950 non-zero exit code. This allows scripts to detect when rclone 6951 operations have failed. 6952 6953 During the startup phase, rclone will exit immediately if an error is 6954 detected in the configuration. There will always be a log message 6955 immediately before exiting. 6956 6957 When rclone is running it will accumulate errors as it goes along, and 6958 only exit with a non-zero exit code if (after retries) there were 6959 still failed transfers. For every error counted there will be a high 6960 priority log message (visible with `-q`) showing the message and 6961 which file caused the problem. A high priority message is also shown 6962 when starting a retry so the user can see that any previous error 6963 messages may not be valid after the retry. If rclone has done a retry 6964 it will log a high priority message if the retry was successful. 6965 6966 ### List of exit codes ### 6967 * `0` - success 6968 * `1` - Syntax or usage error 6969 * `2` - Error not otherwise categorised 6970 * `3` - Directory not found 6971 * `4` - File not found 6972 * `5` - Temporary error (one that more retries might fix) (Retry errors) 6973 * `6` - Less serious errors (like 461 errors from dropbox) (NoRetry errors) 6974 * `7` - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors) 6975 * `8` - Transfer exceeded - limit set by --max-transfer reached 6976 * `9` - Operation successful, but no files transferred 6977 6978 Environment Variables 6979 --------------------- 6980 6981 Rclone can be configured entirely using environment variables. These 6982 can be used to set defaults for options or config file entries. 6983 6984 ### Options ### 6985 6986 Every option in rclone can have its default set by environment 6987 variable. 6988 6989 To find the name of the environment variable, first, take the long 6990 option name, strip the leading `--`, change `-` to `_`, make 6991 upper case and prepend `RCLONE_`. 6992 6993 For example, to always set `--stats 5s`, set the environment variable 6994 `RCLONE_STATS=5s`. If you set stats on the command line this will 6995 override the environment variable setting. 6996 6997 Or to always use the trash in drive `--drive-use-trash`, set 6998 `RCLONE_DRIVE_USE_TRASH=true`. 6999 7000 The same parser is used for the options and the environment variables 7001 so they take exactly the same form. 7002 7003 ### Config file ### 7004 7005 You can set defaults for values in the config file on an individual 7006 remote basis. If you want to use this feature, you will need to 7007 discover the name of the config items that you want. The easiest way 7008 is to run through `rclone config` by hand, then look in the config 7009 file to see what the values are (the config file can be found by 7010 looking at the help for `--config` in `rclone help`). 7011 7012 To find the name of the environment variable, you need to set, take 7013 `RCLONE_CONFIG_` + name of remote + `_` + name of config file option 7014 and make it all uppercase. 7015 7016 For example, to configure an S3 remote named `mys3:` without a config 7017 file (using unix ways of setting environment variables): 7018 7019 ``` 7020 $ export RCLONE_CONFIG_MYS3_TYPE=s3 7021 $ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX 7022 $ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX 7023 $ rclone lsd MYS3: 7024 -1 2016-09-21 12:54:21 -1 my-bucket 7025 $ rclone listremotes | grep mys3 7026 mys3: 7027 ``` 7028 7029 Note that if you want to create a remote using environment variables 7030 you must create the `..._TYPE` variable as above. 7031 7032 ### Other environment variables ### 7033 7034 * `RCLONE_CONFIG_PASS` set to contain your config file password (see [Configuration Encryption](#configuration-encryption) section) 7035 * `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (or the lowercase versions thereof). 7036 * `HTTPS_PROXY` takes precedence over `HTTP_PROXY` for https requests. 7037 * The environment values may be either a complete URL or a "host[:port]" for, in which case the "http" scheme is assumed. 7038 7039 # Configuring rclone on a remote / headless machine # 7040 7041 Some of the configurations (those involving oauth2) require an 7042 Internet connected web browser. 7043 7044 If you are trying to set rclone up on a remote or headless box with no 7045 browser available on it (eg a NAS or a server in a datacenter) then 7046 you will need to use an alternative means of configuration. There are 7047 two ways of doing it, described below. 7048 7049 ## Configuring using rclone authorize ## 7050 7051 On the headless box run `rclone` config but answer `N` to the `Use 7052 auto config?` question. 7053 7054 ``` 7055 ... 7056 Remote config 7057 Use auto config? 7058 * Say Y if not sure 7059 * Say N if you are working on a remote or headless machine 7060 y) Yes (default) 7061 n) No 7062 y/n> n 7063 For this to work, you will need rclone available on a machine that has 7064 a web browser available. 7065 7066 For more help and alternate methods see: https://rclone.org/remote_setup/ 7067 7068 Execute the following on the machine with the web browser (same rclone 7069 version recommended): 7070 7071 rclone authorize "amazon cloud drive" 7072 7073 Then paste the result below: 7074 result> 7075 ``` 7076 7077 Then on your main desktop machine 7078 7079 ``` 7080 rclone authorize "amazon cloud drive" 7081 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 7082 Log in and authorize rclone for access 7083 Waiting for code... 7084 Got code 7085 Paste the following into your remote machine ---> 7086 SECRET_TOKEN 7087 <---End paste 7088 ``` 7089 7090 Then back to the headless box, paste in the code 7091 7092 ``` 7093 result> SECRET_TOKEN 7094 -------------------- 7095 [acd12] 7096 client_id = 7097 client_secret = 7098 token = SECRET_TOKEN 7099 -------------------- 7100 y) Yes this is OK 7101 e) Edit this remote 7102 d) Delete this remote 7103 y/e/d> 7104 ``` 7105 7106 ## Configuring by copying the config file ## 7107 7108 Rclone stores all of its config in a single configuration file. This 7109 can easily be copied to configure a remote rclone. 7110 7111 So first configure rclone on your desktop machine with 7112 7113 rclone config 7114 7115 to set up the config file. 7116 7117 Find the config file by running `rclone config file`, for example 7118 7119 ``` 7120 $ rclone config file 7121 Configuration file is stored at: 7122 /home/user/.rclone.conf 7123 ``` 7124 7125 Now transfer it to the remote box (scp, cut paste, ftp, sftp etc) and 7126 place it in the correct place (use `rclone config file` on the remote 7127 box to find out where). 7128 7129 # Filtering, includes and excludes # 7130 7131 Rclone has a sophisticated set of include and exclude rules. Some of 7132 these are based on patterns and some on other things like file size. 7133 7134 The filters are applied for the `copy`, `sync`, `move`, `ls`, `lsl`, 7135 `md5sum`, `sha1sum`, `size`, `delete` and `check` operations. 7136 Note that `purge` does not obey the filters. 7137 7138 Each path as it passes through rclone is matched against the include 7139 and exclude rules like `--include`, `--exclude`, `--include-from`, 7140 `--exclude-from`, `--filter`, or `--filter-from`. The simplest way to 7141 try them out is using the `ls` command, or `--dry-run` together with 7142 `-v`. `--filter-from`, `--exclude-from`, `--include-from`, `--files-from`, 7143 `--files-from-raw` understand `-` as a file name to mean read from standard 7144 input. 7145 7146 ## Patterns ## 7147 7148 The patterns used to match files for inclusion or exclusion are based 7149 on "file globs" as used by the unix shell. 7150 7151 If the pattern starts with a `/` then it only matches at the top level 7152 of the directory tree, **relative to the root of the remote** (not 7153 necessarily the root of the local drive). If it doesn't start with `/` 7154 then it is matched starting at the **end of the path**, but it will 7155 only match a complete path element: 7156 7157 file.jpg - matches "file.jpg" 7158 - matches "directory/file.jpg" 7159 - doesn't match "afile.jpg" 7160 - doesn't match "directory/afile.jpg" 7161 /file.jpg - matches "file.jpg" in the root directory of the remote 7162 - doesn't match "afile.jpg" 7163 - doesn't match "directory/file.jpg" 7164 7165 **Important** Note that you must use `/` in patterns and not `\` even 7166 if running on Windows. 7167 7168 A `*` matches anything but not a `/`. 7169 7170 *.jpg - matches "file.jpg" 7171 - matches "directory/file.jpg" 7172 - doesn't match "file.jpg/something" 7173 7174 Use `**` to match anything, including slashes (`/`). 7175 7176 dir/** - matches "dir/file.jpg" 7177 - matches "dir/dir1/dir2/file.jpg" 7178 - doesn't match "directory/file.jpg" 7179 - doesn't match "adir/file.jpg" 7180 7181 A `?` matches any character except a slash `/`. 7182 7183 l?ss - matches "less" 7184 - matches "lass" 7185 - doesn't match "floss" 7186 7187 A `[` and `]` together make a character class, such as `[a-z]` or 7188 `[aeiou]` or `[[:alpha:]]`. See the [go regexp 7189 docs](https://golang.org/pkg/regexp/syntax/) for more info on these. 7190 7191 h[ae]llo - matches "hello" 7192 - matches "hallo" 7193 - doesn't match "hullo" 7194 7195 A `{` and `}` define a choice between elements. It should contain a 7196 comma separated list of patterns, any of which might match. These 7197 patterns can contain wildcards. 7198 7199 {one,two}_potato - matches "one_potato" 7200 - matches "two_potato" 7201 - doesn't match "three_potato" 7202 - doesn't match "_potato" 7203 7204 Special characters can be escaped with a `\` before them. 7205 7206 \*.jpg - matches "*.jpg" 7207 \\.jpg - matches "\.jpg" 7208 \[one\].jpg - matches "[one].jpg" 7209 7210 Patterns are case sensitive unless the `--ignore-case` flag is used. 7211 7212 Without `--ignore-case` (default) 7213 7214 potato - matches "potato" 7215 - doesn't match "POTATO" 7216 7217 With `--ignore-case` 7218 7219 potato - matches "potato" 7220 - matches "POTATO" 7221 7222 Note also that rclone filter globs can only be used in one of the 7223 filter command line flags, not in the specification of the remote, so 7224 `rclone copy "remote:dir*.jpg" /path/to/dir` won't work - what is 7225 required is `rclone --include "*.jpg" copy remote:dir /path/to/dir` 7226 7227 ### Directories ### 7228 7229 Rclone keeps track of directories that could match any file patterns. 7230 7231 Eg if you add the include rule 7232 7233 /a/*.jpg 7234 7235 Rclone will synthesize the directory include rule 7236 7237 /a/ 7238 7239 If you put any rules which end in `/` then it will only match 7240 directories. 7241 7242 Directory matches are **only** used to optimise directory access 7243 patterns - you must still match the files that you want to match. 7244 Directory matches won't optimise anything on bucket based remotes (eg 7245 s3, swift, google compute storage, b2) which don't have a concept of 7246 directory. 7247 7248 ### Differences between rsync and rclone patterns ### 7249 7250 Rclone implements bash style `{a,b,c}` glob matching which rsync doesn't. 7251 7252 Rclone always does a wildcard match so `\` must always escape a `\`. 7253 7254 ## How the rules are used ## 7255 7256 Rclone maintains a combined list of include rules and exclude rules. 7257 7258 Each file is matched in order, starting from the top, against the rule 7259 in the list until it finds a match. The file is then included or 7260 excluded according to the rule type. 7261 7262 If the matcher fails to find a match after testing against all the 7263 entries in the list then the path is included. 7264 7265 For example given the following rules, `+` being include, `-` being 7266 exclude, 7267 7268 - secret*.jpg 7269 + *.jpg 7270 + *.png 7271 + file2.avi 7272 - * 7273 7274 This would include 7275 7276 * `file1.jpg` 7277 * `file3.png` 7278 * `file2.avi` 7279 7280 This would exclude 7281 7282 * `secret17.jpg` 7283 * non `*.jpg` and `*.png` 7284 7285 A similar process is done on directory entries before recursing into 7286 them. This only works on remotes which have a concept of directory 7287 (Eg local, google drive, onedrive, amazon drive) and not on bucket 7288 based remotes (eg s3, swift, google compute storage, b2). 7289 7290 ## Adding filtering rules ## 7291 7292 Filtering rules are added with the following command line flags. 7293 7294 ### Repeating options ## 7295 7296 You can repeat the following options to add more than one rule of that 7297 type. 7298 7299 * `--include` 7300 * `--include-from` 7301 * `--exclude` 7302 * `--exclude-from` 7303 * `--filter` 7304 * `--filter-from` 7305 * `--filter-from-raw` 7306 7307 **Important** You should not use `--include*` together with `--exclude*`. 7308 It may produce different results than you expected. In that case try to use: `--filter*`. 7309 7310 Note that all the options of the same type are processed together in 7311 the order above, regardless of what order they were placed on the 7312 command line. 7313 7314 So all `--include` options are processed first in the order they 7315 appeared on the command line, then all `--include-from` options etc. 7316 7317 To mix up the order includes and excludes, the `--filter` flag can be 7318 used. 7319 7320 ### `--exclude` - Exclude files matching pattern ### 7321 7322 Add a single exclude rule with `--exclude`. 7323 7324 This flag can be repeated. See above for the order the flags are 7325 processed in. 7326 7327 Eg `--exclude *.bak` to exclude all bak files from the sync. 7328 7329 ### `--exclude-from` - Read exclude patterns from file ### 7330 7331 Add exclude rules from a file. 7332 7333 This flag can be repeated. See above for the order the flags are 7334 processed in. 7335 7336 Prepare a file like this `exclude-file.txt` 7337 7338 # a sample exclude rule file 7339 *.bak 7340 file2.jpg 7341 7342 Then use as `--exclude-from exclude-file.txt`. This will sync all 7343 files except those ending in `bak` and `file2.jpg`. 7344 7345 This is useful if you have a lot of rules. 7346 7347 ### `--include` - Include files matching pattern ### 7348 7349 Add a single include rule with `--include`. 7350 7351 This flag can be repeated. See above for the order the flags are 7352 processed in. 7353 7354 Eg `--include *.{png,jpg}` to include all `png` and `jpg` files in the 7355 backup and no others. 7356 7357 This adds an implicit `--exclude *` at the very end of the filter 7358 list. This means you can mix `--include` and `--include-from` with the 7359 other filters (eg `--exclude`) but you must include all the files you 7360 want in the include statement. If this doesn't provide enough 7361 flexibility then you must use `--filter-from`. 7362 7363 ### `--include-from` - Read include patterns from file ### 7364 7365 Add include rules from a file. 7366 7367 This flag can be repeated. See above for the order the flags are 7368 processed in. 7369 7370 Prepare a file like this `include-file.txt` 7371 7372 # a sample include rule file 7373 *.jpg 7374 *.png 7375 file2.avi 7376 7377 Then use as `--include-from include-file.txt`. This will sync all 7378 `jpg`, `png` files and `file2.avi`. 7379 7380 This is useful if you have a lot of rules. 7381 7382 This adds an implicit `--exclude *` at the very end of the filter 7383 list. This means you can mix `--include` and `--include-from` with the 7384 other filters (eg `--exclude`) but you must include all the files you 7385 want in the include statement. If this doesn't provide enough 7386 flexibility then you must use `--filter-from`. 7387 7388 ### `--filter` - Add a file-filtering rule ### 7389 7390 This can be used to add a single include or exclude rule. Include 7391 rules start with `+ ` and exclude rules start with `- `. A special 7392 rule called `!` can be used to clear the existing rules. 7393 7394 This flag can be repeated. See above for the order the flags are 7395 processed in. 7396 7397 Eg `--filter "- *.bak"` to exclude all bak files from the sync. 7398 7399 ### `--filter-from` - Read filtering patterns from a file ### 7400 7401 Add include/exclude rules from a file. 7402 7403 This flag can be repeated. See above for the order the flags are 7404 processed in. 7405 7406 Prepare a file like this `filter-file.txt` 7407 7408 # a sample filter rule file 7409 - secret*.jpg 7410 + *.jpg 7411 + *.png 7412 + file2.avi 7413 - /dir/Trash/** 7414 + /dir/** 7415 # exclude everything else 7416 - * 7417 7418 Then use as `--filter-from filter-file.txt`. The rules are processed 7419 in the order that they are defined. 7420 7421 This example will include all `jpg` and `png` files, exclude any files 7422 matching `secret*.jpg` and include `file2.avi`. It will also include 7423 everything in the directory `dir` at the root of the sync, except 7424 `dir/Trash` which it will exclude. Everything else will be excluded 7425 from the sync. 7426 7427 ### `--files-from` - Read list of source-file names ### 7428 7429 This reads a list of file names from the file passed in and **only** 7430 these files are transferred. The **filtering rules are ignored** 7431 completely if you use this option. 7432 7433 `--files-from` expects a list of files as its input. Leading / trailing 7434 whitespace is stripped from the input lines and lines starting with `#` 7435 and `;` are ignored. 7436 7437 Rclone will traverse the file system if you use `--files-from`, 7438 effectively using the files in `--files-from` as a set of filters. 7439 Rclone will not error if any of the files are missing. 7440 7441 If you use `--no-traverse` as well as `--files-from` then rclone will 7442 not traverse the destination file system, it will find each file 7443 individually using approximately 1 API call. This can be more 7444 efficient for small lists of files. 7445 7446 This option can be repeated to read from more than one file. These 7447 are read in the order that they are placed on the command line. 7448 7449 Paths within the `--files-from` file will be interpreted as starting 7450 with the root specified in the command. Leading `/` characters are 7451 ignored. See [--files-from-raw](#files-from-raw-read-list-of-source-file-names-without-any-processing) 7452 if you need the input to be processed in a raw manner. 7453 7454 For example, suppose you had `files-from.txt` with this content: 7455 7456 # comment 7457 file1.jpg 7458 subdir/file2.jpg 7459 7460 You could then use it like this: 7461 7462 rclone copy --files-from files-from.txt /home/me/pics remote:pics 7463 7464 This will transfer these files only (if they exist) 7465 7466 /home/me/pics/file1.jpg → remote:pics/file1.jpg 7467 /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg 7468 7469 To take a more complicated example, let's say you had a few files you 7470 want to back up regularly with these absolute paths: 7471 7472 /home/user1/important 7473 /home/user1/dir/file 7474 /home/user2/stuff 7475 7476 To copy these you'd find a common subdirectory - in this case `/home` 7477 and put the remaining files in `files-from.txt` with or without 7478 leading `/`, eg 7479 7480 user1/important 7481 user1/dir/file 7482 user2/stuff 7483 7484 You could then copy these to a remote like this 7485 7486 rclone copy --files-from files-from.txt /home remote:backup 7487 7488 The 3 files will arrive in `remote:backup` with the paths as in the 7489 `files-from.txt` like this: 7490 7491 /home/user1/important → remote:backup/user1/important 7492 /home/user1/dir/file → remote:backup/user1/dir/file 7493 /home/user2/stuff → remote:backup/user2/stuff 7494 7495 You could of course choose `/` as the root too in which case your 7496 `files-from.txt` might look like this. 7497 7498 /home/user1/important 7499 /home/user1/dir/file 7500 /home/user2/stuff 7501 7502 And you would transfer it like this 7503 7504 rclone copy --files-from files-from.txt / remote:backup 7505 7506 In this case there will be an extra `home` directory on the remote: 7507 7508 /home/user1/important → remote:backup/home/user1/important 7509 /home/user1/dir/file → remote:backup/home/user1/dir/file 7510 /home/user2/stuff → remote:backup/home/user2/stuff 7511 7512 ### `--files-from-raw` - Read list of source-file names without any processing ### 7513 This option is same as `--files-from` with the only difference being that the input 7514 is read in a raw manner. This means that lines with leading/trailing whitespace and 7515 lines starting with `;` or `#` are read without any processing. [rclone lsf](https://rclone.org/commands/rclone_lsf/) 7516 has a compatible format that can be used to export file lists from remotes, which 7517 can then be used as an input to `--files-from-raw`. 7518 7519 ### `--min-size` - Don't transfer any file smaller than this ### 7520 7521 This option controls the minimum size file which will be transferred. 7522 This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be 7523 used. 7524 7525 For example `--min-size 50k` means no files smaller than 50kByte will be 7526 transferred. 7527 7528 ### `--max-size` - Don't transfer any file larger than this ### 7529 7530 This option controls the maximum size file which will be transferred. 7531 This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be 7532 used. 7533 7534 For example `--max-size 1G` means no files larger than 1GByte will be 7535 transferred. 7536 7537 ### `--max-age` - Don't transfer any file older than this ### 7538 7539 This option controls the maximum age of files to transfer. Give in 7540 seconds or with a suffix of: 7541 7542 * `ms` - Milliseconds 7543 * `s` - Seconds 7544 * `m` - Minutes 7545 * `h` - Hours 7546 * `d` - Days 7547 * `w` - Weeks 7548 * `M` - Months 7549 * `y` - Years 7550 7551 For example `--max-age 2d` means no files older than 2 days will be 7552 transferred. 7553 7554 This can also be an absolute time in one of these formats 7555 7556 - RFC3339 - eg "2006-01-02T15:04:05Z07:00" 7557 - ISO8601 Date and time, local timezone - "2006-01-02T15:04:05" 7558 - ISO8601 Date and time, local timezone - "2006-01-02 15:04:05" 7559 - ISO8601 Date - "2006-01-02" (YYYY-MM-DD) 7560 7561 ### `--min-age` - Don't transfer any file younger than this ### 7562 7563 This option controls the minimum age of files to transfer. Give in 7564 seconds or with a suffix (see `--max-age` for list of suffixes) 7565 7566 For example `--min-age 2d` means no files younger than 2 days will be 7567 transferred. 7568 7569 ### `--delete-excluded` - Delete files on dest excluded from sync ### 7570 7571 **Important** this flag is dangerous - use with `--dry-run` and `-v` first. 7572 7573 When doing `rclone sync` this will delete any files which are excluded 7574 from the sync on the destination. 7575 7576 If for example you did a sync from `A` to `B` without the `--min-size 50k` flag 7577 7578 rclone sync A: B: 7579 7580 Then you repeated it like this with the `--delete-excluded` 7581 7582 rclone --min-size 50k --delete-excluded sync A: B: 7583 7584 This would delete all files on `B` which are less than 50 kBytes as 7585 these are now excluded from the sync. 7586 7587 Always test first with `--dry-run` and `-v` before using this flag. 7588 7589 ### `--dump filters` - dump the filters to the output ### 7590 7591 This dumps the defined filters to the output as regular expressions. 7592 7593 Useful for debugging. 7594 7595 ### `--ignore-case` - make searches case insensitive ### 7596 7597 Normally filter patterns are case sensitive. If this flag is supplied 7598 then filter patterns become case insensitive. 7599 7600 Normally a `--include "file.txt"` will not match a file called 7601 `FILE.txt`. However if you use the `--ignore-case` flag then 7602 `--include "file.txt"` this will match a file called `FILE.txt`. 7603 7604 ## Quoting shell metacharacters ## 7605 7606 The examples above may not work verbatim in your shell as they have 7607 shell metacharacters in them (eg `*`), and may require quoting. 7608 7609 Eg linux, OSX 7610 7611 * `--include \*.jpg` 7612 * `--include '*.jpg'` 7613 * `--include='*.jpg'` 7614 7615 In Windows the expansion is done by the command not the shell so this 7616 should work fine 7617 7618 * `--include *.jpg` 7619 7620 ## Exclude directory based on a file ## 7621 7622 It is possible to exclude a directory based on a file, which is 7623 present in this directory. Filename should be specified using the 7624 `--exclude-if-present` flag. This flag has a priority over the other 7625 filtering flags. 7626 7627 Imagine, you have the following directory structure: 7628 7629 dir1/file1 7630 dir1/dir2/file2 7631 dir1/dir2/dir3/file3 7632 dir1/dir2/dir3/.ignore 7633 7634 You can exclude `dir3` from sync by running the following command: 7635 7636 rclone sync --exclude-if-present .ignore dir1 remote:backup 7637 7638 Currently only one filename is supported, i.e. `--exclude-if-present` 7639 should not be used multiple times. 7640 7641 # GUI (Experimental) 7642 7643 Rclone can serve a web based GUI (graphical user interface). This is 7644 somewhat experimental at the moment so things may be subject to 7645 change. 7646 7647 Run this command in a terminal and rclone will download and then 7648 display the GUI in a web browser. 7649 7650 ``` 7651 rclone rcd --rc-web-gui 7652 ``` 7653 7654 This will produce logs like this and rclone needs to continue to run to serve the GUI: 7655 7656 ``` 7657 2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip 7658 2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip] 7659 2019/08/25 11:40:16 NOTICE: Unzipping 7660 2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/ 7661 ``` 7662 7663 This assumes you are running rclone locally on your machine. It is 7664 possible to separate the rclone and the GUI - see below for details. 7665 7666 If you wish to check for updates then you can add `--rc-web-gui-update` 7667 to the command line. 7668 7669 If you find your GUI broken, you may force it to update by add `--rc-web-gui-force-update`. 7670 7671 By default, rclone will open your browser. Add `--rc-web-gui-no-open-browser` 7672 to disable this feature. 7673 7674 ## Using the GUI 7675 7676 Once the GUI opens, you will be looking at the dashboard which has an overall overview. 7677 7678 On the left hand side you will see a series of view buttons you can click on: 7679 7680 - Dashboard - main overview 7681 - Configs - examine and create new configurations 7682 - Explorer - view, download and upload files to the cloud storage systems 7683 - Backend - view or alter the backend config 7684 - Log out 7685 7686 (More docs and walkthrough video to come!) 7687 7688 ## How it works 7689 7690 When you run the `rclone rcd --rc-web-gui` this is what happens 7691 7692 - Rclone starts but only runs the remote control API ("rc"). 7693 - The API is bound to localhost with an auto generated username and password. 7694 - If the API bundle is missing then rclone will download it. 7695 - rclone will start serving the files from the API bundle over the same port as the API 7696 - rclone will open the browser with a `login_token` so it can log straight in. 7697 7698 ## Advanced use 7699 7700 The `rclone rcd` may use any of the [flags documented on the rc page](https://rclone.org/rc/#supported-parameters). 7701 7702 The flag `--rc-web-gui` is shorthand for 7703 7704 - Download the web GUI if necessary 7705 - Check we are using some authentication 7706 - `--rc-user gui` 7707 - `--rc-pass <random password>` 7708 - `--rc-serve` 7709 7710 These flags can be overridden as desired. 7711 7712 See also the [rclone rcd documentation](https://rclone.org/commands/rclone_rcd/). 7713 7714 ### Example: Running a public GUI 7715 7716 For example the GUI could be served on a public port over SSL using an htpasswd file using the following flags: 7717 7718 - `--rc-web-gui` 7719 - `--rc-addr :443` 7720 - `--rc-htpasswd /path/to/htpasswd` 7721 - `--rc-cert /path/to/ssl.crt` 7722 - `--rc-key /path/to/ssl.key` 7723 7724 ### Example: Running a GUI behind a proxy 7725 7726 If you want to run the GUI behind a proxy at `/rclone` you could use these flags: 7727 7728 - `--rc-web-gui` 7729 - `--rc-baseurl rclone` 7730 - `--rc-htpasswd /path/to/htpasswd` 7731 7732 Or instead of htpasswd if you just want a single user and password: 7733 7734 - `--rc-user me` 7735 - `--rc-pass mypassword` 7736 7737 ## Project 7738 7739 The GUI is being developed in the: [rclone/rclone-webui-react repository](https://github.com/rclone/rclone-webui-react). 7740 7741 Bug reports and contributions are very welcome :-) 7742 7743 If you have questions then please ask them on the [rclone forum](https://forum.rclone.org/). 7744 7745 # Remote controlling rclone with its API 7746 7747 If rclone is run with the `--rc` flag then it starts an http server 7748 which can be used to remote control rclone using its API. 7749 7750 If you just want to run a remote control then see the [rcd command](https://rclone.org/commands/rclone_rcd/). 7751 7752 ## Supported parameters 7753 7754 ### --rc 7755 7756 Flag to start the http server listen on remote requests 7757 7758 ### --rc-addr=IP 7759 7760 IPaddress:Port or :Port to bind server to. (default "localhost:5572") 7761 7762 ### --rc-cert=KEY 7763 SSL PEM key (concatenation of certificate and CA certificate) 7764 7765 ### --rc-client-ca=PATH 7766 Client certificate authority to verify clients with 7767 7768 ### --rc-htpasswd=PATH 7769 7770 htpasswd file - if not provided no authentication is done 7771 7772 ### --rc-key=PATH 7773 7774 SSL PEM Private key 7775 7776 ### --rc-max-header-bytes=VALUE 7777 7778 Maximum size of request header (default 4096) 7779 7780 ### --rc-user=VALUE 7781 7782 User name for authentication. 7783 7784 ### --rc-pass=VALUE 7785 7786 Password for authentication. 7787 7788 ### --rc-realm=VALUE 7789 7790 Realm for authentication (default "rclone") 7791 7792 ### --rc-server-read-timeout=DURATION 7793 7794 Timeout for server reading data (default 1h0m0s) 7795 7796 ### --rc-server-write-timeout=DURATION 7797 7798 Timeout for server writing data (default 1h0m0s) 7799 7800 ### --rc-serve 7801 7802 Enable the serving of remote objects via the HTTP interface. This 7803 means objects will be accessible at http://127.0.0.1:5572/ by default, 7804 so you can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/* 7805 to see a listing of the remotes. Objects may be requested from 7806 remotes using this syntax http://127.0.0.1:5572/[remote:path]/path/to/object 7807 7808 Default Off. 7809 7810 ### --rc-files /path/to/directory 7811 7812 Path to local files to serve on the HTTP server. 7813 7814 If this is set then rclone will serve the files in that directory. It 7815 will also open the root in the web browser if specified. This is for 7816 implementing browser based GUIs for rclone functions. 7817 7818 If `--rc-user` or `--rc-pass` is set then the URL that is opened will 7819 have the authorization in the URL in the `http://user:pass@localhost/` 7820 style. 7821 7822 Default Off. 7823 7824 ### --rc-enable-metrics 7825 7826 Enable OpenMetrics/Prometheus compatible endpoint at `/metrics`. 7827 7828 Default Off. 7829 7830 ### --rc-web-gui 7831 7832 Set this flag to serve the default web gui on the same port as rclone. 7833 7834 Default Off. 7835 7836 ### --rc-allow-origin 7837 7838 Set the allowed Access-Control-Allow-Origin for rc requests. 7839 7840 Can be used with --rc-web-gui if the rclone is running on different IP than the web-gui. 7841 7842 Default is IP address on which rc is running. 7843 7844 ### --rc-web-fetch-url 7845 7846 Set the URL to fetch the rclone-web-gui files from. 7847 7848 Default https://api.github.com/repos/rclone/rclone-webui-react/releases/latest. 7849 7850 ### --rc-web-gui-update 7851 7852 Set this flag to check and update rclone-webui-react from the rc-web-fetch-url. 7853 7854 Default Off. 7855 7856 ### --rc-web-gui-force-update 7857 7858 Set this flag to force update rclone-webui-react from the rc-web-fetch-url. 7859 7860 Default Off. 7861 7862 ### --rc-web-gui-no-open-browser 7863 7864 Set this flag to disable opening browser automatically when using web-gui. 7865 7866 Default Off. 7867 7868 ### --rc-job-expire-duration=DURATION 7869 7870 Expire finished async jobs older than DURATION (default 60s). 7871 7872 ### --rc-job-expire-interval=DURATION 7873 7874 Interval duration to check for expired async jobs (default 10s). 7875 7876 ### --rc-no-auth 7877 7878 By default rclone will require authorisation to have been set up on 7879 the rc interface in order to use any methods which access any rclone 7880 remotes. Eg `operations/list` is denied as it involved creating a 7881 remote as is `sync/copy`. 7882 7883 If this is set then no authorisation will be required on the server to 7884 use these methods. The alternative is to use `--rc-user` and 7885 `--rc-pass` and use these credentials in the request. 7886 7887 Default Off. 7888 7889 ## Accessing the remote control via the rclone rc command 7890 7891 Rclone itself implements the remote control protocol in its `rclone 7892 rc` command. 7893 7894 You can use it like this 7895 7896 ``` 7897 $ rclone rc rc/noop param1=one param2=two 7898 { 7899 "param1": "one", 7900 "param2": "two" 7901 } 7902 ``` 7903 7904 Run `rclone rc` on its own to see the help for the installed remote 7905 control commands. 7906 7907 `rclone rc` also supports a `--json` flag which can be used to send 7908 more complicated input parameters. 7909 7910 ``` 7911 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop 7912 { 7913 "p1": [ 7914 1, 7915 "2", 7916 null, 7917 4 7918 ], 7919 "p2": { 7920 "a": 1, 7921 "b": 2 7922 } 7923 } 7924 ``` 7925 7926 ## Special parameters 7927 7928 The rc interface supports some special parameters which apply to 7929 **all** commands. These start with `_` to show they are different. 7930 7931 ### Running asynchronous jobs with _async = true 7932 7933 Each rc call is classified as a job and it is assigned its own id. By default 7934 jobs are executed immediately as they are created or synchronously. 7935 7936 If `_async` has a true value when supplied to an rc call then it will 7937 return immediately with a job id and the task will be run in the 7938 background. The `job/status` call can be used to get information of 7939 the background job. The job can be queried for up to 1 minute after 7940 it has finished. 7941 7942 It is recommended that potentially long running jobs, eg `sync/sync`, 7943 `sync/copy`, `sync/move`, `operations/purge` are run with the `_async` 7944 flag to avoid any potential problems with the HTTP request and 7945 response timing out. 7946 7947 Starting a job with the `_async` flag: 7948 7949 ``` 7950 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop 7951 { 7952 "jobid": 2 7953 } 7954 ``` 7955 7956 Query the status to see if the job has finished. For more information 7957 on the meaning of these return parameters see the `job/status` call. 7958 7959 ``` 7960 $ rclone rc --json '{ "jobid":2 }' job/status 7961 { 7962 "duration": 0.000124163, 7963 "endTime": "2018-10-27T11:38:07.911245881+01:00", 7964 "error": "", 7965 "finished": true, 7966 "id": 2, 7967 "output": { 7968 "_async": true, 7969 "p1": [ 7970 1, 7971 "2", 7972 null, 7973 4 7974 ], 7975 "p2": { 7976 "a": 1, 7977 "b": 2 7978 } 7979 }, 7980 "startTime": "2018-10-27T11:38:07.911121728+01:00", 7981 "success": true 7982 } 7983 ``` 7984 7985 `job/list` can be used to show the running or recently completed jobs 7986 7987 ``` 7988 $ rclone rc job/list 7989 { 7990 "jobids": [ 7991 2 7992 ] 7993 } 7994 ``` 7995 7996 ### Assigning operations to groups with _group = value 7997 7998 Each rc call has its own stats group for tracking its metrics. By default 7999 grouping is done by the composite group name from prefix `job/` and id of the 8000 job like so `job/1`. 8001 8002 If `_group` has a value then stats for that request will be grouped under that 8003 value. This allows caller to group stats under their own name. 8004 8005 Stats for specific group can be accessed by passing `group` to `core/stats`: 8006 8007 ``` 8008 $ rclone rc --json '{ "group": "job/1" }' core/stats 8009 { 8010 "speed": 12345 8011 ... 8012 } 8013 ``` 8014 8015 ## Supported commands 8016 8017 ### backend/command: Runs a backend command. {#backend-command} 8018 8019 This takes the following parameters 8020 8021 - command - a string with the command name 8022 - fs - a remote name string eg "drive:" 8023 - arg - a list of arguments for the backend command 8024 - opt - a map of string to string of options 8025 8026 Returns 8027 8028 - result - result from the backend command 8029 8030 For example 8031 8032 rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2 8033 8034 Returns 8035 8036 ``` 8037 { 8038 "result": { 8039 "arg": [ 8040 "path1", 8041 "path2" 8042 ], 8043 "name": "noop", 8044 "opt": { 8045 "blue": "", 8046 "echo": "yes" 8047 } 8048 } 8049 } 8050 ``` 8051 8052 Note that this is the direct equivalent of using this "backend" 8053 command: 8054 8055 rclone backend noop . -o echo=yes -o blue path1 path2 8056 8057 Note that arguments must be preceded by the "-a" flag 8058 8059 See the [backend](https://rclone.org/commands/rclone_backend/) command for more information. 8060 8061 **Authentication is required for this call.** 8062 8063 ### cache/expire: Purge a remote from cache {#cache-expire} 8064 8065 Purge a remote from the cache backend. Supports either a directory or a file. 8066 Params: 8067 - remote = path to remote (required) 8068 - withData = true/false to delete cached data (chunks) as well (optional) 8069 8070 Eg 8071 8072 rclone rc cache/expire remote=path/to/sub/folder/ 8073 rclone rc cache/expire remote=/ withData=true 8074 8075 ### cache/fetch: Fetch file chunks {#cache-fetch} 8076 8077 Ensure the specified file chunks are cached on disk. 8078 8079 The chunks= parameter specifies the file chunks to check. 8080 It takes a comma separated list of array slice indices. 8081 The slice indices are similar to Python slices: start[:end] 8082 8083 start is the 0 based chunk number from the beginning of the file 8084 to fetch inclusive. end is 0 based chunk number from the beginning 8085 of the file to fetch exclusive. 8086 Both values can be negative, in which case they count from the back 8087 of the file. The value "-5:" represents the last 5 chunks of a file. 8088 8089 Some valid examples are: 8090 ":5,-5:" -> the first and last five chunks 8091 "0,-2" -> the first and the second last chunk 8092 "0:10" -> the first ten chunks 8093 8094 Any parameter with a key that starts with "file" can be used to 8095 specify files to fetch, eg 8096 8097 rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye 8098 8099 File names will automatically be encrypted when the a crypt remote 8100 is used on top of the cache. 8101 8102 ### cache/stats: Get cache stats {#cache-stats} 8103 8104 Show statistics for the cache remote. 8105 8106 ### config/create: create the config for a remote. {#config-create} 8107 8108 This takes the following parameters 8109 8110 - name - name of remote 8111 - parameters - a map of \{ "key": "value" \} pairs 8112 - type - type of the new remote 8113 - obscure - optional bool - forces obscuring of passwords 8114 - noObscure - optional bool - forces passwords not to be obscured 8115 8116 8117 See the [config create command](https://rclone.org/commands/rclone_config_create/) command for more information on the above. 8118 8119 **Authentication is required for this call.** 8120 8121 ### config/delete: Delete a remote in the config file. {#config-delete} 8122 8123 Parameters: 8124 8125 - name - name of remote to delete 8126 8127 See the [config delete command](https://rclone.org/commands/rclone_config_delete/) command for more information on the above. 8128 8129 **Authentication is required for this call.** 8130 8131 ### config/dump: Dumps the config file. {#config-dump} 8132 8133 Returns a JSON object: 8134 - key: value 8135 8136 Where keys are remote names and values are the config parameters. 8137 8138 See the [config dump command](https://rclone.org/commands/rclone_config_dump/) command for more information on the above. 8139 8140 **Authentication is required for this call.** 8141 8142 ### config/get: Get a remote in the config file. {#config-get} 8143 8144 Parameters: 8145 8146 - name - name of remote to get 8147 8148 See the [config dump command](https://rclone.org/commands/rclone_config_dump/) command for more information on the above. 8149 8150 **Authentication is required for this call.** 8151 8152 ### config/listremotes: Lists the remotes in the config file. {#config-listremotes} 8153 8154 Returns 8155 - remotes - array of remote names 8156 8157 See the [listremotes command](https://rclone.org/commands/rclone_listremotes/) command for more information on the above. 8158 8159 **Authentication is required for this call.** 8160 8161 ### config/password: password the config for a remote. {#config-password} 8162 8163 This takes the following parameters 8164 8165 - name - name of remote 8166 - parameters - a map of \{ "key": "value" \} pairs 8167 8168 8169 See the [config password command](https://rclone.org/commands/rclone_config_password/) command for more information on the above. 8170 8171 **Authentication is required for this call.** 8172 8173 ### config/providers: Shows how providers are configured in the config file. {#config-providers} 8174 8175 Returns a JSON object: 8176 - providers - array of objects 8177 8178 See the [config providers command](https://rclone.org/commands/rclone_config_providers/) command for more information on the above. 8179 8180 **Authentication is required for this call.** 8181 8182 ### config/update: update the config for a remote. {#config-update} 8183 8184 This takes the following parameters 8185 8186 - name - name of remote 8187 - parameters - a map of \{ "key": "value" \} pairs 8188 - obscure - optional bool - forces obscuring of passwords 8189 - noObscure - optional bool - forces passwords not to be obscured 8190 8191 8192 See the [config update command](https://rclone.org/commands/rclone_config_update/) command for more information on the above. 8193 8194 **Authentication is required for this call.** 8195 8196 ### core/bwlimit: Set the bandwidth limit. {#core-bwlimit} 8197 8198 This sets the bandwidth limit to that passed in. 8199 8200 Eg 8201 8202 rclone rc core/bwlimit rate=off 8203 { 8204 "bytesPerSecond": -1, 8205 "rate": "off" 8206 } 8207 rclone rc core/bwlimit rate=1M 8208 { 8209 "bytesPerSecond": 1048576, 8210 "rate": "1M" 8211 } 8212 8213 8214 If the rate parameter is not supplied then the bandwidth is queried 8215 8216 rclone rc core/bwlimit 8217 { 8218 "bytesPerSecond": 1048576, 8219 "rate": "1M" 8220 } 8221 8222 The format of the parameter is exactly the same as passed to --bwlimit 8223 except only one bandwidth may be specified. 8224 8225 In either case "rate" is returned as a human readable string, and 8226 "bytesPerSecond" is returned as a number. 8227 8228 ### core/gc: Runs a garbage collection. {#core-gc} 8229 8230 This tells the go runtime to do a garbage collection run. It isn't 8231 necessary to call this normally, but it can be useful for debugging 8232 memory problems. 8233 8234 ### core/group-list: Returns list of stats. {#core-group-list} 8235 8236 This returns list of stats groups currently in memory. 8237 8238 Returns the following values: 8239 ``` 8240 { 8241 "groups": an array of group names: 8242 [ 8243 "group1", 8244 "group2", 8245 ... 8246 ] 8247 } 8248 ``` 8249 8250 ### core/memstats: Returns the memory statistics {#core-memstats} 8251 8252 This returns the memory statistics of the running program. What the values mean 8253 are explained in the go docs: https://golang.org/pkg/runtime/#MemStats 8254 8255 The most interesting values for most people are: 8256 8257 * HeapAlloc: This is the amount of memory rclone is actually using 8258 * HeapSys: This is the amount of memory rclone has obtained from the OS 8259 * Sys: this is the total amount of memory requested from the OS 8260 * It is virtual memory so may include unused memory 8261 8262 ### core/obscure: Obscures a string passed in. {#core-obscure} 8263 8264 Pass a clear string and rclone will obscure it for the config file: 8265 - clear - string 8266 8267 Returns 8268 - obscured - string 8269 8270 ### core/pid: Return PID of current process {#core-pid} 8271 8272 This returns PID of current process. 8273 Useful for stopping rclone process. 8274 8275 ### core/quit: Terminates the app. {#core-quit} 8276 8277 (optional) Pass an exit code to be used for terminating the app: 8278 - exitCode - int 8279 8280 ### core/stats: Returns stats about current transfers. {#core-stats} 8281 8282 This returns all available stats: 8283 8284 rclone rc core/stats 8285 8286 If group is not provided then summed up stats for all groups will be 8287 returned. 8288 8289 Parameters 8290 8291 - group - name of the stats group (string) 8292 8293 Returns the following values: 8294 8295 ``` 8296 { 8297 "speed": average speed in bytes/sec since start of the process, 8298 "bytes": total transferred bytes since the start of the process, 8299 "errors": number of errors, 8300 "fatalError": whether there has been at least one FatalError, 8301 "retryError": whether there has been at least one non-NoRetryError, 8302 "checks": number of checked files, 8303 "transfers": number of transferred files, 8304 "deletes" : number of deleted files, 8305 "renames" : number of renamed files, 8306 "elapsedTime": time in seconds since the start of the process, 8307 "lastError": last occurred error, 8308 "transferring": an array of currently active file transfers: 8309 [ 8310 { 8311 "bytes": total transferred bytes for this file, 8312 "eta": estimated time in seconds until file transfer completion 8313 "name": name of the file, 8314 "percentage": progress of the file transfer in percent, 8315 "speed": speed in bytes/sec, 8316 "speedAvg": speed in bytes/sec as an exponentially weighted moving average, 8317 "size": size of the file in bytes 8318 } 8319 ], 8320 "checking": an array of names of currently active file checks 8321 [] 8322 } 8323 ``` 8324 Values for "transferring", "checking" and "lastError" are only assigned if data is available. 8325 The value for "eta" is null if an eta cannot be determined. 8326 8327 ### core/stats-delete: Delete stats group. {#core-stats-delete} 8328 8329 This deletes entire stats group 8330 8331 Parameters 8332 8333 - group - name of the stats group (string) 8334 8335 ### core/stats-reset: Reset stats. {#core-stats-reset} 8336 8337 This clears counters, errors and finished transfers for all stats or specific 8338 stats group if group is provided. 8339 8340 Parameters 8341 8342 - group - name of the stats group (string) 8343 8344 ### core/transferred: Returns stats about completed transfers. {#core-transferred} 8345 8346 This returns stats about completed transfers: 8347 8348 rclone rc core/transferred 8349 8350 If group is not provided then completed transfers for all groups will be 8351 returned. 8352 8353 Note only the last 100 completed transfers are returned. 8354 8355 Parameters 8356 8357 - group - name of the stats group (string) 8358 8359 Returns the following values: 8360 ``` 8361 { 8362 "transferred": an array of completed transfers (including failed ones): 8363 [ 8364 { 8365 "name": name of the file, 8366 "size": size of the file in bytes, 8367 "bytes": total transferred bytes for this file, 8368 "checked": if the transfer is only checked (skipped, deleted), 8369 "timestamp": integer representing millisecond unix epoch, 8370 "error": string description of the error (empty if successful), 8371 "jobid": id of the job that this transfer belongs to 8372 } 8373 ] 8374 } 8375 ``` 8376 8377 ### core/version: Shows the current version of rclone and the go runtime. {#core-version} 8378 8379 This shows the current version of go and the go runtime 8380 8381 - version - rclone version, eg "v1.44" 8382 - decomposed - version number as [major, minor, patch, subpatch] 8383 - note patch and subpatch will be 999 for a git compiled version 8384 - isGit - boolean - true if this was compiled from the git version 8385 - os - OS in use as according to Go 8386 - arch - cpu architecture in use according to Go 8387 - goVersion - version of Go runtime in use 8388 8389 ### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug-set-block-profile-rate} 8390 8391 SetBlockProfileRate controls the fraction of goroutine blocking events 8392 that are reported in the blocking profile. The profiler aims to sample 8393 an average of one blocking event per rate nanoseconds spent blocked. 8394 8395 To include every blocking event in the profile, pass rate = 1. To turn 8396 off profiling entirely, pass rate <= 0. 8397 8398 After calling this you can use this to see the blocking profile: 8399 8400 go tool pprof http://localhost:5572/debug/pprof/block 8401 8402 Parameters 8403 8404 - rate - int 8405 8406 ### debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling. {#debug-set-mutex-profile-fraction} 8407 8408 SetMutexProfileFraction controls the fraction of mutex contention 8409 events that are reported in the mutex profile. On average 1/rate 8410 events are reported. The previous rate is returned. 8411 8412 To turn off profiling entirely, pass rate 0. To just read the current 8413 rate, pass rate < 0. (For n>1 the details of sampling may change.) 8414 8415 Once this is set you can look use this to profile the mutex contention: 8416 8417 go tool pprof http://localhost:5572/debug/pprof/mutex 8418 8419 Parameters 8420 8421 - rate - int 8422 8423 Results 8424 8425 - previousRate - int 8426 8427 ### job/list: Lists the IDs of the running jobs {#job-list} 8428 8429 Parameters - None 8430 8431 Results 8432 8433 - jobids - array of integer job ids 8434 8435 ### job/status: Reads the status of the job ID {#job-status} 8436 8437 Parameters 8438 8439 - jobid - id of the job (integer) 8440 8441 Results 8442 8443 - finished - boolean 8444 - duration - time in seconds that the job ran for 8445 - endTime - time the job finished (eg "2018-10-26T18:50:20.528746884+01:00") 8446 - error - error from the job or empty string for no error 8447 - finished - boolean whether the job has finished or not 8448 - id - as passed in above 8449 - startTime - time the job started (eg "2018-10-26T18:50:20.528336039+01:00") 8450 - success - boolean - true for success false otherwise 8451 - output - output of the job as would have been returned if called synchronously 8452 - progress - output of the progress related to the underlying job 8453 8454 ### job/stop: Stop the running job {#job-stop} 8455 8456 Parameters 8457 8458 - jobid - id of the job (integer) 8459 8460 ### mount/mount: Create a new mount point {#mount-mount} 8461 8462 rclone allows Linux, FreeBSD, macOS and Windows to mount any of 8463 Rclone's cloud storage systems as a file system with FUSE. 8464 8465 If no mountType is provided, the priority is given as follows: 1. mount 2.cmount 3.mount2 8466 8467 This takes the following parameters 8468 8469 - fs - a remote path to be mounted (required) 8470 - mountPoint: valid path on the local machine (required) 8471 - mountType: One of the values (mount, cmount, mount2) specifies the mount implementation to use 8472 8473 Eg 8474 8475 rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint 8476 rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint mountType=mount 8477 8478 **Authentication is required for this call.** 8479 8480 ### mount/types: Show all possible mount types {#mount-types} 8481 8482 This shows all possible mount types and returns them as a list. 8483 8484 This takes no parameters and returns 8485 8486 - mountTypes: list of mount types 8487 8488 The mount types are strings like "mount", "mount2", "cmount" and can 8489 be passed to mount/mount as the mountType parameter. 8490 8491 Eg 8492 8493 rclone rc mount/types 8494 8495 **Authentication is required for this call.** 8496 8497 ### mount/unmount: Unmount all active mounts {#mount-unmount} 8498 8499 rclone allows Linux, FreeBSD, macOS and Windows to 8500 mount any of Rclone's cloud storage systems as a file system with 8501 FUSE. 8502 8503 This takes the following parameters 8504 8505 - mountPoint: valid path on the local machine where the mount was created (required) 8506 8507 Eg 8508 8509 rclone rc mount/unmount mountPoint=/home/<user>/mountPoint 8510 8511 **Authentication is required for this call.** 8512 8513 ### operations/about: Return the space used on the remote {#operations-about} 8514 8515 This takes the following parameters 8516 8517 - fs - a remote name string eg "drive:" 8518 8519 The result is as returned from rclone about --json 8520 8521 See the [about command](https://rclone.org/commands/rclone_size/) command for more information on the above. 8522 8523 **Authentication is required for this call.** 8524 8525 ### operations/cleanup: Remove trashed files in the remote or path {#operations-cleanup} 8526 8527 This takes the following parameters 8528 8529 - fs - a remote name string eg "drive:" 8530 8531 See the [cleanup command](https://rclone.org/commands/rclone_cleanup/) command for more information on the above. 8532 8533 **Authentication is required for this call.** 8534 8535 ### operations/copyfile: Copy a file from source remote to destination remote {#operations-copyfile} 8536 8537 This takes the following parameters 8538 8539 - srcFs - a remote name string eg "drive:" for the source 8540 - srcRemote - a path within that remote eg "file.txt" for the source 8541 - dstFs - a remote name string eg "drive2:" for the destination 8542 - dstRemote - a path within that remote eg "file2.txt" for the destination 8543 8544 **Authentication is required for this call.** 8545 8546 ### operations/copyurl: Copy the URL to the object {#operations-copyurl} 8547 8548 This takes the following parameters 8549 8550 - fs - a remote name string eg "drive:" 8551 - remote - a path within that remote eg "dir" 8552 - url - string, URL to read from 8553 - autoFilename - boolean, set to true to retrieve destination file name from url 8554 See the [copyurl command](https://rclone.org/commands/rclone_copyurl/) command for more information on the above. 8555 8556 **Authentication is required for this call.** 8557 8558 ### operations/delete: Remove files in the path {#operations-delete} 8559 8560 This takes the following parameters 8561 8562 - fs - a remote name string eg "drive:" 8563 8564 See the [delete command](https://rclone.org/commands/rclone_delete/) command for more information on the above. 8565 8566 **Authentication is required for this call.** 8567 8568 ### operations/deletefile: Remove the single file pointed to {#operations-deletefile} 8569 8570 This takes the following parameters 8571 8572 - fs - a remote name string eg "drive:" 8573 - remote - a path within that remote eg "dir" 8574 8575 See the [deletefile command](https://rclone.org/commands/rclone_deletefile/) command for more information on the above. 8576 8577 **Authentication is required for this call.** 8578 8579 ### operations/fsinfo: Return information about the remote {#operations-fsinfo} 8580 8581 This takes the following parameters 8582 8583 - fs - a remote name string eg "drive:" 8584 8585 This returns info about the remote passed in; 8586 8587 ``` 8588 { 8589 // optional features and whether they are available or not 8590 "Features": { 8591 "About": true, 8592 "BucketBased": false, 8593 "CanHaveEmptyDirectories": true, 8594 "CaseInsensitive": false, 8595 "ChangeNotify": false, 8596 "CleanUp": false, 8597 "Copy": false, 8598 "DirCacheFlush": false, 8599 "DirMove": true, 8600 "DuplicateFiles": false, 8601 "GetTier": false, 8602 "ListR": false, 8603 "MergeDirs": false, 8604 "Move": true, 8605 "OpenWriterAt": true, 8606 "PublicLink": false, 8607 "Purge": true, 8608 "PutStream": true, 8609 "PutUnchecked": false, 8610 "ReadMimeType": false, 8611 "ServerSideAcrossConfigs": false, 8612 "SetTier": false, 8613 "SetWrapper": false, 8614 "UnWrap": false, 8615 "WrapFs": false, 8616 "WriteMimeType": false 8617 }, 8618 // Names of hashes available 8619 "Hashes": [ 8620 "MD5", 8621 "SHA-1", 8622 "DropboxHash", 8623 "QuickXorHash" 8624 ], 8625 "Name": "local", // Name as created 8626 "Precision": 1, // Precision of timestamps in ns 8627 "Root": "/", // Path as created 8628 "String": "Local file system at /" // how the remote will appear in logs 8629 } 8630 ``` 8631 8632 This command does not have a command line equivalent so use this instead: 8633 8634 rclone rc --loopback operations/fsinfo fs=remote: 8635 8636 ### operations/list: List the given remote and path in JSON format {#operations-list} 8637 8638 This takes the following parameters 8639 8640 - fs - a remote name string eg "drive:" 8641 - remote - a path within that remote eg "dir" 8642 - opt - a dictionary of options to control the listing (optional) 8643 - recurse - If set recurse directories 8644 - noModTime - If set return modification time 8645 - showEncrypted - If set show decrypted names 8646 - showOrigIDs - If set show the IDs for each item if known 8647 - showHash - If set return a dictionary of hashes 8648 8649 The result is 8650 8651 - list 8652 - This is an array of objects as described in the lsjson command 8653 8654 See the [lsjson command](https://rclone.org/commands/rclone_lsjson/) for more information on the above and examples. 8655 8656 **Authentication is required for this call.** 8657 8658 ### operations/mkdir: Make a destination directory or container {#operations-mkdir} 8659 8660 This takes the following parameters 8661 8662 - fs - a remote name string eg "drive:" 8663 - remote - a path within that remote eg "dir" 8664 8665 See the [mkdir command](https://rclone.org/commands/rclone_mkdir/) command for more information on the above. 8666 8667 **Authentication is required for this call.** 8668 8669 ### operations/movefile: Move a file from source remote to destination remote {#operations-movefile} 8670 8671 This takes the following parameters 8672 8673 - srcFs - a remote name string eg "drive:" for the source 8674 - srcRemote - a path within that remote eg "file.txt" for the source 8675 - dstFs - a remote name string eg "drive2:" for the destination 8676 - dstRemote - a path within that remote eg "file2.txt" for the destination 8677 8678 **Authentication is required for this call.** 8679 8680 ### operations/publiclink: Create or retrieve a public link to the given file or folder. {#operations-publiclink} 8681 8682 This takes the following parameters 8683 8684 - fs - a remote name string eg "drive:" 8685 - remote - a path within that remote eg "dir" 8686 8687 Returns 8688 8689 - url - URL of the resource 8690 8691 See the [link command](https://rclone.org/commands/rclone_link/) command for more information on the above. 8692 8693 **Authentication is required for this call.** 8694 8695 ### operations/purge: Remove a directory or container and all of its contents {#operations-purge} 8696 8697 This takes the following parameters 8698 8699 - fs - a remote name string eg "drive:" 8700 - remote - a path within that remote eg "dir" 8701 8702 See the [purge command](https://rclone.org/commands/rclone_purge/) command for more information on the above. 8703 8704 **Authentication is required for this call.** 8705 8706 ### operations/rmdir: Remove an empty directory or container {#operations-rmdir} 8707 8708 This takes the following parameters 8709 8710 - fs - a remote name string eg "drive:" 8711 - remote - a path within that remote eg "dir" 8712 8713 See the [rmdir command](https://rclone.org/commands/rclone_rmdir/) command for more information on the above. 8714 8715 **Authentication is required for this call.** 8716 8717 ### operations/rmdirs: Remove all the empty directories in the path {#operations-rmdirs} 8718 8719 This takes the following parameters 8720 8721 - fs - a remote name string eg "drive:" 8722 - remote - a path within that remote eg "dir" 8723 - leaveRoot - boolean, set to true not to delete the root 8724 8725 See the [rmdirs command](https://rclone.org/commands/rclone_rmdirs/) command for more information on the above. 8726 8727 **Authentication is required for this call.** 8728 8729 ### operations/size: Count the number of bytes and files in remote {#operations-size} 8730 8731 This takes the following parameters 8732 8733 - fs - a remote name string eg "drive:path/to/dir" 8734 8735 Returns 8736 8737 - count - number of files 8738 - bytes - number of bytes in those files 8739 8740 See the [size command](https://rclone.org/commands/rclone_size/) command for more information on the above. 8741 8742 **Authentication is required for this call.** 8743 8744 ### options/blocks: List all the option blocks {#options-blocks} 8745 8746 Returns 8747 - options - a list of the options block names 8748 8749 ### options/get: Get all the options {#options-get} 8750 8751 Returns an object where keys are option block names and values are an 8752 object with the current option values in. 8753 8754 This shows the internal names of the option within rclone which should 8755 map to the external options very easily with a few exceptions. 8756 8757 ### options/set: Set an option {#options-set} 8758 8759 Parameters 8760 8761 - option block name containing an object with 8762 - key: value 8763 8764 Repeated as often as required. 8765 8766 Only supply the options you wish to change. If an option is unknown 8767 it will be silently ignored. Not all options will have an effect when 8768 changed like this. 8769 8770 For example: 8771 8772 This sets DEBUG level logs (-vv) 8773 8774 rclone rc options/set --json '{"main": {"LogLevel": 8}}' 8775 8776 And this sets INFO level logs (-v) 8777 8778 rclone rc options/set --json '{"main": {"LogLevel": 7}}' 8779 8780 And this sets NOTICE level logs (normal without -v) 8781 8782 rclone rc options/set --json '{"main": {"LogLevel": 6}}' 8783 8784 ### rc/error: This returns an error {#rc-error} 8785 8786 This returns an error with the input as part of its error string. 8787 Useful for testing error handling. 8788 8789 ### rc/list: List all the registered remote control commands {#rc-list} 8790 8791 This lists all the registered remote control commands as a JSON map in 8792 the commands response. 8793 8794 ### rc/noop: Echo the input to the output parameters {#rc-noop} 8795 8796 This echoes the input parameters to the output parameters for testing 8797 purposes. It can be used to check that rclone is still alive and to 8798 check that parameter passing is working properly. 8799 8800 ### rc/noopauth: Echo the input to the output parameters requiring auth {#rc-noopauth} 8801 8802 This echoes the input parameters to the output parameters for testing 8803 purposes. It can be used to check that rclone is still alive and to 8804 check that parameter passing is working properly. 8805 8806 **Authentication is required for this call.** 8807 8808 ### sync/copy: copy a directory from source remote to destination remote {#sync-copy} 8809 8810 This takes the following parameters 8811 8812 - srcFs - a remote name string eg "drive:src" for the source 8813 - dstFs - a remote name string eg "drive:dst" for the destination 8814 8815 8816 See the [copy command](https://rclone.org/commands/rclone_copy/) command for more information on the above. 8817 8818 **Authentication is required for this call.** 8819 8820 ### sync/move: move a directory from source remote to destination remote {#sync-move} 8821 8822 This takes the following parameters 8823 8824 - srcFs - a remote name string eg "drive:src" for the source 8825 - dstFs - a remote name string eg "drive:dst" for the destination 8826 - deleteEmptySrcDirs - delete empty src directories if set 8827 8828 8829 See the [move command](https://rclone.org/commands/rclone_move/) command for more information on the above. 8830 8831 **Authentication is required for this call.** 8832 8833 ### sync/sync: sync a directory from source remote to destination remote {#sync-sync} 8834 8835 This takes the following parameters 8836 8837 - srcFs - a remote name string eg "drive:src" for the source 8838 - dstFs - a remote name string eg "drive:dst" for the destination 8839 8840 8841 See the [sync command](https://rclone.org/commands/rclone_sync/) command for more information on the above. 8842 8843 **Authentication is required for this call.** 8844 8845 ### vfs/forget: Forget files or directories in the directory cache. {#vfs-forget} 8846 8847 This forgets the paths in the directory cache causing them to be 8848 re-read from the remote when needed. 8849 8850 If no paths are passed in then it will forget all the paths in the 8851 directory cache. 8852 8853 rclone rc vfs/forget 8854 8855 Otherwise pass files or dirs in as file=path or dir=path. Any 8856 parameter key starting with file will forget that file and any 8857 starting with dir will forget that dir, eg 8858 8859 rclone rc vfs/forget file=hello file2=goodbye dir=home/junk 8860 8861 ### vfs/poll-interval: Get the status or update the value of the poll-interval option. {#vfs-poll-interval} 8862 8863 Without any parameter given this returns the current status of the 8864 poll-interval setting. 8865 8866 When the interval=duration parameter is set, the poll-interval value 8867 is updated and the polling function is notified. 8868 Setting interval=0 disables poll-interval. 8869 8870 rclone rc vfs/poll-interval interval=5m 8871 8872 The timeout=duration parameter can be used to specify a time to wait 8873 for the current poll function to apply the new value. 8874 If timeout is less or equal 0, which is the default, wait indefinitely. 8875 8876 The new poll-interval value will only be active when the timeout is 8877 not reached. 8878 8879 If poll-interval is updated or disabled temporarily, some changes 8880 might not get picked up by the polling function, depending on the 8881 used remote. 8882 8883 ### vfs/refresh: Refresh the directory cache. {#vfs-refresh} 8884 8885 This reads the directories for the specified paths and freshens the 8886 directory cache. 8887 8888 If no paths are passed in then it will refresh the root directory. 8889 8890 rclone rc vfs/refresh 8891 8892 Otherwise pass directories in as dir=path. Any parameter key 8893 starting with dir will refresh that directory, eg 8894 8895 rclone rc vfs/refresh dir=home/junk dir2=data/misc 8896 8897 If the parameter recursive=true is given the whole directory tree 8898 will get refreshed. This refresh will use --fast-list if enabled. 8899 8900 8901 8902 ## Accessing the remote control via HTTP 8903 8904 Rclone implements a simple HTTP based protocol. 8905 8906 Each endpoint takes an JSON object and returns a JSON object or an 8907 error. The JSON objects are essentially a map of string names to 8908 values. 8909 8910 All calls must made using POST. 8911 8912 The input objects can be supplied using URL parameters, POST 8913 parameters or by supplying "Content-Type: application/json" and a JSON 8914 blob in the body. There are examples of these below using `curl`. 8915 8916 The response will be a JSON blob in the body of the response. This is 8917 formatted to be reasonably human readable. 8918 8919 ### Error returns 8920 8921 If an error occurs then there will be an HTTP error status (eg 500) 8922 and the body of the response will contain a JSON encoded error object, 8923 eg 8924 8925 ``` 8926 { 8927 "error": "Expecting string value for key \"remote\" (was float64)", 8928 "input": { 8929 "fs": "/tmp", 8930 "remote": 3 8931 }, 8932 "status": 400 8933 "path": "operations/rmdir", 8934 } 8935 ``` 8936 8937 The keys in the error response are 8938 - error - error string 8939 - input - the input parameters to the call 8940 - status - the HTTP status code 8941 - path - the path of the call 8942 8943 ### CORS 8944 8945 The sever implements basic CORS support and allows all origins for that. 8946 The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back. 8947 8948 ### Using POST with URL parameters only 8949 8950 ``` 8951 curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2' 8952 ``` 8953 8954 Response 8955 8956 ``` 8957 { 8958 "potato": "1", 8959 "sausage": "2" 8960 } 8961 ``` 8962 8963 Here is what an error response looks like: 8964 8965 ``` 8966 curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 8967 ``` 8968 8969 ``` 8970 { 8971 "error": "arbitrary error on input map[potato:1 sausage:2]", 8972 "input": { 8973 "potato": "1", 8974 "sausage": "2" 8975 } 8976 } 8977 ``` 8978 8979 Note that curl doesn't return errors to the shell unless you use the `-f` option 8980 8981 ``` 8982 $ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 8983 curl: (22) The requested URL returned error: 400 Bad Request 8984 $ echo $? 8985 22 8986 ``` 8987 8988 ### Using POST with a form 8989 8990 ``` 8991 curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop 8992 ``` 8993 8994 Response 8995 8996 ``` 8997 { 8998 "potato": "1", 8999 "sausage": "2" 9000 } 9001 ``` 9002 9003 Note that you can combine these with URL parameters too with the POST 9004 parameters taking precedence. 9005 9006 ``` 9007 curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4" 9008 ``` 9009 9010 Response 9011 9012 ``` 9013 { 9014 "potato": "1", 9015 "rutabaga": "3", 9016 "sausage": "4" 9017 } 9018 9019 ``` 9020 9021 ### Using POST with a JSON blob 9022 9023 ``` 9024 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop 9025 ``` 9026 9027 response 9028 9029 ``` 9030 { 9031 "password": "xyz", 9032 "username": "xyz" 9033 } 9034 ``` 9035 9036 This can be combined with URL parameters too if required. The JSON 9037 blob takes precedence. 9038 9039 ``` 9040 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4' 9041 ``` 9042 9043 ``` 9044 { 9045 "potato": 2, 9046 "rutabaga": "3", 9047 "sausage": 1 9048 } 9049 ``` 9050 9051 ## Debugging rclone with pprof ## 9052 9053 If you use the `--rc` flag this will also enable the use of the go 9054 profiling tools on the same port. 9055 9056 To use these, first [install go](https://golang.org/doc/install). 9057 9058 ### Debugging memory use 9059 9060 To profile rclone's memory use you can run: 9061 9062 go tool pprof -web http://localhost:5572/debug/pprof/heap 9063 9064 This should open a page in your browser showing what is using what 9065 memory. 9066 9067 You can also use the `-text` flag to produce a textual summary 9068 9069 ``` 9070 $ go tool pprof -text http://localhost:5572/debug/pprof/heap 9071 Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total 9072 flat flat% sum% cum cum% 9073 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode 9074 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize 9075 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init 9076 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init 9077 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init 9078 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init 9079 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init 9080 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0 9081 0 0% 100% 1024.03kB 66.62% main.init 9082 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest 9083 0 0% 100% 513kB 33.38% net/http.(*conn).serve 9084 0 0% 100% 1024.03kB 66.62% runtime.main 9085 ``` 9086 9087 ### Debugging go routine leaks 9088 9089 Memory leaks are most often caused by go routine leaks keeping memory 9090 alive which should have been garbage collected. 9091 9092 See all active go routines using 9093 9094 curl http://localhost:5572/debug/pprof/goroutine?debug=1 9095 9096 Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your browser. 9097 9098 ### Other profiles to look at 9099 9100 You can see a summary of profiles available at http://localhost:5572/debug/pprof/ 9101 9102 Here is how to use some of them: 9103 9104 - Memory: `go tool pprof http://localhost:5572/debug/pprof/heap` 9105 - Go routines: `curl http://localhost:5572/debug/pprof/goroutine?debug=1` 9106 - 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile` 9107 - 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5` 9108 - Goroutine blocking profile 9109 - Enable first with: `rclone rc debug/set-block-profile-rate rate=1` ([docs](#debug/set-block-profile-rate)) 9110 - `go tool pprof http://localhost:5572/debug/pprof/block` 9111 - Contended mutexes: 9112 - Enable first with: `rclone rc debug/set-mutex-profile-fraction rate=1` ([docs](#debug/set-mutex-profile-fraction)) 9113 - `go tool pprof http://localhost:5572/debug/pprof/mutex` 9114 9115 See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/) 9116 for more info on how to use the profiling and for a general overview 9117 see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs). 9118 9119 The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234). 9120 9121 # Overview of cloud storage systems # 9122 9123 Each cloud storage system is slightly different. Rclone attempts to 9124 provide a unified interface to them, but some underlying differences 9125 show through. 9126 9127 ## Features ## 9128 9129 Here is an overview of the major features of each cloud storage system. 9130 9131 | Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | 9132 | ---------------------------- |:-----------:|:-------:|:----------------:|:---------------:|:---------:| 9133 | 1Fichier | Whirlpool | No | No | Yes | R | 9134 | Amazon Drive | MD5 | No | Yes | No | R | 9135 | Amazon S3 | MD5 | Yes | No | No | R/W | 9136 | Backblaze B2 | SHA1 | Yes | No | No | R/W | 9137 | Box | SHA1 | Yes | Yes | No | - | 9138 | Citrix ShareFile | MD5 | Yes | Yes | No | - | 9139 | Dropbox | DBHASH † | Yes | Yes | No | - | 9140 | FTP | - | No | No | No | - | 9141 | Google Cloud Storage | MD5 | Yes | No | No | R/W | 9142 | Google Drive | MD5 | Yes | No | Yes | R/W | 9143 | Google Photos | - | No | No | Yes | R | 9144 | HTTP | - | No | No | No | R | 9145 | Hubic | MD5 | Yes | No | No | R/W | 9146 | Jottacloud | MD5 | Yes | Yes | No | R/W | 9147 | Koofr | MD5 | No | Yes | No | - | 9148 | Mail.ru Cloud | Mailru ‡‡‡ | Yes | Yes | No | - | 9149 | Mega | - | No | No | Yes | - | 9150 | Memory | MD5 | Yes | No | No | - | 9151 | Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W | 9152 | Microsoft OneDrive | SHA1 ‡‡ | Yes | Yes | No | R | 9153 | OpenDrive | MD5 | Yes | Yes | No | - | 9154 | OpenStack Swift | MD5 | Yes | No | No | R/W | 9155 | pCloud | MD5, SHA1 | Yes | No | No | W | 9156 | premiumize.me | - | No | Yes | No | R | 9157 | put.io | CRC-32 | Yes | No | Yes | R | 9158 | QingStor | MD5 | No | No | No | R/W | 9159 | Seafile | - | No | No | No | - | 9160 | SFTP | MD5, SHA1 ‡ | Yes | Depends | No | - | 9161 | SugarSync | - | No | No | No | - | 9162 | Tardigrade | - | Yes | No | No | - | 9163 | WebDAV | MD5, SHA1 ††| Yes ††† | Depends | No | - | 9164 | Yandex Disk | MD5 | Yes | No | No | R/W | 9165 | The local filesystem | All | Yes | Depends | No | - | 9166 9167 ### Hash ### 9168 9169 The cloud storage system supports various hash types of the objects. 9170 The hashes are used when transferring data as an integrity check and 9171 can be specifically used with the `--checksum` flag in syncs and in 9172 the `check` command. 9173 9174 To use the verify checksums when transferring between cloud storage 9175 systems they must support a common hash type. 9176 9177 † Note that Dropbox supports [its own custom 9178 hash](https://www.dropbox.com/developers/reference/content-hash). 9179 This is an SHA256 sum of all the 4MB block SHA256s. 9180 9181 ‡ SFTP supports checksums if the same login has shell access and `md5sum` 9182 or `sha1sum` as well as `echo` are in the remote's PATH. 9183 9184 †† WebDAV supports hashes when used with Owncloud and Nextcloud only. 9185 9186 ††† WebDAV supports modtimes when used with Owncloud and Nextcloud only. 9187 9188 ‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive 9189 for business and SharePoint server support Microsoft's own 9190 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 9191 9192 ‡‡‡ Mail.ru uses its own modified SHA1 hash 9193 9194 ### ModTime ### 9195 9196 The cloud storage system supports setting modification times on 9197 objects. If it does then this enables a using the modification times 9198 as part of the sync. If not then only the size will be checked by 9199 default, though the MD5SUM can be checked with the `--checksum` flag. 9200 9201 All cloud storage systems support some kind of date on the object and 9202 these will be set when transferring from the cloud storage system. 9203 9204 ### Case Insensitive ### 9205 9206 If a cloud storage systems is case sensitive then it is possible to 9207 have two files which differ only in case, eg `file.txt` and 9208 `FILE.txt`. If a cloud storage system is case insensitive then that 9209 isn't possible. 9210 9211 This can cause problems when syncing between a case insensitive 9212 system and a case sensitive system. The symptom of this is that no 9213 matter how many times you run the sync it never completes fully. 9214 9215 The local filesystem and SFTP may or may not be case sensitive 9216 depending on OS. 9217 9218 * Windows - usually case insensitive, though case is preserved 9219 * OSX - usually case insensitive, though it is possible to format case sensitive 9220 * Linux - usually case sensitive, but there are case insensitive file systems (eg FAT formatted USB keys) 9221 9222 Most of the time this doesn't cause any problems as people tend to 9223 avoid files whose name differs only by case even on case sensitive 9224 systems. 9225 9226 ### Duplicate files ### 9227 9228 If a cloud storage system allows duplicate files then it can have two 9229 objects with the same name. 9230 9231 This confuses rclone greatly when syncing - use the `rclone dedupe` 9232 command to rename or remove duplicates. 9233 9234 ### Restricted filenames ### 9235 9236 Some cloud storage systems might have restrictions on the characters 9237 that are usable in file or directory names. 9238 When `rclone` detects such a name during a file upload, it will 9239 transparently replace the restricted characters with similar looking 9240 Unicode characters. 9241 9242 This process is designed to avoid ambiguous file names as much as 9243 possible and allow to move files between many cloud storage systems 9244 transparently. 9245 9246 The name shown by `rclone` to the user or during log output will only 9247 contain a minimal set of [replaced characters](#restricted-characters) 9248 to ensure correct formatting and not necessarily the actual name used 9249 on the cloud storage. 9250 9251 This transformation is reversed when downloading a file or parsing 9252 `rclone` arguments. 9253 For example, when uploading a file named `my file?.txt` to Onedrive 9254 will be displayed as `my file?.txt` on the console, but stored as 9255 `my file?.txt` (the `?` gets replaced by the similar looking `?` 9256 character) to Onedrive. 9257 The reverse transformation allows to read a file`unusual/name.txt` 9258 from Google Drive, by passing the name `unusual/name.txt` (the `/` needs 9259 to be replaced by the similar looking `/` character) on the command line. 9260 9261 #### Default restricted characters {#restricted-characters} 9262 9263 The table below shows the characters that are replaced by default. 9264 9265 When a replacement character is found in a filename, this character 9266 will be escaped with the `‛` character to avoid ambiguous file names. 9267 (e.g. a file named `␀.txt` would shown as `‛␀.txt`) 9268 9269 Each cloud storage backend can use a different set of characters, 9270 which will be specified in the documentation for each backend. 9271 9272 | Character | Value | Replacement | 9273 | --------- |:-----:|:-----------:| 9274 | NUL | 0x00 | ␀ | 9275 | SOH | 0x01 | ␁ | 9276 | STX | 0x02 | ␂ | 9277 | ETX | 0x03 | ␃ | 9278 | EOT | 0x04 | ␄ | 9279 | ENQ | 0x05 | ␅ | 9280 | ACK | 0x06 | ␆ | 9281 | BEL | 0x07 | ␇ | 9282 | BS | 0x08 | ␈ | 9283 | HT | 0x09 | ␉ | 9284 | LF | 0x0A | ␊ | 9285 | VT | 0x0B | ␋ | 9286 | FF | 0x0C | ␌ | 9287 | CR | 0x0D | ␍ | 9288 | SO | 0x0E | ␎ | 9289 | SI | 0x0F | ␏ | 9290 | DLE | 0x10 | ␐ | 9291 | DC1 | 0x11 | ␑ | 9292 | DC2 | 0x12 | ␒ | 9293 | DC3 | 0x13 | ␓ | 9294 | DC4 | 0x14 | ␔ | 9295 | NAK | 0x15 | ␕ | 9296 | SYN | 0x16 | ␖ | 9297 | ETB | 0x17 | ␗ | 9298 | CAN | 0x18 | ␘ | 9299 | EM | 0x19 | ␙ | 9300 | SUB | 0x1A | ␚ | 9301 | ESC | 0x1B | ␛ | 9302 | FS | 0x1C | ␜ | 9303 | GS | 0x1D | ␝ | 9304 | RS | 0x1E | ␞ | 9305 | US | 0x1F | ␟ | 9306 | / | 0x2F | / | 9307 | DEL | 0x7F | ␡ | 9308 9309 The default encoding will also encode these file names as they are 9310 problematic with many cloud storage systems. 9311 9312 | File name | Replacement | 9313 | --------- |:-----------:| 9314 | . | . | 9315 | .. | .. | 9316 9317 #### Invalid UTF-8 bytes {#invalid-utf8} 9318 9319 Some backends only support a sequence of well formed UTF-8 bytes 9320 as file or directory names. 9321 9322 In this case all invalid UTF-8 bytes will be replaced with a quoted 9323 representation of the byte value to allow uploading a file to such a 9324 backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`. 9325 9326 A common source of invalid UTF-8 bytes are local filesystems, that store 9327 names in a different encoding than UTF-8 or UTF-16, like latin1. See the 9328 [local filenames](https://rclone.org/local/#filenames) section for details. 9329 9330 #### Encoding option {#encoding} 9331 9332 Most backends have an encoding options, specified as a flag 9333 `--backend-encoding` where `backend` is the name of the backend, or as 9334 a config parameter `encoding` (you'll need to select the Advanced 9335 config in `rclone config` to see it). 9336 9337 This will have default value which encodes and decodes characters in 9338 such a way as to preserve the maximum number of characters (see 9339 above). 9340 9341 However this can be incorrect in some scenarios, for example if you 9342 have a Windows file system with characters such as `*` and `?` that 9343 you want to remain as those characters on the remote rather than being 9344 translated to `*` and `?`. 9345 9346 The `--backend-encoding` flags allow you to change that. You can 9347 disable the encoding completely with `--backend-encoding None` or set 9348 `encoding = None` in the config file. 9349 9350 Encoding takes a comma separated list of encodings. You can see the 9351 list of all available characters by passing an invalid value to this 9352 flag, eg `--local-encoding "help"` and `rclone help flags encoding` 9353 will show you the defaults for the backends. 9354 9355 | Encoding | Characters | 9356 | --------- | ---------- | 9357 | Asterisk | `*` | 9358 | BackQuote | `` ` `` | 9359 | BackSlash | `\` | 9360 | Colon | `:` | 9361 | CrLf | CR 0x0D, LF 0x0A | 9362 | Ctl | All control characters 0x00-0x1F | 9363 | Del | DEL 0x7F | 9364 | Dollar | `$` | 9365 | Dot | `.` | 9366 | DoubleQuote | `"` | 9367 | Hash | `#` | 9368 | InvalidUtf8 | An invalid UTF-8 character (eg latin1) | 9369 | LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string | 9370 | LeftPeriod | `.` on the left of a string | 9371 | LeftSpace | SPACE on the left of a string | 9372 | LeftTilde | `~` on the left of a string | 9373 | LtGt | `<`, `>` | 9374 | None | No characters are encoded | 9375 | Percent | `%` | 9376 | Pipe | \| | 9377 | Question | `?` | 9378 | RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | 9379 | RightPeriod | `.` on the right of a string | 9380 | RightSpace | SPACE on the right of a string | 9381 | SingleQuote | `'` | 9382 | Slash | `/` | 9383 9384 To take a specific example, the FTP backend's default encoding is 9385 9386 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 9387 9388 However, let's say the FTP server is running on Windows and can't have 9389 any of the invalid Windows characters in file names. You are backing 9390 up Linux servers to this FTP server which do have those characters in 9391 file names. So you would add the Windows set which are 9392 9393 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 9394 9395 to the existing ones, giving: 9396 9397 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 9398 9399 This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file. 9400 9401 Or let's say you have a Windows server but you want to preserve `*` 9402 and `?`, you would then have this as the encoding (the Windows 9403 encoding minus `Asterisk` and `Question`). 9404 9405 Slash,LtGt,DoubleQuote,Colon,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 9406 9407 This can be specified using the `--local-encoding` flag or using an 9408 `encoding` parameter in the config file. 9409 9410 ### MIME Type ### 9411 9412 MIME types (also known as media types) classify types of documents 9413 using a simple text classification, eg `text/html` or 9414 `application/pdf`. 9415 9416 Some cloud storage systems support reading (`R`) the MIME type of 9417 objects and some support writing (`W`) the MIME type of objects. 9418 9419 The MIME type can be important if you are serving files directly to 9420 HTTP from the storage system. 9421 9422 If you are copying from a remote which supports reading (`R`) to a 9423 remote which supports writing (`W`) then rclone will preserve the MIME 9424 types. Otherwise they will be guessed from the extension, or the 9425 remote itself may assign the MIME type. 9426 9427 ## Optional Features ## 9428 9429 All the remotes support a basic set of features, but there are some 9430 optional features supported by some remotes used to make some 9431 operations more efficient. 9432 9433 | Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir | 9434 | ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:| :------: | 9435 | 1Fichier | No | No | No | No | No | No | No | No | No | Yes | 9436 | Amazon Drive | Yes | No | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 9437 | Amazon S3 | No | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 9438 | Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 9439 | Box | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | No | Yes | 9440 | Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | 9441 | Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | Yes | Yes | 9442 | FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 9443 | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 9444 | Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 9445 | Google Photos | No | No | No | No | No | No | No | No | No | No | 9446 | HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 9447 | Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 9448 | Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 9449 | Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 9450 | Mega | Yes | No | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 9451 | Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | 9452 | Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 9453 | Microsoft OneDrive | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | Yes | Yes | Yes | 9454 | OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | 9455 | OpenStack Swift | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 9456 | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 9457 | premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes | 9458 | put.io | Yes | No | Yes | Yes | Yes | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 9459 | QingStor | No | Yes | No | No | Yes | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 9460 | Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 9461 | SFTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 9462 | SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | 9463 | Tardigrade | Yes † | No | No | No | No | Yes | Yes | No | No | No | 9464 | WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 9465 | Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | 9466 | The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 9467 9468 ### Purge ### 9469 9470 This deletes a directory quicker than just deleting all the files in 9471 the directory. 9472 9473 † Note Swift, Hubic, and Tardigrade implement this in order to delete 9474 directory markers but they don't actually have a quicker way of deleting 9475 files other than deleting them individually. 9476 9477 ‡ StreamUpload is not supported with Nextcloud 9478 9479 ### Copy ### 9480 9481 Used when copying an object to and from the same remote. This known 9482 as a server side copy so you can copy a file without downloading it 9483 and uploading it again. It is used if you use `rclone copy` or 9484 `rclone move` if the remote doesn't support `Move` directly. 9485 9486 If the server doesn't support `Copy` directly then for copy operations 9487 the file is downloaded then re-uploaded. 9488 9489 ### Move ### 9490 9491 Used when moving/renaming an object on the same remote. This is known 9492 as a server side move of a file. This is used in `rclone move` if the 9493 server doesn't support `DirMove`. 9494 9495 If the server isn't capable of `Move` then rclone simulates it with 9496 `Copy` then delete. If the server doesn't support `Copy` then rclone 9497 will download the file and re-upload it. 9498 9499 ### DirMove ### 9500 9501 This is used to implement `rclone move` to move a directory if 9502 possible. If it isn't then it will use `Move` on each file (which 9503 falls back to `Copy` then download and upload - see `Move` section). 9504 9505 ### CleanUp ### 9506 9507 This is used for emptying the trash for a remote by `rclone cleanup`. 9508 9509 If the server can't do `CleanUp` then `rclone cleanup` will return an 9510 error. 9511 9512 ### ListR ### 9513 9514 The remote supports a recursive list to list all the contents beneath 9515 a directory quickly. This enables the `--fast-list` flag to work. 9516 See the [rclone docs](https://rclone.org/docs/#fast-list) for more details. 9517 9518 ### StreamUpload ### 9519 9520 Some remotes allow files to be uploaded without knowing the file size 9521 in advance. This allows certain operations to work without spooling the 9522 file to local disk first, e.g. `rclone rcat`. 9523 9524 ### LinkSharing ### 9525 9526 Sets the necessary permissions on a file or folder and prints a link 9527 that allows others to access them, even if they don't have an account 9528 on the particular cloud provider. 9529 9530 ### About ### 9531 9532 This is used to fetch quota information from the remote, like bytes 9533 used/free/quota and bytes used in the trash. 9534 9535 This is also used to return the space used, available for `rclone mount`. 9536 9537 If the server can't do `About` then `rclone about` will return an 9538 error. 9539 9540 ### EmptyDir ### 9541 9542 The remote supports empty directories. See [Limitations](https://rclone.org/bugs/#limitations) 9543 for details. Most Object/Bucket based remotes do not support this. 9544 9545 # Global Flags 9546 9547 This describes the global flags available to every rclone command 9548 split into two groups, non backend and backend flags. 9549 9550 ## Non Backend Flags 9551 9552 These flags are available for every command. 9553 9554 ``` 9555 --ask-password Allow prompt for password for encrypted configuration. (default true) 9556 --auto-confirm If enabled, do not request console confirmation. 9557 --backup-dir string Make backups into hierarchy based in DIR. 9558 --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name. 9559 --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer. (default 16M) 9560 --bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable. 9561 --ca-cert string CA certificate used to verify servers 9562 --cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone") 9563 --check-first Do all the checks before starting transfers. 9564 --checkers int Number of checkers to run in parallel. (default 8) 9565 -c, --checksum Skip based on checksum (if available) & size, not mod-time & size 9566 --client-cert string Client SSL certificate (PEM) for mutual TLS auth 9567 --client-key string Client SSL private key (PEM) for mutual TLS auth 9568 --compare-dest string Include additional server-side path during comparison. 9569 --config string Config file. (default "$HOME/.config/rclone/rclone.conf") 9570 --contimeout duration Connect timeout (default 1m0s) 9571 --copy-dest string Implies --compare-dest but also copies files from path into destination. 9572 --cpuprofile string Write cpu profile to file 9573 --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD") 9574 --delete-after When synchronizing, delete files on destination after transferring (default) 9575 --delete-before When synchronizing, delete files on destination before transferring 9576 --delete-during When synchronizing, delete files during transfer 9577 --delete-excluded Delete files on dest excluded from sync 9578 --disable string Disable a comma separated list of features. Use help to see a list. 9579 -n, --dry-run Do a trial run with no permanent changes 9580 --dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles 9581 --dump-bodies Dump HTTP headers and bodies - may contain sensitive info 9582 --dump-headers Dump HTTP headers - may contain sensitive info 9583 --error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts 9584 --exclude stringArray Exclude files matching pattern 9585 --exclude-from stringArray Read exclude patterns from file (use - to read from stdin) 9586 --exclude-if-present string Exclude directories if filename is present 9587 --expect-continue-timeout duration Timeout when using expect / 100-continue in HTTP (default 1s) 9588 --fast-list Use recursive list if available. Uses more memory but fewer transactions. 9589 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 9590 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 9591 -f, --filter stringArray Add a file-filtering rule 9592 --filter-from stringArray Read filtering patterns from a file (use - to read from stdin) 9593 --header stringArray Set HTTP header for all transactions 9594 --header-download stringArray Set HTTP header for download transactions 9595 --header-upload stringArray Set HTTP header for upload transactions 9596 --ignore-case Ignore case in filters (case insensitive) 9597 --ignore-case-sync Ignore case when synchronizing 9598 --ignore-checksum Skip post copy check of checksums. 9599 --ignore-errors delete even if there are I/O errors 9600 --ignore-existing Skip all files that exist on destination 9601 --ignore-size Ignore size when skipping use mod-time or checksum. 9602 -I, --ignore-times Don't skip files that match size and time - transfer all files 9603 --immutable Do not modify files. Fail if existing files have been modified. 9604 --include stringArray Include files matching pattern 9605 --include-from stringArray Read include patterns from file (use - to read from stdin) 9606 --log-file string Log everything to this file 9607 --log-format string Comma separated list of log format options (default "date,time") 9608 --log-level string Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE") 9609 --low-level-retries int Number of low level retries to do. (default 10) 9610 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9611 --max-backlog int Maximum number of objects in sync or check backlog. (default 10000) 9612 --max-delete int When synchronizing, limit the number of deletes (default -1) 9613 --max-depth int If set limits the recursion depth to this. (default -1) 9614 --max-duration duration Maximum duration rclone will transfer data for. 9615 --max-size SizeSuffix Only transfer files smaller than this in k or suffix b|k|M|G (default off) 9616 --max-stats-groups int Maximum number of stats groups to keep in memory. On max oldest is discarded. (default 1000) 9617 --max-transfer SizeSuffix Maximum size of data to transfer. (default off) 9618 --memprofile string Write memory profile to file 9619 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9620 --min-size SizeSuffix Only transfer files bigger than this in k or suffix b|k|M|G (default off) 9621 --modify-window duration Max time diff to be considered the same (default 1ns) 9622 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size. (default 250M) 9623 --multi-thread-streams int Max number of streams to use for multi-thread downloads. (default 4) 9624 --no-check-certificate Do not verify the server SSL certificate. Insecure. 9625 --no-check-dest Don't check the destination, copy regardless. 9626 --no-gzip-encoding Don't set Accept-Encoding: gzip. 9627 --no-traverse Don't traverse destination file system on copy. 9628 --no-unicode-normalization Don't normalize unicode characters in filenames. 9629 --no-update-modtime Don't update destination mod-time if files identical. 9630 --order-by string Instructions on how to order the transfers, eg 'size,descending' 9631 --password-command SpaceSepList Command for supplying password for encrypted configuration. 9632 -P, --progress Show progress during transfer. 9633 -q, --quiet Print as little stuff as possible 9634 --rc Enable the remote control server. 9635 --rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572") 9636 --rc-allow-origin string Set the allowed origin for CORS. 9637 --rc-baseurl string Prefix for URLs - leave blank for root. 9638 --rc-cert string SSL PEM key (concatenation of certificate and CA certificate) 9639 --rc-client-ca string Client certificate authority to verify clients with 9640 --rc-enable-metrics Enable prometheus metrics on /metrics 9641 --rc-files string Path to local files to serve on the HTTP server. 9642 --rc-htpasswd string htpasswd file - if not provided no authentication is done 9643 --rc-job-expire-duration duration expire finished async jobs older than this value (default 1m0s) 9644 --rc-job-expire-interval duration interval to check for expired async jobs (default 10s) 9645 --rc-key string SSL PEM Private key 9646 --rc-max-header-bytes int Maximum size of request header (default 4096) 9647 --rc-no-auth Don't require auth for certain methods. 9648 --rc-pass string Password for authentication. 9649 --rc-realm string realm for authentication (default "rclone") 9650 --rc-serve Enable the serving of remote objects. 9651 --rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s) 9652 --rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s) 9653 --rc-template string User Specified Template. 9654 --rc-user string User name for authentication. 9655 --rc-web-fetch-url string URL to fetch the releases for webgui. (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest") 9656 --rc-web-gui Launch WebGUI on localhost 9657 --rc-web-gui-force-update Force update to latest version of web gui 9658 --rc-web-gui-no-open-browser Don't open the browser automatically 9659 --rc-web-gui-update Check and update to latest version of web gui 9660 --retries int Retry operations this many times if they fail (default 3) 9661 --retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable) 9662 --size-only Skip based on size only, not mod-time or checksum 9663 --stats duration Interval between printing stats, e.g 500ms, 60s, 5m. (0 to disable) (default 1m0s) 9664 --stats-file-name-length int Max file name length in stats. 0 for no limit (default 45) 9665 --stats-log-level string Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO") 9666 --stats-one-line Make the stats fit on one line. 9667 --stats-one-line-date Enables --stats-one-line and add current date/time prefix. 9668 --stats-one-line-date-format string Enables --stats-one-line-date and uses custom formatted date. Enclose date string in double quotes ("). See https://golang.org/pkg/time/#Time.Format 9669 --stats-unit string Show data rate in stats as either 'bits' or 'bytes'/s (default "bytes") 9670 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown. Upload starts after reaching cutoff or when file ends. (default 100k) 9671 --suffix string Suffix to add to changed files. 9672 --suffix-keep-extension Preserve the extension when using --suffix. 9673 --syslog Use Syslog for logging 9674 --syslog-facility string Facility for syslog, eg KERN,USER,... (default "DAEMON") 9675 --timeout duration IO idle timeout (default 5m0s) 9676 --tpslimit float Limit HTTP transactions per second to this. 9677 --tpslimit-burst int Max burst of transactions for --tpslimit. (default 1) 9678 --track-renames When synchronizing, track file renames and do a server side move if possible 9679 --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime (default "hash") 9680 --transfers int Number of file transfers to run in parallel. (default 4) 9681 -u, --update Skip files that are newer on the destination. 9682 --use-cookies Enable session cookiejar. 9683 --use-json-log Use json log format. 9684 --use-mmap Use mmap allocator (see docs). 9685 --use-server-modtime Use server modified time instead of object metadata 9686 --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.52.2") 9687 -v, --verbose count Print lots more stuff (repeat for more) 9688 ``` 9689 9690 ## Backend Flags 9691 9692 These flags are available for every command. They control the backends 9693 and may be set in the config file. 9694 9695 ``` 9696 --acd-auth-url string Auth server URL. 9697 --acd-client-id string Amazon Application Client ID. 9698 --acd-client-secret string Amazon Application Client Secret. 9699 --acd-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 9700 --acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink. (default 9G) 9701 --acd-token-url string Token server url. 9702 --acd-upload-wait-per-gb Duration Additional time per GB to wait after a failed complete upload to see if it appears. (default 3m0s) 9703 --alias-remote string Remote or path to alias. 9704 --azureblob-access-tier string Access tier of blob: hot, cool or archive. 9705 --azureblob-account string Storage Account Name (leave blank to use SAS URL or Emulator) 9706 --azureblob-chunk-size SizeSuffix Upload chunk size (<= 100MB). (default 4M) 9707 --azureblob-disable-checksum Don't store MD5 checksum with object metadata. 9708 --azureblob-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8) 9709 --azureblob-endpoint string Endpoint for the service 9710 --azureblob-key string Storage Account Key (leave blank to use SAS URL or Emulator) 9711 --azureblob-list-chunk int Size of blob list. (default 5000) 9712 --azureblob-memory-pool-flush-time Duration How often internal memory buffer pools will be flushed. (default 1m0s) 9713 --azureblob-memory-pool-use-mmap Whether to use mmap buffers in internal memory pool. 9714 --azureblob-sas-url string SAS URL for container level access only 9715 --azureblob-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (<= 256MB). (default 256M) 9716 --azureblob-use-emulator Uses local storage emulator if provided as 'true' (leave blank if using real azure storage endpoint) 9717 --b2-account string Account ID or Application Key ID 9718 --b2-chunk-size SizeSuffix Upload chunk size. Must fit in memory. (default 96M) 9719 --b2-disable-checksum Disable checksums for large (> upload cutoff) files 9720 --b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d. (default 1w) 9721 --b2-download-url string Custom endpoint for downloads. 9722 --b2-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9723 --b2-endpoint string Endpoint for the service. 9724 --b2-hard-delete Permanently delete files on remote removal, otherwise hide files. 9725 --b2-key string Application Key 9726 --b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging. 9727 --b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload. (default 200M) 9728 --b2-versions Include old versions in directory listings. 9729 --box-box-config-file string Box App config.json location 9730 --box-box-sub-type string (default "user") 9731 --box-client-id string Box App Client Id. 9732 --box-client-secret string Box App Client Secret 9733 --box-commit-retries int Max number of times to try committing a multipart file. (default 100) 9734 --box-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot) 9735 --box-root-folder-id string Fill in for rclone to use a non root folder as its starting point. 9736 --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50MB). (default 50M) 9737 --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s) 9738 --cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming. 9739 --cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend") 9740 --cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M) 9741 --cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G) 9742 --cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend") 9743 --cache-db-purge Clear all the cached data for this remote on start. 9744 --cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s) 9745 --cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s) 9746 --cache-plex-insecure string Skip all certificate verification when connecting to the Plex server 9747 --cache-plex-password string The password of the Plex user (obscured) 9748 --cache-plex-url string The URL of the Plex server 9749 --cache-plex-username string The username of the Plex user 9750 --cache-read-retries int How many times to retry a read from a cache storage. (default 10) 9751 --cache-remote string Remote to cache. 9752 --cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1) 9753 --cache-tmp-upload-path string Directory to keep temporary files until they are uploaded. 9754 --cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s) 9755 --cache-workers int How many workers should run in parallel to download chunks. (default 4) 9756 --cache-writes Cache file data on writes through the FS 9757 --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks. (default 2G) 9758 --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks. 9759 --chunker-hash-type string Choose how chunker handles hash sums. All modes but "none" require metadata. (default "md5") 9760 --chunker-meta-format string Format of the metadata object or "none". By default "simplejson". (default "simplejson") 9761 --chunker-name-format string String format of chunk file names. (default "*.rclone_chunk.###") 9762 --chunker-remote string Remote to chunk/unchunk. 9763 --chunker-start-from int Minimum valid chunk number. Usually 0 or 1. (default 1) 9764 -L, --copy-links Follow symlinks and copy the pointed to item. 9765 --crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true) 9766 --crypt-filename-encryption string How to encrypt the filenames. (default "standard") 9767 --crypt-password string Password or pass phrase for encryption. (obscured) 9768 --crypt-password2 string Password or pass phrase for salt. Optional but recommended. (obscured) 9769 --crypt-remote string Remote to encrypt/decrypt. 9770 --crypt-show-mapping For all files listed show how the names encrypt. 9771 --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded. 9772 --drive-allow-import-name-change Allow the filetype to change when uploading Google docs (e.g. file.doc to file.docx). This will confuse sync and reupload every time. 9773 --drive-alternate-export Use alternate export URLs for google documents export., 9774 --drive-auth-owner-only Only consider files owned by the authenticated user. 9775 --drive-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 8M) 9776 --drive-client-id string Google Application Client Id 9777 --drive-client-secret string Google Application Client Secret 9778 --drive-disable-http2 Disable drive using http2 (default true) 9779 --drive-encoding MultiEncoder This sets the encoding for the backend. (default InvalidUtf8) 9780 --drive-export-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg") 9781 --drive-formats string Deprecated: see export_formats 9782 --drive-impersonate string Impersonate this user when using a service account. 9783 --drive-import-formats string Comma separated list of preferred formats for uploading Google docs. 9784 --drive-keep-revision-forever Keep new head revision of each file forever. 9785 --drive-list-chunk int Size of listing chunk 100-1000. 0 to disable. (default 1000) 9786 --drive-pacer-burst int Number of API calls to allow without sleeping. (default 100) 9787 --drive-pacer-min-sleep Duration Minimum time to sleep between API calls. (default 100ms) 9788 --drive-root-folder-id string ID of the root folder 9789 --drive-scope string Scope that rclone should use when requesting access from drive. 9790 --drive-server-side-across-configs Allow server side operations (eg copy) to work across different drive configs. 9791 --drive-service-account-credentials string Service Account Credentials JSON blob 9792 --drive-service-account-file string Service Account Credentials JSON file path 9793 --drive-shared-with-me Only show files that are shared with me. 9794 --drive-size-as-quota Show sizes as storage quota usage, not actual size. 9795 --drive-skip-checksum-gphotos Skip MD5 checksum on Google photos and videos only. 9796 --drive-skip-gdocs Skip google documents in all listings. 9797 --drive-skip-shortcuts If set skip shortcut files 9798 --drive-stop-on-upload-limit Make upload limit errors be fatal 9799 --drive-team-drive string ID of the Team Drive 9800 --drive-trashed-only Only show files that are in the trash. 9801 --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8M) 9802 --drive-use-created-date Use file created date instead of modified date., 9803 --drive-use-shared-date Use date file was shared instead of modified date. 9804 --drive-use-trash Send files to the trash instead of deleting permanently. (default true) 9805 --drive-v2-download-min-size SizeSuffix If Object's are greater, use drive v2 API to download. (default off) 9806 --dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M) 9807 --dropbox-client-id string Dropbox App Client Id 9808 --dropbox-client-secret string Dropbox App Client Secret 9809 --dropbox-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot) 9810 --dropbox-impersonate string Impersonate this user when using a business account. 9811 --fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl 9812 --fichier-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot) 9813 --fichier-shared-folder string If you want to download a shared folder, add this parameter 9814 --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited 9815 --ftp-disable-epsv Disable using EPSV even if server advertises support 9816 --ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot) 9817 --ftp-host string FTP host to connect to 9818 --ftp-no-check-certificate Do not verify the TLS certificate of the server 9819 --ftp-pass string FTP password (obscured) 9820 --ftp-port string FTP port, leave blank to use default (21) 9821 --ftp-tls Use FTP over TLS (Implicit) 9822 --ftp-user string FTP username, leave blank for current username, $USER 9823 --gcs-bucket-acl string Access Control List for new buckets. 9824 --gcs-bucket-policy-only Access checks should use bucket-level IAM policies. 9825 --gcs-client-id string Google Application Client Id 9826 --gcs-client-secret string Google Application Client Secret 9827 --gcs-encoding MultiEncoder This sets the encoding for the backend. (default Slash,CrLf,InvalidUtf8,Dot) 9828 --gcs-location string Location for the newly created buckets. 9829 --gcs-object-acl string Access Control List for new objects. 9830 --gcs-project-number string Project number. 9831 --gcs-service-account-file string Service Account Credentials JSON file path 9832 --gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage. 9833 --gphotos-client-id string Google Application Client Id 9834 --gphotos-client-secret string Google Application Client Secret 9835 --gphotos-read-only Set to make the Google Photos backend read only. 9836 --gphotos-read-size Set to read the size of media items. 9837 --gphotos-start-year int Year limits the photos to be downloaded to those which are uploaded after the given year (default 2000) 9838 --http-headers CommaSepList Set HTTP headers for all transactions 9839 --http-no-head Don't use HEAD requests to find file sizes in dir listing 9840 --http-no-slash Set this if the site doesn't end directories with / 9841 --http-url string URL of http host to connect to 9842 --hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G) 9843 --hubic-client-id string Hubic Client Id 9844 --hubic-client-secret string Hubic Client Secret 9845 --hubic-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8) 9846 --hubic-no-chunk Don't chunk files during streaming upload. 9847 --jottacloud-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot) 9848 --jottacloud-hard-delete Delete files permanently rather than putting them into the trash. 9849 --jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required. (default 10M) 9850 --jottacloud-trashed-only Only show files that are in the trash. 9851 --jottacloud-unlink Remove existing public link to file/folder with link command rather than creating. 9852 --jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's. (default 10M) 9853 --koofr-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9854 --koofr-endpoint string The Koofr API endpoint to use (default "https://app.koofr.net") 9855 --koofr-mountid string Mount ID of the mount to use. If omitted, the primary mount is used. 9856 --koofr-password string Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) (obscured) 9857 --koofr-setmtime Does the backend support setting modification time. Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend. (default true) 9858 --koofr-user string Your Koofr user name 9859 -l, --links Translate symlinks to/from regular files with a '.rclonelink' extension 9860 --local-case-insensitive Force the filesystem to report itself as case insensitive 9861 --local-case-sensitive Force the filesystem to report itself as case sensitive. 9862 --local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot) 9863 --local-no-check-updated Don't check to see if the files change during upload 9864 --local-no-sparse Disable sparse files for multi-thread downloads 9865 --local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated) 9866 --local-nounc string Disable UNC (long path names) conversion on Windows 9867 --mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true) 9868 --mailru-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9869 --mailru-pass string Password (obscured) 9870 --mailru-speedup-enable Skip full upload if there is another file with same data hash. (default true) 9871 --mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash). (default "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf") 9872 --mailru-speedup-max-disk SizeSuffix This option allows you to disable speedup (put by hash) for large files (default 3G) 9873 --mailru-speedup-max-memory SizeSuffix Files larger than the size given below will always be hashed on disk. (default 32M) 9874 --mailru-user string User name (usually email) 9875 --mega-debug Output more debug from Mega. 9876 --mega-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 9877 --mega-hard-delete Delete files permanently rather than putting them into the trash. 9878 --mega-pass string Password. (obscured) 9879 --mega-user string User name 9880 -x, --one-file-system Don't cross filesystem boundaries (unix/macOS only). 9881 --onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes). (default 10M) 9882 --onedrive-client-id string Microsoft App Client Id 9883 --onedrive-client-secret string Microsoft App Client Secret 9884 --onedrive-drive-id string The ID of the drive to use 9885 --onedrive-drive-type string The type of the drive ( personal | business | documentLibrary ) 9886 --onedrive-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot) 9887 --onedrive-expose-onenote-files Set to make OneNote files show up in directory listings. 9888 --onedrive-server-side-across-configs Allow server side operations (eg copy) to work across different onedrive configs. 9889 --opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size. (default 10M) 9890 --opendrive-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) 9891 --opendrive-password string Password. (obscured) 9892 --opendrive-username string Username 9893 --pcloud-client-id string Pcloud App Client Id 9894 --pcloud-client-secret string Pcloud App Client Secret 9895 --pcloud-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9896 --pcloud-root-folder-id string Fill in for rclone to use a non root folder as its starting point. (default "d0") 9897 --premiumizeme-encoding MultiEncoder This sets the encoding for the backend. (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9898 --putio-encoding MultiEncoder This sets the encoding for the backend. (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 9899 --qingstor-access-key-id string QingStor Access Key ID 9900 --qingstor-chunk-size SizeSuffix Chunk size to use for uploading. (default 4M) 9901 --qingstor-connection-retries int Number of connection retries. (default 3) 9902 --qingstor-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Ctl,InvalidUtf8) 9903 --qingstor-endpoint string Enter an endpoint URL to connection QingStor API. 9904 --qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 9905 --qingstor-secret-access-key string QingStor Secret Access Key (password) 9906 --qingstor-upload-concurrency int Concurrency for multipart uploads. (default 1) 9907 --qingstor-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200M) 9908 --qingstor-zone string Zone to connect to. 9909 --s3-access-key-id string AWS Access Key ID. 9910 --s3-acl string Canned ACL used when creating buckets and storing or copying objects. 9911 --s3-bucket-acl string Canned ACL used when creating buckets. 9912 --s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M) 9913 --s3-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 5G) 9914 --s3-disable-checksum Don't store MD5 checksum with object metadata 9915 --s3-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8,Dot) 9916 --s3-endpoint string Endpoint for S3 API. 9917 --s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 9918 --s3-force-path-style If true use path style access if false use virtual hosted style. (default true) 9919 --s3-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. 9920 --s3-list-chunk int Size of listing chunk (response list for each ListObject S3 request). (default 1000) 9921 --s3-location-constraint string Location constraint - must be set to match the Region. 9922 --s3-memory-pool-flush-time Duration How often internal memory buffer pools will be flushed. (default 1m0s) 9923 --s3-memory-pool-use-mmap Whether to use mmap buffers in internal memory pool. 9924 --s3-provider string Choose your S3 provider. 9925 --s3-region string Region to connect to. 9926 --s3-secret-access-key string AWS Secret Access Key (password) 9927 --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3. 9928 --s3-session-token string An AWS session token 9929 --s3-sse-customer-algorithm string If using SSE-C, the server-side encryption algorithm used when storing this object in S3. 9930 --s3-sse-customer-key string If using SSE-C you must provide the secret encryption key used to encrypt/decrypt your data. 9931 --s3-sse-customer-key-md5 string If using SSE-C you must provide the secret encryption key MD5 checksum. 9932 --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key. 9933 --s3-storage-class string The storage class to use when storing new objects in S3. 9934 --s3-upload-concurrency int Concurrency for multipart uploads. (default 4) 9935 --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200M) 9936 --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint. 9937 --s3-v2-auth If true use v2 authentication. 9938 --seafile-2fa Two-factor authentication ('true' if the account has 2FA enabled) 9939 --seafile-create-library Should rclone create a library if it doesn't exist 9940 --seafile-encoding MultiEncoder This sets the encoding for the backend. (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8) 9941 --seafile-library string Name of the library. Leave blank to access all non-encrypted libraries. 9942 --seafile-library-key string Library password (for encrypted libraries only). Leave blank if you pass it through the command line. (obscured) 9943 --seafile-pass string Password (obscured) 9944 --seafile-url string URL of seafile host to connect to 9945 --seafile-user string User name (usually email address) 9946 --sftp-ask-password Allow asking for SFTP password when needed. 9947 --sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available. 9948 --sftp-host string SSH host to connect to 9949 --sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent. 9950 --sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. (obscured) 9951 --sftp-key-pem string Raw PEM-encoded private key, If specified, will override key_file parameter. 9952 --sftp-key-use-agent When set forces the usage of the ssh-agent. 9953 --sftp-md5sum-command string The command used to read md5 hashes. Leave blank for autodetect. 9954 --sftp-pass string SSH password, leave blank to use ssh-agent. (obscured) 9955 --sftp-path-override string Override path used by SSH connection. 9956 --sftp-port string SSH port, leave blank to use default (22) 9957 --sftp-set-modtime Set the modified time on the remote if set. (default true) 9958 --sftp-sha1sum-command string The command used to read sha1 hashes. Leave blank for autodetect. 9959 --sftp-skip-links Set to skip any symlinks and any other non regular files. 9960 --sftp-use-insecure-cipher Enable the use of insecure ciphers and key exchange methods. 9961 --sftp-user string SSH username, leave blank for current username, ncw 9962 --sharefile-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 64M) 9963 --sharefile-encoding MultiEncoder This sets the encoding for the backend. (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot) 9964 --sharefile-endpoint string Endpoint for API calls. 9965 --sharefile-root-folder-id string ID of the root folder 9966 --sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload. (default 128M) 9967 --skip-links Don't warn about skipped symlinks. 9968 --sugarsync-access-key-id string Sugarsync Access Key ID. 9969 --sugarsync-app-id string Sugarsync App ID. 9970 --sugarsync-authorization string Sugarsync authorization 9971 --sugarsync-authorization-expiry string Sugarsync authorization expiry 9972 --sugarsync-deleted-id string Sugarsync deleted folder id 9973 --sugarsync-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Ctl,InvalidUtf8,Dot) 9974 --sugarsync-hard-delete Permanently delete files if true 9975 --sugarsync-private-access-key string Sugarsync Private Access Key 9976 --sugarsync-refresh-token string Sugarsync refresh token 9977 --sugarsync-root-id string Sugarsync root id 9978 --sugarsync-user string Sugarsync user 9979 --swift-application-credential-id string Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 9980 --swift-application-credential-name string Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 9981 --swift-application-credential-secret string Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 9982 --swift-auth string Authentication URL for server (OS_AUTH_URL). 9983 --swift-auth-token string Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 9984 --swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 9985 --swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G) 9986 --swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 9987 --swift-encoding MultiEncoder This sets the encoding for the backend. (default Slash,InvalidUtf8) 9988 --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") 9989 --swift-env-auth Get swift credentials from environment variables in standard OpenStack form. 9990 --swift-key string API key or password (OS_PASSWORD). 9991 --swift-no-chunk Don't chunk files during streaming upload. 9992 --swift-region string Region name - optional (OS_REGION_NAME) 9993 --swift-storage-policy string The storage policy to use when creating a new container 9994 --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) 9995 --swift-tenant string Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 9996 --swift-tenant-domain string Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 9997 --swift-tenant-id string Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 9998 --swift-user string User name to log in (OS_USERNAME). 9999 --swift-user-id string User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 10000 --tardigrade-access-grant string Access Grant. 10001 --tardigrade-api-key string API Key. 10002 --tardigrade-passphrase string Encryption Passphrase. To access existing objects enter passphrase used for uploading. 10003 --tardigrade-provider string Choose an authentication method. (default "existing") 10004 --tardigrade-satellite-address <nodeid>@<address>:<port> Satellite Address. Custom satellite address should match the format: <nodeid>@<address>:<port>. (default "us-central-1.tardigrade.io") 10005 --union-action-policy string Policy to choose upstream on ACTION category. (default "epall") 10006 --union-cache-time int Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used. (default 120) 10007 --union-create-policy string Policy to choose upstream on CREATE category. (default "epmfs") 10008 --union-search-policy string Policy to choose upstream on SEARCH category. (default "ff") 10009 --union-upstreams string List of space separated upstreams. 10010 --webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon) 10011 --webdav-bearer-token-command string Command to run to get a bearer token 10012 --webdav-pass string Password. (obscured) 10013 --webdav-url string URL of http host to connect to 10014 --webdav-user string User name 10015 --webdav-vendor string Name of the Webdav site/service/software you are using 10016 --yandex-client-id string Yandex Client Id 10017 --yandex-client-secret string Yandex Client Secret 10018 --yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot) 10019 --yandex-unlink Remove existing public link to file/folder with link command rather than creating. 10020 ``` 10021 10022 1Fichier 10023 ----------------------------------------- 10024 10025 This is a backend for the [1fichier](https://1fichier.com) cloud 10026 storage service. Note that a Premium subscription is required to use 10027 the API. 10028 10029 Paths are specified as `remote:path` 10030 10031 Paths may be as deep as required, eg `remote:directory/subdirectory`. 10032 10033 The initial setup for 1Fichier involves getting the API key from the website which you 10034 need to do in your browser. 10035 10036 Here is an example of how to make a remote called `remote`. First run: 10037 10038 rclone config 10039 10040 This will guide you through an interactive setup process: 10041 10042 ``` 10043 No remotes found - make a new one 10044 n) New remote 10045 s) Set configuration password 10046 q) Quit config 10047 n/s/q> n 10048 name> remote 10049 Type of storage to configure. 10050 Enter a string value. Press Enter for the default (""). 10051 Choose a number from below, or type in your own value 10052 [snip] 10053 XX / 1Fichier 10054 \ "fichier" 10055 [snip] 10056 Storage> fichier 10057 ** See help for fichier backend at: https://rclone.org/fichier/ ** 10058 10059 Your API Key, get it from https://1fichier.com/console/params.pl 10060 Enter a string value. Press Enter for the default (""). 10061 api_key> example_key 10062 10063 Edit advanced config? (y/n) 10064 y) Yes 10065 n) No 10066 y/n> 10067 Remote config 10068 -------------------- 10069 [remote] 10070 type = fichier 10071 api_key = example_key 10072 -------------------- 10073 y) Yes this is OK 10074 e) Edit this remote 10075 d) Delete this remote 10076 y/e/d> y 10077 ``` 10078 10079 Once configured you can then use `rclone` like this, 10080 10081 List directories in top level of your 1Fichier account 10082 10083 rclone lsd remote: 10084 10085 List all the files in your 1Fichier account 10086 10087 rclone ls remote: 10088 10089 To copy a local directory to a 1Fichier directory called backup 10090 10091 rclone copy /home/source remote:backup 10092 10093 ### Modified time and hashes ### 10094 10095 1Fichier does not support modification times. It supports the Whirlpool hash algorithm. 10096 10097 ### Duplicated files ### 10098 10099 1Fichier can have two files with exactly the same name and path (unlike a 10100 normal file system). 10101 10102 Duplicated files cause problems with the syncing and you will see 10103 messages in the log about duplicates. 10104 10105 #### Restricted filename characters 10106 10107 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 10108 the following characters are also replaced: 10109 10110 | Character | Value | Replacement | 10111 | --------- |:-----:|:-----------:| 10112 | \ | 0x5C | \ | 10113 | < | 0x3C | < | 10114 | > | 0x3E | > | 10115 | " | 0x22 | " | 10116 | $ | 0x24 | $ | 10117 | ` | 0x60 | ` | 10118 | ' | 0x27 | ' | 10119 10120 File names can also not start or end with the following characters. 10121 These only get replaced if they are the first or last character in the 10122 name: 10123 10124 | Character | Value | Replacement | 10125 | --------- |:-----:|:-----------:| 10126 | SP | 0x20 | ␠ | 10127 10128 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 10129 as they can't be used in JSON strings. 10130 10131 10132 ### Standard Options 10133 10134 Here are the standard options specific to fichier (1Fichier). 10135 10136 #### --fichier-api-key 10137 10138 Your API Key, get it from https://1fichier.com/console/params.pl 10139 10140 - Config: api_key 10141 - Env Var: RCLONE_FICHIER_API_KEY 10142 - Type: string 10143 - Default: "" 10144 10145 ### Advanced Options 10146 10147 Here are the advanced options specific to fichier (1Fichier). 10148 10149 #### --fichier-shared-folder 10150 10151 If you want to download a shared folder, add this parameter 10152 10153 - Config: shared_folder 10154 - Env Var: RCLONE_FICHIER_SHARED_FOLDER 10155 - Type: string 10156 - Default: "" 10157 10158 #### --fichier-encoding 10159 10160 This sets the encoding for the backend. 10161 10162 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 10163 10164 - Config: encoding 10165 - Env Var: RCLONE_FICHIER_ENCODING 10166 - Type: MultiEncoder 10167 - Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot 10168 10169 10170 10171 Alias 10172 ----------------------------------------- 10173 10174 The `alias` remote provides a new name for another remote. 10175 10176 Paths may be as deep as required or a local path, 10177 eg `remote:directory/subdirectory` or `/directory/subdirectory`. 10178 10179 During the initial setup with `rclone config` you will specify the target 10180 remote. The target remote can either be a local path or another remote. 10181 10182 Subfolders can be used in target remote. Assume an alias remote named `backup` 10183 with the target `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop` 10184 is exactly the same as invoking `rclone mkdir mydrive:private/backup/desktop`. 10185 10186 There will be no special handling of paths containing `..` segments. 10187 Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking 10188 `rclone mkdir mydrive:private/backup/../desktop`. 10189 The empty path is not allowed as a remote. To alias the current directory 10190 use `.` instead. 10191 10192 Here is an example of how to make an alias called `remote` for local folder. 10193 First run: 10194 10195 rclone config 10196 10197 This will guide you through an interactive setup process: 10198 10199 ``` 10200 No remotes found - make a new one 10201 n) New remote 10202 s) Set configuration password 10203 q) Quit config 10204 n/s/q> n 10205 name> remote 10206 Type of storage to configure. 10207 Choose a number from below, or type in your own value 10208 [snip] 10209 XX / Alias for an existing remote 10210 \ "alias" 10211 [snip] 10212 Storage> alias 10213 Remote or path to alias. 10214 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". 10215 remote> /mnt/storage/backup 10216 Remote config 10217 -------------------- 10218 [remote] 10219 remote = /mnt/storage/backup 10220 -------------------- 10221 y) Yes this is OK 10222 e) Edit this remote 10223 d) Delete this remote 10224 y/e/d> y 10225 Current remotes: 10226 10227 Name Type 10228 ==== ==== 10229 remote alias 10230 10231 e) Edit existing remote 10232 n) New remote 10233 d) Delete remote 10234 r) Rename remote 10235 c) Copy remote 10236 s) Set configuration password 10237 q) Quit config 10238 e/n/d/r/c/s/q> q 10239 ``` 10240 10241 Once configured you can then use `rclone` like this, 10242 10243 List directories in top level in `/mnt/storage/backup` 10244 10245 rclone lsd remote: 10246 10247 List all the files in `/mnt/storage/backup` 10248 10249 rclone ls remote: 10250 10251 Copy another local directory to the alias directory called source 10252 10253 rclone copy /home/source remote:source 10254 10255 10256 ### Standard Options 10257 10258 Here are the standard options specific to alias (Alias for an existing remote). 10259 10260 #### --alias-remote 10261 10262 Remote or path to alias. 10263 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". 10264 10265 - Config: remote 10266 - Env Var: RCLONE_ALIAS_REMOTE 10267 - Type: string 10268 - Default: "" 10269 10270 10271 10272 Amazon Drive 10273 ----------------------------------------- 10274 10275 Amazon Drive, formerly known as Amazon Cloud Drive, is a cloud storage 10276 service run by Amazon for consumers. 10277 10278 ## Status 10279 10280 **Important:** rclone supports Amazon Drive only if you have your own 10281 set of API keys. Unfortunately the [Amazon Drive developer 10282 program](https://developer.amazon.com/amazon-drive) is now closed to 10283 new entries so if you don't already have your own set of keys you will 10284 not be able to use rclone with Amazon Drive. 10285 10286 For the history on why rclone no longer has a set of Amazon Drive API 10287 keys see [the forum](https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314). 10288 10289 If you happen to know anyone who works at Amazon then please ask them 10290 to re-instate rclone into the Amazon Drive developer program - thanks! 10291 10292 ## Setup 10293 10294 The initial setup for Amazon Drive involves getting a token from 10295 Amazon which you need to do in your browser. `rclone config` walks 10296 you through it. 10297 10298 The configuration process for Amazon Drive may involve using an [oauth 10299 proxy](https://github.com/ncw/oauthproxy). This is used to keep the 10300 Amazon credentials out of the source code. The proxy runs in Google's 10301 very secure App Engine environment and doesn't store any credentials 10302 which pass through it. 10303 10304 Since rclone doesn't currently have its own Amazon Drive credentials 10305 so you will either need to have your own `client_id` and 10306 `client_secret` with Amazon Drive, or use a third party oauth proxy 10307 in which case you will need to enter `client_id`, `client_secret`, 10308 `auth_url` and `token_url`. 10309 10310 Note also if you are not using Amazon's `auth_url` and `token_url`, 10311 (ie you filled in something for those) then if setting up on a remote 10312 machine you can only use the [copying the config method of 10313 configuration](https://rclone.org/remote_setup/#configuring-by-copying-the-config-file) 10314 - `rclone authorize` will not work. 10315 10316 Here is an example of how to make a remote called `remote`. First run: 10317 10318 rclone config 10319 10320 This will guide you through an interactive setup process: 10321 10322 ``` 10323 No remotes found - make a new one 10324 n) New remote 10325 r) Rename remote 10326 c) Copy remote 10327 s) Set configuration password 10328 q) Quit config 10329 n/r/c/s/q> n 10330 name> remote 10331 Type of storage to configure. 10332 Choose a number from below, or type in your own value 10333 [snip] 10334 XX / Amazon Drive 10335 \ "amazon cloud drive" 10336 [snip] 10337 Storage> amazon cloud drive 10338 Amazon Application Client Id - required. 10339 client_id> your client ID goes here 10340 Amazon Application Client Secret - required. 10341 client_secret> your client secret goes here 10342 Auth server URL - leave blank to use Amazon's. 10343 auth_url> Optional auth URL 10344 Token server url - leave blank to use Amazon's. 10345 token_url> Optional token URL 10346 Remote config 10347 Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config. 10348 Use auto config? 10349 * Say Y if not sure 10350 * Say N if you are working on a remote or headless machine 10351 y) Yes 10352 n) No 10353 y/n> y 10354 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 10355 Log in and authorize rclone for access 10356 Waiting for code... 10357 Got code 10358 -------------------- 10359 [remote] 10360 client_id = your client ID goes here 10361 client_secret = your client secret goes here 10362 auth_url = Optional auth URL 10363 token_url = Optional token URL 10364 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"} 10365 -------------------- 10366 y) Yes this is OK 10367 e) Edit this remote 10368 d) Delete this remote 10369 y/e/d> y 10370 ``` 10371 10372 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 10373 machine with no Internet browser available. 10374 10375 Note that rclone runs a webserver on your local machine to collect the 10376 token as returned from Amazon. This only runs from the moment it 10377 opens your browser to the moment you get back the verification 10378 code. This is on `http://127.0.0.1:53682/` and this it may require 10379 you to unblock it temporarily if you are running a host firewall. 10380 10381 Once configured you can then use `rclone` like this, 10382 10383 List directories in top level of your Amazon Drive 10384 10385 rclone lsd remote: 10386 10387 List all the files in your Amazon Drive 10388 10389 rclone ls remote: 10390 10391 To copy a local directory to an Amazon Drive directory called backup 10392 10393 rclone copy /home/source remote:backup 10394 10395 ### Modified time and MD5SUMs ### 10396 10397 Amazon Drive doesn't allow modification times to be changed via 10398 the API so these won't be accurate or used for syncing. 10399 10400 It does store MD5SUMs so for a more accurate sync, you can use the 10401 `--checksum` flag. 10402 10403 #### Restricted filename characters 10404 10405 | Character | Value | Replacement | 10406 | --------- |:-----:|:-----------:| 10407 | NUL | 0x00 | ␀ | 10408 | / | 0x2F | / | 10409 10410 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 10411 as they can't be used in JSON strings. 10412 10413 ### Deleting files ### 10414 10415 Any files you delete with rclone will end up in the trash. Amazon 10416 don't provide an API to permanently delete files, nor to empty the 10417 trash, so you will have to do that with one of Amazon's apps or via 10418 the Amazon Drive website. As of November 17, 2016, files are 10419 automatically deleted by Amazon from the trash after 30 days. 10420 10421 ### Using with non `.com` Amazon accounts ### 10422 10423 Let's say you usually use `amazon.co.uk`. When you authenticate with 10424 rclone it will take you to an `amazon.com` page to log in. Your 10425 `amazon.co.uk` email and password should work here just fine. 10426 10427 10428 ### Standard Options 10429 10430 Here are the standard options specific to amazon cloud drive (Amazon Drive). 10431 10432 #### --acd-client-id 10433 10434 Amazon Application Client ID. 10435 10436 - Config: client_id 10437 - Env Var: RCLONE_ACD_CLIENT_ID 10438 - Type: string 10439 - Default: "" 10440 10441 #### --acd-client-secret 10442 10443 Amazon Application Client Secret. 10444 10445 - Config: client_secret 10446 - Env Var: RCLONE_ACD_CLIENT_SECRET 10447 - Type: string 10448 - Default: "" 10449 10450 ### Advanced Options 10451 10452 Here are the advanced options specific to amazon cloud drive (Amazon Drive). 10453 10454 #### --acd-auth-url 10455 10456 Auth server URL. 10457 Leave blank to use Amazon's. 10458 10459 - Config: auth_url 10460 - Env Var: RCLONE_ACD_AUTH_URL 10461 - Type: string 10462 - Default: "" 10463 10464 #### --acd-token-url 10465 10466 Token server url. 10467 leave blank to use Amazon's. 10468 10469 - Config: token_url 10470 - Env Var: RCLONE_ACD_TOKEN_URL 10471 - Type: string 10472 - Default: "" 10473 10474 #### --acd-checkpoint 10475 10476 Checkpoint for internal polling (debug). 10477 10478 - Config: checkpoint 10479 - Env Var: RCLONE_ACD_CHECKPOINT 10480 - Type: string 10481 - Default: "" 10482 10483 #### --acd-upload-wait-per-gb 10484 10485 Additional time per GB to wait after a failed complete upload to see if it appears. 10486 10487 Sometimes Amazon Drive gives an error when a file has been fully 10488 uploaded but the file appears anyway after a little while. This 10489 happens sometimes for files over 1GB in size and nearly every time for 10490 files bigger than 10GB. This parameter controls the time rclone waits 10491 for the file to appear. 10492 10493 The default value for this parameter is 3 minutes per GB, so by 10494 default it will wait 3 minutes for every GB uploaded to see if the 10495 file appears. 10496 10497 You can disable this feature by setting it to 0. This may cause 10498 conflict errors as rclone retries the failed upload but the file will 10499 most likely appear correctly eventually. 10500 10501 These values were determined empirically by observing lots of uploads 10502 of big files for a range of file sizes. 10503 10504 Upload with the "-v" flag to see more info about what rclone is doing 10505 in this situation. 10506 10507 - Config: upload_wait_per_gb 10508 - Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB 10509 - Type: Duration 10510 - Default: 3m0s 10511 10512 #### --acd-templink-threshold 10513 10514 Files >= this size will be downloaded via their tempLink. 10515 10516 Files this size or more will be downloaded via their "tempLink". This 10517 is to work around a problem with Amazon Drive which blocks downloads 10518 of files bigger than about 10GB. The default for this is 9GB which 10519 shouldn't need to be changed. 10520 10521 To download files above this threshold, rclone requests a "tempLink" 10522 which downloads the file through a temporary URL directly from the 10523 underlying S3 storage. 10524 10525 - Config: templink_threshold 10526 - Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD 10527 - Type: SizeSuffix 10528 - Default: 9G 10529 10530 #### --acd-encoding 10531 10532 This sets the encoding for the backend. 10533 10534 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 10535 10536 - Config: encoding 10537 - Env Var: RCLONE_ACD_ENCODING 10538 - Type: MultiEncoder 10539 - Default: Slash,InvalidUtf8,Dot 10540 10541 10542 10543 ### Limitations ### 10544 10545 Note that Amazon Drive is case insensitive so you can't have a 10546 file called "Hello.doc" and one called "hello.doc". 10547 10548 Amazon Drive has rate limiting so you may notice errors in the 10549 sync (429 errors). rclone will automatically retry the sync up to 3 10550 times by default (see `--retries` flag) which should hopefully work 10551 around this problem. 10552 10553 Amazon Drive has an internal limit of file sizes that can be uploaded 10554 to the service. This limit is not officially published, but all files 10555 larger than this will fail. 10556 10557 At the time of writing (Jan 2016) is in the area of 50GB per file. 10558 This means that larger files are likely to fail. 10559 10560 Unfortunately there is no way for rclone to see that this failure is 10561 because of file size, so it will retry the operation, as any other 10562 failure. To avoid this problem, use `--max-size 50000M` option to limit 10563 the maximum size of uploaded files. Note that `--max-size` does not split 10564 files into segments, it only ignores files over this size. 10565 10566 Amazon S3 Storage Providers 10567 -------------------------------------------------------- 10568 10569 The S3 backend can be used with a number of different providers: 10570 10571 10572 - AWS S3 10573 - Alibaba Cloud (Aliyun) Object Storage System (OSS) 10574 - Ceph 10575 - DigitalOcean Spaces 10576 - Dreamhost 10577 - IBM COS S3 10578 - Minio 10579 - Scaleway 10580 - StackPath 10581 - Wasabi 10582 10583 10584 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 10585 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 10586 10587 Once you have made a remote (see the provider specific section above) 10588 you can use it like this: 10589 10590 See all buckets 10591 10592 rclone lsd remote: 10593 10594 Make a new bucket 10595 10596 rclone mkdir remote:bucket 10597 10598 List the contents of a bucket 10599 10600 rclone ls remote:bucket 10601 10602 Sync `/home/local/directory` to the remote bucket, deleting any excess 10603 files in the bucket. 10604 10605 rclone sync /home/local/directory remote:bucket 10606 10607 ## AWS S3 {#amazon-s3} 10608 10609 Here is an example of making an s3 configuration. First run 10610 10611 rclone config 10612 10613 This will guide you through an interactive setup process. 10614 10615 ``` 10616 No remotes found - make a new one 10617 n) New remote 10618 s) Set configuration password 10619 q) Quit config 10620 n/s/q> n 10621 name> remote 10622 Type of storage to configure. 10623 Choose a number from below, or type in your own value 10624 [snip] 10625 XX / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio) 10626 \ "s3" 10627 [snip] 10628 Storage> s3 10629 Choose your S3 provider. 10630 Choose a number from below, or type in your own value 10631 1 / Amazon Web Services (AWS) S3 10632 \ "AWS" 10633 2 / Ceph Object Storage 10634 \ "Ceph" 10635 3 / Digital Ocean Spaces 10636 \ "DigitalOcean" 10637 4 / Dreamhost DreamObjects 10638 \ "Dreamhost" 10639 5 / IBM COS S3 10640 \ "IBMCOS" 10641 6 / Minio Object Storage 10642 \ "Minio" 10643 7 / Wasabi Object Storage 10644 \ "Wasabi" 10645 8 / Any other S3 compatible provider 10646 \ "Other" 10647 provider> 1 10648 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 10649 Choose a number from below, or type in your own value 10650 1 / Enter AWS credentials in the next step 10651 \ "false" 10652 2 / Get AWS credentials from the environment (env vars or IAM) 10653 \ "true" 10654 env_auth> 1 10655 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 10656 access_key_id> XXX 10657 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 10658 secret_access_key> YYY 10659 Region to connect to. 10660 Choose a number from below, or type in your own value 10661 / The default endpoint - a good choice if you are unsure. 10662 1 | US Region, Northern Virginia or Pacific Northwest. 10663 | Leave location constraint empty. 10664 \ "us-east-1" 10665 / US East (Ohio) Region 10666 2 | Needs location constraint us-east-2. 10667 \ "us-east-2" 10668 / US West (Oregon) Region 10669 3 | Needs location constraint us-west-2. 10670 \ "us-west-2" 10671 / US West (Northern California) Region 10672 4 | Needs location constraint us-west-1. 10673 \ "us-west-1" 10674 / Canada (Central) Region 10675 5 | Needs location constraint ca-central-1. 10676 \ "ca-central-1" 10677 / EU (Ireland) Region 10678 6 | Needs location constraint EU or eu-west-1. 10679 \ "eu-west-1" 10680 / EU (London) Region 10681 7 | Needs location constraint eu-west-2. 10682 \ "eu-west-2" 10683 / EU (Frankfurt) Region 10684 8 | Needs location constraint eu-central-1. 10685 \ "eu-central-1" 10686 / Asia Pacific (Singapore) Region 10687 9 | Needs location constraint ap-southeast-1. 10688 \ "ap-southeast-1" 10689 / Asia Pacific (Sydney) Region 10690 10 | Needs location constraint ap-southeast-2. 10691 \ "ap-southeast-2" 10692 / Asia Pacific (Tokyo) Region 10693 11 | Needs location constraint ap-northeast-1. 10694 \ "ap-northeast-1" 10695 / Asia Pacific (Seoul) 10696 12 | Needs location constraint ap-northeast-2. 10697 \ "ap-northeast-2" 10698 / Asia Pacific (Mumbai) 10699 13 | Needs location constraint ap-south-1. 10700 \ "ap-south-1" 10701 / Asia Patific (Hong Kong) Region 10702 14 | Needs location constraint ap-east-1. 10703 \ "ap-east-1" 10704 / South America (Sao Paulo) Region 10705 15 | Needs location constraint sa-east-1. 10706 \ "sa-east-1" 10707 region> 1 10708 Endpoint for S3 API. 10709 Leave blank if using AWS to use the default endpoint for the region. 10710 endpoint> 10711 Location constraint - must be set to match the Region. Used when creating buckets only. 10712 Choose a number from below, or type in your own value 10713 1 / Empty for US Region, Northern Virginia or Pacific Northwest. 10714 \ "" 10715 2 / US East (Ohio) Region. 10716 \ "us-east-2" 10717 3 / US West (Oregon) Region. 10718 \ "us-west-2" 10719 4 / US West (Northern California) Region. 10720 \ "us-west-1" 10721 5 / Canada (Central) Region. 10722 \ "ca-central-1" 10723 6 / EU (Ireland) Region. 10724 \ "eu-west-1" 10725 7 / EU (London) Region. 10726 \ "eu-west-2" 10727 8 / EU Region. 10728 \ "EU" 10729 9 / Asia Pacific (Singapore) Region. 10730 \ "ap-southeast-1" 10731 10 / Asia Pacific (Sydney) Region. 10732 \ "ap-southeast-2" 10733 11 / Asia Pacific (Tokyo) Region. 10734 \ "ap-northeast-1" 10735 12 / Asia Pacific (Seoul) 10736 \ "ap-northeast-2" 10737 13 / Asia Pacific (Mumbai) 10738 \ "ap-south-1" 10739 14 / Asia Pacific (Hong Kong) 10740 \ "ap-east-1" 10741 15 / South America (Sao Paulo) Region. 10742 \ "sa-east-1" 10743 location_constraint> 1 10744 Canned ACL used when creating buckets and/or storing objects in S3. 10745 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 10746 Choose a number from below, or type in your own value 10747 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 10748 \ "private" 10749 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 10750 \ "public-read" 10751 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 10752 3 | Granting this on a bucket is generally not recommended. 10753 \ "public-read-write" 10754 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 10755 \ "authenticated-read" 10756 / Object owner gets FULL_CONTROL. Bucket owner gets READ access. 10757 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 10758 \ "bucket-owner-read" 10759 / Both the object owner and the bucket owner get FULL_CONTROL over the object. 10760 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 10761 \ "bucket-owner-full-control" 10762 acl> 1 10763 The server-side encryption algorithm used when storing this object in S3. 10764 Choose a number from below, or type in your own value 10765 1 / None 10766 \ "" 10767 2 / AES256 10768 \ "AES256" 10769 server_side_encryption> 1 10770 The storage class to use when storing objects in S3. 10771 Choose a number from below, or type in your own value 10772 1 / Default 10773 \ "" 10774 2 / Standard storage class 10775 \ "STANDARD" 10776 3 / Reduced redundancy storage class 10777 \ "REDUCED_REDUNDANCY" 10778 4 / Standard Infrequent Access storage class 10779 \ "STANDARD_IA" 10780 5 / One Zone Infrequent Access storage class 10781 \ "ONEZONE_IA" 10782 6 / Glacier storage class 10783 \ "GLACIER" 10784 7 / Glacier Deep Archive storage class 10785 \ "DEEP_ARCHIVE" 10786 8 / Intelligent-Tiering storage class 10787 \ "INTELLIGENT_TIERING" 10788 storage_class> 1 10789 Remote config 10790 -------------------- 10791 [remote] 10792 type = s3 10793 provider = AWS 10794 env_auth = false 10795 access_key_id = XXX 10796 secret_access_key = YYY 10797 region = us-east-1 10798 endpoint = 10799 location_constraint = 10800 acl = private 10801 server_side_encryption = 10802 storage_class = 10803 -------------------- 10804 y) Yes this is OK 10805 e) Edit this remote 10806 d) Delete this remote 10807 y/e/d> 10808 ``` 10809 10810 ### --fast-list ### 10811 10812 This remote supports `--fast-list` which allows you to use fewer 10813 transactions in exchange for more memory. See the [rclone 10814 docs](https://rclone.org/docs/#fast-list) for more details. 10815 10816 ### --update and --use-server-modtime ### 10817 10818 As noted below, the modified time is stored on metadata on the object. It is 10819 used by default for all operations that require checking the time a file was 10820 last updated. It allows rclone to treat the remote more like a true filesystem, 10821 but it is inefficient because it requires an extra API call to retrieve the 10822 metadata. 10823 10824 For many operations, the time the object was last uploaded to the remote is 10825 sufficient to determine if it is "dirty". By using `--update` along with 10826 `--use-server-modtime`, you can avoid the extra API call and simply upload 10827 files whose local modtime is newer than the time it was last uploaded. 10828 10829 ### Modified time ### 10830 10831 The modified time is stored as metadata on the object as 10832 `X-Amz-Meta-Mtime` as floating point since the epoch accurate to 1 ns. 10833 10834 If the modification time needs to be updated rclone will attempt to perform a server 10835 side copy to update the modification if the object can be copied in a single part. 10836 In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive 10837 storage the object will be uploaded rather than copied. 10838 10839 #### Restricted filename characters 10840 10841 S3 allows any valid UTF-8 string as a key. 10842 10843 Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), as 10844 they can't be used in XML. 10845 10846 The following characters are replaced since these are problematic when 10847 dealing with the REST API: 10848 10849 | Character | Value | Replacement | 10850 | --------- |:-----:|:-----------:| 10851 | NUL | 0x00 | ␀ | 10852 | / | 0x2F | / | 10853 10854 The encoding will also encode these file names as they don't seem to 10855 work with the SDK properly: 10856 10857 | File name | Replacement | 10858 | --------- |:-----------:| 10859 | . | . | 10860 | .. | .. | 10861 10862 ### Multipart uploads ### 10863 10864 rclone supports multipart uploads with S3 which means that it can 10865 upload files bigger than 5GB. 10866 10867 Note that files uploaded *both* with multipart upload *and* through 10868 crypt remotes do not have MD5 sums. 10869 10870 rclone switches from single part uploads to multipart uploads at the 10871 point specified by `--s3-upload-cutoff`. This can be a maximum of 5GB 10872 and a minimum of 0 (ie always upload multipart files). 10873 10874 The chunk sizes used in the multipart upload are specified by 10875 `--s3-chunk-size` and the number of chunks uploaded concurrently is 10876 specified by `--s3-upload-concurrency`. 10877 10878 Multipart uploads will use `--transfers` * `--s3-upload-concurrency` * 10879 `--s3-chunk-size` extra memory. Single part uploads to not use extra 10880 memory. 10881 10882 Single part transfers can be faster than multipart transfers or slower 10883 depending on your latency from S3 - the more latency, the more likely 10884 single part transfers will be faster. 10885 10886 Increasing `--s3-upload-concurrency` will increase throughput (8 would 10887 be a sensible value) and increasing `--s3-chunk-size` also increases 10888 throughput (16M would be sensible). Increasing either of these will 10889 use more memory. The default values are high enough to gain most of 10890 the possible performance without using too much memory. 10891 10892 10893 ### Buckets and Regions ### 10894 10895 With Amazon S3 you can list buckets (`rclone lsd`) using any region, 10896 but you can only access the content of a bucket from the region it was 10897 created in. If you attempt to access a bucket from the wrong region, 10898 you will get an error, `incorrect region, the bucket is not in 'XXX' 10899 region`. 10900 10901 ### Authentication ### 10902 10903 There are a number of ways to supply `rclone` with a set of AWS 10904 credentials, with and without using the environment. 10905 10906 The different authentication methods are tried in this order: 10907 10908 - Directly in the rclone configuration file (`env_auth = false` in the config file): 10909 - `access_key_id` and `secret_access_key` are required. 10910 - `session_token` can be optionally set when using AWS STS. 10911 - Runtime configuration (`env_auth = true` in the config file): 10912 - Export the following environment variables before running `rclone`: 10913 - Access Key ID: `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY` 10914 - Secret Access Key: `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY` 10915 - Session Token: `AWS_SESSION_TOKEN` (optional) 10916 - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html): 10917 - Profile files are standard files used by AWS CLI tools 10918 - By default it will use the profile in your home directory (eg `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables: 10919 - `AWS_SHARED_CREDENTIALS_FILE` to control which file. 10920 - `AWS_PROFILE` to control which profile to use. 10921 - Or, run `rclone` in an ECS task with an IAM role (AWS only). 10922 - Or, run `rclone` on an EC2 instance with an IAM role (AWS only). 10923 - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only). 10924 10925 If none of these option actually end up providing `rclone` with AWS 10926 credentials then S3 interaction will be non-authenticated (see below). 10927 10928 ### S3 Permissions ### 10929 10930 When using the `sync` subcommand of `rclone` the following minimum 10931 permissions are required to be available on the bucket being written to: 10932 10933 * `ListBucket` 10934 * `DeleteObject` 10935 * `GetObject` 10936 * `PutObject` 10937 * `PutObjectACL` 10938 10939 When using the `lsd` subcommand, the `ListAllMyBuckets` permission is required. 10940 10941 Example policy: 10942 10943 ``` 10944 { 10945 "Version": "2012-10-17", 10946 "Statement": [ 10947 { 10948 "Effect": "Allow", 10949 "Principal": { 10950 "AWS": "arn:aws:iam::USER_SID:user/USER_NAME" 10951 }, 10952 "Action": [ 10953 "s3:ListBucket", 10954 "s3:DeleteObject", 10955 "s3:GetObject", 10956 "s3:PutObject", 10957 "s3:PutObjectAcl" 10958 ], 10959 "Resource": [ 10960 "arn:aws:s3:::BUCKET_NAME/*", 10961 "arn:aws:s3:::BUCKET_NAME" 10962 ] 10963 }, 10964 { 10965 "Effect": "Allow", 10966 "Action": "s3:ListAllMyBuckets", 10967 "Resource": "arn:aws:s3:::*" 10968 } 10969 ] 10970 } 10971 ``` 10972 10973 Notes on above: 10974 10975 1. This is a policy that can be used when creating bucket. It assumes 10976 that `USER_NAME` has been created. 10977 2. The Resource entry must include both resource ARNs, as one implies 10978 the bucket and the other implies the bucket's objects. 10979 10980 For reference, [here's an Ansible script](https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b) 10981 that will generate one or more buckets that will work with `rclone sync`. 10982 10983 ### Key Management System (KMS) ### 10984 10985 If you are using server side encryption with KMS then you will find 10986 you can't transfer small objects. As a work-around you can use the 10987 `--ignore-checksum` flag. 10988 10989 A proper fix is being worked on in [issue #1824](https://github.com/rclone/rclone/issues/1824). 10990 10991 ### Glacier and Glacier Deep Archive ### 10992 10993 You can upload objects using the glacier storage class or transition them to glacier using a [lifecycle policy](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html). 10994 The bucket can still be synced or copied into normally, but if rclone 10995 tries to access data from the glacier storage class you will see an error like below. 10996 10997 2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file 10998 10999 In this case you need to [restore](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html) 11000 the object(s) in question before using rclone. 11001 11002 Note that rclone only speaks the S3 API it does not speak the Glacier 11003 Vault API, so rclone cannot directly access Glacier Vaults. 11004 11005 11006 ### Standard Options 11007 11008 Here are the standard options specific to s3 (Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)). 11009 11010 #### --s3-provider 11011 11012 Choose your S3 provider. 11013 11014 - Config: provider 11015 - Env Var: RCLONE_S3_PROVIDER 11016 - Type: string 11017 - Default: "" 11018 - Examples: 11019 - "AWS" 11020 - Amazon Web Services (AWS) S3 11021 - "Alibaba" 11022 - Alibaba Cloud Object Storage System (OSS) formerly Aliyun 11023 - "Ceph" 11024 - Ceph Object Storage 11025 - "DigitalOcean" 11026 - Digital Ocean Spaces 11027 - "Dreamhost" 11028 - Dreamhost DreamObjects 11029 - "IBMCOS" 11030 - IBM COS S3 11031 - "Minio" 11032 - Minio Object Storage 11033 - "Netease" 11034 - Netease Object Storage (NOS) 11035 - "StackPath" 11036 - StackPath Object Storage 11037 - "Wasabi" 11038 - Wasabi Object Storage 11039 - "Other" 11040 - Any other S3 compatible provider 11041 11042 #### --s3-env-auth 11043 11044 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 11045 Only applies if access_key_id and secret_access_key is blank. 11046 11047 - Config: env_auth 11048 - Env Var: RCLONE_S3_ENV_AUTH 11049 - Type: bool 11050 - Default: false 11051 - Examples: 11052 - "false" 11053 - Enter AWS credentials in the next step 11054 - "true" 11055 - Get AWS credentials from the environment (env vars or IAM) 11056 11057 #### --s3-access-key-id 11058 11059 AWS Access Key ID. 11060 Leave blank for anonymous access or runtime credentials. 11061 11062 - Config: access_key_id 11063 - Env Var: RCLONE_S3_ACCESS_KEY_ID 11064 - Type: string 11065 - Default: "" 11066 11067 #### --s3-secret-access-key 11068 11069 AWS Secret Access Key (password) 11070 Leave blank for anonymous access or runtime credentials. 11071 11072 - Config: secret_access_key 11073 - Env Var: RCLONE_S3_SECRET_ACCESS_KEY 11074 - Type: string 11075 - Default: "" 11076 11077 #### --s3-region 11078 11079 Region to connect to. 11080 11081 - Config: region 11082 - Env Var: RCLONE_S3_REGION 11083 - Type: string 11084 - Default: "" 11085 - Examples: 11086 - "us-east-1" 11087 - The default endpoint - a good choice if you are unsure. 11088 - US Region, Northern Virginia or Pacific Northwest. 11089 - Leave location constraint empty. 11090 - "us-east-2" 11091 - US East (Ohio) Region 11092 - Needs location constraint us-east-2. 11093 - "us-west-2" 11094 - US West (Oregon) Region 11095 - Needs location constraint us-west-2. 11096 - "us-west-1" 11097 - US West (Northern California) Region 11098 - Needs location constraint us-west-1. 11099 - "ca-central-1" 11100 - Canada (Central) Region 11101 - Needs location constraint ca-central-1. 11102 - "eu-west-1" 11103 - EU (Ireland) Region 11104 - Needs location constraint EU or eu-west-1. 11105 - "eu-west-2" 11106 - EU (London) Region 11107 - Needs location constraint eu-west-2. 11108 - "eu-north-1" 11109 - EU (Stockholm) Region 11110 - Needs location constraint eu-north-1. 11111 - "eu-central-1" 11112 - EU (Frankfurt) Region 11113 - Needs location constraint eu-central-1. 11114 - "ap-southeast-1" 11115 - Asia Pacific (Singapore) Region 11116 - Needs location constraint ap-southeast-1. 11117 - "ap-southeast-2" 11118 - Asia Pacific (Sydney) Region 11119 - Needs location constraint ap-southeast-2. 11120 - "ap-northeast-1" 11121 - Asia Pacific (Tokyo) Region 11122 - Needs location constraint ap-northeast-1. 11123 - "ap-northeast-2" 11124 - Asia Pacific (Seoul) 11125 - Needs location constraint ap-northeast-2. 11126 - "ap-south-1" 11127 - Asia Pacific (Mumbai) 11128 - Needs location constraint ap-south-1. 11129 - "ap-east-1" 11130 - Asia Patific (Hong Kong) Region 11131 - Needs location constraint ap-east-1. 11132 - "sa-east-1" 11133 - South America (Sao Paulo) Region 11134 - Needs location constraint sa-east-1. 11135 11136 #### --s3-region 11137 11138 Region to connect to. 11139 Leave blank if you are using an S3 clone and you don't have a region. 11140 11141 - Config: region 11142 - Env Var: RCLONE_S3_REGION 11143 - Type: string 11144 - Default: "" 11145 - Examples: 11146 - "" 11147 - Use this if unsure. Will use v4 signatures and an empty region. 11148 - "other-v2-signature" 11149 - Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH. 11150 11151 #### --s3-endpoint 11152 11153 Endpoint for S3 API. 11154 Leave blank if using AWS to use the default endpoint for the region. 11155 11156 - Config: endpoint 11157 - Env Var: RCLONE_S3_ENDPOINT 11158 - Type: string 11159 - Default: "" 11160 11161 #### --s3-endpoint 11162 11163 Endpoint for IBM COS S3 API. 11164 Specify if using an IBM COS On Premise. 11165 11166 - Config: endpoint 11167 - Env Var: RCLONE_S3_ENDPOINT 11168 - Type: string 11169 - Default: "" 11170 - Examples: 11171 - "s3-api.us-geo.objectstorage.softlayer.net" 11172 - US Cross Region Endpoint 11173 - "s3-api.dal.us-geo.objectstorage.softlayer.net" 11174 - US Cross Region Dallas Endpoint 11175 - "s3-api.wdc-us-geo.objectstorage.softlayer.net" 11176 - US Cross Region Washington DC Endpoint 11177 - "s3-api.sjc-us-geo.objectstorage.softlayer.net" 11178 - US Cross Region San Jose Endpoint 11179 - "s3-api.us-geo.objectstorage.service.networklayer.com" 11180 - US Cross Region Private Endpoint 11181 - "s3-api.dal-us-geo.objectstorage.service.networklayer.com" 11182 - US Cross Region Dallas Private Endpoint 11183 - "s3-api.wdc-us-geo.objectstorage.service.networklayer.com" 11184 - US Cross Region Washington DC Private Endpoint 11185 - "s3-api.sjc-us-geo.objectstorage.service.networklayer.com" 11186 - US Cross Region San Jose Private Endpoint 11187 - "s3.us-east.objectstorage.softlayer.net" 11188 - US Region East Endpoint 11189 - "s3.us-east.objectstorage.service.networklayer.com" 11190 - US Region East Private Endpoint 11191 - "s3.us-south.objectstorage.softlayer.net" 11192 - US Region South Endpoint 11193 - "s3.us-south.objectstorage.service.networklayer.com" 11194 - US Region South Private Endpoint 11195 - "s3.eu-geo.objectstorage.softlayer.net" 11196 - EU Cross Region Endpoint 11197 - "s3.fra-eu-geo.objectstorage.softlayer.net" 11198 - EU Cross Region Frankfurt Endpoint 11199 - "s3.mil-eu-geo.objectstorage.softlayer.net" 11200 - EU Cross Region Milan Endpoint 11201 - "s3.ams-eu-geo.objectstorage.softlayer.net" 11202 - EU Cross Region Amsterdam Endpoint 11203 - "s3.eu-geo.objectstorage.service.networklayer.com" 11204 - EU Cross Region Private Endpoint 11205 - "s3.fra-eu-geo.objectstorage.service.networklayer.com" 11206 - EU Cross Region Frankfurt Private Endpoint 11207 - "s3.mil-eu-geo.objectstorage.service.networklayer.com" 11208 - EU Cross Region Milan Private Endpoint 11209 - "s3.ams-eu-geo.objectstorage.service.networklayer.com" 11210 - EU Cross Region Amsterdam Private Endpoint 11211 - "s3.eu-gb.objectstorage.softlayer.net" 11212 - Great Britain Endpoint 11213 - "s3.eu-gb.objectstorage.service.networklayer.com" 11214 - Great Britain Private Endpoint 11215 - "s3.ap-geo.objectstorage.softlayer.net" 11216 - APAC Cross Regional Endpoint 11217 - "s3.tok-ap-geo.objectstorage.softlayer.net" 11218 - APAC Cross Regional Tokyo Endpoint 11219 - "s3.hkg-ap-geo.objectstorage.softlayer.net" 11220 - APAC Cross Regional HongKong Endpoint 11221 - "s3.seo-ap-geo.objectstorage.softlayer.net" 11222 - APAC Cross Regional Seoul Endpoint 11223 - "s3.ap-geo.objectstorage.service.networklayer.com" 11224 - APAC Cross Regional Private Endpoint 11225 - "s3.tok-ap-geo.objectstorage.service.networklayer.com" 11226 - APAC Cross Regional Tokyo Private Endpoint 11227 - "s3.hkg-ap-geo.objectstorage.service.networklayer.com" 11228 - APAC Cross Regional HongKong Private Endpoint 11229 - "s3.seo-ap-geo.objectstorage.service.networklayer.com" 11230 - APAC Cross Regional Seoul Private Endpoint 11231 - "s3.mel01.objectstorage.softlayer.net" 11232 - Melbourne Single Site Endpoint 11233 - "s3.mel01.objectstorage.service.networklayer.com" 11234 - Melbourne Single Site Private Endpoint 11235 - "s3.tor01.objectstorage.softlayer.net" 11236 - Toronto Single Site Endpoint 11237 - "s3.tor01.objectstorage.service.networklayer.com" 11238 - Toronto Single Site Private Endpoint 11239 11240 #### --s3-endpoint 11241 11242 Endpoint for OSS API. 11243 11244 - Config: endpoint 11245 - Env Var: RCLONE_S3_ENDPOINT 11246 - Type: string 11247 - Default: "" 11248 - Examples: 11249 - "oss-cn-hangzhou.aliyuncs.com" 11250 - East China 1 (Hangzhou) 11251 - "oss-cn-shanghai.aliyuncs.com" 11252 - East China 2 (Shanghai) 11253 - "oss-cn-qingdao.aliyuncs.com" 11254 - North China 1 (Qingdao) 11255 - "oss-cn-beijing.aliyuncs.com" 11256 - North China 2 (Beijing) 11257 - "oss-cn-zhangjiakou.aliyuncs.com" 11258 - North China 3 (Zhangjiakou) 11259 - "oss-cn-huhehaote.aliyuncs.com" 11260 - North China 5 (Huhehaote) 11261 - "oss-cn-shenzhen.aliyuncs.com" 11262 - South China 1 (Shenzhen) 11263 - "oss-cn-hongkong.aliyuncs.com" 11264 - Hong Kong (Hong Kong) 11265 - "oss-us-west-1.aliyuncs.com" 11266 - US West 1 (Silicon Valley) 11267 - "oss-us-east-1.aliyuncs.com" 11268 - US East 1 (Virginia) 11269 - "oss-ap-southeast-1.aliyuncs.com" 11270 - Southeast Asia Southeast 1 (Singapore) 11271 - "oss-ap-southeast-2.aliyuncs.com" 11272 - Asia Pacific Southeast 2 (Sydney) 11273 - "oss-ap-southeast-3.aliyuncs.com" 11274 - Southeast Asia Southeast 3 (Kuala Lumpur) 11275 - "oss-ap-southeast-5.aliyuncs.com" 11276 - Asia Pacific Southeast 5 (Jakarta) 11277 - "oss-ap-northeast-1.aliyuncs.com" 11278 - Asia Pacific Northeast 1 (Japan) 11279 - "oss-ap-south-1.aliyuncs.com" 11280 - Asia Pacific South 1 (Mumbai) 11281 - "oss-eu-central-1.aliyuncs.com" 11282 - Central Europe 1 (Frankfurt) 11283 - "oss-eu-west-1.aliyuncs.com" 11284 - West Europe (London) 11285 - "oss-me-east-1.aliyuncs.com" 11286 - Middle East 1 (Dubai) 11287 11288 #### --s3-endpoint 11289 11290 Endpoint for StackPath Object Storage. 11291 11292 - Config: endpoint 11293 - Env Var: RCLONE_S3_ENDPOINT 11294 - Type: string 11295 - Default: "" 11296 - Examples: 11297 - "s3.us-east-2.stackpathstorage.com" 11298 - US East Endpoint 11299 - "s3.us-west-1.stackpathstorage.com" 11300 - US West Endpoint 11301 - "s3.eu-central-1.stackpathstorage.com" 11302 - EU Endpoint 11303 11304 #### --s3-endpoint 11305 11306 Endpoint for S3 API. 11307 Required when using an S3 clone. 11308 11309 - Config: endpoint 11310 - Env Var: RCLONE_S3_ENDPOINT 11311 - Type: string 11312 - Default: "" 11313 - Examples: 11314 - "objects-us-east-1.dream.io" 11315 - Dream Objects endpoint 11316 - "nyc3.digitaloceanspaces.com" 11317 - Digital Ocean Spaces New York 3 11318 - "ams3.digitaloceanspaces.com" 11319 - Digital Ocean Spaces Amsterdam 3 11320 - "sgp1.digitaloceanspaces.com" 11321 - Digital Ocean Spaces Singapore 1 11322 - "s3.wasabisys.com" 11323 - Wasabi US East endpoint 11324 - "s3.us-west-1.wasabisys.com" 11325 - Wasabi US West endpoint 11326 - "s3.eu-central-1.wasabisys.com" 11327 - Wasabi EU Central endpoint 11328 11329 #### --s3-location-constraint 11330 11331 Location constraint - must be set to match the Region. 11332 Used when creating buckets only. 11333 11334 - Config: location_constraint 11335 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 11336 - Type: string 11337 - Default: "" 11338 - Examples: 11339 - "" 11340 - Empty for US Region, Northern Virginia or Pacific Northwest. 11341 - "us-east-2" 11342 - US East (Ohio) Region. 11343 - "us-west-2" 11344 - US West (Oregon) Region. 11345 - "us-west-1" 11346 - US West (Northern California) Region. 11347 - "ca-central-1" 11348 - Canada (Central) Region. 11349 - "eu-west-1" 11350 - EU (Ireland) Region. 11351 - "eu-west-2" 11352 - EU (London) Region. 11353 - "eu-north-1" 11354 - EU (Stockholm) Region. 11355 - "EU" 11356 - EU Region. 11357 - "ap-southeast-1" 11358 - Asia Pacific (Singapore) Region. 11359 - "ap-southeast-2" 11360 - Asia Pacific (Sydney) Region. 11361 - "ap-northeast-1" 11362 - Asia Pacific (Tokyo) Region. 11363 - "ap-northeast-2" 11364 - Asia Pacific (Seoul) 11365 - "ap-south-1" 11366 - Asia Pacific (Mumbai) 11367 - "ap-east-1" 11368 - Asia Pacific (Hong Kong) 11369 - "sa-east-1" 11370 - South America (Sao Paulo) Region. 11371 11372 #### --s3-location-constraint 11373 11374 Location constraint - must match endpoint when using IBM Cloud Public. 11375 For on-prem COS, do not make a selection from this list, hit enter 11376 11377 - Config: location_constraint 11378 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 11379 - Type: string 11380 - Default: "" 11381 - Examples: 11382 - "us-standard" 11383 - US Cross Region Standard 11384 - "us-vault" 11385 - US Cross Region Vault 11386 - "us-cold" 11387 - US Cross Region Cold 11388 - "us-flex" 11389 - US Cross Region Flex 11390 - "us-east-standard" 11391 - US East Region Standard 11392 - "us-east-vault" 11393 - US East Region Vault 11394 - "us-east-cold" 11395 - US East Region Cold 11396 - "us-east-flex" 11397 - US East Region Flex 11398 - "us-south-standard" 11399 - US South Region Standard 11400 - "us-south-vault" 11401 - US South Region Vault 11402 - "us-south-cold" 11403 - US South Region Cold 11404 - "us-south-flex" 11405 - US South Region Flex 11406 - "eu-standard" 11407 - EU Cross Region Standard 11408 - "eu-vault" 11409 - EU Cross Region Vault 11410 - "eu-cold" 11411 - EU Cross Region Cold 11412 - "eu-flex" 11413 - EU Cross Region Flex 11414 - "eu-gb-standard" 11415 - Great Britain Standard 11416 - "eu-gb-vault" 11417 - Great Britain Vault 11418 - "eu-gb-cold" 11419 - Great Britain Cold 11420 - "eu-gb-flex" 11421 - Great Britain Flex 11422 - "ap-standard" 11423 - APAC Standard 11424 - "ap-vault" 11425 - APAC Vault 11426 - "ap-cold" 11427 - APAC Cold 11428 - "ap-flex" 11429 - APAC Flex 11430 - "mel01-standard" 11431 - Melbourne Standard 11432 - "mel01-vault" 11433 - Melbourne Vault 11434 - "mel01-cold" 11435 - Melbourne Cold 11436 - "mel01-flex" 11437 - Melbourne Flex 11438 - "tor01-standard" 11439 - Toronto Standard 11440 - "tor01-vault" 11441 - Toronto Vault 11442 - "tor01-cold" 11443 - Toronto Cold 11444 - "tor01-flex" 11445 - Toronto Flex 11446 11447 #### --s3-location-constraint 11448 11449 Location constraint - must be set to match the Region. 11450 Leave blank if not sure. Used when creating buckets only. 11451 11452 - Config: location_constraint 11453 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 11454 - Type: string 11455 - Default: "" 11456 11457 #### --s3-acl 11458 11459 Canned ACL used when creating buckets and storing or copying objects. 11460 11461 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 11462 11463 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 11464 11465 Note that this ACL is applied when server side copying objects as S3 11466 doesn't copy the ACL from the source but rather writes a fresh one. 11467 11468 - Config: acl 11469 - Env Var: RCLONE_S3_ACL 11470 - Type: string 11471 - Default: "" 11472 - Examples: 11473 - "private" 11474 - Owner gets FULL_CONTROL. No one else has access rights (default). 11475 - "public-read" 11476 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. 11477 - "public-read-write" 11478 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 11479 - Granting this on a bucket is generally not recommended. 11480 - "authenticated-read" 11481 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 11482 - "bucket-owner-read" 11483 - Object owner gets FULL_CONTROL. Bucket owner gets READ access. 11484 - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 11485 - "bucket-owner-full-control" 11486 - Both the object owner and the bucket owner get FULL_CONTROL over the object. 11487 - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 11488 - "private" 11489 - Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS 11490 - "public-read" 11491 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS 11492 - "public-read-write" 11493 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS 11494 - "authenticated-read" 11495 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS 11496 11497 #### --s3-server-side-encryption 11498 11499 The server-side encryption algorithm used when storing this object in S3. 11500 11501 - Config: server_side_encryption 11502 - Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION 11503 - Type: string 11504 - Default: "" 11505 - Examples: 11506 - "" 11507 - None 11508 - "AES256" 11509 - AES256 11510 - "aws:kms" 11511 - aws:kms 11512 11513 #### --s3-sse-kms-key-id 11514 11515 If using KMS ID you must provide the ARN of Key. 11516 11517 - Config: sse_kms_key_id 11518 - Env Var: RCLONE_S3_SSE_KMS_KEY_ID 11519 - Type: string 11520 - Default: "" 11521 - Examples: 11522 - "" 11523 - None 11524 - "arn:aws:kms:us-east-1:*" 11525 - arn:aws:kms:* 11526 11527 #### --s3-storage-class 11528 11529 The storage class to use when storing new objects in S3. 11530 11531 - Config: storage_class 11532 - Env Var: RCLONE_S3_STORAGE_CLASS 11533 - Type: string 11534 - Default: "" 11535 - Examples: 11536 - "" 11537 - Default 11538 - "STANDARD" 11539 - Standard storage class 11540 - "REDUCED_REDUNDANCY" 11541 - Reduced redundancy storage class 11542 - "STANDARD_IA" 11543 - Standard Infrequent Access storage class 11544 - "ONEZONE_IA" 11545 - One Zone Infrequent Access storage class 11546 - "GLACIER" 11547 - Glacier storage class 11548 - "DEEP_ARCHIVE" 11549 - Glacier Deep Archive storage class 11550 - "INTELLIGENT_TIERING" 11551 - Intelligent-Tiering storage class 11552 11553 #### --s3-storage-class 11554 11555 The storage class to use when storing new objects in OSS. 11556 11557 - Config: storage_class 11558 - Env Var: RCLONE_S3_STORAGE_CLASS 11559 - Type: string 11560 - Default: "" 11561 - Examples: 11562 - "" 11563 - Default 11564 - "STANDARD" 11565 - Standard storage class 11566 - "GLACIER" 11567 - Archive storage mode. 11568 - "STANDARD_IA" 11569 - Infrequent access storage mode. 11570 11571 ### Advanced Options 11572 11573 Here are the advanced options specific to s3 (Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)). 11574 11575 #### --s3-bucket-acl 11576 11577 Canned ACL used when creating buckets. 11578 11579 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 11580 11581 Note that this ACL is applied when only when creating buckets. If it 11582 isn't set then "acl" is used instead. 11583 11584 - Config: bucket_acl 11585 - Env Var: RCLONE_S3_BUCKET_ACL 11586 - Type: string 11587 - Default: "" 11588 - Examples: 11589 - "private" 11590 - Owner gets FULL_CONTROL. No one else has access rights (default). 11591 - "public-read" 11592 - Owner gets FULL_CONTROL. The AllUsers group gets READ access. 11593 - "public-read-write" 11594 - Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 11595 - Granting this on a bucket is generally not recommended. 11596 - "authenticated-read" 11597 - Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 11598 11599 #### --s3-sse-customer-algorithm 11600 11601 If using SSE-C, the server-side encryption algorithm used when storing this object in S3. 11602 11603 - Config: sse_customer_algorithm 11604 - Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM 11605 - Type: string 11606 - Default: "" 11607 - Examples: 11608 - "" 11609 - None 11610 - "AES256" 11611 - AES256 11612 11613 #### --s3-sse-customer-key 11614 11615 If using SSE-C you must provide the secret encryption key used to encrypt/decrypt your data. 11616 11617 - Config: sse_customer_key 11618 - Env Var: RCLONE_S3_SSE_CUSTOMER_KEY 11619 - Type: string 11620 - Default: "" 11621 - Examples: 11622 - "" 11623 - None 11624 11625 #### --s3-sse-customer-key-md5 11626 11627 If using SSE-C you must provide the secret encryption key MD5 checksum. 11628 11629 - Config: sse_customer_key_md5 11630 - Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5 11631 - Type: string 11632 - Default: "" 11633 - Examples: 11634 - "" 11635 - None 11636 11637 #### --s3-upload-cutoff 11638 11639 Cutoff for switching to chunked upload 11640 11641 Any files larger than this will be uploaded in chunks of chunk_size. 11642 The minimum is 0 and the maximum is 5GB. 11643 11644 - Config: upload_cutoff 11645 - Env Var: RCLONE_S3_UPLOAD_CUTOFF 11646 - Type: SizeSuffix 11647 - Default: 200M 11648 11649 #### --s3-chunk-size 11650 11651 Chunk size to use for uploading. 11652 11653 When uploading files larger than upload_cutoff or files with unknown 11654 size (eg from "rclone rcat" or uploaded with "rclone mount" or google 11655 photos or google docs) they will be uploaded as multipart uploads 11656 using this chunk size. 11657 11658 Note that "--s3-upload-concurrency" chunks of this size are buffered 11659 in memory per transfer. 11660 11661 If you are transferring large files over high speed links and you have 11662 enough memory, then increasing this will speed up the transfers. 11663 11664 Rclone will automatically increase the chunk size when uploading a 11665 large file of known size to stay below the 10,000 chunks limit. 11666 11667 Files of unknown size are uploaded with the configured 11668 chunk_size. Since the default chunk size is 5MB and there can be at 11669 most 10,000 chunks, this means that by default the maximum size of 11670 file you can stream upload is 48GB. If you wish to stream upload 11671 larger files then you will need to increase chunk_size. 11672 11673 - Config: chunk_size 11674 - Env Var: RCLONE_S3_CHUNK_SIZE 11675 - Type: SizeSuffix 11676 - Default: 5M 11677 11678 #### --s3-copy-cutoff 11679 11680 Cutoff for switching to multipart copy 11681 11682 Any files larger than this that need to be server side copied will be 11683 copied in chunks of this size. 11684 11685 The minimum is 0 and the maximum is 5GB. 11686 11687 - Config: copy_cutoff 11688 - Env Var: RCLONE_S3_COPY_CUTOFF 11689 - Type: SizeSuffix 11690 - Default: 5G 11691 11692 #### --s3-disable-checksum 11693 11694 Don't store MD5 checksum with object metadata 11695 11696 Normally rclone will calculate the MD5 checksum of the input before 11697 uploading it so it can add it to metadata on the object. This is great 11698 for data integrity checking but can cause long delays for large files 11699 to start uploading. 11700 11701 - Config: disable_checksum 11702 - Env Var: RCLONE_S3_DISABLE_CHECKSUM 11703 - Type: bool 11704 - Default: false 11705 11706 #### --s3-session-token 11707 11708 An AWS session token 11709 11710 - Config: session_token 11711 - Env Var: RCLONE_S3_SESSION_TOKEN 11712 - Type: string 11713 - Default: "" 11714 11715 #### --s3-upload-concurrency 11716 11717 Concurrency for multipart uploads. 11718 11719 This is the number of chunks of the same file that are uploaded 11720 concurrently. 11721 11722 If you are uploading small numbers of large file over high speed link 11723 and these uploads do not fully utilize your bandwidth, then increasing 11724 this may help to speed up the transfers. 11725 11726 - Config: upload_concurrency 11727 - Env Var: RCLONE_S3_UPLOAD_CONCURRENCY 11728 - Type: int 11729 - Default: 4 11730 11731 #### --s3-force-path-style 11732 11733 If true use path style access if false use virtual hosted style. 11734 11735 If this is true (the default) then rclone will use path style access, 11736 if false then rclone will use virtual path style. See [the AWS S3 11737 docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro) 11738 for more info. 11739 11740 Some providers (eg AWS, Aliyun OSS or Netease COS) require this set to 11741 false - rclone will do this automatically based on the provider 11742 setting. 11743 11744 - Config: force_path_style 11745 - Env Var: RCLONE_S3_FORCE_PATH_STYLE 11746 - Type: bool 11747 - Default: true 11748 11749 #### --s3-v2-auth 11750 11751 If true use v2 authentication. 11752 11753 If this is false (the default) then rclone will use v4 authentication. 11754 If it is set then rclone will use v2 authentication. 11755 11756 Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH. 11757 11758 - Config: v2_auth 11759 - Env Var: RCLONE_S3_V2_AUTH 11760 - Type: bool 11761 - Default: false 11762 11763 #### --s3-use-accelerate-endpoint 11764 11765 If true use the AWS S3 accelerated endpoint. 11766 11767 See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html) 11768 11769 - Config: use_accelerate_endpoint 11770 - Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT 11771 - Type: bool 11772 - Default: false 11773 11774 #### --s3-leave-parts-on-error 11775 11776 If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. 11777 11778 It should be set to true for resuming uploads across different sessions. 11779 11780 WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up. 11781 11782 11783 - Config: leave_parts_on_error 11784 - Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR 11785 - Type: bool 11786 - Default: false 11787 11788 #### --s3-list-chunk 11789 11790 Size of listing chunk (response list for each ListObject S3 request). 11791 11792 This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification. 11793 Most services truncate the response list to 1000 objects even if requested more than that. 11794 In AWS S3 this is a global maximum and cannot be changed, see [AWS S3](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html). 11795 In Ceph, this can be increased with the "rgw list buckets max chunk" option. 11796 11797 11798 - Config: list_chunk 11799 - Env Var: RCLONE_S3_LIST_CHUNK 11800 - Type: int 11801 - Default: 1000 11802 11803 #### --s3-encoding 11804 11805 This sets the encoding for the backend. 11806 11807 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 11808 11809 - Config: encoding 11810 - Env Var: RCLONE_S3_ENCODING 11811 - Type: MultiEncoder 11812 - Default: Slash,InvalidUtf8,Dot 11813 11814 #### --s3-memory-pool-flush-time 11815 11816 How often internal memory buffer pools will be flushed. 11817 Uploads which requires additional buffers (f.e multipart) will use memory pool for allocations. 11818 This option controls how often unused buffers will be removed from the pool. 11819 11820 - Config: memory_pool_flush_time 11821 - Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME 11822 - Type: Duration 11823 - Default: 1m0s 11824 11825 #### --s3-memory-pool-use-mmap 11826 11827 Whether to use mmap buffers in internal memory pool. 11828 11829 - Config: memory_pool_use_mmap 11830 - Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP 11831 - Type: bool 11832 - Default: false 11833 11834 11835 11836 ### Anonymous access to public buckets ### 11837 11838 If you want to use rclone to access a public bucket, configure with a 11839 blank `access_key_id` and `secret_access_key`. Your config should end 11840 up looking like this: 11841 11842 ``` 11843 [anons3] 11844 type = s3 11845 provider = AWS 11846 env_auth = false 11847 access_key_id = 11848 secret_access_key = 11849 region = us-east-1 11850 endpoint = 11851 location_constraint = 11852 acl = private 11853 server_side_encryption = 11854 storage_class = 11855 ``` 11856 11857 Then use it as normal with the name of the public bucket, eg 11858 11859 rclone lsd anons3:1000genomes 11860 11861 You will be able to list and copy data but not upload it. 11862 11863 ### Ceph ### 11864 11865 [Ceph](https://ceph.com/) is an open source unified, distributed 11866 storage system designed for excellent performance, reliability and 11867 scalability. It has an S3 compatible object storage interface. 11868 11869 To use rclone with Ceph, configure as above but leave the region blank 11870 and set the endpoint. You should end up with something like this in 11871 your config: 11872 11873 11874 ``` 11875 [ceph] 11876 type = s3 11877 provider = Ceph 11878 env_auth = false 11879 access_key_id = XXX 11880 secret_access_key = YYY 11881 region = 11882 endpoint = https://ceph.endpoint.example.com 11883 location_constraint = 11884 acl = 11885 server_side_encryption = 11886 storage_class = 11887 ``` 11888 11889 If you are using an older version of CEPH, eg 10.2.x Jewel, then you 11890 may need to supply the parameter `--s3-upload-cutoff 0` or put this in 11891 the config file as `upload_cutoff 0` to work around a bug which causes 11892 uploading of small files to fail. 11893 11894 Note also that Ceph sometimes puts `/` in the passwords it gives 11895 users. If you read the secret access key using the command line tools 11896 you will get a JSON blob with the `/` escaped as `\/`. Make sure you 11897 only write `/` in the secret access key. 11898 11899 Eg the dump from Ceph looks something like this (irrelevant keys 11900 removed). 11901 11902 ``` 11903 { 11904 "user_id": "xxx", 11905 "display_name": "xxxx", 11906 "keys": [ 11907 { 11908 "user": "xxx", 11909 "access_key": "xxxxxx", 11910 "secret_key": "xxxxxx\/xxxx" 11911 } 11912 ], 11913 } 11914 ``` 11915 11916 Because this is a json dump, it is encoding the `/` as `\/`, so if you 11917 use the secret key as `xxxxxx/xxxx` it will work fine. 11918 11919 ### Dreamhost ### 11920 11921 Dreamhost [DreamObjects](https://www.dreamhost.com/cloud/storage/) is 11922 an object storage system based on CEPH. 11923 11924 To use rclone with Dreamhost, configure as above but leave the region blank 11925 and set the endpoint. You should end up with something like this in 11926 your config: 11927 11928 ``` 11929 [dreamobjects] 11930 type = s3 11931 provider = DreamHost 11932 env_auth = false 11933 access_key_id = your_access_key 11934 secret_access_key = your_secret_key 11935 region = 11936 endpoint = objects-us-west-1.dream.io 11937 location_constraint = 11938 acl = private 11939 server_side_encryption = 11940 storage_class = 11941 ``` 11942 11943 ### DigitalOcean Spaces ### 11944 11945 [Spaces](https://www.digitalocean.com/products/object-storage/) is an [S3-interoperable](https://developers.digitalocean.com/documentation/spaces/) object storage service from cloud provider DigitalOcean. 11946 11947 To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "[Applications & API](https://cloud.digitalocean.com/settings/api/tokens)" page of the DigitalOcean control panel. They will be needed when prompted by `rclone config` for your `access_key_id` and `secret_access_key`. 11948 11949 When prompted for a `region` or `location_constraint`, press enter to use the default value. The region must be included in the `endpoint` setting (e.g. `nyc3.digitaloceanspaces.com`). The default values can be used for other settings. 11950 11951 Going through the whole process of creating a new remote by running `rclone config`, each prompt should be answered as shown below: 11952 11953 ``` 11954 Storage> s3 11955 env_auth> 1 11956 access_key_id> YOUR_ACCESS_KEY 11957 secret_access_key> YOUR_SECRET_KEY 11958 region> 11959 endpoint> nyc3.digitaloceanspaces.com 11960 location_constraint> 11961 acl> 11962 storage_class> 11963 ``` 11964 11965 The resulting configuration file should look like: 11966 11967 ``` 11968 [spaces] 11969 type = s3 11970 provider = DigitalOcean 11971 env_auth = false 11972 access_key_id = YOUR_ACCESS_KEY 11973 secret_access_key = YOUR_SECRET_KEY 11974 region = 11975 endpoint = nyc3.digitaloceanspaces.com 11976 location_constraint = 11977 acl = 11978 server_side_encryption = 11979 storage_class = 11980 ``` 11981 11982 Once configured, you can create a new Space and begin copying files. For example: 11983 11984 ``` 11985 rclone mkdir spaces:my-new-space 11986 rclone copy /path/to/files spaces:my-new-space 11987 ``` 11988 11989 ### IBM COS (S3) ### 11990 11991 Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage) 11992 11993 To configure access to IBM COS S3, follow the steps below: 11994 11995 1. Run rclone config and select n for a new remote. 11996 ``` 11997 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults 11998 No remotes found - make a new one 11999 n) New remote 12000 s) Set configuration password 12001 q) Quit config 12002 n/s/q> n 12003 ``` 12004 12005 2. Enter the name for the configuration 12006 ``` 12007 name> <YOUR NAME> 12008 ``` 12009 12010 3. Select "s3" storage. 12011 ``` 12012 Choose a number from below, or type in your own value 12013 1 / Alias for an existing remote 12014 \ "alias" 12015 2 / Amazon Drive 12016 \ "amazon cloud drive" 12017 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, Minio, IBM COS) 12018 \ "s3" 12019 4 / Backblaze B2 12020 \ "b2" 12021 [snip] 12022 23 / http Connection 12023 \ "http" 12024 Storage> 3 12025 ``` 12026 12027 4. Select IBM COS as the S3 Storage Provider. 12028 ``` 12029 Choose the S3 provider. 12030 Choose a number from below, or type in your own value 12031 1 / Choose this option to configure Storage to AWS S3 12032 \ "AWS" 12033 2 / Choose this option to configure Storage to Ceph Systems 12034 \ "Ceph" 12035 3 / Choose this option to configure Storage to Dreamhost 12036 \ "Dreamhost" 12037 4 / Choose this option to the configure Storage to IBM COS S3 12038 \ "IBMCOS" 12039 5 / Choose this option to the configure Storage to Minio 12040 \ "Minio" 12041 Provider>4 12042 ``` 12043 12044 5. Enter the Access Key and Secret. 12045 ``` 12046 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 12047 access_key_id> <> 12048 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 12049 secret_access_key> <> 12050 ``` 12051 12052 6. Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an enpoint address. 12053 ``` 12054 Endpoint for IBM COS S3 API. 12055 Specify if using an IBM COS On Premise. 12056 Choose a number from below, or type in your own value 12057 1 / US Cross Region Endpoint 12058 \ "s3-api.us-geo.objectstorage.softlayer.net" 12059 2 / US Cross Region Dallas Endpoint 12060 \ "s3-api.dal.us-geo.objectstorage.softlayer.net" 12061 3 / US Cross Region Washington DC Endpoint 12062 \ "s3-api.wdc-us-geo.objectstorage.softlayer.net" 12063 4 / US Cross Region San Jose Endpoint 12064 \ "s3-api.sjc-us-geo.objectstorage.softlayer.net" 12065 5 / US Cross Region Private Endpoint 12066 \ "s3-api.us-geo.objectstorage.service.networklayer.com" 12067 6 / US Cross Region Dallas Private Endpoint 12068 \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com" 12069 7 / US Cross Region Washington DC Private Endpoint 12070 \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com" 12071 8 / US Cross Region San Jose Private Endpoint 12072 \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com" 12073 9 / US Region East Endpoint 12074 \ "s3.us-east.objectstorage.softlayer.net" 12075 10 / US Region East Private Endpoint 12076 \ "s3.us-east.objectstorage.service.networklayer.com" 12077 11 / US Region South Endpoint 12078 [snip] 12079 34 / Toronto Single Site Private Endpoint 12080 \ "s3.tor01.objectstorage.service.networklayer.com" 12081 endpoint>1 12082 ``` 12083 12084 12085 7. Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter 12086 ``` 12087 1 / US Cross Region Standard 12088 \ "us-standard" 12089 2 / US Cross Region Vault 12090 \ "us-vault" 12091 3 / US Cross Region Cold 12092 \ "us-cold" 12093 4 / US Cross Region Flex 12094 \ "us-flex" 12095 5 / US East Region Standard 12096 \ "us-east-standard" 12097 6 / US East Region Vault 12098 \ "us-east-vault" 12099 7 / US East Region Cold 12100 \ "us-east-cold" 12101 8 / US East Region Flex 12102 \ "us-east-flex" 12103 9 / US South Region Standard 12104 \ "us-south-standard" 12105 10 / US South Region Vault 12106 \ "us-south-vault" 12107 [snip] 12108 32 / Toronto Flex 12109 \ "tor01-flex" 12110 location_constraint>1 12111 ``` 12112 12113 9. Specify a canned ACL. IBM Cloud (Strorage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs. 12114 ``` 12115 Canned ACL used when creating buckets and/or storing objects in S3. 12116 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 12117 Choose a number from below, or type in your own value 12118 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS 12119 \ "private" 12120 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS 12121 \ "public-read" 12122 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS 12123 \ "public-read-write" 12124 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS 12125 \ "authenticated-read" 12126 acl> 1 12127 ``` 12128 12129 12130 12. Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this 12131 ``` 12132 [xxx] 12133 type = s3 12134 Provider = IBMCOS 12135 access_key_id = xxx 12136 secret_access_key = yyy 12137 endpoint = s3-api.us-geo.objectstorage.softlayer.net 12138 location_constraint = us-standard 12139 acl = private 12140 ``` 12141 12142 13. Execute rclone commands 12143 ``` 12144 1) Create a bucket. 12145 rclone mkdir IBM-COS-XREGION:newbucket 12146 2) List available buckets. 12147 rclone lsd IBM-COS-XREGION: 12148 -1 2017-11-08 21:16:22 -1 test 12149 -1 2018-02-14 20:16:39 -1 newbucket 12150 3) List contents of a bucket. 12151 rclone ls IBM-COS-XREGION:newbucket 12152 18685952 test.exe 12153 4) Copy a file from local to remote. 12154 rclone copy /Users/file.txt IBM-COS-XREGION:newbucket 12155 5) Copy a file from remote to local. 12156 rclone copy IBM-COS-XREGION:newbucket/file.txt . 12157 6) Delete a file on remote. 12158 rclone delete IBM-COS-XREGION:newbucket/file.txt 12159 ``` 12160 12161 ### Minio ### 12162 12163 [Minio](https://minio.io/) is an object storage server built for cloud application developers and devops. 12164 12165 It is very easy to install and provides an S3 compatible server which can be used by rclone. 12166 12167 To use it, install Minio following the instructions [here](https://docs.minio.io/docs/minio-quickstart-guide). 12168 12169 When it configures itself Minio will print something like this 12170 12171 ``` 12172 Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000 12173 AccessKey: USWUXHGYZQYFYFFIT3RE 12174 SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 12175 Region: us-east-1 12176 SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis 12177 12178 Browser Access: 12179 http://192.168.1.106:9000 http://172.23.0.1:9000 12180 12181 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide 12182 $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 12183 12184 Object API (Amazon S3 compatible): 12185 Go: https://docs.minio.io/docs/golang-client-quickstart-guide 12186 Java: https://docs.minio.io/docs/java-client-quickstart-guide 12187 Python: https://docs.minio.io/docs/python-client-quickstart-guide 12188 JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide 12189 .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide 12190 12191 Drive Capacity: 26 GiB Free, 165 GiB Total 12192 ``` 12193 12194 These details need to go into `rclone config` like this. Note that it 12195 is important to put the region in as stated above. 12196 12197 ``` 12198 env_auth> 1 12199 access_key_id> USWUXHGYZQYFYFFIT3RE 12200 secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 12201 region> us-east-1 12202 endpoint> http://192.168.1.106:9000 12203 location_constraint> 12204 server_side_encryption> 12205 ``` 12206 12207 Which makes the config file look like this 12208 12209 ``` 12210 [minio] 12211 type = s3 12212 provider = Minio 12213 env_auth = false 12214 access_key_id = USWUXHGYZQYFYFFIT3RE 12215 secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 12216 region = us-east-1 12217 endpoint = http://192.168.1.106:9000 12218 location_constraint = 12219 server_side_encryption = 12220 ``` 12221 12222 So once set up, for example to copy files into a bucket 12223 12224 ``` 12225 rclone copy /path/to/files minio:bucket 12226 ``` 12227 12228 ### Scaleway {#scaleway} 12229 12230 [Scaleway](https://www.scaleway.com/object-storage/) The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos. 12231 Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool. 12232 12233 Scaleway provides an S3 interface which can be configured for use with rclone like this: 12234 12235 ``` 12236 [scaleway] 12237 type = s3 12238 env_auth = false 12239 endpoint = s3.nl-ams.scw.cloud 12240 access_key_id = SCWXXXXXXXXXXXXXX 12241 secret_access_key = 1111111-2222-3333-44444-55555555555555 12242 region = nl-ams 12243 location_constraint = 12244 acl = private 12245 force_path_style = false 12246 server_side_encryption = 12247 storage_class = 12248 ``` 12249 12250 ### Wasabi ### 12251 12252 [Wasabi](https://wasabi.com) is a cloud-based object storage service for a 12253 broad range of applications and use cases. Wasabi is designed for 12254 individuals and organizations that require a high-performance, 12255 reliable, and secure data storage infrastructure at minimal cost. 12256 12257 Wasabi provides an S3 interface which can be configured for use with 12258 rclone like this. 12259 12260 ``` 12261 No remotes found - make a new one 12262 n) New remote 12263 s) Set configuration password 12264 n/s> n 12265 name> wasabi 12266 Type of storage to configure. 12267 Choose a number from below, or type in your own value 12268 [snip] 12269 XX / Amazon S3 (also Dreamhost, Ceph, Minio) 12270 \ "s3" 12271 [snip] 12272 Storage> s3 12273 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 12274 Choose a number from below, or type in your own value 12275 1 / Enter AWS credentials in the next step 12276 \ "false" 12277 2 / Get AWS credentials from the environment (env vars or IAM) 12278 \ "true" 12279 env_auth> 1 12280 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 12281 access_key_id> YOURACCESSKEY 12282 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 12283 secret_access_key> YOURSECRETACCESSKEY 12284 Region to connect to. 12285 Choose a number from below, or type in your own value 12286 / The default endpoint - a good choice if you are unsure. 12287 1 | US Region, Northern Virginia or Pacific Northwest. 12288 | Leave location constraint empty. 12289 \ "us-east-1" 12290 [snip] 12291 region> us-east-1 12292 Endpoint for S3 API. 12293 Leave blank if using AWS to use the default endpoint for the region. 12294 Specify if using an S3 clone such as Ceph. 12295 endpoint> s3.wasabisys.com 12296 Location constraint - must be set to match the Region. Used when creating buckets only. 12297 Choose a number from below, or type in your own value 12298 1 / Empty for US Region, Northern Virginia or Pacific Northwest. 12299 \ "" 12300 [snip] 12301 location_constraint> 12302 Canned ACL used when creating buckets and/or storing objects in S3. 12303 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 12304 Choose a number from below, or type in your own value 12305 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 12306 \ "private" 12307 [snip] 12308 acl> 12309 The server-side encryption algorithm used when storing this object in S3. 12310 Choose a number from below, or type in your own value 12311 1 / None 12312 \ "" 12313 2 / AES256 12314 \ "AES256" 12315 server_side_encryption> 12316 The storage class to use when storing objects in S3. 12317 Choose a number from below, or type in your own value 12318 1 / Default 12319 \ "" 12320 2 / Standard storage class 12321 \ "STANDARD" 12322 3 / Reduced redundancy storage class 12323 \ "REDUCED_REDUNDANCY" 12324 4 / Standard Infrequent Access storage class 12325 \ "STANDARD_IA" 12326 storage_class> 12327 Remote config 12328 -------------------- 12329 [wasabi] 12330 env_auth = false 12331 access_key_id = YOURACCESSKEY 12332 secret_access_key = YOURSECRETACCESSKEY 12333 region = us-east-1 12334 endpoint = s3.wasabisys.com 12335 location_constraint = 12336 acl = 12337 server_side_encryption = 12338 storage_class = 12339 -------------------- 12340 y) Yes this is OK 12341 e) Edit this remote 12342 d) Delete this remote 12343 y/e/d> y 12344 ``` 12345 12346 This will leave the config file looking like this. 12347 12348 ``` 12349 [wasabi] 12350 type = s3 12351 provider = Wasabi 12352 env_auth = false 12353 access_key_id = YOURACCESSKEY 12354 secret_access_key = YOURSECRETACCESSKEY 12355 region = 12356 endpoint = s3.wasabisys.com 12357 location_constraint = 12358 acl = 12359 server_side_encryption = 12360 storage_class = 12361 ``` 12362 12363 ### Alibaba OSS {#alibaba-oss} 12364 12365 Here is an example of making an [Alibaba Cloud (Aliyun) OSS](https://www.alibabacloud.com/product/oss/) 12366 configuration. First run: 12367 12368 rclone config 12369 12370 This will guide you through an interactive setup process. 12371 12372 ``` 12373 No remotes found - make a new one 12374 n) New remote 12375 s) Set configuration password 12376 q) Quit config 12377 n/s/q> n 12378 name> oss 12379 Type of storage to configure. 12380 Enter a string value. Press Enter for the default (""). 12381 Choose a number from below, or type in your own value 12382 [snip] 12383 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) 12384 \ "s3" 12385 [snip] 12386 Storage> s3 12387 Choose your S3 provider. 12388 Enter a string value. Press Enter for the default (""). 12389 Choose a number from below, or type in your own value 12390 1 / Amazon Web Services (AWS) S3 12391 \ "AWS" 12392 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun 12393 \ "Alibaba" 12394 3 / Ceph Object Storage 12395 \ "Ceph" 12396 [snip] 12397 provider> Alibaba 12398 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 12399 Only applies if access_key_id and secret_access_key is blank. 12400 Enter a boolean value (true or false). Press Enter for the default ("false"). 12401 Choose a number from below, or type in your own value 12402 1 / Enter AWS credentials in the next step 12403 \ "false" 12404 2 / Get AWS credentials from the environment (env vars or IAM) 12405 \ "true" 12406 env_auth> 1 12407 AWS Access Key ID. 12408 Leave blank for anonymous access or runtime credentials. 12409 Enter a string value. Press Enter for the default (""). 12410 access_key_id> accesskeyid 12411 AWS Secret Access Key (password) 12412 Leave blank for anonymous access or runtime credentials. 12413 Enter a string value. Press Enter for the default (""). 12414 secret_access_key> secretaccesskey 12415 Endpoint for OSS API. 12416 Enter a string value. Press Enter for the default (""). 12417 Choose a number from below, or type in your own value 12418 1 / East China 1 (Hangzhou) 12419 \ "oss-cn-hangzhou.aliyuncs.com" 12420 2 / East China 2 (Shanghai) 12421 \ "oss-cn-shanghai.aliyuncs.com" 12422 3 / North China 1 (Qingdao) 12423 \ "oss-cn-qingdao.aliyuncs.com" 12424 [snip] 12425 endpoint> 1 12426 Canned ACL used when creating buckets and storing or copying objects. 12427 12428 Note that this ACL is applied when server side copying objects as S3 12429 doesn't copy the ACL from the source but rather writes a fresh one. 12430 Enter a string value. Press Enter for the default (""). 12431 Choose a number from below, or type in your own value 12432 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 12433 \ "private" 12434 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 12435 \ "public-read" 12436 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 12437 [snip] 12438 acl> 1 12439 The storage class to use when storing new objects in OSS. 12440 Enter a string value. Press Enter for the default (""). 12441 Choose a number from below, or type in your own value 12442 1 / Default 12443 \ "" 12444 2 / Standard storage class 12445 \ "STANDARD" 12446 3 / Archive storage mode. 12447 \ "GLACIER" 12448 4 / Infrequent access storage mode. 12449 \ "STANDARD_IA" 12450 storage_class> 1 12451 Edit advanced config? (y/n) 12452 y) Yes 12453 n) No 12454 y/n> n 12455 Remote config 12456 -------------------- 12457 [oss] 12458 type = s3 12459 provider = Alibaba 12460 env_auth = false 12461 access_key_id = accesskeyid 12462 secret_access_key = secretaccesskey 12463 endpoint = oss-cn-hangzhou.aliyuncs.com 12464 acl = private 12465 storage_class = Standard 12466 -------------------- 12467 y) Yes this is OK 12468 e) Edit this remote 12469 d) Delete this remote 12470 y/e/d> y 12471 ``` 12472 12473 ### Netease NOS ### 12474 12475 For Netease NOS configure as per the configurator `rclone config` 12476 setting the provider `Netease`. This will automatically set 12477 `force_path_style = false` which is necessary for it to run properly. 12478 12479 Backblaze B2 12480 ---------------------------------------- 12481 12482 B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/). 12483 12484 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 12485 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 12486 12487 Here is an example of making a b2 configuration. First run 12488 12489 rclone config 12490 12491 This will guide you through an interactive setup process. To authenticate 12492 you will either need your Account ID (a short hex number) and Master 12493 Application Key (a long hex number) OR an Application Key, which is the 12494 recommended method. See below for further details on generating and using 12495 an Application Key. 12496 12497 ``` 12498 No remotes found - make a new one 12499 n) New remote 12500 q) Quit config 12501 n/q> n 12502 name> remote 12503 Type of storage to configure. 12504 Choose a number from below, or type in your own value 12505 [snip] 12506 XX / Backblaze B2 12507 \ "b2" 12508 [snip] 12509 Storage> b2 12510 Account ID or Application Key ID 12511 account> 123456789abc 12512 Application Key 12513 key> 0123456789abcdef0123456789abcdef0123456789 12514 Endpoint for the service - leave blank normally. 12515 endpoint> 12516 Remote config 12517 -------------------- 12518 [remote] 12519 account = 123456789abc 12520 key = 0123456789abcdef0123456789abcdef0123456789 12521 endpoint = 12522 -------------------- 12523 y) Yes this is OK 12524 e) Edit this remote 12525 d) Delete this remote 12526 y/e/d> y 12527 ``` 12528 12529 This remote is called `remote` and can now be used like this 12530 12531 See all buckets 12532 12533 rclone lsd remote: 12534 12535 Create a new bucket 12536 12537 rclone mkdir remote:bucket 12538 12539 List the contents of a bucket 12540 12541 rclone ls remote:bucket 12542 12543 Sync `/home/local/directory` to the remote bucket, deleting any 12544 excess files in the bucket. 12545 12546 rclone sync /home/local/directory remote:bucket 12547 12548 ### Application Keys ### 12549 12550 B2 supports multiple [Application Keys for different access permission 12551 to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html). 12552 12553 You can use these with rclone too; you will need to use rclone version 1.43 12554 or later. 12555 12556 Follow Backblaze's docs to create an Application Key with the required 12557 permission and add the `applicationKeyId` as the `account` and the 12558 `Application Key` itself as the `key`. 12559 12560 Note that you must put the _applicationKeyId_ as the `account` – you 12561 can't use the master Account ID. If you try then B2 will return 401 12562 errors. 12563 12564 ### --fast-list ### 12565 12566 This remote supports `--fast-list` which allows you to use fewer 12567 transactions in exchange for more memory. See the [rclone 12568 docs](https://rclone.org/docs/#fast-list) for more details. 12569 12570 ### Modified time ### 12571 12572 The modified time is stored as metadata on the object as 12573 `X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01 12574 in the Backblaze standard. Other tools should be able to use this as 12575 a modified time. 12576 12577 Modified times are used in syncing and are fully supported. Note that 12578 if a modification time needs to be updated on an object then it will 12579 create a new version of the object. 12580 12581 #### Restricted filename characters 12582 12583 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 12584 the following characters are also replaced: 12585 12586 | Character | Value | Replacement | 12587 | --------- |:-----:|:-----------:| 12588 | \ | 0x5C | \ | 12589 12590 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 12591 as they can't be used in JSON strings. 12592 12593 ### SHA1 checksums ### 12594 12595 The SHA1 checksums of the files are checked on upload and download and 12596 will be used in the syncing process. 12597 12598 Large files (bigger than the limit in `--b2-upload-cutoff`) which are 12599 uploaded in chunks will store their SHA1 on the object as 12600 `X-Bz-Info-large_file_sha1` as recommended by Backblaze. 12601 12602 For a large file to be uploaded with an SHA1 checksum, the source 12603 needs to support SHA1 checksums. The local disk supports SHA1 12604 checksums so large file transfers from local disk will have an SHA1. 12605 See [the overview](https://rclone.org/overview/#features) for exactly which remotes 12606 support SHA1. 12607 12608 Sources which don't support SHA1, in particular `crypt` will upload 12609 large files without SHA1 checksums. This may be fixed in the future 12610 (see [#1767](https://github.com/rclone/rclone/issues/1767)). 12611 12612 Files sizes below `--b2-upload-cutoff` will always have an SHA1 12613 regardless of the source. 12614 12615 ### Transfers ### 12616 12617 Backblaze recommends that you do lots of transfers simultaneously for 12618 maximum speed. In tests from my SSD equipped laptop the optimum 12619 setting is about `--transfers 32` though higher numbers may be used 12620 for a slight speed improvement. The optimum number for you may vary 12621 depending on your hardware, how big the files are, how much you want 12622 to load your computer, etc. The default of `--transfers 4` is 12623 definitely too low for Backblaze B2 though. 12624 12625 Note that uploading big files (bigger than 200 MB by default) will use 12626 a 96 MB RAM buffer by default. There can be at most `--transfers` of 12627 these in use at any moment, so this sets the upper limit on the memory 12628 used. 12629 12630 ### Versions ### 12631 12632 When rclone uploads a new version of a file it creates a [new version 12633 of it](https://www.backblaze.com/b2/docs/file_versions.html). 12634 Likewise when you delete a file, the old version will be marked hidden 12635 and still be available. Conversely, you may opt in to a "hard delete" 12636 of files with the `--b2-hard-delete` flag which would permanently remove 12637 the file instead of hiding it. 12638 12639 Old versions of files, where available, are visible using the 12640 `--b2-versions` flag. 12641 12642 **NB** Note that `--b2-versions` does not work with crypt at the 12643 moment [#1627](https://github.com/rclone/rclone/issues/1627). Using 12644 [--backup-dir](https://rclone.org/docs/#backup-dir-dir) with rclone is the recommended 12645 way of working around this. 12646 12647 If you wish to remove all the old versions then you can use the 12648 `rclone cleanup remote:bucket` command which will delete all the old 12649 versions of files, leaving the current ones intact. You can also 12650 supply a path and only old versions under that path will be deleted, 12651 eg `rclone cleanup remote:bucket/path/to/stuff`. 12652 12653 Note that `cleanup` will remove partially uploaded files from the bucket 12654 if they are more than a day old. 12655 12656 When you `purge` a bucket, the current and the old versions will be 12657 deleted then the bucket will be deleted. 12658 12659 However `delete` will cause the current versions of the files to 12660 become hidden old versions. 12661 12662 Here is a session showing the listing and retrieval of an old 12663 version followed by a `cleanup` of the old versions. 12664 12665 Show current version and all the versions with `--b2-versions` flag. 12666 12667 ``` 12668 $ rclone -q ls b2:cleanup-test 12669 9 one.txt 12670 12671 $ rclone -q --b2-versions ls b2:cleanup-test 12672 9 one.txt 12673 8 one-v2016-07-04-141032-000.txt 12674 16 one-v2016-07-04-141003-000.txt 12675 15 one-v2016-07-02-155621-000.txt 12676 ``` 12677 12678 Retrieve an old version 12679 12680 ``` 12681 $ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp 12682 12683 $ ls -l /tmp/one-v2016-07-04-141003-000.txt 12684 -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt 12685 ``` 12686 12687 Clean up all the old versions and show that they've gone. 12688 12689 ``` 12690 $ rclone -q cleanup b2:cleanup-test 12691 12692 $ rclone -q ls b2:cleanup-test 12693 9 one.txt 12694 12695 $ rclone -q --b2-versions ls b2:cleanup-test 12696 9 one.txt 12697 ``` 12698 12699 ### Data usage ### 12700 12701 It is useful to know how many requests are sent to the server in different scenarios. 12702 12703 All copy commands send the following 4 requests: 12704 12705 ``` 12706 /b2api/v1/b2_authorize_account 12707 /b2api/v1/b2_create_bucket 12708 /b2api/v1/b2_list_buckets 12709 /b2api/v1/b2_list_file_names 12710 ``` 12711 12712 The `b2_list_file_names` request will be sent once for every 1k files 12713 in the remote path, providing the checksum and modification time of 12714 the listed files. As of version 1.33 issue 12715 [#818](https://github.com/rclone/rclone/issues/818) causes extra requests 12716 to be sent when using B2 with Crypt. When a copy operation does not 12717 require any files to be uploaded, no more requests will be sent. 12718 12719 Uploading files that do not require chunking, will send 2 requests per 12720 file upload: 12721 12722 ``` 12723 /b2api/v1/b2_get_upload_url 12724 /b2api/v1/b2_upload_file/ 12725 ``` 12726 12727 Uploading files requiring chunking, will send 2 requests (one each to 12728 start and finish the upload) and another 2 requests for each chunk: 12729 12730 ``` 12731 /b2api/v1/b2_start_large_file 12732 /b2api/v1/b2_get_upload_part_url 12733 /b2api/v1/b2_upload_part/ 12734 /b2api/v1/b2_finish_large_file 12735 ``` 12736 12737 #### Versions #### 12738 12739 Versions can be viewed with the `--b2-versions` flag. When it is set 12740 rclone will show and act on older versions of files. For example 12741 12742 Listing without `--b2-versions` 12743 12744 ``` 12745 $ rclone -q ls b2:cleanup-test 12746 9 one.txt 12747 ``` 12748 12749 And with 12750 12751 ``` 12752 $ rclone -q --b2-versions ls b2:cleanup-test 12753 9 one.txt 12754 8 one-v2016-07-04-141032-000.txt 12755 16 one-v2016-07-04-141003-000.txt 12756 15 one-v2016-07-02-155621-000.txt 12757 ``` 12758 12759 Showing that the current version is unchanged but older versions can 12760 be seen. These have the UTC date that they were uploaded to the 12761 server to the nearest millisecond appended to them. 12762 12763 Note that when using `--b2-versions` no file write operations are 12764 permitted, so you can't upload files or delete them. 12765 12766 ### B2 and rclone link ### 12767 12768 Rclone supports generating file share links for private B2 buckets. 12769 They can either be for a file for example: 12770 12771 ``` 12772 ./rclone link B2:bucket/path/to/file.txt 12773 https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx 12774 12775 ``` 12776 12777 or if run on a directory you will get: 12778 12779 ``` 12780 ./rclone link B2:bucket/path 12781 https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx 12782 ``` 12783 12784 you can then use the authorization token (the part of the url from the 12785 `?Authorization=` on) on any file path under that directory. For example: 12786 12787 ``` 12788 https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx 12789 https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx 12790 https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx 12791 12792 ``` 12793 12794 12795 ### Standard Options 12796 12797 Here are the standard options specific to b2 (Backblaze B2). 12798 12799 #### --b2-account 12800 12801 Account ID or Application Key ID 12802 12803 - Config: account 12804 - Env Var: RCLONE_B2_ACCOUNT 12805 - Type: string 12806 - Default: "" 12807 12808 #### --b2-key 12809 12810 Application Key 12811 12812 - Config: key 12813 - Env Var: RCLONE_B2_KEY 12814 - Type: string 12815 - Default: "" 12816 12817 #### --b2-hard-delete 12818 12819 Permanently delete files on remote removal, otherwise hide files. 12820 12821 - Config: hard_delete 12822 - Env Var: RCLONE_B2_HARD_DELETE 12823 - Type: bool 12824 - Default: false 12825 12826 ### Advanced Options 12827 12828 Here are the advanced options specific to b2 (Backblaze B2). 12829 12830 #### --b2-endpoint 12831 12832 Endpoint for the service. 12833 Leave blank normally. 12834 12835 - Config: endpoint 12836 - Env Var: RCLONE_B2_ENDPOINT 12837 - Type: string 12838 - Default: "" 12839 12840 #### --b2-test-mode 12841 12842 A flag string for X-Bz-Test-Mode header for debugging. 12843 12844 This is for debugging purposes only. Setting it to one of the strings 12845 below will cause b2 to return specific errors: 12846 12847 * "fail_some_uploads" 12848 * "expire_some_account_authorization_tokens" 12849 * "force_cap_exceeded" 12850 12851 These will be set in the "X-Bz-Test-Mode" header which is documented 12852 in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html). 12853 12854 - Config: test_mode 12855 - Env Var: RCLONE_B2_TEST_MODE 12856 - Type: string 12857 - Default: "" 12858 12859 #### --b2-versions 12860 12861 Include old versions in directory listings. 12862 Note that when using this no file write operations are permitted, 12863 so you can't upload files or delete them. 12864 12865 - Config: versions 12866 - Env Var: RCLONE_B2_VERSIONS 12867 - Type: bool 12868 - Default: false 12869 12870 #### --b2-upload-cutoff 12871 12872 Cutoff for switching to chunked upload. 12873 12874 Files above this size will be uploaded in chunks of "--b2-chunk-size". 12875 12876 This value should be set no larger than 4.657GiB (== 5GB). 12877 12878 - Config: upload_cutoff 12879 - Env Var: RCLONE_B2_UPLOAD_CUTOFF 12880 - Type: SizeSuffix 12881 - Default: 200M 12882 12883 #### --b2-chunk-size 12884 12885 Upload chunk size. Must fit in memory. 12886 12887 When uploading large files, chunk the file into this size. Note that 12888 these chunks are buffered in memory and there might a maximum of 12889 "--transfers" chunks in progress at once. 5,000,000 Bytes is the 12890 minimum size. 12891 12892 - Config: chunk_size 12893 - Env Var: RCLONE_B2_CHUNK_SIZE 12894 - Type: SizeSuffix 12895 - Default: 96M 12896 12897 #### --b2-disable-checksum 12898 12899 Disable checksums for large (> upload cutoff) files 12900 12901 Normally rclone will calculate the SHA1 checksum of the input before 12902 uploading it so it can add it to metadata on the object. This is great 12903 for data integrity checking but can cause long delays for large files 12904 to start uploading. 12905 12906 - Config: disable_checksum 12907 - Env Var: RCLONE_B2_DISABLE_CHECKSUM 12908 - Type: bool 12909 - Default: false 12910 12911 #### --b2-download-url 12912 12913 Custom endpoint for downloads. 12914 12915 This is usually set to a Cloudflare CDN URL as Backblaze offers 12916 free egress for data downloaded through the Cloudflare network. 12917 This is probably only useful for a public bucket. 12918 Leave blank if you want to use the endpoint provided by Backblaze. 12919 12920 - Config: download_url 12921 - Env Var: RCLONE_B2_DOWNLOAD_URL 12922 - Type: string 12923 - Default: "" 12924 12925 #### --b2-download-auth-duration 12926 12927 Time before the authorization token will expire in s or suffix ms|s|m|h|d. 12928 12929 The duration before the download authorization token will expire. 12930 The minimum value is 1 second. The maximum value is one week. 12931 12932 - Config: download_auth_duration 12933 - Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION 12934 - Type: Duration 12935 - Default: 1w 12936 12937 #### --b2-encoding 12938 12939 This sets the encoding for the backend. 12940 12941 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 12942 12943 - Config: encoding 12944 - Env Var: RCLONE_B2_ENCODING 12945 - Type: MultiEncoder 12946 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 12947 12948 12949 12950 Box 12951 ----------------------------------------- 12952 12953 Paths are specified as `remote:path` 12954 12955 Paths may be as deep as required, eg `remote:directory/subdirectory`. 12956 12957 The initial setup for Box involves getting a token from Box which you 12958 can do either in your browser, or with a config.json downloaded from Box 12959 to use JWT authentication. `rclone config` walks you through it. 12960 12961 Here is an example of how to make a remote called `remote`. First run: 12962 12963 rclone config 12964 12965 This will guide you through an interactive setup process: 12966 12967 ``` 12968 No remotes found - make a new one 12969 n) New remote 12970 s) Set configuration password 12971 q) Quit config 12972 n/s/q> n 12973 name> remote 12974 Type of storage to configure. 12975 Choose a number from below, or type in your own value 12976 [snip] 12977 XX / Box 12978 \ "box" 12979 [snip] 12980 Storage> box 12981 Box App Client Id - leave blank normally. 12982 client_id> 12983 Box App Client Secret - leave blank normally. 12984 client_secret> 12985 Box App config.json location 12986 Leave blank normally. 12987 Enter a string value. Press Enter for the default (""). 12988 config_json> 12989 'enterprise' or 'user' depending on the type of token being requested. 12990 Enter a string value. Press Enter for the default ("user"). 12991 box_sub_type> 12992 Remote config 12993 Use auto config? 12994 * Say Y if not sure 12995 * Say N if you are working on a remote or headless machine 12996 y) Yes 12997 n) No 12998 y/n> y 12999 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 13000 Log in and authorize rclone for access 13001 Waiting for code... 13002 Got code 13003 -------------------- 13004 [remote] 13005 client_id = 13006 client_secret = 13007 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} 13008 -------------------- 13009 y) Yes this is OK 13010 e) Edit this remote 13011 d) Delete this remote 13012 y/e/d> y 13013 ``` 13014 13015 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 13016 machine with no Internet browser available. 13017 13018 Note that rclone runs a webserver on your local machine to collect the 13019 token as returned from Box. This only runs from the moment it opens 13020 your browser to the moment you get back the verification code. This 13021 is on `http://127.0.0.1:53682/` and this it may require you to unblock 13022 it temporarily if you are running a host firewall. 13023 13024 Once configured you can then use `rclone` like this, 13025 13026 List directories in top level of your Box 13027 13028 rclone lsd remote: 13029 13030 List all the files in your Box 13031 13032 rclone ls remote: 13033 13034 To copy a local directory to an Box directory called backup 13035 13036 rclone copy /home/source remote:backup 13037 13038 ### Using rclone with an Enterprise account with SSO ### 13039 13040 If you have an "Enterprise" account type with Box with single sign on 13041 (SSO), you need to create a password to use Box with rclone. This can 13042 be done at your Enterprise Box account by going to Settings, "Account" 13043 Tab, and then set the password in the "Authentication" field. 13044 13045 Once you have done this, you can setup your Enterprise Box account 13046 using the same procedure detailed above in the, using the password you 13047 have just set. 13048 13049 ### Invalid refresh token ### 13050 13051 According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens): 13052 13053 > Each refresh_token is valid for one use in 60 days. 13054 13055 This means that if you 13056 13057 * Don't use the box remote for 60 days 13058 * Copy the config file with a box refresh token in and use it in two places 13059 * Get an error on a token refresh 13060 13061 then rclone will return an error which includes the text `Invalid 13062 refresh token`. 13063 13064 To fix this you will need to use oauth2 again to update the refresh 13065 token. You can use the methods in [the remote setup 13066 docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the 13067 config file method, you should not use that remote on the computer you 13068 did the authentication on. 13069 13070 Here is how to do it. 13071 13072 ``` 13073 $ rclone config 13074 Current remotes: 13075 13076 Name Type 13077 ==== ==== 13078 remote box 13079 13080 e) Edit existing remote 13081 n) New remote 13082 d) Delete remote 13083 r) Rename remote 13084 c) Copy remote 13085 s) Set configuration password 13086 q) Quit config 13087 e/n/d/r/c/s/q> e 13088 Choose a number from below, or type in an existing value 13089 1 > remote 13090 remote> remote 13091 -------------------- 13092 [remote] 13093 type = box 13094 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} 13095 -------------------- 13096 Edit remote 13097 Value "client_id" = "" 13098 Edit? (y/n)> 13099 y) Yes 13100 n) No 13101 y/n> n 13102 Value "client_secret" = "" 13103 Edit? (y/n)> 13104 y) Yes 13105 n) No 13106 y/n> n 13107 Remote config 13108 Already have a token - refresh? 13109 y) Yes 13110 n) No 13111 y/n> y 13112 Use auto config? 13113 * Say Y if not sure 13114 * Say N if you are working on a remote or headless machine 13115 y) Yes 13116 n) No 13117 y/n> y 13118 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 13119 Log in and authorize rclone for access 13120 Waiting for code... 13121 Got code 13122 -------------------- 13123 [remote] 13124 type = box 13125 token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} 13126 -------------------- 13127 y) Yes this is OK 13128 e) Edit this remote 13129 d) Delete this remote 13130 y/e/d> y 13131 ``` 13132 13133 ### Modified time and hashes ### 13134 13135 Box allows modification times to be set on objects accurate to 1 13136 second. These will be used to detect whether objects need syncing or 13137 not. 13138 13139 Box supports SHA1 type hashes, so you can use the `--checksum` 13140 flag. 13141 13142 #### Restricted filename characters 13143 13144 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 13145 the following characters are also replaced: 13146 13147 | Character | Value | Replacement | 13148 | --------- |:-----:|:-----------:| 13149 | \ | 0x5C | \ | 13150 13151 File names can also not end with the following characters. 13152 These only get replaced if they are the last character in the name: 13153 13154 | Character | Value | Replacement | 13155 | --------- |:-----:|:-----------:| 13156 | SP | 0x20 | ␠ | 13157 13158 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 13159 as they can't be used in JSON strings. 13160 13161 ### Transfers ### 13162 13163 For files above 50MB rclone will use a chunked transfer. Rclone will 13164 upload up to `--transfers` chunks at the same time (shared among all 13165 the multipart uploads). Chunks are buffered in memory and are 13166 normally 8MB so increasing `--transfers` will increase memory use. 13167 13168 ### Deleting files ### 13169 13170 Depending on the enterprise settings for your user, the item will 13171 either be actually deleted from Box or moved to the trash. 13172 13173 ### Root folder ID ### 13174 13175 You can set the `root_folder_id` for rclone. This is the directory 13176 (identified by its `Folder ID`) that rclone considers to be the root 13177 of your Box drive. 13178 13179 Normally you will leave this blank and rclone will determine the 13180 correct root to use itself. 13181 13182 However you can set this to restrict rclone to a specific folder 13183 hierarchy. 13184 13185 In order to do this you will have to find the `Folder ID` of the 13186 directory you wish rclone to display. This will be the last segment 13187 of the URL when you open the relevant folder in the Box web 13188 interface. 13189 13190 So if the folder you want rclone to use has a URL which looks like 13191 `https://app.box.com/folder/11xxxxxxxxx8` 13192 in the browser, then you use `11xxxxxxxxx8` as 13193 the `root_folder_id` in the config. 13194 13195 13196 ### Standard Options 13197 13198 Here are the standard options specific to box (Box). 13199 13200 #### --box-client-id 13201 13202 Box App Client Id. 13203 Leave blank normally. 13204 13205 - Config: client_id 13206 - Env Var: RCLONE_BOX_CLIENT_ID 13207 - Type: string 13208 - Default: "" 13209 13210 #### --box-client-secret 13211 13212 Box App Client Secret 13213 Leave blank normally. 13214 13215 - Config: client_secret 13216 - Env Var: RCLONE_BOX_CLIENT_SECRET 13217 - Type: string 13218 - Default: "" 13219 13220 #### --box-box-config-file 13221 13222 Box App config.json location 13223 Leave blank normally. 13224 13225 - Config: box_config_file 13226 - Env Var: RCLONE_BOX_BOX_CONFIG_FILE 13227 - Type: string 13228 - Default: "" 13229 13230 #### --box-box-sub-type 13231 13232 13233 13234 - Config: box_sub_type 13235 - Env Var: RCLONE_BOX_BOX_SUB_TYPE 13236 - Type: string 13237 - Default: "user" 13238 - Examples: 13239 - "user" 13240 - Rclone should act on behalf of a user 13241 - "enterprise" 13242 - Rclone should act on behalf of a service account 13243 13244 ### Advanced Options 13245 13246 Here are the advanced options specific to box (Box). 13247 13248 #### --box-root-folder-id 13249 13250 Fill in for rclone to use a non root folder as its starting point. 13251 13252 - Config: root_folder_id 13253 - Env Var: RCLONE_BOX_ROOT_FOLDER_ID 13254 - Type: string 13255 - Default: "0" 13256 13257 #### --box-upload-cutoff 13258 13259 Cutoff for switching to multipart upload (>= 50MB). 13260 13261 - Config: upload_cutoff 13262 - Env Var: RCLONE_BOX_UPLOAD_CUTOFF 13263 - Type: SizeSuffix 13264 - Default: 50M 13265 13266 #### --box-commit-retries 13267 13268 Max number of times to try committing a multipart file. 13269 13270 - Config: commit_retries 13271 - Env Var: RCLONE_BOX_COMMIT_RETRIES 13272 - Type: int 13273 - Default: 100 13274 13275 #### --box-encoding 13276 13277 This sets the encoding for the backend. 13278 13279 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 13280 13281 - Config: encoding 13282 - Env Var: RCLONE_BOX_ENCODING 13283 - Type: MultiEncoder 13284 - Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot 13285 13286 13287 13288 ### Limitations ### 13289 13290 Note that Box is case insensitive so you can't have a file called 13291 "Hello.doc" and one called "hello.doc". 13292 13293 Box file names can't have the `\` character in. rclone maps this to 13294 and from an identical looking unicode equivalent `\`. 13295 13296 Box only supports filenames up to 255 characters in length. 13297 13298 Cache (BETA) 13299 ----------------------------------------- 13300 13301 The `cache` remote wraps another existing remote and stores file structure 13302 and its data for long running tasks like `rclone mount`. 13303 13304 ## Status 13305 13306 The cache backend code is working but it currently doesn't 13307 have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed. 13308 13309 The cache backend is due to be phased out in favour of the VFS caching 13310 layer eventually which is more tightly integrated into rclone. 13311 13312 Until this happens we recommend only using the cache backend if you 13313 find you can't work without it. There are many docs online describing 13314 the use of the cache backend to minimize API hits and by-and-large 13315 these are out of date and the cache backend isn't needed in those 13316 scenarios any more. 13317 13318 ## Setup 13319 13320 To get started you just need to have an existing remote which can be configured 13321 with `cache`. 13322 13323 Here is an example of how to make a remote called `test-cache`. First run: 13324 13325 rclone config 13326 13327 This will guide you through an interactive setup process: 13328 13329 ``` 13330 No remotes found - make a new one 13331 n) New remote 13332 r) Rename remote 13333 c) Copy remote 13334 s) Set configuration password 13335 q) Quit config 13336 n/r/c/s/q> n 13337 name> test-cache 13338 Type of storage to configure. 13339 Choose a number from below, or type in your own value 13340 [snip] 13341 XX / Cache a remote 13342 \ "cache" 13343 [snip] 13344 Storage> cache 13345 Remote to cache. 13346 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13347 "myremote:bucket" or maybe "myremote:" (not recommended). 13348 remote> local:/test 13349 Optional: The URL of the Plex server 13350 plex_url> http://127.0.0.1:32400 13351 Optional: The username of the Plex user 13352 plex_username> dummyusername 13353 Optional: The password of the Plex user 13354 y) Yes type in my own password 13355 g) Generate random password 13356 n) No leave this optional password blank 13357 y/g/n> y 13358 Enter the password: 13359 password: 13360 Confirm the password: 13361 password: 13362 The size of a chunk. Lower value good for slow connections but can affect seamless reading. 13363 Default: 5M 13364 Choose a number from below, or type in your own value 13365 1 / 1MB 13366 \ "1m" 13367 2 / 5 MB 13368 \ "5M" 13369 3 / 10 MB 13370 \ "10M" 13371 chunk_size> 2 13372 How much time should object info (file size, file hashes etc) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. 13373 Accepted units are: "s", "m", "h". 13374 Default: 5m 13375 Choose a number from below, or type in your own value 13376 1 / 1 hour 13377 \ "1h" 13378 2 / 24 hours 13379 \ "24h" 13380 3 / 24 hours 13381 \ "48h" 13382 info_age> 2 13383 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. 13384 Default: 10G 13385 Choose a number from below, or type in your own value 13386 1 / 500 MB 13387 \ "500M" 13388 2 / 1 GB 13389 \ "1G" 13390 3 / 10 GB 13391 \ "10G" 13392 chunk_total_size> 3 13393 Remote config 13394 -------------------- 13395 [test-cache] 13396 remote = local:/test 13397 plex_url = http://127.0.0.1:32400 13398 plex_username = dummyusername 13399 plex_password = *** ENCRYPTED *** 13400 chunk_size = 5M 13401 info_age = 48h 13402 chunk_total_size = 10G 13403 ``` 13404 13405 You can then use it like this, 13406 13407 List directories in top level of your drive 13408 13409 rclone lsd test-cache: 13410 13411 List all the files in your drive 13412 13413 rclone ls test-cache: 13414 13415 To start a cached mount 13416 13417 rclone mount --allow-other test-cache: /var/tmp/test-cache 13418 13419 ### Write Features ### 13420 13421 ### Offline uploading ### 13422 13423 In an effort to make writing through cache more reliable, the backend 13424 now supports this feature which can be activated by specifying a 13425 `cache-tmp-upload-path`. 13426 13427 A files goes through these states when using this feature: 13428 13429 1. An upload is started (usually by copying a file on the cache remote) 13430 2. When the copy to the temporary location is complete the file is part 13431 of the cached remote and looks and behaves like any other file (reading included) 13432 3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move` 13433 is used to move the file to the cloud provider 13434 4. Reading the file still works during the upload but most modifications on it will be prohibited 13435 5. Once the move is complete the file is unlocked for modifications as it 13436 becomes as any other regular file 13437 6. If the file is being read through `cache` when it's actually 13438 deleted from the temporary path then `cache` will simply swap the source 13439 to the cloud provider without interrupting the reading (small blip can happen though) 13440 13441 Files are uploaded in sequence and only one file is uploaded at a time. 13442 Uploads will be stored in a queue and be processed based on the order they were added. 13443 The queue and the temporary storage is persistent across restarts but 13444 can be cleared on startup with the `--cache-db-purge` flag. 13445 13446 ### Write Support ### 13447 13448 Writes are supported through `cache`. 13449 One caveat is that a mounted cache remote does not add any retry or fallback 13450 mechanism to the upload operation. This will depend on the implementation 13451 of the wrapped remote. Consider using `Offline uploading` for reliable writes. 13452 13453 One special case is covered with `cache-writes` which will cache the file 13454 data at the same time as the upload when it is enabled making it available 13455 from the cache store immediately once the upload is finished. 13456 13457 ### Read Features ### 13458 13459 #### Multiple connections #### 13460 13461 To counter the high latency between a local PC where rclone is running 13462 and cloud providers, the cache remote can split multiple requests to the 13463 cloud provider for smaller file chunks and combines them together locally 13464 where they can be available almost immediately before the reader usually 13465 needs them. 13466 13467 This is similar to buffering when media files are played online. Rclone 13468 will stay around the current marker but always try its best to stay ahead 13469 and prepare the data before. 13470 13471 #### Plex Integration #### 13472 13473 There is a direct integration with Plex which allows cache to detect during reading 13474 if the file is in playback or not. This helps cache to adapt how it queries 13475 the cloud provider depending on what is needed for. 13476 13477 Scans will have a minimum amount of workers (1) while in a confirmed playback cache 13478 will deploy the configured number of workers. 13479 13480 This integration opens the doorway to additional performance improvements 13481 which will be explored in the near future. 13482 13483 **Note:** If Plex options are not configured, `cache` will function with its 13484 configured options without adapting any of its settings. 13485 13486 How to enable? Run `rclone config` and add all the Plex options (endpoint, username 13487 and password) in your remote and it will be automatically enabled. 13488 13489 Affected settings: 13490 - `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times 13491 13492 ##### Certificate Validation ##### 13493 13494 When the Plex server is configured to only accept secure connections, it is 13495 possible to use `.plex.direct` URLs to ensure certificate validation succeeds. 13496 These URLs are used by Plex internally to connect to the Plex server securely. 13497 13498 The format for these URLs is the following: 13499 13500 https://ip-with-dots-replaced.server-hash.plex.direct:32400/ 13501 13502 The `ip-with-dots-replaced` part can be any IPv4 address, where the dots 13503 have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`. 13504 13505 To get the `server-hash` part, the easiest way is to visit 13506 13507 https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token 13508 13509 This page will list all the available Plex servers for your account 13510 with at least one `.plex.direct` link for each. Copy one URL and replace 13511 the IP address with the desired address. This can be used as the 13512 `plex_url` value. 13513 13514 ### Known issues ### 13515 13516 #### Mount and --dir-cache-time #### 13517 13518 --dir-cache-time controls the first layer of directory caching which works at the mount layer. 13519 Being an independent caching mechanism from the `cache` backend, it will manage its own entries 13520 based on the configured time. 13521 13522 To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct 13523 one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are 13524 already configured in this way. 13525 13526 #### Windows support - Experimental #### 13527 13528 There are a couple of issues with Windows `mount` functionality that still require some investigations. 13529 It should be considered as experimental thus far as fixes come in for this OS. 13530 13531 Most of the issues seem to be related to the difference between filesystems 13532 on Linux flavors and Windows as cache is heavily dependent on them. 13533 13534 Any reports or feedback on how cache behaves on this OS is greatly appreciated. 13535 13536 - https://github.com/rclone/rclone/issues/1935 13537 - https://github.com/rclone/rclone/issues/1907 13538 - https://github.com/rclone/rclone/issues/1834 13539 13540 #### Risk of throttling #### 13541 13542 Future iterations of the cache backend will make use of the pooling functionality 13543 of the cloud provider to synchronize and at the same time make writing through it 13544 more tolerant to failures. 13545 13546 There are a couple of enhancements in track to add these but in the meantime 13547 there is a valid concern that the expiring cache listings can lead to cloud provider 13548 throttles or bans due to repeated queries on it for very large mounts. 13549 13550 Some recommendations: 13551 - don't use a very small interval for entry information (`--cache-info-age`) 13552 - while writes aren't yet optimised, you can still write through `cache` which gives you the advantage 13553 of adding the file in the cache at the same time if configured to do so. 13554 13555 Future enhancements: 13556 13557 - https://github.com/rclone/rclone/issues/1937 13558 - https://github.com/rclone/rclone/issues/1936 13559 13560 #### cache and crypt #### 13561 13562 One common scenario is to keep your data encrypted in the cloud provider 13563 using the `crypt` remote. `crypt` uses a similar technique to wrap around 13564 an existing remote and handles this translation in a seamless way. 13565 13566 There is an issue with wrapping the remotes in this order: 13567 **cloud remote** -> **crypt** -> **cache** 13568 13569 During testing, I experienced a lot of bans with the remotes in this order. 13570 I suspect it might be related to how crypt opens files on the cloud provider 13571 which makes it think we're downloading the full file instead of small chunks. 13572 Organizing the remotes in this order yields better results: 13573 **cloud remote** -> **cache** -> **crypt** 13574 13575 #### absolute remote paths #### 13576 13577 `cache` can not differentiate between relative and absolute paths for the wrapped remote. 13578 Any path given in the `remote` config setting and on the command line will be passed to 13579 the wrapped remote as is, but for storing the chunks on disk the path will be made 13580 relative by removing any leading `/` character. 13581 13582 This behavior is irrelevant for most backend types, but there are backends where a leading `/` 13583 changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are 13584 relative to the root of the SSH server and paths without are relative to the user home directory. 13585 As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent 13586 a different directory on the SSH server. 13587 13588 ### Cache and Remote Control (--rc) ### 13589 Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points: 13590 By default, the listener is disabled if you do not add the flag. 13591 13592 ### rc cache/expire 13593 Purge a remote from the cache backend. Supports either a directory or a file. 13594 It supports both encrypted and unencrypted file names if cache is wrapped by crypt. 13595 13596 Params: 13597 - **remote** = path to remote **(required)** 13598 - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_ 13599 13600 13601 ### Standard Options 13602 13603 Here are the standard options specific to cache (Cache a remote). 13604 13605 #### --cache-remote 13606 13607 Remote to cache. 13608 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13609 "myremote:bucket" or maybe "myremote:" (not recommended). 13610 13611 - Config: remote 13612 - Env Var: RCLONE_CACHE_REMOTE 13613 - Type: string 13614 - Default: "" 13615 13616 #### --cache-plex-url 13617 13618 The URL of the Plex server 13619 13620 - Config: plex_url 13621 - Env Var: RCLONE_CACHE_PLEX_URL 13622 - Type: string 13623 - Default: "" 13624 13625 #### --cache-plex-username 13626 13627 The username of the Plex user 13628 13629 - Config: plex_username 13630 - Env Var: RCLONE_CACHE_PLEX_USERNAME 13631 - Type: string 13632 - Default: "" 13633 13634 #### --cache-plex-password 13635 13636 The password of the Plex user 13637 13638 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 13639 13640 - Config: plex_password 13641 - Env Var: RCLONE_CACHE_PLEX_PASSWORD 13642 - Type: string 13643 - Default: "" 13644 13645 #### --cache-chunk-size 13646 13647 The size of a chunk (partial file data). 13648 13649 Use lower numbers for slower connections. If the chunk size is 13650 changed, any downloaded chunks will be invalid and cache-chunk-path 13651 will need to be cleared or unexpected EOF errors will occur. 13652 13653 - Config: chunk_size 13654 - Env Var: RCLONE_CACHE_CHUNK_SIZE 13655 - Type: SizeSuffix 13656 - Default: 5M 13657 - Examples: 13658 - "1m" 13659 - 1MB 13660 - "5M" 13661 - 5 MB 13662 - "10M" 13663 - 10 MB 13664 13665 #### --cache-info-age 13666 13667 How long to cache file structure information (directory listings, file size, times etc). 13668 If all write operations are done through the cache then you can safely make 13669 this value very large as the cache store will also be updated in real time. 13670 13671 - Config: info_age 13672 - Env Var: RCLONE_CACHE_INFO_AGE 13673 - Type: Duration 13674 - Default: 6h0m0s 13675 - Examples: 13676 - "1h" 13677 - 1 hour 13678 - "24h" 13679 - 24 hours 13680 - "48h" 13681 - 48 hours 13682 13683 #### --cache-chunk-total-size 13684 13685 The total size that the chunks can take up on the local disk. 13686 13687 If the cache exceeds this value then it will start to delete the 13688 oldest chunks until it goes under this value. 13689 13690 - Config: chunk_total_size 13691 - Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE 13692 - Type: SizeSuffix 13693 - Default: 10G 13694 - Examples: 13695 - "500M" 13696 - 500 MB 13697 - "1G" 13698 - 1 GB 13699 - "10G" 13700 - 10 GB 13701 13702 ### Advanced Options 13703 13704 Here are the advanced options specific to cache (Cache a remote). 13705 13706 #### --cache-plex-token 13707 13708 The plex token for authentication - auto set normally 13709 13710 - Config: plex_token 13711 - Env Var: RCLONE_CACHE_PLEX_TOKEN 13712 - Type: string 13713 - Default: "" 13714 13715 #### --cache-plex-insecure 13716 13717 Skip all certificate verification when connecting to the Plex server 13718 13719 - Config: plex_insecure 13720 - Env Var: RCLONE_CACHE_PLEX_INSECURE 13721 - Type: string 13722 - Default: "" 13723 13724 #### --cache-db-path 13725 13726 Directory to store file structure metadata DB. 13727 The remote name is used as the DB file name. 13728 13729 - Config: db_path 13730 - Env Var: RCLONE_CACHE_DB_PATH 13731 - Type: string 13732 - Default: "$HOME/.cache/rclone/cache-backend" 13733 13734 #### --cache-chunk-path 13735 13736 Directory to cache chunk files. 13737 13738 Path to where partial file data (chunks) are stored locally. The remote 13739 name is appended to the final path. 13740 13741 This config follows the "--cache-db-path". If you specify a custom 13742 location for "--cache-db-path" and don't specify one for "--cache-chunk-path" 13743 then "--cache-chunk-path" will use the same path as "--cache-db-path". 13744 13745 - Config: chunk_path 13746 - Env Var: RCLONE_CACHE_CHUNK_PATH 13747 - Type: string 13748 - Default: "$HOME/.cache/rclone/cache-backend" 13749 13750 #### --cache-db-purge 13751 13752 Clear all the cached data for this remote on start. 13753 13754 - Config: db_purge 13755 - Env Var: RCLONE_CACHE_DB_PURGE 13756 - Type: bool 13757 - Default: false 13758 13759 #### --cache-chunk-clean-interval 13760 13761 How often should the cache perform cleanups of the chunk storage. 13762 The default value should be ok for most people. If you find that the 13763 cache goes over "cache-chunk-total-size" too often then try to lower 13764 this value to force it to perform cleanups more often. 13765 13766 - Config: chunk_clean_interval 13767 - Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL 13768 - Type: Duration 13769 - Default: 1m0s 13770 13771 #### --cache-read-retries 13772 13773 How many times to retry a read from a cache storage. 13774 13775 Since reading from a cache stream is independent from downloading file 13776 data, readers can get to a point where there's no more data in the 13777 cache. Most of the times this can indicate a connectivity issue if 13778 cache isn't able to provide file data anymore. 13779 13780 For really slow connections, increase this to a point where the stream is 13781 able to provide data but your experience will be very stuttering. 13782 13783 - Config: read_retries 13784 - Env Var: RCLONE_CACHE_READ_RETRIES 13785 - Type: int 13786 - Default: 10 13787 13788 #### --cache-workers 13789 13790 How many workers should run in parallel to download chunks. 13791 13792 Higher values will mean more parallel processing (better CPU needed) 13793 and more concurrent requests on the cloud provider. This impacts 13794 several aspects like the cloud provider API limits, more stress on the 13795 hardware that rclone runs on but it also means that streams will be 13796 more fluid and data will be available much more faster to readers. 13797 13798 **Note**: If the optional Plex integration is enabled then this 13799 setting will adapt to the type of reading performed and the value 13800 specified here will be used as a maximum number of workers to use. 13801 13802 - Config: workers 13803 - Env Var: RCLONE_CACHE_WORKERS 13804 - Type: int 13805 - Default: 4 13806 13807 #### --cache-chunk-no-memory 13808 13809 Disable the in-memory cache for storing chunks during streaming. 13810 13811 By default, cache will keep file data during streaming in RAM as well 13812 to provide it to readers as fast as possible. 13813 13814 This transient data is evicted as soon as it is read and the number of 13815 chunks stored doesn't exceed the number of workers. However, depending 13816 on other settings like "cache-chunk-size" and "cache-workers" this footprint 13817 can increase if there are parallel streams too (multiple files being read 13818 at the same time). 13819 13820 If the hardware permits it, use this feature to provide an overall better 13821 performance during streaming but it can also be disabled if RAM is not 13822 available on the local machine. 13823 13824 - Config: chunk_no_memory 13825 - Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY 13826 - Type: bool 13827 - Default: false 13828 13829 #### --cache-rps 13830 13831 Limits the number of requests per second to the source FS (-1 to disable) 13832 13833 This setting places a hard limit on the number of requests per second 13834 that cache will be doing to the cloud provider remote and try to 13835 respect that value by setting waits between reads. 13836 13837 If you find that you're getting banned or limited on the cloud 13838 provider through cache and know that a smaller number of requests per 13839 second will allow you to work with it then you can use this setting 13840 for that. 13841 13842 A good balance of all the other settings should make this setting 13843 useless but it is available to set for more special cases. 13844 13845 **NOTE**: This will limit the number of requests during streams but 13846 other API calls to the cloud provider like directory listings will 13847 still pass. 13848 13849 - Config: rps 13850 - Env Var: RCLONE_CACHE_RPS 13851 - Type: int 13852 - Default: -1 13853 13854 #### --cache-writes 13855 13856 Cache file data on writes through the FS 13857 13858 If you need to read files immediately after you upload them through 13859 cache you can enable this flag to have their data stored in the 13860 cache store at the same time during upload. 13861 13862 - Config: writes 13863 - Env Var: RCLONE_CACHE_WRITES 13864 - Type: bool 13865 - Default: false 13866 13867 #### --cache-tmp-upload-path 13868 13869 Directory to keep temporary files until they are uploaded. 13870 13871 This is the path where cache will use as a temporary storage for new 13872 files that need to be uploaded to the cloud provider. 13873 13874 Specifying a value will enable this feature. Without it, it is 13875 completely disabled and files will be uploaded directly to the cloud 13876 provider 13877 13878 - Config: tmp_upload_path 13879 - Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH 13880 - Type: string 13881 - Default: "" 13882 13883 #### --cache-tmp-wait-time 13884 13885 How long should files be stored in local cache before being uploaded 13886 13887 This is the duration that a file must wait in the temporary location 13888 _cache-tmp-upload-path_ before it is selected for upload. 13889 13890 Note that only one file is uploaded at a time and it can take longer 13891 to start the upload if a queue formed for this purpose. 13892 13893 - Config: tmp_wait_time 13894 - Env Var: RCLONE_CACHE_TMP_WAIT_TIME 13895 - Type: Duration 13896 - Default: 15s 13897 13898 #### --cache-db-wait-time 13899 13900 How long to wait for the DB to be available - 0 is unlimited 13901 13902 Only one process can have the DB open at any one time, so rclone waits 13903 for this duration for the DB to become available before it gives an 13904 error. 13905 13906 If you set it to 0 then it will wait forever. 13907 13908 - Config: db_wait_time 13909 - Env Var: RCLONE_CACHE_DB_WAIT_TIME 13910 - Type: Duration 13911 - Default: 1s 13912 13913 ### Backend commands 13914 13915 Here are the commands specific to the cache backend. 13916 13917 Run them with 13918 13919 rclone backend COMMAND remote: 13920 13921 The help below will explain what arguments each command takes. 13922 13923 See [the "rclone backend" command](https://rclone.org/commands/rclone_backend/) for more 13924 info on how to pass options and arguments. 13925 13926 These can be run on a running backend using the rc command 13927 [backend/command](https://rclone.org/rc/#backend/command). 13928 13929 #### stats 13930 13931 Print stats on the cache backend in JSON format. 13932 13933 rclone backend stats remote: [options] [<arguments>+] 13934 13935 13936 13937 Chunker (BETA) 13938 ---------------------------------------- 13939 13940 The `chunker` overlay transparently splits large files into smaller chunks 13941 during upload to wrapped remote and transparently assembles them back 13942 when the file is downloaded. This allows to effectively overcome size limits 13943 imposed by storage providers. 13944 13945 To use it, first set up the underlying remote following the configuration 13946 instructions for that remote. You can also use a local pathname instead of 13947 a remote. 13948 13949 First check your chosen remote is working - we'll call it `remote:path` here. 13950 Note that anything inside `remote:path` will be chunked and anything outside 13951 won't. This means that if you are using a bucket based remote (eg S3, B2, swift) 13952 then you should probably put the bucket in the remote `s3:bucket`. 13953 13954 Now configure `chunker` using `rclone config`. We will call this one `overlay` 13955 to separate it from the `remote` itself. 13956 13957 ``` 13958 No remotes found - make a new one 13959 n) New remote 13960 s) Set configuration password 13961 q) Quit config 13962 n/s/q> n 13963 name> overlay 13964 Type of storage to configure. 13965 Choose a number from below, or type in your own value 13966 [snip] 13967 XX / Transparently chunk/split large files 13968 \ "chunker" 13969 [snip] 13970 Storage> chunker 13971 Remote to chunk/unchunk. 13972 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 13973 "myremote:bucket" or maybe "myremote:" (not recommended). 13974 Enter a string value. Press Enter for the default (""). 13975 remote> remote:path 13976 Files larger than chunk size will be split in chunks. 13977 Enter a size with suffix k,M,G,T. Press Enter for the default ("2G"). 13978 chunk_size> 100M 13979 Choose how chunker handles hash sums. All modes but "none" require metadata. 13980 Enter a string value. Press Enter for the default ("md5"). 13981 Choose a number from below, or type in your own value 13982 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise 13983 \ "none" 13984 2 / MD5 for composite files 13985 \ "md5" 13986 3 / SHA1 for composite files 13987 \ "sha1" 13988 4 / MD5 for all files 13989 \ "md5all" 13990 5 / SHA1 for all files 13991 \ "sha1all" 13992 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported 13993 \ "md5quick" 13994 7 / Similar to "md5quick" but prefers SHA1 over MD5 13995 \ "sha1quick" 13996 hash_type> md5 13997 Edit advanced config? (y/n) 13998 y) Yes 13999 n) No 14000 y/n> n 14001 Remote config 14002 -------------------- 14003 [overlay] 14004 type = chunker 14005 remote = remote:bucket 14006 chunk_size = 100M 14007 hash_type = md5 14008 -------------------- 14009 y) Yes this is OK 14010 e) Edit this remote 14011 d) Delete this remote 14012 y/e/d> y 14013 ``` 14014 14015 ### Specifying the remote 14016 14017 In normal use, make sure the remote has a `:` in. If you specify the remote 14018 without a `:` then rclone will use a local directory of that name. 14019 So if you use a remote of `/path/to/secret/files` then rclone will 14020 chunk stuff in that directory. If you use a remote of `name` then rclone 14021 will put files in a directory called `name` in the current directory. 14022 14023 14024 ### Chunking 14025 14026 When rclone starts a file upload, chunker checks the file size. If it 14027 doesn't exceed the configured chunk size, chunker will just pass the file 14028 to the wrapped remote. If a file is large, chunker will transparently cut 14029 data in pieces with temporary names and stream them one by one, on the fly. 14030 Each data chunk will contain the specified number of bytes, except for the 14031 last one which may have less data. If file size is unknown in advance 14032 (this is called a streaming upload), chunker will internally create 14033 a temporary copy, record its size and repeat the above process. 14034 14035 When upload completes, temporary chunk files are finally renamed. 14036 This scheme guarantees that operations can be run in parallel and look 14037 from outside as atomic. 14038 A similar method with hidden temporary chunks is used for other operations 14039 (copy/move/rename etc). If an operation fails, hidden chunks are normally 14040 destroyed, and the target composite file stays intact. 14041 14042 When a composite file download is requested, chunker transparently 14043 assembles it by concatenating data chunks in order. As the split is trivial 14044 one could even manually concatenate data chunks together to obtain the 14045 original content. 14046 14047 When the `list` rclone command scans a directory on wrapped remote, 14048 the potential chunk files are accounted for, grouped and assembled into 14049 composite directory entries. Any temporary chunks are hidden. 14050 14051 List and other commands can sometimes come across composite files with 14052 missing or invalid chunks, eg. shadowed by like-named directory or 14053 another file. This usually means that wrapped file system has been directly 14054 tampered with or damaged. If chunker detects a missing chunk it will 14055 by default print warning, skip the whole incomplete group of chunks but 14056 proceed with current command. 14057 You can set the `--chunker-fail-hard` flag to have commands abort with 14058 error message in such cases. 14059 14060 14061 #### Chunk names 14062 14063 The default chunk name format is `*.rclone_chunk.###`, hence by default 14064 chunk names are `BIG_FILE_NAME.rclone_chunk.001`, 14065 `BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format 14066 using the `name_format` configuration file option. The format uses asterisk 14067 `*` as a placeholder for the base file name and one or more consecutive 14068 hash characters `#` as a placeholder for sequential chunk number. 14069 There must be one and only one asterisk. The number of consecutive hash 14070 characters defines the minimum length of a string representing a chunk number. 14071 If decimal chunk number has less digits than the number of hashes, it is 14072 left-padded by zeros. If the decimal string is longer, it is left intact. 14073 By default numbering starts from 1 but there is another option that allows 14074 user to start from 0, eg. for compatibility with legacy software. 14075 14076 For example, if name format is `big_*-##.part` and original file name is 14077 `data.txt` and numbering starts from 0, then the first chunk will be named 14078 `big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part` 14079 and the 302nd chunk will become `big_data.txt-301.part`. 14080 14081 Note that `list` assembles composite directory entries only when chunk names 14082 match the configured format and treats non-conforming file names as normal 14083 non-chunked files. 14084 14085 14086 ### Metadata 14087 14088 Besides data chunks chunker will by default create metadata object for 14089 a composite file. The object is named after the original file. 14090 Chunker allows user to disable metadata completely (the `none` format). 14091 Note that metadata is normally not created for files smaller than the 14092 configured chunk size. This may change in future rclone releases. 14093 14094 #### Simple JSON metadata format 14095 14096 This is the default format. It supports hash sums and chunk validation 14097 for composite files. Meta objects carry the following fields: 14098 14099 - `ver` - version of format, currently `1` 14100 - `size` - total size of composite file 14101 - `nchunks` - number of data chunks in file 14102 - `md5` - MD5 hashsum of composite file (if present) 14103 - `sha1` - SHA1 hashsum (if present) 14104 14105 There is no field for composite file name as it's simply equal to the name 14106 of meta object on the wrapped remote. Please refer to respective sections 14107 for details on hashsums and modified time handling. 14108 14109 #### No metadata 14110 14111 You can disable meta objects by setting the meta format option to `none`. 14112 In this mode chunker will scan directory for all files that follow 14113 configured chunk name format, group them by detecting chunks with the same 14114 base name and show group names as virtual composite files. 14115 This method is more prone to missing chunk errors (especially missing 14116 last chunk) than format with metadata enabled. 14117 14118 14119 ### Hashsums 14120 14121 Chunker supports hashsums only when a compatible metadata is present. 14122 Hence, if you choose metadata format of `none`, chunker will report hashsum 14123 as `UNSUPPORTED`. 14124 14125 Please note that by default metadata is stored only for composite files. 14126 If a file is smaller than configured chunk size, chunker will transparently 14127 redirect hash requests to wrapped remote, so support depends on that. 14128 You will see the empty string as a hashsum of requested type for small 14129 files if the wrapped remote doesn't support it. 14130 14131 Many storage backends support MD5 and SHA1 hash types, so does chunker. 14132 With chunker you can choose one or another but not both. 14133 MD5 is set by default as the most supported type. 14134 Since chunker keeps hashes for composite files and falls back to the 14135 wrapped remote hash for non-chunked ones, we advise you to choose the same 14136 hash type as supported by wrapped remote so that your file listings 14137 look coherent. 14138 14139 If your storage backend does not support MD5 or SHA1 but you need consistent 14140 file hashing, configure chunker with `md5all` or `sha1all`. These two modes 14141 guarantee given hash for all files. If wrapped remote doesn't support it, 14142 chunker will then add metadata to all files, even small. However, this can 14143 double the amount of small files in storage and incur additional service charges. 14144 You can even use chunker to force md5/sha1 support in any other remote 14145 at expense of sidecar meta objects by setting eg. `chunk_type=sha1all` 14146 to force hashsums and `chunk_size=1P` to effectively disable chunking. 14147 14148 Normally, when a file is copied to chunker controlled remote, chunker 14149 will ask the file source for compatible file hash and revert to on-the-fly 14150 calculation if none is found. This involves some CPU overhead but provides 14151 a guarantee that given hashsum is available. Also, chunker will reject 14152 a server-side copy or move operation if source and destination hashsum 14153 types are different resulting in the extra network bandwidth, too. 14154 In some rare cases this may be undesired, so chunker provides two optional 14155 choices: `sha1quick` and `md5quick`. If the source does not support primary 14156 hash type and the quick mode is enabled, chunker will try to fall back to 14157 the secondary type. This will save CPU and bandwidth but can result in empty 14158 hashsums at destination. Beware of consequences: the `sync` command will 14159 revert (sometimes silently) to time/size comparison if compatible hashsums 14160 between source and target are not found. 14161 14162 14163 ### Modified time 14164 14165 Chunker stores modification times using the wrapped remote so support 14166 depends on that. For a small non-chunked file the chunker overlay simply 14167 manipulates modification time of the wrapped remote file. 14168 For a composite file with metadata chunker will get and set 14169 modification time of the metadata object on the wrapped remote. 14170 If file is chunked but metadata format is `none` then chunker will 14171 use modification time of the first data chunk. 14172 14173 14174 ### Migrations 14175 14176 The idiomatic way to migrate to a different chunk size, hash type or 14177 chunk naming scheme is to: 14178 14179 - Collect all your chunked files under a directory and have your 14180 chunker remote point to it. 14181 - Create another directory (most probably on the same cloud storage) 14182 and configure a new remote with desired metadata format, 14183 hash type, chunk naming etc. 14184 - Now run `rclone sync oldchunks: newchunks:` and all your data 14185 will be transparently converted in transfer. 14186 This may take some time, yet chunker will try server-side 14187 copy if possible. 14188 - After checking data integrity you may remove configuration section 14189 of the old remote. 14190 14191 If rclone gets killed during a long operation on a big composite file, 14192 hidden temporary chunks may stay in the directory. They will not be 14193 shown by the `list` command but will eat up your account quota. 14194 Please note that the `deletefile` command deletes only active 14195 chunks of a file. As a workaround, you can use remote of the wrapped 14196 file system to see them. 14197 An easy way to get rid of hidden garbage is to copy littered directory 14198 somewhere using the chunker remote and purge the original directory. 14199 The `copy` command will copy only active chunks while the `purge` will 14200 remove everything including garbage. 14201 14202 14203 ### Caveats and Limitations 14204 14205 Chunker requires wrapped remote to support server side `move` (or `copy` + 14206 `delete`) operations, otherwise it will explicitly refuse to start. 14207 This is because it internally renames temporary chunk files to their final 14208 names when an operation completes successfully. 14209 14210 Chunker encodes chunk number in file name, so with default `name_format` 14211 setting it adds 17 characters. Also chunker adds 7 characters of temporary 14212 suffix during operations. Many file systems limit base file name without path 14213 by 255 characters. Using rclone's crypt remote as a base file system limits 14214 file name by 143 characters. Thus, maximum name length is 231 for most files 14215 and 119 for chunker-over-crypt. A user in need can change name format to 14216 eg. `*.rcc##` and save 10 characters (provided at most 99 chunks per file). 14217 14218 Note that a move implemented using the copy-and-delete method may incur 14219 double charging with some cloud storage providers. 14220 14221 Chunker will not automatically rename existing chunks when you run 14222 `rclone config` on a live remote and change the chunk name format. 14223 Beware that in result of this some files which have been treated as chunks 14224 before the change can pop up in directory listings as normal files 14225 and vice versa. The same warning holds for the chunk size. 14226 If you desperately need to change critical chunking settings, you should 14227 run data migration as described above. 14228 14229 If wrapped remote is case insensitive, the chunker overlay will inherit 14230 that property (so you can't have a file called "Hello.doc" and "hello.doc" 14231 in the same directory). 14232 14233 14234 14235 ### Standard Options 14236 14237 Here are the standard options specific to chunker (Transparently chunk/split large files). 14238 14239 #### --chunker-remote 14240 14241 Remote to chunk/unchunk. 14242 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 14243 "myremote:bucket" or maybe "myremote:" (not recommended). 14244 14245 - Config: remote 14246 - Env Var: RCLONE_CHUNKER_REMOTE 14247 - Type: string 14248 - Default: "" 14249 14250 #### --chunker-chunk-size 14251 14252 Files larger than chunk size will be split in chunks. 14253 14254 - Config: chunk_size 14255 - Env Var: RCLONE_CHUNKER_CHUNK_SIZE 14256 - Type: SizeSuffix 14257 - Default: 2G 14258 14259 #### --chunker-hash-type 14260 14261 Choose how chunker handles hash sums. All modes but "none" require metadata. 14262 14263 - Config: hash_type 14264 - Env Var: RCLONE_CHUNKER_HASH_TYPE 14265 - Type: string 14266 - Default: "md5" 14267 - Examples: 14268 - "none" 14269 - Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise 14270 - "md5" 14271 - MD5 for composite files 14272 - "sha1" 14273 - SHA1 for composite files 14274 - "md5all" 14275 - MD5 for all files 14276 - "sha1all" 14277 - SHA1 for all files 14278 - "md5quick" 14279 - Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported 14280 - "sha1quick" 14281 - Similar to "md5quick" but prefers SHA1 over MD5 14282 14283 ### Advanced Options 14284 14285 Here are the advanced options specific to chunker (Transparently chunk/split large files). 14286 14287 #### --chunker-name-format 14288 14289 String format of chunk file names. 14290 The two placeholders are: base file name (*) and chunk number (#...). 14291 There must be one and only one asterisk and one or more consecutive hash characters. 14292 If chunk number has less digits than the number of hashes, it is left-padded by zeros. 14293 If there are more digits in the number, they are left as is. 14294 Possible chunk files are ignored if their name does not match given format. 14295 14296 - Config: name_format 14297 - Env Var: RCLONE_CHUNKER_NAME_FORMAT 14298 - Type: string 14299 - Default: "*.rclone_chunk.###" 14300 14301 #### --chunker-start-from 14302 14303 Minimum valid chunk number. Usually 0 or 1. 14304 By default chunk numbers start from 1. 14305 14306 - Config: start_from 14307 - Env Var: RCLONE_CHUNKER_START_FROM 14308 - Type: int 14309 - Default: 1 14310 14311 #### --chunker-meta-format 14312 14313 Format of the metadata object or "none". By default "simplejson". 14314 Metadata is a small JSON file named after the composite file. 14315 14316 - Config: meta_format 14317 - Env Var: RCLONE_CHUNKER_META_FORMAT 14318 - Type: string 14319 - Default: "simplejson" 14320 - Examples: 14321 - "none" 14322 - Do not use metadata files at all. Requires hash type "none". 14323 - "simplejson" 14324 - Simple JSON supports hash sums and chunk validation. 14325 - It has the following fields: ver, size, nchunks, md5, sha1. 14326 14327 #### --chunker-fail-hard 14328 14329 Choose how chunker should handle files with missing or invalid chunks. 14330 14331 - Config: fail_hard 14332 - Env Var: RCLONE_CHUNKER_FAIL_HARD 14333 - Type: bool 14334 - Default: false 14335 - Examples: 14336 - "true" 14337 - Report errors and abort current command. 14338 - "false" 14339 - Warn user, skip incomplete file and proceed. 14340 14341 14342 14343 ## Citrix ShareFile 14344 14345 [Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business. 14346 14347 The initial setup for Citrix ShareFile involves getting a token from 14348 Citrix ShareFile which you can in your browser. `rclone config` walks you 14349 through it. 14350 14351 Here is an example of how to make a remote called `remote`. First run: 14352 14353 rclone config 14354 14355 This will guide you through an interactive setup process: 14356 14357 ``` 14358 No remotes found - make a new one 14359 n) New remote 14360 s) Set configuration password 14361 q) Quit config 14362 n/s/q> n 14363 name> remote 14364 Type of storage to configure. 14365 Enter a string value. Press Enter for the default (""). 14366 Choose a number from below, or type in your own value 14367 XX / Citrix Sharefile 14368 \ "sharefile" 14369 Storage> sharefile 14370 ** See help for sharefile backend at: https://rclone.org/sharefile/ ** 14371 14372 ID of the root folder 14373 14374 Leave blank to access "Personal Folders". You can use one of the 14375 standard values here or any folder ID (long hex number ID). 14376 Enter a string value. Press Enter for the default (""). 14377 Choose a number from below, or type in your own value 14378 1 / Access the Personal Folders. (Default) 14379 \ "" 14380 2 / Access the Favorites folder. 14381 \ "favorites" 14382 3 / Access all the shared folders. 14383 \ "allshared" 14384 4 / Access all the individual connectors. 14385 \ "connectors" 14386 5 / Access the home, favorites, and shared folders as well as the connectors. 14387 \ "top" 14388 root_folder_id> 14389 Edit advanced config? (y/n) 14390 y) Yes 14391 n) No 14392 y/n> n 14393 Remote config 14394 Use auto config? 14395 * Say Y if not sure 14396 * Say N if you are working on a remote or headless machine 14397 y) Yes 14398 n) No 14399 y/n> y 14400 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX 14401 Log in and authorize rclone for access 14402 Waiting for code... 14403 Got code 14404 -------------------- 14405 [remote] 14406 type = sharefile 14407 endpoint = https://XXX.sharefile.com 14408 token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} 14409 -------------------- 14410 y) Yes this is OK 14411 e) Edit this remote 14412 d) Delete this remote 14413 y/e/d> y 14414 ``` 14415 14416 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 14417 machine with no Internet browser available. 14418 14419 Note that rclone runs a webserver on your local machine to collect the 14420 token as returned from Citrix ShareFile. This only runs from the moment it opens 14421 your browser to the moment you get back the verification code. This 14422 is on `http://127.0.0.1:53682/` and this it may require you to unblock 14423 it temporarily if you are running a host firewall. 14424 14425 Once configured you can then use `rclone` like this, 14426 14427 List directories in top level of your ShareFile 14428 14429 rclone lsd remote: 14430 14431 List all the files in your ShareFile 14432 14433 rclone ls remote: 14434 14435 To copy a local directory to an ShareFile directory called backup 14436 14437 rclone copy /home/source remote:backup 14438 14439 Paths may be as deep as required, eg `remote:directory/subdirectory`. 14440 14441 ### Modified time and hashes ### 14442 14443 ShareFile allows modification times to be set on objects accurate to 1 14444 second. These will be used to detect whether objects need syncing or 14445 not. 14446 14447 ShareFile supports MD5 type hashes, so you can use the `--checksum` 14448 flag. 14449 14450 ### Transfers ### 14451 14452 For files above 128MB rclone will use a chunked transfer. Rclone will 14453 upload up to `--transfers` chunks at the same time (shared among all 14454 the multipart uploads). Chunks are buffered in memory and are 14455 normally 64MB so increasing `--transfers` will increase memory use. 14456 14457 ### Limitations ### 14458 14459 Note that ShareFile is case insensitive so you can't have a file called 14460 "Hello.doc" and one called "hello.doc". 14461 14462 ShareFile only supports filenames up to 256 characters in length. 14463 14464 #### Restricted filename characters 14465 14466 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 14467 the following characters are also replaced: 14468 14469 | Character | Value | Replacement | 14470 | --------- |:-----:|:-----------:| 14471 | \\ | 0x5C | \ | 14472 | * | 0x2A | * | 14473 | < | 0x3C | < | 14474 | > | 0x3E | > | 14475 | ? | 0x3F | ? | 14476 | : | 0x3A | : | 14477 | \| | 0x7C | | | 14478 | " | 0x22 | " | 14479 14480 File names can also not start or end with the following characters. 14481 These only get replaced if they are the first or last character in the 14482 name: 14483 14484 | Character | Value | Replacement | 14485 | --------- |:-----:|:-----------:| 14486 | SP | 0x20 | ␠ | 14487 | . | 0x2E | . | 14488 14489 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 14490 as they can't be used in JSON strings. 14491 14492 14493 ### Standard Options 14494 14495 Here are the standard options specific to sharefile (Citrix Sharefile). 14496 14497 #### --sharefile-root-folder-id 14498 14499 ID of the root folder 14500 14501 Leave blank to access "Personal Folders". You can use one of the 14502 standard values here or any folder ID (long hex number ID). 14503 14504 - Config: root_folder_id 14505 - Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID 14506 - Type: string 14507 - Default: "" 14508 - Examples: 14509 - "" 14510 - Access the Personal Folders. (Default) 14511 - "favorites" 14512 - Access the Favorites folder. 14513 - "allshared" 14514 - Access all the shared folders. 14515 - "connectors" 14516 - Access all the individual connectors. 14517 - "top" 14518 - Access the home, favorites, and shared folders as well as the connectors. 14519 14520 ### Advanced Options 14521 14522 Here are the advanced options specific to sharefile (Citrix Sharefile). 14523 14524 #### --sharefile-upload-cutoff 14525 14526 Cutoff for switching to multipart upload. 14527 14528 - Config: upload_cutoff 14529 - Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF 14530 - Type: SizeSuffix 14531 - Default: 128M 14532 14533 #### --sharefile-chunk-size 14534 14535 Upload chunk size. Must a power of 2 >= 256k. 14536 14537 Making this larger will improve performance, but note that each chunk 14538 is buffered in memory one per transfer. 14539 14540 Reducing this will reduce memory usage but decrease performance. 14541 14542 - Config: chunk_size 14543 - Env Var: RCLONE_SHAREFILE_CHUNK_SIZE 14544 - Type: SizeSuffix 14545 - Default: 64M 14546 14547 #### --sharefile-endpoint 14548 14549 Endpoint for API calls. 14550 14551 This is usually auto discovered as part of the oauth process, but can 14552 be set manually to something like: https://XXX.sharefile.com 14553 14554 14555 - Config: endpoint 14556 - Env Var: RCLONE_SHAREFILE_ENDPOINT 14557 - Type: string 14558 - Default: "" 14559 14560 #### --sharefile-encoding 14561 14562 This sets the encoding for the backend. 14563 14564 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 14565 14566 - Config: encoding 14567 - Env Var: RCLONE_SHAREFILE_ENCODING 14568 - Type: MultiEncoder 14569 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot 14570 14571 14572 14573 Crypt 14574 ---------------------------------------- 14575 14576 The `crypt` remote encrypts and decrypts another remote. 14577 14578 To use it first set up the underlying remote following the config 14579 instructions for that remote. You can also use a local pathname 14580 instead of a remote which will encrypt and decrypt from that directory 14581 which might be useful for encrypting onto a USB stick for example. 14582 14583 First check your chosen remote is working - we'll call it 14584 `remote:path` in these docs. Note that anything inside `remote:path` 14585 will be encrypted and anything outside won't. This means that if you 14586 are using a bucket based remote (eg S3, B2, swift) then you should 14587 probably put the bucket in the remote `s3:bucket`. If you just use 14588 `s3:` then rclone will make encrypted bucket names too (if using file 14589 name encryption) which may or may not be what you want. 14590 14591 Now configure `crypt` using `rclone config`. We will call this one 14592 `secret` to differentiate it from the `remote`. 14593 14594 ``` 14595 No remotes found - make a new one 14596 n) New remote 14597 s) Set configuration password 14598 q) Quit config 14599 n/s/q> n 14600 name> secret 14601 Type of storage to configure. 14602 Choose a number from below, or type in your own value 14603 [snip] 14604 XX / Encrypt/Decrypt a remote 14605 \ "crypt" 14606 [snip] 14607 Storage> crypt 14608 Remote to encrypt/decrypt. 14609 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 14610 "myremote:bucket" or maybe "myremote:" (not recommended). 14611 remote> remote:path 14612 How to encrypt the filenames. 14613 Choose a number from below, or type in your own value 14614 1 / Don't encrypt the file names. Adds a ".bin" extension only. 14615 \ "off" 14616 2 / Encrypt the filenames see the docs for the details. 14617 \ "standard" 14618 3 / Very simple filename obfuscation. 14619 \ "obfuscate" 14620 filename_encryption> 2 14621 Option to either encrypt directory names or leave them intact. 14622 Choose a number from below, or type in your own value 14623 1 / Encrypt directory names. 14624 \ "true" 14625 2 / Don't encrypt directory names, leave them intact. 14626 \ "false" 14627 filename_encryption> 1 14628 Password or pass phrase for encryption. 14629 y) Yes type in my own password 14630 g) Generate random password 14631 y/g> y 14632 Enter the password: 14633 password: 14634 Confirm the password: 14635 password: 14636 Password or pass phrase for salt. Optional but recommended. 14637 Should be different to the previous password. 14638 y) Yes type in my own password 14639 g) Generate random password 14640 n) No leave this optional password blank 14641 y/g/n> g 14642 Password strength in bits. 14643 64 is just about memorable 14644 128 is secure 14645 1024 is the maximum 14646 Bits> 128 14647 Your password is: JAsJvRcgR-_veXNfy_sGmQ 14648 Use this password? 14649 y) Yes 14650 n) No 14651 y/n> y 14652 Remote config 14653 -------------------- 14654 [secret] 14655 remote = remote:path 14656 filename_encryption = standard 14657 password = *** ENCRYPTED *** 14658 password2 = *** ENCRYPTED *** 14659 -------------------- 14660 y) Yes this is OK 14661 e) Edit this remote 14662 d) Delete this remote 14663 y/e/d> y 14664 ``` 14665 14666 **Important** The password is stored in the config file is lightly 14667 obscured so it isn't immediately obvious what it is. It is in no way 14668 secure unless you use config file encryption. 14669 14670 A long passphrase is recommended, or you can use a random one. 14671 14672 The obscured password is created by using AES-CTR with a static key, with 14673 the salt stored verbatim at the beginning of the obscured password. This 14674 static key is shared by between all versions of rclone. 14675 14676 If you reconfigure rclone with the same passwords/passphrases 14677 elsewhere it will be compatible, but the obscured version will be different 14678 due to the different salt. 14679 14680 Note that rclone does not encrypt 14681 14682 * file length - this can be calculated within 16 bytes 14683 * modification time - used for syncing 14684 14685 ## Specifying the remote ## 14686 14687 In normal use, make sure the remote has a `:` in. If you specify the 14688 remote without a `:` then rclone will use a local directory of that 14689 name. So if you use a remote of `/path/to/secret/files` then rclone 14690 will encrypt stuff to that directory. If you use a remote of `name` 14691 then rclone will put files in a directory called `name` in the current 14692 directory. 14693 14694 If you specify the remote as `remote:path/to/dir` then rclone will 14695 store encrypted files in `path/to/dir` on the remote. If you are using 14696 file name encryption, then when you save files to 14697 `secret:subdir/subfile` this will store them in the unencrypted path 14698 `path/to/dir` but the `subdir/subpath` bit will be encrypted. 14699 14700 Note that unless you want encrypted bucket names (which are difficult 14701 to manage because you won't know what directory they represent in web 14702 interfaces etc), you should probably specify a bucket, eg 14703 `remote:secretbucket` when using bucket based remotes such as S3, 14704 Swift, Hubic, B2, GCS. 14705 14706 ## Example ## 14707 14708 To test I made a little directory of files using "standard" file name 14709 encryption. 14710 14711 ``` 14712 plaintext/ 14713 ├── file0.txt 14714 ├── file1.txt 14715 └── subdir 14716 ├── file2.txt 14717 ├── file3.txt 14718 └── subsubdir 14719 └── file4.txt 14720 ``` 14721 14722 Copy these to the remote and list them back 14723 14724 ``` 14725 $ rclone -q copy plaintext secret: 14726 $ rclone -q ls secret: 14727 7 file1.txt 14728 6 file0.txt 14729 8 subdir/file2.txt 14730 10 subdir/subsubdir/file4.txt 14731 9 subdir/file3.txt 14732 ``` 14733 14734 Now see what that looked like when encrypted 14735 14736 ``` 14737 $ rclone -q ls remote:path 14738 55 hagjclgavj2mbiqm6u6cnjjqcg 14739 54 v05749mltvv1tf4onltun46gls 14740 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 14741 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 14742 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 14743 ``` 14744 14745 Note that this retains the directory structure which means you can do this 14746 14747 ``` 14748 $ rclone -q ls secret:subdir 14749 8 file2.txt 14750 9 file3.txt 14751 10 subsubdir/file4.txt 14752 ``` 14753 14754 If don't use file name encryption then the remote will look like this 14755 - note the `.bin` extensions added to prevent the cloud provider 14756 attempting to interpret the data. 14757 14758 ``` 14759 $ rclone -q ls remote:path 14760 54 file0.txt.bin 14761 57 subdir/file3.txt.bin 14762 56 subdir/file2.txt.bin 14763 58 subdir/subsubdir/file4.txt.bin 14764 55 file1.txt.bin 14765 ``` 14766 14767 ### File name encryption modes ### 14768 14769 Here are some of the features of the file name encryption modes 14770 14771 Off 14772 14773 * doesn't hide file names or directory structure 14774 * allows for longer file names (~246 characters) 14775 * can use sub paths and copy single files 14776 14777 Standard 14778 14779 * file names encrypted 14780 * file names can't be as long (~143 characters) 14781 * can use sub paths and copy single files 14782 * directory structure visible 14783 * identical files names will have identical uploaded names 14784 * can use shortcuts to shorten the directory recursion 14785 14786 Obfuscation 14787 14788 This is a simple "rotate" of the filename, with each file having a rot 14789 distance based on the filename. We store the distance at the beginning 14790 of the filename. So a file called "hello" may become "53.jgnnq". 14791 14792 This is not a strong encryption of filenames, but it may stop automated 14793 scanning tools from picking up on filename patterns. As such it's an 14794 intermediate between "off" and "standard". The advantage is that it 14795 allows for longer path segment names. 14796 14797 There is a possibility with some unicode based filenames that the 14798 obfuscation is weak and may map lower case characters to upper case 14799 equivalents. You can not rely on this for strong protection. 14800 14801 * file names very lightly obfuscated 14802 * file names can be longer than standard encryption 14803 * can use sub paths and copy single files 14804 * directory structure visible 14805 * identical files names will have identical uploaded names 14806 14807 Cloud storage systems have various limits on file name length and 14808 total path length which you are more likely to hit using "Standard" 14809 file name encryption. If you keep your file names to below 156 14810 characters in length then you should be OK on all providers. 14811 14812 There may be an even more secure file name encryption mode in the 14813 future which will address the long file name problem. 14814 14815 ### Directory name encryption ### 14816 Crypt offers the option of encrypting dir names or leaving them intact. 14817 There are two options: 14818 14819 True 14820 14821 Encrypts the whole file path including directory names 14822 Example: 14823 `1/12/123.txt` is encrypted to 14824 `p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0` 14825 14826 False 14827 14828 Only encrypts file names, skips directory names 14829 Example: 14830 `1/12/123.txt` is encrypted to 14831 `1/12/qgm4avr35m5loi1th53ato71v0` 14832 14833 14834 ### Modified time and hashes ### 14835 14836 Crypt stores modification times using the underlying remote so support 14837 depends on that. 14838 14839 Hashes are not stored for crypt. However the data integrity is 14840 protected by an extremely strong crypto authenticator. 14841 14842 Note that you should use the `rclone cryptcheck` command to check the 14843 integrity of a crypted remote instead of `rclone check` which can't 14844 check the checksums properly. 14845 14846 14847 ### Standard Options 14848 14849 Here are the standard options specific to crypt (Encrypt/Decrypt a remote). 14850 14851 #### --crypt-remote 14852 14853 Remote to encrypt/decrypt. 14854 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 14855 "myremote:bucket" or maybe "myremote:" (not recommended). 14856 14857 - Config: remote 14858 - Env Var: RCLONE_CRYPT_REMOTE 14859 - Type: string 14860 - Default: "" 14861 14862 #### --crypt-filename-encryption 14863 14864 How to encrypt the filenames. 14865 14866 - Config: filename_encryption 14867 - Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION 14868 - Type: string 14869 - Default: "standard" 14870 - Examples: 14871 - "standard" 14872 - Encrypt the filenames see the docs for the details. 14873 - "obfuscate" 14874 - Very simple filename obfuscation. 14875 - "off" 14876 - Don't encrypt the file names. Adds a ".bin" extension only. 14877 14878 #### --crypt-directory-name-encryption 14879 14880 Option to either encrypt directory names or leave them intact. 14881 14882 NB If filename_encryption is "off" then this option will do nothing. 14883 14884 - Config: directory_name_encryption 14885 - Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION 14886 - Type: bool 14887 - Default: true 14888 - Examples: 14889 - "true" 14890 - Encrypt directory names. 14891 - "false" 14892 - Don't encrypt directory names, leave them intact. 14893 14894 #### --crypt-password 14895 14896 Password or pass phrase for encryption. 14897 14898 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 14899 14900 - Config: password 14901 - Env Var: RCLONE_CRYPT_PASSWORD 14902 - Type: string 14903 - Default: "" 14904 14905 #### --crypt-password2 14906 14907 Password or pass phrase for salt. Optional but recommended. 14908 Should be different to the previous password. 14909 14910 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 14911 14912 - Config: password2 14913 - Env Var: RCLONE_CRYPT_PASSWORD2 14914 - Type: string 14915 - Default: "" 14916 14917 ### Advanced Options 14918 14919 Here are the advanced options specific to crypt (Encrypt/Decrypt a remote). 14920 14921 #### --crypt-show-mapping 14922 14923 For all files listed show how the names encrypt. 14924 14925 If this flag is set then for each file that the remote is asked to 14926 list, it will log (at level INFO) a line stating the decrypted file 14927 name and the encrypted file name. 14928 14929 This is so you can work out which encrypted names are which decrypted 14930 names just in case you need to do something with the encrypted file 14931 names, or for debugging purposes. 14932 14933 - Config: show_mapping 14934 - Env Var: RCLONE_CRYPT_SHOW_MAPPING 14935 - Type: bool 14936 - Default: false 14937 14938 ### Backend commands 14939 14940 Here are the commands specific to the crypt backend. 14941 14942 Run them with 14943 14944 rclone backend COMMAND remote: 14945 14946 The help below will explain what arguments each command takes. 14947 14948 See [the "rclone backend" command](https://rclone.org/commands/rclone_backend/) for more 14949 info on how to pass options and arguments. 14950 14951 These can be run on a running backend using the rc command 14952 [backend/command](https://rclone.org/rc/#backend/command). 14953 14954 #### encode 14955 14956 Encode the given filename(s) 14957 14958 rclone backend encode remote: [options] [<arguments>+] 14959 14960 This encodes the filenames given as arguments returning a list of 14961 strings of the encoded results. 14962 14963 Usage Example: 14964 14965 rclone backend encode crypt: file1 [file2...] 14966 rclone rc backend/command command=encode fs=crypt: file1 [file2...] 14967 14968 14969 #### decode 14970 14971 Decode the given filename(s) 14972 14973 rclone backend decode remote: [options] [<arguments>+] 14974 14975 This decodes the filenames given as arguments returning a list of 14976 strings of the decoded results. It will return an error if any of the 14977 inputs are invalid. 14978 14979 Usage Example: 14980 14981 rclone backend decode crypt: encryptedfile1 [encryptedfile2...] 14982 rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...] 14983 14984 14985 14986 14987 ## Backing up a crypted remote ## 14988 14989 If you wish to backup a crypted remote, it is recommended that you use 14990 `rclone sync` on the encrypted files, and make sure the passwords are 14991 the same in the new encrypted remote. 14992 14993 This will have the following advantages 14994 14995 * `rclone sync` will check the checksums while copying 14996 * you can use `rclone check` between the encrypted remotes 14997 * you don't decrypt and encrypt unnecessarily 14998 14999 For example, let's say you have your original remote at `remote:` with 15000 the encrypted version at `eremote:` with path `remote:crypt`. You 15001 would then set up the new remote `remote2:` and then the encrypted 15002 version `eremote2:` with path `remote2:crypt` using the same passwords 15003 as `eremote:`. 15004 15005 To sync the two remotes you would do 15006 15007 rclone sync remote:crypt remote2:crypt 15008 15009 And to check the integrity you would do 15010 15011 rclone check remote:crypt remote2:crypt 15012 15013 ## File formats ## 15014 15015 ### File encryption ### 15016 15017 Files are encrypted 1:1 source file to destination object. The file 15018 has a header and is divided into chunks. 15019 15020 #### Header #### 15021 15022 * 8 bytes magic string `RCLONE\x00\x00` 15023 * 24 bytes Nonce (IV) 15024 15025 The initial nonce is generated from the operating systems crypto 15026 strong random number generator. The nonce is incremented for each 15027 chunk read making sure each nonce is unique for each block written. 15028 The chance of a nonce being re-used is minuscule. If you wrote an 15029 exabyte of data (10¹⁸ bytes) you would have a probability of 15030 approximately 2×10⁻³² of re-using a nonce. 15031 15032 #### Chunk #### 15033 15034 Each chunk will contain 64kB of data, except for the last one which 15035 may have less data. The data chunk is in standard NACL secretbox 15036 format. Secretbox uses XSalsa20 and Poly1305 to encrypt and 15037 authenticate messages. 15038 15039 Each chunk contains: 15040 15041 * 16 Bytes of Poly1305 authenticator 15042 * 1 - 65536 bytes XSalsa20 encrypted data 15043 15044 64k chunk size was chosen as the best performing chunk size (the 15045 authenticator takes too much time below this and the performance drops 15046 off due to cache effects above this). Note that these chunks are 15047 buffered in memory so they can't be too big. 15048 15049 This uses a 32 byte (256 bit key) key derived from the user password. 15050 15051 #### Examples #### 15052 15053 1 byte file will encrypt to 15054 15055 * 32 bytes header 15056 * 17 bytes data chunk 15057 15058 49 bytes total 15059 15060 1MB (1048576 bytes) file will encrypt to 15061 15062 * 32 bytes header 15063 * 16 chunks of 65568 bytes 15064 15065 1049120 bytes total (a 0.05% overhead). This is the overhead for big 15066 files. 15067 15068 ### Name encryption ### 15069 15070 File names are encrypted segment by segment - the path is broken up 15071 into `/` separated strings and these are encrypted individually. 15072 15073 File segments are padded using PKCS#7 to a multiple of 16 bytes 15074 before encryption. 15075 15076 They are then encrypted with EME using AES with 256 bit key. EME 15077 (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003 15078 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway. 15079 15080 This makes for deterministic encryption which is what we want - the 15081 same filename must encrypt to the same thing otherwise we can't find 15082 it on the cloud storage system. 15083 15084 This means that 15085 15086 * filenames with the same name will encrypt the same 15087 * filenames which start the same won't have a common prefix 15088 15089 This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of 15090 which are derived from the user password. 15091 15092 After encryption they are written out using a modified version of 15093 standard `base32` encoding as described in RFC4648. The standard 15094 encoding is modified in two ways: 15095 15096 * it becomes lower case (no-one likes upper case filenames!) 15097 * we strip the padding character `=` 15098 15099 `base32` is used rather than the more efficient `base64` so rclone can be 15100 used on case insensitive remotes (eg Windows, Amazon Drive). 15101 15102 ### Key derivation ### 15103 15104 Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an 15105 optional user supplied salt (password2) to derive the 32+32+16 = 80 15106 bytes of key material required. If the user doesn't supply a salt 15107 then rclone uses an internal one. 15108 15109 `scrypt` makes it impractical to mount a dictionary attack on rclone 15110 encrypted data. For full protection against this you should always use 15111 a salt. 15112 15113 Dropbox 15114 --------------------------------- 15115 15116 Paths are specified as `remote:path` 15117 15118 Dropbox paths may be as deep as required, eg 15119 `remote:directory/subdirectory`. 15120 15121 The initial setup for dropbox involves getting a token from Dropbox 15122 which you need to do in your browser. `rclone config` walks you 15123 through it. 15124 15125 Here is an example of how to make a remote called `remote`. First run: 15126 15127 rclone config 15128 15129 This will guide you through an interactive setup process: 15130 15131 ``` 15132 n) New remote 15133 d) Delete remote 15134 q) Quit config 15135 e/n/d/q> n 15136 name> remote 15137 Type of storage to configure. 15138 Choose a number from below, or type in your own value 15139 [snip] 15140 XX / Dropbox 15141 \ "dropbox" 15142 [snip] 15143 Storage> dropbox 15144 Dropbox App Key - leave blank normally. 15145 app_key> 15146 Dropbox App Secret - leave blank normally. 15147 app_secret> 15148 Remote config 15149 Please visit: 15150 https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code 15151 Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX 15152 -------------------- 15153 [remote] 15154 app_key = 15155 app_secret = 15156 token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 15157 -------------------- 15158 y) Yes this is OK 15159 e) Edit this remote 15160 d) Delete this remote 15161 y/e/d> y 15162 ``` 15163 15164 You can then use it like this, 15165 15166 List directories in top level of your dropbox 15167 15168 rclone lsd remote: 15169 15170 List all the files in your dropbox 15171 15172 rclone ls remote: 15173 15174 To copy a local directory to a dropbox directory called backup 15175 15176 rclone copy /home/source remote:backup 15177 15178 ### Dropbox for business ### 15179 15180 Rclone supports Dropbox for business and Team Folders. 15181 15182 When using Dropbox for business `remote:` and `remote:path/to/file` 15183 will refer to your personal folder. 15184 15185 If you wish to see Team Folders you must use a leading `/` in the 15186 path, so `rclone lsd remote:/` will refer to the root and show you all 15187 Team Folders and your User Folder. 15188 15189 You can then use team folders like this `remote:/TeamFolder` and 15190 `remote:/TeamFolder/path/to/file`. 15191 15192 A leading `/` for a Dropbox personal account will do nothing, but it 15193 will take an extra HTTP transaction so it should be avoided. 15194 15195 ### Modified time and Hashes ### 15196 15197 Dropbox supports modified times, but the only way to set a 15198 modification time is to re-upload the file. 15199 15200 This means that if you uploaded your data with an older version of 15201 rclone which didn't support the v2 API and modified times, rclone will 15202 decide to upload all your old data to fix the modification times. If 15203 you don't want this to happen use `--size-only` or `--checksum` flag 15204 to stop it. 15205 15206 Dropbox supports [its own hash 15207 type](https://www.dropbox.com/developers/reference/content-hash) which 15208 is checked for all transfers. 15209 15210 #### Restricted filename characters 15211 15212 | Character | Value | Replacement | 15213 | --------- |:-----:|:-----------:| 15214 | NUL | 0x00 | ␀ | 15215 | / | 0x2F | / | 15216 | DEL | 0x7F | ␡ | 15217 | \ | 0x5C | \ | 15218 15219 File names can also not end with the following characters. 15220 These only get replaced if they are the last character in the name: 15221 15222 | Character | Value | Replacement | 15223 | --------- |:-----:|:-----------:| 15224 | SP | 0x20 | ␠ | 15225 15226 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 15227 as they can't be used in JSON strings. 15228 15229 15230 ### Standard Options 15231 15232 Here are the standard options specific to dropbox (Dropbox). 15233 15234 #### --dropbox-client-id 15235 15236 Dropbox App Client Id 15237 Leave blank normally. 15238 15239 - Config: client_id 15240 - Env Var: RCLONE_DROPBOX_CLIENT_ID 15241 - Type: string 15242 - Default: "" 15243 15244 #### --dropbox-client-secret 15245 15246 Dropbox App Client Secret 15247 Leave blank normally. 15248 15249 - Config: client_secret 15250 - Env Var: RCLONE_DROPBOX_CLIENT_SECRET 15251 - Type: string 15252 - Default: "" 15253 15254 ### Advanced Options 15255 15256 Here are the advanced options specific to dropbox (Dropbox). 15257 15258 #### --dropbox-chunk-size 15259 15260 Upload chunk size. (< 150M). 15261 15262 Any files larger than this will be uploaded in chunks of this size. 15263 15264 Note that chunks are buffered in memory (one at a time) so rclone can 15265 deal with retries. Setting this larger will increase the speed 15266 slightly (at most 10% for 128MB in tests) at the cost of using more 15267 memory. It can be set smaller if you are tight on memory. 15268 15269 - Config: chunk_size 15270 - Env Var: RCLONE_DROPBOX_CHUNK_SIZE 15271 - Type: SizeSuffix 15272 - Default: 48M 15273 15274 #### --dropbox-impersonate 15275 15276 Impersonate this user when using a business account. 15277 15278 - Config: impersonate 15279 - Env Var: RCLONE_DROPBOX_IMPERSONATE 15280 - Type: string 15281 - Default: "" 15282 15283 #### --dropbox-encoding 15284 15285 This sets the encoding for the backend. 15286 15287 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 15288 15289 - Config: encoding 15290 - Env Var: RCLONE_DROPBOX_ENCODING 15291 - Type: MultiEncoder 15292 - Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot 15293 15294 15295 15296 ### Limitations ### 15297 15298 Note that Dropbox is case insensitive so you can't have a file called 15299 "Hello.doc" and one called "hello.doc". 15300 15301 There are some file names such as `thumbs.db` which Dropbox can't 15302 store. There is a full list of them in the ["Ignored Files" section 15303 of this document](https://www.dropbox.com/en/help/145). Rclone will 15304 issue an error message `File name disallowed - not uploading` if it 15305 attempts to upload one of those file names, but the sync won't fail. 15306 15307 If you have more than 10,000 files in a directory then `rclone purge 15308 dropbox:dir` will return the error `Failed to purge: There are too 15309 many files involved in this operation`. As a work-around do an 15310 `rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`. 15311 15312 ### Get your own Dropbox App ID ### 15313 15314 When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users. 15315 15316 Here is how to create your own Dropbox App ID for rclone: 15317 15318 1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not 15319 to be the same account as the Dropbox you want to access) 15320 15321 2. Choose an API => Usually this should be `Dropbox API` 15322 15323 3. Choose the type of access you want to use => `Full Dropbox` or `App Folder` 15324 15325 4. Name your App. The app name is global, so you can't use `rclone` for example 15326 15327 5. Click the button `Create App` 15328 15329 5. Fill `Redirect URIs` as `http://localhost:53682/` 15330 15331 6. Find the `App key` and `App secret` Use these values in rclone config to add a new remote or edit an existing remote. 15332 15333 FTP 15334 ------------------------------ 15335 15336 FTP is the File Transfer Protocol. FTP support is provided using the 15337 [github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp) 15338 package. 15339 15340 Here is an example of making an FTP configuration. First run 15341 15342 rclone config 15343 15344 This will guide you through an interactive setup process. An FTP remote only 15345 needs a host together with and a username and a password. With anonymous FTP 15346 server, you will need to use `anonymous` as username and your email address as 15347 the password. 15348 15349 ``` 15350 No remotes found - make a new one 15351 n) New remote 15352 r) Rename remote 15353 c) Copy remote 15354 s) Set configuration password 15355 q) Quit config 15356 n/r/c/s/q> n 15357 name> remote 15358 Type of storage to configure. 15359 Enter a string value. Press Enter for the default (""). 15360 Choose a number from below, or type in your own value 15361 [snip] 15362 XX / FTP Connection 15363 \ "ftp" 15364 [snip] 15365 Storage> ftp 15366 ** See help for ftp backend at: https://rclone.org/ftp/ ** 15367 15368 FTP host to connect to 15369 Enter a string value. Press Enter for the default (""). 15370 Choose a number from below, or type in your own value 15371 1 / Connect to ftp.example.com 15372 \ "ftp.example.com" 15373 host> ftp.example.com 15374 FTP username, leave blank for current username, ncw 15375 Enter a string value. Press Enter for the default (""). 15376 user> 15377 FTP port, leave blank to use default (21) 15378 Enter a string value. Press Enter for the default (""). 15379 port> 15380 FTP password 15381 y) Yes type in my own password 15382 g) Generate random password 15383 y/g> y 15384 Enter the password: 15385 password: 15386 Confirm the password: 15387 password: 15388 Use FTP over TLS (Implicit) 15389 Enter a boolean value (true or false). Press Enter for the default ("false"). 15390 tls> 15391 Remote config 15392 -------------------- 15393 [remote] 15394 type = ftp 15395 host = ftp.example.com 15396 pass = *** ENCRYPTED *** 15397 -------------------- 15398 y) Yes this is OK 15399 e) Edit this remote 15400 d) Delete this remote 15401 y/e/d> y 15402 ``` 15403 15404 This remote is called `remote` and can now be used like this 15405 15406 See all directories in the home directory 15407 15408 rclone lsd remote: 15409 15410 Make a new directory 15411 15412 rclone mkdir remote:path/to/directory 15413 15414 List the contents of a directory 15415 15416 rclone ls remote:path/to/directory 15417 15418 Sync `/home/local/directory` to the remote directory, deleting any 15419 excess files in the directory. 15420 15421 rclone sync /home/local/directory remote:directory 15422 15423 ### Modified time ### 15424 15425 FTP does not support modified times. Any times you see on the server 15426 will be time of upload. 15427 15428 ### Checksums ### 15429 15430 FTP does not support any checksums. 15431 15432 ### Usage without a config file ### 15433 15434 An example how to use the ftp remote without a config file: 15435 15436 rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=`rclone obscure dummy` 15437 15438 #### Restricted filename characters 15439 15440 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 15441 the following characters are also replaced: 15442 15443 File names can also not end with the following characters. 15444 These only get replaced if they are the last character in the name: 15445 15446 | Character | Value | Replacement | 15447 | --------- |:-----:|:-----------:| 15448 | SP | 0x20 | ␠ | 15449 15450 Note that not all FTP servers can have all characters in file names, for example: 15451 15452 | FTP Server| Forbidden characters | 15453 | --------- |:--------------------:| 15454 | proftpd | `*` | 15455 | pureftpd | `\ [ ]` | 15456 15457 ### Implicit TLS ### 15458 15459 FTP supports implicit FTP over TLS servers (FTPS). This has to be enabled 15460 in the config for the remote. The default FTPS port is `990` so the 15461 port will likely have to be explicitly set in the config for the remote. 15462 15463 15464 ### Standard Options 15465 15466 Here are the standard options specific to ftp (FTP Connection). 15467 15468 #### --ftp-host 15469 15470 FTP host to connect to 15471 15472 - Config: host 15473 - Env Var: RCLONE_FTP_HOST 15474 - Type: string 15475 - Default: "" 15476 - Examples: 15477 - "ftp.example.com" 15478 - Connect to ftp.example.com 15479 15480 #### --ftp-user 15481 15482 FTP username, leave blank for current username, $USER 15483 15484 - Config: user 15485 - Env Var: RCLONE_FTP_USER 15486 - Type: string 15487 - Default: "" 15488 15489 #### --ftp-port 15490 15491 FTP port, leave blank to use default (21) 15492 15493 - Config: port 15494 - Env Var: RCLONE_FTP_PORT 15495 - Type: string 15496 - Default: "" 15497 15498 #### --ftp-pass 15499 15500 FTP password 15501 15502 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 15503 15504 - Config: pass 15505 - Env Var: RCLONE_FTP_PASS 15506 - Type: string 15507 - Default: "" 15508 15509 #### --ftp-tls 15510 15511 Use FTP over TLS (Implicit) 15512 15513 - Config: tls 15514 - Env Var: RCLONE_FTP_TLS 15515 - Type: bool 15516 - Default: false 15517 15518 ### Advanced Options 15519 15520 Here are the advanced options specific to ftp (FTP Connection). 15521 15522 #### --ftp-concurrency 15523 15524 Maximum number of FTP simultaneous connections, 0 for unlimited 15525 15526 - Config: concurrency 15527 - Env Var: RCLONE_FTP_CONCURRENCY 15528 - Type: int 15529 - Default: 0 15530 15531 #### --ftp-no-check-certificate 15532 15533 Do not verify the TLS certificate of the server 15534 15535 - Config: no_check_certificate 15536 - Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE 15537 - Type: bool 15538 - Default: false 15539 15540 #### --ftp-disable-epsv 15541 15542 Disable using EPSV even if server advertises support 15543 15544 - Config: disable_epsv 15545 - Env Var: RCLONE_FTP_DISABLE_EPSV 15546 - Type: bool 15547 - Default: false 15548 15549 #### --ftp-encoding 15550 15551 This sets the encoding for the backend. 15552 15553 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 15554 15555 - Config: encoding 15556 - Env Var: RCLONE_FTP_ENCODING 15557 - Type: MultiEncoder 15558 - Default: Slash,Del,Ctl,RightSpace,Dot 15559 15560 15561 15562 ### Limitations ### 15563 15564 Note that since FTP isn't HTTP based the following flags don't work 15565 with it: `--dump-headers`, `--dump-bodies`, `--dump-auth` 15566 15567 Note that `--timeout` isn't supported (but `--contimeout` is). 15568 15569 Note that `--bind` isn't supported. 15570 15571 FTP could support server side move but doesn't yet. 15572 15573 Note that the ftp backend does not support the `ftp_proxy` environment 15574 variable yet. 15575 15576 Note that while implicit FTP over TLS is supported, 15577 explicit FTP over TLS is not. 15578 15579 Google Cloud Storage 15580 ------------------------------------------------- 15581 15582 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 15583 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 15584 15585 The initial setup for google cloud storage involves getting a token from Google Cloud Storage 15586 which you need to do in your browser. `rclone config` walks you 15587 through it. 15588 15589 Here is an example of how to make a remote called `remote`. First run: 15590 15591 rclone config 15592 15593 This will guide you through an interactive setup process: 15594 15595 ``` 15596 n) New remote 15597 d) Delete remote 15598 q) Quit config 15599 e/n/d/q> n 15600 name> remote 15601 Type of storage to configure. 15602 Choose a number from below, or type in your own value 15603 [snip] 15604 XX / Google Cloud Storage (this is not Google Drive) 15605 \ "google cloud storage" 15606 [snip] 15607 Storage> google cloud storage 15608 Google Application Client Id - leave blank normally. 15609 client_id> 15610 Google Application Client Secret - leave blank normally. 15611 client_secret> 15612 Project number optional - needed only for list/create/delete buckets - see your developer console. 15613 project_number> 12345678 15614 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. 15615 service_account_file> 15616 Access Control List for new objects. 15617 Choose a number from below, or type in your own value 15618 1 / Object owner gets OWNER access, and all Authenticated Users get READER access. 15619 \ "authenticatedRead" 15620 2 / Object owner gets OWNER access, and project team owners get OWNER access. 15621 \ "bucketOwnerFullControl" 15622 3 / Object owner gets OWNER access, and project team owners get READER access. 15623 \ "bucketOwnerRead" 15624 4 / Object owner gets OWNER access [default if left blank]. 15625 \ "private" 15626 5 / Object owner gets OWNER access, and project team members get access according to their roles. 15627 \ "projectPrivate" 15628 6 / Object owner gets OWNER access, and all Users get READER access. 15629 \ "publicRead" 15630 object_acl> 4 15631 Access Control List for new buckets. 15632 Choose a number from below, or type in your own value 15633 1 / Project team owners get OWNER access, and all Authenticated Users get READER access. 15634 \ "authenticatedRead" 15635 2 / Project team owners get OWNER access [default if left blank]. 15636 \ "private" 15637 3 / Project team members get access according to their roles. 15638 \ "projectPrivate" 15639 4 / Project team owners get OWNER access, and all Users get READER access. 15640 \ "publicRead" 15641 5 / Project team owners get OWNER access, and all Users get WRITER access. 15642 \ "publicReadWrite" 15643 bucket_acl> 2 15644 Location for the newly created buckets. 15645 Choose a number from below, or type in your own value 15646 1 / Empty for default location (US). 15647 \ "" 15648 2 / Multi-regional location for Asia. 15649 \ "asia" 15650 3 / Multi-regional location for Europe. 15651 \ "eu" 15652 4 / Multi-regional location for United States. 15653 \ "us" 15654 5 / Taiwan. 15655 \ "asia-east1" 15656 6 / Tokyo. 15657 \ "asia-northeast1" 15658 7 / Singapore. 15659 \ "asia-southeast1" 15660 8 / Sydney. 15661 \ "australia-southeast1" 15662 9 / Belgium. 15663 \ "europe-west1" 15664 10 / London. 15665 \ "europe-west2" 15666 11 / Iowa. 15667 \ "us-central1" 15668 12 / South Carolina. 15669 \ "us-east1" 15670 13 / Northern Virginia. 15671 \ "us-east4" 15672 14 / Oregon. 15673 \ "us-west1" 15674 location> 12 15675 The storage class to use when storing objects in Google Cloud Storage. 15676 Choose a number from below, or type in your own value 15677 1 / Default 15678 \ "" 15679 2 / Multi-regional storage class 15680 \ "MULTI_REGIONAL" 15681 3 / Regional storage class 15682 \ "REGIONAL" 15683 4 / Nearline storage class 15684 \ "NEARLINE" 15685 5 / Coldline storage class 15686 \ "COLDLINE" 15687 6 / Durable reduced availability storage class 15688 \ "DURABLE_REDUCED_AVAILABILITY" 15689 storage_class> 5 15690 Remote config 15691 Use auto config? 15692 * Say Y if not sure 15693 * Say N if you are working on a remote or headless machine or Y didn't work 15694 y) Yes 15695 n) No 15696 y/n> y 15697 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 15698 Log in and authorize rclone for access 15699 Waiting for code... 15700 Got code 15701 -------------------- 15702 [remote] 15703 type = google cloud storage 15704 client_id = 15705 client_secret = 15706 token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} 15707 project_number = 12345678 15708 object_acl = private 15709 bucket_acl = private 15710 -------------------- 15711 y) Yes this is OK 15712 e) Edit this remote 15713 d) Delete this remote 15714 y/e/d> y 15715 ``` 15716 15717 Note that rclone runs a webserver on your local machine to collect the 15718 token as returned from Google if you use auto config mode. This only 15719 runs from the moment it opens your browser to the moment you get back 15720 the verification code. This is on `http://127.0.0.1:53682/` and this 15721 it may require you to unblock it temporarily if you are running a host 15722 firewall, or use manual mode. 15723 15724 This remote is called `remote` and can now be used like this 15725 15726 See all the buckets in your project 15727 15728 rclone lsd remote: 15729 15730 Make a new bucket 15731 15732 rclone mkdir remote:bucket 15733 15734 List the contents of a bucket 15735 15736 rclone ls remote:bucket 15737 15738 Sync `/home/local/directory` to the remote bucket, deleting any excess 15739 files in the bucket. 15740 15741 rclone sync /home/local/directory remote:bucket 15742 15743 ### Service Account support ### 15744 15745 You can set up rclone with Google Cloud Storage in an unattended mode, 15746 i.e. not tied to a specific end-user Google account. This is useful 15747 when you want to synchronise files onto machines that don't have 15748 actively logged-in users, for example build machines. 15749 15750 To get credentials for Google Cloud Platform 15751 [IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts), 15752 please head to the 15753 [Service Account](https://console.cloud.google.com/permissions/serviceaccounts) 15754 section of the Google Developer Console. Service Accounts behave just 15755 like normal `User` permissions in 15756 [Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control), 15757 so you can limit their access (e.g. make them read only). After 15758 creating an account, a JSON file containing the Service Account's 15759 credentials will be downloaded onto your machines. These credentials 15760 are what rclone will use for authentication. 15761 15762 To use a Service Account instead of OAuth2 token flow, enter the path 15763 to your Service Account credentials at the `service_account_file` 15764 prompt and rclone won't use the browser based authentication 15765 flow. If you'd rather stuff the contents of the credentials file into 15766 the rclone config file, you can set `service_account_credentials` with 15767 the actual contents of the file instead, or set the equivalent 15768 environment variable. 15769 15770 ### Application Default Credentials ### 15771 15772 If no other source of credentials is provided, rclone will fall back 15773 to 15774 [Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials) 15775 this is useful both when you already have configured authentication 15776 for your developer account, or in production when running on a google 15777 compute host. Note that if running in docker, you may need to run 15778 additional commands on your google compute machine - 15779 [see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper). 15780 15781 Note that in the case application default credentials are used, there 15782 is no need to explicitly configure a project number. 15783 15784 ### --fast-list ### 15785 15786 This remote supports `--fast-list` which allows you to use fewer 15787 transactions in exchange for more memory. See the [rclone 15788 docs](https://rclone.org/docs/#fast-list) for more details. 15789 15790 ### Custom upload headers ### 15791 15792 You can set custom upload headers with the `--header-upload` 15793 flag. Google Cloud Storage supports the headers as described in the 15794 [working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata) 15795 15796 - Cache-Control 15797 - Content-Disposition 15798 - Content-Encoding 15799 - Content-Language 15800 - Content-Type 15801 - X-Goog-Meta- 15802 15803 Eg `--header-upload "Content-Type text/potato"` 15804 15805 Note that the last of these is for setting custom metadata in the form 15806 `--header-upload "x-goog-meta-key: value"` 15807 15808 ### Modified time ### 15809 15810 Google google cloud storage stores md5sums natively and rclone stores 15811 modification times as metadata on the object, under the "mtime" key in 15812 RFC3339 format accurate to 1ns. 15813 15814 #### Restricted filename characters 15815 15816 | Character | Value | Replacement | 15817 | --------- |:-----:|:-----------:| 15818 | NUL | 0x00 | ␀ | 15819 | LF | 0x0A | ␊ | 15820 | CR | 0x0D | ␍ | 15821 | / | 0x2F | / | 15822 15823 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 15824 as they can't be used in JSON strings. 15825 15826 15827 ### Standard Options 15828 15829 Here are the standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 15830 15831 #### --gcs-client-id 15832 15833 Google Application Client Id 15834 Leave blank normally. 15835 15836 - Config: client_id 15837 - Env Var: RCLONE_GCS_CLIENT_ID 15838 - Type: string 15839 - Default: "" 15840 15841 #### --gcs-client-secret 15842 15843 Google Application Client Secret 15844 Leave blank normally. 15845 15846 - Config: client_secret 15847 - Env Var: RCLONE_GCS_CLIENT_SECRET 15848 - Type: string 15849 - Default: "" 15850 15851 #### --gcs-project-number 15852 15853 Project number. 15854 Optional - needed only for list/create/delete buckets - see your developer console. 15855 15856 - Config: project_number 15857 - Env Var: RCLONE_GCS_PROJECT_NUMBER 15858 - Type: string 15859 - Default: "" 15860 15861 #### --gcs-service-account-file 15862 15863 Service Account Credentials JSON file path 15864 Leave blank normally. 15865 Needed only if you want use SA instead of interactive login. 15866 15867 - Config: service_account_file 15868 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE 15869 - Type: string 15870 - Default: "" 15871 15872 #### --gcs-service-account-credentials 15873 15874 Service Account Credentials JSON blob 15875 Leave blank normally. 15876 Needed only if you want use SA instead of interactive login. 15877 15878 - Config: service_account_credentials 15879 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS 15880 - Type: string 15881 - Default: "" 15882 15883 #### --gcs-object-acl 15884 15885 Access Control List for new objects. 15886 15887 - Config: object_acl 15888 - Env Var: RCLONE_GCS_OBJECT_ACL 15889 - Type: string 15890 - Default: "" 15891 - Examples: 15892 - "authenticatedRead" 15893 - Object owner gets OWNER access, and all Authenticated Users get READER access. 15894 - "bucketOwnerFullControl" 15895 - Object owner gets OWNER access, and project team owners get OWNER access. 15896 - "bucketOwnerRead" 15897 - Object owner gets OWNER access, and project team owners get READER access. 15898 - "private" 15899 - Object owner gets OWNER access [default if left blank]. 15900 - "projectPrivate" 15901 - Object owner gets OWNER access, and project team members get access according to their roles. 15902 - "publicRead" 15903 - Object owner gets OWNER access, and all Users get READER access. 15904 15905 #### --gcs-bucket-acl 15906 15907 Access Control List for new buckets. 15908 15909 - Config: bucket_acl 15910 - Env Var: RCLONE_GCS_BUCKET_ACL 15911 - Type: string 15912 - Default: "" 15913 - Examples: 15914 - "authenticatedRead" 15915 - Project team owners get OWNER access, and all Authenticated Users get READER access. 15916 - "private" 15917 - Project team owners get OWNER access [default if left blank]. 15918 - "projectPrivate" 15919 - Project team members get access according to their roles. 15920 - "publicRead" 15921 - Project team owners get OWNER access, and all Users get READER access. 15922 - "publicReadWrite" 15923 - Project team owners get OWNER access, and all Users get WRITER access. 15924 15925 #### --gcs-bucket-policy-only 15926 15927 Access checks should use bucket-level IAM policies. 15928 15929 If you want to upload objects to a bucket with Bucket Policy Only set 15930 then you will need to set this. 15931 15932 When it is set, rclone: 15933 15934 - ignores ACLs set on buckets 15935 - ignores ACLs set on objects 15936 - creates buckets with Bucket Policy Only set 15937 15938 Docs: https://cloud.google.com/storage/docs/bucket-policy-only 15939 15940 15941 - Config: bucket_policy_only 15942 - Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY 15943 - Type: bool 15944 - Default: false 15945 15946 #### --gcs-location 15947 15948 Location for the newly created buckets. 15949 15950 - Config: location 15951 - Env Var: RCLONE_GCS_LOCATION 15952 - Type: string 15953 - Default: "" 15954 - Examples: 15955 - "" 15956 - Empty for default location (US). 15957 - "asia" 15958 - Multi-regional location for Asia. 15959 - "eu" 15960 - Multi-regional location for Europe. 15961 - "us" 15962 - Multi-regional location for United States. 15963 - "asia-east1" 15964 - Taiwan. 15965 - "asia-east2" 15966 - Hong Kong. 15967 - "asia-northeast1" 15968 - Tokyo. 15969 - "asia-south1" 15970 - Mumbai. 15971 - "asia-southeast1" 15972 - Singapore. 15973 - "australia-southeast1" 15974 - Sydney. 15975 - "europe-north1" 15976 - Finland. 15977 - "europe-west1" 15978 - Belgium. 15979 - "europe-west2" 15980 - London. 15981 - "europe-west3" 15982 - Frankfurt. 15983 - "europe-west4" 15984 - Netherlands. 15985 - "us-central1" 15986 - Iowa. 15987 - "us-east1" 15988 - South Carolina. 15989 - "us-east4" 15990 - Northern Virginia. 15991 - "us-west1" 15992 - Oregon. 15993 - "us-west2" 15994 - California. 15995 15996 #### --gcs-storage-class 15997 15998 The storage class to use when storing objects in Google Cloud Storage. 15999 16000 - Config: storage_class 16001 - Env Var: RCLONE_GCS_STORAGE_CLASS 16002 - Type: string 16003 - Default: "" 16004 - Examples: 16005 - "" 16006 - Default 16007 - "MULTI_REGIONAL" 16008 - Multi-regional storage class 16009 - "REGIONAL" 16010 - Regional storage class 16011 - "NEARLINE" 16012 - Nearline storage class 16013 - "COLDLINE" 16014 - Coldline storage class 16015 - "ARCHIVE" 16016 - Archive storage class 16017 - "DURABLE_REDUCED_AVAILABILITY" 16018 - Durable reduced availability storage class 16019 16020 ### Advanced Options 16021 16022 Here are the advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 16023 16024 #### --gcs-encoding 16025 16026 This sets the encoding for the backend. 16027 16028 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 16029 16030 - Config: encoding 16031 - Env Var: RCLONE_GCS_ENCODING 16032 - Type: MultiEncoder 16033 - Default: Slash,CrLf,InvalidUtf8,Dot 16034 16035 16036 16037 Google Drive 16038 ----------------------------------------- 16039 16040 Paths are specified as `drive:path` 16041 16042 Drive paths may be as deep as required, eg `drive:directory/subdirectory`. 16043 16044 The initial setup for drive involves getting a token from Google drive 16045 which you need to do in your browser. `rclone config` walks you 16046 through it. 16047 16048 Here is an example of how to make a remote called `remote`. First run: 16049 16050 rclone config 16051 16052 This will guide you through an interactive setup process: 16053 16054 ``` 16055 No remotes found - make a new one 16056 n) New remote 16057 r) Rename remote 16058 c) Copy remote 16059 s) Set configuration password 16060 q) Quit config 16061 n/r/c/s/q> n 16062 name> remote 16063 Type of storage to configure. 16064 Choose a number from below, or type in your own value 16065 [snip] 16066 XX / Google Drive 16067 \ "drive" 16068 [snip] 16069 Storage> drive 16070 Google Application Client Id - leave blank normally. 16071 client_id> 16072 Google Application Client Secret - leave blank normally. 16073 client_secret> 16074 Scope that rclone should use when requesting access from drive. 16075 Choose a number from below, or type in your own value 16076 1 / Full access all files, excluding Application Data Folder. 16077 \ "drive" 16078 2 / Read-only access to file metadata and file contents. 16079 \ "drive.readonly" 16080 / Access to files created by rclone only. 16081 3 | These are visible in the drive website. 16082 | File authorization is revoked when the user deauthorizes the app. 16083 \ "drive.file" 16084 / Allows read and write access to the Application Data folder. 16085 4 | This is not visible in the drive website. 16086 \ "drive.appfolder" 16087 / Allows read-only access to file metadata but 16088 5 | does not allow any access to read or download file content. 16089 \ "drive.metadata.readonly" 16090 scope> 1 16091 ID of the root folder - leave blank normally. Fill in to access "Computers" folders. (see docs). 16092 root_folder_id> 16093 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. 16094 service_account_file> 16095 Remote config 16096 Use auto config? 16097 * Say Y if not sure 16098 * Say N if you are working on a remote or headless machine or Y didn't work 16099 y) Yes 16100 n) No 16101 y/n> y 16102 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 16103 Log in and authorize rclone for access 16104 Waiting for code... 16105 Got code 16106 Configure this as a team drive? 16107 y) Yes 16108 n) No 16109 y/n> n 16110 -------------------- 16111 [remote] 16112 client_id = 16113 client_secret = 16114 scope = drive 16115 root_folder_id = 16116 service_account_file = 16117 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} 16118 -------------------- 16119 y) Yes this is OK 16120 e) Edit this remote 16121 d) Delete this remote 16122 y/e/d> y 16123 ``` 16124 16125 Note that rclone runs a webserver on your local machine to collect the 16126 token as returned from Google if you use auto config mode. This only 16127 runs from the moment it opens your browser to the moment you get back 16128 the verification code. This is on `http://127.0.0.1:53682/` and this 16129 it may require you to unblock it temporarily if you are running a host 16130 firewall, or use manual mode. 16131 16132 You can then use it like this, 16133 16134 List directories in top level of your drive 16135 16136 rclone lsd remote: 16137 16138 List all the files in your drive 16139 16140 rclone ls remote: 16141 16142 To copy a local directory to a drive directory called backup 16143 16144 rclone copy /home/source remote:backup 16145 16146 ### Scopes ### 16147 16148 Rclone allows you to select which scope you would like for rclone to 16149 use. This changes what type of token is granted to rclone. [The 16150 scopes are defined 16151 here](https://developers.google.com/drive/v3/web/about-auth). 16152 16153 The scope are 16154 16155 #### drive #### 16156 16157 This is the default scope and allows full access to all files, except 16158 for the Application Data Folder (see below). 16159 16160 Choose this one if you aren't sure. 16161 16162 #### drive.readonly #### 16163 16164 This allows read only access to all files. Files may be listed and 16165 downloaded but not uploaded, renamed or deleted. 16166 16167 #### drive.file #### 16168 16169 With this scope rclone can read/view/modify only those files and 16170 folders it creates. 16171 16172 So if you uploaded files to drive via the web interface (or any other 16173 means) they will not be visible to rclone. 16174 16175 This can be useful if you are using rclone to backup data and you want 16176 to be sure confidential data on your drive is not visible to rclone. 16177 16178 Files created with this scope are visible in the web interface. 16179 16180 #### drive.appfolder #### 16181 16182 This gives rclone its own private area to store files. Rclone will 16183 not be able to see any other files on your drive and you won't be able 16184 to see rclone's files from the web interface either. 16185 16186 #### drive.metadata.readonly #### 16187 16188 This allows read only access to file names only. It does not allow 16189 rclone to download or upload data, or rename or delete files or 16190 directories. 16191 16192 ### Root folder ID ### 16193 16194 You can set the `root_folder_id` for rclone. This is the directory 16195 (identified by its `Folder ID`) that rclone considers to be the root 16196 of your drive. 16197 16198 Normally you will leave this blank and rclone will determine the 16199 correct root to use itself. 16200 16201 However you can set this to restrict rclone to a specific folder 16202 hierarchy or to access data within the "Computers" tab on the drive 16203 web interface (where files from Google's Backup and Sync desktop 16204 program go). 16205 16206 In order to do this you will have to find the `Folder ID` of the 16207 directory you wish rclone to display. This will be the last segment 16208 of the URL when you open the relevant folder in the drive web 16209 interface. 16210 16211 So if the folder you want rclone to use has a URL which looks like 16212 `https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` 16213 in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as 16214 the `root_folder_id` in the config. 16215 16216 **NB** folders under the "Computers" tab seem to be read only (drive 16217 gives a 500 error) when using rclone. 16218 16219 There doesn't appear to be an API to discover the folder IDs of the 16220 "Computers" tab - please contact us if you know otherwise! 16221 16222 Note also that rclone can't access any data under the "Backups" tab on 16223 the google drive web interface yet. 16224 16225 ### Service Account support ### 16226 16227 You can set up rclone with Google Drive in an unattended mode, 16228 i.e. not tied to a specific end-user Google account. This is useful 16229 when you want to synchronise files onto machines that don't have 16230 actively logged-in users, for example build machines. 16231 16232 To use a Service Account instead of OAuth2 token flow, enter the path 16233 to your Service Account credentials at the `service_account_file` 16234 prompt during `rclone config` and rclone won't use the browser based 16235 authentication flow. If you'd rather stuff the contents of the 16236 credentials file into the rclone config file, you can set 16237 `service_account_credentials` with the actual contents of the file 16238 instead, or set the equivalent environment variable. 16239 16240 #### Use case - Google Apps/G-suite account and individual Drive #### 16241 16242 Let's say that you are the administrator of a Google Apps (old) or 16243 G-suite account. 16244 The goal is to store data on an individual's Drive account, who IS 16245 a member of the domain. 16246 We'll call the domain **example.com**, and the user 16247 **foo@example.com**. 16248 16249 There's a few steps we need to go through to accomplish this: 16250 16251 ##### 1. Create a service account for example.com ##### 16252 - To create a service account and obtain its credentials, go to the 16253 [Google Developer Console](https://console.developers.google.com). 16254 - You must have a project - create one if you don't. 16255 - Then go to "IAM & admin" -> "Service Accounts". 16256 - Use the "Create Credentials" button. Fill in "Service account name" 16257 with something that identifies your client. "Role" can be empty. 16258 - Tick "Furnish a new private key" - select "Key type JSON". 16259 - Tick "Enable G Suite Domain-wide Delegation". This option makes 16260 "impersonation" possible, as documented here: 16261 [Delegating domain-wide authority to the service account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority) 16262 - These credentials are what rclone will use for authentication. 16263 If you ever need to remove access, press the "Delete service 16264 account key" button. 16265 16266 ##### 2. Allowing API access to example.com Google Drive ##### 16267 - Go to example.com's admin console 16268 - Go into "Security" (or use the search bar) 16269 - Select "Show more" and then "Advanced settings" 16270 - Select "Manage API client access" in the "Authentication" section 16271 - In the "Client Name" field enter the service account's 16272 "Client ID" - this can be found in the Developer Console under 16273 "IAM & Admin" -> "Service Accounts", then "View Client ID" for 16274 the newly created service account. 16275 It is a ~21 character numerical string. 16276 - In the next field, "One or More API Scopes", enter 16277 `https://www.googleapis.com/auth/drive` 16278 to grant access to Google Drive specifically. 16279 16280 ##### 3. Configure rclone, assuming a new install ##### 16281 16282 ``` 16283 rclone config 16284 16285 n/s/q> n # New 16286 name>gdrive # Gdrive is an example name 16287 Storage> # Select the number shown for Google Drive 16288 client_id> # Can be left blank 16289 client_secret> # Can be left blank 16290 scope> # Select your scope, 1 for example 16291 root_folder_id> # Can be left blank 16292 service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! 16293 y/n> # Auto config, y 16294 16295 ``` 16296 16297 ##### 4. Verify that it's working ##### 16298 - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup` 16299 - The arguments do: 16300 - `-v` - verbose logging 16301 - `--drive-impersonate foo@example.com` - this is what does 16302 the magic, pretending to be user foo. 16303 - `lsf` - list files in a parsing friendly way 16304 - `gdrive:backup` - use the remote called gdrive, work in 16305 the folder named backup. 16306 16307 ### Team drives ### 16308 16309 If you want to configure the remote to point to a Google Team Drive 16310 then answer `y` to the question `Configure this as a team drive?`. 16311 16312 This will fetch the list of Team Drives from google and allow you to 16313 configure which one you want to use. You can also type in a team 16314 drive ID if you prefer. 16315 16316 For example: 16317 16318 ``` 16319 Configure this as a team drive? 16320 y) Yes 16321 n) No 16322 y/n> y 16323 Fetching team drive list... 16324 Choose a number from below, or type in your own value 16325 1 / Rclone Test 16326 \ "xxxxxxxxxxxxxxxxxxxx" 16327 2 / Rclone Test 2 16328 \ "yyyyyyyyyyyyyyyyyyyy" 16329 3 / Rclone Test 3 16330 \ "zzzzzzzzzzzzzzzzzzzz" 16331 Enter a Team Drive ID> 1 16332 -------------------- 16333 [remote] 16334 client_id = 16335 client_secret = 16336 token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} 16337 team_drive = xxxxxxxxxxxxxxxxxxxx 16338 -------------------- 16339 y) Yes this is OK 16340 e) Edit this remote 16341 d) Delete this remote 16342 y/e/d> y 16343 ``` 16344 16345 ### --fast-list ### 16346 16347 This remote supports `--fast-list` which allows you to use fewer 16348 transactions in exchange for more memory. See the [rclone 16349 docs](https://rclone.org/docs/#fast-list) for more details. 16350 16351 It does this by combining multiple `list` calls into a single API request. 16352 16353 This works by combining many `'%s' in parents` filters into one expression. 16354 To list the contents of directories a, b and c, the following requests will be send by the regular `List` function: 16355 ``` 16356 trashed=false and 'a' in parents 16357 trashed=false and 'b' in parents 16358 trashed=false and 'c' in parents 16359 ``` 16360 These can now be combined into a single request: 16361 ``` 16362 trashed=false and ('a' in parents or 'b' in parents or 'c' in parents) 16363 ``` 16364 16365 The implementation of `ListR` will put up to 50 `parents` filters into one request. 16366 It will use the `--checkers` value to specify the number of requests to run in parallel. 16367 16368 In tests, these batch requests were up to 20x faster than the regular method. 16369 Running the following command against different sized folders gives: 16370 ``` 16371 rclone lsjson -vv -R --checkers=6 gdrive:folder 16372 ``` 16373 16374 small folder (220 directories, 700 files): 16375 16376 - without `--fast-list`: 38s 16377 - with `--fast-list`: 10s 16378 16379 large folder (10600 directories, 39000 files): 16380 16381 - without `--fast-list`: 22:05 min 16382 - with `--fast-list`: 58s 16383 16384 ### Modified time ### 16385 16386 Google drive stores modification times accurate to 1 ms. 16387 16388 #### Restricted filename characters 16389 16390 Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), 16391 as they can't be used in JSON strings. 16392 16393 In contrast to other backends, `/` can also be used in names and `.` 16394 or `..` are valid names. 16395 16396 ### Revisions ### 16397 16398 Google drive stores revisions of files. When you upload a change to 16399 an existing file to google drive using rclone it will create a new 16400 revision of that file. 16401 16402 Revisions follow the standard google policy which at time of writing 16403 was 16404 16405 * They are deleted after 30 days or 100 revisions (whatever comes first). 16406 * They do not count towards a user storage quota. 16407 16408 ### Deleting files ### 16409 16410 By default rclone will send all files to the trash when deleting 16411 files. If deleting them permanently is required then use the 16412 `--drive-use-trash=false` flag, or set the equivalent environment 16413 variable. 16414 16415 ### Shortcuts ### 16416 16417 In March 2020 Google introduced a new feature in Google Drive called 16418 [drive shortcuts](https://support.google.com/drive/answer/9700156) 16419 ([API](https://developers.google.com/drive/api/v3/shortcuts)). These 16420 will (by September 2020) [replace the ability for files or folders to 16421 be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models). 16422 16423 Shortcuts are files that link to other files on Google Drive somewhat 16424 like a symlink in unix, except they point to the underlying file data 16425 (eg the inode in unix terms) so they don't break if the source is 16426 renamed or moved about. 16427 16428 Be default rclone treats these as follows. 16429 16430 For shortcuts pointing to files: 16431 16432 - When listing a file shortcut appears as the destination file. 16433 - When downloading the contents of the destination file is downloaded. 16434 - When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut. 16435 - When server side moving (renaming) the shortcut is renamed, not the destination file. 16436 - When server side copying the shortcut is copied, not the contents of the shortcut. 16437 - When deleting the shortcut is deleted not the linked file. 16438 - When setting the modification time, the modification time of the linked file will be set. 16439 16440 For shortcuts pointing to folders: 16441 16442 - When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders) 16443 - When downloading the contents of the linked folder and sub contents are downloaded 16444 - When uploading to a shortcut folder the file will be placed in the linked folder 16445 - When server side moving (renaming) the shortcut is renamed, not the destination folder 16446 - When server side copying the contents of the linked folder is copied, not the shortcut. 16447 - When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder. 16448 - **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted. 16449 16450 The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts. 16451 16452 Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag 16453 or the corresponding `skip_shortcuts` configuration setting. 16454 16455 ### Emptying trash ### 16456 16457 If you wish to empty your trash you can use the `rclone cleanup remote:` 16458 command which will permanently delete all your trashed files. This command 16459 does not take any path arguments. 16460 16461 Note that Google Drive takes some time (minutes to days) to empty the 16462 trash even though the command returns within a few seconds. No output 16463 is echoed, so there will be no confirmation even using -v or -vv. 16464 16465 ### Quota information ### 16466 16467 To view your current quota you can use the `rclone about remote:` 16468 command which will display your usage limit (quota), the usage in Google 16469 Drive, the size of all files in the Trash and the space used by other 16470 Google services such as Gmail. This command does not take any path 16471 arguments. 16472 16473 #### Import/Export of google documents #### 16474 16475 Google documents can be exported from and uploaded to Google Drive. 16476 16477 When rclone downloads a Google doc it chooses a format to download 16478 depending upon the `--drive-export-formats` setting. 16479 By default the export formats are `docx,xlsx,pptx,svg` which are a 16480 sensible default for an editable document. 16481 16482 When choosing a format, rclone runs down the list provided in order 16483 and chooses the first file format the doc can be exported as from the 16484 list. If the file can't be exported to a format on the formats list, 16485 then rclone will choose a format from the default list. 16486 16487 If you prefer an archive copy then you might use `--drive-export-formats 16488 pdf`, or if you prefer openoffice/libreoffice formats you might use 16489 `--drive-export-formats ods,odt,odp`. 16490 16491 Note that rclone adds the extension to the google doc, so if it is 16492 called `My Spreadsheet` on google docs, it will be exported as `My 16493 Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc. 16494 16495 When importing files into Google Drive, rclone will convert all 16496 files with an extension in `--drive-import-formats` to their 16497 associated document type. 16498 rclone will not convert any files by default, since the conversion 16499 is lossy process. 16500 16501 The conversion must result in a file with the same extension when 16502 the `--drive-export-formats` rules are applied to the uploaded document. 16503 16504 Here are some examples for allowed and prohibited conversions. 16505 16506 | export-formats | import-formats | Upload Ext | Document Ext | Allowed | 16507 | -------------- | -------------- | ---------- | ------------ | ------- | 16508 | odt | odt | odt | odt | Yes | 16509 | odt | docx,odt | odt | odt | Yes | 16510 | | docx | docx | docx | Yes | 16511 | | odt | odt | docx | No | 16512 | odt,docx | docx,odt | docx | odt | No | 16513 | docx,odt | docx,odt | docx | docx | Yes | 16514 | docx,odt | docx,odt | odt | docx | No | 16515 16516 This limitation can be disabled by specifying `--drive-allow-import-name-change`. 16517 When using this flag, rclone can convert multiple files types resulting 16518 in the same document type at once, eg with `--drive-import-formats docx,odt,txt`, 16519 all files having these extension would result in a document represented as a docx file. 16520 This brings the additional risk of overwriting a document, if multiple files 16521 have the same stem. Many rclone operations will not handle this name change 16522 in any way. They assume an equal name when copying files and might copy the 16523 file again or delete them when the name changes. 16524 16525 Here are the possible export extensions with their corresponding mime types. 16526 Most of these can also be used for importing, but there more that are not 16527 listed here. Some of these additional ones might only be available when 16528 the operating system provides the correct MIME type entries. 16529 16530 This list can be changed by Google Drive at any time and might not 16531 represent the currently available conversions. 16532 16533 | Extension | Mime Type | Description | 16534 | --------- |-----------| ------------| 16535 | csv | text/csv | Standard CSV format for Spreadsheets | 16536 | docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document | 16537 | epub | application/epub+zip | E-book format | 16538 | html | text/html | An HTML Document | 16539 | jpg | image/jpeg | A JPEG Image File | 16540 | json | application/vnd.google-apps.script+json | JSON Text Format | 16541 | odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation | 16542 | ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 16543 | ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 16544 | odt | application/vnd.oasis.opendocument.text | Openoffice Document | 16545 | pdf | application/pdf | Adobe PDF Format | 16546 | png | image/png | PNG Image Format| 16547 | pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint | 16548 | rtf | application/rtf | Rich Text Format | 16549 | svg | image/svg+xml | Scalable Vector Graphics Format | 16550 | tsv | text/tab-separated-values | Standard TSV format for spreadsheets | 16551 | txt | text/plain | Plain Text | 16552 | xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet | 16553 | zip | application/zip | A ZIP file of HTML, Images CSS | 16554 16555 Google documents can also be exported as link files. These files will 16556 open a browser window for the Google Docs website of that document 16557 when opened. The link file extension has to be specified as a 16558 `--drive-export-formats` parameter. They will match all available 16559 Google Documents. 16560 16561 | Extension | Description | OS Support | 16562 | --------- | ----------- | ---------- | 16563 | desktop | freedesktop.org specified desktop entry | Linux | 16564 | link.html | An HTML Document with a redirect | All | 16565 | url | INI style link file | macOS, Windows | 16566 | webloc | macOS specific XML format | macOS | 16567 16568 16569 ### Standard Options 16570 16571 Here are the standard options specific to drive (Google Drive). 16572 16573 #### --drive-client-id 16574 16575 Google Application Client Id 16576 Setting your own is recommended. 16577 See https://rclone.org/drive/#making-your-own-client-id for how to create your own. 16578 If you leave this blank, it will use an internal key which is low performance. 16579 16580 - Config: client_id 16581 - Env Var: RCLONE_DRIVE_CLIENT_ID 16582 - Type: string 16583 - Default: "" 16584 16585 #### --drive-client-secret 16586 16587 Google Application Client Secret 16588 Setting your own is recommended. 16589 16590 - Config: client_secret 16591 - Env Var: RCLONE_DRIVE_CLIENT_SECRET 16592 - Type: string 16593 - Default: "" 16594 16595 #### --drive-scope 16596 16597 Scope that rclone should use when requesting access from drive. 16598 16599 - Config: scope 16600 - Env Var: RCLONE_DRIVE_SCOPE 16601 - Type: string 16602 - Default: "" 16603 - Examples: 16604 - "drive" 16605 - Full access all files, excluding Application Data Folder. 16606 - "drive.readonly" 16607 - Read-only access to file metadata and file contents. 16608 - "drive.file" 16609 - Access to files created by rclone only. 16610 - These are visible in the drive website. 16611 - File authorization is revoked when the user deauthorizes the app. 16612 - "drive.appfolder" 16613 - Allows read and write access to the Application Data folder. 16614 - This is not visible in the drive website. 16615 - "drive.metadata.readonly" 16616 - Allows read-only access to file metadata but 16617 - does not allow any access to read or download file content. 16618 16619 #### --drive-root-folder-id 16620 16621 ID of the root folder 16622 Leave blank normally. 16623 16624 Fill in to access "Computers" folders (see docs), or for rclone to use 16625 a non root folder as its starting point. 16626 16627 Note that if this is blank, the first time rclone runs it will fill it 16628 in with the ID of the root folder. 16629 16630 16631 - Config: root_folder_id 16632 - Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID 16633 - Type: string 16634 - Default: "" 16635 16636 #### --drive-service-account-file 16637 16638 Service Account Credentials JSON file path 16639 Leave blank normally. 16640 Needed only if you want use SA instead of interactive login. 16641 16642 - Config: service_account_file 16643 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE 16644 - Type: string 16645 - Default: "" 16646 16647 ### Advanced Options 16648 16649 Here are the advanced options specific to drive (Google Drive). 16650 16651 #### --drive-service-account-credentials 16652 16653 Service Account Credentials JSON blob 16654 Leave blank normally. 16655 Needed only if you want use SA instead of interactive login. 16656 16657 - Config: service_account_credentials 16658 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS 16659 - Type: string 16660 - Default: "" 16661 16662 #### --drive-team-drive 16663 16664 ID of the Team Drive 16665 16666 - Config: team_drive 16667 - Env Var: RCLONE_DRIVE_TEAM_DRIVE 16668 - Type: string 16669 - Default: "" 16670 16671 #### --drive-auth-owner-only 16672 16673 Only consider files owned by the authenticated user. 16674 16675 - Config: auth_owner_only 16676 - Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY 16677 - Type: bool 16678 - Default: false 16679 16680 #### --drive-use-trash 16681 16682 Send files to the trash instead of deleting permanently. 16683 Defaults to true, namely sending files to the trash. 16684 Use `--drive-use-trash=false` to delete files permanently instead. 16685 16686 - Config: use_trash 16687 - Env Var: RCLONE_DRIVE_USE_TRASH 16688 - Type: bool 16689 - Default: true 16690 16691 #### --drive-skip-gdocs 16692 16693 Skip google documents in all listings. 16694 If given, gdocs practically become invisible to rclone. 16695 16696 - Config: skip_gdocs 16697 - Env Var: RCLONE_DRIVE_SKIP_GDOCS 16698 - Type: bool 16699 - Default: false 16700 16701 #### --drive-skip-checksum-gphotos 16702 16703 Skip MD5 checksum on Google photos and videos only. 16704 16705 Use this if you get checksum errors when transferring Google photos or 16706 videos. 16707 16708 Setting this flag will cause Google photos and videos to return a 16709 blank MD5 checksum. 16710 16711 Google photos are identified by being in the "photos" space. 16712 16713 Corrupted checksums are caused by Google modifying the image/video but 16714 not updating the checksum. 16715 16716 - Config: skip_checksum_gphotos 16717 - Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS 16718 - Type: bool 16719 - Default: false 16720 16721 #### --drive-shared-with-me 16722 16723 Only show files that are shared with me. 16724 16725 Instructs rclone to operate on your "Shared with me" folder (where 16726 Google Drive lets you access the files and folders others have shared 16727 with you). 16728 16729 This works both with the "list" (lsd, lsl, etc) and the "copy" 16730 commands (copy, sync, etc), and with all other commands too. 16731 16732 - Config: shared_with_me 16733 - Env Var: RCLONE_DRIVE_SHARED_WITH_ME 16734 - Type: bool 16735 - Default: false 16736 16737 #### --drive-trashed-only 16738 16739 Only show files that are in the trash. 16740 This will show trashed files in their original directory structure. 16741 16742 - Config: trashed_only 16743 - Env Var: RCLONE_DRIVE_TRASHED_ONLY 16744 - Type: bool 16745 - Default: false 16746 16747 #### --drive-formats 16748 16749 Deprecated: see export_formats 16750 16751 - Config: formats 16752 - Env Var: RCLONE_DRIVE_FORMATS 16753 - Type: string 16754 - Default: "" 16755 16756 #### --drive-export-formats 16757 16758 Comma separated list of preferred formats for downloading Google docs. 16759 16760 - Config: export_formats 16761 - Env Var: RCLONE_DRIVE_EXPORT_FORMATS 16762 - Type: string 16763 - Default: "docx,xlsx,pptx,svg" 16764 16765 #### --drive-import-formats 16766 16767 Comma separated list of preferred formats for uploading Google docs. 16768 16769 - Config: import_formats 16770 - Env Var: RCLONE_DRIVE_IMPORT_FORMATS 16771 - Type: string 16772 - Default: "" 16773 16774 #### --drive-allow-import-name-change 16775 16776 Allow the filetype to change when uploading Google docs (e.g. file.doc to file.docx). This will confuse sync and reupload every time. 16777 16778 - Config: allow_import_name_change 16779 - Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE 16780 - Type: bool 16781 - Default: false 16782 16783 #### --drive-use-created-date 16784 16785 Use file created date instead of modified date., 16786 16787 Useful when downloading data and you want the creation date used in 16788 place of the last modified date. 16789 16790 **WARNING**: This flag may have some unexpected consequences. 16791 16792 When uploading to your drive all files will be overwritten unless they 16793 haven't been modified since their creation. And the inverse will occur 16794 while downloading. This side effect can be avoided by using the 16795 "--checksum" flag. 16796 16797 This feature was implemented to retain photos capture date as recorded 16798 by google photos. You will first need to check the "Create a Google 16799 Photos folder" option in your google drive settings. You can then copy 16800 or move the photos locally and use the date the image was taken 16801 (created) set as the modification date. 16802 16803 - Config: use_created_date 16804 - Env Var: RCLONE_DRIVE_USE_CREATED_DATE 16805 - Type: bool 16806 - Default: false 16807 16808 #### --drive-use-shared-date 16809 16810 Use date file was shared instead of modified date. 16811 16812 Note that, as with "--drive-use-created-date", this flag may have 16813 unexpected consequences when uploading/downloading files. 16814 16815 If both this flag and "--drive-use-created-date" are set, the created 16816 date is used. 16817 16818 - Config: use_shared_date 16819 - Env Var: RCLONE_DRIVE_USE_SHARED_DATE 16820 - Type: bool 16821 - Default: false 16822 16823 #### --drive-list-chunk 16824 16825 Size of listing chunk 100-1000. 0 to disable. 16826 16827 - Config: list_chunk 16828 - Env Var: RCLONE_DRIVE_LIST_CHUNK 16829 - Type: int 16830 - Default: 1000 16831 16832 #### --drive-impersonate 16833 16834 Impersonate this user when using a service account. 16835 16836 Note that if this is used then "root_folder_id" will be ignored. 16837 16838 16839 - Config: impersonate 16840 - Env Var: RCLONE_DRIVE_IMPERSONATE 16841 - Type: string 16842 - Default: "" 16843 16844 #### --drive-alternate-export 16845 16846 Use alternate export URLs for google documents export., 16847 16848 If this option is set this instructs rclone to use an alternate set of 16849 export URLs for drive documents. Users have reported that the 16850 official export URLs can't export large documents, whereas these 16851 unofficial ones can. 16852 16853 See rclone issue [#2243](https://github.com/rclone/rclone/issues/2243) for background, 16854 [this google drive issue](https://issuetracker.google.com/issues/36761333) and 16855 [this helpful post](https://www.labnol.org/internet/direct-links-for-google-drive/28356/). 16856 16857 - Config: alternate_export 16858 - Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT 16859 - Type: bool 16860 - Default: false 16861 16862 #### --drive-upload-cutoff 16863 16864 Cutoff for switching to chunked upload 16865 16866 - Config: upload_cutoff 16867 - Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF 16868 - Type: SizeSuffix 16869 - Default: 8M 16870 16871 #### --drive-chunk-size 16872 16873 Upload chunk size. Must a power of 2 >= 256k. 16874 16875 Making this larger will improve performance, but note that each chunk 16876 is buffered in memory one per transfer. 16877 16878 Reducing this will reduce memory usage but decrease performance. 16879 16880 - Config: chunk_size 16881 - Env Var: RCLONE_DRIVE_CHUNK_SIZE 16882 - Type: SizeSuffix 16883 - Default: 8M 16884 16885 #### --drive-acknowledge-abuse 16886 16887 Set to allow files which return cannotDownloadAbusiveFile to be downloaded. 16888 16889 If downloading a file returns the error "This file has been identified 16890 as malware or spam and cannot be downloaded" with the error code 16891 "cannotDownloadAbusiveFile" then supply this flag to rclone to 16892 indicate you acknowledge the risks of downloading the file and rclone 16893 will download it anyway. 16894 16895 - Config: acknowledge_abuse 16896 - Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE 16897 - Type: bool 16898 - Default: false 16899 16900 #### --drive-keep-revision-forever 16901 16902 Keep new head revision of each file forever. 16903 16904 - Config: keep_revision_forever 16905 - Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER 16906 - Type: bool 16907 - Default: false 16908 16909 #### --drive-size-as-quota 16910 16911 Show sizes as storage quota usage, not actual size. 16912 16913 Show the size of a file as the storage quota used. This is the 16914 current version plus any older versions that have been set to keep 16915 forever. 16916 16917 **WARNING**: This flag may have some unexpected consequences. 16918 16919 It is not recommended to set this flag in your config - the 16920 recommended usage is using the flag form --drive-size-as-quota when 16921 doing rclone ls/lsl/lsf/lsjson/etc only. 16922 16923 If you do use this flag for syncing (not recommended) then you will 16924 need to use --ignore size also. 16925 16926 - Config: size_as_quota 16927 - Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA 16928 - Type: bool 16929 - Default: false 16930 16931 #### --drive-v2-download-min-size 16932 16933 If Object's are greater, use drive v2 API to download. 16934 16935 - Config: v2_download_min_size 16936 - Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE 16937 - Type: SizeSuffix 16938 - Default: off 16939 16940 #### --drive-pacer-min-sleep 16941 16942 Minimum time to sleep between API calls. 16943 16944 - Config: pacer_min_sleep 16945 - Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP 16946 - Type: Duration 16947 - Default: 100ms 16948 16949 #### --drive-pacer-burst 16950 16951 Number of API calls to allow without sleeping. 16952 16953 - Config: pacer_burst 16954 - Env Var: RCLONE_DRIVE_PACER_BURST 16955 - Type: int 16956 - Default: 100 16957 16958 #### --drive-server-side-across-configs 16959 16960 Allow server side operations (eg copy) to work across different drive configs. 16961 16962 This can be useful if you wish to do a server side copy between two 16963 different Google drives. Note that this isn't enabled by default 16964 because it isn't easy to tell if it will work between any two 16965 configurations. 16966 16967 - Config: server_side_across_configs 16968 - Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS 16969 - Type: bool 16970 - Default: false 16971 16972 #### --drive-disable-http2 16973 16974 Disable drive using http2 16975 16976 There is currently an unsolved issue with the google drive backend and 16977 HTTP/2. HTTP/2 is therefore disabled by default for the drive backend 16978 but can be re-enabled here. When the issue is solved this flag will 16979 be removed. 16980 16981 See: https://github.com/rclone/rclone/issues/3631 16982 16983 16984 16985 - Config: disable_http2 16986 - Env Var: RCLONE_DRIVE_DISABLE_HTTP2 16987 - Type: bool 16988 - Default: true 16989 16990 #### --drive-stop-on-upload-limit 16991 16992 Make upload limit errors be fatal 16993 16994 At the time of writing it is only possible to upload 750GB of data to 16995 Google Drive a day (this is an undocumented limit). When this limit is 16996 reached Google Drive produces a slightly different error message. When 16997 this flag is set it causes these errors to be fatal. These will stop 16998 the in-progress sync. 16999 17000 Note that this detection is relying on error message strings which 17001 Google don't document so it may break in the future. 17002 17003 See: https://github.com/rclone/rclone/issues/3857 17004 17005 17006 - Config: stop_on_upload_limit 17007 - Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT 17008 - Type: bool 17009 - Default: false 17010 17011 #### --drive-skip-shortcuts 17012 17013 If set skip shortcut files 17014 17015 Normally rclone dereferences shortcut files making them appear as if 17016 they are the original file (see [the shortcuts section](#shortcuts)). 17017 If this flag is set then rclone will ignore shortcut files completely. 17018 17019 17020 - Config: skip_shortcuts 17021 - Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS 17022 - Type: bool 17023 - Default: false 17024 17025 #### --drive-encoding 17026 17027 This sets the encoding for the backend. 17028 17029 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 17030 17031 - Config: encoding 17032 - Env Var: RCLONE_DRIVE_ENCODING 17033 - Type: MultiEncoder 17034 - Default: InvalidUtf8 17035 17036 ### Backend commands 17037 17038 Here are the commands specific to the drive backend. 17039 17040 Run them with 17041 17042 rclone backend COMMAND remote: 17043 17044 The help below will explain what arguments each command takes. 17045 17046 See [the "rclone backend" command](https://rclone.org/commands/rclone_backend/) for more 17047 info on how to pass options and arguments. 17048 17049 These can be run on a running backend using the rc command 17050 [backend/command](https://rclone.org/rc/#backend/command). 17051 17052 #### get 17053 17054 Get command for fetching the drive config parameters 17055 17056 rclone backend get remote: [options] [<arguments>+] 17057 17058 This is a get command which will be used to fetch the various drive config parameters 17059 17060 Usage Examples: 17061 17062 rclone backend get drive: [-o service_account_file] [-o chunk_size] 17063 rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size] 17064 17065 17066 Options: 17067 17068 - "chunk_size": show the current upload chunk size 17069 - "service_account_file": show the current service account file 17070 17071 #### set 17072 17073 Set command for updating the drive config parameters 17074 17075 rclone backend set remote: [options] [<arguments>+] 17076 17077 This is a set command which will be used to update the various drive config parameters 17078 17079 Usage Examples: 17080 17081 rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864] 17082 rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864] 17083 17084 17085 Options: 17086 17087 - "chunk_size": update the current upload chunk size 17088 - "service_account_file": update the current service account file 17089 17090 #### shortcut 17091 17092 Create shortcuts from files or directories 17093 17094 rclone backend shortcut remote: [options] [<arguments>+] 17095 17096 This command creates shortcuts from files or directories. 17097 17098 Usage: 17099 17100 rclone backend shortcut drive: source_item destination_shortcut 17101 rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut 17102 17103 In the first example this creates a shortcut from the "source_item" 17104 which can be a file or a directory to the "destination_shortcut". The 17105 "source_item" and the "destination_shortcut" should be relative paths 17106 from "drive:" 17107 17108 In the second example this creates a shortcut from the "source_item" 17109 relative to "drive:" to the "destination_shortcut" relative to 17110 "drive2:". This may fail with a permission error if the user 17111 authenticated with "drive2:" can't read files from "drive:". 17112 17113 17114 Options: 17115 17116 - "target": optional target remote for the shortcut destination 17117 17118 17119 17120 ### Limitations ### 17121 17122 Drive has quite a lot of rate limiting. This causes rclone to be 17123 limited to transferring about 2 files per second only. Individual 17124 files may be transferred much faster at 100s of MBytes/s but lots of 17125 small files can take a long time. 17126 17127 Server side copies are also subject to a separate rate limit. If you 17128 see User rate limit exceeded errors, wait at least 24 hours and retry. 17129 You can disable server side copies with `--disable copy` to download 17130 and upload the files if you prefer. 17131 17132 #### Limitations of Google Docs #### 17133 17134 Google docs will appear as size -1 in `rclone ls` and as size 0 in 17135 anything which uses the VFS layer, eg `rclone mount`, `rclone serve`. 17136 17137 This is because rclone can't find out the size of the Google docs 17138 without downloading them. 17139 17140 Google docs will transfer correctly with `rclone sync`, `rclone copy` 17141 etc as rclone knows to ignore the size when doing the transfer. 17142 17143 However an unfortunate consequence of this is that you may not be able 17144 to download Google docs using `rclone mount`. If it doesn't work you 17145 will get a 0 sized file. If you try again the doc may gain its 17146 correct size and be downloadable. Whether it will work on not depends 17147 on the application accessing the mount and the OS you are running - 17148 experiment to find out if it does work for you! 17149 17150 ### Duplicated files ### 17151 17152 Sometimes, for no reason I've been able to track down, drive will 17153 duplicate a file that rclone uploads. Drive unlike all the other 17154 remotes can have duplicated files. 17155 17156 Duplicated files cause problems with the syncing and you will see 17157 messages in the log about duplicates. 17158 17159 Use `rclone dedupe` to fix duplicated files. 17160 17161 Note that this isn't just a problem with rclone, even Google Photos on 17162 Android duplicates files on drive sometimes. 17163 17164 ### Rclone appears to be re-copying files it shouldn't ### 17165 17166 The most likely cause of this is the duplicated file issue above - run 17167 `rclone dedupe` and check your logs for duplicate object or directory 17168 messages. 17169 17170 This can also be caused by a delay/caching on google drive's end when 17171 comparing directory listings. Specifically with team drives used in 17172 combination with --fast-list. Files that were uploaded recently may 17173 not appear on the directory list sent to rclone when using --fast-list. 17174 17175 Waiting a moderate period of time between attempts (estimated to be 17176 approximately 1 hour) and/or not using --fast-list both seem to be 17177 effective in preventing the problem. 17178 17179 ### Making your own client_id ### 17180 17181 When you use rclone with Google drive in its default configuration you 17182 are using rclone's client_id. This is shared between all the rclone 17183 users. There is a global rate limit on the number of queries per 17184 second that each client_id can do set by Google. rclone already has a 17185 high quota and I will continue to make sure it is high enough by 17186 contacting Google. 17187 17188 It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower. 17189 17190 Here is how to create your own Google Drive client ID for rclone: 17191 17192 1. Log into the [Google API 17193 Console](https://console.developers.google.com/) with your Google 17194 account. It doesn't matter what Google account you use. (It need not 17195 be the same account as the Google Drive you want to access) 17196 17197 2. Select a project or create a new project. 17198 17199 3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the 17200 "Google Drive API". 17201 17202 4. Click "Credentials" in the left-side panel (not "Create 17203 credentials", which opens the wizard), then "Create credentials" 17204 17205 5. If you already configured an "Oauth Consent Screen", then skip 17206 to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button 17207 (near the top right corner of the right panel), then select "External" 17208 and click on "CREATE"; on the next screen, enter an "Application name" 17209 ("rclone" is OK) then click on "Save" (all other data is optional). 17210 Click again on "Credentials" on the left panel to go back to the 17211 "Credentials" screen. 17212 17213 (PS: if you are a GSuite user, you could also select "Internal" instead 17214 of "External" above, but this has not been tested/documented so far). 17215 17216 6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen, 17217 then select "OAuth client ID". 17218 17219 7. Choose an application type of "Desktop app" if you using a Google account or "Other" if 17220 you using a GSuite account and click "Create". (the default name is fine) 17221 17222 8. It will show you a client ID and client secret. Use these values 17223 in rclone config to add a new remote or edit an existing remote. 17224 17225 Be aware that, due to the "enhanced security" recently introduced by 17226 Google, you are theoretically expected to "submit your app for verification" 17227 and then wait a few weeks(!) for their response; in practice, you can go right 17228 ahead and use the client ID and client secret with rclone, the only issue will 17229 be a very scary confirmation screen shown when you connect via your browser 17230 for rclone to be able to get its token-id (but as this only happens during 17231 the remote configuration, it's not such a big deal). 17232 17233 (Thanks to @balazer on github for these instructions.) 17234 17235 Google Photos 17236 ------------------------------------------------- 17237 17238 The rclone backend for [Google Photos](https://www.google.com/photos/about/) is 17239 a specialized backend for transferring photos and videos to and from 17240 Google Photos. 17241 17242 **NB** The Google Photos API which rclone uses has quite a few 17243 limitations, so please read the [limitations section](#limitations) 17244 carefully to make sure it is suitable for your use. 17245 17246 ## Configuring Google Photos 17247 17248 The initial setup for google cloud storage involves getting a token from Google Photos 17249 which you need to do in your browser. `rclone config` walks you 17250 through it. 17251 17252 Here is an example of how to make a remote called `remote`. First run: 17253 17254 rclone config 17255 17256 This will guide you through an interactive setup process: 17257 17258 ``` 17259 No remotes found - make a new one 17260 n) New remote 17261 s) Set configuration password 17262 q) Quit config 17263 n/s/q> n 17264 name> remote 17265 Type of storage to configure. 17266 Enter a string value. Press Enter for the default (""). 17267 Choose a number from below, or type in your own value 17268 [snip] 17269 XX / Google Photos 17270 \ "google photos" 17271 [snip] 17272 Storage> google photos 17273 ** See help for google photos backend at: https://rclone.org/googlephotos/ ** 17274 17275 Google Application Client Id 17276 Leave blank normally. 17277 Enter a string value. Press Enter for the default (""). 17278 client_id> 17279 Google Application Client Secret 17280 Leave blank normally. 17281 Enter a string value. Press Enter for the default (""). 17282 client_secret> 17283 Set to make the Google Photos backend read only. 17284 17285 If you choose read only then rclone will only request read only access 17286 to your photos, otherwise rclone will request full access. 17287 Enter a boolean value (true or false). Press Enter for the default ("false"). 17288 read_only> 17289 Edit advanced config? (y/n) 17290 y) Yes 17291 n) No 17292 y/n> n 17293 Remote config 17294 Use auto config? 17295 * Say Y if not sure 17296 * Say N if you are working on a remote or headless machine 17297 y) Yes 17298 n) No 17299 y/n> y 17300 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 17301 Log in and authorize rclone for access 17302 Waiting for code... 17303 Got code 17304 17305 *** IMPORTANT: All media items uploaded to Google Photos with rclone 17306 *** are stored in full resolution at original quality. These uploads 17307 *** will count towards storage in your Google Account. 17308 17309 -------------------- 17310 [remote] 17311 type = google photos 17312 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} 17313 -------------------- 17314 y) Yes this is OK 17315 e) Edit this remote 17316 d) Delete this remote 17317 y/e/d> y 17318 ``` 17319 17320 Note that rclone runs a webserver on your local machine to collect the 17321 token as returned from Google if you use auto config mode. This only 17322 runs from the moment it opens your browser to the moment you get back 17323 the verification code. This is on `http://127.0.0.1:53682/` and this 17324 may require you to unblock it temporarily if you are running a host 17325 firewall, or use manual mode. 17326 17327 This remote is called `remote` and can now be used like this 17328 17329 See all the albums in your photos 17330 17331 rclone lsd remote:album 17332 17333 Make a new album 17334 17335 rclone mkdir remote:album/newAlbum 17336 17337 List the contents of an album 17338 17339 rclone ls remote:album/newAlbum 17340 17341 Sync `/home/local/images` to the Google Photos, removing any excess 17342 files in the album. 17343 17344 rclone sync /home/local/image remote:album/newAlbum 17345 17346 ## Layout 17347 17348 As Google Photos is not a general purpose cloud storage system the 17349 backend is laid out to help you navigate it. 17350 17351 The directories under `media` show different ways of categorizing the 17352 media. Each file will appear multiple times. So if you want to make 17353 a backup of your google photos you might choose to backup 17354 `remote:media/by-month`. (**NB** `remote:media/by-day` is rather slow 17355 at the moment so avoid for syncing.) 17356 17357 Note that all your photos and videos will appear somewhere under 17358 `media`, but they may not appear under `album` unless you've put them 17359 into albums. 17360 17361 ``` 17362 / 17363 - upload 17364 - file1.jpg 17365 - file2.jpg 17366 - ... 17367 - media 17368 - all 17369 - file1.jpg 17370 - file2.jpg 17371 - ... 17372 - by-year 17373 - 2000 17374 - file1.jpg 17375 - ... 17376 - 2001 17377 - file2.jpg 17378 - ... 17379 - ... 17380 - by-month 17381 - 2000 17382 - 2000-01 17383 - file1.jpg 17384 - ... 17385 - 2000-02 17386 - file2.jpg 17387 - ... 17388 - ... 17389 - by-day 17390 - 2000 17391 - 2000-01-01 17392 - file1.jpg 17393 - ... 17394 - 2000-01-02 17395 - file2.jpg 17396 - ... 17397 - ... 17398 - album 17399 - album name 17400 - album name/sub 17401 - shared-album 17402 - album name 17403 - album name/sub 17404 - feature 17405 - favorites 17406 - file1.jpg 17407 - file2.jpg 17408 ``` 17409 17410 There are two writable parts of the tree, the `upload` directory and 17411 sub directories of the `album` directory. 17412 17413 The `upload` directory is for uploading files you don't want to put 17414 into albums. This will be empty to start with and will contain the 17415 files you've uploaded for one rclone session only, becoming empty 17416 again when you restart rclone. The use case for this would be if you 17417 have a load of files you just want to once off dump into Google 17418 Photos. For repeated syncing, uploading to `album` will work better. 17419 17420 Directories within the `album` directory are also writeable and you 17421 may create new directories (albums) under `album`. If you copy files 17422 with a directory hierarchy in there then rclone will create albums 17423 with the `/` character in them. For example if you do 17424 17425 rclone copy /path/to/images remote:album/images 17426 17427 and the images directory contains 17428 17429 ``` 17430 images 17431 - file1.jpg 17432 dir 17433 file2.jpg 17434 dir2 17435 dir3 17436 file3.jpg 17437 ``` 17438 17439 Then rclone will create the following albums with the following files in 17440 17441 - images 17442 - file1.jpg 17443 - images/dir 17444 - file2.jpg 17445 - images/dir2/dir3 17446 - file3.jpg 17447 17448 This means that you can use the `album` path pretty much like a normal 17449 filesystem and it is a good target for repeated syncing. 17450 17451 The `shared-album` directory shows albums shared with you or by you. 17452 This is similar to the Sharing tab in the Google Photos web interface. 17453 17454 ## Limitations 17455 17456 Only images and videos can be uploaded. If you attempt to upload non 17457 videos or images or formats that Google Photos doesn't understand, 17458 rclone will upload the file, then Google Photos will give an error 17459 when it is put turned into a media item. 17460 17461 Note that all media items uploaded to Google Photos through the API 17462 are stored in full resolution at "original quality" and **will** count 17463 towards your storage quota in your Google Account. The API does 17464 **not** offer a way to upload in "high quality" mode.. 17465 17466 ### Downloading Images 17467 17468 When Images are downloaded this strips EXIF location (according to the 17469 docs and my tests). This is a limitation of the Google Photos API and 17470 is covered by [bug #112096115](https://issuetracker.google.com/issues/112096115). 17471 17472 **The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort** 17473 17474 ### Downloading Videos 17475 17476 When videos are downloaded they are downloaded in a really compressed 17477 version of the video compared to downloading it via the Google Photos 17478 web interface. This is covered by [bug #113672044](https://issuetracker.google.com/issues/113672044). 17479 17480 ### Duplicates 17481 17482 If a file name is duplicated in a directory then rclone will add the 17483 file ID into its name. So two files called `file.jpg` would then 17484 appear as `file {123456}.jpg` and `file {ABCDEF}.jpg` (the actual IDs 17485 are a lot longer alas!). 17486 17487 If you upload the same image (with the same binary data) twice then 17488 Google Photos will deduplicate it. However it will retain the 17489 filename from the first upload which may confuse rclone. For example 17490 if you uploaded an image to `upload` then uploaded the same image to 17491 `album/my_album` the filename of the image in `album/my_album` will be 17492 what it was uploaded with initially, not what you uploaded it with to 17493 `album`. In practise this shouldn't cause too many problems. 17494 17495 ### Modified time 17496 17497 The date shown of media in Google Photos is the creation date as 17498 determined by the EXIF information, or the upload date if that is not 17499 known. 17500 17501 This is not changeable by rclone and is not the modification date of 17502 the media on local disk. This means that rclone cannot use the dates 17503 from Google Photos for syncing purposes. 17504 17505 ### Size 17506 17507 The Google Photos API does not return the size of media. This means 17508 that when syncing to Google Photos, rclone can only do a file 17509 existence check. 17510 17511 It is possible to read the size of the media, but this needs an extra 17512 HTTP HEAD request per media item so is **very slow** and uses up a lot of 17513 transactions. This can be enabled with the `--gphotos-read-size` 17514 option or the `read_size = true` config parameter. 17515 17516 If you want to use the backend with `rclone mount` you may need to 17517 enable this flag (depending on your OS and application using the 17518 photos) otherwise you may not be able to read media off the mount. 17519 You'll need to experiment to see if it works for you without the flag. 17520 17521 ### Albums 17522 17523 Rclone can only upload files to albums it created. This is a 17524 [limitation of the Google Photos API](https://developers.google.com/photos/library/guides/manage-albums). 17525 17526 Rclone can remove files it uploaded from albums it created only. 17527 17528 ### Deleting files 17529 17530 Rclone can remove files from albums it created, but note that the 17531 Google Photos API does not allow media to be deleted permanently so 17532 this media will still remain. See [bug #109759781](https://issuetracker.google.com/issues/109759781). 17533 17534 Rclone cannot delete files anywhere except under `album`. 17535 17536 ### Deleting albums 17537 17538 The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733). 17539 17540 17541 ### Standard Options 17542 17543 Here are the standard options specific to google photos (Google Photos). 17544 17545 #### --gphotos-client-id 17546 17547 Google Application Client Id 17548 Leave blank normally. 17549 17550 - Config: client_id 17551 - Env Var: RCLONE_GPHOTOS_CLIENT_ID 17552 - Type: string 17553 - Default: "" 17554 17555 #### --gphotos-client-secret 17556 17557 Google Application Client Secret 17558 Leave blank normally. 17559 17560 - Config: client_secret 17561 - Env Var: RCLONE_GPHOTOS_CLIENT_SECRET 17562 - Type: string 17563 - Default: "" 17564 17565 #### --gphotos-read-only 17566 17567 Set to make the Google Photos backend read only. 17568 17569 If you choose read only then rclone will only request read only access 17570 to your photos, otherwise rclone will request full access. 17571 17572 - Config: read_only 17573 - Env Var: RCLONE_GPHOTOS_READ_ONLY 17574 - Type: bool 17575 - Default: false 17576 17577 ### Advanced Options 17578 17579 Here are the advanced options specific to google photos (Google Photos). 17580 17581 #### --gphotos-read-size 17582 17583 Set to read the size of media items. 17584 17585 Normally rclone does not read the size of media items since this takes 17586 another transaction. This isn't necessary for syncing. However 17587 rclone mount needs to know the size of files in advance of reading 17588 them, so setting this flag when using rclone mount is recommended if 17589 you want to read the media. 17590 17591 - Config: read_size 17592 - Env Var: RCLONE_GPHOTOS_READ_SIZE 17593 - Type: bool 17594 - Default: false 17595 17596 #### --gphotos-start-year 17597 17598 Year limits the photos to be downloaded to those which are uploaded after the given year 17599 17600 - Config: start_year 17601 - Env Var: RCLONE_GPHOTOS_START_YEAR 17602 - Type: int 17603 - Default: 2000 17604 17605 17606 17607 HTTP 17608 ------------------------------------------------- 17609 17610 The HTTP remote is a read only remote for reading files of a 17611 webserver. The webserver should provide file listings which rclone 17612 will read and turn into a remote. This has been tested with common 17613 webservers such as Apache/Nginx/Caddy and will likely work with file 17614 listings from most web servers. (If it doesn't then please file an 17615 issue, or send a pull request!) 17616 17617 Paths are specified as `remote:` or `remote:path/to/dir`. 17618 17619 Here is an example of how to make a remote called `remote`. First 17620 run: 17621 17622 rclone config 17623 17624 This will guide you through an interactive setup process: 17625 17626 ``` 17627 No remotes found - make a new one 17628 n) New remote 17629 s) Set configuration password 17630 q) Quit config 17631 n/s/q> n 17632 name> remote 17633 Type of storage to configure. 17634 Choose a number from below, or type in your own value 17635 [snip] 17636 XX / http Connection 17637 \ "http" 17638 [snip] 17639 Storage> http 17640 URL of http host to connect to 17641 Choose a number from below, or type in your own value 17642 1 / Connect to example.com 17643 \ "https://example.com" 17644 url> https://beta.rclone.org 17645 Remote config 17646 -------------------- 17647 [remote] 17648 url = https://beta.rclone.org 17649 -------------------- 17650 y) Yes this is OK 17651 e) Edit this remote 17652 d) Delete this remote 17653 y/e/d> y 17654 Current remotes: 17655 17656 Name Type 17657 ==== ==== 17658 remote http 17659 17660 e) Edit existing remote 17661 n) New remote 17662 d) Delete remote 17663 r) Rename remote 17664 c) Copy remote 17665 s) Set configuration password 17666 q) Quit config 17667 e/n/d/r/c/s/q> q 17668 ``` 17669 17670 This remote is called `remote` and can now be used like this 17671 17672 See all the top level directories 17673 17674 rclone lsd remote: 17675 17676 List the contents of a directory 17677 17678 rclone ls remote:directory 17679 17680 Sync the remote `directory` to `/home/local/directory`, deleting any excess files. 17681 17682 rclone sync remote:directory /home/local/directory 17683 17684 ### Read only ### 17685 17686 This remote is read only - you can't upload files to an HTTP server. 17687 17688 ### Modified time ### 17689 17690 Most HTTP servers store time accurate to 1 second. 17691 17692 ### Checksum ### 17693 17694 No checksums are stored. 17695 17696 ### Usage without a config file ### 17697 17698 Since the http remote only has one config parameter it is easy to use 17699 without a config file: 17700 17701 rclone lsd --http-url https://beta.rclone.org :http: 17702 17703 17704 ### Standard Options 17705 17706 Here are the standard options specific to http (http Connection). 17707 17708 #### --http-url 17709 17710 URL of http host to connect to 17711 17712 - Config: url 17713 - Env Var: RCLONE_HTTP_URL 17714 - Type: string 17715 - Default: "" 17716 - Examples: 17717 - "https://example.com" 17718 - Connect to example.com 17719 - "https://user:pass@example.com" 17720 - Connect to example.com using a username and password 17721 17722 ### Advanced Options 17723 17724 Here are the advanced options specific to http (http Connection). 17725 17726 #### --http-headers 17727 17728 Set HTTP headers for all transactions 17729 17730 Use this to set additional HTTP headers for all transactions 17731 17732 The input format is comma separated list of key,value pairs. Standard 17733 [CSV encoding](https://godoc.org/encoding/csv) may be used. 17734 17735 For example to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'. 17736 17737 You can set multiple headers, eg '"Cookie","name=value","Authorization","xxx"'. 17738 17739 17740 - Config: headers 17741 - Env Var: RCLONE_HTTP_HEADERS 17742 - Type: CommaSepList 17743 - Default: 17744 17745 #### --http-no-slash 17746 17747 Set this if the site doesn't end directories with / 17748 17749 Use this if your target website does not use / on the end of 17750 directories. 17751 17752 A / on the end of a path is how rclone normally tells the difference 17753 between files and directories. If this flag is set, then rclone will 17754 treat all files with Content-Type: text/html as directories and read 17755 URLs from them rather than downloading them. 17756 17757 Note that this may cause rclone to confuse genuine HTML files with 17758 directories. 17759 17760 - Config: no_slash 17761 - Env Var: RCLONE_HTTP_NO_SLASH 17762 - Type: bool 17763 - Default: false 17764 17765 #### --http-no-head 17766 17767 Don't use HEAD requests to find file sizes in dir listing 17768 17769 If your site is being very slow to load then you can try this option. 17770 Normally rclone does a HEAD request for each potential file in a 17771 directory listing to: 17772 17773 - find its size 17774 - check it really exists 17775 - check to see if it is a directory 17776 17777 If you set this option, rclone will not do the HEAD request. This will mean 17778 17779 - directory listings are much quicker 17780 - rclone won't have the times or sizes of any files 17781 - some files that don't exist may be in the listing 17782 17783 17784 - Config: no_head 17785 - Env Var: RCLONE_HTTP_NO_HEAD 17786 - Type: bool 17787 - Default: false 17788 17789 17790 17791 Hubic 17792 ----------------------------------------- 17793 17794 Paths are specified as `remote:path` 17795 17796 Paths are specified as `remote:container` (or `remote:` for the `lsd` 17797 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 17798 17799 The initial setup for Hubic involves getting a token from Hubic which 17800 you need to do in your browser. `rclone config` walks you through it. 17801 17802 Here is an example of how to make a remote called `remote`. First run: 17803 17804 rclone config 17805 17806 This will guide you through an interactive setup process: 17807 17808 ``` 17809 n) New remote 17810 s) Set configuration password 17811 n/s> n 17812 name> remote 17813 Type of storage to configure. 17814 Choose a number from below, or type in your own value 17815 [snip] 17816 XX / Hubic 17817 \ "hubic" 17818 [snip] 17819 Storage> hubic 17820 Hubic Client Id - leave blank normally. 17821 client_id> 17822 Hubic Client Secret - leave blank normally. 17823 client_secret> 17824 Remote config 17825 Use auto config? 17826 * Say Y if not sure 17827 * Say N if you are working on a remote or headless machine 17828 y) Yes 17829 n) No 17830 y/n> y 17831 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 17832 Log in and authorize rclone for access 17833 Waiting for code... 17834 Got code 17835 -------------------- 17836 [remote] 17837 client_id = 17838 client_secret = 17839 token = {"access_token":"XXXXXX"} 17840 -------------------- 17841 y) Yes this is OK 17842 e) Edit this remote 17843 d) Delete this remote 17844 y/e/d> y 17845 ``` 17846 17847 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 17848 machine with no Internet browser available. 17849 17850 Note that rclone runs a webserver on your local machine to collect the 17851 token as returned from Hubic. This only runs from the moment it opens 17852 your browser to the moment you get back the verification code. This 17853 is on `http://127.0.0.1:53682/` and this it may require you to unblock 17854 it temporarily if you are running a host firewall. 17855 17856 Once configured you can then use `rclone` like this, 17857 17858 List containers in the top level of your Hubic 17859 17860 rclone lsd remote: 17861 17862 List all the files in your Hubic 17863 17864 rclone ls remote: 17865 17866 To copy a local directory to an Hubic directory called backup 17867 17868 rclone copy /home/source remote:backup 17869 17870 If you want the directory to be visible in the official *Hubic 17871 browser*, you need to copy your files to the `default` directory 17872 17873 rclone copy /home/source remote:default/backup 17874 17875 ### --fast-list ### 17876 17877 This remote supports `--fast-list` which allows you to use fewer 17878 transactions in exchange for more memory. See the [rclone 17879 docs](https://rclone.org/docs/#fast-list) for more details. 17880 17881 ### Modified time ### 17882 17883 The modified time is stored as metadata on the object as 17884 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 17885 ns. 17886 17887 This is a de facto standard (used in the official python-swiftclient 17888 amongst others) for storing the modification time for an object. 17889 17890 Note that Hubic wraps the Swift backend, so most of the properties of 17891 are the same. 17892 17893 17894 ### Standard Options 17895 17896 Here are the standard options specific to hubic (Hubic). 17897 17898 #### --hubic-client-id 17899 17900 Hubic Client Id 17901 Leave blank normally. 17902 17903 - Config: client_id 17904 - Env Var: RCLONE_HUBIC_CLIENT_ID 17905 - Type: string 17906 - Default: "" 17907 17908 #### --hubic-client-secret 17909 17910 Hubic Client Secret 17911 Leave blank normally. 17912 17913 - Config: client_secret 17914 - Env Var: RCLONE_HUBIC_CLIENT_SECRET 17915 - Type: string 17916 - Default: "" 17917 17918 ### Advanced Options 17919 17920 Here are the advanced options specific to hubic (Hubic). 17921 17922 #### --hubic-chunk-size 17923 17924 Above this size files will be chunked into a _segments container. 17925 17926 Above this size files will be chunked into a _segments container. The 17927 default for this is 5GB which is its maximum value. 17928 17929 - Config: chunk_size 17930 - Env Var: RCLONE_HUBIC_CHUNK_SIZE 17931 - Type: SizeSuffix 17932 - Default: 5G 17933 17934 #### --hubic-no-chunk 17935 17936 Don't chunk files during streaming upload. 17937 17938 When doing streaming uploads (eg using rcat or mount) setting this 17939 flag will cause the swift backend to not upload chunked files. 17940 17941 This will limit the maximum upload size to 5GB. However non chunked 17942 files are easier to deal with and have an MD5SUM. 17943 17944 Rclone will still chunk files bigger than chunk_size when doing normal 17945 copy operations. 17946 17947 - Config: no_chunk 17948 - Env Var: RCLONE_HUBIC_NO_CHUNK 17949 - Type: bool 17950 - Default: false 17951 17952 #### --hubic-encoding 17953 17954 This sets the encoding for the backend. 17955 17956 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 17957 17958 - Config: encoding 17959 - Env Var: RCLONE_HUBIC_ENCODING 17960 - Type: MultiEncoder 17961 - Default: Slash,InvalidUtf8 17962 17963 17964 17965 ### Limitations ### 17966 17967 This uses the normal OpenStack Swift mechanism to refresh the Swift 17968 API credentials and ignores the expires field returned by the Hubic 17969 API. 17970 17971 The Swift API doesn't return a correct MD5SUM for segmented files 17972 (Dynamic or Static Large Objects) so rclone won't check or use the 17973 MD5SUM for these. 17974 17975 Jottacloud 17976 ----------------------------------------- 17977 17978 Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters in Norway. 17979 17980 In addition to the official service at [jottacloud.com](https://www.jottacloud.com/), there are 17981 also several whitelabel versions which should work with this backend. 17982 17983 Paths are specified as `remote:path` 17984 17985 Paths may be as deep as required, eg `remote:directory/subdirectory`. 17986 17987 ## Setup 17988 17989 To configure Jottacloud you will need to generate a personal security token in the Jottacloud web interface. 17990 You will the option to do in your [account security settings](https://www.jottacloud.com/web/secure) 17991 (for whitelabel version you need to find this page in its web interface). 17992 Note that the web interface may refer to this token as a JottaCli token. 17993 17994 Here is an example of how to make a remote called `remote`. First run: 17995 17996 rclone config 17997 17998 This will guide you through an interactive setup process: 17999 18000 ``` 18001 No remotes found - make a new one 18002 n) New remote 18003 s) Set configuration password 18004 q) Quit config 18005 n/s/q> n 18006 name> jotta 18007 Type of storage to configure. 18008 Enter a string value. Press Enter for the default (""). 18009 Choose a number from below, or type in your own value 18010 [snip] 18011 XX / Jottacloud 18012 \ "jottacloud" 18013 [snip] 18014 Storage> jottacloud 18015 ** See help for jottacloud backend at: https://rclone.org/jottacloud/ ** 18016 18017 Edit advanced config? (y/n) 18018 y) Yes 18019 n) No 18020 y/n> n 18021 Remote config 18022 18023 Generate a personal login token here: https://www.jottacloud.com/web/secure 18024 Login Token> <your token here> 18025 18026 Do you want to use a non standard device/mountpoint e.g. for accessing files uploaded using the official Jottacloud client? 18027 18028 y) Yes 18029 n) No 18030 y/n> y 18031 Please select the device to use. Normally this will be Jotta 18032 Choose a number from below, or type in an existing value 18033 1 > DESKTOP-3H31129 18034 2 > fla1 18035 3 > Jotta 18036 Devices> 3 18037 Please select the mountpoint to user. Normally this will be Archive 18038 Choose a number from below, or type in an existing value 18039 1 > Archive 18040 2 > Links 18041 3 > Sync 18042 18043 Mountpoints> 1 18044 -------------------- 18045 [jotta] 18046 type = jottacloud 18047 user = 0xC4KE@gmail.com 18048 token = {........} 18049 device = Jotta 18050 mountpoint = Archive 18051 configVersion = 1 18052 -------------------- 18053 y) Yes this is OK 18054 e) Edit this remote 18055 d) Delete this remote 18056 y/e/d> y 18057 ``` 18058 Once configured you can then use `rclone` like this, 18059 18060 List directories in top level of your Jottacloud 18061 18062 rclone lsd remote: 18063 18064 List all the files in your Jottacloud 18065 18066 rclone ls remote: 18067 18068 To copy a local directory to an Jottacloud directory called backup 18069 18070 rclone copy /home/source remote:backup 18071 18072 ### Devices and Mountpoints 18073 18074 The official Jottacloud client registers a device for each computer you install it on, 18075 and then creates a mountpoint for each folder you select for Backup. 18076 The web interface uses a special device called Jotta for the Archive and Sync mountpoints. 18077 In most cases you'll want to use the Jotta/Archive device/mountpoint, however if you want to access 18078 files uploaded by any of the official clients rclone provides the option to select other devices 18079 and mountpoints during config. 18080 18081 The built-in Jotta device may also contain several other mountpoints, such as: Latest, Links, Shared and Trash. 18082 These are special mountpoints with a different internal representation than the "regular" mountpoints. 18083 Rclone will only to a very limited degree support them. Generally you should avoid these, unless you know what you 18084 are doing. 18085 18086 ### --fast-list 18087 18088 This remote supports `--fast-list` which allows you to use fewer 18089 transactions in exchange for more memory. See the [rclone 18090 docs](https://rclone.org/docs/#fast-list) for more details. 18091 18092 Note that the implementation in Jottacloud always uses only a single 18093 API request to get the entire list, so for large folders this could 18094 lead to long wait time before the first results are shown. 18095 18096 ### Modified time and hashes 18097 18098 Jottacloud allows modification times to be set on objects accurate to 1 18099 second. These will be used to detect whether objects need syncing or 18100 not. 18101 18102 Jottacloud supports MD5 type hashes, so you can use the `--checksum` 18103 flag. 18104 18105 Note that Jottacloud requires the MD5 hash before upload so if the 18106 source does not have an MD5 checksum then the file will be cached 18107 temporarily on disk (wherever the `TMPDIR` environment variable points 18108 to) before it is uploaded. Small files will be cached in memory - see 18109 the [--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag. 18110 18111 #### Restricted filename characters 18112 18113 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 18114 the following characters are also replaced: 18115 18116 | Character | Value | Replacement | 18117 | --------- |:-----:|:-----------:| 18118 | " | 0x22 | " | 18119 | * | 0x2A | * | 18120 | : | 0x3A | : | 18121 | < | 0x3C | < | 18122 | > | 0x3E | > | 18123 | ? | 0x3F | ? | 18124 | \| | 0x7C | | | 18125 18126 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 18127 as they can't be used in XML strings. 18128 18129 ### Deleting files 18130 18131 By default rclone will send all files to the trash when deleting files. They will be permanently 18132 deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately 18133 by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable. 18134 Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command. 18135 18136 ### Versions 18137 18138 Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it. 18139 Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website. 18140 18141 ### Quota information 18142 18143 To view your current quota you can use the `rclone about remote:` 18144 command which will display your usage limit (unless it is unlimited) 18145 and the current usage. 18146 18147 18148 ### Advanced Options 18149 18150 Here are the advanced options specific to jottacloud (Jottacloud). 18151 18152 #### --jottacloud-md5-memory-limit 18153 18154 Files bigger than this will be cached on disk to calculate the MD5 if required. 18155 18156 - Config: md5_memory_limit 18157 - Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT 18158 - Type: SizeSuffix 18159 - Default: 10M 18160 18161 #### --jottacloud-trashed-only 18162 18163 Only show files that are in the trash. 18164 This will show trashed files in their original directory structure. 18165 18166 - Config: trashed_only 18167 - Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY 18168 - Type: bool 18169 - Default: false 18170 18171 #### --jottacloud-hard-delete 18172 18173 Delete files permanently rather than putting them into the trash. 18174 18175 - Config: hard_delete 18176 - Env Var: RCLONE_JOTTACLOUD_HARD_DELETE 18177 - Type: bool 18178 - Default: false 18179 18180 #### --jottacloud-unlink 18181 18182 Remove existing public link to file/folder with link command rather than creating. 18183 Default is false, meaning link command will create or retrieve public link. 18184 18185 - Config: unlink 18186 - Env Var: RCLONE_JOTTACLOUD_UNLINK 18187 - Type: bool 18188 - Default: false 18189 18190 #### --jottacloud-upload-resume-limit 18191 18192 Files bigger than this can be resumed if the upload fail's. 18193 18194 - Config: upload_resume_limit 18195 - Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT 18196 - Type: SizeSuffix 18197 - Default: 10M 18198 18199 #### --jottacloud-encoding 18200 18201 This sets the encoding for the backend. 18202 18203 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 18204 18205 - Config: encoding 18206 - Env Var: RCLONE_JOTTACLOUD_ENCODING 18207 - Type: MultiEncoder 18208 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot 18209 18210 18211 18212 ### Limitations 18213 18214 Note that Jottacloud is case insensitive so you can't have a file called 18215 "Hello.doc" and one called "hello.doc". 18216 18217 There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical 18218 looking unicode equivalent. For example if a file has a ? in it will be mapped to ? instead. 18219 18220 Jottacloud only supports filenames up to 255 characters in length. 18221 18222 ### Troubleshooting 18223 18224 Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove 18225 operations to previously deleted paths to fail. Emptying the trash should help in such cases. 18226 18227 Koofr 18228 ----------------------------------------- 18229 18230 Paths are specified as `remote:path` 18231 18232 Paths may be as deep as required, eg `remote:directory/subdirectory`. 18233 18234 The initial setup for Koofr involves creating an application password for 18235 rclone. You can do that by opening the Koofr 18236 [web application](https://app.koofr.net/app/admin/preferences/password), 18237 giving the password a nice name like `rclone` and clicking on generate. 18238 18239 Here is an example of how to make a remote called `koofr`. First run: 18240 18241 rclone config 18242 18243 This will guide you through an interactive setup process: 18244 18245 ``` 18246 No remotes found - make a new one 18247 n) New remote 18248 s) Set configuration password 18249 q) Quit config 18250 n/s/q> n 18251 name> koofr 18252 Type of storage to configure. 18253 Enter a string value. Press Enter for the default (""). 18254 Choose a number from below, or type in your own value 18255 [snip] 18256 XX / Koofr 18257 \ "koofr" 18258 [snip] 18259 Storage> koofr 18260 ** See help for koofr backend at: https://rclone.org/koofr/ ** 18261 18262 Your Koofr user name 18263 Enter a string value. Press Enter for the default (""). 18264 user> USER@NAME 18265 Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) 18266 y) Yes type in my own password 18267 g) Generate random password 18268 y/g> y 18269 Enter the password: 18270 password: 18271 Confirm the password: 18272 password: 18273 Edit advanced config? (y/n) 18274 y) Yes 18275 n) No 18276 y/n> n 18277 Remote config 18278 -------------------- 18279 [koofr] 18280 type = koofr 18281 baseurl = https://app.koofr.net 18282 user = USER@NAME 18283 password = *** ENCRYPTED *** 18284 -------------------- 18285 y) Yes this is OK 18286 e) Edit this remote 18287 d) Delete this remote 18288 y/e/d> y 18289 ``` 18290 18291 You can choose to edit advanced config in order to enter your own service URL 18292 if you use an on-premise or white label Koofr instance, or choose an alternative 18293 mount instead of your primary storage. 18294 18295 Once configured you can then use `rclone` like this, 18296 18297 List directories in top level of your Koofr 18298 18299 rclone lsd koofr: 18300 18301 List all the files in your Koofr 18302 18303 rclone ls koofr: 18304 18305 To copy a local directory to an Koofr directory called backup 18306 18307 rclone copy /home/source remote:backup 18308 18309 #### Restricted filename characters 18310 18311 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 18312 the following characters are also replaced: 18313 18314 | Character | Value | Replacement | 18315 | --------- |:-----:|:-----------:| 18316 | \ | 0x5C | \ | 18317 18318 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 18319 as they can't be used in XML strings. 18320 18321 18322 ### Standard Options 18323 18324 Here are the standard options specific to koofr (Koofr). 18325 18326 #### --koofr-user 18327 18328 Your Koofr user name 18329 18330 - Config: user 18331 - Env Var: RCLONE_KOOFR_USER 18332 - Type: string 18333 - Default: "" 18334 18335 #### --koofr-password 18336 18337 Your Koofr password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) 18338 18339 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 18340 18341 - Config: password 18342 - Env Var: RCLONE_KOOFR_PASSWORD 18343 - Type: string 18344 - Default: "" 18345 18346 ### Advanced Options 18347 18348 Here are the advanced options specific to koofr (Koofr). 18349 18350 #### --koofr-endpoint 18351 18352 The Koofr API endpoint to use 18353 18354 - Config: endpoint 18355 - Env Var: RCLONE_KOOFR_ENDPOINT 18356 - Type: string 18357 - Default: "https://app.koofr.net" 18358 18359 #### --koofr-mountid 18360 18361 Mount ID of the mount to use. If omitted, the primary mount is used. 18362 18363 - Config: mountid 18364 - Env Var: RCLONE_KOOFR_MOUNTID 18365 - Type: string 18366 - Default: "" 18367 18368 #### --koofr-setmtime 18369 18370 Does the backend support setting modification time. Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend. 18371 18372 - Config: setmtime 18373 - Env Var: RCLONE_KOOFR_SETMTIME 18374 - Type: bool 18375 - Default: true 18376 18377 #### --koofr-encoding 18378 18379 This sets the encoding for the backend. 18380 18381 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 18382 18383 - Config: encoding 18384 - Env Var: RCLONE_KOOFR_ENCODING 18385 - Type: MultiEncoder 18386 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 18387 18388 18389 18390 ### Limitations ### 18391 18392 Note that Koofr is case insensitive so you can't have a file called 18393 "Hello.doc" and one called "hello.doc". 18394 18395 Mail.ru Cloud 18396 ---------------------------------------- 18397 18398 [Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/), available only on Windows. (Please note that official sites are in Russian) 18399 18400 Currently it is recommended to disable 2FA on Mail.ru accounts intended for rclone until it gets eventually implemented. 18401 18402 ### Features highlights ### 18403 18404 - Paths may be as deep as required, eg `remote:directory/subdirectory` 18405 - Files have a `last modified time` property, directories don't 18406 - Deleted files are by default moved to the trash 18407 - Files and directories can be shared via public links 18408 - Partial uploads or streaming are not supported, file size must be known before upload 18409 - Maximum file size is limited to 2G for a free account, unlimited for paid accounts 18410 - Storage keeps hash for all files and performs transparent deduplication, 18411 the hash algorithm is a modified SHA1 18412 - If a particular file is already present in storage, one can quickly submit file hash 18413 instead of long file upload (this optimization is supported by rclone) 18414 18415 ### Configuration ### 18416 18417 Here is an example of making a mailru configuration. First create a Mail.ru Cloud 18418 account and choose a tariff, then run 18419 18420 rclone config 18421 18422 This will guide you through an interactive setup process: 18423 18424 ``` 18425 No remotes found - make a new one 18426 n) New remote 18427 s) Set configuration password 18428 q) Quit config 18429 n/s/q> n 18430 name> remote 18431 Type of storage to configure. 18432 Type of storage to configure. 18433 Enter a string value. Press Enter for the default (""). 18434 Choose a number from below, or type in your own value 18435 [snip] 18436 XX / Mail.ru Cloud 18437 \ "mailru" 18438 [snip] 18439 Storage> mailru 18440 User name (usually email) 18441 Enter a string value. Press Enter for the default (""). 18442 user> username@mail.ru 18443 Password 18444 y) Yes type in my own password 18445 g) Generate random password 18446 y/g> y 18447 Enter the password: 18448 password: 18449 Confirm the password: 18450 password: 18451 Skip full upload if there is another file with same data hash. 18452 This feature is called "speedup" or "put by hash". It is especially efficient 18453 in case of generally available files like popular books, video or audio clips 18454 [snip] 18455 Enter a boolean value (true or false). Press Enter for the default ("true"). 18456 Choose a number from below, or type in your own value 18457 1 / Enable 18458 \ "true" 18459 2 / Disable 18460 \ "false" 18461 speedup_enable> 1 18462 Edit advanced config? (y/n) 18463 y) Yes 18464 n) No 18465 y/n> n 18466 Remote config 18467 -------------------- 18468 [remote] 18469 type = mailru 18470 user = username@mail.ru 18471 pass = *** ENCRYPTED *** 18472 speedup_enable = true 18473 -------------------- 18474 y) Yes this is OK 18475 e) Edit this remote 18476 d) Delete this remote 18477 y/e/d> y 18478 ``` 18479 18480 Configuration of this backend does not require a local web browser. 18481 You can use the configured backend as shown below: 18482 18483 See top level directories 18484 18485 rclone lsd remote: 18486 18487 Make a new directory 18488 18489 rclone mkdir remote:directory 18490 18491 List the contents of a directory 18492 18493 rclone ls remote:directory 18494 18495 Sync `/home/local/directory` to the remote path, deleting any 18496 excess files in the path. 18497 18498 rclone sync /home/local/directory remote:directory 18499 18500 ### Modified time ### 18501 18502 Files support a modification time attribute with up to 1 second precision. 18503 Directories do not have a modification time, which is shown as "Jan 1 1970". 18504 18505 ### Hash checksums ### 18506 18507 Hash sums use a custom Mail.ru algorithm based on SHA1. 18508 If file size is less than or equal to the SHA1 block size (20 bytes), 18509 its hash is simply its data right-padded with zero bytes. 18510 Hash sum of a larger file is computed as a SHA1 sum of the file data 18511 bytes concatenated with a decimal representation of the data length. 18512 18513 ### Emptying Trash ### 18514 18515 Removing a file or directory actually moves it to the trash, which is not 18516 visible to rclone but can be seen in a web browser. The trashed file 18517 still occupies part of total quota. If you wish to empty your trash 18518 and free some quota, you can use the `rclone cleanup remote:` command, 18519 which will permanently delete all your trashed files. 18520 This command does not take any path arguments. 18521 18522 ### Quota information ### 18523 18524 To view your current quota you can use the `rclone about remote:` 18525 command which will display your usage limit (quota) and the current usage. 18526 18527 #### Restricted filename characters 18528 18529 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 18530 the following characters are also replaced: 18531 18532 | Character | Value | Replacement | 18533 | --------- |:-----:|:-----------:| 18534 | " | 0x22 | " | 18535 | * | 0x2A | * | 18536 | : | 0x3A | : | 18537 | < | 0x3C | < | 18538 | > | 0x3E | > | 18539 | ? | 0x3F | ? | 18540 | \ | 0x5C | \ | 18541 | \| | 0x7C | | | 18542 18543 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 18544 as they can't be used in JSON strings. 18545 18546 ### Limitations ### 18547 18548 File size limits depend on your account. A single file size is limited by 2G 18549 for a free account and unlimited for paid tariffs. Please refer to the Mail.ru 18550 site for the total uploaded size limits. 18551 18552 Note that Mailru is case insensitive so you can't have a file called 18553 "Hello.doc" and one called "hello.doc". 18554 18555 18556 ### Standard Options 18557 18558 Here are the standard options specific to mailru (Mail.ru Cloud). 18559 18560 #### --mailru-user 18561 18562 User name (usually email) 18563 18564 - Config: user 18565 - Env Var: RCLONE_MAILRU_USER 18566 - Type: string 18567 - Default: "" 18568 18569 #### --mailru-pass 18570 18571 Password 18572 18573 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 18574 18575 - Config: pass 18576 - Env Var: RCLONE_MAILRU_PASS 18577 - Type: string 18578 - Default: "" 18579 18580 #### --mailru-speedup-enable 18581 18582 Skip full upload if there is another file with same data hash. 18583 This feature is called "speedup" or "put by hash". It is especially efficient 18584 in case of generally available files like popular books, video or audio clips, 18585 because files are searched by hash in all accounts of all mailru users. 18586 Please note that rclone may need local memory and disk space to calculate 18587 content hash in advance and decide whether full upload is required. 18588 Also, if rclone does not know file size in advance (e.g. in case of 18589 streaming or partial uploads), it will not even try this optimization. 18590 18591 - Config: speedup_enable 18592 - Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE 18593 - Type: bool 18594 - Default: true 18595 - Examples: 18596 - "true" 18597 - Enable 18598 - "false" 18599 - Disable 18600 18601 ### Advanced Options 18602 18603 Here are the advanced options specific to mailru (Mail.ru Cloud). 18604 18605 #### --mailru-speedup-file-patterns 18606 18607 Comma separated list of file name patterns eligible for speedup (put by hash). 18608 Patterns are case insensitive and can contain '*' or '?' meta characters. 18609 18610 - Config: speedup_file_patterns 18611 - Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS 18612 - Type: string 18613 - Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf" 18614 - Examples: 18615 - "" 18616 - Empty list completely disables speedup (put by hash). 18617 - "*" 18618 - All files will be attempted for speedup. 18619 - "*.mkv,*.avi,*.mp4,*.mp3" 18620 - Only common audio/video files will be tried for put by hash. 18621 - "*.zip,*.gz,*.rar,*.pdf" 18622 - Only common archives or PDF books will be tried for speedup. 18623 18624 #### --mailru-speedup-max-disk 18625 18626 This option allows you to disable speedup (put by hash) for large files 18627 (because preliminary hashing can exhaust you RAM or disk space) 18628 18629 - Config: speedup_max_disk 18630 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK 18631 - Type: SizeSuffix 18632 - Default: 3G 18633 - Examples: 18634 - "0" 18635 - Completely disable speedup (put by hash). 18636 - "1G" 18637 - Files larger than 1Gb will be uploaded directly. 18638 - "3G" 18639 - Choose this option if you have less than 3Gb free on local disk. 18640 18641 #### --mailru-speedup-max-memory 18642 18643 Files larger than the size given below will always be hashed on disk. 18644 18645 - Config: speedup_max_memory 18646 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY 18647 - Type: SizeSuffix 18648 - Default: 32M 18649 - Examples: 18650 - "0" 18651 - Preliminary hashing will always be done in a temporary disk location. 18652 - "32M" 18653 - Do not dedicate more than 32Mb RAM for preliminary hashing. 18654 - "256M" 18655 - You have at most 256Mb RAM free for hash calculations. 18656 18657 #### --mailru-check-hash 18658 18659 What should copy do if file checksum is mismatched or invalid 18660 18661 - Config: check_hash 18662 - Env Var: RCLONE_MAILRU_CHECK_HASH 18663 - Type: bool 18664 - Default: true 18665 - Examples: 18666 - "true" 18667 - Fail with error. 18668 - "false" 18669 - Ignore and continue. 18670 18671 #### --mailru-user-agent 18672 18673 HTTP user agent used internally by client. 18674 Defaults to "rclone/VERSION" or "--user-agent" provided on command line. 18675 18676 - Config: user_agent 18677 - Env Var: RCLONE_MAILRU_USER_AGENT 18678 - Type: string 18679 - Default: "" 18680 18681 #### --mailru-quirks 18682 18683 Comma separated list of internal maintenance flags. 18684 This option must not be used by an ordinary user. It is intended only to 18685 facilitate remote troubleshooting of backend issues. Strict meaning of 18686 flags is not documented and not guaranteed to persist between releases. 18687 Quirks will be removed when the backend grows stable. 18688 Supported quirks: atomicmkdir binlist gzip insecure retry400 18689 18690 - Config: quirks 18691 - Env Var: RCLONE_MAILRU_QUIRKS 18692 - Type: string 18693 - Default: "" 18694 18695 #### --mailru-encoding 18696 18697 This sets the encoding for the backend. 18698 18699 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 18700 18701 - Config: encoding 18702 - Env Var: RCLONE_MAILRU_ENCODING 18703 - Type: MultiEncoder 18704 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot 18705 18706 18707 18708 Mega 18709 ----------------------------------------- 18710 18711 [Mega](https://mega.nz/) is a cloud storage and file hosting service 18712 known for its security feature where all files are encrypted locally 18713 before they are uploaded. This prevents anyone (including employees of 18714 Mega) from accessing the files without knowledge of the key used for 18715 encryption. 18716 18717 This is an rclone backend for Mega which supports the file transfer 18718 features of Mega using the same client side encryption. 18719 18720 Paths are specified as `remote:path` 18721 18722 Paths may be as deep as required, eg `remote:directory/subdirectory`. 18723 18724 Here is an example of how to make a remote called `remote`. First run: 18725 18726 rclone config 18727 18728 This will guide you through an interactive setup process: 18729 18730 ``` 18731 No remotes found - make a new one 18732 n) New remote 18733 s) Set configuration password 18734 q) Quit config 18735 n/s/q> n 18736 name> remote 18737 Type of storage to configure. 18738 Choose a number from below, or type in your own value 18739 [snip] 18740 XX / Mega 18741 \ "mega" 18742 [snip] 18743 Storage> mega 18744 User name 18745 user> you@example.com 18746 Password. 18747 y) Yes type in my own password 18748 g) Generate random password 18749 n) No leave this optional password blank 18750 y/g/n> y 18751 Enter the password: 18752 password: 18753 Confirm the password: 18754 password: 18755 Remote config 18756 -------------------- 18757 [remote] 18758 type = mega 18759 user = you@example.com 18760 pass = *** ENCRYPTED *** 18761 -------------------- 18762 y) Yes this is OK 18763 e) Edit this remote 18764 d) Delete this remote 18765 y/e/d> y 18766 ``` 18767 18768 **NOTE:** The encryption keys need to have been already generated after a regular login 18769 via the browser, otherwise attempting to use the credentials in `rclone` will fail. 18770 18771 Once configured you can then use `rclone` like this, 18772 18773 List directories in top level of your Mega 18774 18775 rclone lsd remote: 18776 18777 List all the files in your Mega 18778 18779 rclone ls remote: 18780 18781 To copy a local directory to an Mega directory called backup 18782 18783 rclone copy /home/source remote:backup 18784 18785 ### Modified time and hashes ### 18786 18787 Mega does not support modification times or hashes yet. 18788 18789 #### Restricted filename characters 18790 18791 | Character | Value | Replacement | 18792 | --------- |:-----:|:-----------:| 18793 | NUL | 0x00 | ␀ | 18794 | / | 0x2F | / | 18795 18796 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 18797 as they can't be used in JSON strings. 18798 18799 ### Duplicated files ### 18800 18801 Mega can have two files with exactly the same name and path (unlike a 18802 normal file system). 18803 18804 Duplicated files cause problems with the syncing and you will see 18805 messages in the log about duplicates. 18806 18807 Use `rclone dedupe` to fix duplicated files. 18808 18809 ### Failure to log-in ### 18810 18811 Mega remotes seem to get blocked (reject logins) under "heavy use". 18812 We haven't worked out the exact blocking rules but it seems to be 18813 related to fast paced, successive rclone commands. 18814 18815 For example, executing this command 90 times in a row `rclone link 18816 remote:file` will cause the remote to become "blocked". This is not an 18817 abnormal situation, for example if you wish to get the public links of 18818 a directory with hundred of files... After more or less a week, the 18819 remote will remote accept rclone logins normally again. 18820 18821 You can mitigate this issue by mounting the remote it with `rclone 18822 mount`. This will log-in when mounting and a log-out when unmounting 18823 only. You can also run `rclone rcd` and then use `rclone rc` to run 18824 the commands over the API to avoid logging in each time. 18825 18826 Rclone does not currently close mega sessions (you can see them in the 18827 web interface), however closing the sessions does not solve the issue. 18828 18829 If you space rclone commands by 3 seconds it will avoid blocking the 18830 remote. We haven't identified the exact blocking rules, so perhaps one 18831 could execute the command 80 times without waiting and avoid blocking 18832 by waiting 3 seconds, then continuing... 18833 18834 Note that this has been observed by trial and error and might not be 18835 set in stone. 18836 18837 Other tools seem not to produce this blocking effect, as they use a 18838 different working approach (state-based, using sessionIDs instead of 18839 log-in) which isn't compatible with the current stateless rclone 18840 approach. 18841 18842 Note that once blocked, the use of other tools (such as megacmd) is 18843 not a sure workaround: following megacmd login times have been 18844 observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30 18845 min, 30min. Web access looks unaffected though. 18846 18847 Investigation is continuing in relation to workarounds based on 18848 timeouts, pacers, retrials and tpslimits - if you discover something 18849 relevant, please post on the forum. 18850 18851 So, if rclone was working nicely and suddenly you are unable to log-in 18852 and you are sure the user and the password are correct, likely you 18853 have got the remote blocked for a while. 18854 18855 18856 ### Standard Options 18857 18858 Here are the standard options specific to mega (Mega). 18859 18860 #### --mega-user 18861 18862 User name 18863 18864 - Config: user 18865 - Env Var: RCLONE_MEGA_USER 18866 - Type: string 18867 - Default: "" 18868 18869 #### --mega-pass 18870 18871 Password. 18872 18873 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 18874 18875 - Config: pass 18876 - Env Var: RCLONE_MEGA_PASS 18877 - Type: string 18878 - Default: "" 18879 18880 ### Advanced Options 18881 18882 Here are the advanced options specific to mega (Mega). 18883 18884 #### --mega-debug 18885 18886 Output more debug from Mega. 18887 18888 If this flag is set (along with -vv) it will print further debugging 18889 information from the mega backend. 18890 18891 - Config: debug 18892 - Env Var: RCLONE_MEGA_DEBUG 18893 - Type: bool 18894 - Default: false 18895 18896 #### --mega-hard-delete 18897 18898 Delete files permanently rather than putting them into the trash. 18899 18900 Normally the mega backend will put all deletions into the trash rather 18901 than permanently deleting them. If you specify this then rclone will 18902 permanently delete objects instead. 18903 18904 - Config: hard_delete 18905 - Env Var: RCLONE_MEGA_HARD_DELETE 18906 - Type: bool 18907 - Default: false 18908 18909 #### --mega-encoding 18910 18911 This sets the encoding for the backend. 18912 18913 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 18914 18915 - Config: encoding 18916 - Env Var: RCLONE_MEGA_ENCODING 18917 - Type: MultiEncoder 18918 - Default: Slash,InvalidUtf8,Dot 18919 18920 18921 18922 ### Limitations ### 18923 18924 This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource 18925 go library implementing the Mega API. There doesn't appear to be any 18926 documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code 18927 so there are likely quite a few errors still remaining in this library. 18928 18929 Mega allows duplicate files which may confuse rclone. 18930 18931 Memory 18932 ----------------------------------------- 18933 18934 The memory backend is an in RAM backend. It does not persist its 18935 data - use the local backend for that. 18936 18937 The memory backend behaves like a bucket based remote (eg like 18938 s3). Because it has no parameters you can just use it with the 18939 `:memory:` remote name. 18940 18941 You can configure it as a remote like this with `rclone config` too if 18942 you want to: 18943 18944 ``` 18945 No remotes found - make a new one 18946 n) New remote 18947 s) Set configuration password 18948 q) Quit config 18949 n/s/q> n 18950 name> remote 18951 Type of storage to configure. 18952 Enter a string value. Press Enter for the default (""). 18953 Choose a number from below, or type in your own value 18954 [snip] 18955 XX / Memory 18956 \ "memory" 18957 [snip] 18958 Storage> memory 18959 ** See help for memory backend at: https://rclone.org/memory/ ** 18960 18961 Remote config 18962 18963 -------------------- 18964 [remote] 18965 type = memory 18966 -------------------- 18967 y) Yes this is OK (default) 18968 e) Edit this remote 18969 d) Delete this remote 18970 y/e/d> y 18971 ``` 18972 18973 Because the memory backend isn't persistent it is most useful for 18974 testing or with an rclone server or rclone mount, eg 18975 18976 rclone mount :memory: /mnt/tmp 18977 rclone serve webdav :memory: 18978 rclone serve sftp :memory: 18979 18980 ### Modified time and hashes ### 18981 18982 The memory backend supports MD5 hashes and modification times accurate to 1 nS. 18983 18984 #### Restricted filename characters 18985 18986 The memory backend replaces the [default restricted characters 18987 set](https://rclone.org/overview/#restricted-characters). 18988 18989 18990 18991 18992 Microsoft Azure Blob Storage 18993 ----------------------------------------- 18994 18995 Paths are specified as `remote:container` (or `remote:` for the `lsd` 18996 command.) You may put subdirectories in too, eg 18997 `remote:container/path/to/dir`. 18998 18999 Here is an example of making a Microsoft Azure Blob Storage 19000 configuration. For a remote called `remote`. First run: 19001 19002 rclone config 19003 19004 This will guide you through an interactive setup process: 19005 19006 ``` 19007 No remotes found - make a new one 19008 n) New remote 19009 s) Set configuration password 19010 q) Quit config 19011 n/s/q> n 19012 name> remote 19013 Type of storage to configure. 19014 Choose a number from below, or type in your own value 19015 [snip] 19016 XX / Microsoft Azure Blob Storage 19017 \ "azureblob" 19018 [snip] 19019 Storage> azureblob 19020 Storage Account Name 19021 account> account_name 19022 Storage Account Key 19023 key> base64encodedkey== 19024 Endpoint for the service - leave blank normally. 19025 endpoint> 19026 Remote config 19027 -------------------- 19028 [remote] 19029 account = account_name 19030 key = base64encodedkey== 19031 endpoint = 19032 -------------------- 19033 y) Yes this is OK 19034 e) Edit this remote 19035 d) Delete this remote 19036 y/e/d> y 19037 ``` 19038 19039 See all containers 19040 19041 rclone lsd remote: 19042 19043 Make a new container 19044 19045 rclone mkdir remote:container 19046 19047 List the contents of a container 19048 19049 rclone ls remote:container 19050 19051 Sync `/home/local/directory` to the remote container, deleting any excess 19052 files in the container. 19053 19054 rclone sync /home/local/directory remote:container 19055 19056 ### --fast-list ### 19057 19058 This remote supports `--fast-list` which allows you to use fewer 19059 transactions in exchange for more memory. See the [rclone 19060 docs](https://rclone.org/docs/#fast-list) for more details. 19061 19062 ### Modified time ### 19063 19064 The modified time is stored as metadata on the object with the `mtime` 19065 key. It is stored using RFC3339 Format time with nanosecond 19066 precision. The metadata is supplied during directory listings so 19067 there is no overhead to using it. 19068 19069 ### Restricted filename characters 19070 19071 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 19072 the following characters are also replaced: 19073 19074 | Character | Value | Replacement | 19075 | --------- |:-----:|:-----------:| 19076 | / | 0x2F | / | 19077 | \ | 0x5C | \ | 19078 19079 File names can also not end with the following characters. 19080 These only get replaced if they are the last character in the name: 19081 19082 | Character | Value | Replacement | 19083 | --------- |:-----:|:-----------:| 19084 | . | 0x2E | . | 19085 19086 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 19087 as they can't be used in JSON strings. 19088 19089 ### Hashes ### 19090 19091 MD5 hashes are stored with blobs. However blobs that were uploaded in 19092 chunks only have an MD5 if the source remote was capable of MD5 19093 hashes, eg the local disk. 19094 19095 ### Authenticating with Azure Blob Storage 19096 19097 Rclone has 3 ways of authenticating with Azure Blob Storage: 19098 19099 #### Account and Key 19100 19101 This is the most straight forward and least flexible way. Just fill 19102 in the `account` and `key` lines and leave the rest blank. 19103 19104 #### SAS URL 19105 19106 This can be an account level SAS URL or container level SAS URL. 19107 19108 To use it leave `account`, `key` blank and fill in `sas_url`. 19109 19110 An account level SAS URL or container level SAS URL can be obtained 19111 from the Azure portal or the Azure Storage Explorer. To get a 19112 container level SAS URL right click on a container in the Azure Blob 19113 explorer in the Azure portal. 19114 19115 If you use a container level SAS URL, rclone operations are permitted 19116 only on a particular container, eg 19117 19118 rclone ls azureblob:container 19119 19120 You can also list the single container from the root. This will only 19121 show the container specified by the SAS URL. 19122 19123 $ rclone lsd azureblob: 19124 container/ 19125 19126 Note that you can't see or access any other containers - this will 19127 fail 19128 19129 rclone ls azureblob:othercontainer 19130 19131 Container level SAS URLs are useful for temporarily allowing third 19132 parties access to a single container or putting credentials into an 19133 untrusted environment such as a CI build server. 19134 19135 ### Multipart uploads ### 19136 19137 Rclone supports multipart uploads with Azure Blob storage. Files 19138 bigger than 256MB will be uploaded using chunked upload by default. 19139 19140 The files will be uploaded in parallel in 4MB chunks (by default). 19141 Note that these chunks are buffered in memory and there may be up to 19142 `--transfers` of them being uploaded at once. 19143 19144 Files can't be split into more than 50,000 chunks so by default, so 19145 the largest file that can be uploaded with 4MB chunk size is 195GB. 19146 Above this rclone will double the chunk size until it creates less 19147 than 50,000 chunks. By default this will mean a maximum file size of 19148 3.2TB can be uploaded. This can be raised to 5TB using 19149 `--azureblob-chunk-size 100M`. 19150 19151 Note that rclone doesn't commit the block list until the end of the 19152 upload which means that there is a limit of 9.5TB of multipart uploads 19153 in progress as Azure won't allow more than that amount of uncommitted 19154 blocks. 19155 19156 19157 ### Standard Options 19158 19159 Here are the standard options specific to azureblob (Microsoft Azure Blob Storage). 19160 19161 #### --azureblob-account 19162 19163 Storage Account Name (leave blank to use SAS URL or Emulator) 19164 19165 - Config: account 19166 - Env Var: RCLONE_AZUREBLOB_ACCOUNT 19167 - Type: string 19168 - Default: "" 19169 19170 #### --azureblob-key 19171 19172 Storage Account Key (leave blank to use SAS URL or Emulator) 19173 19174 - Config: key 19175 - Env Var: RCLONE_AZUREBLOB_KEY 19176 - Type: string 19177 - Default: "" 19178 19179 #### --azureblob-sas-url 19180 19181 SAS URL for container level access only 19182 (leave blank if using account/key or Emulator) 19183 19184 - Config: sas_url 19185 - Env Var: RCLONE_AZUREBLOB_SAS_URL 19186 - Type: string 19187 - Default: "" 19188 19189 #### --azureblob-use-emulator 19190 19191 Uses local storage emulator if provided as 'true' (leave blank if using real azure storage endpoint) 19192 19193 - Config: use_emulator 19194 - Env Var: RCLONE_AZUREBLOB_USE_EMULATOR 19195 - Type: bool 19196 - Default: false 19197 19198 ### Advanced Options 19199 19200 Here are the advanced options specific to azureblob (Microsoft Azure Blob Storage). 19201 19202 #### --azureblob-endpoint 19203 19204 Endpoint for the service 19205 Leave blank normally. 19206 19207 - Config: endpoint 19208 - Env Var: RCLONE_AZUREBLOB_ENDPOINT 19209 - Type: string 19210 - Default: "" 19211 19212 #### --azureblob-upload-cutoff 19213 19214 Cutoff for switching to chunked upload (<= 256MB). 19215 19216 - Config: upload_cutoff 19217 - Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF 19218 - Type: SizeSuffix 19219 - Default: 256M 19220 19221 #### --azureblob-chunk-size 19222 19223 Upload chunk size (<= 100MB). 19224 19225 Note that this is stored in memory and there may be up to 19226 "--transfers" chunks stored at once in memory. 19227 19228 - Config: chunk_size 19229 - Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE 19230 - Type: SizeSuffix 19231 - Default: 4M 19232 19233 #### --azureblob-list-chunk 19234 19235 Size of blob list. 19236 19237 This sets the number of blobs requested in each listing chunk. Default 19238 is the maximum, 5000. "List blobs" requests are permitted 2 minutes 19239 per megabyte to complete. If an operation is taking longer than 2 19240 minutes per megabyte on average, it will time out ( 19241 [source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval) 19242 ). This can be used to limit the number of blobs items to return, to 19243 avoid the time out. 19244 19245 - Config: list_chunk 19246 - Env Var: RCLONE_AZUREBLOB_LIST_CHUNK 19247 - Type: int 19248 - Default: 5000 19249 19250 #### --azureblob-access-tier 19251 19252 Access tier of blob: hot, cool or archive. 19253 19254 Archived blobs can be restored by setting access tier to hot or 19255 cool. Leave blank if you intend to use default access tier, which is 19256 set at account level 19257 19258 If there is no "access tier" specified, rclone doesn't apply any tier. 19259 rclone performs "Set Tier" operation on blobs while uploading, if objects 19260 are not modified, specifying "access tier" to new one will have no effect. 19261 If blobs are in "archive tier" at remote, trying to perform data transfer 19262 operations from remote will not be allowed. User should first restore by 19263 tiering blob to "Hot" or "Cool". 19264 19265 - Config: access_tier 19266 - Env Var: RCLONE_AZUREBLOB_ACCESS_TIER 19267 - Type: string 19268 - Default: "" 19269 19270 #### --azureblob-disable-checksum 19271 19272 Don't store MD5 checksum with object metadata. 19273 19274 Normally rclone will calculate the MD5 checksum of the input before 19275 uploading it so it can add it to metadata on the object. This is great 19276 for data integrity checking but can cause long delays for large files 19277 to start uploading. 19278 19279 - Config: disable_checksum 19280 - Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM 19281 - Type: bool 19282 - Default: false 19283 19284 #### --azureblob-memory-pool-flush-time 19285 19286 How often internal memory buffer pools will be flushed. 19287 Uploads which requires additional buffers (f.e multipart) will use memory pool for allocations. 19288 This option controls how often unused buffers will be removed from the pool. 19289 19290 - Config: memory_pool_flush_time 19291 - Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME 19292 - Type: Duration 19293 - Default: 1m0s 19294 19295 #### --azureblob-memory-pool-use-mmap 19296 19297 Whether to use mmap buffers in internal memory pool. 19298 19299 - Config: memory_pool_use_mmap 19300 - Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP 19301 - Type: bool 19302 - Default: false 19303 19304 #### --azureblob-encoding 19305 19306 This sets the encoding for the backend. 19307 19308 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 19309 19310 - Config: encoding 19311 - Env Var: RCLONE_AZUREBLOB_ENCODING 19312 - Type: MultiEncoder 19313 - Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8 19314 19315 19316 19317 ### Limitations ### 19318 19319 MD5 sums are only uploaded with chunked files if the source has an MD5 19320 sum. This will always be the case for a local to azure copy. 19321 19322 ### Azure Storage Emulator Support ### 19323 You can test rclone with storage emulator locally, to do this make sure azure storage emulator 19324 installed locally and set up a new remote with `rclone config` follow instructions described in 19325 introduction, set `use_emulator` config as `true`, you do not need to provide default account name 19326 or key if using emulator. 19327 19328 Microsoft OneDrive 19329 ----------------------------------------- 19330 19331 Paths are specified as `remote:path` 19332 19333 Paths may be as deep as required, eg `remote:directory/subdirectory`. 19334 19335 The initial setup for OneDrive involves getting a token from 19336 Microsoft which you need to do in your browser. `rclone config` walks 19337 you through it. 19338 19339 Here is an example of how to make a remote called `remote`. First run: 19340 19341 rclone config 19342 19343 This will guide you through an interactive setup process: 19344 19345 ``` 19346 e) Edit existing remote 19347 n) New remote 19348 d) Delete remote 19349 r) Rename remote 19350 c) Copy remote 19351 s) Set configuration password 19352 q) Quit config 19353 e/n/d/r/c/s/q> n 19354 name> remote 19355 Type of storage to configure. 19356 Enter a string value. Press Enter for the default (""). 19357 Choose a number from below, or type in your own value 19358 [snip] 19359 XX / Microsoft OneDrive 19360 \ "onedrive" 19361 [snip] 19362 Storage> onedrive 19363 Microsoft App Client Id 19364 Leave blank normally. 19365 Enter a string value. Press Enter for the default (""). 19366 client_id> 19367 Microsoft App Client Secret 19368 Leave blank normally. 19369 Enter a string value. Press Enter for the default (""). 19370 client_secret> 19371 Edit advanced config? (y/n) 19372 y) Yes 19373 n) No 19374 y/n> n 19375 Remote config 19376 Use auto config? 19377 * Say Y if not sure 19378 * Say N if you are working on a remote or headless machine 19379 y) Yes 19380 n) No 19381 y/n> y 19382 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 19383 Log in and authorize rclone for access 19384 Waiting for code... 19385 Got code 19386 Choose a number from below, or type in an existing value 19387 1 / OneDrive Personal or Business 19388 \ "onedrive" 19389 2 / Sharepoint site 19390 \ "sharepoint" 19391 3 / Type in driveID 19392 \ "driveid" 19393 4 / Type in SiteID 19394 \ "siteid" 19395 5 / Search a Sharepoint site 19396 \ "search" 19397 Your choice> 1 19398 Found 1 drives, please select the one you want to use: 19399 0: OneDrive (business) id=b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk 19400 Chose drive to use:> 0 19401 Found drive 'root' of type 'business', URL: https://org-my.sharepoint.com/personal/you/Documents 19402 Is that okay? 19403 y) Yes 19404 n) No 19405 y/n> y 19406 -------------------- 19407 [remote] 19408 type = onedrive 19409 token = {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} 19410 drive_id = b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk 19411 drive_type = business 19412 -------------------- 19413 y) Yes this is OK 19414 e) Edit this remote 19415 d) Delete this remote 19416 y/e/d> y 19417 ``` 19418 19419 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 19420 machine with no Internet browser available. 19421 19422 Note that rclone runs a webserver on your local machine to collect the 19423 token as returned from Microsoft. This only runs from the moment it 19424 opens your browser to the moment you get back the verification 19425 code. This is on `http://127.0.0.1:53682/` and this it may require 19426 you to unblock it temporarily if you are running a host firewall. 19427 19428 Once configured you can then use `rclone` like this, 19429 19430 List directories in top level of your OneDrive 19431 19432 rclone lsd remote: 19433 19434 List all the files in your OneDrive 19435 19436 rclone ls remote: 19437 19438 To copy a local directory to an OneDrive directory called backup 19439 19440 rclone copy /home/source remote:backup 19441 19442 ### Getting your own Client ID and Key ### 19443 19444 You can use your own Client ID if the default (`client_id` left blank) 19445 one doesn't work for you or you see lots of throttling. The default 19446 Client ID and Key is shared by all rclone users when performing 19447 requests. 19448 19449 If you are having problems with them (E.g., seeing a lot of throttling), you can get your own 19450 Client ID and Key by following the steps below: 19451 19452 1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade, then click `New registration`. 19453 2. Enter a name for your app, choose account type `Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)`, select `Web` in `Redirect URI` Enter `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use. 19454 3. Under `manage` select `Certificates & secrets`, click `New client secret`. Copy and keep that secret for later use. 19455 4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`. 19456 5. Search and select the following permissions: `Files.Read`, `Files.ReadWrite`, `Files.Read.All`, `Files.ReadWrite.All`, `offline_access`, `User.Read`. Once selected click `Add permissions` at the bottom. 19457 19458 Now the application is complete. Run `rclone config` to create or edit a OneDrive remote. 19459 Supply the app ID and password as Client ID and Secret, respectively. rclone will walk you through the remaining steps. 19460 19461 ### Modification time and hashes ### 19462 19463 OneDrive allows modification times to be set on objects accurate to 1 19464 second. These will be used to detect whether objects need syncing or 19465 not. 19466 19467 OneDrive personal supports SHA1 type hashes. OneDrive for business and 19468 Sharepoint Server support 19469 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 19470 19471 For all types of OneDrive you can use the `--checksum` flag. 19472 19473 ### Restricted filename characters ### 19474 19475 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 19476 the following characters are also replaced: 19477 19478 | Character | Value | Replacement | 19479 | --------- |:-----:|:-----------:| 19480 | " | 0x22 | " | 19481 | * | 0x2A | * | 19482 | : | 0x3A | : | 19483 | < | 0x3C | < | 19484 | > | 0x3E | > | 19485 | ? | 0x3F | ? | 19486 | \ | 0x5C | \ | 19487 | \| | 0x7C | | | 19488 | # | 0x23 | # | 19489 | % | 0x25 | % | 19490 19491 File names can also not end with the following characters. 19492 These only get replaced if they are the last character in the name: 19493 19494 | Character | Value | Replacement | 19495 | --------- |:-----:|:-----------:| 19496 | SP | 0x20 | ␠ | 19497 | . | 0x2E | . | 19498 19499 File names can also not begin with the following characters. 19500 These only get replaced if they are the first character in the name: 19501 19502 | Character | Value | Replacement | 19503 | --------- |:-----:|:-----------:| 19504 | SP | 0x20 | ␠ | 19505 | ~ | 0x7E | ~ | 19506 19507 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 19508 as they can't be used in JSON strings. 19509 19510 ### Deleting files ### 19511 19512 Any files you delete with rclone will end up in the trash. Microsoft 19513 doesn't provide an API to permanently delete files, nor to empty the 19514 trash, so you will have to do that with one of Microsoft's apps or via 19515 the OneDrive website. 19516 19517 19518 ### Standard Options 19519 19520 Here are the standard options specific to onedrive (Microsoft OneDrive). 19521 19522 #### --onedrive-client-id 19523 19524 Microsoft App Client Id 19525 Leave blank normally. 19526 19527 - Config: client_id 19528 - Env Var: RCLONE_ONEDRIVE_CLIENT_ID 19529 - Type: string 19530 - Default: "" 19531 19532 #### --onedrive-client-secret 19533 19534 Microsoft App Client Secret 19535 Leave blank normally. 19536 19537 - Config: client_secret 19538 - Env Var: RCLONE_ONEDRIVE_CLIENT_SECRET 19539 - Type: string 19540 - Default: "" 19541 19542 ### Advanced Options 19543 19544 Here are the advanced options specific to onedrive (Microsoft OneDrive). 19545 19546 #### --onedrive-chunk-size 19547 19548 Chunk size to upload files with - must be multiple of 320k (327,680 bytes). 19549 19550 Above this size files will be chunked - must be multiple of 320k (327,680 bytes) and 19551 should not exceed 250M (262,144,000 bytes) else you may encounter \"Microsoft.SharePoint.Client.InvalidClientQueryException: The request message is too big.\" 19552 Note that the chunks will be buffered into memory. 19553 19554 - Config: chunk_size 19555 - Env Var: RCLONE_ONEDRIVE_CHUNK_SIZE 19556 - Type: SizeSuffix 19557 - Default: 10M 19558 19559 #### --onedrive-drive-id 19560 19561 The ID of the drive to use 19562 19563 - Config: drive_id 19564 - Env Var: RCLONE_ONEDRIVE_DRIVE_ID 19565 - Type: string 19566 - Default: "" 19567 19568 #### --onedrive-drive-type 19569 19570 The type of the drive ( personal | business | documentLibrary ) 19571 19572 - Config: drive_type 19573 - Env Var: RCLONE_ONEDRIVE_DRIVE_TYPE 19574 - Type: string 19575 - Default: "" 19576 19577 #### --onedrive-expose-onenote-files 19578 19579 Set to make OneNote files show up in directory listings. 19580 19581 By default rclone will hide OneNote files in directory listings because 19582 operations like "Open" and "Update" won't work on them. But this 19583 behaviour may also prevent you from deleting them. If you want to 19584 delete OneNote files or otherwise want them to show up in directory 19585 listing, set this option. 19586 19587 - Config: expose_onenote_files 19588 - Env Var: RCLONE_ONEDRIVE_EXPOSE_ONENOTE_FILES 19589 - Type: bool 19590 - Default: false 19591 19592 #### --onedrive-server-side-across-configs 19593 19594 Allow server side operations (eg copy) to work across different onedrive configs. 19595 19596 This can be useful if you wish to do a server side copy between two 19597 different Onedrives. Note that this isn't enabled by default 19598 because it isn't easy to tell if it will work between any two 19599 configurations. 19600 19601 - Config: server_side_across_configs 19602 - Env Var: RCLONE_ONEDRIVE_SERVER_SIDE_ACROSS_CONFIGS 19603 - Type: bool 19604 - Default: false 19605 19606 #### --onedrive-encoding 19607 19608 This sets the encoding for the backend. 19609 19610 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 19611 19612 - Config: encoding 19613 - Env Var: RCLONE_ONEDRIVE_ENCODING 19614 - Type: MultiEncoder 19615 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot 19616 19617 19618 19619 ### Limitations ### 19620 19621 #### Naming #### 19622 19623 Note that OneDrive is case insensitive so you can't have a 19624 file called "Hello.doc" and one called "hello.doc". 19625 19626 There are quite a few characters that can't be in OneDrive file 19627 names. These can't occur on Windows platforms, but on non-Windows 19628 platforms they are common. Rclone will map these names to and from an 19629 identical looking unicode equivalent. For example if a file has a `?` 19630 in it will be mapped to `?` instead. 19631 19632 #### File sizes #### 19633 19634 The largest allowed file sizes are 15GB for OneDrive for Business and 100GB for OneDrive Personal (Updated 19 May 2020). 19635 Source: 19636 https://support.office.com/en-us/article/upload-photos-and-files-to-onedrive-b00ad3fe-6643-4b16-9212-de00ef02b586 19637 19638 #### Path length #### 19639 19640 The entire path, including the file name, must contain fewer than 400 characters for OneDrive, OneDrive for Business and SharePoint Online. If you are encrypting file and folder names with rclone, you may want to pay attention to this limitation because the encrypted names are typically longer than the original ones. 19641 19642 #### Number of files #### 19643 19644 OneDrive seems to be OK with at least 50,000 files in a folder, but at 19645 100,000 rclone will get errors listing the directory like `couldn’t 19646 list files: UnknownError:`. See 19647 [#2707](https://github.com/rclone/rclone/issues/2707) for more info. 19648 19649 An official document about the limitations for different types of OneDrive can be found [here](https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa). 19650 19651 ### Versioning issue ### 19652 19653 Every change in OneDrive causes the service to create a new version. 19654 This counts against a users quota. 19655 For example changing the modification time of a file creates a second 19656 version, so the file is using twice the space. 19657 19658 The `copy` is the only rclone command affected by this as we copy 19659 the file and then afterwards set the modification time to match the 19660 source file. 19661 19662 **Note**: Starting October 2018, users will no longer be able to disable versioning by default. This is because Microsoft has brought an [update](https://techcommunity.microsoft.com/t5/Microsoft-OneDrive-Blog/New-Updates-to-OneDrive-and-SharePoint-Team-Site-Versioning/ba-p/204390) to the mechanism. To change this new default setting, a PowerShell command is required to be run by a SharePoint admin. If you are an admin, you can run these commands in PowerShell to change that setting: 19663 19664 1. `Install-Module -Name Microsoft.Online.SharePoint.PowerShell` (in case you haven't installed this already) 19665 1. `Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking` 19666 1. `Connect-SPOService -Url https://YOURSITE-admin.sharepoint.com -Credential YOU@YOURSITE.COM` (replacing `YOURSITE`, `YOU`, `YOURSITE.COM` with the actual values; this will prompt for your credentials) 19667 1. `Set-SPOTenant -EnableMinimumVersionRequirement $False` 19668 1. `Disconnect-SPOService` (to disconnect from the server) 19669 19670 *Below are the steps for normal users to disable versioning. If you don't see the "No Versioning" option, make sure the above requirements are met.* 19671 19672 User [Weropol](https://github.com/Weropol) has found a method to disable 19673 versioning on OneDrive 19674 19675 1. Open the settings menu by clicking on the gear symbol at the top of the OneDrive Business page. 19676 2. Click Site settings. 19677 3. Once on the Site settings page, navigate to Site Administration > Site libraries and lists. 19678 4. Click Customize "Documents". 19679 5. Click General Settings > Versioning Settings. 19680 6. Under Document Version History select the option No versioning. 19681 Note: This will disable the creation of new file versions, but will not remove any previous versions. Your documents are safe. 19682 7. Apply the changes by clicking OK. 19683 8. Use rclone to upload or modify files. (I also use the --no-update-modtime flag) 19684 9. Restore the versioning settings after using rclone. (Optional) 19685 19686 ### Troubleshooting ### 19687 19688 #### Unexpected file size/hash differences on Sharepoint #### 19689 19690 It is a 19691 [known](https://github.com/OneDrive/onedrive-api-docs/issues/935#issuecomment-441741631) 19692 issue that Sharepoint (not OneDrive or OneDrive for Business) silently modifies 19693 uploaded files, mainly Office files (.docx, .xlsx, etc.), causing file size and 19694 hash checks to fail. To use rclone with such affected files on Sharepoint, you 19695 may disable these checks with the following command line arguments: 19696 19697 ``` 19698 --ignore-checksum --ignore-size 19699 ``` 19700 19701 #### Replacing/deleting existing files on Sharepoint gets "item not found" #### 19702 19703 It is a [known](https://github.com/OneDrive/onedrive-api-docs/issues/1068) issue 19704 that Sharepoint (not OneDrive or OneDrive for Business) may return "item not 19705 found" errors when users try to replace or delete uploaded files; this seems to 19706 mainly affect Office files (.docx, .xlsx, etc.). As a workaround, you may use 19707 the `--backup-dir <BACKUP_DIR>` command line argument so rclone moves the 19708 files to be replaced/deleted into a given backup directory (instead of directly 19709 replacing/deleting them). For example, to instruct rclone to move the files into 19710 the directory `rclone-backup-dir` on backend `mysharepoint`, you may use: 19711 19712 ``` 19713 --backup-dir mysharepoint:rclone-backup-dir 19714 ``` 19715 19716 #### access\_denied (AADSTS65005) #### 19717 19718 ``` 19719 Error: access_denied 19720 Code: AADSTS65005 19721 Description: Using application 'rclone' is currently not supported for your organization [YOUR_ORGANIZATION] because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of [YOUR_ORGANIZATION] before the application rclone can be provisioned. 19722 ``` 19723 19724 This means that rclone can't use the OneDrive for Business API with your account. You can't do much about it, maybe write an email to your admins. 19725 19726 However, there are other ways to interact with your OneDrive account. Have a look at the webdav backend: https://rclone.org/webdav/#sharepoint 19727 19728 #### invalid\_grant (AADSTS50076) #### 19729 19730 ``` 19731 Error: invalid_grant 19732 Code: AADSTS50076 19733 Description: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '...'. 19734 ``` 19735 19736 If you see the error above after enabling multi-factor authentication for your account, you can fix it by refreshing your OAuth refresh token. To do that, run `rclone config`, and choose to edit your OneDrive backend. Then, you don't need to actually make any changes until you reach this question: `Already have a token - refresh?`. For this question, answer `y` and go through the process to refresh your token, just like the first time the backend is configured. After this, rclone should work again for this backend. 19737 19738 OpenDrive 19739 ------------------------------------ 19740 19741 Paths are specified as `remote:path` 19742 19743 Paths may be as deep as required, eg `remote:directory/subdirectory`. 19744 19745 Here is an example of how to make a remote called `remote`. First run: 19746 19747 rclone config 19748 19749 This will guide you through an interactive setup process: 19750 19751 ``` 19752 n) New remote 19753 d) Delete remote 19754 q) Quit config 19755 e/n/d/q> n 19756 name> remote 19757 Type of storage to configure. 19758 Choose a number from below, or type in your own value 19759 [snip] 19760 XX / OpenDrive 19761 \ "opendrive" 19762 [snip] 19763 Storage> opendrive 19764 Username 19765 username> 19766 Password 19767 y) Yes type in my own password 19768 g) Generate random password 19769 y/g> y 19770 Enter the password: 19771 password: 19772 Confirm the password: 19773 password: 19774 -------------------- 19775 [remote] 19776 username = 19777 password = *** ENCRYPTED *** 19778 -------------------- 19779 y) Yes this is OK 19780 e) Edit this remote 19781 d) Delete this remote 19782 y/e/d> y 19783 ``` 19784 19785 List directories in top level of your OpenDrive 19786 19787 rclone lsd remote: 19788 19789 List all the files in your OpenDrive 19790 19791 rclone ls remote: 19792 19793 To copy a local directory to an OpenDrive directory called backup 19794 19795 rclone copy /home/source remote:backup 19796 19797 ### Modified time and MD5SUMs ### 19798 19799 OpenDrive allows modification times to be set on objects accurate to 1 19800 second. These will be used to detect whether objects need syncing or 19801 not. 19802 19803 #### Restricted filename characters 19804 19805 | Character | Value | Replacement | 19806 | --------- |:-----:|:-----------:| 19807 | NUL | 0x00 | ␀ | 19808 | / | 0x2F | / | 19809 | " | 0x22 | " | 19810 | * | 0x2A | * | 19811 | : | 0x3A | : | 19812 | < | 0x3C | < | 19813 | > | 0x3E | > | 19814 | ? | 0x3F | ? | 19815 | \ | 0x5C | \ | 19816 | \| | 0x7C | | | 19817 19818 File names can also not begin or end with the following characters. 19819 These only get replaced if they are the first or last character in the name: 19820 19821 | Character | Value | Replacement | 19822 | --------- |:-----:|:-----------:| 19823 | SP | 0x20 | ␠ | 19824 | HT | 0x09 | ␉ | 19825 | LF | 0x0A | ␊ | 19826 | VT | 0x0B | ␋ | 19827 | CR | 0x0D | ␍ | 19828 19829 19830 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 19831 as they can't be used in JSON strings. 19832 19833 19834 ### Standard Options 19835 19836 Here are the standard options specific to opendrive (OpenDrive). 19837 19838 #### --opendrive-username 19839 19840 Username 19841 19842 - Config: username 19843 - Env Var: RCLONE_OPENDRIVE_USERNAME 19844 - Type: string 19845 - Default: "" 19846 19847 #### --opendrive-password 19848 19849 Password. 19850 19851 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 19852 19853 - Config: password 19854 - Env Var: RCLONE_OPENDRIVE_PASSWORD 19855 - Type: string 19856 - Default: "" 19857 19858 ### Advanced Options 19859 19860 Here are the advanced options specific to opendrive (OpenDrive). 19861 19862 #### --opendrive-encoding 19863 19864 This sets the encoding for the backend. 19865 19866 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 19867 19868 - Config: encoding 19869 - Env Var: RCLONE_OPENDRIVE_ENCODING 19870 - Type: MultiEncoder 19871 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot 19872 19873 #### --opendrive-chunk-size 19874 19875 Files will be uploaded in chunks this size. 19876 19877 Note that these chunks are buffered in memory so increasing them will 19878 increase memory use. 19879 19880 - Config: chunk_size 19881 - Env Var: RCLONE_OPENDRIVE_CHUNK_SIZE 19882 - Type: SizeSuffix 19883 - Default: 10M 19884 19885 19886 19887 ### Limitations ### 19888 19889 Note that OpenDrive is case insensitive so you can't have a 19890 file called "Hello.doc" and one called "hello.doc". 19891 19892 There are quite a few characters that can't be in OpenDrive file 19893 names. These can't occur on Windows platforms, but on non-Windows 19894 platforms they are common. Rclone will map these names to and from an 19895 identical looking unicode equivalent. For example if a file has a `?` 19896 in it will be mapped to `?` instead. 19897 19898 QingStor 19899 --------------------------------------- 19900 19901 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 19902 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 19903 19904 Here is an example of making an QingStor configuration. First run 19905 19906 rclone config 19907 19908 This will guide you through an interactive setup process. 19909 19910 ``` 19911 No remotes found - make a new one 19912 n) New remote 19913 r) Rename remote 19914 c) Copy remote 19915 s) Set configuration password 19916 q) Quit config 19917 n/r/c/s/q> n 19918 name> remote 19919 Type of storage to configure. 19920 Choose a number from below, or type in your own value 19921 [snip] 19922 XX / QingStor Object Storage 19923 \ "qingstor" 19924 [snip] 19925 Storage> qingstor 19926 Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 19927 Choose a number from below, or type in your own value 19928 1 / Enter QingStor credentials in the next step 19929 \ "false" 19930 2 / Get QingStor credentials from the environment (env vars or IAM) 19931 \ "true" 19932 env_auth> 1 19933 QingStor Access Key ID - leave blank for anonymous access or runtime credentials. 19934 access_key_id> access_key 19935 QingStor Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 19936 secret_access_key> secret_key 19937 Enter an endpoint URL to connection QingStor API. 19938 Leave blank will use the default value "https://qingstor.com:443" 19939 endpoint> 19940 Zone connect to. Default is "pek3a". 19941 Choose a number from below, or type in your own value 19942 / The Beijing (China) Three Zone 19943 1 | Needs location constraint pek3a. 19944 \ "pek3a" 19945 / The Shanghai (China) First Zone 19946 2 | Needs location constraint sh1a. 19947 \ "sh1a" 19948 zone> 1 19949 Number of connection retry. 19950 Leave blank will use the default value "3". 19951 connection_retries> 19952 Remote config 19953 -------------------- 19954 [remote] 19955 env_auth = false 19956 access_key_id = access_key 19957 secret_access_key = secret_key 19958 endpoint = 19959 zone = pek3a 19960 connection_retries = 19961 -------------------- 19962 y) Yes this is OK 19963 e) Edit this remote 19964 d) Delete this remote 19965 y/e/d> y 19966 ``` 19967 19968 This remote is called `remote` and can now be used like this 19969 19970 See all buckets 19971 19972 rclone lsd remote: 19973 19974 Make a new bucket 19975 19976 rclone mkdir remote:bucket 19977 19978 List the contents of a bucket 19979 19980 rclone ls remote:bucket 19981 19982 Sync `/home/local/directory` to the remote bucket, deleting any excess 19983 files in the bucket. 19984 19985 rclone sync /home/local/directory remote:bucket 19986 19987 ### --fast-list ### 19988 19989 This remote supports `--fast-list` which allows you to use fewer 19990 transactions in exchange for more memory. See the [rclone 19991 docs](https://rclone.org/docs/#fast-list) for more details. 19992 19993 ### Multipart uploads ### 19994 19995 rclone supports multipart uploads with QingStor which means that it can 19996 upload files bigger than 5GB. Note that files uploaded with multipart 19997 upload don't have an MD5SUM. 19998 19999 Note that incomplete multipart uploads older than 24 hours can be 20000 removed with `rclone cleanup remote:bucket` just for one bucket 20001 `rclone cleanup remote:` for all buckets. QingStor does not ever 20002 remove incomplete multipart uploads so it may be necessary to run this 20003 from time to time. 20004 20005 ### Buckets and Zone ### 20006 20007 With QingStor you can list buckets (`rclone lsd`) using any zone, 20008 but you can only access the content of a bucket from the zone it was 20009 created in. If you attempt to access a bucket from the wrong zone, 20010 you will get an error, `incorrect zone, the bucket is not in 'XXX' 20011 zone`. 20012 20013 ### Authentication ### 20014 20015 There are two ways to supply `rclone` with a set of QingStor 20016 credentials. In order of precedence: 20017 20018 - Directly in the rclone configuration file (as configured by `rclone config`) 20019 - set `access_key_id` and `secret_access_key` 20020 - Runtime configuration: 20021 - set `env_auth` to `true` in the config file 20022 - Exporting the following environment variables before running `rclone` 20023 - Access Key ID: `QS_ACCESS_KEY_ID` or `QS_ACCESS_KEY` 20024 - Secret Access Key: `QS_SECRET_ACCESS_KEY` or `QS_SECRET_KEY` 20025 20026 ### Restricted filename characters 20027 20028 The control characters 0x00-0x1F and / are replaced as in the [default 20029 restricted characters set](https://rclone.org/overview/#restricted-characters). Note 20030 that 0x7F is not replaced. 20031 20032 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 20033 as they can't be used in JSON strings. 20034 20035 20036 ### Standard Options 20037 20038 Here are the standard options specific to qingstor (QingCloud Object Storage). 20039 20040 #### --qingstor-env-auth 20041 20042 Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 20043 20044 - Config: env_auth 20045 - Env Var: RCLONE_QINGSTOR_ENV_AUTH 20046 - Type: bool 20047 - Default: false 20048 - Examples: 20049 - "false" 20050 - Enter QingStor credentials in the next step 20051 - "true" 20052 - Get QingStor credentials from the environment (env vars or IAM) 20053 20054 #### --qingstor-access-key-id 20055 20056 QingStor Access Key ID 20057 Leave blank for anonymous access or runtime credentials. 20058 20059 - Config: access_key_id 20060 - Env Var: RCLONE_QINGSTOR_ACCESS_KEY_ID 20061 - Type: string 20062 - Default: "" 20063 20064 #### --qingstor-secret-access-key 20065 20066 QingStor Secret Access Key (password) 20067 Leave blank for anonymous access or runtime credentials. 20068 20069 - Config: secret_access_key 20070 - Env Var: RCLONE_QINGSTOR_SECRET_ACCESS_KEY 20071 - Type: string 20072 - Default: "" 20073 20074 #### --qingstor-endpoint 20075 20076 Enter an endpoint URL to connection QingStor API. 20077 Leave blank will use the default value "https://qingstor.com:443" 20078 20079 - Config: endpoint 20080 - Env Var: RCLONE_QINGSTOR_ENDPOINT 20081 - Type: string 20082 - Default: "" 20083 20084 #### --qingstor-zone 20085 20086 Zone to connect to. 20087 Default is "pek3a". 20088 20089 - Config: zone 20090 - Env Var: RCLONE_QINGSTOR_ZONE 20091 - Type: string 20092 - Default: "" 20093 - Examples: 20094 - "pek3a" 20095 - The Beijing (China) Three Zone 20096 - Needs location constraint pek3a. 20097 - "sh1a" 20098 - The Shanghai (China) First Zone 20099 - Needs location constraint sh1a. 20100 - "gd2a" 20101 - The Guangdong (China) Second Zone 20102 - Needs location constraint gd2a. 20103 20104 ### Advanced Options 20105 20106 Here are the advanced options specific to qingstor (QingCloud Object Storage). 20107 20108 #### --qingstor-connection-retries 20109 20110 Number of connection retries. 20111 20112 - Config: connection_retries 20113 - Env Var: RCLONE_QINGSTOR_CONNECTION_RETRIES 20114 - Type: int 20115 - Default: 3 20116 20117 #### --qingstor-upload-cutoff 20118 20119 Cutoff for switching to chunked upload 20120 20121 Any files larger than this will be uploaded in chunks of chunk_size. 20122 The minimum is 0 and the maximum is 5GB. 20123 20124 - Config: upload_cutoff 20125 - Env Var: RCLONE_QINGSTOR_UPLOAD_CUTOFF 20126 - Type: SizeSuffix 20127 - Default: 200M 20128 20129 #### --qingstor-chunk-size 20130 20131 Chunk size to use for uploading. 20132 20133 When uploading files larger than upload_cutoff they will be uploaded 20134 as multipart uploads using this chunk size. 20135 20136 Note that "--qingstor-upload-concurrency" chunks of this size are buffered 20137 in memory per transfer. 20138 20139 If you are transferring large files over high speed links and you have 20140 enough memory, then increasing this will speed up the transfers. 20141 20142 - Config: chunk_size 20143 - Env Var: RCLONE_QINGSTOR_CHUNK_SIZE 20144 - Type: SizeSuffix 20145 - Default: 4M 20146 20147 #### --qingstor-upload-concurrency 20148 20149 Concurrency for multipart uploads. 20150 20151 This is the number of chunks of the same file that are uploaded 20152 concurrently. 20153 20154 NB if you set this to > 1 then the checksums of multpart uploads 20155 become corrupted (the uploads themselves are not corrupted though). 20156 20157 If you are uploading small numbers of large file over high speed link 20158 and these uploads do not fully utilize your bandwidth, then increasing 20159 this may help to speed up the transfers. 20160 20161 - Config: upload_concurrency 20162 - Env Var: RCLONE_QINGSTOR_UPLOAD_CONCURRENCY 20163 - Type: int 20164 - Default: 1 20165 20166 #### --qingstor-encoding 20167 20168 This sets the encoding for the backend. 20169 20170 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 20171 20172 - Config: encoding 20173 - Env Var: RCLONE_QINGSTOR_ENCODING 20174 - Type: MultiEncoder 20175 - Default: Slash,Ctl,InvalidUtf8 20176 20177 20178 20179 Swift 20180 ---------------------------------------- 20181 20182 Swift refers to [OpenStack Object Storage](https://docs.openstack.org/swift/latest/). 20183 Commercial implementations of that being: 20184 20185 * [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/) 20186 * [Memset Memstore](https://www.memset.com/cloud/storage/) 20187 * [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/) 20188 * [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc) 20189 * [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html) 20190 20191 Paths are specified as `remote:container` (or `remote:` for the `lsd` 20192 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 20193 20194 Here is an example of making a swift configuration. First run 20195 20196 rclone config 20197 20198 This will guide you through an interactive setup process. 20199 20200 ``` 20201 No remotes found - make a new one 20202 n) New remote 20203 s) Set configuration password 20204 q) Quit config 20205 n/s/q> n 20206 name> remote 20207 Type of storage to configure. 20208 Choose a number from below, or type in your own value 20209 [snip] 20210 XX / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 20211 \ "swift" 20212 [snip] 20213 Storage> swift 20214 Get swift credentials from environment variables in standard OpenStack form. 20215 Choose a number from below, or type in your own value 20216 1 / Enter swift credentials in the next step 20217 \ "false" 20218 2 / Get swift credentials from environment vars. Leave other fields blank if using this. 20219 \ "true" 20220 env_auth> true 20221 User name to log in (OS_USERNAME). 20222 user> 20223 API key or password (OS_PASSWORD). 20224 key> 20225 Authentication URL for server (OS_AUTH_URL). 20226 Choose a number from below, or type in your own value 20227 1 / Rackspace US 20228 \ "https://auth.api.rackspacecloud.com/v1.0" 20229 2 / Rackspace UK 20230 \ "https://lon.auth.api.rackspacecloud.com/v1.0" 20231 3 / Rackspace v2 20232 \ "https://identity.api.rackspacecloud.com/v2.0" 20233 4 / Memset Memstore UK 20234 \ "https://auth.storage.memset.com/v1.0" 20235 5 / Memset Memstore UK v2 20236 \ "https://auth.storage.memset.com/v2.0" 20237 6 / OVH 20238 \ "https://auth.cloud.ovh.net/v3" 20239 auth> 20240 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 20241 user_id> 20242 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 20243 domain> 20244 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 20245 tenant> 20246 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 20247 tenant_id> 20248 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 20249 tenant_domain> 20250 Region name - optional (OS_REGION_NAME) 20251 region> 20252 Storage URL - optional (OS_STORAGE_URL) 20253 storage_url> 20254 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 20255 auth_token> 20256 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 20257 auth_version> 20258 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 20259 Choose a number from below, or type in your own value 20260 1 / Public (default, choose this if not sure) 20261 \ "public" 20262 2 / Internal (use internal service net) 20263 \ "internal" 20264 3 / Admin 20265 \ "admin" 20266 endpoint_type> 20267 Remote config 20268 -------------------- 20269 [test] 20270 env_auth = true 20271 user = 20272 key = 20273 auth = 20274 user_id = 20275 domain = 20276 tenant = 20277 tenant_id = 20278 tenant_domain = 20279 region = 20280 storage_url = 20281 auth_token = 20282 auth_version = 20283 endpoint_type = 20284 -------------------- 20285 y) Yes this is OK 20286 e) Edit this remote 20287 d) Delete this remote 20288 y/e/d> y 20289 ``` 20290 20291 This remote is called `remote` and can now be used like this 20292 20293 See all containers 20294 20295 rclone lsd remote: 20296 20297 Make a new container 20298 20299 rclone mkdir remote:container 20300 20301 List the contents of a container 20302 20303 rclone ls remote:container 20304 20305 Sync `/home/local/directory` to the remote container, deleting any 20306 excess files in the container. 20307 20308 rclone sync /home/local/directory remote:container 20309 20310 ### Configuration from an OpenStack credentials file ### 20311 20312 An OpenStack credentials file typically looks something something 20313 like this (without the comments) 20314 20315 ``` 20316 export OS_AUTH_URL=https://a.provider.net/v2.0 20317 export OS_TENANT_ID=ffffffffffffffffffffffffffffffff 20318 export OS_TENANT_NAME="1234567890123456" 20319 export OS_USERNAME="123abc567xy" 20320 echo "Please enter your OpenStack Password: " 20321 read -sr OS_PASSWORD_INPUT 20322 export OS_PASSWORD=$OS_PASSWORD_INPUT 20323 export OS_REGION_NAME="SBG1" 20324 if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi 20325 ``` 20326 20327 The config file needs to look something like this where `$OS_USERNAME` 20328 represents the value of the `OS_USERNAME` variable - `123abc567xy` in 20329 the example above. 20330 20331 ``` 20332 [remote] 20333 type = swift 20334 user = $OS_USERNAME 20335 key = $OS_PASSWORD 20336 auth = $OS_AUTH_URL 20337 tenant = $OS_TENANT_NAME 20338 ``` 20339 20340 Note that you may (or may not) need to set `region` too - try without first. 20341 20342 ### Configuration from the environment ### 20343 20344 If you prefer you can configure rclone to use swift using a standard 20345 set of OpenStack environment variables. 20346 20347 When you run through the config, make sure you choose `true` for 20348 `env_auth` and leave everything else blank. 20349 20350 rclone will then set any empty config parameters from the environment 20351 using standard OpenStack environment variables. There is [a list of 20352 the 20353 variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment) 20354 in the docs for the swift library. 20355 20356 ### Using an alternate authentication method ### 20357 20358 If your OpenStack installation uses a non-standard authentication method 20359 that might not be yet supported by rclone or the underlying swift library, 20360 you can authenticate externally (e.g. calling manually the `openstack` 20361 commands to get a token). Then, you just need to pass the two 20362 configuration variables ``auth_token`` and ``storage_url``. 20363 If they are both provided, the other variables are ignored. rclone will 20364 not try to authenticate but instead assume it is already authenticated 20365 and use these two variables to access the OpenStack installation. 20366 20367 #### Using rclone without a config file #### 20368 20369 You can use rclone with swift without a config file, if desired, like 20370 this: 20371 20372 ``` 20373 source openstack-credentials-file 20374 export RCLONE_CONFIG_MYREMOTE_TYPE=swift 20375 export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true 20376 rclone lsd myremote: 20377 ``` 20378 20379 ### --fast-list ### 20380 20381 This remote supports `--fast-list` which allows you to use fewer 20382 transactions in exchange for more memory. See the [rclone 20383 docs](https://rclone.org/docs/#fast-list) for more details. 20384 20385 ### --update and --use-server-modtime ### 20386 20387 As noted below, the modified time is stored on metadata on the object. It is 20388 used by default for all operations that require checking the time a file was 20389 last updated. It allows rclone to treat the remote more like a true filesystem, 20390 but it is inefficient because it requires an extra API call to retrieve the 20391 metadata. 20392 20393 For many operations, the time the object was last uploaded to the remote is 20394 sufficient to determine if it is "dirty". By using `--update` along with 20395 `--use-server-modtime`, you can avoid the extra API call and simply upload 20396 files whose local modtime is newer than the time it was last uploaded. 20397 20398 20399 ### Standard Options 20400 20401 Here are the standard options specific to swift (OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 20402 20403 #### --swift-env-auth 20404 20405 Get swift credentials from environment variables in standard OpenStack form. 20406 20407 - Config: env_auth 20408 - Env Var: RCLONE_SWIFT_ENV_AUTH 20409 - Type: bool 20410 - Default: false 20411 - Examples: 20412 - "false" 20413 - Enter swift credentials in the next step 20414 - "true" 20415 - Get swift credentials from environment vars. Leave other fields blank if using this. 20416 20417 #### --swift-user 20418 20419 User name to log in (OS_USERNAME). 20420 20421 - Config: user 20422 - Env Var: RCLONE_SWIFT_USER 20423 - Type: string 20424 - Default: "" 20425 20426 #### --swift-key 20427 20428 API key or password (OS_PASSWORD). 20429 20430 - Config: key 20431 - Env Var: RCLONE_SWIFT_KEY 20432 - Type: string 20433 - Default: "" 20434 20435 #### --swift-auth 20436 20437 Authentication URL for server (OS_AUTH_URL). 20438 20439 - Config: auth 20440 - Env Var: RCLONE_SWIFT_AUTH 20441 - Type: string 20442 - Default: "" 20443 - Examples: 20444 - "https://auth.api.rackspacecloud.com/v1.0" 20445 - Rackspace US 20446 - "https://lon.auth.api.rackspacecloud.com/v1.0" 20447 - Rackspace UK 20448 - "https://identity.api.rackspacecloud.com/v2.0" 20449 - Rackspace v2 20450 - "https://auth.storage.memset.com/v1.0" 20451 - Memset Memstore UK 20452 - "https://auth.storage.memset.com/v2.0" 20453 - Memset Memstore UK v2 20454 - "https://auth.cloud.ovh.net/v3" 20455 - OVH 20456 20457 #### --swift-user-id 20458 20459 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 20460 20461 - Config: user_id 20462 - Env Var: RCLONE_SWIFT_USER_ID 20463 - Type: string 20464 - Default: "" 20465 20466 #### --swift-domain 20467 20468 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 20469 20470 - Config: domain 20471 - Env Var: RCLONE_SWIFT_DOMAIN 20472 - Type: string 20473 - Default: "" 20474 20475 #### --swift-tenant 20476 20477 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 20478 20479 - Config: tenant 20480 - Env Var: RCLONE_SWIFT_TENANT 20481 - Type: string 20482 - Default: "" 20483 20484 #### --swift-tenant-id 20485 20486 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 20487 20488 - Config: tenant_id 20489 - Env Var: RCLONE_SWIFT_TENANT_ID 20490 - Type: string 20491 - Default: "" 20492 20493 #### --swift-tenant-domain 20494 20495 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 20496 20497 - Config: tenant_domain 20498 - Env Var: RCLONE_SWIFT_TENANT_DOMAIN 20499 - Type: string 20500 - Default: "" 20501 20502 #### --swift-region 20503 20504 Region name - optional (OS_REGION_NAME) 20505 20506 - Config: region 20507 - Env Var: RCLONE_SWIFT_REGION 20508 - Type: string 20509 - Default: "" 20510 20511 #### --swift-storage-url 20512 20513 Storage URL - optional (OS_STORAGE_URL) 20514 20515 - Config: storage_url 20516 - Env Var: RCLONE_SWIFT_STORAGE_URL 20517 - Type: string 20518 - Default: "" 20519 20520 #### --swift-auth-token 20521 20522 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 20523 20524 - Config: auth_token 20525 - Env Var: RCLONE_SWIFT_AUTH_TOKEN 20526 - Type: string 20527 - Default: "" 20528 20529 #### --swift-application-credential-id 20530 20531 Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 20532 20533 - Config: application_credential_id 20534 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID 20535 - Type: string 20536 - Default: "" 20537 20538 #### --swift-application-credential-name 20539 20540 Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 20541 20542 - Config: application_credential_name 20543 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME 20544 - Type: string 20545 - Default: "" 20546 20547 #### --swift-application-credential-secret 20548 20549 Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 20550 20551 - Config: application_credential_secret 20552 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET 20553 - Type: string 20554 - Default: "" 20555 20556 #### --swift-auth-version 20557 20558 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 20559 20560 - Config: auth_version 20561 - Env Var: RCLONE_SWIFT_AUTH_VERSION 20562 - Type: int 20563 - Default: 0 20564 20565 #### --swift-endpoint-type 20566 20567 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 20568 20569 - Config: endpoint_type 20570 - Env Var: RCLONE_SWIFT_ENDPOINT_TYPE 20571 - Type: string 20572 - Default: "public" 20573 - Examples: 20574 - "public" 20575 - Public (default, choose this if not sure) 20576 - "internal" 20577 - Internal (use internal service net) 20578 - "admin" 20579 - Admin 20580 20581 #### --swift-storage-policy 20582 20583 The storage policy to use when creating a new container 20584 20585 This applies the specified storage policy when creating a new 20586 container. The policy cannot be changed afterwards. The allowed 20587 configuration values and their meaning depend on your Swift storage 20588 provider. 20589 20590 - Config: storage_policy 20591 - Env Var: RCLONE_SWIFT_STORAGE_POLICY 20592 - Type: string 20593 - Default: "" 20594 - Examples: 20595 - "" 20596 - Default 20597 - "pcs" 20598 - OVH Public Cloud Storage 20599 - "pca" 20600 - OVH Public Cloud Archive 20601 20602 ### Advanced Options 20603 20604 Here are the advanced options specific to swift (OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 20605 20606 #### --swift-chunk-size 20607 20608 Above this size files will be chunked into a _segments container. 20609 20610 Above this size files will be chunked into a _segments container. The 20611 default for this is 5GB which is its maximum value. 20612 20613 - Config: chunk_size 20614 - Env Var: RCLONE_SWIFT_CHUNK_SIZE 20615 - Type: SizeSuffix 20616 - Default: 5G 20617 20618 #### --swift-no-chunk 20619 20620 Don't chunk files during streaming upload. 20621 20622 When doing streaming uploads (eg using rcat or mount) setting this 20623 flag will cause the swift backend to not upload chunked files. 20624 20625 This will limit the maximum upload size to 5GB. However non chunked 20626 files are easier to deal with and have an MD5SUM. 20627 20628 Rclone will still chunk files bigger than chunk_size when doing normal 20629 copy operations. 20630 20631 - Config: no_chunk 20632 - Env Var: RCLONE_SWIFT_NO_CHUNK 20633 - Type: bool 20634 - Default: false 20635 20636 #### --swift-encoding 20637 20638 This sets the encoding for the backend. 20639 20640 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 20641 20642 - Config: encoding 20643 - Env Var: RCLONE_SWIFT_ENCODING 20644 - Type: MultiEncoder 20645 - Default: Slash,InvalidUtf8 20646 20647 20648 20649 ### Modified time ### 20650 20651 The modified time is stored as metadata on the object as 20652 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 20653 ns. 20654 20655 This is a de facto standard (used in the official python-swiftclient 20656 amongst others) for storing the modification time for an object. 20657 20658 ### Restricted filename characters 20659 20660 | Character | Value | Replacement | 20661 | --------- |:-----:|:-----------:| 20662 | NUL | 0x00 | ␀ | 20663 | / | 0x2F | / | 20664 20665 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 20666 as they can't be used in JSON strings. 20667 20668 ### Limitations ### 20669 20670 The Swift API doesn't return a correct MD5SUM for segmented files 20671 (Dynamic or Static Large Objects) so rclone won't check or use the 20672 MD5SUM for these. 20673 20674 ### Troubleshooting ### 20675 20676 #### Rclone gives Failed to create file system for "remote:": Bad Request #### 20677 20678 Due to an oddity of the underlying swift library, it gives a "Bad 20679 Request" error rather than a more sensible error when the 20680 authentication fails for Swift. 20681 20682 So this most likely means your username / password is wrong. You can 20683 investigate further with the `--dump-bodies` flag. 20684 20685 This may also be caused by specifying the region when you shouldn't 20686 have (eg OVH). 20687 20688 #### Rclone gives Failed to create file system: Response didn't have storage url and auth token #### 20689 20690 This is most likely caused by forgetting to specify your tenant when 20691 setting up a swift remote. 20692 20693 pCloud 20694 ----------------------------------------- 20695 20696 Paths are specified as `remote:path` 20697 20698 Paths may be as deep as required, eg `remote:directory/subdirectory`. 20699 20700 The initial setup for pCloud involves getting a token from pCloud which you 20701 need to do in your browser. `rclone config` walks you through it. 20702 20703 Here is an example of how to make a remote called `remote`. First run: 20704 20705 rclone config 20706 20707 This will guide you through an interactive setup process: 20708 20709 ``` 20710 No remotes found - make a new one 20711 n) New remote 20712 s) Set configuration password 20713 q) Quit config 20714 n/s/q> n 20715 name> remote 20716 Type of storage to configure. 20717 Choose a number from below, or type in your own value 20718 [snip] 20719 XX / Pcloud 20720 \ "pcloud" 20721 [snip] 20722 Storage> pcloud 20723 Pcloud App Client Id - leave blank normally. 20724 client_id> 20725 Pcloud App Client Secret - leave blank normally. 20726 client_secret> 20727 Remote config 20728 Use auto config? 20729 * Say Y if not sure 20730 * Say N if you are working on a remote or headless machine 20731 y) Yes 20732 n) No 20733 y/n> y 20734 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 20735 Log in and authorize rclone for access 20736 Waiting for code... 20737 Got code 20738 -------------------- 20739 [remote] 20740 client_id = 20741 client_secret = 20742 token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} 20743 -------------------- 20744 y) Yes this is OK 20745 e) Edit this remote 20746 d) Delete this remote 20747 y/e/d> y 20748 ``` 20749 20750 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 20751 machine with no Internet browser available. 20752 20753 Note that rclone runs a webserver on your local machine to collect the 20754 token as returned from pCloud. This only runs from the moment it opens 20755 your browser to the moment you get back the verification code. This 20756 is on `http://127.0.0.1:53682/` and this it may require you to unblock 20757 it temporarily if you are running a host firewall. 20758 20759 Once configured you can then use `rclone` like this, 20760 20761 List directories in top level of your pCloud 20762 20763 rclone lsd remote: 20764 20765 List all the files in your pCloud 20766 20767 rclone ls remote: 20768 20769 To copy a local directory to an pCloud directory called backup 20770 20771 rclone copy /home/source remote:backup 20772 20773 ### Modified time and hashes ### 20774 20775 pCloud allows modification times to be set on objects accurate to 1 20776 second. These will be used to detect whether objects need syncing or 20777 not. In order to set a Modification time pCloud requires the object 20778 be re-uploaded. 20779 20780 pCloud supports MD5 and SHA1 type hashes, so you can use the 20781 `--checksum` flag. 20782 20783 #### Restricted filename characters 20784 20785 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 20786 the following characters are also replaced: 20787 20788 | Character | Value | Replacement | 20789 | --------- |:-----:|:-----------:| 20790 | \ | 0x5C | \ | 20791 20792 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 20793 as they can't be used in JSON strings. 20794 20795 ### Deleting files ### 20796 20797 Deleted files will be moved to the trash. Your subscription level 20798 will determine how long items stay in the trash. `rclone cleanup` can 20799 be used to empty the trash. 20800 20801 ### Root folder ID ### 20802 20803 You can set the `root_folder_id` for rclone. This is the directory 20804 (identified by its `Folder ID`) that rclone considers to be the root 20805 of your pCloud drive. 20806 20807 Normally you will leave this blank and rclone will determine the 20808 correct root to use itself. 20809 20810 However you can set this to restrict rclone to a specific folder 20811 hierarchy. 20812 20813 In order to do this you will have to find the `Folder ID` of the 20814 directory you wish rclone to display. This will be the `folder` field 20815 of the URL when you open the relevant folder in the pCloud web 20816 interface. 20817 20818 So if the folder you want rclone to use has a URL which looks like 20819 `https://my.pcloud.com/#page=filemanager&folder=5xxxxxxxx8&tpl=foldergrid` 20820 in the browser, then you use `5xxxxxxxx8` as 20821 the `root_folder_id` in the config. 20822 20823 20824 ### Standard Options 20825 20826 Here are the standard options specific to pcloud (Pcloud). 20827 20828 #### --pcloud-client-id 20829 20830 Pcloud App Client Id 20831 Leave blank normally. 20832 20833 - Config: client_id 20834 - Env Var: RCLONE_PCLOUD_CLIENT_ID 20835 - Type: string 20836 - Default: "" 20837 20838 #### --pcloud-client-secret 20839 20840 Pcloud App Client Secret 20841 Leave blank normally. 20842 20843 - Config: client_secret 20844 - Env Var: RCLONE_PCLOUD_CLIENT_SECRET 20845 - Type: string 20846 - Default: "" 20847 20848 ### Advanced Options 20849 20850 Here are the advanced options specific to pcloud (Pcloud). 20851 20852 #### --pcloud-encoding 20853 20854 This sets the encoding for the backend. 20855 20856 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 20857 20858 - Config: encoding 20859 - Env Var: RCLONE_PCLOUD_ENCODING 20860 - Type: MultiEncoder 20861 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 20862 20863 #### --pcloud-root-folder-id 20864 20865 Fill in for rclone to use a non root folder as its starting point. 20866 20867 - Config: root_folder_id 20868 - Env Var: RCLONE_PCLOUD_ROOT_FOLDER_ID 20869 - Type: string 20870 - Default: "d0" 20871 20872 20873 20874 premiumize.me 20875 ----------------------------------------- 20876 20877 Paths are specified as `remote:path` 20878 20879 Paths may be as deep as required, eg `remote:directory/subdirectory`. 20880 20881 The initial setup for [premiumize.me](https://premiumize.me/) involves getting a token from premiumize.me which you 20882 need to do in your browser. `rclone config` walks you through it. 20883 20884 Here is an example of how to make a remote called `remote`. First run: 20885 20886 rclone config 20887 20888 This will guide you through an interactive setup process: 20889 20890 ``` 20891 No remotes found - make a new one 20892 n) New remote 20893 s) Set configuration password 20894 q) Quit config 20895 n/s/q> n 20896 name> remote 20897 Type of storage to configure. 20898 Enter a string value. Press Enter for the default (""). 20899 Choose a number from below, or type in your own value 20900 [snip] 20901 XX / premiumize.me 20902 \ "premiumizeme" 20903 [snip] 20904 Storage> premiumizeme 20905 ** See help for premiumizeme backend at: https://rclone.org/premiumizeme/ ** 20906 20907 Remote config 20908 Use auto config? 20909 * Say Y if not sure 20910 * Say N if you are working on a remote or headless machine 20911 y) Yes 20912 n) No 20913 y/n> y 20914 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 20915 Log in and authorize rclone for access 20916 Waiting for code... 20917 Got code 20918 -------------------- 20919 [remote] 20920 type = premiumizeme 20921 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} 20922 -------------------- 20923 y) Yes this is OK 20924 e) Edit this remote 20925 d) Delete this remote 20926 y/e/d> 20927 ``` 20928 20929 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 20930 machine with no Internet browser available. 20931 20932 Note that rclone runs a webserver on your local machine to collect the 20933 token as returned from premiumize.me. This only runs from the moment it opens 20934 your browser to the moment you get back the verification code. This 20935 is on `http://127.0.0.1:53682/` and this it may require you to unblock 20936 it temporarily if you are running a host firewall. 20937 20938 Once configured you can then use `rclone` like this, 20939 20940 List directories in top level of your premiumize.me 20941 20942 rclone lsd remote: 20943 20944 List all the files in your premiumize.me 20945 20946 rclone ls remote: 20947 20948 To copy a local directory to an premiumize.me directory called backup 20949 20950 rclone copy /home/source remote:backup 20951 20952 ### Modified time and hashes ### 20953 20954 premiumize.me does not support modification times or hashes, therefore 20955 syncing will default to `--size-only` checking. Note that using 20956 `--update` will work. 20957 20958 #### Restricted filename characters 20959 20960 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 20961 the following characters are also replaced: 20962 20963 | Character | Value | Replacement | 20964 | --------- |:-----:|:-----------:| 20965 | \ | 0x5C | \ | 20966 | " | 0x22 | " | 20967 20968 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 20969 as they can't be used in JSON strings. 20970 20971 20972 ### Standard Options 20973 20974 Here are the standard options specific to premiumizeme (premiumize.me). 20975 20976 #### --premiumizeme-api-key 20977 20978 API Key. 20979 20980 This is not normally used - use oauth instead. 20981 20982 20983 - Config: api_key 20984 - Env Var: RCLONE_PREMIUMIZEME_API_KEY 20985 - Type: string 20986 - Default: "" 20987 20988 ### Advanced Options 20989 20990 Here are the advanced options specific to premiumizeme (premiumize.me). 20991 20992 #### --premiumizeme-encoding 20993 20994 This sets the encoding for the backend. 20995 20996 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 20997 20998 - Config: encoding 20999 - Env Var: RCLONE_PREMIUMIZEME_ENCODING 21000 - Type: MultiEncoder 21001 - Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot 21002 21003 21004 21005 ### Limitations ### 21006 21007 Note that premiumize.me is case insensitive so you can't have a file called 21008 "Hello.doc" and one called "hello.doc". 21009 21010 premiumize.me file names can't have the `\` or `"` characters in. 21011 rclone maps these to and from an identical looking unicode equivalents 21012 `\` and `"` 21013 21014 premiumize.me only supports filenames up to 255 characters in length. 21015 21016 put.io 21017 --------------------------------- 21018 21019 Paths are specified as `remote:path` 21020 21021 put.io paths may be as deep as required, eg 21022 `remote:directory/subdirectory`. 21023 21024 The initial setup for put.io involves getting a token from put.io 21025 which you need to do in your browser. `rclone config` walks you 21026 through it. 21027 21028 Here is an example of how to make a remote called `remote`. First run: 21029 21030 rclone config 21031 21032 This will guide you through an interactive setup process: 21033 21034 ``` 21035 No remotes found - make a new one 21036 n) New remote 21037 s) Set configuration password 21038 q) Quit config 21039 n/s/q> n 21040 name> putio 21041 Type of storage to configure. 21042 Enter a string value. Press Enter for the default (""). 21043 Choose a number from below, or type in your own value 21044 [snip] 21045 XX / Put.io 21046 \ "putio" 21047 [snip] 21048 Storage> putio 21049 ** See help for putio backend at: https://rclone.org/putio/ ** 21050 21051 Remote config 21052 Use auto config? 21053 * Say Y if not sure 21054 * Say N if you are working on a remote or headless machine 21055 y) Yes 21056 n) No 21057 y/n> y 21058 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 21059 Log in and authorize rclone for access 21060 Waiting for code... 21061 Got code 21062 -------------------- 21063 [putio] 21064 type = putio 21065 token = {"access_token":"XXXXXXXX","expiry":"0001-01-01T00:00:00Z"} 21066 -------------------- 21067 y) Yes this is OK 21068 e) Edit this remote 21069 d) Delete this remote 21070 y/e/d> y 21071 Current remotes: 21072 21073 Name Type 21074 ==== ==== 21075 putio putio 21076 21077 e) Edit existing remote 21078 n) New remote 21079 d) Delete remote 21080 r) Rename remote 21081 c) Copy remote 21082 s) Set configuration password 21083 q) Quit config 21084 e/n/d/r/c/s/q> q 21085 ``` 21086 21087 Note that rclone runs a webserver on your local machine to collect the 21088 token as returned from Google if you use auto config mode. This only 21089 runs from the moment it opens your browser to the moment you get back 21090 the verification code. This is on `http://127.0.0.1:53682/` and this 21091 it may require you to unblock it temporarily if you are running a host 21092 firewall, or use manual mode. 21093 21094 You can then use it like this, 21095 21096 List directories in top level of your put.io 21097 21098 rclone lsd remote: 21099 21100 List all the files in your put.io 21101 21102 rclone ls remote: 21103 21104 To copy a local directory to a put.io directory called backup 21105 21106 rclone copy /home/source remote:backup 21107 21108 #### Restricted filename characters 21109 21110 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 21111 the following characters are also replaced: 21112 21113 | Character | Value | Replacement | 21114 | --------- |:-----:|:-----------:| 21115 | \ | 0x5C | \ | 21116 21117 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 21118 as they can't be used in JSON strings. 21119 21120 21121 ### Advanced Options 21122 21123 Here are the advanced options specific to putio (Put.io). 21124 21125 #### --putio-encoding 21126 21127 This sets the encoding for the backend. 21128 21129 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 21130 21131 - Config: encoding 21132 - Env Var: RCLONE_PUTIO_ENCODING 21133 - Type: MultiEncoder 21134 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 21135 21136 21137 21138 Seafile 21139 ---------------------------------------- 21140 21141 This is a backend for the [Seafile](https://www.seafile.com/) storage service: 21142 - It works with both the free community edition or the professional edition. 21143 - Seafile versions 6.x and 7.x are all supported. 21144 - Encrypted libraries are also supported. 21145 - It supports 2FA enabled users 21146 21147 ### Root mode vs Library mode ### 21148 21149 There are two distinct modes you can setup your remote: 21150 - you point your remote to the **root of the server**, meaning you don't specify a library during the configuration: 21151 Paths are specified as `remote:library`. You may put subdirectories in too, eg `remote:library/path/to/dir`. 21152 - you point your remote to a specific library during the configuration: 21153 Paths are specified as `remote:path/to/dir`. **This is the recommended mode when using encrypted libraries**. (_This mode is possibly slightly faster than the root mode_) 21154 21155 ### Configuration in root mode ### 21156 21157 Here is an example of making a seafile configuration for a user with **no** two-factor authentication. First run 21158 21159 rclone config 21160 21161 This will guide you through an interactive setup process. To authenticate 21162 you will need the URL of your server, your email (or username) and your password. 21163 21164 ``` 21165 No remotes found - make a new one 21166 n) New remote 21167 s) Set configuration password 21168 q) Quit config 21169 n/s/q> n 21170 name> seafile 21171 Type of storage to configure. 21172 Enter a string value. Press Enter for the default (""). 21173 Choose a number from below, or type in your own value 21174 [snip] 21175 XX / Seafile 21176 \ "seafile" 21177 [snip] 21178 Storage> seafile 21179 ** See help for seafile backend at: https://rclone.org/seafile/ ** 21180 21181 URL of seafile host to connect to 21182 Enter a string value. Press Enter for the default (""). 21183 Choose a number from below, or type in your own value 21184 1 / Connect to cloud.seafile.com 21185 \ "https://cloud.seafile.com/" 21186 url> http://my.seafile.server/ 21187 User name (usually email address) 21188 Enter a string value. Press Enter for the default (""). 21189 user> me@example.com 21190 Password 21191 y) Yes type in my own password 21192 g) Generate random password 21193 n) No leave this optional password blank (default) 21194 y/g> y 21195 Enter the password: 21196 password: 21197 Confirm the password: 21198 password: 21199 Two-factor authentication ('true' if the account has 2FA enabled) 21200 Enter a boolean value (true or false). Press Enter for the default ("false"). 21201 2fa> false 21202 Name of the library. Leave blank to access all non-encrypted libraries. 21203 Enter a string value. Press Enter for the default (""). 21204 library> 21205 Library password (for encrypted libraries only). Leave blank if you pass it through the command line. 21206 y) Yes type in my own password 21207 g) Generate random password 21208 n) No leave this optional password blank (default) 21209 y/g/n> n 21210 Edit advanced config? (y/n) 21211 y) Yes 21212 n) No (default) 21213 y/n> n 21214 Remote config 21215 Two-factor authentication is not enabled on this account. 21216 -------------------- 21217 [seafile] 21218 type = seafile 21219 url = http://my.seafile.server/ 21220 user = me@example.com 21221 pass = *** ENCRYPTED *** 21222 2fa = false 21223 -------------------- 21224 y) Yes this is OK (default) 21225 e) Edit this remote 21226 d) Delete this remote 21227 y/e/d> y 21228 ``` 21229 21230 This remote is called `seafile`. It's pointing to the root of your seafile server and can now be used like this: 21231 21232 See all libraries 21233 21234 rclone lsd seafile: 21235 21236 Create a new library 21237 21238 rclone mkdir seafile:library 21239 21240 List the contents of a library 21241 21242 rclone ls seafile:library 21243 21244 Sync `/home/local/directory` to the remote library, deleting any 21245 excess files in the library. 21246 21247 rclone sync /home/local/directory seafile:library 21248 21249 ### Configuration in library mode ### 21250 21251 Here's an example of a configuration in library mode with a user that has the two-factor authentication enabled. Your 2FA code will be asked at the end of the configuration, and will attempt to authenticate you: 21252 21253 ``` 21254 No remotes found - make a new one 21255 n) New remote 21256 s) Set configuration password 21257 q) Quit config 21258 n/s/q> n 21259 name> seafile 21260 Type of storage to configure. 21261 Enter a string value. Press Enter for the default (""). 21262 Choose a number from below, or type in your own value 21263 [snip] 21264 XX / Seafile 21265 \ "seafile" 21266 [snip] 21267 Storage> seafile 21268 ** See help for seafile backend at: https://rclone.org/seafile/ ** 21269 21270 URL of seafile host to connect to 21271 Enter a string value. Press Enter for the default (""). 21272 Choose a number from below, or type in your own value 21273 1 / Connect to cloud.seafile.com 21274 \ "https://cloud.seafile.com/" 21275 url> http://my.seafile.server/ 21276 User name (usually email address) 21277 Enter a string value. Press Enter for the default (""). 21278 user> me@example.com 21279 Password 21280 y) Yes type in my own password 21281 g) Generate random password 21282 n) No leave this optional password blank (default) 21283 y/g> y 21284 Enter the password: 21285 password: 21286 Confirm the password: 21287 password: 21288 Two-factor authentication ('true' if the account has 2FA enabled) 21289 Enter a boolean value (true or false). Press Enter for the default ("false"). 21290 2fa> true 21291 Name of the library. Leave blank to access all non-encrypted libraries. 21292 Enter a string value. Press Enter for the default (""). 21293 library> My Library 21294 Library password (for encrypted libraries only). Leave blank if you pass it through the command line. 21295 y) Yes type in my own password 21296 g) Generate random password 21297 n) No leave this optional password blank (default) 21298 y/g/n> n 21299 Edit advanced config? (y/n) 21300 y) Yes 21301 n) No (default) 21302 y/n> n 21303 Remote config 21304 Two-factor authentication: please enter your 2FA code 21305 2fa code> 123456 21306 Authenticating... 21307 Success! 21308 -------------------- 21309 [seafile] 21310 type = seafile 21311 url = http://my.seafile.server/ 21312 user = me@example.com 21313 pass = 21314 2fa = true 21315 library = My Library 21316 -------------------- 21317 y) Yes this is OK (default) 21318 e) Edit this remote 21319 d) Delete this remote 21320 y/e/d> y 21321 ``` 21322 21323 You'll notice your password is blank in the configuration. It's because we only need the password to authenticate you once. 21324 21325 You specified `My Library` during the configuration. The root of the remote is pointing at the 21326 root of the library `My Library`: 21327 21328 See all files in the library: 21329 21330 rclone lsd seafile: 21331 21332 Create a new directory inside the library 21333 21334 rclone mkdir seafile:directory 21335 21336 List the contents of a directory 21337 21338 rclone ls seafile:directory 21339 21340 Sync `/home/local/directory` to the remote library, deleting any 21341 excess files in the library. 21342 21343 rclone sync /home/local/directory seafile: 21344 21345 21346 ### --fast-list ### 21347 21348 Seafile version 7+ supports `--fast-list` which allows you to use fewer 21349 transactions in exchange for more memory. See the [rclone 21350 docs](https://rclone.org/docs/#fast-list) for more details. 21351 Please note this is not supported on seafile server version 6.x 21352 21353 21354 #### Restricted filename characters 21355 21356 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 21357 the following characters are also replaced: 21358 21359 | Character | Value | Replacement | 21360 | --------- |:-----:|:-----------:| 21361 | / | 0x2F | / | 21362 | " | 0x22 | " | 21363 | \ | 0x5C | \ | 21364 21365 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 21366 as they can't be used in JSON strings. 21367 21368 ### Seafile and rclone link ### 21369 21370 Rclone supports generating share links for non-encrypted libraries only. 21371 They can either be for a file or a directory: 21372 21373 ``` 21374 rclone link seafile:seafile-tutorial.doc 21375 http://my.seafile.server/f/fdcd8a2f93f84b8b90f4/ 21376 21377 ``` 21378 21379 or if run on a directory you will get: 21380 21381 ``` 21382 rclone link seafile:dir 21383 http://my.seafile.server/d/9ea2455f6f55478bbb0d/ 21384 ``` 21385 21386 Please note a share link is unique for each file or directory. If you run a link command on a file/dir 21387 that has already been shared, you will get the exact same link. 21388 21389 ### Compatibility ### 21390 21391 It has been actively tested using the [seafile docker image](https://github.com/haiwen/seafile-docker) of these versions: 21392 - 6.3.4 community edition 21393 - 7.0.5 community edition 21394 - 7.1.3 community edition 21395 21396 Versions below 6.0 are not supported. 21397 Versions between 6.0 and 6.3 haven't been tested and might not work properly. 21398 21399 21400 ### Standard Options 21401 21402 Here are the standard options specific to seafile (seafile). 21403 21404 #### --seafile-url 21405 21406 URL of seafile host to connect to 21407 21408 - Config: url 21409 - Env Var: RCLONE_SEAFILE_URL 21410 - Type: string 21411 - Default: "" 21412 - Examples: 21413 - "https://cloud.seafile.com/" 21414 - Connect to cloud.seafile.com 21415 21416 #### --seafile-user 21417 21418 User name (usually email address) 21419 21420 - Config: user 21421 - Env Var: RCLONE_SEAFILE_USER 21422 - Type: string 21423 - Default: "" 21424 21425 #### --seafile-pass 21426 21427 Password 21428 21429 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 21430 21431 - Config: pass 21432 - Env Var: RCLONE_SEAFILE_PASS 21433 - Type: string 21434 - Default: "" 21435 21436 #### --seafile-2fa 21437 21438 Two-factor authentication ('true' if the account has 2FA enabled) 21439 21440 - Config: 2fa 21441 - Env Var: RCLONE_SEAFILE_2FA 21442 - Type: bool 21443 - Default: false 21444 21445 #### --seafile-library 21446 21447 Name of the library. Leave blank to access all non-encrypted libraries. 21448 21449 - Config: library 21450 - Env Var: RCLONE_SEAFILE_LIBRARY 21451 - Type: string 21452 - Default: "" 21453 21454 #### --seafile-library-key 21455 21456 Library password (for encrypted libraries only). Leave blank if you pass it through the command line. 21457 21458 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 21459 21460 - Config: library_key 21461 - Env Var: RCLONE_SEAFILE_LIBRARY_KEY 21462 - Type: string 21463 - Default: "" 21464 21465 #### --seafile-auth-token 21466 21467 Authentication token 21468 21469 - Config: auth_token 21470 - Env Var: RCLONE_SEAFILE_AUTH_TOKEN 21471 - Type: string 21472 - Default: "" 21473 21474 ### Advanced Options 21475 21476 Here are the advanced options specific to seafile (seafile). 21477 21478 #### --seafile-create-library 21479 21480 Should rclone create a library if it doesn't exist 21481 21482 - Config: create_library 21483 - Env Var: RCLONE_SEAFILE_CREATE_LIBRARY 21484 - Type: bool 21485 - Default: false 21486 21487 #### --seafile-encoding 21488 21489 This sets the encoding for the backend. 21490 21491 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 21492 21493 - Config: encoding 21494 - Env Var: RCLONE_SEAFILE_ENCODING 21495 - Type: MultiEncoder 21496 - Default: Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8 21497 21498 21499 21500 SFTP 21501 ---------------------------------------- 21502 21503 SFTP is the [Secure (or SSH) File Transfer 21504 Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol). 21505 21506 The SFTP backend can be used with a number of different providers: 21507 21508 21509 - C14 21510 - rsync.net 21511 21512 21513 SFTP runs over SSH v2 and is installed as standard with most modern 21514 SSH installations. 21515 21516 Paths are specified as `remote:path`. If the path does not begin with 21517 a `/` it is relative to the home directory of the user. An empty path 21518 `remote:` refers to the user's home directory. 21519 21520 "Note that some SFTP servers will need the leading / - Synology is a 21521 good example of this. rsync.net, on the other hand, requires users to 21522 OMIT the leading /. 21523 21524 Here is an example of making an SFTP configuration. First run 21525 21526 rclone config 21527 21528 This will guide you through an interactive setup process. 21529 21530 ``` 21531 No remotes found - make a new one 21532 n) New remote 21533 s) Set configuration password 21534 q) Quit config 21535 n/s/q> n 21536 name> remote 21537 Type of storage to configure. 21538 Choose a number from below, or type in your own value 21539 [snip] 21540 XX / SSH/SFTP Connection 21541 \ "sftp" 21542 [snip] 21543 Storage> sftp 21544 SSH host to connect to 21545 Choose a number from below, or type in your own value 21546 1 / Connect to example.com 21547 \ "example.com" 21548 host> example.com 21549 SSH username, leave blank for current username, ncw 21550 user> sftpuser 21551 SSH port, leave blank to use default (22) 21552 port> 21553 SSH password, leave blank to use ssh-agent. 21554 y) Yes type in my own password 21555 g) Generate random password 21556 n) No leave this optional password blank 21557 y/g/n> n 21558 Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. 21559 key_file> 21560 Remote config 21561 -------------------- 21562 [remote] 21563 host = example.com 21564 user = sftpuser 21565 port = 21566 pass = 21567 key_file = 21568 -------------------- 21569 y) Yes this is OK 21570 e) Edit this remote 21571 d) Delete this remote 21572 y/e/d> y 21573 ``` 21574 21575 This remote is called `remote` and can now be used like this: 21576 21577 See all directories in the home directory 21578 21579 rclone lsd remote: 21580 21581 Make a new directory 21582 21583 rclone mkdir remote:path/to/directory 21584 21585 List the contents of a directory 21586 21587 rclone ls remote:path/to/directory 21588 21589 Sync `/home/local/directory` to the remote directory, deleting any 21590 excess files in the directory. 21591 21592 rclone sync /home/local/directory remote:directory 21593 21594 ### SSH Authentication ### 21595 21596 The SFTP remote supports three authentication methods: 21597 21598 * Password 21599 * Key file 21600 * ssh-agent 21601 21602 Key files should be PEM-encoded private key files. For instance `/home/$USER/.ssh/id_rsa`. 21603 Only unencrypted OpenSSH or PEM encrypted files are supported. 21604 21605 The key file can be specified in either an external file (key_file) or contained within the 21606 rclone config file (key_pem). If using key_pem in the config file, the entry should be on a 21607 single line with new line ('\n' or '\r\n') separating lines. i.e. 21608 21609 key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY----- 21610 21611 This will generate it correctly for key_pem for use in the config: 21612 21613 awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa 21614 21615 If you don't specify `pass`, `key_file`, or `key_pem` then rclone will attempt to contact an ssh-agent. 21616 21617 You can also specify `key_use_agent` to force the usage of an ssh-agent. In this case 21618 `key_file` or `key_pem` can also be specified to force the usage of a specific key in the ssh-agent. 21619 21620 Using an ssh-agent is the only way to load encrypted OpenSSH keys at the moment. 21621 21622 If you set the `--sftp-ask-password` option, rclone will prompt for a 21623 password when needed and no password has been configured. 21624 21625 ### ssh-agent on macOS ### 21626 21627 Note that there seem to be various problems with using an ssh-agent on 21628 macOS due to recent changes in the OS. The most effective work-around 21629 seems to be to start an ssh-agent in each session, eg 21630 21631 eval `ssh-agent -s` && ssh-add -A 21632 21633 And then at the end of the session 21634 21635 eval `ssh-agent -k` 21636 21637 These commands can be used in scripts of course. 21638 21639 ### Modified time ### 21640 21641 Modified times are stored on the server to 1 second precision. 21642 21643 Modified times are used in syncing and are fully supported. 21644 21645 Some SFTP servers disable setting/modifying the file modification time after 21646 upload (for example, certain configurations of ProFTPd with mod_sftp). If you 21647 are using one of these servers, you can set the option `set_modtime = false` in 21648 your RClone backend configuration to disable this behaviour. 21649 21650 21651 ### Standard Options 21652 21653 Here are the standard options specific to sftp (SSH/SFTP Connection). 21654 21655 #### --sftp-host 21656 21657 SSH host to connect to 21658 21659 - Config: host 21660 - Env Var: RCLONE_SFTP_HOST 21661 - Type: string 21662 - Default: "" 21663 - Examples: 21664 - "example.com" 21665 - Connect to example.com 21666 21667 #### --sftp-user 21668 21669 SSH username, leave blank for current username, ncw 21670 21671 - Config: user 21672 - Env Var: RCLONE_SFTP_USER 21673 - Type: string 21674 - Default: "" 21675 21676 #### --sftp-port 21677 21678 SSH port, leave blank to use default (22) 21679 21680 - Config: port 21681 - Env Var: RCLONE_SFTP_PORT 21682 - Type: string 21683 - Default: "" 21684 21685 #### --sftp-pass 21686 21687 SSH password, leave blank to use ssh-agent. 21688 21689 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 21690 21691 - Config: pass 21692 - Env Var: RCLONE_SFTP_PASS 21693 - Type: string 21694 - Default: "" 21695 21696 #### --sftp-key-pem 21697 21698 Raw PEM-encoded private key, If specified, will override key_file parameter. 21699 21700 - Config: key_pem 21701 - Env Var: RCLONE_SFTP_KEY_PEM 21702 - Type: string 21703 - Default: "" 21704 21705 #### --sftp-key-file 21706 21707 Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent. 21708 21709 - Config: key_file 21710 - Env Var: RCLONE_SFTP_KEY_FILE 21711 - Type: string 21712 - Default: "" 21713 21714 #### --sftp-key-file-pass 21715 21716 The passphrase to decrypt the PEM-encoded private key file. 21717 21718 Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys 21719 in the new OpenSSH format can't be used. 21720 21721 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 21722 21723 - Config: key_file_pass 21724 - Env Var: RCLONE_SFTP_KEY_FILE_PASS 21725 - Type: string 21726 - Default: "" 21727 21728 #### --sftp-key-use-agent 21729 21730 When set forces the usage of the ssh-agent. 21731 21732 When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key is 21733 requested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errors 21734 when the ssh-agent contains many keys. 21735 21736 - Config: key_use_agent 21737 - Env Var: RCLONE_SFTP_KEY_USE_AGENT 21738 - Type: bool 21739 - Default: false 21740 21741 #### --sftp-use-insecure-cipher 21742 21743 Enable the use of insecure ciphers and key exchange methods. 21744 21745 This enables the use of the following insecure ciphers and key exchange methods: 21746 21747 - aes128-cbc 21748 - aes192-cbc 21749 - aes256-cbc 21750 - 3des-cbc 21751 - diffie-hellman-group-exchange-sha256 21752 - diffie-hellman-group-exchange-sha1 21753 21754 Those algorithms are insecure and may allow plaintext data to be recovered by an attacker. 21755 21756 - Config: use_insecure_cipher 21757 - Env Var: RCLONE_SFTP_USE_INSECURE_CIPHER 21758 - Type: bool 21759 - Default: false 21760 - Examples: 21761 - "false" 21762 - Use default Cipher list. 21763 - "true" 21764 - Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange. 21765 21766 #### --sftp-disable-hashcheck 21767 21768 Disable the execution of SSH commands to determine if remote file hashing is available. 21769 Leave blank or set to false to enable hashing (recommended), set to true to disable hashing. 21770 21771 - Config: disable_hashcheck 21772 - Env Var: RCLONE_SFTP_DISABLE_HASHCHECK 21773 - Type: bool 21774 - Default: false 21775 21776 ### Advanced Options 21777 21778 Here are the advanced options specific to sftp (SSH/SFTP Connection). 21779 21780 #### --sftp-ask-password 21781 21782 Allow asking for SFTP password when needed. 21783 21784 If this is set and no password is supplied then rclone will: 21785 - ask for a password 21786 - not contact the ssh agent 21787 21788 21789 - Config: ask_password 21790 - Env Var: RCLONE_SFTP_ASK_PASSWORD 21791 - Type: bool 21792 - Default: false 21793 21794 #### --sftp-path-override 21795 21796 Override path used by SSH connection. 21797 21798 This allows checksum calculation when SFTP and SSH paths are 21799 different. This issue affects among others Synology NAS boxes. 21800 21801 Shared folders can be found in directories representing volumes 21802 21803 rclone sync /home/local/directory remote:/directory --ssh-path-override /volume2/directory 21804 21805 Home directory can be found in a shared folder called "home" 21806 21807 rclone sync /home/local/directory remote:/home/directory --ssh-path-override /volume1/homes/USER/directory 21808 21809 - Config: path_override 21810 - Env Var: RCLONE_SFTP_PATH_OVERRIDE 21811 - Type: string 21812 - Default: "" 21813 21814 #### --sftp-set-modtime 21815 21816 Set the modified time on the remote if set. 21817 21818 - Config: set_modtime 21819 - Env Var: RCLONE_SFTP_SET_MODTIME 21820 - Type: bool 21821 - Default: true 21822 21823 #### --sftp-md5sum-command 21824 21825 The command used to read md5 hashes. Leave blank for autodetect. 21826 21827 - Config: md5sum_command 21828 - Env Var: RCLONE_SFTP_MD5SUM_COMMAND 21829 - Type: string 21830 - Default: "" 21831 21832 #### --sftp-sha1sum-command 21833 21834 The command used to read sha1 hashes. Leave blank for autodetect. 21835 21836 - Config: sha1sum_command 21837 - Env Var: RCLONE_SFTP_SHA1SUM_COMMAND 21838 - Type: string 21839 - Default: "" 21840 21841 #### --sftp-skip-links 21842 21843 Set to skip any symlinks and any other non regular files. 21844 21845 - Config: skip_links 21846 - Env Var: RCLONE_SFTP_SKIP_LINKS 21847 - Type: bool 21848 - Default: false 21849 21850 21851 21852 ### Limitations ### 21853 21854 SFTP supports checksums if the same login has shell access and `md5sum` 21855 or `sha1sum` as well as `echo` are in the remote's PATH. 21856 This remote checksumming (file hashing) is recommended and enabled by default. 21857 Disabling the checksumming may be required if you are connecting to SFTP servers 21858 which are not under your control, and to which the execution of remote commands 21859 is prohibited. Set the configuration option `disable_hashcheck` to `true` to 21860 disable checksumming. 21861 21862 SFTP also supports `about` if the same login has shell 21863 access and `df` are in the remote's PATH. `about` will 21864 return the total space, free space, and used space on the remote 21865 for the disk of the specified path on the remote or, if not set, 21866 the disk of the root on the remote. 21867 `about` will fail if it does not have shell 21868 access or if `df` is not in the remote's PATH. 21869 21870 Note that some SFTP servers (eg Synology) the paths are different for 21871 SSH and SFTP so the hashes can't be calculated properly. For them 21872 using `disable_hashcheck` is a good idea. 21873 21874 The only ssh agent supported under Windows is Putty's pageant. 21875 21876 The Go SSH library disables the use of the aes128-cbc cipher by 21877 default, due to security concerns. This can be re-enabled on a 21878 per-connection basis by setting the `use_insecure_cipher` setting in 21879 the configuration file to `true`. Further details on the insecurity of 21880 this cipher can be found [in this paper] 21881 (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf). 21882 21883 SFTP isn't supported under plan9 until [this 21884 issue](https://github.com/pkg/sftp/issues/156) is fixed. 21885 21886 Note that since SFTP isn't HTTP based the following flags don't work 21887 with it: `--dump-headers`, `--dump-bodies`, `--dump-auth` 21888 21889 Note that `--timeout` isn't supported (but `--contimeout` is). 21890 21891 21892 ## C14 {#c14} 21893 21894 C14 is supported through the SFTP backend. 21895 21896 See [C14's documentation](https://www.online.net/en/storage/c14-cold-storage) 21897 21898 ## rsync.net {#rsync-net} 21899 21900 rsync.net is supported through the SFTP backend. 21901 21902 See [rsync.net's documentation of rclone examples](https://www.rsync.net/products/rclone.html). 21903 21904 SugarSync 21905 ----------------------------------------- 21906 21907 [SugarSync](https://sugarsync.com) is a cloud service that enables 21908 active synchronization of files across computers and other devices for 21909 file backup, access, syncing, and sharing. 21910 21911 The initial setup for SugarSync involves getting a token from SugarSync which you 21912 can do with rclone. `rclone config` walks you through it. 21913 21914 Here is an example of how to make a remote called `remote`. First run: 21915 21916 rclone config 21917 21918 This will guide you through an interactive setup process: 21919 21920 ``` 21921 No remotes found - make a new one 21922 n) New remote 21923 s) Set configuration password 21924 q) Quit config 21925 n/s/q> n 21926 name> remote 21927 Type of storage to configure. 21928 Enter a string value. Press Enter for the default (""). 21929 Choose a number from below, or type in your own value 21930 [snip] 21931 XX / Sugarsync 21932 \ "sugarsync" 21933 [snip] 21934 Storage> sugarsync 21935 ** See help for sugarsync backend at: https://rclone.org/sugarsync/ ** 21936 21937 Sugarsync App ID. 21938 Leave blank to use rclone's. 21939 Enter a string value. Press Enter for the default (""). 21940 app_id> 21941 Sugarsync Access Key ID. 21942 Leave blank to use rclone's. 21943 Enter a string value. Press Enter for the default (""). 21944 access_key_id> 21945 Sugarsync Private Access Key 21946 Leave blank to use rclone's. 21947 Enter a string value. Press Enter for the default (""). 21948 private_access_key> 21949 Permanently delete files if true 21950 otherwise put them in the deleted files. 21951 Enter a boolean value (true or false). Press Enter for the default ("false"). 21952 hard_delete> 21953 Edit advanced config? (y/n) 21954 y) Yes 21955 n) No (default) 21956 y/n> n 21957 Remote config 21958 Username (email address)> nick@craig-wood.com 21959 Your Sugarsync password is only required during setup and will not be stored. 21960 password: 21961 -------------------- 21962 [remote] 21963 type = sugarsync 21964 refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX 21965 -------------------- 21966 y) Yes this is OK (default) 21967 e) Edit this remote 21968 d) Delete this remote 21969 y/e/d> y 21970 ``` 21971 21972 Note that the config asks for your email and password but doesn't 21973 store them, it only uses them to get the initial token. 21974 21975 Once configured you can then use `rclone` like this, 21976 21977 List directories (sync folders) in top level of your SugarSync 21978 21979 rclone lsd remote: 21980 21981 List all the files in your SugarSync folder "Test" 21982 21983 rclone ls remote:Test 21984 21985 To copy a local directory to an SugarSync folder called backup 21986 21987 rclone copy /home/source remote:backup 21988 21989 Paths are specified as `remote:path` 21990 21991 Paths may be as deep as required, eg `remote:directory/subdirectory`. 21992 21993 **NB** you can't create files in the top level folder you have to 21994 create a folder, which rclone will create as a "Sync Folder" with 21995 SugarSync. 21996 21997 ### Modified time and hashes ### 21998 21999 SugarSync does not support modification times or hashes, therefore 22000 syncing will default to `--size-only` checking. Note that using 22001 `--update` will work as rclone can read the time files were uploaded. 22002 22003 #### Restricted filename characters 22004 22005 SugarSync replaces the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 22006 except for DEL. 22007 22008 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 22009 as they can't be used in XML strings. 22010 22011 ### Deleting files ### 22012 22013 Deleted files will be moved to the "Deleted items" folder by default. 22014 22015 However you can supply the flag `--sugarsync-hard-delete` or set the 22016 config parameter `hard_delete = true` if you would like files to be 22017 deleted straight away. 22018 22019 22020 22021 ### Standard Options 22022 22023 Here are the standard options specific to sugarsync (Sugarsync). 22024 22025 #### --sugarsync-app-id 22026 22027 Sugarsync App ID. 22028 22029 Leave blank to use rclone's. 22030 22031 - Config: app_id 22032 - Env Var: RCLONE_SUGARSYNC_APP_ID 22033 - Type: string 22034 - Default: "" 22035 22036 #### --sugarsync-access-key-id 22037 22038 Sugarsync Access Key ID. 22039 22040 Leave blank to use rclone's. 22041 22042 - Config: access_key_id 22043 - Env Var: RCLONE_SUGARSYNC_ACCESS_KEY_ID 22044 - Type: string 22045 - Default: "" 22046 22047 #### --sugarsync-private-access-key 22048 22049 Sugarsync Private Access Key 22050 22051 Leave blank to use rclone's. 22052 22053 - Config: private_access_key 22054 - Env Var: RCLONE_SUGARSYNC_PRIVATE_ACCESS_KEY 22055 - Type: string 22056 - Default: "" 22057 22058 #### --sugarsync-hard-delete 22059 22060 Permanently delete files if true 22061 otherwise put them in the deleted files. 22062 22063 - Config: hard_delete 22064 - Env Var: RCLONE_SUGARSYNC_HARD_DELETE 22065 - Type: bool 22066 - Default: false 22067 22068 ### Advanced Options 22069 22070 Here are the advanced options specific to sugarsync (Sugarsync). 22071 22072 #### --sugarsync-refresh-token 22073 22074 Sugarsync refresh token 22075 22076 Leave blank normally, will be auto configured by rclone. 22077 22078 - Config: refresh_token 22079 - Env Var: RCLONE_SUGARSYNC_REFRESH_TOKEN 22080 - Type: string 22081 - Default: "" 22082 22083 #### --sugarsync-authorization 22084 22085 Sugarsync authorization 22086 22087 Leave blank normally, will be auto configured by rclone. 22088 22089 - Config: authorization 22090 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION 22091 - Type: string 22092 - Default: "" 22093 22094 #### --sugarsync-authorization-expiry 22095 22096 Sugarsync authorization expiry 22097 22098 Leave blank normally, will be auto configured by rclone. 22099 22100 - Config: authorization_expiry 22101 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION_EXPIRY 22102 - Type: string 22103 - Default: "" 22104 22105 #### --sugarsync-user 22106 22107 Sugarsync user 22108 22109 Leave blank normally, will be auto configured by rclone. 22110 22111 - Config: user 22112 - Env Var: RCLONE_SUGARSYNC_USER 22113 - Type: string 22114 - Default: "" 22115 22116 #### --sugarsync-root-id 22117 22118 Sugarsync root id 22119 22120 Leave blank normally, will be auto configured by rclone. 22121 22122 - Config: root_id 22123 - Env Var: RCLONE_SUGARSYNC_ROOT_ID 22124 - Type: string 22125 - Default: "" 22126 22127 #### --sugarsync-deleted-id 22128 22129 Sugarsync deleted folder id 22130 22131 Leave blank normally, will be auto configured by rclone. 22132 22133 - Config: deleted_id 22134 - Env Var: RCLONE_SUGARSYNC_DELETED_ID 22135 - Type: string 22136 - Default: "" 22137 22138 #### --sugarsync-encoding 22139 22140 This sets the encoding for the backend. 22141 22142 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 22143 22144 - Config: encoding 22145 - Env Var: RCLONE_SUGARSYNC_ENCODING 22146 - Type: MultiEncoder 22147 - Default: Slash,Ctl,InvalidUtf8,Dot 22148 22149 22150 22151 Tardigrade 22152 ----------------------------------------- 22153 22154 [Tardigrade](https://tardigrade.io) is an encrypted, secure, and 22155 cost-effective object storage service that enables you to store, back up, and 22156 archive large amounts of data in a decentralized manner. 22157 22158 ## Setup 22159 22160 To make a new Tardigrade configuration you need one of the following: 22161 * Access Grant that someone else shared with you. 22162 * [API Key](https://documentation.tardigrade.io/getting-started/uploading-your-first-object/create-an-api-key) 22163 of a Tardigrade project you are a member of. 22164 22165 Here is an example of how to make a remote called `remote`. First run: 22166 22167 rclone config 22168 22169 This will guide you through an interactive setup process: 22170 22171 ### Setup with access grant 22172 22173 ``` 22174 No remotes found - make a new one 22175 n) New remote 22176 s) Set configuration password 22177 q) Quit config 22178 n/s/q> n 22179 name> remote 22180 Type of storage to configure. 22181 Enter a string value. Press Enter for the default (""). 22182 Choose a number from below, or type in your own value 22183 [snip] 22184 XX / Tardigrade Decentralized Cloud Storage 22185 \ "tardigrade" 22186 [snip] 22187 Storage> tardigrade 22188 ** See help for tardigrade backend at: https://rclone.org/tardigrade/ ** 22189 22190 Choose an authentication method. 22191 Enter a string value. Press Enter for the default ("existing"). 22192 Choose a number from below, or type in your own value 22193 1 / Use an existing access grant. 22194 \ "existing" 22195 2 / Create a new access grant from satellite address, API key, and passphrase. 22196 \ "new" 22197 provider> existing 22198 Access Grant. 22199 Enter a string value. Press Enter for the default (""). 22200 access_grant> your-access-grant-received-by-someone-else 22201 Remote config 22202 -------------------- 22203 [remote] 22204 type = tardigrade 22205 access_grant = your-access-grant-received-by-someone-else 22206 -------------------- 22207 y) Yes this is OK (default) 22208 e) Edit this remote 22209 d) Delete this remote 22210 y/e/d> y 22211 ``` 22212 22213 ### Setup with API key and passhprase 22214 22215 ``` 22216 No remotes found - make a new one 22217 n) New remote 22218 s) Set configuration password 22219 q) Quit config 22220 n/s/q> n 22221 name> remote 22222 Type of storage to configure. 22223 Enter a string value. Press Enter for the default (""). 22224 Choose a number from below, or type in your own value 22225 [snip] 22226 XX / Tardigrade Decentralized Cloud Storage 22227 \ "tardigrade" 22228 [snip] 22229 Storage> tardigrade 22230 ** See help for tardigrade backend at: https://rclone.org/tardigrade/ ** 22231 22232 Choose an authentication method. 22233 Enter a string value. Press Enter for the default ("existing"). 22234 Choose a number from below, or type in your own value 22235 1 / Use an existing access grant. 22236 \ "existing" 22237 2 / Create a new access grant from satellite address, API key, and passphrase. 22238 \ "new" 22239 provider> new 22240 Satellite Address. Custom satellite address should match the format: `<nodeid>@<address>:<port>`. 22241 Enter a string value. Press Enter for the default ("us-central-1.tardigrade.io"). 22242 Choose a number from below, or type in your own value 22243 1 / US Central 1 22244 \ "us-central-1.tardigrade.io" 22245 2 / Europe West 1 22246 \ "europe-west-1.tardigrade.io" 22247 3 / Asia East 1 22248 \ "asia-east-1.tardigrade.io" 22249 satellite_address> 1 22250 API Key. 22251 Enter a string value. Press Enter for the default (""). 22252 api_key> your-api-key-for-your-tardigrade-project 22253 Encryption Passphrase. To access existing objects enter passphrase used for uploading. 22254 Enter a string value. Press Enter for the default (""). 22255 passphrase> your-human-readable-encryption-passphrase 22256 Remote config 22257 -------------------- 22258 [remote] 22259 type = tardigrade 22260 satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us-central-1.tardigrade.io:7777 22261 api_key = your-api-key-for-your-tardigrade-project 22262 passphrase = your-human-readable-encryption-passphrase 22263 access_grant = the-access-grant-generated-from-the-api-key-and-passphrase 22264 -------------------- 22265 y) Yes this is OK (default) 22266 e) Edit this remote 22267 d) Delete this remote 22268 y/e/d> y 22269 ``` 22270 22271 ## Usage 22272 22273 Paths are specified as `remote:bucket` (or `remote:` for the `lsf` 22274 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 22275 22276 Once configured you can then use `rclone` like this. 22277 22278 ### Create a new bucket 22279 22280 Use the `mkdir` command to create new bucket, e.g. `bucket`. 22281 22282 rclone mkdir remote:bucket 22283 22284 ### List all buckets 22285 22286 Use the `lsf` command to list all buckets. 22287 22288 rclone lsf remote: 22289 22290 Note the colon (`:`) character at the end of the command line. 22291 22292 ### Delete a bucket 22293 22294 Use the `rmdir` command to delete an empty bucket. 22295 22296 rclone rmdir remote:bucket 22297 22298 Use the `purge` command to delete a non-empty bucket with all its content. 22299 22300 rclone purge remote:bucket 22301 22302 ### Upload objects 22303 22304 Use the `copy` command to upload an object. 22305 22306 rclone copy --progress /home/local/directory/file.ext remote:bucket/path/to/dir/ 22307 22308 The `--progress` flag is for displaying progress information. 22309 Remove it if you don't need this information. 22310 22311 Use a folder in the local path to upload all its objects. 22312 22313 rclone copy --progress /home/local/directory/ remote:bucket/path/to/dir/ 22314 22315 Only modified files will be copied. 22316 22317 ### List objects 22318 22319 Use the `ls` command to list recursively all objects in a bucket. 22320 22321 rclone ls remote:bucket 22322 22323 Add the folder to the remote path to list recursively all objects in this folder. 22324 22325 rclone ls remote:bucket/path/to/dir/ 22326 22327 Use the `lsf` command to list non-recursively all objects in a bucket or a folder. 22328 22329 rclone lsf remote:bucket/path/to/dir/ 22330 22331 ### Download objects 22332 22333 Use the `copy` command to download an object. 22334 22335 rclone copy --progress remote:bucket/path/to/dir/file.ext /home/local/directory/ 22336 22337 The `--progress` flag is for displaying progress information. 22338 Remove it if you don't need this information. 22339 22340 Use a folder in the remote path to download all its objects. 22341 22342 rclone copy --progress remote:bucket/path/to/dir/ /home/local/directory/ 22343 22344 ### Delete objects 22345 22346 Use the `deletefile` command to delete a single object. 22347 22348 rclone deletefile remote:bucket/path/to/dir/file.ext 22349 22350 Use the `delete` command to delete all object in a folder. 22351 22352 rclone delete remote:bucket/path/to/dir/ 22353 22354 ### Print the total size of objects 22355 22356 Use the `size` command to print the total size of objects in a bucket or a folder. 22357 22358 rclone size remote:bucket/path/to/dir/ 22359 22360 ### Sync two Locations 22361 22362 Use the `sync` command to sync the source to the destination, 22363 changing the destination only, deleting any excess files. 22364 22365 rclone sync --progress /home/local/directory/ remote:bucket/path/to/dir/ 22366 22367 The `--progress` flag is for displaying progress information. 22368 Remove it if you don't need this information. 22369 22370 Since this can cause data loss, test first with the `--dry-run` flag 22371 to see exactly what would be copied and deleted. 22372 22373 The sync can be done also from Tardigrade to the local file system. 22374 22375 rclone sync --progress remote:bucket/path/to/dir/ /home/local/directory/ 22376 22377 Or between two Tardigrade buckets. 22378 22379 rclone sync --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/ 22380 22381 Or even between another cloud storage and Tardigrade. 22382 22383 rclone sync --progress s3:bucket/path/to/dir/ tardigrade:bucket/path/to/dir/ 22384 22385 22386 ### Standard Options 22387 22388 Here are the standard options specific to tardigrade (Tardigrade Decentralized Cloud Storage). 22389 22390 #### --tardigrade-provider 22391 22392 Choose an authentication method. 22393 22394 - Config: provider 22395 - Env Var: RCLONE_TARDIGRADE_PROVIDER 22396 - Type: string 22397 - Default: "existing" 22398 - Examples: 22399 - "existing" 22400 - Use an existing access grant. 22401 - "new" 22402 - Create a new access grant from satellite address, API key, and passphrase. 22403 22404 #### --tardigrade-access-grant 22405 22406 Access Grant. 22407 22408 - Config: access_grant 22409 - Env Var: RCLONE_TARDIGRADE_ACCESS_GRANT 22410 - Type: string 22411 - Default: "" 22412 22413 #### --tardigrade-satellite-address 22414 22415 Satellite Address. Custom satellite address should match the format: `<nodeid>@<address>:<port>`. 22416 22417 - Config: satellite_address 22418 - Env Var: RCLONE_TARDIGRADE_SATELLITE_ADDRESS 22419 - Type: string 22420 - Default: "us-central-1.tardigrade.io" 22421 - Examples: 22422 - "us-central-1.tardigrade.io" 22423 - US Central 1 22424 - "europe-west-1.tardigrade.io" 22425 - Europe West 1 22426 - "asia-east-1.tardigrade.io" 22427 - Asia East 1 22428 22429 #### --tardigrade-api-key 22430 22431 API Key. 22432 22433 - Config: api_key 22434 - Env Var: RCLONE_TARDIGRADE_API_KEY 22435 - Type: string 22436 - Default: "" 22437 22438 #### --tardigrade-passphrase 22439 22440 Encryption Passphrase. To access existing objects enter passphrase used for uploading. 22441 22442 - Config: passphrase 22443 - Env Var: RCLONE_TARDIGRADE_PASSPHRASE 22444 - Type: string 22445 - Default: "" 22446 22447 22448 22449 Union 22450 ----------------------------------------- 22451 22452 The `union` remote provides a unification similar to UnionFS using other remotes. 22453 22454 Paths may be as deep as required or a local path, 22455 eg `remote:directory/subdirectory` or `/directory/subdirectory`. 22456 22457 During the initial setup with `rclone config` you will specify the upstream 22458 remotes as a space separated list. The upstream remotes can either be a local paths or other remotes. 22459 22460 Attribute `:ro` and `:nc` can be attach to the end of path to tag the remote as **read only** or **no create**, 22461 eg `remote:directory/subdirectory:ro` or `remote:directory/subdirectory:nc`. 22462 22463 Subfolders can be used in upstream remotes. Assume a union remote named `backup` 22464 with the remotes `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop` 22465 is exactly the same as invoking `rclone mkdir mydrive2:/backup/desktop`. 22466 22467 There will be no special handling of paths containing `..` segments. 22468 Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking 22469 `rclone mkdir mydrive2:/backup/../desktop`. 22470 22471 ### Behavior / Policies 22472 22473 The behavior of union backend is inspired by [trapexit/mergerfs](https://github.com/trapexit/mergerfs). All functions are grouped into 3 categories: **action**, **create** and **search**. These functions and categories can be assigned a policy which dictates what file or directory is chosen when performing that behavior. Any policy can be assigned to a function or category though some may not be very useful in practice. For instance: **rand** (random) may be useful for file creation (create) but could lead to very odd behavior if used for `delete` if there were more than one copy of the file. 22474 22475 #### Function / Category classifications 22476 22477 | Category | Description | Functions | 22478 |----------|--------------------------|-------------------------------------------------------------------------------------| 22479 | action | Writing Existing file | move, rmdir, rmdirs, delete, purge and copy, sync (as destination when file exist) | 22480 | create | Create non-existing file | copy, sync (as destination when file not exist) | 22481 | search | Reading and listing file | ls, lsd, lsl, cat, md5sum, sha1sum and copy, sync (as source) | 22482 | N/A | | size, about | 22483 22484 #### Path Preservation 22485 22486 Policies, as described below, are of two basic types. `path preserving` and `non-path preserving`. 22487 22488 All policies which start with `ep` (**epff**, **eplfs**, **eplus**, **epmfs**, **eprand**) are `path preserving`. `ep` stands for `existing path`. 22489 22490 A path preserving policy will only consider upstreams where the relative path being accessed already exists. 22491 22492 When using non-path preserving policies paths will be created in target upstreams as necessary. 22493 22494 #### Quota Relevant Policies 22495 22496 Some policies rely on quota information. These policies should be used only if your upstreams support the respective quota fields. 22497 22498 | Policy | Required Field | 22499 |------------|----------------| 22500 | lfs, eplfs | Free | 22501 | mfs, epmfs | Free | 22502 | lus, eplus | Used | 22503 | lno, eplno | Objects | 22504 22505 To check if your upstream supports the field, run `rclone about remote: [flags]` and see if the required field exists. 22506 22507 #### Filters 22508 22509 Policies basically search upstream remotes and create a list of files / paths for functions to work on. The policy is responsible for filtering and sorting. The policy type defines the sorting but filtering is mostly uniform as described below. 22510 22511 * No **search** policies filter. 22512 * All **action** policies will filter out remotes which are tagged as **read-only**. 22513 * All **create** policies will filter out remotes which are tagged **read-only** or **no-create**. 22514 22515 If all remotes are filtered an error will be returned. 22516 22517 #### Policy descriptions 22518 22519 The policies definition are inspired by [trapexit/mergerfs](https://github.com/trapexit/mergerfs) but not exactly the same. Some policy definition could be different due to the much larger latency of remote file systems. 22520 22521 | Policy | Description | 22522 |------------------|------------------------------------------------------------| 22523 | all | Search category: same as **epall**. Action category: same as **epall**. Create category: act on all upstreams. | 22524 | epall (existing path, all) | Search category: Given this order configured, act on the first one found where the relative path exists. Action category: apply to all found. Create category: act on all upstreams where the relative path exists. | 22525 | epff (existing path, first found) | Act on the first one found, by the time upstreams reply, where the relative path exists. | 22526 | eplfs (existing path, least free space) | Of all the upstreams on which the relative path exists choose the one with the least free space. | 22527 | eplus (existing path, least used space) | Of all the upstreams on which the relative path exists choose the one with the least used space. | 22528 | eplno (existing path, least number of objects) | Of all the upstreams on which the relative path exists choose the one with the least number of objects. | 22529 | epmfs (existing path, most free space) | Of all the upstreams on which the relative path exists choose the one with the most free space. | 22530 | eprand (existing path, random) | Calls **epall** and then randomizes. Returns only one upstream. | 22531 | ff (first found) | Search category: same as **epff**. Action category: same as **epff**. Create category: Act on the first one found by the time upstreams reply. | 22532 | lfs (least free space) | Search category: same as **eplfs**. Action category: same as **eplfs**. Create category: Pick the upstream with the least available free space. | 22533 | lus (least used space) | Search category: same as **eplus**. Action category: same as **eplus**. Create category: Pick the upstream with the least used space. | 22534 | lno (least number of objects) | Search category: same as **eplno**. Action category: same as **eplno**. Create category: Pick the upstream with the least number of objects. | 22535 | mfs (most free space) | Search category: same as **epmfs**. Action category: same as **epmfs**. Create category: Pick the upstream with the most available free space. | 22536 | newest | Pick the file / directory with the largest mtime. | 22537 | rand (random) | Calls **all** and then randomizes. Returns only one upstream. | 22538 22539 ### Setup 22540 22541 Here is an example of how to make a union called `remote` for local folders. 22542 First run: 22543 22544 rclone config 22545 22546 This will guide you through an interactive setup process: 22547 22548 ``` 22549 No remotes found - make a new one 22550 n) New remote 22551 s) Set configuration password 22552 q) Quit config 22553 n/s/q> n 22554 name> remote 22555 Type of storage to configure. 22556 Choose a number from below, or type in your own value 22557 [snip] 22558 XX / Union merges the contents of several remotes 22559 \ "union" 22560 [snip] 22561 Storage> union 22562 List of space separated upstreams. 22563 Can be 'upstreama:test/dir upstreamb:', '\"upstreama:test/space:ro dir\" upstreamb:', etc. 22564 Enter a string value. Press Enter for the default (""). 22565 upstreams> 22566 Policy to choose upstream on ACTION class. 22567 Enter a string value. Press Enter for the default ("epall"). 22568 action_policy> 22569 Policy to choose upstream on CREATE class. 22570 Enter a string value. Press Enter for the default ("epmfs"). 22571 create_policy> 22572 Policy to choose upstream on SEARCH class. 22573 Enter a string value. Press Enter for the default ("ff"). 22574 search_policy> 22575 Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used. 22576 Enter a signed integer. Press Enter for the default ("120"). 22577 cache_time> 22578 Remote config 22579 -------------------- 22580 [remote] 22581 type = union 22582 upstreams = C:\dir1 C:\dir2 C:\dir3 22583 -------------------- 22584 y) Yes this is OK 22585 e) Edit this remote 22586 d) Delete this remote 22587 y/e/d> y 22588 Current remotes: 22589 22590 Name Type 22591 ==== ==== 22592 remote union 22593 22594 e) Edit existing remote 22595 n) New remote 22596 d) Delete remote 22597 r) Rename remote 22598 c) Copy remote 22599 s) Set configuration password 22600 q) Quit config 22601 e/n/d/r/c/s/q> q 22602 ``` 22603 22604 Once configured you can then use `rclone` like this, 22605 22606 List directories in top level in `C:\dir1`, `C:\dir2` and `C:\dir3` 22607 22608 rclone lsd remote: 22609 22610 List all the files in `C:\dir1`, `C:\dir2` and `C:\dir3` 22611 22612 rclone ls remote: 22613 22614 Copy another local directory to the union directory called source, which will be placed into `C:\dir3` 22615 22616 rclone copy C:\source remote:source 22617 22618 22619 ### Standard Options 22620 22621 Here are the standard options specific to union (Union merges the contents of several upstream fs). 22622 22623 #### --union-upstreams 22624 22625 List of space separated upstreams. 22626 Can be 'upstreama:test/dir upstreamb:', '"upstreama:test/space:ro dir" upstreamb:', etc. 22627 22628 22629 - Config: upstreams 22630 - Env Var: RCLONE_UNION_UPSTREAMS 22631 - Type: string 22632 - Default: "" 22633 22634 #### --union-action-policy 22635 22636 Policy to choose upstream on ACTION category. 22637 22638 - Config: action_policy 22639 - Env Var: RCLONE_UNION_ACTION_POLICY 22640 - Type: string 22641 - Default: "epall" 22642 22643 #### --union-create-policy 22644 22645 Policy to choose upstream on CREATE category. 22646 22647 - Config: create_policy 22648 - Env Var: RCLONE_UNION_CREATE_POLICY 22649 - Type: string 22650 - Default: "epmfs" 22651 22652 #### --union-search-policy 22653 22654 Policy to choose upstream on SEARCH category. 22655 22656 - Config: search_policy 22657 - Env Var: RCLONE_UNION_SEARCH_POLICY 22658 - Type: string 22659 - Default: "ff" 22660 22661 #### --union-cache-time 22662 22663 Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used. 22664 22665 - Config: cache_time 22666 - Env Var: RCLONE_UNION_CACHE_TIME 22667 - Type: int 22668 - Default: 120 22669 22670 22671 22672 WebDAV 22673 ----------------------------------------- 22674 22675 Paths are specified as `remote:path` 22676 22677 Paths may be as deep as required, eg `remote:directory/subdirectory`. 22678 22679 To configure the WebDAV remote you will need to have a URL for it, and 22680 a username and password. If you know what kind of system you are 22681 connecting to then rclone can enable extra features. 22682 22683 Here is an example of how to make a remote called `remote`. First run: 22684 22685 rclone config 22686 22687 This will guide you through an interactive setup process: 22688 22689 ``` 22690 No remotes found - make a new one 22691 n) New remote 22692 s) Set configuration password 22693 q) Quit config 22694 n/s/q> n 22695 name> remote 22696 Type of storage to configure. 22697 Choose a number from below, or type in your own value 22698 [snip] 22699 XX / Webdav 22700 \ "webdav" 22701 [snip] 22702 Storage> webdav 22703 URL of http host to connect to 22704 Choose a number from below, or type in your own value 22705 1 / Connect to example.com 22706 \ "https://example.com" 22707 url> https://example.com/remote.php/webdav/ 22708 Name of the Webdav site/service/software you are using 22709 Choose a number from below, or type in your own value 22710 1 / Nextcloud 22711 \ "nextcloud" 22712 2 / Owncloud 22713 \ "owncloud" 22714 3 / Sharepoint 22715 \ "sharepoint" 22716 4 / Other site/service or software 22717 \ "other" 22718 vendor> 1 22719 User name 22720 user> user 22721 Password. 22722 y) Yes type in my own password 22723 g) Generate random password 22724 n) No leave this optional password blank 22725 y/g/n> y 22726 Enter the password: 22727 password: 22728 Confirm the password: 22729 password: 22730 Bearer token instead of user/pass (eg a Macaroon) 22731 bearer_token> 22732 Remote config 22733 -------------------- 22734 [remote] 22735 type = webdav 22736 url = https://example.com/remote.php/webdav/ 22737 vendor = nextcloud 22738 user = user 22739 pass = *** ENCRYPTED *** 22740 bearer_token = 22741 -------------------- 22742 y) Yes this is OK 22743 e) Edit this remote 22744 d) Delete this remote 22745 y/e/d> y 22746 ``` 22747 22748 Once configured you can then use `rclone` like this, 22749 22750 List directories in top level of your WebDAV 22751 22752 rclone lsd remote: 22753 22754 List all the files in your WebDAV 22755 22756 rclone ls remote: 22757 22758 To copy a local directory to an WebDAV directory called backup 22759 22760 rclone copy /home/source remote:backup 22761 22762 ### Modified time and hashes ### 22763 22764 Plain WebDAV does not support modified times. However when used with 22765 Owncloud or Nextcloud rclone will support modified times. 22766 22767 Likewise plain WebDAV does not support hashes, however when used with 22768 Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes. 22769 Depending on the exact version of Owncloud or Nextcloud hashes may 22770 appear on all objects, or only on objects which had a hash uploaded 22771 with them. 22772 22773 22774 ### Standard Options 22775 22776 Here are the standard options specific to webdav (Webdav). 22777 22778 #### --webdav-url 22779 22780 URL of http host to connect to 22781 22782 - Config: url 22783 - Env Var: RCLONE_WEBDAV_URL 22784 - Type: string 22785 - Default: "" 22786 - Examples: 22787 - "https://example.com" 22788 - Connect to example.com 22789 22790 #### --webdav-vendor 22791 22792 Name of the Webdav site/service/software you are using 22793 22794 - Config: vendor 22795 - Env Var: RCLONE_WEBDAV_VENDOR 22796 - Type: string 22797 - Default: "" 22798 - Examples: 22799 - "nextcloud" 22800 - Nextcloud 22801 - "owncloud" 22802 - Owncloud 22803 - "sharepoint" 22804 - Sharepoint 22805 - "other" 22806 - Other site/service or software 22807 22808 #### --webdav-user 22809 22810 User name 22811 22812 - Config: user 22813 - Env Var: RCLONE_WEBDAV_USER 22814 - Type: string 22815 - Default: "" 22816 22817 #### --webdav-pass 22818 22819 Password. 22820 22821 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 22822 22823 - Config: pass 22824 - Env Var: RCLONE_WEBDAV_PASS 22825 - Type: string 22826 - Default: "" 22827 22828 #### --webdav-bearer-token 22829 22830 Bearer token instead of user/pass (eg a Macaroon) 22831 22832 - Config: bearer_token 22833 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN 22834 - Type: string 22835 - Default: "" 22836 22837 ### Advanced Options 22838 22839 Here are the advanced options specific to webdav (Webdav). 22840 22841 #### --webdav-bearer-token-command 22842 22843 Command to run to get a bearer token 22844 22845 - Config: bearer_token_command 22846 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN_COMMAND 22847 - Type: string 22848 - Default: "" 22849 22850 22851 22852 ## Provider notes ## 22853 22854 See below for notes on specific providers. 22855 22856 ### Owncloud ### 22857 22858 Click on the settings cog in the bottom right of the page and this 22859 will show the WebDAV URL that rclone needs in the config step. It 22860 will look something like `https://example.com/remote.php/webdav/`. 22861 22862 Owncloud supports modified times using the `X-OC-Mtime` header. 22863 22864 ### Nextcloud ### 22865 22866 This is configured in an identical way to Owncloud. Note that 22867 Nextcloud does not support streaming of files (`rcat`) whereas 22868 Owncloud does. This [may be 22869 fixed](https://github.com/nextcloud/nextcloud-snap/issues/365) in the 22870 future. 22871 22872 ### Sharepoint ### 22873 22874 Rclone can be used with Sharepoint provided by OneDrive for Business 22875 or Office365 Education Accounts. 22876 This feature is only needed for a few of these Accounts, 22877 mostly Office365 Education ones. These accounts are sometimes not 22878 verified by the domain owner [github#1975](https://github.com/rclone/rclone/issues/1975) 22879 22880 This means that these accounts can't be added using the official 22881 API (other Accounts should work with the "onedrive" option). However, 22882 it is possible to access them using webdav. 22883 22884 To use a sharepoint remote with rclone, add it like this: 22885 First, you need to get your remote's URL: 22886 22887 - Go [here](https://onedrive.live.com/about/en-us/signin/) 22888 to open your OneDrive or to sign in 22889 - Now take a look at your address bar, the URL should look like this: 22890 `https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/_layouts/15/onedrive.aspx` 22891 22892 You'll only need this URL up to the email address. After that, you'll 22893 most likely want to add "/Documents". That subdirectory contains 22894 the actual data stored on your OneDrive. 22895 22896 Add the remote to rclone like this: 22897 Configure the `url` as `https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents` 22898 and use your normal account email and password for `user` and `pass`. 22899 If you have 2FA enabled, you have to generate an app password. 22900 Set the `vendor` to `sharepoint`. 22901 22902 Your config file should look like this: 22903 22904 ``` 22905 [sharepoint] 22906 type = webdav 22907 url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents 22908 vendor = other 22909 user = YourEmailAddress 22910 pass = encryptedpassword 22911 ``` 22912 22913 #### Required Flags for SharePoint #### 22914 As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer. 22915 22916 For Rclone calls copying files (especially Office files such as .docx, .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents: 22917 22918 ``` 22919 --ignore-size --ignore-checksum --update 22920 ``` 22921 22922 ### dCache ### 22923 22924 dCache is a storage system that supports many protocols and 22925 authentication/authorisation schemes. For WebDAV clients, it allows 22926 users to authenticate with username and password (BASIC), X.509, 22927 Kerberos, and various bearer tokens, including 22928 [Macaroons](https://www.dcache.org/manuals/workshop-2017-05-29-Umea/000-Final/anupam_macaroons_v02.pdf) 22929 and [OpenID-Connect](https://en.wikipedia.org/wiki/OpenID_Connect) 22930 access tokens. 22931 22932 Configure as normal using the `other` type. Don't enter a username or 22933 password, instead enter your Macaroon as the `bearer_token`. 22934 22935 The config will end up looking something like this. 22936 22937 ``` 22938 [dcache] 22939 type = webdav 22940 url = https://dcache... 22941 vendor = other 22942 user = 22943 pass = 22944 bearer_token = your-macaroon 22945 ``` 22946 22947 There is a [script](https://github.com/sara-nl/GridScripts/blob/master/get-macaroon) that 22948 obtains a Macaroon from a dCache WebDAV endpoint, and creates an rclone config file. 22949 22950 Macaroons may also be obtained from the dCacheView 22951 web-browser/JavaScript client that comes with dCache. 22952 22953 ### OpenID-Connect ### 22954 22955 dCache also supports authenticating with OpenID-Connect access tokens. 22956 OpenID-Connect is a protocol (based on OAuth 2.0) that allows services 22957 to identify users who have authenticated with some central service. 22958 22959 Support for OpenID-Connect in rclone is currently achieved using 22960 another software package called 22961 [oidc-agent](https://github.com/indigo-dc/oidc-agent). This is a 22962 command-line tool that facilitates obtaining an access token. Once 22963 installed and configured, an access token is obtained by running the 22964 `oidc-token` command. The following example shows a (shortened) 22965 access token obtained from the *XDC* OIDC Provider. 22966 22967 ``` 22968 paul@celebrimbor:~$ oidc-token XDC 22969 eyJraWQ[...]QFXDt0 22970 paul@celebrimbor:~$ 22971 ``` 22972 22973 **Note** Before the `oidc-token` command will work, the refresh token 22974 must be loaded into the oidc agent. This is done with the `oidc-add` 22975 command (e.g., `oidc-add XDC`). This is typically done once per login 22976 session. Full details on this and how to register oidc-agent with 22977 your OIDC Provider are provided in the [oidc-agent 22978 documentation](https://indigo-dc.gitbooks.io/oidc-agent/). 22979 22980 The rclone `bearer_token_command` configuration option is used to 22981 fetch the access token from oidc-agent. 22982 22983 Configure as a normal WebDAV endpoint, using the 'other' vendor, 22984 leaving the username and password empty. When prompted, choose to 22985 edit the advanced config and enter the command to get a bearer token 22986 (e.g., `oidc-agent XDC`). 22987 22988 The following example config shows a WebDAV endpoint that uses 22989 oidc-agent to supply an access token from the *XDC* OIDC Provider. 22990 22991 ``` 22992 [dcache] 22993 type = webdav 22994 url = https://dcache.example.org/ 22995 vendor = other 22996 bearer_token_command = oidc-token XDC 22997 ``` 22998 22999 Yandex Disk 23000 ---------------------------------------- 23001 23002 [Yandex Disk](https://disk.yandex.com) is a cloud storage solution created by [Yandex](https://yandex.com). 23003 23004 Here is an example of making a yandex configuration. First run 23005 23006 rclone config 23007 23008 This will guide you through an interactive setup process: 23009 23010 ``` 23011 No remotes found - make a new one 23012 n) New remote 23013 s) Set configuration password 23014 n/s> n 23015 name> remote 23016 Type of storage to configure. 23017 Choose a number from below, or type in your own value 23018 [snip] 23019 XX / Yandex Disk 23020 \ "yandex" 23021 [snip] 23022 Storage> yandex 23023 Yandex Client Id - leave blank normally. 23024 client_id> 23025 Yandex Client Secret - leave blank normally. 23026 client_secret> 23027 Remote config 23028 Use auto config? 23029 * Say Y if not sure 23030 * Say N if you are working on a remote or headless machine 23031 y) Yes 23032 n) No 23033 y/n> y 23034 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 23035 Log in and authorize rclone for access 23036 Waiting for code... 23037 Got code 23038 -------------------- 23039 [remote] 23040 client_id = 23041 client_secret = 23042 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","expiry":"2016-12-29T12:27:11.362788025Z"} 23043 -------------------- 23044 y) Yes this is OK 23045 e) Edit this remote 23046 d) Delete this remote 23047 y/e/d> y 23048 ``` 23049 23050 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 23051 machine with no Internet browser available. 23052 23053 Note that rclone runs a webserver on your local machine to collect the 23054 token as returned from Yandex Disk. This only runs from the moment it 23055 opens your browser to the moment you get back the verification code. 23056 This is on `http://127.0.0.1:53682/` and this it may require you to 23057 unblock it temporarily if you are running a host firewall. 23058 23059 Once configured you can then use `rclone` like this, 23060 23061 See top level directories 23062 23063 rclone lsd remote: 23064 23065 Make a new directory 23066 23067 rclone mkdir remote:directory 23068 23069 List the contents of a directory 23070 23071 rclone ls remote:directory 23072 23073 Sync `/home/local/directory` to the remote path, deleting any 23074 excess files in the path. 23075 23076 rclone sync /home/local/directory remote:directory 23077 23078 Yandex paths may be as deep as required, eg `remote:directory/subdirectory`. 23079 23080 ### Modified time ### 23081 23082 Modified times are supported and are stored accurate to 1 ns in custom 23083 metadata called `rclone_modified` in RFC3339 with nanoseconds format. 23084 23085 ### MD5 checksums ### 23086 23087 MD5 checksums are natively supported by Yandex Disk. 23088 23089 ### Emptying Trash ### 23090 23091 If you wish to empty your trash you can use the `rclone cleanup remote:` 23092 command which will permanently delete all your trashed files. This command 23093 does not take any path arguments. 23094 23095 ### Quota information ### 23096 23097 To view your current quota you can use the `rclone about remote:` 23098 command which will display your usage limit (quota) and the current usage. 23099 23100 #### Restricted filename characters 23101 23102 The [default restricted characters set](https://rclone.org/overview/#restricted-characters) 23103 are replaced. 23104 23105 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 23106 as they can't be used in JSON strings. 23107 23108 ### Limitations ### 23109 23110 When uploading very large files (bigger than about 5GB) you will need 23111 to increase the `--timeout` parameter. This is because Yandex pauses 23112 (perhaps to calculate the MD5SUM for the entire file) before returning 23113 confirmation that the file has been uploaded. The default handling of 23114 timeouts in rclone is to assume a 5 minute pause is an error and close 23115 the connection - you'll see `net/http: timeout awaiting response 23116 headers` errors in the logs if this is happening. Setting the timeout 23117 to twice the max size of file in GB should be enough, so if you want 23118 to upload a 30GB file set a timeout of `2 * 30 = 60m`, that is 23119 `--timeout 60m`. 23120 23121 23122 ### Standard Options 23123 23124 Here are the standard options specific to yandex (Yandex Disk). 23125 23126 #### --yandex-client-id 23127 23128 Yandex Client Id 23129 Leave blank normally. 23130 23131 - Config: client_id 23132 - Env Var: RCLONE_YANDEX_CLIENT_ID 23133 - Type: string 23134 - Default: "" 23135 23136 #### --yandex-client-secret 23137 23138 Yandex Client Secret 23139 Leave blank normally. 23140 23141 - Config: client_secret 23142 - Env Var: RCLONE_YANDEX_CLIENT_SECRET 23143 - Type: string 23144 - Default: "" 23145 23146 ### Advanced Options 23147 23148 Here are the advanced options specific to yandex (Yandex Disk). 23149 23150 #### --yandex-unlink 23151 23152 Remove existing public link to file/folder with link command rather than creating. 23153 Default is false, meaning link command will create or retrieve public link. 23154 23155 - Config: unlink 23156 - Env Var: RCLONE_YANDEX_UNLINK 23157 - Type: bool 23158 - Default: false 23159 23160 #### --yandex-encoding 23161 23162 This sets the encoding for the backend. 23163 23164 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 23165 23166 - Config: encoding 23167 - Env Var: RCLONE_YANDEX_ENCODING 23168 - Type: MultiEncoder 23169 - Default: Slash,Del,Ctl,InvalidUtf8,Dot 23170 23171 23172 23173 Local Filesystem 23174 ------------------------------------------- 23175 23176 Local paths are specified as normal filesystem paths, eg `/path/to/wherever`, so 23177 23178 rclone sync /home/source /tmp/destination 23179 23180 Will sync `/home/source` to `/tmp/destination` 23181 23182 These can be configured into the config file for consistencies sake, 23183 but it is probably easier not to. 23184 23185 ### Modified time ### 23186 23187 Rclone reads and writes the modified time using an accuracy determined by 23188 the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second 23189 on OS X. 23190 23191 ### Filenames ### 23192 23193 Filenames should be encoded in UTF-8 on disk. This is the normal case 23194 for Windows and OS X. 23195 23196 There is a bit more uncertainty in the Linux world, but new 23197 distributions will have UTF-8 encoded files names. If you are using an 23198 old Linux filesystem with non UTF-8 file names (eg latin1) then you 23199 can use the `convmv` tool to convert the filesystem to UTF-8. This 23200 tool is available in most distributions' package managers. 23201 23202 If an invalid (non-UTF8) filename is read, the invalid characters will 23203 be replaced with a quoted representation of the invalid bytes. The name 23204 `gro\xdf` will be transferred as `gro‛DF`. `rclone` will emit a debug 23205 message in this case (use `-v` to see), eg 23206 23207 ``` 23208 Local file system at .: Replacing invalid UTF-8 characters in "gro\xdf" 23209 ``` 23210 23211 #### Restricted characters 23212 23213 On non Windows platforms the following characters are replaced when 23214 handling file names. 23215 23216 | Character | Value | Replacement | 23217 | --------- |:-----:|:-----------:| 23218 | NUL | 0x00 | ␀ | 23219 | / | 0x2F | / | 23220 23221 When running on Windows the following characters are replaced. This 23222 list is based on the [Windows file naming conventions](https://docs.microsoft.com/de-de/windows/desktop/FileIO/naming-a-file#naming-conventions). 23223 23224 | Character | Value | Replacement | 23225 | --------- |:-----:|:-----------:| 23226 | NUL | 0x00 | ␀ | 23227 | SOH | 0x01 | ␁ | 23228 | STX | 0x02 | ␂ | 23229 | ETX | 0x03 | ␃ | 23230 | EOT | 0x04 | ␄ | 23231 | ENQ | 0x05 | ␅ | 23232 | ACK | 0x06 | ␆ | 23233 | BEL | 0x07 | ␇ | 23234 | BS | 0x08 | ␈ | 23235 | HT | 0x09 | ␉ | 23236 | LF | 0x0A | ␊ | 23237 | VT | 0x0B | ␋ | 23238 | FF | 0x0C | ␌ | 23239 | CR | 0x0D | ␍ | 23240 | SO | 0x0E | ␎ | 23241 | SI | 0x0F | ␏ | 23242 | DLE | 0x10 | ␐ | 23243 | DC1 | 0x11 | ␑ | 23244 | DC2 | 0x12 | ␒ | 23245 | DC3 | 0x13 | ␓ | 23246 | DC4 | 0x14 | ␔ | 23247 | NAK | 0x15 | ␕ | 23248 | SYN | 0x16 | ␖ | 23249 | ETB | 0x17 | ␗ | 23250 | CAN | 0x18 | ␘ | 23251 | EM | 0x19 | ␙ | 23252 | SUB | 0x1A | ␚ | 23253 | ESC | 0x1B | ␛ | 23254 | FS | 0x1C | ␜ | 23255 | GS | 0x1D | ␝ | 23256 | RS | 0x1E | ␞ | 23257 | US | 0x1F | ␟ | 23258 | / | 0x2F | / | 23259 | " | 0x22 | " | 23260 | * | 0x2A | * | 23261 | : | 0x3A | : | 23262 | < | 0x3C | < | 23263 | > | 0x3E | > | 23264 | ? | 0x3F | ? | 23265 | \ | 0x5C | \ | 23266 | \| | 0x7C | | | 23267 23268 File names on Windows can also not end with the following characters. 23269 These only get replaced if they are the last character in the name: 23270 23271 | Character | Value | Replacement | 23272 | --------- |:-----:|:-----------:| 23273 | SP | 0x20 | ␠ | 23274 | . | 0x2E | . | 23275 23276 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 23277 as they can't be converted to UTF-16. 23278 23279 ### Long paths on Windows ### 23280 23281 Rclone handles long paths automatically, by converting all paths to long 23282 [UNC paths](https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath) 23283 which allows paths up to 32,767 characters. 23284 23285 This is why you will see that your paths, for instance `c:\files` is 23286 converted to the UNC path `\\?\c:\files` in the output, 23287 and `\\server\share` is converted to `\\?\UNC\server\share`. 23288 23289 However, in rare cases this may cause problems with buggy file 23290 system drivers like [EncFS](https://github.com/rclone/rclone/issues/261). 23291 To disable UNC conversion globally, add this to your `.rclone.conf` file: 23292 23293 ``` 23294 [local] 23295 nounc = true 23296 ``` 23297 23298 If you want to selectively disable UNC, you can add it to a separate entry like this: 23299 23300 ``` 23301 [nounc] 23302 type = local 23303 nounc = true 23304 ``` 23305 And use rclone like this: 23306 23307 `rclone copy c:\src nounc:z:\dst` 23308 23309 This will use UNC paths on `c:\src` but not on `z:\dst`. 23310 Of course this will cause problems if the absolute path length of a 23311 file exceeds 258 characters on z, so only use this option if you have to. 23312 23313 ### Symlinks / Junction points 23314 23315 Normally rclone will ignore symlinks or junction points (which behave 23316 like symlinks under Windows). 23317 23318 If you supply `--copy-links` or `-L` then rclone will follow the 23319 symlink and copy the pointed to file or directory. Note that this 23320 flag is incompatible with `-links` / `-l`. 23321 23322 This flag applies to all commands. 23323 23324 For example, supposing you have a directory structure like this 23325 23326 ``` 23327 $ tree /tmp/a 23328 /tmp/a 23329 ├── b -> ../b 23330 ├── expected -> ../expected 23331 ├── one 23332 └── two 23333 └── three 23334 ``` 23335 23336 Then you can see the difference with and without the flag like this 23337 23338 ``` 23339 $ rclone ls /tmp/a 23340 6 one 23341 6 two/three 23342 ``` 23343 23344 and 23345 23346 ``` 23347 $ rclone -L ls /tmp/a 23348 4174 expected 23349 6 one 23350 6 two/three 23351 6 b/two 23352 6 b/one 23353 ``` 23354 23355 #### --links, -l 23356 23357 Normally rclone will ignore symlinks or junction points (which behave 23358 like symlinks under Windows). 23359 23360 If you supply this flag then rclone will copy symbolic links from the local storage, 23361 and store them as text files, with a '.rclonelink' suffix in the remote storage. 23362 23363 The text file will contain the target of the symbolic link (see example). 23364 23365 This flag applies to all commands. 23366 23367 For example, supposing you have a directory structure like this 23368 23369 ``` 23370 $ tree /tmp/a 23371 /tmp/a 23372 ├── file1 -> ./file4 23373 └── file2 -> /home/user/file3 23374 ``` 23375 23376 Copying the entire directory with '-l' 23377 23378 ``` 23379 $ rclone copyto -l /tmp/a/file1 remote:/tmp/a/ 23380 ``` 23381 23382 The remote files are created with a '.rclonelink' suffix 23383 23384 ``` 23385 $ rclone ls remote:/tmp/a 23386 5 file1.rclonelink 23387 14 file2.rclonelink 23388 ``` 23389 23390 The remote files will contain the target of the symbolic links 23391 23392 ``` 23393 $ rclone cat remote:/tmp/a/file1.rclonelink 23394 ./file4 23395 23396 $ rclone cat remote:/tmp/a/file2.rclonelink 23397 /home/user/file3 23398 ``` 23399 23400 Copying them back with '-l' 23401 23402 ``` 23403 $ rclone copyto -l remote:/tmp/a/ /tmp/b/ 23404 23405 $ tree /tmp/b 23406 /tmp/b 23407 ├── file1 -> ./file4 23408 └── file2 -> /home/user/file3 23409 ``` 23410 23411 However, if copied back without '-l' 23412 23413 ``` 23414 $ rclone copyto remote:/tmp/a/ /tmp/b/ 23415 23416 $ tree /tmp/b 23417 /tmp/b 23418 ├── file1.rclonelink 23419 └── file2.rclonelink 23420 ```` 23421 23422 Note that this flag is incompatible with `-copy-links` / `-L`. 23423 23424 ### Restricting filesystems with --one-file-system 23425 23426 Normally rclone will recurse through filesystems as mounted. 23427 23428 However if you set `--one-file-system` or `-x` this tells rclone to 23429 stay in the filesystem specified by the root and not to recurse into 23430 different file systems. 23431 23432 For example if you have a directory hierarchy like this 23433 23434 ``` 23435 root 23436 ├── disk1 - disk1 mounted on the root 23437 │ └── file3 - stored on disk1 23438 ├── disk2 - disk2 mounted on the root 23439 │ └── file4 - stored on disk12 23440 ├── file1 - stored on the root disk 23441 └── file2 - stored on the root disk 23442 ``` 23443 23444 Using `rclone --one-file-system copy root remote:` will only copy `file1` and `file2`. Eg 23445 23446 ``` 23447 $ rclone -q --one-file-system ls root 23448 0 file1 23449 0 file2 23450 ``` 23451 23452 ``` 23453 $ rclone -q ls root 23454 0 disk1/file3 23455 0 disk2/file4 23456 0 file1 23457 0 file2 23458 ``` 23459 23460 **NB** Rclone (like most unix tools such as `du`, `rsync` and `tar`) 23461 treats a bind mount to the same device as being on the same 23462 filesystem. 23463 23464 **NB** This flag is only available on Unix based systems. On systems 23465 where it isn't supported (eg Windows) it will be ignored. 23466 23467 23468 ### Standard Options 23469 23470 Here are the standard options specific to local (Local Disk). 23471 23472 #### --local-nounc 23473 23474 Disable UNC (long path names) conversion on Windows 23475 23476 - Config: nounc 23477 - Env Var: RCLONE_LOCAL_NOUNC 23478 - Type: string 23479 - Default: "" 23480 - Examples: 23481 - "true" 23482 - Disables long file names 23483 23484 ### Advanced Options 23485 23486 Here are the advanced options specific to local (Local Disk). 23487 23488 #### --copy-links / -L 23489 23490 Follow symlinks and copy the pointed to item. 23491 23492 - Config: copy_links 23493 - Env Var: RCLONE_LOCAL_COPY_LINKS 23494 - Type: bool 23495 - Default: false 23496 23497 #### --links / -l 23498 23499 Translate symlinks to/from regular files with a '.rclonelink' extension 23500 23501 - Config: links 23502 - Env Var: RCLONE_LOCAL_LINKS 23503 - Type: bool 23504 - Default: false 23505 23506 #### --skip-links 23507 23508 Don't warn about skipped symlinks. 23509 This flag disables warning messages on skipped symlinks or junction 23510 points, as you explicitly acknowledge that they should be skipped. 23511 23512 - Config: skip_links 23513 - Env Var: RCLONE_LOCAL_SKIP_LINKS 23514 - Type: bool 23515 - Default: false 23516 23517 #### --local-no-unicode-normalization 23518 23519 Don't apply unicode normalization to paths and filenames (Deprecated) 23520 23521 This flag is deprecated now. Rclone no longer normalizes unicode file 23522 names, but it compares them with unicode normalization in the sync 23523 routine instead. 23524 23525 - Config: no_unicode_normalization 23526 - Env Var: RCLONE_LOCAL_NO_UNICODE_NORMALIZATION 23527 - Type: bool 23528 - Default: false 23529 23530 #### --local-no-check-updated 23531 23532 Don't check to see if the files change during upload 23533 23534 Normally rclone checks the size and modification time of files as they 23535 are being uploaded and aborts with a message which starts "can't copy 23536 - source file is being updated" if the file changes during upload. 23537 23538 However on some file systems this modification time check may fail (eg 23539 [Glusterfs #2206](https://github.com/rclone/rclone/issues/2206)) so this 23540 check can be disabled with this flag. 23541 23542 - Config: no_check_updated 23543 - Env Var: RCLONE_LOCAL_NO_CHECK_UPDATED 23544 - Type: bool 23545 - Default: false 23546 23547 #### --one-file-system / -x 23548 23549 Don't cross filesystem boundaries (unix/macOS only). 23550 23551 - Config: one_file_system 23552 - Env Var: RCLONE_LOCAL_ONE_FILE_SYSTEM 23553 - Type: bool 23554 - Default: false 23555 23556 #### --local-case-sensitive 23557 23558 Force the filesystem to report itself as case sensitive. 23559 23560 Normally the local backend declares itself as case insensitive on 23561 Windows/macOS and case sensitive for everything else. Use this flag 23562 to override the default choice. 23563 23564 - Config: case_sensitive 23565 - Env Var: RCLONE_LOCAL_CASE_SENSITIVE 23566 - Type: bool 23567 - Default: false 23568 23569 #### --local-case-insensitive 23570 23571 Force the filesystem to report itself as case insensitive 23572 23573 Normally the local backend declares itself as case insensitive on 23574 Windows/macOS and case sensitive for everything else. Use this flag 23575 to override the default choice. 23576 23577 - Config: case_insensitive 23578 - Env Var: RCLONE_LOCAL_CASE_INSENSITIVE 23579 - Type: bool 23580 - Default: false 23581 23582 #### --local-no-sparse 23583 23584 Disable sparse files for multi-thread downloads 23585 23586 On Windows platforms rclone will make sparse files when doing 23587 multi-thread downloads. This avoids long pauses on large files where 23588 the OS zeros the file. However sparse files may be undesirable as they 23589 cause disk fragmentation and can be slow to work with. 23590 23591 - Config: no_sparse 23592 - Env Var: RCLONE_LOCAL_NO_SPARSE 23593 - Type: bool 23594 - Default: false 23595 23596 #### --local-encoding 23597 23598 This sets the encoding for the backend. 23599 23600 See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 23601 23602 - Config: encoding 23603 - Env Var: RCLONE_LOCAL_ENCODING 23604 - Type: MultiEncoder 23605 - Default: Slash,Dot 23606 23607 ### Backend commands 23608 23609 Here are the commands specific to the local backend. 23610 23611 Run them with 23612 23613 rclone backend COMMAND remote: 23614 23615 The help below will explain what arguments each command takes. 23616 23617 See [the "rclone backend" command](https://rclone.org/commands/rclone_backend/) for more 23618 info on how to pass options and arguments. 23619 23620 These can be run on a running backend using the rc command 23621 [backend/command](https://rclone.org/rc/#backend/command). 23622 23623 #### noop 23624 23625 A null operation for testing backend commands 23626 23627 rclone backend noop remote: [options] [<arguments>+] 23628 23629 This is a test command which has some options 23630 you can try to change the output. 23631 23632 Options: 23633 23634 - "echo": echo the input arguments 23635 - "error": return an error based on option value 23636 23637 23638 23639 # Changelog 23640 23641 ## v1.52.2 - 2020-06-24 23642 23643 [See commits](https://github.com/rclone/rclone/compare/v1.52.1...v1.52.2) 23644 23645 * Bug Fixes 23646 * build 23647 * Fix docker release build action (Nick Craig-Wood) 23648 * Fix custom timezone in Docker image (NoLooseEnds) 23649 * check: Fix misleading message which printed errors instead of differences (Nick Craig-Wood) 23650 * errors: Add WSAECONNREFUSED and more to the list of retriable Windows errors (Nick Craig-Wood) 23651 * rcd: Fix incorrect prometheus metrics (Gary Kim) 23652 * serve restic: Fix flags so they use environment variables (Nick Craig-Wood) 23653 * serve webdav: Fix flags so they use environment variables (Nick Craig-Wood) 23654 * sync: Fix --track-renames-strategy modtime (Nick Craig-Wood) 23655 * Drive 23656 * Fix not being able to delete a directory with a trashed shortcut (Nick Craig-Wood) 23657 * Fix creating a directory inside a shortcut (Nick Craig-Wood) 23658 * Fix --drive-impersonate with cached root_folder_id (Nick Craig-Wood) 23659 * SFTP 23660 * Fix SSH key PEM loading (Zac Rubin) 23661 * Swift 23662 * Speed up deletes by not retrying segment container deletes (Nick Craig-Wood) 23663 * Tardigrade 23664 * Upgrade to uplink v1.1.1 (Caleb Case) 23665 * WebDAV 23666 * Fix free/used display for rclone about/df for certain backends (Nick Craig-Wood) 23667 23668 ## v1.52.1 - 2020-06-10 23669 23670 [See commits](https://github.com/rclone/rclone/compare/v1.52.0...v1.52.1) 23671 23672 * Bug Fixes 23673 * lib/file: Fix SetSparse on Windows 7 which fixes downloads of files > 250MB (Nick Craig-Wood) 23674 * build 23675 * Update go.mod to go1.14 to enable -mod=vendor build (Nick Craig-Wood) 23676 * Remove quicktest from Dockerfile (Nick Craig-Wood) 23677 * Build Docker images with GitHub actions (Matteo Pietro Dazzi) 23678 * Update Docker build workflows (Nick Craig-Wood) 23679 * Set user_allow_other in /etc/fuse.conf in the Docker image (Nick Craig-Wood) 23680 * Fix xgo build after go1.14 go.mod update (Nick Craig-Wood) 23681 * docs 23682 * Add link to source and modified time to footer of every page (Nick Craig-Wood) 23683 * Remove manually set dates and use git dates instead (Nick Craig-Wood) 23684 * Minor tense, punctuation, brevity and positivity changes for the home page (edwardxml) 23685 * Remove leading slash in page reference in footer when present (Nick Craig-Wood) 23686 * Note commands which need obscured input in the docs (Nick Craig-Wood) 23687 * obscure: Write more help as we are referencing it elsewhere (Nick Craig-Wood) 23688 * VFS 23689 * Fix OS vs Unix path confusion - fixes ChangeNotify on Windows (Nick Craig-Wood) 23690 * Drive 23691 * Fix missing items when listing using --fast-list / ListR (Nick Craig-Wood) 23692 * Putio 23693 * Fix panic on Object.Open (Cenk Alti) 23694 * S3 23695 * Fix upload of single files into buckets without create permission (Nick Craig-Wood) 23696 * Fix --header-upload (Nick Craig-Wood) 23697 * Tardigrade 23698 * Fix listing bug by upgrading to v1.0.7 23699 * Set UserAgent to rclone (Caleb Case) 23700 23701 ## v1.52.0 - 2020-05-27 23702 23703 Special thanks to Martin Michlmayr for proof reading and correcting 23704 all the docs and Edward Barker for helping re-write the front page. 23705 23706 [See commits](https://github.com/rclone/rclone/compare/v1.51.0...v1.52.0) 23707 23708 * New backends 23709 * [Tardigrade](https://rclone.org/tardigrade/) backend for use with storj.io (Caleb Case) 23710 * [Union](https://rclone.org/union/) re-write to have multiple writable remotes (Max Sum) 23711 * [Seafile](/seafile) for Seafile server (Fred @creativeprojects) 23712 * New commands 23713 * backend: command for backend specific commands (see backends) (Nick Craig-Wood) 23714 * cachestats: Deprecate in favour of `rclone backend stats cache:` (Nick Craig-Wood) 23715 * dbhashsum: Deprecate in favour of `rclone hashsum DropboxHash` (Nick Craig-Wood) 23716 * New Features 23717 * Add `--header-download` and `--header-upload` flags for setting HTTP headers when uploading/downloading (Tim Gallant) 23718 * Add `--header` flag to add HTTP headers to every HTTP transaction (Nick Craig-Wood) 23719 * Add `--check-first` to do all checking before starting transfers (Nick Craig-Wood) 23720 * Add `--track-renames-strategy` for configurable matching criteria for `--track-renames` (Bernd Schoolmann) 23721 * Add `--cutoff-mode` hard,soft,catious (Shing Kit Chan & Franklyn Tackitt) 23722 * Filter flags (eg `--files-from -`) can read from stdin (fishbullet) 23723 * Add `--error-on-no-transfer` option (Jon Fautley) 23724 * Implement `--order-by xxx,mixed` for copying some small and some big files (Nick Craig-Wood) 23725 * Allow `--max-backlog` to be negative meaning as large as possible (Nick Craig-Wood) 23726 * Added `--no-unicode-normalization` flag to allow Unicode filenames to remain unique (Ben Zenker) 23727 * Allow `--min-age`/`--max-age` to take a date as well as a duration (Nick Craig-Wood) 23728 * Add rename statistics for file and directory renames (Nick Craig-Wood) 23729 * Add statistics output to JSON log (reddi) 23730 * Make stats be printed on non-zero exit code (Nick Craig-Wood) 23731 * When running `--password-command` allow use of stdin (Sébastien Gross) 23732 * Stop empty strings being a valid remote path (Nick Craig-Wood) 23733 * accounting: support WriterTo for less memory copying (Nick Craig-Wood) 23734 * build 23735 * Update to use go1.14 for the build (Nick Craig-Wood) 23736 * Add `-trimpath` to release build for reproduceable builds (Nick Craig-Wood) 23737 * Remove GOOS and GOARCH from Dockerfile (Brandon Philips) 23738 * config 23739 * Fsync the config file after writing to save more reliably (Nick Craig-Wood) 23740 * Add `--obscure` and `--no-obscure` flags to `config create`/`update` (Nick Craig-Wood) 23741 * Make `config show` take `remote:` as well as `remote` (Nick Craig-Wood) 23742 * copyurl: Add `--no-clobber` flag (Denis) 23743 * delete: Added `--rmdirs` flag to delete directories as well (Kush) 23744 * filter: Added `--files-from-raw` flag (Ankur Gupta) 23745 * genautocomplete: Add support for fish shell (Matan Rosenberg) 23746 * log: Add support for syslog LOCAL facilities (Patryk Jakuszew) 23747 * lsjson: Add `--hash-type` parameter and use it in lsf to speed up hashing (Nick Craig-Wood) 23748 * rc 23749 * Add `-o`/`--opt` and `-a`/`--arg` for more structured input (Nick Craig-Wood) 23750 * Implement `backend/command` for running backend specific commands remotely (Nick Craig-Wood) 23751 * Add `mount/mount` command for starting `rclone mount` via the API (Chaitanya) 23752 * rcd: Add Prometheus metrics support (Gary Kim) 23753 * serve http 23754 * Added a `--template` flag for user defined markup (calistri) 23755 * Add Last-Modified headers to files and directories (Nick Craig-Wood) 23756 * serve sftp: Add support for multiple host keys by repeating `--key` flag (Maxime Suret) 23757 * touch: Add `--localtime` flag to make `--timestamp` localtime not UTC (Nick Craig-Wood) 23758 * Bug Fixes 23759 * accounting 23760 * Restore "Max number of stats groups reached" log line (Michał Matczuk) 23761 * Correct exitcode on Transfer Limit Exceeded flag. (Anuar Serdaliyev) 23762 * Reset bytes read during copy retry (Ankur Gupta) 23763 * Fix race clearing stats (Nick Craig-Wood) 23764 * copy: Only create empty directories when they don't exist on the remote (Ishuah Kariuki) 23765 * dedupe: Stop dedupe deleting files with identical IDs (Nick Craig-Wood) 23766 * oauth 23767 * Use custom http client so that `--no-check-certificate` is honored by oauth token fetch (Mark Spieth) 23768 * Replace deprecated oauth2.NoContext (Lars Lehtonen) 23769 * operations 23770 * Fix setting the timestamp on Windows for multithread copy (Nick Craig-Wood) 23771 * Make rcat obey `--ignore-checksum` (Nick Craig-Wood) 23772 * Make `--max-transfer` more accurate (Nick Craig-Wood) 23773 * rc 23774 * Fix dropped error (Lars Lehtonen) 23775 * Fix misplaced http server config (Xiaoxing Ye) 23776 * Disable duplicate log (ElonH) 23777 * serve dlna 23778 * Cds: don't specify childCount at all when unknown (Dan Walters) 23779 * Cds: use modification time as date in dlna metadata (Dan Walters) 23780 * serve restic: Fix tests after restic project removed vendoring (Nick Craig-Wood) 23781 * sync 23782 * Fix incorrect "nothing to transfer" message using `--delete-before` (Nick Craig-Wood) 23783 * Only create empty directories when they don't exist on the remote (Ishuah Kariuki) 23784 * Mount 23785 * Add `--async-read` flag to disable asynchronous reads (Nick Craig-Wood) 23786 * Ignore `--allow-root` flag with a warning as it has been removed upstream (Nick Craig-Wood) 23787 * Warn if `--allow-non-empty` used on Windows and clarify docs (Nick Craig-Wood) 23788 * Constrain to go1.13 or above otherwise bazil.org/fuse fails to compile (Nick Craig-Wood) 23789 * Fix fail because of too long volume name (evileye) 23790 * Report 1PB free for unknown disk sizes (Nick Craig-Wood) 23791 * Map more rclone errors into file systems errors (Nick Craig-Wood) 23792 * Fix disappearing cwd problem (Nick Craig-Wood) 23793 * Use ReaddirPlus on Windows to improve directory listing performance (Nick Craig-Wood) 23794 * Send a hint as to whether the filesystem is case insensitive or not (Nick Craig-Wood) 23795 * Add rc command `mount/types` (Nick Craig-Wood) 23796 * Change maximum leaf name length to 1024 bytes (Nick Craig-Wood) 23797 * VFS 23798 * Add `--vfs-read-wait` and `--vfs-write-wait` flags to control time waiting for a sequential read/write (Nick Craig-Wood) 23799 * Change default `--vfs-read-wait` to 20ms (it was 5ms and not configurable) (Nick Craig-Wood) 23800 * Make `df` output more consistent on a rclone mount. (Yves G) 23801 * Report 1PB free for unknown disk sizes (Nick Craig-Wood) 23802 * Fix race condition caused by unlocked reading of Dir.path (Nick Craig-Wood) 23803 * Make File lock and Dir lock not overlap to avoid deadlock (Nick Craig-Wood) 23804 * Implement lock ordering between File and Dir to eliminate deadlocks (Nick Craig-Wood) 23805 * Factor the vfs cache into its own package (Nick Craig-Wood) 23806 * Pin the Fs in use in the Fs cache (Nick Craig-Wood) 23807 * Add SetSys() methods to Node to allow caching stuff on a node (Nick Craig-Wood) 23808 * Ignore file not found errors from Hash in Read.Release (Nick Craig-Wood) 23809 * Fix hang in read wait code (Nick Craig-Wood) 23810 * Local 23811 * Speed up multi thread downloads by using sparse files on Windows (Nick Craig-Wood) 23812 * Implement `--local-no-sparse` flag for disabling sparse files (Nick Craig-Wood) 23813 * Implement `rclone backend noop` for testing purposes (Nick Craig-Wood) 23814 * Fix "file not found" errors on post transfer Hash calculation (Nick Craig-Wood) 23815 * Cache 23816 * Implement `rclone backend stats` command (Nick Craig-Wood) 23817 * Fix Server Side Copy with Temp Upload (Brandon McNama) 23818 * Remove Unused Functions (Lars Lehtonen) 23819 * Disable race tests until bbolt is fixed (Nick Craig-Wood) 23820 * Move methods used for testing into test file (greatroar) 23821 * Add Pin and Unpin and canonicalised lookup (Nick Craig-Wood) 23822 * Use proper import path go.etcd.io/bbolt (Robert-André Mauchin) 23823 * Crypt 23824 * Calculate hashes for uploads from local disk (Nick Craig-Wood) 23825 * This allows crypted Jottacloud uploads without using local disk 23826 * This means crypted s3/b2 uploads will now have hashes 23827 * Added `rclone backend decode`/`encode` commands to replicate functionality of `cryptdecode` (Anagh Kumar Baranwal) 23828 * Get rid of the unused Cipher interface as it obfuscated the code (Nick Craig-Wood) 23829 * Azure Blob 23830 * Implement streaming of unknown sized files so `rcat` is now supported (Nick Craig-Wood) 23831 * Implement memory pooling to control memory use (Nick Craig-Wood) 23832 * Add `--azureblob-disable-checksum` flag (Nick Craig-Wood) 23833 * Retry `InvalidBlobOrBlock` error as it may indicate block concurrency problems (Nick Craig-Wood) 23834 * Remove unused `Object.parseTimeString()` (Lars Lehtonen) 23835 * Fix permission error on SAS URL limited to container (Nick Craig-Wood) 23836 * B2 23837 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23838 * Ignore directory markers at the root also (Nick Craig-Wood) 23839 * Force the case of the SHA1 to lowercase (Nick Craig-Wood) 23840 * Remove unused `largeUpload.clearUploadURL()` (Lars Lehtonen) 23841 * Box 23842 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23843 * Implement About to read size used (Nick Craig-Wood) 23844 * Add token renew function for jwt auth (David Bramwell) 23845 * Added support for interchangeable root folder for Box backend (Sunil Patra) 23846 * Remove unnecessary iat from jws claims (David) 23847 * Drive 23848 * Follow shortcuts by default, skip with `--drive-skip-shortcuts` (Nick Craig-Wood) 23849 * Implement `rclone backend shortcut` command for creating shortcuts (Nick Craig-Wood) 23850 * Added `rclone backend` command to change `service_account_file` and `chunk_size` (Anagh Kumar Baranwal) 23851 * Fix missing files when using `--fast-list` and `--drive-shared-with-me` (Nick Craig-Wood) 23852 * Fix duplicate items when using `--drive-shared-with-me` (Nick Craig-Wood) 23853 * Extend `--drive-stop-on-upload-limit` to respond to `teamDriveFileLimitExceeded`. (harry) 23854 * Don't delete files with multiple parents to avoid data loss (Nick Craig-Wood) 23855 * Server side copy docs use default description if empty (Nick Craig-Wood) 23856 * Dropbox 23857 * Make error insufficient space to be fatal (harry) 23858 * Add info about required redirect url (Elan Ruusamäe) 23859 * Fichier 23860 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23861 * Implement custom pacer to deal with the new rate limiting (buengese) 23862 * FTP 23863 * Fix lockup when using concurrency limit on failed connections (Nick Craig-Wood) 23864 * Fix lockup on failed upload when using concurrency limit (Nick Craig-Wood) 23865 * Fix lockup on Close failures when using concurrency limit (Nick Craig-Wood) 23866 * Work around pureftp sending spurious 150 messages (Nick Craig-Wood) 23867 * Google Cloud Storage 23868 * Add support for `--header-upload` and `--header-download` (Nick Craig-Wood) 23869 * Add `ARCHIVE` storage class to help (Adam Stroud) 23870 * Ignore directory markers at the root (Nick Craig-Wood) 23871 * Googlephotos 23872 * Make the start year configurable (Daven) 23873 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23874 * Create feature/favorites directory (Brandon Philips) 23875 * Fix "concurrent map write" error (Nick Craig-Wood) 23876 * Don't put an image in error message (Nick Craig-Wood) 23877 * HTTP 23878 * Improved directory listing with new template from Caddy project (calisro) 23879 * Jottacloud 23880 * Implement `--jottacloud-trashed-only` (buengese) 23881 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23882 * Use `RawURLEncoding` when decoding base64 encoded login token (buengese) 23883 * Implement cleanup (buengese) 23884 * Update docs regarding cleanup, removed remains from old auth, and added warning about special mountpoints. (albertony) 23885 * Mailru 23886 * Describe 2FA requirements (valery1707) 23887 * Onedrive 23888 * Implement `--onedrive-server-side-across-configs` (Nick Craig-Wood) 23889 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23890 * Fix occasional 416 errors on multipart uploads (Nick Craig-Wood) 23891 * Added maximum chunk size limit warning in the docs (Harry) 23892 * Fix missing drive on config (Nick Craig-Wood) 23893 * Make error `quotaLimitReached` to be fatal (harry) 23894 * Opendrive 23895 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23896 * Pcloud 23897 * Added support for interchangeable root folder for pCloud backend (Sunil Patra) 23898 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23899 * Fix initial config "Auth state doesn't match" message (Nick Craig-Wood) 23900 * Premiumizeme 23901 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23902 * Prune unused functions (Lars Lehtonen) 23903 * Putio 23904 * Add support for `--header-upload` and `--header-download` (Nick Craig-Wood) 23905 * Make downloading files use the rclone http Client (Nick Craig-Wood) 23906 * Fix parsing of remotes with leading and trailing / (Nick Craig-Wood) 23907 * Qingstor 23908 * Make `rclone cleanup` remove pending multipart uploads older than 24h (Nick Craig-Wood) 23909 * Try harder to cancel failed multipart uploads (Nick Craig-Wood) 23910 * Prune `multiUploader.list()` (Lars Lehtonen) 23911 * Lint fix (Lars Lehtonen) 23912 * S3 23913 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23914 * Use memory pool for buffer allocations (Maciej Zimnoch) 23915 * Add SSE-C support for AWS, Ceph, and MinIO (Jack Anderson) 23916 * Fail fast multipart upload (Michał Matczuk) 23917 * Report errors on bucket creation (mkdir) correctly (Nick Craig-Wood) 23918 * Specify that Minio supports URL encoding in listings (Nick Craig-Wood) 23919 * Added 500 as retryErrorCode (Michał Matczuk) 23920 * Use `--low-level-retries` as the number of SDK retries (Aleksandar Janković) 23921 * Fix multipart abort context (Aleksandar Jankovic) 23922 * Replace deprecated `session.New()` with `session.NewSession()` (Lars Lehtonen) 23923 * Use the provided size parameter when allocating a new memory pool (Joachim Brandon LeBlanc) 23924 * Use rclone's low level retries instead of AWS SDK to fix listing retries (Nick Craig-Wood) 23925 * Ignore directory markers at the root also (Nick Craig-Wood) 23926 * Use single memory pool (Michał Matczuk) 23927 * Do not resize buf on put to memBuf (Michał Matczuk) 23928 * Improve docs for `--s3-disable-checksum` (Nick Craig-Wood) 23929 * Don't leak memory or tokens in edge cases for multipart upload (Nick Craig-Wood) 23930 * Seafile 23931 * Implement 2FA (Fred) 23932 * SFTP 23933 * Added `--sftp-pem-key` to support inline key files (calisro) 23934 * Fix post transfer copies failing with 0 size when using `set_modtime=false` (Nick Craig-Wood) 23935 * Sharefile 23936 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23937 * Sugarsync 23938 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23939 * Swift 23940 * Add support for `--header-upload` and `--header-download` (Nick Craig-Wood) 23941 * Fix cosmetic issue in error message (Martin Michlmayr) 23942 * Union 23943 * Implement multiple writable remotes (Max Sum) 23944 * Fix server-side copy (Max Sum) 23945 * Implement ListR (Max Sum) 23946 * Enable ListR when upstreams contain local (Max Sum) 23947 * WebDAV 23948 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23949 * Fix `X-OC-Mtime` header for Transip compatibility (Nick Craig-Wood) 23950 * Report full and consistent usage with `about` (Yves G) 23951 * Yandex 23952 * Add support for `--header-upload` and `--header-download` (Tim Gallant) 23953 23954 ## v1.51.0 - 2020-02-01 23955 23956 * New backends 23957 * [Memory](https://rclone.org/memory/) (Nick Craig-Wood) 23958 * [Sugarsync](https://rclone.org/sugarsync/) (Nick Craig-Wood) 23959 * New Features 23960 * Adjust all backends to have `--backend-encoding` parameter (Nick Craig-Wood) 23961 * this enables the encoding for special characters to be adjusted or disabled 23962 * Add `--max-duration` flag to control the maximum duration of a transfer session (boosh) 23963 * Add `--expect-continue-timeout` flag, default 1s (Nick Craig-Wood) 23964 * Add `--no-check-dest` flag for copying without testing the destination (Nick Craig-Wood) 23965 * Implement `--order-by` flag to order transfers (Nick Craig-Wood) 23966 * accounting 23967 * Don't show entries in both transferring and checking (Nick Craig-Wood) 23968 * Add option to delete stats (Aleksandar Jankovic) 23969 * build 23970 * Compress the test builds with gzip (Nick Craig-Wood) 23971 * Implement a framework for starting test servers during tests (Nick Craig-Wood) 23972 * cmd: Always print elapsed time to tenth place seconds in progress (Gary Kim) 23973 * config 23974 * Add `--password-command` to allow dynamic config password (Damon Permezel) 23975 * Give config questions default values (Nick Craig-Wood) 23976 * Check a remote exists when creating a new one (Nick Craig-Wood) 23977 * copyurl: Add `--stdout` flag to write to stdout (Nick Craig-Wood) 23978 * dedupe: Implement keep smallest too (Nick Craig-Wood) 23979 * hashsum: Add flag `--base64` flag (landall) 23980 * lsf: Speed up on s3/swift/etc by not reading mimetype by default (Nick Craig-Wood) 23981 * lsjson: Add `--no-mimetype` flag (Nick Craig-Wood) 23982 * rc: Add methods to turn on blocking and mutex profiling (Nick Craig-Wood) 23983 * rcd 23984 * Adding group parameter to stats (Chaitanya) 23985 * Move webgui apart; option to disable browser (Xiaoxing Ye) 23986 * serve sftp: Add support for public key with auth proxy (Paul Tinsley) 23987 * stats: Show deletes in stats and hide zero stats (anuar45) 23988 * Bug Fixes 23989 * accounting 23990 * Fix error counter counting multiple times (Ankur Gupta) 23991 * Fix error count shown as checks (Cnly) 23992 * Clear finished transfer in stats-reset (Maciej Zimnoch) 23993 * Added StatsInfo locking in statsGroups sum function (Michał Matczuk) 23994 * asyncreader: Fix EOF error (buengese) 23995 * check: Fix `--one-way` recursing more directories than it needs to (Nick Craig-Wood) 23996 * chunkedreader: Disable hash calculation for first segment (Nick Craig-Wood) 23997 * config 23998 * Do not open browser on headless on drive/gcs/google photos (Xiaoxing Ye) 23999 * SetValueAndSave ignore error if config section does not exist yet (buengese) 24000 * cmd: Fix completion with an encrypted config (Danil Semelenov) 24001 * dbhashsum: Stop it returning UNSUPPORTED on dropbox (Nick Craig-Wood) 24002 * dedupe: Add missing modes to help string (Nick Craig-Wood) 24003 * operations 24004 * Fix dedupe continuing on errors like insufficientFilePermisson (SezalAgrawal) 24005 * Clear accounting before low level retry (Maciej Zimnoch) 24006 * Write debug message when hashes could not be checked (Ole Schütt) 24007 * Move interface assertion to tests to remove pflag dependency (Nick Craig-Wood) 24008 * Make NewOverrideObjectInfo public and factor uses (Nick Craig-Wood) 24009 * proxy: Replace use of bcrypt with sha256 (Nick Craig-Wood) 24010 * vendor 24011 * Update bazil.org/fuse to fix FreeBSD 12.1 (Nick Craig-Wood) 24012 * Update github.com/t3rm1n4l/go-mega to fix mega "illegal base64 data at input byte 22" (Nick Craig-Wood) 24013 * Update termbox-go to fix ncdu command on FreeBSD (Kuang-che Wu) 24014 * Update t3rm1n4l/go-mega - fixes mega: couldn't login: crypto/aes: invalid key size 0 (Nick Craig-Wood) 24015 * Mount 24016 * Enable async reads for a 20% speedup (Nick Craig-Wood) 24017 * Replace use of WriteAt with Write for cache mode >= writes and O_APPEND (Brett Dutro) 24018 * Make sure we call unmount when exiting (Nick Craig-Wood) 24019 * Don't build on go1.10 as bazil/fuse no longer supports it (Nick Craig-Wood) 24020 * When setting dates discard out of range dates (Nick Craig-Wood) 24021 * VFS 24022 * Add a newly created file straight into the directory (Nick Craig-Wood) 24023 * Only calculate one hash for reads for a speedup (Nick Craig-Wood) 24024 * Make ReadAt for non cached files work better with non-sequential reads (Nick Craig-Wood) 24025 * Fix edge cases when reading ModTime from file (Nick Craig-Wood) 24026 * Make sure existing files opened for write show correct size (Nick Craig-Wood) 24027 * Don't cache the path in RW file objects to fix renaming (Nick Craig-Wood) 24028 * Fix rename of open files when using the VFS cache (Nick Craig-Wood) 24029 * When renaming files in the cache, rename the cache item in memory too (Nick Craig-Wood) 24030 * Fix open file renaming on drive when using `--vfs-cache-mode writes` (Nick Craig-Wood) 24031 * Fix incorrect modtime for mv into mount with `--vfs-cache-modes writes` (Nick Craig-Wood) 24032 * On rename, rename in cache too if the file exists (Anagh Kumar Baranwal) 24033 * Local 24034 * Make source file being updated errors be NoLowLevelRetry errors (Nick Craig-Wood) 24035 * Fix update of hidden files on Windows (Nick Craig-Wood) 24036 * Cache 24037 * Follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt (Nick Craig-Wood) 24038 * Fix `fatal error: concurrent map writes` (Nick Craig-Wood) 24039 * Crypt 24040 * Reorder the filename encryption options (Thomas Eales) 24041 * Correctly handle trailing dot (buengese) 24042 * Chunker 24043 * Reduce length of temporary suffix (Ivan Andreev) 24044 * Drive 24045 * Add `--drive-stop-on-upload-limit` flag to stop syncs when upload limit reached (Nick Craig-Wood) 24046 * Add `--drive-use-shared-date` to use date file was shared instead of modified date (Garry McNulty) 24047 * Make sure invalid auth for teamdrives always reports an error (Nick Craig-Wood) 24048 * Fix `--fast-list` when using appDataFolder (Nick Craig-Wood) 24049 * Use multipart resumable uploads for streaming and uploads in mount (Nick Craig-Wood) 24050 * Log an ERROR if an incomplete search is returned (Nick Craig-Wood) 24051 * Hide dangerous config from the configurator (Nick Craig-Wood) 24052 * Dropbox 24053 * Treat `insufficient_space` errors as non retriable errors (Nick Craig-Wood) 24054 * Jottacloud 24055 * Use new auth method used by official client (buengese) 24056 * Add URL to generate Login Token to config wizard (Nick Craig-Wood) 24057 * Add support whitelabel versions (buengese) 24058 * Koofr 24059 * Use rclone HTTP client. (jaKa) 24060 * Onedrive 24061 * Add Sites.Read.All permission (Benjamin Richter) 24062 * Add support "Retry-After" header (Motonori IWAMURO) 24063 * Opendrive 24064 * Implement `--opendrive-chunk-size` (Nick Craig-Wood) 24065 * S3 24066 * Re-implement multipart upload to fix memory issues (Nick Craig-Wood) 24067 * Add `--s3-copy-cutoff` for size to switch to multipart copy (Nick Craig-Wood) 24068 * Add new region Asia Patific (Hong Kong) (Outvi V) 24069 * Reduce memory usage streaming files by reducing max stream upload size (Nick Craig-Wood) 24070 * Add `--s3-list-chunk` option for bucket listing (Thomas Kriechbaumer) 24071 * Force path style bucket access to off for AWS deprecation (Nick Craig-Wood) 24072 * Use AWS web identity role provider if available (Tennix) 24073 * Add StackPath Object Storage Support (Dave Koston) 24074 * Fix ExpiryWindow value (Aleksandar Jankovic) 24075 * Fix DisableChecksum condition (Aleksandar Janković) 24076 * Fix URL decoding of NextMarker (Nick Craig-Wood) 24077 * SFTP 24078 * Add `--sftp-skip-links` to skip symlinks and non regular files (Nick Craig-Wood) 24079 * Retry Creation of Connection (Sebastian Brandt) 24080 * Fix "failed to parse private key file: ssh: not an encrypted key" error (Nick Craig-Wood) 24081 * Open files for update write only to fix AWS SFTP interop (Nick Craig-Wood) 24082 * Swift 24083 * Reserve segments of dynamic large object when delete objects in container what was enabled versioning. (Nguyễn Hữu Luân) 24084 * Fix parsing of X-Object-Manifest (Nick Craig-Wood) 24085 * Update OVH API endpoint (unbelauscht) 24086 * WebDAV 24087 * Make nextcloud only upload SHA1 checksums (Nick Craig-Wood) 24088 * Fix case of "Bearer" in Authorization: header to agree with RFC (Nick Craig-Wood) 24089 * Add Referer header to fix problems with WAFs (Nick Craig-Wood) 24090 24091 ## v1.50.2 - 2019-11-19 24092 24093 * Bug Fixes 24094 * accounting: Fix memory leak on retries operations (Nick Craig-Wood) 24095 * Drive 24096 * Fix listing of the root directory with drive.files scope (Nick Craig-Wood) 24097 * Fix --drive-root-folder-id with team/shared drives (Nick Craig-Wood) 24098 24099 ## v1.50.1 - 2019-11-02 24100 24101 * Bug Fixes 24102 * hash: Fix accidentally changed hash names for `DropboxHash` and `CRC-32` (Nick Craig-Wood) 24103 * fshttp: Fix error reporting on tpslimit token bucket errors (Nick Craig-Wood) 24104 * fshttp: Don't print token bucket errors on context cancelled (Nick Craig-Wood) 24105 * Local 24106 * Fix listings of . on Windows (Nick Craig-Wood) 24107 * Onedrive 24108 * Fix DirMove/Move after Onedrive change (Xiaoxing Ye) 24109 24110 ## v1.50.0 - 2019-10-26 24111 24112 * New backends 24113 * [Citrix Sharefile](https://rclone.org/sharefile/) (Nick Craig-Wood) 24114 * [Chunker](https://rclone.org/chunker/) - an overlay backend to split files into smaller parts (Ivan Andreev) 24115 * [Mail.ru Cloud](https://rclone.org/mailru/) (Ivan Andreev) 24116 * New Features 24117 * encodings (Fabian Möller & Nick Craig-Wood) 24118 * All backends now use file name encoding to ensure any file name can be written to any backend. 24119 * See the [restricted file name docs](https://rclone.org/overview/#restricted-filenames) for more info and the [local backend docs](/local/#filenames). 24120 * Some file names may look different in rclone if you are using any control characters in names or [unicode FULLWIDTH symbols](https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)). 24121 * build 24122 * Update to use go1.13 for the build (Nick Craig-Wood) 24123 * Drop support for go1.9 (Nick Craig-Wood) 24124 * Build rclone with GitHub actions (Nick Craig-Wood) 24125 * Convert python scripts to python3 (Nick Craig-Wood) 24126 * Swap Azure/go-ansiterm for mattn/go-colorable (Nick Craig-Wood) 24127 * Dockerfile fixes (Matei David) 24128 * Add [plugin support](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-plugin) for backends and commands (Richard Patel) 24129 * config 24130 * Use alternating Red/Green in config to make more obvious (Nick Craig-Wood) 24131 * contrib 24132 * Add sample DLNA server Docker Compose manifest. (pataquets) 24133 * Add sample WebDAV server Docker Compose manifest. (pataquets) 24134 * copyurl 24135 * Add `--auto-filename` flag for using file name from URL in destination path (Denis) 24136 * serve dlna: 24137 * Many compatibility improvements (Dan Walters) 24138 * Support for external srt subtitles (Dan Walters) 24139 * rc 24140 * Added command core/quit (Saksham Khanna) 24141 * Bug Fixes 24142 * sync 24143 * Make `--update`/`-u` not transfer files that haven't changed (Nick Craig-Wood) 24144 * Free objects after they come out of the transfer pipe to save memory (Nick Craig-Wood) 24145 * Fix `--files-from without --no-traverse` doing a recursive scan (Nick Craig-Wood) 24146 * operations 24147 * Fix accounting for server side copies (Nick Craig-Wood) 24148 * Display 'All duplicates removed' only if dedupe successful (Sezal Agrawal) 24149 * Display 'Deleted X extra copies' only if dedupe successful (Sezal Agrawal) 24150 * accounting 24151 * Only allow up to 100 completed transfers in the accounting list to save memory (Nick Craig-Wood) 24152 * Cull the old time ranges when possible to save memory (Nick Craig-Wood) 24153 * Fix panic due to server-side copy fallback (Ivan Andreev) 24154 * Fix memory leak noticeable for transfers of large numbers of objects (Nick Craig-Wood) 24155 * Fix total duration calculation (Nick Craig-Wood) 24156 * cmd 24157 * Fix environment variables not setting command line flags (Nick Craig-Wood) 24158 * Make autocomplete compatible with bash's posix mode for macOS (Danil Semelenov) 24159 * Make `--progress` work in git bash on Windows (Nick Craig-Wood) 24160 * Fix 'compopt: command not found' on autocomplete on macOS (Danil Semelenov) 24161 * config 24162 * Fix setting of non top level flags from environment variables (Nick Craig-Wood) 24163 * Check config names more carefully and report errors (Nick Craig-Wood) 24164 * Remove error: can't use `--size-only` and `--ignore-size` together. (Nick Craig-Wood) 24165 * filter: Prevent mixing options when `--files-from` is in use (Michele Caci) 24166 * serve sftp: Fix crash on unsupported operations (eg Readlink) (Nick Craig-Wood) 24167 * Mount 24168 * Allow files of unknown size to be read properly (Nick Craig-Wood) 24169 * Skip tests on <= 2 CPUs to avoid lockup (Nick Craig-Wood) 24170 * Fix panic on File.Open (Nick Craig-Wood) 24171 * Fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD (Nick Craig-Wood) 24172 * Don't pass huge filenames (>4k) to FUSE as it can't cope (Nick Craig-Wood) 24173 * VFS 24174 * Add flag `--vfs-case-insensitive` for windows/macOS mounts (Ivan Andreev) 24175 * Make objects of unknown size readable through the VFS (Nick Craig-Wood) 24176 * Move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush() (Brett Dutro) 24177 * Stop empty dirs disappearing when renamed on bucket based remotes (Nick Craig-Wood) 24178 * Stop change notify polling clearing so much of the directory cache (Nick Craig-Wood) 24179 * Azure Blob 24180 * Disable logging to the Windows event log (Nick Craig-Wood) 24181 * B2 24182 * Remove `unverified:` prefix on sha1 to improve interop (eg with CyberDuck) (Nick Craig-Wood) 24183 * Box 24184 * Add options to get access token via JWT auth (David) 24185 * Drive 24186 * Disable HTTP/2 by default to work around INTERNAL_ERROR problems (Nick Craig-Wood) 24187 * Make sure that drive root ID is always canonical (Nick Craig-Wood) 24188 * Fix `--drive-shared-with-me` from the root with lsand `--fast-list` (Nick Craig-Wood) 24189 * Fix ChangeNotify polling for shared drives (Nick Craig-Wood) 24190 * Fix change notify polling when using appDataFolder (Nick Craig-Wood) 24191 * Dropbox 24192 * Make disallowed filenames errors not retry (Nick Craig-Wood) 24193 * Fix nil pointer exception on restricted files (Nick Craig-Wood) 24194 * Fichier 24195 * Fix accessing files > 2GB on 32 bit systems (Nick Craig-Wood) 24196 * FTP 24197 * Allow disabling EPSV mode (Jon Fautley) 24198 * HTTP 24199 * HEAD directory entries in parallel to speedup (Nick Craig-Wood) 24200 * Add `--http-no-head` to stop rclone doing HEAD in listings (Nick Craig-Wood) 24201 * Putio 24202 * Add ability to resume uploads (Cenk Alti) 24203 * S3 24204 * Fix signature v2_auth headers (Anthony Rusdi) 24205 * Fix encoding for control characters (Nick Craig-Wood) 24206 * Only ask for URL encoded directory listings if we need them on Ceph (Nick Craig-Wood) 24207 * Add option for multipart failure behaviour (Aleksandar Jankovic) 24208 * Support for multipart copy (庄天翼) 24209 * Fix nil pointer reference if no metadata returned for object (Nick Craig-Wood) 24210 * SFTP 24211 * Fix `--sftp-ask-password` trying to contact the ssh agent (Nick Craig-Wood) 24212 * Fix hashes of files with backslashes (Nick Craig-Wood) 24213 * Include more ciphers with `--sftp-use-insecure-cipher` (Carlos Ferreyra) 24214 * WebDAV 24215 * Parse and return Sharepoint error response (Henning Surmeier) 24216 24217 ## v1.49.5 - 2019-10-05 24218 24219 * Bug Fixes 24220 * Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing issues (Nick Craig-Wood) 24221 * Fix rpm packages by using master builds of nfpm (Nick Craig-Wood) 24222 * Fix macOS build after brew changes (Nick Craig-Wood) 24223 24224 ## v1.49.4 - 2019-09-29 24225 24226 * Bug Fixes 24227 * cmd/rcd: Address ZipSlip vulnerability (Richard Patel) 24228 * accounting: Fix file handle leak on errors (Nick Craig-Wood) 24229 * oauthutil: Fix security problem when running with two users on the same machine (Nick Craig-Wood) 24230 * FTP 24231 * Fix listing of an empty root returning: error dir not found (Nick Craig-Wood) 24232 * S3 24233 * Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier (Nick Craig-Wood) 24234 24235 ## v1.49.3 - 2019-09-15 24236 24237 * Bug Fixes 24238 * accounting 24239 * Fix total duration calculation (Aleksandar Jankovic) 24240 * Fix "file already closed" on transfer retries (Nick Craig-Wood) 24241 24242 ## v1.49.2 - 2019-09-08 24243 24244 * New Features 24245 * build: Add Docker workflow support (Alfonso Montero) 24246 * Bug Fixes 24247 * accounting: Fix locking in Transfer to avoid deadlock with `--progress` (Nick Craig-Wood) 24248 * docs: Fix template argument for mktemp in install.sh (Cnly) 24249 * operations: Fix `-u`/`--update` with google photos / files of unknown size (Nick Craig-Wood) 24250 * rc: Fix docs for config/create /update /password (Nick Craig-Wood) 24251 * Google Cloud Storage 24252 * Fix need for elevated permissions on SetModTime (Nick Craig-Wood) 24253 24254 ## v1.49.1 - 2019-08-28 24255 24256 * Bug Fixes 24257 * config: Fix generated passwords being stored as empty password (Nick Craig-Wood) 24258 * rcd: Added missing parameter for web-gui info logs. (Chaitanya) 24259 * Googlephotos 24260 * Fix crash on error response (Nick Craig-Wood) 24261 * Onedrive 24262 * Fix crash on error response (Nick Craig-Wood) 24263 24264 ## v1.49.0 - 2019-08-26 24265 24266 * New backends 24267 * [1fichier](https://rclone.org/fichier/) (Laura Hausmann) 24268 * [Google Photos](https://rclone.org/googlephotos/) (Nick Craig-Wood) 24269 * [Putio](https://rclone.org/putio/) (Cenk Alti) 24270 * [premiumize.me](https://rclone.org/premiumizeme/) (Nick Craig-Wood) 24271 * New Features 24272 * Experimental [web GUI](https://rclone.org/gui/) (Chaitanya Bankanhal) 24273 * Implement `--compare-dest` & `--copy-dest` (yparitcher) 24274 * Implement `--suffix` without `--backup-dir` for backup to current dir (yparitcher) 24275 * `config reconnect` to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood) 24276 * `config userinfo` to discover which user you are logged in as. (Nick Craig-Wood) 24277 * `config disconnect` to disconnect you (log out) from the backend. (Nick Craig-Wood) 24278 * Add `--use-json-log` for JSON logging (justinalin) 24279 * Add context propagation to rclone (Aleksandar Jankovic) 24280 * Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic) 24281 * Add Higher units for ETA (AbelThar) 24282 * Update rclone logos to new design (Andreas Chlupka) 24283 * hash: Add CRC-32 support (Cenk Alti) 24284 * help showbackend: Fixed advanced option category when there are no standard options (buengese) 24285 * ncdu: Display/Copy to Clipboard Current Path (Gary Kim) 24286 * operations: 24287 * Run hashing operations in parallel (Nick Craig-Wood) 24288 * Don't calculate checksums when using `--ignore-checksum` (Nick Craig-Wood) 24289 * Check transfer hashes when using `--size-only` mode (Nick Craig-Wood) 24290 * Disable multi thread copy for local to local copies (Nick Craig-Wood) 24291 * Debug successful hashes as well as failures (Nick Craig-Wood) 24292 * rc 24293 * Add ability to stop async jobs (Aleksandar Jankovic) 24294 * Return current settings if core/bwlimit called without parameters (Nick Craig-Wood) 24295 * Rclone-WebUI integration with rclone (Chaitanya Bankanhal) 24296 * Added command line parameter to control the cross origin resource sharing (CORS) in the rcd. (Security Improvement) (Chaitanya Bankanhal) 24297 * Add anchor tags to the docs so links are consistent (Nick Craig-Wood) 24298 * Remove _async key from input parameters after parsing so later operations won't get confused (buengese) 24299 * Add call to clear stats (Aleksandar Jankovic) 24300 * rcd 24301 * Auto-login for web-gui (Chaitanya Bankanhal) 24302 * Implement `--baseurl` for rcd and web-gui (Chaitanya Bankanhal) 24303 * serve dlna 24304 * Only select interfaces which can multicast for SSDP (Nick Craig-Wood) 24305 * Add more builtin mime types to cover standard audio/video (Nick Craig-Wood) 24306 * Fix missing mime types on Android causing missing videos (Nick Craig-Wood) 24307 * serve ftp 24308 * Refactor to bring into line with other serve commands (Nick Craig-Wood) 24309 * Implement `--auth-proxy` (Nick Craig-Wood) 24310 * serve http: Implement `--baseurl` (Nick Craig-Wood) 24311 * serve restic: Implement `--baseurl` (Nick Craig-Wood) 24312 * serve sftp 24313 * Implement auth proxy (Nick Craig-Wood) 24314 * Fix detection of whether server is authorized (Nick Craig-Wood) 24315 * serve webdav 24316 * Implement `--baseurl` (Nick Craig-Wood) 24317 * Support `--auth-proxy` (Nick Craig-Wood) 24318 * Bug Fixes 24319 * Make "bad record MAC" a retriable error (Nick Craig-Wood) 24320 * copyurl: Fix copying files that return HTTP errors (Nick Craig-Wood) 24321 * march: Fix checking sub-directories when using `--no-traverse` (buengese) 24322 * rc 24323 * Fix unmarshalable http.AuthFn in options and put in test for marshalability (Nick Craig-Wood) 24324 * Move job expire flags to rc to fix initialization problem (Nick Craig-Wood) 24325 * Fix `--loopback` with rc/list and others (Nick Craig-Wood) 24326 * rcat: Fix slowdown on systems with multiple hashes (Nick Craig-Wood) 24327 * rcd: Fix permissions problems on cache directory with web gui download (Nick Craig-Wood) 24328 * Mount 24329 * Default `--deamon-timout` to 15 minutes on macOS and FreeBSD (Nick Craig-Wood) 24330 * Update docs to show mounting from root OK for bucket based (Nick Craig-Wood) 24331 * Remove nonseekable flag from write files (Nick Craig-Wood) 24332 * VFS 24333 * Make write without cache more efficient (Nick Craig-Wood) 24334 * Fix `--vfs-cache-mode minimal` and `writes` ignoring cached files (Nick Craig-Wood) 24335 * Local 24336 * Add `--local-case-sensitive` and `--local-case-insensitive` (Nick Craig-Wood) 24337 * Avoid polluting page cache when uploading local files to remote backends (Michał Matczuk) 24338 * Don't calculate any hashes by default (Nick Craig-Wood) 24339 * Fadvise run syscall on a dedicated go routine (Michał Matczuk) 24340 * Azure Blob 24341 * Azure Storage Emulator support (Sandeep) 24342 * Updated config help details to remove connection string references (Sandeep) 24343 * Make all operations work from the root (Nick Craig-Wood) 24344 * B2 24345 * Implement link sharing (yparitcher) 24346 * Enable server side copy to copy between buckets (Nick Craig-Wood) 24347 * Make all operations work from the root (Nick Craig-Wood) 24348 * Drive 24349 * Fix server side copy of big files (Nick Craig-Wood) 24350 * Update API for teamdrive use (Nick Craig-Wood) 24351 * Add error for purge with `--drive-trashed-only` (ginvine) 24352 * Fichier 24353 * Make FolderID int and adjust related code (buengese) 24354 * Google Cloud Storage 24355 * Reduce oauth scope requested as suggested by Google (Nick Craig-Wood) 24356 * Make all operations work from the root (Nick Craig-Wood) 24357 * HTTP 24358 * Add `--http-headers` flag for setting arbitrary headers (Nick Craig-Wood) 24359 * Jottacloud 24360 * Use new api for retrieving internal username (buengese) 24361 * Refactor configuration and minor cleanup (buengese) 24362 * Koofr 24363 * Support setting modification times on Koofr backend. (jaKa) 24364 * Opendrive 24365 * Refactor to use existing lib/rest facilities for uploads (Nick Craig-Wood) 24366 * Qingstor 24367 * Upgrade to v3 SDK and fix listing loop (Nick Craig-Wood) 24368 * Make all operations work from the root (Nick Craig-Wood) 24369 * S3 24370 * Add INTELLIGENT_TIERING storage class (Matti Niemenmaa) 24371 * Make all operations work from the root (Nick Craig-Wood) 24372 * SFTP 24373 * Add missing interface check and fix About (Nick Craig-Wood) 24374 * Completely ignore all modtime checks if SetModTime=false (Jon Fautley) 24375 * Support md5/sha1 with rsync.net (Nick Craig-Wood) 24376 * Save the md5/sha1 command in use to the config file for efficiency (Nick Craig-Wood) 24377 * Opt-in support for diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 (Yi FU) 24378 * Swift 24379 * Use FixRangeOption to fix 0 length files via the VFS (Nick Craig-Wood) 24380 * Fix upload when using no_chunk to return the correct size (Nick Craig-Wood) 24381 * Make all operations work from the root (Nick Craig-Wood) 24382 * Fix segments leak during failed large file uploads. (nguyenhuuluan434) 24383 * WebDAV 24384 * Add `--webdav-bearer-token-command` (Nick Craig-Wood) 24385 * Refresh token when it expires with `--webdav-bearer-token-command` (Nick Craig-Wood) 24386 * Add docs for using bearer_token_command with oidc-agent (Paul Millar) 24387 24388 ## v1.48.0 - 2019-06-15 24389 24390 * New commands 24391 * serve sftp: Serve an rclone remote over SFTP (Nick Craig-Wood) 24392 * New Features 24393 * Multi threaded downloads to local storage (Nick Craig-Wood) 24394 * controlled with `--multi-thread-cutoff` and `--multi-thread-streams` 24395 * Use rclone.conf from rclone executable directory to enable portable use (albertony) 24396 * Allow sync of a file and a directory with the same name (forgems) 24397 * this is common on bucket based remotes, eg s3, gcs 24398 * Add `--ignore-case-sync` for forced case insensitivity (garry415) 24399 * Implement `--stats-one-line-date` and `--stats-one-line-date-format` (Peter Berbec) 24400 * Log an ERROR for all commands which exit with non-zero status (Nick Craig-Wood) 24401 * Use go-homedir to read the home directory more reliably (Nick Craig-Wood) 24402 * Enable creating encrypted config through external script invocation (Wojciech Smigielski) 24403 * build: Drop support for go1.8 (Nick Craig-Wood) 24404 * config: Make config create/update encrypt passwords where necessary (Nick Craig-Wood) 24405 * copyurl: Honor `--no-check-certificate` (Stefan Breunig) 24406 * install: Linux skip man pages if no mandb (didil) 24407 * lsf: Support showing the Tier of the object (Nick Craig-Wood) 24408 * lsjson 24409 * Added EncryptedPath to output (calisro) 24410 * Support showing the Tier of the object (Nick Craig-Wood) 24411 * Add IsBucket field for bucket based remote listing of the root (Nick Craig-Wood) 24412 * rc 24413 * Add `--loopback` flag to run commands directly without a server (Nick Craig-Wood) 24414 * Add operations/fsinfo: Return information about the remote (Nick Craig-Wood) 24415 * Skip auth for OPTIONS request (Nick Craig-Wood) 24416 * cmd/providers: Add DefaultStr, ValueStr and Type fields (Nick Craig-Wood) 24417 * jobs: Make job expiry timeouts configurable (Aleksandar Jankovic) 24418 * serve dlna reworked and improved (Dan Walters) 24419 * serve ftp: add `--ftp-public-ip` flag to specify public IP (calistri) 24420 * serve restic: Add support for `--private-repos` in `serve restic` (Florian Apolloner) 24421 * serve webdav: Combine serve webdav and serve http (Gary Kim) 24422 * size: Ignore negative sizes when calculating total (Garry McNulty) 24423 * Bug Fixes 24424 * Make move and copy individual files obey `--backup-dir` (Nick Craig-Wood) 24425 * If `--ignore-checksum` is in effect, don't calculate checksum (Nick Craig-Wood) 24426 * moveto: Fix case-insensitive same remote move (Gary Kim) 24427 * rc: Fix serving bucket based objects with `--rc-serve` (Nick Craig-Wood) 24428 * serve webdav: Fix serveDir not being updated with changes from webdav (Gary Kim) 24429 * Mount 24430 * Fix poll interval documentation (Animosity022) 24431 * VFS 24432 * Make WriteAt for non cached files work with non-sequential writes (Nick Craig-Wood) 24433 * Local 24434 * Only calculate the required hashes for big speedup (Nick Craig-Wood) 24435 * Log errors when listing instead of returning an error (Nick Craig-Wood) 24436 * Fix preallocate warning on Linux with ZFS (Nick Craig-Wood) 24437 * Crypt 24438 * Make rclone dedupe work through crypt (Nick Craig-Wood) 24439 * Fix wrapping of ChangeNotify to decrypt directories properly (Nick Craig-Wood) 24440 * Support PublicLink (rclone link) of underlying backend (Nick Craig-Wood) 24441 * Implement Optional methods SetTier, GetTier (Nick Craig-Wood) 24442 * B2 24443 * Implement server side copy (Nick Craig-Wood) 24444 * Implement SetModTime (Nick Craig-Wood) 24445 * Drive 24446 * Fix move and copy from TeamDrive to GDrive (Fionera) 24447 * Add notes that cleanup works in the background on drive (Nick Craig-Wood) 24448 * Add `--drive-server-side-across-configs` to default back to old server side copy semantics by default (Nick Craig-Wood) 24449 * Add `--drive-size-as-quota` to show storage quota usage for file size (Garry McNulty) 24450 * FTP 24451 * Add FTP List timeout (Jeff Quinn) 24452 * Add FTP over TLS support (Gary Kim) 24453 * Add `--ftp-no-check-certificate` option for FTPS (Gary Kim) 24454 * Google Cloud Storage 24455 * Fix upload errors when uploading pre 1970 files (Nick Craig-Wood) 24456 * Jottacloud 24457 * Add support for selecting device and mountpoint. (buengese) 24458 * Mega 24459 * Add cleanup support (Gary Kim) 24460 * Onedrive 24461 * More accurately check if root is found (Cnly) 24462 * S3 24463 * Support S3 Accelerated endpoints with `--s3-use-accelerate-endpoint` (Nick Craig-Wood) 24464 * Add config info for Wasabi's EU Central endpoint (Robert Marko) 24465 * Make SetModTime work for GLACIER while syncing (Philip Harvey) 24466 * SFTP 24467 * Add About support (Gary Kim) 24468 * Fix about parsing of `df` results so it can cope with -ve results (Nick Craig-Wood) 24469 * Send custom client version and debug server version (Nick Craig-Wood) 24470 * WebDAV 24471 * Retry on 423 Locked errors (Nick Craig-Wood) 24472 24473 ## v1.47.0 - 2019-04-13 24474 24475 * New backends 24476 * Backend for Koofr cloud storage service. (jaKa) 24477 * New Features 24478 * Resume downloads if the reader fails in copy (Nick Craig-Wood) 24479 * this means rclone will restart transfers if the source has an error 24480 * this is most useful for downloads or cloud to cloud copies 24481 * Use `--fast-list` for listing operations where it won't use more memory (Nick Craig-Wood) 24482 * this should speed up the following operations on remotes which support `ListR` 24483 * `dedupe`, `serve restic` `lsf`, `ls`, `lsl`, `lsjson`, `lsd`, `md5sum`, `sha1sum`, `hashsum`, `size`, `delete`, `cat`, `settier` 24484 * use `--disable ListR` to get old behaviour if required 24485 * Make `--files-from` traverse the destination unless `--no-traverse` is set (Nick Craig-Wood) 24486 * this fixes `--files-from` with Google drive and excessive API use in general. 24487 * Make server side copy account bytes and obey `--max-transfer` (Nick Craig-Wood) 24488 * Add `--create-empty-src-dirs` flag and default to not creating empty dirs (ishuah) 24489 * Add client side TLS/SSL flags `--ca-cert`/`--client-cert`/`--client-key` (Nick Craig-Wood) 24490 * Implement `--suffix-keep-extension` for use with `--suffix` (Nick Craig-Wood) 24491 * build: 24492 * Switch to semvar compliant version tags to be go modules compliant (Nick Craig-Wood) 24493 * Update to use go1.12.x for the build (Nick Craig-Wood) 24494 * serve dlna: Add connection manager service description to improve compatibility (Dan Walters) 24495 * lsf: Add 'e' format to show encrypted names and 'o' for original IDs (Nick Craig-Wood) 24496 * lsjson: Added `--files-only` and `--dirs-only` flags (calistri) 24497 * rc: Implement operations/publiclink the equivalent of `rclone link` (Nick Craig-Wood) 24498 * Bug Fixes 24499 * accounting: Fix total ETA when `--stats-unit bits` is in effect (Nick Craig-Wood) 24500 * Bash TAB completion 24501 * Use private custom func to fix clash between rclone and kubectl (Nick Craig-Wood) 24502 * Fix for remotes with underscores in their names (Six) 24503 * Fix completion of remotes (Florian Gamböck) 24504 * Fix autocompletion of remote paths with spaces (Danil Semelenov) 24505 * serve dlna: Fix root XML service descriptor (Dan Walters) 24506 * ncdu: Fix display corruption with Chinese characters (Nick Craig-Wood) 24507 * Add SIGTERM to signals which run the exit handlers on unix (Nick Craig-Wood) 24508 * rc: Reload filter when the options are set via the rc (Nick Craig-Wood) 24509 * VFS / Mount 24510 * Fix FreeBSD: Ignore Truncate if called with no readers and already the correct size (Nick Craig-Wood) 24511 * Read directory and check for a file before mkdir (Nick Craig-Wood) 24512 * Shorten the locking window for vfs/refresh (Nick Craig-Wood) 24513 * Azure Blob 24514 * Enable MD5 checksums when uploading files bigger than the "Cutoff" (Dr.Rx) 24515 * Fix SAS URL support (Nick Craig-Wood) 24516 * B2 24517 * Allow manual configuration of backblaze downloadUrl (Vince) 24518 * Ignore already_hidden error on remove (Nick Craig-Wood) 24519 * Ignore malformed `src_last_modified_millis` (Nick Craig-Wood) 24520 * Drive 24521 * Add `--skip-checksum-gphotos` to ignore incorrect checksums on Google Photos (Nick Craig-Wood) 24522 * Allow server side move/copy between different remotes. (Fionera) 24523 * Add docs on team drives and `--fast-list` eventual consistency (Nestar47) 24524 * Fix imports of text files (Nick Craig-Wood) 24525 * Fix range requests on 0 length files (Nick Craig-Wood) 24526 * Fix creation of duplicates with server side copy (Nick Craig-Wood) 24527 * Dropbox 24528 * Retry blank errors to fix long listings (Nick Craig-Wood) 24529 * FTP 24530 * Add `--ftp-concurrency` to limit maximum number of connections (Nick Craig-Wood) 24531 * Google Cloud Storage 24532 * Fall back to default application credentials (marcintustin) 24533 * Allow bucket policy only buckets (Nick Craig-Wood) 24534 * HTTP 24535 * Add `--http-no-slash` for websites with directories with no slashes (Nick Craig-Wood) 24536 * Remove duplicates from listings (Nick Craig-Wood) 24537 * Fix socket leak on 404 errors (Nick Craig-Wood) 24538 * Jottacloud 24539 * Fix token refresh (Sebastian Bünger) 24540 * Add device registration (Oliver Heyme) 24541 * Onedrive 24542 * Implement graceful cancel of multipart uploads if rclone is interrupted (Cnly) 24543 * Always add trailing colon to path when addressing items, (Cnly) 24544 * Return errors instead of panic for invalid uploads (Fabian Möller) 24545 * S3 24546 * Add support for "Glacier Deep Archive" storage class (Manu) 24547 * Update Dreamhost endpoint (Nick Craig-Wood) 24548 * Note incompatibility with CEPH Jewel (Nick Craig-Wood) 24549 * SFTP 24550 * Allow custom ssh client config (Alexandru Bumbacea) 24551 * Swift 24552 * Obey Retry-After to enable OVH restore from cold storage (Nick Craig-Wood) 24553 * Work around token expiry on CEPH (Nick Craig-Wood) 24554 * WebDAV 24555 * Allow IsCollection property to be integer or boolean (Nick Craig-Wood) 24556 * Fix race when creating directories (Nick Craig-Wood) 24557 * Fix About/df when reading the available/total returns 0 (Nick Craig-Wood) 24558 24559 ## v1.46 - 2019-02-09 24560 24561 * New backends 24562 * Support Alibaba Cloud (Aliyun) OSS via the s3 backend (Nick Craig-Wood) 24563 * New commands 24564 * serve dlna: serves a remove via DLNA for the local network (nicolov) 24565 * New Features 24566 * copy, move: Restore deprecated `--no-traverse` flag (Nick Craig-Wood) 24567 * This is useful for when transferring a small number of files into a large destination 24568 * genautocomplete: Add remote path completion for bash completion (Christopher Peterson & Danil Semelenov) 24569 * Buffer memory handling reworked to return memory to the OS better (Nick Craig-Wood) 24570 * Buffer recycling library to replace sync.Pool 24571 * Optionally use memory mapped memory for better memory shrinking 24572 * Enable with `--use-mmap` if having memory problems - not default yet 24573 * Parallelise reading of files specified by `--files-from` (Nick Craig-Wood) 24574 * check: Add stats showing total files matched. (Dario Guzik) 24575 * Allow rename/delete open files under Windows (Nick Craig-Wood) 24576 * lsjson: Use exactly the correct number of decimal places in the seconds (Nick Craig-Wood) 24577 * Add cookie support with cmdline switch `--use-cookies` for all HTTP based remotes (qip) 24578 * Warn if `--checksum` is set but there are no hashes available (Nick Craig-Wood) 24579 * Rework rate limiting (pacer) to be more accurate and allow bursting (Nick Craig-Wood) 24580 * Improve error reporting for too many/few arguments in commands (Nick Craig-Wood) 24581 * listremotes: Remove `-l` short flag as it conflicts with the new global flag (weetmuts) 24582 * Make http serving with auth generate INFO messages on auth fail (Nick Craig-Wood) 24583 * Bug Fixes 24584 * Fix layout of stats (Nick Craig-Wood) 24585 * Fix `--progress` crash under Windows Jenkins (Nick Craig-Wood) 24586 * Fix transfer of google/onedrive docs by calling Rcat in Copy when size is -1 (Cnly) 24587 * copyurl: Fix checking of `--dry-run` (Denis Skovpen) 24588 * Mount 24589 * Check that mountpoint and local directory to mount don't overlap (Nick Craig-Wood) 24590 * Fix mount size under 32 bit Windows (Nick Craig-Wood) 24591 * VFS 24592 * Implement renaming of directories for backends without DirMove (Nick Craig-Wood) 24593 * now all backends except b2 support renaming directories 24594 * Implement `--vfs-cache-max-size` to limit the total size of the cache (Nick Craig-Wood) 24595 * Add `--dir-perms` and `--file-perms` flags to set default permissions (Nick Craig-Wood) 24596 * Fix deadlock on concurrent operations on a directory (Nick Craig-Wood) 24597 * Fix deadlock between RWFileHandle.close and File.Remove (Nick Craig-Wood) 24598 * Fix renaming/deleting open files with cache mode "writes" under Windows (Nick Craig-Wood) 24599 * Fix panic on rename with `--dry-run` set (Nick Craig-Wood) 24600 * Fix vfs/refresh with recurse=true needing the `--fast-list` flag 24601 * Local 24602 * Add support for `-l`/`--links` (symbolic link translation) (yair@unicorn) 24603 * this works by showing links as `link.rclonelink` - see local backend docs for more info 24604 * this errors if used with `-L`/`--copy-links` 24605 * Fix renaming/deleting open files on Windows (Nick Craig-Wood) 24606 * Crypt 24607 * Check for maximum length before decrypting filename to fix panic (Garry McNulty) 24608 * Azure Blob 24609 * Allow building azureblob backend on *BSD (themylogin) 24610 * Use the rclone HTTP client to support `--dump headers`, `--tpslimit` etc (Nick Craig-Wood) 24611 * Use the s3 pacer for 0 delay in non error conditions (Nick Craig-Wood) 24612 * Ignore directory markers (Nick Craig-Wood) 24613 * Stop Mkdir attempting to create existing containers (Nick Craig-Wood) 24614 * B2 24615 * cleanup: will remove unfinished large files >24hrs old (Garry McNulty) 24616 * For a bucket limited application key check the bucket name (Nick Craig-Wood) 24617 * before this, rclone would use the authorised bucket regardless of what you put on the command line 24618 * Added `--b2-disable-checksum` flag (Wojciech Smigielski) 24619 * this enables large files to be uploaded without a SHA-1 hash for speed reasons 24620 * Drive 24621 * Set default pacer to 100ms for 10 tps (Nick Craig-Wood) 24622 * This fits the Google defaults much better and reduces the 403 errors massively 24623 * Add `--drive-pacer-min-sleep` and `--drive-pacer-burst` to control the pacer 24624 * Improve ChangeNotify support for items with multiple parents (Fabian Möller) 24625 * Fix ListR for items with multiple parents - this fixes oddities with `vfs/refresh` (Fabian Möller) 24626 * Fix using `--drive-impersonate` and appfolders (Nick Craig-Wood) 24627 * Fix google docs in rclone mount for some (not all) applications (Nick Craig-Wood) 24628 * Dropbox 24629 * Retry-After support for Dropbox backend (Mathieu Carbou) 24630 * FTP 24631 * Wait for 60 seconds for a connection to Close then declare it dead (Nick Craig-Wood) 24632 * helps with indefinite hangs on some FTP servers 24633 * Google Cloud Storage 24634 * Update google cloud storage endpoints (weetmuts) 24635 * HTTP 24636 * Add an example with username and password which is supported but wasn't documented (Nick Craig-Wood) 24637 * Fix backend with `--files-from` and non-existent files (Nick Craig-Wood) 24638 * Hubic 24639 * Make error message more informative if authentication fails (Nick Craig-Wood) 24640 * Jottacloud 24641 * Resume and deduplication support (Oliver Heyme) 24642 * Use token auth for all API requests Don't store password anymore (Sebastian Bünger) 24643 * Add support for 2-factor authentication (Sebastian Bünger) 24644 * Mega 24645 * Implement v2 account login which fixes logins for newer Mega accounts (Nick Craig-Wood) 24646 * Return error if an unknown length file is attempted to be uploaded (Nick Craig-Wood) 24647 * Add new error codes for better error reporting (Nick Craig-Wood) 24648 * Onedrive 24649 * Fix broken support for "shared with me" folders (Alex Chen) 24650 * Fix root ID not normalised (Cnly) 24651 * Return err instead of panic on unknown-sized uploads (Cnly) 24652 * Qingstor 24653 * Fix go routine leak on multipart upload errors (Nick Craig-Wood) 24654 * Add upload chunk size/concurrency/cutoff control (Nick Craig-Wood) 24655 * Default `--qingstor-upload-concurrency` to 1 to work around bug (Nick Craig-Wood) 24656 * S3 24657 * Implement `--s3-upload-cutoff` for single part uploads below this (Nick Craig-Wood) 24658 * Change `--s3-upload-concurrency` default to 4 to increase performance (Nick Craig-Wood) 24659 * Add `--s3-bucket-acl` to control bucket ACL (Nick Craig-Wood) 24660 * Auto detect region for buckets on operation failure (Nick Craig-Wood) 24661 * Add GLACIER storage class (William Cocker) 24662 * Add Scaleway to s3 documentation (Rémy Léone) 24663 * Add AWS endpoint eu-north-1 (weetmuts) 24664 * SFTP 24665 * Add support for PEM encrypted private keys (Fabian Möller) 24666 * Add option to force the usage of an ssh-agent (Fabian Möller) 24667 * Perform environment variable expansion on key-file (Fabian Möller) 24668 * Fix rmdir on Windows based servers (eg CrushFTP) (Nick Craig-Wood) 24669 * Fix rmdir deleting directory contents on some SFTP servers (Nick Craig-Wood) 24670 * Fix error on dangling symlinks (Nick Craig-Wood) 24671 * Swift 24672 * Add `--swift-no-chunk` to disable segmented uploads in rcat/mount (Nick Craig-Wood) 24673 * Introduce application credential auth support (kayrus) 24674 * Fix memory usage by slimming Object (Nick Craig-Wood) 24675 * Fix extra requests on upload (Nick Craig-Wood) 24676 * Fix reauth on big files (Nick Craig-Wood) 24677 * Union 24678 * Fix poll-interval not working (Nick Craig-Wood) 24679 * WebDAV 24680 * Support About which means rclone mount will show the correct disk size (Nick Craig-Wood) 24681 * Support MD5 and SHA1 hashes with Owncloud and Nextcloud (Nick Craig-Wood) 24682 * Fail soft on time parsing errors (Nick Craig-Wood) 24683 * Fix infinite loop on failed directory creation (Nick Craig-Wood) 24684 * Fix identification of directories for Bitrix Site Manager (Nick Craig-Wood) 24685 * Fix upload of 0 length files on some servers (Nick Craig-Wood) 24686 * Fix if MKCOL fails with 423 Locked assume the directory exists (Nick Craig-Wood) 24687 24688 ## v1.45 - 2018-11-24 24689 24690 * New backends 24691 * The Yandex backend was re-written - see below for details (Sebastian Bünger) 24692 * New commands 24693 * rcd: New command just to serve the remote control API (Nick Craig-Wood) 24694 * New Features 24695 * The remote control API (rc) was greatly expanded to allow full control over rclone (Nick Craig-Wood) 24696 * sensitive operations require authorization or the `--rc-no-auth` flag 24697 * config/* operations to configure rclone 24698 * options/* for reading/setting command line flags 24699 * operations/* for all low level operations, eg copy file, list directory 24700 * sync/* for sync, copy and move 24701 * `--rc-files` flag to serve files on the rc http server 24702 * this is for building web native GUIs for rclone 24703 * Optionally serving objects on the rc http server 24704 * Ensure rclone fails to start up if the `--rc` port is in use already 24705 * See [the rc docs](https://rclone.org/rc/) for more info 24706 * sync/copy/move 24707 * Make `--files-from` only read the objects specified and don't scan directories (Nick Craig-Wood) 24708 * This is a huge speed improvement for destinations with lots of files 24709 * filter: Add `--ignore-case` flag (Nick Craig-Wood) 24710 * ncdu: Add remove function ('d' key) (Henning Surmeier) 24711 * rc command 24712 * Add `--json` flag for structured JSON input (Nick Craig-Wood) 24713 * Add `--user` and `--pass` flags and interpret `--rc-user`, `--rc-pass`, `--rc-addr` (Nick Craig-Wood) 24714 * build 24715 * Require go1.8 or later for compilation (Nick Craig-Wood) 24716 * Enable softfloat on MIPS arch (Scott Edlund) 24717 * Integration test framework revamped with a better report and better retries (Nick Craig-Wood) 24718 * Bug Fixes 24719 * cmd: Make `--progress` update the stats correctly at the end (Nick Craig-Wood) 24720 * config: Create config directory on save if it is missing (Nick Craig-Wood) 24721 * dedupe: Check for existing filename before renaming a dupe file (ssaqua) 24722 * move: Don't create directories with `--dry-run` (Nick Craig-Wood) 24723 * operations: Fix Purge and Rmdirs when dir is not the root (Nick Craig-Wood) 24724 * serve http/webdav/restic: Ensure rclone exits if the port is in use (Nick Craig-Wood) 24725 * Mount 24726 * Make `--volname` work for Windows and macOS (Nick Craig-Wood) 24727 * Azure Blob 24728 * Avoid context deadline exceeded error by setting a large TryTimeout value (brused27) 24729 * Fix erroneous Rmdir error "directory not empty" (Nick Craig-Wood) 24730 * Wait for up to 60s to create a just deleted container (Nick Craig-Wood) 24731 * Dropbox 24732 * Add dropbox impersonate support (Jake Coggiano) 24733 * Jottacloud 24734 * Fix bug in `--fast-list` handing of empty folders (albertony) 24735 * Opendrive 24736 * Fix transfer of files with `+` and `&` in (Nick Craig-Wood) 24737 * Fix retries of upload chunks (Nick Craig-Wood) 24738 * S3 24739 * Set ACL for server side copies to that provided by the user (Nick Craig-Wood) 24740 * Fix role_arn, credential_source, ... (Erik Swanson) 24741 * Add config info for Wasabi's US-West endpoint (Henry Ptasinski) 24742 * SFTP 24743 * Ensure file hash checking is really disabled (Jon Fautley) 24744 * Swift 24745 * Add pacer for retries to make swift more reliable (Nick Craig-Wood) 24746 * WebDAV 24747 * Add Content-Type to PUT requests (Nick Craig-Wood) 24748 * Fix config parsing so `--webdav-user` and `--webdav-pass` flags work (Nick Craig-Wood) 24749 * Add RFC3339 date format (Ralf Hemberger) 24750 * Yandex 24751 * The yandex backend was re-written (Sebastian Bünger) 24752 * This implements low level retries (Sebastian Bünger) 24753 * Copy, Move, DirMove, PublicLink and About optional interfaces (Sebastian Bünger) 24754 * Improved general error handling (Sebastian Bünger) 24755 * Removed ListR for now due to inconsistent behaviour (Sebastian Bünger) 24756 24757 ## v1.44 - 2018-10-15 24758 24759 * New commands 24760 * serve ftp: Add ftp server (Antoine GIRARD) 24761 * settier: perform storage tier changes on supported remotes (sandeepkru) 24762 * New Features 24763 * Reworked command line help 24764 * Make default help less verbose (Nick Craig-Wood) 24765 * Split flags up into global and backend flags (Nick Craig-Wood) 24766 * Implement specialised help for flags and backends (Nick Craig-Wood) 24767 * Show URL of backend help page when starting config (Nick Craig-Wood) 24768 * stats: Long names now split in center (Joanna Marek) 24769 * Add `--log-format` flag for more control over log output (dcpu) 24770 * rc: Add support for OPTIONS and basic CORS (frenos) 24771 * stats: show FatalErrors and NoRetryErrors in stats (Cédric Connes) 24772 * Bug Fixes 24773 * Fix -P not ending with a new line (Nick Craig-Wood) 24774 * config: don't create default config dir when user supplies `--config` (albertony) 24775 * Don't print non-ASCII characters with `--progress` on windows (Nick Craig-Wood) 24776 * Correct logs for excluded items (ssaqua) 24777 * Mount 24778 * Remove EXPERIMENTAL tags (Nick Craig-Wood) 24779 * VFS 24780 * Fix race condition detected by serve ftp tests (Nick Craig-Wood) 24781 * Add vfs/poll-interval rc command (Fabian Möller) 24782 * Enable rename for nearly all remotes using server side Move or Copy (Nick Craig-Wood) 24783 * Reduce directory cache cleared by poll-interval (Fabian Möller) 24784 * Remove EXPERIMENTAL tags (Nick Craig-Wood) 24785 * Local 24786 * Skip bad symlinks in dir listing with -L enabled (Cédric Connes) 24787 * Preallocate files on Windows to reduce fragmentation (Nick Craig-Wood) 24788 * Preallocate files on linux with fallocate(2) (Nick Craig-Wood) 24789 * Cache 24790 * Add cache/fetch rc function (Fabian Möller) 24791 * Fix worker scale down (Fabian Möller) 24792 * Improve performance by not sending info requests for cached chunks (dcpu) 24793 * Fix error return value of cache/fetch rc method (Fabian Möller) 24794 * Documentation fix for cache-chunk-total-size (Anagh Kumar Baranwal) 24795 * Preserve leading / in wrapped remote path (Fabian Möller) 24796 * Add plex_insecure option to skip certificate validation (Fabian Möller) 24797 * Remove entries that no longer exist in the source (dcpu) 24798 * Crypt 24799 * Preserve leading / in wrapped remote path (Fabian Möller) 24800 * Alias 24801 * Fix handling of Windows network paths (Nick Craig-Wood) 24802 * Azure Blob 24803 * Add `--azureblob-list-chunk` parameter (Santiago Rodríguez) 24804 * Implemented settier command support on azureblob remote. (sandeepkru) 24805 * Work around SDK bug which causes errors for chunk-sized files (Nick Craig-Wood) 24806 * Box 24807 * Implement link sharing. (Sebastian Bünger) 24808 * Drive 24809 * Add `--drive-import-formats` - google docs can now be imported (Fabian Möller) 24810 * Rewrite mime type and extension handling (Fabian Möller) 24811 * Add document links (Fabian Möller) 24812 * Add support for multipart document extensions (Fabian Möller) 24813 * Add support for apps-script to json export (Fabian Möller) 24814 * Fix escaped chars in documents during list (Fabian Möller) 24815 * Add `--drive-v2-download-min-size` a workaround for slow downloads (Fabian Möller) 24816 * Improve directory notifications in ChangeNotify (Fabian Möller) 24817 * When listing team drives in config, continue on failure (Nick Craig-Wood) 24818 * FTP 24819 * Add a small pause after failed upload before deleting file (Nick Craig-Wood) 24820 * Google Cloud Storage 24821 * Fix service_account_file being ignored (Fabian Möller) 24822 * Jottacloud 24823 * Minor improvement in quota info (omit if unlimited) (albertony) 24824 * Add `--fast-list` support (albertony) 24825 * Add permanent delete support: `--jottacloud-hard-delete` (albertony) 24826 * Add link sharing support (albertony) 24827 * Fix handling of reserved characters. (Sebastian Bünger) 24828 * Fix socket leak on Object.Remove (Nick Craig-Wood) 24829 * Onedrive 24830 * Rework to support Microsoft Graph (Cnly) 24831 * **NB** this will require re-authenticating the remote 24832 * Removed upload cutoff and always do session uploads (Oliver Heyme) 24833 * Use single-part upload for empty files (Cnly) 24834 * Fix new fields not saved when editing old config (Alex Chen) 24835 * Fix sometimes special chars in filenames not replaced (Alex Chen) 24836 * Ignore OneNote files by default (Alex Chen) 24837 * Add link sharing support (jackyzy823) 24838 * S3 24839 * Use custom pacer, to retry operations when reasonable (Craig Miskell) 24840 * Use configured server-side-encryption and storage class options when calling CopyObject() (Paul Kohout) 24841 * Make `--s3-v2-auth` flag (Nick Craig-Wood) 24842 * Fix v2 auth on files with spaces (Nick Craig-Wood) 24843 * Union 24844 * Implement union backend which reads from multiple backends (Felix Brucker) 24845 * Implement optional interfaces (Move, DirMove, Copy etc) (Nick Craig-Wood) 24846 * Fix ChangeNotify to support multiple remotes (Fabian Möller) 24847 * Fix `--backup-dir` on union backend (Nick Craig-Wood) 24848 * WebDAV 24849 * Add another time format (Nick Craig-Wood) 24850 * Add a small pause after failed upload before deleting file (Nick Craig-Wood) 24851 * Add workaround for missing mtime (buergi) 24852 * Sharepoint: Renew cookies after 12hrs (Henning Surmeier) 24853 * Yandex 24854 * Remove redundant nil checks (teresy) 24855 24856 ## v1.43.1 - 2018-09-07 24857 24858 Point release to fix hubic and azureblob backends. 24859 24860 * Bug Fixes 24861 * ncdu: Return error instead of log.Fatal in Show (Fabian Möller) 24862 * cmd: Fix crash with `--progress` and `--stats 0` (Nick Craig-Wood) 24863 * docs: Tidy website display (Anagh Kumar Baranwal) 24864 * Azure Blob: 24865 * Fix multi-part uploads. (sandeepkru) 24866 * Hubic 24867 * Fix uploads (Nick Craig-Wood) 24868 * Retry auth fetching if it fails to make hubic more reliable (Nick Craig-Wood) 24869 24870 ## v1.43 - 2018-09-01 24871 24872 * New backends 24873 * Jottacloud (Sebastian Bünger) 24874 * New commands 24875 * copyurl: copies a URL to a remote (Denis) 24876 * New Features 24877 * Reworked config for backends (Nick Craig-Wood) 24878 * All backend config can now be supplied by command line, env var or config file 24879 * Advanced section in the config wizard for the optional items 24880 * A large step towards rclone backends being usable in other go software 24881 * Allow on the fly remotes with :backend: syntax 24882 * Stats revamp 24883 * Add `--progress`/`-P` flag to show interactive progress (Nick Craig-Wood) 24884 * Show the total progress of the sync in the stats (Nick Craig-Wood) 24885 * Add `--stats-one-line` flag for single line stats (Nick Craig-Wood) 24886 * Added weekday schedule into `--bwlimit` (Mateusz) 24887 * lsjson: Add option to show the original object IDs (Fabian Möller) 24888 * serve webdav: Make Content-Type without reading the file and add `--etag-hash` (Nick Craig-Wood) 24889 * build 24890 * Build macOS with native compiler (Nick Craig-Wood) 24891 * Update to use go1.11 for the build (Nick Craig-Wood) 24892 * rc 24893 * Added core/stats to return the stats (reddi1) 24894 * `version --check`: Prints the current release and beta versions (Nick Craig-Wood) 24895 * Bug Fixes 24896 * accounting 24897 * Fix time to completion estimates (Nick Craig-Wood) 24898 * Fix moving average speed for file stats (Nick Craig-Wood) 24899 * config: Fix error reading password from piped input (Nick Craig-Wood) 24900 * move: Fix `--delete-empty-src-dirs` flag to delete all empty dirs on move (ishuah) 24901 * Mount 24902 * Implement `--daemon-timeout` flag for OSXFUSE (Nick Craig-Wood) 24903 * Fix mount `--daemon` not working with encrypted config (Alex Chen) 24904 * Clip the number of blocks to 2^32-1 on macOS - fixes borg backup (Nick Craig-Wood) 24905 * VFS 24906 * Enable vfs-read-chunk-size by default (Fabian Möller) 24907 * Add the vfs/refresh rc command (Fabian Möller) 24908 * Add non recursive mode to vfs/refresh rc command (Fabian Möller) 24909 * Try to seek buffer on read only files (Fabian Möller) 24910 * Local 24911 * Fix crash when deprecated `--local-no-unicode-normalization` is supplied (Nick Craig-Wood) 24912 * Fix mkdir error when trying to copy files to the root of a drive on windows (Nick Craig-Wood) 24913 * Cache 24914 * Fix nil pointer deref when using lsjson on cached directory (Nick Craig-Wood) 24915 * Fix nil pointer deref for occasional crash on playback (Nick Craig-Wood) 24916 * Crypt 24917 * Fix accounting when checking hashes on upload (Nick Craig-Wood) 24918 * Amazon Cloud Drive 24919 * Make very clear in the docs that rclone has no ACD keys (Nick Craig-Wood) 24920 * Azure Blob 24921 * Add connection string and SAS URL auth (Nick Craig-Wood) 24922 * List the container to see if it exists (Nick Craig-Wood) 24923 * Port new Azure Blob Storage SDK (sandeepkru) 24924 * Added blob tier, tier between Hot, Cool and Archive. (sandeepkru) 24925 * Remove leading / from paths (Nick Craig-Wood) 24926 * B2 24927 * Support Application Keys (Nick Craig-Wood) 24928 * Remove leading / from paths (Nick Craig-Wood) 24929 * Box 24930 * Fix upload of > 2GB files on 32 bit platforms (Nick Craig-Wood) 24931 * Make `--box-commit-retries` flag defaulting to 100 to fix large uploads (Nick Craig-Wood) 24932 * Drive 24933 * Add `--drive-keep-revision-forever` flag (lewapm) 24934 * Handle gdocs when filtering file names in list (Fabian Möller) 24935 * Support using `--fast-list` for large speedups (Fabian Möller) 24936 * FTP 24937 * Fix Put mkParentDir failed: 521 for BunnyCDN (Nick Craig-Wood) 24938 * Google Cloud Storage 24939 * Fix index out of range error with `--fast-list` (Nick Craig-Wood) 24940 * Jottacloud 24941 * Fix MD5 error check (Oliver Heyme) 24942 * Handle empty time values (Martin Polden) 24943 * Calculate missing MD5s (Oliver Heyme) 24944 * Docs, fixes and tests for MD5 calculation (Nick Craig-Wood) 24945 * Add optional MimeTyper interface. (Sebastian Bünger) 24946 * Implement optional About interface (for `df` support). (Sebastian Bünger) 24947 * Mega 24948 * Wait for events instead of arbitrary sleeping (Nick Craig-Wood) 24949 * Add `--mega-hard-delete` flag (Nick Craig-Wood) 24950 * Fix failed logins with upper case chars in email (Nick Craig-Wood) 24951 * Onedrive 24952 * Shared folder support (Yoni Jah) 24953 * Implement DirMove (Cnly) 24954 * Fix rmdir sometimes deleting directories with contents (Nick Craig-Wood) 24955 * Pcloud 24956 * Delete half uploaded files on upload error (Nick Craig-Wood) 24957 * Qingstor 24958 * Remove leading / from paths (Nick Craig-Wood) 24959 * S3 24960 * Fix index out of range error with `--fast-list` (Nick Craig-Wood) 24961 * Add `--s3-force-path-style` (Nick Craig-Wood) 24962 * Add support for KMS Key ID (bsteiss) 24963 * Remove leading / from paths (Nick Craig-Wood) 24964 * Swift 24965 * Add `storage_policy` (Ruben Vandamme) 24966 * Make it so just `storage_url` or `auth_token` can be overridden (Nick Craig-Wood) 24967 * Fix server side copy bug for unusual file names (Nick Craig-Wood) 24968 * Remove leading / from paths (Nick Craig-Wood) 24969 * WebDAV 24970 * Ensure we call MKCOL with a URL with a trailing / for QNAP interop (Nick Craig-Wood) 24971 * If root ends with / then don't check if it is a file (Nick Craig-Wood) 24972 * Don't accept redirects when reading metadata (Nick Craig-Wood) 24973 * Add bearer token (Macaroon) support for dCache (Nick Craig-Wood) 24974 * Document dCache and Macaroons (Onno Zweers) 24975 * Sharepoint recursion with different depth (Henning) 24976 * Attempt to remove failed uploads (Nick Craig-Wood) 24977 * Yandex 24978 * Fix listing/deleting files in the root (Nick Craig-Wood) 24979 24980 ## v1.42 - 2018-06-16 24981 24982 * New backends 24983 * OpenDrive (Oliver Heyme, Jakub Karlicek, ncw) 24984 * New commands 24985 * deletefile command (Filip Bartodziej) 24986 * New Features 24987 * copy, move: Copy single files directly, don't use `--files-from` work-around 24988 * this makes them much more efficient 24989 * Implement `--max-transfer` flag to quit transferring at a limit 24990 * make exit code 8 for `--max-transfer` exceeded 24991 * copy: copy empty source directories to destination (Ishuah Kariuki) 24992 * check: Add `--one-way` flag (Kasper Byrdal Nielsen) 24993 * Add siginfo handler for macOS for ctrl-T stats (kubatasiemski) 24994 * rc 24995 * add core/gc to run a garbage collection on demand 24996 * enable go profiling by default on the `--rc` port 24997 * return error from remote on failure 24998 * lsf 24999 * Add `--absolute` flag to add a leading / onto path names 25000 * Add `--csv` flag for compliant CSV output 25001 * Add 'm' format specifier to show the MimeType 25002 * Implement 'i' format for showing object ID 25003 * lsjson 25004 * Add MimeType to the output 25005 * Add ID field to output to show Object ID 25006 * Add `--retries-sleep` flag (Benjamin Joseph Dag) 25007 * Oauth tidy up web page and error handling (Henning Surmeier) 25008 * Bug Fixes 25009 * Password prompt output with `--log-file` fixed for unix (Filip Bartodziej) 25010 * Calculate ModifyWindow each time on the fly to fix various problems (Stefan Breunig) 25011 * Mount 25012 * Only print "File.rename error" if there actually is an error (Stefan Breunig) 25013 * Delay rename if file has open writers instead of failing outright (Stefan Breunig) 25014 * Ensure atexit gets run on interrupt 25015 * macOS enhancements 25016 * Make `--noappledouble` `--noapplexattr` 25017 * Add `--volname` flag and remove special chars from it 25018 * Make Get/List/Set/Remove xattr return ENOSYS for efficiency 25019 * Make `--daemon` work for macOS without CGO 25020 * VFS 25021 * Add `--vfs-read-chunk-size` and `--vfs-read-chunk-size-limit` (Fabian Möller) 25022 * Fix ChangeNotify for new or changed folders (Fabian Möller) 25023 * Local 25024 * Fix symlink/junction point directory handling under Windows 25025 * **NB** you will need to add `-L` to your command line to copy files with reparse points 25026 * Cache 25027 * Add non cached dirs on notifications (Remus Bunduc) 25028 * Allow root to be expired from rc (Remus Bunduc) 25029 * Clean remaining empty folders from temp upload path (Remus Bunduc) 25030 * Cache lists using batch writes (Remus Bunduc) 25031 * Use secure websockets for HTTPS Plex addresses (John Clayton) 25032 * Reconnect plex websocket on failures (Remus Bunduc) 25033 * Fix panic when running without plex configs (Remus Bunduc) 25034 * Fix root folder caching (Remus Bunduc) 25035 * Crypt 25036 * Check the crypted hash of files when uploading for extra data security 25037 * Dropbox 25038 * Make Dropbox for business folders accessible using an initial `/` in the path 25039 * Google Cloud Storage 25040 * Low level retry all operations if necessary 25041 * Google Drive 25042 * Add `--drive-acknowledge-abuse` to download flagged files 25043 * Add `--drive-alternate-export` to fix large doc export 25044 * Don't attempt to choose Team Drives when using rclone config create 25045 * Fix change list polling with team drives 25046 * Fix ChangeNotify for folders (Fabian Möller) 25047 * Fix about (and df on a mount) for team drives 25048 * Onedrive 25049 * Errorhandler for onedrive for business requests (Henning Surmeier) 25050 * S3 25051 * Adjust upload concurrency with `--s3-upload-concurrency` (themylogin) 25052 * Fix `--s3-chunk-size` which was always using the minimum 25053 * SFTP 25054 * Add `--ssh-path-override` flag (Piotr Oleszczyk) 25055 * Fix slow downloads for long latency connections 25056 * Webdav 25057 * Add workarounds for biz.mail.ru 25058 * Ignore Reason-Phrase in status line to fix 4shared (Rodrigo) 25059 * Better error message generation 25060 25061 ## v1.41 - 2018-04-28 25062 25063 * New backends 25064 * Mega support added 25065 * Webdav now supports SharePoint cookie authentication (hensur) 25066 * New commands 25067 * link: create public link to files and folders (Stefan Breunig) 25068 * about: gets quota info from a remote (a-roussos, ncw) 25069 * hashsum: a generic tool for any hash to produce md5sum like output 25070 * New Features 25071 * lsd: Add -R flag and fix and update docs for all ls commands 25072 * ncdu: added a "refresh" key - CTRL-L (Keith Goldfarb) 25073 * serve restic: Add append-only mode (Steve Kriss) 25074 * serve restic: Disallow overwriting files in append-only mode (Alexander Neumann) 25075 * serve restic: Print actual listener address (Matt Holt) 25076 * size: Add --json flag (Matthew Holt) 25077 * sync: implement --ignore-errors (Mateusz Pabian) 25078 * dedupe: Add dedupe largest functionality (Richard Yang) 25079 * fs: Extend SizeSuffix to include TB and PB for rclone about 25080 * fs: add --dump goroutines and --dump openfiles for debugging 25081 * rc: implement core/memstats to print internal memory usage info 25082 * rc: new call rc/pid (Michael P. Dubner) 25083 * Compile 25084 * Drop support for go1.6 25085 * Release 25086 * Fix `make tarball` (Chih-Hsuan Yen) 25087 * Bug Fixes 25088 * filter: fix --min-age and --max-age together check 25089 * fs: limit MaxIdleConns and MaxIdleConnsPerHost in transport 25090 * lsd,lsf: make sure all times we output are in local time 25091 * rc: fix setting bwlimit to unlimited 25092 * rc: take note of the --rc-addr flag too as per the docs 25093 * Mount 25094 * Use About to return the correct disk total/used/free (eg in `df`) 25095 * Set `--attr-timeout default` to `1s` - fixes: 25096 * rclone using too much memory 25097 * rclone not serving files to samba 25098 * excessive time listing directories 25099 * Fix `df -i` (upstream fix) 25100 * VFS 25101 * Filter files `.` and `..` from directory listing 25102 * Only make the VFS cache if --vfs-cache-mode > Off 25103 * Local 25104 * Add --local-no-check-updated to disable updated file checks 25105 * Retry remove on Windows sharing violation error 25106 * Cache 25107 * Flush the memory cache after close 25108 * Purge file data on notification 25109 * Always forget parent dir for notifications 25110 * Integrate with Plex websocket 25111 * Add rc cache/stats (seuffert) 25112 * Add info log on notification 25113 * Box 25114 * Fix failure reading large directories - parse file/directory size as float 25115 * Dropbox 25116 * Fix crypt+obfuscate on dropbox 25117 * Fix repeatedly uploading the same files 25118 * FTP 25119 * Work around strange response from box FTP server 25120 * More workarounds for FTP servers to fix mkParentDir error 25121 * Fix no error on listing non-existent directory 25122 * Google Cloud Storage 25123 * Add service_account_credentials (Matt Holt) 25124 * Detect bucket presence by listing it - minimises permissions needed 25125 * Ignore zero length directory markers 25126 * Google Drive 25127 * Add service_account_credentials (Matt Holt) 25128 * Fix directory move leaving a hardlinked directory behind 25129 * Return proper google errors when Opening files 25130 * When initialized with a filepath, optional features used incorrect root path (Stefan Breunig) 25131 * HTTP 25132 * Fix sync for servers which don't return Content-Length in HEAD 25133 * Onedrive 25134 * Add QuickXorHash support for OneDrive for business 25135 * Fix socket leak in multipart session upload 25136 * S3 25137 * Look in S3 named profile files for credentials 25138 * Add `--s3-disable-checksum` to disable checksum uploading (Chris Redekop) 25139 * Hierarchical configuration support (Giri Badanahatti) 25140 * Add in config for all the supported S3 providers 25141 * Add One Zone Infrequent Access storage class (Craig Rachel) 25142 * Add --use-server-modtime support (Peter Baumgartner) 25143 * Add --s3-chunk-size option to control multipart uploads 25144 * Ignore zero length directory markers 25145 * SFTP 25146 * Update docs to match code, fix typos and clarify disable_hashcheck prompt (Michael G. Noll) 25147 * Update docs with Synology quirks 25148 * Fail soft with a debug on hash failure 25149 * Swift 25150 * Add --use-server-modtime support (Peter Baumgartner) 25151 * Webdav 25152 * Support SharePoint cookie authentication (hensur) 25153 * Strip leading and trailing / off root 25154 25155 ## v1.40 - 2018-03-19 25156 25157 * New backends 25158 * Alias backend to create aliases for existing remote names (Fabian Möller) 25159 * New commands 25160 * `lsf`: list for parsing purposes (Jakub Tasiemski) 25161 * by default this is a simple non recursive list of files and directories 25162 * it can be configured to add more info in an easy to parse way 25163 * `serve restic`: for serving a remote as a Restic REST endpoint 25164 * This enables restic to use any backends that rclone can access 25165 * Thanks Alexander Neumann for help, patches and review 25166 * `rc`: enable the remote control of a running rclone 25167 * The running rclone must be started with --rc and related flags. 25168 * Currently there is support for bwlimit, and flushing for mount and cache. 25169 * New Features 25170 * `--max-delete` flag to add a delete threshold (Bjørn Erik Pedersen) 25171 * All backends now support RangeOption for ranged Open 25172 * `cat`: Use RangeOption for limited fetches to make more efficient 25173 * `cryptcheck`: make reading of nonce more efficient with RangeOption 25174 * serve http/webdav/restic 25175 * support SSL/TLS 25176 * add `--user` `--pass` and `--htpasswd` for authentication 25177 * `copy`/`move`: detect file size change during copy/move and abort transfer (ishuah) 25178 * `cryptdecode`: added option to return encrypted file names. (ishuah) 25179 * `lsjson`: add `--encrypted` to show encrypted name (Jakub Tasiemski) 25180 * Add `--stats-file-name-length` to specify the printed file name length for stats (Will Gunn) 25181 * Compile 25182 * Code base was shuffled and factored 25183 * backends moved into a backend directory 25184 * large packages split up 25185 * See the CONTRIBUTING.md doc for info as to what lives where now 25186 * Update to using go1.10 as the default go version 25187 * Implement daily [full integration tests](https://pub.rclone.org/integration-tests/) 25188 * Release 25189 * Include a source tarball and sign it and the binaries 25190 * Sign the git tags as part of the release process 25191 * Add .deb and .rpm packages as part of the build 25192 * Make a beta release for all branches on the main repo (but not pull requests) 25193 * Bug Fixes 25194 * config: fixes errors on non existing config by loading config file only on first access 25195 * config: retry saving the config after failure (Mateusz) 25196 * sync: when using `--backup-dir` don't delete files if we can't set their modtime 25197 * this fixes odd behaviour with Dropbox and `--backup-dir` 25198 * fshttp: fix idle timeouts for HTTP connections 25199 * `serve http`: fix serving files with : in - fixes 25200 * Fix `--exclude-if-present` to ignore directories which it doesn't have permission for (Iakov Davydov) 25201 * Make accounting work properly with crypt and b2 25202 * remove `--no-traverse` flag because it is obsolete 25203 * Mount 25204 * Add `--attr-timeout` flag to control attribute caching in kernel 25205 * this now defaults to 0 which is correct but less efficient 25206 * see [the mount docs](https://rclone.org/commands/rclone_mount/#attribute-caching) for more info 25207 * Add `--daemon` flag to allow mount to run in the background (ishuah) 25208 * Fix: Return ENOSYS rather than EIO on attempted link 25209 * This fixes FileZilla accessing an rclone mount served over sftp. 25210 * Fix setting modtime twice 25211 * Mount tests now run on CI for Linux (mount & cmount)/Mac/Windows 25212 * Many bugs fixed in the VFS layer - see below 25213 * VFS 25214 * Many fixes for `--vfs-cache-mode` writes and above 25215 * Update cached copy if we know it has changed (fixes stale data) 25216 * Clean path names before using them in the cache 25217 * Disable cache cleaner if `--vfs-cache-poll-interval=0` 25218 * Fill and clean the cache immediately on startup 25219 * Fix Windows opening every file when it stats the file 25220 * Fix applying modtime for an open Write Handle 25221 * Fix creation of files when truncating 25222 * Write 0 bytes when flushing unwritten handles to avoid race conditions in FUSE 25223 * Downgrade "poll-interval is not supported" message to Info 25224 * Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 25225 * Local 25226 * Downgrade "invalid cross-device link: trying copy" to debug 25227 * Make DirMove return fs.ErrorCantDirMove to allow fallback to Copy for cross device 25228 * Fix race conditions updating the hashes 25229 * Cache 25230 * Add support for polling - cache will update when remote changes on supported backends 25231 * Reduce log level for Plex api 25232 * Fix dir cache issue 25233 * Implement `--cache-db-wait-time` flag 25234 * Improve efficiency with RangeOption and RangeSeek 25235 * Fix dirmove with temp fs enabled 25236 * Notify vfs when using temp fs 25237 * Offline uploading 25238 * Remote control support for path flushing 25239 * Amazon cloud drive 25240 * Rclone no longer has any working keys - disable integration tests 25241 * Implement DirChangeNotify to notify cache/vfs/mount of changes 25242 * Azureblob 25243 * Don't check for bucket/container presence if listing was OK 25244 * this makes rclone do one less request per invocation 25245 * Improve accounting for chunked uploads 25246 * Backblaze B2 25247 * Don't check for bucket/container presence if listing was OK 25248 * this makes rclone do one less request per invocation 25249 * Box 25250 * Improve accounting for chunked uploads 25251 * Dropbox 25252 * Fix custom oauth client parameters 25253 * Google Cloud Storage 25254 * Don't check for bucket/container presence if listing was OK 25255 * this makes rclone do one less request per invocation 25256 * Google Drive 25257 * Migrate to api v3 (Fabian Möller) 25258 * Add scope configuration and root folder selection 25259 * Add `--drive-impersonate` for service accounts 25260 * thanks to everyone who tested, explored and contributed docs 25261 * Add `--drive-use-created-date` to use created date as modified date (nbuchanan) 25262 * Request the export formats only when required 25263 * This makes rclone quicker when there are no google docs 25264 * Fix finding paths with latin1 chars (a workaround for a drive bug) 25265 * Fix copying of a single Google doc file 25266 * Fix `--drive-auth-owner-only` to look in all directories 25267 * HTTP 25268 * Fix handling of directories with & in 25269 * Onedrive 25270 * Removed upload cutoff and always do session uploads 25271 * this stops the creation of multiple versions on business onedrive 25272 * Overwrite object size value with real size when reading file. (Victor) 25273 * this fixes oddities when onedrive misreports the size of images 25274 * Pcloud 25275 * Remove unused chunked upload flag and code 25276 * Qingstor 25277 * Don't check for bucket/container presence if listing was OK 25278 * this makes rclone do one less request per invocation 25279 * S3 25280 * Support hashes for multipart files (Chris Redekop) 25281 * Initial support for IBM COS (S3) (Giri Badanahatti) 25282 * Update docs to discourage use of v2 auth with CEPH and others 25283 * Don't check for bucket/container presence if listing was OK 25284 * this makes rclone do one less request per invocation 25285 * Fix server side copy and set modtime on files with + in 25286 * SFTP 25287 * Add option to disable remote hash check command execution (Jon Fautley) 25288 * Add `--sftp-ask-password` flag to prompt for password when needed (Leo R. Lundgren) 25289 * Add `set_modtime` configuration option 25290 * Fix following of symlinks 25291 * Fix reading config file outside of Fs setup 25292 * Fix reading $USER in username fallback not $HOME 25293 * Fix running under crontab - Use correct OS way of reading username 25294 * Swift 25295 * Fix refresh of authentication token 25296 * in v1.39 a bug was introduced which ignored new tokens - this fixes it 25297 * Fix extra HEAD transaction when uploading a new file 25298 * Don't check for bucket/container presence if listing was OK 25299 * this makes rclone do one less request per invocation 25300 * Webdav 25301 * Add new time formats to support mydrive.ch and others 25302 25303 ## v1.39 - 2017-12-23 25304 25305 * New backends 25306 * WebDAV 25307 * tested with nextcloud, owncloud, put.io and others! 25308 * Pcloud 25309 * cache - wraps a cache around other backends (Remus Bunduc) 25310 * useful in combination with mount 25311 * NB this feature is in beta so use with care 25312 * New commands 25313 * serve command with subcommands: 25314 * serve webdav: this implements a webdav server for any rclone remote. 25315 * serve http: command to serve a remote over HTTP 25316 * config: add sub commands for full config file management 25317 * create/delete/dump/edit/file/password/providers/show/update 25318 * touch: to create or update the timestamp of a file (Jakub Tasiemski) 25319 * New Features 25320 * curl install for rclone (Filip Bartodziej) 25321 * --stats now shows percentage, size, rate and ETA in condensed form (Ishuah Kariuki) 25322 * --exclude-if-present to exclude a directory if a file is present (Iakov Davydov) 25323 * rmdirs: add --leave-root flag (lewapm) 25324 * move: add --delete-empty-src-dirs flag to remove dirs after move (Ishuah Kariuki) 25325 * Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters 25326 * Obscure X-Auth-Token: from headers when dumping too 25327 * Document and implement exit codes for different failure modes (Ishuah Kariuki) 25328 * Compile 25329 * Bug Fixes 25330 * Retry lots more different types of errors to make multipart transfers more reliable 25331 * Save the config before asking for a token, fixes disappearing oauth config 25332 * Warn the user if --include and --exclude are used together (Ernest Borowski) 25333 * Fix duplicate files (eg on Google drive) causing spurious copies 25334 * Allow trailing and leading whitespace for passwords (Jason Rose) 25335 * ncdu: fix crashes on empty directories 25336 * rcat: fix goroutine leak 25337 * moveto/copyto: Fix to allow copying to the same name 25338 * Mount 25339 * --vfs-cache mode to make writes into mounts more reliable. 25340 * this requires caching files on the disk (see --cache-dir) 25341 * As this is a new feature, use with care 25342 * Use sdnotify to signal systemd the mount is ready (Fabian Möller) 25343 * Check if directory is not empty before mounting (Ernest Borowski) 25344 * Local 25345 * Add error message for cross file system moves 25346 * Fix equality check for times 25347 * Dropbox 25348 * Rework multipart upload 25349 * buffer the chunks when uploading large files so they can be retried 25350 * change default chunk size to 48MB now we are buffering them in memory 25351 * retry every error after the first chunk is done successfully 25352 * Fix error when renaming directories 25353 * Swift 25354 * Fix crash on bad authentication 25355 * Google Drive 25356 * Add service account support (Tim Cooijmans) 25357 * S3 25358 * Make it work properly with Digital Ocean Spaces (Andrew Starr-Bochicchio) 25359 * Fix crash if a bad listing is received 25360 * Add support for ECS task IAM roles (David Minor) 25361 * Backblaze B2 25362 * Fix multipart upload retries 25363 * Fix --hard-delete to make it work 100% of the time 25364 * Swift 25365 * Allow authentication with storage URL and auth key (Giovanni Pizzi) 25366 * Add new fields for swift configuration to support IBM Bluemix Swift (Pierre Carlson) 25367 * Add OS_TENANT_ID and OS_USER_ID to config 25368 * Allow configs with user id instead of user name 25369 * Check if swift segments container exists before creating (John Leach) 25370 * Fix memory leak in swift transfers (upstream fix) 25371 * SFTP 25372 * Add option to enable the use of aes128-cbc cipher (Jon Fautley) 25373 * Amazon cloud drive 25374 * Fix download of large files failing with "Only one auth mechanism allowed" 25375 * crypt 25376 * Option to encrypt directory names or leave them intact 25377 * Implement DirChangeNotify (Fabian Möller) 25378 * onedrive 25379 * Add option to choose resourceURL during setup of OneDrive Business account if more than one is available for user 25380 25381 ## v1.38 - 2017-09-30 25382 25383 * New backends 25384 * Azure Blob Storage (thanks Andrei Dragomir) 25385 * Box 25386 * Onedrive for Business (thanks Oliver Heyme) 25387 * QingStor from QingCloud (thanks wuyu) 25388 * New commands 25389 * `rcat` - read from standard input and stream upload 25390 * `tree` - shows a nicely formatted recursive listing 25391 * `cryptdecode` - decode crypted file names (thanks ishuah) 25392 * `config show` - print the config file 25393 * `config file` - print the config file location 25394 * New Features 25395 * Empty directories are deleted on `sync` 25396 * `dedupe` - implement merging of duplicate directories 25397 * `check` and `cryptcheck` made more consistent and use less memory 25398 * `cleanup` for remaining remotes (thanks ishuah) 25399 * `--immutable` for ensuring that files don't change (thanks Jacob McNamee) 25400 * `--user-agent` option (thanks Alex McGrath Kraak) 25401 * `--disable` flag to disable optional features 25402 * `--bind` flag for choosing the local addr on outgoing connections 25403 * Support for zsh auto-completion (thanks bpicode) 25404 * Stop normalizing file names but do a normalized compare in `sync` 25405 * Compile 25406 * Update to using go1.9 as the default go version 25407 * Remove snapd build due to maintenance problems 25408 * Bug Fixes 25409 * Improve retriable error detection which makes multipart uploads better 25410 * Make `check` obey `--ignore-size` 25411 * Fix bwlimit toggle in conjunction with schedules (thanks cbruegg) 25412 * `config` ensures newly written config is on the same mount 25413 * Local 25414 * Revert to copy when moving file across file system boundaries 25415 * `--skip-links` to suppress symlink warnings (thanks Zhiming Wang) 25416 * Mount 25417 * Re-use `rcat` internals to support uploads from all remotes 25418 * Dropbox 25419 * Fix "entry doesn't belong in directory" error 25420 * Stop using deprecated API methods 25421 * Swift 25422 * Fix server side copy to empty container with `--fast-list` 25423 * Google Drive 25424 * Change the default for `--drive-use-trash` to `true` 25425 * S3 25426 * Set session token when using STS (thanks Girish Ramakrishnan) 25427 * Glacier docs and error messages (thanks Jan Varho) 25428 * Read 1000 (not 1024) items in dir listings to fix Wasabi 25429 * Backblaze B2 25430 * Fix SHA1 mismatch when downloading files with no SHA1 25431 * Calculate missing hashes on the fly instead of spooling 25432 * `--b2-hard-delete` to permanently delete (not hide) files (thanks John Papandriopoulos) 25433 * Hubic 25434 * Fix creating containers - no longer have to use the `default` container 25435 * Swift 25436 * Optionally configure from a standard set of OpenStack environment vars 25437 * Add `endpoint_type` config 25438 * Google Cloud Storage 25439 * Fix bucket creation to work with limited permission users 25440 * SFTP 25441 * Implement connection pooling for multiple ssh connections 25442 * Limit new connections per second 25443 * Add support for MD5 and SHA1 hashes where available (thanks Christian Brüggemann) 25444 * HTTP 25445 * Fix URL encoding issues 25446 * Fix directories with `:` in 25447 * Fix panic with URL encoded content 25448 25449 ## v1.37 - 2017-07-22 25450 25451 * New backends 25452 * FTP - thanks to Antonio Messina 25453 * HTTP - thanks to Vasiliy Tolstov 25454 * New commands 25455 * rclone ncdu - for exploring a remote with a text based user interface. 25456 * rclone lsjson - for listing with a machine readable output 25457 * rclone dbhashsum - to show Dropbox style hashes of files (local or Dropbox) 25458 * New Features 25459 * Implement --fast-list flag 25460 * This allows remotes to list recursively if they can 25461 * This uses less transactions (important if you pay for them) 25462 * This may or may not be quicker 25463 * This will use more memory as it has to hold the listing in memory 25464 * --old-sync-method deprecated - the remaining uses are covered by --fast-list 25465 * This involved a major re-write of all the listing code 25466 * Add --tpslimit and --tpslimit-burst to limit transactions per second 25467 * this is useful in conjunction with `rclone mount` to limit external apps 25468 * Add --stats-log-level so can see --stats without -v 25469 * Print password prompts to stderr - Hraban Luyat 25470 * Warn about duplicate files when syncing 25471 * Oauth improvements 25472 * allow auth_url and token_url to be set in the config file 25473 * Print redirection URI if using own credentials. 25474 * Don't Mkdir at the start of sync to save transactions 25475 * Compile 25476 * Update build to go1.8.3 25477 * Require go1.6 for building rclone 25478 * Compile 386 builds with "GO386=387" for maximum compatibility 25479 * Bug Fixes 25480 * Fix menu selection when no remotes 25481 * Config saving reworked to not kill the file if disk gets full 25482 * Don't delete remote if name does not change while renaming 25483 * moveto, copyto: report transfers and checks as per move and copy 25484 * Local 25485 * Add --local-no-unicode-normalization flag - Bob Potter 25486 * Mount 25487 * Now supported on Windows using cgofuse and WinFsp - thanks to Bill Zissimopoulos for much help 25488 * Compare checksums on upload/download via FUSE 25489 * Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM - Jérôme Vizcaino 25490 * On read only open of file, make open pending until first read 25491 * Make --read-only reject modify operations 25492 * Implement ModTime via FUSE for remotes that support it 25493 * Allow modTime to be changed even before all writers are closed 25494 * Fix panic on renames 25495 * Fix hang on errored upload 25496 * Crypt 25497 * Report the name:root as specified by the user 25498 * Add an "obfuscate" option for filename encryption - Stephen Harris 25499 * Amazon Drive 25500 * Fix initialization order for token renewer 25501 * Remove revoked credentials, allow oauth proxy config and update docs 25502 * B2 25503 * Reduce minimum chunk size to 5MB 25504 * Drive 25505 * Add team drive support 25506 * Reduce bandwidth by adding fields for partial responses - Martin Kristensen 25507 * Implement --drive-shared-with-me flag to view shared with me files - Danny Tsai 25508 * Add --drive-trashed-only to read only the files in the trash 25509 * Remove obsolete --drive-full-list 25510 * Add missing seek to start on retries of chunked uploads 25511 * Fix stats accounting for upload 25512 * Convert / in names to a unicode equivalent (/) 25513 * Poll for Google Drive changes when mounted 25514 * OneDrive 25515 * Fix the uploading of files with spaces 25516 * Fix initialization order for token renewer 25517 * Display speeds accurately when uploading - Yoni Jah 25518 * Swap to using http://localhost:53682/ as redirect URL - Michael Ledin 25519 * Retry on token expired error, reset upload body on retry - Yoni Jah 25520 * Google Cloud Storage 25521 * Add ability to specify location and storage class via config and command line - thanks gdm85 25522 * Create container if necessary on server side copy 25523 * Increase directory listing chunk to 1000 to increase performance 25524 * Obtain a refresh token for GCS - Steven Lu 25525 * Yandex 25526 * Fix the name reported in log messages (was empty) 25527 * Correct error return for listing empty directory 25528 * Dropbox 25529 * Rewritten to use the v2 API 25530 * Now supports ModTime 25531 * Can only set by uploading the file again 25532 * If you uploaded with an old rclone, rclone may upload everything again 25533 * Use `--size-only` or `--checksum` to avoid this 25534 * Now supports the Dropbox content hashing scheme 25535 * Now supports low level retries 25536 * S3 25537 * Work around eventual consistency in bucket creation 25538 * Create container if necessary on server side copy 25539 * Add us-east-2 (Ohio) and eu-west-2 (London) S3 regions - Zahiar Ahmed 25540 * Swift, Hubic 25541 * Fix zero length directory markers showing in the subdirectory listing 25542 * this caused lots of duplicate transfers 25543 * Fix paged directory listings 25544 * this caused duplicate directory errors 25545 * Create container if necessary on server side copy 25546 * Increase directory listing chunk to 1000 to increase performance 25547 * Make sensible error if the user forgets the container 25548 * SFTP 25549 * Add support for using ssh key files 25550 * Fix under Windows 25551 * Fix ssh agent on Windows 25552 * Adapt to latest version of library - Igor Kharin 25553 25554 ## v1.36 - 2017-03-18 25555 25556 * New Features 25557 * SFTP remote (Jack Schmidt) 25558 * Re-implement sync routine to work a directory at a time reducing memory usage 25559 * Logging revamped to be more inline with rsync - now much quieter 25560 * -v only shows transfers 25561 * -vv is for full debug 25562 * --syslog to log to syslog on capable platforms 25563 * Implement --backup-dir and --suffix 25564 * Implement --track-renames (initial implementation by Bjørn Erik Pedersen) 25565 * Add time-based bandwidth limits (Lukas Loesche) 25566 * rclone cryptcheck: checks integrity of crypt remotes 25567 * Allow all config file variables and options to be set from environment variables 25568 * Add --buffer-size parameter to control buffer size for copy 25569 * Make --delete-after the default 25570 * Add --ignore-checksum flag (fixed by Hisham Zarka) 25571 * rclone check: Add --download flag to check all the data, not just hashes 25572 * rclone cat: add --head, --tail, --offset, --count and --discard 25573 * rclone config: when choosing from a list, allow the value to be entered too 25574 * rclone config: allow rename and copy of remotes 25575 * rclone obscure: for generating encrypted passwords for rclone's config (T.C. Ferguson) 25576 * Comply with XDG Base Directory specification (Dario Giovannetti) 25577 * this moves the default location of the config file in a backwards compatible way 25578 * Release changes 25579 * Ubuntu snap support (Dedsec1) 25580 * Compile with go 1.8 25581 * MIPS/Linux big and little endian support 25582 * Bug Fixes 25583 * Fix copyto copying things to the wrong place if the destination dir didn't exist 25584 * Fix parsing of remotes in moveto and copyto 25585 * Fix --delete-before deleting files on copy 25586 * Fix --files-from with an empty file copying everything 25587 * Fix sync: don't update mod times if --dry-run set 25588 * Fix MimeType propagation 25589 * Fix filters to add ** rules to directory rules 25590 * Local 25591 * Implement -L, --copy-links flag to allow rclone to follow symlinks 25592 * Open files in write only mode so rclone can write to an rclone mount 25593 * Fix unnormalised unicode causing problems reading directories 25594 * Fix interaction between -x flag and --max-depth 25595 * Mount 25596 * Implement proper directory handling (mkdir, rmdir, renaming) 25597 * Make include and exclude filters apply to mount 25598 * Implement read and write async buffers - control with --buffer-size 25599 * Fix fsync on for directories 25600 * Fix retry on network failure when reading off crypt 25601 * Crypt 25602 * Add --crypt-show-mapping to show encrypted file mapping 25603 * Fix crypt writer getting stuck in a loop 25604 * **IMPORTANT** this bug had the potential to cause data corruption when 25605 * reading data from a network based remote and 25606 * writing to a crypt on Google Drive 25607 * Use the cryptcheck command to validate your data if you are concerned 25608 * If syncing two crypt remotes, sync the unencrypted remote 25609 * Amazon Drive 25610 * Fix panics on Move (rename) 25611 * Fix panic on token expiry 25612 * B2 25613 * Fix inconsistent listings and rclone check 25614 * Fix uploading empty files with go1.8 25615 * Constrain memory usage when doing multipart uploads 25616 * Fix upload url not being refreshed properly 25617 * Drive 25618 * Fix Rmdir on directories with trashed files 25619 * Fix "Ignoring unknown object" when downloading 25620 * Add --drive-list-chunk 25621 * Add --drive-skip-gdocs (Károly Oláh) 25622 * OneDrive 25623 * Implement Move 25624 * Fix Copy 25625 * Fix overwrite detection in Copy 25626 * Fix waitForJob to parse errors correctly 25627 * Use token renewer to stop auth errors on long uploads 25628 * Fix uploading empty files with go1.8 25629 * Google Cloud Storage 25630 * Fix depth 1 directory listings 25631 * Yandex 25632 * Fix single level directory listing 25633 * Dropbox 25634 * Normalise the case for single level directory listings 25635 * Fix depth 1 listing 25636 * S3 25637 * Added ca-central-1 region (Jon Yergatian) 25638 25639 ## v1.35 - 2017-01-02 25640 25641 * New Features 25642 * moveto and copyto commands for choosing a destination name on copy/move 25643 * rmdirs command to recursively delete empty directories 25644 * Allow repeated --include/--exclude/--filter options 25645 * Only show transfer stats on commands which transfer stuff 25646 * show stats on any command using the `--stats` flag 25647 * Allow overlapping directories in move when server side dir move is supported 25648 * Add --stats-unit option - thanks Scott McGillivray 25649 * Bug Fixes 25650 * Fix the config file being overwritten when two rclone instances are running 25651 * Make rclone lsd obey the filters properly 25652 * Fix compilation on mips 25653 * Fix not transferring files that don't differ in size 25654 * Fix panic on nil retry/fatal error 25655 * Mount 25656 * Retry reads on error - should help with reliability a lot 25657 * Report the modification times for directories from the remote 25658 * Add bandwidth accounting and limiting (fixes --bwlimit) 25659 * If --stats provided will show stats and which files are transferring 25660 * Support R/W files if truncate is set. 25661 * Implement statfs interface so df works 25662 * Note that write is now supported on Amazon Drive 25663 * Report number of blocks in a file - thanks Stefan Breunig 25664 * Crypt 25665 * Prevent the user pointing crypt at itself 25666 * Fix failed to authenticate decrypted block errors 25667 * these will now return the underlying unexpected EOF instead 25668 * Amazon Drive 25669 * Add support for server side move and directory move - thanks Stefan Breunig 25670 * Fix nil pointer deref on size attribute 25671 * B2 25672 * Use new prefix and delimiter parameters in directory listings 25673 * This makes --max-depth 1 dir listings as used in mount much faster 25674 * Reauth the account while doing uploads too - should help with token expiry 25675 * Drive 25676 * Make DirMove more efficient and complain about moving the root 25677 * Create destination directory on Move() 25678 25679 ## v1.34 - 2016-11-06 25680 25681 * New Features 25682 * Stop single file and `--files-from` operations iterating through the source bucket. 25683 * Stop removing failed upload to cloud storage remotes 25684 * Make ContentType be preserved for cloud to cloud copies 25685 * Add support to toggle bandwidth limits via SIGUSR2 - thanks Marco Paganini 25686 * `rclone check` shows count of hashes that couldn't be checked 25687 * `rclone listremotes` command 25688 * Support linux/arm64 build - thanks Fredrik Fornwall 25689 * Remove `Authorization:` lines from `--dump-headers` output 25690 * Bug Fixes 25691 * Ignore files with control characters in the names 25692 * Fix `rclone move` command 25693 * Delete src files which already existed in dst 25694 * Fix deletion of src file when dst file older 25695 * Fix `rclone check` on crypted file systems 25696 * Make failed uploads not count as "Transferred" 25697 * Make sure high level retries show with `-q` 25698 * Use a vendor directory with godep for repeatable builds 25699 * `rclone mount` - FUSE 25700 * Implement FUSE mount options 25701 * `--no-modtime`, `--debug-fuse`, `--read-only`, `--allow-non-empty`, `--allow-root`, `--allow-other` 25702 * `--default-permissions`, `--write-back-cache`, `--max-read-ahead`, `--umask`, `--uid`, `--gid` 25703 * Add `--dir-cache-time` to control caching of directory entries 25704 * Implement seek for files opened for read (useful for video players) 25705 * with `-no-seek` flag to disable 25706 * Fix crash on 32 bit ARM (alignment of 64 bit counter) 25707 * ...and many more internal fixes and improvements! 25708 * Crypt 25709 * Don't show encrypted password in configurator to stop confusion 25710 * Amazon Drive 25711 * New wait for upload option `--acd-upload-wait-per-gb` 25712 * upload timeouts scale by file size and can be disabled 25713 * Add 502 Bad Gateway to list of errors we retry 25714 * Fix overwriting a file with a zero length file 25715 * Fix ACD file size warning limit - thanks Felix Bünemann 25716 * Local 25717 * Unix: implement `-x`/`--one-file-system` to stay on a single file system 25718 * thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana 25719 * Windows: ignore the symlink bit on files 25720 * Windows: Ignore directory based junction points 25721 * B2 25722 * Make sure each upload has at least one upload slot - fixes strange upload stats 25723 * Fix uploads when using crypt 25724 * Fix download of large files (sha1 mismatch) 25725 * Return error when we try to create a bucket which someone else owns 25726 * Update B2 docs with Data usage, and Crypt section - thanks Tomasz Mazur 25727 * S3 25728 * Command line and config file support for 25729 * Setting/overriding ACL - thanks Radek Senfeld 25730 * Setting storage class - thanks Asko Tamm 25731 * Drive 25732 * Make exponential backoff work exactly as per Google specification 25733 * add `.epub`, `.odp` and `.tsv` as export formats. 25734 * Swift 25735 * Don't read metadata for directory marker objects 25736 25737 ## v1.33 - 2016-08-24 25738 25739 * New Features 25740 * Implement encryption 25741 * data encrypted in NACL secretbox format 25742 * with optional file name encryption 25743 * New commands 25744 * rclone mount - implements FUSE mounting of remotes (EXPERIMENTAL) 25745 * works on Linux, FreeBSD and OS X (need testers for the last 2!) 25746 * rclone cat - outputs remote file or files to the terminal 25747 * rclone genautocomplete - command to make a bash completion script for rclone 25748 * Editing a remote using `rclone config` now goes through the wizard 25749 * Compile with go 1.7 - this fixes rclone on macOS Sierra and on 386 processors 25750 * Use cobra for sub commands and docs generation 25751 * drive 25752 * Document how to make your own client_id 25753 * s3 25754 * User-configurable Amazon S3 ACL (thanks Radek Šenfeld) 25755 * b2 25756 * Fix stats accounting for upload - no more jumping to 100% done 25757 * On cleanup delete hide marker if it is the current file 25758 * New B2 API endpoint (thanks Per Cederberg) 25759 * Set maximum backoff to 5 Minutes 25760 * onedrive 25761 * Fix URL escaping in file names - eg uploading files with `+` in them. 25762 * amazon cloud drive 25763 * Fix token expiry during large uploads 25764 * Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors 25765 * local 25766 * Fix filenames with invalid UTF-8 not being uploaded 25767 * Fix problem with some UTF-8 characters on OS X 25768 25769 ## v1.32 - 2016-07-13 25770 25771 * Backblaze B2 25772 * Fix upload of files large files not in root 25773 25774 ## v1.31 - 2016-07-13 25775 25776 * New Features 25777 * Reduce memory on sync by about 50% 25778 * Implement --no-traverse flag to stop copy traversing the destination remote. 25779 * This can be used to reduce memory usage down to the smallest possible. 25780 * Useful to copy a small number of files into a large destination folder. 25781 * Implement cleanup command for emptying trash / removing old versions of files 25782 * Currently B2 only 25783 * Single file handling improved 25784 * Now copied with --files-from 25785 * Automatically sets --no-traverse when copying a single file 25786 * Info on using installing with ansible - thanks Stefan Weichinger 25787 * Implement --no-update-modtime flag to stop rclone fixing the remote modified times. 25788 * Bug Fixes 25789 * Fix move command - stop it running for overlapping Fses - this was causing data loss. 25790 * Local 25791 * Fix incomplete hashes - this was causing problems for B2. 25792 * Amazon Drive 25793 * Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed. 25794 * Swift 25795 * Add support for non-default project domain - thanks Antonio Messina. 25796 * S3 25797 * Add instructions on how to use rclone with minio. 25798 * Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions. 25799 * Skip setting the modified time for objects > 5GB as it isn't possible. 25800 * Backblaze B2 25801 * Add --b2-versions flag so old versions can be listed and retrieved. 25802 * Treat 403 errors (eg cap exceeded) as fatal. 25803 * Implement cleanup command for deleting old file versions. 25804 * Make error handling compliant with B2 integrations notes. 25805 * Fix handling of token expiry. 25806 * Implement --b2-test-mode to set `X-Bz-Test-Mode` header. 25807 * Set cutoff for chunked upload to 200MB as per B2 guidelines. 25808 * Make upload multi-threaded. 25809 * Dropbox 25810 * Don't retry 461 errors. 25811 25812 ## v1.30 - 2016-06-18 25813 25814 * New Features 25815 * Directory listing code reworked for more features and better error reporting (thanks to Klaus Post for help). This enables 25816 * Directory include filtering for efficiency 25817 * --max-depth parameter 25818 * Better error reporting 25819 * More to come 25820 * Retry more errors 25821 * Add --ignore-size flag - for uploading images to onedrive 25822 * Log -v output to stdout by default 25823 * Display the transfer stats in more human readable form 25824 * Make 0 size files specifiable with `--max-size 0b` 25825 * Add `b` suffix so we can specify bytes in --bwlimit, --min-size etc 25826 * Use "password:" instead of "password>" prompt - thanks Klaus Post and Leigh Klotz 25827 * Bug Fixes 25828 * Fix retry doing one too many retries 25829 * Local 25830 * Fix problems with OS X and UTF-8 characters 25831 * Amazon Drive 25832 * Check a file exists before uploading to help with 408 Conflict errors 25833 * Reauth on 401 errors - this has been causing a lot of problems 25834 * Work around spurious 403 errors 25835 * Restart directory listings on error 25836 * Google Drive 25837 * Check a file exists before uploading to help with duplicates 25838 * Fix retry of multipart uploads 25839 * Backblaze B2 25840 * Implement large file uploading 25841 * S3 25842 * Add AES256 server-side encryption for - thanks Justin R. Wilson 25843 * Google Cloud Storage 25844 * Make sure we don't use conflicting content types on upload 25845 * Add service account support - thanks Michal Witkowski 25846 * Swift 25847 * Add auth version parameter 25848 * Add domain option for openstack (v3 auth) - thanks Fabian Ruff 25849 25850 ## v1.29 - 2016-04-18 25851 25852 * New Features 25853 * Implement `-I, --ignore-times` for unconditional upload 25854 * Improve `dedupe`command 25855 * Now removes identical copies without asking 25856 * Now obeys `--dry-run` 25857 * Implement `--dedupe-mode` for non interactive running 25858 * `--dedupe-mode interactive` - interactive the default. 25859 * `--dedupe-mode skip` - removes identical files then skips anything left. 25860 * `--dedupe-mode first` - removes identical files then keeps the first one. 25861 * `--dedupe-mode newest` - removes identical files then keeps the newest one. 25862 * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. 25863 * `--dedupe-mode rename` - removes identical files then renames the rest to be different. 25864 * Bug fixes 25865 * Make rclone check obey the `--size-only` flag. 25866 * Use "application/octet-stream" if discovered mime type is invalid. 25867 * Fix missing "quit" option when there are no remotes. 25868 * Google Drive 25869 * Increase default chunk size to 8 MB - increases upload speed of big files 25870 * Speed up directory listings and make more reliable 25871 * Add missing retries for Move and DirMove - increases reliability 25872 * Preserve mime type on file update 25873 * Backblaze B2 25874 * Enable mod time syncing 25875 * This means that B2 will now check modification times 25876 * It will upload new files to update the modification times 25877 * (there isn't an API to just set the mod time.) 25878 * If you want the old behaviour use `--size-only`. 25879 * Update API to new version 25880 * Fix parsing of mod time when not in metadata 25881 * Swift/Hubic 25882 * Don't return an MD5SUM for static large objects 25883 * S3 25884 * Fix uploading files bigger than 50GB 25885 25886 ## v1.28 - 2016-03-01 25887 25888 * New Features 25889 * Configuration file encryption - thanks Klaus Post 25890 * Improve `rclone config` adding more help and making it easier to understand 25891 * Implement `-u`/`--update` so creation times can be used on all remotes 25892 * Implement `--low-level-retries` flag 25893 * Optionally disable gzip compression on downloads with `--no-gzip-encoding` 25894 * Bug fixes 25895 * Don't make directories if `--dry-run` set 25896 * Fix and document the `move` command 25897 * Fix redirecting stderr on unix-like OSes when using `--log-file` 25898 * Fix `delete` command to wait until all finished - fixes missing deletes. 25899 * Backblaze B2 25900 * Use one upload URL per go routine fixes `more than one upload using auth token` 25901 * Add pacing, retries and reauthentication - fixes token expiry problems 25902 * Upload without using a temporary file from local (and remotes which support SHA1) 25903 * Fix reading metadata for all files when it shouldn't have been 25904 * Drive 25905 * Fix listing drive documents at root 25906 * Disable copy and move for Google docs 25907 * Swift 25908 * Fix uploading of chunked files with non ASCII characters 25909 * Allow setting of `storage_url` in the config - thanks Xavier Lucas 25910 * S3 25911 * Allow IAM role and credentials from environment variables - thanks Brian Stengaard 25912 * Allow low privilege users to use S3 (check if directory exists during Mkdir) - thanks Jakub Gedeon 25913 * Amazon Drive 25914 * Retry on more things to make directory listings more reliable 25915 25916 ## v1.27 - 2016-01-31 25917 25918 * New Features 25919 * Easier headless configuration with `rclone authorize` 25920 * Add support for multiple hash types - we now check SHA1 as well as MD5 hashes. 25921 * `delete` command which does obey the filters (unlike `purge`) 25922 * `dedupe` command to deduplicate a remote. Useful with Google Drive. 25923 * Add `--ignore-existing` flag to skip all files that exist on destination. 25924 * Add `--delete-before`, `--delete-during`, `--delete-after` flags. 25925 * Add `--memprofile` flag to debug memory use. 25926 * Warn the user about files with same name but different case 25927 * Make `--include` rules add their implicit exclude * at the end of the filter list 25928 * Deprecate compiling with go1.3 25929 * Amazon Drive 25930 * Fix download of files > 10 GB 25931 * Fix directory traversal ("Next token is expired") for large directory listings 25932 * Remove 409 conflict from error codes we will retry - stops very long pauses 25933 * Backblaze B2 25934 * SHA1 hashes now checked by rclone core 25935 * Drive 25936 * Add `--drive-auth-owner-only` to only consider files owned by the user - thanks Björn Harrtell 25937 * Export Google documents 25938 * Dropbox 25939 * Make file exclusion error controllable with -q 25940 * Swift 25941 * Fix upload from unprivileged user. 25942 * S3 25943 * Fix updating of mod times of files with `+` in. 25944 * Local 25945 * Add local file system option to disable UNC on Windows. 25946 25947 ## v1.26 - 2016-01-02 25948 25949 * New Features 25950 * Yandex storage backend - thank you Dmitry Burdeev ("dibu") 25951 * Implement Backblaze B2 storage backend 25952 * Add --min-age and --max-age flags - thank you Adriano Aurélio Meirelles 25953 * Make ls/lsl/md5sum/size/check obey includes and excludes 25954 * Fixes 25955 * Fix crash in http logging 25956 * Upload releases to github too 25957 * Swift 25958 * Fix sync for chunked files 25959 * OneDrive 25960 * Re-enable server side copy 25961 * Don't mask HTTP error codes with JSON decode error 25962 * S3 25963 * Fix corrupting Content-Type on mod time update (thanks Joseph Spurrier) 25964 25965 ## v1.25 - 2015-11-14 25966 25967 * New features 25968 * Implement Hubic storage system 25969 * Fixes 25970 * Fix deletion of some excluded files without --delete-excluded 25971 * This could have deleted files unexpectedly on sync 25972 * Always check first with `--dry-run`! 25973 * Swift 25974 * Stop SetModTime losing metadata (eg X-Object-Manifest) 25975 * This could have caused data loss for files > 5GB in size 25976 * Use ContentType from Object to avoid lookups in listings 25977 * OneDrive 25978 * disable server side copy as it seems to be broken at Microsoft 25979 25980 ## v1.24 - 2015-11-07 25981 25982 * New features 25983 * Add support for Microsoft OneDrive 25984 * Add `--no-check-certificate` option to disable server certificate verification 25985 * Add async readahead buffer for faster transfer of big files 25986 * Fixes 25987 * Allow spaces in remotes and check remote names for validity at creation time 25988 * Allow '&' and disallow ':' in Windows filenames. 25989 * Swift 25990 * Ignore directory marker objects where appropriate - allows working with Hubic 25991 * Don't delete the container if fs wasn't at root 25992 * S3 25993 * Don't delete the bucket if fs wasn't at root 25994 * Google Cloud Storage 25995 * Don't delete the bucket if fs wasn't at root 25996 25997 ## v1.23 - 2015-10-03 25998 25999 * New features 26000 * Implement `rclone size` for measuring remotes 26001 * Fixes 26002 * Fix headless config for drive and gcs 26003 * Tell the user they should try again if the webserver method failed 26004 * Improve output of `--dump-headers` 26005 * S3 26006 * Allow anonymous access to public buckets 26007 * Swift 26008 * Stop chunked operations logging "Failed to read info: Object Not Found" 26009 * Use Content-Length on uploads for extra reliability 26010 26011 ## v1.22 - 2015-09-28 26012 26013 * Implement rsync like include and exclude flags 26014 * swift 26015 * Support files > 5GB - thanks Sergey Tolmachev 26016 26017 ## v1.21 - 2015-09-22 26018 26019 * New features 26020 * Display individual transfer progress 26021 * Make lsl output times in localtime 26022 * Fixes 26023 * Fix allowing user to override credentials again in Drive, GCS and ACD 26024 * Amazon Drive 26025 * Implement compliant pacing scheme 26026 * Google Drive 26027 * Make directory reads concurrent for increased speed. 26028 26029 ## v1.20 - 2015-09-15 26030 26031 * New features 26032 * Amazon Drive support 26033 * Oauth support redone - fix many bugs and improve usability 26034 * Use "golang.org/x/oauth2" as oauth library of choice 26035 * Improve oauth usability for smoother initial signup 26036 * drive, googlecloudstorage: optionally use auto config for the oauth token 26037 * Implement --dump-headers and --dump-bodies debug flags 26038 * Show multiple matched commands if abbreviation too short 26039 * Implement server side move where possible 26040 * local 26041 * Always use UNC paths internally on Windows - fixes a lot of bugs 26042 * dropbox 26043 * force use of our custom transport which makes timeouts work 26044 * Thanks to Klaus Post for lots of help with this release 26045 26046 ## v1.19 - 2015-08-28 26047 26048 * New features 26049 * Server side copies for s3/swift/drive/dropbox/gcs 26050 * Move command - uses server side copies if it can 26051 * Implement --retries flag - tries 3 times by default 26052 * Build for plan9/amd64 and solaris/amd64 too 26053 * Fixes 26054 * Make a current version download with a fixed URL for scripting 26055 * Ignore rmdir in limited fs rather than throwing error 26056 * dropbox 26057 * Increase chunk size to improve upload speeds massively 26058 * Issue an error message when trying to upload bad file name 26059 26060 ## v1.18 - 2015-08-17 26061 26062 * drive 26063 * Add `--drive-use-trash` flag so rclone trashes instead of deletes 26064 * Add "Forbidden to download" message for files with no downloadURL 26065 * dropbox 26066 * Remove datastore 26067 * This was deprecated and it caused a lot of problems 26068 * Modification times and MD5SUMs no longer stored 26069 * Fix uploading files > 2GB 26070 * s3 26071 * use official AWS SDK from github.com/aws/aws-sdk-go 26072 * **NB** will most likely require you to delete and recreate remote 26073 * enable multipart upload which enables files > 5GB 26074 * tested with Ceph / RadosGW / S3 emulation 26075 * many thanks to Sam Liston and Brian Haymore at the [Utah Center for High Performance Computing](https://www.chpc.utah.edu/) for a Ceph test account 26076 * misc 26077 * Show errors when reading the config file 26078 * Do not print stats in quiet mode - thanks Leonid Shalupov 26079 * Add FAQ 26080 * Fix created directories not obeying umask 26081 * Linux installation instructions - thanks Shimon Doodkin 26082 26083 ## v1.17 - 2015-06-14 26084 26085 * dropbox: fix case insensitivity issues - thanks Leonid Shalupov 26086 26087 ## v1.16 - 2015-06-09 26088 26089 * Fix uploading big files which was causing timeouts or panics 26090 * Don't check md5sum after download with --size-only 26091 26092 ## v1.15 - 2015-06-06 26093 26094 * Add --checksum flag to only discard transfers by MD5SUM - thanks Alex Couper 26095 * Implement --size-only flag to sync on size not checksum & modtime 26096 * Expand docs and remove duplicated information 26097 * Document rclone's limitations with directories 26098 * dropbox: update docs about case insensitivity 26099 26100 ## v1.14 - 2015-05-21 26101 26102 * local: fix encoding of non utf-8 file names - fixes a duplicate file problem 26103 * drive: docs about rate limiting 26104 * google cloud storage: Fix compile after API change in "google.golang.org/api/storage/v1" 26105 26106 ## v1.13 - 2015-05-10 26107 26108 * Revise documentation (especially sync) 26109 * Implement --timeout and --conntimeout 26110 * s3: ignore etags from multipart uploads which aren't md5sums 26111 26112 ## v1.12 - 2015-03-15 26113 26114 * drive: Use chunked upload for files above a certain size 26115 * drive: add --drive-chunk-size and --drive-upload-cutoff parameters 26116 * drive: switch to insert from update when a failed copy deletes the upload 26117 * core: Log duplicate files if they are detected 26118 26119 ## v1.11 - 2015-03-04 26120 26121 * swift: add region parameter 26122 * drive: fix crash on failed to update remote mtime 26123 * In remote paths, change native directory separators to / 26124 * Add synchronization to ls/lsl/lsd output to stop corruptions 26125 * Ensure all stats/log messages to go stderr 26126 * Add --log-file flag to log everything (including panics) to file 26127 * Make it possible to disable stats printing with --stats=0 26128 * Implement --bwlimit to limit data transfer bandwidth 26129 26130 ## v1.10 - 2015-02-12 26131 26132 * s3: list an unlimited number of items 26133 * Fix getting stuck in the configurator 26134 26135 ## v1.09 - 2015-02-07 26136 26137 * windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:) 26138 * local: Fix directory separators on Windows 26139 * drive: fix rate limit exceeded errors 26140 26141 ## v1.08 - 2015-02-04 26142 26143 * drive: fix subdirectory listing to not list entire drive 26144 * drive: Fix SetModTime 26145 * dropbox: adapt code to recent library changes 26146 26147 ## v1.07 - 2014-12-23 26148 26149 * google cloud storage: fix memory leak 26150 26151 ## v1.06 - 2014-12-12 26152 26153 * Fix "Couldn't find home directory" on OSX 26154 * swift: Add tenant parameter 26155 * Use new location of Google API packages 26156 26157 ## v1.05 - 2014-08-09 26158 26159 * Improved tests and consequently lots of minor fixes 26160 * core: Fix race detected by go race detector 26161 * core: Fixes after running errcheck 26162 * drive: reset root directory on Rmdir and Purge 26163 * fs: Document that Purger returns error on empty directory, test and fix 26164 * google cloud storage: fix ListDir on subdirectory 26165 * google cloud storage: re-read metadata in SetModTime 26166 * s3: make reading metadata more reliable to work around eventual consistency problems 26167 * s3: strip trailing / from ListDir() 26168 * swift: return directories without / in ListDir 26169 26170 ## v1.04 - 2014-07-21 26171 26172 * google cloud storage: Fix crash on Update 26173 26174 ## v1.03 - 2014-07-20 26175 26176 * swift, s3, dropbox: fix updated files being marked as corrupted 26177 * Make compile with go 1.1 again 26178 26179 ## v1.02 - 2014-07-19 26180 26181 * Implement Dropbox remote 26182 * Implement Google Cloud Storage remote 26183 * Verify Md5sums and Sizes after copies 26184 * Remove times from "ls" command - lists sizes only 26185 * Add add "lsl" - lists times and sizes 26186 * Add "md5sum" command 26187 26188 ## v1.01 - 2014-07-04 26189 26190 * drive: fix transfer of big files using up lots of memory 26191 26192 ## v1.00 - 2014-07-03 26193 26194 * drive: fix whole second dates 26195 26196 ## v0.99 - 2014-06-26 26197 26198 * Fix --dry-run not working 26199 * Make compatible with go 1.1 26200 26201 ## v0.98 - 2014-05-30 26202 26203 * s3: Treat missing Content-Length as 0 for some ceph installations 26204 * rclonetest: add file with a space in 26205 26206 ## v0.97 - 2014-05-05 26207 26208 * Implement copying of single files 26209 * s3 & swift: support paths inside containers/buckets 26210 26211 ## v0.96 - 2014-04-24 26212 26213 * drive: Fix multiple files of same name being created 26214 * drive: Use o.Update and fs.Put to optimise transfers 26215 * Add version number, -V and --version 26216 26217 ## v0.95 - 2014-03-28 26218 26219 * rclone.org: website, docs and graphics 26220 * drive: fix path parsing 26221 26222 ## v0.94 - 2014-03-27 26223 26224 * Change remote format one last time 26225 * GNU style flags 26226 26227 ## v0.93 - 2014-03-16 26228 26229 * drive: store token in config file 26230 * cross compile other versions 26231 * set strict permissions on config file 26232 26233 ## v0.92 - 2014-03-15 26234 26235 * Config fixes and --config option 26236 26237 ## v0.91 - 2014-03-15 26238 26239 * Make config file 26240 26241 ## v0.90 - 2013-06-27 26242 26243 * Project named rclone 26244 26245 ## v0.00 - 2012-11-18 26246 26247 * Project started 26248 26249 # Bugs and Limitations 26250 26251 ## Limitations 26252 26253 ### Directory timestamps aren't preserved 26254 26255 Rclone doesn't currently preserve the timestamps of directories. This 26256 is because rclone only really considers objects when syncing. 26257 26258 ### Rclone struggles with millions of files in a directory 26259 26260 Currently rclone loads each directory entirely into memory before 26261 using it. Since each Rclone object takes 0.5k-1k of memory this can 26262 take a very long time and use an extremely large amount of memory. 26263 26264 Millions of files in a directory tend caused by software writing cloud 26265 storage (eg S3 buckets). 26266 26267 ### Bucket based remotes and folders 26268 26269 Bucket based remotes (eg S3/GCS/Swift/B2) do not have a concept of 26270 directories. Rclone therefore cannot create directories in them which 26271 means that empty directories on a bucket based remote will tend to 26272 disappear. 26273 26274 Some software creates empty keys ending in `/` as directory markers. 26275 Rclone doesn't do this as it potentially creates more objects and 26276 costs more. It may do in future (probably with a flag). 26277 26278 ## Bugs 26279 26280 Bugs are stored in rclone's GitHub project: 26281 26282 * [Reported bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug) 26283 * [Known issues](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Known+Problem%22) 26284 26285 Frequently Asked Questions 26286 -------------------------- 26287 26288 ### Do all cloud storage systems support all rclone commands ### 26289 26290 Yes they do. All the rclone commands (eg `sync`, `copy` etc) will 26291 work on all the remote storage systems. 26292 26293 ### Can I copy the config from one machine to another ### 26294 26295 Sure! Rclone stores all of its config in a single file. If you want 26296 to find this file, run `rclone config file` which will tell you where 26297 it is. 26298 26299 See the [remote setup docs](https://rclone.org/remote_setup/) for more info. 26300 26301 ### How do I configure rclone on a remote / headless box with no browser? ### 26302 26303 This has now been documented in its own [remote setup page](https://rclone.org/remote_setup/). 26304 26305 ### Can rclone sync directly from drive to s3 ### 26306 26307 Rclone can sync between two remote cloud storage systems just fine. 26308 26309 Note that it effectively downloads the file and uploads it again, so 26310 the node running rclone would need to have lots of bandwidth. 26311 26312 The syncs would be incremental (on a file by file basis). 26313 26314 Eg 26315 26316 rclone sync drive:Folder s3:bucket 26317 26318 26319 ### Using rclone from multiple locations at the same time ### 26320 26321 You can use rclone from multiple places at the same time if you choose 26322 different subdirectory for the output, eg 26323 26324 ``` 26325 Server A> rclone sync /tmp/whatever remote:ServerA 26326 Server B> rclone sync /tmp/whatever remote:ServerB 26327 ``` 26328 26329 If you sync to the same directory then you should use rclone copy 26330 otherwise the two instances of rclone may delete each other's files, eg 26331 26332 ``` 26333 Server A> rclone copy /tmp/whatever remote:Backup 26334 Server B> rclone copy /tmp/whatever remote:Backup 26335 ``` 26336 26337 The file names you upload from Server A and Server B should be 26338 different in this case, otherwise some file systems (eg Drive) may 26339 make duplicates. 26340 26341 ### Why doesn't rclone support partial transfers / binary diffs like rsync? ### 26342 26343 Rclone stores each file you transfer as a native object on the remote 26344 cloud storage system. This means that you can see the files you 26345 upload as expected using alternative access methods (eg using the 26346 Google Drive web interface). There is a 1:1 mapping between files on 26347 your hard disk and objects created in the cloud storage system. 26348 26349 Cloud storage systems (at least none I've come across yet) don't 26350 support partially uploading an object. You can't take an existing 26351 object, and change some bytes in the middle of it. 26352 26353 It would be possible to make a sync system which stored binary diffs 26354 instead of whole objects like rclone does, but that would break the 26355 1:1 mapping of files on your hard disk to objects in the remote cloud 26356 storage system. 26357 26358 All the cloud storage systems support partial downloads of content, so 26359 it would be possible to make partial downloads work. However to make 26360 this work efficiently this would require storing a significant amount 26361 of metadata, which breaks the desired 1:1 mapping of files to objects. 26362 26363 ### Can rclone do bi-directional sync? ### 26364 26365 No, not at present. rclone only does uni-directional sync from A -> 26366 B. It may do in the future though since it has all the primitives - it 26367 just requires writing the algorithm to do it. 26368 26369 ### Can I use rclone with an HTTP proxy? ### 26370 26371 Yes. rclone will follow the standard environment variables for 26372 proxies, similar to cURL and other programs. 26373 26374 In general the variables are called `http_proxy` (for services reached 26375 over `http`) and `https_proxy` (for services reached over `https`). Most 26376 public services will be using `https`, but you may wish to set both. 26377 26378 The content of the variable is `protocol://server:port`. The protocol 26379 value is the one used to talk to the proxy server, itself, and is commonly 26380 either `http` or `socks5`. 26381 26382 Slightly annoyingly, there is no _standard_ for the name; some applications 26383 may use `http_proxy` but another one `HTTP_PROXY`. The `Go` libraries 26384 used by `rclone` will try both variations, but you may wish to set all 26385 possibilities. So, on Linux, you may end up with code similar to 26386 26387 export http_proxy=http://proxyserver:12345 26388 export https_proxy=$http_proxy 26389 export HTTP_PROXY=$http_proxy 26390 export HTTPS_PROXY=$http_proxy 26391 26392 The `NO_PROXY` allows you to disable the proxy for specific hosts. 26393 Hosts must be comma separated, and can contain domains or parts. 26394 For instance "foo.com" also matches "bar.foo.com". 26395 26396 e.g. 26397 26398 export no_proxy=localhost,127.0.0.0/8,my.host.name 26399 export NO_PROXY=$no_proxy 26400 26401 Note that the ftp backend does not support `ftp_proxy` yet. 26402 26403 ### Rclone gives x509: failed to load system roots and no roots provided error ### 26404 26405 This means that `rclone` can't file the SSL root certificates. Likely 26406 you are running `rclone` on a NAS with a cut-down Linux OS, or 26407 possibly on Solaris. 26408 26409 Rclone (via the Go runtime) tries to load the root certificates from 26410 these places on Linux. 26411 26412 "/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc. 26413 "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 26414 "/etc/ssl/ca-bundle.pem", // OpenSUSE 26415 "/etc/pki/tls/cacert.pem", // OpenELEC 26416 26417 So doing something like this should fix the problem. It also sets the 26418 time which is important for SSL to work properly. 26419 26420 ``` 26421 mkdir -p /etc/ssl/certs/ 26422 curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 26423 ntpclient -s -h pool.ntp.org 26424 ``` 26425 26426 The two environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR`, mentioned in the [x509 package](https://godoc.org/crypto/x509), 26427 provide an additional way to provide the SSL root certificates. 26428 26429 Note that you may need to add the `--insecure` option to the `curl` command line if it doesn't work without. 26430 26431 ``` 26432 curl --insecure -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 26433 ``` 26434 26435 ### Rclone gives Failed to load config file: function not implemented error ### 26436 26437 Likely this means that you are running rclone on Linux version not 26438 supported by the go runtime, ie earlier than version 2.6.23. 26439 26440 See the [system requirements section in the go install 26441 docs](https://golang.org/doc/install) for full details. 26442 26443 ### All my uploaded docx/xlsx/pptx files appear as archive/zip ### 26444 26445 This is caused by uploading these files from a Windows computer which 26446 hasn't got the Microsoft Office suite installed. The easiest way to 26447 fix is to install the Word viewer and the Microsoft Office 26448 Compatibility Pack for Word, Excel, and PowerPoint 2007 and later 26449 versions' file formats 26450 26451 ### tcp lookup some.domain.com no such host ### 26452 26453 This happens when rclone cannot resolve a domain. Please check that 26454 your DNS setup is generally working, e.g. 26455 26456 ``` 26457 # both should print a long list of possible IP addresses 26458 dig www.googleapis.com # resolve using your default DNS 26459 dig www.googleapis.com @8.8.8.8 # resolve with Google's DNS server 26460 ``` 26461 26462 If you are using `systemd-resolved` (default on Arch Linux), ensure it 26463 is at version 233 or higher. Previous releases contain a bug which 26464 causes not all domains to be resolved properly. 26465 26466 Additionally with the `GODEBUG=netdns=` environment variable the Go 26467 resolver decision can be influenced. This also allows to resolve certain 26468 issues with DNS resolution. See the [name resolution section in the go docs](https://golang.org/pkg/net/#hdr-Name_Resolution). 26469 26470 ### The total size reported in the stats for a sync is wrong and keeps changing 26471 26472 It is likely you have more than 10,000 files that need to be 26473 synced. By default rclone only gets 10,000 files ahead in a sync so as 26474 not to use up too much memory. You can change this default with the 26475 [--max-backlog](https://rclone.org/docs/#max-backlog-n) flag. 26476 26477 ### Rclone is using too much memory or appears to have a memory leak 26478 26479 Rclone is written in Go which uses a garbage collector. The default 26480 settings for the garbage collector mean that it runs when the heap 26481 size has doubled. 26482 26483 However it is possible to tune the garbage collector to use less 26484 memory by [setting GOGC](https://dave.cheney.net/tag/gogc) to a lower 26485 value, say `export GOGC=20`. This will make the garbage collector 26486 work harder, reducing memory size at the expense of CPU usage. 26487 26488 The most common cause of rclone using lots of memory is a single 26489 directory with thousands or millions of files in. Rclone has to load 26490 this entirely into memory as rclone objects. Each rclone object takes 26491 0.5k-1k of memory. 26492 26493 License 26494 ------- 26495 26496 This is free software under the terms of MIT the license (check the 26497 COPYING file included with the source code). 26498 26499 ``` 26500 Copyright (C) 2019 by Nick Craig-Wood https://www.craig-wood.com/nick/ 26501 26502 Permission is hereby granted, free of charge, to any person obtaining a copy 26503 of this software and associated documentation files (the "Software"), to deal 26504 in the Software without restriction, including without limitation the rights 26505 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 26506 copies of the Software, and to permit persons to whom the Software is 26507 furnished to do so, subject to the following conditions: 26508 26509 The above copyright notice and this permission notice shall be included in 26510 all copies or substantial portions of the Software. 26511 26512 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 26513 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 26514 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 26515 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 26516 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 26517 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 26518 THE SOFTWARE. 26519 ``` 26520 26521 Authors 26522 ------- 26523 26524 * Nick Craig-Wood <nick@craig-wood.com> 26525 26526 Contributors 26527 ------------ 26528 26529 {{< rem `email addresses removed from here need to be addeed to 26530 bin/.ignore-emails to make sure update-authors.py doesn't immediately 26531 put them back in again.` >}} 26532 26533 * Alex Couper <amcouper@gmail.com> 26534 * Leonid Shalupov <leonid@shalupov.com> <shalupov@diverse.org.ru> 26535 * Shimon Doodkin <helpmepro1@gmail.com> 26536 * Colin Nicholson <colin@colinn.com> 26537 * Klaus Post <klauspost@gmail.com> 26538 * Sergey Tolmachev <tolsi.ru@gmail.com> 26539 * Adriano Aurélio Meirelles <adriano@atinge.com> 26540 * C. Bess <cbess@users.noreply.github.com> 26541 * Dmitry Burdeev <dibu28@gmail.com> 26542 * Joseph Spurrier <github@josephspurrier.com> 26543 * Björn Harrtell <bjorn@wololo.org> 26544 * Xavier Lucas <xavier.lucas@corp.ovh.com> 26545 * Werner Beroux <werner@beroux.com> 26546 * Brian Stengaard <brian@stengaard.eu> 26547 * Jakub Gedeon <jgedeon@sofi.com> 26548 * Jim Tittsler <jwt@onjapan.net> 26549 * Michal Witkowski <michal@improbable.io> 26550 * Fabian Ruff <fabian.ruff@sap.com> 26551 * Leigh Klotz <klotz@quixey.com> 26552 * Romain Lapray <lapray.romain@gmail.com> 26553 * Justin R. Wilson <jrw972@gmail.com> 26554 * Antonio Messina <antonio.s.messina@gmail.com> 26555 * Stefan G. Weichinger <office@oops.co.at> 26556 * Per Cederberg <cederberg@gmail.com> 26557 * Radek Šenfeld <rush@logic.cz> 26558 * Fredrik Fornwall <fredrik@fornwall.net> 26559 * Asko Tamm <asko@deekit.net> 26560 * xor-zz <xor@gstocco.com> 26561 * Tomasz Mazur <tmazur90@gmail.com> 26562 * Marco Paganini <paganini@paganini.net> 26563 * Felix Bünemann <buenemann@louis.info> 26564 * Durval Menezes <jmrclone@durval.com> 26565 * Luiz Carlos Rumbelsperger Viana <maxd13_luiz_carlos@hotmail.com> 26566 * Stefan Breunig <stefan-github@yrden.de> 26567 * Alishan Ladhani <ali-l@users.noreply.github.com> 26568 * 0xJAKE <0xJAKE@users.noreply.github.com> 26569 * Thibault Molleman <thibaultmol@users.noreply.github.com> 26570 * Scott McGillivray <scott.mcgillivray@gmail.com> 26571 * Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com> 26572 * Lukas Loesche <lukas@mesosphere.io> 26573 * emyarod <allllaboutyou@gmail.com> 26574 * T.C. Ferguson <tcf909@gmail.com> 26575 * Brandur <brandur@mutelight.org> 26576 * Dario Giovannetti <dev@dariogiovannetti.net> 26577 * Károly Oláh <okaresz@aol.com> 26578 * Jon Yergatian <jon@macfanatic.ca> 26579 * Jack Schmidt <github@mowsey.org> 26580 * Dedsec1 <Dedsec1@users.noreply.github.com> 26581 * Hisham Zarka <hzarka@gmail.com> 26582 * Jérôme Vizcaino <jerome.vizcaino@gmail.com> 26583 * Mike Tesch <mjt6129@rit.edu> 26584 * Marvin Watson <marvwatson@users.noreply.github.com> 26585 * Danny Tsai <danny8376@gmail.com> 26586 * Yoni Jah <yonjah+git@gmail.com> <yonjah+github@gmail.com> 26587 * Stephen Harris <github@spuddy.org> <sweharris@users.noreply.github.com> 26588 * Ihor Dvoretskyi <ihor.dvoretskyi@gmail.com> 26589 * Jon Craton <jncraton@gmail.com> 26590 * Hraban Luyat <hraban@0brg.net> 26591 * Michael Ledin <mledin89@gmail.com> 26592 * Martin Kristensen <me@azgul.com> 26593 * Too Much IO <toomuchio@users.noreply.github.com> 26594 * Anisse Astier <anisse@astier.eu> 26595 * Zahiar Ahmed <zahiar@live.com> 26596 * Igor Kharin <igorkharin@gmail.com> 26597 * Bill Zissimopoulos <billziss@navimatics.com> 26598 * Bob Potter <bobby.potter@gmail.com> 26599 * Steven Lu <tacticalazn@gmail.com> 26600 * Sjur Fredriksen <sjurtf@ifi.uio.no> 26601 * Ruwbin <hubus12345@gmail.com> 26602 * Fabian Möller <fabianm88@gmail.com> <f.moeller@nynex.de> 26603 * Edward Q. Bridges <github@eqbridges.com> 26604 * Vasiliy Tolstov <v.tolstov@selfip.ru> 26605 * Harshavardhana <harsha@minio.io> 26606 * sainaen <sainaen@gmail.com> 26607 * gdm85 <gdm85@users.noreply.github.com> 26608 * Yaroslav Halchenko <debian@onerussian.com> 26609 * John Papandriopoulos <jpap@users.noreply.github.com> 26610 * Zhiming Wang <zmwangx@gmail.com> 26611 * Andy Pilate <cubox@cubox.me> 26612 * Oliver Heyme <olihey@googlemail.com> <olihey@users.noreply.github.com> <de8olihe@lego.com> 26613 * wuyu <wuyu@yunify.com> 26614 * Andrei Dragomir <adragomi@adobe.com> 26615 * Christian Brüggemann <mail@cbruegg.com> 26616 * Alex McGrath Kraak <amkdude@gmail.com> 26617 * bpicode <bjoern.pirnay@googlemail.com> 26618 * Daniel Jagszent <daniel@jagszent.de> 26619 * Josiah White <thegenius2009@gmail.com> 26620 * Ishuah Kariuki <kariuki@ishuah.com> <ishuah91@gmail.com> 26621 * Jan Varho <jan@varho.org> 26622 * Girish Ramakrishnan <girish@cloudron.io> 26623 * LingMan <LingMan@users.noreply.github.com> 26624 * Jacob McNamee <jacobmcnamee@gmail.com> 26625 * jersou <jertux@gmail.com> 26626 * thierry <thierry@substantiel.fr> 26627 * Simon Leinen <simon.leinen@gmail.com> <ubuntu@s3-test.novalocal> 26628 * Dan Dascalescu <ddascalescu+github@gmail.com> 26629 * Jason Rose <jason@jro.io> 26630 * Andrew Starr-Bochicchio <a.starr.b@gmail.com> 26631 * John Leach <john@johnleach.co.uk> 26632 * Corban Raun <craun@instructure.com> 26633 * Pierre Carlson <mpcarl@us.ibm.com> 26634 * Ernest Borowski <er.borowski@gmail.com> 26635 * Remus Bunduc <remus.bunduc@gmail.com> 26636 * Iakov Davydov <iakov.davydov@unil.ch> <dav05.gith@myths.ru> 26637 * Jakub Tasiemski <tasiemski@gmail.com> 26638 * David Minor <dminor@saymedia.com> 26639 * Tim Cooijmans <cooijmans.tim@gmail.com> 26640 * Laurence <liuxy6@gmail.com> 26641 * Giovanni Pizzi <gio.piz@gmail.com> 26642 * Filip Bartodziej <filipbartodziej@gmail.com> 26643 * Jon Fautley <jon@dead.li> 26644 * lewapm <32110057+lewapm@users.noreply.github.com> 26645 * Yassine Imounachen <yassine256@gmail.com> 26646 * Chris Redekop <chris-redekop@users.noreply.github.com> <chris.redekop@gmail.com> 26647 * Jon Fautley <jon@adenoid.appstal.co.uk> 26648 * Will Gunn <WillGunn@users.noreply.github.com> 26649 * Lucas Bremgartner <lucas@bremis.ch> 26650 * Jody Frankowski <jody.frankowski@gmail.com> 26651 * Andreas Roussos <arouss1980@gmail.com> 26652 * nbuchanan <nbuchanan@utah.gov> 26653 * Durval Menezes <rclone@durval.com> 26654 * Victor <vb-github@viblo.se> 26655 * Mateusz <pabian.mateusz@gmail.com> 26656 * Daniel Loader <spicypixel@gmail.com> 26657 * David0rk <davidork@gmail.com> 26658 * Alexander Neumann <alexander@bumpern.de> 26659 * Giri Badanahatti <gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local> 26660 * Leo R. Lundgren <leo@finalresort.org> 26661 * wolfv <wolfv6@users.noreply.github.com> 26662 * Dave Pedu <dave@davepedu.com> 26663 * Stefan Lindblom <lindblom@spotify.com> 26664 * seuffert <oliver@seuffert.biz> 26665 * gbadanahatti <37121690+gbadanahatti@users.noreply.github.com> 26666 * Keith Goldfarb <barkofdelight@gmail.com> 26667 * Steve Kriss <steve@heptio.com> 26668 * Chih-Hsuan Yen <yan12125@gmail.com> 26669 * Alexander Neumann <fd0@users.noreply.github.com> 26670 * Matt Holt <mholt@users.noreply.github.com> 26671 * Eri Bastos <bastos.eri@gmail.com> 26672 * Michael P. Dubner <pywebmail@list.ru> 26673 * Antoine GIRARD <sapk@users.noreply.github.com> 26674 * Mateusz Piotrowski <mpp302@gmail.com> 26675 * Animosity022 <animosity22@users.noreply.github.com> <earl.texter@gmail.com> 26676 * Peter Baumgartner <pete@lincolnloop.com> 26677 * Craig Rachel <craig@craigrachel.com> 26678 * Michael G. Noll <miguno@users.noreply.github.com> 26679 * hensur <me@hensur.de> 26680 * Oliver Heyme <de8olihe@lego.com> 26681 * Richard Yang <richard@yenforyang.com> 26682 * Piotr Oleszczyk <piotr.oleszczyk@gmail.com> 26683 * Rodrigo <rodarima@gmail.com> 26684 * NoLooseEnds <NoLooseEnds@users.noreply.github.com> 26685 * Jakub Karlicek <jakub@karlicek.me> 26686 * John Clayton <john@codemonkeylabs.com> 26687 * Kasper Byrdal Nielsen <byrdal76@gmail.com> 26688 * Benjamin Joseph Dag <bjdag1234@users.noreply.github.com> 26689 * themylogin <themylogin@gmail.com> 26690 * Onno Zweers <onno.zweers@surfsara.nl> 26691 * Jasper Lievisse Adriaanse <jasper@humppa.nl> 26692 * sandeepkru <sandeep.ummadi@gmail.com> <sandeepkru@users.noreply.github.com> 26693 * HerrH <atomtigerzoo@users.noreply.github.com> 26694 * Andrew <4030760+sparkyman215@users.noreply.github.com> 26695 * dan smith <XX1011@gmail.com> 26696 * Oleg Kovalov <iamolegkovalov@gmail.com> 26697 * Ruben Vandamme <github-com-00ff86@vandamme.email> 26698 * Cnly <minecnly@gmail.com> 26699 * Andres Alvarez <1671935+kir4h@users.noreply.github.com> 26700 * reddi1 <xreddi@gmail.com> 26701 * Matt Tucker <matthewtckr@gmail.com> 26702 * Sebastian Bünger <buengese@gmail.com> 26703 * Martin Polden <mpolden@mpolden.no> 26704 * Alex Chen <Cnly@users.noreply.github.com> 26705 * Denis <deniskovpen@gmail.com> 26706 * bsteiss <35940619+bsteiss@users.noreply.github.com> 26707 * Cédric Connes <cedric.connes@gmail.com> 26708 * Dr. Tobias Quathamer <toddy15@users.noreply.github.com> 26709 * dcpu <42736967+dcpu@users.noreply.github.com> 26710 * Sheldon Rupp <me@shel.io> 26711 * albertony <12441419+albertony@users.noreply.github.com> 26712 * cron410 <cron410@gmail.com> 26713 * Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com> 26714 * Felix Brucker <felix@felixbrucker.com> 26715 * Santiago Rodríguez <scollazo@users.noreply.github.com> 26716 * Craig Miskell <craig.miskell@fluxfederation.com> 26717 * Antoine GIRARD <sapk@sapk.fr> 26718 * Joanna Marek <joanna.marek@u2i.com> 26719 * frenos <frenos@users.noreply.github.com> 26720 * ssaqua <ssaqua@users.noreply.github.com> 26721 * xnaas <me@xnaas.info> 26722 * Frantisek Fuka <fuka@fuxoft.cz> 26723 * Paul Kohout <pauljkohout@yahoo.com> 26724 * dcpu <43330287+dcpu@users.noreply.github.com> 26725 * jackyzy823 <jackyzy823@gmail.com> 26726 * David Haguenauer <ml@kurokatta.org> 26727 * teresy <hi.teresy@gmail.com> 26728 * buergi <patbuergi@gmx.de> 26729 * Florian Gamboeck <mail@floga.de> 26730 * Ralf Hemberger <10364191+rhemberger@users.noreply.github.com> 26731 * Scott Edlund <sedlund@users.noreply.github.com> 26732 * Erik Swanson <erik@retailnext.net> 26733 * Jake Coggiano <jake@stripe.com> 26734 * brused27 <brused27@noemailaddress> 26735 * Peter Kaminski <kaminski@istori.com> 26736 * Henry Ptasinski <henry@logout.com> 26737 * Alexander <kharkovalexander@gmail.com> 26738 * Garry McNulty <garrmcnu@gmail.com> 26739 * Mathieu Carbou <mathieu.carbou@gmail.com> 26740 * Mark Otway <mark@otway.com> 26741 * William Cocker <37018962+WilliamCocker@users.noreply.github.com> 26742 * François Leurent <131.js@cloudyks.org> 26743 * Arkadius Stefanski <arkste@gmail.com> 26744 * Jay <dev@jaygoel.com> 26745 * andrea rota <a@xelera.eu> 26746 * nicolov <nicolov@users.noreply.github.com> 26747 * Dario Guzik <dario@guzik.com.ar> 26748 * qip <qip@users.noreply.github.com> 26749 * yair@unicorn <yair@unicorn> 26750 * Matt Robinson <brimstone@the.narro.ws> 26751 * kayrus <kay.diam@gmail.com> 26752 * Rémy Léone <remy.leone@gmail.com> 26753 * Wojciech Smigielski <wojciech.hieronim.smigielski@gmail.com> 26754 * weetmuts <oehrstroem@gmail.com> 26755 * Jonathan <vanillajonathan@users.noreply.github.com> 26756 * James Carpenter <orbsmiv@users.noreply.github.com> 26757 * Vince <vince0villamora@gmail.com> 26758 * Nestar47 <47841759+Nestar47@users.noreply.github.com> 26759 * Six <brbsix@gmail.com> 26760 * Alexandru Bumbacea <alexandru.bumbacea@booking.com> 26761 * calisro <robert.calistri@gmail.com> 26762 * Dr.Rx <david.rey@nventive.com> 26763 * marcintustin <marcintustin@users.noreply.github.com> 26764 * jaKa Močnik <jaka@koofr.net> 26765 * Fionera <fionera@fionera.de> 26766 * Dan Walters <dan@walters.io> 26767 * Danil Semelenov <sgtpep@users.noreply.github.com> 26768 * xopez <28950736+xopez@users.noreply.github.com> 26769 * Ben Boeckel <mathstuf@gmail.com> 26770 * Manu <manu@snapdragon.cc> 26771 * Kyle E. Mitchell <kyle@kemitchell.com> 26772 * Gary Kim <gary@garykim.dev> 26773 * Jon <jonathn@github.com> 26774 * Jeff Quinn <jeffrey.quinn@bluevoyant.com> 26775 * Peter Berbec <peter@berbec.com> 26776 * didil <1284255+didil@users.noreply.github.com> 26777 * id01 <gaviniboom@gmail.com> 26778 * Robert Marko <robimarko@gmail.com> 26779 * Philip Harvey <32467456+pharveybattelle@users.noreply.github.com> 26780 * JorisE <JorisE@users.noreply.github.com> 26781 * garry415 <garry.415@gmail.com> 26782 * forgems <forgems@gmail.com> 26783 * Florian Apolloner <florian@apolloner.eu> 26784 * Aleksandar Janković <office@ajankovic.com> <ajankovic@users.noreply.github.com> 26785 * Maran <maran@protonmail.com> 26786 * nguyenhuuluan434 <nguyenhuuluan434@gmail.com> 26787 * Laura Hausmann <zotan@zotan.pw> <laura@hausmann.dev> 26788 * yparitcher <y@paritcher.com> 26789 * AbelThar <abela.tharen@gmail.com> 26790 * Matti Niemenmaa <matti.niemenmaa+git@iki.fi> 26791 * Russell Davis <russelldavis@users.noreply.github.com> 26792 * Yi FU <yi.fu@tink.se> 26793 * Paul Millar <paul.millar@desy.de> 26794 * justinalin <justinalin@qnap.com> 26795 * EliEron <subanimehd@gmail.com> 26796 * justina777 <chiahuei.lin@gmail.com> 26797 * Chaitanya Bankanhal <bchaitanya15@gmail.com> 26798 * Michał Matczuk <michal@scylladb.com> 26799 * Macavirus <macavirus@zoho.com> 26800 * Abhinav Sharma <abhi18av@users.noreply.github.com> 26801 * ginvine <34869051+ginvine@users.noreply.github.com> 26802 * Patrick Wang <mail6543210@yahoo.com.tw> 26803 * Cenk Alti <cenkalti@gmail.com> 26804 * Andreas Chlupka <andy@chlupka.com> 26805 * Alfonso Montero <amontero@tinet.org> 26806 * Ivan Andreev <ivandeex@gmail.com> 26807 * David Baumgold <david@davidbaumgold.com> 26808 * Lars Lehtonen <lars.lehtonen@gmail.com> 26809 * Matei David <matei.david@gmail.com> 26810 * David <david.bramwell@endemolshine.com> 26811 * Anthony Rusdi <33247310+antrusd@users.noreply.github.com> 26812 * Richard Patel <me@terorie.dev> 26813 * 庄天翼 <zty0826@gmail.com> 26814 * SwitchJS <dev@switchjs.com> 26815 * Raphael <PowershellNinja@users.noreply.github.com> 26816 * Sezal Agrawal <sezalagrawal@gmail.com> 26817 * Tyler <TylerNakamura@users.noreply.github.com> 26818 * Brett Dutro <brett.dutro@gmail.com> 26819 * Vighnesh SK <booterror99@gmail.com> 26820 * Arijit Biswas <dibbyo456@gmail.com> 26821 * Michele Caci <michele.caci@gmail.com> 26822 * AlexandrBoltris <ua2fgb@gmail.com> 26823 * Bryce Larson <blarson@saltstack.com> 26824 * Carlos Ferreyra <crypticmind@gmail.com> 26825 * Saksham Khanna <sakshamkhanna@outlook.com> 26826 * dausruddin <5763466+dausruddin@users.noreply.github.com> 26827 * zero-24 <zero-24@users.noreply.github.com> 26828 * Xiaoxing Ye <ye@xiaoxing.us> 26829 * Barry Muldrey <barry@muldrey.net> 26830 * Sebastian Brandt <sebastian.brandt@friday.de> 26831 * Marco Molteni <marco.molteni@mailbox.org> 26832 * Ankur Gupta <ankur0493@gmail.com> <7876747+ankur0493@users.noreply.github.com> 26833 * Maciej Zimnoch <maciej@scylladb.com> 26834 * anuar45 <serdaliyev.anuar@gmail.com> 26835 * Fernando <ferferga@users.noreply.github.com> 26836 * David Cole <david.cole@sohonet.com> 26837 * Wei He <git@weispot.com> 26838 * Outvi V <19144373+outloudvi@users.noreply.github.com> 26839 * Thomas Kriechbaumer <thomas@kriechbaumer.name> 26840 * Tennix <tennix@users.noreply.github.com> 26841 * Ole Schütt <ole@schuett.name> 26842 * Kuang-che Wu <kcwu@csie.org> 26843 * Thomas Eales <wingsuit@users.noreply.github.com> 26844 * Paul Tinsley <paul.tinsley@vitalsource.com> 26845 * Felix Hungenberg <git@shiftgeist.com> 26846 * Benjamin Richter <github@dev.telepath.de> 26847 * landall <cst_zf@qq.com> 26848 * thestigma <thestigma@gmail.com> 26849 * jtagcat <38327267+jtagcat@users.noreply.github.com> 26850 * Damon Permezel <permezel@me.com> 26851 * boosh <boosh@users.noreply.github.com> 26852 * unbelauscht <58393353+unbelauscht@users.noreply.github.com> 26853 * Motonori IWAMURO <vmi@nifty.com> 26854 * Benjapol Worakan <benwrk@live.com> 26855 * Dave Koston <dave.koston@stackpath.com> 26856 * Durval Menezes <DurvalMenezes@users.noreply.github.com> 26857 * Tim Gallant <me@timgallant.us> 26858 * Frederick Zhang <frederick888@tsundere.moe> 26859 * valery1707 <valery1707@gmail.com> 26860 * Yves G <theYinYeti@yalis.fr> 26861 * Shing Kit Chan <chanshingkit@gmail.com> 26862 * Franklyn Tackitt <franklyn@tackitt.net> 26863 * Robert-André Mauchin <zebob.m@gmail.com> 26864 * evileye <48332831+ibiruai@users.noreply.github.com> 26865 * Joachim Brandon LeBlanc <brandon@leblanc.codes> 26866 * Patryk Jakuszew <patryk.jakuszew@gmail.com> 26867 * fishbullet <shindu666@gmail.com> 26868 * greatroar <@> 26869 * Bernd Schoolmann <mail@quexten.com> 26870 * Elan Ruusamäe <glen@pld-linux.org> 26871 * Max Sum <max@lolyculture.com> 26872 * Mark Spieth <mspieth@users.noreply.github.com> 26873 * harry <me@harry.plus> 26874 * Samantha McVey <samantham@posteo.net> 26875 * Jack Anderson <jack.anderson@metaswitch.com> 26876 * Michael G <draget@speciesm.net> 26877 * Brandon Philips <brandon@ifup.org> 26878 * Daven <dooven@users.noreply.github.com> 26879 * Martin Stone <martin@d7415.co.uk> 26880 * David Bramwell <13053834+dbramwell@users.noreply.github.com> 26881 * Sunil Patra <snl_su@live.com> 26882 * Adam Stroud <adam.stroud@gmail.com> 26883 * Kush <kushsharma@users.noreply.github.com> 26884 * Matan Rosenberg <matan129@gmail.com> 26885 * gitch1 <63495046+gitch1@users.noreply.github.com> 26886 * ElonH <elonhhuang@gmail.com> 26887 * Fred <fred@creativeprojects.tech> 26888 * Sébastien Gross <renard@users.noreply.github.com> 26889 * Maxime Suret <11944422+msuret@users.noreply.github.com> 26890 * Caleb Case <caleb@storj.io> 26891 * Ben Zenker <imbenzenker@gmail.com> 26892 * Martin Michlmayr <tbm@cyrius.com> 26893 * Brandon McNama <bmcnama@pagerduty.com> 26894 * Daniel Slyman <github@skylayer.eu> 26895 26896 # Contact the rclone project # 26897 26898 ## Forum ## 26899 26900 Forum for questions and general discussion: 26901 26902 * https://forum.rclone.org 26903 26904 ## GitHub repository ## 26905 26906 The project's repository is located at: 26907 26908 * https://github.com/rclone/rclone 26909 26910 There you can file bug reports or contribute with pull requests. 26911 26912 ## Twitter ## 26913 26914 You can also follow me on twitter for rclone announcements: 26915 26916 * [@njcw](https://twitter.com/njcw) 26917 26918 ## Email ## 26919 26920 Or if all else fails or you want to ask something private or 26921 confidential email [Nick Craig-Wood](mailto:nick@craig-wood.com). 26922 Please don't email me requests for help - those are better directed to 26923 the forum. Thanks! 26924