github.com/rclone/rclone@v1.66.1-0.20240517100346-7b89735ae726/MANUAL.txt (about) 1 rclone(1) User Manual 2 Nick Craig-Wood 3 Mar 10, 2024 4 5 Rclone syncs your files to cloud storage 6 7 - About rclone 8 - What can rclone do for you? 9 - What features does rclone have? 10 - What providers does rclone support? 11 - Download 12 - Install 13 - Donate. 14 15 About rclone 16 17 Rclone is a command-line program to manage files on cloud storage. It is 18 a feature-rich alternative to cloud vendors' web storage interfaces. 19 Over 70 cloud storage products support rclone including S3 object 20 stores, business & consumer file storage services, as well as standard 21 transfer protocols. 22 23 Rclone has powerful cloud equivalents to the unix commands rsync, cp, 24 mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax 25 includes shell pipeline support, and --dry-run protection. It is used at 26 the command line, in scripts or via its API. 27 28 Users call rclone "The Swiss army knife of cloud storage", and 29 "Technology indistinguishable from magic". 30 31 Rclone really looks after your data. It preserves timestamps and 32 verifies checksums at all times. Transfers over limited bandwidth; 33 intermittent connections, or subject to quota can be restarted, from the 34 last good file transferred. You can check the integrity of your files. 35 Where possible, rclone employs server-side transfers to minimise local 36 bandwidth use and transfers from one provider to another without using 37 local disk. 38 39 Virtual backends wrap local and cloud file systems to apply encryption, 40 compression, chunking, hashing and joining. 41 42 Rclone mounts any local, cloud or virtual filesystem as a disk on 43 Windows, macOS, linux and FreeBSD, and also serves these over SFTP, 44 HTTP, WebDAV, FTP and DLNA. 45 46 Rclone is mature, open-source software originally inspired by rsync and 47 written in Go. The friendly support community is familiar with varied 48 use cases. Official Ubuntu, Debian, Fedora, Brew and Chocolatey repos. 49 include rclone. For the latest version downloading from rclone.org is 50 recommended. 51 52 Rclone is widely used on Linux, Windows and Mac. Third-party developers 53 create innovative backup, restore, GUI and business process solutions 54 using the rclone command line or API. 55 56 Rclone does the heavy lifting of communicating with cloud storage. 57 58 What can rclone do for you? 59 60 Rclone helps you: 61 62 - Backup (and encrypt) files to cloud storage 63 - Restore (and decrypt) files from cloud storage 64 - Mirror cloud data to other cloud services or locally 65 - Migrate data to the cloud, or between cloud storage vendors 66 - Mount multiple, encrypted, cached or diverse cloud storage as a disk 67 - Analyse and account for data held on cloud storage using lsf, ljson, 68 size, ncdu 69 - Union file systems together to present multiple local and/or cloud 70 file systems as one 71 72 Features 73 74 - Transfers 75 - MD5, SHA1 hashes are checked at all times for file integrity 76 - Timestamps are preserved on files 77 - Operations can be restarted at any time 78 - Can be to and from network, e.g. two different cloud providers 79 - Can use multi-threaded downloads to local disk 80 - Copy new or changed files to cloud storage 81 - Sync (one way) to make a directory identical 82 - Bisync (two way) to keep two directories in sync bidirectionally 83 - Move files to cloud storage deleting the local after verification 84 - Check hashes and for missing/extra files 85 - Mount your cloud storage as a network disk 86 - Serve local or remote files over HTTP/WebDav/FTP/SFTP/DLNA 87 - Experimental Web based GUI 88 89 Supported providers 90 91 (There are many others, built on standard protocols such as WebDAV or 92 S3, that work out of the box.) 93 94 - 1Fichier 95 - Akamai Netstorage 96 - Alibaba Cloud (Aliyun) Object Storage System (OSS) 97 - Amazon S3 98 - Backblaze B2 99 - Box 100 - Ceph 101 - China Mobile Ecloud Elastic Object Storage (EOS) 102 - Arvan Cloud Object Storage (AOS) 103 - Citrix ShareFile 104 - Cloudflare R2 105 - DigitalOcean Spaces 106 - Digi Storage 107 - Dreamhost 108 - Dropbox 109 - Enterprise File Fabric 110 - Fastmail Files 111 - FTP 112 - Google Cloud Storage 113 - Google Drive 114 - Google Photos 115 - HDFS 116 - Hetzner Storage Box 117 - HiDrive 118 - HTTP 119 - ImageKit 120 - Internet Archive 121 - Jottacloud 122 - IBM COS S3 123 - IDrive e2 124 - IONOS Cloud 125 - Koofr 126 - Leviia Object Storage 127 - Liara Object Storage 128 - Linkbox 129 - Linode Object Storage 130 - Mail.ru Cloud 131 - Memset Memstore 132 - Mega 133 - Memory 134 - Microsoft Azure Blob Storage 135 - Microsoft Azure Files Storage 136 - Microsoft OneDrive 137 - Minio 138 - Nextcloud 139 - OVH 140 - Blomp Cloud Storage 141 - OpenDrive 142 - OpenStack Swift 143 - Oracle Cloud Storage Swift 144 - Oracle Object Storage 145 - ownCloud 146 - pCloud 147 - Petabox 148 - PikPak 149 - premiumize.me 150 - put.io 151 - Proton Drive 152 - QingStor 153 - Qiniu Cloud Object Storage (Kodo) 154 - Quatrix by Maytech 155 - Rackspace Cloud Files 156 - rsync.net 157 - Scaleway 158 - Seafile 159 - Seagate Lyve Cloud 160 - SeaweedFS 161 - SFTP 162 - Sia 163 - SMB / CIFS 164 - StackPath 165 - Storj 166 - Synology 167 - SugarSync 168 - Tencent Cloud Object Storage (COS) 169 - Uptobox 170 - Wasabi 171 - WebDAV 172 - Yandex Disk 173 - Zoho WorkDrive 174 - The local filesystem 175 176 Virtual providers 177 178 These backends adapt or modify other storage providers: 179 180 - Alias: Rename existing remotes 181 - Cache: Cache remotes (DEPRECATED) 182 - Chunker: Split large files 183 - Combine: Combine multiple remotes into a directory tree 184 - Compress: Compress files 185 - Crypt: Encrypt files 186 - Hasher: Hash files 187 - Union: Join multiple remotes to work together 188 189 Links 190 191 - Home page 192 - GitHub project page for source and bug tracker 193 - Rclone Forum 194 - Downloads 195 196 Install 197 198 Rclone is a Go program and comes as a single binary file. 199 200 Quickstart 201 202 - Download the relevant binary. 203 - Extract the rclone executable, rclone.exe on Windows, from the 204 archive. 205 - Run rclone config to setup. See rclone config docs for more details. 206 - Optionally configure automatic execution. 207 208 See below for some expanded Linux / macOS / Windows instructions. 209 210 See the usage docs for how to use rclone, or run rclone -h. 211 212 Already installed rclone can be easily updated to the latest version 213 using the rclone selfupdate command. 214 215 See the release signing docs for how to verify signatures on the 216 release. 217 218 Script installation 219 220 To install rclone on Linux/macOS/BSD systems, run: 221 222 sudo -v ; curl https://rclone.org/install.sh | sudo bash 223 224 For beta installation, run: 225 226 sudo -v ; curl https://rclone.org/install.sh | sudo bash -s beta 227 228 Note that this script checks the version of rclone installed first and 229 won't re-download if not needed. 230 231 Linux installation 232 233 Precompiled binary 234 235 Fetch and unpack 236 237 curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip 238 unzip rclone-current-linux-amd64.zip 239 cd rclone-*-linux-amd64 240 241 Copy binary file 242 243 sudo cp rclone /usr/bin/ 244 sudo chown root:root /usr/bin/rclone 245 sudo chmod 755 /usr/bin/rclone 246 247 Install manpage 248 249 sudo mkdir -p /usr/local/share/man/man1 250 sudo cp rclone.1 /usr/local/share/man/man1/ 251 sudo mandb 252 253 Run rclone config to setup. See rclone config docs for more details. 254 255 rclone config 256 257 macOS installation 258 259 Installation with brew 260 261 brew install rclone 262 263 NOTE: This version of rclone will not support mount any more (see 264 #5373). If mounting is wanted on macOS, either install a precompiled 265 binary or enable the relevant option when installing from source. 266 267 Note that this is a third party installer not controlled by the rclone 268 developers so it may be out of date. Its current version is as below. 269 270 [Homebrew package] 271 272 Installation with MacPorts (#macos-macports) 273 274 On macOS, rclone can also be installed via MacPorts: 275 276 sudo port install rclone 277 278 Note that this is a third party installer not controlled by the rclone 279 developers so it may be out of date. Its current version is as below. 280 281 [MacPorts port] 282 283 More information here. 284 285 Precompiled binary, using curl 286 287 To avoid problems with macOS gatekeeper enforcing the binary to be 288 signed and notarized it is enough to download with curl. 289 290 Download the latest version of rclone. 291 292 cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip 293 294 Unzip the download and cd to the extracted folder. 295 296 unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64 297 298 Move rclone to your $PATH. You will be prompted for your password. 299 300 sudo mkdir -p /usr/local/bin 301 sudo mv rclone /usr/local/bin/ 302 303 (the mkdir command is safe to run, even if the directory already 304 exists). 305 306 Remove the leftover files. 307 308 cd .. && rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip 309 310 Run rclone config to setup. See rclone config docs for more details. 311 312 rclone config 313 314 Precompiled binary, using a web browser 315 316 When downloading a binary with a web browser, the browser will set the 317 macOS gatekeeper quarantine attribute. Starting from Catalina, when 318 attempting to run rclone, a pop-up will appear saying: 319 320 "rclone" cannot be opened because the developer cannot be verified. 321 macOS cannot verify that this app is free from malware. 322 323 The simplest fix is to run 324 325 xattr -d com.apple.quarantine rclone 326 327 Windows installation 328 329 Precompiled binary 330 331 Fetch the correct binary for your processor type by clicking on these 332 links. If not sure, use the first link. 333 334 - Intel/AMD - 64 Bit 335 - Intel/AMD - 32 Bit 336 - ARM - 64 Bit 337 338 Open this file in the Explorer and extract rclone.exe. Rclone is a 339 portable executable so you can place it wherever is convenient. 340 341 Open a CMD window (or powershell) and run the binary. Note that rclone 342 does not launch a GUI by default, it runs in the CMD Window. 343 344 - Run rclone.exe config to setup. See rclone config docs for more 345 details. 346 - Optionally configure automatic execution. 347 348 If you are planning to use the rclone mount feature then you will need 349 to install the third party utility WinFsp also. 350 351 Windows package manager (Winget) 352 353 Winget comes pre-installed with the latest versions of Windows. If not, 354 update the App Installer package from the Microsoft store. 355 356 To install rclone 357 358 winget install Rclone.Rclone 359 360 To uninstall rclone 361 362 winget uninstall Rclone.Rclone --force 363 364 Chocolatey package manager 365 366 Make sure you have Choco installed 367 368 choco search rclone 369 choco install rclone 370 371 This will install rclone on your Windows machine. If you are planning to 372 use rclone mount then 373 374 choco install winfsp 375 376 will install that too. 377 378 Note that this is a third party installer not controlled by the rclone 379 developers so it may be out of date. Its current version is as below. 380 381 [Chocolatey package] 382 383 Scoop package manager 384 385 Make sure you have Scoop installed 386 387 scoop install rclone 388 389 Note that this is a third party installer not controlled by the rclone 390 developers so it may be out of date. Its current version is as below. 391 392 [Scoop package] 393 394 Package manager installation 395 396 Many Linux, Windows, macOS and other OS distributions package and 397 distribute rclone. 398 399 The distributed versions of rclone are often quite out of date and for 400 this reason we recommend one of the other installation methods if 401 possible. 402 403 You can get an idea of how up to date or not your OS distribution's 404 package is here. 405 406 [Packaging status] 407 408 Docker installation 409 410 The rclone developers maintain a docker image for rclone. 411 412 These images are built as part of the release process based on a minimal 413 Alpine Linux. 414 415 The :latest tag will always point to the latest stable release. You can 416 use the :beta tag to get the latest build from master. You can also use 417 version tags, e.g. :1.49.1, :1.49 or :1. 418 419 $ docker pull rclone/rclone:latest 420 latest: Pulling from rclone/rclone 421 Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11 422 ... 423 $ docker run --rm rclone/rclone:latest version 424 rclone v1.49.1 425 - os/arch: linux/amd64 426 - go version: go1.12.9 427 428 There are a few command line options to consider when starting an rclone 429 Docker container from the rclone image. 430 431 - You need to mount the host rclone config dir at /config/rclone into 432 the Docker container. Due to the fact that rclone updates tokens 433 inside its config file, and that the update process involves a file 434 rename, you need to mount the whole host rclone config dir, not just 435 the single host rclone config file. 436 437 - You need to mount a host data dir at /data into the Docker 438 container. 439 440 - By default, the rclone binary inside a Docker container runs with 441 UID=0 (root). As a result, all files created in a run will have 442 UID=0. If your config and data files reside on the host with a 443 non-root UID:GID, you need to pass these on the container start 444 command line. 445 446 - If you want to access the RC interface (either via the API or the 447 Web UI), it is required to set the --rc-addr to :5572 in order to 448 connect to it from outside the container. An explanation about why 449 this is necessary is present here. 450 451 - NOTE: Users running this container with the docker network set 452 to host should probably set it to listen to localhost only, with 453 127.0.0.1:5572 as the value for --rc-addr 454 455 - It is possible to use rclone mount inside a userspace Docker 456 container, and expose the resulting fuse mount to the host. The 457 exact docker run options to do that might vary slightly between 458 hosts. See, e.g. the discussion in this thread. 459 460 You also need to mount the host /etc/passwd and /etc/group for fuse 461 to work inside the container. 462 463 Here are some commands tested on an Ubuntu 18.04.3 host: 464 465 # config on host at ~/.config/rclone/rclone.conf 466 # data on host at ~/data 467 468 # add a remote interactively 469 docker run --rm -it \ 470 --volume ~/.config/rclone:/config/rclone \ 471 --user $(id -u):$(id -g) \ 472 rclone/rclone \ 473 config 474 475 # make sure the config is ok by listing the remotes 476 docker run --rm \ 477 --volume ~/.config/rclone:/config/rclone \ 478 --user $(id -u):$(id -g) \ 479 rclone/rclone \ 480 listremotes 481 482 # perform mount inside Docker container, expose result to host 483 mkdir -p ~/data/mount 484 docker run --rm \ 485 --volume ~/.config/rclone:/config/rclone \ 486 --volume ~/data:/data:shared \ 487 --user $(id -u):$(id -g) \ 488 --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \ 489 --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \ 490 rclone/rclone \ 491 mount dropbox:Photos /data/mount & 492 ls ~/data/mount 493 kill %1 494 495 Snap installation 496 497 [Get it from the Snap Store] 498 499 Make sure you have Snapd installed 500 501 $ sudo snap install rclone 502 503 Due to the strict confinement of Snap, rclone snap cannot access real 504 /home/$USER/.config/rclone directory, default config path is as below. 505 506 - Default config directory: 507 - /home/$USER/snap/rclone/current/.config/rclone 508 509 Note: Due to the strict confinement of Snap, rclone mount feature is not 510 supported. 511 512 If mounting is wanted, either install a precompiled binary or enable the 513 relevant option when installing from source. 514 515 Note that this is controlled by community maintainer not the rclone 516 developers so it may be out of date. Its current version is as below. 517 518 [rclone] 519 520 Source installation 521 522 Make sure you have git and Go installed. Go version 1.18 or newer is 523 required, the latest release is recommended. You can get it from your 524 package manager, or download it from golang.org/dl. Then you can run the 525 following: 526 527 git clone https://github.com/rclone/rclone.git 528 cd rclone 529 go build 530 531 This will check out the rclone source in subfolder rclone, which you can 532 later modify and send pull requests with. Then it will build the rclone 533 executable in the same folder. As an initial check you can now run 534 ./rclone version (.\rclone version on Windows). 535 536 Note that on macOS and Windows the mount command will not be available 537 unless you specify an additional build tag cmount. 538 539 go build -tags cmount 540 541 This assumes you have a GCC compatible C compiler (GCC or Clang) in your 542 PATH, as it uses cgo. But on Windows, the cgofuse library that the 543 cmount implementation is based on, also supports building without cgo, 544 i.e. by setting environment variable CGO_ENABLED to value 0 (static 545 linking). This is how the official Windows release of rclone is being 546 built, starting with version 1.59. It is still possible to build with 547 cgo on Windows as well, by using the MinGW port of GCC, e.g. by 548 installing it in a MSYS2 distribution (make sure you install it in the 549 classic mingw64 subsystem, the ucrt64 version is not compatible). 550 551 Additionally, to build with mount on Windows, you must install the third 552 party utility WinFsp, with the "Developer" feature selected. If building 553 with cgo, you must also set environment variable CPATH pointing to the 554 fuse include directory within the WinFsp installation (normally 555 C:\Program Files (x86)\WinFsp\inc\fuse). 556 557 You may add arguments -ldflags -s to omit symbol table and debug 558 information, making the executable file smaller, and -trimpath to remove 559 references to local file system paths. The official rclone releases are 560 built with both of these. 561 562 go build -trimpath -ldflags -s -tags cmount 563 564 If you want to customize the version string, as reported by the 565 rclone version command, you can set one of the variables fs.Version, 566 fs.VersionTag (to keep default suffix but customize the number), or 567 fs.VersionSuffix (to keep default number but customize the suffix). This 568 can be done from the build command, by adding to the -ldflags argument 569 value as shown below. 570 571 go build -trimpath -ldflags "-s -X github.com/rclone/rclone/fs.Version=v9.9.9-test" -tags cmount 572 573 On Windows, the official executables also have the version information, 574 as well as a file icon, embedded as binary resources. To get that with 575 your own build you need to run the following command before the build 576 command. It generates a Windows resource system object file, with 577 extension .syso, e.g. resource_windows_amd64.syso, that will be 578 automatically picked up by future build commands. 579 580 go run bin/resource_windows.go 581 582 The above command will generate a resource file containing version 583 information based on the fs.Version variable in source at the time you 584 run the command, which means if the value of this variable changes you 585 need to re-run the command for it to be reflected in the version 586 information. Also, if you override this version variable in the build 587 command as described above, you need to do that also when generating the 588 resource file, or else it will still use the value from the source. 589 590 go run bin/resource_windows.go -version v9.9.9-test 591 592 Instead of executing the go build command directly, you can run it via 593 the Makefile. The default target changes the version suffix from "-DEV" 594 to "-beta" followed by additional commit details, embeds version 595 information binary resources on Windows, and copies the resulting rclone 596 executable into your GOPATH bin folder ($(go env GOPATH)/bin, which 597 corresponds to ~/go/bin/rclone by default). 598 599 make 600 601 To include mount command on macOS and Windows with Makefile build: 602 603 make GOTAGS=cmount 604 605 There are other make targets that can be used for more advanced builds, 606 such as cross-compiling for all supported os/architectures, and 607 packaging results into release artifacts. See Makefile and 608 cross-compile.go for details. 609 610 Another alternative method for source installation is to download the 611 source, build and install rclone - all in one operation, as a regular Go 612 package. The source will be stored it in the Go module cache, and the 613 resulting executable will be in your GOPATH bin folder 614 ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default). 615 616 go install github.com/rclone/rclone@latest 617 618 Ansible installation 619 620 This can be done with Stefan Weichinger's ansible role. 621 622 Instructions 623 624 1. git clone https://github.com/stefangweichinger/ansible-rclone.git 625 into your local roles-directory 626 2. add the role to the hosts you want rclone installed to: 627 628 - hosts: rclone-hosts 629 roles: 630 - rclone 631 632 Portable installation 633 634 As mentioned above, rclone is single executable (rclone, or rclone.exe 635 on Windows) that you can download as a zip archive and extract into a 636 location of your choosing. When executing different commands, it may 637 create files in different locations, such as a configuration file and 638 various temporary files. By default the locations for these are 639 according to your operating system, e.g. configuration file in your user 640 profile directory and temporary files in the standard temporary 641 directory, but you can customize all of them, e.g. to make a completely 642 self-contained, portable installation. 643 644 Run the config paths command to see the locations that rclone will use. 645 646 To override them set the corresponding options (as command-line 647 arguments, or as environment variables): - --config - --cache-dir - 648 --temp-dir 649 650 Autostart 651 652 After installing and configuring rclone, as described above, you are 653 ready to use rclone as an interactive command line utility. If your goal 654 is to perform periodic operations, such as a regular sync, you will 655 probably want to configure your rclone command in your operating 656 system's scheduler. If you need to expose service-like features, such as 657 remote control, GUI, serve or mount, you will often want an rclone 658 command always running in the background, and configuring it to run in a 659 service infrastructure may be a better option. Below are some 660 alternatives on how to achieve this on different operating systems. 661 662 NOTE: Before setting up autorun it is highly recommended that you have 663 tested your command manually from a Command Prompt first. 664 665 Autostart on Windows 666 667 The most relevant alternatives for autostart on Windows are: - Run at 668 user log on using the Startup folder - Run at user log on, at system 669 startup or at schedule using Task Scheduler - Run at system startup 670 using Windows service 671 672 Running in background 673 674 Rclone is a console application, so if not starting from an existing 675 Command Prompt, e.g. when starting rclone.exe from a shortcut, it will 676 open a Command Prompt window. When configuring rclone to run from task 677 scheduler and windows service you are able to set it to run hidden in 678 background. From rclone version 1.54 you can also make it run hidden 679 from anywhere by adding option --no-console (it may still flash briefly 680 when the program starts). Since rclone normally writes information and 681 any error messages to the console, you must redirect this to a file to 682 be able to see it. Rclone has a built-in option --log-file for that. 683 684 Example command to run a sync in background: 685 686 c:\rclone\rclone.exe sync c:\files remote:/files --no-console --log-file c:\rclone\logs\sync_files.txt 687 688 User account 689 690 As mentioned in the mount documentation, mounted drives created as 691 Administrator are not visible to other accounts, not even the account 692 that was elevated as Administrator. By running the mount command as the 693 built-in SYSTEM user account, it will create drives accessible for 694 everyone on the system. Both scheduled task and Windows service can be 695 used to achieve this. 696 697 NOTE: Remember that when rclone runs as the SYSTEM user, the user 698 profile that it sees will not be yours. This means that if you normally 699 run rclone with configuration file in the default location, to be able 700 to use the same configuration when running as the system user you must 701 explicitly tell rclone where to find it with the --config option, or 702 else it will look in the system users profile path 703 (C:\Windows\System32\config\systemprofile). To test your command 704 manually from a Command Prompt, you can run it with the PsExec utility 705 from Microsoft's Sysinternals suite, which takes option -s to execute 706 commands as the SYSTEM user. 707 708 Start from Startup folder 709 710 To quickly execute an rclone command you can simply create a standard 711 Windows Explorer shortcut for the complete rclone command you want to 712 run. If you store this shortcut in the special "Startup" start-menu 713 folder, Windows will automatically run it at login. To open this folder 714 in Windows Explorer, enter path 715 %APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup, or 716 C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp if you want 717 the command to start for every user that logs in. 718 719 This is the easiest approach to autostarting of rclone, but it offers no 720 functionality to set it to run as different user, or to set conditions 721 or actions on certain events. Setting up a scheduled task as described 722 below will often give you better results. 723 724 Start from Task Scheduler 725 726 Task Scheduler is an administrative tool built into Windows, and it can 727 be used to configure rclone to be started automatically in a highly 728 configurable way, e.g. periodically on a schedule, on user log on, or at 729 system startup. It can run be configured to run as the current user, or 730 for a mount command that needs to be available to all users it can run 731 as the SYSTEM user. For technical information, see 732 https://docs.microsoft.com/windows/win32/taskschd/task-scheduler-start-page. 733 734 Run as service 735 736 For running rclone at system startup, you can create a Windows service 737 that executes your rclone command, as an alternative to scheduled task 738 configured to run at startup. 739 740 Mount command built-in service integration 741 742 For mount commands, rclone has a built-in Windows service integration 743 via the third-party WinFsp library it uses. Registering as a regular 744 Windows service easy, as you just have to execute the built-in 745 PowerShell command New-Service (requires administrative privileges). 746 747 Example of a PowerShell command that creates a Windows service for 748 mounting some remote:/files as drive letter X:, for all users (service 749 will be running as the local system account): 750 751 New-Service -Name Rclone -BinaryPathName 'c:\rclone\rclone.exe mount remote:/files X: --config c:\rclone\config\rclone.conf --log-file c:\rclone\logs\mount.txt' 752 753 The WinFsp service infrastructure supports incorporating services for 754 file system implementations, such as rclone, into its own launcher 755 service, as kind of "child services". This has the additional advantage 756 that it also implements a network provider that integrates into Windows 757 standard methods for managing network drives. This is currently not 758 officially supported by Rclone, but with WinFsp version 2019.3 B2 / 759 v1.5B2 or later it should be possible through path rewriting as 760 described here. 761 762 Third-party service integration 763 764 To Windows service running any rclone command, the excellent third-party 765 utility NSSM, the "Non-Sucking Service Manager", can be used. It 766 includes some advanced features such as adjusting process priority, 767 defining process environment variables, redirect to file anything 768 written to stdout, and customized response to different exit codes, with 769 a GUI to configure everything from (although it can also be used from 770 command line ). 771 772 There are also several other alternatives. To mention one more, WinSW, 773 "Windows Service Wrapper", is worth checking out. It requires .NET 774 Framework, but it is preinstalled on newer versions of Windows, and it 775 also provides alternative standalone distributions which includes 776 necessary runtime (.NET 5). WinSW is a command-line only utility, where 777 you have to manually create an XML file with service configuration. This 778 may be a drawback for some, but it can also be an advantage as it is 779 easy to back up and reuse the configuration settings, without having go 780 through manual steps in a GUI. One thing to note is that by default it 781 does not restart the service on error, one have to explicit enable this 782 in the configuration file (via the "onfailure" parameter). 783 784 Autostart on Linux 785 786 Start as a service 787 788 To always run rclone in background, relevant for mount commands etc, you 789 can use systemd to set up rclone as a system or user service. Running as 790 a system service ensures that it is run at startup even if the user it 791 is running as has no active session. Running rclone as a user service 792 ensures that it only starts after the configured user has logged into 793 the system. 794 795 Run periodically from cron 796 797 To run a periodic command, such as a copy/sync, you can set up a cron 798 job. 799 800 Usage 801 802 Rclone is a command line program to manage files on cloud storage. After 803 download and install, continue here to learn how to use it: Initial 804 configuration, what the basic syntax looks like, describes the various 805 subcommands, the various options, and more. 806 807 Configure 808 809 First, you'll need to configure rclone. As the object storage systems 810 have quite complicated authentication these are kept in a config file. 811 (See the --config entry for how to find the config file and choose its 812 location.) 813 814 The easiest way to make the config is to run rclone with the config 815 option: 816 817 rclone config 818 819 See the following for detailed instructions for 820 821 - 1Fichier 822 - Akamai Netstorage 823 - Alias 824 - Amazon S3 825 - Backblaze B2 826 - Box 827 - Chunker - transparently splits large files for other remotes 828 - Citrix ShareFile 829 - Compress 830 - Combine 831 - Crypt - to encrypt other remotes 832 - DigitalOcean Spaces 833 - Digi Storage 834 - Dropbox 835 - Enterprise File Fabric 836 - FTP 837 - Google Cloud Storage 838 - Google Drive 839 - Google Photos 840 - Hasher - to handle checksums for other remotes 841 - HDFS 842 - HiDrive 843 - HTTP 844 - Internet Archive 845 - Jottacloud 846 - Koofr 847 - Linkbox 848 - Mail.ru Cloud 849 - Mega 850 - Memory 851 - Microsoft Azure Blob Storage 852 - Microsoft Azure Files Storage 853 - Microsoft OneDrive 854 - OpenStack Swift / Rackspace Cloudfiles / Blomp Cloud Storage / 855 Memset Memstore 856 - OpenDrive 857 - Oracle Object Storage 858 - Pcloud 859 - PikPak 860 - premiumize.me 861 - put.io 862 - Proton Drive 863 - QingStor 864 - Quatrix by Maytech 865 - Seafile 866 - SFTP 867 - Sia 868 - SMB 869 - Storj 870 - SugarSync 871 - Union 872 - Uptobox 873 - WebDAV 874 - Yandex Disk 875 - Zoho WorkDrive 876 - The local filesystem 877 878 Basic syntax 879 880 Rclone syncs a directory tree from one storage system to another. 881 882 Its syntax is like this 883 884 Syntax: [options] subcommand <parameters> <parameters...> 885 886 Source and destination paths are specified by the name you gave the 887 storage system in the config file then the sub path, e.g. 888 "drive:myfolder" to look at "myfolder" in Google drive. 889 890 You can define as many storage paths as you like in the config file. 891 892 Please use the --interactive/-i flag while learning rclone to avoid 893 accidental data loss. 894 895 Subcommands 896 897 rclone uses a system of subcommands. For example 898 899 rclone ls remote:path # lists a remote 900 rclone copy /local/path remote:path # copies /local/path to the remote 901 rclone sync --interactive /local/path remote:path # syncs /local/path to the remote 902 903 rclone config 904 905 Enter an interactive configuration session. 906 907 Synopsis 908 909 Enter an interactive configuration session where you can setup new 910 remotes and manage existing ones. You may also set or remove a password 911 to protect your configuration. 912 913 rclone config [flags] 914 915 Options 916 917 -h, --help help for config 918 919 See the global flags page for global options not listed here. 920 921 SEE ALSO 922 923 - rclone - Show help for rclone commands, flags and backends. 924 - rclone config create - Create a new remote with name, type and 925 options. 926 - rclone config delete - Delete an existing remote. 927 - rclone config disconnect - Disconnects user from remote 928 - rclone config dump - Dump the config file as JSON. 929 - rclone config edit - Enter an interactive configuration session. 930 - rclone config file - Show path of configuration file in use. 931 - rclone config password - Update password in an existing remote. 932 - rclone config paths - Show paths used for configuration, cache, temp 933 etc. 934 - rclone config providers - List in JSON format all the providers and 935 options. 936 - rclone config reconnect - Re-authenticates user with remote. 937 - rclone config redacted - Print redacted (decrypted) config file, or 938 the redacted config for a single remote. 939 - rclone config show - Print (decrypted) config file, or the config 940 for a single remote. 941 - rclone config touch - Ensure configuration file exists. 942 - rclone config update - Update options in an existing remote. 943 - rclone config userinfo - Prints info about logged in user of remote. 944 945 rclone copy 946 947 Copy files from source to dest, skipping identical files. 948 949 Synopsis 950 951 Copy the source to the destination. Does not transfer files that are 952 identical on source and destination, testing by size and modification 953 time or MD5SUM. Doesn't delete files from the destination. If you want 954 to also delete files from destination, to make it match source, use the 955 sync command instead. 956 957 Note that it is always the contents of the directory that is synced, not 958 the directory itself. So when source:path is a directory, it's the 959 contents of source:path that are copied, not the directory name and 960 contents. 961 962 To copy single files, use the copyto command instead. 963 964 If dest:path doesn't exist, it is created and the source:path contents 965 go there. 966 967 For example 968 969 rclone copy source:sourcepath dest:destpath 970 971 Let's say there are two files in sourcepath 972 973 sourcepath/one.txt 974 sourcepath/two.txt 975 976 This copies them to 977 978 destpath/one.txt 979 destpath/two.txt 980 981 Not to 982 983 destpath/sourcepath/one.txt 984 destpath/sourcepath/two.txt 985 986 If you are familiar with rsync, rclone always works as if you had 987 written a trailing / - meaning "copy the contents of this directory". 988 This applies to all commands and whether you are talking about the 989 source or destination. 990 991 See the --no-traverse option for controlling whether rclone lists the 992 destination directory or not. Supplying this option when copying a small 993 number of files into a large destination can speed transfers up greatly. 994 995 For example, if you have many files in /path/to/src but only a few of 996 them change every day, you can copy all the files which have changed 997 recently very efficiently like this: 998 999 rclone copy --max-age 24h --no-traverse /path/to/src remote: 1000 1001 Rclone will sync the modification times of files and directories if the 1002 backend supports it. If metadata syncing is required then use the 1003 --metadata flag. 1004 1005 Note that the modification time and metadata for the root directory will 1006 not be synced. See https://github.com/rclone/rclone/issues/7652 for more 1007 info. 1008 1009 Note: Use the -P/--progress flag to view real-time transfer statistics. 1010 1011 Note: Use the --dry-run or the --interactive/-i flag to test without 1012 copying anything. 1013 1014 rclone copy source:path dest:path [flags] 1015 1016 Options 1017 1018 --create-empty-src-dirs Create empty source dirs on destination after copy 1019 -h, --help help for copy 1020 1021 Copy Options 1022 1023 Flags for anything which can Copy a file. 1024 1025 --check-first Do all the checks before starting transfers 1026 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 1027 --compare-dest stringArray Include additional comma separated server-side paths during comparison 1028 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 1029 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 1030 --ignore-case-sync Ignore case when synchronizing 1031 --ignore-checksum Skip post copy check of checksums 1032 --ignore-existing Skip all files that exist on destination 1033 --ignore-size Ignore size when skipping use modtime or checksum 1034 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 1035 --immutable Do not modify files, fail if existing files have been modified 1036 --inplace Download directly to destination file instead of atomic download to temp/rename 1037 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 1038 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 1039 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 1040 -M, --metadata If set, preserve metadata when copying objects 1041 --modify-window Duration Max time diff to be considered the same (default 1ns) 1042 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 1043 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 1044 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 1045 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 1046 --no-check-dest Don't check the destination, copy regardless 1047 --no-traverse Don't traverse destination file system on copy 1048 --no-update-dir-modtime Don't update directory modification times 1049 --no-update-modtime Don't update destination modtime if files identical 1050 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 1051 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 1052 --refresh-times Refresh the modtime of remote files 1053 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 1054 --size-only Skip based on size only, not modtime or checksum 1055 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 1056 -u, --update Skip files that are newer on the destination 1057 1058 Important Options 1059 1060 Important flags useful for most commands. 1061 1062 -n, --dry-run Do a trial run with no permanent changes 1063 -i, --interactive Enable interactive mode 1064 -v, --verbose count Print lots more stuff (repeat for more) 1065 1066 Filter Options 1067 1068 Flags for filtering directory listings. 1069 1070 --delete-excluded Delete files on dest excluded from sync 1071 --exclude stringArray Exclude files matching pattern 1072 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1073 --exclude-if-present stringArray Exclude directories if filename is present 1074 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1075 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1076 -f, --filter stringArray Add a file filtering rule 1077 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1078 --ignore-case Ignore case in filters (case insensitive) 1079 --include stringArray Include files matching pattern 1080 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1081 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1082 --max-depth int If set limits the recursion depth to this (default -1) 1083 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1084 --metadata-exclude stringArray Exclude metadatas matching pattern 1085 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1086 --metadata-filter stringArray Add a metadata filtering rule 1087 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1088 --metadata-include stringArray Include metadatas matching pattern 1089 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1090 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1091 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1092 1093 Listing Options 1094 1095 Flags for listing directories. 1096 1097 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1098 --fast-list Use recursive list if available; uses more memory but fewer transactions 1099 1100 See the global flags page for global options not listed here. 1101 1102 SEE ALSO 1103 1104 - rclone - Show help for rclone commands, flags and backends. 1105 1106 rclone sync 1107 1108 Make source and dest identical, modifying destination only. 1109 1110 Synopsis 1111 1112 Sync the source to the destination, changing the destination only. 1113 Doesn't transfer files that are identical on source and destination, 1114 testing by size and modification time or MD5SUM. Destination is updated 1115 to match source, including deleting files if necessary (except duplicate 1116 objects, see below). If you don't want to delete files from destination, 1117 use the copy command instead. 1118 1119 Important: Since this can cause data loss, test first with the --dry-run 1120 or the --interactive/-i flag. 1121 1122 rclone sync --interactive SOURCE remote:DESTINATION 1123 1124 Note that files in the destination won't be deleted if there were any 1125 errors at any point. Duplicate objects (files with the same name, on 1126 those providers that support it) are also not yet handled. 1127 1128 It is always the contents of the directory that is synced, not the 1129 directory itself. So when source:path is a directory, it's the contents 1130 of source:path that are copied, not the directory name and contents. See 1131 extended explanation in the copy command if unsure. 1132 1133 If dest:path doesn't exist, it is created and the source:path contents 1134 go there. 1135 1136 It is not possible to sync overlapping remotes. However, you may exclude 1137 the destination from the sync with a filter rule or by putting an 1138 exclude-if-present file inside the destination directory and sync to a 1139 destination that is inside the source directory. 1140 1141 Rclone will sync the modification times of files and directories if the 1142 backend supports it. If metadata syncing is required then use the 1143 --metadata flag. 1144 1145 Note that the modification time and metadata for the root directory will 1146 not be synced. See https://github.com/rclone/rclone/issues/7652 for more 1147 info. 1148 1149 Note: Use the -P/--progress flag to view real-time transfer statistics 1150 1151 Note: Use the rclone dedupe command to deal with "Duplicate 1152 object/directory found in source/destination - ignoring" errors. See 1153 this forum post for more info. 1154 1155 Logger Flags 1156 1157 The --differ, --missing-on-dst, --missing-on-src, --match and --error 1158 flags write paths, one per line, to the file name (or stdout if it is -) 1159 supplied. What they write is described in the help below. For example 1160 --differ will write all paths which are present on both the source and 1161 destination but different. 1162 1163 The --combined flag will write a file (or stdout) which contains all 1164 file paths with a symbol and then a space and then the path to tell you 1165 what happened to it. These are reminiscent of diff files. 1166 1167 - = path means path was found in source and destination and was 1168 identical 1169 - `- path` means path was missing on the source, so only in the 1170 destination 1171 - `+ path` means path was missing on the destination, so only in the 1172 source 1173 - `* path` means path was present in source and destination but 1174 different. 1175 - ! path means there was an error reading or hashing the source or 1176 dest. 1177 1178 The --dest-after flag writes a list file using the same format flags as 1179 lsf (including customizable options for hash, modtime, etc.) 1180 Conceptually it is similar to rsync's --itemize-changes, but not 1181 identical -- it should output an accurate list of what will be on the 1182 destination after the sync. 1183 1184 Note that these logger flags have a few limitations, and certain 1185 scenarios are not currently supported: 1186 1187 - --max-duration / CutoffModeHard 1188 - --compare-dest / --copy-dest 1189 - server-side moves of an entire dir at once 1190 - High-level retries, because there would be duplicates (use 1191 --retries 1 to disable) 1192 - Possibly some unusual error scenarios 1193 1194 Note also that each file is logged during the sync, as opposed to after, 1195 so it is most useful as a predictor of what SHOULD happen to each file 1196 (which may or may not match what actually DID.) 1197 1198 rclone sync source:path dest:path [flags] 1199 1200 Options 1201 1202 --absolute Put a leading / in front of path names 1203 --combined string Make a combined report of changes to this file 1204 --create-empty-src-dirs Create empty source dirs on destination after sync 1205 --csv Output in CSV format 1206 --dest-after string Report all files that exist on the dest post-sync 1207 --differ string Report all non-matching files to this file 1208 -d, --dir-slash Append a slash to directory names (default true) 1209 --dirs-only Only list directories 1210 --error string Report all files with errors (hashing or reading) to this file 1211 --files-only Only list files (default true) 1212 -F, --format string Output format - see lsf help for details (default "p") 1213 --hash h Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "md5") 1214 -h, --help help for sync 1215 --match string Report all matching files to this file 1216 --missing-on-dst string Report all files missing from the destination to this file 1217 --missing-on-src string Report all files missing from the source to this file 1218 -s, --separator string Separator for the items in the format (default ";") 1219 -t, --timeformat string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) 1220 1221 Copy Options 1222 1223 Flags for anything which can Copy a file. 1224 1225 --check-first Do all the checks before starting transfers 1226 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 1227 --compare-dest stringArray Include additional comma separated server-side paths during comparison 1228 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 1229 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 1230 --ignore-case-sync Ignore case when synchronizing 1231 --ignore-checksum Skip post copy check of checksums 1232 --ignore-existing Skip all files that exist on destination 1233 --ignore-size Ignore size when skipping use modtime or checksum 1234 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 1235 --immutable Do not modify files, fail if existing files have been modified 1236 --inplace Download directly to destination file instead of atomic download to temp/rename 1237 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 1238 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 1239 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 1240 -M, --metadata If set, preserve metadata when copying objects 1241 --modify-window Duration Max time diff to be considered the same (default 1ns) 1242 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 1243 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 1244 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 1245 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 1246 --no-check-dest Don't check the destination, copy regardless 1247 --no-traverse Don't traverse destination file system on copy 1248 --no-update-dir-modtime Don't update directory modification times 1249 --no-update-modtime Don't update destination modtime if files identical 1250 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 1251 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 1252 --refresh-times Refresh the modtime of remote files 1253 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 1254 --size-only Skip based on size only, not modtime or checksum 1255 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 1256 -u, --update Skip files that are newer on the destination 1257 1258 Sync Options 1259 1260 Flags just used for rclone sync. 1261 1262 --backup-dir string Make backups into hierarchy based in DIR 1263 --delete-after When synchronizing, delete files on destination after transferring (default) 1264 --delete-before When synchronizing, delete files on destination before transferring 1265 --delete-during When synchronizing, delete files during transfer 1266 --fix-case Force rename of case insensitive dest to match source 1267 --ignore-errors Delete even if there are I/O errors 1268 --max-delete int When synchronizing, limit the number of deletes (default -1) 1269 --max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off) 1270 --suffix string Suffix to add to changed files 1271 --suffix-keep-extension Preserve the extension when using --suffix 1272 --track-renames When synchronizing, track file renames and do a server-side move if possible 1273 --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash") 1274 1275 Important Options 1276 1277 Important flags useful for most commands. 1278 1279 -n, --dry-run Do a trial run with no permanent changes 1280 -i, --interactive Enable interactive mode 1281 -v, --verbose count Print lots more stuff (repeat for more) 1282 1283 Filter Options 1284 1285 Flags for filtering directory listings. 1286 1287 --delete-excluded Delete files on dest excluded from sync 1288 --exclude stringArray Exclude files matching pattern 1289 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1290 --exclude-if-present stringArray Exclude directories if filename is present 1291 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1292 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1293 -f, --filter stringArray Add a file filtering rule 1294 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1295 --ignore-case Ignore case in filters (case insensitive) 1296 --include stringArray Include files matching pattern 1297 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1298 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1299 --max-depth int If set limits the recursion depth to this (default -1) 1300 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1301 --metadata-exclude stringArray Exclude metadatas matching pattern 1302 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1303 --metadata-filter stringArray Add a metadata filtering rule 1304 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1305 --metadata-include stringArray Include metadatas matching pattern 1306 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1307 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1308 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1309 1310 Listing Options 1311 1312 Flags for listing directories. 1313 1314 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1315 --fast-list Use recursive list if available; uses more memory but fewer transactions 1316 1317 See the global flags page for global options not listed here. 1318 1319 SEE ALSO 1320 1321 - rclone - Show help for rclone commands, flags and backends. 1322 1323 rclone move 1324 1325 Move files from source to dest. 1326 1327 Synopsis 1328 1329 Moves the contents of the source directory to the destination directory. 1330 Rclone will error if the source and destination overlap and the remote 1331 does not support a server-side directory move operation. 1332 1333 To move single files, use the moveto command instead. 1334 1335 If no filters are in use and if possible this will server-side move 1336 source:path into dest:path. After this source:path will no longer exist. 1337 1338 Otherwise for each file in source:path selected by the filters (if any) 1339 this will move it into dest:path. If possible a server-side move will be 1340 used, otherwise it will copy it (server-side if possible) into dest:path 1341 then delete the original (if no errors on copy) in source:path. 1342 1343 If you want to delete empty source directories after move, use the 1344 --delete-empty-src-dirs flag. 1345 1346 See the --no-traverse option for controlling whether rclone lists the 1347 destination directory or not. Supplying this option when moving a small 1348 number of files into a large destination can speed transfers up greatly. 1349 1350 Rclone will sync the modification times of files and directories if the 1351 backend supports it. If metadata syncing is required then use the 1352 --metadata flag. 1353 1354 Note that the modification time and metadata for the root directory will 1355 not be synced. See https://github.com/rclone/rclone/issues/7652 for more 1356 info. 1357 1358 Important: Since this can cause data loss, test first with the --dry-run 1359 or the --interactive/-i flag. 1360 1361 Note: Use the -P/--progress flag to view real-time transfer statistics. 1362 1363 rclone move source:path dest:path [flags] 1364 1365 Options 1366 1367 --create-empty-src-dirs Create empty source dirs on destination after move 1368 --delete-empty-src-dirs Delete empty source dirs after move 1369 -h, --help help for move 1370 1371 Copy Options 1372 1373 Flags for anything which can Copy a file. 1374 1375 --check-first Do all the checks before starting transfers 1376 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 1377 --compare-dest stringArray Include additional comma separated server-side paths during comparison 1378 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 1379 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 1380 --ignore-case-sync Ignore case when synchronizing 1381 --ignore-checksum Skip post copy check of checksums 1382 --ignore-existing Skip all files that exist on destination 1383 --ignore-size Ignore size when skipping use modtime or checksum 1384 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 1385 --immutable Do not modify files, fail if existing files have been modified 1386 --inplace Download directly to destination file instead of atomic download to temp/rename 1387 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 1388 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 1389 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 1390 -M, --metadata If set, preserve metadata when copying objects 1391 --modify-window Duration Max time diff to be considered the same (default 1ns) 1392 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 1393 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 1394 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 1395 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 1396 --no-check-dest Don't check the destination, copy regardless 1397 --no-traverse Don't traverse destination file system on copy 1398 --no-update-dir-modtime Don't update directory modification times 1399 --no-update-modtime Don't update destination modtime if files identical 1400 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 1401 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 1402 --refresh-times Refresh the modtime of remote files 1403 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 1404 --size-only Skip based on size only, not modtime or checksum 1405 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 1406 -u, --update Skip files that are newer on the destination 1407 1408 Important Options 1409 1410 Important flags useful for most commands. 1411 1412 -n, --dry-run Do a trial run with no permanent changes 1413 -i, --interactive Enable interactive mode 1414 -v, --verbose count Print lots more stuff (repeat for more) 1415 1416 Filter Options 1417 1418 Flags for filtering directory listings. 1419 1420 --delete-excluded Delete files on dest excluded from sync 1421 --exclude stringArray Exclude files matching pattern 1422 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1423 --exclude-if-present stringArray Exclude directories if filename is present 1424 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1425 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1426 -f, --filter stringArray Add a file filtering rule 1427 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1428 --ignore-case Ignore case in filters (case insensitive) 1429 --include stringArray Include files matching pattern 1430 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1431 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1432 --max-depth int If set limits the recursion depth to this (default -1) 1433 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1434 --metadata-exclude stringArray Exclude metadatas matching pattern 1435 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1436 --metadata-filter stringArray Add a metadata filtering rule 1437 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1438 --metadata-include stringArray Include metadatas matching pattern 1439 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1440 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1441 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1442 1443 Listing Options 1444 1445 Flags for listing directories. 1446 1447 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1448 --fast-list Use recursive list if available; uses more memory but fewer transactions 1449 1450 See the global flags page for global options not listed here. 1451 1452 SEE ALSO 1453 1454 - rclone - Show help for rclone commands, flags and backends. 1455 1456 rclone delete 1457 1458 Remove the files in path. 1459 1460 Synopsis 1461 1462 Remove the files in path. Unlike purge it obeys include/exclude filters 1463 so can be used to selectively delete files. 1464 1465 rclone delete only deletes files but leaves the directory structure 1466 alone. If you want to delete a directory and all of its contents use the 1467 purge command. 1468 1469 If you supply the --rmdirs flag, it will remove all empty directories 1470 along with it. You can also use the separate command rmdir or rmdirs to 1471 delete empty directories only. 1472 1473 For example, to delete all files bigger than 100 MiB, you may first want 1474 to check what would be deleted (use either): 1475 1476 rclone --min-size 100M lsl remote:path 1477 rclone --dry-run --min-size 100M delete remote:path 1478 1479 Then proceed with the actual delete: 1480 1481 rclone --min-size 100M delete remote:path 1482 1483 That reads "delete everything with a minimum size of 100 MiB", hence 1484 delete all files bigger than 100 MiB. 1485 1486 Important: Since this can cause data loss, test first with the --dry-run 1487 or the --interactive/-i flag. 1488 1489 rclone delete remote:path [flags] 1490 1491 Options 1492 1493 -h, --help help for delete 1494 --rmdirs rmdirs removes empty directories but leaves root intact 1495 1496 Important Options 1497 1498 Important flags useful for most commands. 1499 1500 -n, --dry-run Do a trial run with no permanent changes 1501 -i, --interactive Enable interactive mode 1502 -v, --verbose count Print lots more stuff (repeat for more) 1503 1504 Filter Options 1505 1506 Flags for filtering directory listings. 1507 1508 --delete-excluded Delete files on dest excluded from sync 1509 --exclude stringArray Exclude files matching pattern 1510 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1511 --exclude-if-present stringArray Exclude directories if filename is present 1512 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1513 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1514 -f, --filter stringArray Add a file filtering rule 1515 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1516 --ignore-case Ignore case in filters (case insensitive) 1517 --include stringArray Include files matching pattern 1518 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1519 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1520 --max-depth int If set limits the recursion depth to this (default -1) 1521 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1522 --metadata-exclude stringArray Exclude metadatas matching pattern 1523 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1524 --metadata-filter stringArray Add a metadata filtering rule 1525 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1526 --metadata-include stringArray Include metadatas matching pattern 1527 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1528 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1529 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1530 1531 Listing Options 1532 1533 Flags for listing directories. 1534 1535 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1536 --fast-list Use recursive list if available; uses more memory but fewer transactions 1537 1538 See the global flags page for global options not listed here. 1539 1540 SEE ALSO 1541 1542 - rclone - Show help for rclone commands, flags and backends. 1543 1544 rclone purge 1545 1546 Remove the path and all of its contents. 1547 1548 Synopsis 1549 1550 Remove the path and all of its contents. Note that this does not obey 1551 include/exclude filters - everything will be removed. Use the delete 1552 command if you want to selectively delete files. To delete empty 1553 directories only, use command rmdir or rmdirs. 1554 1555 Important: Since this can cause data loss, test first with the --dry-run 1556 or the --interactive/-i flag. 1557 1558 rclone purge remote:path [flags] 1559 1560 Options 1561 1562 -h, --help help for purge 1563 1564 Important Options 1565 1566 Important flags useful for most commands. 1567 1568 -n, --dry-run Do a trial run with no permanent changes 1569 -i, --interactive Enable interactive mode 1570 -v, --verbose count Print lots more stuff (repeat for more) 1571 1572 See the global flags page for global options not listed here. 1573 1574 SEE ALSO 1575 1576 - rclone - Show help for rclone commands, flags and backends. 1577 1578 rclone mkdir 1579 1580 Make the path if it doesn't already exist. 1581 1582 rclone mkdir remote:path [flags] 1583 1584 Options 1585 1586 -h, --help help for mkdir 1587 1588 Important Options 1589 1590 Important flags useful for most commands. 1591 1592 -n, --dry-run Do a trial run with no permanent changes 1593 -i, --interactive Enable interactive mode 1594 -v, --verbose count Print lots more stuff (repeat for more) 1595 1596 See the global flags page for global options not listed here. 1597 1598 SEE ALSO 1599 1600 - rclone - Show help for rclone commands, flags and backends. 1601 1602 rclone rmdir 1603 1604 Remove the empty directory at path. 1605 1606 Synopsis 1607 1608 This removes empty directory given by path. Will not remove the path if 1609 it has any objects in it, not even empty subdirectories. Use command 1610 rmdirs (or delete with option --rmdirs) to do that. 1611 1612 To delete a path and any objects in it, use purge command. 1613 1614 rclone rmdir remote:path [flags] 1615 1616 Options 1617 1618 -h, --help help for rmdir 1619 1620 Important Options 1621 1622 Important flags useful for most commands. 1623 1624 -n, --dry-run Do a trial run with no permanent changes 1625 -i, --interactive Enable interactive mode 1626 -v, --verbose count Print lots more stuff (repeat for more) 1627 1628 See the global flags page for global options not listed here. 1629 1630 SEE ALSO 1631 1632 - rclone - Show help for rclone commands, flags and backends. 1633 1634 rclone check 1635 1636 Checks the files in the source and destination match. 1637 1638 Synopsis 1639 1640 Checks the files in the source and destination match. It compares sizes 1641 and hashes (MD5 or SHA1) and logs a report of files that don't match. It 1642 doesn't alter the source or destination. 1643 1644 For the crypt remote there is a dedicated command, cryptcheck, that are 1645 able to check the checksums of the encrypted files. 1646 1647 If you supply the --size-only flag, it will only compare the sizes not 1648 the hashes as well. Use this for a quick check. 1649 1650 If you supply the --download flag, it will download the data from both 1651 remotes and check them against each other on the fly. This can be useful 1652 for remotes that don't support hashes or if you really want to check all 1653 the data. 1654 1655 If you supply the --checkfile HASH flag with a valid hash name, the 1656 source:path must point to a text file in the SUM format. 1657 1658 If you supply the --one-way flag, it will only check that files in the 1659 source match the files in the destination, not the other way around. 1660 This means that extra files in the destination that are not in the 1661 source will not be detected. 1662 1663 The --differ, --missing-on-dst, --missing-on-src, --match and --error 1664 flags write paths, one per line, to the file name (or stdout if it is -) 1665 supplied. What they write is described in the help below. For example 1666 --differ will write all paths which are present on both the source and 1667 destination but different. 1668 1669 The --combined flag will write a file (or stdout) which contains all 1670 file paths with a symbol and then a space and then the path to tell you 1671 what happened to it. These are reminiscent of diff files. 1672 1673 - = path means path was found in source and destination and was 1674 identical 1675 - `- path` means path was missing on the source, so only in the 1676 destination 1677 - `+ path` means path was missing on the destination, so only in the 1678 source 1679 - `* path` means path was present in source and destination but 1680 different. 1681 - ! path means there was an error reading or hashing the source or 1682 dest. 1683 1684 The default number of parallel checks is 8. See the --checkers=N option 1685 for more information. 1686 1687 rclone check source:path dest:path [flags] 1688 1689 Options 1690 1691 -C, --checkfile string Treat source:path as a SUM file with hashes of given type 1692 --combined string Make a combined report of changes to this file 1693 --differ string Report all non-matching files to this file 1694 --download Check by downloading rather than with hash 1695 --error string Report all files with errors (hashing or reading) to this file 1696 -h, --help help for check 1697 --match string Report all matching files to this file 1698 --missing-on-dst string Report all files missing from the destination to this file 1699 --missing-on-src string Report all files missing from the source to this file 1700 --one-way Check one way only, source files must exist on remote 1701 1702 Check Options 1703 1704 Flags used for rclone check. 1705 1706 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 1707 1708 Filter Options 1709 1710 Flags for filtering directory listings. 1711 1712 --delete-excluded Delete files on dest excluded from sync 1713 --exclude stringArray Exclude files matching pattern 1714 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1715 --exclude-if-present stringArray Exclude directories if filename is present 1716 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1717 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1718 -f, --filter stringArray Add a file filtering rule 1719 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1720 --ignore-case Ignore case in filters (case insensitive) 1721 --include stringArray Include files matching pattern 1722 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1723 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1724 --max-depth int If set limits the recursion depth to this (default -1) 1725 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1726 --metadata-exclude stringArray Exclude metadatas matching pattern 1727 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1728 --metadata-filter stringArray Add a metadata filtering rule 1729 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1730 --metadata-include stringArray Include metadatas matching pattern 1731 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1732 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1733 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1734 1735 Listing Options 1736 1737 Flags for listing directories. 1738 1739 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1740 --fast-list Use recursive list if available; uses more memory but fewer transactions 1741 1742 See the global flags page for global options not listed here. 1743 1744 SEE ALSO 1745 1746 - rclone - Show help for rclone commands, flags and backends. 1747 1748 rclone ls 1749 1750 List the objects in the path with size and path. 1751 1752 Synopsis 1753 1754 Lists the objects in the source path to standard output in a human 1755 readable format with size and path. Recurses by default. 1756 1757 Eg 1758 1759 $ rclone ls swift:bucket 1760 60295 bevajer5jef 1761 90613 canole 1762 94467 diwogej7 1763 37600 fubuwic 1764 1765 Any of the filtering options can be applied to this command. 1766 1767 There are several related list commands 1768 1769 - ls to list size and path of objects only 1770 - lsl to list modification time, size and path of objects only 1771 - lsd to list directories only 1772 - lsf to list objects and directories in easy to parse format 1773 - lsjson to list objects and directories in JSON format 1774 1775 ls,lsl,lsd are designed to be human-readable. lsf is designed to be 1776 human and machine-readable. lsjson is designed to be machine-readable. 1777 1778 Note that ls and lsl recurse by default - use --max-depth 1 to stop the 1779 recursion. 1780 1781 The other list commands lsd,lsf,lsjson do not recurse by default - use 1782 -R to make them recurse. 1783 1784 Listing a nonexistent directory will produce an error except for remotes 1785 which can't have empty directories (e.g. s3, swift, or gcs - the 1786 bucket-based remotes). 1787 1788 rclone ls remote:path [flags] 1789 1790 Options 1791 1792 -h, --help help for ls 1793 1794 Filter Options 1795 1796 Flags for filtering directory listings. 1797 1798 --delete-excluded Delete files on dest excluded from sync 1799 --exclude stringArray Exclude files matching pattern 1800 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1801 --exclude-if-present stringArray Exclude directories if filename is present 1802 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1803 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1804 -f, --filter stringArray Add a file filtering rule 1805 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1806 --ignore-case Ignore case in filters (case insensitive) 1807 --include stringArray Include files matching pattern 1808 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1809 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1810 --max-depth int If set limits the recursion depth to this (default -1) 1811 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1812 --metadata-exclude stringArray Exclude metadatas matching pattern 1813 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1814 --metadata-filter stringArray Add a metadata filtering rule 1815 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1816 --metadata-include stringArray Include metadatas matching pattern 1817 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1818 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1819 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1820 1821 Listing Options 1822 1823 Flags for listing directories. 1824 1825 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1826 --fast-list Use recursive list if available; uses more memory but fewer transactions 1827 1828 See the global flags page for global options not listed here. 1829 1830 SEE ALSO 1831 1832 - rclone - Show help for rclone commands, flags and backends. 1833 1834 rclone lsd 1835 1836 List all directories/containers/buckets in the path. 1837 1838 Synopsis 1839 1840 Lists the directories in the source path to standard output. Does not 1841 recurse by default. Use the -R flag to recurse. 1842 1843 This command lists the total size of the directory (if known, -1 if 1844 not), the modification time (if known, the current time if not), the 1845 number of objects in the directory (if known, -1 if not) and the name of 1846 the directory, Eg 1847 1848 $ rclone lsd swift: 1849 494000 2018-04-26 08:43:20 10000 10000files 1850 65 2018-04-26 08:43:20 1 1File 1851 1852 Or 1853 1854 $ rclone lsd drive:test 1855 -1 2016-10-17 17:41:53 -1 1000files 1856 -1 2017-01-03 14:40:54 -1 2500files 1857 -1 2017-07-08 14:39:28 -1 4000files 1858 1859 If you just want the directory names use rclone lsf --dirs-only. 1860 1861 Any of the filtering options can be applied to this command. 1862 1863 There are several related list commands 1864 1865 - ls to list size and path of objects only 1866 - lsl to list modification time, size and path of objects only 1867 - lsd to list directories only 1868 - lsf to list objects and directories in easy to parse format 1869 - lsjson to list objects and directories in JSON format 1870 1871 ls,lsl,lsd are designed to be human-readable. lsf is designed to be 1872 human and machine-readable. lsjson is designed to be machine-readable. 1873 1874 Note that ls and lsl recurse by default - use --max-depth 1 to stop the 1875 recursion. 1876 1877 The other list commands lsd,lsf,lsjson do not recurse by default - use 1878 -R to make them recurse. 1879 1880 Listing a nonexistent directory will produce an error except for remotes 1881 which can't have empty directories (e.g. s3, swift, or gcs - the 1882 bucket-based remotes). 1883 1884 rclone lsd remote:path [flags] 1885 1886 Options 1887 1888 -h, --help help for lsd 1889 -R, --recursive Recurse into the listing 1890 1891 Filter Options 1892 1893 Flags for filtering directory listings. 1894 1895 --delete-excluded Delete files on dest excluded from sync 1896 --exclude stringArray Exclude files matching pattern 1897 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1898 --exclude-if-present stringArray Exclude directories if filename is present 1899 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1900 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1901 -f, --filter stringArray Add a file filtering rule 1902 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1903 --ignore-case Ignore case in filters (case insensitive) 1904 --include stringArray Include files matching pattern 1905 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1906 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1907 --max-depth int If set limits the recursion depth to this (default -1) 1908 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1909 --metadata-exclude stringArray Exclude metadatas matching pattern 1910 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1911 --metadata-filter stringArray Add a metadata filtering rule 1912 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 1913 --metadata-include stringArray Include metadatas matching pattern 1914 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 1915 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1916 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 1917 1918 Listing Options 1919 1920 Flags for listing directories. 1921 1922 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 1923 --fast-list Use recursive list if available; uses more memory but fewer transactions 1924 1925 See the global flags page for global options not listed here. 1926 1927 SEE ALSO 1928 1929 - rclone - Show help for rclone commands, flags and backends. 1930 1931 rclone lsl 1932 1933 List the objects in path with modification time, size and path. 1934 1935 Synopsis 1936 1937 Lists the objects in the source path to standard output in a human 1938 readable format with modification time, size and path. Recurses by 1939 default. 1940 1941 Eg 1942 1943 $ rclone lsl swift:bucket 1944 60295 2016-06-25 18:55:41.062626927 bevajer5jef 1945 90613 2016-06-25 18:55:43.302607074 canole 1946 94467 2016-06-25 18:55:43.046609333 diwogej7 1947 37600 2016-06-25 18:55:40.814629136 fubuwic 1948 1949 Any of the filtering options can be applied to this command. 1950 1951 There are several related list commands 1952 1953 - ls to list size and path of objects only 1954 - lsl to list modification time, size and path of objects only 1955 - lsd to list directories only 1956 - lsf to list objects and directories in easy to parse format 1957 - lsjson to list objects and directories in JSON format 1958 1959 ls,lsl,lsd are designed to be human-readable. lsf is designed to be 1960 human and machine-readable. lsjson is designed to be machine-readable. 1961 1962 Note that ls and lsl recurse by default - use --max-depth 1 to stop the 1963 recursion. 1964 1965 The other list commands lsd,lsf,lsjson do not recurse by default - use 1966 -R to make them recurse. 1967 1968 Listing a nonexistent directory will produce an error except for remotes 1969 which can't have empty directories (e.g. s3, swift, or gcs - the 1970 bucket-based remotes). 1971 1972 rclone lsl remote:path [flags] 1973 1974 Options 1975 1976 -h, --help help for lsl 1977 1978 Filter Options 1979 1980 Flags for filtering directory listings. 1981 1982 --delete-excluded Delete files on dest excluded from sync 1983 --exclude stringArray Exclude files matching pattern 1984 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 1985 --exclude-if-present stringArray Exclude directories if filename is present 1986 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 1987 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 1988 -f, --filter stringArray Add a file filtering rule 1989 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 1990 --ignore-case Ignore case in filters (case insensitive) 1991 --include stringArray Include files matching pattern 1992 --include-from stringArray Read file include patterns from file (use - to read from stdin) 1993 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 1994 --max-depth int If set limits the recursion depth to this (default -1) 1995 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 1996 --metadata-exclude stringArray Exclude metadatas matching pattern 1997 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 1998 --metadata-filter stringArray Add a metadata filtering rule 1999 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2000 --metadata-include stringArray Include metadatas matching pattern 2001 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2002 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2003 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2004 2005 Listing Options 2006 2007 Flags for listing directories. 2008 2009 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2010 --fast-list Use recursive list if available; uses more memory but fewer transactions 2011 2012 See the global flags page for global options not listed here. 2013 2014 SEE ALSO 2015 2016 - rclone - Show help for rclone commands, flags and backends. 2017 2018 rclone md5sum 2019 2020 Produces an md5sum file for all the objects in the path. 2021 2022 Synopsis 2023 2024 Produces an md5sum file for all the objects in the path. This is in the 2025 same format as the standard md5sum tool produces. 2026 2027 By default, the hash is requested from the remote. If MD5 is not 2028 supported by the remote, no hash will be returned. With the download 2029 flag, the file will be downloaded from the remote and hashed locally 2030 enabling MD5 for any remote. 2031 2032 For other algorithms, see the hashsum command. Running 2033 rclone md5sum remote:path is equivalent to running 2034 rclone hashsum MD5 remote:path. 2035 2036 This command can also hash data received on standard input (stdin), by 2037 not passing a remote:path, or by passing a hyphen as remote:path when 2038 there is data to read (if not, the hyphen will be treated literally, as 2039 a relative path). 2040 2041 rclone md5sum remote:path [flags] 2042 2043 Options 2044 2045 --base64 Output base64 encoded hashsum 2046 -C, --checkfile string Validate hashes against a given SUM file instead of printing them 2047 --download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote 2048 -h, --help help for md5sum 2049 --output-file string Output hashsums to a file rather than the terminal 2050 2051 Filter Options 2052 2053 Flags for filtering directory listings. 2054 2055 --delete-excluded Delete files on dest excluded from sync 2056 --exclude stringArray Exclude files matching pattern 2057 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2058 --exclude-if-present stringArray Exclude directories if filename is present 2059 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2060 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2061 -f, --filter stringArray Add a file filtering rule 2062 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2063 --ignore-case Ignore case in filters (case insensitive) 2064 --include stringArray Include files matching pattern 2065 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2066 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2067 --max-depth int If set limits the recursion depth to this (default -1) 2068 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2069 --metadata-exclude stringArray Exclude metadatas matching pattern 2070 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2071 --metadata-filter stringArray Add a metadata filtering rule 2072 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2073 --metadata-include stringArray Include metadatas matching pattern 2074 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2075 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2076 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2077 2078 Listing Options 2079 2080 Flags for listing directories. 2081 2082 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2083 --fast-list Use recursive list if available; uses more memory but fewer transactions 2084 2085 See the global flags page for global options not listed here. 2086 2087 SEE ALSO 2088 2089 - rclone - Show help for rclone commands, flags and backends. 2090 2091 rclone sha1sum 2092 2093 Produces an sha1sum file for all the objects in the path. 2094 2095 Synopsis 2096 2097 Produces an sha1sum file for all the objects in the path. This is in the 2098 same format as the standard sha1sum tool produces. 2099 2100 By default, the hash is requested from the remote. If SHA-1 is not 2101 supported by the remote, no hash will be returned. With the download 2102 flag, the file will be downloaded from the remote and hashed locally 2103 enabling SHA-1 for any remote. 2104 2105 For other algorithms, see the hashsum command. Running 2106 rclone sha1sum remote:path is equivalent to running 2107 rclone hashsum SHA1 remote:path. 2108 2109 This command can also hash data received on standard input (stdin), by 2110 not passing a remote:path, or by passing a hyphen as remote:path when 2111 there is data to read (if not, the hyphen will be treated literally, as 2112 a relative path). 2113 2114 This command can also hash data received on STDIN, if not passing a 2115 remote:path. 2116 2117 rclone sha1sum remote:path [flags] 2118 2119 Options 2120 2121 --base64 Output base64 encoded hashsum 2122 -C, --checkfile string Validate hashes against a given SUM file instead of printing them 2123 --download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote 2124 -h, --help help for sha1sum 2125 --output-file string Output hashsums to a file rather than the terminal 2126 2127 Filter Options 2128 2129 Flags for filtering directory listings. 2130 2131 --delete-excluded Delete files on dest excluded from sync 2132 --exclude stringArray Exclude files matching pattern 2133 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2134 --exclude-if-present stringArray Exclude directories if filename is present 2135 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2136 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2137 -f, --filter stringArray Add a file filtering rule 2138 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2139 --ignore-case Ignore case in filters (case insensitive) 2140 --include stringArray Include files matching pattern 2141 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2142 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2143 --max-depth int If set limits the recursion depth to this (default -1) 2144 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2145 --metadata-exclude stringArray Exclude metadatas matching pattern 2146 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2147 --metadata-filter stringArray Add a metadata filtering rule 2148 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2149 --metadata-include stringArray Include metadatas matching pattern 2150 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2151 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2152 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2153 2154 Listing Options 2155 2156 Flags for listing directories. 2157 2158 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2159 --fast-list Use recursive list if available; uses more memory but fewer transactions 2160 2161 See the global flags page for global options not listed here. 2162 2163 SEE ALSO 2164 2165 - rclone - Show help for rclone commands, flags and backends. 2166 2167 rclone size 2168 2169 Prints the total size and number of objects in remote:path. 2170 2171 Synopsis 2172 2173 Counts objects in the path and calculates the total size. Prints the 2174 result to standard output. 2175 2176 By default the output is in human-readable format, but shows values in 2177 both human-readable format as well as the raw numbers (global option 2178 --human-readable is not considered). Use option --json to format output 2179 as JSON instead. 2180 2181 Recurses by default, use --max-depth 1 to stop the recursion. 2182 2183 Some backends do not always provide file sizes, see for example Google 2184 Photos and Google Docs. Rclone will then show a notice in the log 2185 indicating how many such files were encountered, and count them in as 2186 empty files in the output of the size command. 2187 2188 rclone size remote:path [flags] 2189 2190 Options 2191 2192 -h, --help help for size 2193 --json Format output as JSON 2194 2195 Filter Options 2196 2197 Flags for filtering directory listings. 2198 2199 --delete-excluded Delete files on dest excluded from sync 2200 --exclude stringArray Exclude files matching pattern 2201 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2202 --exclude-if-present stringArray Exclude directories if filename is present 2203 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2204 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2205 -f, --filter stringArray Add a file filtering rule 2206 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2207 --ignore-case Ignore case in filters (case insensitive) 2208 --include stringArray Include files matching pattern 2209 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2210 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2211 --max-depth int If set limits the recursion depth to this (default -1) 2212 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2213 --metadata-exclude stringArray Exclude metadatas matching pattern 2214 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2215 --metadata-filter stringArray Add a metadata filtering rule 2216 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2217 --metadata-include stringArray Include metadatas matching pattern 2218 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2219 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2220 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2221 2222 Listing Options 2223 2224 Flags for listing directories. 2225 2226 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2227 --fast-list Use recursive list if available; uses more memory but fewer transactions 2228 2229 See the global flags page for global options not listed here. 2230 2231 SEE ALSO 2232 2233 - rclone - Show help for rclone commands, flags and backends. 2234 2235 rclone version 2236 2237 Show the version number. 2238 2239 Synopsis 2240 2241 Show the rclone version number, the go version, the build target OS and 2242 architecture, the runtime OS and kernel version and bitness, build tags 2243 and the type of executable (static or dynamic). 2244 2245 For example: 2246 2247 $ rclone version 2248 rclone v1.55.0 2249 - os/version: ubuntu 18.04 (64 bit) 2250 - os/kernel: 4.15.0-136-generic (x86_64) 2251 - os/type: linux 2252 - os/arch: amd64 2253 - go/version: go1.16 2254 - go/linking: static 2255 - go/tags: none 2256 2257 Note: before rclone version 1.55 the os/type and os/arch lines were 2258 merged, and the "go/version" line was tagged as "go version". 2259 2260 If you supply the --check flag, then it will do an online check to 2261 compare your version with the latest release and the latest beta. 2262 2263 $ rclone version --check 2264 yours: 1.42.0.6 2265 latest: 1.42 (released 2018-06-16) 2266 beta: 1.42.0.5 (released 2018-06-17) 2267 2268 Or 2269 2270 $ rclone version --check 2271 yours: 1.41 2272 latest: 1.42 (released 2018-06-16) 2273 upgrade: https://downloads.rclone.org/v1.42 2274 beta: 1.42.0.5 (released 2018-06-17) 2275 upgrade: https://beta.rclone.org/v1.42-005-g56e1e820 2276 2277 rclone version [flags] 2278 2279 Options 2280 2281 --check Check for new version 2282 -h, --help help for version 2283 2284 See the global flags page for global options not listed here. 2285 2286 SEE ALSO 2287 2288 - rclone - Show help for rclone commands, flags and backends. 2289 2290 rclone cleanup 2291 2292 Clean up the remote if possible. 2293 2294 Synopsis 2295 2296 Clean up the remote if possible. Empty the trash or delete old file 2297 versions. Not supported by all remotes. 2298 2299 rclone cleanup remote:path [flags] 2300 2301 Options 2302 2303 -h, --help help for cleanup 2304 2305 Important Options 2306 2307 Important flags useful for most commands. 2308 2309 -n, --dry-run Do a trial run with no permanent changes 2310 -i, --interactive Enable interactive mode 2311 -v, --verbose count Print lots more stuff (repeat for more) 2312 2313 See the global flags page for global options not listed here. 2314 2315 SEE ALSO 2316 2317 - rclone - Show help for rclone commands, flags and backends. 2318 2319 rclone dedupe 2320 2321 Interactively find duplicate filenames and delete/rename them. 2322 2323 Synopsis 2324 2325 By default dedupe interactively finds files with duplicate names and 2326 offers to delete all but one or rename them to be different. This is 2327 known as deduping by name. 2328 2329 Deduping by name is only useful with a small group of backends (e.g. 2330 Google Drive, Opendrive) that can have duplicate file names. It can be 2331 run on wrapping backends (e.g. crypt) if they wrap a backend which 2332 supports duplicate file names. 2333 2334 However if --by-hash is passed in then dedupe will find files with 2335 duplicate hashes instead which will work on any backend which supports 2336 at least one hash. This can be used to find files with duplicate 2337 content. This is known as deduping by hash. 2338 2339 If deduping by name, first rclone will merge directories with the same 2340 name. It will do this iteratively until all the identically named 2341 directories have been merged. 2342 2343 Next, if deduping by name, for every group of duplicate file names / 2344 hashes, it will delete all but one identical file it finds without 2345 confirmation. This means that for most duplicated files the dedupe 2346 command will not be interactive. 2347 2348 dedupe considers files to be identical if they have the same file path 2349 and the same hash. If the backend does not support hashes (e.g. crypt 2350 wrapping Google Drive) then they will never be found to be identical. If 2351 you use the --size-only flag then files will be considered identical if 2352 they have the same size (any hash will be ignored). This can be useful 2353 on crypt backends which do not support hashes. 2354 2355 Next rclone will resolve the remaining duplicates. Exactly which action 2356 is taken depends on the dedupe mode. By default, rclone will 2357 interactively query the user for each one. 2358 2359 Important: Since this can cause data loss, test first with the --dry-run 2360 or the --interactive/-i flag. 2361 2362 Here is an example run. 2363 2364 Before - with duplicates 2365 2366 $ rclone lsl drive:dupes 2367 6048320 2016-03-05 16:23:16.798000000 one.txt 2368 6048320 2016-03-05 16:23:11.775000000 one.txt 2369 564374 2016-03-05 16:23:06.731000000 one.txt 2370 6048320 2016-03-05 16:18:26.092000000 one.txt 2371 6048320 2016-03-05 16:22:46.185000000 two.txt 2372 1744073 2016-03-05 16:22:38.104000000 two.txt 2373 564374 2016-03-05 16:22:52.118000000 two.txt 2374 2375 Now the dedupe session 2376 2377 $ rclone dedupe drive:dupes 2378 2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode. 2379 one.txt: Found 4 files with duplicate names 2380 one.txt: Deleting 2/3 identical duplicates (MD5 "1eedaa9fe86fd4b8632e2ac549403b36") 2381 one.txt: 2 duplicates remain 2382 1: 6048320 bytes, 2016-03-05 16:23:16.798000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36 2383 2: 564374 bytes, 2016-03-05 16:23:06.731000000, MD5 7594e7dc9fc28f727c42ee3e0749de81 2384 s) Skip and do nothing 2385 k) Keep just one (choose which in next step) 2386 r) Rename all to be different (by changing file.jpg to file-1.jpg) 2387 s/k/r> k 2388 Enter the number of the file to keep> 1 2389 one.txt: Deleted 1 extra copies 2390 two.txt: Found 3 files with duplicate names 2391 two.txt: 3 duplicates remain 2392 1: 564374 bytes, 2016-03-05 16:22:52.118000000, MD5 7594e7dc9fc28f727c42ee3e0749de81 2393 2: 6048320 bytes, 2016-03-05 16:22:46.185000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36 2394 3: 1744073 bytes, 2016-03-05 16:22:38.104000000, MD5 851957f7fb6f0bc4ce76be966d336802 2395 s) Skip and do nothing 2396 k) Keep just one (choose which in next step) 2397 r) Rename all to be different (by changing file.jpg to file-1.jpg) 2398 s/k/r> r 2399 two-1.txt: renamed from: two.txt 2400 two-2.txt: renamed from: two.txt 2401 two-3.txt: renamed from: two.txt 2402 2403 The result being 2404 2405 $ rclone lsl drive:dupes 2406 6048320 2016-03-05 16:23:16.798000000 one.txt 2407 564374 2016-03-05 16:22:52.118000000 two-1.txt 2408 6048320 2016-03-05 16:22:46.185000000 two-2.txt 2409 1744073 2016-03-05 16:22:38.104000000 two-3.txt 2410 2411 Dedupe can be run non interactively using the --dedupe-mode flag or by 2412 using an extra parameter with the same value 2413 2414 - --dedupe-mode interactive - interactive as above. 2415 - --dedupe-mode skip - removes identical files then skips anything 2416 left. 2417 - --dedupe-mode first - removes identical files then keeps the first 2418 one. 2419 - --dedupe-mode newest - removes identical files then keeps the newest 2420 one. 2421 - --dedupe-mode oldest - removes identical files then keeps the oldest 2422 one. 2423 - --dedupe-mode largest - removes identical files then keeps the 2424 largest one. 2425 - --dedupe-mode smallest - removes identical files then keeps the 2426 smallest one. 2427 - --dedupe-mode rename - removes identical files then renames the rest 2428 to be different. 2429 - --dedupe-mode list - lists duplicate dirs and files only and changes 2430 nothing. 2431 2432 For example, to rename all the identically named photos in your Google 2433 Photos directory, do 2434 2435 rclone dedupe --dedupe-mode rename "drive:Google Photos" 2436 2437 Or 2438 2439 rclone dedupe rename "drive:Google Photos" 2440 2441 rclone dedupe [mode] remote:path [flags] 2442 2443 Options 2444 2445 --by-hash Find identical hashes rather than names 2446 --dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename (default "interactive") 2447 -h, --help help for dedupe 2448 2449 Important Options 2450 2451 Important flags useful for most commands. 2452 2453 -n, --dry-run Do a trial run with no permanent changes 2454 -i, --interactive Enable interactive mode 2455 -v, --verbose count Print lots more stuff (repeat for more) 2456 2457 See the global flags page for global options not listed here. 2458 2459 SEE ALSO 2460 2461 - rclone - Show help for rclone commands, flags and backends. 2462 2463 rclone about 2464 2465 Get quota information from the remote. 2466 2467 Synopsis 2468 2469 rclone about prints quota information about a remote to standard output. 2470 The output is typically used, free, quota and trash contents. 2471 2472 E.g. Typical output from rclone about remote: is: 2473 2474 Total: 17 GiB 2475 Used: 7.444 GiB 2476 Free: 1.315 GiB 2477 Trashed: 100.000 MiB 2478 Other: 8.241 GiB 2479 2480 Where the fields are: 2481 2482 - Total: Total size available. 2483 - Used: Total size used. 2484 - Free: Total space available to this user. 2485 - Trashed: Total space used by trash. 2486 - Other: Total amount in other storage (e.g. Gmail, Google Photos). 2487 - Objects: Total number of objects in the storage. 2488 2489 All sizes are in number of bytes. 2490 2491 Applying a --full flag to the command prints the bytes in full, e.g. 2492 2493 Total: 18253611008 2494 Used: 7993453766 2495 Free: 1411001220 2496 Trashed: 104857602 2497 Other: 8849156022 2498 2499 A --json flag generates conveniently machine-readable output, e.g. 2500 2501 { 2502 "total": 18253611008, 2503 "used": 7993453766, 2504 "trashed": 104857602, 2505 "other": 8849156022, 2506 "free": 1411001220 2507 } 2508 2509 Not all backends print all fields. Information is not included if it is 2510 not provided by a backend. Where the value is unlimited it is omitted. 2511 2512 Some backends does not support the rclone about command at all, see 2513 complete list in documentation. 2514 2515 rclone about remote: [flags] 2516 2517 Options 2518 2519 --full Full numbers instead of human-readable 2520 -h, --help help for about 2521 --json Format output as JSON 2522 2523 See the global flags page for global options not listed here. 2524 2525 SEE ALSO 2526 2527 - rclone - Show help for rclone commands, flags and backends. 2528 2529 rclone authorize 2530 2531 Remote authorization. 2532 2533 Synopsis 2534 2535 Remote authorization. Used to authorize a remote or headless rclone from 2536 a machine with a browser - use as instructed by rclone config. 2537 2538 Use --auth-no-open-browser to prevent rclone to open auth link in 2539 default browser automatically. 2540 2541 Use --template to generate HTML output via a custom Go template. If a 2542 blank string is provided as an argument to this flag, the default 2543 template is used. 2544 2545 rclone authorize [flags] 2546 2547 Options 2548 2549 --auth-no-open-browser Do not automatically open auth link in default browser 2550 -h, --help help for authorize 2551 --template string The path to a custom Go template for generating HTML responses 2552 2553 See the global flags page for global options not listed here. 2554 2555 SEE ALSO 2556 2557 - rclone - Show help for rclone commands, flags and backends. 2558 2559 rclone backend 2560 2561 Run a backend-specific command. 2562 2563 Synopsis 2564 2565 This runs a backend-specific command. The commands themselves (except 2566 for "help" and "features") are defined by the backends and you should 2567 see the backend docs for definitions. 2568 2569 You can discover what commands a backend implements by using 2570 2571 rclone backend help remote: 2572 rclone backend help <backendname> 2573 2574 You can also discover information about the backend using (see 2575 operations/fsinfo in the remote control docs for more info). 2576 2577 rclone backend features remote: 2578 2579 Pass options to the backend command with -o. This should be key=value or 2580 key, e.g.: 2581 2582 rclone backend stats remote:path stats -o format=json -o long 2583 2584 Pass arguments to the backend by placing them on the end of the line 2585 2586 rclone backend cleanup remote:path file1 file2 file3 2587 2588 Note to run these commands on a running backend then see backend/command 2589 in the rc docs. 2590 2591 rclone backend <command> remote:path [opts] <args> [flags] 2592 2593 Options 2594 2595 -h, --help help for backend 2596 --json Always output in JSON format 2597 -o, --option stringArray Option in the form name=value or name 2598 2599 Important Options 2600 2601 Important flags useful for most commands. 2602 2603 -n, --dry-run Do a trial run with no permanent changes 2604 -i, --interactive Enable interactive mode 2605 -v, --verbose count Print lots more stuff (repeat for more) 2606 2607 See the global flags page for global options not listed here. 2608 2609 SEE ALSO 2610 2611 - rclone - Show help for rclone commands, flags and backends. 2612 2613 rclone bisync 2614 2615 Perform bidirectional synchronization between two paths. 2616 2617 Synopsis 2618 2619 Perform bidirectional synchronization between two paths. 2620 2621 Bisync provides a bidirectional cloud sync solution in rclone. It 2622 retains the Path1 and Path2 filesystem listings from the prior run. On 2623 each successive run it will: - list files on Path1 and Path2, and check 2624 for changes on each side. Changes include New, Newer, Older, and Deleted 2625 files. - Propagate changes on Path1 to Path2, and vice-versa. 2626 2627 Bisync is in beta and is considered an advanced command, so use with 2628 care. Make sure you have read and understood the entire manual 2629 (especially the Limitations section) before using, or data loss can 2630 result. Questions can be asked in the Rclone Forum. 2631 2632 See full bisync description for details. 2633 2634 rclone bisync remote1:path1 remote2:path2 [flags] 2635 2636 Options 2637 2638 --backup-dir1 string --backup-dir for Path1. Must be a non-overlapping path on the same remote. 2639 --backup-dir2 string --backup-dir for Path2. Must be a non-overlapping path on the same remote. 2640 --check-access Ensure expected RCLONE_TEST files are found on both Path1 and Path2 filesystems, else abort. 2641 --check-filename string Filename for --check-access (default: RCLONE_TEST) 2642 --check-sync string Controls comparison of final listings: true|false|only (default: true) (default "true") 2643 --compare string Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime') 2644 --conflict-loser ConflictLoserAction Action to take on the loser of a sync conflict (when there is a winner) or on both files (when there is no winner): , num, pathname, delete (default: num) 2645 --conflict-resolve string Automatically resolve conflicts by preferring the version that is: none, path1, path2, newer, older, larger, smaller (default: none) (default "none") 2646 --conflict-suffix string Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict') 2647 --create-empty-src-dirs Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs) 2648 --download-hash Compute hash by downloading when otherwise unavailable. (warning: may be slow and use lots of data!) 2649 --filters-file string Read filtering patterns from a file 2650 --force Bypass --max-delete safety check and run the sync. Consider using with --verbose 2651 -h, --help help for bisync 2652 --ignore-listing-checksum Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks) 2653 --max-lock Duration Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m) (default 0s) 2654 --no-cleanup Retain working files (useful for troubleshooting and testing). 2655 --no-slow-hash Ignore listing checksums only on backends where they are slow 2656 --recover Automatically recover from interruptions without requiring --resync. 2657 --remove-empty-dirs Remove ALL empty directories at the final cleanup step. 2658 --resilient Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk! 2659 -1, --resync Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first. 2660 --resync-mode string During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.) (default "none") 2661 --slow-hash-sync-only Ignore slow checksums for listings and deltas, but still consider them during sync calls. 2662 --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) 2663 2664 Copy Options 2665 2666 Flags for anything which can Copy a file. 2667 2668 --check-first Do all the checks before starting transfers 2669 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 2670 --compare-dest stringArray Include additional comma separated server-side paths during comparison 2671 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 2672 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 2673 --ignore-case-sync Ignore case when synchronizing 2674 --ignore-checksum Skip post copy check of checksums 2675 --ignore-existing Skip all files that exist on destination 2676 --ignore-size Ignore size when skipping use modtime or checksum 2677 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 2678 --immutable Do not modify files, fail if existing files have been modified 2679 --inplace Download directly to destination file instead of atomic download to temp/rename 2680 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 2681 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 2682 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 2683 -M, --metadata If set, preserve metadata when copying objects 2684 --modify-window Duration Max time diff to be considered the same (default 1ns) 2685 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 2686 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 2687 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 2688 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 2689 --no-check-dest Don't check the destination, copy regardless 2690 --no-traverse Don't traverse destination file system on copy 2691 --no-update-dir-modtime Don't update directory modification times 2692 --no-update-modtime Don't update destination modtime if files identical 2693 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 2694 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 2695 --refresh-times Refresh the modtime of remote files 2696 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 2697 --size-only Skip based on size only, not modtime or checksum 2698 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 2699 -u, --update Skip files that are newer on the destination 2700 2701 Important Options 2702 2703 Important flags useful for most commands. 2704 2705 -n, --dry-run Do a trial run with no permanent changes 2706 -i, --interactive Enable interactive mode 2707 -v, --verbose count Print lots more stuff (repeat for more) 2708 2709 Filter Options 2710 2711 Flags for filtering directory listings. 2712 2713 --delete-excluded Delete files on dest excluded from sync 2714 --exclude stringArray Exclude files matching pattern 2715 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2716 --exclude-if-present stringArray Exclude directories if filename is present 2717 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2718 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2719 -f, --filter stringArray Add a file filtering rule 2720 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2721 --ignore-case Ignore case in filters (case insensitive) 2722 --include stringArray Include files matching pattern 2723 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2724 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2725 --max-depth int If set limits the recursion depth to this (default -1) 2726 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2727 --metadata-exclude stringArray Exclude metadatas matching pattern 2728 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2729 --metadata-filter stringArray Add a metadata filtering rule 2730 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2731 --metadata-include stringArray Include metadatas matching pattern 2732 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2733 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2734 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2735 2736 See the global flags page for global options not listed here. 2737 2738 SEE ALSO 2739 2740 - rclone - Show help for rclone commands, flags and backends. 2741 2742 rclone cat 2743 2744 Concatenates any files and sends them to stdout. 2745 2746 Synopsis 2747 2748 rclone cat sends any files to standard output. 2749 2750 You can use it like this to output a single file 2751 2752 rclone cat remote:path/to/file 2753 2754 Or like this to output any file in dir or its subdirectories. 2755 2756 rclone cat remote:path/to/dir 2757 2758 Or like this to output any .txt files in dir or its subdirectories. 2759 2760 rclone --include "*.txt" cat remote:path/to/dir 2761 2762 Use the --head flag to print characters only at the start, --tail for 2763 the end and --offset and --count to print a section in the middle. Note 2764 that if offset is negative it will count from the end, so 2765 --offset -1 --count 1 is equivalent to --tail 1. 2766 2767 Use the --separator flag to print a separator value between files. Be 2768 sure to shell-escape special characters. For example, to print a newline 2769 between files, use: 2770 2771 - bash: 2772 2773 rclone --include "*.txt" --separator $'\n' cat remote:path/to/dir 2774 2775 - powershell: 2776 2777 rclone --include "*.txt" --separator "`n" cat remote:path/to/dir 2778 2779 rclone cat remote:path [flags] 2780 2781 Options 2782 2783 --count int Only print N characters (default -1) 2784 --discard Discard the output instead of printing 2785 --head int Only print the first N characters 2786 -h, --help help for cat 2787 --offset int Start printing at offset N (or from end if -ve) 2788 --separator string Separator to use between objects when printing multiple files 2789 --tail int Only print the last N characters 2790 2791 Filter Options 2792 2793 Flags for filtering directory listings. 2794 2795 --delete-excluded Delete files on dest excluded from sync 2796 --exclude stringArray Exclude files matching pattern 2797 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2798 --exclude-if-present stringArray Exclude directories if filename is present 2799 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2800 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2801 -f, --filter stringArray Add a file filtering rule 2802 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2803 --ignore-case Ignore case in filters (case insensitive) 2804 --include stringArray Include files matching pattern 2805 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2806 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2807 --max-depth int If set limits the recursion depth to this (default -1) 2808 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2809 --metadata-exclude stringArray Exclude metadatas matching pattern 2810 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2811 --metadata-filter stringArray Add a metadata filtering rule 2812 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2813 --metadata-include stringArray Include metadatas matching pattern 2814 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2815 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2816 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2817 2818 Listing Options 2819 2820 Flags for listing directories. 2821 2822 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2823 --fast-list Use recursive list if available; uses more memory but fewer transactions 2824 2825 See the global flags page for global options not listed here. 2826 2827 SEE ALSO 2828 2829 - rclone - Show help for rclone commands, flags and backends. 2830 2831 rclone checksum 2832 2833 Checks the files in the destination against a SUM file. 2834 2835 Synopsis 2836 2837 Checks that hashsums of destination files match the SUM file. It 2838 compares hashes (MD5, SHA1, etc) and logs a report of files which don't 2839 match. It doesn't alter the file system. 2840 2841 The sumfile is treated as the source and the dst:path is treated as the 2842 destination for the purposes of the output. 2843 2844 If you supply the --download flag, it will download the data from the 2845 remote and calculate the content hash on the fly. This can be useful for 2846 remotes that don't support hashes or if you really want to check all the 2847 data. 2848 2849 Note that hash values in the SUM file are treated as case insensitive. 2850 2851 If you supply the --one-way flag, it will only check that files in the 2852 source match the files in the destination, not the other way around. 2853 This means that extra files in the destination that are not in the 2854 source will not be detected. 2855 2856 The --differ, --missing-on-dst, --missing-on-src, --match and --error 2857 flags write paths, one per line, to the file name (or stdout if it is -) 2858 supplied. What they write is described in the help below. For example 2859 --differ will write all paths which are present on both the source and 2860 destination but different. 2861 2862 The --combined flag will write a file (or stdout) which contains all 2863 file paths with a symbol and then a space and then the path to tell you 2864 what happened to it. These are reminiscent of diff files. 2865 2866 - = path means path was found in source and destination and was 2867 identical 2868 - `- path` means path was missing on the source, so only in the 2869 destination 2870 - `+ path` means path was missing on the destination, so only in the 2871 source 2872 - `* path` means path was present in source and destination but 2873 different. 2874 - ! path means there was an error reading or hashing the source or 2875 dest. 2876 2877 The default number of parallel checks is 8. See the --checkers=N option 2878 for more information. 2879 2880 rclone checksum <hash> sumfile dst:path [flags] 2881 2882 Options 2883 2884 --combined string Make a combined report of changes to this file 2885 --differ string Report all non-matching files to this file 2886 --download Check by hashing the contents 2887 --error string Report all files with errors (hashing or reading) to this file 2888 -h, --help help for checksum 2889 --match string Report all matching files to this file 2890 --missing-on-dst string Report all files missing from the destination to this file 2891 --missing-on-src string Report all files missing from the source to this file 2892 --one-way Check one way only, source files must exist on remote 2893 2894 Filter Options 2895 2896 Flags for filtering directory listings. 2897 2898 --delete-excluded Delete files on dest excluded from sync 2899 --exclude stringArray Exclude files matching pattern 2900 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 2901 --exclude-if-present stringArray Exclude directories if filename is present 2902 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 2903 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 2904 -f, --filter stringArray Add a file filtering rule 2905 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 2906 --ignore-case Ignore case in filters (case insensitive) 2907 --include stringArray Include files matching pattern 2908 --include-from stringArray Read file include patterns from file (use - to read from stdin) 2909 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2910 --max-depth int If set limits the recursion depth to this (default -1) 2911 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 2912 --metadata-exclude stringArray Exclude metadatas matching pattern 2913 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 2914 --metadata-filter stringArray Add a metadata filtering rule 2915 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 2916 --metadata-include stringArray Include metadatas matching pattern 2917 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 2918 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 2919 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 2920 2921 Listing Options 2922 2923 Flags for listing directories. 2924 2925 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 2926 --fast-list Use recursive list if available; uses more memory but fewer transactions 2927 2928 See the global flags page for global options not listed here. 2929 2930 SEE ALSO 2931 2932 - rclone - Show help for rclone commands, flags and backends. 2933 2934 rclone completion 2935 2936 Output completion script for a given shell. 2937 2938 Synopsis 2939 2940 Generates a shell completion script for rclone. Run with --help to list 2941 the supported shells. 2942 2943 Options 2944 2945 -h, --help help for completion 2946 2947 See the global flags page for global options not listed here. 2948 2949 SEE ALSO 2950 2951 - rclone - Show help for rclone commands, flags and backends. 2952 - rclone completion bash - Output bash completion script for rclone. 2953 - rclone completion fish - Output fish completion script for rclone. 2954 - rclone completion powershell - Output powershell completion script 2955 for rclone. 2956 - rclone completion zsh - Output zsh completion script for rclone. 2957 2958 rclone completion bash 2959 2960 Output bash completion script for rclone. 2961 2962 Synopsis 2963 2964 Generates a bash shell autocompletion script for rclone. 2965 2966 This writes to /etc/bash_completion.d/rclone by default so will probably 2967 need to be run with sudo or as root, e.g. 2968 2969 sudo rclone genautocomplete bash 2970 2971 Logout and login again to use the autocompletion scripts, or source them 2972 directly 2973 2974 . /etc/bash_completion 2975 2976 If you supply a command line argument the script will be written there. 2977 2978 If output_file is "-", then the output will be written to stdout. 2979 2980 rclone completion bash [output_file] [flags] 2981 2982 Options 2983 2984 -h, --help help for bash 2985 2986 See the global flags page for global options not listed here. 2987 2988 SEE ALSO 2989 2990 - rclone completion - Output completion script for a given shell. 2991 2992 rclone completion fish 2993 2994 Output fish completion script for rclone. 2995 2996 Synopsis 2997 2998 Generates a fish autocompletion script for rclone. 2999 3000 This writes to /etc/fish/completions/rclone.fish by default so will 3001 probably need to be run with sudo or as root, e.g. 3002 3003 sudo rclone genautocomplete fish 3004 3005 Logout and login again to use the autocompletion scripts, or source them 3006 directly 3007 3008 . /etc/fish/completions/rclone.fish 3009 3010 If you supply a command line argument the script will be written there. 3011 3012 If output_file is "-", then the output will be written to stdout. 3013 3014 rclone completion fish [output_file] [flags] 3015 3016 Options 3017 3018 -h, --help help for fish 3019 3020 See the global flags page for global options not listed here. 3021 3022 SEE ALSO 3023 3024 - rclone completion - Output completion script for a given shell. 3025 3026 rclone completion powershell 3027 3028 Output powershell completion script for rclone. 3029 3030 Synopsis 3031 3032 Generate the autocompletion script for powershell. 3033 3034 To load completions in your current shell session: 3035 3036 rclone completion powershell | Out-String | Invoke-Expression 3037 3038 To load completions for every new session, add the output of the above 3039 command to your powershell profile. 3040 3041 If output_file is "-" or missing, then the output will be written to 3042 stdout. 3043 3044 rclone completion powershell [output_file] [flags] 3045 3046 Options 3047 3048 -h, --help help for powershell 3049 3050 See the global flags page for global options not listed here. 3051 3052 SEE ALSO 3053 3054 - rclone completion - Output completion script for a given shell. 3055 3056 rclone completion zsh 3057 3058 Output zsh completion script for rclone. 3059 3060 Synopsis 3061 3062 Generates a zsh autocompletion script for rclone. 3063 3064 This writes to /usr/share/zsh/vendor-completions/_rclone by default so 3065 will probably need to be run with sudo or as root, e.g. 3066 3067 sudo rclone genautocomplete zsh 3068 3069 Logout and login again to use the autocompletion scripts, or source them 3070 directly 3071 3072 autoload -U compinit && compinit 3073 3074 If you supply a command line argument the script will be written there. 3075 3076 If output_file is "-", then the output will be written to stdout. 3077 3078 rclone completion zsh [output_file] [flags] 3079 3080 Options 3081 3082 -h, --help help for zsh 3083 3084 See the global flags page for global options not listed here. 3085 3086 SEE ALSO 3087 3088 - rclone completion - Output completion script for a given shell. 3089 3090 rclone config create 3091 3092 Create a new remote with name, type and options. 3093 3094 Synopsis 3095 3096 Create a new remote of name with type and options. The options should be 3097 passed in pairs of key value or as key=value. 3098 3099 For example, to make a swift remote of name myremote using auto config 3100 you would do: 3101 3102 rclone config create myremote swift env_auth true 3103 rclone config create myremote swift env_auth=true 3104 3105 So for example if you wanted to configure a Google Drive remote but 3106 using remote authorization you would do this: 3107 3108 rclone config create mydrive drive config_is_local=false 3109 3110 Note that if the config process would normally ask a question the 3111 default is taken (unless --non-interactive is used). Each time that 3112 happens rclone will print or DEBUG a message saying how to affect the 3113 value taken. 3114 3115 If any of the parameters passed is a password field, then rclone will 3116 automatically obscure them if they aren't already obscured before 3117 putting them in the config file. 3118 3119 NB If the password parameter is 22 characters or longer and consists 3120 only of base64 characters then rclone can get confused about whether the 3121 password is already obscured or not and put unobscured passwords into 3122 the config file. If you want to be 100% certain that the passwords get 3123 obscured then use the --obscure flag, or if you are 100% certain you are 3124 already passing obscured passwords then use --no-obscure. You can also 3125 set obscured passwords using the rclone config password command. 3126 3127 The flag --non-interactive is for use by applications that wish to 3128 configure rclone themselves, rather than using rclone's text based 3129 configuration questions. If this flag is set, and rclone needs to ask 3130 the user a question, a JSON blob will be returned with the question in 3131 it. 3132 3133 This will look something like (some irrelevant detail removed): 3134 3135 { 3136 "State": "*oauth-islocal,teamdrive,,", 3137 "Option": { 3138 "Name": "config_is_local", 3139 "Help": "Use web browser to automatically authenticate rclone with remote?\n * Say Y if the machine running rclone has a web browser you can use\n * Say N if running rclone on a (remote) machine without web browser access\nIf not sure try Y. If Y failed, try N.\n", 3140 "Default": true, 3141 "Examples": [ 3142 { 3143 "Value": "true", 3144 "Help": "Yes" 3145 }, 3146 { 3147 "Value": "false", 3148 "Help": "No" 3149 } 3150 ], 3151 "Required": false, 3152 "IsPassword": false, 3153 "Type": "bool", 3154 "Exclusive": true, 3155 }, 3156 "Error": "", 3157 } 3158 3159 The format of Option is the same as returned by rclone config providers. 3160 The question should be asked to the user and returned to rclone as the 3161 --result option along with the --state parameter. 3162 3163 The keys of Option are used as follows: 3164 3165 - Name - name of variable - show to user 3166 - Help - help text. Hard wrapped at 80 chars. Any URLs should be 3167 clicky. 3168 - Default - default value - return this if the user just wants the 3169 default. 3170 - Examples - the user should be able to choose one of these 3171 - Required - the value should be non-empty 3172 - IsPassword - the value is a password and should be edited as such 3173 - Type - type of value, eg bool, string, int and others 3174 - Exclusive - if set no free-form entry allowed only the Examples 3175 - Irrelevant keys Provider, ShortOpt, Hide, NoPrefix, Advanced 3176 3177 If Error is set then it should be shown to the user at the same time as 3178 the question. 3179 3180 rclone config update name --continue --state "*oauth-islocal,teamdrive,," --result "true" 3181 3182 Note that when using --continue all passwords should be passed in the 3183 clear (not obscured). Any default config values should be passed in with 3184 each invocation of --continue. 3185 3186 At the end of the non interactive process, rclone will return a result 3187 with State as empty string. 3188 3189 If --all is passed then rclone will ask all the config questions, not 3190 just the post config questions. Any parameters are used as defaults for 3191 questions as usual. 3192 3193 Note that bin/config.py in the rclone source implements this protocol as 3194 a readable demonstration. 3195 3196 rclone config create name type [key value]* [flags] 3197 3198 Options 3199 3200 --all Ask the full set of config questions 3201 --continue Continue the configuration process with an answer 3202 -h, --help help for create 3203 --no-obscure Force any passwords not to be obscured 3204 --non-interactive Don't interact with user and return questions 3205 --obscure Force any passwords to be obscured 3206 --result string Result - use with --continue 3207 --state string State - use with --continue 3208 3209 See the global flags page for global options not listed here. 3210 3211 SEE ALSO 3212 3213 - rclone config - Enter an interactive configuration session. 3214 3215 rclone config delete 3216 3217 Delete an existing remote. 3218 3219 rclone config delete name [flags] 3220 3221 Options 3222 3223 -h, --help help for delete 3224 3225 See the global flags page for global options not listed here. 3226 3227 SEE ALSO 3228 3229 - rclone config - Enter an interactive configuration session. 3230 3231 rclone config disconnect 3232 3233 Disconnects user from remote 3234 3235 Synopsis 3236 3237 This disconnects the remote: passed in to the cloud storage system. 3238 3239 This normally means revoking the oauth token. 3240 3241 To reconnect use "rclone config reconnect". 3242 3243 rclone config disconnect remote: [flags] 3244 3245 Options 3246 3247 -h, --help help for disconnect 3248 3249 See the global flags page for global options not listed here. 3250 3251 SEE ALSO 3252 3253 - rclone config - Enter an interactive configuration session. 3254 3255 rclone config dump 3256 3257 Dump the config file as JSON. 3258 3259 rclone config dump [flags] 3260 3261 Options 3262 3263 -h, --help help for dump 3264 3265 See the global flags page for global options not listed here. 3266 3267 SEE ALSO 3268 3269 - rclone config - Enter an interactive configuration session. 3270 3271 rclone config edit 3272 3273 Enter an interactive configuration session. 3274 3275 Synopsis 3276 3277 Enter an interactive configuration session where you can setup new 3278 remotes and manage existing ones. You may also set or remove a password 3279 to protect your configuration. 3280 3281 rclone config edit [flags] 3282 3283 Options 3284 3285 -h, --help help for edit 3286 3287 See the global flags page for global options not listed here. 3288 3289 SEE ALSO 3290 3291 - rclone config - Enter an interactive configuration session. 3292 3293 rclone config file 3294 3295 Show path of configuration file in use. 3296 3297 rclone config file [flags] 3298 3299 Options 3300 3301 -h, --help help for file 3302 3303 See the global flags page for global options not listed here. 3304 3305 SEE ALSO 3306 3307 - rclone config - Enter an interactive configuration session. 3308 3309 rclone config password 3310 3311 Update password in an existing remote. 3312 3313 Synopsis 3314 3315 Update an existing remote's password. The password should be passed in 3316 pairs of key password or as key=password. The password should be passed 3317 in in clear (unobscured). 3318 3319 For example, to set password of a remote of name myremote you would do: 3320 3321 rclone config password myremote fieldname mypassword 3322 rclone config password myremote fieldname=mypassword 3323 3324 This command is obsolete now that "config update" and "config create" 3325 both support obscuring passwords directly. 3326 3327 rclone config password name [key value]+ [flags] 3328 3329 Options 3330 3331 -h, --help help for password 3332 3333 See the global flags page for global options not listed here. 3334 3335 SEE ALSO 3336 3337 - rclone config - Enter an interactive configuration session. 3338 3339 rclone config paths 3340 3341 Show paths used for configuration, cache, temp etc. 3342 3343 rclone config paths [flags] 3344 3345 Options 3346 3347 -h, --help help for paths 3348 3349 See the global flags page for global options not listed here. 3350 3351 SEE ALSO 3352 3353 - rclone config - Enter an interactive configuration session. 3354 3355 rclone config providers 3356 3357 List in JSON format all the providers and options. 3358 3359 rclone config providers [flags] 3360 3361 Options 3362 3363 -h, --help help for providers 3364 3365 See the global flags page for global options not listed here. 3366 3367 SEE ALSO 3368 3369 - rclone config - Enter an interactive configuration session. 3370 3371 rclone config reconnect 3372 3373 Re-authenticates user with remote. 3374 3375 Synopsis 3376 3377 This reconnects remote: passed in to the cloud storage system. 3378 3379 To disconnect the remote use "rclone config disconnect". 3380 3381 This normally means going through the interactive oauth flow again. 3382 3383 rclone config reconnect remote: [flags] 3384 3385 Options 3386 3387 -h, --help help for reconnect 3388 3389 See the global flags page for global options not listed here. 3390 3391 SEE ALSO 3392 3393 - rclone config - Enter an interactive configuration session. 3394 3395 rclone config redacted 3396 3397 Print redacted (decrypted) config file, or the redacted config for a 3398 single remote. 3399 3400 Synopsis 3401 3402 This prints a redacted copy of the config file, either the whole config 3403 file or for a given remote. 3404 3405 The config file will be redacted by replacing all passwords and other 3406 sensitive info with XXX. 3407 3408 This makes the config file suitable for posting online for support. 3409 3410 It should be double checked before posting as the redaction may not be 3411 perfect. 3412 3413 rclone config redacted [<remote>] [flags] 3414 3415 Options 3416 3417 -h, --help help for redacted 3418 3419 See the global flags page for global options not listed here. 3420 3421 SEE ALSO 3422 3423 - rclone config - Enter an interactive configuration session. 3424 3425 rclone config show 3426 3427 Print (decrypted) config file, or the config for a single remote. 3428 3429 rclone config show [<remote>] [flags] 3430 3431 Options 3432 3433 -h, --help help for show 3434 3435 See the global flags page for global options not listed here. 3436 3437 SEE ALSO 3438 3439 - rclone config - Enter an interactive configuration session. 3440 3441 rclone config touch 3442 3443 Ensure configuration file exists. 3444 3445 rclone config touch [flags] 3446 3447 Options 3448 3449 -h, --help help for touch 3450 3451 See the global flags page for global options not listed here. 3452 3453 SEE ALSO 3454 3455 - rclone config - Enter an interactive configuration session. 3456 3457 rclone config update 3458 3459 Update options in an existing remote. 3460 3461 Synopsis 3462 3463 Update an existing remote's options. The options should be passed in 3464 pairs of key value or as key=value. 3465 3466 For example, to update the env_auth field of a remote of name myremote 3467 you would do: 3468 3469 rclone config update myremote env_auth true 3470 rclone config update myremote env_auth=true 3471 3472 If the remote uses OAuth the token will be updated, if you don't require 3473 this add an extra parameter thus: 3474 3475 rclone config update myremote env_auth=true config_refresh_token=false 3476 3477 Note that if the config process would normally ask a question the 3478 default is taken (unless --non-interactive is used). Each time that 3479 happens rclone will print or DEBUG a message saying how to affect the 3480 value taken. 3481 3482 If any of the parameters passed is a password field, then rclone will 3483 automatically obscure them if they aren't already obscured before 3484 putting them in the config file. 3485 3486 NB If the password parameter is 22 characters or longer and consists 3487 only of base64 characters then rclone can get confused about whether the 3488 password is already obscured or not and put unobscured passwords into 3489 the config file. If you want to be 100% certain that the passwords get 3490 obscured then use the --obscure flag, or if you are 100% certain you are 3491 already passing obscured passwords then use --no-obscure. You can also 3492 set obscured passwords using the rclone config password command. 3493 3494 The flag --non-interactive is for use by applications that wish to 3495 configure rclone themselves, rather than using rclone's text based 3496 configuration questions. If this flag is set, and rclone needs to ask 3497 the user a question, a JSON blob will be returned with the question in 3498 it. 3499 3500 This will look something like (some irrelevant detail removed): 3501 3502 { 3503 "State": "*oauth-islocal,teamdrive,,", 3504 "Option": { 3505 "Name": "config_is_local", 3506 "Help": "Use web browser to automatically authenticate rclone with remote?\n * Say Y if the machine running rclone has a web browser you can use\n * Say N if running rclone on a (remote) machine without web browser access\nIf not sure try Y. If Y failed, try N.\n", 3507 "Default": true, 3508 "Examples": [ 3509 { 3510 "Value": "true", 3511 "Help": "Yes" 3512 }, 3513 { 3514 "Value": "false", 3515 "Help": "No" 3516 } 3517 ], 3518 "Required": false, 3519 "IsPassword": false, 3520 "Type": "bool", 3521 "Exclusive": true, 3522 }, 3523 "Error": "", 3524 } 3525 3526 The format of Option is the same as returned by rclone config providers. 3527 The question should be asked to the user and returned to rclone as the 3528 --result option along with the --state parameter. 3529 3530 The keys of Option are used as follows: 3531 3532 - Name - name of variable - show to user 3533 - Help - help text. Hard wrapped at 80 chars. Any URLs should be 3534 clicky. 3535 - Default - default value - return this if the user just wants the 3536 default. 3537 - Examples - the user should be able to choose one of these 3538 - Required - the value should be non-empty 3539 - IsPassword - the value is a password and should be edited as such 3540 - Type - type of value, eg bool, string, int and others 3541 - Exclusive - if set no free-form entry allowed only the Examples 3542 - Irrelevant keys Provider, ShortOpt, Hide, NoPrefix, Advanced 3543 3544 If Error is set then it should be shown to the user at the same time as 3545 the question. 3546 3547 rclone config update name --continue --state "*oauth-islocal,teamdrive,," --result "true" 3548 3549 Note that when using --continue all passwords should be passed in the 3550 clear (not obscured). Any default config values should be passed in with 3551 each invocation of --continue. 3552 3553 At the end of the non interactive process, rclone will return a result 3554 with State as empty string. 3555 3556 If --all is passed then rclone will ask all the config questions, not 3557 just the post config questions. Any parameters are used as defaults for 3558 questions as usual. 3559 3560 Note that bin/config.py in the rclone source implements this protocol as 3561 a readable demonstration. 3562 3563 rclone config update name [key value]+ [flags] 3564 3565 Options 3566 3567 --all Ask the full set of config questions 3568 --continue Continue the configuration process with an answer 3569 -h, --help help for update 3570 --no-obscure Force any passwords not to be obscured 3571 --non-interactive Don't interact with user and return questions 3572 --obscure Force any passwords to be obscured 3573 --result string Result - use with --continue 3574 --state string State - use with --continue 3575 3576 See the global flags page for global options not listed here. 3577 3578 SEE ALSO 3579 3580 - rclone config - Enter an interactive configuration session. 3581 3582 rclone config userinfo 3583 3584 Prints info about logged in user of remote. 3585 3586 Synopsis 3587 3588 This prints the details of the person logged in to the cloud storage 3589 system. 3590 3591 rclone config userinfo remote: [flags] 3592 3593 Options 3594 3595 -h, --help help for userinfo 3596 --json Format output as JSON 3597 3598 See the global flags page for global options not listed here. 3599 3600 SEE ALSO 3601 3602 - rclone config - Enter an interactive configuration session. 3603 3604 rclone copyto 3605 3606 Copy files from source to dest, skipping identical files. 3607 3608 Synopsis 3609 3610 If source:path is a file or directory then it copies it to a file or 3611 directory named dest:path. 3612 3613 This can be used to upload single files to other than their current 3614 name. If the source is a directory then it acts exactly like the copy 3615 command. 3616 3617 So 3618 3619 rclone copyto src dst 3620 3621 where src and dst are rclone paths, either remote:path or /path/to/local 3622 or C:. 3623 3624 This will: 3625 3626 if src is file 3627 copy it to dst, overwriting an existing file if it exists 3628 if src is directory 3629 copy it to dst, overwriting existing files if they exist 3630 see copy command for full details 3631 3632 This doesn't transfer files that are identical on src and dst, testing 3633 by size and modification time or MD5SUM. It doesn't delete files from 3634 the destination. 3635 3636 Note: Use the -P/--progress flag to view real-time transfer statistics 3637 3638 rclone copyto source:path dest:path [flags] 3639 3640 Options 3641 3642 -h, --help help for copyto 3643 3644 Copy Options 3645 3646 Flags for anything which can Copy a file. 3647 3648 --check-first Do all the checks before starting transfers 3649 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 3650 --compare-dest stringArray Include additional comma separated server-side paths during comparison 3651 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 3652 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 3653 --ignore-case-sync Ignore case when synchronizing 3654 --ignore-checksum Skip post copy check of checksums 3655 --ignore-existing Skip all files that exist on destination 3656 --ignore-size Ignore size when skipping use modtime or checksum 3657 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 3658 --immutable Do not modify files, fail if existing files have been modified 3659 --inplace Download directly to destination file instead of atomic download to temp/rename 3660 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 3661 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 3662 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 3663 -M, --metadata If set, preserve metadata when copying objects 3664 --modify-window Duration Max time diff to be considered the same (default 1ns) 3665 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 3666 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 3667 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 3668 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 3669 --no-check-dest Don't check the destination, copy regardless 3670 --no-traverse Don't traverse destination file system on copy 3671 --no-update-dir-modtime Don't update directory modification times 3672 --no-update-modtime Don't update destination modtime if files identical 3673 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 3674 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 3675 --refresh-times Refresh the modtime of remote files 3676 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 3677 --size-only Skip based on size only, not modtime or checksum 3678 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 3679 -u, --update Skip files that are newer on the destination 3680 3681 Important Options 3682 3683 Important flags useful for most commands. 3684 3685 -n, --dry-run Do a trial run with no permanent changes 3686 -i, --interactive Enable interactive mode 3687 -v, --verbose count Print lots more stuff (repeat for more) 3688 3689 Filter Options 3690 3691 Flags for filtering directory listings. 3692 3693 --delete-excluded Delete files on dest excluded from sync 3694 --exclude stringArray Exclude files matching pattern 3695 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 3696 --exclude-if-present stringArray Exclude directories if filename is present 3697 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 3698 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 3699 -f, --filter stringArray Add a file filtering rule 3700 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 3701 --ignore-case Ignore case in filters (case insensitive) 3702 --include stringArray Include files matching pattern 3703 --include-from stringArray Read file include patterns from file (use - to read from stdin) 3704 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 3705 --max-depth int If set limits the recursion depth to this (default -1) 3706 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 3707 --metadata-exclude stringArray Exclude metadatas matching pattern 3708 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 3709 --metadata-filter stringArray Add a metadata filtering rule 3710 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 3711 --metadata-include stringArray Include metadatas matching pattern 3712 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 3713 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 3714 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 3715 3716 Listing Options 3717 3718 Flags for listing directories. 3719 3720 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 3721 --fast-list Use recursive list if available; uses more memory but fewer transactions 3722 3723 See the global flags page for global options not listed here. 3724 3725 SEE ALSO 3726 3727 - rclone - Show help for rclone commands, flags and backends. 3728 3729 rclone copyurl 3730 3731 Copy the contents of the URL supplied content to dest:path. 3732 3733 Synopsis 3734 3735 Download a URL's content and copy it to the destination without saving 3736 it in temporary storage. 3737 3738 Setting --auto-filename will attempt to automatically determine the 3739 filename from the URL (after any redirections) and used in the 3740 destination path. 3741 3742 With --auto-filename-header in addition, if a specific filename is set 3743 in HTTP headers, it will be used instead of the name from the URL. With 3744 --print-filename in addition, the resulting file name will be printed. 3745 3746 Setting --no-clobber will prevent overwriting file on the destination if 3747 there is one with the same name. 3748 3749 Setting --stdout or making the output file name - will cause the output 3750 to be written to standard output. 3751 3752 Troublshooting 3753 3754 If you can't get rclone copyurl to work then here are some things you 3755 can try: 3756 3757 - --disable-http2 rclone will use HTTP2 if available - try disabling 3758 it 3759 - --bind 0.0.0.0 rclone will use IPv6 if available - try disabling it 3760 - --bind ::0 to disable IPv4 3761 - --user agent curl - some sites have whitelists for curl's 3762 user-agent - try that 3763 - Make sure the site works with curl directly 3764 3765 rclone copyurl https://example.com dest:path [flags] 3766 3767 Options 3768 3769 -a, --auto-filename Get the file name from the URL and use it for destination file path 3770 --header-filename Get the file name from the Content-Disposition header 3771 -h, --help help for copyurl 3772 --no-clobber Prevent overwriting file with same name 3773 -p, --print-filename Print the resulting name from --auto-filename 3774 --stdout Write the output to stdout rather than a file 3775 3776 Important Options 3777 3778 Important flags useful for most commands. 3779 3780 -n, --dry-run Do a trial run with no permanent changes 3781 -i, --interactive Enable interactive mode 3782 -v, --verbose count Print lots more stuff (repeat for more) 3783 3784 See the global flags page for global options not listed here. 3785 3786 SEE ALSO 3787 3788 - rclone - Show help for rclone commands, flags and backends. 3789 3790 rclone cryptcheck 3791 3792 Cryptcheck checks the integrity of an encrypted remote. 3793 3794 Synopsis 3795 3796 rclone cryptcheck checks a remote against a crypted remote. This is the 3797 equivalent of running rclone check, but able to check the checksums of 3798 the encrypted remote. 3799 3800 For it to work the underlying remote of the cryptedremote must support 3801 some kind of checksum. 3802 3803 It works by reading the nonce from each file on the cryptedremote: and 3804 using that to encrypt each file on the remote:. It then checks the 3805 checksum of the underlying file on the cryptedremote: against the 3806 checksum of the file it has just encrypted. 3807 3808 Use it like this 3809 3810 rclone cryptcheck /path/to/files encryptedremote:path 3811 3812 You can use it like this also, but that will involve downloading all the 3813 files in remote:path. 3814 3815 rclone cryptcheck remote:path encryptedremote:path 3816 3817 After it has run it will log the status of the encryptedremote:. 3818 3819 If you supply the --one-way flag, it will only check that files in the 3820 source match the files in the destination, not the other way around. 3821 This means that extra files in the destination that are not in the 3822 source will not be detected. 3823 3824 The --differ, --missing-on-dst, --missing-on-src, --match and --error 3825 flags write paths, one per line, to the file name (or stdout if it is -) 3826 supplied. What they write is described in the help below. For example 3827 --differ will write all paths which are present on both the source and 3828 destination but different. 3829 3830 The --combined flag will write a file (or stdout) which contains all 3831 file paths with a symbol and then a space and then the path to tell you 3832 what happened to it. These are reminiscent of diff files. 3833 3834 - = path means path was found in source and destination and was 3835 identical 3836 - `- path` means path was missing on the source, so only in the 3837 destination 3838 - `+ path` means path was missing on the destination, so only in the 3839 source 3840 - `* path` means path was present in source and destination but 3841 different. 3842 - ! path means there was an error reading or hashing the source or 3843 dest. 3844 3845 The default number of parallel checks is 8. See the --checkers=N option 3846 for more information. 3847 3848 rclone cryptcheck remote:path cryptedremote:path [flags] 3849 3850 Options 3851 3852 --combined string Make a combined report of changes to this file 3853 --differ string Report all non-matching files to this file 3854 --error string Report all files with errors (hashing or reading) to this file 3855 -h, --help help for cryptcheck 3856 --match string Report all matching files to this file 3857 --missing-on-dst string Report all files missing from the destination to this file 3858 --missing-on-src string Report all files missing from the source to this file 3859 --one-way Check one way only, source files must exist on remote 3860 3861 Check Options 3862 3863 Flags used for rclone check. 3864 3865 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 3866 3867 Filter Options 3868 3869 Flags for filtering directory listings. 3870 3871 --delete-excluded Delete files on dest excluded from sync 3872 --exclude stringArray Exclude files matching pattern 3873 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 3874 --exclude-if-present stringArray Exclude directories if filename is present 3875 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 3876 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 3877 -f, --filter stringArray Add a file filtering rule 3878 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 3879 --ignore-case Ignore case in filters (case insensitive) 3880 --include stringArray Include files matching pattern 3881 --include-from stringArray Read file include patterns from file (use - to read from stdin) 3882 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 3883 --max-depth int If set limits the recursion depth to this (default -1) 3884 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 3885 --metadata-exclude stringArray Exclude metadatas matching pattern 3886 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 3887 --metadata-filter stringArray Add a metadata filtering rule 3888 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 3889 --metadata-include stringArray Include metadatas matching pattern 3890 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 3891 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 3892 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 3893 3894 Listing Options 3895 3896 Flags for listing directories. 3897 3898 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 3899 --fast-list Use recursive list if available; uses more memory but fewer transactions 3900 3901 See the global flags page for global options not listed here. 3902 3903 SEE ALSO 3904 3905 - rclone - Show help for rclone commands, flags and backends. 3906 3907 rclone cryptdecode 3908 3909 Cryptdecode returns unencrypted file names. 3910 3911 Synopsis 3912 3913 rclone cryptdecode returns unencrypted file names when provided with a 3914 list of encrypted file names. List limit is 10 items. 3915 3916 If you supply the --reverse flag, it will return encrypted file names. 3917 3918 use it like this 3919 3920 rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2 3921 3922 rclone cryptdecode --reverse encryptedremote: filename1 filename2 3923 3924 Another way to accomplish this is by using the rclone backend encode (or 3925 decode) command. See the documentation on the crypt overlay for more 3926 info. 3927 3928 rclone cryptdecode encryptedremote: encryptedfilename [flags] 3929 3930 Options 3931 3932 -h, --help help for cryptdecode 3933 --reverse Reverse cryptdecode, encrypts filenames 3934 3935 See the global flags page for global options not listed here. 3936 3937 SEE ALSO 3938 3939 - rclone - Show help for rclone commands, flags and backends. 3940 3941 rclone deletefile 3942 3943 Remove a single file from remote. 3944 3945 Synopsis 3946 3947 Remove a single file from remote. Unlike delete it cannot be used to 3948 remove a directory and it doesn't obey include/exclude filters - if the 3949 specified file exists, it will always be removed. 3950 3951 rclone deletefile remote:path [flags] 3952 3953 Options 3954 3955 -h, --help help for deletefile 3956 3957 Important Options 3958 3959 Important flags useful for most commands. 3960 3961 -n, --dry-run Do a trial run with no permanent changes 3962 -i, --interactive Enable interactive mode 3963 -v, --verbose count Print lots more stuff (repeat for more) 3964 3965 See the global flags page for global options not listed here. 3966 3967 SEE ALSO 3968 3969 - rclone - Show help for rclone commands, flags and backends. 3970 3971 rclone genautocomplete 3972 3973 Output completion script for a given shell. 3974 3975 Synopsis 3976 3977 Generates a shell completion script for rclone. Run with --help to list 3978 the supported shells. 3979 3980 Options 3981 3982 -h, --help help for genautocomplete 3983 3984 See the global flags page for global options not listed here. 3985 3986 SEE ALSO 3987 3988 - rclone - Show help for rclone commands, flags and backends. 3989 - rclone genautocomplete bash - Output bash completion script for 3990 rclone. 3991 - rclone genautocomplete fish - Output fish completion script for 3992 rclone. 3993 - rclone genautocomplete zsh - Output zsh completion script for 3994 rclone. 3995 3996 rclone genautocomplete bash 3997 3998 Output bash completion script for rclone. 3999 4000 Synopsis 4001 4002 Generates a bash shell autocompletion script for rclone. 4003 4004 This writes to /etc/bash_completion.d/rclone by default so will probably 4005 need to be run with sudo or as root, e.g. 4006 4007 sudo rclone genautocomplete bash 4008 4009 Logout and login again to use the autocompletion scripts, or source them 4010 directly 4011 4012 . /etc/bash_completion 4013 4014 If you supply a command line argument the script will be written there. 4015 4016 If output_file is "-", then the output will be written to stdout. 4017 4018 rclone genautocomplete bash [output_file] [flags] 4019 4020 Options 4021 4022 -h, --help help for bash 4023 4024 See the global flags page for global options not listed here. 4025 4026 SEE ALSO 4027 4028 - rclone genautocomplete - Output completion script for a given shell. 4029 4030 rclone genautocomplete fish 4031 4032 Output fish completion script for rclone. 4033 4034 Synopsis 4035 4036 Generates a fish autocompletion script for rclone. 4037 4038 This writes to /etc/fish/completions/rclone.fish by default so will 4039 probably need to be run with sudo or as root, e.g. 4040 4041 sudo rclone genautocomplete fish 4042 4043 Logout and login again to use the autocompletion scripts, or source them 4044 directly 4045 4046 . /etc/fish/completions/rclone.fish 4047 4048 If you supply a command line argument the script will be written there. 4049 4050 If output_file is "-", then the output will be written to stdout. 4051 4052 rclone genautocomplete fish [output_file] [flags] 4053 4054 Options 4055 4056 -h, --help help for fish 4057 4058 See the global flags page for global options not listed here. 4059 4060 SEE ALSO 4061 4062 - rclone genautocomplete - Output completion script for a given shell. 4063 4064 rclone genautocomplete zsh 4065 4066 Output zsh completion script for rclone. 4067 4068 Synopsis 4069 4070 Generates a zsh autocompletion script for rclone. 4071 4072 This writes to /usr/share/zsh/vendor-completions/_rclone by default so 4073 will probably need to be run with sudo or as root, e.g. 4074 4075 sudo rclone genautocomplete zsh 4076 4077 Logout and login again to use the autocompletion scripts, or source them 4078 directly 4079 4080 autoload -U compinit && compinit 4081 4082 If you supply a command line argument the script will be written there. 4083 4084 If output_file is "-", then the output will be written to stdout. 4085 4086 rclone genautocomplete zsh [output_file] [flags] 4087 4088 Options 4089 4090 -h, --help help for zsh 4091 4092 See the global flags page for global options not listed here. 4093 4094 SEE ALSO 4095 4096 - rclone genautocomplete - Output completion script for a given shell. 4097 4098 rclone gendocs 4099 4100 Output markdown docs for rclone to the directory supplied. 4101 4102 Synopsis 4103 4104 This produces markdown docs for the rclone commands to the directory 4105 supplied. These are in a format suitable for hugo to render into the 4106 rclone.org website. 4107 4108 rclone gendocs output_directory [flags] 4109 4110 Options 4111 4112 -h, --help help for gendocs 4113 4114 See the global flags page for global options not listed here. 4115 4116 SEE ALSO 4117 4118 - rclone - Show help for rclone commands, flags and backends. 4119 4120 rclone hashsum 4121 4122 Produces a hashsum file for all the objects in the path. 4123 4124 Synopsis 4125 4126 Produces a hash file for all the objects in the path using the hash 4127 named. The output is in the same format as the standard md5sum/sha1sum 4128 tool. 4129 4130 By default, the hash is requested from the remote. If the hash is not 4131 supported by the remote, no hash will be returned. With the download 4132 flag, the file will be downloaded from the remote and hashed locally 4133 enabling any hash for any remote. 4134 4135 For the MD5 and SHA1 algorithms there are also dedicated commands, 4136 md5sum and sha1sum. 4137 4138 This command can also hash data received on standard input (stdin), by 4139 not passing a remote:path, or by passing a hyphen as remote:path when 4140 there is data to read (if not, the hyphen will be treated literally, as 4141 a relative path). 4142 4143 Run without a hash to see the list of all supported hashes, e.g. 4144 4145 $ rclone hashsum 4146 Supported hashes are: 4147 * md5 4148 * sha1 4149 * whirlpool 4150 * crc32 4151 * sha256 4152 4153 Then 4154 4155 $ rclone hashsum MD5 remote:path 4156 4157 Note that hash names are case insensitive and values are output in lower 4158 case. 4159 4160 rclone hashsum [<hash> remote:path] [flags] 4161 4162 Options 4163 4164 --base64 Output base64 encoded hashsum 4165 -C, --checkfile string Validate hashes against a given SUM file instead of printing them 4166 --download Download the file and hash it locally; if this flag is not specified, the hash is requested from the remote 4167 -h, --help help for hashsum 4168 --output-file string Output hashsums to a file rather than the terminal 4169 4170 Filter Options 4171 4172 Flags for filtering directory listings. 4173 4174 --delete-excluded Delete files on dest excluded from sync 4175 --exclude stringArray Exclude files matching pattern 4176 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 4177 --exclude-if-present stringArray Exclude directories if filename is present 4178 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 4179 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 4180 -f, --filter stringArray Add a file filtering rule 4181 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 4182 --ignore-case Ignore case in filters (case insensitive) 4183 --include stringArray Include files matching pattern 4184 --include-from stringArray Read file include patterns from file (use - to read from stdin) 4185 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4186 --max-depth int If set limits the recursion depth to this (default -1) 4187 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 4188 --metadata-exclude stringArray Exclude metadatas matching pattern 4189 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 4190 --metadata-filter stringArray Add a metadata filtering rule 4191 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 4192 --metadata-include stringArray Include metadatas matching pattern 4193 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 4194 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4195 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 4196 4197 Listing Options 4198 4199 Flags for listing directories. 4200 4201 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 4202 --fast-list Use recursive list if available; uses more memory but fewer transactions 4203 4204 See the global flags page for global options not listed here. 4205 4206 SEE ALSO 4207 4208 - rclone - Show help for rclone commands, flags and backends. 4209 4210 rclone link 4211 4212 Generate public link to file/folder. 4213 4214 Synopsis 4215 4216 rclone link will create, retrieve or remove a public link to the given 4217 file or folder. 4218 4219 rclone link remote:path/to/file 4220 rclone link remote:path/to/folder/ 4221 rclone link --unlink remote:path/to/folder/ 4222 rclone link --expire 1d remote:path/to/file 4223 4224 If you supply the --expire flag, it will set the expiration time 4225 otherwise it will use the default (100 years). Note not all backends 4226 support the --expire flag - if the backend doesn't support it then the 4227 link returned won't expire. 4228 4229 Use the --unlink flag to remove existing public links to the file or 4230 folder. Note not all backends support "--unlink" flag - those that don't 4231 will just ignore it. 4232 4233 If successful, the last line of the output will contain the link. Exact 4234 capabilities depend on the remote, but the link will always by default 4235 be created with the least constraints – e.g. no expiry, no password 4236 protection, accessible without account. 4237 4238 rclone link remote:path [flags] 4239 4240 Options 4241 4242 --expire Duration The amount of time that the link will be valid (default off) 4243 -h, --help help for link 4244 --unlink Remove existing public link to file/folder 4245 4246 See the global flags page for global options not listed here. 4247 4248 SEE ALSO 4249 4250 - rclone - Show help for rclone commands, flags and backends. 4251 4252 rclone listremotes 4253 4254 List all the remotes in the config file and defined in environment 4255 variables. 4256 4257 Synopsis 4258 4259 rclone listremotes lists all the available remotes from the config file. 4260 4261 When used with the --long flag it lists the types and the descriptions 4262 too. 4263 4264 rclone listremotes [flags] 4265 4266 Options 4267 4268 -h, --help help for listremotes 4269 --long Show the type and the description as well as names 4270 4271 See the global flags page for global options not listed here. 4272 4273 SEE ALSO 4274 4275 - rclone - Show help for rclone commands, flags and backends. 4276 4277 rclone lsf 4278 4279 List directories and objects in remote:path formatted for parsing. 4280 4281 Synopsis 4282 4283 List the contents of the source path (directories and objects) to 4284 standard output in a form which is easy to parse by scripts. By default 4285 this will just be the names of the objects and directories, one per 4286 line. The directories will have a / suffix. 4287 4288 Eg 4289 4290 $ rclone lsf swift:bucket 4291 bevajer5jef 4292 canole 4293 diwogej7 4294 ferejej3gux/ 4295 fubuwic 4296 4297 Use the --format option to control what gets listed. By default this is 4298 just the path, but you can use these parameters to control the output: 4299 4300 p - path 4301 s - size 4302 t - modification time 4303 h - hash 4304 i - ID of object 4305 o - Original ID of underlying object 4306 m - MimeType of object if known 4307 e - encrypted name 4308 T - tier of storage if known, e.g. "Hot" or "Cool" 4309 M - Metadata of object in JSON blob format, eg {"key":"value"} 4310 4311 So if you wanted the path, size and modification time, you would use 4312 --format "pst", or maybe --format "tsp" to put the path last. 4313 4314 Eg 4315 4316 $ rclone lsf --format "tsp" swift:bucket 4317 2016-06-25 18:55:41;60295;bevajer5jef 4318 2016-06-25 18:55:43;90613;canole 4319 2016-06-25 18:55:43;94467;diwogej7 4320 2018-04-26 08:50:45;0;ferejej3gux/ 4321 2016-06-25 18:55:40;37600;fubuwic 4322 4323 If you specify "h" in the format you will get the MD5 hash by default, 4324 use the --hash flag to change which hash you want. Note that this can be 4325 returned as an empty string if it isn't available on the object (and for 4326 directories), "ERROR" if there was an error reading it from the object 4327 and "UNSUPPORTED" if that object does not support that hash type. 4328 4329 For example, to emulate the md5sum command you can use 4330 4331 rclone lsf -R --hash MD5 --format hp --separator " " --files-only . 4332 4333 Eg 4334 4335 $ rclone lsf -R --hash MD5 --format hp --separator " " --files-only swift:bucket 4336 7908e352297f0f530b84a756f188baa3 bevajer5jef 4337 cd65ac234e6fea5925974a51cdd865cc canole 4338 03b5341b4f234b9d984d03ad076bae91 diwogej7 4339 8fd37c3810dd660778137ac3a66cc06d fubuwic 4340 99713e14a4c4ff553acaf1930fad985b gixacuh7ku 4341 4342 (Though "rclone md5sum ." is an easier way of typing this.) 4343 4344 By default the separator is ";" this can be changed with the --separator 4345 flag. Note that separators aren't escaped in the path so putting it last 4346 is a good strategy. 4347 4348 Eg 4349 4350 $ rclone lsf --separator "," --format "tshp" swift:bucket 4351 2016-06-25 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef 4352 2016-06-25 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole 4353 2016-06-25 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7 4354 2018-04-26 08:52:53,0,,ferejej3gux/ 4355 2016-06-25 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic 4356 4357 You can output in CSV standard format. This will escape things in " if 4358 they contain , 4359 4360 Eg 4361 4362 $ rclone lsf --csv --files-only --format ps remote:path 4363 test.log,22355 4364 test.sh,449 4365 "this file contains a comma, in the file name.txt",6 4366 4367 Note that the --absolute parameter is useful for making lists of files 4368 to pass to an rclone copy with the --files-from-raw flag. 4369 4370 For example, to find all the files modified within one day and copy 4371 those only (without traversing the whole directory structure): 4372 4373 rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files 4374 rclone copy --files-from-raw new_files /path/to/local remote:path 4375 4376 The default time format is '2006-01-02 15:04:05'. Other formats can be 4377 specified with the --time-format flag. Examples: 4378 4379 rclone lsf remote:path --format pt --time-format 'Jan 2, 2006 at 3:04pm (MST)' 4380 rclone lsf remote:path --format pt --time-format '2006-01-02 15:04:05.000000000' 4381 rclone lsf remote:path --format pt --time-format '2006-01-02T15:04:05.999999999Z07:00' 4382 rclone lsf remote:path --format pt --time-format RFC3339 4383 rclone lsf remote:path --format pt --time-format DateOnly 4384 rclone lsf remote:path --format pt --time-format max 4385 4386 --time-format max will automatically truncate 4387 '2006-01-02 15:04:05.000000000' to the maximum precision supported by 4388 the remote. 4389 4390 Any of the filtering options can be applied to this command. 4391 4392 There are several related list commands 4393 4394 - ls to list size and path of objects only 4395 - lsl to list modification time, size and path of objects only 4396 - lsd to list directories only 4397 - lsf to list objects and directories in easy to parse format 4398 - lsjson to list objects and directories in JSON format 4399 4400 ls,lsl,lsd are designed to be human-readable. lsf is designed to be 4401 human and machine-readable. lsjson is designed to be machine-readable. 4402 4403 Note that ls and lsl recurse by default - use --max-depth 1 to stop the 4404 recursion. 4405 4406 The other list commands lsd,lsf,lsjson do not recurse by default - use 4407 -R to make them recurse. 4408 4409 Listing a nonexistent directory will produce an error except for remotes 4410 which can't have empty directories (e.g. s3, swift, or gcs - the 4411 bucket-based remotes). 4412 4413 rclone lsf remote:path [flags] 4414 4415 Options 4416 4417 --absolute Put a leading / in front of path names 4418 --csv Output in CSV format 4419 -d, --dir-slash Append a slash to directory names (default true) 4420 --dirs-only Only list directories 4421 --files-only Only list files 4422 -F, --format string Output format - see help for details (default "p") 4423 --hash h Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "md5") 4424 -h, --help help for lsf 4425 -R, --recursive Recurse into the listing 4426 -s, --separator string Separator for the items in the format (default ";") 4427 -t, --time-format string Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05) 4428 4429 Filter Options 4430 4431 Flags for filtering directory listings. 4432 4433 --delete-excluded Delete files on dest excluded from sync 4434 --exclude stringArray Exclude files matching pattern 4435 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 4436 --exclude-if-present stringArray Exclude directories if filename is present 4437 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 4438 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 4439 -f, --filter stringArray Add a file filtering rule 4440 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 4441 --ignore-case Ignore case in filters (case insensitive) 4442 --include stringArray Include files matching pattern 4443 --include-from stringArray Read file include patterns from file (use - to read from stdin) 4444 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4445 --max-depth int If set limits the recursion depth to this (default -1) 4446 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 4447 --metadata-exclude stringArray Exclude metadatas matching pattern 4448 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 4449 --metadata-filter stringArray Add a metadata filtering rule 4450 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 4451 --metadata-include stringArray Include metadatas matching pattern 4452 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 4453 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4454 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 4455 4456 Listing Options 4457 4458 Flags for listing directories. 4459 4460 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 4461 --fast-list Use recursive list if available; uses more memory but fewer transactions 4462 4463 See the global flags page for global options not listed here. 4464 4465 SEE ALSO 4466 4467 - rclone - Show help for rclone commands, flags and backends. 4468 4469 rclone lsjson 4470 4471 List directories and objects in the path in JSON format. 4472 4473 Synopsis 4474 4475 List directories and objects in the path in JSON format. 4476 4477 The output is an array of Items, where each Item looks like this 4478 4479 { 4480 "Hashes" : { 4481 "SHA-1" : "f572d396fae9206628714fb2ce00f72e94f2258f", 4482 "MD5" : "b1946ac92492d2347c6235b4d2611184", 4483 "DropboxHash" : "ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc" 4484 }, 4485 "ID": "y2djkhiujf83u33", 4486 "OrigID": "UYOJVTUW00Q1RzTDA", 4487 "IsBucket" : false, 4488 "IsDir" : false, 4489 "MimeType" : "application/octet-stream", 4490 "ModTime" : "2017-05-31T16:15:57.034468261+01:00", 4491 "Name" : "file.txt", 4492 "Encrypted" : "v0qpsdq8anpci8n929v3uu9338", 4493 "EncryptedPath" : "kja9098349023498/v0qpsdq8anpci8n929v3uu9338", 4494 "Path" : "full/path/goes/here/file.txt", 4495 "Size" : 6, 4496 "Tier" : "hot", 4497 } 4498 4499 If --hash is not specified the Hashes property won't be emitted. The 4500 types of hash can be specified with the --hash-type parameter (which may 4501 be repeated). If --hash-type is set then it implies --hash. 4502 4503 If --no-modtime is specified then ModTime will be blank. This can speed 4504 things up on remotes where reading the ModTime takes an extra request 4505 (e.g. s3, swift). 4506 4507 If --no-mimetype is specified then MimeType will be blank. This can 4508 speed things up on remotes where reading the MimeType takes an extra 4509 request (e.g. s3, swift). 4510 4511 If --encrypted is not specified the Encrypted won't be emitted. 4512 4513 If --dirs-only is not specified files in addition to directories are 4514 returned 4515 4516 If --files-only is not specified directories in addition to the files 4517 will be returned. 4518 4519 If --metadata is set then an additional Metadata key will be returned. 4520 This will have metadata in rclone standard format as a JSON object. 4521 4522 if --stat is set then a single JSON blob will be returned about the item 4523 pointed to. This will return an error if the item isn't found. However 4524 on bucket based backends (like s3, gcs, b2, azureblob etc) if the item 4525 isn't found it will return an empty directory as it isn't possible to 4526 tell empty directories from missing directories there. 4527 4528 The Path field will only show folders below the remote path being 4529 listed. If "remote:path" contains the file "subfolder/file.txt", the 4530 Path for "file.txt" will be "subfolder/file.txt", not 4531 "remote:path/subfolder/file.txt". When used without --recursive the Path 4532 will always be the same as Name. 4533 4534 If the directory is a bucket in a bucket-based backend, then "IsBucket" 4535 will be set to true. This key won't be present unless it is "true". 4536 4537 The time is in RFC3339 format with up to nanosecond precision. The 4538 number of decimal digits in the seconds will depend on the precision 4539 that the remote can hold the times, so if times are accurate to the 4540 nearest millisecond (e.g. Google Drive) then 3 digits will always be 4541 shown ("2017-05-31T16:15:57.034+01:00") whereas if the times are 4542 accurate to the nearest second (Dropbox, Box, WebDav, etc.) no digits 4543 will be shown ("2017-05-31T16:15:57+01:00"). 4544 4545 The whole output can be processed as a JSON blob, or alternatively it 4546 can be processed line by line as each item is written one to a line. 4547 4548 Any of the filtering options can be applied to this command. 4549 4550 There are several related list commands 4551 4552 - ls to list size and path of objects only 4553 - lsl to list modification time, size and path of objects only 4554 - lsd to list directories only 4555 - lsf to list objects and directories in easy to parse format 4556 - lsjson to list objects and directories in JSON format 4557 4558 ls,lsl,lsd are designed to be human-readable. lsf is designed to be 4559 human and machine-readable. lsjson is designed to be machine-readable. 4560 4561 Note that ls and lsl recurse by default - use --max-depth 1 to stop the 4562 recursion. 4563 4564 The other list commands lsd,lsf,lsjson do not recurse by default - use 4565 -R to make them recurse. 4566 4567 Listing a nonexistent directory will produce an error except for remotes 4568 which can't have empty directories (e.g. s3, swift, or gcs - the 4569 bucket-based remotes). 4570 4571 rclone lsjson remote:path [flags] 4572 4573 Options 4574 4575 --dirs-only Show only directories in the listing 4576 --encrypted Show the encrypted names 4577 --files-only Show only files in the listing 4578 --hash Include hashes in the output (may take longer) 4579 --hash-type stringArray Show only this hash type (may be repeated) 4580 -h, --help help for lsjson 4581 -M, --metadata Add metadata to the listing 4582 --no-mimetype Don't read the mime type (can speed things up) 4583 --no-modtime Don't read the modification time (can speed things up) 4584 --original Show the ID of the underlying Object 4585 -R, --recursive Recurse into the listing 4586 --stat Just return the info for the pointed to file 4587 4588 Filter Options 4589 4590 Flags for filtering directory listings. 4591 4592 --delete-excluded Delete files on dest excluded from sync 4593 --exclude stringArray Exclude files matching pattern 4594 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 4595 --exclude-if-present stringArray Exclude directories if filename is present 4596 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 4597 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 4598 -f, --filter stringArray Add a file filtering rule 4599 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 4600 --ignore-case Ignore case in filters (case insensitive) 4601 --include stringArray Include files matching pattern 4602 --include-from stringArray Read file include patterns from file (use - to read from stdin) 4603 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4604 --max-depth int If set limits the recursion depth to this (default -1) 4605 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 4606 --metadata-exclude stringArray Exclude metadatas matching pattern 4607 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 4608 --metadata-filter stringArray Add a metadata filtering rule 4609 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 4610 --metadata-include stringArray Include metadatas matching pattern 4611 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 4612 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 4613 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 4614 4615 Listing Options 4616 4617 Flags for listing directories. 4618 4619 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 4620 --fast-list Use recursive list if available; uses more memory but fewer transactions 4621 4622 See the global flags page for global options not listed here. 4623 4624 SEE ALSO 4625 4626 - rclone - Show help for rclone commands, flags and backends. 4627 4628 rclone mount 4629 4630 Mount the remote as file system on a mountpoint. 4631 4632 Synopsis 4633 4634 rclone mount allows Linux, FreeBSD, macOS and Windows to mount any of 4635 Rclone's cloud storage systems as a file system with FUSE. 4636 4637 First set up your remote using rclone config. Check it works with 4638 rclone ls etc. 4639 4640 On Linux and macOS, you can run mount in either foreground or background 4641 (aka daemon) mode. Mount runs in foreground mode by default. Use the 4642 --daemon flag to force background mode. On Windows you can run mount in 4643 foreground only, the flag is ignored. 4644 4645 In background mode rclone acts as a generic Unix mount program: the main 4646 program starts, spawns background rclone process to setup and maintain 4647 the mount, waits until success or timeout and exits with appropriate 4648 code (killing the child process if it fails). 4649 4650 On Linux/macOS/FreeBSD start the mount like this, where 4651 /path/to/local/mount is an empty existing directory: 4652 4653 rclone mount remote:path/to/files /path/to/local/mount 4654 4655 On Windows you can start a mount in different ways. See below for 4656 details. If foreground mount is used interactively from a console 4657 window, rclone will serve the mount and occupy the console so another 4658 window should be used to work with the mount until rclone is interrupted 4659 e.g. by pressing Ctrl-C. 4660 4661 The following examples will mount to an automatically assigned drive, to 4662 specific drive letter X:, to path C:\path\parent\mount (where parent 4663 directory or drive must exist, and mount must not exist, and is not 4664 supported when mounting as a network drive), and the last example will 4665 mount as network share \\cloud\remote and map it to an automatically 4666 assigned drive: 4667 4668 rclone mount remote:path/to/files * 4669 rclone mount remote:path/to/files X: 4670 rclone mount remote:path/to/files C:\path\parent\mount 4671 rclone mount remote:path/to/files \\cloud\remote 4672 4673 When the program ends while in foreground mode, either via Ctrl+C or 4674 receiving a SIGINT or SIGTERM signal, the mount should be automatically 4675 stopped. 4676 4677 When running in background mode the user will have to stop the mount 4678 manually: 4679 4680 # Linux 4681 fusermount -u /path/to/local/mount 4682 # OS X 4683 umount /path/to/local/mount 4684 4685 The umount operation can fail, for example when the mountpoint is busy. 4686 When that happens, it is the user's responsibility to stop the mount 4687 manually. 4688 4689 The size of the mounted file system will be set according to information 4690 retrieved from the remote, the same as returned by the rclone about 4691 command. Remotes with unlimited storage may report the used size only, 4692 then an additional 1 PiB of free space is assumed. If the remote does 4693 not support the about feature at all, then 1 PiB is set as both the 4694 total and the free size. 4695 4696 Installing on Windows 4697 4698 To run rclone mount on Windows, you will need to download and install 4699 WinFsp. 4700 4701 WinFsp is an open-source Windows File System Proxy which makes it easy 4702 to write user space file systems for Windows. It provides a FUSE 4703 emulation layer which rclone uses combination with cgofuse. Both of 4704 these packages are by Bill Zissimopoulos who was very helpful during the 4705 implementation of rclone mount for Windows. 4706 4707 Mounting modes on windows 4708 4709 Unlike other operating systems, Microsoft Windows provides a different 4710 filesystem type for network and fixed drives. It optimises access on the 4711 assumption fixed disk drives are fast and reliable, while network drives 4712 have relatively high latency and less reliability. Some settings can 4713 also be differentiated between the two types, for example that Windows 4714 Explorer should just display icons and not create preview thumbnails for 4715 image and video files on network drives. 4716 4717 In most cases, rclone will mount the remote as a normal, fixed disk 4718 drive by default. However, you can also choose to mount it as a remote 4719 network drive, often described as a network share. If you mount an 4720 rclone remote using the default, fixed drive mode and experience 4721 unexpected program errors, freezes or other issues, consider mounting as 4722 a network drive instead. 4723 4724 When mounting as a fixed disk drive you can either mount to an unused 4725 drive letter, or to a path representing a nonexistent subdirectory of an 4726 existing parent directory or drive. Using the special value * will tell 4727 rclone to automatically assign the next available drive letter, starting 4728 with Z: and moving backward. Examples: 4729 4730 rclone mount remote:path/to/files * 4731 rclone mount remote:path/to/files X: 4732 rclone mount remote:path/to/files C:\path\parent\mount 4733 rclone mount remote:path/to/files X: 4734 4735 Option --volname can be used to set a custom volume name for the mounted 4736 file system. The default is to use the remote name and path. 4737 4738 To mount as network drive, you can add option --network-mode to your 4739 mount command. Mounting to a directory path is not supported in this 4740 mode, it is a limitation Windows imposes on junctions, so the remote 4741 must always be mounted to a drive letter. 4742 4743 rclone mount remote:path/to/files X: --network-mode 4744 4745 A volume name specified with --volname will be used to create the 4746 network share path. A complete UNC path, such as \\cloud\remote, 4747 optionally with path \\cloud\remote\madeup\path, will be used as is. Any 4748 other string will be used as the share part, after a default prefix 4749 \\server\. If no volume name is specified then \\server\share will be 4750 used. You must make sure the volume name is unique when you are mounting 4751 more than one drive, or else the mount command will fail. The share name 4752 will treated as the volume label for the mapped drive, shown in Windows 4753 Explorer etc, while the complete \\server\share will be reported as the 4754 remote UNC path by net use etc, just like a normal network drive 4755 mapping. 4756 4757 If you specify a full network share UNC path with --volname, this will 4758 implicitly set the --network-mode option, so the following two examples 4759 have same result: 4760 4761 rclone mount remote:path/to/files X: --network-mode 4762 rclone mount remote:path/to/files X: --volname \\server\share 4763 4764 You may also specify the network share UNC path as the mountpoint 4765 itself. Then rclone will automatically assign a drive letter, same as 4766 with * and use that as mountpoint, and instead use the UNC path 4767 specified as the volume name, as if it were specified with the --volname 4768 option. This will also implicitly set the --network-mode option. This 4769 means the following two examples have same result: 4770 4771 rclone mount remote:path/to/files \\cloud\remote 4772 rclone mount remote:path/to/files * --volname \\cloud\remote 4773 4774 There is yet another way to enable network mode, and to set the share 4775 path, and that is to pass the "native" libfuse/WinFsp option directly: 4776 --fuse-flag --VolumePrefix=\server\share. Note that the path must be 4777 with just a single backslash prefix in this case. 4778 4779 Note: In previous versions of rclone this was the only supported method. 4780 4781 Read more about drive mapping 4782 4783 See also Limitations section below. 4784 4785 Windows filesystem permissions 4786 4787 The FUSE emulation layer on Windows must convert between the POSIX-based 4788 permission model used in FUSE, and the permission model used in Windows, 4789 based on access-control lists (ACL). 4790 4791 The mounted filesystem will normally get three entries in its 4792 access-control list (ACL), representing permissions for the POSIX 4793 permission scopes: Owner, group and others. By default, the owner and 4794 group will be taken from the current user, and the built-in group 4795 "Everyone" will be used to represent others. The user/group can be 4796 customized with FUSE options "UserName" and "GroupName", e.g. 4797 -o UserName=user123 -o GroupName="Authenticated Users". The permissions 4798 on each entry will be set according to options --dir-perms and 4799 --file-perms, which takes a value in traditional Unix numeric notation. 4800 4801 The default permissions corresponds to 4802 --file-perms 0666 --dir-perms 0777, i.e. read and write permissions to 4803 everyone. This means you will not be able to start any programs from the 4804 mount. To be able to do that you must add execute permissions, e.g. 4805 --file-perms 0777 --dir-perms 0777 to add it to everyone. If the program 4806 needs to write files, chances are you will have to enable VFS File 4807 Caching as well (see also limitations). Note that the default write 4808 permission have some restrictions for accounts other than the owner, 4809 specifically it lacks the "write extended attributes", as explained 4810 next. 4811 4812 The mapping of permissions is not always trivial, and the result you see 4813 in Windows Explorer may not be exactly like you expected. For example, 4814 when setting a value that includes write access for the group or others 4815 scope, this will be mapped to individual permissions "write attributes", 4816 "write data" and "append data", but not "write extended attributes". 4817 Windows will then show this as basic permission "Special" instead of 4818 "Write", because "Write" also covers the "write extended attributes" 4819 permission. When setting digit 0 for group or others, to indicate no 4820 permissions, they will still get individual permissions "read 4821 attributes", "read extended attributes" and "read permissions". This is 4822 done for compatibility reasons, e.g. to allow users without additional 4823 permissions to be able to read basic metadata about files like in Unix. 4824 4825 WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", 4826 that allows the complete specification of file security descriptors 4827 using SDDL. With this you get detailed control of the resulting 4828 permissions, compared to use of the POSIX permissions described above, 4829 and no additional permissions will be added automatically for 4830 compatibility with Unix. Some example use cases will following. 4831 4832 If you set POSIX permissions for only allowing access to the owner, 4833 using --file-perms 0600 --dir-perms 0700, the user group and the 4834 built-in "Everyone" group will still be given some special permissions, 4835 as described above. Some programs may then (incorrectly) interpret this 4836 as the file being accessible by everyone, for example an SSH client may 4837 warn about "unprotected private key file". You can work around this by 4838 specifying -o FileSecurity="D:P(A;;FA;;;OW)", which sets file all access 4839 (FA) to the owner (OW), and nothing else. 4840 4841 When setting write permissions then, except for the owner, this does not 4842 include the "write extended attributes" permission, as mentioned above. 4843 This may prevent applications from writing to files, giving permission 4844 denied error instead. To set working write permissions for the built-in 4845 "Everyone" group, similar to what it gets by default but with the 4846 addition of the "write extended attributes", you can specify 4847 -o FileSecurity="D:P(A;;FRFW;;;WD)", which sets file read (FR) and file 4848 write (FW) to everyone (WD). If file execute (FX) is also needed, then 4849 change to -o FileSecurity="D:P(A;;FRFWFX;;;WD)", or set file all access 4850 (FA) to get full access permissions, including delete, with 4851 -o FileSecurity="D:P(A;;FA;;;WD)". 4852 4853 Windows caveats 4854 4855 Drives created as Administrator are not visible to other accounts, not 4856 even an account that was elevated to Administrator with the User Account 4857 Control (UAC) feature. A result of this is that if you mount to a drive 4858 letter from a Command Prompt run as Administrator, and then try to 4859 access the same drive from Windows Explorer (which does not run as 4860 Administrator), you will not be able to see the mounted drive. 4861 4862 If you don't need to access the drive from applications running with 4863 administrative privileges, the easiest way around this is to always 4864 create the mount from a non-elevated command prompt. 4865 4866 To make mapped drives available to the user account that created them 4867 regardless if elevated or not, there is a special Windows setting called 4868 linked connections that can be enabled. 4869 4870 It is also possible to make a drive mount available to everyone on the 4871 system, by running the process creating it as the built-in SYSTEM 4872 account. There are several ways to do this: One is to use the 4873 command-line utility PsExec, from Microsoft's Sysinternals suite, which 4874 has option -s to start processes as the SYSTEM account. Another 4875 alternative is to run the mount command from a Windows Scheduled Task, 4876 or a Windows Service, configured to run as the SYSTEM account. A third 4877 alternative is to use the WinFsp.Launcher infrastructure). Read more in 4878 the install documentation. Note that when running rclone as another 4879 user, it will not use the configuration file from your profile unless 4880 you tell it to with the --config option. Note also that it is now the 4881 SYSTEM account that will have the owner permissions, and other accounts 4882 will have permissions according to the group or others scopes. As 4883 mentioned above, these will then not get the "write extended attributes" 4884 permission, and this may prevent writing to files. You can work around 4885 this with the FileSecurity option, see example above. 4886 4887 Note that mapping to a directory path, instead of a drive letter, does 4888 not suffer from the same limitations. 4889 4890 Mounting on macOS 4891 4892 Mounting on macOS can be done either via built-in NFS server, macFUSE 4893 (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver 4894 utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE 4895 system which "mounts" via an NFSv4 local server. 4896 4897 Unicode Normalization 4898 4899 It is highly recommended to keep the default of 4900 --no-unicode-normalization=false for all mount and serve commands on 4901 macOS. For details, see vfs-case-sensitivity. 4902 4903 NFS mount 4904 4905 This method spins up an NFS server using serve nfs command and mounts it 4906 to the specified mountpoint. If you run this in background mode using 4907 |--daemon|, you will need to send SIGTERM signal to the rclone process 4908 using |kill| command to stop the mount. 4909 4910 Note that --nfs-cache-handle-limit controls the maximum number of cached 4911 file handles stored by the nfsmount caching handler. This should not be 4912 set too low or you may experience errors when trying to access files. 4913 The default is 1000000, but consider lowering this limit if the server's 4914 system resource usage causes problems. 4915 4916 macFUSE Notes 4917 4918 If installing macFUSE using dmg packages from the website, rclone will 4919 locate the macFUSE libraries without any further intervention. If 4920 however, macFUSE is installed using the macports package manager, the 4921 following addition steps are required. 4922 4923 sudo mkdir /usr/local/lib 4924 cd /usr/local/lib 4925 sudo ln -s /opt/local/lib/libfuse.2.dylib 4926 4927 FUSE-T Limitations, Caveats, and Notes 4928 4929 There are some limitations, caveats, and notes about how it works. These 4930 are current as of FUSE-T version 1.0.14. 4931 4932 ModTime update on read 4933 4934 As per the FUSE-T wiki: 4935 4936 File access and modification times cannot be set separately as it 4937 seems to be an issue with the NFS client which always modifies both. 4938 Can be reproduced with 'touch -m' and 'touch -a' commands 4939 4940 This means that viewing files with various tools, notably macOS Finder, 4941 will cause rlcone to update the modification time of the file. This may 4942 make rclone upload a full new copy of the file. 4943 4944 Read Only mounts 4945 4946 When mounting with --read-only, attempts to write to files will fail 4947 silently as opposed to with a clear warning as in macFUSE. 4948 4949 Limitations 4950 4951 Without the use of --vfs-cache-mode this can only write files 4952 sequentially, it can only seek when reading. This means that many 4953 applications won't work with their files on an rclone mount without 4954 --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File 4955 Caching section for more info. When using NFS mount on macOS, if you 4956 don't specify |--vfs-cache-mode| the mount point will be read-only. 4957 4958 The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do 4959 not support the concept of empty directories, so empty directories will 4960 have a tendency to disappear once they fall out of the directory cache. 4961 4962 When rclone mount is invoked on Unix with --daemon flag, the main rclone 4963 program will wait for the background mount to become ready or until the 4964 timeout specified by the --daemon-wait flag. On Linux it can check mount 4965 status using ProcFS so the flag in fact sets maximum time to wait, while 4966 the real wait can be less. On macOS / BSD the time to wait is constant 4967 and the check is performed only at the end. We advise you to set wait 4968 time on macOS reasonably. 4969 4970 Only supported on Linux, FreeBSD, OS X and Windows at the moment. 4971 4972 rclone mount vs rclone sync/copy 4973 4974 File systems expect things to be 100% reliable, whereas cloud storage 4975 systems are a long way from 100% reliable. The rclone sync/copy commands 4976 cope with this with lots of retries. However rclone mount can't use 4977 retries in the same way without making local copies of the uploads. Look 4978 at the VFS File Caching for solutions to make mount more reliable. 4979 4980 Attribute caching 4981 4982 You can use the flag --attr-timeout to set the time the kernel caches 4983 the attributes (size, modification time, etc.) for directory entries. 4984 4985 The default is 1s which caches files just long enough to avoid too many 4986 callbacks to rclone from the kernel. 4987 4988 In theory 0s should be the correct value for filesystems which can 4989 change outside the control of the kernel. However this causes quite a 4990 few problems such as rclone using too much memory, rclone not serving 4991 files to samba and excessive time listing directories. 4992 4993 The kernel can cache the info about a file for the time given by 4994 --attr-timeout. You may see corruption if the remote file changes length 4995 during this window. It will show up as either a truncated file or a file 4996 with garbage on the end. With --attr-timeout 1s this is very unlikely 4997 but not impossible. The higher you set --attr-timeout the more likely it 4998 is. The default setting of "1s" is the lowest setting which mitigates 4999 the problems above. 5000 5001 If you set it higher (10s or 1m say) then the kernel will call back to 5002 rclone less often making it more efficient, however there is more chance 5003 of the corruption issue above. 5004 5005 If files don't change on the remote outside of the control of rclone 5006 then there is no chance of corruption. 5007 5008 This is the same as setting the attr_timeout option in mount.fuse. 5009 5010 Filters 5011 5012 Note that all the rclone filters can be used to select a subset of the 5013 files to be visible in the mount. 5014 5015 systemd 5016 5017 When running rclone mount as a systemd service, it is possible to use 5018 Type=notify. In this case the service will enter the started state after 5019 the mountpoint has been successfully set up. Units having the rclone 5020 mount service specified as a requirement will see all files and folders 5021 immediately in this mode. 5022 5023 Note that systemd runs mount units without any environment variables 5024 including PATH or HOME. This means that tilde (~) expansion will not 5025 work and you should provide --config and --cache-dir explicitly as 5026 absolute paths via rclone arguments. Since mounting requires the 5027 fusermount program, rclone will use the fallback PATH of /bin:/usr/bin 5028 in this scenario. Please ensure that fusermount is present on this PATH. 5029 5030 Rclone as Unix mount helper 5031 5032 The core Unix program /bin/mount normally takes the -t FSTYPE argument 5033 then runs the /sbin/mount.FSTYPE helper program passing it mount options 5034 as -o key=val,... or --opt=.... Automount (classic or systemd) behaves 5035 in a similar way. 5036 5037 rclone by default expects GNU-style flags --key val. To run it as a 5038 mount helper you should symlink rclone binary to /sbin/mount.rclone and 5039 optionally /usr/bin/rclonefs, e.g. 5040 ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and 5041 translate command-line arguments appropriately. 5042 5043 Now you can run classic mounts like this: 5044 5045 mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem 5046 5047 or create systemd mount units: 5048 5049 # /etc/systemd/system/mnt-data.mount 5050 [Unit] 5051 Description=Mount for /mnt/data 5052 [Mount] 5053 Type=rclone 5054 What=sftp1:subdir 5055 Where=/mnt/data 5056 Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone 5057 5058 optionally accompanied by systemd automount unit 5059 5060 # /etc/systemd/system/mnt-data.automount 5061 [Unit] 5062 Description=AutoMount for /mnt/data 5063 [Automount] 5064 Where=/mnt/data 5065 TimeoutIdleSec=600 5066 [Install] 5067 WantedBy=multi-user.target 5068 5069 or add in /etc/fstab a line like 5070 5071 sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 5072 5073 or use classic Automountd. Remember to provide explicit 5074 config=...,cache-dir=... as a workaround for mount units being run 5075 without HOME. 5076 5077 Rclone in the mount helper mode will split -o argument(s) by comma, 5078 replace _ by - and prepend -- to get the command-line flags. Options 5079 containing commas or spaces can be wrapped in single or double quotes. 5080 Any inner quotes inside outer quotes of the same type should be doubled. 5081 5082 Mount option syntax includes a few extra options treated specially: 5083 5084 - env.NAME=VALUE will set an environment variable for the mount 5085 process. This helps with Automountd and Systemd.mount which don't 5086 allow setting custom environment for mount helpers. Typically you 5087 will use env.HTTPS_PROXY=proxy.host:3128 or env.HOME=/root 5088 - command=cmount can be used to run cmount or any other rclone command 5089 rather than the default mount. 5090 - args2env will pass mount options to the mount helper running in 5091 background via environment variables instead of command line 5092 arguments. This allows to hide secrets from such commands as ps or 5093 pgrep. 5094 - vv... will be transformed into appropriate --verbose=N 5095 - standard mount options like x-systemd.automount, _netdev, nosuid and 5096 alike are intended only for Automountd and ignored by rclone. ## 5097 VFS - Virtual File System 5098 5099 This command uses the VFS layer. This adapts the cloud storage objects 5100 that rclone uses into something which looks much more like a disk filing 5101 system. 5102 5103 Cloud storage objects have lots of properties which aren't like disk 5104 files - you can't extend them or write to the middle of them, so the VFS 5105 layer has to deal with that. Because there is no one right way of doing 5106 this there are various options explained below. 5107 5108 The VFS layer also implements a directory cache - this caches info about 5109 files and directories (but not the data) in memory. 5110 5111 VFS Directory Cache 5112 5113 Using the --dir-cache-time flag, you can control how long a directory 5114 should be considered up to date and not refreshed from the backend. 5115 Changes made through the VFS will appear immediately or invalidate the 5116 cache. 5117 5118 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 5119 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 5120 5121 However, changes made directly on the cloud storage by the web interface 5122 or a different copy of rclone will only be picked up once the directory 5123 cache expires if the backend configured does not support polling for 5124 changes. If the backend supports polling, changes will be picked up 5125 within the polling interval. 5126 5127 You can send a SIGHUP signal to rclone for it to flush all directory 5128 caches, regardless of how old they are. Assuming only one rclone 5129 instance is running, you can reset the cache like this: 5130 5131 kill -SIGHUP $(pidof rclone) 5132 5133 If you configure rclone with a remote control then you can use rclone rc 5134 to flush the whole directory cache: 5135 5136 rclone rc vfs/forget 5137 5138 Or individual files or directories: 5139 5140 rclone rc vfs/forget file=path/to/file dir=path/to/dir 5141 5142 VFS File Buffering 5143 5144 The --buffer-size flag determines the amount of memory, that will be 5145 used to buffer data in advance. 5146 5147 Each open file will try to keep the specified amount of data in memory 5148 at all times. The buffered data is bound to one open file and won't be 5149 shared. 5150 5151 This flag is a upper limit for the used memory per open file. The buffer 5152 will only use memory for data that is downloaded but not not yet read. 5153 If the buffer is empty, only a small amount of memory will be used. 5154 5155 The maximum memory used by rclone for buffering can be up to 5156 --buffer-size * open files. 5157 5158 VFS File Caching 5159 5160 These flags control the VFS file caching options. File caching is 5161 necessary to make the VFS layer appear compatible with a normal file 5162 system. It can be disabled at the cost of some compatibility. 5163 5164 For example you'll need to enable VFS caching if you want to read and 5165 write simultaneously to a file. See below for more details. 5166 5167 Note that the VFS cache is separate from the cache backend and you may 5168 find that you need one or the other or both. 5169 5170 --cache-dir string Directory rclone will use for caching. 5171 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 5172 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 5173 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 5174 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 5175 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 5176 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 5177 5178 If run with -vv rclone will print the location of the file cache. The 5179 files are stored in the user cache file area which is OS dependent but 5180 can be controlled with --cache-dir or setting the appropriate 5181 environment variable. 5182 5183 The cache has 4 different modes selected by --vfs-cache-mode. The higher 5184 the cache mode the more compatible rclone becomes at the cost of using 5185 disk space. 5186 5187 Note that files are written back to the remote only when they are closed 5188 and if they haven't been accessed for --vfs-write-back seconds. If 5189 rclone is quit or dies with files that haven't been uploaded, these will 5190 be uploaded next time rclone is run with the same flags. 5191 5192 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 5193 cache may exceed these quotas for two reasons. Firstly because it is 5194 only checked every --vfs-cache-poll-interval. Secondly because open 5195 files cannot be evicted from the cache. When --vfs-cache-max-size or 5196 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 5197 least accessed files from the cache first. rclone will start with files 5198 that haven't been accessed for the longest. This cache flushing strategy 5199 is efficient and more relevant files are likely to remain cached. 5200 5201 The --vfs-cache-max-age will evict files from the cache after the set 5202 time since last access has passed. The default value of 1 hour will 5203 start evicting files from cache that haven't been accessed for 1 hour. 5204 When a cached file is accessed the 1 hour timer is reset to 0 and will 5205 wait for 1 more hour before evicting. Specify the time with standard 5206 notation, s, m, h, d, w . 5207 5208 You should not run two copies of rclone using the same VFS cache with 5209 the same or overlapping remotes if using --vfs-cache-mode > off. This 5210 can potentially cause data corruption if you do. You can work around 5211 this by giving each rclone its own cache hierarchy with --cache-dir. You 5212 don't need to worry about this if the remotes in use don't overlap. 5213 5214 --vfs-cache-mode off 5215 5216 In this mode (the default) the cache will read directly from the remote 5217 and write directly to the remote without caching anything on disk. 5218 5219 This will mean some operations are not possible 5220 5221 - Files can't be opened for both read AND write 5222 - Files opened for write can't be seeked 5223 - Existing files opened for write must have O_TRUNC set 5224 - Files open for read with O_TRUNC will be opened write only 5225 - Files open for write only will behave as if O_TRUNC was supplied 5226 - Open modes O_APPEND, O_TRUNC are ignored 5227 - If an upload fails it can't be retried 5228 5229 --vfs-cache-mode minimal 5230 5231 This is very similar to "off" except that files opened for read AND 5232 write will be buffered to disk. This means that files opened for write 5233 will be a lot more compatible, but uses the minimal disk space. 5234 5235 These operations are not possible 5236 5237 - Files opened for write only can't be seeked 5238 - Existing files opened for write must have O_TRUNC set 5239 - Files opened for write only will ignore O_APPEND, O_TRUNC 5240 - If an upload fails it can't be retried 5241 5242 --vfs-cache-mode writes 5243 5244 In this mode files opened for read only are still read directly from the 5245 remote, write only and read/write files are buffered to disk first. 5246 5247 This mode should support all normal file system operations. 5248 5249 If an upload fails it will be retried at exponentially increasing 5250 intervals up to 1 minute. 5251 5252 --vfs-cache-mode full 5253 5254 In this mode all reads and writes are buffered to and from disk. When 5255 data is read from the remote this is buffered to disk as well. 5256 5257 In this mode the files in the cache will be sparse files and rclone will 5258 keep track of which bits of the files it has downloaded. 5259 5260 So if an application only reads the starts of each file, then rclone 5261 will only buffer the start of the file. These files will appear to be 5262 their full size in the cache, but they will be sparse files with only 5263 the data that has been downloaded present in them. 5264 5265 This mode should support all normal file system operations and is 5266 otherwise identical to --vfs-cache-mode writes. 5267 5268 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 5269 bytes ahead. The --buffer-size is buffered in memory whereas the 5270 --vfs-read-ahead is buffered on disk. 5271 5272 When using this mode it is recommended that --buffer-size is not set too 5273 large and --vfs-read-ahead is set large if required. 5274 5275 IMPORTANT not all file systems support sparse files. In particular 5276 FAT/exFAT do not. Rclone will perform very badly if the cache directory 5277 is on a filesystem which doesn't support sparse files and it will log an 5278 ERROR message if one is detected. 5279 5280 Fingerprinting 5281 5282 Various parts of the VFS use fingerprinting to see if a local file copy 5283 has changed relative to a remote file. Fingerprints are made from: 5284 5285 - size 5286 - modification time 5287 - hash 5288 5289 where available on an object. 5290 5291 On some backends some of these attributes are slow to read (they take an 5292 extra API call per object, or extra work per object). 5293 5294 For example hash is slow with the local and sftp backends as they have 5295 to read the entire file and hash it, and modtime is slow with the s3, 5296 swift, ftp and qinqstor backends because they need to do an extra API 5297 call to fetch it. 5298 5299 If you use the --vfs-fast-fingerprint flag then rclone will not include 5300 the slow operations in the fingerprint. This makes the fingerprinting 5301 less accurate but much faster and will improve the opening time of 5302 cached files. 5303 5304 If you are running a vfs cache over local, s3 or swift backends then 5305 using this flag is recommended. 5306 5307 Note that if you change the value of this flag, the fingerprints of the 5308 files in the cache may be invalidated and the files will need to be 5309 downloaded again. 5310 5311 VFS Chunked Reading 5312 5313 When rclone reads files from a remote it reads them in chunks. This 5314 means that rather than requesting the whole file rclone reads the chunk 5315 specified. This can reduce the used download quota for some remotes by 5316 requesting only chunks from the remote that are actually read, at the 5317 cost of an increased number of requests. 5318 5319 These flags control the chunking: 5320 5321 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 5322 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 5323 5324 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 5325 then double the size for each read. When --vfs-read-chunk-size-limit is 5326 specified, and greater than --vfs-read-chunk-size, the chunk size for 5327 each open file will get doubled only until the specified value is 5328 reached. If the value is "off", which is the default, the limit is 5329 disabled and the chunk size will grow indefinitely. 5330 5331 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 5332 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 5333 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 5334 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 5335 1200M-1700M and so on. 5336 5337 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 5338 5339 VFS Performance 5340 5341 These flags may be used to enable/disable features of the VFS for 5342 performance or other reasons. See also the chunked reading feature. 5343 5344 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 5345 use --use-server-modtime for a slightly different effect) as each read 5346 of the modification time takes a transaction. 5347 5348 --no-checksum Don't compare checksums on up/download. 5349 --no-modtime Don't read/write the modification time (can speed things up). 5350 --no-seek Don't allow seeking in files. 5351 --read-only Only allow read-only access. 5352 5353 Sometimes rclone is delivered reads or writes out of order. Rather than 5354 seeking rclone will wait a short time for the in sequence read or write 5355 to come in. These flags only come into effect when not using an on disk 5356 cache file. 5357 5358 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 5359 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 5360 5361 When using VFS write caching (--vfs-cache-mode with value writes or 5362 full), the global flag --transfers can be set to adjust the number of 5363 parallel uploads of modified files from the cache (the related global 5364 flag --checkers has no effect on the VFS). 5365 5366 --transfers int Number of file transfers to run in parallel (default 4) 5367 5368 VFS Case Sensitivity 5369 5370 Linux file systems are case-sensitive: two files can differ only by 5371 case, and the exact case must be used when opening a file. 5372 5373 File systems in modern Windows are case-insensitive but case-preserving: 5374 although existing files can be opened using any case, the exact case 5375 used to create the file is preserved and available for programs to 5376 query. It is not allowed for two files in the same directory to differ 5377 only by case. 5378 5379 Usually file systems on macOS are case-insensitive. It is possible to 5380 make macOS file systems case-sensitive but that is not the default. 5381 5382 The --vfs-case-insensitive VFS flag controls how rclone handles these 5383 two cases. If its value is "false", rclone passes file names to the 5384 remote as-is. If the flag is "true" (or appears without a value on the 5385 command line), rclone may perform a "fixup" as explained below. 5386 5387 The user may specify a file name to open/delete/rename/etc with a case 5388 different than what is stored on the remote. If an argument refers to an 5389 existing file with exactly the same name, then the case of the existing 5390 file on the disk will be used. However, if a file name with exactly the 5391 same name is not found but a name differing only by case exists, rclone 5392 will transparently fixup the name. This fixup happens only when an 5393 existing file is requested. Case sensitivity of file names created anew 5394 by rclone is controlled by the underlying remote. 5395 5396 Note that case sensitivity of the operating system running rclone (the 5397 target) may differ from case sensitivity of a file system presented by 5398 rclone (the source). The flag controls whether "fixup" is performed to 5399 satisfy the target. 5400 5401 If the flag is not provided on the command line, then its default value 5402 depends on the operating system where rclone runs: "true" on Windows and 5403 macOS, "false" otherwise. If the flag is provided without a value, then 5404 it is "true". 5405 5406 The --no-unicode-normalization flag controls whether a similar "fixup" 5407 is performed for filenames that differ but are canonically equivalent 5408 with respect to unicode. Unicode normalization can be particularly 5409 helpful for users of macOS, which prefers form NFD instead of the NFC 5410 used by most other platforms. It is therefore highly recommended to keep 5411 the default of false on macOS, to avoid encoding compatibility issues. 5412 5413 In the (probably unlikely) event that a directory has multiple duplicate 5414 filenames after applying case and unicode normalization, the 5415 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 5416 with a performance tradeoff, as rclone will have to scan the entire 5417 directory for duplicates when listing a directory. For this reason, it 5418 is recommended to leave this disabled if not needed. However, macOS 5419 users may wish to consider using it, as otherwise, if a remote directory 5420 contains both NFC and NFD versions of the same filename, an odd 5421 situation will occur: both versions of the file will be visible in the 5422 mount, and both will appear to be editable, however, editing either 5423 version will actually result in only the NFD version getting edited 5424 under the hood. --vfs-block- norm-dupes prevents this confusion by 5425 detecting this scenario, hiding the duplicates, and logging an error, 5426 similar to how this is handled in rclone sync. 5427 5428 VFS Disk Options 5429 5430 This flag allows you to manually set the statistics about the filing 5431 system. It can be useful when those statistics cannot be read correctly 5432 automatically. 5433 5434 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 5435 5436 Alternate report of used bytes 5437 5438 Some backends, most notably S3, do not report the amount of bytes used. 5439 If you need this information to be available when running df on the 5440 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 5441 flag set, instead of relying on the backend to report this information, 5442 rclone will scan the whole remote similar to rclone size and compute the 5443 total used space itself. 5444 5445 WARNING. Contrary to rclone size, this flag ignores filters so that the 5446 result is accurate. However, this is very inefficient and may cost lots 5447 of API calls resulting in extra charges. Use it as a last resort and 5448 only with caching. 5449 5450 rclone mount remote:path /path/to/mountpoint [flags] 5451 5452 Options 5453 5454 --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) 5455 --allow-other Allow access to other users (not supported on Windows) 5456 --allow-root Allow access to root user (not supported on Windows) 5457 --async-read Use asynchronous reads (not supported on Windows) (default true) 5458 --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) 5459 --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) 5460 --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) 5461 --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) 5462 --debug-fuse Debug the FUSE internals - needs -v 5463 --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) 5464 --devname string Set the device name - default is remote:path 5465 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 5466 --dir-perms FileMode Directory permissions (default 0777) 5467 --file-perms FileMode File permissions (default 0666) 5468 --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) 5469 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 5470 -h, --help help for mount 5471 --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) 5472 --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) 5473 --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) 5474 --no-checksum Don't compare checksums on up/download 5475 --no-modtime Don't read/write the modification time (can speed things up) 5476 --no-seek Don't allow seeking in files 5477 --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) 5478 --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) 5479 -o, --option stringArray Option for libfuse/WinFsp (repeat if required) 5480 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 5481 --read-only Only allow read-only access 5482 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 5483 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 5484 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 5485 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 5486 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 5487 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 5488 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 5489 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 5490 --vfs-case-insensitive If a file name not found, find a case insensitive match 5491 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 5492 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 5493 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 5494 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 5495 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 5496 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 5497 --vfs-refresh Refreshes the directory cache recursively in the background on start 5498 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 5499 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 5500 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 5501 --volname string Set the volume name (supported on Windows and OSX only) 5502 --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) 5503 5504 Filter Options 5505 5506 Flags for filtering directory listings. 5507 5508 --delete-excluded Delete files on dest excluded from sync 5509 --exclude stringArray Exclude files matching pattern 5510 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 5511 --exclude-if-present stringArray Exclude directories if filename is present 5512 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 5513 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 5514 -f, --filter stringArray Add a file filtering rule 5515 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 5516 --ignore-case Ignore case in filters (case insensitive) 5517 --include stringArray Include files matching pattern 5518 --include-from stringArray Read file include patterns from file (use - to read from stdin) 5519 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5520 --max-depth int If set limits the recursion depth to this (default -1) 5521 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 5522 --metadata-exclude stringArray Exclude metadatas matching pattern 5523 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 5524 --metadata-filter stringArray Add a metadata filtering rule 5525 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 5526 --metadata-include stringArray Include metadatas matching pattern 5527 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 5528 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5529 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 5530 5531 See the global flags page for global options not listed here. 5532 5533 SEE ALSO 5534 5535 - rclone - Show help for rclone commands, flags and backends. 5536 5537 rclone moveto 5538 5539 Move file or directory from source to dest. 5540 5541 Synopsis 5542 5543 If source:path is a file or directory then it moves it to a file or 5544 directory named dest:path. 5545 5546 This can be used to rename files or upload single files to other than 5547 their existing name. If the source is a directory then it acts exactly 5548 like the move command. 5549 5550 So 5551 5552 rclone moveto src dst 5553 5554 where src and dst are rclone paths, either remote:path or /path/to/local 5555 or C:. 5556 5557 This will: 5558 5559 if src is file 5560 move it to dst, overwriting an existing file if it exists 5561 if src is directory 5562 move it to dst, overwriting existing files if they exist 5563 see move command for full details 5564 5565 This doesn't transfer files that are identical on src and dst, testing 5566 by size and modification time or MD5SUM. src will be deleted on 5567 successful transfer. 5568 5569 Important: Since this can cause data loss, test first with the --dry-run 5570 or the --interactive/-i flag. 5571 5572 Note: Use the -P/--progress flag to view real-time transfer statistics. 5573 5574 rclone moveto source:path dest:path [flags] 5575 5576 Options 5577 5578 -h, --help help for moveto 5579 5580 Copy Options 5581 5582 Flags for anything which can Copy a file. 5583 5584 --check-first Do all the checks before starting transfers 5585 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 5586 --compare-dest stringArray Include additional comma separated server-side paths during comparison 5587 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 5588 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 5589 --ignore-case-sync Ignore case when synchronizing 5590 --ignore-checksum Skip post copy check of checksums 5591 --ignore-existing Skip all files that exist on destination 5592 --ignore-size Ignore size when skipping use modtime or checksum 5593 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 5594 --immutable Do not modify files, fail if existing files have been modified 5595 --inplace Download directly to destination file instead of atomic download to temp/rename 5596 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 5597 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 5598 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 5599 -M, --metadata If set, preserve metadata when copying objects 5600 --modify-window Duration Max time diff to be considered the same (default 1ns) 5601 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 5602 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 5603 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 5604 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 5605 --no-check-dest Don't check the destination, copy regardless 5606 --no-traverse Don't traverse destination file system on copy 5607 --no-update-dir-modtime Don't update directory modification times 5608 --no-update-modtime Don't update destination modtime if files identical 5609 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 5610 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 5611 --refresh-times Refresh the modtime of remote files 5612 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 5613 --size-only Skip based on size only, not modtime or checksum 5614 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 5615 -u, --update Skip files that are newer on the destination 5616 5617 Important Options 5618 5619 Important flags useful for most commands. 5620 5621 -n, --dry-run Do a trial run with no permanent changes 5622 -i, --interactive Enable interactive mode 5623 -v, --verbose count Print lots more stuff (repeat for more) 5624 5625 Filter Options 5626 5627 Flags for filtering directory listings. 5628 5629 --delete-excluded Delete files on dest excluded from sync 5630 --exclude stringArray Exclude files matching pattern 5631 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 5632 --exclude-if-present stringArray Exclude directories if filename is present 5633 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 5634 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 5635 -f, --filter stringArray Add a file filtering rule 5636 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 5637 --ignore-case Ignore case in filters (case insensitive) 5638 --include stringArray Include files matching pattern 5639 --include-from stringArray Read file include patterns from file (use - to read from stdin) 5640 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5641 --max-depth int If set limits the recursion depth to this (default -1) 5642 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 5643 --metadata-exclude stringArray Exclude metadatas matching pattern 5644 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 5645 --metadata-filter stringArray Add a metadata filtering rule 5646 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 5647 --metadata-include stringArray Include metadatas matching pattern 5648 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 5649 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5650 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 5651 5652 Listing Options 5653 5654 Flags for listing directories. 5655 5656 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 5657 --fast-list Use recursive list if available; uses more memory but fewer transactions 5658 5659 See the global flags page for global options not listed here. 5660 5661 SEE ALSO 5662 5663 - rclone - Show help for rclone commands, flags and backends. 5664 5665 rclone ncdu 5666 5667 Explore a remote with a text based user interface. 5668 5669 Synopsis 5670 5671 This displays a text based user interface allowing the navigation of a 5672 remote. It is most useful for answering the question - "What is using 5673 all my disk space?". 5674 5675 To make the user interface it first scans the entire remote given and 5676 builds an in memory representation. rclone ncdu can be used during this 5677 scanning phase and you will see it building up the directory structure 5678 as it goes along. 5679 5680 You can interact with the user interface using key presses, press '?' to 5681 toggle the help on and off. The supported keys are: 5682 5683 ↑,↓ or k,j to Move 5684 →,l to enter 5685 ←,h to return 5686 g toggle graph 5687 c toggle counts 5688 a toggle average size in directory 5689 m toggle modified time 5690 u toggle human-readable format 5691 n,s,C,A,M sort by name,size,count,asize,mtime 5692 d delete file/directory 5693 v select file/directory 5694 V enter visual select mode 5695 D delete selected files/directories 5696 y copy current path to clipboard 5697 Y display current path 5698 ^L refresh screen (fix screen corruption) 5699 r recalculate file sizes 5700 ? to toggle help on and off 5701 q/ESC/^c to quit 5702 5703 Listed files/directories may be prefixed by a one-character flag, some 5704 of them combined with a description in brackets at end of line. These 5705 flags have the following meaning: 5706 5707 e means this is an empty directory, i.e. contains no files (but 5708 may contain empty subdirectories) 5709 ~ means this is a directory where some of the files (possibly in 5710 subdirectories) have unknown size, and therefore the directory 5711 size may be underestimated (and average size inaccurate, as it 5712 is average of the files with known sizes). 5713 . means an error occurred while reading a subdirectory, and 5714 therefore the directory size may be underestimated (and average 5715 size inaccurate) 5716 ! means an error occurred while reading this directory 5717 5718 This an homage to the ncdu tool but for rclone remotes. It is missing 5719 lots of features at the moment but is useful as it stands. 5720 5721 Note that it might take some time to delete big files/directories. The 5722 UI won't respond in the meantime since the deletion is done 5723 synchronously. 5724 5725 For a non-interactive listing of the remote, see the tree command. To 5726 just get the total size of the remote you can also use the size command. 5727 5728 rclone ncdu remote:path [flags] 5729 5730 Options 5731 5732 -h, --help help for ncdu 5733 5734 Filter Options 5735 5736 Flags for filtering directory listings. 5737 5738 --delete-excluded Delete files on dest excluded from sync 5739 --exclude stringArray Exclude files matching pattern 5740 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 5741 --exclude-if-present stringArray Exclude directories if filename is present 5742 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 5743 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 5744 -f, --filter stringArray Add a file filtering rule 5745 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 5746 --ignore-case Ignore case in filters (case insensitive) 5747 --include stringArray Include files matching pattern 5748 --include-from stringArray Read file include patterns from file (use - to read from stdin) 5749 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5750 --max-depth int If set limits the recursion depth to this (default -1) 5751 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 5752 --metadata-exclude stringArray Exclude metadatas matching pattern 5753 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 5754 --metadata-filter stringArray Add a metadata filtering rule 5755 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 5756 --metadata-include stringArray Include metadatas matching pattern 5757 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 5758 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 5759 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 5760 5761 Listing Options 5762 5763 Flags for listing directories. 5764 5765 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 5766 --fast-list Use recursive list if available; uses more memory but fewer transactions 5767 5768 See the global flags page for global options not listed here. 5769 5770 SEE ALSO 5771 5772 - rclone - Show help for rclone commands, flags and backends. 5773 5774 rclone nfsmount 5775 5776 Mount the remote as file system on a mountpoint. 5777 5778 Synopsis 5779 5780 rclone nfsmount allows Linux, FreeBSD, macOS and Windows to mount any of 5781 Rclone's cloud storage systems as a file system with FUSE. 5782 5783 First set up your remote using rclone config. Check it works with 5784 rclone ls etc. 5785 5786 On Linux and macOS, you can run mount in either foreground or background 5787 (aka daemon) mode. Mount runs in foreground mode by default. Use the 5788 --daemon flag to force background mode. On Windows you can run mount in 5789 foreground only, the flag is ignored. 5790 5791 In background mode rclone acts as a generic Unix mount program: the main 5792 program starts, spawns background rclone process to setup and maintain 5793 the mount, waits until success or timeout and exits with appropriate 5794 code (killing the child process if it fails). 5795 5796 On Linux/macOS/FreeBSD start the mount like this, where 5797 /path/to/local/mount is an empty existing directory: 5798 5799 rclone nfsmount remote:path/to/files /path/to/local/mount 5800 5801 On Windows you can start a mount in different ways. See below for 5802 details. If foreground mount is used interactively from a console 5803 window, rclone will serve the mount and occupy the console so another 5804 window should be used to work with the mount until rclone is interrupted 5805 e.g. by pressing Ctrl-C. 5806 5807 The following examples will mount to an automatically assigned drive, to 5808 specific drive letter X:, to path C:\path\parent\mount (where parent 5809 directory or drive must exist, and mount must not exist, and is not 5810 supported when mounting as a network drive), and the last example will 5811 mount as network share \\cloud\remote and map it to an automatically 5812 assigned drive: 5813 5814 rclone nfsmount remote:path/to/files * 5815 rclone nfsmount remote:path/to/files X: 5816 rclone nfsmount remote:path/to/files C:\path\parent\mount 5817 rclone nfsmount remote:path/to/files \\cloud\remote 5818 5819 When the program ends while in foreground mode, either via Ctrl+C or 5820 receiving a SIGINT or SIGTERM signal, the mount should be automatically 5821 stopped. 5822 5823 When running in background mode the user will have to stop the mount 5824 manually: 5825 5826 # Linux 5827 fusermount -u /path/to/local/mount 5828 # OS X 5829 umount /path/to/local/mount 5830 5831 The umount operation can fail, for example when the mountpoint is busy. 5832 When that happens, it is the user's responsibility to stop the mount 5833 manually. 5834 5835 The size of the mounted file system will be set according to information 5836 retrieved from the remote, the same as returned by the rclone about 5837 command. Remotes with unlimited storage may report the used size only, 5838 then an additional 1 PiB of free space is assumed. If the remote does 5839 not support the about feature at all, then 1 PiB is set as both the 5840 total and the free size. 5841 5842 Installing on Windows 5843 5844 To run rclone nfsmount on Windows, you will need to download and install 5845 WinFsp. 5846 5847 WinFsp is an open-source Windows File System Proxy which makes it easy 5848 to write user space file systems for Windows. It provides a FUSE 5849 emulation layer which rclone uses combination with cgofuse. Both of 5850 these packages are by Bill Zissimopoulos who was very helpful during the 5851 implementation of rclone nfsmount for Windows. 5852 5853 Mounting modes on windows 5854 5855 Unlike other operating systems, Microsoft Windows provides a different 5856 filesystem type for network and fixed drives. It optimises access on the 5857 assumption fixed disk drives are fast and reliable, while network drives 5858 have relatively high latency and less reliability. Some settings can 5859 also be differentiated between the two types, for example that Windows 5860 Explorer should just display icons and not create preview thumbnails for 5861 image and video files on network drives. 5862 5863 In most cases, rclone will mount the remote as a normal, fixed disk 5864 drive by default. However, you can also choose to mount it as a remote 5865 network drive, often described as a network share. If you mount an 5866 rclone remote using the default, fixed drive mode and experience 5867 unexpected program errors, freezes or other issues, consider mounting as 5868 a network drive instead. 5869 5870 When mounting as a fixed disk drive you can either mount to an unused 5871 drive letter, or to a path representing a nonexistent subdirectory of an 5872 existing parent directory or drive. Using the special value * will tell 5873 rclone to automatically assign the next available drive letter, starting 5874 with Z: and moving backward. Examples: 5875 5876 rclone nfsmount remote:path/to/files * 5877 rclone nfsmount remote:path/to/files X: 5878 rclone nfsmount remote:path/to/files C:\path\parent\mount 5879 rclone nfsmount remote:path/to/files X: 5880 5881 Option --volname can be used to set a custom volume name for the mounted 5882 file system. The default is to use the remote name and path. 5883 5884 To mount as network drive, you can add option --network-mode to your 5885 nfsmount command. Mounting to a directory path is not supported in this 5886 mode, it is a limitation Windows imposes on junctions, so the remote 5887 must always be mounted to a drive letter. 5888 5889 rclone nfsmount remote:path/to/files X: --network-mode 5890 5891 A volume name specified with --volname will be used to create the 5892 network share path. A complete UNC path, such as \\cloud\remote, 5893 optionally with path \\cloud\remote\madeup\path, will be used as is. Any 5894 other string will be used as the share part, after a default prefix 5895 \\server\. If no volume name is specified then \\server\share will be 5896 used. You must make sure the volume name is unique when you are mounting 5897 more than one drive, or else the mount command will fail. The share name 5898 will treated as the volume label for the mapped drive, shown in Windows 5899 Explorer etc, while the complete \\server\share will be reported as the 5900 remote UNC path by net use etc, just like a normal network drive 5901 mapping. 5902 5903 If you specify a full network share UNC path with --volname, this will 5904 implicitly set the --network-mode option, so the following two examples 5905 have same result: 5906 5907 rclone nfsmount remote:path/to/files X: --network-mode 5908 rclone nfsmount remote:path/to/files X: --volname \\server\share 5909 5910 You may also specify the network share UNC path as the mountpoint 5911 itself. Then rclone will automatically assign a drive letter, same as 5912 with * and use that as mountpoint, and instead use the UNC path 5913 specified as the volume name, as if it were specified with the --volname 5914 option. This will also implicitly set the --network-mode option. This 5915 means the following two examples have same result: 5916 5917 rclone nfsmount remote:path/to/files \\cloud\remote 5918 rclone nfsmount remote:path/to/files * --volname \\cloud\remote 5919 5920 There is yet another way to enable network mode, and to set the share 5921 path, and that is to pass the "native" libfuse/WinFsp option directly: 5922 --fuse-flag --VolumePrefix=\server\share. Note that the path must be 5923 with just a single backslash prefix in this case. 5924 5925 Note: In previous versions of rclone this was the only supported method. 5926 5927 Read more about drive mapping 5928 5929 See also Limitations section below. 5930 5931 Windows filesystem permissions 5932 5933 The FUSE emulation layer on Windows must convert between the POSIX-based 5934 permission model used in FUSE, and the permission model used in Windows, 5935 based on access-control lists (ACL). 5936 5937 The mounted filesystem will normally get three entries in its 5938 access-control list (ACL), representing permissions for the POSIX 5939 permission scopes: Owner, group and others. By default, the owner and 5940 group will be taken from the current user, and the built-in group 5941 "Everyone" will be used to represent others. The user/group can be 5942 customized with FUSE options "UserName" and "GroupName", e.g. 5943 -o UserName=user123 -o GroupName="Authenticated Users". The permissions 5944 on each entry will be set according to options --dir-perms and 5945 --file-perms, which takes a value in traditional Unix numeric notation. 5946 5947 The default permissions corresponds to 5948 --file-perms 0666 --dir-perms 0777, i.e. read and write permissions to 5949 everyone. This means you will not be able to start any programs from the 5950 mount. To be able to do that you must add execute permissions, e.g. 5951 --file-perms 0777 --dir-perms 0777 to add it to everyone. If the program 5952 needs to write files, chances are you will have to enable VFS File 5953 Caching as well (see also limitations). Note that the default write 5954 permission have some restrictions for accounts other than the owner, 5955 specifically it lacks the "write extended attributes", as explained 5956 next. 5957 5958 The mapping of permissions is not always trivial, and the result you see 5959 in Windows Explorer may not be exactly like you expected. For example, 5960 when setting a value that includes write access for the group or others 5961 scope, this will be mapped to individual permissions "write attributes", 5962 "write data" and "append data", but not "write extended attributes". 5963 Windows will then show this as basic permission "Special" instead of 5964 "Write", because "Write" also covers the "write extended attributes" 5965 permission. When setting digit 0 for group or others, to indicate no 5966 permissions, they will still get individual permissions "read 5967 attributes", "read extended attributes" and "read permissions". This is 5968 done for compatibility reasons, e.g. to allow users without additional 5969 permissions to be able to read basic metadata about files like in Unix. 5970 5971 WinFsp 2021 (version 1.9) introduced a new FUSE option "FileSecurity", 5972 that allows the complete specification of file security descriptors 5973 using SDDL. With this you get detailed control of the resulting 5974 permissions, compared to use of the POSIX permissions described above, 5975 and no additional permissions will be added automatically for 5976 compatibility with Unix. Some example use cases will following. 5977 5978 If you set POSIX permissions for only allowing access to the owner, 5979 using --file-perms 0600 --dir-perms 0700, the user group and the 5980 built-in "Everyone" group will still be given some special permissions, 5981 as described above. Some programs may then (incorrectly) interpret this 5982 as the file being accessible by everyone, for example an SSH client may 5983 warn about "unprotected private key file". You can work around this by 5984 specifying -o FileSecurity="D:P(A;;FA;;;OW)", which sets file all access 5985 (FA) to the owner (OW), and nothing else. 5986 5987 When setting write permissions then, except for the owner, this does not 5988 include the "write extended attributes" permission, as mentioned above. 5989 This may prevent applications from writing to files, giving permission 5990 denied error instead. To set working write permissions for the built-in 5991 "Everyone" group, similar to what it gets by default but with the 5992 addition of the "write extended attributes", you can specify 5993 -o FileSecurity="D:P(A;;FRFW;;;WD)", which sets file read (FR) and file 5994 write (FW) to everyone (WD). If file execute (FX) is also needed, then 5995 change to -o FileSecurity="D:P(A;;FRFWFX;;;WD)", or set file all access 5996 (FA) to get full access permissions, including delete, with 5997 -o FileSecurity="D:P(A;;FA;;;WD)". 5998 5999 Windows caveats 6000 6001 Drives created as Administrator are not visible to other accounts, not 6002 even an account that was elevated to Administrator with the User Account 6003 Control (UAC) feature. A result of this is that if you mount to a drive 6004 letter from a Command Prompt run as Administrator, and then try to 6005 access the same drive from Windows Explorer (which does not run as 6006 Administrator), you will not be able to see the mounted drive. 6007 6008 If you don't need to access the drive from applications running with 6009 administrative privileges, the easiest way around this is to always 6010 create the mount from a non-elevated command prompt. 6011 6012 To make mapped drives available to the user account that created them 6013 regardless if elevated or not, there is a special Windows setting called 6014 linked connections that can be enabled. 6015 6016 It is also possible to make a drive mount available to everyone on the 6017 system, by running the process creating it as the built-in SYSTEM 6018 account. There are several ways to do this: One is to use the 6019 command-line utility PsExec, from Microsoft's Sysinternals suite, which 6020 has option -s to start processes as the SYSTEM account. Another 6021 alternative is to run the mount command from a Windows Scheduled Task, 6022 or a Windows Service, configured to run as the SYSTEM account. A third 6023 alternative is to use the WinFsp.Launcher infrastructure). Read more in 6024 the install documentation. Note that when running rclone as another 6025 user, it will not use the configuration file from your profile unless 6026 you tell it to with the --config option. Note also that it is now the 6027 SYSTEM account that will have the owner permissions, and other accounts 6028 will have permissions according to the group or others scopes. As 6029 mentioned above, these will then not get the "write extended attributes" 6030 permission, and this may prevent writing to files. You can work around 6031 this with the FileSecurity option, see example above. 6032 6033 Note that mapping to a directory path, instead of a drive letter, does 6034 not suffer from the same limitations. 6035 6036 Mounting on macOS 6037 6038 Mounting on macOS can be done either via built-in NFS server, macFUSE 6039 (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver 6040 utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE 6041 system which "mounts" via an NFSv4 local server. 6042 6043 Unicode Normalization 6044 6045 It is highly recommended to keep the default of 6046 --no-unicode-normalization=false for all mount and serve commands on 6047 macOS. For details, see vfs-case-sensitivity. 6048 6049 NFS mount 6050 6051 This method spins up an NFS server using serve nfs command and mounts it 6052 to the specified mountpoint. If you run this in background mode using 6053 |--daemon|, you will need to send SIGTERM signal to the rclone process 6054 using |kill| command to stop the mount. 6055 6056 Note that --nfs-cache-handle-limit controls the maximum number of cached 6057 file handles stored by the nfsmount caching handler. This should not be 6058 set too low or you may experience errors when trying to access files. 6059 The default is 1000000, but consider lowering this limit if the server's 6060 system resource usage causes problems. 6061 6062 macFUSE Notes 6063 6064 If installing macFUSE using dmg packages from the website, rclone will 6065 locate the macFUSE libraries without any further intervention. If 6066 however, macFUSE is installed using the macports package manager, the 6067 following addition steps are required. 6068 6069 sudo mkdir /usr/local/lib 6070 cd /usr/local/lib 6071 sudo ln -s /opt/local/lib/libfuse.2.dylib 6072 6073 FUSE-T Limitations, Caveats, and Notes 6074 6075 There are some limitations, caveats, and notes about how it works. These 6076 are current as of FUSE-T version 1.0.14. 6077 6078 ModTime update on read 6079 6080 As per the FUSE-T wiki: 6081 6082 File access and modification times cannot be set separately as it 6083 seems to be an issue with the NFS client which always modifies both. 6084 Can be reproduced with 'touch -m' and 'touch -a' commands 6085 6086 This means that viewing files with various tools, notably macOS Finder, 6087 will cause rlcone to update the modification time of the file. This may 6088 make rclone upload a full new copy of the file. 6089 6090 Read Only mounts 6091 6092 When mounting with --read-only, attempts to write to files will fail 6093 silently as opposed to with a clear warning as in macFUSE. 6094 6095 Limitations 6096 6097 Without the use of --vfs-cache-mode this can only write files 6098 sequentially, it can only seek when reading. This means that many 6099 applications won't work with their files on an rclone mount without 6100 --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File 6101 Caching section for more info. When using NFS mount on macOS, if you 6102 don't specify |--vfs-cache-mode| the mount point will be read-only. 6103 6104 The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do 6105 not support the concept of empty directories, so empty directories will 6106 have a tendency to disappear once they fall out of the directory cache. 6107 6108 When rclone mount is invoked on Unix with --daemon flag, the main rclone 6109 program will wait for the background mount to become ready or until the 6110 timeout specified by the --daemon-wait flag. On Linux it can check mount 6111 status using ProcFS so the flag in fact sets maximum time to wait, while 6112 the real wait can be less. On macOS / BSD the time to wait is constant 6113 and the check is performed only at the end. We advise you to set wait 6114 time on macOS reasonably. 6115 6116 Only supported on Linux, FreeBSD, OS X and Windows at the moment. 6117 6118 rclone nfsmount vs rclone sync/copy 6119 6120 File systems expect things to be 100% reliable, whereas cloud storage 6121 systems are a long way from 100% reliable. The rclone sync/copy commands 6122 cope with this with lots of retries. However rclone nfsmount can't use 6123 retries in the same way without making local copies of the uploads. Look 6124 at the VFS File Caching for solutions to make nfsmount more reliable. 6125 6126 Attribute caching 6127 6128 You can use the flag --attr-timeout to set the time the kernel caches 6129 the attributes (size, modification time, etc.) for directory entries. 6130 6131 The default is 1s which caches files just long enough to avoid too many 6132 callbacks to rclone from the kernel. 6133 6134 In theory 0s should be the correct value for filesystems which can 6135 change outside the control of the kernel. However this causes quite a 6136 few problems such as rclone using too much memory, rclone not serving 6137 files to samba and excessive time listing directories. 6138 6139 The kernel can cache the info about a file for the time given by 6140 --attr-timeout. You may see corruption if the remote file changes length 6141 during this window. It will show up as either a truncated file or a file 6142 with garbage on the end. With --attr-timeout 1s this is very unlikely 6143 but not impossible. The higher you set --attr-timeout the more likely it 6144 is. The default setting of "1s" is the lowest setting which mitigates 6145 the problems above. 6146 6147 If you set it higher (10s or 1m say) then the kernel will call back to 6148 rclone less often making it more efficient, however there is more chance 6149 of the corruption issue above. 6150 6151 If files don't change on the remote outside of the control of rclone 6152 then there is no chance of corruption. 6153 6154 This is the same as setting the attr_timeout option in mount.fuse. 6155 6156 Filters 6157 6158 Note that all the rclone filters can be used to select a subset of the 6159 files to be visible in the mount. 6160 6161 systemd 6162 6163 When running rclone nfsmount as a systemd service, it is possible to use 6164 Type=notify. In this case the service will enter the started state after 6165 the mountpoint has been successfully set up. Units having the rclone 6166 nfsmount service specified as a requirement will see all files and 6167 folders immediately in this mode. 6168 6169 Note that systemd runs mount units without any environment variables 6170 including PATH or HOME. This means that tilde (~) expansion will not 6171 work and you should provide --config and --cache-dir explicitly as 6172 absolute paths via rclone arguments. Since mounting requires the 6173 fusermount program, rclone will use the fallback PATH of /bin:/usr/bin 6174 in this scenario. Please ensure that fusermount is present on this PATH. 6175 6176 Rclone as Unix mount helper 6177 6178 The core Unix program /bin/mount normally takes the -t FSTYPE argument 6179 then runs the /sbin/mount.FSTYPE helper program passing it mount options 6180 as -o key=val,... or --opt=.... Automount (classic or systemd) behaves 6181 in a similar way. 6182 6183 rclone by default expects GNU-style flags --key val. To run it as a 6184 mount helper you should symlink rclone binary to /sbin/mount.rclone and 6185 optionally /usr/bin/rclonefs, e.g. 6186 ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and 6187 translate command-line arguments appropriately. 6188 6189 Now you can run classic mounts like this: 6190 6191 mount sftp1:subdir /mnt/data -t rclone -o vfs_cache_mode=writes,sftp_key_file=/path/to/pem 6192 6193 or create systemd mount units: 6194 6195 # /etc/systemd/system/mnt-data.mount 6196 [Unit] 6197 Description=Mount for /mnt/data 6198 [Mount] 6199 Type=rclone 6200 What=sftp1:subdir 6201 Where=/mnt/data 6202 Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/etc/rclone.conf,cache-dir=/var/rclone 6203 6204 optionally accompanied by systemd automount unit 6205 6206 # /etc/systemd/system/mnt-data.automount 6207 [Unit] 6208 Description=AutoMount for /mnt/data 6209 [Automount] 6210 Where=/mnt/data 6211 TimeoutIdleSec=600 6212 [Install] 6213 WantedBy=multi-user.target 6214 6215 or add in /etc/fstab a line like 6216 6217 sftp1:subdir /mnt/data rclone rw,noauto,nofail,_netdev,x-systemd.automount,args2env,vfs_cache_mode=writes,config=/etc/rclone.conf,cache_dir=/var/cache/rclone 0 0 6218 6219 or use classic Automountd. Remember to provide explicit 6220 config=...,cache-dir=... as a workaround for mount units being run 6221 without HOME. 6222 6223 Rclone in the mount helper mode will split -o argument(s) by comma, 6224 replace _ by - and prepend -- to get the command-line flags. Options 6225 containing commas or spaces can be wrapped in single or double quotes. 6226 Any inner quotes inside outer quotes of the same type should be doubled. 6227 6228 Mount option syntax includes a few extra options treated specially: 6229 6230 - env.NAME=VALUE will set an environment variable for the mount 6231 process. This helps with Automountd and Systemd.mount which don't 6232 allow setting custom environment for mount helpers. Typically you 6233 will use env.HTTPS_PROXY=proxy.host:3128 or env.HOME=/root 6234 - command=cmount can be used to run cmount or any other rclone command 6235 rather than the default mount. 6236 - args2env will pass mount options to the mount helper running in 6237 background via environment variables instead of command line 6238 arguments. This allows to hide secrets from such commands as ps or 6239 pgrep. 6240 - vv... will be transformed into appropriate --verbose=N 6241 - standard mount options like x-systemd.automount, _netdev, nosuid and 6242 alike are intended only for Automountd and ignored by rclone. ## 6243 VFS - Virtual File System 6244 6245 This command uses the VFS layer. This adapts the cloud storage objects 6246 that rclone uses into something which looks much more like a disk filing 6247 system. 6248 6249 Cloud storage objects have lots of properties which aren't like disk 6250 files - you can't extend them or write to the middle of them, so the VFS 6251 layer has to deal with that. Because there is no one right way of doing 6252 this there are various options explained below. 6253 6254 The VFS layer also implements a directory cache - this caches info about 6255 files and directories (but not the data) in memory. 6256 6257 VFS Directory Cache 6258 6259 Using the --dir-cache-time flag, you can control how long a directory 6260 should be considered up to date and not refreshed from the backend. 6261 Changes made through the VFS will appear immediately or invalidate the 6262 cache. 6263 6264 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 6265 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 6266 6267 However, changes made directly on the cloud storage by the web interface 6268 or a different copy of rclone will only be picked up once the directory 6269 cache expires if the backend configured does not support polling for 6270 changes. If the backend supports polling, changes will be picked up 6271 within the polling interval. 6272 6273 You can send a SIGHUP signal to rclone for it to flush all directory 6274 caches, regardless of how old they are. Assuming only one rclone 6275 instance is running, you can reset the cache like this: 6276 6277 kill -SIGHUP $(pidof rclone) 6278 6279 If you configure rclone with a remote control then you can use rclone rc 6280 to flush the whole directory cache: 6281 6282 rclone rc vfs/forget 6283 6284 Or individual files or directories: 6285 6286 rclone rc vfs/forget file=path/to/file dir=path/to/dir 6287 6288 VFS File Buffering 6289 6290 The --buffer-size flag determines the amount of memory, that will be 6291 used to buffer data in advance. 6292 6293 Each open file will try to keep the specified amount of data in memory 6294 at all times. The buffered data is bound to one open file and won't be 6295 shared. 6296 6297 This flag is a upper limit for the used memory per open file. The buffer 6298 will only use memory for data that is downloaded but not not yet read. 6299 If the buffer is empty, only a small amount of memory will be used. 6300 6301 The maximum memory used by rclone for buffering can be up to 6302 --buffer-size * open files. 6303 6304 VFS File Caching 6305 6306 These flags control the VFS file caching options. File caching is 6307 necessary to make the VFS layer appear compatible with a normal file 6308 system. It can be disabled at the cost of some compatibility. 6309 6310 For example you'll need to enable VFS caching if you want to read and 6311 write simultaneously to a file. See below for more details. 6312 6313 Note that the VFS cache is separate from the cache backend and you may 6314 find that you need one or the other or both. 6315 6316 --cache-dir string Directory rclone will use for caching. 6317 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 6318 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 6319 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 6320 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 6321 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 6322 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 6323 6324 If run with -vv rclone will print the location of the file cache. The 6325 files are stored in the user cache file area which is OS dependent but 6326 can be controlled with --cache-dir or setting the appropriate 6327 environment variable. 6328 6329 The cache has 4 different modes selected by --vfs-cache-mode. The higher 6330 the cache mode the more compatible rclone becomes at the cost of using 6331 disk space. 6332 6333 Note that files are written back to the remote only when they are closed 6334 and if they haven't been accessed for --vfs-write-back seconds. If 6335 rclone is quit or dies with files that haven't been uploaded, these will 6336 be uploaded next time rclone is run with the same flags. 6337 6338 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 6339 cache may exceed these quotas for two reasons. Firstly because it is 6340 only checked every --vfs-cache-poll-interval. Secondly because open 6341 files cannot be evicted from the cache. When --vfs-cache-max-size or 6342 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 6343 least accessed files from the cache first. rclone will start with files 6344 that haven't been accessed for the longest. This cache flushing strategy 6345 is efficient and more relevant files are likely to remain cached. 6346 6347 The --vfs-cache-max-age will evict files from the cache after the set 6348 time since last access has passed. The default value of 1 hour will 6349 start evicting files from cache that haven't been accessed for 1 hour. 6350 When a cached file is accessed the 1 hour timer is reset to 0 and will 6351 wait for 1 more hour before evicting. Specify the time with standard 6352 notation, s, m, h, d, w . 6353 6354 You should not run two copies of rclone using the same VFS cache with 6355 the same or overlapping remotes if using --vfs-cache-mode > off. This 6356 can potentially cause data corruption if you do. You can work around 6357 this by giving each rclone its own cache hierarchy with --cache-dir. You 6358 don't need to worry about this if the remotes in use don't overlap. 6359 6360 --vfs-cache-mode off 6361 6362 In this mode (the default) the cache will read directly from the remote 6363 and write directly to the remote without caching anything on disk. 6364 6365 This will mean some operations are not possible 6366 6367 - Files can't be opened for both read AND write 6368 - Files opened for write can't be seeked 6369 - Existing files opened for write must have O_TRUNC set 6370 - Files open for read with O_TRUNC will be opened write only 6371 - Files open for write only will behave as if O_TRUNC was supplied 6372 - Open modes O_APPEND, O_TRUNC are ignored 6373 - If an upload fails it can't be retried 6374 6375 --vfs-cache-mode minimal 6376 6377 This is very similar to "off" except that files opened for read AND 6378 write will be buffered to disk. This means that files opened for write 6379 will be a lot more compatible, but uses the minimal disk space. 6380 6381 These operations are not possible 6382 6383 - Files opened for write only can't be seeked 6384 - Existing files opened for write must have O_TRUNC set 6385 - Files opened for write only will ignore O_APPEND, O_TRUNC 6386 - If an upload fails it can't be retried 6387 6388 --vfs-cache-mode writes 6389 6390 In this mode files opened for read only are still read directly from the 6391 remote, write only and read/write files are buffered to disk first. 6392 6393 This mode should support all normal file system operations. 6394 6395 If an upload fails it will be retried at exponentially increasing 6396 intervals up to 1 minute. 6397 6398 --vfs-cache-mode full 6399 6400 In this mode all reads and writes are buffered to and from disk. When 6401 data is read from the remote this is buffered to disk as well. 6402 6403 In this mode the files in the cache will be sparse files and rclone will 6404 keep track of which bits of the files it has downloaded. 6405 6406 So if an application only reads the starts of each file, then rclone 6407 will only buffer the start of the file. These files will appear to be 6408 their full size in the cache, but they will be sparse files with only 6409 the data that has been downloaded present in them. 6410 6411 This mode should support all normal file system operations and is 6412 otherwise identical to --vfs-cache-mode writes. 6413 6414 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 6415 bytes ahead. The --buffer-size is buffered in memory whereas the 6416 --vfs-read-ahead is buffered on disk. 6417 6418 When using this mode it is recommended that --buffer-size is not set too 6419 large and --vfs-read-ahead is set large if required. 6420 6421 IMPORTANT not all file systems support sparse files. In particular 6422 FAT/exFAT do not. Rclone will perform very badly if the cache directory 6423 is on a filesystem which doesn't support sparse files and it will log an 6424 ERROR message if one is detected. 6425 6426 Fingerprinting 6427 6428 Various parts of the VFS use fingerprinting to see if a local file copy 6429 has changed relative to a remote file. Fingerprints are made from: 6430 6431 - size 6432 - modification time 6433 - hash 6434 6435 where available on an object. 6436 6437 On some backends some of these attributes are slow to read (they take an 6438 extra API call per object, or extra work per object). 6439 6440 For example hash is slow with the local and sftp backends as they have 6441 to read the entire file and hash it, and modtime is slow with the s3, 6442 swift, ftp and qinqstor backends because they need to do an extra API 6443 call to fetch it. 6444 6445 If you use the --vfs-fast-fingerprint flag then rclone will not include 6446 the slow operations in the fingerprint. This makes the fingerprinting 6447 less accurate but much faster and will improve the opening time of 6448 cached files. 6449 6450 If you are running a vfs cache over local, s3 or swift backends then 6451 using this flag is recommended. 6452 6453 Note that if you change the value of this flag, the fingerprints of the 6454 files in the cache may be invalidated and the files will need to be 6455 downloaded again. 6456 6457 VFS Chunked Reading 6458 6459 When rclone reads files from a remote it reads them in chunks. This 6460 means that rather than requesting the whole file rclone reads the chunk 6461 specified. This can reduce the used download quota for some remotes by 6462 requesting only chunks from the remote that are actually read, at the 6463 cost of an increased number of requests. 6464 6465 These flags control the chunking: 6466 6467 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 6468 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 6469 6470 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 6471 then double the size for each read. When --vfs-read-chunk-size-limit is 6472 specified, and greater than --vfs-read-chunk-size, the chunk size for 6473 each open file will get doubled only until the specified value is 6474 reached. If the value is "off", which is the default, the limit is 6475 disabled and the chunk size will grow indefinitely. 6476 6477 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 6478 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 6479 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 6480 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 6481 1200M-1700M and so on. 6482 6483 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 6484 6485 VFS Performance 6486 6487 These flags may be used to enable/disable features of the VFS for 6488 performance or other reasons. See also the chunked reading feature. 6489 6490 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 6491 use --use-server-modtime for a slightly different effect) as each read 6492 of the modification time takes a transaction. 6493 6494 --no-checksum Don't compare checksums on up/download. 6495 --no-modtime Don't read/write the modification time (can speed things up). 6496 --no-seek Don't allow seeking in files. 6497 --read-only Only allow read-only access. 6498 6499 Sometimes rclone is delivered reads or writes out of order. Rather than 6500 seeking rclone will wait a short time for the in sequence read or write 6501 to come in. These flags only come into effect when not using an on disk 6502 cache file. 6503 6504 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 6505 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 6506 6507 When using VFS write caching (--vfs-cache-mode with value writes or 6508 full), the global flag --transfers can be set to adjust the number of 6509 parallel uploads of modified files from the cache (the related global 6510 flag --checkers has no effect on the VFS). 6511 6512 --transfers int Number of file transfers to run in parallel (default 4) 6513 6514 VFS Case Sensitivity 6515 6516 Linux file systems are case-sensitive: two files can differ only by 6517 case, and the exact case must be used when opening a file. 6518 6519 File systems in modern Windows are case-insensitive but case-preserving: 6520 although existing files can be opened using any case, the exact case 6521 used to create the file is preserved and available for programs to 6522 query. It is not allowed for two files in the same directory to differ 6523 only by case. 6524 6525 Usually file systems on macOS are case-insensitive. It is possible to 6526 make macOS file systems case-sensitive but that is not the default. 6527 6528 The --vfs-case-insensitive VFS flag controls how rclone handles these 6529 two cases. If its value is "false", rclone passes file names to the 6530 remote as-is. If the flag is "true" (or appears without a value on the 6531 command line), rclone may perform a "fixup" as explained below. 6532 6533 The user may specify a file name to open/delete/rename/etc with a case 6534 different than what is stored on the remote. If an argument refers to an 6535 existing file with exactly the same name, then the case of the existing 6536 file on the disk will be used. However, if a file name with exactly the 6537 same name is not found but a name differing only by case exists, rclone 6538 will transparently fixup the name. This fixup happens only when an 6539 existing file is requested. Case sensitivity of file names created anew 6540 by rclone is controlled by the underlying remote. 6541 6542 Note that case sensitivity of the operating system running rclone (the 6543 target) may differ from case sensitivity of a file system presented by 6544 rclone (the source). The flag controls whether "fixup" is performed to 6545 satisfy the target. 6546 6547 If the flag is not provided on the command line, then its default value 6548 depends on the operating system where rclone runs: "true" on Windows and 6549 macOS, "false" otherwise. If the flag is provided without a value, then 6550 it is "true". 6551 6552 The --no-unicode-normalization flag controls whether a similar "fixup" 6553 is performed for filenames that differ but are canonically equivalent 6554 with respect to unicode. Unicode normalization can be particularly 6555 helpful for users of macOS, which prefers form NFD instead of the NFC 6556 used by most other platforms. It is therefore highly recommended to keep 6557 the default of false on macOS, to avoid encoding compatibility issues. 6558 6559 In the (probably unlikely) event that a directory has multiple duplicate 6560 filenames after applying case and unicode normalization, the 6561 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 6562 with a performance tradeoff, as rclone will have to scan the entire 6563 directory for duplicates when listing a directory. For this reason, it 6564 is recommended to leave this disabled if not needed. However, macOS 6565 users may wish to consider using it, as otherwise, if a remote directory 6566 contains both NFC and NFD versions of the same filename, an odd 6567 situation will occur: both versions of the file will be visible in the 6568 mount, and both will appear to be editable, however, editing either 6569 version will actually result in only the NFD version getting edited 6570 under the hood. --vfs-block- norm-dupes prevents this confusion by 6571 detecting this scenario, hiding the duplicates, and logging an error, 6572 similar to how this is handled in rclone sync. 6573 6574 VFS Disk Options 6575 6576 This flag allows you to manually set the statistics about the filing 6577 system. It can be useful when those statistics cannot be read correctly 6578 automatically. 6579 6580 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 6581 6582 Alternate report of used bytes 6583 6584 Some backends, most notably S3, do not report the amount of bytes used. 6585 If you need this information to be available when running df on the 6586 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 6587 flag set, instead of relying on the backend to report this information, 6588 rclone will scan the whole remote similar to rclone size and compute the 6589 total used space itself. 6590 6591 WARNING. Contrary to rclone size, this flag ignores filters so that the 6592 result is accurate. However, this is very inefficient and may cost lots 6593 of API calls resulting in extra charges. Use it as a last resort and 6594 only with caching. 6595 6596 rclone nfsmount remote:path /path/to/mountpoint [flags] 6597 6598 Options 6599 6600 --addr string IPaddress:Port or :Port to bind server to 6601 --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) 6602 --allow-other Allow access to other users (not supported on Windows) 6603 --allow-root Allow access to root user (not supported on Windows) 6604 --async-read Use asynchronous reads (not supported on Windows) (default true) 6605 --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) 6606 --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) 6607 --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) 6608 --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) 6609 --debug-fuse Debug the FUSE internals - needs -v 6610 --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) 6611 --devname string Set the device name - default is remote:path 6612 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 6613 --dir-perms FileMode Directory permissions (default 0777) 6614 --file-perms FileMode File permissions (default 0666) 6615 --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) 6616 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 6617 -h, --help help for nfsmount 6618 --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) 6619 --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) 6620 --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) 6621 --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) 6622 --no-checksum Don't compare checksums on up/download 6623 --no-modtime Don't read/write the modification time (can speed things up) 6624 --no-seek Don't allow seeking in files 6625 --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) 6626 --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) 6627 -o, --option stringArray Option for libfuse/WinFsp (repeat if required) 6628 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 6629 --read-only Only allow read-only access 6630 --sudo Use sudo to run the mount command as root. 6631 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 6632 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 6633 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 6634 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 6635 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 6636 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 6637 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 6638 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 6639 --vfs-case-insensitive If a file name not found, find a case insensitive match 6640 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 6641 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 6642 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 6643 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 6644 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 6645 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 6646 --vfs-refresh Refreshes the directory cache recursively in the background on start 6647 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 6648 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 6649 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 6650 --volname string Set the volume name (supported on Windows and OSX only) 6651 --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) 6652 6653 Filter Options 6654 6655 Flags for filtering directory listings. 6656 6657 --delete-excluded Delete files on dest excluded from sync 6658 --exclude stringArray Exclude files matching pattern 6659 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 6660 --exclude-if-present stringArray Exclude directories if filename is present 6661 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 6662 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 6663 -f, --filter stringArray Add a file filtering rule 6664 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 6665 --ignore-case Ignore case in filters (case insensitive) 6666 --include stringArray Include files matching pattern 6667 --include-from stringArray Read file include patterns from file (use - to read from stdin) 6668 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 6669 --max-depth int If set limits the recursion depth to this (default -1) 6670 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 6671 --metadata-exclude stringArray Exclude metadatas matching pattern 6672 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 6673 --metadata-filter stringArray Add a metadata filtering rule 6674 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 6675 --metadata-include stringArray Include metadatas matching pattern 6676 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 6677 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 6678 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 6679 6680 See the global flags page for global options not listed here. 6681 6682 SEE ALSO 6683 6684 - rclone - Show help for rclone commands, flags and backends. 6685 6686 rclone obscure 6687 6688 Obscure password for use in the rclone config file. 6689 6690 Synopsis 6691 6692 In the rclone config file, human-readable passwords are obscured. 6693 Obscuring them is done by encrypting them and writing them out in 6694 base64. This is not a secure way of encrypting these passwords as rclone 6695 can decrypt them - it is to prevent "eyedropping" - namely someone 6696 seeing a password in the rclone config file by accident. 6697 6698 Many equally important things (like access tokens) are not obscured in 6699 the config file. However it is very hard to shoulder surf a 64 character 6700 hex token. 6701 6702 This command can also accept a password through STDIN instead of an 6703 argument by passing a hyphen as an argument. This will use the first 6704 line of STDIN as the password not including the trailing newline. 6705 6706 echo "secretpassword" | rclone obscure - 6707 6708 If there is no data on STDIN to read, rclone obscure will default to 6709 obfuscating the hyphen itself. 6710 6711 If you want to encrypt the config file then please use config file 6712 encryption - see rclone config for more info. 6713 6714 rclone obscure password [flags] 6715 6716 Options 6717 6718 -h, --help help for obscure 6719 6720 See the global flags page for global options not listed here. 6721 6722 SEE ALSO 6723 6724 - rclone - Show help for rclone commands, flags and backends. 6725 6726 rclone rc 6727 6728 Run a command against a running rclone. 6729 6730 Synopsis 6731 6732 This runs a command against a running rclone. Use the --url flag to 6733 specify an non default URL to connect on. This can be either a ":port" 6734 which is taken to mean "http://localhost:port" or a "host:port" which is 6735 taken to mean "http://host:port" 6736 6737 A username and password can be passed in with --user and --pass. 6738 6739 Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, 6740 --user, --pass. 6741 6742 Arguments should be passed in as parameter=value. 6743 6744 The result will be returned as a JSON object by default. 6745 6746 The --json parameter can be used to pass in a JSON blob as an input 6747 instead of key=value arguments. This is the only way of passing in more 6748 complicated values. 6749 6750 The -o/--opt option can be used to set a key "opt" with key, value 6751 options in the form -o key=value or -o key. It can be repeated as many 6752 times as required. This is useful for rc commands which take the "opt" 6753 parameter which by convention is a dictionary of strings. 6754 6755 -o key=value -o key2 6756 6757 Will place this in the "opt" value 6758 6759 {"key":"value", "key2","") 6760 6761 The -a/--arg option can be used to set strings in the "arg" value. It 6762 can be repeated as many times as required. This is useful for rc 6763 commands which take the "arg" parameter which by convention is a list of 6764 strings. 6765 6766 -a value -a value2 6767 6768 Will place this in the "arg" value 6769 6770 ["value", "value2"] 6771 6772 Use --loopback to connect to the rclone instance running rclone rc. This 6773 is very useful for testing commands without having to run an rclone rc 6774 server, e.g.: 6775 6776 rclone rc --loopback operations/about fs=/ 6777 6778 Use rclone rc to see a list of all possible commands. 6779 6780 rclone rc commands parameter [flags] 6781 6782 Options 6783 6784 -a, --arg stringArray Argument placed in the "arg" array 6785 -h, --help help for rc 6786 --json string Input JSON - use instead of key=value args 6787 --loopback If set connect to this rclone instance not via HTTP 6788 --no-output If set, don't output the JSON result 6789 -o, --opt stringArray Option in the form name=value or name placed in the "opt" array 6790 --pass string Password to use to connect to rclone remote control 6791 --url string URL to connect to rclone remote control (default "http://localhost:5572/") 6792 --user string Username to use to rclone remote control 6793 6794 See the global flags page for global options not listed here. 6795 6796 SEE ALSO 6797 6798 - rclone - Show help for rclone commands, flags and backends. 6799 6800 rclone rcat 6801 6802 Copies standard input to file on remote. 6803 6804 Synopsis 6805 6806 rclone rcat reads from standard input (stdin) and copies it to a single 6807 remote file. 6808 6809 echo "hello world" | rclone rcat remote:path/to/file 6810 ffmpeg - | rclone rcat remote:path/to/file 6811 6812 If the remote file already exists, it will be overwritten. 6813 6814 rcat will try to upload small files in a single request, which is 6815 usually more efficient than the streaming/chunked upload endpoints, 6816 which use multiple requests. Exact behaviour depends on the remote. What 6817 is considered a small file may be set through --streaming-upload-cutoff. 6818 Uploading only starts after the cutoff is reached or if the file ends 6819 before that. The data must fit into RAM. The cutoff needs to be small 6820 enough to adhere the limits of your remote, please see there. Generally 6821 speaking, setting this cutoff too high will decrease your performance. 6822 6823 Use the --size flag to preallocate the file in advance at the remote end 6824 and actually stream it, even if remote backend doesn't support 6825 streaming. 6826 6827 --size should be the exact size of the input stream in bytes. If the 6828 size of the stream is different in length to the --size passed in then 6829 the transfer will likely fail. 6830 6831 Note that the upload cannot be retried because the data is not stored. 6832 If the backend supports multipart uploading then individual chunks can 6833 be retried. If you need to transfer a lot of data, you may be better off 6834 caching it locally and then rclone move it to the destination which can 6835 use retries. 6836 6837 rclone rcat remote:path [flags] 6838 6839 Options 6840 6841 -h, --help help for rcat 6842 --size int File size hint to preallocate (default -1) 6843 6844 Important Options 6845 6846 Important flags useful for most commands. 6847 6848 -n, --dry-run Do a trial run with no permanent changes 6849 -i, --interactive Enable interactive mode 6850 -v, --verbose count Print lots more stuff (repeat for more) 6851 6852 See the global flags page for global options not listed here. 6853 6854 SEE ALSO 6855 6856 - rclone - Show help for rclone commands, flags and backends. 6857 6858 rclone rcd 6859 6860 Run rclone listening to remote control commands only. 6861 6862 Synopsis 6863 6864 This runs rclone so that it only listens to remote control commands. 6865 6866 This is useful if you are controlling rclone via the rc API. 6867 6868 If you pass in a path to a directory, rclone will serve that directory 6869 for GET requests on the URL passed in. It will also open the URL in the 6870 browser when rclone is run. 6871 6872 See the rc documentation for more info on the rc flags. 6873 6874 Server options 6875 6876 Use --rc-addr to specify which IP address and port the server should 6877 listen on, eg --rc-addr 1.2.3.4:8000 or --rc-addr :8080 to listen to all 6878 IPs. By default it only listens on localhost. You can use port :0 to let 6879 the OS choose an available port. 6880 6881 If you set --rc-addr to listen on a public or LAN accessible IP address 6882 then using Authentication is advised - see the next section for info. 6883 6884 You can use a unix socket by setting the url to unix:///path/to/socket 6885 or just by using an absolute path name. Note that unix sockets bypass 6886 the authentication - this is expected to be done with file system 6887 permissions. 6888 6889 --rc-addr may be repeated to listen on multiple IPs/ports/sockets. 6890 6891 --rc-server-read-timeout and --rc-server-write-timeout can be used to 6892 control the timeouts on the server. Note that this is the total time for 6893 a transfer. 6894 6895 --rc-max-header-bytes controls the maximum number of bytes the server 6896 will accept in the HTTP header. 6897 6898 --rc-baseurl controls the URL prefix that rclone serves from. By default 6899 rclone will serve from the root. If you used --rc-baseurl "/rclone" then 6900 rclone would serve from a URL starting with "/rclone/". This is useful 6901 if you wish to proxy rclone serve. Rclone automatically inserts leading 6902 and trailing "/" on --rc-baseurl, so --rc-baseurl "rclone", 6903 --rc-baseurl "/rclone" and --rc-baseurl "/rclone/" are all treated 6904 identically. 6905 6906 TLS (SSL) 6907 6908 By default this will serve over http. If you want you can serve over 6909 https. You will need to supply the --rc-cert and --rc-key flags. If you 6910 wish to do client side certificate validation then you will need to 6911 supply --rc-client-ca also. 6912 6913 --rc-cert should be a either a PEM encoded certificate or a 6914 concatenation of that with the CA certificate. --krc-ey should be the 6915 PEM encoded private key and --rc-client-ca should be the PEM encoded 6916 client certificate authority certificate. 6917 6918 --rc-min-tls-version is minimum TLS version that is acceptable. Valid 6919 values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 6920 6921 Template 6922 6923 --rc-template allows a user to specify a custom markup template for HTTP 6924 and WebDAV serve functions. The server exports the following markup to 6925 be used within the template to server pages: 6926 6927 ----------------------------------------------------------------------- 6928 Parameter Description 6929 ----------------------------------- ----------------------------------- 6930 .Name The full path of a file/directory. 6931 6932 .Title Directory listing of .Name 6933 6934 .Sort The current sort used. This is 6935 changeable via ?sort= parameter 6936 6937 Sort Options: 6938 namedirfirst,name,size,time 6939 (default namedirfirst) 6940 6941 .Order The current ordering used. This is 6942 changeable via ?order= parameter 6943 6944 Order Options: asc,desc (default 6945 asc) 6946 6947 .Query Currently unused. 6948 6949 .Breadcrumb Allows for creating a relative 6950 navigation 6951 6952 -- .Link The relative to the root link of 6953 the Text. 6954 6955 -- .Text The Name of the directory. 6956 6957 .Entries Information about a specific 6958 file/directory. 6959 6960 -- .URL The 'url' of an entry. 6961 6962 -- .Leaf Currently same as 'URL' but 6963 intended to be 'just' the name. 6964 6965 -- .IsDir Boolean for if an entry is a 6966 directory or not. 6967 6968 -- .Size Size in Bytes of the entry. 6969 6970 -- .ModTime The UTC timestamp of an entry. 6971 ----------------------------------------------------------------------- 6972 6973 The server also makes the following functions available so that they can 6974 be used within the template. These functions help extend the options for 6975 dynamic rendering of HTML. They can be used to render HTML based on 6976 specific conditions. 6977 6978 ----------------------------------------------------------------------- 6979 Function Description 6980 ----------------------------------- ----------------------------------- 6981 afterEpoch Returns the time since the epoch 6982 for the given time. 6983 6984 contains Checks whether a given substring is 6985 present or not in a given string. 6986 6987 hasPrefix Checks whether the given string 6988 begins with the specified prefix. 6989 6990 hasSuffix Checks whether the given string end 6991 with the specified suffix. 6992 ----------------------------------------------------------------------- 6993 6994 Authentication 6995 6996 By default this will serve files without needing a login. 6997 6998 You can either use an htpasswd file which can take lots of users, or set 6999 a single username and password with the --rc-user and --rc-pass flags. 7000 7001 If no static users are configured by either of the above methods, and 7002 client certificates are required by the --client-ca flag passed to the 7003 server, the client certificate common name will be considered as the 7004 username. 7005 7006 Use --rc-htpasswd /path/to/htpasswd to provide an htpasswd file. This is 7007 in standard apache format and supports MD5, SHA1 and BCrypt for basic 7008 authentication. Bcrypt is recommended. 7009 7010 To create an htpasswd file: 7011 7012 touch htpasswd 7013 htpasswd -B htpasswd user 7014 htpasswd -B htpasswd anotherUser 7015 7016 The password file can be updated while rclone is running. 7017 7018 Use --rc-realm to set the authentication realm. 7019 7020 Use --rc-salt to change the password hashing salt from the default. 7021 7022 rclone rcd <path to files to serve>* [flags] 7023 7024 Options 7025 7026 -h, --help help for rcd 7027 7028 RC Options 7029 7030 Flags to control the Remote Control API. 7031 7032 --rc Enable the remote control server 7033 --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) 7034 --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from 7035 --rc-baseurl string Prefix for URLs - leave blank for root 7036 --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) 7037 --rc-client-ca string Client certificate authority to verify clients with 7038 --rc-enable-metrics Enable prometheus metrics on /metrics 7039 --rc-files string Path to local files to serve on the HTTP server 7040 --rc-htpasswd string A htpasswd file - if not provided no authentication is done 7041 --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) 7042 --rc-job-expire-interval Duration Interval to check for expired async jobs (default 10s) 7043 --rc-key string TLS PEM Private key 7044 --rc-max-header-bytes int Maximum size of request header (default 4096) 7045 --rc-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 7046 --rc-no-auth Don't require auth for certain methods 7047 --rc-pass string Password for authentication 7048 --rc-realm string Realm for authentication 7049 --rc-salt string Password hashing salt (default "dlPL2MqE") 7050 --rc-serve Enable the serving of remote objects 7051 --rc-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 7052 --rc-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 7053 --rc-template string User-specified template 7054 --rc-user string User name for authentication 7055 --rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest") 7056 --rc-web-gui Launch WebGUI on localhost 7057 --rc-web-gui-force-update Force update to latest version of web gui 7058 --rc-web-gui-no-open-browser Don't open the browser automatically 7059 --rc-web-gui-update Check and update to latest version of web gui 7060 7061 See the global flags page for global options not listed here. 7062 7063 SEE ALSO 7064 7065 - rclone - Show help for rclone commands, flags and backends. 7066 7067 rclone rmdirs 7068 7069 Remove empty directories under the path. 7070 7071 Synopsis 7072 7073 This recursively removes any empty directories (including directories 7074 that only contain empty directories), that it finds under the path. The 7075 root path itself will also be removed if it is empty, unless you supply 7076 the --leave-root flag. 7077 7078 Use command rmdir to delete just the empty directory given by path, not 7079 recurse. 7080 7081 This is useful for tidying up remotes that rclone has left a lot of 7082 empty directories in. For example the delete command will delete files 7083 but leave the directory structure (unless used with option --rmdirs). 7084 7085 This will delete --checkers directories concurrently so if you have 7086 thousands of empty directories consider increasing this number. 7087 7088 To delete a path and any objects in it, use the purge command. 7089 7090 rclone rmdirs remote:path [flags] 7091 7092 Options 7093 7094 -h, --help help for rmdirs 7095 --leave-root Do not remove root directory if empty 7096 7097 Important Options 7098 7099 Important flags useful for most commands. 7100 7101 -n, --dry-run Do a trial run with no permanent changes 7102 -i, --interactive Enable interactive mode 7103 -v, --verbose count Print lots more stuff (repeat for more) 7104 7105 See the global flags page for global options not listed here. 7106 7107 SEE ALSO 7108 7109 - rclone - Show help for rclone commands, flags and backends. 7110 7111 rclone selfupdate 7112 7113 Update the rclone binary. 7114 7115 Synopsis 7116 7117 This command downloads the latest release of rclone and replaces the 7118 currently running binary. The download is verified with a hashsum and 7119 cryptographically signed signature; see the release signing docs for 7120 details. 7121 7122 If used without flags (or with implied --stable flag), this command will 7123 install the latest stable release. However, some issues may be fixed (or 7124 features added) only in the latest beta release. In such cases you 7125 should run the command with the --beta flag, i.e. 7126 rclone selfupdate --beta. You can check in advance what version would be 7127 installed by adding the --check flag, then repeat the command without it 7128 when you are satisfied. 7129 7130 Sometimes the rclone team may recommend you a concrete beta or stable 7131 rclone release to troubleshoot your issue or add a bleeding edge 7132 feature. The --version VER flag, if given, will update to the concrete 7133 version instead of the latest one. If you omit micro version from VER 7134 (for example 1.53), the latest matching micro version will be used. 7135 7136 Upon successful update rclone will print a message that contains a 7137 previous version number. You will need it if you later decide to revert 7138 your update for some reason. Then you'll have to note the previous 7139 version and run the following command: 7140 rclone selfupdate [--beta] OLDVER. If the old version contains only dots 7141 and digits (for example v1.54.0) then it's a stable release so you won't 7142 need the --beta flag. Beta releases have an additional information 7143 similar to v1.54.0-beta.5111.06f1c0c61. (if you are a developer and use 7144 a locally built rclone, the version number will end with -DEV, you will 7145 have to rebuild it as it obviously can't be distributed). 7146 7147 If you previously installed rclone via a package manager, the package 7148 may include local documentation or configure services. You may wish to 7149 update with the flag --package deb or --package rpm (whichever is 7150 correct for your OS) to update these too. This command with the default 7151 --package zip will update only the rclone executable so the local manual 7152 may become inaccurate after it. 7153 7154 The rclone mount command may or may not support extended FUSE options 7155 depending on the build and OS. selfupdate will refuse to update if the 7156 capability would be discarded. 7157 7158 Note: Windows forbids deletion of a currently running executable so this 7159 command will rename the old executable to 'rclone.old.exe' upon success. 7160 7161 Please note that this command was not available before rclone version 7162 1.55. If it fails for you with the message unknown command "selfupdate" 7163 then you will need to update manually following the install instructions 7164 located at https://rclone.org/install/ 7165 7166 rclone selfupdate [flags] 7167 7168 Options 7169 7170 --beta Install beta release 7171 --check Check for latest release, do not download 7172 -h, --help help for selfupdate 7173 --output string Save the downloaded binary at a given path (default: replace running binary) 7174 --package string Package format: zip|deb|rpm (default: zip) 7175 --stable Install stable release (this is the default) 7176 --version string Install the given rclone version (default: latest) 7177 7178 See the global flags page for global options not listed here. 7179 7180 SEE ALSO 7181 7182 - rclone - Show help for rclone commands, flags and backends. 7183 7184 rclone serve 7185 7186 Serve a remote over a protocol. 7187 7188 Synopsis 7189 7190 Serve a remote over a given protocol. Requires the use of a subcommand 7191 to specify the protocol, e.g. 7192 7193 rclone serve http remote: 7194 7195 Each subcommand has its own options which you can see in their help. 7196 7197 rclone serve <protocol> [opts] <remote> [flags] 7198 7199 Options 7200 7201 -h, --help help for serve 7202 7203 See the global flags page for global options not listed here. 7204 7205 SEE ALSO 7206 7207 - rclone - Show help for rclone commands, flags and backends. 7208 - rclone serve dlna - Serve remote:path over DLNA 7209 - rclone serve docker - Serve any remote on docker's volume plugin 7210 API. 7211 - rclone serve ftp - Serve remote:path over FTP. 7212 - rclone serve http - Serve the remote over HTTP. 7213 - rclone serve nfs - Serve the remote as an NFS mount 7214 - rclone serve restic - Serve the remote for restic's REST API. 7215 - rclone serve s3 - Serve remote:path over s3. 7216 - rclone serve sftp - Serve the remote over SFTP. 7217 - rclone serve webdav - Serve remote:path over WebDAV. 7218 7219 rclone serve dlna 7220 7221 Serve remote:path over DLNA 7222 7223 Synopsis 7224 7225 Run a DLNA media server for media stored in an rclone remote. Many 7226 devices, such as the Xbox and PlayStation, can automatically discover 7227 this server in the LAN and play audio/video from it. VLC is also 7228 supported. Service discovery uses UDP multicast packets (SSDP) and will 7229 thus only work on LANs. 7230 7231 Rclone will list all files present in the remote, without filtering 7232 based on media formats or file extensions. Additionally, there is no 7233 media transcoding support. This means that some players might show files 7234 that they are not able to play back correctly. 7235 7236 Server options 7237 7238 Use --addr to specify which IP address and port the server should listen 7239 on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. 7240 7241 Use --name to choose the friendly server name, which is by default 7242 "rclone (hostname)". 7243 7244 Use --log-trace in conjunction with -vv to enable additional debug 7245 logging of all UPNP traffic. ## VFS - Virtual File System 7246 7247 This command uses the VFS layer. This adapts the cloud storage objects 7248 that rclone uses into something which looks much more like a disk filing 7249 system. 7250 7251 Cloud storage objects have lots of properties which aren't like disk 7252 files - you can't extend them or write to the middle of them, so the VFS 7253 layer has to deal with that. Because there is no one right way of doing 7254 this there are various options explained below. 7255 7256 The VFS layer also implements a directory cache - this caches info about 7257 files and directories (but not the data) in memory. 7258 7259 VFS Directory Cache 7260 7261 Using the --dir-cache-time flag, you can control how long a directory 7262 should be considered up to date and not refreshed from the backend. 7263 Changes made through the VFS will appear immediately or invalidate the 7264 cache. 7265 7266 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 7267 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 7268 7269 However, changes made directly on the cloud storage by the web interface 7270 or a different copy of rclone will only be picked up once the directory 7271 cache expires if the backend configured does not support polling for 7272 changes. If the backend supports polling, changes will be picked up 7273 within the polling interval. 7274 7275 You can send a SIGHUP signal to rclone for it to flush all directory 7276 caches, regardless of how old they are. Assuming only one rclone 7277 instance is running, you can reset the cache like this: 7278 7279 kill -SIGHUP $(pidof rclone) 7280 7281 If you configure rclone with a remote control then you can use rclone rc 7282 to flush the whole directory cache: 7283 7284 rclone rc vfs/forget 7285 7286 Or individual files or directories: 7287 7288 rclone rc vfs/forget file=path/to/file dir=path/to/dir 7289 7290 VFS File Buffering 7291 7292 The --buffer-size flag determines the amount of memory, that will be 7293 used to buffer data in advance. 7294 7295 Each open file will try to keep the specified amount of data in memory 7296 at all times. The buffered data is bound to one open file and won't be 7297 shared. 7298 7299 This flag is a upper limit for the used memory per open file. The buffer 7300 will only use memory for data that is downloaded but not not yet read. 7301 If the buffer is empty, only a small amount of memory will be used. 7302 7303 The maximum memory used by rclone for buffering can be up to 7304 --buffer-size * open files. 7305 7306 VFS File Caching 7307 7308 These flags control the VFS file caching options. File caching is 7309 necessary to make the VFS layer appear compatible with a normal file 7310 system. It can be disabled at the cost of some compatibility. 7311 7312 For example you'll need to enable VFS caching if you want to read and 7313 write simultaneously to a file. See below for more details. 7314 7315 Note that the VFS cache is separate from the cache backend and you may 7316 find that you need one or the other or both. 7317 7318 --cache-dir string Directory rclone will use for caching. 7319 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 7320 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 7321 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 7322 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 7323 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 7324 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 7325 7326 If run with -vv rclone will print the location of the file cache. The 7327 files are stored in the user cache file area which is OS dependent but 7328 can be controlled with --cache-dir or setting the appropriate 7329 environment variable. 7330 7331 The cache has 4 different modes selected by --vfs-cache-mode. The higher 7332 the cache mode the more compatible rclone becomes at the cost of using 7333 disk space. 7334 7335 Note that files are written back to the remote only when they are closed 7336 and if they haven't been accessed for --vfs-write-back seconds. If 7337 rclone is quit or dies with files that haven't been uploaded, these will 7338 be uploaded next time rclone is run with the same flags. 7339 7340 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 7341 cache may exceed these quotas for two reasons. Firstly because it is 7342 only checked every --vfs-cache-poll-interval. Secondly because open 7343 files cannot be evicted from the cache. When --vfs-cache-max-size or 7344 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 7345 least accessed files from the cache first. rclone will start with files 7346 that haven't been accessed for the longest. This cache flushing strategy 7347 is efficient and more relevant files are likely to remain cached. 7348 7349 The --vfs-cache-max-age will evict files from the cache after the set 7350 time since last access has passed. The default value of 1 hour will 7351 start evicting files from cache that haven't been accessed for 1 hour. 7352 When a cached file is accessed the 1 hour timer is reset to 0 and will 7353 wait for 1 more hour before evicting. Specify the time with standard 7354 notation, s, m, h, d, w . 7355 7356 You should not run two copies of rclone using the same VFS cache with 7357 the same or overlapping remotes if using --vfs-cache-mode > off. This 7358 can potentially cause data corruption if you do. You can work around 7359 this by giving each rclone its own cache hierarchy with --cache-dir. You 7360 don't need to worry about this if the remotes in use don't overlap. 7361 7362 --vfs-cache-mode off 7363 7364 In this mode (the default) the cache will read directly from the remote 7365 and write directly to the remote without caching anything on disk. 7366 7367 This will mean some operations are not possible 7368 7369 - Files can't be opened for both read AND write 7370 - Files opened for write can't be seeked 7371 - Existing files opened for write must have O_TRUNC set 7372 - Files open for read with O_TRUNC will be opened write only 7373 - Files open for write only will behave as if O_TRUNC was supplied 7374 - Open modes O_APPEND, O_TRUNC are ignored 7375 - If an upload fails it can't be retried 7376 7377 --vfs-cache-mode minimal 7378 7379 This is very similar to "off" except that files opened for read AND 7380 write will be buffered to disk. This means that files opened for write 7381 will be a lot more compatible, but uses the minimal disk space. 7382 7383 These operations are not possible 7384 7385 - Files opened for write only can't be seeked 7386 - Existing files opened for write must have O_TRUNC set 7387 - Files opened for write only will ignore O_APPEND, O_TRUNC 7388 - If an upload fails it can't be retried 7389 7390 --vfs-cache-mode writes 7391 7392 In this mode files opened for read only are still read directly from the 7393 remote, write only and read/write files are buffered to disk first. 7394 7395 This mode should support all normal file system operations. 7396 7397 If an upload fails it will be retried at exponentially increasing 7398 intervals up to 1 minute. 7399 7400 --vfs-cache-mode full 7401 7402 In this mode all reads and writes are buffered to and from disk. When 7403 data is read from the remote this is buffered to disk as well. 7404 7405 In this mode the files in the cache will be sparse files and rclone will 7406 keep track of which bits of the files it has downloaded. 7407 7408 So if an application only reads the starts of each file, then rclone 7409 will only buffer the start of the file. These files will appear to be 7410 their full size in the cache, but they will be sparse files with only 7411 the data that has been downloaded present in them. 7412 7413 This mode should support all normal file system operations and is 7414 otherwise identical to --vfs-cache-mode writes. 7415 7416 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 7417 bytes ahead. The --buffer-size is buffered in memory whereas the 7418 --vfs-read-ahead is buffered on disk. 7419 7420 When using this mode it is recommended that --buffer-size is not set too 7421 large and --vfs-read-ahead is set large if required. 7422 7423 IMPORTANT not all file systems support sparse files. In particular 7424 FAT/exFAT do not. Rclone will perform very badly if the cache directory 7425 is on a filesystem which doesn't support sparse files and it will log an 7426 ERROR message if one is detected. 7427 7428 Fingerprinting 7429 7430 Various parts of the VFS use fingerprinting to see if a local file copy 7431 has changed relative to a remote file. Fingerprints are made from: 7432 7433 - size 7434 - modification time 7435 - hash 7436 7437 where available on an object. 7438 7439 On some backends some of these attributes are slow to read (they take an 7440 extra API call per object, or extra work per object). 7441 7442 For example hash is slow with the local and sftp backends as they have 7443 to read the entire file and hash it, and modtime is slow with the s3, 7444 swift, ftp and qinqstor backends because they need to do an extra API 7445 call to fetch it. 7446 7447 If you use the --vfs-fast-fingerprint flag then rclone will not include 7448 the slow operations in the fingerprint. This makes the fingerprinting 7449 less accurate but much faster and will improve the opening time of 7450 cached files. 7451 7452 If you are running a vfs cache over local, s3 or swift backends then 7453 using this flag is recommended. 7454 7455 Note that if you change the value of this flag, the fingerprints of the 7456 files in the cache may be invalidated and the files will need to be 7457 downloaded again. 7458 7459 VFS Chunked Reading 7460 7461 When rclone reads files from a remote it reads them in chunks. This 7462 means that rather than requesting the whole file rclone reads the chunk 7463 specified. This can reduce the used download quota for some remotes by 7464 requesting only chunks from the remote that are actually read, at the 7465 cost of an increased number of requests. 7466 7467 These flags control the chunking: 7468 7469 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 7470 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 7471 7472 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 7473 then double the size for each read. When --vfs-read-chunk-size-limit is 7474 specified, and greater than --vfs-read-chunk-size, the chunk size for 7475 each open file will get doubled only until the specified value is 7476 reached. If the value is "off", which is the default, the limit is 7477 disabled and the chunk size will grow indefinitely. 7478 7479 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 7480 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 7481 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 7482 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 7483 1200M-1700M and so on. 7484 7485 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 7486 7487 VFS Performance 7488 7489 These flags may be used to enable/disable features of the VFS for 7490 performance or other reasons. See also the chunked reading feature. 7491 7492 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 7493 use --use-server-modtime for a slightly different effect) as each read 7494 of the modification time takes a transaction. 7495 7496 --no-checksum Don't compare checksums on up/download. 7497 --no-modtime Don't read/write the modification time (can speed things up). 7498 --no-seek Don't allow seeking in files. 7499 --read-only Only allow read-only access. 7500 7501 Sometimes rclone is delivered reads or writes out of order. Rather than 7502 seeking rclone will wait a short time for the in sequence read or write 7503 to come in. These flags only come into effect when not using an on disk 7504 cache file. 7505 7506 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 7507 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 7508 7509 When using VFS write caching (--vfs-cache-mode with value writes or 7510 full), the global flag --transfers can be set to adjust the number of 7511 parallel uploads of modified files from the cache (the related global 7512 flag --checkers has no effect on the VFS). 7513 7514 --transfers int Number of file transfers to run in parallel (default 4) 7515 7516 VFS Case Sensitivity 7517 7518 Linux file systems are case-sensitive: two files can differ only by 7519 case, and the exact case must be used when opening a file. 7520 7521 File systems in modern Windows are case-insensitive but case-preserving: 7522 although existing files can be opened using any case, the exact case 7523 used to create the file is preserved and available for programs to 7524 query. It is not allowed for two files in the same directory to differ 7525 only by case. 7526 7527 Usually file systems on macOS are case-insensitive. It is possible to 7528 make macOS file systems case-sensitive but that is not the default. 7529 7530 The --vfs-case-insensitive VFS flag controls how rclone handles these 7531 two cases. If its value is "false", rclone passes file names to the 7532 remote as-is. If the flag is "true" (or appears without a value on the 7533 command line), rclone may perform a "fixup" as explained below. 7534 7535 The user may specify a file name to open/delete/rename/etc with a case 7536 different than what is stored on the remote. If an argument refers to an 7537 existing file with exactly the same name, then the case of the existing 7538 file on the disk will be used. However, if a file name with exactly the 7539 same name is not found but a name differing only by case exists, rclone 7540 will transparently fixup the name. This fixup happens only when an 7541 existing file is requested. Case sensitivity of file names created anew 7542 by rclone is controlled by the underlying remote. 7543 7544 Note that case sensitivity of the operating system running rclone (the 7545 target) may differ from case sensitivity of a file system presented by 7546 rclone (the source). The flag controls whether "fixup" is performed to 7547 satisfy the target. 7548 7549 If the flag is not provided on the command line, then its default value 7550 depends on the operating system where rclone runs: "true" on Windows and 7551 macOS, "false" otherwise. If the flag is provided without a value, then 7552 it is "true". 7553 7554 The --no-unicode-normalization flag controls whether a similar "fixup" 7555 is performed for filenames that differ but are canonically equivalent 7556 with respect to unicode. Unicode normalization can be particularly 7557 helpful for users of macOS, which prefers form NFD instead of the NFC 7558 used by most other platforms. It is therefore highly recommended to keep 7559 the default of false on macOS, to avoid encoding compatibility issues. 7560 7561 In the (probably unlikely) event that a directory has multiple duplicate 7562 filenames after applying case and unicode normalization, the 7563 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 7564 with a performance tradeoff, as rclone will have to scan the entire 7565 directory for duplicates when listing a directory. For this reason, it 7566 is recommended to leave this disabled if not needed. However, macOS 7567 users may wish to consider using it, as otherwise, if a remote directory 7568 contains both NFC and NFD versions of the same filename, an odd 7569 situation will occur: both versions of the file will be visible in the 7570 mount, and both will appear to be editable, however, editing either 7571 version will actually result in only the NFD version getting edited 7572 under the hood. --vfs-block- norm-dupes prevents this confusion by 7573 detecting this scenario, hiding the duplicates, and logging an error, 7574 similar to how this is handled in rclone sync. 7575 7576 VFS Disk Options 7577 7578 This flag allows you to manually set the statistics about the filing 7579 system. It can be useful when those statistics cannot be read correctly 7580 automatically. 7581 7582 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 7583 7584 Alternate report of used bytes 7585 7586 Some backends, most notably S3, do not report the amount of bytes used. 7587 If you need this information to be available when running df on the 7588 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 7589 flag set, instead of relying on the backend to report this information, 7590 rclone will scan the whole remote similar to rclone size and compute the 7591 total used space itself. 7592 7593 WARNING. Contrary to rclone size, this flag ignores filters so that the 7594 result is accurate. However, this is very inefficient and may cost lots 7595 of API calls resulting in extra charges. Use it as a last resort and 7596 only with caching. 7597 7598 rclone serve dlna remote:path [flags] 7599 7600 Options 7601 7602 --addr string The ip:port or :port to bind the DLNA http server to (default ":7879") 7603 --announce-interval Duration The interval between SSDP announcements (default 12m0s) 7604 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 7605 --dir-perms FileMode Directory permissions (default 0777) 7606 --file-perms FileMode File permissions (default 0666) 7607 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 7608 -h, --help help for dlna 7609 --interface stringArray The interface to use for SSDP (repeat as necessary) 7610 --log-trace Enable trace logging of SOAP traffic 7611 --name string Name of DLNA server 7612 --no-checksum Don't compare checksums on up/download 7613 --no-modtime Don't read/write the modification time (can speed things up) 7614 --no-seek Don't allow seeking in files 7615 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 7616 --read-only Only allow read-only access 7617 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 7618 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 7619 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 7620 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 7621 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 7622 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 7623 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 7624 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 7625 --vfs-case-insensitive If a file name not found, find a case insensitive match 7626 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 7627 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 7628 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 7629 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 7630 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 7631 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 7632 --vfs-refresh Refreshes the directory cache recursively in the background on start 7633 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 7634 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 7635 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 7636 7637 Filter Options 7638 7639 Flags for filtering directory listings. 7640 7641 --delete-excluded Delete files on dest excluded from sync 7642 --exclude stringArray Exclude files matching pattern 7643 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 7644 --exclude-if-present stringArray Exclude directories if filename is present 7645 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 7646 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 7647 -f, --filter stringArray Add a file filtering rule 7648 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 7649 --ignore-case Ignore case in filters (case insensitive) 7650 --include stringArray Include files matching pattern 7651 --include-from stringArray Read file include patterns from file (use - to read from stdin) 7652 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 7653 --max-depth int If set limits the recursion depth to this (default -1) 7654 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 7655 --metadata-exclude stringArray Exclude metadatas matching pattern 7656 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 7657 --metadata-filter stringArray Add a metadata filtering rule 7658 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 7659 --metadata-include stringArray Include metadatas matching pattern 7660 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 7661 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 7662 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 7663 7664 See the global flags page for global options not listed here. 7665 7666 SEE ALSO 7667 7668 - rclone serve - Serve a remote over a protocol. 7669 7670 rclone serve docker 7671 7672 Serve any remote on docker's volume plugin API. 7673 7674 Synopsis 7675 7676 This command implements the Docker volume plugin API allowing docker to 7677 use rclone as a data storage mechanism for various cloud providers. 7678 rclone provides docker volume plugin based on it. 7679 7680 To create a docker plugin, one must create a Unix or TCP socket that 7681 Docker will look for when you use the plugin and then it listens for 7682 commands from docker daemon and runs the corresponding code when 7683 necessary. Docker plugins can run as a managed plugin under control of 7684 the docker daemon or as an independent native service. For testing, you 7685 can just run it directly from the command line, for example: 7686 7687 sudo rclone serve docker --base-dir /tmp/rclone-volumes --socket-addr localhost:8787 -vv 7688 7689 Running rclone serve docker will create the said socket, listening for 7690 commands from Docker to create the necessary Volumes. Normally you need 7691 not give the --socket-addr flag. The API will listen on the unix domain 7692 socket at /run/docker/plugins/rclone.sock. In the example above rclone 7693 will create a TCP socket and a small file 7694 /etc/docker/plugins/rclone.spec containing the socket address. We use 7695 sudo because both paths are writeable only by the root user. 7696 7697 If you later decide to change listening socket, the docker daemon must 7698 be restarted to reconnect to /run/docker/plugins/rclone.sock or parse 7699 new /etc/docker/plugins/rclone.spec. Until you restart, any volume 7700 related docker commands will timeout trying to access the old socket. 7701 Running directly is supported on Linux only, not on Windows or MacOS. 7702 This is not a problem with managed plugin mode described in details in 7703 the full documentation. 7704 7705 The command will create volume mounts under the path given by --base-dir 7706 (by default /var/lib/docker-volumes/rclone available only to root) and 7707 maintain the JSON formatted file docker-plugin.state in the rclone cache 7708 directory with book-keeping records of created and mounted volumes. 7709 7710 All mount and VFS options are submitted by the docker daemon via API, 7711 but you can also provide defaults on the command line as well as set 7712 path to the config file and cache directory or adjust logging verbosity. 7713 ## VFS - Virtual File System 7714 7715 This command uses the VFS layer. This adapts the cloud storage objects 7716 that rclone uses into something which looks much more like a disk filing 7717 system. 7718 7719 Cloud storage objects have lots of properties which aren't like disk 7720 files - you can't extend them or write to the middle of them, so the VFS 7721 layer has to deal with that. Because there is no one right way of doing 7722 this there are various options explained below. 7723 7724 The VFS layer also implements a directory cache - this caches info about 7725 files and directories (but not the data) in memory. 7726 7727 VFS Directory Cache 7728 7729 Using the --dir-cache-time flag, you can control how long a directory 7730 should be considered up to date and not refreshed from the backend. 7731 Changes made through the VFS will appear immediately or invalidate the 7732 cache. 7733 7734 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 7735 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 7736 7737 However, changes made directly on the cloud storage by the web interface 7738 or a different copy of rclone will only be picked up once the directory 7739 cache expires if the backend configured does not support polling for 7740 changes. If the backend supports polling, changes will be picked up 7741 within the polling interval. 7742 7743 You can send a SIGHUP signal to rclone for it to flush all directory 7744 caches, regardless of how old they are. Assuming only one rclone 7745 instance is running, you can reset the cache like this: 7746 7747 kill -SIGHUP $(pidof rclone) 7748 7749 If you configure rclone with a remote control then you can use rclone rc 7750 to flush the whole directory cache: 7751 7752 rclone rc vfs/forget 7753 7754 Or individual files or directories: 7755 7756 rclone rc vfs/forget file=path/to/file dir=path/to/dir 7757 7758 VFS File Buffering 7759 7760 The --buffer-size flag determines the amount of memory, that will be 7761 used to buffer data in advance. 7762 7763 Each open file will try to keep the specified amount of data in memory 7764 at all times. The buffered data is bound to one open file and won't be 7765 shared. 7766 7767 This flag is a upper limit for the used memory per open file. The buffer 7768 will only use memory for data that is downloaded but not not yet read. 7769 If the buffer is empty, only a small amount of memory will be used. 7770 7771 The maximum memory used by rclone for buffering can be up to 7772 --buffer-size * open files. 7773 7774 VFS File Caching 7775 7776 These flags control the VFS file caching options. File caching is 7777 necessary to make the VFS layer appear compatible with a normal file 7778 system. It can be disabled at the cost of some compatibility. 7779 7780 For example you'll need to enable VFS caching if you want to read and 7781 write simultaneously to a file. See below for more details. 7782 7783 Note that the VFS cache is separate from the cache backend and you may 7784 find that you need one or the other or both. 7785 7786 --cache-dir string Directory rclone will use for caching. 7787 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 7788 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 7789 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 7790 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 7791 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 7792 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 7793 7794 If run with -vv rclone will print the location of the file cache. The 7795 files are stored in the user cache file area which is OS dependent but 7796 can be controlled with --cache-dir or setting the appropriate 7797 environment variable. 7798 7799 The cache has 4 different modes selected by --vfs-cache-mode. The higher 7800 the cache mode the more compatible rclone becomes at the cost of using 7801 disk space. 7802 7803 Note that files are written back to the remote only when they are closed 7804 and if they haven't been accessed for --vfs-write-back seconds. If 7805 rclone is quit or dies with files that haven't been uploaded, these will 7806 be uploaded next time rclone is run with the same flags. 7807 7808 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 7809 cache may exceed these quotas for two reasons. Firstly because it is 7810 only checked every --vfs-cache-poll-interval. Secondly because open 7811 files cannot be evicted from the cache. When --vfs-cache-max-size or 7812 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 7813 least accessed files from the cache first. rclone will start with files 7814 that haven't been accessed for the longest. This cache flushing strategy 7815 is efficient and more relevant files are likely to remain cached. 7816 7817 The --vfs-cache-max-age will evict files from the cache after the set 7818 time since last access has passed. The default value of 1 hour will 7819 start evicting files from cache that haven't been accessed for 1 hour. 7820 When a cached file is accessed the 1 hour timer is reset to 0 and will 7821 wait for 1 more hour before evicting. Specify the time with standard 7822 notation, s, m, h, d, w . 7823 7824 You should not run two copies of rclone using the same VFS cache with 7825 the same or overlapping remotes if using --vfs-cache-mode > off. This 7826 can potentially cause data corruption if you do. You can work around 7827 this by giving each rclone its own cache hierarchy with --cache-dir. You 7828 don't need to worry about this if the remotes in use don't overlap. 7829 7830 --vfs-cache-mode off 7831 7832 In this mode (the default) the cache will read directly from the remote 7833 and write directly to the remote without caching anything on disk. 7834 7835 This will mean some operations are not possible 7836 7837 - Files can't be opened for both read AND write 7838 - Files opened for write can't be seeked 7839 - Existing files opened for write must have O_TRUNC set 7840 - Files open for read with O_TRUNC will be opened write only 7841 - Files open for write only will behave as if O_TRUNC was supplied 7842 - Open modes O_APPEND, O_TRUNC are ignored 7843 - If an upload fails it can't be retried 7844 7845 --vfs-cache-mode minimal 7846 7847 This is very similar to "off" except that files opened for read AND 7848 write will be buffered to disk. This means that files opened for write 7849 will be a lot more compatible, but uses the minimal disk space. 7850 7851 These operations are not possible 7852 7853 - Files opened for write only can't be seeked 7854 - Existing files opened for write must have O_TRUNC set 7855 - Files opened for write only will ignore O_APPEND, O_TRUNC 7856 - If an upload fails it can't be retried 7857 7858 --vfs-cache-mode writes 7859 7860 In this mode files opened for read only are still read directly from the 7861 remote, write only and read/write files are buffered to disk first. 7862 7863 This mode should support all normal file system operations. 7864 7865 If an upload fails it will be retried at exponentially increasing 7866 intervals up to 1 minute. 7867 7868 --vfs-cache-mode full 7869 7870 In this mode all reads and writes are buffered to and from disk. When 7871 data is read from the remote this is buffered to disk as well. 7872 7873 In this mode the files in the cache will be sparse files and rclone will 7874 keep track of which bits of the files it has downloaded. 7875 7876 So if an application only reads the starts of each file, then rclone 7877 will only buffer the start of the file. These files will appear to be 7878 their full size in the cache, but they will be sparse files with only 7879 the data that has been downloaded present in them. 7880 7881 This mode should support all normal file system operations and is 7882 otherwise identical to --vfs-cache-mode writes. 7883 7884 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 7885 bytes ahead. The --buffer-size is buffered in memory whereas the 7886 --vfs-read-ahead is buffered on disk. 7887 7888 When using this mode it is recommended that --buffer-size is not set too 7889 large and --vfs-read-ahead is set large if required. 7890 7891 IMPORTANT not all file systems support sparse files. In particular 7892 FAT/exFAT do not. Rclone will perform very badly if the cache directory 7893 is on a filesystem which doesn't support sparse files and it will log an 7894 ERROR message if one is detected. 7895 7896 Fingerprinting 7897 7898 Various parts of the VFS use fingerprinting to see if a local file copy 7899 has changed relative to a remote file. Fingerprints are made from: 7900 7901 - size 7902 - modification time 7903 - hash 7904 7905 where available on an object. 7906 7907 On some backends some of these attributes are slow to read (they take an 7908 extra API call per object, or extra work per object). 7909 7910 For example hash is slow with the local and sftp backends as they have 7911 to read the entire file and hash it, and modtime is slow with the s3, 7912 swift, ftp and qinqstor backends because they need to do an extra API 7913 call to fetch it. 7914 7915 If you use the --vfs-fast-fingerprint flag then rclone will not include 7916 the slow operations in the fingerprint. This makes the fingerprinting 7917 less accurate but much faster and will improve the opening time of 7918 cached files. 7919 7920 If you are running a vfs cache over local, s3 or swift backends then 7921 using this flag is recommended. 7922 7923 Note that if you change the value of this flag, the fingerprints of the 7924 files in the cache may be invalidated and the files will need to be 7925 downloaded again. 7926 7927 VFS Chunked Reading 7928 7929 When rclone reads files from a remote it reads them in chunks. This 7930 means that rather than requesting the whole file rclone reads the chunk 7931 specified. This can reduce the used download quota for some remotes by 7932 requesting only chunks from the remote that are actually read, at the 7933 cost of an increased number of requests. 7934 7935 These flags control the chunking: 7936 7937 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 7938 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 7939 7940 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 7941 then double the size for each read. When --vfs-read-chunk-size-limit is 7942 specified, and greater than --vfs-read-chunk-size, the chunk size for 7943 each open file will get doubled only until the specified value is 7944 reached. If the value is "off", which is the default, the limit is 7945 disabled and the chunk size will grow indefinitely. 7946 7947 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 7948 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 7949 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 7950 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 7951 1200M-1700M and so on. 7952 7953 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 7954 7955 VFS Performance 7956 7957 These flags may be used to enable/disable features of the VFS for 7958 performance or other reasons. See also the chunked reading feature. 7959 7960 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 7961 use --use-server-modtime for a slightly different effect) as each read 7962 of the modification time takes a transaction. 7963 7964 --no-checksum Don't compare checksums on up/download. 7965 --no-modtime Don't read/write the modification time (can speed things up). 7966 --no-seek Don't allow seeking in files. 7967 --read-only Only allow read-only access. 7968 7969 Sometimes rclone is delivered reads or writes out of order. Rather than 7970 seeking rclone will wait a short time for the in sequence read or write 7971 to come in. These flags only come into effect when not using an on disk 7972 cache file. 7973 7974 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 7975 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 7976 7977 When using VFS write caching (--vfs-cache-mode with value writes or 7978 full), the global flag --transfers can be set to adjust the number of 7979 parallel uploads of modified files from the cache (the related global 7980 flag --checkers has no effect on the VFS). 7981 7982 --transfers int Number of file transfers to run in parallel (default 4) 7983 7984 VFS Case Sensitivity 7985 7986 Linux file systems are case-sensitive: two files can differ only by 7987 case, and the exact case must be used when opening a file. 7988 7989 File systems in modern Windows are case-insensitive but case-preserving: 7990 although existing files can be opened using any case, the exact case 7991 used to create the file is preserved and available for programs to 7992 query. It is not allowed for two files in the same directory to differ 7993 only by case. 7994 7995 Usually file systems on macOS are case-insensitive. It is possible to 7996 make macOS file systems case-sensitive but that is not the default. 7997 7998 The --vfs-case-insensitive VFS flag controls how rclone handles these 7999 two cases. If its value is "false", rclone passes file names to the 8000 remote as-is. If the flag is "true" (or appears without a value on the 8001 command line), rclone may perform a "fixup" as explained below. 8002 8003 The user may specify a file name to open/delete/rename/etc with a case 8004 different than what is stored on the remote. If an argument refers to an 8005 existing file with exactly the same name, then the case of the existing 8006 file on the disk will be used. However, if a file name with exactly the 8007 same name is not found but a name differing only by case exists, rclone 8008 will transparently fixup the name. This fixup happens only when an 8009 existing file is requested. Case sensitivity of file names created anew 8010 by rclone is controlled by the underlying remote. 8011 8012 Note that case sensitivity of the operating system running rclone (the 8013 target) may differ from case sensitivity of a file system presented by 8014 rclone (the source). The flag controls whether "fixup" is performed to 8015 satisfy the target. 8016 8017 If the flag is not provided on the command line, then its default value 8018 depends on the operating system where rclone runs: "true" on Windows and 8019 macOS, "false" otherwise. If the flag is provided without a value, then 8020 it is "true". 8021 8022 The --no-unicode-normalization flag controls whether a similar "fixup" 8023 is performed for filenames that differ but are canonically equivalent 8024 with respect to unicode. Unicode normalization can be particularly 8025 helpful for users of macOS, which prefers form NFD instead of the NFC 8026 used by most other platforms. It is therefore highly recommended to keep 8027 the default of false on macOS, to avoid encoding compatibility issues. 8028 8029 In the (probably unlikely) event that a directory has multiple duplicate 8030 filenames after applying case and unicode normalization, the 8031 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 8032 with a performance tradeoff, as rclone will have to scan the entire 8033 directory for duplicates when listing a directory. For this reason, it 8034 is recommended to leave this disabled if not needed. However, macOS 8035 users may wish to consider using it, as otherwise, if a remote directory 8036 contains both NFC and NFD versions of the same filename, an odd 8037 situation will occur: both versions of the file will be visible in the 8038 mount, and both will appear to be editable, however, editing either 8039 version will actually result in only the NFD version getting edited 8040 under the hood. --vfs-block- norm-dupes prevents this confusion by 8041 detecting this scenario, hiding the duplicates, and logging an error, 8042 similar to how this is handled in rclone sync. 8043 8044 VFS Disk Options 8045 8046 This flag allows you to manually set the statistics about the filing 8047 system. It can be useful when those statistics cannot be read correctly 8048 automatically. 8049 8050 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 8051 8052 Alternate report of used bytes 8053 8054 Some backends, most notably S3, do not report the amount of bytes used. 8055 If you need this information to be available when running df on the 8056 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 8057 flag set, instead of relying on the backend to report this information, 8058 rclone will scan the whole remote similar to rclone size and compute the 8059 total used space itself. 8060 8061 WARNING. Contrary to rclone size, this flag ignores filters so that the 8062 result is accurate. However, this is very inefficient and may cost lots 8063 of API calls resulting in extra charges. Use it as a last resort and 8064 only with caching. 8065 8066 rclone serve docker [flags] 8067 8068 Options 8069 8070 --allow-non-empty Allow mounting over a non-empty directory (not supported on Windows) 8071 --allow-other Allow access to other users (not supported on Windows) 8072 --allow-root Allow access to root user (not supported on Windows) 8073 --async-read Use asynchronous reads (not supported on Windows) (default true) 8074 --attr-timeout Duration Time for which file/directory attributes are cached (default 1s) 8075 --base-dir string Base directory for volumes (default "/var/lib/docker-volumes/rclone") 8076 --daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows) 8077 --daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s) 8078 --daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s) 8079 --debug-fuse Debug the FUSE internals - needs -v 8080 --default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows) 8081 --devname string Set the device name - default is remote:path 8082 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 8083 --dir-perms FileMode Directory permissions (default 0777) 8084 --file-perms FileMode File permissions (default 0666) 8085 --forget-state Skip restoring previous state 8086 --fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required) 8087 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 8088 -h, --help help for docker 8089 --max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki) 8090 --mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset) 8091 --network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only) 8092 --no-checksum Don't compare checksums on up/download 8093 --no-modtime Don't read/write the modification time (can speed things up) 8094 --no-seek Don't allow seeking in files 8095 --no-spec Do not write spec file 8096 --noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true) 8097 --noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only) 8098 -o, --option stringArray Option for libfuse/WinFsp (repeat if required) 8099 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 8100 --read-only Only allow read-only access 8101 --socket-addr string Address <host:port> or absolute path (default: /run/docker/plugins/rclone.sock) 8102 --socket-gid int GID for unix socket (default: current process GID) (default 1000) 8103 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 8104 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 8105 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 8106 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 8107 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 8108 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 8109 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 8110 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 8111 --vfs-case-insensitive If a file name not found, find a case insensitive match 8112 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 8113 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 8114 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 8115 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 8116 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 8117 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 8118 --vfs-refresh Refreshes the directory cache recursively in the background on start 8119 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 8120 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 8121 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 8122 --volname string Set the volume name (supported on Windows and OSX only) 8123 --write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows) 8124 8125 Filter Options 8126 8127 Flags for filtering directory listings. 8128 8129 --delete-excluded Delete files on dest excluded from sync 8130 --exclude stringArray Exclude files matching pattern 8131 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 8132 --exclude-if-present stringArray Exclude directories if filename is present 8133 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 8134 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 8135 -f, --filter stringArray Add a file filtering rule 8136 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 8137 --ignore-case Ignore case in filters (case insensitive) 8138 --include stringArray Include files matching pattern 8139 --include-from stringArray Read file include patterns from file (use - to read from stdin) 8140 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8141 --max-depth int If set limits the recursion depth to this (default -1) 8142 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 8143 --metadata-exclude stringArray Exclude metadatas matching pattern 8144 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 8145 --metadata-filter stringArray Add a metadata filtering rule 8146 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 8147 --metadata-include stringArray Include metadatas matching pattern 8148 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 8149 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8150 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 8151 8152 See the global flags page for global options not listed here. 8153 8154 SEE ALSO 8155 8156 - rclone serve - Serve a remote over a protocol. 8157 8158 rclone serve ftp 8159 8160 Serve remote:path over FTP. 8161 8162 Synopsis 8163 8164 Run a basic FTP server to serve a remote over FTP protocol. This can be 8165 viewed with a FTP client or you can make a remote of type FTP to read 8166 and write it. 8167 8168 Server options 8169 8170 Use --addr to specify which IP address and port the server should listen 8171 on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By 8172 default it only listens on localhost. You can use port :0 to let the OS 8173 choose an available port. 8174 8175 If you set --addr to listen on a public or LAN accessible IP address 8176 then using Authentication is advised - see the next section for info. 8177 8178 Authentication 8179 8180 By default this will serve files without needing a login. 8181 8182 You can set a single username and password with the --user and --pass 8183 flags. ## VFS - Virtual File System 8184 8185 This command uses the VFS layer. This adapts the cloud storage objects 8186 that rclone uses into something which looks much more like a disk filing 8187 system. 8188 8189 Cloud storage objects have lots of properties which aren't like disk 8190 files - you can't extend them or write to the middle of them, so the VFS 8191 layer has to deal with that. Because there is no one right way of doing 8192 this there are various options explained below. 8193 8194 The VFS layer also implements a directory cache - this caches info about 8195 files and directories (but not the data) in memory. 8196 8197 VFS Directory Cache 8198 8199 Using the --dir-cache-time flag, you can control how long a directory 8200 should be considered up to date and not refreshed from the backend. 8201 Changes made through the VFS will appear immediately or invalidate the 8202 cache. 8203 8204 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 8205 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 8206 8207 However, changes made directly on the cloud storage by the web interface 8208 or a different copy of rclone will only be picked up once the directory 8209 cache expires if the backend configured does not support polling for 8210 changes. If the backend supports polling, changes will be picked up 8211 within the polling interval. 8212 8213 You can send a SIGHUP signal to rclone for it to flush all directory 8214 caches, regardless of how old they are. Assuming only one rclone 8215 instance is running, you can reset the cache like this: 8216 8217 kill -SIGHUP $(pidof rclone) 8218 8219 If you configure rclone with a remote control then you can use rclone rc 8220 to flush the whole directory cache: 8221 8222 rclone rc vfs/forget 8223 8224 Or individual files or directories: 8225 8226 rclone rc vfs/forget file=path/to/file dir=path/to/dir 8227 8228 VFS File Buffering 8229 8230 The --buffer-size flag determines the amount of memory, that will be 8231 used to buffer data in advance. 8232 8233 Each open file will try to keep the specified amount of data in memory 8234 at all times. The buffered data is bound to one open file and won't be 8235 shared. 8236 8237 This flag is a upper limit for the used memory per open file. The buffer 8238 will only use memory for data that is downloaded but not not yet read. 8239 If the buffer is empty, only a small amount of memory will be used. 8240 8241 The maximum memory used by rclone for buffering can be up to 8242 --buffer-size * open files. 8243 8244 VFS File Caching 8245 8246 These flags control the VFS file caching options. File caching is 8247 necessary to make the VFS layer appear compatible with a normal file 8248 system. It can be disabled at the cost of some compatibility. 8249 8250 For example you'll need to enable VFS caching if you want to read and 8251 write simultaneously to a file. See below for more details. 8252 8253 Note that the VFS cache is separate from the cache backend and you may 8254 find that you need one or the other or both. 8255 8256 --cache-dir string Directory rclone will use for caching. 8257 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 8258 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 8259 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 8260 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 8261 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 8262 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 8263 8264 If run with -vv rclone will print the location of the file cache. The 8265 files are stored in the user cache file area which is OS dependent but 8266 can be controlled with --cache-dir or setting the appropriate 8267 environment variable. 8268 8269 The cache has 4 different modes selected by --vfs-cache-mode. The higher 8270 the cache mode the more compatible rclone becomes at the cost of using 8271 disk space. 8272 8273 Note that files are written back to the remote only when they are closed 8274 and if they haven't been accessed for --vfs-write-back seconds. If 8275 rclone is quit or dies with files that haven't been uploaded, these will 8276 be uploaded next time rclone is run with the same flags. 8277 8278 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 8279 cache may exceed these quotas for two reasons. Firstly because it is 8280 only checked every --vfs-cache-poll-interval. Secondly because open 8281 files cannot be evicted from the cache. When --vfs-cache-max-size or 8282 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 8283 least accessed files from the cache first. rclone will start with files 8284 that haven't been accessed for the longest. This cache flushing strategy 8285 is efficient and more relevant files are likely to remain cached. 8286 8287 The --vfs-cache-max-age will evict files from the cache after the set 8288 time since last access has passed. The default value of 1 hour will 8289 start evicting files from cache that haven't been accessed for 1 hour. 8290 When a cached file is accessed the 1 hour timer is reset to 0 and will 8291 wait for 1 more hour before evicting. Specify the time with standard 8292 notation, s, m, h, d, w . 8293 8294 You should not run two copies of rclone using the same VFS cache with 8295 the same or overlapping remotes if using --vfs-cache-mode > off. This 8296 can potentially cause data corruption if you do. You can work around 8297 this by giving each rclone its own cache hierarchy with --cache-dir. You 8298 don't need to worry about this if the remotes in use don't overlap. 8299 8300 --vfs-cache-mode off 8301 8302 In this mode (the default) the cache will read directly from the remote 8303 and write directly to the remote without caching anything on disk. 8304 8305 This will mean some operations are not possible 8306 8307 - Files can't be opened for both read AND write 8308 - Files opened for write can't be seeked 8309 - Existing files opened for write must have O_TRUNC set 8310 - Files open for read with O_TRUNC will be opened write only 8311 - Files open for write only will behave as if O_TRUNC was supplied 8312 - Open modes O_APPEND, O_TRUNC are ignored 8313 - If an upload fails it can't be retried 8314 8315 --vfs-cache-mode minimal 8316 8317 This is very similar to "off" except that files opened for read AND 8318 write will be buffered to disk. This means that files opened for write 8319 will be a lot more compatible, but uses the minimal disk space. 8320 8321 These operations are not possible 8322 8323 - Files opened for write only can't be seeked 8324 - Existing files opened for write must have O_TRUNC set 8325 - Files opened for write only will ignore O_APPEND, O_TRUNC 8326 - If an upload fails it can't be retried 8327 8328 --vfs-cache-mode writes 8329 8330 In this mode files opened for read only are still read directly from the 8331 remote, write only and read/write files are buffered to disk first. 8332 8333 This mode should support all normal file system operations. 8334 8335 If an upload fails it will be retried at exponentially increasing 8336 intervals up to 1 minute. 8337 8338 --vfs-cache-mode full 8339 8340 In this mode all reads and writes are buffered to and from disk. When 8341 data is read from the remote this is buffered to disk as well. 8342 8343 In this mode the files in the cache will be sparse files and rclone will 8344 keep track of which bits of the files it has downloaded. 8345 8346 So if an application only reads the starts of each file, then rclone 8347 will only buffer the start of the file. These files will appear to be 8348 their full size in the cache, but they will be sparse files with only 8349 the data that has been downloaded present in them. 8350 8351 This mode should support all normal file system operations and is 8352 otherwise identical to --vfs-cache-mode writes. 8353 8354 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 8355 bytes ahead. The --buffer-size is buffered in memory whereas the 8356 --vfs-read-ahead is buffered on disk. 8357 8358 When using this mode it is recommended that --buffer-size is not set too 8359 large and --vfs-read-ahead is set large if required. 8360 8361 IMPORTANT not all file systems support sparse files. In particular 8362 FAT/exFAT do not. Rclone will perform very badly if the cache directory 8363 is on a filesystem which doesn't support sparse files and it will log an 8364 ERROR message if one is detected. 8365 8366 Fingerprinting 8367 8368 Various parts of the VFS use fingerprinting to see if a local file copy 8369 has changed relative to a remote file. Fingerprints are made from: 8370 8371 - size 8372 - modification time 8373 - hash 8374 8375 where available on an object. 8376 8377 On some backends some of these attributes are slow to read (they take an 8378 extra API call per object, or extra work per object). 8379 8380 For example hash is slow with the local and sftp backends as they have 8381 to read the entire file and hash it, and modtime is slow with the s3, 8382 swift, ftp and qinqstor backends because they need to do an extra API 8383 call to fetch it. 8384 8385 If you use the --vfs-fast-fingerprint flag then rclone will not include 8386 the slow operations in the fingerprint. This makes the fingerprinting 8387 less accurate but much faster and will improve the opening time of 8388 cached files. 8389 8390 If you are running a vfs cache over local, s3 or swift backends then 8391 using this flag is recommended. 8392 8393 Note that if you change the value of this flag, the fingerprints of the 8394 files in the cache may be invalidated and the files will need to be 8395 downloaded again. 8396 8397 VFS Chunked Reading 8398 8399 When rclone reads files from a remote it reads them in chunks. This 8400 means that rather than requesting the whole file rclone reads the chunk 8401 specified. This can reduce the used download quota for some remotes by 8402 requesting only chunks from the remote that are actually read, at the 8403 cost of an increased number of requests. 8404 8405 These flags control the chunking: 8406 8407 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 8408 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 8409 8410 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 8411 then double the size for each read. When --vfs-read-chunk-size-limit is 8412 specified, and greater than --vfs-read-chunk-size, the chunk size for 8413 each open file will get doubled only until the specified value is 8414 reached. If the value is "off", which is the default, the limit is 8415 disabled and the chunk size will grow indefinitely. 8416 8417 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 8418 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 8419 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 8420 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 8421 1200M-1700M and so on. 8422 8423 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 8424 8425 VFS Performance 8426 8427 These flags may be used to enable/disable features of the VFS for 8428 performance or other reasons. See also the chunked reading feature. 8429 8430 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 8431 use --use-server-modtime for a slightly different effect) as each read 8432 of the modification time takes a transaction. 8433 8434 --no-checksum Don't compare checksums on up/download. 8435 --no-modtime Don't read/write the modification time (can speed things up). 8436 --no-seek Don't allow seeking in files. 8437 --read-only Only allow read-only access. 8438 8439 Sometimes rclone is delivered reads or writes out of order. Rather than 8440 seeking rclone will wait a short time for the in sequence read or write 8441 to come in. These flags only come into effect when not using an on disk 8442 cache file. 8443 8444 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 8445 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 8446 8447 When using VFS write caching (--vfs-cache-mode with value writes or 8448 full), the global flag --transfers can be set to adjust the number of 8449 parallel uploads of modified files from the cache (the related global 8450 flag --checkers has no effect on the VFS). 8451 8452 --transfers int Number of file transfers to run in parallel (default 4) 8453 8454 VFS Case Sensitivity 8455 8456 Linux file systems are case-sensitive: two files can differ only by 8457 case, and the exact case must be used when opening a file. 8458 8459 File systems in modern Windows are case-insensitive but case-preserving: 8460 although existing files can be opened using any case, the exact case 8461 used to create the file is preserved and available for programs to 8462 query. It is not allowed for two files in the same directory to differ 8463 only by case. 8464 8465 Usually file systems on macOS are case-insensitive. It is possible to 8466 make macOS file systems case-sensitive but that is not the default. 8467 8468 The --vfs-case-insensitive VFS flag controls how rclone handles these 8469 two cases. If its value is "false", rclone passes file names to the 8470 remote as-is. If the flag is "true" (or appears without a value on the 8471 command line), rclone may perform a "fixup" as explained below. 8472 8473 The user may specify a file name to open/delete/rename/etc with a case 8474 different than what is stored on the remote. If an argument refers to an 8475 existing file with exactly the same name, then the case of the existing 8476 file on the disk will be used. However, if a file name with exactly the 8477 same name is not found but a name differing only by case exists, rclone 8478 will transparently fixup the name. This fixup happens only when an 8479 existing file is requested. Case sensitivity of file names created anew 8480 by rclone is controlled by the underlying remote. 8481 8482 Note that case sensitivity of the operating system running rclone (the 8483 target) may differ from case sensitivity of a file system presented by 8484 rclone (the source). The flag controls whether "fixup" is performed to 8485 satisfy the target. 8486 8487 If the flag is not provided on the command line, then its default value 8488 depends on the operating system where rclone runs: "true" on Windows and 8489 macOS, "false" otherwise. If the flag is provided without a value, then 8490 it is "true". 8491 8492 The --no-unicode-normalization flag controls whether a similar "fixup" 8493 is performed for filenames that differ but are canonically equivalent 8494 with respect to unicode. Unicode normalization can be particularly 8495 helpful for users of macOS, which prefers form NFD instead of the NFC 8496 used by most other platforms. It is therefore highly recommended to keep 8497 the default of false on macOS, to avoid encoding compatibility issues. 8498 8499 In the (probably unlikely) event that a directory has multiple duplicate 8500 filenames after applying case and unicode normalization, the 8501 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 8502 with a performance tradeoff, as rclone will have to scan the entire 8503 directory for duplicates when listing a directory. For this reason, it 8504 is recommended to leave this disabled if not needed. However, macOS 8505 users may wish to consider using it, as otherwise, if a remote directory 8506 contains both NFC and NFD versions of the same filename, an odd 8507 situation will occur: both versions of the file will be visible in the 8508 mount, and both will appear to be editable, however, editing either 8509 version will actually result in only the NFD version getting edited 8510 under the hood. --vfs-block- norm-dupes prevents this confusion by 8511 detecting this scenario, hiding the duplicates, and logging an error, 8512 similar to how this is handled in rclone sync. 8513 8514 VFS Disk Options 8515 8516 This flag allows you to manually set the statistics about the filing 8517 system. It can be useful when those statistics cannot be read correctly 8518 automatically. 8519 8520 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 8521 8522 Alternate report of used bytes 8523 8524 Some backends, most notably S3, do not report the amount of bytes used. 8525 If you need this information to be available when running df on the 8526 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 8527 flag set, instead of relying on the backend to report this information, 8528 rclone will scan the whole remote similar to rclone size and compute the 8529 total used space itself. 8530 8531 WARNING. Contrary to rclone size, this flag ignores filters so that the 8532 result is accurate. However, this is very inefficient and may cost lots 8533 of API calls resulting in extra charges. Use it as a last resort and 8534 only with caching. 8535 8536 Auth Proxy 8537 8538 If you supply the parameter --auth-proxy /path/to/program then rclone 8539 will use that program to generate backends on the fly which then are 8540 used to authenticate incoming requests. This uses a simple JSON based 8541 protocol with input on STDIN and output on STDOUT. 8542 8543 PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, 8544 if --auth-proxy is set the authorized keys option will be ignored. 8545 8546 There is an example program bin/test_proxy.py in the rclone source code. 8547 8548 The program's job is to take a user and pass on the input and turn those 8549 into the config for a backend on STDOUT in JSON format. This config will 8550 have any default parameters for the backend added, but it won't use 8551 configuration from environment variables or command line options - it is 8552 the job of the proxy program to make a complete config. 8553 8554 This config generated must have this extra parameter - _root - root to 8555 use for the backend 8556 8557 And it may have this parameter - _obscure - comma separated strings for 8558 parameters to obscure 8559 8560 If password authentication was used by the client, input to the proxy 8561 process (on STDIN) would look similar to this: 8562 8563 { 8564 "user": "me", 8565 "pass": "mypassword" 8566 } 8567 8568 If public-key authentication was used by the client, input to the proxy 8569 process (on STDIN) would look similar to this: 8570 8571 { 8572 "user": "me", 8573 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 8574 } 8575 8576 And as an example return this on STDOUT 8577 8578 { 8579 "type": "sftp", 8580 "_root": "", 8581 "_obscure": "pass", 8582 "user": "me", 8583 "pass": "mypassword", 8584 "host": "sftp.example.com" 8585 } 8586 8587 This would mean that an SFTP backend would be created on the fly for the 8588 user and pass/public_key returned in the output to the host given. Note 8589 that since _obscure is set to pass, rclone will obscure the pass 8590 parameter before creating the backend (which is required for sftp 8591 backends). 8592 8593 The program can manipulate the supplied user in any way, for example to 8594 make proxy to many different sftp backends, you could make the user be 8595 user@example.com and then set the host to example.com in the output and 8596 the user to user. For security you'd probably want to restrict the host 8597 to a limited list. 8598 8599 Note that an internal cache is keyed on user so only use that for 8600 configuration, don't use pass or public_key. This also means that if a 8601 user's password or public-key is changed the cache will need to expire 8602 (which takes 5 mins) before it takes effect. 8603 8604 This can be used to build general purpose proxies to any kind of backend 8605 that rclone supports. 8606 8607 rclone serve ftp remote:path [flags] 8608 8609 Options 8610 8611 --addr string IPaddress:Port or :Port to bind server to (default "localhost:2121") 8612 --auth-proxy string A program to use to create the backend from the auth 8613 --cert string TLS PEM key (concatenation of certificate and CA certificate) 8614 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 8615 --dir-perms FileMode Directory permissions (default 0777) 8616 --file-perms FileMode File permissions (default 0666) 8617 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 8618 -h, --help help for ftp 8619 --key string TLS PEM Private key 8620 --no-checksum Don't compare checksums on up/download 8621 --no-modtime Don't read/write the modification time (can speed things up) 8622 --no-seek Don't allow seeking in files 8623 --pass string Password for authentication (empty value allow every password) 8624 --passive-port string Passive port range to use (default "30000-32000") 8625 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 8626 --public-ip string Public IP address to advertise for passive connections 8627 --read-only Only allow read-only access 8628 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 8629 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 8630 --user string User name for authentication (default "anonymous") 8631 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 8632 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 8633 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 8634 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 8635 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 8636 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 8637 --vfs-case-insensitive If a file name not found, find a case insensitive match 8638 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 8639 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 8640 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 8641 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 8642 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 8643 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 8644 --vfs-refresh Refreshes the directory cache recursively in the background on start 8645 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 8646 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 8647 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 8648 8649 Filter Options 8650 8651 Flags for filtering directory listings. 8652 8653 --delete-excluded Delete files on dest excluded from sync 8654 --exclude stringArray Exclude files matching pattern 8655 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 8656 --exclude-if-present stringArray Exclude directories if filename is present 8657 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 8658 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 8659 -f, --filter stringArray Add a file filtering rule 8660 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 8661 --ignore-case Ignore case in filters (case insensitive) 8662 --include stringArray Include files matching pattern 8663 --include-from stringArray Read file include patterns from file (use - to read from stdin) 8664 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8665 --max-depth int If set limits the recursion depth to this (default -1) 8666 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 8667 --metadata-exclude stringArray Exclude metadatas matching pattern 8668 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 8669 --metadata-filter stringArray Add a metadata filtering rule 8670 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 8671 --metadata-include stringArray Include metadatas matching pattern 8672 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 8673 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 8674 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 8675 8676 See the global flags page for global options not listed here. 8677 8678 SEE ALSO 8679 8680 - rclone serve - Serve a remote over a protocol. 8681 8682 rclone serve http 8683 8684 Serve the remote over HTTP. 8685 8686 Synopsis 8687 8688 Run a basic web server to serve a remote over HTTP. This can be viewed 8689 in a web browser or you can make a remote of type http read from it. 8690 8691 You can use the filter flags (e.g. --include, --exclude) to control what 8692 is served. 8693 8694 The server will log errors. Use -v to see access logs. 8695 8696 --bwlimit will be respected for file transfers. Use --stats to control 8697 the stats printing. 8698 8699 Server options 8700 8701 Use --addr to specify which IP address and port the server should listen 8702 on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By 8703 default it only listens on localhost. You can use port :0 to let the OS 8704 choose an available port. 8705 8706 If you set --addr to listen on a public or LAN accessible IP address 8707 then using Authentication is advised - see the next section for info. 8708 8709 You can use a unix socket by setting the url to unix:///path/to/socket 8710 or just by using an absolute path name. Note that unix sockets bypass 8711 the authentication - this is expected to be done with file system 8712 permissions. 8713 8714 --addr may be repeated to listen on multiple IPs/ports/sockets. 8715 8716 --server-read-timeout and --server-write-timeout can be used to control 8717 the timeouts on the server. Note that this is the total time for a 8718 transfer. 8719 8720 --max-header-bytes controls the maximum number of bytes the server will 8721 accept in the HTTP header. 8722 8723 --baseurl controls the URL prefix that rclone serves from. By default 8724 rclone will serve from the root. If you used --baseurl "/rclone" then 8725 rclone would serve from a URL starting with "/rclone/". This is useful 8726 if you wish to proxy rclone serve. Rclone automatically inserts leading 8727 and trailing "/" on --baseurl, so --baseurl "rclone", 8728 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 8729 identically. 8730 8731 TLS (SSL) 8732 8733 By default this will serve over http. If you want you can serve over 8734 https. You will need to supply the --cert and --key flags. If you wish 8735 to do client side certificate validation then you will need to supply 8736 --client-ca also. 8737 8738 --cert should be a either a PEM encoded certificate or a concatenation 8739 of that with the CA certificate. --key should be the PEM encoded private 8740 key and --client-ca should be the PEM encoded client certificate 8741 authority certificate. 8742 8743 --min-tls-version is minimum TLS version that is acceptable. Valid 8744 values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 8745 8746 Template 8747 8748 --template allows a user to specify a custom markup template for HTTP 8749 and WebDAV serve functions. The server exports the following markup to 8750 be used within the template to server pages: 8751 8752 ----------------------------------------------------------------------- 8753 Parameter Description 8754 ----------------------------------- ----------------------------------- 8755 .Name The full path of a file/directory. 8756 8757 .Title Directory listing of .Name 8758 8759 .Sort The current sort used. This is 8760 changeable via ?sort= parameter 8761 8762 Sort Options: 8763 namedirfirst,name,size,time 8764 (default namedirfirst) 8765 8766 .Order The current ordering used. This is 8767 changeable via ?order= parameter 8768 8769 Order Options: asc,desc (default 8770 asc) 8771 8772 .Query Currently unused. 8773 8774 .Breadcrumb Allows for creating a relative 8775 navigation 8776 8777 -- .Link The relative to the root link of 8778 the Text. 8779 8780 -- .Text The Name of the directory. 8781 8782 .Entries Information about a specific 8783 file/directory. 8784 8785 -- .URL The 'url' of an entry. 8786 8787 -- .Leaf Currently same as 'URL' but 8788 intended to be 'just' the name. 8789 8790 -- .IsDir Boolean for if an entry is a 8791 directory or not. 8792 8793 -- .Size Size in Bytes of the entry. 8794 8795 -- .ModTime The UTC timestamp of an entry. 8796 ----------------------------------------------------------------------- 8797 8798 The server also makes the following functions available so that they can 8799 be used within the template. These functions help extend the options for 8800 dynamic rendering of HTML. They can be used to render HTML based on 8801 specific conditions. 8802 8803 ----------------------------------------------------------------------- 8804 Function Description 8805 ----------------------------------- ----------------------------------- 8806 afterEpoch Returns the time since the epoch 8807 for the given time. 8808 8809 contains Checks whether a given substring is 8810 present or not in a given string. 8811 8812 hasPrefix Checks whether the given string 8813 begins with the specified prefix. 8814 8815 hasSuffix Checks whether the given string end 8816 with the specified suffix. 8817 ----------------------------------------------------------------------- 8818 8819 Authentication 8820 8821 By default this will serve files without needing a login. 8822 8823 You can either use an htpasswd file which can take lots of users, or set 8824 a single username and password with the --user and --pass flags. 8825 8826 If no static users are configured by either of the above methods, and 8827 client certificates are required by the --client-ca flag passed to the 8828 server, the client certificate common name will be considered as the 8829 username. 8830 8831 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in 8832 standard apache format and supports MD5, SHA1 and BCrypt for basic 8833 authentication. Bcrypt is recommended. 8834 8835 To create an htpasswd file: 8836 8837 touch htpasswd 8838 htpasswd -B htpasswd user 8839 htpasswd -B htpasswd anotherUser 8840 8841 The password file can be updated while rclone is running. 8842 8843 Use --realm to set the authentication realm. 8844 8845 Use --salt to change the password hashing salt from the default. ## VFS 8846 - Virtual File System 8847 8848 This command uses the VFS layer. This adapts the cloud storage objects 8849 that rclone uses into something which looks much more like a disk filing 8850 system. 8851 8852 Cloud storage objects have lots of properties which aren't like disk 8853 files - you can't extend them or write to the middle of them, so the VFS 8854 layer has to deal with that. Because there is no one right way of doing 8855 this there are various options explained below. 8856 8857 The VFS layer also implements a directory cache - this caches info about 8858 files and directories (but not the data) in memory. 8859 8860 VFS Directory Cache 8861 8862 Using the --dir-cache-time flag, you can control how long a directory 8863 should be considered up to date and not refreshed from the backend. 8864 Changes made through the VFS will appear immediately or invalidate the 8865 cache. 8866 8867 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 8868 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 8869 8870 However, changes made directly on the cloud storage by the web interface 8871 or a different copy of rclone will only be picked up once the directory 8872 cache expires if the backend configured does not support polling for 8873 changes. If the backend supports polling, changes will be picked up 8874 within the polling interval. 8875 8876 You can send a SIGHUP signal to rclone for it to flush all directory 8877 caches, regardless of how old they are. Assuming only one rclone 8878 instance is running, you can reset the cache like this: 8879 8880 kill -SIGHUP $(pidof rclone) 8881 8882 If you configure rclone with a remote control then you can use rclone rc 8883 to flush the whole directory cache: 8884 8885 rclone rc vfs/forget 8886 8887 Or individual files or directories: 8888 8889 rclone rc vfs/forget file=path/to/file dir=path/to/dir 8890 8891 VFS File Buffering 8892 8893 The --buffer-size flag determines the amount of memory, that will be 8894 used to buffer data in advance. 8895 8896 Each open file will try to keep the specified amount of data in memory 8897 at all times. The buffered data is bound to one open file and won't be 8898 shared. 8899 8900 This flag is a upper limit for the used memory per open file. The buffer 8901 will only use memory for data that is downloaded but not not yet read. 8902 If the buffer is empty, only a small amount of memory will be used. 8903 8904 The maximum memory used by rclone for buffering can be up to 8905 --buffer-size * open files. 8906 8907 VFS File Caching 8908 8909 These flags control the VFS file caching options. File caching is 8910 necessary to make the VFS layer appear compatible with a normal file 8911 system. It can be disabled at the cost of some compatibility. 8912 8913 For example you'll need to enable VFS caching if you want to read and 8914 write simultaneously to a file. See below for more details. 8915 8916 Note that the VFS cache is separate from the cache backend and you may 8917 find that you need one or the other or both. 8918 8919 --cache-dir string Directory rclone will use for caching. 8920 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 8921 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 8922 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 8923 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 8924 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 8925 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 8926 8927 If run with -vv rclone will print the location of the file cache. The 8928 files are stored in the user cache file area which is OS dependent but 8929 can be controlled with --cache-dir or setting the appropriate 8930 environment variable. 8931 8932 The cache has 4 different modes selected by --vfs-cache-mode. The higher 8933 the cache mode the more compatible rclone becomes at the cost of using 8934 disk space. 8935 8936 Note that files are written back to the remote only when they are closed 8937 and if they haven't been accessed for --vfs-write-back seconds. If 8938 rclone is quit or dies with files that haven't been uploaded, these will 8939 be uploaded next time rclone is run with the same flags. 8940 8941 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 8942 cache may exceed these quotas for two reasons. Firstly because it is 8943 only checked every --vfs-cache-poll-interval. Secondly because open 8944 files cannot be evicted from the cache. When --vfs-cache-max-size or 8945 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 8946 least accessed files from the cache first. rclone will start with files 8947 that haven't been accessed for the longest. This cache flushing strategy 8948 is efficient and more relevant files are likely to remain cached. 8949 8950 The --vfs-cache-max-age will evict files from the cache after the set 8951 time since last access has passed. The default value of 1 hour will 8952 start evicting files from cache that haven't been accessed for 1 hour. 8953 When a cached file is accessed the 1 hour timer is reset to 0 and will 8954 wait for 1 more hour before evicting. Specify the time with standard 8955 notation, s, m, h, d, w . 8956 8957 You should not run two copies of rclone using the same VFS cache with 8958 the same or overlapping remotes if using --vfs-cache-mode > off. This 8959 can potentially cause data corruption if you do. You can work around 8960 this by giving each rclone its own cache hierarchy with --cache-dir. You 8961 don't need to worry about this if the remotes in use don't overlap. 8962 8963 --vfs-cache-mode off 8964 8965 In this mode (the default) the cache will read directly from the remote 8966 and write directly to the remote without caching anything on disk. 8967 8968 This will mean some operations are not possible 8969 8970 - Files can't be opened for both read AND write 8971 - Files opened for write can't be seeked 8972 - Existing files opened for write must have O_TRUNC set 8973 - Files open for read with O_TRUNC will be opened write only 8974 - Files open for write only will behave as if O_TRUNC was supplied 8975 - Open modes O_APPEND, O_TRUNC are ignored 8976 - If an upload fails it can't be retried 8977 8978 --vfs-cache-mode minimal 8979 8980 This is very similar to "off" except that files opened for read AND 8981 write will be buffered to disk. This means that files opened for write 8982 will be a lot more compatible, but uses the minimal disk space. 8983 8984 These operations are not possible 8985 8986 - Files opened for write only can't be seeked 8987 - Existing files opened for write must have O_TRUNC set 8988 - Files opened for write only will ignore O_APPEND, O_TRUNC 8989 - If an upload fails it can't be retried 8990 8991 --vfs-cache-mode writes 8992 8993 In this mode files opened for read only are still read directly from the 8994 remote, write only and read/write files are buffered to disk first. 8995 8996 This mode should support all normal file system operations. 8997 8998 If an upload fails it will be retried at exponentially increasing 8999 intervals up to 1 minute. 9000 9001 --vfs-cache-mode full 9002 9003 In this mode all reads and writes are buffered to and from disk. When 9004 data is read from the remote this is buffered to disk as well. 9005 9006 In this mode the files in the cache will be sparse files and rclone will 9007 keep track of which bits of the files it has downloaded. 9008 9009 So if an application only reads the starts of each file, then rclone 9010 will only buffer the start of the file. These files will appear to be 9011 their full size in the cache, but they will be sparse files with only 9012 the data that has been downloaded present in them. 9013 9014 This mode should support all normal file system operations and is 9015 otherwise identical to --vfs-cache-mode writes. 9016 9017 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 9018 bytes ahead. The --buffer-size is buffered in memory whereas the 9019 --vfs-read-ahead is buffered on disk. 9020 9021 When using this mode it is recommended that --buffer-size is not set too 9022 large and --vfs-read-ahead is set large if required. 9023 9024 IMPORTANT not all file systems support sparse files. In particular 9025 FAT/exFAT do not. Rclone will perform very badly if the cache directory 9026 is on a filesystem which doesn't support sparse files and it will log an 9027 ERROR message if one is detected. 9028 9029 Fingerprinting 9030 9031 Various parts of the VFS use fingerprinting to see if a local file copy 9032 has changed relative to a remote file. Fingerprints are made from: 9033 9034 - size 9035 - modification time 9036 - hash 9037 9038 where available on an object. 9039 9040 On some backends some of these attributes are slow to read (they take an 9041 extra API call per object, or extra work per object). 9042 9043 For example hash is slow with the local and sftp backends as they have 9044 to read the entire file and hash it, and modtime is slow with the s3, 9045 swift, ftp and qinqstor backends because they need to do an extra API 9046 call to fetch it. 9047 9048 If you use the --vfs-fast-fingerprint flag then rclone will not include 9049 the slow operations in the fingerprint. This makes the fingerprinting 9050 less accurate but much faster and will improve the opening time of 9051 cached files. 9052 9053 If you are running a vfs cache over local, s3 or swift backends then 9054 using this flag is recommended. 9055 9056 Note that if you change the value of this flag, the fingerprints of the 9057 files in the cache may be invalidated and the files will need to be 9058 downloaded again. 9059 9060 VFS Chunked Reading 9061 9062 When rclone reads files from a remote it reads them in chunks. This 9063 means that rather than requesting the whole file rclone reads the chunk 9064 specified. This can reduce the used download quota for some remotes by 9065 requesting only chunks from the remote that are actually read, at the 9066 cost of an increased number of requests. 9067 9068 These flags control the chunking: 9069 9070 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 9071 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 9072 9073 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 9074 then double the size for each read. When --vfs-read-chunk-size-limit is 9075 specified, and greater than --vfs-read-chunk-size, the chunk size for 9076 each open file will get doubled only until the specified value is 9077 reached. If the value is "off", which is the default, the limit is 9078 disabled and the chunk size will grow indefinitely. 9079 9080 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 9081 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 9082 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 9083 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 9084 1200M-1700M and so on. 9085 9086 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 9087 9088 VFS Performance 9089 9090 These flags may be used to enable/disable features of the VFS for 9091 performance or other reasons. See also the chunked reading feature. 9092 9093 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 9094 use --use-server-modtime for a slightly different effect) as each read 9095 of the modification time takes a transaction. 9096 9097 --no-checksum Don't compare checksums on up/download. 9098 --no-modtime Don't read/write the modification time (can speed things up). 9099 --no-seek Don't allow seeking in files. 9100 --read-only Only allow read-only access. 9101 9102 Sometimes rclone is delivered reads or writes out of order. Rather than 9103 seeking rclone will wait a short time for the in sequence read or write 9104 to come in. These flags only come into effect when not using an on disk 9105 cache file. 9106 9107 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 9108 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 9109 9110 When using VFS write caching (--vfs-cache-mode with value writes or 9111 full), the global flag --transfers can be set to adjust the number of 9112 parallel uploads of modified files from the cache (the related global 9113 flag --checkers has no effect on the VFS). 9114 9115 --transfers int Number of file transfers to run in parallel (default 4) 9116 9117 VFS Case Sensitivity 9118 9119 Linux file systems are case-sensitive: two files can differ only by 9120 case, and the exact case must be used when opening a file. 9121 9122 File systems in modern Windows are case-insensitive but case-preserving: 9123 although existing files can be opened using any case, the exact case 9124 used to create the file is preserved and available for programs to 9125 query. It is not allowed for two files in the same directory to differ 9126 only by case. 9127 9128 Usually file systems on macOS are case-insensitive. It is possible to 9129 make macOS file systems case-sensitive but that is not the default. 9130 9131 The --vfs-case-insensitive VFS flag controls how rclone handles these 9132 two cases. If its value is "false", rclone passes file names to the 9133 remote as-is. If the flag is "true" (or appears without a value on the 9134 command line), rclone may perform a "fixup" as explained below. 9135 9136 The user may specify a file name to open/delete/rename/etc with a case 9137 different than what is stored on the remote. If an argument refers to an 9138 existing file with exactly the same name, then the case of the existing 9139 file on the disk will be used. However, if a file name with exactly the 9140 same name is not found but a name differing only by case exists, rclone 9141 will transparently fixup the name. This fixup happens only when an 9142 existing file is requested. Case sensitivity of file names created anew 9143 by rclone is controlled by the underlying remote. 9144 9145 Note that case sensitivity of the operating system running rclone (the 9146 target) may differ from case sensitivity of a file system presented by 9147 rclone (the source). The flag controls whether "fixup" is performed to 9148 satisfy the target. 9149 9150 If the flag is not provided on the command line, then its default value 9151 depends on the operating system where rclone runs: "true" on Windows and 9152 macOS, "false" otherwise. If the flag is provided without a value, then 9153 it is "true". 9154 9155 The --no-unicode-normalization flag controls whether a similar "fixup" 9156 is performed for filenames that differ but are canonically equivalent 9157 with respect to unicode. Unicode normalization can be particularly 9158 helpful for users of macOS, which prefers form NFD instead of the NFC 9159 used by most other platforms. It is therefore highly recommended to keep 9160 the default of false on macOS, to avoid encoding compatibility issues. 9161 9162 In the (probably unlikely) event that a directory has multiple duplicate 9163 filenames after applying case and unicode normalization, the 9164 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 9165 with a performance tradeoff, as rclone will have to scan the entire 9166 directory for duplicates when listing a directory. For this reason, it 9167 is recommended to leave this disabled if not needed. However, macOS 9168 users may wish to consider using it, as otherwise, if a remote directory 9169 contains both NFC and NFD versions of the same filename, an odd 9170 situation will occur: both versions of the file will be visible in the 9171 mount, and both will appear to be editable, however, editing either 9172 version will actually result in only the NFD version getting edited 9173 under the hood. --vfs-block- norm-dupes prevents this confusion by 9174 detecting this scenario, hiding the duplicates, and logging an error, 9175 similar to how this is handled in rclone sync. 9176 9177 VFS Disk Options 9178 9179 This flag allows you to manually set the statistics about the filing 9180 system. It can be useful when those statistics cannot be read correctly 9181 automatically. 9182 9183 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 9184 9185 Alternate report of used bytes 9186 9187 Some backends, most notably S3, do not report the amount of bytes used. 9188 If you need this information to be available when running df on the 9189 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 9190 flag set, instead of relying on the backend to report this information, 9191 rclone will scan the whole remote similar to rclone size and compute the 9192 total used space itself. 9193 9194 WARNING. Contrary to rclone size, this flag ignores filters so that the 9195 result is accurate. However, this is very inefficient and may cost lots 9196 of API calls resulting in extra charges. Use it as a last resort and 9197 only with caching. 9198 9199 Auth Proxy 9200 9201 If you supply the parameter --auth-proxy /path/to/program then rclone 9202 will use that program to generate backends on the fly which then are 9203 used to authenticate incoming requests. This uses a simple JSON based 9204 protocol with input on STDIN and output on STDOUT. 9205 9206 PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, 9207 if --auth-proxy is set the authorized keys option will be ignored. 9208 9209 There is an example program bin/test_proxy.py in the rclone source code. 9210 9211 The program's job is to take a user and pass on the input and turn those 9212 into the config for a backend on STDOUT in JSON format. This config will 9213 have any default parameters for the backend added, but it won't use 9214 configuration from environment variables or command line options - it is 9215 the job of the proxy program to make a complete config. 9216 9217 This config generated must have this extra parameter - _root - root to 9218 use for the backend 9219 9220 And it may have this parameter - _obscure - comma separated strings for 9221 parameters to obscure 9222 9223 If password authentication was used by the client, input to the proxy 9224 process (on STDIN) would look similar to this: 9225 9226 { 9227 "user": "me", 9228 "pass": "mypassword" 9229 } 9230 9231 If public-key authentication was used by the client, input to the proxy 9232 process (on STDIN) would look similar to this: 9233 9234 { 9235 "user": "me", 9236 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 9237 } 9238 9239 And as an example return this on STDOUT 9240 9241 { 9242 "type": "sftp", 9243 "_root": "", 9244 "_obscure": "pass", 9245 "user": "me", 9246 "pass": "mypassword", 9247 "host": "sftp.example.com" 9248 } 9249 9250 This would mean that an SFTP backend would be created on the fly for the 9251 user and pass/public_key returned in the output to the host given. Note 9252 that since _obscure is set to pass, rclone will obscure the pass 9253 parameter before creating the backend (which is required for sftp 9254 backends). 9255 9256 The program can manipulate the supplied user in any way, for example to 9257 make proxy to many different sftp backends, you could make the user be 9258 user@example.com and then set the host to example.com in the output and 9259 the user to user. For security you'd probably want to restrict the host 9260 to a limited list. 9261 9262 Note that an internal cache is keyed on user so only use that for 9263 configuration, don't use pass or public_key. This also means that if a 9264 user's password or public-key is changed the cache will need to expire 9265 (which takes 5 mins) before it takes effect. 9266 9267 This can be used to build general purpose proxies to any kind of backend 9268 that rclone supports. 9269 9270 rclone serve http remote:path [flags] 9271 9272 Options 9273 9274 --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) 9275 --allow-origin string Origin which cross-domain request (CORS) can be executed from 9276 --auth-proxy string A program to use to create the backend from the auth 9277 --baseurl string Prefix for URLs - leave blank for root 9278 --cert string TLS PEM key (concatenation of certificate and CA certificate) 9279 --client-ca string Client certificate authority to verify clients with 9280 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 9281 --dir-perms FileMode Directory permissions (default 0777) 9282 --file-perms FileMode File permissions (default 0666) 9283 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 9284 -h, --help help for http 9285 --htpasswd string A htpasswd file - if not provided no authentication is done 9286 --key string TLS PEM Private key 9287 --max-header-bytes int Maximum size of request header (default 4096) 9288 --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 9289 --no-checksum Don't compare checksums on up/download 9290 --no-modtime Don't read/write the modification time (can speed things up) 9291 --no-seek Don't allow seeking in files 9292 --pass string Password for authentication 9293 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 9294 --read-only Only allow read-only access 9295 --realm string Realm for authentication 9296 --salt string Password hashing salt (default "dlPL2MqE") 9297 --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 9298 --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 9299 --template string User-specified template 9300 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 9301 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 9302 --user string User name for authentication 9303 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 9304 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 9305 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 9306 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 9307 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 9308 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 9309 --vfs-case-insensitive If a file name not found, find a case insensitive match 9310 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 9311 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 9312 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 9313 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 9314 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 9315 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 9316 --vfs-refresh Refreshes the directory cache recursively in the background on start 9317 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 9318 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 9319 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 9320 9321 Filter Options 9322 9323 Flags for filtering directory listings. 9324 9325 --delete-excluded Delete files on dest excluded from sync 9326 --exclude stringArray Exclude files matching pattern 9327 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 9328 --exclude-if-present stringArray Exclude directories if filename is present 9329 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 9330 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 9331 -f, --filter stringArray Add a file filtering rule 9332 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 9333 --ignore-case Ignore case in filters (case insensitive) 9334 --include stringArray Include files matching pattern 9335 --include-from stringArray Read file include patterns from file (use - to read from stdin) 9336 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9337 --max-depth int If set limits the recursion depth to this (default -1) 9338 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 9339 --metadata-exclude stringArray Exclude metadatas matching pattern 9340 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 9341 --metadata-filter stringArray Add a metadata filtering rule 9342 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 9343 --metadata-include stringArray Include metadatas matching pattern 9344 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 9345 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9346 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 9347 9348 See the global flags page for global options not listed here. 9349 9350 SEE ALSO 9351 9352 - rclone serve - Serve a remote over a protocol. 9353 9354 rclone serve nfs 9355 9356 Serve the remote as an NFS mount 9357 9358 Synopsis 9359 9360 Create an NFS server that serves the given remote over the network. 9361 9362 The primary purpose for this command is to enable mount command on 9363 recent macOS versions where installing FUSE is very cumbersome. 9364 9365 Since this is running on NFSv3, no authentication method is available. 9366 Any client will be able to access the data. To limit access, you can use 9367 serve NFS on loopback address and rely on secure tunnels (such as SSH). 9368 For this reason, by default, a random TCP port is chosen and loopback 9369 interface is used for the listening address; meaning that it is only 9370 available to the local machine. If you want other machines to access the 9371 NFS mount over local network, you need to specify the listening address 9372 and port using --addr flag. 9373 9374 Modifying files through NFS protocol requires VFS caching. Usually you 9375 will need to specify --vfs-cache-mode in order to be able to write to 9376 the mountpoint (full is recommended). If you don't specify VFS cache 9377 mode, the mount will be read-only. Note also that 9378 --nfs-cache-handle-limit controls the maximum number of cached file 9379 handles stored by the caching handler. This should not be set too low or 9380 you may experience errors when trying to access files. The default is 9381 1000000, but consider lowering this limit if the server's system 9382 resource usage causes problems. 9383 9384 To serve NFS over the network use following command: 9385 9386 rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full 9387 9388 We specify a specific port that we can use in the mount command: 9389 9390 To mount the server under Linux/macOS, use the following command: 9391 9392 mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint 9393 9394 Where $PORT is the same port number we used in the serve nfs command. 9395 9396 This feature is only available on Unix platforms. 9397 9398 VFS - Virtual File System 9399 9400 This command uses the VFS layer. This adapts the cloud storage objects 9401 that rclone uses into something which looks much more like a disk filing 9402 system. 9403 9404 Cloud storage objects have lots of properties which aren't like disk 9405 files - you can't extend them or write to the middle of them, so the VFS 9406 layer has to deal with that. Because there is no one right way of doing 9407 this there are various options explained below. 9408 9409 The VFS layer also implements a directory cache - this caches info about 9410 files and directories (but not the data) in memory. 9411 9412 VFS Directory Cache 9413 9414 Using the --dir-cache-time flag, you can control how long a directory 9415 should be considered up to date and not refreshed from the backend. 9416 Changes made through the VFS will appear immediately or invalidate the 9417 cache. 9418 9419 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 9420 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 9421 9422 However, changes made directly on the cloud storage by the web interface 9423 or a different copy of rclone will only be picked up once the directory 9424 cache expires if the backend configured does not support polling for 9425 changes. If the backend supports polling, changes will be picked up 9426 within the polling interval. 9427 9428 You can send a SIGHUP signal to rclone for it to flush all directory 9429 caches, regardless of how old they are. Assuming only one rclone 9430 instance is running, you can reset the cache like this: 9431 9432 kill -SIGHUP $(pidof rclone) 9433 9434 If you configure rclone with a remote control then you can use rclone rc 9435 to flush the whole directory cache: 9436 9437 rclone rc vfs/forget 9438 9439 Or individual files or directories: 9440 9441 rclone rc vfs/forget file=path/to/file dir=path/to/dir 9442 9443 VFS File Buffering 9444 9445 The --buffer-size flag determines the amount of memory, that will be 9446 used to buffer data in advance. 9447 9448 Each open file will try to keep the specified amount of data in memory 9449 at all times. The buffered data is bound to one open file and won't be 9450 shared. 9451 9452 This flag is a upper limit for the used memory per open file. The buffer 9453 will only use memory for data that is downloaded but not not yet read. 9454 If the buffer is empty, only a small amount of memory will be used. 9455 9456 The maximum memory used by rclone for buffering can be up to 9457 --buffer-size * open files. 9458 9459 VFS File Caching 9460 9461 These flags control the VFS file caching options. File caching is 9462 necessary to make the VFS layer appear compatible with a normal file 9463 system. It can be disabled at the cost of some compatibility. 9464 9465 For example you'll need to enable VFS caching if you want to read and 9466 write simultaneously to a file. See below for more details. 9467 9468 Note that the VFS cache is separate from the cache backend and you may 9469 find that you need one or the other or both. 9470 9471 --cache-dir string Directory rclone will use for caching. 9472 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 9473 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 9474 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 9475 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 9476 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 9477 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 9478 9479 If run with -vv rclone will print the location of the file cache. The 9480 files are stored in the user cache file area which is OS dependent but 9481 can be controlled with --cache-dir or setting the appropriate 9482 environment variable. 9483 9484 The cache has 4 different modes selected by --vfs-cache-mode. The higher 9485 the cache mode the more compatible rclone becomes at the cost of using 9486 disk space. 9487 9488 Note that files are written back to the remote only when they are closed 9489 and if they haven't been accessed for --vfs-write-back seconds. If 9490 rclone is quit or dies with files that haven't been uploaded, these will 9491 be uploaded next time rclone is run with the same flags. 9492 9493 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 9494 cache may exceed these quotas for two reasons. Firstly because it is 9495 only checked every --vfs-cache-poll-interval. Secondly because open 9496 files cannot be evicted from the cache. When --vfs-cache-max-size or 9497 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 9498 least accessed files from the cache first. rclone will start with files 9499 that haven't been accessed for the longest. This cache flushing strategy 9500 is efficient and more relevant files are likely to remain cached. 9501 9502 The --vfs-cache-max-age will evict files from the cache after the set 9503 time since last access has passed. The default value of 1 hour will 9504 start evicting files from cache that haven't been accessed for 1 hour. 9505 When a cached file is accessed the 1 hour timer is reset to 0 and will 9506 wait for 1 more hour before evicting. Specify the time with standard 9507 notation, s, m, h, d, w . 9508 9509 You should not run two copies of rclone using the same VFS cache with 9510 the same or overlapping remotes if using --vfs-cache-mode > off. This 9511 can potentially cause data corruption if you do. You can work around 9512 this by giving each rclone its own cache hierarchy with --cache-dir. You 9513 don't need to worry about this if the remotes in use don't overlap. 9514 9515 --vfs-cache-mode off 9516 9517 In this mode (the default) the cache will read directly from the remote 9518 and write directly to the remote without caching anything on disk. 9519 9520 This will mean some operations are not possible 9521 9522 - Files can't be opened for both read AND write 9523 - Files opened for write can't be seeked 9524 - Existing files opened for write must have O_TRUNC set 9525 - Files open for read with O_TRUNC will be opened write only 9526 - Files open for write only will behave as if O_TRUNC was supplied 9527 - Open modes O_APPEND, O_TRUNC are ignored 9528 - If an upload fails it can't be retried 9529 9530 --vfs-cache-mode minimal 9531 9532 This is very similar to "off" except that files opened for read AND 9533 write will be buffered to disk. This means that files opened for write 9534 will be a lot more compatible, but uses the minimal disk space. 9535 9536 These operations are not possible 9537 9538 - Files opened for write only can't be seeked 9539 - Existing files opened for write must have O_TRUNC set 9540 - Files opened for write only will ignore O_APPEND, O_TRUNC 9541 - If an upload fails it can't be retried 9542 9543 --vfs-cache-mode writes 9544 9545 In this mode files opened for read only are still read directly from the 9546 remote, write only and read/write files are buffered to disk first. 9547 9548 This mode should support all normal file system operations. 9549 9550 If an upload fails it will be retried at exponentially increasing 9551 intervals up to 1 minute. 9552 9553 --vfs-cache-mode full 9554 9555 In this mode all reads and writes are buffered to and from disk. When 9556 data is read from the remote this is buffered to disk as well. 9557 9558 In this mode the files in the cache will be sparse files and rclone will 9559 keep track of which bits of the files it has downloaded. 9560 9561 So if an application only reads the starts of each file, then rclone 9562 will only buffer the start of the file. These files will appear to be 9563 their full size in the cache, but they will be sparse files with only 9564 the data that has been downloaded present in them. 9565 9566 This mode should support all normal file system operations and is 9567 otherwise identical to --vfs-cache-mode writes. 9568 9569 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 9570 bytes ahead. The --buffer-size is buffered in memory whereas the 9571 --vfs-read-ahead is buffered on disk. 9572 9573 When using this mode it is recommended that --buffer-size is not set too 9574 large and --vfs-read-ahead is set large if required. 9575 9576 IMPORTANT not all file systems support sparse files. In particular 9577 FAT/exFAT do not. Rclone will perform very badly if the cache directory 9578 is on a filesystem which doesn't support sparse files and it will log an 9579 ERROR message if one is detected. 9580 9581 Fingerprinting 9582 9583 Various parts of the VFS use fingerprinting to see if a local file copy 9584 has changed relative to a remote file. Fingerprints are made from: 9585 9586 - size 9587 - modification time 9588 - hash 9589 9590 where available on an object. 9591 9592 On some backends some of these attributes are slow to read (they take an 9593 extra API call per object, or extra work per object). 9594 9595 For example hash is slow with the local and sftp backends as they have 9596 to read the entire file and hash it, and modtime is slow with the s3, 9597 swift, ftp and qinqstor backends because they need to do an extra API 9598 call to fetch it. 9599 9600 If you use the --vfs-fast-fingerprint flag then rclone will not include 9601 the slow operations in the fingerprint. This makes the fingerprinting 9602 less accurate but much faster and will improve the opening time of 9603 cached files. 9604 9605 If you are running a vfs cache over local, s3 or swift backends then 9606 using this flag is recommended. 9607 9608 Note that if you change the value of this flag, the fingerprints of the 9609 files in the cache may be invalidated and the files will need to be 9610 downloaded again. 9611 9612 VFS Chunked Reading 9613 9614 When rclone reads files from a remote it reads them in chunks. This 9615 means that rather than requesting the whole file rclone reads the chunk 9616 specified. This can reduce the used download quota for some remotes by 9617 requesting only chunks from the remote that are actually read, at the 9618 cost of an increased number of requests. 9619 9620 These flags control the chunking: 9621 9622 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 9623 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 9624 9625 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 9626 then double the size for each read. When --vfs-read-chunk-size-limit is 9627 specified, and greater than --vfs-read-chunk-size, the chunk size for 9628 each open file will get doubled only until the specified value is 9629 reached. If the value is "off", which is the default, the limit is 9630 disabled and the chunk size will grow indefinitely. 9631 9632 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 9633 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 9634 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 9635 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 9636 1200M-1700M and so on. 9637 9638 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 9639 9640 VFS Performance 9641 9642 These flags may be used to enable/disable features of the VFS for 9643 performance or other reasons. See also the chunked reading feature. 9644 9645 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 9646 use --use-server-modtime for a slightly different effect) as each read 9647 of the modification time takes a transaction. 9648 9649 --no-checksum Don't compare checksums on up/download. 9650 --no-modtime Don't read/write the modification time (can speed things up). 9651 --no-seek Don't allow seeking in files. 9652 --read-only Only allow read-only access. 9653 9654 Sometimes rclone is delivered reads or writes out of order. Rather than 9655 seeking rclone will wait a short time for the in sequence read or write 9656 to come in. These flags only come into effect when not using an on disk 9657 cache file. 9658 9659 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 9660 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 9661 9662 When using VFS write caching (--vfs-cache-mode with value writes or 9663 full), the global flag --transfers can be set to adjust the number of 9664 parallel uploads of modified files from the cache (the related global 9665 flag --checkers has no effect on the VFS). 9666 9667 --transfers int Number of file transfers to run in parallel (default 4) 9668 9669 VFS Case Sensitivity 9670 9671 Linux file systems are case-sensitive: two files can differ only by 9672 case, and the exact case must be used when opening a file. 9673 9674 File systems in modern Windows are case-insensitive but case-preserving: 9675 although existing files can be opened using any case, the exact case 9676 used to create the file is preserved and available for programs to 9677 query. It is not allowed for two files in the same directory to differ 9678 only by case. 9679 9680 Usually file systems on macOS are case-insensitive. It is possible to 9681 make macOS file systems case-sensitive but that is not the default. 9682 9683 The --vfs-case-insensitive VFS flag controls how rclone handles these 9684 two cases. If its value is "false", rclone passes file names to the 9685 remote as-is. If the flag is "true" (or appears without a value on the 9686 command line), rclone may perform a "fixup" as explained below. 9687 9688 The user may specify a file name to open/delete/rename/etc with a case 9689 different than what is stored on the remote. If an argument refers to an 9690 existing file with exactly the same name, then the case of the existing 9691 file on the disk will be used. However, if a file name with exactly the 9692 same name is not found but a name differing only by case exists, rclone 9693 will transparently fixup the name. This fixup happens only when an 9694 existing file is requested. Case sensitivity of file names created anew 9695 by rclone is controlled by the underlying remote. 9696 9697 Note that case sensitivity of the operating system running rclone (the 9698 target) may differ from case sensitivity of a file system presented by 9699 rclone (the source). The flag controls whether "fixup" is performed to 9700 satisfy the target. 9701 9702 If the flag is not provided on the command line, then its default value 9703 depends on the operating system where rclone runs: "true" on Windows and 9704 macOS, "false" otherwise. If the flag is provided without a value, then 9705 it is "true". 9706 9707 The --no-unicode-normalization flag controls whether a similar "fixup" 9708 is performed for filenames that differ but are canonically equivalent 9709 with respect to unicode. Unicode normalization can be particularly 9710 helpful for users of macOS, which prefers form NFD instead of the NFC 9711 used by most other platforms. It is therefore highly recommended to keep 9712 the default of false on macOS, to avoid encoding compatibility issues. 9713 9714 In the (probably unlikely) event that a directory has multiple duplicate 9715 filenames after applying case and unicode normalization, the 9716 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 9717 with a performance tradeoff, as rclone will have to scan the entire 9718 directory for duplicates when listing a directory. For this reason, it 9719 is recommended to leave this disabled if not needed. However, macOS 9720 users may wish to consider using it, as otherwise, if a remote directory 9721 contains both NFC and NFD versions of the same filename, an odd 9722 situation will occur: both versions of the file will be visible in the 9723 mount, and both will appear to be editable, however, editing either 9724 version will actually result in only the NFD version getting edited 9725 under the hood. --vfs-block- norm-dupes prevents this confusion by 9726 detecting this scenario, hiding the duplicates, and logging an error, 9727 similar to how this is handled in rclone sync. 9728 9729 VFS Disk Options 9730 9731 This flag allows you to manually set the statistics about the filing 9732 system. It can be useful when those statistics cannot be read correctly 9733 automatically. 9734 9735 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 9736 9737 Alternate report of used bytes 9738 9739 Some backends, most notably S3, do not report the amount of bytes used. 9740 If you need this information to be available when running df on the 9741 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 9742 flag set, instead of relying on the backend to report this information, 9743 rclone will scan the whole remote similar to rclone size and compute the 9744 total used space itself. 9745 9746 WARNING. Contrary to rclone size, this flag ignores filters so that the 9747 result is accurate. However, this is very inefficient and may cost lots 9748 of API calls resulting in extra charges. Use it as a last resort and 9749 only with caching. 9750 9751 rclone serve nfs remote:path [flags] 9752 9753 Options 9754 9755 --addr string IPaddress:Port or :Port to bind server to 9756 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 9757 --dir-perms FileMode Directory permissions (default 0777) 9758 --file-perms FileMode File permissions (default 0666) 9759 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 9760 -h, --help help for nfs 9761 --nfs-cache-handle-limit int max file handles cached simultaneously (min 5) (default 1000000) 9762 --no-checksum Don't compare checksums on up/download 9763 --no-modtime Don't read/write the modification time (can speed things up) 9764 --no-seek Don't allow seeking in files 9765 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 9766 --read-only Only allow read-only access 9767 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 9768 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 9769 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 9770 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 9771 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 9772 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 9773 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 9774 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 9775 --vfs-case-insensitive If a file name not found, find a case insensitive match 9776 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 9777 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 9778 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 9779 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 9780 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 9781 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 9782 --vfs-refresh Refreshes the directory cache recursively in the background on start 9783 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 9784 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 9785 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 9786 9787 Filter Options 9788 9789 Flags for filtering directory listings. 9790 9791 --delete-excluded Delete files on dest excluded from sync 9792 --exclude stringArray Exclude files matching pattern 9793 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 9794 --exclude-if-present stringArray Exclude directories if filename is present 9795 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 9796 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 9797 -f, --filter stringArray Add a file filtering rule 9798 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 9799 --ignore-case Ignore case in filters (case insensitive) 9800 --include stringArray Include files matching pattern 9801 --include-from stringArray Read file include patterns from file (use - to read from stdin) 9802 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9803 --max-depth int If set limits the recursion depth to this (default -1) 9804 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 9805 --metadata-exclude stringArray Exclude metadatas matching pattern 9806 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 9807 --metadata-filter stringArray Add a metadata filtering rule 9808 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 9809 --metadata-include stringArray Include metadatas matching pattern 9810 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 9811 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 9812 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 9813 9814 See the global flags page for global options not listed here. 9815 9816 SEE ALSO 9817 9818 - rclone serve - Serve a remote over a protocol. 9819 9820 rclone serve restic 9821 9822 Serve the remote for restic's REST API. 9823 9824 Synopsis 9825 9826 Run a basic web server to serve a remote over restic's REST backend API 9827 over HTTP. This allows restic to use rclone as a data storage mechanism 9828 for cloud providers that restic does not support directly. 9829 9830 Restic is a command-line program for doing backups. 9831 9832 The server will log errors. Use -v to see access logs. 9833 9834 --bwlimit will be respected for file transfers. Use --stats to control 9835 the stats printing. 9836 9837 Setting up rclone for use by restic 9838 9839 First set up a remote for your chosen cloud provider. 9840 9841 Once you have set up the remote, check it is working with, for example 9842 "rclone lsd remote:". You may have called the remote something other 9843 than "remote:" - just substitute whatever you called it in the following 9844 instructions. 9845 9846 Now start the rclone restic server 9847 9848 rclone serve restic -v remote:backup 9849 9850 Where you can replace "backup" in the above by whatever path in the 9851 remote you wish to use. 9852 9853 By default this will serve on "localhost:8080" you can change this with 9854 use of the --addr flag. 9855 9856 You might wish to start this server on boot. 9857 9858 Adding --cache-objects=false will cause rclone to stop caching objects 9859 returned from the List call. Caching is normally desirable as it speeds 9860 up downloading objects, saves transactions and uses very little memory. 9861 9862 Setting up restic to use rclone 9863 9864 Now you can follow the restic instructions on setting up restic. 9865 9866 Note that you will need restic 0.8.2 or later to interoperate with 9867 rclone. 9868 9869 For the example above you will want to use "http://localhost:8080/" as 9870 the URL for the REST server. 9871 9872 For example: 9873 9874 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/ 9875 $ export RESTIC_PASSWORD=yourpassword 9876 $ restic init 9877 created restic backend 8b1a4b56ae at rest:http://localhost:8080/ 9878 9879 Please note that knowledge of your password is required to access 9880 the repository. Losing your password means that your data is 9881 irrecoverably lost. 9882 $ restic backup /path/to/files/to/backup 9883 scan [/path/to/files/to/backup] 9884 scanned 189 directories, 312 files in 0:00 9885 [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00 9886 duration: 0:00 9887 snapshot 45c8fdd8 saved 9888 9889 Multiple repositories 9890 9891 Note that you can use the endpoint to host multiple repositories. Do 9892 this by adding a directory name or path after the URL. Note that these 9893 must end with /. Eg 9894 9895 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/ 9896 # backup user1 stuff 9897 $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/ 9898 # backup user2 stuff 9899 9900 Private repositories 9901 9902 The--private-repos flag can be used to limit users to repositories 9903 starting with a path of /<username>/. 9904 9905 Server options 9906 9907 Use --addr to specify which IP address and port the server should listen 9908 on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By 9909 default it only listens on localhost. You can use port :0 to let the OS 9910 choose an available port. 9911 9912 If you set --addr to listen on a public or LAN accessible IP address 9913 then using Authentication is advised - see the next section for info. 9914 9915 You can use a unix socket by setting the url to unix:///path/to/socket 9916 or just by using an absolute path name. Note that unix sockets bypass 9917 the authentication - this is expected to be done with file system 9918 permissions. 9919 9920 --addr may be repeated to listen on multiple IPs/ports/sockets. 9921 9922 --server-read-timeout and --server-write-timeout can be used to control 9923 the timeouts on the server. Note that this is the total time for a 9924 transfer. 9925 9926 --max-header-bytes controls the maximum number of bytes the server will 9927 accept in the HTTP header. 9928 9929 --baseurl controls the URL prefix that rclone serves from. By default 9930 rclone will serve from the root. If you used --baseurl "/rclone" then 9931 rclone would serve from a URL starting with "/rclone/". This is useful 9932 if you wish to proxy rclone serve. Rclone automatically inserts leading 9933 and trailing "/" on --baseurl, so --baseurl "rclone", 9934 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 9935 identically. 9936 9937 TLS (SSL) 9938 9939 By default this will serve over http. If you want you can serve over 9940 https. You will need to supply the --cert and --key flags. If you wish 9941 to do client side certificate validation then you will need to supply 9942 --client-ca also. 9943 9944 --cert should be a either a PEM encoded certificate or a concatenation 9945 of that with the CA certificate. --key should be the PEM encoded private 9946 key and --client-ca should be the PEM encoded client certificate 9947 authority certificate. 9948 9949 --min-tls-version is minimum TLS version that is acceptable. Valid 9950 values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 9951 9952 Authentication 9953 9954 By default this will serve files without needing a login. 9955 9956 You can either use an htpasswd file which can take lots of users, or set 9957 a single username and password with the --user and --pass flags. 9958 9959 If no static users are configured by either of the above methods, and 9960 client certificates are required by the --client-ca flag passed to the 9961 server, the client certificate common name will be considered as the 9962 username. 9963 9964 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in 9965 standard apache format and supports MD5, SHA1 and BCrypt for basic 9966 authentication. Bcrypt is recommended. 9967 9968 To create an htpasswd file: 9969 9970 touch htpasswd 9971 htpasswd -B htpasswd user 9972 htpasswd -B htpasswd anotherUser 9973 9974 The password file can be updated while rclone is running. 9975 9976 Use --realm to set the authentication realm. 9977 9978 Use --salt to change the password hashing salt from the default. 9979 9980 rclone serve restic remote:path [flags] 9981 9982 Options 9983 9984 --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) 9985 --allow-origin string Origin which cross-domain request (CORS) can be executed from 9986 --append-only Disallow deletion of repository data 9987 --baseurl string Prefix for URLs - leave blank for root 9988 --cache-objects Cache listed objects (default true) 9989 --cert string TLS PEM key (concatenation of certificate and CA certificate) 9990 --client-ca string Client certificate authority to verify clients with 9991 -h, --help help for restic 9992 --htpasswd string A htpasswd file - if not provided no authentication is done 9993 --key string TLS PEM Private key 9994 --max-header-bytes int Maximum size of request header (default 4096) 9995 --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 9996 --pass string Password for authentication 9997 --private-repos Users can only access their private repo 9998 --realm string Realm for authentication 9999 --salt string Password hashing salt (default "dlPL2MqE") 10000 --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 10001 --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 10002 --stdio Run an HTTP2 server on stdin/stdout 10003 --user string User name for authentication 10004 10005 See the global flags page for global options not listed here. 10006 10007 SEE ALSO 10008 10009 - rclone serve - Serve a remote over a protocol. 10010 10011 rclone serve s3 10012 10013 Serve remote:path over s3. 10014 10015 Synopsis 10016 10017 serve s3 implements a basic s3 server that serves a remote via s3. This 10018 can be viewed with an s3 client, or you can make an s3 type remote to 10019 read and write to it with rclone. 10020 10021 serve s3 is considered Experimental so use with care. 10022 10023 S3 server supports Signature Version 4 authentication. Just use 10024 --auth-key accessKey,secretKey and set the Authorization header 10025 correctly in the request. (See the AWS docs). 10026 10027 --auth-key can be repeated for multiple auth pairs. If --auth-key is not 10028 provided then serve s3 will allow anonymous access. 10029 10030 Please note that some clients may require HTTPS endpoints. See the SSL 10031 docs for more information. 10032 10033 This command uses the VFS directory cache. All the functionality will 10034 work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes) 10035 can be used to cache objects locally to improve performance. 10036 10037 Use --force-path-style=false if you want to use the bucket name as a 10038 part of the hostname (such as mybucket.local) 10039 10040 Use --etag-hash if you want to change the hash uses for the ETag. Note 10041 that using anything other than MD5 (the default) is likely to cause 10042 problems for S3 clients which rely on the Etag being the MD5. 10043 10044 Quickstart 10045 10046 For a simple set up, to serve remote:path over s3, run the server like 10047 this: 10048 10049 rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path 10050 10051 This will be compatible with an rclone remote which is defined like 10052 this: 10053 10054 [serves3] 10055 type = s3 10056 provider = Rclone 10057 endpoint = http://127.0.0.1:8080/ 10058 access_key_id = ACCESS_KEY_ID 10059 secret_access_key = SECRET_ACCESS_KEY 10060 use_multipart_uploads = false 10061 10062 Note that setting disable_multipart_uploads = true is to work around a 10063 bug which will be fixed in due course. 10064 10065 Bugs 10066 10067 When uploading multipart files serve s3 holds all the parts in memory 10068 (see #7453). This is a limitaton of the library rclone uses for serving 10069 S3 and will hopefully be fixed at some point. 10070 10071 Multipart server side copies do not work (see #7454). These take a very 10072 long time and eventually fail. The default threshold for multipart 10073 server side copies is 5G which is the maximum it can be, so files above 10074 this side will fail to be server side copied. 10075 10076 For a current list of serve s3 bugs see the serve s3 bug category on 10077 GitHub. 10078 10079 Limitations 10080 10081 serve s3 will treat all directories in the root as buckets and ignore 10082 all files in the root. You can use CreateBucket to create folders under 10083 the root, but you can't create empty folders under other folders not in 10084 the root. 10085 10086 When using PutObject or DeleteObject, rclone will automatically create 10087 or clean up empty folders. If you don't want to clean up empty folders 10088 automatically, use --no-cleanup. 10089 10090 When using ListObjects, rclone will use / when the delimiter is empty. 10091 This reduces backend requests with no effect on most operations, but if 10092 the delimiter is something other than / and empty, rclone will do a full 10093 recursive search of the backend, which can take some time. 10094 10095 Versioning is not currently supported. 10096 10097 Metadata will only be saved in memory other than the rclone mtime 10098 metadata which will be set as the modification time of the file. 10099 10100 Supported operations 10101 10102 serve s3 currently supports the following operations. 10103 10104 - Bucket 10105 - ListBuckets 10106 - CreateBucket 10107 - DeleteBucket 10108 - Object 10109 - HeadObject 10110 - ListObjects 10111 - GetObject 10112 - PutObject 10113 - DeleteObject 10114 - DeleteObjects 10115 - CreateMultipartUpload 10116 - CompleteMultipartUpload 10117 - AbortMultipartUpload 10118 - CopyObject 10119 - UploadPart 10120 10121 Other operations will return error Unimplemented. 10122 10123 Server options 10124 10125 Use --addr to specify which IP address and port the server should listen 10126 on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By 10127 default it only listens on localhost. You can use port :0 to let the OS 10128 choose an available port. 10129 10130 If you set --addr to listen on a public or LAN accessible IP address 10131 then using Authentication is advised - see the next section for info. 10132 10133 You can use a unix socket by setting the url to unix:///path/to/socket 10134 or just by using an absolute path name. Note that unix sockets bypass 10135 the authentication - this is expected to be done with file system 10136 permissions. 10137 10138 --addr may be repeated to listen on multiple IPs/ports/sockets. 10139 10140 --server-read-timeout and --server-write-timeout can be used to control 10141 the timeouts on the server. Note that this is the total time for a 10142 transfer. 10143 10144 --max-header-bytes controls the maximum number of bytes the server will 10145 accept in the HTTP header. 10146 10147 --baseurl controls the URL prefix that rclone serves from. By default 10148 rclone will serve from the root. If you used --baseurl "/rclone" then 10149 rclone would serve from a URL starting with "/rclone/". This is useful 10150 if you wish to proxy rclone serve. Rclone automatically inserts leading 10151 and trailing "/" on --baseurl, so --baseurl "rclone", 10152 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 10153 identically. 10154 10155 TLS (SSL) 10156 10157 By default this will serve over http. If you want you can serve over 10158 https. You will need to supply the --cert and --key flags. If you wish 10159 to do client side certificate validation then you will need to supply 10160 --client-ca also. 10161 10162 --cert should be a either a PEM encoded certificate or a concatenation 10163 of that with the CA certificate. --key should be the PEM encoded private 10164 key and --client-ca should be the PEM encoded client certificate 10165 authority certificate. 10166 10167 --min-tls-version is minimum TLS version that is acceptable. Valid 10168 values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 10169 ## VFS - Virtual File System 10170 10171 This command uses the VFS layer. This adapts the cloud storage objects 10172 that rclone uses into something which looks much more like a disk filing 10173 system. 10174 10175 Cloud storage objects have lots of properties which aren't like disk 10176 files - you can't extend them or write to the middle of them, so the VFS 10177 layer has to deal with that. Because there is no one right way of doing 10178 this there are various options explained below. 10179 10180 The VFS layer also implements a directory cache - this caches info about 10181 files and directories (but not the data) in memory. 10182 10183 VFS Directory Cache 10184 10185 Using the --dir-cache-time flag, you can control how long a directory 10186 should be considered up to date and not refreshed from the backend. 10187 Changes made through the VFS will appear immediately or invalidate the 10188 cache. 10189 10190 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 10191 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 10192 10193 However, changes made directly on the cloud storage by the web interface 10194 or a different copy of rclone will only be picked up once the directory 10195 cache expires if the backend configured does not support polling for 10196 changes. If the backend supports polling, changes will be picked up 10197 within the polling interval. 10198 10199 You can send a SIGHUP signal to rclone for it to flush all directory 10200 caches, regardless of how old they are. Assuming only one rclone 10201 instance is running, you can reset the cache like this: 10202 10203 kill -SIGHUP $(pidof rclone) 10204 10205 If you configure rclone with a remote control then you can use rclone rc 10206 to flush the whole directory cache: 10207 10208 rclone rc vfs/forget 10209 10210 Or individual files or directories: 10211 10212 rclone rc vfs/forget file=path/to/file dir=path/to/dir 10213 10214 VFS File Buffering 10215 10216 The --buffer-size flag determines the amount of memory, that will be 10217 used to buffer data in advance. 10218 10219 Each open file will try to keep the specified amount of data in memory 10220 at all times. The buffered data is bound to one open file and won't be 10221 shared. 10222 10223 This flag is a upper limit for the used memory per open file. The buffer 10224 will only use memory for data that is downloaded but not not yet read. 10225 If the buffer is empty, only a small amount of memory will be used. 10226 10227 The maximum memory used by rclone for buffering can be up to 10228 --buffer-size * open files. 10229 10230 VFS File Caching 10231 10232 These flags control the VFS file caching options. File caching is 10233 necessary to make the VFS layer appear compatible with a normal file 10234 system. It can be disabled at the cost of some compatibility. 10235 10236 For example you'll need to enable VFS caching if you want to read and 10237 write simultaneously to a file. See below for more details. 10238 10239 Note that the VFS cache is separate from the cache backend and you may 10240 find that you need one or the other or both. 10241 10242 --cache-dir string Directory rclone will use for caching. 10243 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 10244 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 10245 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 10246 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 10247 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 10248 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 10249 10250 If run with -vv rclone will print the location of the file cache. The 10251 files are stored in the user cache file area which is OS dependent but 10252 can be controlled with --cache-dir or setting the appropriate 10253 environment variable. 10254 10255 The cache has 4 different modes selected by --vfs-cache-mode. The higher 10256 the cache mode the more compatible rclone becomes at the cost of using 10257 disk space. 10258 10259 Note that files are written back to the remote only when they are closed 10260 and if they haven't been accessed for --vfs-write-back seconds. If 10261 rclone is quit or dies with files that haven't been uploaded, these will 10262 be uploaded next time rclone is run with the same flags. 10263 10264 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 10265 cache may exceed these quotas for two reasons. Firstly because it is 10266 only checked every --vfs-cache-poll-interval. Secondly because open 10267 files cannot be evicted from the cache. When --vfs-cache-max-size or 10268 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 10269 least accessed files from the cache first. rclone will start with files 10270 that haven't been accessed for the longest. This cache flushing strategy 10271 is efficient and more relevant files are likely to remain cached. 10272 10273 The --vfs-cache-max-age will evict files from the cache after the set 10274 time since last access has passed. The default value of 1 hour will 10275 start evicting files from cache that haven't been accessed for 1 hour. 10276 When a cached file is accessed the 1 hour timer is reset to 0 and will 10277 wait for 1 more hour before evicting. Specify the time with standard 10278 notation, s, m, h, d, w . 10279 10280 You should not run two copies of rclone using the same VFS cache with 10281 the same or overlapping remotes if using --vfs-cache-mode > off. This 10282 can potentially cause data corruption if you do. You can work around 10283 this by giving each rclone its own cache hierarchy with --cache-dir. You 10284 don't need to worry about this if the remotes in use don't overlap. 10285 10286 --vfs-cache-mode off 10287 10288 In this mode (the default) the cache will read directly from the remote 10289 and write directly to the remote without caching anything on disk. 10290 10291 This will mean some operations are not possible 10292 10293 - Files can't be opened for both read AND write 10294 - Files opened for write can't be seeked 10295 - Existing files opened for write must have O_TRUNC set 10296 - Files open for read with O_TRUNC will be opened write only 10297 - Files open for write only will behave as if O_TRUNC was supplied 10298 - Open modes O_APPEND, O_TRUNC are ignored 10299 - If an upload fails it can't be retried 10300 10301 --vfs-cache-mode minimal 10302 10303 This is very similar to "off" except that files opened for read AND 10304 write will be buffered to disk. This means that files opened for write 10305 will be a lot more compatible, but uses the minimal disk space. 10306 10307 These operations are not possible 10308 10309 - Files opened for write only can't be seeked 10310 - Existing files opened for write must have O_TRUNC set 10311 - Files opened for write only will ignore O_APPEND, O_TRUNC 10312 - If an upload fails it can't be retried 10313 10314 --vfs-cache-mode writes 10315 10316 In this mode files opened for read only are still read directly from the 10317 remote, write only and read/write files are buffered to disk first. 10318 10319 This mode should support all normal file system operations. 10320 10321 If an upload fails it will be retried at exponentially increasing 10322 intervals up to 1 minute. 10323 10324 --vfs-cache-mode full 10325 10326 In this mode all reads and writes are buffered to and from disk. When 10327 data is read from the remote this is buffered to disk as well. 10328 10329 In this mode the files in the cache will be sparse files and rclone will 10330 keep track of which bits of the files it has downloaded. 10331 10332 So if an application only reads the starts of each file, then rclone 10333 will only buffer the start of the file. These files will appear to be 10334 their full size in the cache, but they will be sparse files with only 10335 the data that has been downloaded present in them. 10336 10337 This mode should support all normal file system operations and is 10338 otherwise identical to --vfs-cache-mode writes. 10339 10340 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 10341 bytes ahead. The --buffer-size is buffered in memory whereas the 10342 --vfs-read-ahead is buffered on disk. 10343 10344 When using this mode it is recommended that --buffer-size is not set too 10345 large and --vfs-read-ahead is set large if required. 10346 10347 IMPORTANT not all file systems support sparse files. In particular 10348 FAT/exFAT do not. Rclone will perform very badly if the cache directory 10349 is on a filesystem which doesn't support sparse files and it will log an 10350 ERROR message if one is detected. 10351 10352 Fingerprinting 10353 10354 Various parts of the VFS use fingerprinting to see if a local file copy 10355 has changed relative to a remote file. Fingerprints are made from: 10356 10357 - size 10358 - modification time 10359 - hash 10360 10361 where available on an object. 10362 10363 On some backends some of these attributes are slow to read (they take an 10364 extra API call per object, or extra work per object). 10365 10366 For example hash is slow with the local and sftp backends as they have 10367 to read the entire file and hash it, and modtime is slow with the s3, 10368 swift, ftp and qinqstor backends because they need to do an extra API 10369 call to fetch it. 10370 10371 If you use the --vfs-fast-fingerprint flag then rclone will not include 10372 the slow operations in the fingerprint. This makes the fingerprinting 10373 less accurate but much faster and will improve the opening time of 10374 cached files. 10375 10376 If you are running a vfs cache over local, s3 or swift backends then 10377 using this flag is recommended. 10378 10379 Note that if you change the value of this flag, the fingerprints of the 10380 files in the cache may be invalidated and the files will need to be 10381 downloaded again. 10382 10383 VFS Chunked Reading 10384 10385 When rclone reads files from a remote it reads them in chunks. This 10386 means that rather than requesting the whole file rclone reads the chunk 10387 specified. This can reduce the used download quota for some remotes by 10388 requesting only chunks from the remote that are actually read, at the 10389 cost of an increased number of requests. 10390 10391 These flags control the chunking: 10392 10393 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 10394 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 10395 10396 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 10397 then double the size for each read. When --vfs-read-chunk-size-limit is 10398 specified, and greater than --vfs-read-chunk-size, the chunk size for 10399 each open file will get doubled only until the specified value is 10400 reached. If the value is "off", which is the default, the limit is 10401 disabled and the chunk size will grow indefinitely. 10402 10403 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 10404 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 10405 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 10406 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 10407 1200M-1700M and so on. 10408 10409 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 10410 10411 VFS Performance 10412 10413 These flags may be used to enable/disable features of the VFS for 10414 performance or other reasons. See also the chunked reading feature. 10415 10416 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 10417 use --use-server-modtime for a slightly different effect) as each read 10418 of the modification time takes a transaction. 10419 10420 --no-checksum Don't compare checksums on up/download. 10421 --no-modtime Don't read/write the modification time (can speed things up). 10422 --no-seek Don't allow seeking in files. 10423 --read-only Only allow read-only access. 10424 10425 Sometimes rclone is delivered reads or writes out of order. Rather than 10426 seeking rclone will wait a short time for the in sequence read or write 10427 to come in. These flags only come into effect when not using an on disk 10428 cache file. 10429 10430 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 10431 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 10432 10433 When using VFS write caching (--vfs-cache-mode with value writes or 10434 full), the global flag --transfers can be set to adjust the number of 10435 parallel uploads of modified files from the cache (the related global 10436 flag --checkers has no effect on the VFS). 10437 10438 --transfers int Number of file transfers to run in parallel (default 4) 10439 10440 VFS Case Sensitivity 10441 10442 Linux file systems are case-sensitive: two files can differ only by 10443 case, and the exact case must be used when opening a file. 10444 10445 File systems in modern Windows are case-insensitive but case-preserving: 10446 although existing files can be opened using any case, the exact case 10447 used to create the file is preserved and available for programs to 10448 query. It is not allowed for two files in the same directory to differ 10449 only by case. 10450 10451 Usually file systems on macOS are case-insensitive. It is possible to 10452 make macOS file systems case-sensitive but that is not the default. 10453 10454 The --vfs-case-insensitive VFS flag controls how rclone handles these 10455 two cases. If its value is "false", rclone passes file names to the 10456 remote as-is. If the flag is "true" (or appears without a value on the 10457 command line), rclone may perform a "fixup" as explained below. 10458 10459 The user may specify a file name to open/delete/rename/etc with a case 10460 different than what is stored on the remote. If an argument refers to an 10461 existing file with exactly the same name, then the case of the existing 10462 file on the disk will be used. However, if a file name with exactly the 10463 same name is not found but a name differing only by case exists, rclone 10464 will transparently fixup the name. This fixup happens only when an 10465 existing file is requested. Case sensitivity of file names created anew 10466 by rclone is controlled by the underlying remote. 10467 10468 Note that case sensitivity of the operating system running rclone (the 10469 target) may differ from case sensitivity of a file system presented by 10470 rclone (the source). The flag controls whether "fixup" is performed to 10471 satisfy the target. 10472 10473 If the flag is not provided on the command line, then its default value 10474 depends on the operating system where rclone runs: "true" on Windows and 10475 macOS, "false" otherwise. If the flag is provided without a value, then 10476 it is "true". 10477 10478 The --no-unicode-normalization flag controls whether a similar "fixup" 10479 is performed for filenames that differ but are canonically equivalent 10480 with respect to unicode. Unicode normalization can be particularly 10481 helpful for users of macOS, which prefers form NFD instead of the NFC 10482 used by most other platforms. It is therefore highly recommended to keep 10483 the default of false on macOS, to avoid encoding compatibility issues. 10484 10485 In the (probably unlikely) event that a directory has multiple duplicate 10486 filenames after applying case and unicode normalization, the 10487 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 10488 with a performance tradeoff, as rclone will have to scan the entire 10489 directory for duplicates when listing a directory. For this reason, it 10490 is recommended to leave this disabled if not needed. However, macOS 10491 users may wish to consider using it, as otherwise, if a remote directory 10492 contains both NFC and NFD versions of the same filename, an odd 10493 situation will occur: both versions of the file will be visible in the 10494 mount, and both will appear to be editable, however, editing either 10495 version will actually result in only the NFD version getting edited 10496 under the hood. --vfs-block- norm-dupes prevents this confusion by 10497 detecting this scenario, hiding the duplicates, and logging an error, 10498 similar to how this is handled in rclone sync. 10499 10500 VFS Disk Options 10501 10502 This flag allows you to manually set the statistics about the filing 10503 system. It can be useful when those statistics cannot be read correctly 10504 automatically. 10505 10506 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 10507 10508 Alternate report of used bytes 10509 10510 Some backends, most notably S3, do not report the amount of bytes used. 10511 If you need this information to be available when running df on the 10512 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 10513 flag set, instead of relying on the backend to report this information, 10514 rclone will scan the whole remote similar to rclone size and compute the 10515 total used space itself. 10516 10517 WARNING. Contrary to rclone size, this flag ignores filters so that the 10518 result is accurate. However, this is very inefficient and may cost lots 10519 of API calls resulting in extra charges. Use it as a last resort and 10520 only with caching. 10521 10522 rclone serve s3 remote:path [flags] 10523 10524 Options 10525 10526 --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) 10527 --allow-origin string Origin which cross-domain request (CORS) can be executed from 10528 --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key 10529 --baseurl string Prefix for URLs - leave blank for root 10530 --cert string TLS PEM key (concatenation of certificate and CA certificate) 10531 --client-ca string Client certificate authority to verify clients with 10532 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 10533 --dir-perms FileMode Directory permissions (default 0777) 10534 --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5") 10535 --file-perms FileMode File permissions (default 0666) 10536 --force-path-style If true use path style access if false use virtual hosted style (default true) (default true) 10537 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 10538 -h, --help help for s3 10539 --key string TLS PEM Private key 10540 --max-header-bytes int Maximum size of request header (default 4096) 10541 --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 10542 --no-checksum Don't compare checksums on up/download 10543 --no-cleanup Not to cleanup empty folder after object is deleted 10544 --no-modtime Don't read/write the modification time (can speed things up) 10545 --no-seek Don't allow seeking in files 10546 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 10547 --read-only Only allow read-only access 10548 --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 10549 --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 10550 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 10551 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 10552 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 10553 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 10554 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 10555 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 10556 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 10557 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 10558 --vfs-case-insensitive If a file name not found, find a case insensitive match 10559 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 10560 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 10561 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 10562 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 10563 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 10564 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 10565 --vfs-refresh Refreshes the directory cache recursively in the background on start 10566 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 10567 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 10568 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 10569 10570 Filter Options 10571 10572 Flags for filtering directory listings. 10573 10574 --delete-excluded Delete files on dest excluded from sync 10575 --exclude stringArray Exclude files matching pattern 10576 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 10577 --exclude-if-present stringArray Exclude directories if filename is present 10578 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 10579 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 10580 -f, --filter stringArray Add a file filtering rule 10581 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 10582 --ignore-case Ignore case in filters (case insensitive) 10583 --include stringArray Include files matching pattern 10584 --include-from stringArray Read file include patterns from file (use - to read from stdin) 10585 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 10586 --max-depth int If set limits the recursion depth to this (default -1) 10587 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 10588 --metadata-exclude stringArray Exclude metadatas matching pattern 10589 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 10590 --metadata-filter stringArray Add a metadata filtering rule 10591 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 10592 --metadata-include stringArray Include metadatas matching pattern 10593 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 10594 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 10595 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 10596 10597 See the global flags page for global options not listed here. 10598 10599 SEE ALSO 10600 10601 - rclone serve - Serve a remote over a protocol. 10602 10603 rclone serve sftp 10604 10605 Serve the remote over SFTP. 10606 10607 Synopsis 10608 10609 Run an SFTP server to serve a remote over SFTP. This can be used with an 10610 SFTP client or you can make a remote of type sftp to use with it. 10611 10612 You can use the filter flags (e.g. --include, --exclude) to control what 10613 is served. 10614 10615 The server will respond to a small number of shell commands, mainly 10616 md5sum, sha1sum and df, which enable it to provide support for checksums 10617 and the about feature when accessed from an sftp remote. 10618 10619 Note that this server uses standard 32 KiB packet payload size, which 10620 means you must not configure the client to expect anything else, e.g. 10621 with the chunk_size option on an sftp remote. 10622 10623 The server will log errors. Use -v to see access logs. 10624 10625 --bwlimit will be respected for file transfers. Use --stats to control 10626 the stats printing. 10627 10628 You must provide some means of authentication, either with 10629 --user/--pass, an authorized keys file (specify location with 10630 --authorized-keys - the default is the same as ssh), an --auth-proxy, or 10631 set the --no-auth flag for no authentication when logging in. 10632 10633 If you don't supply a host --key then rclone will generate rsa, ecdsa 10634 and ed25519 variants, and cache them for later use in rclone's cache 10635 directory (see rclone help flags cache-dir) in the "serve-sftp" 10636 directory. 10637 10638 By default the server binds to localhost:2022 - if you want it to be 10639 reachable externally then supply --addr :2022 for example. 10640 10641 Note that the default of --vfs-cache-mode off is fine for the rclone 10642 sftp backend, but it may not be with other SFTP clients. 10643 10644 If --stdio is specified, rclone will serve SFTP over stdio, which can be 10645 used with sshd via ~/.ssh/authorized_keys, for example: 10646 10647 restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ... 10648 10649 On the client you need to set --transfers 1 when using --stdio. 10650 Otherwise multiple instances of the rclone server are started by OpenSSH 10651 which can lead to "corrupted on transfer" errors. This is the case 10652 because the client chooses indiscriminately which server to send 10653 commands to while the servers all have different views of the state of 10654 the filing system. 10655 10656 The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from 10657 being used. Omitting "restrict" and using --sftp-path-override to enable 10658 checksumming is possible but less secure and you could use the SFTP 10659 server provided by OpenSSH in this case. 10660 10661 VFS - Virtual File System 10662 10663 This command uses the VFS layer. This adapts the cloud storage objects 10664 that rclone uses into something which looks much more like a disk filing 10665 system. 10666 10667 Cloud storage objects have lots of properties which aren't like disk 10668 files - you can't extend them or write to the middle of them, so the VFS 10669 layer has to deal with that. Because there is no one right way of doing 10670 this there are various options explained below. 10671 10672 The VFS layer also implements a directory cache - this caches info about 10673 files and directories (but not the data) in memory. 10674 10675 VFS Directory Cache 10676 10677 Using the --dir-cache-time flag, you can control how long a directory 10678 should be considered up to date and not refreshed from the backend. 10679 Changes made through the VFS will appear immediately or invalidate the 10680 cache. 10681 10682 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 10683 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 10684 10685 However, changes made directly on the cloud storage by the web interface 10686 or a different copy of rclone will only be picked up once the directory 10687 cache expires if the backend configured does not support polling for 10688 changes. If the backend supports polling, changes will be picked up 10689 within the polling interval. 10690 10691 You can send a SIGHUP signal to rclone for it to flush all directory 10692 caches, regardless of how old they are. Assuming only one rclone 10693 instance is running, you can reset the cache like this: 10694 10695 kill -SIGHUP $(pidof rclone) 10696 10697 If you configure rclone with a remote control then you can use rclone rc 10698 to flush the whole directory cache: 10699 10700 rclone rc vfs/forget 10701 10702 Or individual files or directories: 10703 10704 rclone rc vfs/forget file=path/to/file dir=path/to/dir 10705 10706 VFS File Buffering 10707 10708 The --buffer-size flag determines the amount of memory, that will be 10709 used to buffer data in advance. 10710 10711 Each open file will try to keep the specified amount of data in memory 10712 at all times. The buffered data is bound to one open file and won't be 10713 shared. 10714 10715 This flag is a upper limit for the used memory per open file. The buffer 10716 will only use memory for data that is downloaded but not not yet read. 10717 If the buffer is empty, only a small amount of memory will be used. 10718 10719 The maximum memory used by rclone for buffering can be up to 10720 --buffer-size * open files. 10721 10722 VFS File Caching 10723 10724 These flags control the VFS file caching options. File caching is 10725 necessary to make the VFS layer appear compatible with a normal file 10726 system. It can be disabled at the cost of some compatibility. 10727 10728 For example you'll need to enable VFS caching if you want to read and 10729 write simultaneously to a file. See below for more details. 10730 10731 Note that the VFS cache is separate from the cache backend and you may 10732 find that you need one or the other or both. 10733 10734 --cache-dir string Directory rclone will use for caching. 10735 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 10736 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 10737 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 10738 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 10739 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 10740 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 10741 10742 If run with -vv rclone will print the location of the file cache. The 10743 files are stored in the user cache file area which is OS dependent but 10744 can be controlled with --cache-dir or setting the appropriate 10745 environment variable. 10746 10747 The cache has 4 different modes selected by --vfs-cache-mode. The higher 10748 the cache mode the more compatible rclone becomes at the cost of using 10749 disk space. 10750 10751 Note that files are written back to the remote only when they are closed 10752 and if they haven't been accessed for --vfs-write-back seconds. If 10753 rclone is quit or dies with files that haven't been uploaded, these will 10754 be uploaded next time rclone is run with the same flags. 10755 10756 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 10757 cache may exceed these quotas for two reasons. Firstly because it is 10758 only checked every --vfs-cache-poll-interval. Secondly because open 10759 files cannot be evicted from the cache. When --vfs-cache-max-size or 10760 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 10761 least accessed files from the cache first. rclone will start with files 10762 that haven't been accessed for the longest. This cache flushing strategy 10763 is efficient and more relevant files are likely to remain cached. 10764 10765 The --vfs-cache-max-age will evict files from the cache after the set 10766 time since last access has passed. The default value of 1 hour will 10767 start evicting files from cache that haven't been accessed for 1 hour. 10768 When a cached file is accessed the 1 hour timer is reset to 0 and will 10769 wait for 1 more hour before evicting. Specify the time with standard 10770 notation, s, m, h, d, w . 10771 10772 You should not run two copies of rclone using the same VFS cache with 10773 the same or overlapping remotes if using --vfs-cache-mode > off. This 10774 can potentially cause data corruption if you do. You can work around 10775 this by giving each rclone its own cache hierarchy with --cache-dir. You 10776 don't need to worry about this if the remotes in use don't overlap. 10777 10778 --vfs-cache-mode off 10779 10780 In this mode (the default) the cache will read directly from the remote 10781 and write directly to the remote without caching anything on disk. 10782 10783 This will mean some operations are not possible 10784 10785 - Files can't be opened for both read AND write 10786 - Files opened for write can't be seeked 10787 - Existing files opened for write must have O_TRUNC set 10788 - Files open for read with O_TRUNC will be opened write only 10789 - Files open for write only will behave as if O_TRUNC was supplied 10790 - Open modes O_APPEND, O_TRUNC are ignored 10791 - If an upload fails it can't be retried 10792 10793 --vfs-cache-mode minimal 10794 10795 This is very similar to "off" except that files opened for read AND 10796 write will be buffered to disk. This means that files opened for write 10797 will be a lot more compatible, but uses the minimal disk space. 10798 10799 These operations are not possible 10800 10801 - Files opened for write only can't be seeked 10802 - Existing files opened for write must have O_TRUNC set 10803 - Files opened for write only will ignore O_APPEND, O_TRUNC 10804 - If an upload fails it can't be retried 10805 10806 --vfs-cache-mode writes 10807 10808 In this mode files opened for read only are still read directly from the 10809 remote, write only and read/write files are buffered to disk first. 10810 10811 This mode should support all normal file system operations. 10812 10813 If an upload fails it will be retried at exponentially increasing 10814 intervals up to 1 minute. 10815 10816 --vfs-cache-mode full 10817 10818 In this mode all reads and writes are buffered to and from disk. When 10819 data is read from the remote this is buffered to disk as well. 10820 10821 In this mode the files in the cache will be sparse files and rclone will 10822 keep track of which bits of the files it has downloaded. 10823 10824 So if an application only reads the starts of each file, then rclone 10825 will only buffer the start of the file. These files will appear to be 10826 their full size in the cache, but they will be sparse files with only 10827 the data that has been downloaded present in them. 10828 10829 This mode should support all normal file system operations and is 10830 otherwise identical to --vfs-cache-mode writes. 10831 10832 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 10833 bytes ahead. The --buffer-size is buffered in memory whereas the 10834 --vfs-read-ahead is buffered on disk. 10835 10836 When using this mode it is recommended that --buffer-size is not set too 10837 large and --vfs-read-ahead is set large if required. 10838 10839 IMPORTANT not all file systems support sparse files. In particular 10840 FAT/exFAT do not. Rclone will perform very badly if the cache directory 10841 is on a filesystem which doesn't support sparse files and it will log an 10842 ERROR message if one is detected. 10843 10844 Fingerprinting 10845 10846 Various parts of the VFS use fingerprinting to see if a local file copy 10847 has changed relative to a remote file. Fingerprints are made from: 10848 10849 - size 10850 - modification time 10851 - hash 10852 10853 where available on an object. 10854 10855 On some backends some of these attributes are slow to read (they take an 10856 extra API call per object, or extra work per object). 10857 10858 For example hash is slow with the local and sftp backends as they have 10859 to read the entire file and hash it, and modtime is slow with the s3, 10860 swift, ftp and qinqstor backends because they need to do an extra API 10861 call to fetch it. 10862 10863 If you use the --vfs-fast-fingerprint flag then rclone will not include 10864 the slow operations in the fingerprint. This makes the fingerprinting 10865 less accurate but much faster and will improve the opening time of 10866 cached files. 10867 10868 If you are running a vfs cache over local, s3 or swift backends then 10869 using this flag is recommended. 10870 10871 Note that if you change the value of this flag, the fingerprints of the 10872 files in the cache may be invalidated and the files will need to be 10873 downloaded again. 10874 10875 VFS Chunked Reading 10876 10877 When rclone reads files from a remote it reads them in chunks. This 10878 means that rather than requesting the whole file rclone reads the chunk 10879 specified. This can reduce the used download quota for some remotes by 10880 requesting only chunks from the remote that are actually read, at the 10881 cost of an increased number of requests. 10882 10883 These flags control the chunking: 10884 10885 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 10886 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 10887 10888 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 10889 then double the size for each read. When --vfs-read-chunk-size-limit is 10890 specified, and greater than --vfs-read-chunk-size, the chunk size for 10891 each open file will get doubled only until the specified value is 10892 reached. If the value is "off", which is the default, the limit is 10893 disabled and the chunk size will grow indefinitely. 10894 10895 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 10896 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 10897 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 10898 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 10899 1200M-1700M and so on. 10900 10901 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 10902 10903 VFS Performance 10904 10905 These flags may be used to enable/disable features of the VFS for 10906 performance or other reasons. See also the chunked reading feature. 10907 10908 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 10909 use --use-server-modtime for a slightly different effect) as each read 10910 of the modification time takes a transaction. 10911 10912 --no-checksum Don't compare checksums on up/download. 10913 --no-modtime Don't read/write the modification time (can speed things up). 10914 --no-seek Don't allow seeking in files. 10915 --read-only Only allow read-only access. 10916 10917 Sometimes rclone is delivered reads or writes out of order. Rather than 10918 seeking rclone will wait a short time for the in sequence read or write 10919 to come in. These flags only come into effect when not using an on disk 10920 cache file. 10921 10922 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 10923 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 10924 10925 When using VFS write caching (--vfs-cache-mode with value writes or 10926 full), the global flag --transfers can be set to adjust the number of 10927 parallel uploads of modified files from the cache (the related global 10928 flag --checkers has no effect on the VFS). 10929 10930 --transfers int Number of file transfers to run in parallel (default 4) 10931 10932 VFS Case Sensitivity 10933 10934 Linux file systems are case-sensitive: two files can differ only by 10935 case, and the exact case must be used when opening a file. 10936 10937 File systems in modern Windows are case-insensitive but case-preserving: 10938 although existing files can be opened using any case, the exact case 10939 used to create the file is preserved and available for programs to 10940 query. It is not allowed for two files in the same directory to differ 10941 only by case. 10942 10943 Usually file systems on macOS are case-insensitive. It is possible to 10944 make macOS file systems case-sensitive but that is not the default. 10945 10946 The --vfs-case-insensitive VFS flag controls how rclone handles these 10947 two cases. If its value is "false", rclone passes file names to the 10948 remote as-is. If the flag is "true" (or appears without a value on the 10949 command line), rclone may perform a "fixup" as explained below. 10950 10951 The user may specify a file name to open/delete/rename/etc with a case 10952 different than what is stored on the remote. If an argument refers to an 10953 existing file with exactly the same name, then the case of the existing 10954 file on the disk will be used. However, if a file name with exactly the 10955 same name is not found but a name differing only by case exists, rclone 10956 will transparently fixup the name. This fixup happens only when an 10957 existing file is requested. Case sensitivity of file names created anew 10958 by rclone is controlled by the underlying remote. 10959 10960 Note that case sensitivity of the operating system running rclone (the 10961 target) may differ from case sensitivity of a file system presented by 10962 rclone (the source). The flag controls whether "fixup" is performed to 10963 satisfy the target. 10964 10965 If the flag is not provided on the command line, then its default value 10966 depends on the operating system where rclone runs: "true" on Windows and 10967 macOS, "false" otherwise. If the flag is provided without a value, then 10968 it is "true". 10969 10970 The --no-unicode-normalization flag controls whether a similar "fixup" 10971 is performed for filenames that differ but are canonically equivalent 10972 with respect to unicode. Unicode normalization can be particularly 10973 helpful for users of macOS, which prefers form NFD instead of the NFC 10974 used by most other platforms. It is therefore highly recommended to keep 10975 the default of false on macOS, to avoid encoding compatibility issues. 10976 10977 In the (probably unlikely) event that a directory has multiple duplicate 10978 filenames after applying case and unicode normalization, the 10979 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 10980 with a performance tradeoff, as rclone will have to scan the entire 10981 directory for duplicates when listing a directory. For this reason, it 10982 is recommended to leave this disabled if not needed. However, macOS 10983 users may wish to consider using it, as otherwise, if a remote directory 10984 contains both NFC and NFD versions of the same filename, an odd 10985 situation will occur: both versions of the file will be visible in the 10986 mount, and both will appear to be editable, however, editing either 10987 version will actually result in only the NFD version getting edited 10988 under the hood. --vfs-block- norm-dupes prevents this confusion by 10989 detecting this scenario, hiding the duplicates, and logging an error, 10990 similar to how this is handled in rclone sync. 10991 10992 VFS Disk Options 10993 10994 This flag allows you to manually set the statistics about the filing 10995 system. It can be useful when those statistics cannot be read correctly 10996 automatically. 10997 10998 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 10999 11000 Alternate report of used bytes 11001 11002 Some backends, most notably S3, do not report the amount of bytes used. 11003 If you need this information to be available when running df on the 11004 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 11005 flag set, instead of relying on the backend to report this information, 11006 rclone will scan the whole remote similar to rclone size and compute the 11007 total used space itself. 11008 11009 WARNING. Contrary to rclone size, this flag ignores filters so that the 11010 result is accurate. However, this is very inefficient and may cost lots 11011 of API calls resulting in extra charges. Use it as a last resort and 11012 only with caching. 11013 11014 Auth Proxy 11015 11016 If you supply the parameter --auth-proxy /path/to/program then rclone 11017 will use that program to generate backends on the fly which then are 11018 used to authenticate incoming requests. This uses a simple JSON based 11019 protocol with input on STDIN and output on STDOUT. 11020 11021 PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, 11022 if --auth-proxy is set the authorized keys option will be ignored. 11023 11024 There is an example program bin/test_proxy.py in the rclone source code. 11025 11026 The program's job is to take a user and pass on the input and turn those 11027 into the config for a backend on STDOUT in JSON format. This config will 11028 have any default parameters for the backend added, but it won't use 11029 configuration from environment variables or command line options - it is 11030 the job of the proxy program to make a complete config. 11031 11032 This config generated must have this extra parameter - _root - root to 11033 use for the backend 11034 11035 And it may have this parameter - _obscure - comma separated strings for 11036 parameters to obscure 11037 11038 If password authentication was used by the client, input to the proxy 11039 process (on STDIN) would look similar to this: 11040 11041 { 11042 "user": "me", 11043 "pass": "mypassword" 11044 } 11045 11046 If public-key authentication was used by the client, input to the proxy 11047 process (on STDIN) would look similar to this: 11048 11049 { 11050 "user": "me", 11051 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 11052 } 11053 11054 And as an example return this on STDOUT 11055 11056 { 11057 "type": "sftp", 11058 "_root": "", 11059 "_obscure": "pass", 11060 "user": "me", 11061 "pass": "mypassword", 11062 "host": "sftp.example.com" 11063 } 11064 11065 This would mean that an SFTP backend would be created on the fly for the 11066 user and pass/public_key returned in the output to the host given. Note 11067 that since _obscure is set to pass, rclone will obscure the pass 11068 parameter before creating the backend (which is required for sftp 11069 backends). 11070 11071 The program can manipulate the supplied user in any way, for example to 11072 make proxy to many different sftp backends, you could make the user be 11073 user@example.com and then set the host to example.com in the output and 11074 the user to user. For security you'd probably want to restrict the host 11075 to a limited list. 11076 11077 Note that an internal cache is keyed on user so only use that for 11078 configuration, don't use pass or public_key. This also means that if a 11079 user's password or public-key is changed the cache will need to expire 11080 (which takes 5 mins) before it takes effect. 11081 11082 This can be used to build general purpose proxies to any kind of backend 11083 that rclone supports. 11084 11085 rclone serve sftp remote:path [flags] 11086 11087 Options 11088 11089 --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022") 11090 --auth-proxy string A program to use to create the backend from the auth 11091 --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys") 11092 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 11093 --dir-perms FileMode Directory permissions (default 0777) 11094 --file-perms FileMode File permissions (default 0666) 11095 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 11096 -h, --help help for sftp 11097 --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate) 11098 --no-auth Allow connections with no authentication if set 11099 --no-checksum Don't compare checksums on up/download 11100 --no-modtime Don't read/write the modification time (can speed things up) 11101 --no-seek Don't allow seeking in files 11102 --pass string Password for authentication 11103 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 11104 --read-only Only allow read-only access 11105 --stdio Run an sftp server on stdin/stdout 11106 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 11107 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 11108 --user string User name for authentication 11109 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 11110 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 11111 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 11112 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 11113 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 11114 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 11115 --vfs-case-insensitive If a file name not found, find a case insensitive match 11116 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 11117 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 11118 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 11119 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 11120 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 11121 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 11122 --vfs-refresh Refreshes the directory cache recursively in the background on start 11123 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 11124 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 11125 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 11126 11127 Filter Options 11128 11129 Flags for filtering directory listings. 11130 11131 --delete-excluded Delete files on dest excluded from sync 11132 --exclude stringArray Exclude files matching pattern 11133 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 11134 --exclude-if-present stringArray Exclude directories if filename is present 11135 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 11136 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 11137 -f, --filter stringArray Add a file filtering rule 11138 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 11139 --ignore-case Ignore case in filters (case insensitive) 11140 --include stringArray Include files matching pattern 11141 --include-from stringArray Read file include patterns from file (use - to read from stdin) 11142 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 11143 --max-depth int If set limits the recursion depth to this (default -1) 11144 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 11145 --metadata-exclude stringArray Exclude metadatas matching pattern 11146 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 11147 --metadata-filter stringArray Add a metadata filtering rule 11148 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 11149 --metadata-include stringArray Include metadatas matching pattern 11150 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 11151 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 11152 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 11153 11154 See the global flags page for global options not listed here. 11155 11156 SEE ALSO 11157 11158 - rclone serve - Serve a remote over a protocol. 11159 11160 rclone serve webdav 11161 11162 Serve remote:path over WebDAV. 11163 11164 Synopsis 11165 11166 Run a basic WebDAV server to serve a remote over HTTP via the WebDAV 11167 protocol. This can be viewed with a WebDAV client, through a web 11168 browser, or you can make a remote of type WebDAV to read and write it. 11169 11170 WebDAV options 11171 11172 --etag-hash 11173 11174 This controls the ETag header. Without this flag the ETag will be based 11175 on the ModTime and Size of the object. 11176 11177 If this flag is set to "auto" then rclone will choose the first 11178 supported hash on the backend or you can use a named hash such as "MD5" 11179 or "SHA-1". Use the hashsum command to see the full list. 11180 11181 Access WebDAV on Windows 11182 11183 WebDAV shared folder can be mapped as a drive on Windows, however the 11184 default settings prevent it. Windows will fail to connect to the server 11185 using insecure Basic authentication. It will not even display any login 11186 dialog. Windows requires SSL / HTTPS connection to be used with Basic. 11187 If you try to connect via Add Network Location Wizard you will get the 11188 following error: "The folder you entered does not appear to be valid. 11189 Please choose another". However, you still can connect if you set the 11190 following registry key on a client machine: HKEY_LOCAL_MACHINEto 2. The 11191 BasicAuthLevel can be set to the following values: 0 - Basic 11192 authentication disabled 1 - Basic authentication enabled for SSL 11193 connections only 2 - Basic authentication enabled for SSL connections 11194 and for non-SSL connections If required, increase the 11195 FileSizeLimitInBytes to a higher value. Navigate to the Services 11196 interface, then restart the WebClient service. 11197 11198 Access Office applications on WebDAV 11199 11200 Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create 11201 a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication 11202 disabled 1 - Basic authentication enabled for SSL connections only 2 - 11203 Basic authentication enabled for SSL and for non-SSL connections 11204 11205 https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint 11206 11207 Server options 11208 11209 Use --addr to specify which IP address and port the server should listen 11210 on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By 11211 default it only listens on localhost. You can use port :0 to let the OS 11212 choose an available port. 11213 11214 If you set --addr to listen on a public or LAN accessible IP address 11215 then using Authentication is advised - see the next section for info. 11216 11217 You can use a unix socket by setting the url to unix:///path/to/socket 11218 or just by using an absolute path name. Note that unix sockets bypass 11219 the authentication - this is expected to be done with file system 11220 permissions. 11221 11222 --addr may be repeated to listen on multiple IPs/ports/sockets. 11223 11224 --server-read-timeout and --server-write-timeout can be used to control 11225 the timeouts on the server. Note that this is the total time for a 11226 transfer. 11227 11228 --max-header-bytes controls the maximum number of bytes the server will 11229 accept in the HTTP header. 11230 11231 --baseurl controls the URL prefix that rclone serves from. By default 11232 rclone will serve from the root. If you used --baseurl "/rclone" then 11233 rclone would serve from a URL starting with "/rclone/". This is useful 11234 if you wish to proxy rclone serve. Rclone automatically inserts leading 11235 and trailing "/" on --baseurl, so --baseurl "rclone", 11236 --baseurl "/rclone" and --baseurl "/rclone/" are all treated 11237 identically. 11238 11239 TLS (SSL) 11240 11241 By default this will serve over http. If you want you can serve over 11242 https. You will need to supply the --cert and --key flags. If you wish 11243 to do client side certificate validation then you will need to supply 11244 --client-ca also. 11245 11246 --cert should be a either a PEM encoded certificate or a concatenation 11247 of that with the CA certificate. --key should be the PEM encoded private 11248 key and --client-ca should be the PEM encoded client certificate 11249 authority certificate. 11250 11251 --min-tls-version is minimum TLS version that is acceptable. Valid 11252 values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 11253 11254 Template 11255 11256 --template allows a user to specify a custom markup template for HTTP 11257 and WebDAV serve functions. The server exports the following markup to 11258 be used within the template to server pages: 11259 11260 ----------------------------------------------------------------------- 11261 Parameter Description 11262 ----------------------------------- ----------------------------------- 11263 .Name The full path of a file/directory. 11264 11265 .Title Directory listing of .Name 11266 11267 .Sort The current sort used. This is 11268 changeable via ?sort= parameter 11269 11270 Sort Options: 11271 namedirfirst,name,size,time 11272 (default namedirfirst) 11273 11274 .Order The current ordering used. This is 11275 changeable via ?order= parameter 11276 11277 Order Options: asc,desc (default 11278 asc) 11279 11280 .Query Currently unused. 11281 11282 .Breadcrumb Allows for creating a relative 11283 navigation 11284 11285 -- .Link The relative to the root link of 11286 the Text. 11287 11288 -- .Text The Name of the directory. 11289 11290 .Entries Information about a specific 11291 file/directory. 11292 11293 -- .URL The 'url' of an entry. 11294 11295 -- .Leaf Currently same as 'URL' but 11296 intended to be 'just' the name. 11297 11298 -- .IsDir Boolean for if an entry is a 11299 directory or not. 11300 11301 -- .Size Size in Bytes of the entry. 11302 11303 -- .ModTime The UTC timestamp of an entry. 11304 ----------------------------------------------------------------------- 11305 11306 The server also makes the following functions available so that they can 11307 be used within the template. These functions help extend the options for 11308 dynamic rendering of HTML. They can be used to render HTML based on 11309 specific conditions. 11310 11311 ----------------------------------------------------------------------- 11312 Function Description 11313 ----------------------------------- ----------------------------------- 11314 afterEpoch Returns the time since the epoch 11315 for the given time. 11316 11317 contains Checks whether a given substring is 11318 present or not in a given string. 11319 11320 hasPrefix Checks whether the given string 11321 begins with the specified prefix. 11322 11323 hasSuffix Checks whether the given string end 11324 with the specified suffix. 11325 ----------------------------------------------------------------------- 11326 11327 Authentication 11328 11329 By default this will serve files without needing a login. 11330 11331 You can either use an htpasswd file which can take lots of users, or set 11332 a single username and password with the --user and --pass flags. 11333 11334 If no static users are configured by either of the above methods, and 11335 client certificates are required by the --client-ca flag passed to the 11336 server, the client certificate common name will be considered as the 11337 username. 11338 11339 Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in 11340 standard apache format and supports MD5, SHA1 and BCrypt for basic 11341 authentication. Bcrypt is recommended. 11342 11343 To create an htpasswd file: 11344 11345 touch htpasswd 11346 htpasswd -B htpasswd user 11347 htpasswd -B htpasswd anotherUser 11348 11349 The password file can be updated while rclone is running. 11350 11351 Use --realm to set the authentication realm. 11352 11353 Use --salt to change the password hashing salt from the default. ## VFS 11354 - Virtual File System 11355 11356 This command uses the VFS layer. This adapts the cloud storage objects 11357 that rclone uses into something which looks much more like a disk filing 11358 system. 11359 11360 Cloud storage objects have lots of properties which aren't like disk 11361 files - you can't extend them or write to the middle of them, so the VFS 11362 layer has to deal with that. Because there is no one right way of doing 11363 this there are various options explained below. 11364 11365 The VFS layer also implements a directory cache - this caches info about 11366 files and directories (but not the data) in memory. 11367 11368 VFS Directory Cache 11369 11370 Using the --dir-cache-time flag, you can control how long a directory 11371 should be considered up to date and not refreshed from the backend. 11372 Changes made through the VFS will appear immediately or invalidate the 11373 cache. 11374 11375 --dir-cache-time duration Time to cache directory entries for (default 5m0s) 11376 --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s) 11377 11378 However, changes made directly on the cloud storage by the web interface 11379 or a different copy of rclone will only be picked up once the directory 11380 cache expires if the backend configured does not support polling for 11381 changes. If the backend supports polling, changes will be picked up 11382 within the polling interval. 11383 11384 You can send a SIGHUP signal to rclone for it to flush all directory 11385 caches, regardless of how old they are. Assuming only one rclone 11386 instance is running, you can reset the cache like this: 11387 11388 kill -SIGHUP $(pidof rclone) 11389 11390 If you configure rclone with a remote control then you can use rclone rc 11391 to flush the whole directory cache: 11392 11393 rclone rc vfs/forget 11394 11395 Or individual files or directories: 11396 11397 rclone rc vfs/forget file=path/to/file dir=path/to/dir 11398 11399 VFS File Buffering 11400 11401 The --buffer-size flag determines the amount of memory, that will be 11402 used to buffer data in advance. 11403 11404 Each open file will try to keep the specified amount of data in memory 11405 at all times. The buffered data is bound to one open file and won't be 11406 shared. 11407 11408 This flag is a upper limit for the used memory per open file. The buffer 11409 will only use memory for data that is downloaded but not not yet read. 11410 If the buffer is empty, only a small amount of memory will be used. 11411 11412 The maximum memory used by rclone for buffering can be up to 11413 --buffer-size * open files. 11414 11415 VFS File Caching 11416 11417 These flags control the VFS file caching options. File caching is 11418 necessary to make the VFS layer appear compatible with a normal file 11419 system. It can be disabled at the cost of some compatibility. 11420 11421 For example you'll need to enable VFS caching if you want to read and 11422 write simultaneously to a file. See below for more details. 11423 11424 Note that the VFS cache is separate from the cache backend and you may 11425 find that you need one or the other or both. 11426 11427 --cache-dir string Directory rclone will use for caching. 11428 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 11429 --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s) 11430 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 11431 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 11432 --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s) 11433 --vfs-write-back duration Time to writeback files after last use when using cache (default 5s) 11434 11435 If run with -vv rclone will print the location of the file cache. The 11436 files are stored in the user cache file area which is OS dependent but 11437 can be controlled with --cache-dir or setting the appropriate 11438 environment variable. 11439 11440 The cache has 4 different modes selected by --vfs-cache-mode. The higher 11441 the cache mode the more compatible rclone becomes at the cost of using 11442 disk space. 11443 11444 Note that files are written back to the remote only when they are closed 11445 and if they haven't been accessed for --vfs-write-back seconds. If 11446 rclone is quit or dies with files that haven't been uploaded, these will 11447 be uploaded next time rclone is run with the same flags. 11448 11449 If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the 11450 cache may exceed these quotas for two reasons. Firstly because it is 11451 only checked every --vfs-cache-poll-interval. Secondly because open 11452 files cannot be evicted from the cache. When --vfs-cache-max-size or 11453 --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the 11454 least accessed files from the cache first. rclone will start with files 11455 that haven't been accessed for the longest. This cache flushing strategy 11456 is efficient and more relevant files are likely to remain cached. 11457 11458 The --vfs-cache-max-age will evict files from the cache after the set 11459 time since last access has passed. The default value of 1 hour will 11460 start evicting files from cache that haven't been accessed for 1 hour. 11461 When a cached file is accessed the 1 hour timer is reset to 0 and will 11462 wait for 1 more hour before evicting. Specify the time with standard 11463 notation, s, m, h, d, w . 11464 11465 You should not run two copies of rclone using the same VFS cache with 11466 the same or overlapping remotes if using --vfs-cache-mode > off. This 11467 can potentially cause data corruption if you do. You can work around 11468 this by giving each rclone its own cache hierarchy with --cache-dir. You 11469 don't need to worry about this if the remotes in use don't overlap. 11470 11471 --vfs-cache-mode off 11472 11473 In this mode (the default) the cache will read directly from the remote 11474 and write directly to the remote without caching anything on disk. 11475 11476 This will mean some operations are not possible 11477 11478 - Files can't be opened for both read AND write 11479 - Files opened for write can't be seeked 11480 - Existing files opened for write must have O_TRUNC set 11481 - Files open for read with O_TRUNC will be opened write only 11482 - Files open for write only will behave as if O_TRUNC was supplied 11483 - Open modes O_APPEND, O_TRUNC are ignored 11484 - If an upload fails it can't be retried 11485 11486 --vfs-cache-mode minimal 11487 11488 This is very similar to "off" except that files opened for read AND 11489 write will be buffered to disk. This means that files opened for write 11490 will be a lot more compatible, but uses the minimal disk space. 11491 11492 These operations are not possible 11493 11494 - Files opened for write only can't be seeked 11495 - Existing files opened for write must have O_TRUNC set 11496 - Files opened for write only will ignore O_APPEND, O_TRUNC 11497 - If an upload fails it can't be retried 11498 11499 --vfs-cache-mode writes 11500 11501 In this mode files opened for read only are still read directly from the 11502 remote, write only and read/write files are buffered to disk first. 11503 11504 This mode should support all normal file system operations. 11505 11506 If an upload fails it will be retried at exponentially increasing 11507 intervals up to 1 minute. 11508 11509 --vfs-cache-mode full 11510 11511 In this mode all reads and writes are buffered to and from disk. When 11512 data is read from the remote this is buffered to disk as well. 11513 11514 In this mode the files in the cache will be sparse files and rclone will 11515 keep track of which bits of the files it has downloaded. 11516 11517 So if an application only reads the starts of each file, then rclone 11518 will only buffer the start of the file. These files will appear to be 11519 their full size in the cache, but they will be sparse files with only 11520 the data that has been downloaded present in them. 11521 11522 This mode should support all normal file system operations and is 11523 otherwise identical to --vfs-cache-mode writes. 11524 11525 When reading a file rclone will read --buffer-size plus --vfs-read-ahead 11526 bytes ahead. The --buffer-size is buffered in memory whereas the 11527 --vfs-read-ahead is buffered on disk. 11528 11529 When using this mode it is recommended that --buffer-size is not set too 11530 large and --vfs-read-ahead is set large if required. 11531 11532 IMPORTANT not all file systems support sparse files. In particular 11533 FAT/exFAT do not. Rclone will perform very badly if the cache directory 11534 is on a filesystem which doesn't support sparse files and it will log an 11535 ERROR message if one is detected. 11536 11537 Fingerprinting 11538 11539 Various parts of the VFS use fingerprinting to see if a local file copy 11540 has changed relative to a remote file. Fingerprints are made from: 11541 11542 - size 11543 - modification time 11544 - hash 11545 11546 where available on an object. 11547 11548 On some backends some of these attributes are slow to read (they take an 11549 extra API call per object, or extra work per object). 11550 11551 For example hash is slow with the local and sftp backends as they have 11552 to read the entire file and hash it, and modtime is slow with the s3, 11553 swift, ftp and qinqstor backends because they need to do an extra API 11554 call to fetch it. 11555 11556 If you use the --vfs-fast-fingerprint flag then rclone will not include 11557 the slow operations in the fingerprint. This makes the fingerprinting 11558 less accurate but much faster and will improve the opening time of 11559 cached files. 11560 11561 If you are running a vfs cache over local, s3 or swift backends then 11562 using this flag is recommended. 11563 11564 Note that if you change the value of this flag, the fingerprints of the 11565 files in the cache may be invalidated and the files will need to be 11566 downloaded again. 11567 11568 VFS Chunked Reading 11569 11570 When rclone reads files from a remote it reads them in chunks. This 11571 means that rather than requesting the whole file rclone reads the chunk 11572 specified. This can reduce the used download quota for some remotes by 11573 requesting only chunks from the remote that are actually read, at the 11574 cost of an increased number of requests. 11575 11576 These flags control the chunking: 11577 11578 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M) 11579 --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off) 11580 11581 Rclone will start reading a chunk of size --vfs-read-chunk-size, and 11582 then double the size for each read. When --vfs-read-chunk-size-limit is 11583 specified, and greater than --vfs-read-chunk-size, the chunk size for 11584 each open file will get doubled only until the specified value is 11585 reached. If the value is "off", which is the default, the limit is 11586 disabled and the chunk size will grow indefinitely. 11587 11588 With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the 11589 following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 11590 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, 11591 the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 11592 1200M-1700M and so on. 11593 11594 Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading. 11595 11596 VFS Performance 11597 11598 These flags may be used to enable/disable features of the VFS for 11599 performance or other reasons. See also the chunked reading feature. 11600 11601 In particular S3 and Swift benefit hugely from the --no-modtime flag (or 11602 use --use-server-modtime for a slightly different effect) as each read 11603 of the modification time takes a transaction. 11604 11605 --no-checksum Don't compare checksums on up/download. 11606 --no-modtime Don't read/write the modification time (can speed things up). 11607 --no-seek Don't allow seeking in files. 11608 --read-only Only allow read-only access. 11609 11610 Sometimes rclone is delivered reads or writes out of order. Rather than 11611 seeking rclone will wait a short time for the in sequence read or write 11612 to come in. These flags only come into effect when not using an on disk 11613 cache file. 11614 11615 --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms) 11616 --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s) 11617 11618 When using VFS write caching (--vfs-cache-mode with value writes or 11619 full), the global flag --transfers can be set to adjust the number of 11620 parallel uploads of modified files from the cache (the related global 11621 flag --checkers has no effect on the VFS). 11622 11623 --transfers int Number of file transfers to run in parallel (default 4) 11624 11625 VFS Case Sensitivity 11626 11627 Linux file systems are case-sensitive: two files can differ only by 11628 case, and the exact case must be used when opening a file. 11629 11630 File systems in modern Windows are case-insensitive but case-preserving: 11631 although existing files can be opened using any case, the exact case 11632 used to create the file is preserved and available for programs to 11633 query. It is not allowed for two files in the same directory to differ 11634 only by case. 11635 11636 Usually file systems on macOS are case-insensitive. It is possible to 11637 make macOS file systems case-sensitive but that is not the default. 11638 11639 The --vfs-case-insensitive VFS flag controls how rclone handles these 11640 two cases. If its value is "false", rclone passes file names to the 11641 remote as-is. If the flag is "true" (or appears without a value on the 11642 command line), rclone may perform a "fixup" as explained below. 11643 11644 The user may specify a file name to open/delete/rename/etc with a case 11645 different than what is stored on the remote. If an argument refers to an 11646 existing file with exactly the same name, then the case of the existing 11647 file on the disk will be used. However, if a file name with exactly the 11648 same name is not found but a name differing only by case exists, rclone 11649 will transparently fixup the name. This fixup happens only when an 11650 existing file is requested. Case sensitivity of file names created anew 11651 by rclone is controlled by the underlying remote. 11652 11653 Note that case sensitivity of the operating system running rclone (the 11654 target) may differ from case sensitivity of a file system presented by 11655 rclone (the source). The flag controls whether "fixup" is performed to 11656 satisfy the target. 11657 11658 If the flag is not provided on the command line, then its default value 11659 depends on the operating system where rclone runs: "true" on Windows and 11660 macOS, "false" otherwise. If the flag is provided without a value, then 11661 it is "true". 11662 11663 The --no-unicode-normalization flag controls whether a similar "fixup" 11664 is performed for filenames that differ but are canonically equivalent 11665 with respect to unicode. Unicode normalization can be particularly 11666 helpful for users of macOS, which prefers form NFD instead of the NFC 11667 used by most other platforms. It is therefore highly recommended to keep 11668 the default of false on macOS, to avoid encoding compatibility issues. 11669 11670 In the (probably unlikely) event that a directory has multiple duplicate 11671 filenames after applying case and unicode normalization, the 11672 --vfs-block-norm-dupes flag allows hiding these duplicates. This comes 11673 with a performance tradeoff, as rclone will have to scan the entire 11674 directory for duplicates when listing a directory. For this reason, it 11675 is recommended to leave this disabled if not needed. However, macOS 11676 users may wish to consider using it, as otherwise, if a remote directory 11677 contains both NFC and NFD versions of the same filename, an odd 11678 situation will occur: both versions of the file will be visible in the 11679 mount, and both will appear to be editable, however, editing either 11680 version will actually result in only the NFD version getting edited 11681 under the hood. --vfs-block- norm-dupes prevents this confusion by 11682 detecting this scenario, hiding the duplicates, and logging an error, 11683 similar to how this is handled in rclone sync. 11684 11685 VFS Disk Options 11686 11687 This flag allows you to manually set the statistics about the filing 11688 system. It can be useful when those statistics cannot be read correctly 11689 automatically. 11690 11691 --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1) 11692 11693 Alternate report of used bytes 11694 11695 Some backends, most notably S3, do not report the amount of bytes used. 11696 If you need this information to be available when running df on the 11697 filesystem, then pass the flag --vfs-used-is-size to rclone. With this 11698 flag set, instead of relying on the backend to report this information, 11699 rclone will scan the whole remote similar to rclone size and compute the 11700 total used space itself. 11701 11702 WARNING. Contrary to rclone size, this flag ignores filters so that the 11703 result is accurate. However, this is very inefficient and may cost lots 11704 of API calls resulting in extra charges. Use it as a last resort and 11705 only with caching. 11706 11707 Auth Proxy 11708 11709 If you supply the parameter --auth-proxy /path/to/program then rclone 11710 will use that program to generate backends on the fly which then are 11711 used to authenticate incoming requests. This uses a simple JSON based 11712 protocol with input on STDIN and output on STDOUT. 11713 11714 PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, 11715 if --auth-proxy is set the authorized keys option will be ignored. 11716 11717 There is an example program bin/test_proxy.py in the rclone source code. 11718 11719 The program's job is to take a user and pass on the input and turn those 11720 into the config for a backend on STDOUT in JSON format. This config will 11721 have any default parameters for the backend added, but it won't use 11722 configuration from environment variables or command line options - it is 11723 the job of the proxy program to make a complete config. 11724 11725 This config generated must have this extra parameter - _root - root to 11726 use for the backend 11727 11728 And it may have this parameter - _obscure - comma separated strings for 11729 parameters to obscure 11730 11731 If password authentication was used by the client, input to the proxy 11732 process (on STDIN) would look similar to this: 11733 11734 { 11735 "user": "me", 11736 "pass": "mypassword" 11737 } 11738 11739 If public-key authentication was used by the client, input to the proxy 11740 process (on STDIN) would look similar to this: 11741 11742 { 11743 "user": "me", 11744 "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf" 11745 } 11746 11747 And as an example return this on STDOUT 11748 11749 { 11750 "type": "sftp", 11751 "_root": "", 11752 "_obscure": "pass", 11753 "user": "me", 11754 "pass": "mypassword", 11755 "host": "sftp.example.com" 11756 } 11757 11758 This would mean that an SFTP backend would be created on the fly for the 11759 user and pass/public_key returned in the output to the host given. Note 11760 that since _obscure is set to pass, rclone will obscure the pass 11761 parameter before creating the backend (which is required for sftp 11762 backends). 11763 11764 The program can manipulate the supplied user in any way, for example to 11765 make proxy to many different sftp backends, you could make the user be 11766 user@example.com and then set the host to example.com in the output and 11767 the user to user. For security you'd probably want to restrict the host 11768 to a limited list. 11769 11770 Note that an internal cache is keyed on user so only use that for 11771 configuration, don't use pass or public_key. This also means that if a 11772 user's password or public-key is changed the cache will need to expire 11773 (which takes 5 mins) before it takes effect. 11774 11775 This can be used to build general purpose proxies to any kind of backend 11776 that rclone supports. 11777 11778 rclone serve webdav remote:path [flags] 11779 11780 Options 11781 11782 --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080]) 11783 --allow-origin string Origin which cross-domain request (CORS) can be executed from 11784 --auth-proxy string A program to use to create the backend from the auth 11785 --baseurl string Prefix for URLs - leave blank for root 11786 --cert string TLS PEM key (concatenation of certificate and CA certificate) 11787 --client-ca string Client certificate authority to verify clients with 11788 --dir-cache-time Duration Time to cache directory entries for (default 5m0s) 11789 --dir-perms FileMode Directory permissions (default 0777) 11790 --disable-dir-list Disable HTML directory list on GET request for a directory 11791 --etag-hash string Which hash to use for the ETag, or auto or blank for off 11792 --file-perms FileMode File permissions (default 0666) 11793 --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000) 11794 -h, --help help for webdav 11795 --htpasswd string A htpasswd file - if not provided no authentication is done 11796 --key string TLS PEM Private key 11797 --max-header-bytes int Maximum size of request header (default 4096) 11798 --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 11799 --no-checksum Don't compare checksums on up/download 11800 --no-modtime Don't read/write the modification time (can speed things up) 11801 --no-seek Don't allow seeking in files 11802 --pass string Password for authentication 11803 --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s) 11804 --read-only Only allow read-only access 11805 --realm string Realm for authentication 11806 --salt string Password hashing salt (default "dlPL2MqE") 11807 --server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 11808 --server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 11809 --template string User-specified template 11810 --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000) 11811 --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2) 11812 --user string User name for authentication 11813 --vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost) 11814 --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s) 11815 --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off) 11816 --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off) 11817 --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off) 11818 --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s) 11819 --vfs-case-insensitive If a file name not found, find a case insensitive match 11820 --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off) 11821 --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection 11822 --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full 11823 --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) 11824 --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) 11825 --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) 11826 --vfs-refresh Refreshes the directory cache recursively in the background on start 11827 --vfs-used-is-size rclone size Use the rclone size algorithm for Used size 11828 --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) 11829 --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) 11830 11831 Filter Options 11832 11833 Flags for filtering directory listings. 11834 11835 --delete-excluded Delete files on dest excluded from sync 11836 --exclude stringArray Exclude files matching pattern 11837 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 11838 --exclude-if-present stringArray Exclude directories if filename is present 11839 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 11840 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 11841 -f, --filter stringArray Add a file filtering rule 11842 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 11843 --ignore-case Ignore case in filters (case insensitive) 11844 --include stringArray Include files matching pattern 11845 --include-from stringArray Read file include patterns from file (use - to read from stdin) 11846 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 11847 --max-depth int If set limits the recursion depth to this (default -1) 11848 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 11849 --metadata-exclude stringArray Exclude metadatas matching pattern 11850 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 11851 --metadata-filter stringArray Add a metadata filtering rule 11852 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 11853 --metadata-include stringArray Include metadatas matching pattern 11854 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 11855 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 11856 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 11857 11858 See the global flags page for global options not listed here. 11859 11860 SEE ALSO 11861 11862 - rclone serve - Serve a remote over a protocol. 11863 11864 rclone settier 11865 11866 Changes storage class/tier of objects in remote. 11867 11868 Synopsis 11869 11870 rclone settier changes storage tier or class at remote if supported. Few 11871 cloud storage services provides different storage classes on objects, 11872 for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and 11873 Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc. 11874 11875 Note that, certain tier changes make objects not available to access 11876 immediately. For example tiering to archive in azure blob storage makes 11877 objects in frozen state, user can restore by setting tier to Hot/Cool, 11878 similarly S3 to Glacier makes object inaccessible.true 11879 11880 You can use it to tier single object 11881 11882 rclone settier Cool remote:path/file 11883 11884 Or use rclone filters to set tier on only specific files 11885 11886 rclone --include "*.txt" settier Hot remote:path/dir 11887 11888 Or just provide remote directory and all files in directory will be 11889 tiered 11890 11891 rclone settier tier remote:path/dir 11892 11893 rclone settier tier remote:path [flags] 11894 11895 Options 11896 11897 -h, --help help for settier 11898 11899 See the global flags page for global options not listed here. 11900 11901 SEE ALSO 11902 11903 - rclone - Show help for rclone commands, flags and backends. 11904 11905 rclone test 11906 11907 Run a test command 11908 11909 Synopsis 11910 11911 Rclone test is used to run test commands. 11912 11913 Select which test command you want with the subcommand, eg 11914 11915 rclone test memory remote: 11916 11917 Each subcommand has its own options which you can see in their help. 11918 11919 NB Be careful running these commands, they may do strange things so 11920 reading their documentation first is recommended. 11921 11922 Options 11923 11924 -h, --help help for test 11925 11926 See the global flags page for global options not listed here. 11927 11928 SEE ALSO 11929 11930 - rclone - Show help for rclone commands, flags and backends. 11931 - rclone test changenotify - Log any change notify requests for the 11932 remote passed in. 11933 - rclone test histogram - Makes a histogram of file name characters. 11934 - rclone test info - Discovers file name or other limitations for 11935 paths. 11936 - rclone test makefile - Make files with random contents of the size 11937 given 11938 - rclone test makefiles - Make a random file hierarchy in a directory 11939 - rclone test memory - Load all the objects at remote:path into memory 11940 and report memory stats. 11941 11942 rclone test changenotify 11943 11944 Log any change notify requests for the remote passed in. 11945 11946 rclone test changenotify remote: [flags] 11947 11948 Options 11949 11950 -h, --help help for changenotify 11951 --poll-interval Duration Time to wait between polling for changes (default 10s) 11952 11953 See the global flags page for global options not listed here. 11954 11955 SEE ALSO 11956 11957 - rclone test - Run a test command 11958 11959 rclone test histogram 11960 11961 Makes a histogram of file name characters. 11962 11963 Synopsis 11964 11965 This command outputs JSON which shows the histogram of characters used 11966 in filenames in the remote:path specified. 11967 11968 The data doesn't contain any identifying information but is useful for 11969 the rclone developers when developing filename compression. 11970 11971 rclone test histogram [remote:path] [flags] 11972 11973 Options 11974 11975 -h, --help help for histogram 11976 11977 See the global flags page for global options not listed here. 11978 11979 SEE ALSO 11980 11981 - rclone test - Run a test command 11982 11983 rclone test info 11984 11985 Discovers file name or other limitations for paths. 11986 11987 Synopsis 11988 11989 rclone info discovers what filenames and upload methods are possible to 11990 write to the paths passed in and how long they can be. It can take some 11991 time. It will write test files into the remote:path passed in. It 11992 outputs a bit of go code for each one. 11993 11994 NB this can create undeletable files and other hazards - use with care 11995 11996 rclone test info [remote:path]+ [flags] 11997 11998 Options 11999 12000 --all Run all tests 12001 --check-base32768 Check can store all possible base32768 characters 12002 --check-control Check control characters 12003 --check-length Check max filename length 12004 --check-normalization Check UTF-8 Normalization 12005 --check-streaming Check uploads with indeterminate file size 12006 -h, --help help for info 12007 --upload-wait Duration Wait after writing a file (default 0s) 12008 --write-json string Write results to file 12009 12010 See the global flags page for global options not listed here. 12011 12012 SEE ALSO 12013 12014 - rclone test - Run a test command 12015 12016 rclone test makefile 12017 12018 Make files with random contents of the size given 12019 12020 rclone test makefile <size> [<file>]+ [flags] 12021 12022 Options 12023 12024 --ascii Fill files with random ASCII printable bytes only 12025 --chargen Fill files with a ASCII chargen pattern 12026 -h, --help help for makefile 12027 --pattern Fill files with a periodic pattern 12028 --seed int Seed for the random number generator (0 for random) (default 1) 12029 --sparse Make the files sparse (appear to be filled with ASCII 0x00) 12030 --zero Fill files with ASCII 0x00 12031 12032 See the global flags page for global options not listed here. 12033 12034 SEE ALSO 12035 12036 - rclone test - Run a test command 12037 12038 rclone test makefiles 12039 12040 Make a random file hierarchy in a directory 12041 12042 rclone test makefiles <dir> [flags] 12043 12044 Options 12045 12046 --ascii Fill files with random ASCII printable bytes only 12047 --chargen Fill files with a ASCII chargen pattern 12048 --files int Number of files to create (default 1000) 12049 --files-per-directory int Average number of files per directory (default 10) 12050 -h, --help help for makefiles 12051 --max-depth int Maximum depth of directory hierarchy (default 10) 12052 --max-file-size SizeSuffix Maximum size of files to create (default 100) 12053 --max-name-length int Maximum size of file names (default 12) 12054 --min-file-size SizeSuffix Minimum size of file to create 12055 --min-name-length int Minimum size of file names (default 4) 12056 --pattern Fill files with a periodic pattern 12057 --seed int Seed for the random number generator (0 for random) (default 1) 12058 --sparse Make the files sparse (appear to be filled with ASCII 0x00) 12059 --zero Fill files with ASCII 0x00 12060 12061 See the global flags page for global options not listed here. 12062 12063 SEE ALSO 12064 12065 - rclone test - Run a test command 12066 12067 rclone test memory 12068 12069 Load all the objects at remote:path into memory and report memory stats. 12070 12071 rclone test memory remote:path [flags] 12072 12073 Options 12074 12075 -h, --help help for memory 12076 12077 See the global flags page for global options not listed here. 12078 12079 SEE ALSO 12080 12081 - rclone test - Run a test command 12082 12083 rclone touch 12084 12085 Create new file or change file modification time. 12086 12087 Synopsis 12088 12089 Set the modification time on file(s) as specified by remote:path to have 12090 the current time. 12091 12092 If remote:path does not exist then a zero sized file will be created, 12093 unless --no-create or --recursive is provided. 12094 12095 If --recursive is used then recursively sets the modification time on 12096 all existing files that is found under the path. Filters are supported, 12097 and you can test with the --dry-run or the --interactive/-i flag. 12098 12099 If --timestamp is used then sets the modification time to that time 12100 instead of the current time. Times may be specified as one of: 12101 12102 - 'YYMMDD' - e.g. 17.10.30 12103 - 'YYYY-MM-DDTHH:MM:SS' - e.g. 2006-01-02T15:04:05 12104 - 'YYYY-MM-DDTHH:MM:SS.SSS' - e.g. 2006-01-02T15:04:05.123456789 12105 12106 Note that value of --timestamp is in UTC. If you want local time then 12107 add the --localtime flag. 12108 12109 rclone touch remote:path [flags] 12110 12111 Options 12112 12113 -h, --help help for touch 12114 --localtime Use localtime for timestamp, not UTC 12115 -C, --no-create Do not create the file if it does not exist (implied with --recursive) 12116 -R, --recursive Recursively touch all files 12117 -t, --timestamp string Use specified time instead of the current time of day 12118 12119 Important Options 12120 12121 Important flags useful for most commands. 12122 12123 -n, --dry-run Do a trial run with no permanent changes 12124 -i, --interactive Enable interactive mode 12125 -v, --verbose count Print lots more stuff (repeat for more) 12126 12127 Filter Options 12128 12129 Flags for filtering directory listings. 12130 12131 --delete-excluded Delete files on dest excluded from sync 12132 --exclude stringArray Exclude files matching pattern 12133 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 12134 --exclude-if-present stringArray Exclude directories if filename is present 12135 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 12136 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 12137 -f, --filter stringArray Add a file filtering rule 12138 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 12139 --ignore-case Ignore case in filters (case insensitive) 12140 --include stringArray Include files matching pattern 12141 --include-from stringArray Read file include patterns from file (use - to read from stdin) 12142 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 12143 --max-depth int If set limits the recursion depth to this (default -1) 12144 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 12145 --metadata-exclude stringArray Exclude metadatas matching pattern 12146 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 12147 --metadata-filter stringArray Add a metadata filtering rule 12148 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 12149 --metadata-include stringArray Include metadatas matching pattern 12150 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 12151 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 12152 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 12153 12154 Listing Options 12155 12156 Flags for listing directories. 12157 12158 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 12159 --fast-list Use recursive list if available; uses more memory but fewer transactions 12160 12161 See the global flags page for global options not listed here. 12162 12163 SEE ALSO 12164 12165 - rclone - Show help for rclone commands, flags and backends. 12166 12167 rclone tree 12168 12169 List the contents of the remote in a tree like fashion. 12170 12171 Synopsis 12172 12173 rclone tree lists the contents of a remote in a similar way to the unix 12174 tree command. 12175 12176 For example 12177 12178 $ rclone tree remote:path 12179 / 12180 ├── file1 12181 ├── file2 12182 ├── file3 12183 └── subdir 12184 ├── file4 12185 └── file5 12186 12187 1 directories, 5 files 12188 12189 You can use any of the filtering options with the tree command (e.g. 12190 --include and --exclude. You can also use --fast-list. 12191 12192 The tree command has many options for controlling the listing which are 12193 compatible with the tree command, for example you can include file sizes 12194 with --size. Note that not all of them have short options as they 12195 conflict with rclone's short options. 12196 12197 For a more interactive navigation of the remote see the ncdu command. 12198 12199 rclone tree remote:path [flags] 12200 12201 Options 12202 12203 -a, --all All files are listed (list . files too) 12204 -d, --dirs-only List directories only 12205 --dirsfirst List directories before files (-U disables) 12206 --full-path Print the full path prefix for each file 12207 -h, --help help for tree 12208 --level int Descend only level directories deep 12209 -D, --modtime Print the date of last modification. 12210 --noindent Don't print indentation lines 12211 --noreport Turn off file/directory count at end of tree listing 12212 -o, --output string Output to file instead of stdout 12213 -p, --protections Print the protections for each file. 12214 -Q, --quote Quote filenames with double quotes. 12215 -s, --size Print the size in bytes of each file. 12216 --sort string Select sort: name,version,size,mtime,ctime 12217 --sort-ctime Sort files by last status change time 12218 -t, --sort-modtime Sort files by last modification time 12219 -r, --sort-reverse Reverse the order of the sort 12220 -U, --unsorted Leave files unsorted 12221 --version Sort files alphanumerically by version 12222 12223 Filter Options 12224 12225 Flags for filtering directory listings. 12226 12227 --delete-excluded Delete files on dest excluded from sync 12228 --exclude stringArray Exclude files matching pattern 12229 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 12230 --exclude-if-present stringArray Exclude directories if filename is present 12231 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 12232 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 12233 -f, --filter stringArray Add a file filtering rule 12234 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 12235 --ignore-case Ignore case in filters (case insensitive) 12236 --include stringArray Include files matching pattern 12237 --include-from stringArray Read file include patterns from file (use - to read from stdin) 12238 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 12239 --max-depth int If set limits the recursion depth to this (default -1) 12240 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 12241 --metadata-exclude stringArray Exclude metadatas matching pattern 12242 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 12243 --metadata-filter stringArray Add a metadata filtering rule 12244 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 12245 --metadata-include stringArray Include metadatas matching pattern 12246 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 12247 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 12248 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 12249 12250 Listing Options 12251 12252 Flags for listing directories. 12253 12254 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 12255 --fast-list Use recursive list if available; uses more memory but fewer transactions 12256 12257 See the global flags page for global options not listed here. 12258 12259 SEE ALSO 12260 12261 - rclone - Show help for rclone commands, flags and backends. 12262 12263 Copying single files 12264 12265 rclone normally syncs or copies directories. However, if the source 12266 remote points to a file, rclone will just copy that file. The 12267 destination remote must point to a directory - rclone will give the 12268 error 12269 Failed to create file system for "remote:file": is a file not a directory 12270 if it isn't. 12271 12272 For example, suppose you have a remote with a file in called test.jpg, 12273 then you could copy just that file like this 12274 12275 rclone copy remote:test.jpg /tmp/download 12276 12277 The file test.jpg will be placed inside /tmp/download. 12278 12279 This is equivalent to specifying 12280 12281 rclone copy --files-from /tmp/files remote: /tmp/download 12282 12283 Where /tmp/files contains the single line 12284 12285 test.jpg 12286 12287 It is recommended to use copy when copying individual files, not sync. 12288 They have pretty much the same effect but copy will use a lot less 12289 memory. 12290 12291 Syntax of remote paths 12292 12293 The syntax of the paths passed to the rclone command are as follows. 12294 12295 /path/to/dir 12296 12297 This refers to the local file system. 12298 12299 On Windows \ may be used instead of / in local paths only, non local 12300 paths must use /. See local filesystem documentation for more about 12301 Windows-specific paths. 12302 12303 These paths needn't start with a leading / - if they don't then they 12304 will be relative to the current directory. 12305 12306 remote:path/to/dir 12307 12308 This refers to a directory path/to/dir on remote: as defined in the 12309 config file (configured with rclone config). 12310 12311 remote:/path/to/dir 12312 12313 On most backends this is refers to the same directory as 12314 remote:path/to/dir and that format should be preferred. On a very small 12315 number of remotes (FTP, SFTP, Dropbox for business) this will refer to a 12316 different directory. On these, paths without a leading / will refer to 12317 your "home" directory and paths with a leading / will refer to the root. 12318 12319 :backend:path/to/dir 12320 12321 This is an advanced form for creating remotes on the fly. backend should 12322 be the name or prefix of a backend (the type in the config file) and all 12323 the configuration for the backend should be provided on the command line 12324 (or in environment variables). 12325 12326 Here are some examples: 12327 12328 rclone lsd --http-url https://pub.rclone.org :http: 12329 12330 To list all the directories in the root of https://pub.rclone.org/. 12331 12332 rclone lsf --http-url https://example.com :http:path/to/dir 12333 12334 To list files and directories in https://example.com/path/to/dir/ 12335 12336 rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir 12337 12338 To copy files and directories in https://example.com/path/to/dir to 12339 /tmp/dir. 12340 12341 rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir 12342 12343 To copy files and directories from example.com in the relative directory 12344 path/to/dir to /tmp/dir using sftp. 12345 12346 Connection strings 12347 12348 The above examples can also be written using a connection string syntax, 12349 so instead of providing the arguments as command line parameters 12350 --http-url https://pub.rclone.org they are provided as part of the 12351 remote specification as a kind of connection string. 12352 12353 rclone lsd ":http,url='https://pub.rclone.org':" 12354 rclone lsf ":http,url='https://example.com':path/to/dir" 12355 rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir 12356 rclone copy :sftp,host=example.com:path/to/dir /tmp/dir 12357 12358 These can apply to modify existing remotes as well as create new remotes 12359 with the on the fly syntax. This example is equivalent to adding the 12360 --drive-shared-with-me parameter to the remote gdrive:. 12361 12362 rclone lsf "gdrive,shared_with_me:path/to/dir" 12363 12364 The major advantage to using the connection string style syntax is that 12365 it only applies to the remote, not to all the remotes of that type of 12366 the command line. A common confusion is this attempt to copy a file 12367 shared on google drive to the normal drive which does not work because 12368 the --drive-shared-with-me flag applies to both the source and the 12369 destination. 12370 12371 rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive: 12372 12373 However using the connection string syntax, this does work. 12374 12375 rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive: 12376 12377 Note that the connection string only affects the options of the 12378 immediate backend. If for example gdriveCrypt is a crypt based on 12379 gdrive, then the following command will not work as intended, because 12380 shared_with_me is ignored by the crypt backend: 12381 12382 rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt: 12383 12384 The connection strings have the following syntax 12385 12386 remote,parameter=value,parameter2=value2:path/to/dir 12387 :backend,parameter=value,parameter2=value2:path/to/dir 12388 12389 If the parameter has a : or , then it must be placed in quotes " or ', 12390 so 12391 12392 remote,parameter="colon:value",parameter2="comma,value":path/to/dir 12393 :backend,parameter='colon:value',parameter2='comma,value':path/to/dir 12394 12395 If a quoted value needs to include that quote, then it should be 12396 doubled, so 12397 12398 remote,parameter="with""quote",parameter2='with''quote':path/to/dir 12399 12400 This will make parameter be with"quote and parameter2 be with'quote. 12401 12402 If you leave off the =parameter then rclone will substitute =true which 12403 works very well with flags. For example, to use s3 configured in the 12404 environment you could use: 12405 12406 rclone lsd :s3,env_auth: 12407 12408 Which is equivalent to 12409 12410 rclone lsd :s3,env_auth=true: 12411 12412 Note that on the command line you might need to surround these 12413 connection strings with " or ' to stop the shell interpreting any 12414 special characters within them. 12415 12416 If you are a shell master then you'll know which strings are OK and 12417 which aren't, but if you aren't sure then enclose them in " and use ' as 12418 the inside quote. This syntax works on all OSes. 12419 12420 rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir 12421 12422 On Linux/macOS some characters are still interpreted inside " strings in 12423 the shell (notably \ and $ and ") so if your strings contain those you 12424 can swap the roles of " and ' thus. (This syntax does not work on 12425 Windows.) 12426 12427 rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir 12428 12429 Connection strings, config and logging 12430 12431 If you supply extra configuration to a backend by command line flag, 12432 environment variable or connection string then rclone will add a suffix 12433 based on the hash of the config to the name of the remote, eg 12434 12435 rclone -vv lsf --s3-chunk-size 20M s3: 12436 12437 Has the log message 12438 12439 DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name 12440 12441 This is so rclone can tell the modified remote apart from the unmodified 12442 remote when caching the backends. 12443 12444 This should only be noticeable in the logs. 12445 12446 This means that on the fly backends such as 12447 12448 rclone -vv lsf :s3,env_auth: 12449 12450 Will get their own names 12451 12452 DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name 12453 12454 Valid remote names 12455 12456 Remote names are case sensitive, and must adhere to the following rules: 12457 - May contain number, letter, _, -, ., +, @ and space. - May not start 12458 with - or space. - May not end with space. 12459 12460 Starting with rclone version 1.61, any Unicode numbers and letters are 12461 allowed, while in older versions it was limited to plain ASCII (0-9, 12462 A-Z, a-z). If you use the same rclone configuration from different 12463 shells, which may be configured with different character encoding, you 12464 must be cautious to use characters that are possible to write in all of 12465 them. This is mostly a problem on Windows, where the console 12466 traditionally uses a non-Unicode character set - defined by the 12467 so-called "code page". 12468 12469 Do not use single character names on Windows as it creates ambiguity 12470 with Windows drives' names, e.g.: remote called C is indistinguishable 12471 from C drive. Rclone will always assume that single letter name refers 12472 to a drive. 12473 12474 Quoting and the shell 12475 12476 When you are typing commands to your computer you are using something 12477 called the command line shell. This interprets various characters in an 12478 OS specific way. 12479 12480 Here are some gotchas which may help users unfamiliar with the shell 12481 rules 12482 12483 Linux / OSX 12484 12485 If your names have spaces or shell metacharacters (e.g. *, ?, $, ', ", 12486 etc.) then you must quote them. Use single quotes ' by default. 12487 12488 rclone copy 'Important files?' remote:backup 12489 12490 If you want to send a ' you will need to use ", e.g. 12491 12492 rclone copy "O'Reilly Reviews" remote:backup 12493 12494 The rules for quoting metacharacters are complicated and if you want the 12495 full details you'll have to consult the manual page for your shell. 12496 12497 Windows 12498 12499 If your names have spaces in you need to put them in ", e.g. 12500 12501 rclone copy "E:\folder name\folder name\folder name" remote:backup 12502 12503 If you are using the root directory on its own then don't quote it (see 12504 #464 for why), e.g. 12505 12506 rclone copy E:\ remote:backup 12507 12508 Copying files or directories with : in the names 12509 12510 rclone uses : to mark a remote name. This is, however, a valid filename 12511 component in non-Windows OSes. The remote name parser will only search 12512 for a : up to the first / so if you need to act on a file or directory 12513 like this then use the full path starting with a /, or use ./ as a 12514 current directory prefix. 12515 12516 So to sync a directory called sync:me to a remote called remote: use 12517 12518 rclone sync --interactive ./sync:me remote:path 12519 12520 or 12521 12522 rclone sync --interactive /full/path/to/sync:me remote:path 12523 12524 Server Side Copy 12525 12526 Most remotes (but not all - see the overview) support server-side copy. 12527 12528 This means if you want to copy one folder to another then rclone won't 12529 download all the files and re-upload them; it will instruct the server 12530 to copy them in place. 12531 12532 Eg 12533 12534 rclone copy s3:oldbucket s3:newbucket 12535 12536 Will copy the contents of oldbucket to newbucket without downloading and 12537 re-uploading. 12538 12539 Remotes which don't support server-side copy will download and re-upload 12540 in this case. 12541 12542 Server side copies are used with sync and copy and will be identified in 12543 the log when using the -v flag. The move command may also use them if 12544 remote doesn't support server-side move directly. This is done by 12545 issuing a server-side copy then a delete which is much quicker than a 12546 download and re-upload. 12547 12548 Server side copies will only be attempted if the remote names are the 12549 same. 12550 12551 This can be used when scripting to make aged backups efficiently, e.g. 12552 12553 rclone sync --interactive remote:current-backup remote:previous-backup 12554 rclone sync --interactive /path/to/files remote:current-backup 12555 12556 Metadata support 12557 12558 Metadata is data about a file (or directory) which isn't the contents of 12559 the file (or directory). Normally rclone only preserves the modification 12560 time and the content (MIME) type where possible. 12561 12562 Rclone supports preserving all the available metadata on files and 12563 directories when using the --metadata or -M flag. 12564 12565 Exactly what metadata is supported and what that support means depends 12566 on the backend. Backends that support metadata have a metadata section 12567 in their docs and are listed in the features table (Eg local, s3) 12568 12569 Some backends don't support metadata, some only support metadata on 12570 files and some support metadata on both files and directories. 12571 12572 Rclone only supports a one-time sync of metadata. This means that 12573 metadata will be synced from the source object to the destination object 12574 only when the source object has changed and needs to be re-uploaded. If 12575 the metadata subsequently changes on the source object without changing 12576 the object itself then it won't be synced to the destination object. 12577 This is in line with the way rclone syncs Content-Type without the 12578 --metadata flag. 12579 12580 Using --metadata when syncing from local to local will preserve file 12581 attributes such as file mode, owner, extended attributes (not Windows). 12582 12583 Note that arbitrary metadata may be added to objects using the 12584 --metadata-set key=value flag when the object is first uploaded. This 12585 flag can be repeated as many times as necessary. 12586 12587 The --metadata-mapper flag can be used to pass the name of a program in 12588 which can transform metadata when it is being copied from source to 12589 destination. 12590 12591 Rclone supports --metadata-set and --metadata-mapper when doing sever 12592 side Move and server side Copy, but not when doing server side DirMove 12593 (renaming a directory) as this would involve recursing into the 12594 directory. Note that you can disable DirMove with --disable DirMove and 12595 rclone will revert back to using Move for each individual object where 12596 --metadata-set and --metadata-mapper are supported. 12597 12598 Types of metadata 12599 12600 Metadata is divided into two type. System metadata and User metadata. 12601 12602 Metadata which the backend uses itself is called system metadata. For 12603 example on the local backend the system metadata uid will store the user 12604 ID of the file when used on a unix based platform. 12605 12606 Arbitrary metadata is called user metadata and this can be set however 12607 is desired. 12608 12609 When objects are copied from backend to backend, they will attempt to 12610 interpret system metadata if it is supplied. Metadata may change from 12611 being user metadata to system metadata as objects are copied between 12612 different backends. For example copying an object from s3 sets the 12613 content-type metadata. In a backend which understands this (like 12614 azureblob) this will become the Content-Type of the object. In a backend 12615 which doesn't understand this (like the local backend) this will become 12616 user metadata. However should the local object be copied back to s3, the 12617 Content-Type will be set correctly. 12618 12619 Metadata framework 12620 12621 Rclone implements a metadata framework which can read metadata from an 12622 object and write it to the object when (and only when) it is being 12623 uploaded. 12624 12625 This metadata is stored as a dictionary with string keys and string 12626 values. 12627 12628 There are some limits on the names of the keys (these may be clarified 12629 further in the future). 12630 12631 - must be lower case 12632 - may be a-z 0-9 containing . - or _ 12633 - length is backend dependent 12634 12635 Each backend can provide system metadata that it understands. Some 12636 backends can also store arbitrary user metadata. 12637 12638 Where possible the key names are standardized, so, for example, it is 12639 possible to copy object metadata from s3 to azureblob for example and 12640 metadata will be translated appropriately. 12641 12642 Some backends have limits on the size of the metadata and rclone will 12643 give errors on upload if they are exceeded. 12644 12645 Metadata preservation 12646 12647 The goal of the implementation is to 12648 12649 1. Preserve metadata if at all possible 12650 2. Interpret metadata if at all possible 12651 12652 The consequences of 1 is that you can copy an S3 object to a local disk 12653 then back to S3 losslessly. Likewise you can copy a local file with file 12654 attributes and xattrs from local disk to s3 and back again losslessly. 12655 12656 The consequence of 2 is that you can copy an S3 object with metadata to 12657 Azureblob (say) and have the metadata appear on the Azureblob object 12658 also. 12659 12660 Standard system metadata 12661 12662 Here is a table of standard system metadata which, if appropriate, a 12663 backend may implement. 12664 12665 ---------------------------------------------------------------------------------------------- 12666 key description example 12667 ---------------------------------- --------------------- ------------------------------------- 12668 mode File type and mode: 0100664 12669 octal, unix style 12670 12671 uid User ID of owner: 500 12672 decimal number 12673 12674 gid Group ID of owner: 500 12675 decimal number 12676 12677 rdev Device ID (if special 0 12678 file) => hexadecimal 12679 12680 atime Time of last access: 2006-01-02T15:04:05.999999999Z07:00 12681 RFC 3339 12682 12683 mtime Time of last 2006-01-02T15:04:05.999999999Z07:00 12684 modification: RFC 12685 3339 12686 12687 btime Time of file creation 2006-01-02T15:04:05.999999999Z07:00 12688 (birth): RFC 3339 12689 12690 utime Time of file upload: 2006-01-02T15:04:05.999999999Z07:00 12691 RFC 3339 12692 12693 cache-control Cache-Control header no-cache 12694 12695 content-disposition Content-Disposition inline 12696 header 12697 12698 content-encoding Content-Encoding gzip 12699 header 12700 12701 content-language Content-Language en-US 12702 header 12703 12704 content-type Content-Type header text/plain 12705 ---------------------------------------------------------------------------------------------- 12706 12707 The metadata keys mtime and content-type will take precedence if 12708 supplied in the metadata over reading the Content-Type or modification 12709 time of the source object. 12710 12711 Hashes are not included in system metadata as there is a well defined 12712 way of reading those already. 12713 12714 Options 12715 12716 Rclone has a number of options to control its behaviour. 12717 12718 Options that take parameters can have the values passed in two ways, 12719 --option=value or --option value. However boolean (true/false) options 12720 behave slightly differently to the other options in that --boolean sets 12721 the option to true and the absence of the flag sets it to false. It is 12722 also possible to specify --boolean=false or --boolean=true. Note that 12723 --boolean false is not valid - this is parsed as --boolean and the false 12724 is parsed as an extra command line argument for rclone. 12725 12726 Time or duration options 12727 12728 TIME or DURATION options can be specified as a duration string or a time 12729 string. 12730 12731 A duration string is a possibly signed sequence of decimal numbers, each 12732 with optional fraction and a unit suffix, such as "300ms", "-1.5h" or 12733 "2h45m". Default units are seconds or the following abbreviations are 12734 valid: 12735 12736 - ms - Milliseconds 12737 - s - Seconds 12738 - m - Minutes 12739 - h - Hours 12740 - d - Days 12741 - w - Weeks 12742 - M - Months 12743 - y - Years 12744 12745 These can also be specified as an absolute time in the following 12746 formats: 12747 12748 - RFC3339 - e.g. 2006-01-02T15:04:05Z or 2006-01-02T15:04:05+07:00 12749 - ISO8601 Date and time, local timezone - 2006-01-02T15:04:05 12750 - ISO8601 Date and time, local timezone - 2006-01-02 15:04:05 12751 - ISO8601 Date - 2006-01-02 (YYYY-MM-DD) 12752 12753 Size options 12754 12755 Options which use SIZE use KiB (multiples of 1024 bytes) by default. 12756 However, a suffix of B for Byte, K for KiB, M for MiB, G for GiB, T for 12757 TiB and P for PiB may be used. These are the binary units, e.g. 1, 12758 2**10, 2**20, 2**30 respectively. 12759 12760 --backup-dir=DIR 12761 12762 When using sync, copy or move any files which would have been 12763 overwritten or deleted are moved in their original hierarchy into this 12764 directory. 12765 12766 If --suffix is set, then the moved files will have the suffix added to 12767 them. If there is a file with the same path (after the suffix has been 12768 added) in DIR, then it will be overwritten. 12769 12770 The remote in use must support server-side move or copy and you must use 12771 the same remote as the destination of the sync. The backup directory 12772 must not overlap the destination directory without it being excluded by 12773 a filter rule. 12774 12775 For example 12776 12777 rclone sync --interactive /path/to/local remote:current --backup-dir remote:old 12778 12779 will sync /path/to/local to remote:current, but for any files which 12780 would have been updated or deleted will be stored in remote:old. 12781 12782 If running rclone from a script you might want to use today's date as 12783 the directory name passed to --backup-dir to store the old files, or you 12784 might want to pass --suffix with today's date. 12785 12786 See --compare-dest and --copy-dest. 12787 12788 --bind string 12789 12790 Local address to bind to for outgoing connections. This can be an IPv4 12791 address (1.2.3.4), an IPv6 address (1234::789A) or host name. If the 12792 host name doesn't resolve or resolves to more than one IP address it 12793 will give an error. 12794 12795 You can use --bind 0.0.0.0 to force rclone to use IPv4 addresses and 12796 --bind ::0 to force rclone to use IPv6 addresses. 12797 12798 --bwlimit=BANDWIDTH_SPEC 12799 12800 This option controls the bandwidth limit. For example 12801 12802 --bwlimit 10M 12803 12804 would mean limit the upload and download bandwidth to 10 MiB/s. NB this 12805 is bytes per second not bits per second. To use a single limit, specify 12806 the desired bandwidth in KiB/s, or use a suffix B|K|M|G|T|P. The default 12807 is 0 which means to not limit bandwidth. 12808 12809 The upload and download bandwidth can be specified separately, as 12810 --bwlimit UP:DOWN, so 12811 12812 --bwlimit 10M:100k 12813 12814 would mean limit the upload bandwidth to 10 MiB/s and the download 12815 bandwidth to 100 KiB/s. Either limit can be "off" meaning no limit, so 12816 to just limit the upload bandwidth you would use 12817 12818 --bwlimit 10M:off 12819 12820 this would limit the upload bandwidth to 10 MiB/s but the download 12821 bandwidth would be unlimited. 12822 12823 When specified as above the bandwidth limits last for the duration of 12824 run of the rclone binary. 12825 12826 It is also possible to specify a "timetable" of limits, which will cause 12827 certain limits to be applied at certain times. To specify a timetable, 12828 format your entries as 12829 WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH... where: WEEKDAY is 12830 optional element. 12831 12832 - BANDWIDTH can be a single number, e.g.100k or a pair of numbers for 12833 upload:download, e.g.10M:1M. 12834 - WEEKDAY can be written as the whole word or only using the first 3 12835 characters. It is optional. 12836 - HH:MM is an hour from 00:00 to 23:59. 12837 12838 An example of a typical timetable to avoid link saturation during 12839 daytime working hours could be: 12840 12841 --bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off" 12842 12843 In this example, the transfer bandwidth will be set to 512 KiB/s at 8am 12844 every day. At noon, it will rise to 10 MiB/s, and drop back to 512 12845 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and 12846 at 11pm it will be completely disabled (full speed). Anything between 12847 11pm and 8am will remain unlimited. 12848 12849 An example of timetable with WEEKDAY could be: 12850 12851 --bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off" 12852 12853 It means that, the transfer bandwidth will be set to 512 KiB/s on 12854 Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00 on 12855 Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will be 12856 unlimited. 12857 12858 Timeslots without WEEKDAY are extended to the whole week. So this 12859 example: 12860 12861 --bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off" 12862 12863 Is equivalent to this: 12864 12865 --bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off" 12866 12867 Bandwidth limit apply to the data transfer for all backends. For most 12868 backends the directory listing bandwidth is also included (exceptions 12869 being the non HTTP backends, ftp, sftp and storj). 12870 12871 Note that the units are Byte/s, not bit/s. Typically connections are 12872 measured in bit/s - to convert divide by 8. For example, let's say you 12873 have a 10 Mbit/s connection and you wish rclone to use half of it - 5 12874 Mbit/s. This is 5/8 = 0.625 MiB/s so you would use a --bwlimit 0.625M 12875 parameter for rclone. 12876 12877 On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled 12878 by sending a SIGUSR2 signal to rclone. This allows to remove the 12879 limitations of a long running rclone transfer and to restore it back to 12880 the value specified with --bwlimit quickly when needed. Assuming there 12881 is only one rclone instance running, you can toggle the limiter like 12882 this: 12883 12884 kill -SIGUSR2 $(pidof rclone) 12885 12886 If you configure rclone with a remote control then you can use change 12887 the bwlimit dynamically: 12888 12889 rclone rc core/bwlimit rate=1M 12890 12891 --bwlimit-file=BANDWIDTH_SPEC 12892 12893 This option controls per file bandwidth limit. For the options see the 12894 --bwlimit flag. 12895 12896 For example use this to allow no transfers to be faster than 1 MiB/s 12897 12898 --bwlimit-file 1M 12899 12900 This can be used in conjunction with --bwlimit. 12901 12902 Note that if a schedule is provided the file will use the schedule in 12903 effect at the start of the transfer. 12904 12905 --buffer-size=SIZE 12906 12907 Use this sized buffer to speed up file transfers. Each --transfer will 12908 use this much memory for buffering. 12909 12910 When using mount or cmount each open file descriptor will use this much 12911 memory for buffering. See the mount documentation for more details. 12912 12913 Set to 0 to disable the buffering for the minimum memory usage. 12914 12915 Note that the memory allocation of the buffers is influenced by the 12916 --use-mmap flag. 12917 12918 --cache-dir=DIR 12919 12920 Specify the directory rclone will use for caching, to override the 12921 default. 12922 12923 Default value is depending on operating system: - Windows 12924 %LocalAppData%\rclone, if LocalAppData is defined. - macOS 12925 $HOME/Library/Caches/rclone if HOME is defined. - Unix 12926 $XDG_CACHE_HOME/rclone if XDG_CACHE_HOME is defined, else 12927 $HOME/.cache/rclone if HOME is defined. - Fallback (on all OS) to 12928 $TMPDIR/rclone, where TMPDIR is the value from --temp-dir. 12929 12930 You can use the config paths command to see the current value. 12931 12932 Cache directory is heavily used by the VFS File Caching mount feature, 12933 but also by serve, GUI and other parts of rclone. 12934 12935 --check-first 12936 12937 If this flag is set then in a sync, copy or move, rclone will do all the 12938 checks to see whether files need to be transferred before doing any of 12939 the transfers. Normally rclone would start running transfers as soon as 12940 possible. 12941 12942 This flag can be useful on IO limited systems where transfers interfere 12943 with checking. 12944 12945 It can also be useful to ensure perfect ordering when using --order-by. 12946 12947 If both --check-first and --order-by are set when doing rclone move then 12948 rclone will use the transfer thread to delete source files which don't 12949 need transferring. This will enable perfect ordering of the transfers 12950 and deletes but will cause the transfer stats to have more items in than 12951 expected. 12952 12953 Using this flag can use more memory as it effectively sets --max-backlog 12954 to infinite. This means that all the info on the objects to transfer is 12955 held in memory before the transfers start. 12956 12957 --checkers=N 12958 12959 Originally controlling just the number of file checkers to run in 12960 parallel, e.g. by rclone copy. Now a fairly universal parallelism 12961 control used by rclone in several places. 12962 12963 Note: checkers do the equality checking of files during a sync. For some 12964 storage systems (e.g. S3, Swift, Dropbox) this can take a significant 12965 amount of time so they are run in parallel. 12966 12967 The default is to run 8 checkers in parallel. However, in case of 12968 slow-reacting backends you may need to lower (rather than increase) this 12969 default by setting --checkers to 4 or less threads. This is especially 12970 advised if you are experiencing backend server crashes during file 12971 checking phase (e.g. on subsequent or top-up backups where little or no 12972 file copying is done and checking takes up most of the time). Increase 12973 this setting only with utmost care, while monitoring your server health 12974 and file checking throughput. 12975 12976 -c, --checksum 12977 12978 Normally rclone will look at modification time and size of files to see 12979 if they are equal. If you set this flag then rclone will check the file 12980 hash and size to determine if files are equal. 12981 12982 This is useful when the remote doesn't support setting modified time and 12983 a more accurate sync is desired than just checking the file size. 12984 12985 This is very useful when transferring between remotes which store the 12986 same hash type on the object, e.g. Drive and Swift. For details of which 12987 remotes support which hash type see the table in the overview section. 12988 12989 Eg rclone --checksum sync s3:/bucket swift:/bucket would run much 12990 quicker than without the --checksum flag. 12991 12992 When using this flag, rclone won't update mtimes of remote files if they 12993 are incorrect as it would normally. 12994 12995 --color WHEN 12996 12997 Specify when colors (and other ANSI codes) should be added to the 12998 output. 12999 13000 AUTO (default) only allows ANSI codes when the output is a terminal 13001 13002 NEVER never allow ANSI codes 13003 13004 ALWAYS always add ANSI codes, regardless of the output format (terminal 13005 or file) 13006 13007 --compare-dest=DIR 13008 13009 When using sync, copy or move DIR is checked in addition to the 13010 destination for files. If a file identical to the source is found that 13011 file is NOT copied from source. This is useful to copy just files that 13012 have changed since the last backup. 13013 13014 You must use the same remote as the destination of the sync. The compare 13015 directory must not overlap the destination directory. 13016 13017 See --copy-dest and --backup-dir. 13018 13019 --config=CONFIG_FILE 13020 13021 Specify the location of the rclone configuration file, to override the 13022 default. E.g. rclone config --config="rclone.conf". 13023 13024 The exact default is a bit complex to describe, due to changes 13025 introduced through different versions of rclone while preserving 13026 backwards compatibility, but in most cases it is as simple as: 13027 13028 - %APPDATA%/rclone/rclone.conf on Windows 13029 - ~/.config/rclone/rclone.conf on other 13030 13031 The complete logic is as follows: Rclone will look for an existing 13032 configuration file in any of the following locations, in priority order: 13033 13034 1. rclone.conf (in program directory, where rclone executable is) 13035 2. %APPDATA%/rclone/rclone.conf (only on Windows) 13036 3. $XDG_CONFIG_HOME/rclone/rclone.conf (on all systems, including 13037 Windows) 13038 4. ~/.config/rclone/rclone.conf (see below for explanation of ~ symbol) 13039 5. ~/.rclone.conf 13040 13041 If no existing configuration file is found, then a new one will be 13042 created in the following location: 13043 13044 - On Windows: Location 2 listed above, except in the unlikely event 13045 that APPDATA is not defined, then location 4 is used instead. 13046 - On Unix: Location 3 if XDG_CONFIG_HOME is defined, else location 4. 13047 - Fallback to location 5 (on all OS), when the rclone directory cannot 13048 be created, but if also a home directory was not found then path 13049 .rclone.conf relative to current working directory will be used as a 13050 final resort. 13051 13052 The ~ symbol in paths above represent the home directory of the current 13053 user on any OS, and the value is defined as following: 13054 13055 - On Windows: %HOME% if defined, else %USERPROFILE%, or else 13056 %HOMEDRIVE%\%HOMEPATH%. 13057 - On Unix: $HOME if defined, else by looking up current user in 13058 OS-specific user database (e.g. passwd file), or else use the result 13059 from shell command cd && pwd. 13060 13061 If you run rclone config file you will see where the default location is 13062 for you. 13063 13064 The fact that an existing file rclone.conf in the same directory as the 13065 rclone executable is always preferred, means that it is easy to run in 13066 "portable" mode by downloading rclone executable to a writable directory 13067 and then create an empty file rclone.conf in the same directory. 13068 13069 If the location is set to empty string "" or path to a file with name 13070 notfound, or the os null device represented by value NUL on Windows and 13071 /dev/null on Unix systems, then rclone will keep the config file in 13072 memory only. 13073 13074 The file format is basic INI: Sections of text, led by a [section] 13075 header and followed by key=value entries on separate lines. In rclone 13076 each remote is represented by its own section, where the section name 13077 defines the name of the remote. Options are specified as the key=value 13078 entries, where the key is the option name without the --backend- prefix, 13079 in lowercase and with _ instead of -. E.g. option --mega-hard-delete 13080 corresponds to key hard_delete. Only backend options can be specified. A 13081 special, and required, key type identifies the storage system, where the 13082 value is the internal lowercase name as returned by command 13083 rclone help backends. Comments are indicated by ; or # at the beginning 13084 of a line. 13085 13086 Example: 13087 13088 [megaremote] 13089 type = mega 13090 user = you@example.com 13091 pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH 13092 13093 Note that passwords are in obscured form. Also, many storage systems 13094 uses token-based authentication instead of passwords, and this requires 13095 additional steps. It is easier, and safer, to use the interactive 13096 command rclone config instead of manually editing the configuration 13097 file. 13098 13099 The configuration file will typically contain login information, and 13100 should therefore have restricted permissions so that only the current 13101 user can read it. Rclone tries to ensure this when it writes the file. 13102 You may also choose to encrypt the file. 13103 13104 When token-based authentication are used, the configuration file must be 13105 writable, because rclone needs to update the tokens inside it. 13106 13107 To reduce risk of corrupting an existing configuration file, rclone will 13108 not write directly to it when saving changes. Instead it will first 13109 write to a new, temporary, file. If a configuration file already 13110 existed, it will (on Unix systems) try to mirror its permissions to the 13111 new file. Then it will rename the existing file to a temporary name as 13112 backup. Next, rclone will rename the new file to the correct name, 13113 before finally cleaning up by deleting the backup file. 13114 13115 If the configuration file path used by rclone is a symbolic link, then 13116 this will be evaluated and rclone will write to the resolved path, 13117 instead of overwriting the symbolic link. Temporary files used in the 13118 process (described above) will be written to the same parent directory 13119 as that of the resolved configuration file, but if this directory is 13120 also a symbolic link it will not be resolved and the temporary files 13121 will be written to the location of the directory symbolic link. 13122 13123 --contimeout=TIME 13124 13125 Set the connection timeout. This should be in go time format which looks 13126 like 5s for 5 seconds, 10m for 10 minutes, or 3h30m. 13127 13128 The connection timeout is the amount of time rclone will wait for a 13129 connection to go through to a remote object storage system. It is 1m by 13130 default. 13131 13132 --copy-dest=DIR 13133 13134 When using sync, copy or move DIR is checked in addition to the 13135 destination for files. If a file identical to the source is found that 13136 file is server-side copied from DIR to the destination. This is useful 13137 for incremental backup. 13138 13139 The remote in use must support server-side copy and you must use the 13140 same remote as the destination of the sync. The compare directory must 13141 not overlap the destination directory. 13142 13143 See --compare-dest and --backup-dir. 13144 13145 --dedupe-mode MODE 13146 13147 Mode to run dedupe command in. One of interactive, skip, first, newest, 13148 oldest, rename. The default is interactive. 13149 See the dedupe command for more information as to what these options 13150 mean. 13151 13152 --default-time TIME 13153 13154 If a file or directory does have a modification time rclone can read 13155 then rclone will display this fixed time instead. 13156 13157 The default is 2000-01-01 00:00:00 UTC. This can be configured in any of 13158 the ways shown in the time or duration options. 13159 13160 For example --default-time 2020-06-01 to set the default time to the 1st 13161 of June 2020 or --default-time 0s to set the default time to the time 13162 rclone started up. 13163 13164 --disable FEATURE,FEATURE,... 13165 13166 This disables a comma separated list of optional features. For example 13167 to disable server-side move and server-side copy use: 13168 13169 --disable move,copy 13170 13171 The features can be put in any case. 13172 13173 To see a list of which features can be disabled use: 13174 13175 --disable help 13176 13177 The features a remote has can be seen in JSON format with: 13178 13179 rclone backend features remote: 13180 13181 See the overview features and optional features to get an idea of which 13182 feature does what. 13183 13184 Note that some features can be set to true if they are true/false 13185 feature flag features by prefixing them with !. For example the 13186 CaseInsensitive feature can be forced to false with 13187 --disable CaseInsensitive and forced to true with 13188 --disable '!CaseInsensitive'. In general it isn't a good idea doing this 13189 but it may be useful in extremis. 13190 13191 (Note that ! is a shell command which you will need to escape with 13192 single quotes or a backslash on unix like platforms.) 13193 13194 This flag can be useful for debugging and in exceptional circumstances 13195 (e.g. Google Drive limiting the total volume of Server Side Copies to 13196 100 GiB/day). 13197 13198 --disable-http2 13199 13200 This stops rclone from trying to use HTTP/2 if available. This can 13201 sometimes speed up transfers due to a problem in the Go standard 13202 library. 13203 13204 --dscp VALUE 13205 13206 Specify a DSCP value or name to use in connections. This could help QoS 13207 system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are 13208 allowed. 13209 13210 See the description of differentiated services to get an idea of this 13211 field. Setting this to 1 (LE) to identify the flow to SCAVENGER class 13212 can avoid occupying too much bandwidth in a network with DiffServ 13213 support (RFC 8622). 13214 13215 For example, if you configured QoS on router to handle LE properly. 13216 Running: 13217 13218 rclone copy --dscp LE from:/from to:/to 13219 13220 would make the priority lower than usual internet flows. 13221 13222 This option has no effect on Windows (see golang/go#42728). 13223 13224 -n, --dry-run 13225 13226 Do a trial run with no permanent changes. Use this to see what rclone 13227 would do without actually doing it. Useful when setting up the sync 13228 command which deletes files in the destination. 13229 13230 --expect-continue-timeout=TIME 13231 13232 This specifies the amount of time to wait for a server's first response 13233 headers after fully writing the request headers if the request has an 13234 "Expect: 100-continue" header. Not all backends support using this. 13235 13236 Zero means no timeout and causes the body to be sent immediately, 13237 without waiting for the server to approve. This time does not include 13238 the time to send the request header. 13239 13240 The default is 1s. Set to 0 to disable. 13241 13242 --error-on-no-transfer 13243 13244 By default, rclone will exit with return code 0 if there were no errors. 13245 13246 This option allows rclone to return exit code 9 if no files were 13247 transferred between the source and destination. This allows using rclone 13248 in scripts, and triggering follow-on actions if data was copied, or 13249 skipping if not. 13250 13251 NB: Enabling this option turns a usually non-fatal error into a 13252 potentially fatal one - please check and adjust your scripts 13253 accordingly! 13254 13255 --fix-case 13256 13257 Normally, a sync to a case insensitive dest (such as macOS / Windows) 13258 will not result in a matching filename if the source and dest filenames 13259 have casing differences but are otherwise identical. For example, 13260 syncing hello.txt to HELLO.txt will normally result in the dest filename 13261 remaining HELLO.txt. If --fix-case is set, then HELLO.txt will be 13262 renamed to hello.txt to match the source. 13263 13264 NB: - directory names with incorrect casing will also be fixed - 13265 --fix-case will be ignored if --immutable is set - using 13266 --local-case-sensitive instead is not advisable; it will cause HELLO.txt 13267 to get deleted! - the old dest filename must not be excluded by filters. 13268 Be especially careful with --files-from, which does not respect 13269 --ignore-case! - on remotes that do not support server-side move, 13270 --fix-case will require downloading the file and re-uploading it. To 13271 avoid this, do not use --fix-case. 13272 13273 --fs-cache-expire-duration=TIME 13274 13275 When using rclone via the API rclone caches created remotes for 5 13276 minutes by default in the "fs cache". This means that if you do repeated 13277 actions on the same remote then rclone won't have to build it again from 13278 scratch, which makes it more efficient. 13279 13280 This flag sets the time that the remotes are cached for. If you set it 13281 to 0 (or negative) then rclone won't cache the remotes at all. 13282 13283 Note that if you use some flags, eg --backup-dir and if this is set to 0 13284 rclone may build two remotes (one for the source or destination and one 13285 for the --backup-dir where it may have only built one before. 13286 13287 --fs-cache-expire-interval=TIME 13288 13289 This controls how often rclone checks for cached remotes to expire. See 13290 the --fs-cache-expire-duration documentation above for more info. The 13291 default is 60s, set to 0 to disable expiry. 13292 13293 --header 13294 13295 Add an HTTP header for all transactions. The flag can be repeated to add 13296 multiple headers. 13297 13298 If you want to add headers only for uploads use --header-upload and if 13299 you want to add headers only for downloads use --header-download. 13300 13301 This flag is supported for all HTTP based backends even those not 13302 supported by --header-upload and --header-download so may be used as a 13303 workaround for those with care. 13304 13305 rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes" 13306 13307 --header-download 13308 13309 Add an HTTP header for all download transactions. The flag can be 13310 repeated to add multiple headers. 13311 13312 rclone sync --interactive s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar" 13313 13314 See the GitHub issue here for currently supported backends. 13315 13316 --header-upload 13317 13318 Add an HTTP header for all upload transactions. The flag can be repeated 13319 to add multiple headers. 13320 13321 rclone sync --interactive ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar" 13322 13323 See the GitHub issue here for currently supported backends. 13324 13325 --human-readable 13326 13327 Rclone commands output values for sizes (e.g. number of bytes) and 13328 counts (e.g. number of files) either as raw numbers, or in 13329 human-readable format. 13330 13331 In human-readable format the values are scaled to larger units, 13332 indicated with a suffix shown after the value, and rounded to three 13333 decimals. Rclone consistently uses binary units (powers of 2) for sizes 13334 and decimal units (powers of 10) for counts. The unit prefix for size is 13335 according to IEC standard notation, e.g. Ki for kibi. Used with byte 13336 unit, 1 KiB means 1024 Byte. In list type of output, only the unit 13337 prefix appended to the value (e.g. 9.762Ki), while in more textual 13338 output the full unit is shown (e.g. 9.762 KiB). For counts the SI 13339 standard notation is used, e.g. prefix k for kilo. Used with file 13340 counts, 1k means 1000 files. 13341 13342 The various list commands output raw numbers by default. Option 13343 --human-readable will make them output values in human-readable format 13344 instead (with the short unit prefix). 13345 13346 The about command outputs human-readable by default, with a 13347 command-specific option --full to output the raw numbers instead. 13348 13349 Command size outputs both human-readable and raw numbers in the same 13350 output. 13351 13352 The tree command also considers --human-readable, but it will not use 13353 the exact same notation as the other commands: It rounds to one decimal, 13354 and uses single letter suffix, e.g. K instead of Ki. The reason for this 13355 is that it relies on an external library. 13356 13357 The interactive command ncdu shows human-readable by default, and 13358 responds to key u for toggling human-readable format. 13359 13360 --ignore-case-sync 13361 13362 Using this option will cause rclone to ignore the case of the files when 13363 synchronizing so files will not be copied/synced when the existing 13364 filenames are the same, even if the casing is different. 13365 13366 --ignore-checksum 13367 13368 Normally rclone will check that the checksums of transferred files 13369 match, and give an error "corrupted on transfer" if they don't. 13370 13371 You can use this option to skip that check. You should only use it if 13372 you have had the "corrupted on transfer" error message and you are sure 13373 you might want to transfer potentially corrupted data. 13374 13375 --ignore-existing 13376 13377 Using this option will make rclone unconditionally skip all files that 13378 exist on the destination, no matter the content of these files. 13379 13380 While this isn't a generally recommended option, it can be useful in 13381 cases where your files change due to encryption. However, it cannot 13382 correct partial transfers in case a transfer was interrupted. 13383 13384 When performing a move/moveto command, this flag will leave skipped 13385 files in the source location unchanged when a file with the same name 13386 exists on the destination. 13387 13388 --ignore-size 13389 13390 Normally rclone will look at modification time and size of files to see 13391 if they are equal. If you set this flag then rclone will check only the 13392 modification time. If --checksum is set then it only checks the 13393 checksum. 13394 13395 It will also cause rclone to skip verifying the sizes are the same after 13396 transfer. 13397 13398 This can be useful for transferring files to and from OneDrive which 13399 occasionally misreports the size of image files (see #399 for more 13400 info). 13401 13402 -I, --ignore-times 13403 13404 Using this option will cause rclone to unconditionally upload all files 13405 regardless of the state of files on the destination. 13406 13407 Normally rclone would skip any files that have the same modification 13408 time and are the same size (or have the same checksum if using 13409 --checksum). 13410 13411 --immutable 13412 13413 Treat source and destination files as immutable and disallow 13414 modification. 13415 13416 With this option set, files will be created and deleted as requested, 13417 but existing files will never be updated. If an existing file does not 13418 match between the source and destination, rclone will give the error 13419 Source and destination exist but do not match: immutable file modified. 13420 13421 Note that only commands which transfer files (e.g. sync, copy, move) are 13422 affected by this behavior, and only modification is disallowed. Files 13423 may still be deleted explicitly (e.g. delete, purge) or implicitly (e.g. 13424 sync, move). Use copy --immutable if it is desired to avoid deletion as 13425 well as modification. 13426 13427 This can be useful as an additional layer of protection for immutable or 13428 append-only data sets (notably backup archives), where modification 13429 implies corruption and should not be propagated. 13430 13431 --inplace 13432 13433 The --inplace flag changes the behaviour of rclone when uploading files 13434 to some backends (backends with the PartialUploads feature flag set) 13435 such as: 13436 13437 - local 13438 - ftp 13439 - sftp 13440 13441 Without --inplace (the default) rclone will first upload to a temporary 13442 file with an extension like this, where XXXXXX represents a random 13443 string and .partial is --partial-suffix value (.partial by default). 13444 13445 original-file-name.XXXXXX.partial 13446 13447 (rclone will make sure the final name is no longer than 100 characters 13448 by truncating the original-file-name part if necessary). 13449 13450 When the upload is complete, rclone will rename the .partial file to the 13451 correct name, overwriting any existing file at that point. If the upload 13452 fails then the .partial file will be deleted. 13453 13454 This prevents other users of the backend from seeing partially uploaded 13455 files in their new names and prevents overwriting the old file until the 13456 new one is completely uploaded. 13457 13458 If the --inplace flag is supplied, rclone will upload directly to the 13459 final name without creating a .partial file. 13460 13461 This means that an incomplete file will be visible in the directory 13462 listings while the upload is in progress and any existing files will be 13463 overwritten as soon as the upload starts. If the transfer fails then the 13464 file will be deleted. This can cause data loss of the existing file if 13465 the transfer fails. 13466 13467 Note that on the local file system if you don't use --inplace hard links 13468 (Unix only) will be broken. And if you do use --inplace you won't be 13469 able to update in use executables. 13470 13471 Note also that versions of rclone prior to v1.63.0 behave as if the 13472 --inplace flag is always supplied. 13473 13474 -i, --interactive 13475 13476 This flag can be used to tell rclone that you wish a manual confirmation 13477 before destructive operations. 13478 13479 It is recommended that you use this flag while learning rclone 13480 especially with rclone sync. 13481 13482 For example 13483 13484 $ rclone delete --interactive /tmp/dir 13485 rclone: delete "important-file.txt"? 13486 y) Yes, this is OK (default) 13487 n) No, skip this 13488 s) Skip all delete operations with no more questions 13489 !) Do all delete operations with no more questions 13490 q) Exit rclone now. 13491 y/n/s/!/q> n 13492 13493 The options mean 13494 13495 - y: Yes, this operation should go ahead. You can also press Return 13496 for this to happen. You'll be asked every time unless you choose s 13497 or !. 13498 - n: No, do not do this operation. You'll be asked every time unless 13499 you choose s or !. 13500 - s: Skip all the following operations of this type with no more 13501 questions. This takes effect until rclone exits. If there are any 13502 different kind of operations you'll be prompted for them. 13503 - !: Do all the following operations with no more questions. Useful if 13504 you've decided that you don't mind rclone doing that kind of 13505 operation. This takes effect until rclone exits . If there are any 13506 different kind of operations you'll be prompted for them. 13507 - q: Quit rclone now, just in case! 13508 13509 --leave-root 13510 13511 During rmdirs it will not remove root directory, even if it's empty. 13512 13513 --log-file=FILE 13514 13515 Log all of rclone's output to FILE. This is not active by default. This 13516 can be useful for tracking down problems with syncs in combination with 13517 the -v flag. See the Logging section for more info. 13518 13519 If FILE exists then rclone will append to it. 13520 13521 Note that if you are using the logrotate program to manage rclone's 13522 logs, then you should use the copytruncate option as rclone doesn't have 13523 a signal to rotate logs. 13524 13525 --log-format LIST 13526 13527 Comma separated list of log format options. Accepted options are date, 13528 time, microseconds, pid, longfile, shortfile, UTC. Any other keywords 13529 will be silently ignored. pid will tag log messages with process 13530 identifier which useful with rclone mount --daemon. Other accepted 13531 options are explained in the go documentation. The default log format is 13532 "date,time". 13533 13534 --log-level LEVEL 13535 13536 This sets the log level for rclone. The default log level is NOTICE. 13537 13538 DEBUG is equivalent to -vv. It outputs lots of debug info - useful for 13539 bug reports and really finding out what rclone is doing. 13540 13541 INFO is equivalent to -v. It outputs information about each transfer and 13542 prints stats once a minute by default. 13543 13544 NOTICE is the default log level if no logging flags are supplied. It 13545 outputs very little when things are working normally. It outputs 13546 warnings and significant events. 13547 13548 ERROR is equivalent to -q. It only outputs error messages. 13549 13550 --use-json-log 13551 13552 This switches the log format to JSON for rclone. The fields of json log 13553 are level, msg, source, time. 13554 13555 --low-level-retries NUMBER 13556 13557 This controls the number of low level retries rclone does. 13558 13559 A low level retry is used to retry a failing operation - typically one 13560 HTTP request. This might be uploading a chunk of a big file for example. 13561 You will see low level retries in the log with the -v flag. 13562 13563 This shouldn't need to be changed from the default in normal operations. 13564 However, if you get a lot of low level retries you may wish to reduce 13565 the value so rclone moves on to a high level retry (see the --retries 13566 flag) quicker. 13567 13568 Disable low level retries with --low-level-retries 1. 13569 13570 --max-backlog=N 13571 13572 This is the maximum allowable backlog of files in a sync/copy/move 13573 queued for being checked or transferred. 13574 13575 This can be set arbitrarily large. It will only use memory when the 13576 queue is in use. Note that it will use in the order of N KiB of memory 13577 when the backlog is in use. 13578 13579 Setting this large allows rclone to calculate how many files are pending 13580 more accurately, give a more accurate estimated finish time and make 13581 --order-by work more accurately. 13582 13583 Setting this small will make rclone more synchronous to the listings of 13584 the remote which may be desirable. 13585 13586 Setting this to a negative number will make the backlog as large as 13587 possible. 13588 13589 --max-delete=N 13590 13591 This tells rclone not to delete more than N files. If that limit is 13592 exceeded then a fatal error will be generated and rclone will stop the 13593 operation in progress. 13594 13595 --max-delete-size=SIZE 13596 13597 Rclone will stop deleting files when the total size of deletions has 13598 reached the size specified. It defaults to off. 13599 13600 If that limit is exceeded then a fatal error will be generated and 13601 rclone will stop the operation in progress. 13602 13603 --max-depth=N 13604 13605 This modifies the recursion depth for all the commands except purge. 13606 13607 So if you do rclone --max-depth 1 ls remote:path you will see only the 13608 files in the top level directory. Using --max-depth 2 means you will see 13609 all the files in first two directory levels and so on. 13610 13611 For historical reasons the lsd command defaults to using a --max-depth 13612 of 1 - you can override this with the command line flag. 13613 13614 You can use this command to disable recursion (with --max-depth 1). 13615 13616 Note that if you use this with sync and --delete-excluded the files not 13617 recursed through are considered excluded and will be deleted on the 13618 destination. Test first with --dry-run if you are not sure what will 13619 happen. 13620 13621 --max-duration=TIME 13622 13623 Rclone will stop transferring when it has run for the duration 13624 specified. Defaults to off. 13625 13626 When the limit is reached all transfers will stop immediately. Use 13627 --cutoff-mode to modify this behaviour. 13628 13629 Rclone will exit with exit code 10 if the duration limit is reached. 13630 13631 --max-transfer=SIZE 13632 13633 Rclone will stop transferring when it has reached the size specified. 13634 Defaults to off. 13635 13636 When the limit is reached all transfers will stop immediately. Use 13637 --cutoff-mode to modify this behaviour. 13638 13639 Rclone will exit with exit code 8 if the transfer limit is reached. 13640 13641 --cutoff-mode=hard|soft|cautious 13642 13643 This modifies the behavior of --max-transfer and --max-duration Defaults 13644 to --cutoff-mode=hard. 13645 13646 Specifying --cutoff-mode=hard will stop transferring immediately when 13647 Rclone reaches the limit. 13648 13649 Specifying --cutoff-mode=soft will stop starting new transfers when 13650 Rclone reaches the limit. 13651 13652 Specifying --cutoff-mode=cautious will try to prevent Rclone from 13653 reaching the limit. Only applicable for --max-transfer 13654 13655 -M, --metadata 13656 13657 Setting this flag enables rclone to copy the metadata from the source to 13658 the destination. For local backends this is ownership, permissions, 13659 xattr etc. See the metadata section for more info. 13660 13661 --metadata-mapper SpaceSepList 13662 13663 If you supply the parameter --metadata-mapper /path/to/program then 13664 rclone will use that program to map metadata from source object to 13665 destination object. 13666 13667 The argument to this flag should be a command with an optional space 13668 separated list of arguments. If one of the arguments has a space in then 13669 enclose it in ", if you want a literal " in an argument then enclose the 13670 argument in " and double the ". See CSV encoding for more info. 13671 13672 --metadata-mapper "python bin/test_metadata_mapper.py" 13673 --metadata-mapper 'python bin/test_metadata_mapper.py "argument with a space"' 13674 --metadata-mapper 'python bin/test_metadata_mapper.py "argument with ""two"" quotes"' 13675 13676 This uses a simple JSON based protocol with input on STDIN and output on 13677 STDOUT. This will be called for every file and directory copied and may 13678 be called concurrently. 13679 13680 The program's job is to take a metadata blob on the input and turn it 13681 into a metadata blob on the output suitable for the destination backend. 13682 13683 Input to the program (via STDIN) might look like this. This provides 13684 some context for the Metadata which may be important. 13685 13686 - SrcFs is the config string for the remote that the object is 13687 currently on. 13688 - SrcFsType is the name of the source backend. 13689 - DstFs is the config string for the remote that the object is being 13690 copied to 13691 - DstFsType is the name of the destination backend. 13692 - Remote is the path of the object relative to the root. 13693 - Size, MimeType, ModTime are attributes of the object. 13694 - IsDir is true if this is a directory (not yet implemented). 13695 - ID is the source ID of the object if known. 13696 - Metadata is the backend specific metadata as described in the 13697 backend docs. 13698 13699 { 13700 "SrcFs": "gdrive:", 13701 "SrcFsType": "drive", 13702 "DstFs": "newdrive:user", 13703 "DstFsType": "onedrive", 13704 "Remote": "test.txt", 13705 "Size": 6, 13706 "MimeType": "text/plain; charset=utf-8", 13707 "ModTime": "2022-10-11T17:53:10.286745272+01:00", 13708 "IsDir": false, 13709 "ID": "xyz", 13710 "Metadata": { 13711 "btime": "2022-10-11T16:53:11Z", 13712 "content-type": "text/plain; charset=utf-8", 13713 "mtime": "2022-10-11T17:53:10.286745272+01:00", 13714 "owner": "user1@domain1.com", 13715 "permissions": "...", 13716 "description": "my nice file", 13717 "starred": "false" 13718 } 13719 } 13720 13721 The program should then modify the input as desired and send it to 13722 STDOUT. The returned Metadata field will be used in its entirety for the 13723 destination object. Any other fields will be ignored. Note in this 13724 example we translate user names and permissions and add something to the 13725 description: 13726 13727 { 13728 "Metadata": { 13729 "btime": "2022-10-11T16:53:11Z", 13730 "content-type": "text/plain; charset=utf-8", 13731 "mtime": "2022-10-11T17:53:10.286745272+01:00", 13732 "owner": "user1@domain2.com", 13733 "permissions": "...", 13734 "description": "my nice file [migrated from domain1]", 13735 "starred": "false" 13736 } 13737 } 13738 13739 Metadata can be removed here too. 13740 13741 An example python program might look something like this to implement 13742 the above transformations. 13743 13744 import sys, json 13745 13746 i = json.load(sys.stdin) 13747 metadata = i["Metadata"] 13748 # Add tag to description 13749 if "description" in metadata: 13750 metadata["description"] += " [migrated from domain1]" 13751 else: 13752 metadata["description"] = "[migrated from domain1]" 13753 # Modify owner 13754 if "owner" in metadata: 13755 metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com") 13756 o = { "Metadata": metadata } 13757 json.dump(o, sys.stdout, indent="\t") 13758 13759 You can find this example (slightly expanded) in the rclone source code 13760 at bin/test_metadata_mapper.py. 13761 13762 If you want to see the input to the metadata mapper and the output 13763 returned from it in the log you can use -vv --dump mapper. 13764 13765 See the metadata section for more info. 13766 13767 --metadata-set key=value 13768 13769 Add metadata key = value when uploading. This can be repeated as many 13770 times as required. See the metadata section for more info. 13771 13772 --modify-window=TIME 13773 13774 When checking whether a file has been modified, this is the maximum 13775 allowed time difference that a file can have and still be considered 13776 equivalent. 13777 13778 The default is 1ns unless this is overridden by a remote. For example OS 13779 X only stores modification times to the nearest second so if you are 13780 reading and writing to an OS X filing system this will be 1s by default. 13781 13782 This command line flag allows you to override that computed default. 13783 13784 --multi-thread-write-buffer-size=SIZE 13785 13786 When transferring with multiple threads, rclone will buffer SIZE bytes 13787 in memory before writing to disk for each thread. 13788 13789 This can improve performance if the underlying filesystem does not deal 13790 well with a lot of small writes in different positions of the file, so 13791 if you see transfers being limited by disk write speed, you might want 13792 to experiment with different values. Specially for magnetic drives and 13793 remote file systems a higher value can be useful. 13794 13795 Nevertheless, the default of 128k should be fine for almost all use 13796 cases, so before changing it ensure that network is not really your 13797 bottleneck. 13798 13799 As a final hint, size is not the only factor: block size (or similar 13800 concept) can have an impact. In one case, we observed that exact 13801 multiples of 16k performed much better than other values. 13802 13803 --multi-thread-chunk-size=SizeSuffix 13804 13805 Normally the chunk size for multi thread transfers is set by the 13806 backend. However some backends such as local and smb (which implement 13807 OpenWriterAt but not OpenChunkWriter) don't have a natural chunk size. 13808 13809 In this case the value of this option is used (default 64Mi). 13810 13811 --multi-thread-cutoff=SIZE 13812 13813 When transferring files above SIZE to capable backends, rclone will use 13814 multiple threads to transfer the file (default 256M). 13815 13816 Capable backends are marked in the overview as MultithreadUpload. (They 13817 need to implement either the OpenWriterAt or OpenChunkWriter internal 13818 interfaces). These include include, local, s3, azureblob, b2, 13819 oracleobjectstorage and smb at the time of writing. 13820 13821 On the local disk, rclone preallocates the file (using 13822 fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on 13823 Windows both of which takes no time) then each thread writes directly 13824 into the file at the correct place. This means that rclone won't create 13825 fragmented or sparse files and there won't be any assembly time at the 13826 end of the transfer. 13827 13828 The number of threads used to transfer is controlled by 13829 --multi-thread-streams. 13830 13831 Use -vv if you wish to see info about the threads. 13832 13833 This will work with the sync/copy/move commands and friends 13834 copyto/moveto. Multi thread transfers will be used with rclone mount and 13835 rclone serve if --vfs-cache-mode is set to writes or above. 13836 13837 NB that this only works with supported backends as the destination but 13838 will work with any backend as the source. 13839 13840 NB that multi-thread copies are disabled for local to local copies as 13841 they are faster without unless --multi-thread-streams is set explicitly. 13842 13843 NB on Windows using multi-thread transfers to the local disk will cause 13844 the resulting files to be sparse. Use --local-no-sparse to disable 13845 sparse files (which may cause long delays at the start of transfers) or 13846 disable multi-thread transfers with --multi-thread-streams 0 13847 13848 --multi-thread-streams=N 13849 13850 When using multi thread transfers (see above --multi-thread-cutoff) this 13851 sets the number of streams to use. Set to 0 to disable multi thread 13852 transfers (Default 4). 13853 13854 If the backend has a --backend-upload-concurrency setting (eg 13855 --s3-upload-concurrency) then this setting will be used as the number of 13856 transfers instead if it is larger than the value of 13857 --multi-thread-streams or --multi-thread-streams isn't set. 13858 13859 --no-check-dest 13860 13861 The --no-check-dest can be used with move or copy and it causes rclone 13862 not to check the destination at all when copying files. 13863 13864 This means that: 13865 13866 - the destination is not listed minimising the API calls 13867 - files are always transferred 13868 - this can cause duplicates on remotes which allow it (e.g. Google 13869 Drive) 13870 - --retries 1 is recommended otherwise you'll transfer everything 13871 again on a retry 13872 13873 This flag is useful to minimise the transactions if you know that none 13874 of the files are on the destination. 13875 13876 This is a specialized flag which should be ignored by most users! 13877 13878 --no-gzip-encoding 13879 13880 Don't set Accept-Encoding: gzip. This means that rclone won't ask the 13881 server for compressed files automatically. Useful if you've set the 13882 server to return files with Content-Encoding: gzip but you uploaded 13883 compressed files. 13884 13885 There is no need to set this in normal operation, and doing so will 13886 decrease the network transfer efficiency of rclone. 13887 13888 --no-traverse 13889 13890 The --no-traverse flag controls whether the destination file system is 13891 traversed when using the copy or move commands. --no-traverse is not 13892 compatible with sync and will be ignored if you supply it with sync. 13893 13894 If you are only copying a small number of files (or are filtering most 13895 of the files) and/or have a large number of files on the destination 13896 then --no-traverse will stop rclone listing the destination and save 13897 time. 13898 13899 However, if you are copying a large number of files, especially if you 13900 are doing a copy where lots of the files under consideration haven't 13901 changed and won't need copying then you shouldn't use --no-traverse. 13902 13903 See rclone copy for an example of how to use it. 13904 13905 --no-unicode-normalization 13906 13907 Don't normalize unicode characters in filenames during the sync routine. 13908 13909 Sometimes, an operating system will store filenames containing unicode 13910 parts in their decomposed form (particularly macOS). Some cloud storage 13911 systems will then recompose the unicode, resulting in duplicate files if 13912 the data is ever copied back to a local filesystem. 13913 13914 Using this flag will disable that functionality, treating each unicode 13915 character as unique. For example, by default é and é will be normalized 13916 into the same character. With --no-unicode-normalization they will be 13917 treated as unique characters. 13918 13919 --no-update-modtime 13920 13921 When using this flag, rclone won't update modification times of remote 13922 files if they are incorrect as it would normally. 13923 13924 This can be used if the remote is being synced with another tool also 13925 (e.g. the Google Drive client). 13926 13927 --no-update-dir-modtime 13928 13929 When using this flag, rclone won't update modification times of remote 13930 directories if they are incorrect as it would normally. 13931 13932 --order-by string 13933 13934 The --order-by flag controls the order in which files in the backlog are 13935 processed in rclone sync, rclone copy and rclone move. 13936 13937 The order by string is constructed like this. The first part describes 13938 what aspect is being measured: 13939 13940 - size - order by the size of the files 13941 - name - order by the full path of the files 13942 - modtime - order by the modification date of the files 13943 13944 This can have a modifier appended with a comma: 13945 13946 - ascending or asc - order so that the smallest (or oldest) is 13947 processed first 13948 - descending or desc - order so that the largest (or newest) is 13949 processed first 13950 - mixed - order so that the smallest is processed first for some 13951 threads and the largest for others 13952 13953 If the modifier is mixed then it can have an optional percentage (which 13954 defaults to 50), e.g. size,mixed,25 which means that 25% of the threads 13955 should be taking the smallest items and 75% the largest. The threads 13956 which take the smallest first will always take the smallest first and 13957 likewise the largest first threads. The mixed mode can be useful to 13958 minimise the transfer time when you are transferring a mixture of large 13959 and small files - the large files are guaranteed upload threads and 13960 bandwidth and the small files will be processed continuously. 13961 13962 If no modifier is supplied then the order is ascending. 13963 13964 For example 13965 13966 - --order-by size,desc - send the largest files first 13967 - --order-by modtime,ascending - send the oldest files first 13968 - --order-by name - send the files with alphabetically by path first 13969 13970 If the --order-by flag is not supplied or it is supplied with an empty 13971 string then the default ordering will be used which is as scanned. With 13972 --checkers 1 this is mostly alphabetical, however with the default 13973 --checkers 8 it is somewhat random. 13974 13975 Limitations 13976 13977 The --order-by flag does not do a separate pass over the data. This 13978 means that it may transfer some files out of the order specified if 13979 13980 - there are no files in the backlog or the source has not been fully 13981 scanned yet 13982 - there are more than --max-backlog files in the backlog 13983 13984 Rclone will do its best to transfer the best file it has so in practice 13985 this should not cause a problem. Think of --order-by as being more of a 13986 best efforts flag rather than a perfect ordering. 13987 13988 If you want perfect ordering then you will need to specify --check-first 13989 which will find all the files which need transferring first before 13990 transferring any. 13991 13992 --partial-suffix 13993 13994 When --inplace is not used, it causes rclone to use the --partial-suffix 13995 as suffix for temporary files. 13996 13997 Suffix length limit is 16 characters. 13998 13999 The default is .partial. 14000 14001 --password-command SpaceSepList 14002 14003 This flag supplies a program which should supply the config password 14004 when run. This is an alternative to rclone prompting for the password or 14005 setting the RCLONE_CONFIG_PASS variable. 14006 14007 The argument to this should be a command with a space separated list of 14008 arguments. If one of the arguments has a space in then enclose it in ", 14009 if you want a literal " in an argument then enclose the argument in " 14010 and double the ". See CSV encoding for more info. 14011 14012 Eg 14013 14014 --password-command "echo hello" 14015 --password-command 'echo "hello with space"' 14016 --password-command 'echo "hello with ""quotes"" and space"' 14017 14018 See the Configuration Encryption for more info. 14019 14020 See a Windows PowerShell example on the Wiki. 14021 14022 -P, --progress 14023 14024 This flag makes rclone update the stats in a static block in the 14025 terminal providing a realtime overview of the transfer. 14026 14027 Any log messages will scroll above the static block. Log messages will 14028 push the static block down to the bottom of the terminal where it will 14029 stay. 14030 14031 Normally this is updated every 500mS but this period can be overridden 14032 with the --stats flag. 14033 14034 This can be used with the --stats-one-line flag for a simpler display. 14035 14036 Note: On Windows until this bug is fixed all non-ASCII characters will 14037 be replaced with . when --progress is in use. 14038 14039 --progress-terminal-title 14040 14041 This flag, when used with -P/--progress, will print the string ETA: %s 14042 to the terminal title. 14043 14044 -q, --quiet 14045 14046 This flag will limit rclone's output to error messages only. 14047 14048 --refresh-times 14049 14050 The --refresh-times flag can be used to update modification times of 14051 existing files when they are out of sync on backends which don't support 14052 hashes. 14053 14054 This is useful if you uploaded files with the incorrect timestamps and 14055 you now wish to correct them. 14056 14057 This flag is only useful for destinations which don't support hashes 14058 (e.g. crypt). 14059 14060 This can be used any of the sync commands sync, copy or move. 14061 14062 To use this flag you will need to be doing a modification time sync (so 14063 not using --size-only or --checksum). The flag will have no effect when 14064 using --size-only or --checksum. 14065 14066 If this flag is used when rclone comes to upload a file it will check to 14067 see if there is an existing file on the destination. If this file 14068 matches the source with size (and checksum if available) but has a 14069 differing timestamp then instead of re-uploading it, rclone will update 14070 the timestamp on the destination file. If the checksum does not match 14071 rclone will upload the new file. If the checksum is absent (e.g. on a 14072 crypt backend) then rclone will update the timestamp. 14073 14074 Note that some remotes can't set the modification time without 14075 re-uploading the file so this flag is less useful on them. 14076 14077 Normally if you are doing a modification time sync rclone will update 14078 modification times without --refresh-times provided that the remote 14079 supports checksums and the checksums match on the file. However if the 14080 checksums are absent then rclone will upload the file rather than 14081 setting the timestamp as this is the safe behaviour. 14082 14083 --retries int 14084 14085 Retry the entire sync if it fails this many times it fails (default 3). 14086 14087 Some remotes can be unreliable and a few retries help pick up the files 14088 which didn't get transferred because of errors. 14089 14090 Disable retries with --retries 1. 14091 14092 --retries-sleep=TIME 14093 14094 This sets the interval between each retry specified by --retries 14095 14096 The default is 0. Use 0 to disable. 14097 14098 --server-side-across-configs 14099 14100 Allow server-side operations (e.g. copy or move) to work across 14101 different configurations. 14102 14103 This can be useful if you wish to do a server-side copy or move between 14104 two remotes which use the same backend but are configured differently. 14105 14106 Note that this isn't enabled by default because it isn't easy for rclone 14107 to tell if it will work between any two configurations. 14108 14109 --size-only 14110 14111 Normally rclone will look at modification time and size of files to see 14112 if they are equal. If you set this flag then rclone will check only the 14113 size. 14114 14115 This can be useful transferring files from Dropbox which have been 14116 modified by the desktop sync client which doesn't set checksums of 14117 modification times in the same way as rclone. 14118 14119 --stats=TIME 14120 14121 Commands which transfer data (sync, copy, copyto, move, moveto) will 14122 print data transfer stats at regular intervals to show their progress. 14123 14124 This sets the interval. 14125 14126 The default is 1m. Use 0 to disable. 14127 14128 If you set the stats interval then all commands can show stats. This can 14129 be useful when running other commands, check or mount for example. 14130 14131 Stats are logged at INFO level by default which means they won't show at 14132 default log level NOTICE. Use --stats-log-level NOTICE or -v to make 14133 them show. See the Logging section for more info on log levels. 14134 14135 Note that on macOS you can send a SIGINFO (which is normally ctrl-T in 14136 the terminal) to make the stats print immediately. 14137 14138 --stats-file-name-length integer 14139 14140 By default, the --stats output will truncate file names and paths longer 14141 than 40 characters. This is equivalent to providing 14142 --stats-file-name-length 40. Use --stats-file-name-length 0 to disable 14143 any truncation of file names printed by stats. 14144 14145 --stats-log-level string 14146 14147 Log level to show --stats output at. This can be DEBUG, INFO, NOTICE, or 14148 ERROR. The default is INFO. This means at the default level of logging 14149 which is NOTICE the stats won't show - if you want them to then use 14150 --stats-log-level NOTICE. See the Logging section for more info on log 14151 levels. 14152 14153 --stats-one-line 14154 14155 When this is specified, rclone condenses the stats into a single line 14156 showing the most important stats only. 14157 14158 --stats-one-line-date 14159 14160 When this is specified, rclone enables the single-line stats and 14161 prepends the display with a date string. The default is 14162 2006/01/02 15:04:05 - 14163 14164 --stats-one-line-date-format 14165 14166 When this is specified, rclone enables the single-line stats and 14167 prepends the display with a user-supplied date string. The date string 14168 MUST be enclosed in quotes. Follow golang specs for date formatting 14169 syntax. 14170 14171 --stats-unit=bits|bytes 14172 14173 By default, data transfer rates will be printed in bytes per second. 14174 14175 This option allows the data rate to be printed in bits per second. 14176 14177 Data transfer volume will still be reported in bytes. 14178 14179 The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals 14180 1,048,576 bit/s and not 1,000,000 bit/s. 14181 14182 The default is bytes. 14183 14184 --suffix=SUFFIX 14185 14186 When using sync, copy or move any files which would have been 14187 overwritten or deleted will have the suffix added to them. If there is a 14188 file with the same path (after the suffix has been added), then it will 14189 be overwritten. 14190 14191 The remote in use must support server-side move or copy and you must use 14192 the same remote as the destination of the sync. 14193 14194 This is for use with files to add the suffix in the current directory or 14195 with --backup-dir. See --backup-dir for more info. 14196 14197 For example 14198 14199 rclone copy --interactive /path/to/local/file remote:current --suffix .bak 14200 14201 will copy /path/to/local to remote:current, but for any files which 14202 would have been updated or deleted have .bak added. 14203 14204 If using rclone sync with --suffix and without --backup-dir then it is 14205 recommended to put a filter rule in excluding the suffix otherwise the 14206 sync will delete the backup files. 14207 14208 rclone sync --interactive /path/to/local/file remote:current --suffix .bak --exclude "*.bak" 14209 14210 --suffix-keep-extension 14211 14212 When using --suffix, setting this causes rclone put the SUFFIX before 14213 the extension of the files that it backs up rather than after. 14214 14215 So let's say we had --suffix -2019-01-01, without the flag file.txt 14216 would be backed up to file.txt-2019-01-01 and with the flag it would be 14217 backed up to file-2019-01-01.txt. This can be helpful to make sure the 14218 suffixed files can still be opened. 14219 14220 If a file has two (or more) extensions and the second (or subsequent) 14221 extension is recognised as a valid mime type, then the suffix will go 14222 before that extension. So file.tar.gz would be backed up to 14223 file-2019-01-01.tar.gz whereas file.badextension.gz would be backed up 14224 to file.badextension-2019-01-01.gz. 14225 14226 --syslog 14227 14228 On capable OSes (not Windows or Plan9) send all log output to syslog. 14229 14230 This can be useful for running rclone in a script or rclone mount. 14231 14232 --syslog-facility string 14233 14234 If using --syslog this sets the syslog facility (e.g. KERN, USER). See 14235 man syslog for a list of possible facilities. The default facility is 14236 DAEMON. 14237 14238 --temp-dir=DIR 14239 14240 Specify the directory rclone will use for temporary files, to override 14241 the default. Make sure the directory exists and have accessible 14242 permissions. 14243 14244 By default the operating system's temp directory will be used: - On Unix 14245 systems, $TMPDIR if non-empty, else /tmp. - On Windows, the first 14246 non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows 14247 directory. 14248 14249 When overriding the default with this option, the specified path will be 14250 set as value of environment variable TMPDIR on Unix systems and TMP and 14251 TEMP on Windows. 14252 14253 You can use the config paths command to see the current value. 14254 14255 --tpslimit float 14256 14257 Limit transactions per second to this number. Default is 0 which is used 14258 to mean unlimited transactions per second. 14259 14260 A transaction is roughly defined as an API call; its exact meaning will 14261 depend on the backend. For HTTP based backends it is an HTTP 14262 PUT/GET/POST/etc and its response. For FTP/SFTP it is a round trip 14263 transaction over TCP. 14264 14265 For example, to limit rclone to 10 transactions per second use 14266 --tpslimit 10, or to 1 transaction every 2 seconds use --tpslimit 0.5. 14267 14268 Use this when the number of transactions per second from rclone is 14269 causing a problem with the cloud storage provider (e.g. getting you 14270 banned or rate limited). 14271 14272 This can be very useful for rclone mount to control the behaviour of 14273 applications using it. 14274 14275 This limit applies to all HTTP based backends and to the FTP and SFTP 14276 backends. It does not apply to the local backend or the Storj backend. 14277 14278 See also --tpslimit-burst. 14279 14280 --tpslimit-burst int 14281 14282 Max burst of transactions for --tpslimit (default 1). 14283 14284 Normally --tpslimit will do exactly the number of transaction per second 14285 specified. However if you supply --tps-burst then rclone can save up 14286 some transactions from when it was idle giving a burst of up to the 14287 parameter supplied. 14288 14289 For example if you provide --tpslimit-burst 10 then if rclone has been 14290 idle for more than 10*--tpslimit then it can do 10 transactions very 14291 quickly before they are limited again. 14292 14293 This may be used to increase performance of --tpslimit without changing 14294 the long term average number of transactions per second. 14295 14296 --track-renames 14297 14298 By default, rclone doesn't keep track of renamed files, so if you rename 14299 a file locally then sync it to a remote, rclone will delete the old file 14300 on the remote and upload a new copy. 14301 14302 An rclone sync with --track-renames runs like a normal sync, but keeps 14303 track of objects which exist in the destination but not in the source 14304 (which would normally be deleted), and which objects exist in the source 14305 but not the destination (which would normally be transferred). These 14306 objects are then candidates for renaming. 14307 14308 After the sync, rclone matches up the source only and destination only 14309 objects using the --track-renames-strategy specified and either renames 14310 the destination object or transfers the source and deletes the 14311 destination object. --track-renames is stateless like all of rclone's 14312 syncs. 14313 14314 To use this flag the destination must support server-side copy or 14315 server-side move, and to use a hash based --track-renames-strategy (the 14316 default) the source and the destination must have a compatible hash. 14317 14318 If the destination does not support server-side copy or move, rclone 14319 will fall back to the default behaviour and log an error level message 14320 to the console. 14321 14322 Encrypted destinations are not currently supported by --track-renames if 14323 --track-renames-strategy includes hash. 14324 14325 Note that --track-renames is incompatible with --no-traverse and that it 14326 uses extra memory to keep track of all the rename candidates. 14327 14328 Note also that --track-renames is incompatible with --delete-before and 14329 will select --delete-after instead of --delete-during. 14330 14331 --track-renames-strategy (hash,modtime,leaf,size) 14332 14333 This option changes the file matching criteria for --track-renames. 14334 14335 The matching is controlled by a comma separated selection of these 14336 tokens: 14337 14338 - modtime - the modification time of the file - not supported on all 14339 backends 14340 - hash - the hash of the file contents - not supported on all backends 14341 - leaf - the name of the file not including its directory name 14342 - size - the size of the file (this is always enabled) 14343 14344 The default option is hash. 14345 14346 Using --track-renames-strategy modtime,leaf would match files based on 14347 modification time, the leaf of the file name and the size only. 14348 14349 Using --track-renames-strategy modtime or leaf can enable 14350 --track-renames support for encrypted destinations. 14351 14352 Note that the hash strategy is not supported with encrypted 14353 destinations. 14354 14355 --delete-(before,during,after) 14356 14357 This option allows you to specify when files on your destination are 14358 deleted when you sync folders. 14359 14360 Specifying the value --delete-before will delete all files present on 14361 the destination, but not on the source before starting the transfer of 14362 any new or updated files. This uses two passes through the file systems, 14363 one for the deletions and one for the copies. 14364 14365 Specifying --delete-during will delete files while checking and 14366 uploading files. This is the fastest option and uses the least memory. 14367 14368 Specifying --delete-after (the default value) will delay deletion of 14369 files until all new/updated files have been successfully transferred. 14370 The files to be deleted are collected in the copy pass then deleted 14371 after the copy pass has completed successfully. The files to be deleted 14372 are held in memory so this mode may use more memory. This is the safest 14373 mode as it will only delete files if there have been no errors 14374 subsequent to that. If there have been errors before the deletions start 14375 then you will get the message 14376 not deleting files as there were IO errors. 14377 14378 --fast-list 14379 14380 When doing anything which involves a directory listing (e.g. sync, copy, 14381 ls - in fact nearly every command), rclone has different strategies to 14382 choose from. 14383 14384 The basic strategy is to list one directory and processes it before 14385 using more directory lists to process any subdirectories. This is a 14386 mandatory backend feature, called List, which means it is supported by 14387 all backends. This strategy uses small amount of memory, and because it 14388 can be parallelised it is fast for operations involving processing of 14389 the list results. 14390 14391 Some backends provide the support for an alternative strategy, where all 14392 files beneath a directory can be listed in one (or a small number) of 14393 transactions. Rclone supports this alternative strategy through an 14394 optional backend feature called ListR. You can see in the storage system 14395 overview documentation's optional features section which backends it is 14396 enabled for (these tend to be the bucket-based ones, e.g. S3, B2, GCS, 14397 Swift). This strategy requires fewer transactions for highly recursive 14398 operations, which is important on backends where this is charged or 14399 heavily rate limited. It may be faster (due to fewer transactions) or 14400 slower (because it can't be parallelized) depending on different 14401 parameters, and may require more memory if rclone has to keep the whole 14402 listing in memory. 14403 14404 Which listing strategy rclone picks for a given operation is 14405 complicated, but in general it tries to choose the best possible. It 14406 will prefer ListR in situations where it doesn't need to store the 14407 listed files in memory, e.g. for unlimited recursive ls command 14408 variants. In other situations it will prefer List, e.g. for sync and 14409 copy, where it needs to keep the listed files in memory, and is 14410 performing operations on them where parallelization may be a huge 14411 advantage. 14412 14413 Rclone is not able to take all relevant parameters into account for 14414 deciding the best strategy, and therefore allows you to influence the 14415 choice in two ways: You can stop rclone from using ListR by disabling 14416 the feature, using the --disable option (--disable ListR), or you can 14417 allow rclone to use ListR where it would normally choose not to do so 14418 due to higher memory usage, using the --fast-list option. Rclone should 14419 always produce identical results either way. Using --disable ListR or 14420 --fast-list on a remote which doesn't support ListR does nothing, rclone 14421 will just ignore it. 14422 14423 A rule of thumb is that if you pay for transactions and can fit your 14424 entire sync listing into memory, then --fast-list is recommended. If you 14425 have a very big sync to do, then don't use --fast-list, otherwise you 14426 will run out of memory. Run some tests and compare before you decide, 14427 and if in doubt then just leave the default, let rclone decide, i.e. not 14428 use --fast-list. 14429 14430 --timeout=TIME 14431 14432 This sets the IO idle timeout. If a transfer has started but then 14433 becomes idle for this long it is considered broken and disconnected. 14434 14435 The default is 5m. Set to 0 to disable. 14436 14437 --transfers=N 14438 14439 The number of file transfers to run in parallel. It can sometimes be 14440 useful to set this to a smaller number if the remote is giving a lot of 14441 timeouts or bigger if you have lots of bandwidth and a fast remote. 14442 14443 The default is to run 4 file transfers in parallel. 14444 14445 Look at --multi-thread-streams if you would like to control single file 14446 transfers. 14447 14448 -u, --update 14449 14450 This forces rclone to skip any files which exist on the destination and 14451 have a modified time that is newer than the source file. 14452 14453 This can be useful in avoiding needless transfers when transferring to a 14454 remote which doesn't support modification times directly (or when using 14455 --use-server-modtime to avoid extra API calls) as it is more accurate 14456 than a --size-only check and faster than using --checksum. On such 14457 remotes (or when using --use-server-modtime) the time checked will be 14458 the uploaded time. 14459 14460 If an existing destination file has a modification time older than the 14461 source file's, it will be updated if the sizes are different. If the 14462 sizes are the same, it will be updated if the checksum is different or 14463 not available. 14464 14465 If an existing destination file has a modification time equal (within 14466 the computed modify window) to the source file's, it will be updated if 14467 the sizes are different. The checksum will not be checked in this case 14468 unless the --checksum flag is provided. 14469 14470 In all other cases the file will not be updated. 14471 14472 Consider using the --modify-window flag to compensate for time skews 14473 between the source and the backend, for backends that do not support mod 14474 times, and instead use uploaded times. However, if the backend does not 14475 support checksums, note that syncing or copying within the time skew 14476 window may still result in additional transfers for safety. 14477 14478 --use-mmap 14479 14480 If this flag is set then rclone will use anonymous memory allocated by 14481 mmap on Unix based platforms and VirtualAlloc on Windows for its 14482 transfer buffers (size controlled by --buffer-size). Memory allocated 14483 like this does not go on the Go heap and can be returned to the OS 14484 immediately when it is finished with. 14485 14486 If this flag is not set then rclone will allocate and free the buffers 14487 using the Go memory allocator which may use more memory as memory pages 14488 are returned less aggressively to the OS. 14489 14490 It is possible this does not work well on all platforms so it is 14491 disabled by default; in the future it may be enabled by default. 14492 14493 --use-server-modtime 14494 14495 Some object-store backends (e.g, Swift, S3) do not preserve file 14496 modification times (modtime). On these backends, rclone stores the 14497 original modtime as additional metadata on the object. By default it 14498 will make an API call to retrieve the metadata when the modtime is 14499 needed by an operation. 14500 14501 Use this flag to disable the extra API call and rely instead on the 14502 server's modified time. In cases such as a local to remote sync using 14503 --update, knowing the local file is newer than the time it was last 14504 uploaded to the remote is sufficient. In those cases, this flag can 14505 speed up the process and reduce the number of API calls necessary. 14506 14507 Using this flag on a sync operation without also using --update would 14508 cause all files modified at any time other than the last upload time to 14509 be uploaded again, which is probably not what you want. 14510 14511 -v, -vv, --verbose 14512 14513 With -v rclone will tell you about each file that is transferred and a 14514 small number of significant events. 14515 14516 With -vv rclone will become very verbose telling you about every file it 14517 considers and transfers. Please send bug reports with a log with this 14518 setting. 14519 14520 When setting verbosity as an environment variable, use RCLONE_VERBOSE=1 14521 or RCLONE_VERBOSE=2 for -v and -vv respectively. 14522 14523 -V, --version 14524 14525 Prints the version number 14526 14527 SSL/TLS options 14528 14529 The outgoing SSL/TLS connections rclone makes can be controlled with 14530 these options. For example this can be very useful with the HTTP or 14531 WebDAV backends. Rclone HTTP servers have their own set of configuration 14532 for SSL/TLS which you can find in their documentation. 14533 14534 --ca-cert stringArray 14535 14536 This loads the PEM encoded certificate authority certificates and uses 14537 it to verify the certificates of the servers rclone connects to. 14538 14539 If you have generated certificates signed with a local CA then you will 14540 need this flag to connect to servers using those certificates. 14541 14542 --client-cert string 14543 14544 This loads the PEM encoded client side certificate. 14545 14546 This is used for mutual TLS authentication. 14547 14548 The --client-key flag is required too when using this. 14549 14550 --client-key string 14551 14552 This loads the PEM encoded client side private key used for mutual TLS 14553 authentication. Used in conjunction with --client-cert. 14554 14555 --no-check-certificate=true/false 14556 14557 --no-check-certificate controls whether a client verifies the server's 14558 certificate chain and host name. If --no-check-certificate is true, TLS 14559 accepts any certificate presented by the server and any host name in 14560 that certificate. In this mode, TLS is susceptible to man-in-the-middle 14561 attacks. 14562 14563 This option defaults to false. 14564 14565 This should be used only for testing. 14566 14567 Configuration Encryption 14568 14569 Your configuration file contains information for logging in to your 14570 cloud services. This means that you should keep your rclone.conf file in 14571 a secure location. 14572 14573 If you are in an environment where that isn't possible, you can add a 14574 password to your configuration. This means that you will have to supply 14575 the password every time you start rclone. 14576 14577 To add a password to your rclone configuration, execute rclone config. 14578 14579 >rclone config 14580 Current remotes: 14581 14582 e) Edit existing remote 14583 n) New remote 14584 d) Delete remote 14585 s) Set configuration password 14586 q) Quit config 14587 e/n/d/s/q> 14588 14589 Go into s, Set configuration password: 14590 14591 e/n/d/s/q> s 14592 Your configuration is not encrypted. 14593 If you add a password, you will protect your login information to cloud services. 14594 a) Add Password 14595 q) Quit to main menu 14596 a/q> a 14597 Enter NEW configuration password: 14598 password: 14599 Confirm NEW password: 14600 password: 14601 Password set 14602 Your configuration is encrypted. 14603 c) Change Password 14604 u) Unencrypt configuration 14605 q) Quit to main menu 14606 c/u/q> 14607 14608 Your configuration is now encrypted, and every time you start rclone you 14609 will have to supply the password. See below for details. In the same 14610 menu, you can change the password or completely remove encryption from 14611 your configuration. 14612 14613 There is no way to recover the configuration if you lose your password. 14614 14615 rclone uses nacl secretbox which in turn uses XSalsa20 and Poly1305 to 14616 encrypt and authenticate your configuration with secret-key 14617 cryptography. The password is SHA-256 hashed, which produces the key for 14618 secretbox. The hashed password is not stored. 14619 14620 While this provides very good security, we do not recommend storing your 14621 encrypted rclone configuration in public if it contains sensitive 14622 information, maybe except if you use a very strong password. 14623 14624 If it is safe in your environment, you can set the RCLONE_CONFIG_PASS 14625 environment variable to contain your password, in which case it will be 14626 used for decrypting the configuration. 14627 14628 You can set this for a session from a script. For unix like systems save 14629 this to a file called set-rclone-password: 14630 14631 #!/bin/echo Source this file don't run it 14632 14633 read -s RCLONE_CONFIG_PASS 14634 export RCLONE_CONFIG_PASS 14635 14636 Then source the file when you want to use it. From the shell you would 14637 do source set-rclone-password. It will then ask you for the password and 14638 set it in the environment variable. 14639 14640 An alternate means of supplying the password is to provide a script 14641 which will retrieve the password and print on standard output. This 14642 script should have a fully specified path name and not rely on any 14643 environment variables. The script is supplied either via 14644 --password-command="..." command line argument or via the 14645 RCLONE_PASSWORD_COMMAND environment variable. 14646 14647 One useful example of this is using the passwordstore application to 14648 retrieve the password: 14649 14650 export RCLONE_PASSWORD_COMMAND="pass rclone/config" 14651 14652 If the passwordstore password manager holds the password for the rclone 14653 configuration, using the script method means the password is primarily 14654 protected by the passwordstore system, and is never embedded in the 14655 clear in scripts, nor available for examination using the standard 14656 commands available. It is quite possible with long running rclone 14657 sessions for copies of passwords to be innocently captured in log files 14658 or terminal scroll buffers, etc. Using the script method of supplying 14659 the password enhances the security of the config password considerably. 14660 14661 If you are running rclone inside a script, unless you are using the 14662 --password-command method, you might want to disable password prompts. 14663 To do that, pass the parameter --ask-password=false to rclone. This will 14664 make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS 14665 doesn't contain a valid password, and --password-command has not been 14666 supplied. 14667 14668 Whenever running commands that may be affected by options in a 14669 configuration file, rclone will look for an existing file according to 14670 the rules described above, and load any it finds. If an encrypted file 14671 is found, this includes decrypting it, with the possible consequence of 14672 a password prompt. When executing a command line that you know are not 14673 actually using anything from such a configuration file, you can avoid it 14674 being loaded by overriding the location, e.g. with one of the documented 14675 special values for memory-only configuration. Since only backend options 14676 can be stored in configuration files, this is normally unnecessary for 14677 commands that do not operate on backends, e.g. genautocomplete. However, 14678 it will be relevant for commands that do operate on backends in general, 14679 but are used without referencing a stored remote, e.g. listing local 14680 filesystem paths, or connection strings: rclone --config="" ls . 14681 14682 Developer options 14683 14684 These options are useful when developing or debugging rclone. There are 14685 also some more remote specific options which aren't documented here 14686 which are used for testing. These start with remote name e.g. 14687 --drive-test-option - see the docs for the remote in question. 14688 14689 --cpuprofile=FILE 14690 14691 Write CPU profile to file. This can be analysed with go tool pprof. 14692 14693 --dump flag,flag,flag 14694 14695 The --dump flag takes a comma separated list of flags to dump info 14696 about. 14697 14698 Note that some headers including Accept-Encoding as shown may not be 14699 correct in the request and the response may not show Content-Encoding if 14700 the go standard libraries auto gzip encoding was in effect. In this case 14701 the body of the request will be gunzipped before showing it. 14702 14703 The available flags are: 14704 14705 --dump headers 14706 14707 Dump HTTP headers with Authorization: lines removed. May still contain 14708 sensitive info. Can be very verbose. Useful for debugging only. 14709 14710 Use --dump auth if you do want the Authorization: headers. 14711 14712 --dump bodies 14713 14714 Dump HTTP headers and bodies - may contain sensitive info. Can be very 14715 verbose. Useful for debugging only. 14716 14717 Note that the bodies are buffered in memory so don't use this for 14718 enormous files. 14719 14720 --dump requests 14721 14722 Like --dump bodies but dumps the request bodies and the response 14723 headers. Useful for debugging download problems. 14724 14725 --dump responses 14726 14727 Like --dump bodies but dumps the response bodies and the request 14728 headers. Useful for debugging upload problems. 14729 14730 --dump auth 14731 14732 Dump HTTP headers - will contain sensitive info such as Authorization: 14733 headers - use --dump headers to dump without Authorization: headers. Can 14734 be very verbose. Useful for debugging only. 14735 14736 --dump filters 14737 14738 Dump the filters to the output. Useful to see exactly what include and 14739 exclude options are filtering on. 14740 14741 --dump goroutines 14742 14743 This dumps a list of the running go-routines at the end of the command 14744 to standard output. 14745 14746 --dump openfiles 14747 14748 This dumps a list of the open files at the end of the command. It uses 14749 the lsof command to do that so you'll need that installed to use it. 14750 14751 --dump mapper 14752 14753 This shows the JSON blobs being sent to the program supplied with 14754 --metadata-mapper and received from it. It can be useful for debugging 14755 the metadata mapper interface. 14756 14757 --memprofile=FILE 14758 14759 Write memory profile to file. This can be analysed with go tool pprof. 14760 14761 Filtering 14762 14763 For the filtering options 14764 14765 - --delete-excluded 14766 - --filter 14767 - --filter-from 14768 - --exclude 14769 - --exclude-from 14770 - --exclude-if-present 14771 - --include 14772 - --include-from 14773 - --files-from 14774 - --files-from-raw 14775 - --min-size 14776 - --max-size 14777 - --min-age 14778 - --max-age 14779 - --dump filters 14780 - --metadata-include 14781 - --metadata-include-from 14782 - --metadata-exclude 14783 - --metadata-exclude-from 14784 - --metadata-filter 14785 - --metadata-filter-from 14786 14787 See the filtering section. 14788 14789 Remote control 14790 14791 For the remote control options and for instructions on how to remote 14792 control rclone 14793 14794 - --rc 14795 - and anything starting with --rc- 14796 14797 See the remote control section. 14798 14799 Logging 14800 14801 rclone has 4 levels of logging, ERROR, NOTICE, INFO and DEBUG. 14802 14803 By default, rclone logs to standard error. This means you can redirect 14804 standard error and still see the normal output of rclone commands (e.g. 14805 rclone ls). 14806 14807 By default, rclone will produce Error and Notice level messages. 14808 14809 If you use the -q flag, rclone will only produce Error messages. 14810 14811 If you use the -v flag, rclone will produce Error, Notice and Info 14812 messages. 14813 14814 If you use the -vv flag, rclone will produce Error, Notice, Info and 14815 Debug messages. 14816 14817 You can also control the log levels with the --log-level flag. 14818 14819 If you use the --log-file=FILE option, rclone will redirect Error, Info 14820 and Debug messages along with standard error to FILE. 14821 14822 If you use the --syslog flag then rclone will log to syslog and the 14823 --syslog-facility control which facility it uses. 14824 14825 Rclone prefixes all log messages with their level in capitals, e.g. INFO 14826 which makes it easy to grep the log file for different kinds of 14827 information. 14828 14829 Exit Code 14830 14831 If any errors occur during the command execution, rclone will exit with 14832 a non-zero exit code. This allows scripts to detect when rclone 14833 operations have failed. 14834 14835 During the startup phase, rclone will exit immediately if an error is 14836 detected in the configuration. There will always be a log message 14837 immediately before exiting. 14838 14839 When rclone is running it will accumulate errors as it goes along, and 14840 only exit with a non-zero exit code if (after retries) there were still 14841 failed transfers. For every error counted there will be a high priority 14842 log message (visible with -q) showing the message and which file caused 14843 the problem. A high priority message is also shown when starting a retry 14844 so the user can see that any previous error messages may not be valid 14845 after the retry. If rclone has done a retry it will log a high priority 14846 message if the retry was successful. 14847 14848 List of exit codes 14849 14850 - 0 - success 14851 - 1 - Syntax or usage error 14852 - 2 - Error not otherwise categorised 14853 - 3 - Directory not found 14854 - 4 - File not found 14855 - 5 - Temporary error (one that more retries might fix) (Retry errors) 14856 - 6 - Less serious errors (like 461 errors from dropbox) (NoRetry 14857 errors) 14858 - 7 - Fatal error (one that more retries won't fix, like account 14859 suspended) (Fatal errors) 14860 - 8 - Transfer exceeded - limit set by --max-transfer reached 14861 - 9 - Operation successful, but no files transferred 14862 - 10 - Duration exceeded - limit set by --max-duration reached 14863 14864 Environment Variables 14865 14866 Rclone can be configured entirely using environment variables. These can 14867 be used to set defaults for options or config file entries. 14868 14869 Options 14870 14871 Every option in rclone can have its default set by environment variable. 14872 14873 To find the name of the environment variable, first, take the long 14874 option name, strip the leading --, change - to _, make upper case and 14875 prepend RCLONE_. 14876 14877 For example, to always set --stats 5s, set the environment variable 14878 RCLONE_STATS=5s. If you set stats on the command line this will override 14879 the environment variable setting. 14880 14881 Or to always use the trash in drive --drive-use-trash, set 14882 RCLONE_DRIVE_USE_TRASH=true. 14883 14884 Verbosity is slightly different, the environment variable equivalent of 14885 --verbose or -v is RCLONE_VERBOSE=1, or for -vv, RCLONE_VERBOSE=2. 14886 14887 The same parser is used for the options and the environment variables so 14888 they take exactly the same form. 14889 14890 The options set by environment variables can be seen with the -vv flag, 14891 e.g. rclone version -vv. 14892 14893 Config file 14894 14895 You can set defaults for values in the config file on an individual 14896 remote basis. The names of the config items are documented in the page 14897 for each backend. 14898 14899 To find the name of the environment variable, you need to set, take 14900 RCLONE_CONFIG_ + name of remote + _ + name of config file option and 14901 make it all uppercase. Note one implication here is the remote's name 14902 must be convertible into a valid environment variable name, so it can 14903 only contain letters, digits, or the _ (underscore) character. 14904 14905 For example, to configure an S3 remote named mys3: without a config file 14906 (using unix ways of setting environment variables): 14907 14908 $ export RCLONE_CONFIG_MYS3_TYPE=s3 14909 $ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX 14910 $ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX 14911 $ rclone lsd mys3: 14912 -1 2016-09-21 12:54:21 -1 my-bucket 14913 $ rclone listremotes | grep mys3 14914 mys3: 14915 14916 Note that if you want to create a remote using environment variables you 14917 must create the ..._TYPE variable as above. 14918 14919 Note that the name of a remote created using environment variable is 14920 case insensitive, in contrast to regular remotes stored in config file 14921 as documented above. You must write the name in uppercase in the 14922 environment variable, but as seen from example above it will be listed 14923 and can be accessed in lowercase, while you can also refer to the same 14924 remote in uppercase: 14925 14926 $ rclone lsd mys3: 14927 -1 2016-09-21 12:54:21 -1 my-bucket 14928 $ rclone lsd MYS3: 14929 -1 2016-09-21 12:54:21 -1 my-bucket 14930 14931 Note that you can only set the options of the immediate backend, so 14932 RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is a 14933 crypt remote based on an S3 remote. However RCLONE_S3_ACCESS_KEY_ID will 14934 set the access key of all remotes using S3, including myS3Crypt. 14935 14936 Note also that now rclone has connection strings, it is probably easier 14937 to use those instead which makes the above example 14938 14939 rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX: 14940 14941 Precedence 14942 14943 The various different methods of backend configuration are read in this 14944 order and the first one with a value is used. 14945 14946 - Parameters in connection strings, e.g. myRemote,skip_links: 14947 - Flag values as supplied on the command line, e.g. --skip-links 14948 - Remote specific environment vars, e.g. 14949 RCLONE_CONFIG_MYREMOTE_SKIP_LINKS (see above). 14950 - Backend-specific environment vars, e.g. RCLONE_LOCAL_SKIP_LINKS. 14951 - Backend generic environment vars, e.g. RCLONE_SKIP_LINKS. 14952 - Config file, e.g. skip_links = true. 14953 - Default values, e.g. false - these can't be changed. 14954 14955 So if both --skip-links is supplied on the command line and an 14956 environment variable RCLONE_LOCAL_SKIP_LINKS is set, the command line 14957 flag will take preference. 14958 14959 The backend configurations set by environment variables can be seen with 14960 the -vv flag, e.g. rclone about myRemote: -vv. 14961 14962 For non backend configuration the order is as follows: 14963 14964 - Flag values as supplied on the command line, e.g. --stats 5s. 14965 - Environment vars, e.g. RCLONE_STATS=5s. 14966 - Default values, e.g. 1m - these can't be changed. 14967 14968 Other environment variables 14969 14970 - RCLONE_CONFIG_PASS set to contain your config file password (see 14971 Configuration Encryption section) 14972 - HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions 14973 thereof). 14974 - HTTPS_PROXY takes precedence over HTTP_PROXY for https requests. 14975 - The environment values may be either a complete URL or a 14976 "host[:port]" for, in which case the "http" scheme is assumed. 14977 - USER and LOGNAME values are used as fallbacks for current username. 14978 The primary method for looking up username is OS-specific: Windows 14979 API on Windows, real user ID in /etc/passwd on Unix systems. In the 14980 documentation the current username is simply referred to as $USER. 14981 - RCLONE_CONFIG_DIR - rclone sets this variable for use in config 14982 files and sub processes to point to the directory holding the config 14983 file. 14984 14985 The options set by environment variables can be seen with the -vv and 14986 --log-level=DEBUG flags, e.g. rclone version -vv. 14987 14988 Configuring rclone on a remote / headless machine 14989 14990 Some of the configurations (those involving oauth2) require an Internet 14991 connected web browser. 14992 14993 If you are trying to set rclone up on a remote or headless box with no 14994 browser available on it (e.g. a NAS or a server in a datacenter) then 14995 you will need to use an alternative means of configuration. There are 14996 two ways of doing it, described below. 14997 14998 Configuring using rclone authorize 14999 15000 On the headless box run rclone config but answer N to the 15001 Use web browser to automatically authenticate? question. 15002 15003 ... 15004 Remote config 15005 Use web browser to automatically authenticate rclone with remote? 15006 * Say Y if the machine running rclone has a web browser you can use 15007 * Say N if running rclone on a (remote) machine without web browser access 15008 If not sure try Y. If Y failed, try N. 15009 y) Yes (default) 15010 n) No 15011 y/n> n 15012 For this to work, you will need rclone available on a machine that has 15013 a web browser available. 15014 15015 For more help and alternate methods see: https://rclone.org/remote_setup/ 15016 15017 Execute the following on the machine with the web browser (same rclone 15018 version recommended): 15019 15020 rclone authorize "dropbox" 15021 15022 Then paste the result below: 15023 result> 15024 15025 Then on your main desktop machine 15026 15027 rclone authorize "dropbox" 15028 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 15029 Log in and authorize rclone for access 15030 Waiting for code... 15031 Got code 15032 Paste the following into your remote machine ---> 15033 SECRET_TOKEN 15034 <---End paste 15035 15036 Then back to the headless box, paste in the code 15037 15038 result> SECRET_TOKEN 15039 -------------------- 15040 [acd12] 15041 client_id = 15042 client_secret = 15043 token = SECRET_TOKEN 15044 -------------------- 15045 y) Yes this is OK 15046 e) Edit this remote 15047 d) Delete this remote 15048 y/e/d> 15049 15050 Configuring by copying the config file 15051 15052 Rclone stores all of its config in a single configuration file. This can 15053 easily be copied to configure a remote rclone. 15054 15055 So first configure rclone on your desktop machine with 15056 15057 rclone config 15058 15059 to set up the config file. 15060 15061 Find the config file by running rclone config file, for example 15062 15063 $ rclone config file 15064 Configuration file is stored at: 15065 /home/user/.rclone.conf 15066 15067 Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and 15068 place it in the correct place (use rclone config file on the remote box 15069 to find out where). 15070 15071 Configuring using SSH Tunnel 15072 15073 Linux and MacOS users can utilize SSH Tunnel to redirect the headless 15074 box port 53682 to local machine by using the following command: 15075 15076 ssh -L localhost:53682:localhost:53682 username@remote_server 15077 15078 Then on the headless box run rclone config and answer Y to the 15079 Use web browser to automatically authenticate? question. 15080 15081 ... 15082 Remote config 15083 Use web browser to automatically authenticate rclone with remote? 15084 * Say Y if the machine running rclone has a web browser you can use 15085 * Say N if running rclone on a (remote) machine without web browser access 15086 If not sure try Y. If Y failed, try N. 15087 y) Yes (default) 15088 n) No 15089 y/n> y 15090 15091 Then copy and paste the auth url 15092 http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx to the browser on your 15093 local machine, complete the auth and it is done. 15094 15095 Filtering, includes and excludes 15096 15097 Filter flags determine which files rclone sync, move, ls, lsl, md5sum, 15098 sha1sum, size, delete, check and similar commands apply to. 15099 15100 They are specified in terms of path/file name patterns; path/file lists; 15101 file age and size, or presence of a file in a directory. Bucket based 15102 remotes without the concept of directory apply filters to object key, 15103 age and size in an analogous way. 15104 15105 Rclone purge does not obey filters. 15106 15107 To test filters without risk of damage to data, apply them to rclone ls, 15108 or with the --dry-run and -vv flags. 15109 15110 Rclone filter patterns can only be used in filter command line options, 15111 not in the specification of a remote. 15112 15113 E.g. rclone copy "remote:dir*.jpg" /path/to/dir does not have a filter 15114 effect. rclone copy remote:dir /path/to/dir --include "*.jpg" does. 15115 15116 Important Avoid mixing any two of --include..., --exclude... or 15117 --filter... flags in an rclone command. The results might not be what 15118 you expect. Instead use a --filter... flag. 15119 15120 Patterns for matching path/file names 15121 15122 Pattern syntax 15123 15124 Here is a formal definition of the pattern syntax, examples are below. 15125 15126 Rclone matching rules follow a glob style: 15127 15128 * matches any sequence of non-separator (/) characters 15129 ** matches any sequence of characters including / separators 15130 ? matches any single non-separator (/) character 15131 [ [ ! ] { character-range } ] 15132 character class (must be non-empty) 15133 { pattern-list } 15134 pattern alternatives 15135 {{ regexp }} 15136 regular expression to match 15137 c matches character c (c != *, **, ?, \, [, {, }) 15138 \c matches reserved character c (c = *, **, ?, \, [, {, }) or character class 15139 15140 character-range: 15141 15142 c matches character c (c != \, -, ]) 15143 \c matches reserved character c (c = \, -, ]) 15144 lo - hi matches character c for lo <= c <= hi 15145 15146 pattern-list: 15147 15148 pattern { , pattern } 15149 comma-separated (without spaces) patterns 15150 15151 character classes (see Go regular expression reference) include: 15152 15153 Named character classes (e.g. [\d], [^\d], [\D], [^\D]) 15154 Perl character classes (e.g. \s, \S, \w, \W) 15155 ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]]) 15156 15157 regexp for advanced users to insert a regular expression - see below for 15158 more info: 15159 15160 Any re2 regular expression not containing `}}` 15161 15162 If the filter pattern starts with a / then it only matches at the top 15163 level of the directory tree, relative to the root of the remote (not 15164 necessarily the root of the drive). If it does not start with / then it 15165 is matched starting at the end of the path/file name but it only matches 15166 a complete path element - it must match from a / separator or the 15167 beginning of the path/file. 15168 15169 file.jpg - matches "file.jpg" 15170 - matches "directory/file.jpg" 15171 - doesn't match "afile.jpg" 15172 - doesn't match "directory/afile.jpg" 15173 /file.jpg - matches "file.jpg" in the root directory of the remote 15174 - doesn't match "afile.jpg" 15175 - doesn't match "directory/file.jpg" 15176 15177 The top level of the remote might not be the top level of the drive. 15178 15179 E.g. for a Microsoft Windows local directory structure 15180 15181 F: 15182 ├── bkp 15183 ├── data 15184 │ ├── excl 15185 │ │ ├── 123.jpg 15186 │ │ └── 456.jpg 15187 │ ├── incl 15188 │ │ └── document.pdf 15189 15190 To copy the contents of folder data into folder bkp excluding the 15191 contents of subfolder exclthe following command treats F:\data and 15192 F:\bkp as top level for filtering. 15193 15194 rclone copy F:\data\ F:\bkp\ --exclude=/excl/** 15195 15196 Important Use / in path/file name patterns and not \ even if running on 15197 Microsoft Windows. 15198 15199 Simple patterns are case sensitive unless the --ignore-case flag is 15200 used. 15201 15202 Without --ignore-case (default) 15203 15204 potato - matches "potato" 15205 - doesn't match "POTATO" 15206 15207 With --ignore-case 15208 15209 potato - matches "potato" 15210 - matches "POTATO" 15211 15212 Using regular expressions in filter patterns 15213 15214 The syntax of filter patterns is glob style matching (like bash uses) to 15215 make things easy for users. However this does not provide absolute 15216 control over the matching, so for advanced users rclone also provides a 15217 regular expression syntax. 15218 15219 The regular expressions used are as defined in the Go regular expression 15220 reference. Regular expressions should be enclosed in {{ }}. They will 15221 match only the last path segment if the glob doesn't start with / or the 15222 whole path name if it does. Note that rclone does not attempt to parse 15223 the supplied regular expression, meaning that using any regular 15224 expression filter will prevent rclone from using directory filter rules, 15225 as it will instead check every path against the supplied regular 15226 expression(s). 15227 15228 Here is how the {{regexp}} is transformed into an full regular 15229 expression to match the entire path: 15230 15231 {{regexp}} becomes (^|/)(regexp)$ 15232 /{{regexp}} becomes ^(regexp)$ 15233 15234 Regexp syntax can be mixed with glob syntax, for example 15235 15236 *.{{jpe?g}} to match file.jpg, file.jpeg but not file.png 15237 15238 You can also use regexp flags - to set case insensitive, for example 15239 15240 *.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png 15241 15242 Be careful with wildcards in regular expressions - you don't want them 15243 to match path separators normally. To match any file name starting with 15244 start and ending with end write 15245 15246 {{start[^/]*end\.jpg}} 15247 15248 Not 15249 15250 {{start.*end\.jpg}} 15251 15252 Which will match a directory called start with a file called end.jpg in 15253 it as the .* will match / characters. 15254 15255 Note that you can use -vv --dump filters to show the filter patterns in 15256 regexp format - rclone implements the glob patterns by transforming them 15257 into regular expressions. 15258 15259 Filter pattern examples 15260 15261 Description Pattern Matches Does not match 15262 --------------- ---------------- ------------------------------- ------------------ 15263 Wildcard *.jpg /file.jpg /file.png 15264 /dir/file.jpg /dir/file.png 15265 Rooted /*.jpg /file.jpg /file.png 15266 /file2.jpg /dir/file.jpg 15267 Alternates *.{jpg,png} /file.jpg /file.gif 15268 /dir/file.png /dir/file.gif 15269 Path Wildcard dir/** /dir/anyfile file.png 15270 /subdir/dir/subsubdir/anyfile /subdir/file.png 15271 Any Char *.t?t /file.txt /file.qxt 15272 /dir/file.tzt /dir/file.png 15273 Range *.[a-z] /file.a /file.0 15274 /dir/file.b /dir/file.1 15275 Escape *.\?\?\? /file.??? /file.abc 15276 /dir/file.??? /dir/file.def 15277 Class *.\d\d\d /file.012 /file.abc 15278 /dir/file.345 /dir/file.def 15279 Regexp *.{{jpe?g}} /file.jpeg /file.png 15280 /dir/file.jpg /dir/file.jpeeg 15281 Rooted Regexp /{{.*\.jpe?g}} /file.jpeg /file.png 15282 /file.jpg /dir/file.jpg 15283 15284 How filter rules are applied to files 15285 15286 Rclone path/file name filters are made up of one or more of the 15287 following flags: 15288 15289 - --include 15290 - --include-from 15291 - --exclude 15292 - --exclude-from 15293 - --filter 15294 - --filter-from 15295 15296 There can be more than one instance of individual flags. 15297 15298 Rclone internally uses a combined list of all the include and exclude 15299 rules. The order in which rules are processed can influence the result 15300 of the filter. 15301 15302 All flags of the same type are processed together in the order above, 15303 regardless of what order the different types of flags are included on 15304 the command line. 15305 15306 Multiple instances of the same flag are processed from left to right 15307 according to their position in the command line. 15308 15309 To mix up the order of processing includes and excludes use --filter... 15310 flags. 15311 15312 Within --include-from, --exclude-from and --filter-from flags rules are 15313 processed from top to bottom of the referenced file. 15314 15315 If there is an --include or --include-from flag specified, rclone 15316 implies a - ** rule which it adds to the bottom of the internal rule 15317 list. Specifying a + rule with a --filter... flag does not imply that 15318 rule. 15319 15320 Each path/file name passed through rclone is matched against the 15321 combined filter list. At first match to a rule the path/file name is 15322 included or excluded and no further filter rules are processed for that 15323 path/file. 15324 15325 If rclone does not find a match, after testing against all rules 15326 (including the implied rule if appropriate), the path/file name is 15327 included. 15328 15329 Any path/file included at that stage is processed by the rclone command. 15330 15331 --files-from and --files-from-raw flags over-ride and cannot be combined 15332 with other filter options. 15333 15334 To see the internal combined rule list, in regular expression form, for 15335 a command add the --dump filters flag. Running an rclone command with 15336 --dump filters and -vv flags lists the internal filter elements and 15337 shows how they are applied to each source path/file. There is not 15338 currently a means provided to pass regular expression filter options 15339 into rclone directly though character class filter rules contain 15340 character classes. Go regular expression reference 15341 15342 How filter rules are applied to directories 15343 15344 Rclone commands are applied to path/file names not directories. The 15345 entire contents of a directory can be matched to a filter by the pattern 15346 directory/* or recursively by directory/**. 15347 15348 Directory filter rules are defined with a closing / separator. 15349 15350 E.g. /directory/subdirectory/ is an rclone directory filter rule. 15351 15352 Rclone commands can use directory filter rules to determine whether they 15353 recurse into subdirectories. This potentially optimises access to a 15354 remote by avoiding listing unnecessary directories. Whether optimisation 15355 is desirable depends on the specific filter rules and source remote 15356 content. 15357 15358 If any regular expression filters are in use, then no directory 15359 recursion optimisation is possible, as rclone must check every path 15360 against the supplied regular expression(s). 15361 15362 Directory recursion optimisation occurs if either: 15363 15364 - A source remote does not support the rclone ListR primitive. local, 15365 sftp, Microsoft OneDrive and WebDAV do not support ListR. Google 15366 Drive and most bucket type storage do. Full list 15367 15368 - On other remotes (those that support ListR), if the rclone command 15369 is not naturally recursive, and provided it is not run with the 15370 --fast-list flag. ls, lsf -R and size are naturally recursive but 15371 sync, copy and move are not. 15372 15373 - Whenever the --disable ListR flag is applied to an rclone command. 15374 15375 Rclone commands imply directory filter rules from path/file filter 15376 rules. To view the directory filter rules rclone has implied for a 15377 command specify the --dump filters flag. 15378 15379 E.g. for an include rule 15380 15381 /a/*.jpg 15382 15383 Rclone implies the directory include rule 15384 15385 /a/ 15386 15387 Directory filter rules specified in an rclone command can limit the 15388 scope of an rclone command but path/file filters still have to be 15389 specified. 15390 15391 E.g. rclone ls remote: --include /directory/ will not match any files. 15392 Because it is an --include option the --exclude ** rule is implied, and 15393 the /directory/ pattern serves only to optimise access to the remote by 15394 ignoring everything outside of that directory. 15395 15396 E.g. rclone ls remote: --filter-from filter-list.txt with a file 15397 filter-list.txt: 15398 15399 - /dir1/ 15400 - /dir2/ 15401 + *.pdf 15402 - ** 15403 15404 All files in directories dir1 or dir2 or their subdirectories are 15405 completely excluded from the listing. Only files of suffix pdf in the 15406 root of remote: or its subdirectories are listed. The - ** rule prevents 15407 listing of any path/files not previously matched by the rules above. 15408 15409 Option exclude-if-present creates a directory exclude rule based on the 15410 presence of a file in a directory and takes precedence over other rclone 15411 directory filter rules. 15412 15413 When using pattern list syntax, if a pattern item contains either / or 15414 **, then rclone will not able to imply a directory filter rule from this 15415 pattern list. 15416 15417 E.g. for an include rule 15418 15419 {dir1/**,dir2/**} 15420 15421 Rclone will match files below directories dir1 or dir2 only, but will 15422 not be able to use this filter to exclude a directory dir3 from being 15423 traversed. 15424 15425 Directory recursion optimisation may affect performance, but normally 15426 not the result. One exception to this is sync operations with option 15427 --create-empty-src-dirs, where any traversed empty directories will be 15428 created. With the pattern list example {dir1/**,dir2/**} above, this 15429 would create an empty directory dir3 on destination (when it exists on 15430 source). Changing the filter to {dir1,dir2}/**, or splitting it into two 15431 include rules --include dir1/** --include dir2/**, will match the same 15432 files while also filtering directories, with the result that an empty 15433 directory dir3 will no longer be created. 15434 15435 --exclude - Exclude files matching pattern 15436 15437 Excludes path/file names from an rclone command based on a single 15438 exclude rule. 15439 15440 This flag can be repeated. See above for the order filter flags are 15441 processed in. 15442 15443 --exclude should not be used with --include, --include-from, --filter or 15444 --filter-from flags. 15445 15446 --exclude has no effect when combined with --files-from or 15447 --files-from-raw flags. 15448 15449 E.g. rclone ls remote: --exclude *.bak excludes all .bak files from 15450 listing. 15451 15452 E.g. rclone size remote: "--exclude /dir/**" returns the total size of 15453 all files on remote: excluding those in root directory dir and sub 15454 directories. 15455 15456 E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*" 15457 lists the files in remote: without [JP] or [KR] or [HK] in their name. 15458 Quotes prevent the shell from interpreting the \ characters.\ characters 15459 escape the [ and ] so an rclone filter treats them literally rather than 15460 as a character-range. The { and } define an rclone pattern list. For 15461 other operating systems single quotes are required ie 15462 rclone ls remote: --exclude '*\[{JP,KR,HK}\]*' 15463 15464 --exclude-from - Read exclude patterns from file 15465 15466 Excludes path/file names from an rclone command based on rules in a 15467 named file. The file contains a list of remarks and pattern rules. 15468 15469 For an example exclude-file.txt: 15470 15471 # a sample exclude rule file 15472 *.bak 15473 file2.jpg 15474 15475 rclone ls remote: --exclude-from exclude-file.txt lists the files on 15476 remote: except those named file2.jpg or with a suffix .bak. That is 15477 equivalent to rclone ls remote: --exclude file2.jpg --exclude "*.bak". 15478 15479 This flag can be repeated. See above for the order filter flags are 15480 processed in. 15481 15482 The --exclude-from flag is useful where multiple exclude filter rules 15483 are applied to an rclone command. 15484 15485 --exclude-from should not be used with --include, --include-from, 15486 --filter or --filter-from flags. 15487 15488 --exclude-from has no effect when combined with --files-from or 15489 --files-from-raw flags. 15490 15491 --exclude-from followed by - reads filter rules from standard input. 15492 15493 --include - Include files matching pattern 15494 15495 Adds a single include rule based on path/file names to an rclone 15496 command. 15497 15498 This flag can be repeated. See above for the order filter flags are 15499 processed in. 15500 15501 --include has no effect when combined with --files-from or 15502 --files-from-raw flags. 15503 15504 --include implies --exclude ** at the end of an rclone internal filter 15505 list. Therefore if you mix --include and --include-from flags with 15506 --exclude, --exclude-from, --filter or --filter-from, you must use 15507 include rules for all the files you want in the include statement. For 15508 more flexibility use the --filter-from flag. 15509 15510 E.g. rclone ls remote: --include "*.{png,jpg}" lists the files on 15511 remote: with suffix .png and .jpg. All other files are excluded. 15512 15513 E.g. multiple rclone copy commands can be combined with --include and a 15514 pattern-list. 15515 15516 rclone copy /vol1/A remote:A 15517 rclone copy /vol1/B remote:B 15518 15519 is equivalent to: 15520 15521 rclone copy /vol1 remote: --include "{A,B}/**" 15522 15523 E.g. rclone ls remote:/wheat --include "??[^[:punct:]]*" lists the files 15524 remote: directory wheat (and subdirectories) whose third character is 15525 not punctuation. This example uses an ASCII character class. 15526 15527 --include-from - Read include patterns from file 15528 15529 Adds path/file names to an rclone command based on rules in a named 15530 file. The file contains a list of remarks and pattern rules. 15531 15532 For an example include-file.txt: 15533 15534 # a sample include rule file 15535 *.jpg 15536 file2.avi 15537 15538 rclone ls remote: --include-from include-file.txt lists the files on 15539 remote: with name file2.avi or suffix .jpg. That is equivalent to 15540 rclone ls remote: --include file2.avi --include "*.jpg". 15541 15542 This flag can be repeated. See above for the order filter flags are 15543 processed in. 15544 15545 The --include-from flag is useful where multiple include filter rules 15546 are applied to an rclone command. 15547 15548 --include-from implies --exclude ** at the end of an rclone internal 15549 filter list. Therefore if you mix --include and --include-from flags 15550 with --exclude, --exclude-from, --filter or --filter-from, you must use 15551 include rules for all the files you want in the include statement. For 15552 more flexibility use the --filter-from flag. 15553 15554 --exclude-from has no effect when combined with --files-from or 15555 --files-from-raw flags. 15556 15557 --exclude-from followed by - reads filter rules from standard input. 15558 15559 --filter - Add a file-filtering rule 15560 15561 Specifies path/file names to an rclone command, based on a single 15562 include or exclude rule, in + or - format. 15563 15564 This flag can be repeated. See above for the order filter flags are 15565 processed in. 15566 15567 --filter + differs from --include. In the case of --include rclone 15568 implies an --exclude * rule which it adds to the bottom of the internal 15569 rule list. --filter...+ does not imply that rule. 15570 15571 --filter has no effect when combined with --files-from or 15572 --files-from-raw flags. 15573 15574 --filter should not be used with --include, --include-from, --exclude or 15575 --exclude-from flags. 15576 15577 E.g. rclone ls remote: --filter "- *.bak" excludes all .bak files from a 15578 list of remote:. 15579 15580 --filter-from - Read filtering patterns from a file 15581 15582 Adds path/file names to an rclone command based on rules in a named 15583 file. The file contains a list of remarks and pattern rules. Include 15584 rules start with + and exclude rules with -. ! clears existing rules. 15585 Rules are processed in the order they are defined. 15586 15587 This flag can be repeated. See above for the order filter flags are 15588 processed in. 15589 15590 Arrange the order of filter rules with the most restrictive first and 15591 work down. 15592 15593 E.g. for filter-file.txt: 15594 15595 # a sample filter rule file 15596 - secret*.jpg 15597 + *.jpg 15598 + *.png 15599 + file2.avi 15600 - /dir/Trash/** 15601 + /dir/** 15602 # exclude everything else 15603 - * 15604 15605 rclone ls remote: --filter-from filter-file.txt lists the path/files on 15606 remote: including all jpg and png files, excluding any matching 15607 secret*.jpg and including file2.avi. It also includes everything in the 15608 directory dir at the root of remote, except remote:dir/Trash which it 15609 excludes. Everything else is excluded. 15610 15611 E.g. for an alternative filter-file.txt: 15612 15613 - secret*.jpg 15614 + *.jpg 15615 + *.png 15616 + file2.avi 15617 - * 15618 15619 Files file1.jpg, file3.png and file2.avi are listed whilst secret17.jpg 15620 and files without the suffix .jpg or .png are excluded. 15621 15622 E.g. for an alternative filter-file.txt: 15623 15624 + *.jpg 15625 + *.gif 15626 ! 15627 + 42.doc 15628 - * 15629 15630 Only file 42.doc is listed. Prior rules are cleared by the !. 15631 15632 --files-from - Read list of source-file names 15633 15634 Adds path/files to an rclone command from a list in a named file. Rclone 15635 processes the path/file names in the order of the list, and no others. 15636 15637 Other filter flags (--include, --include-from, --exclude, 15638 --exclude-from, --filter and --filter-from) are ignored when 15639 --files-from is used. 15640 15641 --files-from expects a list of files as its input. Leading or trailing 15642 whitespace is stripped from the input lines. Lines starting with # or ; 15643 are ignored. 15644 15645 Rclone commands with a --files-from flag traverse the remote, treating 15646 the names in --files-from as a set of filters. 15647 15648 If the --no-traverse and --files-from flags are used together an rclone 15649 command does not traverse the remote. Instead it addresses each 15650 path/file named in the file individually. For each path/file name, that 15651 requires typically 1 API call. This can be efficient for a short 15652 --files-from list and a remote containing many files. 15653 15654 Rclone commands do not error if any names in the --files-from file are 15655 missing from the source remote. 15656 15657 The --files-from flag can be repeated in a single rclone command to read 15658 path/file names from more than one file. The files are read from left to 15659 right along the command line. 15660 15661 Paths within the --files-from file are interpreted as starting with the 15662 root specified in the rclone command. Leading / separators are ignored. 15663 See --files-from-raw if you need the input to be processed in a raw 15664 manner. 15665 15666 E.g. for a file files-from.txt: 15667 15668 # comment 15669 file1.jpg 15670 subdir/file2.jpg 15671 15672 rclone copy --files-from files-from.txt /home/me/pics remote:pics copies 15673 the following, if they exist, and only those files. 15674 15675 /home/me/pics/file1.jpg → remote:pics/file1.jpg 15676 /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg 15677 15678 E.g. to copy the following files referenced by their absolute paths: 15679 15680 /home/user1/42 15681 /home/user1/dir/ford 15682 /home/user2/prefect 15683 15684 First find a common subdirectory - in this case /home and put the 15685 remaining files in files-from.txt with or without leading /, e.g. 15686 15687 user1/42 15688 user1/dir/ford 15689 user2/prefect 15690 15691 Then copy these to a remote: 15692 15693 rclone copy --files-from files-from.txt /home remote:backup 15694 15695 The three files are transferred as follows: 15696 15697 /home/user1/42 → remote:backup/user1/important 15698 /home/user1/dir/ford → remote:backup/user1/dir/file 15699 /home/user2/prefect → remote:backup/user2/stuff 15700 15701 Alternatively if / is chosen as root files-from.txt will be: 15702 15703 /home/user1/42 15704 /home/user1/dir/ford 15705 /home/user2/prefect 15706 15707 The copy command will be: 15708 15709 rclone copy --files-from files-from.txt / remote:backup 15710 15711 Then there will be an extra home directory on the remote: 15712 15713 /home/user1/42 → remote:backup/home/user1/42 15714 /home/user1/dir/ford → remote:backup/home/user1/dir/ford 15715 /home/user2/prefect → remote:backup/home/user2/prefect 15716 15717 --files-from-raw - Read list of source-file names without any processing 15718 15719 This flag is the same as --files-from except that input is read in a raw 15720 manner. Lines with leading / trailing whitespace, and lines starting 15721 with ; or # are read without any processing. rclone lsf has a compatible 15722 format that can be used to export file lists from remotes for input to 15723 --files-from-raw. 15724 15725 --ignore-case - make searches case insensitive 15726 15727 By default, rclone filter patterns are case sensitive. The --ignore-case 15728 flag makes all of the filters patterns on the command line case 15729 insensitive. 15730 15731 E.g. --include "zaphod.txt" does not match a file Zaphod.txt. With 15732 --ignore-case a match is made. 15733 15734 Quoting shell metacharacters 15735 15736 Rclone commands with filter patterns containing shell metacharacters may 15737 not as work as expected in your shell and may require quoting. 15738 15739 E.g. linux, OSX (* metacharacter) 15740 15741 - --include \*.jpg 15742 - --include '*.jpg' 15743 - --include='*.jpg' 15744 15745 Microsoft Windows expansion is done by the command, not shell, so 15746 --include *.jpg does not require quoting. 15747 15748 If the rclone error 15749 Command .... needs .... arguments maximum: you provided .... non flag arguments: 15750 is encountered, the cause is commonly spaces within the name of a remote 15751 or flag value. The fix then is to quote values containing spaces. 15752 15753 Other filters 15754 15755 --min-size - Don't transfer any file smaller than this 15756 15757 Controls the minimum size file within the scope of an rclone command. 15758 Default units are KiB but abbreviations K, M, G, T or P are valid. 15759 15760 E.g. rclone ls remote: --min-size 50k lists files on remote: of 50 KiB 15761 size or larger. 15762 15763 See the size option docs for more info. 15764 15765 --max-size - Don't transfer any file larger than this 15766 15767 Controls the maximum size file within the scope of an rclone command. 15768 Default units are KiB but abbreviations K, M, G, T or P are valid. 15769 15770 E.g. rclone ls remote: --max-size 1G lists files on remote: of 1 GiB 15771 size or smaller. 15772 15773 See the size option docs for more info. 15774 15775 --max-age - Don't transfer any file older than this 15776 15777 Controls the maximum age of files within the scope of an rclone command. 15778 15779 --max-age applies only to files and not to directories. 15780 15781 E.g. rclone ls remote: --max-age 2d lists files on remote: of 2 days old 15782 or less. 15783 15784 See the time option docs for valid formats. 15785 15786 --min-age - Don't transfer any file younger than this 15787 15788 Controls the minimum age of files within the scope of an rclone command. 15789 (see --max-age for valid formats) 15790 15791 --min-age applies only to files and not to directories. 15792 15793 E.g. rclone ls remote: --min-age 2d lists files on remote: of 2 days old 15794 or more. 15795 15796 See the time option docs for valid formats. 15797 15798 Other flags 15799 15800 --delete-excluded - Delete files on dest excluded from sync 15801 15802 Important this flag is dangerous to your data - use with --dry-run and 15803 -v first. 15804 15805 In conjunction with rclone sync, --delete-excluded deletes any files on 15806 the destination which are excluded from the command. 15807 15808 E.g. the scope of rclone sync --interactive A: B: can be restricted: 15809 15810 rclone --min-size 50k --delete-excluded sync A: B: 15811 15812 All files on B: which are less than 50 KiB are deleted because they are 15813 excluded from the rclone sync command. 15814 15815 --dump filters - dump the filters to the output 15816 15817 Dumps the defined filters to standard output in regular expression 15818 format. 15819 15820 Useful for debugging. 15821 15822 Exclude directory based on a file 15823 15824 The --exclude-if-present flag controls whether a directory is within the 15825 scope of an rclone command based on the presence of a named file within 15826 it. The flag can be repeated to check for multiple file names, presence 15827 of any of them will exclude the directory. 15828 15829 This flag has a priority over other filter flags. 15830 15831 E.g. for the following directory structure: 15832 15833 dir1/file1 15834 dir1/dir2/file2 15835 dir1/dir2/dir3/file3 15836 dir1/dir2/dir3/.ignore 15837 15838 The command rclone ls --exclude-if-present .ignore dir1 does not list 15839 dir3, file3 or .ignore. 15840 15841 Metadata filters 15842 15843 The metadata filters work in a very similar way to the normal file name 15844 filters, except they match metadata on the object. 15845 15846 The metadata should be specified as key=value patterns. This may be 15847 wildcarded using the normal filter patterns or regular expressions. 15848 15849 For example if you wished to list only local files with a mode of 100664 15850 you could do that with: 15851 15852 rclone lsf -M --files-only --metadata-include "mode=100664" . 15853 15854 Or if you wished to show files with an atime, mtime or btime at a given 15855 date: 15856 15857 rclone lsf -M --files-only --metadata-include "[abm]time=2022-12-16*" . 15858 15859 Like file filtering, metadata filtering only applies to files not to 15860 directories. 15861 15862 The filters can be applied using these flags. 15863 15864 - --metadata-include - Include metadatas matching pattern 15865 - --metadata-include-from - Read metadata include patterns from file 15866 (use - to read from stdin) 15867 - --metadata-exclude - Exclude metadatas matching pattern 15868 - --metadata-exclude-from - Read metadata exclude patterns from file 15869 (use - to read from stdin) 15870 - --metadata-filter - Add a metadata filtering rule 15871 - --metadata-filter-from - Read metadata filtering patterns from a 15872 file (use - to read from stdin) 15873 15874 Each flag can be repeated. See the section on how filter rules are 15875 applied for more details - these flags work in an identical way to the 15876 file name filtering flags, but instead of file name patterns have 15877 metadata patterns. 15878 15879 Common pitfalls 15880 15881 The most frequent filter support issues on the rclone forum are: 15882 15883 - Not using paths relative to the root of the remote 15884 - Not using / to match from the root of a remote 15885 - Not using ** to match the contents of a directory 15886 15887 GUI (Experimental) 15888 15889 Rclone can serve a web based GUI (graphical user interface). This is 15890 somewhat experimental at the moment so things may be subject to change. 15891 15892 Run this command in a terminal and rclone will download and then display 15893 the GUI in a web browser. 15894 15895 rclone rcd --rc-web-gui 15896 15897 This will produce logs like this and rclone needs to continue to run to 15898 serve the GUI: 15899 15900 2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip 15901 2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip] 15902 2019/08/25 11:40:16 NOTICE: Unzipping 15903 2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/ 15904 15905 This assumes you are running rclone locally on your machine. It is 15906 possible to separate the rclone and the GUI - see below for details. 15907 15908 If you wish to check for updates then you can add --rc-web-gui-update to 15909 the command line. 15910 15911 If you find your GUI broken, you may force it to update by add 15912 --rc-web-gui-force-update. 15913 15914 By default, rclone will open your browser. Add 15915 --rc-web-gui-no-open-browser to disable this feature. 15916 15917 Using the GUI 15918 15919 Once the GUI opens, you will be looking at the dashboard which has an 15920 overall overview. 15921 15922 On the left hand side you will see a series of view buttons you can 15923 click on: 15924 15925 - Dashboard - main overview 15926 - Configs - examine and create new configurations 15927 - Explorer - view, download and upload files to the cloud storage 15928 systems 15929 - Backend - view or alter the backend config 15930 - Log out 15931 15932 (More docs and walkthrough video to come!) 15933 15934 How it works 15935 15936 When you run the rclone rcd --rc-web-gui this is what happens 15937 15938 - Rclone starts but only runs the remote control API ("rc"). 15939 - The API is bound to localhost with an auto-generated username and 15940 password. 15941 - If the API bundle is missing then rclone will download it. 15942 - rclone will start serving the files from the API bundle over the 15943 same port as the API 15944 - rclone will open the browser with a login_token so it can log 15945 straight in. 15946 15947 Advanced use 15948 15949 The rclone rcd may use any of the flags documented on the rc page. 15950 15951 The flag --rc-web-gui is shorthand for 15952 15953 - Download the web GUI if necessary 15954 - Check we are using some authentication 15955 - --rc-user gui 15956 - --rc-pass <random password> 15957 - --rc-serve 15958 15959 These flags can be overridden as desired. 15960 15961 See also the rclone rcd documentation. 15962 15963 Example: Running a public GUI 15964 15965 For example the GUI could be served on a public port over SSL using an 15966 htpasswd file using the following flags: 15967 15968 - --rc-web-gui 15969 - --rc-addr :443 15970 - --rc-htpasswd /path/to/htpasswd 15971 - --rc-cert /path/to/ssl.crt 15972 - --rc-key /path/to/ssl.key 15973 15974 Example: Running a GUI behind a proxy 15975 15976 If you want to run the GUI behind a proxy at /rclone you could use these 15977 flags: 15978 15979 - --rc-web-gui 15980 - --rc-baseurl rclone 15981 - --rc-htpasswd /path/to/htpasswd 15982 15983 Or instead of htpasswd if you just want a single user and password: 15984 15985 - --rc-user me 15986 - --rc-pass mypassword 15987 15988 Project 15989 15990 The GUI is being developed in the: rclone/rclone-webui-react repository. 15991 15992 Bug reports and contributions are very welcome :-) 15993 15994 If you have questions then please ask them on the rclone forum. 15995 15996 Remote controlling rclone with its API 15997 15998 If rclone is run with the --rc flag then it starts an HTTP server which 15999 can be used to remote control rclone using its API. 16000 16001 You can either use the rc command to access the API or use HTTP 16002 directly. 16003 16004 If you just want to run a remote control then see the rcd command. 16005 16006 Supported parameters 16007 16008 --rc 16009 16010 Flag to start the http server listen on remote requests 16011 16012 --rc-addr=IP 16013 16014 IPaddress:Port or :Port to bind server to. (default "localhost:5572") 16015 16016 --rc-cert=KEY 16017 16018 SSL PEM key (concatenation of certificate and CA certificate) 16019 16020 --rc-client-ca=PATH 16021 16022 Client certificate authority to verify clients with 16023 16024 --rc-htpasswd=PATH 16025 16026 htpasswd file - if not provided no authentication is done 16027 16028 --rc-key=PATH 16029 16030 SSL PEM Private key 16031 16032 --rc-max-header-bytes=VALUE 16033 16034 Maximum size of request header (default 4096) 16035 16036 --rc-min-tls-version=VALUE 16037 16038 The minimum TLS version that is acceptable. Valid values are "tls1.0", 16039 "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). 16040 16041 --rc-user=VALUE 16042 16043 User name for authentication. 16044 16045 --rc-pass=VALUE 16046 16047 Password for authentication. 16048 16049 --rc-realm=VALUE 16050 16051 Realm for authentication (default "rclone") 16052 16053 --rc-server-read-timeout=DURATION 16054 16055 Timeout for server reading data (default 1h0m0s) 16056 16057 --rc-server-write-timeout=DURATION 16058 16059 Timeout for server writing data (default 1h0m0s) 16060 16061 --rc-serve 16062 16063 Enable the serving of remote objects via the HTTP interface. This means 16064 objects will be accessible at http://127.0.0.1:5572/ by default, so you 16065 can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/* to see a 16066 listing of the remotes. Objects may be requested from remotes using this 16067 syntax http://127.0.0.1:5572/[remote:path]/path/to/object 16068 16069 Default Off. 16070 16071 --rc-files /path/to/directory 16072 16073 Path to local files to serve on the HTTP server. 16074 16075 If this is set then rclone will serve the files in that directory. It 16076 will also open the root in the web browser if specified. This is for 16077 implementing browser based GUIs for rclone functions. 16078 16079 If --rc-user or --rc-pass is set then the URL that is opened will have 16080 the authorization in the URL in the http://user:pass@localhost/ style. 16081 16082 Default Off. 16083 16084 --rc-enable-metrics 16085 16086 Enable OpenMetrics/Prometheus compatible endpoint at /metrics. 16087 16088 Default Off. 16089 16090 --rc-web-gui 16091 16092 Set this flag to serve the default web gui on the same port as rclone. 16093 16094 Default Off. 16095 16096 --rc-allow-origin 16097 16098 Set the allowed Access-Control-Allow-Origin for rc requests. 16099 16100 Can be used with --rc-web-gui if the rclone is running on different IP 16101 than the web-gui. 16102 16103 Default is IP address on which rc is running. 16104 16105 --rc-web-fetch-url 16106 16107 Set the URL to fetch the rclone-web-gui files from. 16108 16109 Default 16110 https://api.github.com/repos/rclone/rclone-webui-react/releases/latest. 16111 16112 --rc-web-gui-update 16113 16114 Set this flag to check and update rclone-webui-react from the 16115 rc-web-fetch-url. 16116 16117 Default Off. 16118 16119 --rc-web-gui-force-update 16120 16121 Set this flag to force update rclone-webui-react from the 16122 rc-web-fetch-url. 16123 16124 Default Off. 16125 16126 --rc-web-gui-no-open-browser 16127 16128 Set this flag to disable opening browser automatically when using 16129 web-gui. 16130 16131 Default Off. 16132 16133 --rc-job-expire-duration=DURATION 16134 16135 Expire finished async jobs older than DURATION (default 60s). 16136 16137 --rc-job-expire-interval=DURATION 16138 16139 Interval duration to check for expired async jobs (default 10s). 16140 16141 --rc-no-auth 16142 16143 By default rclone will require authorisation to have been set up on the 16144 rc interface in order to use any methods which access any rclone 16145 remotes. Eg operations/list is denied as it involved creating a remote 16146 as is sync/copy. 16147 16148 If this is set then no authorisation will be required on the server to 16149 use these methods. The alternative is to use --rc-user and --rc-pass and 16150 use these credentials in the request. 16151 16152 Default Off. 16153 16154 --rc-baseurl 16155 16156 Prefix for URLs. 16157 16158 Default is root 16159 16160 --rc-template 16161 16162 User-specified template. 16163 16164 Accessing the remote control via the rclone rc command 16165 16166 Rclone itself implements the remote control protocol in its rclone rc 16167 command. 16168 16169 You can use it like this 16170 16171 $ rclone rc rc/noop param1=one param2=two 16172 { 16173 "param1": "one", 16174 "param2": "two" 16175 } 16176 16177 Run rclone rc on its own to see the help for the installed remote 16178 control commands. 16179 16180 JSON input 16181 16182 rclone rc also supports a --json flag which can be used to send more 16183 complicated input parameters. 16184 16185 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop 16186 { 16187 "p1": [ 16188 1, 16189 "2", 16190 null, 16191 4 16192 ], 16193 "p2": { 16194 "a": 1, 16195 "b": 2 16196 } 16197 } 16198 16199 If the parameter being passed is an object then it can be passed as a 16200 JSON string rather than using the --json flag which simplifies the 16201 command line. 16202 16203 rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}' 16204 16205 Rather than 16206 16207 rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}' 16208 16209 Special parameters 16210 16211 The rc interface supports some special parameters which apply to all 16212 commands. These start with _ to show they are different. 16213 16214 Running asynchronous jobs with _async = true 16215 16216 Each rc call is classified as a job and it is assigned its own id. By 16217 default jobs are executed immediately as they are created or 16218 synchronously. 16219 16220 If _async has a true value when supplied to an rc call then it will 16221 return immediately with a job id and the task will be run in the 16222 background. The job/status call can be used to get information of the 16223 background job. The job can be queried for up to 1 minute after it has 16224 finished. 16225 16226 It is recommended that potentially long running jobs, e.g. sync/sync, 16227 sync/copy, sync/move, operations/purge are run with the _async flag to 16228 avoid any potential problems with the HTTP request and response timing 16229 out. 16230 16231 Starting a job with the _async flag: 16232 16233 $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop 16234 { 16235 "jobid": 2 16236 } 16237 16238 Query the status to see if the job has finished. For more information on 16239 the meaning of these return parameters see the job/status call. 16240 16241 $ rclone rc --json '{ "jobid":2 }' job/status 16242 { 16243 "duration": 0.000124163, 16244 "endTime": "2018-10-27T11:38:07.911245881+01:00", 16245 "error": "", 16246 "finished": true, 16247 "id": 2, 16248 "output": { 16249 "_async": true, 16250 "p1": [ 16251 1, 16252 "2", 16253 null, 16254 4 16255 ], 16256 "p2": { 16257 "a": 1, 16258 "b": 2 16259 } 16260 }, 16261 "startTime": "2018-10-27T11:38:07.911121728+01:00", 16262 "success": true 16263 } 16264 16265 job/list can be used to show the running or recently completed jobs 16266 16267 $ rclone rc job/list 16268 { 16269 "jobids": [ 16270 2 16271 ] 16272 } 16273 16274 Setting config flags with _config 16275 16276 If you wish to set config (the equivalent of the global flags) for the 16277 duration of an rc call only then pass in the _config parameter. 16278 16279 This should be in the same format as the config key returned by 16280 options/get. 16281 16282 For example, if you wished to run a sync with the --checksum parameter, 16283 you would pass this parameter in your JSON blob. 16284 16285 "_config":{"CheckSum": true} 16286 16287 If using rclone rc this could be passed as 16288 16289 rclone rc sync/sync ... _config='{"CheckSum": true}' 16290 16291 Any config parameters you don't set will inherit the global defaults 16292 which were set with command line flags or environment variables. 16293 16294 Note that it is possible to set some values as strings or integers - see 16295 data types for more info. Here is an example setting the equivalent of 16296 --buffer-size in string or integer format. 16297 16298 "_config":{"BufferSize": "42M"} 16299 "_config":{"BufferSize": 44040192} 16300 16301 If you wish to check the _config assignment has worked properly then 16302 calling options/local will show what the value got set to. 16303 16304 Setting filter flags with _filter 16305 16306 If you wish to set filters for the duration of an rc call only then pass 16307 in the _filter parameter. 16308 16309 This should be in the same format as the filter key returned by 16310 options/get. 16311 16312 For example, if you wished to run a sync with these flags 16313 16314 --max-size 1M --max-age 42s --include "a" --include "b" 16315 16316 you would pass this parameter in your JSON blob. 16317 16318 "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"} 16319 16320 If using rclone rc this could be passed as 16321 16322 rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}' 16323 16324 Any filter parameters you don't set will inherit the global defaults 16325 which were set with command line flags or environment variables. 16326 16327 Note that it is possible to set some values as strings or integers - see 16328 data types for more info. Here is an example setting the equivalent of 16329 --buffer-size in string or integer format. 16330 16331 "_filter":{"MinSize": "42M"} 16332 "_filter":{"MinSize": 44040192} 16333 16334 If you wish to check the _filter assignment has worked properly then 16335 calling options/local will show what the value got set to. 16336 16337 Assigning operations to groups with _group = value 16338 16339 Each rc call has its own stats group for tracking its metrics. By 16340 default grouping is done by the composite group name from prefix job/ 16341 and id of the job like so job/1. 16342 16343 If _group has a value then stats for that request will be grouped under 16344 that value. This allows caller to group stats under their own name. 16345 16346 Stats for specific group can be accessed by passing group to core/stats: 16347 16348 $ rclone rc --json '{ "group": "job/1" }' core/stats 16349 { 16350 "speed": 12345 16351 ... 16352 } 16353 16354 Data types 16355 16356 When the API returns types, these will mostly be straight forward 16357 integer, string or boolean types. 16358 16359 However some of the types returned by the options/get call and taken by 16360 the options/set calls as well as the vfsOpt, mountOpt and the _config 16361 parameters. 16362 16363 - Duration - these are returned as an integer duration in nanoseconds. 16364 They may be set as an integer, or they may be set with time string, 16365 eg "5s". See the options section for more info. 16366 - Size - these are returned as an integer number of bytes. They may be 16367 set as an integer or they may be set with a size suffix string, eg 16368 "10M". See the options section for more info. 16369 - Enumerated type (such as CutoffMode, DumpFlags, LogLevel, 16370 VfsCacheMode - these will be returned as an integer and may be set 16371 as an integer but more conveniently they can be set as a string, eg 16372 "HARD" for CutoffMode or DEBUG for LogLevel. 16373 - BandwidthSpec - this will be set and returned as a string, eg "1M". 16374 16375 Specifying remotes to work on 16376 16377 Remotes are specified with the fs=, srcFs=, dstFs= parameters depending 16378 on the command being used. 16379 16380 The parameters can be a string as per the rest of rclone, eg 16381 s3:bucket/path or :sftp:/my/dir. They can also be specified as JSON 16382 blobs. 16383 16384 If specifying a JSON blob it should be a object mapping strings to 16385 strings. These values will be used to configure the remote. There are 3 16386 special values which may be set: 16387 16388 - type - set to type to specify a remote called :type: 16389 - _name - set to name to specify a remote called name: 16390 - _root - sets the root of the remote - may be empty 16391 16392 One of _name or type should normally be set. If the local backend is 16393 desired then type should be set to local. If _root isn't specified then 16394 it defaults to the root of the remote. 16395 16396 For example this JSON is equivalent to remote:/tmp 16397 16398 { 16399 "_name": "remote", 16400 "_path": "/tmp" 16401 } 16402 16403 And this is equivalent to :sftp,host='example.com':/tmp 16404 16405 { 16406 "type": "sftp", 16407 "host": "example.com", 16408 "_path": "/tmp" 16409 } 16410 16411 And this is equivalent to /tmp/dir 16412 16413 { 16414 type = "local", 16415 _ path = "/tmp/dir" 16416 } 16417 16418 Supported commands 16419 16420 backend/command: Runs a backend command. 16421 16422 This takes the following parameters: 16423 16424 - command - a string with the command name 16425 - fs - a remote name string e.g. "drive:" 16426 - arg - a list of arguments for the backend command 16427 - opt - a map of string to string of options 16428 16429 Returns: 16430 16431 - result - result from the backend command 16432 16433 Example: 16434 16435 rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2 16436 16437 Returns 16438 16439 { 16440 "result": { 16441 "arg": [ 16442 "path1", 16443 "path2" 16444 ], 16445 "name": "noop", 16446 "opt": { 16447 "blue": "", 16448 "echo": "yes" 16449 } 16450 } 16451 } 16452 16453 Note that this is the direct equivalent of using this "backend" command: 16454 16455 rclone backend noop . -o echo=yes -o blue path1 path2 16456 16457 Note that arguments must be preceded by the "-a" flag 16458 16459 See the backend command for more information. 16460 16461 Authentication is required for this call. 16462 16463 cache/expire: Purge a remote from cache 16464 16465 Purge a remote from the cache backend. Supports either a directory or a 16466 file. Params: - remote = path to remote (required) - withData = 16467 true/false to delete cached data (chunks) as well (optional) 16468 16469 Eg 16470 16471 rclone rc cache/expire remote=path/to/sub/folder/ 16472 rclone rc cache/expire remote=/ withData=true 16473 16474 cache/fetch: Fetch file chunks 16475 16476 Ensure the specified file chunks are cached on disk. 16477 16478 The chunks= parameter specifies the file chunks to check. It takes a 16479 comma separated list of array slice indices. The slice indices are 16480 similar to Python slices: start[:end] 16481 16482 start is the 0 based chunk number from the beginning of the file to 16483 fetch inclusive. end is 0 based chunk number from the beginning of the 16484 file to fetch exclusive. Both values can be negative, in which case they 16485 count from the back of the file. The value "-5:" represents the last 5 16486 chunks of a file. 16487 16488 Some valid examples are: ":5,-5:" -> the first and last five chunks 16489 "0,-2" -> the first and the second last chunk "0:10" -> the first ten 16490 chunks 16491 16492 Any parameter with a key that starts with "file" can be used to specify 16493 files to fetch, e.g. 16494 16495 rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye 16496 16497 File names will automatically be encrypted when the a crypt remote is 16498 used on top of the cache. 16499 16500 cache/stats: Get cache stats 16501 16502 Show statistics for the cache remote. 16503 16504 config/create: create the config for a remote. 16505 16506 This takes the following parameters: 16507 16508 - name - name of remote 16509 - parameters - a map of { "key": "value" } pairs 16510 - type - type of the new remote 16511 - opt - a dictionary of options to control the configuration 16512 - obscure - declare passwords are plain and need obscuring 16513 - noObscure - declare passwords are already obscured and don't 16514 need obscuring 16515 - nonInteractive - don't interact with a user, return questions 16516 - continue - continue the config process with an answer 16517 - all - ask all the config questions not just the post config ones 16518 - state - state to restart with - used with continue 16519 - result - result to restart with - used with continue 16520 16521 See the config create command for more information on the above. 16522 16523 Authentication is required for this call. 16524 16525 config/delete: Delete a remote in the config file. 16526 16527 Parameters: 16528 16529 - name - name of remote to delete 16530 16531 See the config delete command for more information on the above. 16532 16533 Authentication is required for this call. 16534 16535 config/dump: Dumps the config file. 16536 16537 Returns a JSON object: - key: value 16538 16539 Where keys are remote names and values are the config parameters. 16540 16541 See the config dump command for more information on the above. 16542 16543 Authentication is required for this call. 16544 16545 config/get: Get a remote in the config file. 16546 16547 Parameters: 16548 16549 - name - name of remote to get 16550 16551 See the config dump command for more information on the above. 16552 16553 Authentication is required for this call. 16554 16555 config/listremotes: Lists the remotes in the config file and defined in environment variables. 16556 16557 Returns - remotes - array of remote names 16558 16559 See the listremotes command for more information on the above. 16560 16561 Authentication is required for this call. 16562 16563 config/password: password the config for a remote. 16564 16565 This takes the following parameters: 16566 16567 - name - name of remote 16568 - parameters - a map of { "key": "value" } pairs 16569 16570 See the config password command for more information on the above. 16571 16572 Authentication is required for this call. 16573 16574 config/paths: Reads the config file path and other important paths. 16575 16576 Returns a JSON object with the following keys: 16577 16578 - config: path to config file 16579 - cache: path to root of cache directory 16580 - temp: path to root of temporary directory 16581 16582 Eg 16583 16584 { 16585 "cache": "/home/USER/.cache/rclone", 16586 "config": "/home/USER/.rclone.conf", 16587 "temp": "/tmp" 16588 } 16589 16590 See the config paths command for more information on the above. 16591 16592 Authentication is required for this call. 16593 16594 config/providers: Shows how providers are configured in the config file. 16595 16596 Returns a JSON object: - providers - array of objects 16597 16598 See the config providers command for more information on the above. 16599 16600 Authentication is required for this call. 16601 16602 config/setpath: Set the path of the config file 16603 16604 Parameters: 16605 16606 - path - path to the config file to use 16607 16608 Authentication is required for this call. 16609 16610 config/update: update the config for a remote. 16611 16612 This takes the following parameters: 16613 16614 - name - name of remote 16615 - parameters - a map of { "key": "value" } pairs 16616 - opt - a dictionary of options to control the configuration 16617 - obscure - declare passwords are plain and need obscuring 16618 - noObscure - declare passwords are already obscured and don't 16619 need obscuring 16620 - nonInteractive - don't interact with a user, return questions 16621 - continue - continue the config process with an answer 16622 - all - ask all the config questions not just the post config ones 16623 - state - state to restart with - used with continue 16624 - result - result to restart with - used with continue 16625 16626 See the config update command for more information on the above. 16627 16628 Authentication is required for this call. 16629 16630 core/bwlimit: Set the bandwidth limit. 16631 16632 This sets the bandwidth limit to the string passed in. This should be a 16633 single bandwidth limit entry or a pair of upload:download bandwidth. 16634 16635 Eg 16636 16637 rclone rc core/bwlimit rate=off 16638 { 16639 "bytesPerSecond": -1, 16640 "bytesPerSecondTx": -1, 16641 "bytesPerSecondRx": -1, 16642 "rate": "off" 16643 } 16644 rclone rc core/bwlimit rate=1M 16645 { 16646 "bytesPerSecond": 1048576, 16647 "bytesPerSecondTx": 1048576, 16648 "bytesPerSecondRx": 1048576, 16649 "rate": "1M" 16650 } 16651 rclone rc core/bwlimit rate=1M:100k 16652 { 16653 "bytesPerSecond": 1048576, 16654 "bytesPerSecondTx": 1048576, 16655 "bytesPerSecondRx": 131072, 16656 "rate": "1M" 16657 } 16658 16659 If the rate parameter is not supplied then the bandwidth is queried 16660 16661 rclone rc core/bwlimit 16662 { 16663 "bytesPerSecond": 1048576, 16664 "bytesPerSecondTx": 1048576, 16665 "bytesPerSecondRx": 1048576, 16666 "rate": "1M" 16667 } 16668 16669 The format of the parameter is exactly the same as passed to --bwlimit 16670 except only one bandwidth may be specified. 16671 16672 In either case "rate" is returned as a human-readable string, and 16673 "bytesPerSecond" is returned as a number. 16674 16675 core/command: Run a rclone terminal command over rc. 16676 16677 This takes the following parameters: 16678 16679 - command - a string with the command name. 16680 - arg - a list of arguments for the backend command. 16681 - opt - a map of string to string of options. 16682 - returnType - one of ("COMBINED_OUTPUT", "STREAM", 16683 "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR"). 16684 - Defaults to "COMBINED_OUTPUT" if not set. 16685 - The STREAM returnTypes will write the output to the body of the 16686 HTTP message. 16687 - The COMBINED_OUTPUT will write the output to the "result" 16688 parameter. 16689 16690 Returns: 16691 16692 - result - result from the backend command. 16693 - Only set when using returnType "COMBINED_OUTPUT". 16694 - error - set if rclone exits with an error code. 16695 - returnType - one of ("COMBINED_OUTPUT", "STREAM", 16696 "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR"). 16697 16698 Example: 16699 16700 rclone rc core/command command=ls -a mydrive:/ -o max-depth=1 16701 rclone rc core/command -a ls -a mydrive:/ -o max-depth=1 16702 16703 Returns: 16704 16705 { 16706 "error": false, 16707 "result": "<Raw command line output>" 16708 } 16709 16710 OR 16711 { 16712 "error": true, 16713 "result": "<Raw command line output>" 16714 } 16715 16716 Authentication is required for this call. 16717 16718 core/du: Returns disk usage of a locally attached disk. 16719 16720 This returns the disk usage for the local directory passed in as dir. 16721 16722 If the directory is not passed in, it defaults to the directory pointed 16723 to by --cache-dir. 16724 16725 - dir - string (optional) 16726 16727 Returns: 16728 16729 { 16730 "dir": "/", 16731 "info": { 16732 "Available": 361769115648, 16733 "Free": 361785892864, 16734 "Total": 982141468672 16735 } 16736 } 16737 16738 core/gc: Runs a garbage collection. 16739 16740 This tells the go runtime to do a garbage collection run. It isn't 16741 necessary to call this normally, but it can be useful for debugging 16742 memory problems. 16743 16744 core/group-list: Returns list of stats. 16745 16746 This returns list of stats groups currently in memory. 16747 16748 Returns the following values: 16749 16750 { 16751 "groups": an array of group names: 16752 [ 16753 "group1", 16754 "group2", 16755 ... 16756 ] 16757 } 16758 16759 core/memstats: Returns the memory statistics 16760 16761 This returns the memory statistics of the running program. What the 16762 values mean are explained in the go docs: 16763 https://golang.org/pkg/runtime/#MemStats 16764 16765 The most interesting values for most people are: 16766 16767 - HeapAlloc - this is the amount of memory rclone is actually using 16768 - HeapSys - this is the amount of memory rclone has obtained from the 16769 OS 16770 - Sys - this is the total amount of memory requested from the OS 16771 - It is virtual memory so may include unused memory 16772 16773 core/obscure: Obscures a string passed in. 16774 16775 Pass a clear string and rclone will obscure it for the config file: - 16776 clear - string 16777 16778 Returns: - obscured - string 16779 16780 core/pid: Return PID of current process 16781 16782 This returns PID of current process. Useful for stopping rclone process. 16783 16784 core/quit: Terminates the app. 16785 16786 (Optional) Pass an exit code to be used for terminating the app: - 16787 exitCode - int 16788 16789 core/stats: Returns stats about current transfers. 16790 16791 This returns all available stats: 16792 16793 rclone rc core/stats 16794 16795 If group is not provided then summed up stats for all groups will be 16796 returned. 16797 16798 Parameters 16799 16800 - group - name of the stats group (string) 16801 16802 Returns the following values: 16803 16804 { 16805 "bytes": total transferred bytes since the start of the group, 16806 "checks": number of files checked, 16807 "deletes" : number of files deleted, 16808 "elapsedTime": time in floating point seconds since rclone was started, 16809 "errors": number of errors, 16810 "eta": estimated time in seconds until the group completes, 16811 "fatalError": boolean whether there has been at least one fatal error, 16812 "lastError": last error string, 16813 "renames" : number of files renamed, 16814 "retryError": boolean showing whether there has been at least one non-NoRetryError, 16815 "serverSideCopies": number of server side copies done, 16816 "serverSideCopyBytes": number bytes server side copied, 16817 "serverSideMoves": number of server side moves done, 16818 "serverSideMoveBytes": number bytes server side moved, 16819 "speed": average speed in bytes per second since start of the group, 16820 "totalBytes": total number of bytes in the group, 16821 "totalChecks": total number of checks in the group, 16822 "totalTransfers": total number of transfers in the group, 16823 "transferTime" : total time spent on running jobs, 16824 "transfers": number of transferred files, 16825 "transferring": an array of currently active file transfers: 16826 [ 16827 { 16828 "bytes": total transferred bytes for this file, 16829 "eta": estimated time in seconds until file transfer completion 16830 "name": name of the file, 16831 "percentage": progress of the file transfer in percent, 16832 "speed": average speed over the whole transfer in bytes per second, 16833 "speedAvg": current speed in bytes per second as an exponentially weighted moving average, 16834 "size": size of the file in bytes 16835 } 16836 ], 16837 "checking": an array of names of currently active file checks 16838 [] 16839 } 16840 16841 Values for "transferring", "checking" and "lastError" are only assigned 16842 if data is available. The value for "eta" is null if an eta cannot be 16843 determined. 16844 16845 core/stats-delete: Delete stats group. 16846 16847 This deletes entire stats group. 16848 16849 Parameters 16850 16851 - group - name of the stats group (string) 16852 16853 core/stats-reset: Reset stats. 16854 16855 This clears counters, errors and finished transfers for all stats or 16856 specific stats group if group is provided. 16857 16858 Parameters 16859 16860 - group - name of the stats group (string) 16861 16862 core/transferred: Returns stats about completed transfers. 16863 16864 This returns stats about completed transfers: 16865 16866 rclone rc core/transferred 16867 16868 If group is not provided then completed transfers for all groups will be 16869 returned. 16870 16871 Note only the last 100 completed transfers are returned. 16872 16873 Parameters 16874 16875 - group - name of the stats group (string) 16876 16877 Returns the following values: 16878 16879 { 16880 "transferred": an array of completed transfers (including failed ones): 16881 [ 16882 { 16883 "name": name of the file, 16884 "size": size of the file in bytes, 16885 "bytes": total transferred bytes for this file, 16886 "checked": if the transfer is only checked (skipped, deleted), 16887 "timestamp": integer representing millisecond unix epoch, 16888 "error": string description of the error (empty if successful), 16889 "jobid": id of the job that this transfer belongs to 16890 } 16891 ] 16892 } 16893 16894 core/version: Shows the current version of rclone and the go runtime. 16895 16896 This shows the current version of go and the go runtime: 16897 16898 - version - rclone version, e.g. "v1.53.0" 16899 - decomposed - version number as [major, minor, patch] 16900 - isGit - boolean - true if this was compiled from the git version 16901 - isBeta - boolean - true if this is a beta version 16902 - os - OS in use as according to Go 16903 - arch - cpu architecture in use according to Go 16904 - goVersion - version of Go runtime in use 16905 - linking - type of rclone executable (static or dynamic) 16906 - goTags - space separated build tags or "none" 16907 16908 debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. 16909 16910 SetBlockProfileRate controls the fraction of goroutine blocking events 16911 that are reported in the blocking profile. The profiler aims to sample 16912 an average of one blocking event per rate nanoseconds spent blocked. 16913 16914 To include every blocking event in the profile, pass rate = 1. To turn 16915 off profiling entirely, pass rate <= 0. 16916 16917 After calling this you can use this to see the blocking profile: 16918 16919 go tool pprof http://localhost:5572/debug/pprof/block 16920 16921 Parameters: 16922 16923 - rate - int 16924 16925 debug/set-gc-percent: Call runtime/debug.SetGCPercent for setting the garbage collection target percentage. 16926 16927 SetGCPercent sets the garbage collection target percentage: a collection 16928 is triggered when the ratio of freshly allocated data to live data 16929 remaining after the previous collection reaches this percentage. 16930 SetGCPercent returns the previous setting. The initial setting is the 16931 value of the GOGC environment variable at startup, or 100 if the 16932 variable is not set. 16933 16934 This setting may be effectively reduced in order to maintain a memory 16935 limit. A negative percentage effectively disables garbage collection, 16936 unless the memory limit is reached. 16937 16938 See https://pkg.go.dev/runtime/debug#SetMemoryLimit for more details. 16939 16940 Parameters: 16941 16942 - gc-percent - int 16943 16944 debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling. 16945 16946 SetMutexProfileFraction controls the fraction of mutex contention events 16947 that are reported in the mutex profile. On average 1/rate events are 16948 reported. The previous rate is returned. 16949 16950 To turn off profiling entirely, pass rate 0. To just read the current 16951 rate, pass rate < 0. (For n>1 the details of sampling may change.) 16952 16953 Once this is set you can look use this to profile the mutex contention: 16954 16955 go tool pprof http://localhost:5572/debug/pprof/mutex 16956 16957 Parameters: 16958 16959 - rate - int 16960 16961 Results: 16962 16963 - previousRate - int 16964 16965 debug/set-soft-memory-limit: Call runtime/debug.SetMemoryLimit for setting a soft memory limit for the runtime. 16966 16967 SetMemoryLimit provides the runtime with a soft memory limit. 16968 16969 The runtime undertakes several processes to try to respect this memory 16970 limit, including adjustments to the frequency of garbage collections and 16971 returning memory to the underlying system more aggressively. This limit 16972 will be respected even if GOGC=off (or, if SetGCPercent(-1) is 16973 executed). 16974 16975 The input limit is provided as bytes, and includes all memory mapped, 16976 managed, and not released by the Go runtime. Notably, it does not 16977 account for space used by the Go binary and memory external to Go, such 16978 as memory managed by the underlying system on behalf of the process, or 16979 memory managed by non-Go code inside the same process. Examples of 16980 excluded memory sources include: OS kernel memory held on behalf of the 16981 process, memory allocated by C code, and memory mapped by syscall.Mmap 16982 (because it is not managed by the Go runtime). 16983 16984 A zero limit or a limit that's lower than the amount of memory used by 16985 the Go runtime may cause the garbage collector to run nearly 16986 continuously. However, the application may still make progress. 16987 16988 The memory limit is always respected by the Go runtime, so to 16989 effectively disable this behavior, set the limit very high. 16990 math.MaxInt64 is the canonical value for disabling the limit, but values 16991 much greater than the available memory on the underlying system work 16992 just as well. 16993 16994 See https://go.dev/doc/gc-guide for a detailed guide explaining the soft 16995 memory limit in more detail, as well as a variety of common use-cases 16996 and scenarios. 16997 16998 SetMemoryLimit returns the previously set memory limit. A negative input 16999 does not adjust the limit, and allows for retrieval of the currently set 17000 memory limit. 17001 17002 Parameters: 17003 17004 - mem-limit - int 17005 17006 fscache/clear: Clear the Fs cache. 17007 17008 This clears the fs cache. This is where remotes created from backends 17009 are cached for a short while to make repeated rc calls more efficient. 17010 17011 If you change the parameters of a backend then you may want to call this 17012 to clear an existing remote out of the cache before re-creating it. 17013 17014 Authentication is required for this call. 17015 17016 fscache/entries: Returns the number of entries in the fs cache. 17017 17018 This returns the number of entries in the fs cache. 17019 17020 Returns - entries - number of items in the cache 17021 17022 Authentication is required for this call. 17023 17024 job/list: Lists the IDs of the running jobs 17025 17026 Parameters: None. 17027 17028 Results: 17029 17030 - executeId - string id of rclone executing (change after restart) 17031 - jobids - array of integer job ids (starting at 1 on each restart) 17032 17033 job/status: Reads the status of the job ID 17034 17035 Parameters: 17036 17037 - jobid - id of the job (integer). 17038 17039 Results: 17040 17041 - finished - boolean 17042 - duration - time in seconds that the job ran for 17043 - endTime - time the job finished (e.g. 17044 "2018-10-26T18:50:20.528746884+01:00") 17045 - error - error from the job or empty string for no error 17046 - finished - boolean whether the job has finished or not 17047 - id - as passed in above 17048 - startTime - time the job started (e.g. 17049 "2018-10-26T18:50:20.528336039+01:00") 17050 - success - boolean - true for success false otherwise 17051 - output - output of the job as would have been returned if called 17052 synchronously 17053 - progress - output of the progress related to the underlying job 17054 17055 job/stop: Stop the running job 17056 17057 Parameters: 17058 17059 - jobid - id of the job (integer). 17060 17061 job/stopgroup: Stop all running jobs in a group 17062 17063 Parameters: 17064 17065 - group - name of the group (string). 17066 17067 mount/listmounts: Show current mount points 17068 17069 This shows currently mounted points, which can be used for performing an 17070 unmount. 17071 17072 This takes no parameters and returns 17073 17074 - mountPoints: list of current mount points 17075 17076 Eg 17077 17078 rclone rc mount/listmounts 17079 17080 Authentication is required for this call. 17081 17082 mount/mount: Create a new mount point 17083 17084 rclone allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's 17085 cloud storage systems as a file system with FUSE. 17086 17087 If no mountType is provided, the priority is given as follows: 1. mount 17088 2.cmount 3.mount2 17089 17090 This takes the following parameters: 17091 17092 - fs - a remote path to be mounted (required) 17093 - mountPoint: valid path on the local machine (required) 17094 - mountType: one of the values (mount, cmount, mount2) specifies the 17095 mount implementation to use 17096 - mountOpt: a JSON object with Mount options in. 17097 - vfsOpt: a JSON object with VFS options in. 17098 17099 Example: 17100 17101 rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint 17102 rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint mountType=mount 17103 rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"CacheMode": 2}' mountOpt='{"AllowOther": true}' 17104 17105 The vfsOpt are as described in options/get and can be seen in the the 17106 "vfs" section when running and the mountOpt can be seen in the "mount" 17107 section: 17108 17109 rclone rc options/get 17110 17111 Authentication is required for this call. 17112 17113 mount/types: Show all possible mount types 17114 17115 This shows all possible mount types and returns them as a list. 17116 17117 This takes no parameters and returns 17118 17119 - mountTypes: list of mount types 17120 17121 The mount types are strings like "mount", "mount2", "cmount" and can be 17122 passed to mount/mount as the mountType parameter. 17123 17124 Eg 17125 17126 rclone rc mount/types 17127 17128 Authentication is required for this call. 17129 17130 mount/unmount: Unmount selected active mount 17131 17132 rclone allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's 17133 cloud storage systems as a file system with FUSE. 17134 17135 This takes the following parameters: 17136 17137 - mountPoint: valid path on the local machine where the mount was 17138 created (required) 17139 17140 Example: 17141 17142 rclone rc mount/unmount mountPoint=/home/<user>/mountPoint 17143 17144 Authentication is required for this call. 17145 17146 mount/unmountall: Unmount all active mounts 17147 17148 rclone allows Linux, FreeBSD, macOS and Windows to mount any of Rclone's 17149 cloud storage systems as a file system with FUSE. 17150 17151 This takes no parameters and returns error if unmount does not succeed. 17152 17153 Eg 17154 17155 rclone rc mount/unmountall 17156 17157 Authentication is required for this call. 17158 17159 operations/about: Return the space used on the remote 17160 17161 This takes the following parameters: 17162 17163 - fs - a remote name string e.g. "drive:" 17164 17165 The result is as returned from rclone about --json 17166 17167 See the about command for more information on the above. 17168 17169 Authentication is required for this call. 17170 17171 operations/check: check the source and destination are the same 17172 17173 Checks the files in the source and destination match. It compares sizes 17174 and hashes and logs a report of files that don't match. It doesn't alter 17175 the source or destination. 17176 17177 This takes the following parameters: 17178 17179 - srcFs - a remote name string e.g. "drive:" for the source, "/" for 17180 local filesystem 17181 - dstFs - a remote name string e.g. "drive2:" for the destination, "/" 17182 for local filesystem 17183 - download - check by downloading rather than with hash 17184 - checkFileHash - treat checkFileFs:checkFileRemote as a SUM file with 17185 hashes of given type 17186 - checkFileFs - treat checkFileFs:checkFileRemote as a SUM file with 17187 hashes of given type 17188 - checkFileRemote - treat checkFileFs:checkFileRemote as a SUM file 17189 with hashes of given type 17190 - oneWay - check one way only, source files must exist on remote 17191 - combined - make a combined report of changes (default false) 17192 - missingOnSrc - report all files missing from the source (default 17193 true) 17194 - missingOnDst - report all files missing from the destination 17195 (default true) 17196 - match - report all matching files (default false) 17197 - differ - report all non-matching files (default true) 17198 - error - report all files with errors (hashing or reading) (default 17199 true) 17200 17201 If you supply the download flag, it will download the data from both 17202 remotes and check them against each other on the fly. This can be useful 17203 for remotes that don't support hashes or if you really want to check all 17204 the data. 17205 17206 If you supply the size-only global flag, it will only compare the sizes 17207 not the hashes as well. Use this for a quick check. 17208 17209 If you supply the checkFileHash option with a valid hash name, the 17210 checkFileFs:checkFileRemote must point to a text file in the SUM format. 17211 This treats the checksum file as the source and dstFs as the 17212 destination. Note that srcFs is not used and should not be supplied in 17213 this case. 17214 17215 Returns: 17216 17217 - success - true if no error, false otherwise 17218 - status - textual summary of check, OK or text string 17219 - hashType - hash used in check, may be missing 17220 - combined - array of strings of combined report of changes 17221 - missingOnSrc - array of strings of all files missing from the source 17222 - missingOnDst - array of strings of all files missing from the 17223 destination 17224 - match - array of strings of all matching files 17225 - differ - array of strings of all non-matching files 17226 - error - array of strings of all files with errors (hashing or 17227 reading) 17228 17229 Authentication is required for this call. 17230 17231 operations/cleanup: Remove trashed files in the remote or path 17232 17233 This takes the following parameters: 17234 17235 - fs - a remote name string e.g. "drive:" 17236 17237 See the cleanup command for more information on the above. 17238 17239 Authentication is required for this call. 17240 17241 operations/copyfile: Copy a file from source remote to destination remote 17242 17243 This takes the following parameters: 17244 17245 - srcFs - a remote name string e.g. "drive:" for the source, "/" for 17246 local filesystem 17247 - srcRemote - a path within that remote e.g. "file.txt" for the source 17248 - dstFs - a remote name string e.g. "drive2:" for the destination, "/" 17249 for local filesystem 17250 - dstRemote - a path within that remote e.g. "file2.txt" for the 17251 destination 17252 17253 Authentication is required for this call. 17254 17255 operations/copyurl: Copy the URL to the object 17256 17257 This takes the following parameters: 17258 17259 - fs - a remote name string e.g. "drive:" 17260 - remote - a path within that remote e.g. "dir" 17261 - url - string, URL to read from 17262 - autoFilename - boolean, set to true to retrieve destination file 17263 name from url 17264 17265 See the copyurl command for more information on the above. 17266 17267 Authentication is required for this call. 17268 17269 operations/delete: Remove files in the path 17270 17271 This takes the following parameters: 17272 17273 - fs - a remote name string e.g. "drive:" 17274 17275 See the delete command for more information on the above. 17276 17277 Authentication is required for this call. 17278 17279 operations/deletefile: Remove the single file pointed to 17280 17281 This takes the following parameters: 17282 17283 - fs - a remote name string e.g. "drive:" 17284 - remote - a path within that remote e.g. "dir" 17285 17286 See the deletefile command for more information on the above. 17287 17288 Authentication is required for this call. 17289 17290 operations/fsinfo: Return information about the remote 17291 17292 This takes the following parameters: 17293 17294 - fs - a remote name string e.g. "drive:" 17295 17296 This returns info about the remote passed in; 17297 17298 { 17299 // optional features and whether they are available or not 17300 "Features": { 17301 "About": true, 17302 "BucketBased": false, 17303 "BucketBasedRootOK": false, 17304 "CanHaveEmptyDirectories": true, 17305 "CaseInsensitive": false, 17306 "ChangeNotify": false, 17307 "CleanUp": false, 17308 "Command": true, 17309 "Copy": false, 17310 "DirCacheFlush": false, 17311 "DirMove": true, 17312 "Disconnect": false, 17313 "DuplicateFiles": false, 17314 "GetTier": false, 17315 "IsLocal": true, 17316 "ListR": false, 17317 "MergeDirs": false, 17318 "MetadataInfo": true, 17319 "Move": true, 17320 "OpenWriterAt": true, 17321 "PublicLink": false, 17322 "Purge": true, 17323 "PutStream": true, 17324 "PutUnchecked": false, 17325 "ReadMetadata": true, 17326 "ReadMimeType": false, 17327 "ServerSideAcrossConfigs": false, 17328 "SetTier": false, 17329 "SetWrapper": false, 17330 "Shutdown": false, 17331 "SlowHash": true, 17332 "SlowModTime": false, 17333 "UnWrap": false, 17334 "UserInfo": false, 17335 "UserMetadata": true, 17336 "WrapFs": false, 17337 "WriteMetadata": true, 17338 "WriteMimeType": false 17339 }, 17340 // Names of hashes available 17341 "Hashes": [ 17342 "md5", 17343 "sha1", 17344 "whirlpool", 17345 "crc32", 17346 "sha256", 17347 "dropbox", 17348 "mailru", 17349 "quickxor" 17350 ], 17351 "Name": "local", // Name as created 17352 "Precision": 1, // Precision of timestamps in ns 17353 "Root": "/", // Path as created 17354 "String": "Local file system at /", // how the remote will appear in logs 17355 // Information about the system metadata for this backend 17356 "MetadataInfo": { 17357 "System": { 17358 "atime": { 17359 "Help": "Time of last access", 17360 "Type": "RFC 3339", 17361 "Example": "2006-01-02T15:04:05.999999999Z07:00" 17362 }, 17363 "btime": { 17364 "Help": "Time of file birth (creation)", 17365 "Type": "RFC 3339", 17366 "Example": "2006-01-02T15:04:05.999999999Z07:00" 17367 }, 17368 "gid": { 17369 "Help": "Group ID of owner", 17370 "Type": "decimal number", 17371 "Example": "500" 17372 }, 17373 "mode": { 17374 "Help": "File type and mode", 17375 "Type": "octal, unix style", 17376 "Example": "0100664" 17377 }, 17378 "mtime": { 17379 "Help": "Time of last modification", 17380 "Type": "RFC 3339", 17381 "Example": "2006-01-02T15:04:05.999999999Z07:00" 17382 }, 17383 "rdev": { 17384 "Help": "Device ID (if special file)", 17385 "Type": "hexadecimal", 17386 "Example": "1abc" 17387 }, 17388 "uid": { 17389 "Help": "User ID of owner", 17390 "Type": "decimal number", 17391 "Example": "500" 17392 } 17393 }, 17394 "Help": "Textual help string\n" 17395 } 17396 } 17397 17398 This command does not have a command line equivalent so use this 17399 instead: 17400 17401 rclone rc --loopback operations/fsinfo fs=remote: 17402 17403 operations/hashsum: Produces a hashsum file for all the objects in the path. 17404 17405 Produces a hash file for all the objects in the path using the hash 17406 named. The output is in the same format as the standard md5sum/sha1sum 17407 tool. 17408 17409 This takes the following parameters: 17410 17411 - fs - a remote name string e.g. "drive:" for the source, "/" for 17412 local filesystem 17413 - this can point to a file and just that file will be returned in 17414 the listing. 17415 - hashType - type of hash to be used 17416 - download - check by downloading rather than with hash (boolean) 17417 - base64 - output the hashes in base64 rather than hex (boolean) 17418 17419 If you supply the download flag, it will download the data from the 17420 remote and create the hash on the fly. This can be useful for remotes 17421 that don't support the given hash or if you really want to check all the 17422 data. 17423 17424 Note that if you wish to supply a checkfile to check hashes against the 17425 current files then you should use operations/check instead of 17426 operations/hashsum. 17427 17428 Returns: 17429 17430 - hashsum - array of strings of the hashes 17431 - hashType - type of hash used 17432 17433 Example: 17434 17435 $ rclone rc --loopback operations/hashsum fs=bin hashType=MD5 download=true base64=true 17436 { 17437 "hashType": "md5", 17438 "hashsum": [ 17439 "WTSVLpuiXyJO_kGzJerRLg== backend-versions.sh", 17440 "v1b_OlWCJO9LtNq3EIKkNQ== bisect-go-rclone.sh", 17441 "VHbmHzHh4taXzgag8BAIKQ== bisect-rclone.sh", 17442 ] 17443 } 17444 17445 See the hashsum command for more information on the above. 17446 17447 Authentication is required for this call. 17448 17449 operations/list: List the given remote and path in JSON format 17450 17451 This takes the following parameters: 17452 17453 - fs - a remote name string e.g. "drive:" 17454 - remote - a path within that remote e.g. "dir" 17455 - opt - a dictionary of options to control the listing (optional) 17456 - recurse - If set recurse directories 17457 - noModTime - If set return modification time 17458 - showEncrypted - If set show decrypted names 17459 - showOrigIDs - If set show the IDs for each item if known 17460 - showHash - If set return a dictionary of hashes 17461 - noMimeType - If set don't show mime types 17462 - dirsOnly - If set only show directories 17463 - filesOnly - If set only show files 17464 - metadata - If set return metadata of objects also 17465 - hashTypes - array of strings of hash types to show if showHash 17466 set 17467 17468 Returns: 17469 17470 - list 17471 - This is an array of objects as described in the lsjson command 17472 17473 See the lsjson command for more information on the above and examples. 17474 17475 Authentication is required for this call. 17476 17477 operations/mkdir: Make a destination directory or container 17478 17479 This takes the following parameters: 17480 17481 - fs - a remote name string e.g. "drive:" 17482 - remote - a path within that remote e.g. "dir" 17483 17484 See the mkdir command for more information on the above. 17485 17486 Authentication is required for this call. 17487 17488 operations/movefile: Move a file from source remote to destination remote 17489 17490 This takes the following parameters: 17491 17492 - srcFs - a remote name string e.g. "drive:" for the source, "/" for 17493 local filesystem 17494 - srcRemote - a path within that remote e.g. "file.txt" for the source 17495 - dstFs - a remote name string e.g. "drive2:" for the destination, "/" 17496 for local filesystem 17497 - dstRemote - a path within that remote e.g. "file2.txt" for the 17498 destination 17499 17500 Authentication is required for this call. 17501 17502 operations/publiclink: Create or retrieve a public link to the given file or folder. 17503 17504 This takes the following parameters: 17505 17506 - fs - a remote name string e.g. "drive:" 17507 - remote - a path within that remote e.g. "dir" 17508 - unlink - boolean - if set removes the link rather than adding it 17509 (optional) 17510 - expire - string - the expiry time of the link e.g. "1d" (optional) 17511 17512 Returns: 17513 17514 - url - URL of the resource 17515 17516 See the link command for more information on the above. 17517 17518 Authentication is required for this call. 17519 17520 operations/purge: Remove a directory or container and all of its contents 17521 17522 This takes the following parameters: 17523 17524 - fs - a remote name string e.g. "drive:" 17525 - remote - a path within that remote e.g. "dir" 17526 17527 See the purge command for more information on the above. 17528 17529 Authentication is required for this call. 17530 17531 operations/rmdir: Remove an empty directory or container 17532 17533 This takes the following parameters: 17534 17535 - fs - a remote name string e.g. "drive:" 17536 - remote - a path within that remote e.g. "dir" 17537 17538 See the rmdir command for more information on the above. 17539 17540 Authentication is required for this call. 17541 17542 operations/rmdirs: Remove all the empty directories in the path 17543 17544 This takes the following parameters: 17545 17546 - fs - a remote name string e.g. "drive:" 17547 - remote - a path within that remote e.g. "dir" 17548 - leaveRoot - boolean, set to true not to delete the root 17549 17550 See the rmdirs command for more information on the above. 17551 17552 Authentication is required for this call. 17553 17554 operations/settier: Changes storage tier or class on all files in the path 17555 17556 This takes the following parameters: 17557 17558 - fs - a remote name string e.g. "drive:" 17559 17560 See the settier command for more information on the above. 17561 17562 Authentication is required for this call. 17563 17564 operations/settierfile: Changes storage tier or class on the single file pointed to 17565 17566 This takes the following parameters: 17567 17568 - fs - a remote name string e.g. "drive:" 17569 - remote - a path within that remote e.g. "dir" 17570 17571 See the settierfile command for more information on the above. 17572 17573 Authentication is required for this call. 17574 17575 operations/size: Count the number of bytes and files in remote 17576 17577 This takes the following parameters: 17578 17579 - fs - a remote name string e.g. "drive:path/to/dir" 17580 17581 Returns: 17582 17583 - count - number of files 17584 - bytes - number of bytes in those files 17585 17586 See the size command for more information on the above. 17587 17588 Authentication is required for this call. 17589 17590 operations/stat: Give information about the supplied file or directory 17591 17592 This takes the following parameters 17593 17594 - fs - a remote name string eg "drive:" 17595 - remote - a path within that remote eg "dir" 17596 - opt - a dictionary of options to control the listing (optional) 17597 - see operations/list for the options 17598 17599 The result is 17600 17601 - item - an object as described in the lsjson command. Will be null if 17602 not found. 17603 17604 Note that if you are only interested in files then it is much more 17605 efficient to set the filesOnly flag in the options. 17606 17607 See the lsjson command for more information on the above and examples. 17608 17609 Authentication is required for this call. 17610 17611 operations/uploadfile: Upload file using multiform/form-data 17612 17613 This takes the following parameters: 17614 17615 - fs - a remote name string e.g. "drive:" 17616 - remote - a path within that remote e.g. "dir" 17617 - each part in body represents a file to be uploaded 17618 17619 See the uploadfile command for more information on the above. 17620 17621 Authentication is required for this call. 17622 17623 options/blocks: List all the option blocks 17624 17625 Returns: - options - a list of the options block names 17626 17627 options/get: Get all the global options 17628 17629 Returns an object where keys are option block names and values are an 17630 object with the current option values in. 17631 17632 Note that these are the global options which are unaffected by use of 17633 the _config and _filter parameters. If you wish to read the parameters 17634 set in _config then use options/config and for _filter use 17635 options/filter. 17636 17637 This shows the internal names of the option within rclone which should 17638 map to the external options very easily with a few exceptions. 17639 17640 options/local: Get the currently active config for this call 17641 17642 Returns an object with the keys "config" and "filter". The "config" key 17643 contains the local config and the "filter" key contains the local 17644 filters. 17645 17646 Note that these are the local options specific to this rc call. If 17647 _config was not supplied then they will be the global options. Likewise 17648 with "_filter". 17649 17650 This call is mostly useful for seeing if _config and _filter passing is 17651 working. 17652 17653 This shows the internal names of the option within rclone which should 17654 map to the external options very easily with a few exceptions. 17655 17656 options/set: Set an option 17657 17658 Parameters: 17659 17660 - option block name containing an object with 17661 - key: value 17662 17663 Repeated as often as required. 17664 17665 Only supply the options you wish to change. If an option is unknown it 17666 will be silently ignored. Not all options will have an effect when 17667 changed like this. 17668 17669 For example: 17670 17671 This sets DEBUG level logs (-vv) (these can be set by number or string) 17672 17673 rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}' 17674 rclone rc options/set --json '{"main": {"LogLevel": 8}}' 17675 17676 And this sets INFO level logs (-v) 17677 17678 rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}' 17679 17680 And this sets NOTICE level logs (normal without -v) 17681 17682 rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}' 17683 17684 pluginsctl/addPlugin: Add a plugin using url 17685 17686 Used for adding a plugin to the webgui. 17687 17688 This takes the following parameters: 17689 17690 - url - http url of the github repo where the plugin is hosted 17691 (http://github.com/rclone/rclone-webui-react). 17692 17693 Example: 17694 17695 rclone rc pluginsctl/addPlugin 17696 17697 Authentication is required for this call. 17698 17699 pluginsctl/getPluginsForType: Get plugins with type criteria 17700 17701 This shows all possible plugins by a mime type. 17702 17703 This takes the following parameters: 17704 17705 - type - supported mime type by a loaded plugin e.g. (video/mp4, 17706 audio/mp3). 17707 - pluginType - filter plugins based on their type e.g. (DASHBOARD, 17708 FILE_HANDLER, TERMINAL). 17709 17710 Returns: 17711 17712 - loadedPlugins - list of current production plugins. 17713 - testPlugins - list of temporarily loaded development plugins, 17714 usually running on a different server. 17715 17716 Example: 17717 17718 rclone rc pluginsctl/getPluginsForType type=video/mp4 17719 17720 Authentication is required for this call. 17721 17722 pluginsctl/listPlugins: Get the list of currently loaded plugins 17723 17724 This allows you to get the currently enabled plugins and their details. 17725 17726 This takes no parameters and returns: 17727 17728 - loadedPlugins - list of current production plugins. 17729 - testPlugins - list of temporarily loaded development plugins, 17730 usually running on a different server. 17731 17732 E.g. 17733 17734 rclone rc pluginsctl/listPlugins 17735 17736 Authentication is required for this call. 17737 17738 pluginsctl/listTestPlugins: Show currently loaded test plugins 17739 17740 Allows listing of test plugins with the rclone.test set to true in 17741 package.json of the plugin. 17742 17743 This takes no parameters and returns: 17744 17745 - loadedTestPlugins - list of currently available test plugins. 17746 17747 E.g. 17748 17749 rclone rc pluginsctl/listTestPlugins 17750 17751 Authentication is required for this call. 17752 17753 pluginsctl/removePlugin: Remove a loaded plugin 17754 17755 This allows you to remove a plugin using it's name. 17756 17757 This takes parameters: 17758 17759 - name - name of the plugin in the format author/plugin_name. 17760 17761 E.g. 17762 17763 rclone rc pluginsctl/removePlugin name=rclone/video-plugin 17764 17765 Authentication is required for this call. 17766 17767 pluginsctl/removeTestPlugin: Remove a test plugin 17768 17769 This allows you to remove a plugin using it's name. 17770 17771 This takes the following parameters: 17772 17773 - name - name of the plugin in the format author/plugin_name. 17774 17775 Example: 17776 17777 rclone rc pluginsctl/removeTestPlugin name=rclone/rclone-webui-react 17778 17779 Authentication is required for this call. 17780 17781 rc/error: This returns an error 17782 17783 This returns an error with the input as part of its error string. Useful 17784 for testing error handling. 17785 17786 rc/list: List all the registered remote control commands 17787 17788 This lists all the registered remote control commands as a JSON map in 17789 the commands response. 17790 17791 rc/noop: Echo the input to the output parameters 17792 17793 This echoes the input parameters to the output parameters for testing 17794 purposes. It can be used to check that rclone is still alive and to 17795 check that parameter passing is working properly. 17796 17797 rc/noopauth: Echo the input to the output parameters requiring auth 17798 17799 This echoes the input parameters to the output parameters for testing 17800 purposes. It can be used to check that rclone is still alive and to 17801 check that parameter passing is working properly. 17802 17803 Authentication is required for this call. 17804 17805 sync/bisync: Perform bidirectional synchronization between two paths. 17806 17807 This takes the following parameters 17808 17809 - path1 - a remote directory string e.g. drive:path1 17810 - path2 - a remote directory string e.g. drive:path2 17811 - dryRun - dry-run mode 17812 - resync - performs the resync run 17813 - checkAccess - abort if RCLONE_TEST files are not found on both 17814 filesystems 17815 - checkFilename - file name for checkAccess (default: RCLONE_TEST) 17816 - maxDelete - abort sync if percentage of deleted files is above this 17817 threshold (default: 50) 17818 - force - Bypass maxDelete safety check and run the sync 17819 - checkSync - true by default, false disables comparison of final 17820 listings, only will skip sync, only compare listings from the last 17821 run 17822 - createEmptySrcDirs - Sync creation and deletion of empty 17823 directories. (Not compatible with --remove-empty-dirs) 17824 - removeEmptyDirs - remove empty directories at the final cleanup step 17825 - filtersFile - read filtering patterns from a file 17826 - ignoreListingChecksum - Do not use checksums for listings 17827 - resilient - Allow future runs to retry after certain less-serious 17828 errors, instead of requiring resync. Use at your own risk! 17829 - workdir - server directory for history files (default: 17830 ~/.cache/rclone/bisync) 17831 - backupdir1 - --backup-dir for Path1. Must be a non-overlapping path 17832 on the same remote. 17833 - backupdir2 - --backup-dir for Path2. Must be a non-overlapping path 17834 on the same remote. 17835 - noCleanup - retain working files 17836 17837 See bisync command help and full bisync description for more 17838 information. 17839 17840 Authentication is required for this call. 17841 17842 sync/copy: copy a directory from source remote to destination remote 17843 17844 This takes the following parameters: 17845 17846 - srcFs - a remote name string e.g. "drive:src" for the source 17847 - dstFs - a remote name string e.g. "drive:dst" for the destination 17848 - createEmptySrcDirs - create empty src directories on destination if 17849 set 17850 17851 See the copy command for more information on the above. 17852 17853 Authentication is required for this call. 17854 17855 sync/move: move a directory from source remote to destination remote 17856 17857 This takes the following parameters: 17858 17859 - srcFs - a remote name string e.g. "drive:src" for the source 17860 - dstFs - a remote name string e.g. "drive:dst" for the destination 17861 - createEmptySrcDirs - create empty src directories on destination if 17862 set 17863 - deleteEmptySrcDirs - delete empty src directories if set 17864 17865 See the move command for more information on the above. 17866 17867 Authentication is required for this call. 17868 17869 sync/sync: sync a directory from source remote to destination remote 17870 17871 This takes the following parameters: 17872 17873 - srcFs - a remote name string e.g. "drive:src" for the source 17874 - dstFs - a remote name string e.g. "drive:dst" for the destination 17875 - createEmptySrcDirs - create empty src directories on destination if 17876 set 17877 17878 See the sync command for more information on the above. 17879 17880 Authentication is required for this call. 17881 17882 vfs/forget: Forget files or directories in the directory cache. 17883 17884 This forgets the paths in the directory cache causing them to be re-read 17885 from the remote when needed. 17886 17887 If no paths are passed in then it will forget all the paths in the 17888 directory cache. 17889 17890 rclone rc vfs/forget 17891 17892 Otherwise pass files or dirs in as file=path or dir=path. Any parameter 17893 key starting with file will forget that file and any starting with dir 17894 will forget that dir, e.g. 17895 17896 rclone rc vfs/forget file=hello file2=goodbye dir=home/junk 17897 17898 This command takes an "fs" parameter. If this parameter is not supplied 17899 and if there is only one VFS in use then that VFS will be used. If there 17900 is more than one VFS in use then the "fs" parameter must be supplied. 17901 17902 vfs/list: List active VFSes. 17903 17904 This lists the active VFSes. 17905 17906 It returns a list under the key "vfses" where the values are the VFS 17907 names that could be passed to the other VFS commands in the "fs" 17908 parameter. 17909 17910 vfs/poll-interval: Get the status or update the value of the poll-interval option. 17911 17912 Without any parameter given this returns the current status of the 17913 poll-interval setting. 17914 17915 When the interval=duration parameter is set, the poll-interval value is 17916 updated and the polling function is notified. Setting interval=0 17917 disables poll-interval. 17918 17919 rclone rc vfs/poll-interval interval=5m 17920 17921 The timeout=duration parameter can be used to specify a time to wait for 17922 the current poll function to apply the new value. If timeout is less or 17923 equal 0, which is the default, wait indefinitely. 17924 17925 The new poll-interval value will only be active when the timeout is not 17926 reached. 17927 17928 If poll-interval is updated or disabled temporarily, some changes might 17929 not get picked up by the polling function, depending on the used remote. 17930 17931 This command takes an "fs" parameter. If this parameter is not supplied 17932 and if there is only one VFS in use then that VFS will be used. If there 17933 is more than one VFS in use then the "fs" parameter must be supplied. 17934 17935 vfs/refresh: Refresh the directory cache. 17936 17937 This reads the directories for the specified paths and freshens the 17938 directory cache. 17939 17940 If no paths are passed in then it will refresh the root directory. 17941 17942 rclone rc vfs/refresh 17943 17944 Otherwise pass directories in as dir=path. Any parameter key starting 17945 with dir will refresh that directory, e.g. 17946 17947 rclone rc vfs/refresh dir=home/junk dir2=data/misc 17948 17949 If the parameter recursive=true is given the whole directory tree will 17950 get refreshed. This refresh will use --fast-list if enabled. 17951 17952 This command takes an "fs" parameter. If this parameter is not supplied 17953 and if there is only one VFS in use then that VFS will be used. If there 17954 is more than one VFS in use then the "fs" parameter must be supplied. 17955 17956 vfs/stats: Stats for a VFS. 17957 17958 This returns stats for the selected VFS. 17959 17960 { 17961 // Status of the disk cache - only present if --vfs-cache-mode > off 17962 "diskCache": { 17963 "bytesUsed": 0, 17964 "erroredFiles": 0, 17965 "files": 0, 17966 "hashType": 1, 17967 "outOfSpace": false, 17968 "path": "/home/user/.cache/rclone/vfs/local/mnt/a", 17969 "pathMeta": "/home/user/.cache/rclone/vfsMeta/local/mnt/a", 17970 "uploadsInProgress": 0, 17971 "uploadsQueued": 0 17972 }, 17973 "fs": "/mnt/a", 17974 "inUse": 1, 17975 // Status of the in memory metadata cache 17976 "metadataCache": { 17977 "dirs": 1, 17978 "files": 0 17979 }, 17980 // Options as returned by options/get 17981 "opt": { 17982 "CacheMaxAge": 3600000000000, 17983 // ... 17984 "WriteWait": 1000000000 17985 } 17986 } 17987 17988 This command takes an "fs" parameter. If this parameter is not supplied 17989 and if there is only one VFS in use then that VFS will be used. If there 17990 is more than one VFS in use then the "fs" parameter must be supplied. 17991 17992 Accessing the remote control via HTTP 17993 17994 Rclone implements a simple HTTP based protocol. 17995 17996 Each endpoint takes an JSON object and returns a JSON object or an 17997 error. The JSON objects are essentially a map of string names to values. 17998 17999 All calls must made using POST. 18000 18001 The input objects can be supplied using URL parameters, POST parameters 18002 or by supplying "Content-Type: application/json" and a JSON blob in the 18003 body. There are examples of these below using curl. 18004 18005 The response will be a JSON blob in the body of the response. This is 18006 formatted to be reasonably human-readable. 18007 18008 Error returns 18009 18010 If an error occurs then there will be an HTTP error status (e.g. 500) 18011 and the body of the response will contain a JSON encoded error object, 18012 e.g. 18013 18014 { 18015 "error": "Expecting string value for key \"remote\" (was float64)", 18016 "input": { 18017 "fs": "/tmp", 18018 "remote": 3 18019 }, 18020 "status": 400 18021 "path": "operations/rmdir", 18022 } 18023 18024 The keys in the error response are - error - error string - input - the 18025 input parameters to the call - status - the HTTP status code - path - 18026 the path of the call 18027 18028 CORS 18029 18030 The sever implements basic CORS support and allows all origins for that. 18031 The response to a preflight OPTIONS request will echo the requested 18032 "Access-Control-Request-Headers" back. 18033 18034 Using POST with URL parameters only 18035 18036 curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2' 18037 18038 Response 18039 18040 { 18041 "potato": "1", 18042 "sausage": "2" 18043 } 18044 18045 Here is what an error response looks like: 18046 18047 curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 18048 18049 { 18050 "error": "arbitrary error on input map[potato:1 sausage:2]", 18051 "input": { 18052 "potato": "1", 18053 "sausage": "2" 18054 } 18055 } 18056 18057 Note that curl doesn't return errors to the shell unless you use the -f 18058 option 18059 18060 $ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2' 18061 curl: (22) The requested URL returned error: 400 Bad Request 18062 $ echo $? 18063 22 18064 18065 Using POST with a form 18066 18067 curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop 18068 18069 Response 18070 18071 { 18072 "potato": "1", 18073 "sausage": "2" 18074 } 18075 18076 Note that you can combine these with URL parameters too with the POST 18077 parameters taking precedence. 18078 18079 curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4" 18080 18081 Response 18082 18083 { 18084 "potato": "1", 18085 "rutabaga": "3", 18086 "sausage": "4" 18087 } 18088 18089 Using POST with a JSON blob 18090 18091 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop 18092 18093 response 18094 18095 { 18096 "password": "xyz", 18097 "username": "xyz" 18098 } 18099 18100 This can be combined with URL parameters too if required. The JSON blob 18101 takes precedence. 18102 18103 curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4' 18104 18105 { 18106 "potato": 2, 18107 "rutabaga": "3", 18108 "sausage": 1 18109 } 18110 18111 Debugging rclone with pprof 18112 18113 If you use the --rc flag this will also enable the use of the go 18114 profiling tools on the same port. 18115 18116 To use these, first install go. 18117 18118 Debugging memory use 18119 18120 To profile rclone's memory use you can run: 18121 18122 go tool pprof -web http://localhost:5572/debug/pprof/heap 18123 18124 This should open a page in your browser showing what is using what 18125 memory. 18126 18127 You can also use the -text flag to produce a textual summary 18128 18129 $ go tool pprof -text http://localhost:5572/debug/pprof/heap 18130 Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total 18131 flat flat% sum% cum cum% 18132 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode 18133 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize 18134 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init 18135 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init 18136 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init 18137 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init 18138 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init 18139 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0 18140 0 0% 100% 1024.03kB 66.62% main.init 18141 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest 18142 0 0% 100% 513kB 33.38% net/http.(*conn).serve 18143 0 0% 100% 1024.03kB 66.62% runtime.main 18144 18145 Debugging go routine leaks 18146 18147 Memory leaks are most often caused by go routine leaks keeping memory 18148 alive which should have been garbage collected. 18149 18150 See all active go routines using 18151 18152 curl http://localhost:5572/debug/pprof/goroutine?debug=1 18153 18154 Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your 18155 browser. 18156 18157 Other profiles to look at 18158 18159 You can see a summary of profiles available at 18160 http://localhost:5572/debug/pprof/ 18161 18162 Here is how to use some of them: 18163 18164 - Memory: go tool pprof http://localhost:5572/debug/pprof/heap 18165 - Go routines: 18166 curl http://localhost:5572/debug/pprof/goroutine?debug=1 18167 - 30-second CPU profile: 18168 go tool pprof http://localhost:5572/debug/pprof/profile 18169 - 5-second execution trace: 18170 wget http://localhost:5572/debug/pprof/trace?seconds=5 18171 - Goroutine blocking profile 18172 - Enable first with: rclone rc debug/set-block-profile-rate rate=1 18173 (docs) 18174 - go tool pprof http://localhost:5572/debug/pprof/block 18175 - Contended mutexes: 18176 - Enable first with: 18177 rclone rc debug/set-mutex-profile-fraction rate=1 (docs) 18178 - go tool pprof http://localhost:5572/debug/pprof/mutex 18179 18180 See the net/http/pprof docs for more info on how to use the profiling 18181 and for a general overview see the Go team's blog post on profiling go 18182 programs. 18183 18184 The profiling hook is zero overhead unless it is used. 18185 18186 Overview of cloud storage systems 18187 18188 Each cloud storage system is slightly different. Rclone attempts to 18189 provide a unified interface to them, but some underlying differences 18190 show through. 18191 18192 Features 18193 18194 Here is an overview of the major features of each cloud storage system. 18195 18196 Name Hash ModTime Case Insensitive Duplicate Files MIME Type Metadata 18197 ------------------------------- ------------------- --------- ------------------ ----------------- ----------- ---------- 18198 1Fichier Whirlpool - No Yes R - 18199 Akamai Netstorage MD5, SHA256 R/W No No R - 18200 Amazon S3 (or S3 compatible) MD5 R/W No No R/W RWU 18201 Backblaze B2 SHA1 R/W No No R/W - 18202 Box SHA1 R/W Yes No - - 18203 Citrix ShareFile MD5 R/W Yes No - - 18204 Dropbox DBHASH ¹ R Yes No - - 18205 Enterprise File Fabric - R/W Yes No R/W - 18206 FTP - R/W ¹⁰ No No - - 18207 Google Cloud Storage MD5 R/W No No R/W - 18208 Google Drive MD5, SHA1, SHA256 DR/W No Yes R/W DRWU 18209 Google Photos - - No Yes R - 18210 HDFS - R/W No No - - 18211 HiDrive HiDrive ¹² R/W No No - - 18212 HTTP - R No No R - 18213 Internet Archive MD5, SHA1, CRC32 R/W ¹¹ No No - RWU 18214 Jottacloud MD5 R/W Yes No R RW 18215 Koofr MD5 - Yes No - - 18216 Linkbox - R No No - - 18217 Mail.ru Cloud Mailru ⁶ R/W Yes No - - 18218 Mega - - No Yes - - 18219 Memory MD5 R/W No No - - 18220 Microsoft Azure Blob Storage MD5 R/W No No R/W - 18221 Microsoft Azure Files Storage MD5 R/W Yes No R/W - 18222 Microsoft OneDrive QuickXorHash ⁵ DR/W Yes No R DRW 18223 OpenDrive MD5 R/W Yes Partial ⁸ - - 18224 OpenStack Swift MD5 R/W No No R/W - 18225 Oracle Object Storage MD5 R/W No No R/W - 18226 pCloud MD5, SHA1 ⁷ R No No W - 18227 PikPak MD5 R No No R - 18228 premiumize.me - - Yes No R - 18229 put.io CRC-32 R/W No Yes R - 18230 Proton Drive SHA1 R/W No No R - 18231 QingStor MD5 - ⁹ No No R/W - 18232 Quatrix by Maytech - R/W No No - - 18233 Seafile - - No No - - 18234 SFTP MD5, SHA1 ² DR/W Depends No - - 18235 Sia - - No No - - 18236 SMB - R/W Yes No - - 18237 SugarSync - - No No - - 18238 Storj - R No No - - 18239 Uptobox - - No Yes - - 18240 WebDAV MD5, SHA1 ³ R ⁴ Depends No - - 18241 Yandex Disk MD5 R/W No No R - 18242 Zoho WorkDrive - - No No - - 18243 The local filesystem All DR/W Depends No - DRWU 18244 18245 ¹ Dropbox supports its own custom hash. This is an SHA256 sum of all the 18246 4 MiB block SHA256s. 18247 18248 ² SFTP supports checksums if the same login has shell access and md5sum 18249 or sha1sum as well as echo are in the remote's PATH. 18250 18251 ³ WebDAV supports hashes when used with Fastmail Files, Owncloud and 18252 Nextcloud only. 18253 18254 ⁴ WebDAV supports modtimes when used with Fastmail Files, Owncloud and 18255 Nextcloud only. 18256 18257 ⁵ QuickXorHash is Microsoft's own hash. 18258 18259 ⁶ Mail.ru uses its own modified SHA1 hash 18260 18261 ⁷ pCloud only supports SHA1 (not MD5) in its EU region 18262 18263 ⁸ Opendrive does not support creation of duplicate files using their web 18264 client interface or other stock clients, but the underlying storage 18265 platform has been determined to allow duplicate files, and it is 18266 possible to create them with rclone. It may be that this is a mistake or 18267 an unsupported feature. 18268 18269 ⁹ QingStor does not support SetModTime for objects bigger than 5 GiB. 18270 18271 ¹⁰ FTP supports modtimes for the major FTP servers, and also others if 18272 they advertised required protocol extensions. See this for more details. 18273 18274 ¹¹ Internet Archive requires option wait_archive to be set to a non-zero 18275 value for full modtime support. 18276 18277 ¹² HiDrive supports its own custom hash. It combines SHA1 sums for each 18278 4 KiB block hierarchically to a single top-level sum. 18279 18280 Hash 18281 18282 The cloud storage system supports various hash types of the objects. The 18283 hashes are used when transferring data as an integrity check and can be 18284 specifically used with the --checksum flag in syncs and in the check 18285 command. 18286 18287 To use the verify checksums when transferring between cloud storage 18288 systems they must support a common hash type. 18289 18290 ModTime 18291 18292 Almost all cloud storage systems store some sort of timestamp on 18293 objects, but several of them not something that is appropriate to use 18294 for syncing. E.g. some backends will only write a timestamp that 18295 represents the time of the upload. To be relevant for syncing it should 18296 be able to store the modification time of the source object. If this is 18297 not the case, rclone will only check the file size by default, though 18298 can be configured to check the file hash (with the --checksum flag). 18299 Ideally it should also be possible to change the timestamp of an 18300 existing file without having to re-upload it. 18301 18302 ----------------------------------------------------------------------- 18303 Key Explanation 18304 ------------------- --------------------------------------------------- 18305 - ModTimes not supported - times likely the upload 18306 time 18307 18308 R ModTimes supported on files but can't be changed 18309 without re-upload 18310 18311 R/W Read and Write ModTimes fully supported on files 18312 18313 DR ModTimes supported on files and directories but 18314 can't be changed without re-upload 18315 18316 DR/W Read and Write ModTimes fully supported on files 18317 and directories 18318 ----------------------------------------------------------------------- 18319 18320 Storage systems with a - in the ModTime column, means the modification 18321 read on objects is not the modification time of the file when uploaded. 18322 It is most likely the time the file was uploaded, or possibly something 18323 else (like the time the picture was taken in Google Photos). 18324 18325 Storage systems with a R (for read-only) in the ModTime column, means 18326 the it keeps modification times on objects, and updates them when 18327 uploading objects, but it does not support changing only the 18328 modification time (SetModTime operation) without re-uploading, possibly 18329 not even without deleting existing first. Some operations in rclone, 18330 such as copy and sync commands, will automatically check for SetModTime 18331 support and re-upload if necessary to keep the modification times in 18332 sync. Other commands will not work without SetModTime support, e.g. 18333 touch command on an existing file will fail, and changes to modification 18334 time only on a files in a mount will be silently ignored. 18335 18336 Storage systems with R/W (for read/write) in the ModTime column, means 18337 they do also support modtime-only operations. 18338 18339 Storage systems with D in the ModTime column means that the following 18340 symbols apply to directories as well as files. 18341 18342 Case Insensitive 18343 18344 If a cloud storage systems is case sensitive then it is possible to have 18345 two files which differ only in case, e.g. file.txt and FILE.txt. If a 18346 cloud storage system is case insensitive then that isn't possible. 18347 18348 This can cause problems when syncing between a case insensitive system 18349 and a case sensitive system. The symptom of this is that no matter how 18350 many times you run the sync it never completes fully. 18351 18352 The local filesystem and SFTP may or may not be case sensitive depending 18353 on OS. 18354 18355 - Windows - usually case insensitive, though case is preserved 18356 - OSX - usually case insensitive, though it is possible to format case 18357 sensitive 18358 - Linux - usually case sensitive, but there are case insensitive file 18359 systems (e.g. FAT formatted USB keys) 18360 18361 Most of the time this doesn't cause any problems as people tend to avoid 18362 files whose name differs only by case even on case sensitive systems. 18363 18364 Duplicate files 18365 18366 If a cloud storage system allows duplicate files then it can have two 18367 objects with the same name. 18368 18369 This confuses rclone greatly when syncing - use the rclone dedupe 18370 command to rename or remove duplicates. 18371 18372 Restricted filenames 18373 18374 Some cloud storage systems might have restrictions on the characters 18375 that are usable in file or directory names. When rclone detects such a 18376 name during a file upload, it will transparently replace the restricted 18377 characters with similar looking Unicode characters. To handle the 18378 different sets of restricted characters for different backends, rclone 18379 uses something it calls encoding. 18380 18381 This process is designed to avoid ambiguous file names as much as 18382 possible and allow to move files between many cloud storage systems 18383 transparently. 18384 18385 The name shown by rclone to the user or during log output will only 18386 contain a minimal set of replaced characters to ensure correct 18387 formatting and not necessarily the actual name used on the cloud 18388 storage. 18389 18390 This transformation is reversed when downloading a file or parsing 18391 rclone arguments. For example, when uploading a file named my file?.txt 18392 to Onedrive, it will be displayed as my file?.txt on the console, but 18393 stored as my file?.txt to Onedrive (the ? gets replaced by the similar 18394 looking ? character, the so-called "fullwidth question mark"). The 18395 reverse transformation allows to read a file unusual/name.txt from 18396 Google Drive, by passing the name unusual/name.txt on the command line 18397 (the / needs to be replaced by the similar looking / character). 18398 18399 Caveats 18400 18401 The filename encoding system works well in most cases, at least where 18402 file names are written in English or similar languages. You might not 18403 even notice it: It just works. In some cases it may lead to issues, 18404 though. E.g. when file names are written in Chinese, or Japanese, where 18405 it is always the Unicode fullwidth variants of the punctuation marks 18406 that are used. 18407 18408 On Windows, the characters :, * and ? are examples of restricted 18409 characters. If these are used in filenames on a remote that supports it, 18410 Rclone will transparently convert them to their fullwidth Unicode 18411 variants *, ? and : when downloading to Windows, and back again when 18412 uploading. This way files with names that are not allowed on Windows can 18413 still be stored. 18414 18415 However, if you have files on your Windows system originally with these 18416 same Unicode characters in their names, they will be included in the 18417 same conversion process. E.g. if you create a file in your Windows 18418 filesystem with name Test:1.jpg, where : is the Unicode fullwidth 18419 colon symbol, and use rclone to upload it to Google Drive, which 18420 supports regular : (halfwidth question mark), rclone will replace the 18421 fullwidth : with the halfwidth : and store the file as Test:1.jpg in 18422 Google Drive. Since both Windows and Google Drive allows the name 18423 Test:1.jpg, it would probably be better if rclone just kept the name as 18424 is in this case. 18425 18426 With the opposite situation; if you have a file named Test:1.jpg, in 18427 your Google Drive, e.g. uploaded from a Linux system where : is valid in 18428 file names. Then later use rclone to copy this file to your Windows 18429 computer you will notice that on your local disk it gets renamed to 18430 Test:1.jpg. The original filename is not legal on Windows, due to the 18431 :, and rclone therefore renames it to make the copy possible. That is 18432 all good. However, this can also lead to an issue: If you already had a 18433 different file named Test:1.jpg on Windows, and then use rclone to copy 18434 either way. Rclone will then treat the file originally named Test:1.jpg 18435 on Google Drive and the file originally named Test:1.jpg on Windows as 18436 the same file, and replace the contents from one with the other. 18437 18438 Its virtually impossible to handle all cases like these correctly in all 18439 situations, but by customizing the encoding option, changing the set of 18440 characters that rclone should convert, you should be able to create a 18441 configuration that works well for your specific situation. See also the 18442 example below. 18443 18444 (Windows was used as an example of a file system with many restricted 18445 characters, and Google drive a storage system with few.) 18446 18447 Default restricted characters 18448 18449 The table below shows the characters that are replaced by default. 18450 18451 When a replacement character is found in a filename, this character will 18452 be escaped with the ‛ character to avoid ambiguous file names. (e.g. a 18453 file named ␀.txt would shown as ‛␀.txt) 18454 18455 Each cloud storage backend can use a different set of characters, which 18456 will be specified in the documentation for each backend. 18457 18458 Character Value Replacement 18459 ----------- ------- ------------- 18460 NUL 0x00 ␀ 18461 SOH 0x01 ␁ 18462 STX 0x02 ␂ 18463 ETX 0x03 ␃ 18464 EOT 0x04 ␄ 18465 ENQ 0x05 ␅ 18466 ACK 0x06 ␆ 18467 BEL 0x07 ␇ 18468 BS 0x08 ␈ 18469 HT 0x09 ␉ 18470 LF 0x0A ␊ 18471 VT 0x0B ␋ 18472 FF 0x0C ␌ 18473 CR 0x0D ␍ 18474 SO 0x0E ␎ 18475 SI 0x0F ␏ 18476 DLE 0x10 ␐ 18477 DC1 0x11 ␑ 18478 DC2 0x12 ␒ 18479 DC3 0x13 ␓ 18480 DC4 0x14 ␔ 18481 NAK 0x15 ␕ 18482 SYN 0x16 ␖ 18483 ETB 0x17 ␗ 18484 CAN 0x18 ␘ 18485 EM 0x19 ␙ 18486 SUB 0x1A ␚ 18487 ESC 0x1B ␛ 18488 FS 0x1C ␜ 18489 GS 0x1D ␝ 18490 RS 0x1E ␞ 18491 US 0x1F ␟ 18492 / 0x2F / 18493 DEL 0x7F ␡ 18494 18495 The default encoding will also encode these file names as they are 18496 problematic with many cloud storage systems. 18497 18498 File name Replacement 18499 ----------- ------------- 18500 . . 18501 .. .. 18502 18503 Invalid UTF-8 bytes 18504 18505 Some backends only support a sequence of well formed UTF-8 bytes as file 18506 or directory names. 18507 18508 In this case all invalid UTF-8 bytes will be replaced with a quoted 18509 representation of the byte value to allow uploading a file to such a 18510 backend. For example, the invalid byte 0xFE will be encoded as ‛FE. 18511 18512 A common source of invalid UTF-8 bytes are local filesystems, that store 18513 names in a different encoding than UTF-8 or UTF-16, like latin1. See the 18514 local filenames section for details. 18515 18516 Encoding option 18517 18518 Most backends have an encoding option, specified as a flag 18519 --backend-encoding where backend is the name of the backend, or as a 18520 config parameter encoding (you'll need to select the Advanced config in 18521 rclone config to see it). 18522 18523 This will have default value which encodes and decodes characters in 18524 such a way as to preserve the maximum number of characters (see above). 18525 18526 However this can be incorrect in some scenarios, for example if you have 18527 a Windows file system with Unicode fullwidth characters *, ? or :, 18528 that you want to remain as those characters on the remote rather than 18529 being translated to regular (halfwidth) *, ? and :. 18530 18531 The --backend-encoding flags allow you to change that. You can disable 18532 the encoding completely with --backend-encoding None or set 18533 encoding = None in the config file. 18534 18535 Encoding takes a comma separated list of encodings. You can see the list 18536 of all possible values by passing an invalid value to this flag, e.g. 18537 --local-encoding "help". The command rclone help flags encoding will 18538 show you the defaults for the backends. 18539 18540 ---------------------------------------------------------------------------------- 18541 Encoding Characters Encoded as 18542 ---------------------- ------------------------ ---------------------------------- 18543 Asterisk * * 18544 18545 BackQuote ` ` 18546 18547 BackSlash \ \ 18548 18549 Colon : : 18550 18551 CrLf CR 0x0D, LF 0x0A ␍, ␊ 18552 18553 Ctl All control characters ␀␁␂␃␄␅␆␇␈␉␊␋␌␍␎␏␐␑␒␓␔␕␖␗␘␙␚␛␜␝␞␟ 18554 0x00-0x1F 18555 18556 Del DEL 0x7F ␡ 18557 18558 Dollar $ $ 18559 18560 Dot . or .. as entire string ., .. 18561 18562 DoubleQuote " " 18563 18564 Hash # # 18565 18566 InvalidUtf8 An invalid UTF-8 � 18567 character (e.g. latin1) 18568 18569 LeftCrLfHtVt CR 0x0D, LF 0x0A, HT ␍, ␊, ␉, ␋ 18570 0x09, VT 0x0B on the 18571 left of a string 18572 18573 LeftPeriod . on the left of a . 18574 string 18575 18576 LeftSpace SPACE on the left of a ␠ 18577 string 18578 18579 LeftTilde ~ on the left of a ~ 18580 string 18581 18582 LtGt <, > <, > 18583 18584 None No characters are 18585 encoded 18586 18587 Percent % % 18588 18589 Pipe | | 18590 18591 Question ? ? 18592 18593 RightCrLfHtVt CR 0x0D, LF 0x0A, HT ␍, ␊, ␉, ␋ 18594 0x09, VT 0x0B on the 18595 right of a string 18596 18597 RightPeriod . on the right of a . 18598 string 18599 18600 RightSpace SPACE on the right of a ␠ 18601 string 18602 18603 Semicolon ; ; 18604 18605 SingleQuote ' ' 18606 18607 Slash / / 18608 18609 SquareBracket [, ] [, ] 18610 ---------------------------------------------------------------------------------- 18611 18612 Encoding example: FTP 18613 18614 To take a specific example, the FTP backend's default encoding is 18615 18616 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 18617 18618 However, let's say the FTP server is running on Windows and can't have 18619 any of the invalid Windows characters in file names. You are backing up 18620 Linux servers to this FTP server which do have those characters in file 18621 names. So you would add the Windows set which are 18622 18623 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 18624 18625 to the existing ones, giving: 18626 18627 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 18628 18629 This can be specified using the --ftp-encoding flag or using an encoding 18630 parameter in the config file. 18631 18632 Encoding example: Windows 18633 18634 As a nother example, take a Windows system where there is a file with 18635 name Test:1.jpg, where : is the Unicode fullwidth colon symbol. When 18636 using rclone to copy this to a remote which supports :, the regular 18637 (halfwidth) colon (such as Google Drive), you will notice that the file 18638 gets renamed to Test:1.jpg. 18639 18640 To avoid this you can change the set of characters rclone should convert 18641 for the local filesystem, using command-line argument --local-encoding. 18642 Rclone's default behavior on Windows corresponds to 18643 18644 --local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" 18645 18646 If you want to use fullwidth characters :, * and ? in your filenames 18647 without rclone changing them when uploading to a remote, then set the 18648 same as the default value but without Colon,Question,Asterisk: 18649 18650 --local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" 18651 18652 Alternatively, you can disable the conversion of any characters with 18653 --local-encoding None. 18654 18655 Instead of using command-line argument --local-encoding, you may also 18656 set it as environment variable RCLONE_LOCAL_ENCODING, or configure a 18657 remote of type local in your config, and set the encoding option there. 18658 18659 The risk by doing this is that if you have a filename with the regular 18660 (halfwidth) :, * and ? in your cloud storage, and you try to download it 18661 to your Windows filesystem, this will fail. These characters are not 18662 valid in filenames on Windows, and you have told rclone not to work 18663 around this by converting them to valid fullwidth variants. 18664 18665 MIME Type 18666 18667 MIME types (also known as media types) classify types of documents using 18668 a simple text classification, e.g. text/html or application/pdf. 18669 18670 Some cloud storage systems support reading (R) the MIME type of objects 18671 and some support writing (W) the MIME type of objects. 18672 18673 The MIME type can be important if you are serving files directly to HTTP 18674 from the storage system. 18675 18676 If you are copying from a remote which supports reading (R) to a remote 18677 which supports writing (W) then rclone will preserve the MIME types. 18678 Otherwise they will be guessed from the extension, or the remote itself 18679 may assign the MIME type. 18680 18681 Metadata 18682 18683 Backends may or may support reading or writing metadata. They may 18684 support reading and writing system metadata (metadata intrinsic to that 18685 backend) and/or user metadata (general purpose metadata). 18686 18687 The levels of metadata support are 18688 18689 ----------------------------------------------------------------------- 18690 Key Explanation 18691 ------------------- --------------------------------------------------- 18692 R Read only System Metadata on files only 18693 18694 RW Read and write System Metadata on files only 18695 18696 RWU Read and write System Metadata and read and write 18697 User Metadata on files only 18698 18699 DR Read only System Metadata on files and directories 18700 18701 DRW Read and write System Metadata on files and 18702 directories 18703 18704 DRWU Read and write System Metadata and read and write 18705 User Metadata on files and directories 18706 ----------------------------------------------------------------------- 18707 18708 See the metadata docs for more info. 18709 18710 Optional Features 18711 18712 All rclone remotes support a base command set. Other features depend 18713 upon backend-specific capabilities. 18714 18715 ------------------------------------------------------------------------------------------------------------------------------------- 18716 Name Purge Copy Move DirMove CleanUp ListR StreamUpload MultithreadUpload LinkSharing About EmptyDir 18717 --------------- ------- ------ ------ --------- --------- ------- -------------- ------------------- ------------- ------- ---------- 18718 1Fichier No Yes Yes No No No No No Yes No Yes 18719 18720 Akamai Yes No No No No Yes Yes No No No Yes 18721 Netstorage 18722 18723 Amazon S3 (or No Yes No No Yes Yes Yes Yes Yes No No 18724 S3 compatible) 18725 18726 Backblaze B2 No Yes No No Yes Yes Yes Yes Yes No No 18727 18728 Box Yes Yes Yes Yes Yes No Yes No Yes Yes Yes 18729 18730 Citrix Yes Yes Yes Yes No No No No No No Yes 18731 ShareFile 18732 18733 Dropbox Yes Yes Yes Yes No No Yes No Yes Yes Yes 18734 18735 Enterprise File Yes Yes Yes Yes Yes No No No No No Yes 18736 Fabric 18737 18738 FTP No No Yes Yes No No Yes No No No Yes 18739 18740 Google Cloud Yes Yes No No No Yes Yes No No No No 18741 Storage 18742 18743 Google Drive Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes 18744 18745 Google Photos No No No No No No No No No No No 18746 18747 HDFS Yes No Yes Yes No No Yes No No Yes Yes 18748 18749 HiDrive Yes Yes Yes Yes No No Yes No No No Yes 18750 18751 HTTP No No No No No No No No No No Yes 18752 18753 ImageKit Yes Yes Yes No No No No No No No Yes 18754 18755 Internet No Yes No No Yes Yes No No Yes Yes No 18756 Archive 18757 18758 Jottacloud Yes Yes Yes Yes Yes Yes No No Yes Yes Yes 18759 18760 Koofr Yes Yes Yes Yes No No Yes No Yes Yes Yes 18761 18762 Mail.ru Cloud Yes Yes Yes Yes Yes No No No Yes Yes Yes 18763 18764 Mega Yes No Yes Yes Yes No No No Yes Yes Yes 18765 18766 Memory No Yes No No No Yes Yes No No No No 18767 18768 Microsoft Azure Yes Yes No No No Yes Yes Yes No No No 18769 Blob Storage 18770 18771 Microsoft Azure No Yes Yes Yes No No Yes Yes No Yes Yes 18772 Files Storage 18773 18774 Microsoft Yes Yes Yes Yes Yes Yes ⁵ No No Yes Yes Yes 18775 OneDrive 18776 18777 OpenDrive Yes Yes Yes Yes No No No No No No Yes 18778 18779 OpenStack Swift Yes ¹ Yes No No No Yes Yes No No Yes No 18780 18781 Oracle Object No Yes No No Yes Yes Yes Yes No No No 18782 Storage 18783 18784 pCloud Yes Yes Yes Yes Yes No No No Yes Yes Yes 18785 18786 PikPak Yes Yes Yes Yes Yes No No No Yes Yes Yes 18787 18788 premiumize.me Yes No Yes Yes No No No No Yes Yes Yes 18789 18790 put.io Yes No Yes Yes Yes No Yes No No Yes Yes 18791 18792 Proton Drive Yes No Yes Yes Yes No No No No Yes Yes 18793 18794 QingStor No Yes No No Yes Yes No No No No No 18795 18796 Quatrix by Yes Yes Yes Yes No No No No No Yes Yes 18797 Maytech 18798 18799 Seafile Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes 18800 18801 SFTP No Yes ⁴ Yes Yes No No Yes No No Yes Yes 18802 18803 Sia No No No No No No Yes No No No Yes 18804 18805 SMB No No Yes Yes No No Yes Yes No No Yes 18806 18807 SugarSync Yes Yes Yes Yes No No Yes No Yes No Yes 18808 18809 Storj Yes ² Yes Yes No No Yes Yes No Yes No No 18810 18811 Uptobox No Yes Yes Yes No No No No No No No 18812 18813 WebDAV Yes Yes Yes Yes No No Yes ³ No No Yes Yes 18814 18815 Yandex Disk Yes Yes Yes Yes Yes No Yes No Yes Yes Yes 18816 18817 Zoho WorkDrive Yes Yes Yes Yes No No No No No Yes Yes 18818 18819 The local No No Yes Yes No No Yes Yes No Yes Yes 18820 filesystem 18821 ------------------------------------------------------------------------------------------------------------------------------------- 18822 18823 ¹ Note Swift implements this in order to delete directory markers but it 18824 doesn't actually have a quicker way of deleting files other than 18825 deleting them individually. 18826 18827 ² Storj implements this efficiently only for entire buckets. If purging 18828 a directory inside a bucket, files are deleted individually. 18829 18830 ³ StreamUpload is not supported with Nextcloud 18831 18832 ⁴ Use the --sftp-copy-is-hardlink flag to enable. 18833 18834 ⁵ Use the --onedrive-delta flag to enable. 18835 18836 Purge 18837 18838 This deletes a directory quicker than just deleting all the files in the 18839 directory. 18840 18841 Copy 18842 18843 Used when copying an object to and from the same remote. This known as a 18844 server-side copy so you can copy a file without downloading it and 18845 uploading it again. It is used if you use rclone copy or rclone move if 18846 the remote doesn't support Move directly. 18847 18848 If the server doesn't support Copy directly then for copy operations the 18849 file is downloaded then re-uploaded. 18850 18851 Move 18852 18853 Used when moving/renaming an object on the same remote. This is known as 18854 a server-side move of a file. This is used in rclone move if the server 18855 doesn't support DirMove. 18856 18857 If the server isn't capable of Move then rclone simulates it with Copy 18858 then delete. If the server doesn't support Copy then rclone will 18859 download the file and re-upload it. 18860 18861 DirMove 18862 18863 This is used to implement rclone move to move a directory if possible. 18864 If it isn't then it will use Move on each file (which falls back to Copy 18865 then download and upload - see Move section). 18866 18867 CleanUp 18868 18869 This is used for emptying the trash for a remote by rclone cleanup. 18870 18871 If the server can't do CleanUp then rclone cleanup will return an error. 18872 18873 ‡‡ Note that while Box implements this it has to delete every file 18874 individually so it will be slower than emptying the trash via the WebUI 18875 18876 ListR 18877 18878 The remote supports a recursive list to list all the contents beneath a 18879 directory quickly. This enables the --fast-list flag to work. See the 18880 rclone docs for more details. 18881 18882 StreamUpload 18883 18884 Some remotes allow files to be uploaded without knowing the file size in 18885 advance. This allows certain operations to work without spooling the 18886 file to local disk first, e.g. rclone rcat. 18887 18888 MultithreadUpload 18889 18890 Some remotes allow transfers to the remote to be sent as chunks in 18891 parallel. If this is supported then rclone will use multi-thread copying 18892 to transfer files much faster. 18893 18894 LinkSharing 18895 18896 Sets the necessary permissions on a file or folder and prints a link 18897 that allows others to access them, even if they don't have an account on 18898 the particular cloud provider. 18899 18900 About 18901 18902 Rclone about prints quota information for a remote. Typical output 18903 includes bytes used, free, quota and in trash. 18904 18905 If a remote lacks about capability rclone about remote:returns an error. 18906 18907 Backends without about capability cannot determine free space for an 18908 rclone mount, or use policy mfs (most free space) as a member of an 18909 rclone union remote. 18910 18911 See rclone about command 18912 18913 EmptyDir 18914 18915 The remote supports empty directories. See Limitations for details. Most 18916 Object/Bucket-based remotes do not support this. 18917 18918 Global Flags 18919 18920 This describes the global flags available to every rclone command split 18921 into groups. 18922 18923 Copy 18924 18925 Flags for anything which can Copy a file. 18926 18927 --check-first Do all the checks before starting transfers 18928 -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). 18929 --compare-dest stringArray Include additional comma separated server-side paths during comparison 18930 --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination 18931 --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) 18932 --ignore-case-sync Ignore case when synchronizing 18933 --ignore-checksum Skip post copy check of checksums 18934 --ignore-existing Skip all files that exist on destination 18935 --ignore-size Ignore size when skipping use modtime or checksum 18936 -I, --ignore-times Don't skip items that match size and time - transfer all unconditionally 18937 --immutable Do not modify files, fail if existing files have been modified 18938 --inplace Download directly to destination file instead of atomic download to temp/rename 18939 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 18940 --max-duration Duration Maximum duration rclone will transfer data for (default 0s) 18941 --max-transfer SizeSuffix Maximum size of data to transfer (default off) 18942 -M, --metadata If set, preserve metadata when copying objects 18943 --modify-window Duration Max time diff to be considered the same (default 1ns) 18944 --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi) 18945 --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi) 18946 --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4) 18947 --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) 18948 --no-check-dest Don't check the destination, copy regardless 18949 --no-traverse Don't traverse destination file system on copy 18950 --no-update-dir-modtime Don't update directory modification times 18951 --no-update-modtime Don't update destination modtime if files identical 18952 --order-by string Instructions on how to order the transfers, e.g. 'size,descending' 18953 --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") 18954 --refresh-times Refresh the modtime of remote files 18955 --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs 18956 --size-only Skip based on size only, not modtime or checksum 18957 --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) 18958 -u, --update Skip files that are newer on the destination 18959 18960 Sync 18961 18962 Flags just used for rclone sync. 18963 18964 --backup-dir string Make backups into hierarchy based in DIR 18965 --delete-after When synchronizing, delete files on destination after transferring (default) 18966 --delete-before When synchronizing, delete files on destination before transferring 18967 --delete-during When synchronizing, delete files during transfer 18968 --fix-case Force rename of case insensitive dest to match source 18969 --ignore-errors Delete even if there are I/O errors 18970 --max-delete int When synchronizing, limit the number of deletes (default -1) 18971 --max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off) 18972 --suffix string Suffix to add to changed files 18973 --suffix-keep-extension Preserve the extension when using --suffix 18974 --track-renames When synchronizing, track file renames and do a server-side move if possible 18975 --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash") 18976 18977 Important 18978 18979 Important flags useful for most commands. 18980 18981 -n, --dry-run Do a trial run with no permanent changes 18982 -i, --interactive Enable interactive mode 18983 -v, --verbose count Print lots more stuff (repeat for more) 18984 18985 Check 18986 18987 Flags used for rclone check. 18988 18989 --max-backlog int Maximum number of objects in sync or check backlog (default 10000) 18990 18991 Networking 18992 18993 General networking and HTTP stuff. 18994 18995 --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name 18996 --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable 18997 --bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable 18998 --ca-cert stringArray CA certificate used to verify servers 18999 --client-cert string Client SSL certificate (PEM) for mutual TLS auth 19000 --client-key string Client SSL private key (PEM) for mutual TLS auth 19001 --contimeout Duration Connect timeout (default 1m0s) 19002 --disable-http-keep-alives Disable HTTP keep-alives and use each connection once. 19003 --disable-http2 Disable HTTP/2 in the global transport 19004 --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21 19005 --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s) 19006 --header stringArray Set HTTP header for all transactions 19007 --header-download stringArray Set HTTP header for download transactions 19008 --header-upload stringArray Set HTTP header for upload transactions 19009 --no-check-certificate Do not verify the server SSL certificate (insecure) 19010 --no-gzip-encoding Don't set Accept-Encoding: gzip 19011 --timeout Duration IO idle timeout (default 5m0s) 19012 --tpslimit float Limit HTTP transactions per second to this 19013 --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) 19014 --use-cookies Enable session cookiejar 19015 --user-agent string Set the user-agent to a specified string (default "rclone/v1.66.0") 19016 19017 Performance 19018 19019 Flags helpful for increasing performance. 19020 19021 --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi) 19022 --checkers int Number of checkers to run in parallel (default 8) 19023 --transfers int Number of file transfers to run in parallel (default 4) 19024 19025 Config 19026 19027 General configuration of rclone. 19028 19029 --ask-password Allow prompt for password for encrypted configuration (default true) 19030 --auto-confirm If enabled, do not request console confirmation 19031 --cache-dir string Directory rclone will use for caching (default "$HOME/.cache/rclone") 19032 --color AUTO|NEVER|ALWAYS When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default AUTO) 19033 --config string Config file (default "$HOME/.config/rclone/rclone.conf") 19034 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 19035 --disable string Disable a comma separated list of features (use --disable help to see a list) 19036 -n, --dry-run Do a trial run with no permanent changes 19037 --error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts 19038 --fs-cache-expire-duration Duration Cache remotes for this long (0 to disable caching) (default 5m0s) 19039 --fs-cache-expire-interval Duration Interval to check for expired remotes (default 1m0s) 19040 --human-readable Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi 19041 -i, --interactive Enable interactive mode 19042 --kv-lock-time Duration Maximum time to keep key-value database locked by process (default 1s) 19043 --low-level-retries int Number of low level retries to do (default 10) 19044 --no-console Hide console window (supported on Windows only) 19045 --no-unicode-normalization Don't normalize unicode characters in filenames 19046 --password-command SpaceSepList Command for supplying password for encrypted configuration 19047 --retries int Retry operations this many times if they fail (default 3) 19048 --retries-sleep Duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s) 19049 --temp-dir string Directory rclone will use for temporary files (default "/tmp") 19050 --use-mmap Use mmap allocator (see docs) 19051 --use-server-modtime Use server modified time instead of object metadata 19052 19053 Debugging 19054 19055 Flags for developers. 19056 19057 --cpuprofile string Write cpu profile to file 19058 --dump DumpFlags List of items to dump from: headers, bodies, requests, responses, auth, filters, goroutines, openfiles, mapper 19059 --dump-bodies Dump HTTP headers and bodies - may contain sensitive info 19060 --dump-headers Dump HTTP headers - may contain sensitive info 19061 --memprofile string Write memory profile to file 19062 19063 Filter 19064 19065 Flags for filtering directory listings. 19066 19067 --delete-excluded Delete files on dest excluded from sync 19068 --exclude stringArray Exclude files matching pattern 19069 --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin) 19070 --exclude-if-present stringArray Exclude directories if filename is present 19071 --files-from stringArray Read list of source-file names from file (use - to read from stdin) 19072 --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin) 19073 -f, --filter stringArray Add a file filtering rule 19074 --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin) 19075 --ignore-case Ignore case in filters (case insensitive) 19076 --include stringArray Include files matching pattern 19077 --include-from stringArray Read file include patterns from file (use - to read from stdin) 19078 --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off) 19079 --max-depth int If set limits the recursion depth to this (default -1) 19080 --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off) 19081 --metadata-exclude stringArray Exclude metadatas matching pattern 19082 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 19083 --metadata-filter stringArray Add a metadata filtering rule 19084 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 19085 --metadata-include stringArray Include metadatas matching pattern 19086 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 19087 --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) 19088 --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off) 19089 19090 Listing 19091 19092 Flags for listing directories. 19093 19094 --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z) 19095 --fast-list Use recursive list if available; uses more memory but fewer transactions 19096 19097 Logging 19098 19099 Logging and statistics. 19100 19101 --log-file string Log everything to this file 19102 --log-format string Comma separated list of log format options (default "date,time") 19103 --log-level LogLevel Log level DEBUG|INFO|NOTICE|ERROR (default NOTICE) 19104 --log-systemd Activate systemd integration for the logger 19105 --max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000) 19106 -P, --progress Show progress during transfer 19107 --progress-terminal-title Show progress on the terminal title (requires -P/--progress) 19108 -q, --quiet Print as little stuff as possible 19109 --stats Duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s) 19110 --stats-file-name-length int Max file name length in stats (0 for no limit) (default 45) 19111 --stats-log-level LogLevel Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default INFO) 19112 --stats-one-line Make the stats fit on one line 19113 --stats-one-line-date Enable --stats-one-line and add current date/time prefix 19114 --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format 19115 --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes") 19116 --syslog Use Syslog for logging 19117 --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON") 19118 --use-json-log Use json log format 19119 -v, --verbose count Print lots more stuff (repeat for more) 19120 19121 Metadata 19122 19123 Flags to control metadata. 19124 19125 -M, --metadata If set, preserve metadata when copying objects 19126 --metadata-exclude stringArray Exclude metadatas matching pattern 19127 --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin) 19128 --metadata-filter stringArray Add a metadata filtering rule 19129 --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin) 19130 --metadata-include stringArray Include metadatas matching pattern 19131 --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin) 19132 --metadata-mapper SpaceSepList Program to run to transforming metadata before upload 19133 --metadata-set stringArray Add metadata key=value when uploading 19134 19135 RC 19136 19137 Flags to control the Remote Control API. 19138 19139 --rc Enable the remote control server 19140 --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572]) 19141 --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from 19142 --rc-baseurl string Prefix for URLs - leave blank for root 19143 --rc-cert string TLS PEM key (concatenation of certificate and CA certificate) 19144 --rc-client-ca string Client certificate authority to verify clients with 19145 --rc-enable-metrics Enable prometheus metrics on /metrics 19146 --rc-files string Path to local files to serve on the HTTP server 19147 --rc-htpasswd string A htpasswd file - if not provided no authentication is done 19148 --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s) 19149 --rc-job-expire-interval Duration Interval to check for expired async jobs (default 10s) 19150 --rc-key string TLS PEM Private key 19151 --rc-max-header-bytes int Maximum size of request header (default 4096) 19152 --rc-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0") 19153 --rc-no-auth Don't require auth for certain methods 19154 --rc-pass string Password for authentication 19155 --rc-realm string Realm for authentication 19156 --rc-salt string Password hashing salt (default "dlPL2MqE") 19157 --rc-serve Enable the serving of remote objects 19158 --rc-server-read-timeout Duration Timeout for server reading data (default 1h0m0s) 19159 --rc-server-write-timeout Duration Timeout for server writing data (default 1h0m0s) 19160 --rc-template string User-specified template 19161 --rc-user string User name for authentication 19162 --rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest") 19163 --rc-web-gui Launch WebGUI on localhost 19164 --rc-web-gui-force-update Force update to latest version of web gui 19165 --rc-web-gui-no-open-browser Don't open the browser automatically 19166 --rc-web-gui-update Check and update to latest version of web gui 19167 19168 Backend 19169 19170 Backend only flags. These can be set in the config file also. 19171 19172 --alias-description string Description of the remote 19173 --alias-remote string Remote or path to alias 19174 --azureblob-access-tier string Access tier of blob: hot, cool, cold or archive 19175 --azureblob-account string Azure Storage Account Name 19176 --azureblob-archive-tier-delete Delete archive tier blobs before overwriting 19177 --azureblob-chunk-size SizeSuffix Upload chunk size (default 4Mi) 19178 --azureblob-client-certificate-password string Password for the certificate file (optional) (obscured) 19179 --azureblob-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key 19180 --azureblob-client-id string The ID of the client in use 19181 --azureblob-client-secret string One of the service principal's client secrets 19182 --azureblob-client-send-certificate-chain Send the certificate chain when using certificate auth 19183 --azureblob-delete-snapshots string Set to specify how to deal with snapshots on blob deletion 19184 --azureblob-description string Description of the remote 19185 --azureblob-directory-markers Upload an empty object with a trailing slash when a new directory is created 19186 --azureblob-disable-checksum Don't store MD5 checksum with object metadata 19187 --azureblob-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8) 19188 --azureblob-endpoint string Endpoint for the service 19189 --azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI) 19190 --azureblob-key string Storage Account Shared Key 19191 --azureblob-list-chunk int Size of blob list (default 5000) 19192 --azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any 19193 --azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any 19194 --azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any 19195 --azureblob-no-check-container If set, don't attempt to check the container exists or create it 19196 --azureblob-no-head-object If set, do not do HEAD before GET when getting objects 19197 --azureblob-password string The user's password (obscured) 19198 --azureblob-public-access string Public access level of a container: blob or container 19199 --azureblob-sas-url string SAS URL for container level access only 19200 --azureblob-service-principal-file string Path to file containing credentials for use with a service principal 19201 --azureblob-tenant string ID of the service principal's tenant. Also called its directory ID 19202 --azureblob-upload-concurrency int Concurrency for multipart uploads (default 16) 19203 --azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated) 19204 --azureblob-use-emulator Uses local storage emulator if provided as 'true' 19205 --azureblob-use-msi Use a managed service identity to authenticate (only works in Azure) 19206 --azureblob-username string User name (usually an email address) 19207 --azurefiles-account string Azure Storage Account Name 19208 --azurefiles-chunk-size SizeSuffix Upload chunk size (default 4Mi) 19209 --azurefiles-client-certificate-password string Password for the certificate file (optional) (obscured) 19210 --azurefiles-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key 19211 --azurefiles-client-id string The ID of the client in use 19212 --azurefiles-client-secret string One of the service principal's client secrets 19213 --azurefiles-client-send-certificate-chain Send the certificate chain when using certificate auth 19214 --azurefiles-connection-string string Azure Files Connection String 19215 --azurefiles-description string Description of the remote 19216 --azurefiles-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot) 19217 --azurefiles-endpoint string Endpoint for the service 19218 --azurefiles-env-auth Read credentials from runtime (environment variables, CLI or MSI) 19219 --azurefiles-key string Storage Account Shared Key 19220 --azurefiles-max-stream-size SizeSuffix Max size for streamed files (default 10Gi) 19221 --azurefiles-msi-client-id string Object ID of the user-assigned MSI to use, if any 19222 --azurefiles-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any 19223 --azurefiles-msi-object-id string Object ID of the user-assigned MSI to use, if any 19224 --azurefiles-password string The user's password (obscured) 19225 --azurefiles-sas-url string SAS URL 19226 --azurefiles-service-principal-file string Path to file containing credentials for use with a service principal 19227 --azurefiles-share-name string Azure Files Share Name 19228 --azurefiles-tenant string ID of the service principal's tenant. Also called its directory ID 19229 --azurefiles-upload-concurrency int Concurrency for multipart uploads (default 16) 19230 --azurefiles-use-msi Use a managed service identity to authenticate (only works in Azure) 19231 --azurefiles-username string User name (usually an email address) 19232 --b2-account string Account ID or Application Key ID 19233 --b2-chunk-size SizeSuffix Upload chunk size (default 96Mi) 19234 --b2-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4Gi) 19235 --b2-description string Description of the remote 19236 --b2-disable-checksum Disable checksums for large (> upload cutoff) files 19237 --b2-download-auth-duration Duration Time before the public link authorization token will expire in s or suffix ms|s|m|h|d (default 1w) 19238 --b2-download-url string Custom endpoint for downloads 19239 --b2-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19240 --b2-endpoint string Endpoint for the service 19241 --b2-hard-delete Permanently delete files on remote removal, otherwise hide files 19242 --b2-key string Application Key 19243 --b2-lifecycle int Set the number of days deleted files should be kept when creating a bucket 19244 --b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging 19245 --b2-upload-concurrency int Concurrency for multipart uploads (default 4) 19246 --b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) 19247 --b2-version-at Time Show file versions as they were at the specified time (default off) 19248 --b2-versions Include old versions in directory listings 19249 --box-access-token string Box App Primary Access Token 19250 --box-auth-url string Auth server URL 19251 --box-box-config-file string Box App config.json location 19252 --box-box-sub-type string (default "user") 19253 --box-client-id string OAuth Client Id 19254 --box-client-secret string OAuth Client Secret 19255 --box-commit-retries int Max number of times to try committing a multipart file (default 100) 19256 --box-description string Description of the remote 19257 --box-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot) 19258 --box-impersonate string Impersonate this user ID when using a service account 19259 --box-list-chunk int Size of listing chunk 1-1000 (default 1000) 19260 --box-owned-by string Only show items owned by the login (email address) passed in 19261 --box-root-folder-id string Fill in for rclone to use a non root folder as its starting point 19262 --box-token string OAuth Access Token as a JSON blob 19263 --box-token-url string Token server url 19264 --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi) 19265 --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage (default 1m0s) 19266 --cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming 19267 --cache-chunk-path string Directory to cache chunk files (default "$HOME/.cache/rclone/cache-backend") 19268 --cache-chunk-size SizeSuffix The size of a chunk (partial file data) (default 5Mi) 19269 --cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk (default 10Gi) 19270 --cache-db-path string Directory to store file structure metadata DB (default "$HOME/.cache/rclone/cache-backend") 19271 --cache-db-purge Clear all the cached data for this remote on start 19272 --cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s) 19273 --cache-description string Description of the remote 19274 --cache-info-age Duration How long to cache file structure information (directory listings, file size, times, etc.) (default 6h0m0s) 19275 --cache-plex-insecure string Skip all certificate verification when connecting to the Plex server 19276 --cache-plex-password string The password of the Plex user (obscured) 19277 --cache-plex-url string The URL of the Plex server 19278 --cache-plex-username string The username of the Plex user 19279 --cache-read-retries int How many times to retry a read from a cache storage (default 10) 19280 --cache-remote string Remote to cache 19281 --cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1) 19282 --cache-tmp-upload-path string Directory to keep temporary files until they are uploaded 19283 --cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s) 19284 --cache-workers int How many workers should run in parallel to download chunks (default 4) 19285 --cache-writes Cache file data on writes through the FS 19286 --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks (default 2Gi) 19287 --chunker-description string Description of the remote 19288 --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks 19289 --chunker-hash-type string Choose how chunker handles hash sums (default "md5") 19290 --chunker-remote string Remote to chunk/unchunk 19291 --combine-description string Description of the remote 19292 --combine-upstreams SpaceSepList Upstreams for combining 19293 --compress-description string Description of the remote 19294 --compress-level int GZIP compression level (-2 to 9) (default -1) 19295 --compress-mode string Compression mode (default "gzip") 19296 --compress-ram-cache-limit SizeSuffix Some remotes don't allow the upload of files with unknown size (default 20Mi) 19297 --compress-remote string Remote to compress 19298 -L, --copy-links Follow symlinks and copy the pointed to item 19299 --crypt-description string Description of the remote 19300 --crypt-directory-name-encryption Option to either encrypt directory names or leave them intact (default true) 19301 --crypt-filename-encoding string How to encode the encrypted filename to text string (default "base32") 19302 --crypt-filename-encryption string How to encrypt the filenames (default "standard") 19303 --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted 19304 --crypt-pass-bad-blocks If set this will pass bad blocks through as all 0 19305 --crypt-password string Password or pass phrase for encryption (obscured) 19306 --crypt-password2 string Password or pass phrase for salt (obscured) 19307 --crypt-remote string Remote to encrypt/decrypt 19308 --crypt-server-side-across-configs Deprecated: use --server-side-across-configs instead 19309 --crypt-show-mapping For all files listed show how the names encrypt 19310 --crypt-strict-names If set, this will raise an error when crypt comes across a filename that can't be decrypted 19311 --crypt-suffix string If this is set it will override the default suffix of ".bin" (default ".bin") 19312 --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded 19313 --drive-allow-import-name-change Allow the filetype to change when uploading Google docs 19314 --drive-auth-owner-only Only consider files owned by the authenticated user 19315 --drive-auth-url string Auth server URL 19316 --drive-chunk-size SizeSuffix Upload chunk size (default 8Mi) 19317 --drive-client-id string Google Application Client Id 19318 --drive-client-secret string OAuth Client Secret 19319 --drive-copy-shortcut-content Server side copy contents of shortcuts instead of the shortcut 19320 --drive-description string Description of the remote 19321 --drive-disable-http2 Disable drive using http2 (default true) 19322 --drive-encoding Encoding The encoding for the backend (default InvalidUtf8) 19323 --drive-env-auth Get IAM credentials from runtime (environment variables or instance meta data if no env vars) 19324 --drive-export-formats string Comma separated list of preferred formats for downloading Google docs (default "docx,xlsx,pptx,svg") 19325 --drive-fast-list-bug-fix Work around a bug in Google Drive listing (default true) 19326 --drive-formats string Deprecated: See export_formats 19327 --drive-impersonate string Impersonate this user when using a service account 19328 --drive-import-formats string Comma separated list of preferred formats for uploading Google docs 19329 --drive-keep-revision-forever Keep new head revision of each file forever 19330 --drive-list-chunk int Size of listing chunk 100-1000, 0 to disable (default 1000) 19331 --drive-metadata-labels Bits Control whether labels should be read or written in metadata (default off) 19332 --drive-metadata-owner Bits Control whether owner should be read or written in metadata (default read) 19333 --drive-metadata-permissions Bits Control whether permissions should be read or written in metadata (default off) 19334 --drive-pacer-burst int Number of API calls to allow without sleeping (default 100) 19335 --drive-pacer-min-sleep Duration Minimum time to sleep between API calls (default 100ms) 19336 --drive-resource-key string Resource key for accessing a link-shared file 19337 --drive-root-folder-id string ID of the root folder 19338 --drive-scope string Comma separated list of scopes that rclone should use when requesting access from drive 19339 --drive-server-side-across-configs Deprecated: use --server-side-across-configs instead 19340 --drive-service-account-credentials string Service Account Credentials JSON blob 19341 --drive-service-account-file string Service Account Credentials JSON file path 19342 --drive-shared-with-me Only show files that are shared with me 19343 --drive-show-all-gdocs Show all Google Docs including non-exportable ones in listings 19344 --drive-size-as-quota Show sizes as storage quota usage, not actual size 19345 --drive-skip-checksum-gphotos Skip checksums on Google photos and videos only 19346 --drive-skip-dangling-shortcuts If set skip dangling shortcut files 19347 --drive-skip-gdocs Skip google documents in all listings 19348 --drive-skip-shortcuts If set skip shortcut files 19349 --drive-starred-only Only show files that are starred 19350 --drive-stop-on-download-limit Make download limit errors be fatal 19351 --drive-stop-on-upload-limit Make upload limit errors be fatal 19352 --drive-team-drive string ID of the Shared Drive (Team Drive) 19353 --drive-token string OAuth Access Token as a JSON blob 19354 --drive-token-url string Token server url 19355 --drive-trashed-only Only show files that are in the trash 19356 --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi) 19357 --drive-use-created-date Use file created date instead of modified date 19358 --drive-use-shared-date Use date file was shared instead of modified date 19359 --drive-use-trash Send files to the trash instead of deleting permanently (default true) 19360 --drive-v2-download-min-size SizeSuffix If Object's are greater, use drive v2 API to download (default off) 19361 --dropbox-auth-url string Auth server URL 19362 --dropbox-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) 19363 --dropbox-batch-mode string Upload file batching sync|async|off (default "sync") 19364 --dropbox-batch-size int Max number of files in upload batch 19365 --dropbox-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s) 19366 --dropbox-chunk-size SizeSuffix Upload chunk size (< 150Mi) (default 48Mi) 19367 --dropbox-client-id string OAuth Client Id 19368 --dropbox-client-secret string OAuth Client Secret 19369 --dropbox-description string Description of the remote 19370 --dropbox-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot) 19371 --dropbox-impersonate string Impersonate this user when using a business account 19372 --dropbox-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) 19373 --dropbox-shared-files Instructs rclone to work on individual shared files 19374 --dropbox-shared-folders Instructs rclone to work on shared folders 19375 --dropbox-token string OAuth Access Token as a JSON blob 19376 --dropbox-token-url string Token server url 19377 --fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl 19378 --fichier-cdn Set if you wish to use CDN download links 19379 --fichier-description string Description of the remote 19380 --fichier-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot) 19381 --fichier-file-password string If you want to download a shared file that is password protected, add this parameter (obscured) 19382 --fichier-folder-password string If you want to list the files in a shared folder that is password protected, add this parameter (obscured) 19383 --fichier-shared-folder string If you want to download a shared folder, add this parameter 19384 --filefabric-description string Description of the remote 19385 --filefabric-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) 19386 --filefabric-permanent-token string Permanent Authentication Token 19387 --filefabric-root-folder-id string ID of the root folder 19388 --filefabric-token string Session Token 19389 --filefabric-token-expiry string Token expiry time 19390 --filefabric-url string URL of the Enterprise File Fabric to connect to 19391 --filefabric-version string Version read from the file fabric 19392 --ftp-ask-password Allow asking for FTP password when needed 19393 --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s) 19394 --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited 19395 --ftp-description string Description of the remote 19396 --ftp-disable-epsv Disable using EPSV even if server advertises support 19397 --ftp-disable-mlsd Disable using MLSD even if server advertises support 19398 --ftp-disable-tls13 Disable TLS 1.3 (workaround for FTP servers with buggy TLS) 19399 --ftp-disable-utf8 Disable using UTF-8 even if server advertises support 19400 --ftp-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot) 19401 --ftp-explicit-tls Use Explicit FTPS (FTP over TLS) 19402 --ftp-force-list-hidden Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD 19403 --ftp-host string FTP host to connect to 19404 --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s) 19405 --ftp-no-check-certificate Do not verify the TLS certificate of the server 19406 --ftp-pass string FTP password (obscured) 19407 --ftp-port int FTP port number (default 21) 19408 --ftp-shut-timeout Duration Maximum time to wait for data connection closing status (default 1m0s) 19409 --ftp-socks-proxy string Socks 5 proxy host 19410 --ftp-tls Use Implicit FTPS (FTP over TLS) 19411 --ftp-tls-cache-size int Size of TLS session cache for all control and data connections (default 32) 19412 --ftp-user string FTP username (default "$USER") 19413 --ftp-writing-mdtm Use MDTM to set modification time (VsFtpd quirk) 19414 --gcs-anonymous Access public buckets and objects without credentials 19415 --gcs-auth-url string Auth server URL 19416 --gcs-bucket-acl string Access Control List for new buckets 19417 --gcs-bucket-policy-only Access checks should use bucket-level IAM policies 19418 --gcs-client-id string OAuth Client Id 19419 --gcs-client-secret string OAuth Client Secret 19420 --gcs-decompress If set this will decompress gzip encoded objects 19421 --gcs-description string Description of the remote 19422 --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created 19423 --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) 19424 --gcs-endpoint string Endpoint for the service 19425 --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) 19426 --gcs-location string Location for the newly created buckets 19427 --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it 19428 --gcs-object-acl string Access Control List for new objects 19429 --gcs-project-number string Project number 19430 --gcs-service-account-file string Service Account Credentials JSON file path 19431 --gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage 19432 --gcs-token string OAuth Access Token as a JSON blob 19433 --gcs-token-url string Token server url 19434 --gcs-user-project string User project 19435 --gphotos-auth-url string Auth server URL 19436 --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s) 19437 --gphotos-batch-mode string Upload file batching sync|async|off (default "sync") 19438 --gphotos-batch-size int Max number of files in upload batch 19439 --gphotos-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s) 19440 --gphotos-client-id string OAuth Client Id 19441 --gphotos-client-secret string OAuth Client Secret 19442 --gphotos-description string Description of the remote 19443 --gphotos-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) 19444 --gphotos-include-archived Also view and download archived media 19445 --gphotos-read-only Set to make the Google Photos backend read only 19446 --gphotos-read-size Set to read the size of media items 19447 --gphotos-start-year int Year limits the photos to be downloaded to those which are uploaded after the given year (default 2000) 19448 --gphotos-token string OAuth Access Token as a JSON blob 19449 --gphotos-token-url string Token server url 19450 --hasher-auto-size SizeSuffix Auto-update checksum for files smaller than this size (disabled by default) 19451 --hasher-description string Description of the remote 19452 --hasher-hashes CommaSepList Comma separated list of supported checksum types (default md5,sha1) 19453 --hasher-max-age Duration Maximum time to keep checksums in cache (0 = no cache, off = cache forever) (default off) 19454 --hasher-remote string Remote to cache checksums for (e.g. myRemote:path) 19455 --hdfs-data-transfer-protection string Kerberos data transfer protection: authentication|integrity|privacy 19456 --hdfs-description string Description of the remote 19457 --hdfs-encoding Encoding The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot) 19458 --hdfs-namenode CommaSepList Hadoop name nodes and ports 19459 --hdfs-service-principal-name string Kerberos service principal name for the namenode 19460 --hdfs-username string Hadoop user name 19461 --hidrive-auth-url string Auth server URL 19462 --hidrive-chunk-size SizeSuffix Chunksize for chunked uploads (default 48Mi) 19463 --hidrive-client-id string OAuth Client Id 19464 --hidrive-client-secret string OAuth Client Secret 19465 --hidrive-description string Description of the remote 19466 --hidrive-disable-fetching-member-count Do not fetch number of objects in directories unless it is absolutely necessary 19467 --hidrive-encoding Encoding The encoding for the backend (default Slash,Dot) 19468 --hidrive-endpoint string Endpoint for the service (default "https://api.hidrive.strato.com/2.1") 19469 --hidrive-root-prefix string The root/parent folder for all paths (default "/") 19470 --hidrive-scope-access string Access permissions that rclone should use when requesting access from HiDrive (default "rw") 19471 --hidrive-scope-role string User-level that rclone should use when requesting access from HiDrive (default "user") 19472 --hidrive-token string OAuth Access Token as a JSON blob 19473 --hidrive-token-url string Token server url 19474 --hidrive-upload-concurrency int Concurrency for chunked uploads (default 4) 19475 --hidrive-upload-cutoff SizeSuffix Cutoff/Threshold for chunked uploads (default 96Mi) 19476 --http-description string Description of the remote 19477 --http-headers CommaSepList Set HTTP headers for all transactions 19478 --http-no-head Don't use HEAD requests 19479 --http-no-slash Set this if the site doesn't end directories with / 19480 --http-url string URL of HTTP host to connect to 19481 --imagekit-description string Description of the remote 19482 --imagekit-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket) 19483 --imagekit-endpoint string You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 19484 --imagekit-only-signed Restrict unsigned image URLs If you have configured Restrict unsigned image URLs in your dashboard settings, set this to true 19485 --imagekit-private-key string You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 19486 --imagekit-public-key string You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 19487 --imagekit-upload-tags string Tags to add to the uploaded files, e.g. "tag1,tag2" 19488 --imagekit-versions Include old versions in directory listings 19489 --internetarchive-access-key-id string IAS3 Access Key 19490 --internetarchive-description string Description of the remote 19491 --internetarchive-disable-checksum Don't ask the server to test against MD5 checksum calculated by rclone (default true) 19492 --internetarchive-encoding Encoding The encoding for the backend (default Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot) 19493 --internetarchive-endpoint string IAS3 Endpoint (default "https://s3.us.archive.org") 19494 --internetarchive-front-endpoint string Host of InternetArchive Frontend (default "https://archive.org") 19495 --internetarchive-secret-access-key string IAS3 Secret Key (password) 19496 --internetarchive-wait-archive Duration Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish (default 0s) 19497 --jottacloud-auth-url string Auth server URL 19498 --jottacloud-client-id string OAuth Client Id 19499 --jottacloud-client-secret string OAuth Client Secret 19500 --jottacloud-description string Description of the remote 19501 --jottacloud-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot) 19502 --jottacloud-hard-delete Delete files permanently rather than putting them into the trash 19503 --jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required (default 10Mi) 19504 --jottacloud-no-versions Avoid server side versioning by deleting files and recreating files instead of overwriting them 19505 --jottacloud-token string OAuth Access Token as a JSON blob 19506 --jottacloud-token-url string Token server url 19507 --jottacloud-trashed-only Only show files that are in the trash 19508 --jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's (default 10Mi) 19509 --koofr-description string Description of the remote 19510 --koofr-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19511 --koofr-endpoint string The Koofr API endpoint to use 19512 --koofr-mountid string Mount ID of the mount to use 19513 --koofr-password string Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) (obscured) 19514 --koofr-provider string Choose your storage provider 19515 --koofr-setmtime Does the backend support setting modification time (default true) 19516 --koofr-user string Your user name 19517 --linkbox-description string Description of the remote 19518 --linkbox-token string Token from https://www.linkbox.to/admin/account 19519 -l, --links Translate symlinks to/from regular files with a '.rclonelink' extension 19520 --local-case-insensitive Force the filesystem to report itself as case insensitive 19521 --local-case-sensitive Force the filesystem to report itself as case sensitive 19522 --local-description string Description of the remote 19523 --local-encoding Encoding The encoding for the backend (default Slash,Dot) 19524 --local-no-check-updated Don't check to see if the files change during upload 19525 --local-no-preallocate Disable preallocation of disk space for transferred files 19526 --local-no-set-modtime Disable setting modtime 19527 --local-no-sparse Disable sparse files for multi-thread downloads 19528 --local-nounc Disable UNC (long path names) conversion on Windows 19529 --local-unicode-normalization Apply unicode NFC normalization to paths and filenames 19530 --local-zero-size-links Assume the Stat size of links is zero (and read them instead) (deprecated) 19531 --mailru-auth-url string Auth server URL 19532 --mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true) 19533 --mailru-client-id string OAuth Client Id 19534 --mailru-client-secret string OAuth Client Secret 19535 --mailru-description string Description of the remote 19536 --mailru-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19537 --mailru-pass string Password (obscured) 19538 --mailru-speedup-enable Skip full upload if there is another file with same data hash (default true) 19539 --mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash) (default "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf") 19540 --mailru-speedup-max-disk SizeSuffix This option allows you to disable speedup (put by hash) for large files (default 3Gi) 19541 --mailru-speedup-max-memory SizeSuffix Files larger than the size given below will always be hashed on disk (default 32Mi) 19542 --mailru-token string OAuth Access Token as a JSON blob 19543 --mailru-token-url string Token server url 19544 --mailru-user string User name (usually email) 19545 --mega-debug Output more debug from Mega 19546 --mega-description string Description of the remote 19547 --mega-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot) 19548 --mega-hard-delete Delete files permanently rather than putting them into the trash 19549 --mega-pass string Password (obscured) 19550 --mega-use-https Use HTTPS for transfers 19551 --mega-user string User name 19552 --memory-description string Description of the remote 19553 --netstorage-account string Set the NetStorage account name 19554 --netstorage-description string Description of the remote 19555 --netstorage-host string Domain+path of NetStorage host to connect to 19556 --netstorage-protocol string Select between HTTP or HTTPS protocol (default "https") 19557 --netstorage-secret string Set the NetStorage account secret/G2O key for authentication (obscured) 19558 -x, --one-file-system Don't cross filesystem boundaries (unix/macOS only) 19559 --onedrive-access-scopes SpaceSepList Set scopes to be requested by rclone (default Files.Read Files.ReadWrite Files.Read.All Files.ReadWrite.All Sites.Read.All offline_access) 19560 --onedrive-auth-url string Auth server URL 19561 --onedrive-av-override Allows download of files the server thinks has a virus 19562 --onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes) (default 10Mi) 19563 --onedrive-client-id string OAuth Client Id 19564 --onedrive-client-secret string OAuth Client Secret 19565 --onedrive-delta If set rclone will use delta listing to implement recursive listings 19566 --onedrive-description string Description of the remote 19567 --onedrive-drive-id string The ID of the drive to use 19568 --onedrive-drive-type string The type of the drive (personal | business | documentLibrary) 19569 --onedrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot) 19570 --onedrive-expose-onenote-files Set to make OneNote files show up in directory listings 19571 --onedrive-hash-type string Specify the hash in use for the backend (default "auto") 19572 --onedrive-link-password string Set the password for links created by the link command 19573 --onedrive-link-scope string Set the scope of the links created by the link command (default "anonymous") 19574 --onedrive-link-type string Set the type of the links created by the link command (default "view") 19575 --onedrive-list-chunk int Size of listing chunk (default 1000) 19576 --onedrive-metadata-permissions Bits Control whether permissions should be read or written in metadata (default off) 19577 --onedrive-no-versions Remove all versions on modifying operations 19578 --onedrive-region string Choose national cloud region for OneDrive (default "global") 19579 --onedrive-root-folder-id string ID of the root folder 19580 --onedrive-server-side-across-configs Deprecated: use --server-side-across-configs instead 19581 --onedrive-token string OAuth Access Token as a JSON blob 19582 --onedrive-token-url string Token server url 19583 --oos-attempt-resume-upload If true attempt to resume previously started multipart upload for the object 19584 --oos-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi) 19585 --oos-compartment string Object storage compartment OCID 19586 --oos-config-file string Path to OCI config file (default "~/.oci/config") 19587 --oos-config-profile string Profile name inside the oci config file (default "Default") 19588 --oos-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4.656Gi) 19589 --oos-copy-timeout Duration Timeout for copy (default 1m0s) 19590 --oos-description string Description of the remote 19591 --oos-disable-checksum Don't store MD5 checksum with object metadata 19592 --oos-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot) 19593 --oos-endpoint string Endpoint for Object storage API 19594 --oos-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery 19595 --oos-max-upload-parts int Maximum number of parts in a multipart upload (default 10000) 19596 --oos-namespace string Object storage namespace 19597 --oos-no-check-bucket If set, don't attempt to check the bucket exists or create it 19598 --oos-provider string Choose your Auth Provider (default "env_auth") 19599 --oos-region string Object storage Region 19600 --oos-sse-customer-algorithm string If using SSE-C, the optional header that specifies "AES256" as the encryption algorithm 19601 --oos-sse-customer-key string To use SSE-C, the optional header that specifies the base64-encoded 256-bit encryption key to use to 19602 --oos-sse-customer-key-file string To use SSE-C, a file containing the base64-encoded string of the AES-256 encryption key associated 19603 --oos-sse-customer-key-sha256 string If using SSE-C, The optional header that specifies the base64-encoded SHA256 hash of the encryption 19604 --oos-sse-kms-key-id string if using your own master key in vault, this header specifies the 19605 --oos-storage-tier string The storage class to use when storing new objects in storage. https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/understandingstoragetiers.htm (default "Standard") 19606 --oos-upload-concurrency int Concurrency for multipart uploads (default 10) 19607 --oos-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) 19608 --opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size (default 10Mi) 19609 --opendrive-description string Description of the remote 19610 --opendrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot) 19611 --opendrive-password string Password (obscured) 19612 --opendrive-username string Username 19613 --pcloud-auth-url string Auth server URL 19614 --pcloud-client-id string OAuth Client Id 19615 --pcloud-client-secret string OAuth Client Secret 19616 --pcloud-description string Description of the remote 19617 --pcloud-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19618 --pcloud-hostname string Hostname to connect to (default "api.pcloud.com") 19619 --pcloud-password string Your pcloud password (obscured) 19620 --pcloud-root-folder-id string Fill in for rclone to use a non root folder as its starting point (default "d0") 19621 --pcloud-token string OAuth Access Token as a JSON blob 19622 --pcloud-token-url string Token server url 19623 --pcloud-username string Your pcloud username 19624 --pikpak-auth-url string Auth server URL 19625 --pikpak-client-id string OAuth Client Id 19626 --pikpak-client-secret string OAuth Client Secret 19627 --pikpak-description string Description of the remote 19628 --pikpak-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot) 19629 --pikpak-hash-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate hash if required (default 10Mi) 19630 --pikpak-pass string Pikpak password (obscured) 19631 --pikpak-root-folder-id string ID of the root folder 19632 --pikpak-token string OAuth Access Token as a JSON blob 19633 --pikpak-token-url string Token server url 19634 --pikpak-trashed-only Only show files that are in the trash 19635 --pikpak-use-trash Send files to the trash instead of deleting permanently (default true) 19636 --pikpak-user string Pikpak username 19637 --premiumizeme-auth-url string Auth server URL 19638 --premiumizeme-client-id string OAuth Client Id 19639 --premiumizeme-client-secret string OAuth Client Secret 19640 --premiumizeme-description string Description of the remote 19641 --premiumizeme-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19642 --premiumizeme-token string OAuth Access Token as a JSON blob 19643 --premiumizeme-token-url string Token server url 19644 --protondrive-2fa string The 2FA code 19645 --protondrive-app-version string The app version string (default "macos-drive@1.0.0-alpha.1+rclone") 19646 --protondrive-description string Description of the remote 19647 --protondrive-enable-caching Caches the files and folders metadata to reduce API calls (default true) 19648 --protondrive-encoding Encoding The encoding for the backend (default Slash,LeftSpace,RightSpace,InvalidUtf8,Dot) 19649 --protondrive-mailbox-password string The mailbox password of your two-password proton account (obscured) 19650 --protondrive-original-file-size Return the file size before encryption (default true) 19651 --protondrive-password string The password of your proton account (obscured) 19652 --protondrive-replace-existing-draft Create a new revision when filename conflict is detected 19653 --protondrive-username string The username of your proton account 19654 --putio-auth-url string Auth server URL 19655 --putio-client-id string OAuth Client Id 19656 --putio-client-secret string OAuth Client Secret 19657 --putio-description string Description of the remote 19658 --putio-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19659 --putio-token string OAuth Access Token as a JSON blob 19660 --putio-token-url string Token server url 19661 --qingstor-access-key-id string QingStor Access Key ID 19662 --qingstor-chunk-size SizeSuffix Chunk size to use for uploading (default 4Mi) 19663 --qingstor-connection-retries int Number of connection retries (default 3) 19664 --qingstor-description string Description of the remote 19665 --qingstor-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8) 19666 --qingstor-endpoint string Enter an endpoint URL to connection QingStor API 19667 --qingstor-env-auth Get QingStor credentials from runtime 19668 --qingstor-secret-access-key string QingStor Secret Access Key (password) 19669 --qingstor-upload-concurrency int Concurrency for multipart uploads (default 1) 19670 --qingstor-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) 19671 --qingstor-zone string Zone to connect to 19672 --quatrix-api-key string API key for accessing Quatrix account 19673 --quatrix-description string Description of the remote 19674 --quatrix-effective-upload-time string Wanted upload time for one chunk (default "4s") 19675 --quatrix-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot) 19676 --quatrix-hard-delete Delete files permanently rather than putting them into the trash 19677 --quatrix-host string Host name of Quatrix account 19678 --quatrix-maximal-summary-chunk-size SizeSuffix The maximal summary for all chunks. It should not be less than 'transfers'*'minimal_chunk_size' (default 95.367Mi) 19679 --quatrix-minimal-chunk-size SizeSuffix The minimal size for one chunk (default 9.537Mi) 19680 --quatrix-skip-project-folders Skip project folders in operations 19681 --s3-access-key-id string AWS Access Key ID 19682 --s3-acl string Canned ACL used when creating buckets and storing or copying objects 19683 --s3-bucket-acl string Canned ACL used when creating buckets 19684 --s3-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi) 19685 --s3-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4.656Gi) 19686 --s3-decompress If set this will decompress gzip encoded objects 19687 --s3-description string Description of the remote 19688 --s3-directory-markers Upload an empty object with a trailing slash when a new directory is created 19689 --s3-disable-checksum Don't store MD5 checksum with object metadata 19690 --s3-disable-http2 Disable usage of http2 for S3 backends 19691 --s3-download-url string Custom endpoint for downloads 19692 --s3-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot) 19693 --s3-endpoint string Endpoint for S3 API 19694 --s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars) 19695 --s3-force-path-style If true use path style access if false use virtual hosted style (default true) 19696 --s3-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery 19697 --s3-list-chunk int Size of listing chunk (response list for each ListObject S3 request) (default 1000) 19698 --s3-list-url-encode Tristate Whether to url encode listings: true/false/unset (default unset) 19699 --s3-list-version int Version of ListObjects to use: 1,2 or 0 for auto 19700 --s3-location-constraint string Location constraint - must be set to match the Region 19701 --s3-max-upload-parts int Maximum number of parts in a multipart upload (default 10000) 19702 --s3-might-gzip Tristate Set this if the backend might gzip objects (default unset) 19703 --s3-no-check-bucket If set, don't attempt to check the bucket exists or create it 19704 --s3-no-head If set, don't HEAD uploaded objects to check integrity 19705 --s3-no-head-object If set, do not do HEAD before GET when getting objects 19706 --s3-no-system-metadata Suppress setting and reading of system metadata 19707 --s3-profile string Profile to use in the shared credentials file 19708 --s3-provider string Choose your S3 provider 19709 --s3-region string Region to connect to 19710 --s3-requester-pays Enables requester pays option when interacting with S3 bucket 19711 --s3-secret-access-key string AWS Secret Access Key (password) 19712 --s3-server-side-encryption string The server-side encryption algorithm used when storing this object in S3 19713 --s3-session-token string An AWS session token 19714 --s3-shared-credentials-file string Path to the shared credentials file 19715 --s3-sse-customer-algorithm string If using SSE-C, the server-side encryption algorithm used when storing this object in S3 19716 --s3-sse-customer-key string To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data 19717 --s3-sse-customer-key-base64 string If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data 19718 --s3-sse-customer-key-md5 string If using SSE-C you may provide the secret encryption key MD5 checksum (optional) 19719 --s3-sse-kms-key-id string If using KMS ID you must provide the ARN of Key 19720 --s3-storage-class string The storage class to use when storing new objects in S3 19721 --s3-sts-endpoint string Endpoint for STS 19722 --s3-upload-concurrency int Concurrency for multipart uploads and copies (default 4) 19723 --s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi) 19724 --s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint 19725 --s3-use-accept-encoding-gzip Accept-Encoding: gzip Whether to send Accept-Encoding: gzip header (default unset) 19726 --s3-use-already-exists Tristate Set if rclone should report BucketAlreadyExists errors on bucket creation (default unset) 19727 --s3-use-dual-stack If true use AWS S3 dual-stack endpoint (IPv6 support) 19728 --s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset) 19729 --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset) 19730 --s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads 19731 --s3-v2-auth If true use v2 authentication 19732 --s3-version-at Time Show file versions as they were at the specified time (default off) 19733 --s3-version-deleted Show deleted file markers when using versions 19734 --s3-versions Include old versions in directory listings 19735 --seafile-2fa Two-factor authentication ('true' if the account has 2FA enabled) 19736 --seafile-create-library Should rclone create a library if it doesn't exist 19737 --seafile-description string Description of the remote 19738 --seafile-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8) 19739 --seafile-library string Name of the library 19740 --seafile-library-key string Library password (for encrypted libraries only) (obscured) 19741 --seafile-pass string Password (obscured) 19742 --seafile-url string URL of seafile host to connect to 19743 --seafile-user string User name (usually email address) 19744 --sftp-ask-password Allow asking for SFTP password when needed 19745 --sftp-chunk-size SizeSuffix Upload and download chunk size (default 32Ki) 19746 --sftp-ciphers SpaceSepList Space separated list of ciphers to be used for session encryption, ordered by preference 19747 --sftp-concurrency int The maximum number of outstanding requests for one file (default 64) 19748 --sftp-copy-is-hardlink Set to enable server side copies using hardlinks 19749 --sftp-description string Description of the remote 19750 --sftp-disable-concurrent-reads If set don't use concurrent reads 19751 --sftp-disable-concurrent-writes If set don't use concurrent writes 19752 --sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available 19753 --sftp-host string SSH host to connect to 19754 --sftp-host-key-algorithms SpaceSepList Space separated list of host key algorithms, ordered by preference 19755 --sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s) 19756 --sftp-key-exchange SpaceSepList Space separated list of key exchange algorithms, ordered by preference 19757 --sftp-key-file string Path to PEM-encoded private key file 19758 --sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file (obscured) 19759 --sftp-key-pem string Raw PEM-encoded private key 19760 --sftp-key-use-agent When set forces the usage of the ssh-agent 19761 --sftp-known-hosts-file string Optional path to known_hosts file 19762 --sftp-macs SpaceSepList Space separated list of MACs (message authentication code) algorithms, ordered by preference 19763 --sftp-md5sum-command string The command used to read md5 hashes 19764 --sftp-pass string SSH password, leave blank to use ssh-agent (obscured) 19765 --sftp-path-override string Override path used by SSH shell commands 19766 --sftp-port int SSH port number (default 22) 19767 --sftp-pubkey-file string Optional path to public key file 19768 --sftp-server-command string Specifies the path or command to run a sftp server on the remote host 19769 --sftp-set-env SpaceSepList Environment variables to pass to sftp and commands 19770 --sftp-set-modtime Set the modified time on the remote if set (default true) 19771 --sftp-sha1sum-command string The command used to read sha1 hashes 19772 --sftp-shell-type string The type of SSH shell on remote server, if any 19773 --sftp-skip-links Set to skip any symlinks and any other non regular files 19774 --sftp-socks-proxy string Socks 5 proxy host 19775 --sftp-ssh SpaceSepList Path and arguments to external ssh binary 19776 --sftp-subsystem string Specifies the SSH2 subsystem on the remote host (default "sftp") 19777 --sftp-use-fstat If set use fstat instead of stat 19778 --sftp-use-insecure-cipher Enable the use of insecure ciphers and key exchange methods 19779 --sftp-user string SSH username (default "$USER") 19780 --sharefile-auth-url string Auth server URL 19781 --sharefile-chunk-size SizeSuffix Upload chunk size (default 64Mi) 19782 --sharefile-client-id string OAuth Client Id 19783 --sharefile-client-secret string OAuth Client Secret 19784 --sharefile-description string Description of the remote 19785 --sharefile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot) 19786 --sharefile-endpoint string Endpoint for API calls 19787 --sharefile-root-folder-id string ID of the root folder 19788 --sharefile-token string OAuth Access Token as a JSON blob 19789 --sharefile-token-url string Token server url 19790 --sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (default 128Mi) 19791 --sia-api-password string Sia Daemon API Password (obscured) 19792 --sia-api-url string Sia daemon API URL, like http://sia.daemon.host:9980 (default "http://127.0.0.1:9980") 19793 --sia-description string Description of the remote 19794 --sia-encoding Encoding The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot) 19795 --sia-user-agent string Siad User Agent (default "Sia-Agent") 19796 --skip-links Don't warn about skipped symlinks 19797 --smb-case-insensitive Whether the server is configured to be case-insensitive (default true) 19798 --smb-description string Description of the remote 19799 --smb-domain string Domain name for NTLM authentication (default "WORKGROUP") 19800 --smb-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot) 19801 --smb-hide-special-share Hide special shares (e.g. print$) which users aren't supposed to access (default true) 19802 --smb-host string SMB server hostname to connect to 19803 --smb-idle-timeout Duration Max time before closing idle connections (default 1m0s) 19804 --smb-pass string SMB password (obscured) 19805 --smb-port int SMB port number (default 445) 19806 --smb-spn string Service principal name 19807 --smb-user string SMB username (default "$USER") 19808 --storj-access-grant string Access grant 19809 --storj-api-key string API key 19810 --storj-description string Description of the remote 19811 --storj-passphrase string Encryption passphrase 19812 --storj-provider string Choose an authentication method (default "existing") 19813 --storj-satellite-address string Satellite address (default "us1.storj.io") 19814 --sugarsync-access-key-id string Sugarsync Access Key ID 19815 --sugarsync-app-id string Sugarsync App ID 19816 --sugarsync-authorization string Sugarsync authorization 19817 --sugarsync-authorization-expiry string Sugarsync authorization expiry 19818 --sugarsync-deleted-id string Sugarsync deleted folder id 19819 --sugarsync-description string Description of the remote 19820 --sugarsync-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot) 19821 --sugarsync-hard-delete Permanently delete files if true 19822 --sugarsync-private-access-key string Sugarsync Private Access Key 19823 --sugarsync-refresh-token string Sugarsync refresh token 19824 --sugarsync-root-id string Sugarsync root id 19825 --sugarsync-user string Sugarsync user 19826 --swift-application-credential-id string Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 19827 --swift-application-credential-name string Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 19828 --swift-application-credential-secret string Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 19829 --swift-auth string Authentication URL for server (OS_AUTH_URL) 19830 --swift-auth-token string Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 19831 --swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 19832 --swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container (default 5Gi) 19833 --swift-description string Description of the remote 19834 --swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 19835 --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8) 19836 --swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public") 19837 --swift-env-auth Get swift credentials from environment variables in standard OpenStack form 19838 --swift-key string API key or password (OS_PASSWORD) 19839 --swift-leave-parts-on-error If true avoid calling abort upload on a failure 19840 --swift-no-chunk Don't chunk files during streaming upload 19841 --swift-no-large-objects Disable support for static and dynamic large objects 19842 --swift-region string Region name - optional (OS_REGION_NAME) 19843 --swift-storage-policy string The storage policy to use when creating a new container 19844 --swift-storage-url string Storage URL - optional (OS_STORAGE_URL) 19845 --swift-tenant string Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 19846 --swift-tenant-domain string Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 19847 --swift-tenant-id string Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 19848 --swift-user string User name to log in (OS_USERNAME) 19849 --swift-user-id string User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID) 19850 --union-action-policy string Policy to choose upstream on ACTION category (default "epall") 19851 --union-cache-time int Cache time of usage and free space (in seconds) (default 120) 19852 --union-create-policy string Policy to choose upstream on CREATE category (default "epmfs") 19853 --union-description string Description of the remote 19854 --union-min-free-space SizeSuffix Minimum viable free space for lfs/eplfs policies (default 1Gi) 19855 --union-search-policy string Policy to choose upstream on SEARCH category (default "ff") 19856 --union-upstreams string List of space separated upstreams 19857 --uptobox-access-token string Your access token 19858 --uptobox-description string Description of the remote 19859 --uptobox-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot) 19860 --uptobox-private Set to make uploaded files private 19861 --webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon) 19862 --webdav-bearer-token-command string Command to run to get a bearer token 19863 --webdav-description string Description of the remote 19864 --webdav-encoding string The encoding for the backend 19865 --webdav-headers CommaSepList Set HTTP headers for all transactions 19866 --webdav-nextcloud-chunk-size SizeSuffix Nextcloud upload chunk size (default 10Mi) 19867 --webdav-owncloud-exclude-shares Exclude ownCloud shares 19868 --webdav-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms) 19869 --webdav-pass string Password (obscured) 19870 --webdav-url string URL of http host to connect to 19871 --webdav-user string User name 19872 --webdav-vendor string Name of the WebDAV site/service/software you are using 19873 --yandex-auth-url string Auth server URL 19874 --yandex-client-id string OAuth Client Id 19875 --yandex-client-secret string OAuth Client Secret 19876 --yandex-description string Description of the remote 19877 --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot) 19878 --yandex-hard-delete Delete files permanently rather than putting them into the trash 19879 --yandex-token string OAuth Access Token as a JSON blob 19880 --yandex-token-url string Token server url 19881 --zoho-auth-url string Auth server URL 19882 --zoho-client-id string OAuth Client Id 19883 --zoho-client-secret string OAuth Client Secret 19884 --zoho-description string Description of the remote 19885 --zoho-encoding Encoding The encoding for the backend (default Del,Ctl,InvalidUtf8) 19886 --zoho-region string Zoho region to connect to 19887 --zoho-token string OAuth Access Token as a JSON blob 19888 --zoho-token-url string Token server url 19889 19890 Docker Volume Plugin 19891 19892 Introduction 19893 19894 Docker 1.9 has added support for creating named volumes via command-line 19895 interface and mounting them in containers as a way to share data between 19896 them. Since Docker 1.10 you can create named volumes with Docker Compose 19897 by descriptions in docker-compose.yml files for use by container groups 19898 on a single host. As of Docker 1.12 volumes are supported by Docker 19899 Swarm included with Docker Engine and created from descriptions in swarm 19900 compose v3 files for use with swarm stacks across multiple cluster 19901 nodes. 19902 19903 Docker Volume Plugins augment the default local volume driver included 19904 in Docker with stateful volumes shared across containers and hosts. 19905 Unlike local volumes, your data will not be deleted when such volume is 19906 removed. Plugins can run managed by the docker daemon, as a native 19907 system service (under systemd, sysv or upstart) or as a standalone 19908 executable. Rclone can run as docker volume plugin in all these modes. 19909 It interacts with the local docker daemon via plugin API and handles 19910 mounting of remote file systems into docker containers so it must run on 19911 the same host as the docker daemon or on every Swarm node. 19912 19913 Getting started 19914 19915 In the first example we will use the SFTP rclone volume with Docker 19916 engine on a standalone Ubuntu machine. 19917 19918 Start from installing Docker on the host. 19919 19920 The FUSE driver is a prerequisite for rclone mounting and should be 19921 installed on host: 19922 19923 sudo apt-get -y install fuse 19924 19925 Create two directories required by rclone docker plugin: 19926 19927 sudo mkdir -p /var/lib/docker-plugins/rclone/config 19928 sudo mkdir -p /var/lib/docker-plugins/rclone/cache 19929 19930 Install the managed rclone docker plugin for your architecture (here 19931 amd64): 19932 19933 docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions 19934 docker plugin list 19935 19936 Create your SFTP volume: 19937 19938 docker volume create firstvolume -d rclone -o type=sftp -o sftp-host=_hostname_ -o sftp-user=_username_ -o sftp-pass=_password_ -o allow-other=true 19939 19940 Note that since all options are static, you don't even have to run 19941 rclone config or create the rclone.conf file (but the config directory 19942 should still be present). In the simplest case you can use localhost as 19943 hostname and your SSH credentials as username and password. You can also 19944 change the remote path to your home directory on the host, for example 19945 -o path=/home/username. 19946 19947 Time to create a test container and mount the volume into it: 19948 19949 docker run --rm -it -v firstvolume:/mnt --workdir /mnt ubuntu:latest bash 19950 19951 If all goes well, you will enter the new container and change right to 19952 the mounted SFTP remote. You can type ls to list the mounted directory 19953 or otherwise play with it. Type exit when you are done. The container 19954 will stop but the volume will stay, ready to be reused. When it's not 19955 needed anymore, remove it: 19956 19957 docker volume list 19958 docker volume remove firstvolume 19959 19960 Now let us try something more elaborate: Google Drive volume on 19961 multi-node Docker Swarm. 19962 19963 You should start from installing Docker and FUSE, creating plugin 19964 directories and installing rclone plugin on every swarm node. Then setup 19965 the Swarm. 19966 19967 Google Drive volumes need an access token which can be setup via web 19968 browser and will be periodically renewed by rclone. The managed plugin 19969 cannot run a browser so we will use a technique similar to the rclone 19970 setup on a headless box. 19971 19972 Run rclone config on another machine equipped with web browser and 19973 graphical user interface. Create the Google Drive remote. When done, 19974 transfer the resulting rclone.conf to the Swarm cluster and save as 19975 /var/lib/docker-plugins/rclone/config/rclone.conf on every node. By 19976 default this location is accessible only to the root user so you will 19977 need appropriate privileges. The resulting config will look like this: 19978 19979 [gdrive] 19980 type = drive 19981 scope = drive 19982 drive_id = 1234567... 19983 root_folder_id = 0Abcd... 19984 token = {"access_token":...} 19985 19986 Now create the file named example.yml with a swarm stack description 19987 like this: 19988 19989 version: '3' 19990 services: 19991 heimdall: 19992 image: linuxserver/heimdall:latest 19993 ports: [8080:80] 19994 volumes: [configdata:/config] 19995 volumes: 19996 configdata: 19997 driver: rclone 19998 driver_opts: 19999 remote: 'gdrive:heimdall' 20000 allow_other: 'true' 20001 vfs_cache_mode: full 20002 poll_interval: 0 20003 20004 and run the stack: 20005 20006 docker stack deploy example -c ./example.yml 20007 20008 After a few seconds docker will spread the parsed stack description over 20009 cluster, create the example_heimdall service on port 8080, run service 20010 containers on one or more cluster nodes and request the 20011 example_configdata volume from rclone plugins on the node hosts. You can 20012 use the following commands to confirm results: 20013 20014 docker service ls 20015 docker service ps example_heimdall 20016 docker volume ls 20017 20018 Point your browser to http://cluster.host.address:8080 and play with the 20019 service. Stop it with docker stack remove example when you are done. 20020 Note that the example_configdata volume(s) created on demand at the 20021 cluster nodes will not be automatically removed together with the stack 20022 but stay for future reuse. You can remove them manually by invoking the 20023 docker volume remove example_configdata command on every node. 20024 20025 Creating Volumes via CLI 20026 20027 Volumes can be created with docker volume create. Here are a few 20028 examples: 20029 20030 docker volume create vol1 -d rclone -o remote=storj: -o vfs-cache-mode=full 20031 docker volume create vol2 -d rclone -o remote=:storj,access_grant=xxx:heimdall 20032 docker volume create vol3 -d rclone -o type=storj -o path=heimdall -o storj-access-grant=xxx -o poll-interval=0 20033 20034 Note the -d rclone flag that tells docker to request volume from the 20035 rclone driver. This works even if you installed managed driver by its 20036 full name rclone/docker-volume-rclone because you provided the 20037 --alias rclone option. 20038 20039 Volumes can be inspected as follows: 20040 20041 docker volume list 20042 docker volume inspect vol1 20043 20044 Volume Configuration 20045 20046 Rclone flags and volume options are set via the -o flag to the 20047 docker volume create command. They include backend-specific parameters 20048 as well as mount and VFS options. Also there are a few special -o 20049 options: remote, fs, type, path, mount-type and persist. 20050 20051 remote determines an existing remote name from the config file, with 20052 trailing colon and optionally with a remote path. See the full syntax in 20053 the rclone documentation. This option can be aliased as fs to prevent 20054 confusion with the remote parameter of such backends as crypt or alias. 20055 20056 The remote=:backend:dir/subdir syntax can be used to create on-the-fly 20057 (config-less) remotes, while the type and path options provide a simpler 20058 alternative for this. Using two split options 20059 20060 -o type=backend -o path=dir/subdir 20061 20062 is equivalent to the combined syntax 20063 20064 -o remote=:backend:dir/subdir 20065 20066 but is arguably easier to parameterize in scripts. The path part is 20067 optional. 20068 20069 Mount and VFS options as well as backend parameters are named like their 20070 twin command-line flags without the -- CLI prefix. Optionally you can 20071 use underscores instead of dashes in option names. For example, 20072 --vfs-cache-mode full becomes -o vfs-cache-mode=full or 20073 -o vfs_cache_mode=full. Boolean CLI flags without value will gain the 20074 true value, e.g. --allow-other becomes -o allow-other=true or 20075 -o allow_other=true. 20076 20077 Please note that you can provide parameters only for the backend 20078 immediately referenced by the backend type of mounted remote. If this is 20079 a wrapping backend like alias, chunker or crypt, you cannot provide 20080 options for the referred to remote or backend. This limitation is 20081 imposed by the rclone connection string parser. The only workaround is 20082 to feed plugin with rclone.conf or configure plugin arguments (see 20083 below). 20084 20085 Special Volume Options 20086 20087 mount-type determines the mount method and in general can be one of: 20088 mount, cmount, or mount2. This can be aliased as mount_type. It should 20089 be noted that the managed rclone docker plugin currently does not 20090 support the cmount method and mount2 is rarely needed. This option 20091 defaults to the first found method, which is usually mount so you 20092 generally won't need it. 20093 20094 persist is a reserved boolean (true/false) option. In future it will 20095 allow to persist on-the-fly remotes in the plugin rclone.conf file. 20096 20097 Connection Strings 20098 20099 The remote value can be extended with connection strings as an 20100 alternative way to supply backend parameters. This is equivalent to the 20101 -o backend options with one syntactic difference. Inside connection 20102 string the backend prefix must be dropped from parameter names but in 20103 the -o param=value array it must be present. For instance, compare the 20104 following option array 20105 20106 -o remote=:sftp:/home -o sftp-host=localhost 20107 20108 with equivalent connection string: 20109 20110 -o remote=:sftp,host=localhost:/home 20111 20112 This difference exists because flag options -o key=val include not only 20113 backend parameters but also mount/VFS flags and possibly other settings. 20114 Also it allows to discriminate the remote option from the crypt-remote 20115 (or similarly named backend parameters) and arguably simplifies 20116 scripting due to clearer value substitution. 20117 20118 Using with Swarm or Compose 20119 20120 Both Docker Swarm and Docker Compose use YAML-formatted text files to 20121 describe groups (stacks) of containers, their properties, networks and 20122 volumes. Compose uses the compose v2 format, Swarm uses the compose v3 20123 format. They are mostly similar, differences are explained in the docker 20124 documentation. 20125 20126 Volumes are described by the children of the top-level volumes: node. 20127 Each of them should be named after its volume and have at least two 20128 elements, the self-explanatory driver: rclone value and the driver_opts: 20129 structure playing the same role as -o key=val CLI flags: 20130 20131 volumes: 20132 volume_name_1: 20133 driver: rclone 20134 driver_opts: 20135 remote: 'gdrive:' 20136 allow_other: 'true' 20137 vfs_cache_mode: full 20138 token: '{"type": "borrower", "expires": "2021-12-31"}' 20139 poll_interval: 0 20140 20141 Notice a few important details: - YAML prefers _ in option names instead 20142 of -. - YAML treats single and double quotes interchangeably. Simple 20143 strings and integers can be left unquoted. - Boolean values must be 20144 quoted like 'true' or "false" because these two words are reserved by 20145 YAML. - The filesystem string is keyed with remote (or with fs). 20146 Normally you can omit quotes here, but if the string ends with colon, 20147 you must quote it like remote: "storage_box:". - YAML is picky about 20148 surrounding braces in values as this is in fact another syntax for 20149 key/value mappings. For example, JSON access tokens usually contain 20150 double quotes and surrounding braces, so you must put them in single 20151 quotes. 20152 20153 Installing as Managed Plugin 20154 20155 Docker daemon can install plugins from an image registry and run them 20156 managed. We maintain the docker-volume-rclone plugin image on Docker 20157 Hub. 20158 20159 Rclone volume plugin requires Docker Engine >= 19.03.15 20160 20161 The plugin requires presence of two directories on the host before it 20162 can be installed. Note that plugin will not create them automatically. 20163 By default they must exist on host at the following locations (though 20164 you can tweak the paths): - /var/lib/docker-plugins/rclone/config is 20165 reserved for the rclone.conf config file and must exist even if it's 20166 empty and the config file is not present. - 20167 /var/lib/docker-plugins/rclone/cache holds the plugin state file as well 20168 as optional VFS caches. 20169 20170 You can install managed plugin with default settings as follows: 20171 20172 docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone 20173 20174 The :amd64 part of the image specification after colon is called a tag. 20175 Usually you will want to install the latest plugin for your 20176 architecture. In this case the tag will just name it, like amd64 above. 20177 The following plugin architectures are currently available: - amd64 - 20178 arm64 - arm-v7 20179 20180 Sometimes you might want a concrete plugin version, not the latest one. 20181 Then you should use image tag in the form :ARCHITECTURE-VERSION. For 20182 example, to install plugin version v1.56.2 on architecture arm64 you 20183 will use tag arm64-1.56.2 (note the removed v) so the full image 20184 specification becomes rclone/docker-volume-rclone:arm64-1.56.2. 20185 20186 We also provide the latest plugin tag, but since docker does not support 20187 multi-architecture plugins as of the time of this writing, this tag is 20188 currently an alias for amd64. By convention the latest tag is the 20189 default one and can be omitted, thus both 20190 rclone/docker-volume-rclone:latest and just rclone/docker-volume-rclone 20191 will refer to the latest plugin release for the amd64 platform. 20192 20193 Also the amd64 part can be omitted from the versioned rclone plugin 20194 tags. For example, rclone image reference 20195 rclone/docker-volume-rclone:amd64-1.56.2 can be abbreviated as 20196 rclone/docker-volume-rclone:1.56.2 for convenience. However, for 20197 non-intel architectures you still have to use the full tag as amd64 or 20198 latest will fail to start. 20199 20200 Managed plugin is in fact a special container running in a namespace 20201 separate from normal docker containers. Inside it runs the 20202 rclone serve docker command. The config and cache directories are 20203 bind-mounted into the container at start. The docker daemon connects to 20204 a unix socket created by the command inside the container. The command 20205 creates on-demand remote mounts right inside, then docker machinery 20206 propagates them through kernel mount namespaces and bind-mounts into 20207 requesting user containers. 20208 20209 You can tweak a few plugin settings after installation when it's 20210 disabled (not in use), for instance: 20211 20212 docker plugin disable rclone 20213 docker plugin set rclone RCLONE_VERBOSE=2 config=/etc/rclone args="--vfs-cache-mode=writes --allow-other" 20214 docker plugin enable rclone 20215 docker plugin inspect rclone 20216 20217 Note that if docker refuses to disable the plugin, you should find and 20218 remove all active volumes connected with it as well as containers and 20219 swarm services that use them. This is rather tedious so please carefully 20220 plan in advance. 20221 20222 You can tweak the following settings: args, config, cache, HTTP_PROXY, 20223 HTTPS_PROXY, NO_PROXY and RCLONE_VERBOSE. It's your task to keep plugin 20224 settings in sync across swarm cluster nodes. 20225 20226 args sets command-line arguments for the rclone serve docker command 20227 (none by default). Arguments should be separated by space so you will 20228 normally want to put them in quotes on the docker plugin set command 20229 line. Both serve docker flags and generic rclone flags are supported, 20230 including backend parameters that will be used as defaults for volume 20231 creation. Note that plugin will fail (due to this docker bug) if the 20232 args value is empty. Use e.g. args="-v" as a workaround. 20233 20234 config=/host/dir sets alternative host location for the config 20235 directory. Plugin will look for rclone.conf here. It's not an error if 20236 the config file is not present but the directory must exist. Please note 20237 that plugin can periodically rewrite the config file, for example when 20238 it renews storage access tokens. Keep this in mind and try to avoid 20239 races between the plugin and other instances of rclone on the host that 20240 might try to change the config simultaneously resulting in corrupted 20241 rclone.conf. You can also put stuff like private key files for SFTP 20242 remotes in this directory. Just note that it's bind-mounted inside the 20243 plugin container at the predefined path /data/config. For example, if 20244 your key file is named sftp-box1.key on the host, the corresponding 20245 volume config option should read 20246 -o sftp-key-file=/data/config/sftp-box1.key. 20247 20248 cache=/host/dir sets alternative host location for the cache directory. 20249 The plugin will keep VFS caches here. Also it will create and maintain 20250 the docker-plugin.state file in this directory. When the plugin is 20251 restarted or reinstalled, it will look in this file to recreate any 20252 volumes that existed previously. However, they will not be re-mounted 20253 into consuming containers after restart. Usually this is not a problem 20254 as the docker daemon normally will restart affected user containers 20255 after failures, daemon restarts or host reboots. 20256 20257 RCLONE_VERBOSE sets plugin verbosity from 0 (errors only, by default) to 20258 2 (debugging). Verbosity can be also tweaked via args="-v [-v] ...". 20259 Since arguments are more generic, you will rarely need this setting. The 20260 plugin output by default feeds the docker daemon log on local host. Log 20261 entries are reflected as errors in the docker log but retain their 20262 actual level assigned by rclone in the encapsulated message string. 20263 20264 HTTP_PROXY, HTTPS_PROXY, NO_PROXY customize the plugin proxy settings. 20265 20266 You can set custom plugin options right when you install it, in one go: 20267 20268 docker plugin remove rclone 20269 docker plugin install rclone/docker-volume-rclone:amd64 \ 20270 --alias rclone --grant-all-permissions \ 20271 args="-v --allow-other" config=/etc/rclone 20272 docker plugin inspect rclone 20273 20274 Healthchecks 20275 20276 The docker plugin volume protocol doesn't provide a way for plugins to 20277 inform the docker daemon that a volume is (un-)available. As a 20278 workaround you can setup a healthcheck to verify that the mount is 20279 responding, for example: 20280 20281 services: 20282 my_service: 20283 image: my_image 20284 healthcheck: 20285 test: ls /path/to/rclone/mount || exit 1 20286 interval: 1m 20287 timeout: 15s 20288 retries: 3 20289 start_period: 15s 20290 20291 Running Plugin under Systemd 20292 20293 In most cases you should prefer managed mode. Moreover, MacOS and 20294 Windows do not support native Docker plugins. Please use managed mode on 20295 these systems. Proceed further only if you are on Linux. 20296 20297 First, install rclone. You can just run it (type rclone serve docker and 20298 hit enter) for the test. 20299 20300 Install FUSE: 20301 20302 sudo apt-get -y install fuse 20303 20304 Download two systemd configuration files: docker-volume-rclone.service 20305 and docker-volume-rclone.socket. 20306 20307 Put them to the /etc/systemd/system/ directory: 20308 20309 cp docker-volume-plugin.service /etc/systemd/system/ 20310 cp docker-volume-plugin.socket /etc/systemd/system/ 20311 20312 Please note that all commands in this section must be run as root but we 20313 omit sudo prefix for brevity. Now create directories required by the 20314 service: 20315 20316 mkdir -p /var/lib/docker-volumes/rclone 20317 mkdir -p /var/lib/docker-plugins/rclone/config 20318 mkdir -p /var/lib/docker-plugins/rclone/cache 20319 20320 Run the docker plugin service in the socket activated mode: 20321 20322 systemctl daemon-reload 20323 systemctl start docker-volume-rclone.service 20324 systemctl enable docker-volume-rclone.socket 20325 systemctl start docker-volume-rclone.socket 20326 systemctl restart docker 20327 20328 Or run the service directly: - run systemctl daemon-reload to let 20329 systemd pick up new config - run 20330 systemctl enable docker-volume-rclone.service to make the new service 20331 start automatically when you power on your machine. - run 20332 systemctl start docker-volume-rclone.service to start the service now. - 20333 run systemctl restart docker to restart docker daemon and let it detect 20334 the new plugin socket. Note that this step is not needed in managed mode 20335 where docker knows about plugin state changes. 20336 20337 The two methods are equivalent from the user perspective, but I 20338 personally prefer socket activation. 20339 20340 Troubleshooting 20341 20342 You can see managed plugin settings with 20343 20344 docker plugin list 20345 docker plugin inspect rclone 20346 20347 Note that docker (including latest 20.10.7) will not show actual values 20348 of args, just the defaults. 20349 20350 Use journalctl --unit docker to see managed plugin output as part of the 20351 docker daemon log. Note that docker reflects plugin lines as errors but 20352 their actual level can be seen from encapsulated message string. 20353 20354 You will usually install the latest version of managed plugin for your 20355 platform. Use the following commands to print the actual installed 20356 version: 20357 20358 PLUGID=$(docker plugin list --no-trunc | awk '/rclone/{print$1}') 20359 sudo runc --root /run/docker/runtime-runc/plugins.moby exec $PLUGID rclone version 20360 20361 You can even use runc to run shell inside the plugin container: 20362 20363 sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $PLUGID bash 20364 20365 Also you can use curl to check the plugin socket connectivity: 20366 20367 docker plugin list --no-trunc 20368 PLUGID=123abc... 20369 sudo curl -H Content-Type:application/json -XPOST -d {} --unix-socket /run/docker/plugins/$PLUGID/rclone.sock http://localhost/Plugin.Activate 20370 20371 though this is rarely needed. 20372 20373 Caveats 20374 20375 Finally I'd like to mention a caveat with updating volume settings. 20376 Docker CLI does not have a dedicated command like docker volume update. 20377 It may be tempting to invoke docker volume create with updated options 20378 on existing volume, but there is a gotcha. The command will do nothing, 20379 it won't even return an error. I hope that docker maintainers will fix 20380 this some day. In the meantime be aware that you must remove your volume 20381 before recreating it with new settings: 20382 20383 docker volume remove my_vol 20384 docker volume create my_vol -d rclone -o opt1=new_val1 ... 20385 20386 and verify that settings did update: 20387 20388 docker volume list 20389 docker volume inspect my_vol 20390 20391 If docker refuses to remove the volume, you should find containers or 20392 swarm services that use it and stop them first. 20393 20394 Bisync 20395 20396 bisync is in beta and is considered an advanced command, so use with 20397 care. Make sure you have read and understood the entire manual 20398 (especially the Limitations section) before using, or data loss can 20399 result. Questions can be asked in the Rclone Forum. 20400 20401 Getting started 20402 20403 - Install rclone and setup your remotes. 20404 - Bisync will create its working directory at ~/.cache/rclone/bisync 20405 on Linux, /Users/yourusername/Library/Caches/rclone/bisync on Mac, 20406 or C:\Users\MyLogin\AppData\Local\rclone\bisync on Windows. Make 20407 sure that this location is writable. 20408 - Run bisync with the --resync flag, specifying the paths to the local 20409 and remote sync directory roots. 20410 - For successive sync runs, leave off the --resync flag. (Important!) 20411 - Consider using a filters file for excluding unnecessary files and 20412 directories from the sync. 20413 - Consider setting up the --check-access feature for safety. 20414 - On Linux or Mac, consider setting up a crontab entry. bisync can 20415 safely run in concurrent cron jobs thanks to lock files it 20416 maintains. 20417 20418 For example, your first command might look like this: 20419 20420 rclone bisync remote1:path1 remote2:path2 --create-empty-src-dirs --compare size,modtime,checksum --slow-hash-sync-only --resilient -MvP --drive-skip-gdocs --fix-case --resync --dry-run 20421 20422 If all looks good, run it again without --dry-run. After that, remove 20423 --resync as well. 20424 20425 Here is a typical run log (with timestamps removed for clarity): 20426 20427 rclone bisync /testdir/path1/ /testdir/path2/ --verbose 20428 INFO : Synching Path1 "/testdir/path1/" with Path2 "/testdir/path2/" 20429 INFO : Path1 checking for diffs 20430 INFO : - Path1 File is new - file11.txt 20431 INFO : - Path1 File is newer - file2.txt 20432 INFO : - Path1 File is newer - file5.txt 20433 INFO : - Path1 File is newer - file7.txt 20434 INFO : - Path1 File was deleted - file4.txt 20435 INFO : - Path1 File was deleted - file6.txt 20436 INFO : - Path1 File was deleted - file8.txt 20437 INFO : Path1: 7 changes: 1 new, 3 newer, 0 older, 3 deleted 20438 INFO : Path2 checking for diffs 20439 INFO : - Path2 File is new - file10.txt 20440 INFO : - Path2 File is newer - file1.txt 20441 INFO : - Path2 File is newer - file5.txt 20442 INFO : - Path2 File is newer - file6.txt 20443 INFO : - Path2 File was deleted - file3.txt 20444 INFO : - Path2 File was deleted - file7.txt 20445 INFO : - Path2 File was deleted - file8.txt 20446 INFO : Path2: 7 changes: 1 new, 3 newer, 0 older, 3 deleted 20447 INFO : Applying changes 20448 INFO : - Path1 Queue copy to Path2 - /testdir/path2/file11.txt 20449 INFO : - Path1 Queue copy to Path2 - /testdir/path2/file2.txt 20450 INFO : - Path2 Queue delete - /testdir/path2/file4.txt 20451 NOTICE: - WARNING New or changed in both paths - file5.txt 20452 NOTICE: - Path1 Renaming Path1 copy - /testdir/path1/file5.txt..path1 20453 NOTICE: - Path1 Queue copy to Path2 - /testdir/path2/file5.txt..path1 20454 NOTICE: - Path2 Renaming Path2 copy - /testdir/path2/file5.txt..path2 20455 NOTICE: - Path2 Queue copy to Path1 - /testdir/path1/file5.txt..path2 20456 INFO : - Path2 Queue copy to Path1 - /testdir/path1/file6.txt 20457 INFO : - Path1 Queue copy to Path2 - /testdir/path2/file7.txt 20458 INFO : - Path2 Queue copy to Path1 - /testdir/path1/file1.txt 20459 INFO : - Path2 Queue copy to Path1 - /testdir/path1/file10.txt 20460 INFO : - Path1 Queue delete - /testdir/path1/file3.txt 20461 INFO : - Path2 Do queued copies to - Path1 20462 INFO : - Path1 Do queued copies to - Path2 20463 INFO : - Do queued deletes on - Path1 20464 INFO : - Do queued deletes on - Path2 20465 INFO : Updating listings 20466 INFO : Validating listings for Path1 "/testdir/path1/" vs Path2 "/testdir/path2/" 20467 INFO : Bisync successful 20468 20469 Command line syntax 20470 20471 $ rclone bisync --help 20472 Usage: 20473 rclone bisync remote1:path1 remote2:path2 [flags] 20474 20475 Positional arguments: 20476 Path1, Path2 Local path, or remote storage with ':' plus optional path. 20477 Type 'rclone listremotes' for list of configured remotes. 20478 20479 Optional Flags: 20480 --backup-dir1 string --backup-dir for Path1. Must be a non-overlapping path on the same remote. 20481 --backup-dir2 string --backup-dir for Path2. Must be a non-overlapping path on the same remote. 20482 --check-access Ensure expected RCLONE_TEST files are found on both Path1 and Path2 filesystems, else abort. 20483 --check-filename string Filename for --check-access (default: RCLONE_TEST) 20484 --check-sync string Controls comparison of final listings: true|false|only (default: true) (default "true") 20485 --compare string Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime') 20486 --conflict-loser ConflictLoserAction Action to take on the loser of a sync conflict (when there is a winner) or on both files (when there is no winner): , num, pathname, delete (default: num) 20487 --conflict-resolve string Automatically resolve conflicts by preferring the version that is: none, path1, path2, newer, older, larger, smaller (default: none) (default "none") 20488 --conflict-suffix string Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict') 20489 --create-empty-src-dirs Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs) 20490 --download-hash Compute hash by downloading when otherwise unavailable. (warning: may be slow and use lots of data!) 20491 --filters-file string Read filtering patterns from a file 20492 --force Bypass --max-delete safety check and run the sync. Consider using with --verbose 20493 -h, --help help for bisync 20494 --ignore-listing-checksum Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks) 20495 --max-lock Duration Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m) (default 0s) 20496 --no-cleanup Retain working files (useful for troubleshooting and testing). 20497 --no-slow-hash Ignore listing checksums only on backends where they are slow 20498 --recover Automatically recover from interruptions without requiring --resync. 20499 --remove-empty-dirs Remove ALL empty directories at the final cleanup step. 20500 --resilient Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk! 20501 -1, --resync Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first. 20502 --resync-mode string During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.) (default "none") 20503 --retries int Retry operations this many times if they fail (requires --resilient). (default 3) 20504 --retries-sleep Duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s) 20505 --slow-hash-sync-only Ignore slow checksums for listings and deltas, but still consider them during sync calls. 20506 --workdir string Use custom working dir - useful for testing. (default: {WORKDIR}) 20507 --max-delete PERCENT Safety check on maximum percentage of deleted files allowed. If exceeded, the bisync run will abort. (default: 50%) 20508 -n, --dry-run Go through the motions - No files are copied/deleted. 20509 -v, --verbose Increases logging verbosity. May be specified more than once for more details. 20510 20511 Arbitrary rclone flags may be specified on the bisync command line, for 20512 example 20513 rclone bisync ./testdir/path1/ gdrive:testdir/path2/ --drive-skip-gdocs -v -v --timeout 10s 20514 Note that interactions of various rclone flags with bisync process flow 20515 has not been fully tested yet. 20516 20517 Paths 20518 20519 Path1 and Path2 arguments may be references to any mix of local 20520 directory paths (absolute or relative), UNC paths (//server/share/path), 20521 Windows drive paths (with a drive letter and :) or configured remotes 20522 with optional subdirectory paths. Cloud references are distinguished by 20523 having a : in the argument (see Windows support below). 20524 20525 Path1 and Path2 are treated equally, in that neither has priority for 20526 file changes (except during --resync), and access efficiency does not 20527 change whether a remote is on Path1 or Path2. 20528 20529 The listings in bisync working directory (default: 20530 ~/.cache/rclone/bisync) are named based on the Path1 and Path2 arguments 20531 so that separate syncs to individual directories within the tree may be 20532 set up, e.g.: path_to_local_tree..dropbox_subdir.lst. 20533 20534 Any empty directories after the sync on both the Path1 and Path2 20535 filesystems are not deleted by default, unless --create-empty-src-dirs 20536 is specified. If the --remove-empty-dirs flag is specified, then both 20537 paths will have ALL empty directories purged as the last step in the 20538 process. 20539 20540 Command-line flags 20541 20542 --resync 20543 20544 This will effectively make both Path1 and Path2 filesystems contain a 20545 matching superset of all files. By default, Path2 files that do not 20546 exist in Path1 will be copied to Path1, and the process will then copy 20547 the Path1 tree to Path2. 20548 20549 The --resync sequence is roughly equivalent to the following (but see 20550 --resync-mode for other options): 20551 20552 rclone copy Path2 Path1 --ignore-existing [--create-empty-src-dirs] 20553 rclone copy Path1 Path2 [--create-empty-src-dirs] 20554 20555 The base directories on both Path1 and Path2 filesystems must exist or 20556 bisync will fail. This is required for safety - that bisync can verify 20557 that both paths are valid. 20558 20559 When using --resync, a newer version of a file on the Path2 filesystem 20560 will (by default) be overwritten by the Path1 filesystem version. (Note 20561 that this is NOT entirely symmetrical, and more symmetrical options can 20562 be specified with the --resync-mode flag.) Carefully evaluate deltas 20563 using --dry-run. 20564 20565 For a resync run, one of the paths may be empty (no files in the path 20566 tree). The resync run should result in files on both paths, else a 20567 normal non-resync run will fail. 20568 20569 For a non-resync run, either path being empty (no files in the tree) 20570 fails with 20571 Empty current PathN listing. Cannot sync to an empty directory: X.pathN.lst 20572 This is a safety check that an unexpected empty path does not result in 20573 deleting everything in the other path. 20574 20575 Note that --resync implies --resync-mode path1 unless a different 20576 --resync-mode is explicitly specified. It is not necessary to use both 20577 the --resync and --resync-mode flags -- either one is sufficient without 20578 the other. 20579 20580 Note: --resync (including --resync-mode) should only be used under three 20581 specific (rare) circumstances: 1. It is your first bisync run (between 20582 these two paths) 2. You've just made changes to your bisync settings 20583 (such as editing the contents of your --filters-file) 3. There was an 20584 error on the prior run, and as a result, bisync now requires --resync to 20585 recover 20586 20587 The rest of the time, you should omit --resync. The reason is because 20588 --resync will only copy (not sync) each side to the other. Therefore, if 20589 you included --resync for every bisync run, it would never be possible 20590 to delete a file -- the deleted file would always keep reappearing at 20591 the end of every run (because it's being copied from the other side 20592 where it still exists). Similarly, renaming a file would always result 20593 in a duplicate copy (both old and new name) on both sides. 20594 20595 If you find that frequent interruptions from #3 are an issue, rather 20596 than automatically running --resync, the recommended alternative is to 20597 use the --resilient, --recover, and --conflict-resolve flags, (along 20598 with Graceful Shutdown mode, when needed) for a very robust 20599 "set-it-and-forget-it" bisync setup that can automatically bounce back 20600 from almost any interruption it might encounter. Consider adding 20601 something like the following: 20602 20603 --resilient --recover --max-lock 2m --conflict-resolve newer 20604 20605 --resync-mode CHOICE 20606 20607 In the event that a file differs on both sides during a --resync, 20608 --resync-mode controls which version will overwrite the other. The 20609 supported options are similar to --conflict-resolve. For all of the 20610 following options, the version that is kept is referred to as the 20611 "winner", and the version that is overwritten (deleted) is referred to 20612 as the "loser". The options are named after the "winner": 20613 20614 - path1 - (the default) - the version from Path1 is unconditionally 20615 considered the winner (regardless of modtime and size, if any). This 20616 can be useful if one side is more trusted or up-to-date than the 20617 other, at the time of the --resync. 20618 - path2 - same as path1, except the path2 version is considered the 20619 winner. 20620 - newer - the newer file (by modtime) is considered the winner, 20621 regardless of which side it came from. This may result in having a 20622 mix of some winners from Path1, and some winners from Path2. (The 20623 implementation is analogous to running rclone copy --update in both 20624 directions.) 20625 - older - same as newer, except the older file is considered the 20626 winner, and the newer file is considered the loser. 20627 - larger - the larger file (by size) is considered the winner 20628 (regardless of modtime, if any). This can be a useful option for 20629 remotes without modtime support, or with the kinds of files (such as 20630 logs) that tend to grow but not shrink, over time. 20631 - smaller - the smaller file (by size) is considered the winner 20632 (regardless of modtime, if any). 20633 20634 For all of the above options, note the following: - If either of the 20635 underlying remotes lacks support for the chosen method, it will be 20636 ignored and will fall back to the default of path1. (For example, if 20637 --resync-mode newer is set, but one of the paths uses a remote that 20638 doesn't support modtime.) - If a winner can't be determined because the 20639 chosen method's attribute is missing or equal, it will be ignored, and 20640 bisync will instead try to determine whether the files differ by looking 20641 at the other --compare methods in effect. (For example, if 20642 --resync-mode newer is set, but the Path1 and Path2 modtimes are 20643 identical, bisync will compare the sizes.) If bisync concludes that they 20644 differ, preference is given to whichever is the "source" at that moment. 20645 (In practice, this gives a slight advantage to Path2, as the 2to1 copy 20646 comes before the 1to2 copy.) If the files do not differ, nothing is 20647 copied (as both sides are already correct). - These options apply only 20648 to files that exist on both sides (with the same name and relative 20649 path). Files that exist only on one side and not the other are always 20650 copied to the other, during --resync (this is one of the main 20651 differences between resync and non-resync runs.). - --conflict-resolve, 20652 --conflict-loser, and --conflict-suffix do not apply during --resync, 20653 and unlike these flags, nothing is renamed during --resync. When a file 20654 differs on both sides during --resync, one version always overwrites the 20655 other (much like in rclone copy.) (Consider using --backup-dir to retain 20656 a backup of the losing version.) - Unlike for --conflict-resolve, 20657 --resync-mode none is not a valid option (or rather, it will be 20658 interpreted as "no resync", unless --resync has also been specified, in 20659 which case it will be ignored.) - Winners and losers are decided at the 20660 individual file-level only (there is not currently an option to pick an 20661 entire winning directory atomically, although the path1 and path2 20662 options typically produce a similar result.) - To maintain 20663 backward-compatibility, the --resync flag implies --resync-mode path1 20664 unless a different --resync-mode is explicitly specified. Similarly, all 20665 --resync-mode options (except none) imply --resync, so it is not 20666 necessary to use both the --resync and --resync-mode flags 20667 simultaneously -- either one is sufficient without the other. 20668 20669 --check-access 20670 20671 Access check files are an additional safety measure against data loss. 20672 bisync will ensure it can find matching RCLONE_TEST files in the same 20673 places in the Path1 and Path2 filesystems. RCLONE_TEST files are not 20674 generated automatically. For --check-access to succeed, you must first 20675 either: A) Place one or more RCLONE_TEST files in both systems, or B) 20676 Set --check-filename to a filename already in use in various locations 20677 throughout your sync'd fileset. Recommended methods for A) include: * 20678 rclone touch Path1/RCLONE_TEST (create a new file) * 20679 rclone copyto Path1/RCLONE_TEST Path2/RCLONE_TEST (copy an existing 20680 file) * 20681 rclone copy Path1/RCLONE_TEST Path2/RCLONE_TEST --include "RCLONE_TEST" 20682 (copy multiple files at once, recursively) * create the files manually 20683 (outside of rclone) * run bisync once without --check-access to set 20684 matching files on both filesystems will also work, but is not preferred, 20685 due to potential for user error (you are temporarily disabling the 20686 safety feature). 20687 20688 Note that --check-access is still enforced on --resync, so 20689 bisync --resync --check-access will not work as a method of initially 20690 setting the files (this is to ensure that bisync can't inadvertently 20691 circumvent its own safety switch.) 20692 20693 Time stamps and file contents for RCLONE_TEST files are not important, 20694 just the names and locations. If you have symbolic links in your sync 20695 tree it is recommended to place RCLONE_TEST files in the linked-to 20696 directory tree to protect against bisync assuming a bunch of deleted 20697 files if the linked-to tree should not be accessible. See also the 20698 --check-filename flag. 20699 20700 --check-filename 20701 20702 Name of the file(s) used in access health validation. The default 20703 --check-filename is RCLONE_TEST. One or more files having this filename 20704 must exist, synchronized between your source and destination filesets, 20705 in order for --check-access to succeed. See --check-access for 20706 additional details. 20707 20708 --compare 20709 20710 As of v1.66, bisync fully supports comparing based on any combination of 20711 size, modtime, and checksum (lifting the prior restriction on backends 20712 without modtime support.) 20713 20714 By default (without the --compare flag), bisync inherits the same 20715 comparison options as sync (that is: size and modtime by default, unless 20716 modified with flags such as --checksum or --size-only.) 20717 20718 If the --compare flag is set, it will override these defaults. This can 20719 be useful if you wish to compare based on combinations not currently 20720 supported in sync, such as comparing all three of size AND modtime AND 20721 checksum simultaneously (or just modtime AND checksum). 20722 20723 --compare takes a comma-separated list, with the currently supported 20724 values being size, modtime, and checksum. For example, if you want to 20725 compare size and checksum, but not modtime, you would do: 20726 20727 --compare size,checksum 20728 20729 Or if you want to compare all three: 20730 20731 --compare size,modtime,checksum 20732 20733 --compare overrides any conflicting flags. For example, if you set the 20734 conflicting flags --compare checksum --size-only, --size-only will be 20735 ignored, and bisync will compare checksum and not size. To avoid 20736 confusion, it is recommended to use either --compare or the normal sync 20737 flags, but not both. 20738 20739 If --compare includes checksum and both remotes support checksums but 20740 have no hash types in common with each other, checksums will be 20741 considered only for comparisons within the same side (to determine what 20742 has changed since the prior sync), but not for comparisons against the 20743 opposite side. If one side supports checksums and the other does not, 20744 checksums will only be considered on the side that supports them. 20745 20746 When comparing with checksum and/or size without modtime, bisync cannot 20747 determine whether a file is newer or older -- only whether it is changed 20748 or unchanged. (If it is changed on both sides, bisync still does the 20749 standard equality-check to avoid declaring a sync conflict unless it 20750 absolutely has to.) 20751 20752 It is recommended to do a --resync when changing --compare settings, as 20753 otherwise your prior listing files may not contain the attributes you 20754 wish to compare (for example, they will not have stored checksums if you 20755 were not previously comparing checksums.) 20756 20757 --ignore-listing-checksum 20758 20759 When --checksum or --compare checksum is set, bisync will retrieve (or 20760 generate) checksums (for backends that support them) when creating the 20761 listings for both paths, and store the checksums in the listing files. 20762 --ignore-listing-checksum will disable this behavior, which may speed 20763 things up considerably, especially on backends (such as local) where 20764 hashes must be computed on the fly instead of retrieved. Please note the 20765 following: 20766 20767 - As of v1.66, --ignore-listing-checksum is now automatically set when 20768 neither --checksum nor --compare checksum are in use (as the 20769 checksums would not be used for anything.) 20770 - --ignore-listing-checksum is NOT the same as --ignore-checksum, and 20771 you may wish to use one or the other, or both. In a nutshell: 20772 --ignore-listing-checksum controls whether checksums are considered 20773 when scanning for diffs, while --ignore-checksum controls whether 20774 checksums are considered during the copy/sync operations that 20775 follow, if there ARE diffs. 20776 - Unless --ignore-listing-checksum is passed, bisync currently 20777 computes hashes for one path even when there's no common hash with 20778 the other path (for example, a crypt remote.) This can still be 20779 beneficial, as the hashes will still be used to detect changes 20780 within the same side (if --checksum or --compare checksum is set), 20781 even if they can't be used to compare against the opposite side. 20782 - If you wish to ignore listing checksums only on remotes where they 20783 are slow to compute, consider using --no-slow-hash (or 20784 --slow-hash-sync-only) instead of --ignore-listing-checksum. 20785 - If --ignore-listing-checksum is used simultaneously with 20786 --compare checksum (or --checksum), checksums will be ignored for 20787 bisync deltas, but still considered during the sync operations that 20788 follow (if deltas are detected based on modtime and/or size.) 20789 20790 --no-slow-hash 20791 20792 On some remotes (notably local), checksums can dramatically slow down a 20793 bisync run, because hashes cannot be stored and need to be computed in 20794 real-time when they are requested. On other remotes (such as drive), 20795 they add practically no time at all. The --no-slow-hash flag will 20796 automatically skip checksums on remotes where they are slow, while still 20797 comparing them on others (assuming --compare includes checksum.) This 20798 can be useful when one of your bisync paths is slow but you still want 20799 to check checksums on the other, for a more robust sync. 20800 20801 --slow-hash-sync-only 20802 20803 Same as --no-slow-hash, except slow hashes are still considered during 20804 sync calls. They are still NOT considered for determining deltas, nor or 20805 they included in listings. They are also skipped during --resync. The 20806 main use case for this flag is when you have a large number of files, 20807 but relatively few of them change from run to run -- so you don't want 20808 to check your entire tree every time (it would take too long), but you 20809 still want to consider checksums for the smaller group of files for 20810 which a modtime or size change was detected. Keep in mind that this 20811 speed savings comes with a safety trade-off: if a file's content were to 20812 change without a change to its modtime or size, bisync would not detect 20813 it, and it would not be synced. 20814 20815 --slow-hash-sync-only is only useful if both remotes share a common hash 20816 type (if they don't, bisync will automatically fall back to 20817 --no-slow-hash.) Both --no-slow-hash and --slow-hash-sync-only have no 20818 effect without --compare checksum (or --checksum). 20819 20820 --download-hash 20821 20822 If --download-hash is set, bisync will use best efforts to obtain an MD5 20823 checksum by downloading and computing on-the-fly, when checksums are not 20824 otherwise available (for example, a remote that doesn't support them.) 20825 Note that since rclone has to download the entire file, this may 20826 dramatically slow down your bisync runs, and is also likely to use a lot 20827 of data, so it is probably not practical for bisync paths with a large 20828 total file size. However, it can be a good option for syncing 20829 small-but-important files with maximum accuracy (for example, a source 20830 code repo on a crypt remote.) An additional advantage over methods like 20831 cryptcheck is that the original file is not required for comparison (for 20832 example, --download-hash can be used to bisync two different crypt 20833 remotes with different passwords.) 20834 20835 When --download-hash is set, bisync still looks for more efficient 20836 checksums first, and falls back to downloading only when none are found. 20837 It takes priority over conflicting flags such as --no-slow-hash. 20838 --download-hash is not suitable for Google Docs and other files of 20839 unknown size, as their checksums would change from run to run (due to 20840 small variances in the internals of the generated export file.) 20841 Therefore, bisync automatically skips --download-hash for files with a 20842 size less than 0. 20843 20844 See also: Hasher backend, cryptcheck command, rclone check --download 20845 option, md5sum command 20846 20847 --max-delete 20848 20849 As a safety check, if greater than the --max-delete percent of files 20850 were deleted on either the Path1 or Path2 filesystem, then bisync will 20851 abort with a warning message, without making any changes. The default 20852 --max-delete is 50%. One way to trigger this limit is to rename a 20853 directory that contains more than half of your files. This will appear 20854 to bisync as a bunch of deleted files and a bunch of new files. This 20855 safety check is intended to block bisync from deleting all of the files 20856 on both filesystems due to a temporary network access issue, or if the 20857 user had inadvertently deleted the files on one side or the other. To 20858 force the sync, either set a different delete percentage limit, e.g. 20859 --max-delete 75 (allows up to 75% deletion), or use --force to bypass 20860 the check. 20861 20862 Also see the all files changed check. 20863 20864 --filters-file 20865 20866 By using rclone filter features you can exclude file types or directory 20867 sub-trees from the sync. See the bisync filters section and generic 20868 --filter-from documentation. An example filters file contains filters 20869 for non-allowed files for synching with Dropbox. 20870 20871 If you make changes to your filters file then bisync requires a run with 20872 --resync. This is a safety feature, which prevents existing files on the 20873 Path1 and/or Path2 side from seeming to disappear from view (since they 20874 are excluded in the new listings), which would fool bisync into seeing 20875 them as deleted (as compared to the prior run listings), and then bisync 20876 would proceed to delete them for real. 20877 20878 To block this from happening, bisync calculates an MD5 hash of the 20879 filters file and stores the hash in a .md5 file in the same place as 20880 your filters file. On the next run with --filters-file set, bisync 20881 re-calculates the MD5 hash of the current filters file and compares it 20882 to the hash stored in the .md5 file. If they don't match, the run aborts 20883 with a critical error and thus forces you to do a --resync, likely 20884 avoiding a disaster. 20885 20886 --conflict-resolve CHOICE 20887 20888 In bisync, a "conflict" is a file that is new or changed on both sides 20889 (relative to the prior run) AND is not currently identical on both 20890 sides. --conflict-resolve controls how bisync handles such a scenario. 20891 The currently supported options are: 20892 20893 - none - (the default) - do not attempt to pick a winner, keep and 20894 rename both files according to --conflict-loser and 20895 --conflict-suffix settings. For example, with the default settings, 20896 file.txt on Path1 is renamed file.txt.conflict1 and file.txt on 20897 Path2 is renamed file.txt.conflict2. Both are copied to the opposite 20898 path during the run, so both sides end up with a copy of both files. 20899 (As none is the default, it is not necessary to specify 20900 --conflict-resolve none -- you can just omit the flag.) 20901 - newer - the newer file (by modtime) is considered the winner and is 20902 copied without renaming. The older file (the "loser") is handled 20903 according to --conflict-loser and --conflict-suffix settings (either 20904 renamed or deleted.) For example, if file.txt on Path1 is newer than 20905 file.txt on Path2, the result on both sides (with other default 20906 settings) will be file.txt (winner from Path1) and 20907 file.txt.conflict1 (loser from Path2). 20908 - older - same as newer, except the older file is considered the 20909 winner, and the newer file is considered the loser. 20910 - larger - the larger file (by size) is considered the winner 20911 (regardless of modtime, if any). 20912 - smaller - the smaller file (by size) is considered the winner 20913 (regardless of modtime, if any). 20914 - path1 - the version from Path1 is unconditionally considered the 20915 winner (regardless of modtime and size, if any). This can be useful 20916 if one side is usually more trusted or up-to-date than the other. 20917 - path2 - same as path1, except the path2 version is considered the 20918 winner. 20919 20920 For all of the above options, note the following: - If either of the 20921 underlying remotes lacks support for the chosen method, it will be 20922 ignored and fall back to none. (For example, if --conflict-resolve newer 20923 is set, but one of the paths uses a remote that doesn't support 20924 modtime.) - If a winner can't be determined because the chosen method's 20925 attribute is missing or equal, it will be ignored and fall back to none. 20926 (For example, if --conflict-resolve newer is set, but the Path1 and 20927 Path2 modtimes are identical, even if the sizes may differ.) - If the 20928 file's content is currently identical on both sides, it is not 20929 considered a "conflict", even if new or changed on both sides since the 20930 prior sync. (For example, if you made a change on one side and then 20931 synced it to the other side by other means.) Therefore, none of the 20932 conflict resolution flags apply in this scenario. - The conflict 20933 resolution flags do not apply during a --resync, as there is no "prior 20934 run" to speak of (but see --resync-mode for similar options.) 20935 20936 --conflict-loser CHOICE 20937 20938 --conflict-loser determines what happens to the "loser" of a sync 20939 conflict (when --conflict-resolve determines a winner) or to both files 20940 (when there is no winner.) The currently supported options are: 20941 20942 - num - (the default) - auto-number the conflicts by automatically 20943 appending the next available number to the --conflict-suffix, in 20944 chronological order. For example, with the default settings, the 20945 first conflict for file.txt will be renamed file.txt.conflict1. If 20946 file.txt.conflict1 already exists, file.txt.conflict2 will be used 20947 instead (etc., up to a maximum of 9223372036854775807 conflicts.) 20948 - pathname - rename the conflicts according to which side they came 20949 from, which was the default behavior prior to v1.66. For example, 20950 with --conflict-suffix path, file.txt from Path1 will be renamed 20951 file.txt.path1, and file.txt from Path2 will be renamed 20952 file.txt.path2. If two non-identical suffixes are provided (ex. 20953 --conflict-suffix cloud,local), the trailing digit is omitted. 20954 Importantly, note that with pathname, there is no auto-numbering 20955 beyond 2, so if file.txt.path2 somehow already exists, it will be 20956 overwritten. Using a dynamic date variable in your --conflict-suffix 20957 (see below) is one possible way to avoid this. Note also that 20958 conflicts-of-conflicts are possible, if the original conflict is not 20959 manually resolved -- for example, if for some reason you edited 20960 file.txt.path1 on both sides, and those edits were different, the 20961 result would be file.txt.path1.path1 and file.txt.path1.path2 (in 20962 addition to file.txt.path2.) 20963 - delete - keep the winner only and delete the loser, instead of 20964 renaming it. If a winner cannot be determined (see 20965 --conflict-resolve for details on how this could happen), delete is 20966 ignored and the default num is used instead (i.e. both versions are 20967 kept and renamed, and neither is deleted.) delete is inherently the 20968 most destructive option, so use it only with care. 20969 20970 For all of the above options, note that if a winner cannot be determined 20971 (see --conflict-resolve for details on how this could happen), or if 20972 --conflict-resolve is not in use, both files will be renamed. 20973 20974 --conflict-suffix STRING[,STRING] 20975 20976 --conflict-suffix controls the suffix that is appended when bisync 20977 renames a --conflict-loser (default: conflict). --conflict-suffix will 20978 accept either one string or two comma-separated strings to assign 20979 different suffixes to Path1 vs. Path2. This may be helpful later in 20980 identifying the source of the conflict. (For example, 20981 --conflict-suffix dropboxconflict,laptopconflict) 20982 20983 With --conflict-loser num, a number is always appended to the suffix. 20984 With --conflict-loser pathname, a number is appended only when one 20985 suffix is specified (or when two identical suffixes are specified.) i.e. 20986 with --conflict-loser pathname, all of the following would produce 20987 exactly the same result: 20988 20989 --conflict-suffix path 20990 --conflict-suffix path,path 20991 --conflict-suffix path1,path2 20992 20993 Suffixes may be as short as 1 character. By default, the suffix is 20994 appended after any other extensions (ex. file.jpg.conflict1), however, 20995 this can be changed with the --suffix-keep-extension flag (i.e. to 20996 instead result in file.conflict1.jpg). 20997 20998 --conflict-suffix supports several dynamic date variables when enclosed 20999 in curly braces as globs. This can be helpful to track the date and/or 21000 time that each conflict was handled by bisync. For example: 21001 21002 --conflict-suffix {DateOnly}-conflict 21003 // result: myfile.txt.2006-01-02-conflict1 21004 21005 All of the formats described here and here are supported, but take care 21006 to ensure that your chosen format does not use any characters that are 21007 illegal on your remotes (for example, macOS does not allow colons in 21008 filenames, and slashes are also best avoided as they are often 21009 interpreted as directory separators.) To address this particular issue, 21010 an additional {MacFriendlyTime} (or just {mac}) option is supported, 21011 which results in 2006-01-02 0304PM. 21012 21013 Note that --conflict-suffix is entirely separate from rclone's main 21014 --sufix flag. This is intentional, as users may wish to use both flags 21015 simultaneously, if also using --backup-dir. 21016 21017 Finally, note that the default in bisync prior to v1.66 was to rename 21018 conflicts with ..path1 and ..path2 (with two periods, and path instead 21019 of conflict.) Bisync now defaults to a single dot instead of a double 21020 dot, but additional dots can be added by including them in the specified 21021 suffix string. For example, for behavior equivalent to the previous 21022 default, use: 21023 21024 [--conflict-resolve none] --conflict-loser pathname --conflict-suffix .path 21025 21026 --check-sync 21027 21028 Enabled by default, the check-sync function checks that all of the same 21029 files exist in both the Path1 and Path2 history listings. This 21030 check-sync integrity check is performed at the end of the sync run by 21031 default. Any untrapped failing copy/deletes between the two paths might 21032 result in differences between the two listings and in the untracked file 21033 content differences between the two paths. A resync run would correct 21034 the error. 21035 21036 Note that the default-enabled integrity check locally executes a load of 21037 both the final Path1 and Path2 listings, and thus adds to the run time 21038 of a sync. Using --check-sync=false will disable it and may 21039 significantly reduce the sync run times for very large numbers of files. 21040 21041 The check may be run manually with --check-sync=only. It runs only the 21042 integrity check and terminates without actually synching. 21043 21044 Note that currently, --check-sync only checks listing snapshots and NOT 21045 the actual files on the remotes. Note also that the listing snapshots 21046 will not know about any changes that happened during or after the latest 21047 bisync run, as those will be discovered on the next run. Therefore, 21048 while listings should always match each other at the end of a bisync 21049 run, it is expected that they will not match the underlying remotes, nor 21050 will the remotes match each other, if there were changes during or after 21051 the run. This is normal, and any differences will be detected and synced 21052 on the next run. 21053 21054 For a robust integrity check of the current state of the remotes (as 21055 opposed to just their listing snapshots), consider using check (or 21056 cryptcheck, if at least one path is a crypt remote) instead of 21057 --check-sync, keeping in mind that differences are expected if files 21058 changed during or after your last bisync run. 21059 21060 For example, a possible sequence could look like this: 21061 21062 1. Normally scheduled bisync run: 21063 21064 rclone bisync Path1 Path2 -MPc --check-access --max-delete 10 --filters-file /path/to/filters.txt -v --no-cleanup --ignore-listing-checksum --disable ListR --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient 21065 21066 2. Periodic independent integrity check (perhaps scheduled nightly or 21067 weekly): 21068 21069 rclone check -MvPc Path1 Path2 --filter-from /path/to/filters.txt 21070 21071 3. If diffs are found, you have some choices to correct them. If one 21072 side is more up-to-date and you want to make the other side match 21073 it, you could run: 21074 21075 rclone sync Path1 Path2 --filter-from /path/to/filters.txt --create-empty-src-dirs -MPc -v 21076 21077 (or switch Path1 and Path2 to make Path2 the source-of-truth) 21078 21079 Or, if neither side is totally up-to-date, you could run a --resync to 21080 bring them back into agreement (but remember that this could cause 21081 deleted files to re-appear.) 21082 21083 *Note also that rclone check does not currently include empty 21084 directories, so if you want to know if any empty directories are out of 21085 sync, consider alternatively running the above rclone sync command with 21086 --dry-run added. 21087 21088 See also: Concurrent modifications, --resilient 21089 21090 --resilient 21091 21092 Caution: this is an experimental feature. Use at your own risk! 21093 21094 By default, most errors or interruptions will cause bisync to abort and 21095 require --resync to recover. This is a safety feature, to prevent bisync 21096 from running again until a user checks things out. However, in some 21097 cases, bisync can go too far and enforce a lockout when one isn't 21098 actually necessary, like for certain less-serious errors that might 21099 resolve themselves on the next run. When --resilient is specified, 21100 bisync tries its best to recover and self-correct, and only requires 21101 --resync as a last resort when a human's involvement is absolutely 21102 necessary. The intended use case is for running bisync as a background 21103 process (such as via scheduled cron). 21104 21105 When using --resilient mode, bisync will still report the error and 21106 abort, however it will not lock out future runs -- allowing the 21107 possibility of retrying at the next normally scheduled time, without 21108 requiring a --resync first. Examples of such retryable errors include 21109 access test failures, missing listing files, and filter change 21110 detections. These safety features will still prevent the current run 21111 from proceeding -- the difference is that if conditions have improved by 21112 the time of the next run, that next run will be allowed to proceed. 21113 Certain more serious errors will still enforce a --resync lockout, even 21114 in --resilient mode, to prevent data loss. 21115 21116 Behavior of --resilient may change in a future version. (See also: 21117 --recover, --max-lock, Graceful Shutdown) 21118 21119 --recover 21120 21121 If --recover is set, in the event of a sudden interruption or other 21122 un-graceful shutdown, bisync will attempt to automatically recover on 21123 the next run, instead of requiring --resync. Bisync is able to recover 21124 robustly by keeping one "backup" listing at all times, representing the 21125 state of both paths after the last known successful sync. Bisync can 21126 then compare the current state with this snapshot to determine which 21127 changes it needs to retry. Changes that were synced after this snapshot 21128 (during the run that was later interrupted) will appear to bisync as if 21129 they are "new or changed on both sides", but in most cases this is not a 21130 problem, as bisync will simply do its usual "equality check" and learn 21131 that no action needs to be taken on these files, since they are already 21132 identical on both sides. 21133 21134 In the rare event that a file is synced successfully during a run that 21135 later aborts, and then that same file changes AGAIN before the next run, 21136 bisync will think it is a sync conflict, and handle it accordingly. 21137 (From bisync's perspective, the file has changed on both sides since the 21138 last trusted sync, and the files on either side are not currently 21139 identical.) Therefore, --recover carries with it a slightly increased 21140 chance of having conflicts -- though in practice this is pretty rare, as 21141 the conditions required to cause it are quite specific. This risk can be 21142 reduced by using bisync's "Graceful Shutdown" mode (triggered by sending 21143 SIGINT or Ctrl+C), when you have the choice, instead of forcing a sudden 21144 termination. 21145 21146 --recover and --resilient are similar, but distinct -- the main 21147 difference is that --resilient is about retrying, while --recover is 21148 about recovering. Most users will probably want both. --resilient allows 21149 retrying when bisync has chosen to abort itself due to safety features 21150 such as failing --check-access or detecting a filter change. --resilient 21151 does not cover external interruptions such as a user shutting down their 21152 computer in the middle of a sync -- that is what --recover is for. 21153 21154 --max-lock 21155 21156 Bisync uses lock files as a safety feature to prevent interference from 21157 other bisync runs while it is running. Bisync normally removes these 21158 lock files at the end of a run, but if bisync is abruptly interrupted, 21159 these files will be left behind. By default, they will lock out all 21160 future runs, until the user has a chance to manually check things out 21161 and remove the lock. As an alternative, --max-lock can be used to make 21162 them automatically expire after a certain period of time, so that future 21163 runs are not locked out forever, and auto-recovery is possible. 21164 --max-lock can be any duration 2m or greater (or 0 to disable). If set, 21165 lock files older than this will be considered "expired", and future runs 21166 will be allowed to disregard them and proceed. (Note that the --max-lock 21167 duration must be set by the process that left the lock file -- not the 21168 later one interpreting it.) 21169 21170 If set, bisync will also "renew" these lock files every 21171 --max-lock minus one minute throughout a run, for extra safety. (For 21172 example, with --max-lock 5m, bisync would renew the lock file (for 21173 another 5 minutes) every 4 minutes until the run has completed.) In 21174 other words, it should not be possible for a lock file to pass its 21175 expiration time while the process that created it is still running -- 21176 and you can therefore be reasonably sure that any expired lock file you 21177 may find was left there by an interrupted run, not one that is still 21178 running and just taking awhile. 21179 21180 If --max-lock is 0 or not set, the default is that lock files will never 21181 expire, and will block future runs (of these same two bisync paths) 21182 indefinitely. 21183 21184 For maximum resilience from disruptions, consider setting a relatively 21185 short duration like --max-lock 2m along with --resilient and --recover, 21186 and a relatively frequent cron schedule. The result will be a very 21187 robust "set-it-and-forget-it" bisync run that can automatically bounce 21188 back from almost any interruption it might encounter, without requiring 21189 the user to get involved and run a --resync. (See also: Graceful 21190 Shutdown mode) 21191 21192 --backup-dir1 and --backup-dir2 21193 21194 As of v1.66, --backup-dir is supported in bisync. Because --backup-dir 21195 must be a non-overlapping path on the same remote, Bisync has introduced 21196 new --backup-dir1 and --backup-dir2 flags to support separate 21197 backup-dirs for Path1 and Path2 (bisyncing between different remotes 21198 with --backup-dir would not otherwise be possible.) --backup-dir1 and 21199 --backup-dir2 can use different remotes from each other, but 21200 --backup-dir1 must use the same remote as Path1, and --backup-dir2 must 21201 use the same remote as Path2. Each backup directory must not overlap its 21202 respective bisync Path without being excluded by a filter rule. 21203 21204 The standard --backup-dir will also work, if both paths use the same 21205 remote (but note that deleted files from both paths would be mixed 21206 together in the same dir). If either --backup-dir1 and --backup-dir2 are 21207 set, they will override --backup-dir. 21208 21209 Example: 21210 21211 rclone bisync /Users/someuser/some/local/path/Bisync gdrive:Bisync --backup-dir1 /Users/someuser/some/local/path/BackupDir --backup-dir2 gdrive:BackupDir --suffix -2023-08-26 --suffix-keep-extension --check-access --max-delete 10 --filters-file /Users/someuser/some/local/path/bisync_filters.txt --no-cleanup --ignore-listing-checksum --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient -MvP --drive-skip-gdocs --fix-case 21212 21213 In this example, if the user deletes a file in 21214 /Users/someuser/some/local/path/Bisync, bisync will propagate the delete 21215 to the other side by moving the corresponding file from gdrive:Bisync to 21216 gdrive:BackupDir. If the user deletes a file from gdrive:Bisync, bisync 21217 moves it from /Users/someuser/some/local/path/Bisync to 21218 /Users/someuser/some/local/path/BackupDir. 21219 21220 In the event of a rename due to a sync conflict, the rename is not 21221 considered a delete, unless a previous conflict with the same name 21222 already exists and would get overwritten. 21223 21224 See also: --suffix, --suffix-keep-extension 21225 21226 Operation 21227 21228 Runtime flow details 21229 21230 bisync retains the listings of the Path1 and Path2 filesystems from the 21231 prior run. On each successive run it will: 21232 21233 - list files on path1 and path2, and check for changes on each side. 21234 Changes include New, Newer, Older, and Deleted files. 21235 - Propagate changes on path1 to path2, and vice-versa. 21236 21237 Safety measures 21238 21239 - Lock file prevents multiple simultaneous runs when taking a while. 21240 This can be particularly useful if bisync is run by cron scheduler. 21241 - Handle change conflicts non-destructively by creating .conflict1, 21242 .conflict2, etc. file versions, according to --conflict-resolve, 21243 --conflict-loser, and --conflict-suffix settings. 21244 - File system access health check using RCLONE_TEST files (see the 21245 --check-access flag). 21246 - Abort on excessive deletes - protects against a failed listing being 21247 interpreted as all the files were deleted. See the --max-delete and 21248 --force flags. 21249 - If something evil happens, bisync goes into a safe state to block 21250 damage by later runs. (See Error Handling) 21251 21252 Normal sync checks 21253 21254 ------------------------------------------------------------------------ 21255 Type Description Result Implementation 21256 --------- ---------------------------- --------------- ----------------- 21257 Path2 new File is new on Path2, does Path2 version rclone copy Path2 21258 not exist on Path1 survives to Path1 21259 21260 Path2 File is newer on Path2, Path2 version rclone copy Path2 21261 newer unchanged on Path1 survives to Path1 21262 21263 Path2 File is deleted on Path2, File is deleted rclone delete 21264 deleted unchanged on Path1 Path1 21265 21266 Path1 new File is new on Path1, does Path1 version rclone copy Path1 21267 not exist on Path2 survives to Path2 21268 21269 Path1 File is newer on Path1, Path1 version rclone copy Path1 21270 newer unchanged on Path2 survives to Path2 21271 21272 Path1 File is older on Path1, Path1 version rclone copy Path1 21273 older unchanged on Path2 survives to Path2 21274 21275 Path2 File is older on Path2, Path2 version rclone copy Path2 21276 older unchanged on Path1 survives to Path1 21277 21278 Path1 File no longer exists on File is deleted rclone delete 21279 deleted Path1 Path2 21280 ------------------------------------------------------------------------ 21281 21282 Unusual sync checks 21283 21284 ------------------------------------------------------------------------------ 21285 Type Description Result Implementation 21286 ----------------- --------------------- -------------------- ----------------- 21287 Path1 new/changed File is new/changed No change None 21288 AND Path2 on Path1 AND 21289 new/changed AND new/changed on Path2 21290 Path1 == Path2 AND Path1 version is 21291 currently identical 21292 to Path2 21293 21294 Path1 new AND File is new on Path1 Conflicts handled default: 21295 Path2 new AND new on Path2 (and according to rclone copy 21296 Path1 version is NOT --conflict-resolve & renamed 21297 identical to Path2) --conflict-loser Path2.conflict2 21298 settings file to Path1, 21299 rclone copy 21300 renamed 21301 Path1.conflict1 21302 file to Path2 21303 21304 Path2 newer AND File is newer on Conflicts handled default: 21305 Path1 changed Path2 AND also according to rclone copy 21306 changed --conflict-resolve & renamed 21307 (newer/older/size) on --conflict-loser Path2.conflict2 21308 Path1 (and Path1 settings file to Path1, 21309 version is NOT rclone copy 21310 identical to Path2) renamed 21311 Path1.conflict1 21312 file to Path2 21313 21314 Path2 newer AND File is newer on Path2 version rclone copy Path2 21315 Path1 deleted Path2 AND also survives to Path1 21316 deleted on Path1 21317 21318 Path2 deleted AND File is deleted on Path1 version rclone copy Path1 21319 Path1 changed Path2 AND changed survives to Path2 21320 (newer/older/size) on 21321 Path1 21322 21323 Path1 deleted AND File is deleted on Path2 version rclone copy Path2 21324 Path2 changed Path1 AND changed survives to Path1 21325 (newer/older/size) on 21326 Path2 21327 ------------------------------------------------------------------------------ 21328 21329 As of rclone v1.64, bisync is now better at detecting false positive 21330 sync conflicts, which would previously have resulted in unnecessary 21331 renames and duplicates. Now, when bisync comes to a file that it wants 21332 to rename (because it is new/changed on both sides), it first checks 21333 whether the Path1 and Path2 versions are currently identical (using the 21334 same underlying function as check.) If bisync concludes that the files 21335 are identical, it will skip them and move on. Otherwise, it will create 21336 renamed duplicates, as before. This behavior also improves the 21337 experience of renaming directories, as a --resync is no longer required, 21338 so long as the same change has been made on both sides. 21339 21340 All files changed check 21341 21342 If all prior existing files on either of the filesystems have changed 21343 (e.g. timestamps have changed due to changing the system's timezone) 21344 then bisync will abort without making any changes. Any new files are not 21345 considered for this check. You could use --force to force the sync 21346 (whichever side has the changed timestamp files wins). Alternately, a 21347 --resync may be used (Path1 versions will be pushed to Path2). Consider 21348 the situation carefully and perhaps use --dry-run before you commit to 21349 the changes. 21350 21351 Modification times 21352 21353 By default, bisync compares files by modification time and size. If you 21354 or your application should change the content of a file without changing 21355 the modification time and size, then bisync will not notice the change, 21356 and thus will not copy it to the other side. As an alternative, consider 21357 comparing by checksum (if your remotes support it). See --compare for 21358 details. 21359 21360 Error handling 21361 21362 Certain bisync critical errors, such as file copy/move failing, will 21363 result in a bisync lockout of following runs. The lockout is asserted 21364 because the sync status and history of the Path1 and Path2 filesystems 21365 cannot be trusted, so it is safer to block any further changes until 21366 someone checks things out. The recovery is to do a --resync again. 21367 21368 It is recommended to use --resync --dry-run --verbose initially and 21369 carefully review what changes will be made before running the --resync 21370 without --dry-run. 21371 21372 Most of these events come up due to an error status from an internal 21373 call. On such a critical error the {...}.path1.lst and {...}.path2.lst 21374 listing files are renamed to extension .lst-err, which blocks any future 21375 bisync runs (since the normal .lst files are not found). Bisync keeps 21376 them under bisync subdirectory of the rclone cache directory, typically 21377 at ${HOME}/.cache/rclone/bisync/ on Linux. 21378 21379 Some errors are considered temporary and re-running the bisync is not 21380 blocked. The critical return blocks further bisync runs. 21381 21382 See also: --resilient, --recover, --max-lock, Graceful Shutdown 21383 21384 Lock file 21385 21386 When bisync is running, a lock file is created in the bisync working 21387 directory, typically at ~/.cache/rclone/bisync/PATH1..PATH2.lck on 21388 Linux. If bisync should crash or hang, the lock file will remain in 21389 place and block any further runs of bisync for the same paths. Delete 21390 the lock file as part of debugging the situation. The lock file 21391 effectively blocks follow-on (e.g., scheduled by cron) runs when the 21392 prior invocation is taking a long time. The lock file contains PID of 21393 the blocking process, which may help in debug. Lock files can be set to 21394 automatically expire after a certain amount of time, using the 21395 --max-lock flag. 21396 21397 Note that while concurrent bisync runs are allowed, be very cautious 21398 that there is no overlap in the trees being synched between concurrent 21399 runs, lest there be replicated files, deleted files and general mayhem. 21400 21401 Return codes 21402 21403 rclone bisync returns the following codes to calling program: - 0 on a 21404 successful run, - 1 for a non-critical failing run (a rerun may be 21405 successful), - 2 for a critically aborted run (requires a --resync to 21406 recover). 21407 21408 Graceful Shutdown 21409 21410 Bisync has a "Graceful Shutdown" mode which is activated by sending 21411 SIGINT or pressing Ctrl+C during a run. Once triggered, bisync will use 21412 best efforts to exit cleanly before the timer runs out. If bisync is in 21413 the middle of transferring files, it will attempt to cleanly empty its 21414 queue by finishing what it has started but not taking more. If it cannot 21415 do so within 30 seconds, it will cancel the in-progress transfers at 21416 that point and then give itself a maximum of 60 seconds to wrap up, save 21417 its state for next time, and exit. With the -vP flags you will see 21418 constant status updates and a final confirmation of whether or not the 21419 graceful shutdown was successful. 21420 21421 At any point during the "Graceful Shutdown" sequence, a second SIGINT or 21422 Ctrl+C will trigger an immediate, un-graceful exit, which will leave 21423 things in a messier state. Usually a robust recovery will still be 21424 possible if using --recover mode, otherwise you will need to do a 21425 --resync. 21426 21427 If you plan to use Graceful Shutdown mode, it is recommended to use 21428 --resilient and --recover, and it is important to NOT use --inplace, 21429 otherwise you risk leaving partially-written files on one side, which 21430 may be confused for real files on the next run. Note also that in the 21431 event of an abrupt interruption, a lock file will be left behind to 21432 block concurrent runs. You will need to delete it before you can proceed 21433 with the next run (or wait for it to expire on its own, if using 21434 --max-lock.) 21435 21436 Limitations 21437 21438 Supported backends 21439 21440 Bisync is considered BETA and has been tested with the following 21441 backends: - Local filesystem - Google Drive - Dropbox - OneDrive - S3 - 21442 SFTP - Yandex Disk - Crypt 21443 21444 It has not been fully tested with other services yet. If it works, or 21445 sorta works, please let us know and we'll update the list. Run the test 21446 suite to check for proper operation as described below. 21447 21448 The first release of rclone bisync required both underlying backends to 21449 support modification times, and refused to run otherwise. This 21450 limitation has been lifted as of v1.66, as bisync now supports comparing 21451 checksum and/or size instead of (or in addition to) modtime. See 21452 --compare for details. 21453 21454 Concurrent modifications 21455 21456 When using Local, FTP or SFTP remotes with --inplace, rclone does not 21457 create temporary files at the destination when copying, and thus if the 21458 connection is lost the created file may be corrupt, which will likely 21459 propagate back to the original path on the next sync, resulting in data 21460 loss. It is therefore recommended to omit --inplace. 21461 21462 Files that change during a bisync run may result in data loss. Prior to 21463 rclone v1.66, this was commonly seen in highly dynamic environments, 21464 where the filesystem was getting hammered by running processes during 21465 the sync. As of rclone v1.66, bisync was redesigned to use a "snapshot" 21466 model, greatly reducing the risks from changes during a sync. Changes 21467 that are not detected during the current sync will now be detected 21468 during the following sync, and will no longer cause the entire run to 21469 throw a critical error. There is additionally a mechanism to mark files 21470 as needing to be internally rechecked next time, for added safety. It 21471 should therefore no longer be necessary to sync only at quiet times -- 21472 however, note that an error can still occur if a file happens to change 21473 at the exact moment it's being read/written by bisync (same as would 21474 happen in rclone sync.) (See also: --ignore-checksum, 21475 --local-no-check-updated) 21476 21477 Empty directories 21478 21479 By default, new/deleted empty directories on one path are not propagated 21480 to the other side. This is because bisync (and rclone) natively works on 21481 files, not directories. However, this can be changed with the 21482 --create-empty-src-dirs flag, which works in much the same way as in 21483 sync and copy. When used, empty directories created or deleted on one 21484 side will also be created or deleted on the other side. The following 21485 should be noted: * --create-empty-src-dirs is not compatible with 21486 --remove-empty-dirs. Use only one or the other (or neither). * It is not 21487 recommended to switch back and forth between --create-empty-src-dirs and 21488 the default (no --create-empty-src-dirs) without running --resync. This 21489 is because it may appear as though all directories (not just the empty 21490 ones) were created/deleted, when actually you've just toggled between 21491 making them visible/invisible to bisync. It looks scarier than it is, 21492 but it's still probably best to stick to one or the other, and use 21493 --resync when you need to switch. 21494 21495 Renamed directories 21496 21497 By default, renaming a folder on the Path1 side results in deleting all 21498 files on the Path2 side and then copying all files again from Path1 to 21499 Path2. Bisync sees this as all files in the old directory name as 21500 deleted and all files in the new directory name as new. 21501 21502 A recommended solution is to use --track-renames, which is now supported 21503 in bisync as of rclone v1.66. Note that --track-renames is not available 21504 during --resync, as --resync does not delete anything (--track-renames 21505 only supports sync, not copy.) 21506 21507 Otherwise, the most effective and efficient method of renaming a 21508 directory is to rename it to the same name on both sides. (As of 21509 rclone v1.64, a --resync is no longer required after doing so, as bisync 21510 will automatically detect that Path1 and Path2 are in agreement.) 21511 21512 --fast-list used by default 21513 21514 Unlike most other rclone commands, bisync uses --fast-list by default, 21515 for backends that support it. In many cases this is desirable, however, 21516 there are some scenarios in which bisync could be faster without 21517 --fast-list, and there is also a known issue concerning Google Drive 21518 users with many empty directories. For now, the recommended way to avoid 21519 using --fast-list is to add --disable ListR to all bisync commands. The 21520 default behavior may change in a future version. 21521 21522 Case (and unicode) sensitivity 21523 21524 As of v1.66, case and unicode form differences no longer cause critical 21525 errors, and normalization (when comparing between filesystems) is 21526 handled according to the same flags and defaults as rclone sync. See the 21527 following options (all of which are supported by bisync) to control this 21528 behavior more granularly: - --fix-case - --ignore-case-sync - 21529 --no-unicode-normalization - --local-unicode-normalization and 21530 --local-case-sensitive (caution: these are normally not what you want.) 21531 21532 Note that in the (probably rare) event that --fix-case is used AND a 21533 file is new/changed on both sides AND the checksums match AND the 21534 filename case does not match, the Path1 filename is considered the 21535 winner, for the purposes of --fix-case (Path2 will be renamed to match 21536 it). 21537 21538 Windows support 21539 21540 Bisync has been tested on Windows 8.1, Windows 10 Pro 64-bit and on 21541 Windows GitHub runners. 21542 21543 Drive letters are allowed, including drive letters mapped to network 21544 drives (rclone bisync J:\localsync GDrive:). If a drive letter is 21545 omitted, the shell current drive is the default. Drive letters are a 21546 single character follows by :, so cloud names must be more than one 21547 character long. 21548 21549 Absolute paths (with or without a drive letter), and relative paths 21550 (with or without a drive letter) are supported. 21551 21552 Working directory is created at 21553 C:\Users\MyLogin\AppData\Local\rclone\bisync. 21554 21555 Note that bisync output may show a mix of forward / and back \ slashes. 21556 21557 Be careful of case independent directory and file naming on Windows vs. 21558 case dependent Linux 21559 21560 Filtering 21561 21562 See filtering documentation for how filter rules are written and 21563 interpreted. 21564 21565 Bisync's --filters-file flag slightly extends the rclone's --filter-from 21566 filtering mechanism. For a given bisync run you may provide only one 21567 --filters-file. The --include*, --exclude*, and --filter flags are also 21568 supported. 21569 21570 How to filter directories 21571 21572 Filtering portions of the directory tree is a critical feature for 21573 synching. 21574 21575 Examples of directory trees (always beneath the Path1/Path2 root level) 21576 you may want to exclude from your sync: - Directory trees containing 21577 only software build intermediate files. - Directory trees containing 21578 application temporary files and data such as the Windows 21579 C:\Users\MyLogin\AppData\ tree. - Directory trees containing files that 21580 are large, less important, or are getting thrashed continuously by 21581 ongoing processes. 21582 21583 On the other hand, there may be only select directories that you 21584 actually want to sync, and exclude all others. See the Example 21585 include-style filters for Windows user directories below. 21586 21587 Filters file writing guidelines 21588 21589 1. Begin with excluding directory trees: 21590 - e.g. `- /AppData/` 21591 - ** on the end is not necessary. Once a given directory level is 21592 excluded then everything beneath it won't be looked at by 21593 rclone. 21594 - Exclude such directories that are unneeded, are big, dynamically 21595 thrashed, or where there may be access permission issues. 21596 - Excluding such dirs first will make rclone operations (much) 21597 faster. 21598 - Specific files may also be excluded, as with the Dropbox 21599 exclusions example below. 21600 2. Decide if it's easier (or cleaner) to: 21601 - Include select directories and therefore exclude everything else 21602 -- or -- 21603 - Exclude select directories and therefore include everything else 21604 3. Include select directories: 21605 - Add lines like: `+ /Documents/PersonalFiles/**` to select which 21606 directories to include in the sync. 21607 - ** on the end specifies to include the full depth of the 21608 specified tree. 21609 - With Include-style filters, files at the Path1/Path2 root are 21610 not included. They may be included with `+ /*`. 21611 - Place RCLONE_TEST files within these included directory trees. 21612 They will only be looked for in these directory trees. 21613 - Finish by excluding everything else by adding `- **` at the end 21614 of the filters file. 21615 - Disregard step 4. 21616 4. Exclude select directories: 21617 - Add more lines like in step 1. For example: 21618 -/Desktop/tempfiles/, or `- /testdir/. Again, a**` on the end 21619 is not necessary. 21620 - Do not add a `- **` in the file. Without this line, everything 21621 will be included that has not been explicitly excluded. 21622 - Disregard step 3. 21623 21624 A few rules for the syntax of a filter file expanding on filtering 21625 documentation: 21626 21627 - Lines may start with spaces and tabs - rclone strips leading 21628 whitespace. 21629 - If the first non-whitespace character is a # then the line is a 21630 comment and will be ignored. 21631 - Blank lines are ignored. 21632 - The first non-whitespace character on a filter line must be a + or 21633 -. 21634 - Exactly 1 space is allowed between the +/- and the path term. 21635 - Only forward slashes (/) are used in path terms, even on Windows. 21636 - The rest of the line is taken as the path term. Trailing whitespace 21637 is taken literally, and probably is an error. 21638 21639 Example include-style filters for Windows user directories 21640 21641 This Windows include-style example is based on the sync root (Path1) set 21642 to C:\Users\MyLogin. The strategy is to select specific directories to 21643 be synched with a network drive (Path2). 21644 21645 - `- /AppData/` excludes an entire tree of Windows stored stuff that 21646 need not be synched. In my case, AppData has >11 GB of stuff I don't 21647 care about, and there are some subdirectories beneath AppData that 21648 are not accessible to my user login, resulting in bisync critical 21649 aborts. 21650 - Windows creates cache files starting with both upper and lowercase 21651 NTUSER at C:\Users\MyLogin. These files may be dynamic, locked, and 21652 are generally don't care. 21653 - There are just a few directories with my data that I do want 21654 synched, in the form of `+ 21655 /. By selecting only the directory trees I want to avoid the dozen plus directories that various apps make atC:`. 21656 - Include files in the root of the sync point, C:\Users\MyLogin, by 21657 adding the `+ /*` line. 21658 - This is an Include-style filters file, therefore it ends with `- **` 21659 which excludes everything not explicitly included. 21660 21661 - /AppData/ 21662 - NTUSER* 21663 - ntuser* 21664 + /Documents/Family/** 21665 + /Documents/Sketchup/** 21666 + /Documents/Microcapture_Photo/** 21667 + /Documents/Microcapture_Video/** 21668 + /Desktop/** 21669 + /Pictures/** 21670 + /* 21671 - ** 21672 21673 Note also that Windows implements several "library" links such as 21674 C:\Users\MyLogin\My Documents\My Music pointing to 21675 C:\Users\MyLogin\Music. rclone sees these as links, so you must add 21676 --links to the bisync command line if you which to follow these links. I 21677 find that I get permission errors in trying to follow the links, so I 21678 don't include the rclone --links flag, but then you get lots of 21679 Can't follow symlink… noise from rclone about not following the links. 21680 This noise can be quashed by adding --quiet to the bisync command line. 21681 21682 Example exclude-style filters files for use with Dropbox 21683 21684 - Dropbox disallows synching the listed temporary and 21685 configuration/data files. The `- ` filters exclude these files where 21686 ever they may occur in the sync tree. Consider adding similar 21687 exclusions for file types you don't need to sync, such as core dump 21688 and software build files. 21689 - bisync testing creates /testdir/ at the top level of the sync tree, 21690 and usually deletes the tree after the test. If a normal sync should 21691 run while the /testdir/ tree exists the --check-access phase may 21692 fail due to unbalanced RCLONE_TEST files. The `- /testdir/` filter 21693 blocks this tree from being synched. You don't need this exclusion 21694 if you are not doing bisync development testing. 21695 - Everything else beneath the Path1/Path2 root will be synched. 21696 - RCLONE_TEST files may be placed anywhere within the tree, including 21697 the root. 21698 21699 Example filters file for Dropbox 21700 21701 # Filter file for use with bisync 21702 # See https://rclone.org/filtering/ for filtering rules 21703 # NOTICE: If you make changes to this file you MUST do a --resync run. 21704 # Run with --dry-run to see what changes will be made. 21705 21706 # Dropbox won't sync some files so filter them away here. 21707 # See https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing 21708 - .dropbox.attr 21709 - ~*.tmp 21710 - ~$* 21711 - .~* 21712 - desktop.ini 21713 - .dropbox 21714 21715 # Used for bisync testing, so excluded from normal runs 21716 - /testdir/ 21717 21718 # Other example filters 21719 #- /TiBU/ 21720 #- /Photos/ 21721 21722 How --check-access handles filters 21723 21724 At the start of a bisync run, listings are gathered for Path1 and Path2 21725 while using the user's --filters-file. During the check access phase, 21726 bisync scans these listings for RCLONE_TEST files. Any RCLONE_TEST files 21727 hidden by the --filters-file are not in the listings and thus not 21728 checked during the check access phase. 21729 21730 Troubleshooting 21731 21732 Reading bisync logs 21733 21734 Here are two normal runs. The first one has a newer file on the remote. 21735 The second has no deltas between local and remote. 21736 21737 2021/05/16 00:24:38 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/" 21738 2021/05/16 00:24:38 INFO : Path1 checking for diffs 21739 2021/05/16 00:24:38 INFO : - Path1 File is new - file.txt 21740 2021/05/16 00:24:38 INFO : Path1: 1 changes: 1 new, 0 newer, 0 older, 0 deleted 21741 2021/05/16 00:24:38 INFO : Path2 checking for diffs 21742 2021/05/16 00:24:38 INFO : Applying changes 21743 2021/05/16 00:24:38 INFO : - Path1 Queue copy to Path2 - dropbox:/file.txt 21744 2021/05/16 00:24:38 INFO : - Path1 Do queued copies to - Path2 21745 2021/05/16 00:24:38 INFO : Updating listings 21746 2021/05/16 00:24:38 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/" 21747 2021/05/16 00:24:38 INFO : Bisync successful 21748 21749 2021/05/16 00:36:52 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/" 21750 2021/05/16 00:36:52 INFO : Path1 checking for diffs 21751 2021/05/16 00:36:52 INFO : Path2 checking for diffs 21752 2021/05/16 00:36:52 INFO : No changes found 21753 2021/05/16 00:36:52 INFO : Updating listings 21754 2021/05/16 00:36:52 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/" 21755 2021/05/16 00:36:52 INFO : Bisync successful 21756 21757 Dry run oddity 21758 21759 The --dry-run messages may indicate that it would try to delete some 21760 files. For example, if a file is new on Path2 and does not exist on 21761 Path1 then it would normally be copied to Path1, but with --dry-run 21762 enabled those copies don't happen, which leads to the attempted delete 21763 on Path2, blocked again by --dry-run: ... Not deleting as --dry-run. 21764 21765 This whole confusing situation is an artifact of the --dry-run flag. 21766 Scrutinize the proposed deletes carefully, and if the files would have 21767 been copied to Path1 then the threatened deletes on Path2 may be 21768 disregarded. 21769 21770 Retries 21771 21772 Rclone has built-in retries. If you run with --verbose you'll see error 21773 and retry messages such as shown below. This is usually not a bug. If at 21774 the end of the run, you see Bisync successful and not 21775 Bisync critical error or Bisync aborted then the run was successful, and 21776 you can ignore the error messages. 21777 21778 The following run shows an intermittent fail. Lines 5 and _6- are 21779 low-level messages. Line 6 is a bubbled-up warning message, conveying 21780 the error. Rclone normally retries failing commands, so there may be 21781 numerous such messages in the log. 21782 21783 Since there are no final error/warning messages on line 7, rclone has 21784 recovered from failure after a retry, and the overall sync was 21785 successful. 21786 21787 1: 2021/05/14 00:44:12 INFO : Synching Path1 "/path/to/local/tree" with Path2 "dropbox:" 21788 2: 2021/05/14 00:44:12 INFO : Path1 checking for diffs 21789 3: 2021/05/14 00:44:12 INFO : Path2 checking for diffs 21790 4: 2021/05/14 00:44:12 INFO : Path2: 113 changes: 22 new, 0 newer, 0 older, 91 deleted 21791 5: 2021/05/14 00:44:12 ERROR : /path/to/local/tree/objects/af: error listing: unexpected end of JSON input 21792 6: 2021/05/14 00:44:12 NOTICE: WARNING listing try 1 failed. - dropbox: 21793 7: 2021/05/14 00:44:12 INFO : Bisync successful 21794 21795 This log shows a Critical failure which requires a --resync to recover 21796 from. See the Runtime Error Handling section. 21797 21798 2021/05/12 00:49:40 INFO : Google drive root '': Waiting for checks to finish 21799 2021/05/12 00:49:40 INFO : Google drive root '': Waiting for transfers to finish 21800 2021/05/12 00:49:40 INFO : Google drive root '': not deleting files as there were IO errors 21801 2021/05/12 00:49:40 ERROR : Attempt 3/3 failed with 3 errors and: not deleting files as there were IO errors 21802 2021/05/12 00:49:40 ERROR : Failed to sync: not deleting files as there were IO errors 21803 2021/05/12 00:49:40 NOTICE: WARNING rclone sync try 3 failed. - /path/to/local/tree/ 21804 2021/05/12 00:49:40 ERROR : Bisync aborted. Must run --resync to recover. 21805 21806 Denied downloads of "infected" or "abusive" files 21807 21808 Google Drive has a filter for certain file types (.exe, .apk, et cetera) 21809 that by default cannot be copied from Google Drive to the local 21810 filesystem. If you are having problems, run with --verbose to see 21811 specifically which files are generating complaints. If the error is 21812 This file has been identified as malware or spam and cannot be downloaded, 21813 consider using the flag --drive-acknowledge-abuse. 21814 21815 Google Docs (and other files of unknown size) 21816 21817 As of v1.66, Google Docs (including Google Sheets, Slides, etc.) are now 21818 supported in bisync, subject to the same options, defaults, and 21819 limitations as in rclone sync. When bisyncing drive with non-drive 21820 backends, the drive -> non-drive direction is controlled by 21821 --drive-export-formats (default "docx,xlsx,pptx,svg") and the non-drive 21822 -> drive direction is controlled by --drive-import-formats (default 21823 none.) 21824 21825 For example, with the default export/import formats, a Google Sheet on 21826 the drive side will be synced to an .xlsx file on the non-drive side. In 21827 the reverse direction, .xlsx files with filenames that match an existing 21828 Google Sheet will be synced to that Google Sheet, while .xlsx files that 21829 do NOT match an existing Google Sheet will be copied to drive as normal 21830 .xlsx files (without conversion to Sheets, although the Google Drive web 21831 browser UI may still give you the option to open it as one.) 21832 21833 If --drive-import-formats is set (it's not, by default), then all of the 21834 specified formats will be converted to Google Docs, if there is no 21835 existing Google Doc with a matching name. Caution: such conversion can 21836 be quite lossy, and in most cases it's probably not what you want! 21837 21838 To bisync Google Docs as URL shortcut links (in a manner similar to 21839 "Drive for Desktop"), use: --drive-export-formats url (or alternatives.) 21840 21841 Note that these link files cannot be edited on the non-drive side -- you 21842 will get errors if you try to sync an edited link file back to drive. 21843 They CAN be deleted (it will result in deleting the corresponding Google 21844 Doc.) If you create a .url file on the non-drive side that does not 21845 match an existing Google Doc, bisyncing it will just result in copying 21846 the literal .url file over to drive (no Google Doc will be created.) So, 21847 as a general rule of thumb, think of them as read-only placeholders on 21848 the non-drive side, and make all your changes on the drive side. 21849 21850 Likewise, even with other export-formats, it is best to only move/rename 21851 Google Docs on the drive side. This is because otherwise, bisync will 21852 interpret this as a file deleted and another created, and accordingly, 21853 it will delete the Google Doc and create a new file at the new path. 21854 (Whether or not that new file is a Google Doc depends on 21855 --drive-import-formats.) 21856 21857 Lastly, take note that all Google Docs on the drive side have a size of 21858 -1 and no checksum. Therefore, they cannot be reliably synced with the 21859 --checksum or --size-only flags. (To be exact: they will still get 21860 created/deleted, and bisync's delta engine will notice changes and queue 21861 them for syncing, but the underlying sync function will consider them 21862 identical and skip them.) To work around this, use the default (modtime 21863 and size) instead of --checksum or --size-only. 21864 21865 To ignore Google Docs entirely, use --drive-skip-gdocs. 21866 21867 Usage examples 21868 21869 Cron 21870 21871 Rclone does not yet have a built-in capability to monitor the local file 21872 system for changes and must be blindly run periodically. On Windows this 21873 can be done using a Task Scheduler, on Linux you can use Cron which is 21874 described below. 21875 21876 The 1st example runs a sync every 5 minutes between a local directory 21877 and an OwnCloud server, with output logged to a runlog file: 21878 21879 # Minute (0-59) 21880 # Hour (0-23) 21881 # Day of Month (1-31) 21882 # Month (1-12 or Jan-Dec) 21883 # Day of Week (0-6 or Sun-Sat) 21884 # Command 21885 */5 * * * * /path/to/rclone bisync /local/files MyCloud: --check-access --filters-file /path/to/bysync-filters.txt --log-file /path/to//bisync.log 21886 21887 See crontab syntax for the details of crontab time interval expressions. 21888 21889 If you run rclone bisync as a cron job, redirect stdout/stderr to a 21890 file. The 2nd example runs a sync to Dropbox every hour and logs all 21891 stdout (via the >>) and stderr (via 2>&1) to a log file. 21892 21893 0 * * * * /path/to/rclone bisync /path/to/local/dropbox Dropbox: --check-access --filters-file /home/user/filters.txt >> /path/to/logs/dropbox-run.log 2>&1 21894 21895 Sharing an encrypted folder tree between hosts 21896 21897 bisync can keep a local folder in sync with a cloud service, but what if 21898 you have some highly sensitive files to be synched? 21899 21900 Usage of a cloud service is for exchanging both routine and sensitive 21901 personal files between one's home network, one's personal notebook when 21902 on the road, and with one's work computer. The routine data is not 21903 sensitive. For the sensitive data, configure an rclone crypt remote to 21904 point to a subdirectory within the local disk tree that is bisync'd to 21905 Dropbox, and then set up an bisync for this local crypt directory to a 21906 directory outside of the main sync tree. 21907 21908 Linux server setup 21909 21910 - /path/to/DBoxroot is the root of my local sync tree. There are 21911 numerous subdirectories. 21912 - /path/to/DBoxroot/crypt is the root subdirectory for files that are 21913 encrypted. This local directory target is setup as an rclone crypt 21914 remote named Dropcrypt:. See rclone.conf snippet below. 21915 - /path/to/my/unencrypted/files is the root of my sensitive files - 21916 not encrypted, not within the tree synched to Dropbox. 21917 - To sync my local unencrypted files with the encrypted Dropbox 21918 versions I manually run 21919 bisync /path/to/my/unencrypted/files DropCrypt:. This step could be 21920 bundled into a script to run before and after the full Dropbox tree 21921 sync in the last step, thus actively keeping the sensitive files in 21922 sync. 21923 - bisync /path/to/DBoxroot Dropbox: runs periodically via cron, 21924 keeping my full local sync tree in sync with Dropbox. 21925 21926 Windows notebook setup 21927 21928 - The Dropbox client runs keeping the local tree 21929 C:\Users\MyLogin\Dropbox always in sync with Dropbox. I could have 21930 used rclone bisync instead. 21931 - A separate directory tree at C:\Users\MyLogin\Documents\DropLocal 21932 hosts the tree of unencrypted files/folders. 21933 - To sync my local unencrypted files with the encrypted Dropbox 21934 versions I manually run the following command: 21935 rclone bisync C:\Users\MyLogin\Documents\DropLocal Dropcrypt:. 21936 - The Dropbox client then syncs the changes with Dropbox. 21937 21938 rclone.conf snippet 21939 21940 [Dropbox] 21941 type = dropbox 21942 ... 21943 21944 [Dropcrypt] 21945 type = crypt 21946 remote = /path/to/DBoxroot/crypt # on the Linux server 21947 remote = C:\Users\MyLogin\Dropbox\crypt # on the Windows notebook 21948 filename_encryption = standard 21949 directory_name_encryption = true 21950 password = ... 21951 ... 21952 21953 Testing 21954 21955 You should read this section only if you are developing for rclone. You 21956 need to have rclone source code locally to work with bisync tests. 21957 21958 Bisync has a dedicated test framework implemented in the bisync_test.go 21959 file located in the rclone source tree. The test suite is based on the 21960 go test command. Series of tests are stored in subdirectories below the 21961 cmd/bisync/testdata directory. Individual tests can be invoked by their 21962 directory name, e.g. 21963 go test . -case basic -remote local -remote2 gdrive: -v 21964 21965 Tests will make a temporary folder on remote and purge it afterwards. If 21966 during test run there are intermittent errors and rclone retries, these 21967 errors will be captured and flagged as invalid MISCOMPAREs. Rerunning 21968 the test will let it pass. Consider such failures as noise. 21969 21970 Test command syntax 21971 21972 usage: go test ./cmd/bisync [options...] 21973 21974 Options: 21975 -case NAME Name(s) of the test case(s) to run. Multiple names should 21976 be separated by commas. You can remove the `test_` prefix 21977 and replace `_` by `-` in test name for convenience. 21978 If not `all`, the name(s) should map to a directory under 21979 `./cmd/bisync/testdata`. 21980 Use `all` to run all tests (default: all) 21981 -remote PATH1 `local` or name of cloud service with `:` (default: local) 21982 -remote2 PATH2 `local` or name of cloud service with `:` (default: local) 21983 -no-compare Disable comparing test results with the golden directory 21984 (default: compare) 21985 -no-cleanup Disable cleanup of Path1 and Path2 testdirs. 21986 Useful for troubleshooting. (default: cleanup) 21987 -golden Store results in the golden directory (default: false) 21988 This flag can be used with multiple tests. 21989 -debug Print debug messages 21990 -stop-at NUM Stop test after given step number. (default: run to the end) 21991 Implies `-no-compare` and `-no-cleanup`, if the test really 21992 ends prematurely. Only meaningful for a single test case. 21993 -refresh-times Force refreshing the target modtime, useful for Dropbox 21994 (default: false) 21995 -verbose Run tests verbosely 21996 21997 Note: unlike rclone flags which must be prefixed by double dash (--), 21998 the test command flags can be equally prefixed by a single - or double 21999 dash. 22000 22001 Running tests 22002 22003 - go test . -case basic -remote local -remote2 local runs the 22004 test_basic test case using only the local filesystem, synching one 22005 local directory with another local directory. Test script output is 22006 to the console, while commands within scenario.txt have their output 22007 sent to the .../workdir/test.log file, which is finally compared to 22008 the golden copy. 22009 - The first argument after go test should be a relative name of the 22010 directory containing bisync source code. If you run tests right from 22011 there, the argument will be . (current directory) as in most 22012 examples below. If you run bisync tests from the rclone source 22013 directory, the command should be go test ./cmd/bisync .... 22014 - The test engine will mangle rclone output to ensure comparability 22015 with golden listings and logs. 22016 - Test scenarios are located in ./cmd/bisync/testdata. The test -case 22017 argument should match the full name of a subdirectory under that 22018 directory. Every test subdirectory name on disk must start with 22019 test_, this prefix can be omitted on command line for brevity. Also, 22020 underscores in the name can be replaced by dashes for convenience. 22021 - go test . -remote local -remote2 local -case all runs all tests. 22022 - Path1 and Path2 may either be the keyword local or may be names of 22023 configured cloud services. 22024 go test . -remote gdrive: -remote2 dropbox: -case basic will run the 22025 test between these two services, without transferring any files to 22026 the local filesystem. 22027 - Test run stdout and stderr console output may be directed to a file, 22028 e.g. 22029 go test . -remote gdrive: -remote2 local -case all > runlog.txt 2>&1 22030 22031 Test execution flow 22032 22033 1. The base setup in the initial directory of the testcase is applied 22034 on the Path1 and Path2 filesystems (via rclone copy the initial 22035 directory to Path1, then rclone sync Path1 to Path2). 22036 2. The commands in the scenario.txt file are applied, with output 22037 directed to the test.log file in the test working directory. 22038 Typically, the first actual command in the scenario.txt file is to 22039 do a --resync, which establishes the baseline {...}.path1.lst and 22040 {...}.path2.lst files in the test working directory (.../workdir/ 22041 relative to the temporary test directory). Various commands and 22042 listing snapshots are done within the test. 22043 3. Finally, the contents of the test working directory are compared to 22044 the contents of the testcase's golden directory. 22045 22046 Notes about testing 22047 22048 - Test cases are in individual directories beneath 22049 ./cmd/bisync/testdata. A command line reference to a test is 22050 understood to reference a directory beneath testdata. For example, 22051 go test ./cmd/bisync -case dry-run -remote gdrive: -remote2 local 22052 refers to the test case in ./cmd/bisync/testdata/test_dry_run. 22053 - The test working directory is located at .../workdir relative to a 22054 temporary test directory, usually under /tmp on Linux. 22055 - The local test sync tree is created at a temporary directory named 22056 like bisync.XXX under system temporary directory. 22057 - The remote test sync tree is located at a temporary directory under 22058 <remote:>/bisync.XXX/. 22059 - path1 and/or path2 subdirectories are created in a temporary 22060 directory under the respective local or cloud test remote. 22061 - By default, the Path1 and Path2 test dirs and workdir will be 22062 deleted after each test run. The -no-cleanup flag disables purging 22063 these directories when validating and debugging a given test. These 22064 directories will be flushed before running another test, independent 22065 of the -no-cleanup usage. 22066 - You will likely want to add `- 22067 /testdir/to your normal bisync--filters-fileso that normal syncs do not attempt to sync the test temporary directories, which may haveRCLONE_TESTmiscompares in some testcases which would otherwise trip the--check-accesssystem. The--check-accessmechanism is hard-coded to ignoreRCLONE_TESTfiles beneathbisync/testdata`, 22068 so the test cases may reside on the synched tree even if there are 22069 check file mismatches in the test tree. 22070 - Some Dropbox tests can fail, notably printing the following message: 22071 src and dst identical but can't set mod time without deleting and re-uploading 22072 This is expected and happens due to the way Dropbox handles 22073 modification times. You should use the -refresh-times test flag to 22074 make up for this. 22075 - If Dropbox tests hit request limit for you and print error message 22076 too_many_requests/...: Too many requests or write operations. then 22077 follow the Dropbox App ID instructions. 22078 22079 Updating golden results 22080 22081 Sometimes even a slight change in the bisync source can cause little 22082 changes spread around many log files. Updating them manually would be a 22083 nightmare. 22084 22085 The -golden flag will store the test.log and *.lst listings from each 22086 test case into respective golden directories. Golden results will 22087 automatically contain generic strings instead of local or cloud paths 22088 which means that they should match when run with a different cloud 22089 service. 22090 22091 Your normal workflow might be as follows: 1. Git-clone the rclone 22092 sources locally 2. Modify bisync source and check that it builds 3. Run 22093 the whole test suite go test ./cmd/bisync -remote local 4. If some tests 22094 show log difference, recheck them individually, e.g.: 22095 go test ./cmd/bisync -remote local -case basic 5. If you are convinced 22096 with the difference, goldenize all tests at once: 22097 go test ./cmd/bisync -remote local -golden 6. Use word diff: 22098 git diff --word-diff ./cmd/bisync/testdata/. Please note that normal 22099 line-level diff is generally useless here. 7. Check the difference 22100 carefully! 8. Commit the change (git commit) only if you are sure. If 22101 unsure, save your code changes then wipe the log diffs from git: 22102 git reset [--hard]. 22103 22104 Structure of test scenarios 22105 22106 - <testname>/initial/ contains a tree of files that will be set as the 22107 initial condition on both Path1 and Path2 testdirs. 22108 - <testname>/modfiles/ contains files that will be used to modify the 22109 Path1 and/or Path2 filesystems. 22110 - <testname>/golden/ contains the expected content of the test working 22111 directory (workdir) at the completion of the testcase. 22112 - <testname>/scenario.txt contains the body of the test, in the form 22113 of various commands to modify files, run bisync, and snapshot 22114 listings. Output from these commands is captured to 22115 .../workdir/test.log for comparison to the golden files. 22116 22117 Supported test commands 22118 22119 - test <some message> Print the line to the console and to the 22120 test.log: test sync is working correctly with options x, y, z 22121 - copy-listings <prefix> Save a copy of all .lst listings in the test 22122 working directory with the specified prefix: 22123 save-listings exclude-pass-run 22124 - move-listings <prefix> Similar to copy-listings but removes the 22125 source 22126 - purge-children <dir> This will delete all child files and purge all 22127 child subdirs under given directory but keep the parent intact. This 22128 behavior is important for tests with Google Drive because removing 22129 and re-creating the parent would change its ID. 22130 - delete-file <file> Delete a single file. 22131 - delete-glob <dir> <pattern> Delete a group of files located one 22132 level deep in the given directory with names matching a given glob 22133 pattern. 22134 - touch-glob YYYY-MM-DD <dir> <pattern> Change modification time on a 22135 group of files. 22136 - touch-copy YYYY-MM-DD <source-file> <dest-dir> Change file 22137 modification time then copy it to destination. 22138 - copy-file <source-file> <dest-dir> Copy a single file to given 22139 directory. 22140 - copy-as <source-file> <dest-file> Similar to above but destination 22141 must include both directory and the new file name at destination. 22142 - copy-dir <src> <dst> and sync-dir <src> <dst> Copy/sync a directory. 22143 Equivalent of rclone copy and rclone sync. 22144 - list-dirs <dir> Equivalent to rclone lsf -R --dirs-only <dir> 22145 - bisync [options] Runs bisync against -remote and -remote2. 22146 22147 Supported substitution terms 22148 22149 - {testdir/} - the root dir of the testcase 22150 - {datadir/} - the modfiles dir under the testcase root 22151 - {workdir/} - the temporary test working directory 22152 - {path1/} - the root of the Path1 test directory tree 22153 - {path2/} - the root of the Path2 test directory tree 22154 - {session} - base name of the test listings 22155 - {/} - OS-specific path separator 22156 - {spc}, {tab}, {eol} - whitespace 22157 - {chr:HH} - raw byte with given hexadecimal code 22158 22159 Substitution results of the terms named like {dir/} will end with / (or 22160 backslash on Windows), so it is not necessary to include slash in the 22161 usage, for example delete-file {path1/}file1.txt. 22162 22163 Benchmarks 22164 22165 This section is work in progress. 22166 22167 Here are a few data points for scale, execution times, and memory usage. 22168 22169 The first set of data was taken between a local disk to Dropbox. The 22170 speedtest.net download speed was ~170 Mbps, and upload speed was ~10 22171 Mbps. 500 files (~9.5 MB each) had been already synched. 50 files were 22172 added in a new directory, each ~9.5 MB, ~475 MB total. 22173 22174 ------------------------------------------------------------------------ 22175 Change Operations and times Overall run 22176 time 22177 ------------------------ ----------------------------------- ----------- 22178 500 files synched 1x listings for Path1 & Path2 1.5 sec 22179 (nothing to move) 22180 22181 500 files synched with 1x listings for Path1 & Path2 1.5 sec 22182 --check-access 22183 22184 50 new files on remote Queued 50 copies down: 27 sec 29 sec 22185 22186 Moved local dir Queued 50 copies up: 410 sec, 50 421 sec 22187 deletes up: 9 sec 22188 22189 Moved remote dir Queued 50 copies down: 31 sec, 50 33 sec 22190 deletes down: <1 sec 22191 22192 Delete local dir Queued 50 deletes up: 9 sec 13 sec 22193 ------------------------------------------------------------------------ 22194 22195 This next data is from a user's application. They had ~400GB of data 22196 over 1.96 million files being sync'ed between a Windows local disk and 22197 some remote cloud. The file full path length was on average 35 22198 characters (which factors into load time and RAM required). 22199 22200 - Loading the prior listing into memory (1.96 million files, listing 22201 file size 140 MB) took ~30 sec and occupied about 1 GB of RAM. 22202 - Getting a fresh listing of the local file system (producing the 140 22203 MB output file) took about XXX sec. 22204 - Getting a fresh listing of the remote file system (producing the 140 22205 MB output file) took about XXX sec. The network download speed was 22206 measured at XXX Mb/s. 22207 - Once the prior and current Path1 and Path2 listings were loaded (a 22208 total of four to be loaded, two at a time), determining the deltas 22209 was pretty quick (a few seconds for this test case), and the 22210 transfer time for any files to be copied was dominated by the 22211 network bandwidth. 22212 22213 References 22214 22215 rclone's bisync implementation was derived from the rclonesync-V2 22216 project, including documentation and test mechanisms, with 22217 [@cjnaz](https://github.com/cjnaz)'s full support and encouragement. 22218 22219 rclone bisync is similar in nature to a range of other projects: 22220 22221 - unison 22222 - syncthing 22223 - cjnaz/rclonesync 22224 - ConorWilliams/rsinc 22225 - jwink3101/syncrclone 22226 - DavideRossi/upback 22227 22228 Bisync adopts the differential synchronization technique, which is based 22229 on keeping history of changes performed by both synchronizing sides. See 22230 the Dual Shadow Method section in Neil Fraser's article. 22231 22232 Also note a number of academic publications by Benjamin Pierce about 22233 Unison and synchronization in general. 22234 22235 Changelog 22236 22237 v1.66 22238 22239 - Copies and deletes are now handled in one operation instead of two 22240 - --track-renames and --backup-dir are now supported 22241 - Partial uploads known issue on local/ftp/sftp has been resolved 22242 (unless using --inplace) 22243 - Final listings are now generated from sync results, to avoid needing 22244 to re-list 22245 - Bisync is now much more resilient to changes that happen during a 22246 bisync run, and far less prone to critical errors / undetected 22247 changes 22248 - Bisync is now capable of rolling a file listing back in cases of 22249 uncertainty, essentially marking the file as needing to be rechecked 22250 next time. 22251 - A few basic terminal colors are now supported, controllable with 22252 --color (AUTO|NEVER|ALWAYS) 22253 - Initial listing snapshots of Path1 and Path2 are now generated 22254 concurrently, using the same "march" infrastructure as check and 22255 sync, for performance improvements and less risk of error. 22256 - Fixed handling of unicode normalization and case insensitivity, 22257 support for --fix-case, --ignore-case-sync, 22258 --no-unicode-normalization 22259 - --resync is now much more efficient (especially for users of 22260 --create-empty-src-dirs) 22261 - Google Docs (and other files of unknown size) are now supported 22262 (with the same options as in sync) 22263 - Equality checks before a sync conflict rename now fall back to 22264 cryptcheck (when possible) or --download, instead of of --size-only, 22265 when check is not available. 22266 - Bisync no longer fails to find the correct listing file when configs 22267 are overridden with backend-specific flags. 22268 - Bisync now fully supports comparing based on any combination of 22269 size, modtime, and checksum, lifting the prior restriction on 22270 backends without modtime support. 22271 - Bisync now supports a "Graceful Shutdown" mode to cleanly cancel a 22272 run early without requiring --resync. 22273 - New --recover flag allows robust recovery in the event of 22274 interruptions, without requiring --resync. 22275 - A new --max-lock setting allows lock files to automatically renew 22276 and expire, for better automatic recovery when a run is interrupted. 22277 - Bisync now supports auto-resolving sync conflicts and customizing 22278 rename behavior with new --conflict-resolve, --conflict-loser, and 22279 --conflict-suffix flags. 22280 - A new --resync-mode flag allows more control over which version of a 22281 file gets kept during a --resync. 22282 - Bisync now supports --retries and --retries-sleep (when --resilient 22283 is set.) 22284 22285 v1.64 22286 22287 - Fixed an issue causing dry runs to inadvertently commit filter 22288 changes 22289 - Fixed an issue causing --resync to erroneously delete empty folders 22290 and duplicate files unique to Path2 22291 - --check-access is now enforced during --resync, preventing data loss 22292 in certain user error scenarios 22293 - Fixed an issue causing bisync to consider more files than necessary 22294 due to overbroad filters during delete operations 22295 - Improved detection of false positive change conflicts (identical 22296 files are now left alone instead of renamed) 22297 - Added support for --create-empty-src-dirs 22298 - Added experimental --resilient mode to allow recovery from 22299 self-correctable errors 22300 - Added new --ignore-listing-checksum flag to distinguish from 22301 --ignore-checksum 22302 - Performance improvements for large remotes 22303 - Documentation and testing improvements 22304 22305 Release signing 22306 22307 The hashes of the binary artefacts of the rclone release are signed with 22308 a public PGP/GPG key. This can be verified manually as described below. 22309 22310 The same mechanism is also used by rclone selfupdate to verify that the 22311 release has not been tampered with before the new update is installed. 22312 This checks the SHA256 hash and the signature with a public key compiled 22313 into the rclone binary. 22314 22315 Release signing key 22316 22317 You may obtain the release signing key from: 22318 22319 - From KEYS on this website - this file contains all past signing keys 22320 also. 22321 - The git repository hosted on GitHub - 22322 https://github.com/rclone/rclone/blob/master/docs/content/KEYS 22323 - gpg --keyserver hkps://keys.openpgp.org --search nick@craig-wood.com 22324 - gpg --keyserver hkps://keyserver.ubuntu.com --search nick@craig-wood.com 22325 - https://www.craig-wood.com/nick/pub/pgp-key.txt 22326 22327 After importing the key, verify that the fingerprint of one of the keys 22328 matches: FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA as this key is used 22329 for signing. 22330 22331 We recommend that you cross-check the fingerprint shown above through 22332 the domains listed below. By cross-checking the integrity of the 22333 fingerprint across multiple domains you can be confident that you 22334 obtained the correct key. 22335 22336 - The source for this page on GitHub. 22337 - Through DNS dig key.rclone.org txt 22338 22339 If you find anything that doesn't not match, please contact the 22340 developers at once. 22341 22342 How to verify the release 22343 22344 In the release directory you will see the release files and some files 22345 called MD5SUMS, SHA1SUMS and SHA256SUMS. 22346 22347 $ rclone lsf --http-url https://downloads.rclone.org/v1.63.1 :http: 22348 MD5SUMS 22349 SHA1SUMS 22350 SHA256SUMS 22351 rclone-v1.63.1-freebsd-386.zip 22352 rclone-v1.63.1-freebsd-amd64.zip 22353 ... 22354 rclone-v1.63.1-windows-arm64.zip 22355 rclone-v1.63.1.tar.gz 22356 version.txt 22357 22358 The MD5SUMS, SHA1SUMS and SHA256SUMS contain hashes of the binary files 22359 in the release directory along with a signature. 22360 22361 For example: 22362 22363 $ rclone cat --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS 22364 -----BEGIN PGP SIGNED MESSAGE----- 22365 Hash: SHA1 22366 22367 f6d1b2d7477475ce681bdce8cb56f7870f174cb6b2a9ac5d7b3764296ea4a113 rclone-v1.63.1-freebsd-386.zip 22368 7266febec1f01a25d6575de51c44ddf749071a4950a6384e4164954dff7ac37e rclone-v1.63.1-freebsd-amd64.zip 22369 ... 22370 66ca083757fb22198309b73879831ed2b42309892394bf193ff95c75dff69c73 rclone-v1.63.1-windows-amd64.zip 22371 bbb47c16882b6c5f2e8c1b04229378e28f68734c613321ef0ea2263760f74cd0 rclone-v1.63.1-windows-arm64.zip 22372 -----BEGIN PGP SIGNATURE----- 22373 22374 iF0EARECAB0WIQT79zfs6firGGBL0qyTk14C/ztU+gUCZLVKJQAKCRCTk14C/ztU 22375 +pZuAJ0XJ+QWLP/3jCtkmgcgc4KAwd/rrwCcCRZQ7E+oye1FPY46HOVzCFU3L7g= 22376 =8qrL 22377 -----END PGP SIGNATURE----- 22378 22379 Download the files 22380 22381 The first step is to download the binary and SUMs file and verify that 22382 the SUMs you have downloaded match. Here we download 22383 rclone-v1.63.1-windows-amd64.zip - choose the binary (or binaries) 22384 appropriate to your architecture. We've also chosen the SHA256SUMS as 22385 these are the most secure. You could verify the other types of hash also 22386 for extra security. rclone selfupdate verifies just the SHA256SUMS. 22387 22388 $ mkdir /tmp/check 22389 $ cd /tmp/check 22390 $ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS . 22391 $ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:rclone-v1.63.1-windows-amd64.zip . 22392 22393 Verify the signatures 22394 22395 First verify the signatures on the SHA256 file. 22396 22397 Import the key. See above for ways to verify this key is correct. 22398 22399 $ gpg --keyserver keyserver.ubuntu.com --receive-keys FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA 22400 gpg: key 93935E02FF3B54FA: public key "Nick Craig-Wood <nick@craig-wood.com>" imported 22401 gpg: Total number processed: 1 22402 gpg: imported: 1 22403 22404 Then check the signature: 22405 22406 $ gpg --verify SHA256SUMS 22407 gpg: Signature made Mon 17 Jul 2023 15:03:17 BST 22408 gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA 22409 gpg: Good signature from "Nick Craig-Wood <nick@craig-wood.com>" [ultimate] 22410 22411 Verify the signature was good and is using the fingerprint shown above. 22412 22413 Repeat for MD5SUMS and SHA1SUMS if desired. 22414 22415 Verify the hashes 22416 22417 Now that we know the signatures on the hashes are OK we can verify the 22418 binaries match the hashes, completing the verification. 22419 22420 $ sha256sum -c SHA256SUMS 2>&1 | grep OK 22421 rclone-v1.63.1-windows-amd64.zip: OK 22422 22423 Or do the check with rclone 22424 22425 $ rclone hashsum sha256 -C SHA256SUMS rclone-v1.63.1-windows-amd64.zip 22426 2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 0 22427 2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 1 22428 2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 49 22429 2023/09/11 10:53:58 NOTICE: SHA256SUMS: 4 warning(s) suppressed... 22430 = rclone-v1.63.1-windows-amd64.zip 22431 2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 0 differences found 22432 2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 1 matching files 22433 22434 Verify signatures and hashes together 22435 22436 You can verify the signatures and hashes in one command line like this: 22437 22438 $ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing 22439 gpg: Signature made Mon 17 Jul 2023 15:03:17 BST 22440 gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA 22441 gpg: Good signature from "Nick Craig-Wood <nick@craig-wood.com>" [ultimate] 22442 gpg: aka "Nick Craig-Wood <nick@memset.com>" [unknown] 22443 rclone-v1.63.1-windows-amd64.zip: OK 22444 22445 1Fichier 22446 22447 This is a backend for the 1fichier cloud storage service. Note that a 22448 Premium subscription is required to use the API. 22449 22450 Paths are specified as remote:path 22451 22452 Paths may be as deep as required, e.g. remote:directory/subdirectory. 22453 22454 Configuration 22455 22456 The initial setup for 1Fichier involves getting the API key from the 22457 website which you need to do in your browser. 22458 22459 Here is an example of how to make a remote called remote. First run: 22460 22461 rclone config 22462 22463 This will guide you through an interactive setup process: 22464 22465 No remotes found, make a new one? 22466 n) New remote 22467 s) Set configuration password 22468 q) Quit config 22469 n/s/q> n 22470 name> remote 22471 Type of storage to configure. 22472 Enter a string value. Press Enter for the default (""). 22473 Choose a number from below, or type in your own value 22474 [snip] 22475 XX / 1Fichier 22476 \ "fichier" 22477 [snip] 22478 Storage> fichier 22479 ** See help for fichier backend at: https://rclone.org/fichier/ ** 22480 22481 Your API Key, get it from https://1fichier.com/console/params.pl 22482 Enter a string value. Press Enter for the default (""). 22483 api_key> example_key 22484 22485 Edit advanced config? (y/n) 22486 y) Yes 22487 n) No 22488 y/n> 22489 Remote config 22490 -------------------- 22491 [remote] 22492 type = fichier 22493 api_key = example_key 22494 -------------------- 22495 y) Yes this is OK 22496 e) Edit this remote 22497 d) Delete this remote 22498 y/e/d> y 22499 22500 Once configured you can then use rclone like this, 22501 22502 List directories in top level of your 1Fichier account 22503 22504 rclone lsd remote: 22505 22506 List all the files in your 1Fichier account 22507 22508 rclone ls remote: 22509 22510 To copy a local directory to a 1Fichier directory called backup 22511 22512 rclone copy /home/source remote:backup 22513 22514 Modification times and hashes 22515 22516 1Fichier does not support modification times. It supports the Whirlpool 22517 hash algorithm. 22518 22519 Duplicated files 22520 22521 1Fichier can have two files with exactly the same name and path (unlike 22522 a normal file system). 22523 22524 Duplicated files cause problems with the syncing and you will see 22525 messages in the log about duplicates. 22526 22527 Restricted filename characters 22528 22529 In addition to the default restricted characters set the following 22530 characters are also replaced: 22531 22532 Character Value Replacement 22533 ----------- ------- ------------- 22534 \ 0x5C \ 22535 < 0x3C < 22536 > 0x3E > 22537 " 0x22 " 22538 $ 0x24 $ 22539 ` 0x60 ` 22540 ' 0x27 ' 22541 22542 File names can also not start or end with the following characters. 22543 These only get replaced if they are the first or last character in the 22544 name: 22545 22546 Character Value Replacement 22547 ----------- ------- ------------- 22548 SP 0x20 ␠ 22549 22550 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 22551 strings. 22552 22553 Standard options 22554 22555 Here are the Standard options specific to fichier (1Fichier). 22556 22557 --fichier-api-key 22558 22559 Your API Key, get it from https://1fichier.com/console/params.pl. 22560 22561 Properties: 22562 22563 - Config: api_key 22564 - Env Var: RCLONE_FICHIER_API_KEY 22565 - Type: string 22566 - Required: false 22567 22568 Advanced options 22569 22570 Here are the Advanced options specific to fichier (1Fichier). 22571 22572 --fichier-shared-folder 22573 22574 If you want to download a shared folder, add this parameter. 22575 22576 Properties: 22577 22578 - Config: shared_folder 22579 - Env Var: RCLONE_FICHIER_SHARED_FOLDER 22580 - Type: string 22581 - Required: false 22582 22583 --fichier-file-password 22584 22585 If you want to download a shared file that is password protected, add 22586 this parameter. 22587 22588 NB Input to this must be obscured - see rclone obscure. 22589 22590 Properties: 22591 22592 - Config: file_password 22593 - Env Var: RCLONE_FICHIER_FILE_PASSWORD 22594 - Type: string 22595 - Required: false 22596 22597 --fichier-folder-password 22598 22599 If you want to list the files in a shared folder that is password 22600 protected, add this parameter. 22601 22602 NB Input to this must be obscured - see rclone obscure. 22603 22604 Properties: 22605 22606 - Config: folder_password 22607 - Env Var: RCLONE_FICHIER_FOLDER_PASSWORD 22608 - Type: string 22609 - Required: false 22610 22611 --fichier-cdn 22612 22613 Set if you wish to use CDN download links. 22614 22615 Properties: 22616 22617 - Config: cdn 22618 - Env Var: RCLONE_FICHIER_CDN 22619 - Type: bool 22620 - Default: false 22621 22622 --fichier-encoding 22623 22624 The encoding for the backend. 22625 22626 See the encoding section in the overview for more info. 22627 22628 Properties: 22629 22630 - Config: encoding 22631 - Env Var: RCLONE_FICHIER_ENCODING 22632 - Type: Encoding 22633 - Default: 22634 Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot 22635 22636 --fichier-description 22637 22638 Description of the remote 22639 22640 Properties: 22641 22642 - Config: description 22643 - Env Var: RCLONE_FICHIER_DESCRIPTION 22644 - Type: string 22645 - Required: false 22646 22647 Limitations 22648 22649 rclone about is not supported by the 1Fichier backend. Backends without 22650 this capability cannot determine free space for an rclone mount or use 22651 policy mfs (most free space) as a member of an rclone union remote. 22652 22653 See List of backends that do not support rclone about and rclone about 22654 22655 Alias 22656 22657 The alias remote provides a new name for another remote. 22658 22659 Paths may be as deep as required or a local path, e.g. 22660 remote:directory/subdirectory or /directory/subdirectory. 22661 22662 During the initial setup with rclone config you will specify the target 22663 remote. The target remote can either be a local path or another remote. 22664 22665 Subfolders can be used in target remote. Assume an alias remote named 22666 backup with the target mydrive:private/backup. Invoking 22667 rclone mkdir backup:desktop is exactly the same as invoking 22668 rclone mkdir mydrive:private/backup/desktop. 22669 22670 There will be no special handling of paths containing .. segments. 22671 Invoking rclone mkdir backup:../desktop is exactly the same as invoking 22672 rclone mkdir mydrive:private/backup/../desktop. The empty path is not 22673 allowed as a remote. To alias the current directory use . instead. 22674 22675 The target remote can also be a connection string. This can be used to 22676 modify the config of a remote for different uses, e.g. the alias 22677 myDriveTrash with the target remote myDrive,trashed_only: can be used to 22678 only show the trashed files in myDrive. 22679 22680 Configuration 22681 22682 Here is an example of how to make an alias called remote for local 22683 folder. First run: 22684 22685 rclone config 22686 22687 This will guide you through an interactive setup process: 22688 22689 No remotes found, make a new one? 22690 n) New remote 22691 s) Set configuration password 22692 q) Quit config 22693 n/s/q> n 22694 name> remote 22695 Type of storage to configure. 22696 Choose a number from below, or type in your own value 22697 [snip] 22698 XX / Alias for an existing remote 22699 \ "alias" 22700 [snip] 22701 Storage> alias 22702 Remote or path to alias. 22703 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path". 22704 remote> /mnt/storage/backup 22705 Remote config 22706 -------------------- 22707 [remote] 22708 remote = /mnt/storage/backup 22709 -------------------- 22710 y) Yes this is OK 22711 e) Edit this remote 22712 d) Delete this remote 22713 y/e/d> y 22714 Current remotes: 22715 22716 Name Type 22717 ==== ==== 22718 remote alias 22719 22720 e) Edit existing remote 22721 n) New remote 22722 d) Delete remote 22723 r) Rename remote 22724 c) Copy remote 22725 s) Set configuration password 22726 q) Quit config 22727 e/n/d/r/c/s/q> q 22728 22729 Once configured you can then use rclone like this, 22730 22731 List directories in top level in /mnt/storage/backup 22732 22733 rclone lsd remote: 22734 22735 List all the files in /mnt/storage/backup 22736 22737 rclone ls remote: 22738 22739 Copy another local directory to the alias directory called source 22740 22741 rclone copy /home/source remote:source 22742 22743 Standard options 22744 22745 Here are the Standard options specific to alias (Alias for an existing 22746 remote). 22747 22748 --alias-remote 22749 22750 Remote or path to alias. 22751 22752 Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or 22753 "/local/path". 22754 22755 Properties: 22756 22757 - Config: remote 22758 - Env Var: RCLONE_ALIAS_REMOTE 22759 - Type: string 22760 - Required: true 22761 22762 Advanced options 22763 22764 Here are the Advanced options specific to alias (Alias for an existing 22765 remote). 22766 22767 --alias-description 22768 22769 Description of the remote 22770 22771 Properties: 22772 22773 - Config: description 22774 - Env Var: RCLONE_ALIAS_DESCRIPTION 22775 - Type: string 22776 - Required: false 22777 22778 Amazon S3 Storage Providers 22779 22780 The S3 backend can be used with a number of different providers: 22781 22782 - AWS S3 22783 - Alibaba Cloud (Aliyun) Object Storage System (OSS) 22784 - Ceph 22785 - China Mobile Ecloud Elastic Object Storage (EOS) 22786 - Cloudflare R2 22787 - Arvan Cloud Object Storage (AOS) 22788 - DigitalOcean Spaces 22789 - Dreamhost 22790 - GCS 22791 - Huawei OBS 22792 - IBM COS S3 22793 - IDrive e2 22794 - IONOS Cloud 22795 - Leviia Object Storage 22796 - Liara Object Storage 22797 - Linode Object Storage 22798 - Minio 22799 - Petabox 22800 - Qiniu Cloud Object Storage (Kodo) 22801 - RackCorp Object Storage 22802 - Rclone Serve S3 22803 - Scaleway 22804 - Seagate Lyve Cloud 22805 - SeaweedFS 22806 - StackPath 22807 - Storj 22808 - Synology C2 Object Storage 22809 - Tencent Cloud Object Storage (COS) 22810 - Wasabi 22811 22812 Paths are specified as remote:bucket (or remote: for the lsd command.) 22813 You may put subdirectories in too, e.g. remote:bucket/path/to/dir. 22814 22815 Once you have made a remote (see the provider specific section above) 22816 you can use it like this: 22817 22818 See all buckets 22819 22820 rclone lsd remote: 22821 22822 Make a new bucket 22823 22824 rclone mkdir remote:bucket 22825 22826 List the contents of a bucket 22827 22828 rclone ls remote:bucket 22829 22830 Sync /home/local/directory to the remote bucket, deleting any excess 22831 files in the bucket. 22832 22833 rclone sync --interactive /home/local/directory remote:bucket 22834 22835 Configuration 22836 22837 Here is an example of making an s3 configuration for the AWS S3 22838 provider. Most applies to the other providers as well, any differences 22839 are described below. 22840 22841 First run 22842 22843 rclone config 22844 22845 This will guide you through an interactive setup process. 22846 22847 No remotes found, make a new one? 22848 n) New remote 22849 s) Set configuration password 22850 q) Quit config 22851 n/s/q> n 22852 name> remote 22853 Type of storage to configure. 22854 Choose a number from below, or type in your own value 22855 [snip] 22856 XX / Amazon S3 Compliant Storage Providers including AWS, ... 22857 \ "s3" 22858 [snip] 22859 Storage> s3 22860 Choose your S3 provider. 22861 Choose a number from below, or type in your own value 22862 1 / Amazon Web Services (AWS) S3 22863 \ "AWS" 22864 2 / Ceph Object Storage 22865 \ "Ceph" 22866 3 / DigitalOcean Spaces 22867 \ "DigitalOcean" 22868 4 / Dreamhost DreamObjects 22869 \ "Dreamhost" 22870 5 / IBM COS S3 22871 \ "IBMCOS" 22872 6 / Minio Object Storage 22873 \ "Minio" 22874 7 / Wasabi Object Storage 22875 \ "Wasabi" 22876 8 / Any other S3 compatible provider 22877 \ "Other" 22878 provider> 1 22879 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 22880 Choose a number from below, or type in your own value 22881 1 / Enter AWS credentials in the next step 22882 \ "false" 22883 2 / Get AWS credentials from the environment (env vars or IAM) 22884 \ "true" 22885 env_auth> 1 22886 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 22887 access_key_id> XXX 22888 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 22889 secret_access_key> YYY 22890 Region to connect to. 22891 Choose a number from below, or type in your own value 22892 / The default endpoint - a good choice if you are unsure. 22893 1 | US Region, Northern Virginia, or Pacific Northwest. 22894 | Leave location constraint empty. 22895 \ "us-east-1" 22896 / US East (Ohio) Region 22897 2 | Needs location constraint us-east-2. 22898 \ "us-east-2" 22899 / US West (Oregon) Region 22900 3 | Needs location constraint us-west-2. 22901 \ "us-west-2" 22902 / US West (Northern California) Region 22903 4 | Needs location constraint us-west-1. 22904 \ "us-west-1" 22905 / Canada (Central) Region 22906 5 | Needs location constraint ca-central-1. 22907 \ "ca-central-1" 22908 / EU (Ireland) Region 22909 6 | Needs location constraint EU or eu-west-1. 22910 \ "eu-west-1" 22911 / EU (London) Region 22912 7 | Needs location constraint eu-west-2. 22913 \ "eu-west-2" 22914 / EU (Frankfurt) Region 22915 8 | Needs location constraint eu-central-1. 22916 \ "eu-central-1" 22917 / Asia Pacific (Singapore) Region 22918 9 | Needs location constraint ap-southeast-1. 22919 \ "ap-southeast-1" 22920 / Asia Pacific (Sydney) Region 22921 10 | Needs location constraint ap-southeast-2. 22922 \ "ap-southeast-2" 22923 / Asia Pacific (Tokyo) Region 22924 11 | Needs location constraint ap-northeast-1. 22925 \ "ap-northeast-1" 22926 / Asia Pacific (Seoul) 22927 12 | Needs location constraint ap-northeast-2. 22928 \ "ap-northeast-2" 22929 / Asia Pacific (Mumbai) 22930 13 | Needs location constraint ap-south-1. 22931 \ "ap-south-1" 22932 / Asia Pacific (Hong Kong) Region 22933 14 | Needs location constraint ap-east-1. 22934 \ "ap-east-1" 22935 / South America (Sao Paulo) Region 22936 15 | Needs location constraint sa-east-1. 22937 \ "sa-east-1" 22938 region> 1 22939 Endpoint for S3 API. 22940 Leave blank if using AWS to use the default endpoint for the region. 22941 endpoint> 22942 Location constraint - must be set to match the Region. Used when creating buckets only. 22943 Choose a number from below, or type in your own value 22944 1 / Empty for US Region, Northern Virginia, or Pacific Northwest. 22945 \ "" 22946 2 / US East (Ohio) Region. 22947 \ "us-east-2" 22948 3 / US West (Oregon) Region. 22949 \ "us-west-2" 22950 4 / US West (Northern California) Region. 22951 \ "us-west-1" 22952 5 / Canada (Central) Region. 22953 \ "ca-central-1" 22954 6 / EU (Ireland) Region. 22955 \ "eu-west-1" 22956 7 / EU (London) Region. 22957 \ "eu-west-2" 22958 8 / EU Region. 22959 \ "EU" 22960 9 / Asia Pacific (Singapore) Region. 22961 \ "ap-southeast-1" 22962 10 / Asia Pacific (Sydney) Region. 22963 \ "ap-southeast-2" 22964 11 / Asia Pacific (Tokyo) Region. 22965 \ "ap-northeast-1" 22966 12 / Asia Pacific (Seoul) 22967 \ "ap-northeast-2" 22968 13 / Asia Pacific (Mumbai) 22969 \ "ap-south-1" 22970 14 / Asia Pacific (Hong Kong) 22971 \ "ap-east-1" 22972 15 / South America (Sao Paulo) Region. 22973 \ "sa-east-1" 22974 location_constraint> 1 22975 Canned ACL used when creating buckets and/or storing objects in S3. 22976 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 22977 Choose a number from below, or type in your own value 22978 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 22979 \ "private" 22980 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 22981 \ "public-read" 22982 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 22983 3 | Granting this on a bucket is generally not recommended. 22984 \ "public-read-write" 22985 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. 22986 \ "authenticated-read" 22987 / Object owner gets FULL_CONTROL. Bucket owner gets READ access. 22988 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 22989 \ "bucket-owner-read" 22990 / Both the object owner and the bucket owner get FULL_CONTROL over the object. 22991 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 22992 \ "bucket-owner-full-control" 22993 acl> 1 22994 The server-side encryption algorithm used when storing this object in S3. 22995 Choose a number from below, or type in your own value 22996 1 / None 22997 \ "" 22998 2 / AES256 22999 \ "AES256" 23000 server_side_encryption> 1 23001 The storage class to use when storing objects in S3. 23002 Choose a number from below, or type in your own value 23003 1 / Default 23004 \ "" 23005 2 / Standard storage class 23006 \ "STANDARD" 23007 3 / Reduced redundancy storage class 23008 \ "REDUCED_REDUNDANCY" 23009 4 / Standard Infrequent Access storage class 23010 \ "STANDARD_IA" 23011 5 / One Zone Infrequent Access storage class 23012 \ "ONEZONE_IA" 23013 6 / Glacier storage class 23014 \ "GLACIER" 23015 7 / Glacier Deep Archive storage class 23016 \ "DEEP_ARCHIVE" 23017 8 / Intelligent-Tiering storage class 23018 \ "INTELLIGENT_TIERING" 23019 9 / Glacier Instant Retrieval storage class 23020 \ "GLACIER_IR" 23021 storage_class> 1 23022 Remote config 23023 -------------------- 23024 [remote] 23025 type = s3 23026 provider = AWS 23027 env_auth = false 23028 access_key_id = XXX 23029 secret_access_key = YYY 23030 region = us-east-1 23031 endpoint = 23032 location_constraint = 23033 acl = private 23034 server_side_encryption = 23035 storage_class = 23036 -------------------- 23037 y) Yes this is OK 23038 e) Edit this remote 23039 d) Delete this remote 23040 y/e/d> 23041 23042 Modification times and hashes 23043 23044 Modification times 23045 23046 The modified time is stored as metadata on the object as 23047 X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns. 23048 23049 If the modification time needs to be updated rclone will attempt to 23050 perform a server side copy to update the modification if the object can 23051 be copied in a single part. In the case the object is larger than 5Gb or 23052 is in Glacier or Glacier Deep Archive storage the object will be 23053 uploaded rather than copied. 23054 23055 Note that reading this from the object takes an additional HEAD request 23056 as the metadata isn't returned in object listings. 23057 23058 Hashes 23059 23060 For small objects which weren't uploaded as multipart uploads (objects 23061 sized below --s3-upload-cutoff if uploaded with rclone) rclone uses the 23062 ETag: header as an MD5 checksum. 23063 23064 However for objects which were uploaded as multipart uploads or with 23065 server side encryption (SSE-AWS or SSE-C) the ETag header is no longer 23066 the MD5 sum of the data, so rclone adds an additional piece of metadata 23067 X-Amz-Meta-Md5chksum which is a base64 encoded MD5 hash (in the same 23068 format as is required for Content-MD5). You can use base64 -d and 23069 hexdump to check this value manually: 23070 23071 echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump 23072 23073 or you can use rclone check to verify the hashes are OK. 23074 23075 For large objects, calculating this hash can take some time so the 23076 addition of this hash can be disabled with --s3-disable-checksum. This 23077 will mean that these objects do not have an MD5 checksum. 23078 23079 Note that reading this from the object takes an additional HEAD request 23080 as the metadata isn't returned in object listings. 23081 23082 Reducing costs 23083 23084 Avoiding HEAD requests to read the modification time 23085 23086 By default, rclone will use the modification time of objects stored in 23087 S3 for syncing. This is stored in object metadata which unfortunately 23088 takes an extra HEAD request to read which can be expensive (in time and 23089 money). 23090 23091 The modification time is used by default for all operations that require 23092 checking the time a file was last updated. It allows rclone to treat the 23093 remote more like a true filesystem, but it is inefficient on S3 because 23094 it requires an extra API call to retrieve the metadata. 23095 23096 The extra API calls can be avoided when syncing (using rclone sync or 23097 rclone copy) in a few different ways, each with its own tradeoffs. 23098 23099 - --size-only 23100 - Only checks the size of files. 23101 - Uses no extra transactions. 23102 - If the file doesn't change size then rclone won't detect it has 23103 changed. 23104 - rclone sync --size-only /path/to/source s3:bucket 23105 - --checksum 23106 - Checks the size and MD5 checksum of files. 23107 - Uses no extra transactions. 23108 - The most accurate detection of changes possible. 23109 - Will cause the source to read an MD5 checksum which, if it is a 23110 local disk, will cause lots of disk activity. 23111 - If the source and destination are both S3 this is the 23112 recommended flag to use for maximum efficiency. 23113 - rclone sync --checksum /path/to/source s3:bucket 23114 - --update --use-server-modtime 23115 - Uses no extra transactions. 23116 - Modification time becomes the time the object was uploaded. 23117 - For many operations this is sufficient to determine if it needs 23118 uploading. 23119 - Using --update along with --use-server-modtime, avoids the extra 23120 API call and uploads files whose local modification time is 23121 newer than the time it was last uploaded. 23122 - Files created with timestamps in the past will be missed by the 23123 sync. 23124 - rclone sync --update --use-server-modtime /path/to/source s3:bucket 23125 23126 These flags can and should be used in combination with --fast-list - see 23127 below. 23128 23129 If using rclone mount or any command using the VFS (eg rclone serve) 23130 commands then you might want to consider using the VFS flag --no-modtime 23131 which will stop rclone reading the modification time for every object. 23132 You could also use --use-server-modtime if you are happy with the 23133 modification times of the objects being the time of upload. 23134 23135 Avoiding GET requests to read directory listings 23136 23137 Rclone's default directory traversal is to process each directory 23138 individually. This takes one API call per directory. Using the 23139 --fast-list flag will read all info about the objects into memory first 23140 using a smaller number of API calls (one per 1000 objects). See the 23141 rclone docs for more details. 23142 23143 rclone sync --fast-list --checksum /path/to/source s3:bucket 23144 23145 --fast-list trades off API transactions for memory use. As a rough guide 23146 rclone uses 1k of memory per object stored, so using --fast-list on a 23147 sync of a million objects will use roughly 1 GiB of RAM. 23148 23149 If you are only copying a small number of files into a big repository 23150 then using --no-traverse is a good idea. This finds objects directly 23151 instead of through directory listings. You can do a "top-up" sync very 23152 cheaply by using --max-age and --no-traverse to copy only recent files, 23153 eg 23154 23155 rclone copy --max-age 24h --no-traverse /path/to/source s3:bucket 23156 23157 You'd then do a full rclone sync less often. 23158 23159 Note that --fast-list isn't required in the top-up sync. 23160 23161 Avoiding HEAD requests after PUT 23162 23163 By default, rclone will HEAD every object it uploads. It does this to 23164 check the object got uploaded correctly. 23165 23166 You can disable this with the --s3-no-head option - see there for more 23167 details. 23168 23169 Setting this flag increases the chance for undetected upload failures. 23170 23171 Versions 23172 23173 When bucket versioning is enabled (this can be done with rclone with the 23174 rclone backend versioning command) when rclone uploads a new version of 23175 a file it creates a new version of it Likewise when you delete a file, 23176 the old version will be marked hidden and still be available. 23177 23178 Old versions of files, where available, are visible using the 23179 --s3-versions flag. 23180 23181 It is also possible to view a bucket as it was at a certain point in 23182 time, using the --s3-version-at flag. This will show the file versions 23183 as they were at that time, showing files that have been deleted 23184 afterwards, and hiding files that were created since. 23185 23186 If you wish to remove all the old versions then you can use the 23187 rclone backend cleanup-hidden remote:bucket command which will delete 23188 all the old hidden versions of files, leaving the current ones intact. 23189 You can also supply a path and only old versions under that path will be 23190 deleted, e.g. rclone backend cleanup-hidden remote:bucket/path/to/stuff. 23191 23192 When you purge a bucket, the current and the old versions will be 23193 deleted then the bucket will be deleted. 23194 23195 However delete will cause the current versions of the files to become 23196 hidden old versions. 23197 23198 Here is a session showing the listing and retrieval of an old version 23199 followed by a cleanup of the old versions. 23200 23201 Show current version and all the versions with --s3-versions flag. 23202 23203 $ rclone -q ls s3:cleanup-test 23204 9 one.txt 23205 23206 $ rclone -q --s3-versions ls s3:cleanup-test 23207 9 one.txt 23208 8 one-v2016-07-04-141032-000.txt 23209 16 one-v2016-07-04-141003-000.txt 23210 15 one-v2016-07-02-155621-000.txt 23211 23212 Retrieve an old version 23213 23214 $ rclone -q --s3-versions copy s3:cleanup-test/one-v2016-07-04-141003-000.txt /tmp 23215 23216 $ ls -l /tmp/one-v2016-07-04-141003-000.txt 23217 -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt 23218 23219 Clean up all the old versions and show that they've gone. 23220 23221 $ rclone -q backend cleanup-hidden s3:cleanup-test 23222 23223 $ rclone -q ls s3:cleanup-test 23224 9 one.txt 23225 23226 $ rclone -q --s3-versions ls s3:cleanup-test 23227 9 one.txt 23228 23229 Versions naming caveat 23230 23231 When using --s3-versions flag rclone is relying on the file name to work 23232 out whether the objects are versions or not. Versions' names are created 23233 by inserting timestamp between file name and its extension. 23234 23235 9 file.txt 23236 8 file-v2023-07-17-161032-000.txt 23237 16 file-v2023-06-15-141003-000.txt 23238 23239 If there are real files present with the same names as versions, then 23240 behaviour of --s3-versions can be unpredictable. 23241 23242 Cleanup 23243 23244 If you run rclone cleanup s3:bucket then it will remove all pending 23245 multipart uploads older than 24 hours. You can use the --interactive/i 23246 or --dry-run flag to see exactly what it will do. If you want more 23247 control over the expiry date then run 23248 rclone backend cleanup s3:bucket -o max-age=1h to expire all uploads 23249 older than one hour. You can use 23250 rclone backend list-multipart-uploads s3:bucket to see the pending 23251 multipart uploads. 23252 23253 Restricted filename characters 23254 23255 S3 allows any valid UTF-8 string as a key. 23256 23257 Invalid UTF-8 bytes will be replaced, as they can't be used in XML. 23258 23259 The following characters are replaced since these are problematic when 23260 dealing with the REST API: 23261 23262 Character Value Replacement 23263 ----------- ------- ------------- 23264 NUL 0x00 ␀ 23265 / 0x2F / 23266 23267 The encoding will also encode these file names as they don't seem to 23268 work with the SDK properly: 23269 23270 File name Replacement 23271 ----------- ------------- 23272 . . 23273 .. .. 23274 23275 Multipart uploads 23276 23277 rclone supports multipart uploads with S3 which means that it can upload 23278 files bigger than 5 GiB. 23279 23280 Note that files uploaded both with multipart upload and through crypt 23281 remotes do not have MD5 sums. 23282 23283 rclone switches from single part uploads to multipart uploads at the 23284 point specified by --s3-upload-cutoff. This can be a maximum of 5 GiB 23285 and a minimum of 0 (ie always upload multipart files). 23286 23287 The chunk sizes used in the multipart upload are specified by 23288 --s3-chunk-size and the number of chunks uploaded concurrently is 23289 specified by --s3-upload-concurrency. 23290 23291 Multipart uploads will use --transfers * --s3-upload-concurrency * 23292 --s3-chunk-size extra memory. Single part uploads to not use extra 23293 memory. 23294 23295 Single part transfers can be faster than multipart transfers or slower 23296 depending on your latency from S3 - the more latency, the more likely 23297 single part transfers will be faster. 23298 23299 Increasing --s3-upload-concurrency will increase throughput (8 would be 23300 a sensible value) and increasing --s3-chunk-size also increases 23301 throughput (16M would be sensible). Increasing either of these will use 23302 more memory. The default values are high enough to gain most of the 23303 possible performance without using too much memory. 23304 23305 Buckets and Regions 23306 23307 With Amazon S3 you can list buckets (rclone lsd) using any region, but 23308 you can only access the content of a bucket from the region it was 23309 created in. If you attempt to access a bucket from the wrong region, you 23310 will get an error, incorrect region, the bucket is not in 'XXX' region. 23311 23312 Authentication 23313 23314 There are a number of ways to supply rclone with a set of AWS 23315 credentials, with and without using the environment. 23316 23317 The different authentication methods are tried in this order: 23318 23319 - Directly in the rclone configuration file (env_auth = false in the 23320 config file): 23321 - access_key_id and secret_access_key are required. 23322 - session_token can be optionally set when using AWS STS. 23323 - Runtime configuration (env_auth = true in the config file): 23324 - Export the following environment variables before running 23325 rclone: 23326 - Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY 23327 - Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY 23328 - Session Token: AWS_SESSION_TOKEN (optional) 23329 - Or, use a named profile: 23330 - Profile files are standard files used by AWS CLI tools 23331 - By default it will use the profile in your home directory 23332 (e.g. ~/.aws/credentials on unix based systems) file and the 23333 "default" profile, to change set these environment 23334 variables: 23335 - AWS_SHARED_CREDENTIALS_FILE to control which file. 23336 - AWS_PROFILE to control which profile to use. 23337 - Or, run rclone in an ECS task with an IAM role (AWS only). 23338 - Or, run rclone on an EC2 instance with an IAM role (AWS only). 23339 - Or, run rclone in an EKS pod with an IAM role that is associated 23340 with a service account (AWS only). 23341 23342 If none of these option actually end up providing rclone with AWS 23343 credentials then S3 interaction will be non-authenticated (see below). 23344 23345 S3 Permissions 23346 23347 When using the sync subcommand of rclone the following minimum 23348 permissions are required to be available on the bucket being written to: 23349 23350 - ListBucket 23351 - DeleteObject 23352 - GetObject 23353 - PutObject 23354 - PutObjectACL 23355 - CreateBucket (unless using s3-no-check-bucket) 23356 23357 When using the lsd subcommand, the ListAllMyBuckets permission is 23358 required. 23359 23360 Example policy: 23361 23362 { 23363 "Version": "2012-10-17", 23364 "Statement": [ 23365 { 23366 "Effect": "Allow", 23367 "Principal": { 23368 "AWS": "arn:aws:iam::USER_SID:user/USER_NAME" 23369 }, 23370 "Action": [ 23371 "s3:ListBucket", 23372 "s3:DeleteObject", 23373 "s3:GetObject", 23374 "s3:PutObject", 23375 "s3:PutObjectAcl" 23376 ], 23377 "Resource": [ 23378 "arn:aws:s3:::BUCKET_NAME/*", 23379 "arn:aws:s3:::BUCKET_NAME" 23380 ] 23381 }, 23382 { 23383 "Effect": "Allow", 23384 "Action": "s3:ListAllMyBuckets", 23385 "Resource": "arn:aws:s3:::*" 23386 } 23387 ] 23388 } 23389 23390 Notes on above: 23391 23392 1. This is a policy that can be used when creating bucket. It assumes 23393 that USER_NAME has been created. 23394 2. The Resource entry must include both resource ARNs, as one implies 23395 the bucket and the other implies the bucket's objects. 23396 3. When using s3-no-check-bucket and the bucket already exsits, the 23397 "arn:aws:s3:::BUCKET_NAME" doesn't have to be included. 23398 23399 For reference, here's an Ansible script that will generate one or more 23400 buckets that will work with rclone sync. 23401 23402 Key Management System (KMS) 23403 23404 If you are using server-side encryption with KMS then you must make sure 23405 rclone is configured with server_side_encryption = aws:kms otherwise you 23406 will find you can't transfer small objects - these will create checksum 23407 errors. 23408 23409 Glacier and Glacier Deep Archive 23410 23411 You can upload objects using the glacier storage class or transition 23412 them to glacier using a lifecycle policy. The bucket can still be synced 23413 or copied into normally, but if rclone tries to access data from the 23414 glacier storage class you will see an error like below. 23415 23416 2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file 23417 23418 In this case you need to restore the object(s) in question before using 23419 rclone. 23420 23421 Note that rclone only speaks the S3 API it does not speak the Glacier 23422 Vault API, so rclone cannot directly access Glacier Vaults. 23423 23424 Object-lock enabled S3 bucket 23425 23426 According to AWS's documentation on S3 Object Lock: 23427 23428 If you configure a default retention period on a bucket, requests to 23429 upload objects in such a bucket must include the Content-MD5 header. 23430 23431 As mentioned in the Modification times and hashes section, small files 23432 that are not uploaded as multipart, use a different tag, causing the 23433 upload to fail. A simple solution is to set the --s3-upload-cutoff 0 and 23434 force all the files to be uploaded as multipart. 23435 23436 Standard options 23437 23438 Here are the Standard options specific to s3 (Amazon S3 Compliant 23439 Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, 23440 Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, 23441 IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, 23442 RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, 23443 TencentCOS, Wasabi, Qiniu and others). 23444 23445 --s3-provider 23446 23447 Choose your S3 provider. 23448 23449 Properties: 23450 23451 - Config: provider 23452 - Env Var: RCLONE_S3_PROVIDER 23453 - Type: string 23454 - Required: false 23455 - Examples: 23456 - "AWS" 23457 - Amazon Web Services (AWS) S3 23458 - "Alibaba" 23459 - Alibaba Cloud Object Storage System (OSS) formerly Aliyun 23460 - "ArvanCloud" 23461 - Arvan Cloud Object Storage (AOS) 23462 - "Ceph" 23463 - Ceph Object Storage 23464 - "ChinaMobile" 23465 - China Mobile Ecloud Elastic Object Storage (EOS) 23466 - "Cloudflare" 23467 - Cloudflare R2 Storage 23468 - "DigitalOcean" 23469 - DigitalOcean Spaces 23470 - "Dreamhost" 23471 - Dreamhost DreamObjects 23472 - "GCS" 23473 - Google Cloud Storage 23474 - "HuaweiOBS" 23475 - Huawei Object Storage Service 23476 - "IBMCOS" 23477 - IBM COS S3 23478 - "IDrive" 23479 - IDrive e2 23480 - "IONOS" 23481 - IONOS Cloud 23482 - "LyveCloud" 23483 - Seagate Lyve Cloud 23484 - "Leviia" 23485 - Leviia Object Storage 23486 - "Liara" 23487 - Liara Object Storage 23488 - "Linode" 23489 - Linode Object Storage 23490 - "Minio" 23491 - Minio Object Storage 23492 - "Netease" 23493 - Netease Object Storage (NOS) 23494 - "Petabox" 23495 - Petabox Object Storage 23496 - "RackCorp" 23497 - RackCorp Object Storage 23498 - "Rclone" 23499 - Rclone S3 Server 23500 - "Scaleway" 23501 - Scaleway Object Storage 23502 - "SeaweedFS" 23503 - SeaweedFS S3 23504 - "StackPath" 23505 - StackPath Object Storage 23506 - "Storj" 23507 - Storj (S3 Compatible Gateway) 23508 - "Synology" 23509 - Synology C2 Object Storage 23510 - "TencentCOS" 23511 - Tencent Cloud Object Storage (COS) 23512 - "Wasabi" 23513 - Wasabi Object Storage 23514 - "Qiniu" 23515 - Qiniu Object Storage (Kodo) 23516 - "Other" 23517 - Any other S3 compatible provider 23518 23519 --s3-env-auth 23520 23521 Get AWS credentials from runtime (environment variables or EC2/ECS meta 23522 data if no env vars). 23523 23524 Only applies if access_key_id and secret_access_key is blank. 23525 23526 Properties: 23527 23528 - Config: env_auth 23529 - Env Var: RCLONE_S3_ENV_AUTH 23530 - Type: bool 23531 - Default: false 23532 - Examples: 23533 - "false" 23534 - Enter AWS credentials in the next step. 23535 - "true" 23536 - Get AWS credentials from the environment (env vars or IAM). 23537 23538 --s3-access-key-id 23539 23540 AWS Access Key ID. 23541 23542 Leave blank for anonymous access or runtime credentials. 23543 23544 Properties: 23545 23546 - Config: access_key_id 23547 - Env Var: RCLONE_S3_ACCESS_KEY_ID 23548 - Type: string 23549 - Required: false 23550 23551 --s3-secret-access-key 23552 23553 AWS Secret Access Key (password). 23554 23555 Leave blank for anonymous access or runtime credentials. 23556 23557 Properties: 23558 23559 - Config: secret_access_key 23560 - Env Var: RCLONE_S3_SECRET_ACCESS_KEY 23561 - Type: string 23562 - Required: false 23563 23564 --s3-region 23565 23566 Region to connect to. 23567 23568 Properties: 23569 23570 - Config: region 23571 - Env Var: RCLONE_S3_REGION 23572 - Provider: AWS 23573 - Type: string 23574 - Required: false 23575 - Examples: 23576 - "us-east-1" 23577 - The default endpoint - a good choice if you are unsure. 23578 - US Region, Northern Virginia, or Pacific Northwest. 23579 - Leave location constraint empty. 23580 - "us-east-2" 23581 - US East (Ohio) Region. 23582 - Needs location constraint us-east-2. 23583 - "us-west-1" 23584 - US West (Northern California) Region. 23585 - Needs location constraint us-west-1. 23586 - "us-west-2" 23587 - US West (Oregon) Region. 23588 - Needs location constraint us-west-2. 23589 - "ca-central-1" 23590 - Canada (Central) Region. 23591 - Needs location constraint ca-central-1. 23592 - "eu-west-1" 23593 - EU (Ireland) Region. 23594 - Needs location constraint EU or eu-west-1. 23595 - "eu-west-2" 23596 - EU (London) Region. 23597 - Needs location constraint eu-west-2. 23598 - "eu-west-3" 23599 - EU (Paris) Region. 23600 - Needs location constraint eu-west-3. 23601 - "eu-north-1" 23602 - EU (Stockholm) Region. 23603 - Needs location constraint eu-north-1. 23604 - "eu-south-1" 23605 - EU (Milan) Region. 23606 - Needs location constraint eu-south-1. 23607 - "eu-central-1" 23608 - EU (Frankfurt) Region. 23609 - Needs location constraint eu-central-1. 23610 - "ap-southeast-1" 23611 - Asia Pacific (Singapore) Region. 23612 - Needs location constraint ap-southeast-1. 23613 - "ap-southeast-2" 23614 - Asia Pacific (Sydney) Region. 23615 - Needs location constraint ap-southeast-2. 23616 - "ap-northeast-1" 23617 - Asia Pacific (Tokyo) Region. 23618 - Needs location constraint ap-northeast-1. 23619 - "ap-northeast-2" 23620 - Asia Pacific (Seoul). 23621 - Needs location constraint ap-northeast-2. 23622 - "ap-northeast-3" 23623 - Asia Pacific (Osaka-Local). 23624 - Needs location constraint ap-northeast-3. 23625 - "ap-south-1" 23626 - Asia Pacific (Mumbai). 23627 - Needs location constraint ap-south-1. 23628 - "ap-east-1" 23629 - Asia Pacific (Hong Kong) Region. 23630 - Needs location constraint ap-east-1. 23631 - "sa-east-1" 23632 - South America (Sao Paulo) Region. 23633 - Needs location constraint sa-east-1. 23634 - "me-south-1" 23635 - Middle East (Bahrain) Region. 23636 - Needs location constraint me-south-1. 23637 - "af-south-1" 23638 - Africa (Cape Town) Region. 23639 - Needs location constraint af-south-1. 23640 - "cn-north-1" 23641 - China (Beijing) Region. 23642 - Needs location constraint cn-north-1. 23643 - "cn-northwest-1" 23644 - China (Ningxia) Region. 23645 - Needs location constraint cn-northwest-1. 23646 - "us-gov-east-1" 23647 - AWS GovCloud (US-East) Region. 23648 - Needs location constraint us-gov-east-1. 23649 - "us-gov-west-1" 23650 - AWS GovCloud (US) Region. 23651 - Needs location constraint us-gov-west-1. 23652 23653 --s3-endpoint 23654 23655 Endpoint for S3 API. 23656 23657 Leave blank if using AWS to use the default endpoint for the region. 23658 23659 Properties: 23660 23661 - Config: endpoint 23662 - Env Var: RCLONE_S3_ENDPOINT 23663 - Provider: AWS 23664 - Type: string 23665 - Required: false 23666 23667 --s3-location-constraint 23668 23669 Location constraint - must be set to match the Region. 23670 23671 Used when creating buckets only. 23672 23673 Properties: 23674 23675 - Config: location_constraint 23676 - Env Var: RCLONE_S3_LOCATION_CONSTRAINT 23677 - Provider: AWS 23678 - Type: string 23679 - Required: false 23680 - Examples: 23681 - "" 23682 - Empty for US Region, Northern Virginia, or Pacific Northwest 23683 - "us-east-2" 23684 - US East (Ohio) Region 23685 - "us-west-1" 23686 - US West (Northern California) Region 23687 - "us-west-2" 23688 - US West (Oregon) Region 23689 - "ca-central-1" 23690 - Canada (Central) Region 23691 - "eu-west-1" 23692 - EU (Ireland) Region 23693 - "eu-west-2" 23694 - EU (London) Region 23695 - "eu-west-3" 23696 - EU (Paris) Region 23697 - "eu-north-1" 23698 - EU (Stockholm) Region 23699 - "eu-south-1" 23700 - EU (Milan) Region 23701 - "EU" 23702 - EU Region 23703 - "ap-southeast-1" 23704 - Asia Pacific (Singapore) Region 23705 - "ap-southeast-2" 23706 - Asia Pacific (Sydney) Region 23707 - "ap-northeast-1" 23708 - Asia Pacific (Tokyo) Region 23709 - "ap-northeast-2" 23710 - Asia Pacific (Seoul) Region 23711 - "ap-northeast-3" 23712 - Asia Pacific (Osaka-Local) Region 23713 - "ap-south-1" 23714 - Asia Pacific (Mumbai) Region 23715 - "ap-east-1" 23716 - Asia Pacific (Hong Kong) Region 23717 - "sa-east-1" 23718 - South America (Sao Paulo) Region 23719 - "me-south-1" 23720 - Middle East (Bahrain) Region 23721 - "af-south-1" 23722 - Africa (Cape Town) Region 23723 - "cn-north-1" 23724 - China (Beijing) Region 23725 - "cn-northwest-1" 23726 - China (Ningxia) Region 23727 - "us-gov-east-1" 23728 - AWS GovCloud (US-East) Region 23729 - "us-gov-west-1" 23730 - AWS GovCloud (US) Region 23731 23732 --s3-acl 23733 23734 Canned ACL used when creating buckets and storing or copying objects. 23735 23736 This ACL is used for creating objects and if bucket_acl isn't set, for 23737 creating buckets too. 23738 23739 For more info visit 23740 https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 23741 23742 Note that this ACL is applied when server-side copying objects as S3 23743 doesn't copy the ACL from the source but rather writes a fresh one. 23744 23745 If the acl is an empty string then no X-Amz-Acl: header is added and the 23746 default (private) will be used. 23747 23748 Properties: 23749 23750 - Config: acl 23751 - Env Var: RCLONE_S3_ACL 23752 - Provider: !Storj,Synology,Cloudflare 23753 - Type: string 23754 - Required: false 23755 - Examples: 23756 - "default" 23757 - Owner gets Full_CONTROL. 23758 - No one else has access rights (default). 23759 - "private" 23760 - Owner gets FULL_CONTROL. 23761 - No one else has access rights (default). 23762 - "public-read" 23763 - Owner gets FULL_CONTROL. 23764 - The AllUsers group gets READ access. 23765 - "public-read-write" 23766 - Owner gets FULL_CONTROL. 23767 - The AllUsers group gets READ and WRITE access. 23768 - Granting this on a bucket is generally not recommended. 23769 - "authenticated-read" 23770 - Owner gets FULL_CONTROL. 23771 - The AuthenticatedUsers group gets READ access. 23772 - "bucket-owner-read" 23773 - Object owner gets FULL_CONTROL. 23774 - Bucket owner gets READ access. 23775 - If you specify this canned ACL when creating a bucket, 23776 Amazon S3 ignores it. 23777 - "bucket-owner-full-control" 23778 - Both the object owner and the bucket owner get FULL_CONTROL 23779 over the object. 23780 - If you specify this canned ACL when creating a bucket, 23781 Amazon S3 ignores it. 23782 - "private" 23783 - Owner gets FULL_CONTROL. 23784 - No one else has access rights (default). 23785 - This acl is available on IBM Cloud (Infra), IBM Cloud 23786 (Storage), On-Premise COS. 23787 - "public-read" 23788 - Owner gets FULL_CONTROL. 23789 - The AllUsers group gets READ access. 23790 - This acl is available on IBM Cloud (Infra), IBM Cloud 23791 (Storage), On-Premise IBM COS. 23792 - "public-read-write" 23793 - Owner gets FULL_CONTROL. 23794 - The AllUsers group gets READ and WRITE access. 23795 - This acl is available on IBM Cloud (Infra), On-Premise IBM 23796 COS. 23797 - "authenticated-read" 23798 - Owner gets FULL_CONTROL. 23799 - The AuthenticatedUsers group gets READ access. 23800 - Not supported on Buckets. 23801 - This acl is available on IBM Cloud (Infra) and On-Premise 23802 IBM COS. 23803 23804 --s3-server-side-encryption 23805 23806 The server-side encryption algorithm used when storing this object in 23807 S3. 23808 23809 Properties: 23810 23811 - Config: server_side_encryption 23812 - Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION 23813 - Provider: AWS,Ceph,ChinaMobile,Minio 23814 - Type: string 23815 - Required: false 23816 - Examples: 23817 - "" 23818 - None 23819 - "AES256" 23820 - AES256 23821 - "aws:kms" 23822 - aws:kms 23823 23824 --s3-sse-kms-key-id 23825 23826 If using KMS ID you must provide the ARN of Key. 23827 23828 Properties: 23829 23830 - Config: sse_kms_key_id 23831 - Env Var: RCLONE_S3_SSE_KMS_KEY_ID 23832 - Provider: AWS,Ceph,Minio 23833 - Type: string 23834 - Required: false 23835 - Examples: 23836 - "" 23837 - None 23838 - "arn:aws:kms:us-east-1:*" 23839 - arn:aws:kms:* 23840 23841 --s3-storage-class 23842 23843 The storage class to use when storing new objects in S3. 23844 23845 Properties: 23846 23847 - Config: storage_class 23848 - Env Var: RCLONE_S3_STORAGE_CLASS 23849 - Provider: AWS 23850 - Type: string 23851 - Required: false 23852 - Examples: 23853 - "" 23854 - Default 23855 - "STANDARD" 23856 - Standard storage class 23857 - "REDUCED_REDUNDANCY" 23858 - Reduced redundancy storage class 23859 - "STANDARD_IA" 23860 - Standard Infrequent Access storage class 23861 - "ONEZONE_IA" 23862 - One Zone Infrequent Access storage class 23863 - "GLACIER" 23864 - Glacier storage class 23865 - "DEEP_ARCHIVE" 23866 - Glacier Deep Archive storage class 23867 - "INTELLIGENT_TIERING" 23868 - Intelligent-Tiering storage class 23869 - "GLACIER_IR" 23870 - Glacier Instant Retrieval storage class 23871 23872 Advanced options 23873 23874 Here are the Advanced options specific to s3 (Amazon S3 Compliant 23875 Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, 23876 Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, 23877 IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, 23878 RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, 23879 TencentCOS, Wasabi, Qiniu and others). 23880 23881 --s3-bucket-acl 23882 23883 Canned ACL used when creating buckets. 23884 23885 For more info visit 23886 https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 23887 23888 Note that this ACL is applied when only when creating buckets. If it 23889 isn't set then "acl" is used instead. 23890 23891 If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl: 23892 header is added and the default (private) will be used. 23893 23894 Properties: 23895 23896 - Config: bucket_acl 23897 - Env Var: RCLONE_S3_BUCKET_ACL 23898 - Type: string 23899 - Required: false 23900 - Examples: 23901 - "private" 23902 - Owner gets FULL_CONTROL. 23903 - No one else has access rights (default). 23904 - "public-read" 23905 - Owner gets FULL_CONTROL. 23906 - The AllUsers group gets READ access. 23907 - "public-read-write" 23908 - Owner gets FULL_CONTROL. 23909 - The AllUsers group gets READ and WRITE access. 23910 - Granting this on a bucket is generally not recommended. 23911 - "authenticated-read" 23912 - Owner gets FULL_CONTROL. 23913 - The AuthenticatedUsers group gets READ access. 23914 23915 --s3-requester-pays 23916 23917 Enables requester pays option when interacting with S3 bucket. 23918 23919 Properties: 23920 23921 - Config: requester_pays 23922 - Env Var: RCLONE_S3_REQUESTER_PAYS 23923 - Provider: AWS 23924 - Type: bool 23925 - Default: false 23926 23927 --s3-sse-customer-algorithm 23928 23929 If using SSE-C, the server-side encryption algorithm used when storing 23930 this object in S3. 23931 23932 Properties: 23933 23934 - Config: sse_customer_algorithm 23935 - Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM 23936 - Provider: AWS,Ceph,ChinaMobile,Minio 23937 - Type: string 23938 - Required: false 23939 - Examples: 23940 - "" 23941 - None 23942 - "AES256" 23943 - AES256 23944 23945 --s3-sse-customer-key 23946 23947 To use SSE-C you may provide the secret encryption key used to 23948 encrypt/decrypt your data. 23949 23950 Alternatively you can provide --sse-customer-key-base64. 23951 23952 Properties: 23953 23954 - Config: sse_customer_key 23955 - Env Var: RCLONE_S3_SSE_CUSTOMER_KEY 23956 - Provider: AWS,Ceph,ChinaMobile,Minio 23957 - Type: string 23958 - Required: false 23959 - Examples: 23960 - "" 23961 - None 23962 23963 --s3-sse-customer-key-base64 23964 23965 If using SSE-C you must provide the secret encryption key encoded in 23966 base64 format to encrypt/decrypt your data. 23967 23968 Alternatively you can provide --sse-customer-key. 23969 23970 Properties: 23971 23972 - Config: sse_customer_key_base64 23973 - Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64 23974 - Provider: AWS,Ceph,ChinaMobile,Minio 23975 - Type: string 23976 - Required: false 23977 - Examples: 23978 - "" 23979 - None 23980 23981 --s3-sse-customer-key-md5 23982 23983 If using SSE-C you may provide the secret encryption key MD5 checksum 23984 (optional). 23985 23986 If you leave it blank, this is calculated automatically from the 23987 sse_customer_key provided. 23988 23989 Properties: 23990 23991 - Config: sse_customer_key_md5 23992 - Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5 23993 - Provider: AWS,Ceph,ChinaMobile,Minio 23994 - Type: string 23995 - Required: false 23996 - Examples: 23997 - "" 23998 - None 23999 24000 --s3-upload-cutoff 24001 24002 Cutoff for switching to chunked upload. 24003 24004 Any files larger than this will be uploaded in chunks of chunk_size. The 24005 minimum is 0 and the maximum is 5 GiB. 24006 24007 Properties: 24008 24009 - Config: upload_cutoff 24010 - Env Var: RCLONE_S3_UPLOAD_CUTOFF 24011 - Type: SizeSuffix 24012 - Default: 200Mi 24013 24014 --s3-chunk-size 24015 24016 Chunk size to use for uploading. 24017 24018 When uploading files larger than upload_cutoff or files with unknown 24019 size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google 24020 photos or google docs) they will be uploaded as multipart uploads using 24021 this chunk size. 24022 24023 Note that "--s3-upload-concurrency" chunks of this size are buffered in 24024 memory per transfer. 24025 24026 If you are transferring large files over high-speed links and you have 24027 enough memory, then increasing this will speed up the transfers. 24028 24029 Rclone will automatically increase the chunk size when uploading a large 24030 file of known size to stay below the 10,000 chunks limit. 24031 24032 Files of unknown size are uploaded with the configured chunk_size. Since 24033 the default chunk size is 5 MiB and there can be at most 10,000 chunks, 24034 this means that by default the maximum size of a file you can stream 24035 upload is 48 GiB. If you wish to stream upload larger files then you 24036 will need to increase chunk_size. 24037 24038 Increasing the chunk size decreases the accuracy of the progress 24039 statistics displayed with "-P" flag. Rclone treats chunk as sent when 24040 it's buffered by the AWS SDK, when in fact it may still be uploading. A 24041 bigger chunk size means a bigger AWS SDK buffer and progress reporting 24042 more deviating from the truth. 24043 24044 Properties: 24045 24046 - Config: chunk_size 24047 - Env Var: RCLONE_S3_CHUNK_SIZE 24048 - Type: SizeSuffix 24049 - Default: 5Mi 24050 24051 --s3-max-upload-parts 24052 24053 Maximum number of parts in a multipart upload. 24054 24055 This option defines the maximum number of multipart chunks to use when 24056 doing a multipart upload. 24057 24058 This can be useful if a service does not support the AWS S3 24059 specification of 10,000 chunks. 24060 24061 Rclone will automatically increase the chunk size when uploading a large 24062 file of a known size to stay below this number of chunks limit. 24063 24064 Properties: 24065 24066 - Config: max_upload_parts 24067 - Env Var: RCLONE_S3_MAX_UPLOAD_PARTS 24068 - Type: int 24069 - Default: 10000 24070 24071 --s3-copy-cutoff 24072 24073 Cutoff for switching to multipart copy. 24074 24075 Any files larger than this that need to be server-side copied will be 24076 copied in chunks of this size. 24077 24078 The minimum is 0 and the maximum is 5 GiB. 24079 24080 Properties: 24081 24082 - Config: copy_cutoff 24083 - Env Var: RCLONE_S3_COPY_CUTOFF 24084 - Type: SizeSuffix 24085 - Default: 4.656Gi 24086 24087 --s3-disable-checksum 24088 24089 Don't store MD5 checksum with object metadata. 24090 24091 Normally rclone will calculate the MD5 checksum of the input before 24092 uploading it so it can add it to metadata on the object. This is great 24093 for data integrity checking but can cause long delays for large files to 24094 start uploading. 24095 24096 Properties: 24097 24098 - Config: disable_checksum 24099 - Env Var: RCLONE_S3_DISABLE_CHECKSUM 24100 - Type: bool 24101 - Default: false 24102 24103 --s3-shared-credentials-file 24104 24105 Path to the shared credentials file. 24106 24107 If env_auth = true then rclone can use a shared credentials file. 24108 24109 If this variable is empty rclone will look for the 24110 "AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty it 24111 will default to the current user's home directory. 24112 24113 Linux/OSX: "$HOME/.aws/credentials" 24114 Windows: "%USERPROFILE%\.aws\credentials" 24115 24116 Properties: 24117 24118 - Config: shared_credentials_file 24119 - Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE 24120 - Type: string 24121 - Required: false 24122 24123 --s3-profile 24124 24125 Profile to use in the shared credentials file. 24126 24127 If env_auth = true then rclone can use a shared credentials file. This 24128 variable controls which profile is used in that file. 24129 24130 If empty it will default to the environment variable "AWS_PROFILE" or 24131 "default" if that environment variable is also not set. 24132 24133 Properties: 24134 24135 - Config: profile 24136 - Env Var: RCLONE_S3_PROFILE 24137 - Type: string 24138 - Required: false 24139 24140 --s3-session-token 24141 24142 An AWS session token. 24143 24144 Properties: 24145 24146 - Config: session_token 24147 - Env Var: RCLONE_S3_SESSION_TOKEN 24148 - Type: string 24149 - Required: false 24150 24151 --s3-upload-concurrency 24152 24153 Concurrency for multipart uploads and copies. 24154 24155 This is the number of chunks of the same file that are uploaded 24156 concurrently for multipart uploads and copies. 24157 24158 If you are uploading small numbers of large files over high-speed links 24159 and these uploads do not fully utilize your bandwidth, then increasing 24160 this may help to speed up the transfers. 24161 24162 Properties: 24163 24164 - Config: upload_concurrency 24165 - Env Var: RCLONE_S3_UPLOAD_CONCURRENCY 24166 - Type: int 24167 - Default: 4 24168 24169 --s3-force-path-style 24170 24171 If true use path style access if false use virtual hosted style. 24172 24173 If this is true (the default) then rclone will use path style access, if 24174 false then rclone will use virtual path style. See the AWS S3 docs for 24175 more info. 24176 24177 Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) 24178 require this set to false - rclone will do this automatically based on 24179 the provider setting. 24180 24181 Properties: 24182 24183 - Config: force_path_style 24184 - Env Var: RCLONE_S3_FORCE_PATH_STYLE 24185 - Type: bool 24186 - Default: true 24187 24188 --s3-v2-auth 24189 24190 If true use v2 authentication. 24191 24192 If this is false (the default) then rclone will use v4 authentication. 24193 If it is set then rclone will use v2 authentication. 24194 24195 Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH. 24196 24197 Properties: 24198 24199 - Config: v2_auth 24200 - Env Var: RCLONE_S3_V2_AUTH 24201 - Type: bool 24202 - Default: false 24203 24204 --s3-use-dual-stack 24205 24206 If true use AWS S3 dual-stack endpoint (IPv6 support). 24207 24208 See AWS Docs on Dualstack Endpoints 24209 24210 Properties: 24211 24212 - Config: use_dual_stack 24213 - Env Var: RCLONE_S3_USE_DUAL_STACK 24214 - Type: bool 24215 - Default: false 24216 24217 --s3-use-accelerate-endpoint 24218 24219 If true use the AWS S3 accelerated endpoint. 24220 24221 See: AWS S3 Transfer acceleration 24222 24223 Properties: 24224 24225 - Config: use_accelerate_endpoint 24226 - Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT 24227 - Provider: AWS 24228 - Type: bool 24229 - Default: false 24230 24231 --s3-leave-parts-on-error 24232 24233 If true avoid calling abort upload on a failure, leaving all 24234 successfully uploaded parts on S3 for manual recovery. 24235 24236 It should be set to true for resuming uploads across different sessions. 24237 24238 WARNING: Storing parts of an incomplete multipart upload counts towards 24239 space usage on S3 and will add additional costs if not cleaned up. 24240 24241 Properties: 24242 24243 - Config: leave_parts_on_error 24244 - Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR 24245 - Provider: AWS 24246 - Type: bool 24247 - Default: false 24248 24249 --s3-list-chunk 24250 24251 Size of listing chunk (response list for each ListObject S3 request). 24252 24253 This option is also known as "MaxKeys", "max-items", or "page-size" from 24254 the AWS S3 specification. Most services truncate the response list to 24255 1000 objects even if requested more than that. In AWS S3 this is a 24256 global maximum and cannot be changed, see AWS S3. In Ceph, this can be 24257 increased with the "rgw list buckets max chunk" option. 24258 24259 Properties: 24260 24261 - Config: list_chunk 24262 - Env Var: RCLONE_S3_LIST_CHUNK 24263 - Type: int 24264 - Default: 1000 24265 24266 --s3-list-version 24267 24268 Version of ListObjects to use: 1,2 or 0 for auto. 24269 24270 When S3 originally launched it only provided the ListObjects call to 24271 enumerate objects in a bucket. 24272 24273 However in May 2016 the ListObjectsV2 call was introduced. This is much 24274 higher performance and should be used if at all possible. 24275 24276 If set to the default, 0, rclone will guess according to the provider 24277 set which list objects method to call. If it guesses wrong, then it may 24278 be set manually here. 24279 24280 Properties: 24281 24282 - Config: list_version 24283 - Env Var: RCLONE_S3_LIST_VERSION 24284 - Type: int 24285 - Default: 0 24286 24287 --s3-list-url-encode 24288 24289 Whether to url encode listings: true/false/unset 24290 24291 Some providers support URL encoding listings and where this is available 24292 this is more reliable when using control characters in file names. If 24293 this is set to unset (the default) then rclone will choose according to 24294 the provider setting what to apply, but you can override rclone's choice 24295 here. 24296 24297 Properties: 24298 24299 - Config: list_url_encode 24300 - Env Var: RCLONE_S3_LIST_URL_ENCODE 24301 - Type: Tristate 24302 - Default: unset 24303 24304 --s3-no-check-bucket 24305 24306 If set, don't attempt to check the bucket exists or create it. 24307 24308 This can be useful when trying to minimise the number of transactions 24309 rclone does if you know the bucket exists already. 24310 24311 It can also be needed if the user you are using does not have bucket 24312 creation permissions. Before v1.52.0 this would have passed silently due 24313 to a bug. 24314 24315 Properties: 24316 24317 - Config: no_check_bucket 24318 - Env Var: RCLONE_S3_NO_CHECK_BUCKET 24319 - Type: bool 24320 - Default: false 24321 24322 --s3-no-head 24323 24324 If set, don't HEAD uploaded objects to check integrity. 24325 24326 This can be useful when trying to minimise the number of transactions 24327 rclone does. 24328 24329 Setting it means that if rclone receives a 200 OK message after 24330 uploading an object with PUT then it will assume that it got uploaded 24331 properly. 24332 24333 In particular it will assume: 24334 24335 - the metadata, including modtime, storage class and content type was 24336 as uploaded 24337 - the size was as uploaded 24338 24339 It reads the following items from the response for a single part PUT: 24340 24341 - the MD5SUM 24342 - The uploaded date 24343 24344 For multipart uploads these items aren't read. 24345 24346 If an source object of unknown length is uploaded then rclone will do a 24347 HEAD request. 24348 24349 Setting this flag increases the chance for undetected upload failures, 24350 in particular an incorrect size, so it isn't recommended for normal 24351 operation. In practice the chance of an undetected upload failure is 24352 very small even with this flag. 24353 24354 Properties: 24355 24356 - Config: no_head 24357 - Env Var: RCLONE_S3_NO_HEAD 24358 - Type: bool 24359 - Default: false 24360 24361 --s3-no-head-object 24362 24363 If set, do not do HEAD before GET when getting objects. 24364 24365 Properties: 24366 24367 - Config: no_head_object 24368 - Env Var: RCLONE_S3_NO_HEAD_OBJECT 24369 - Type: bool 24370 - Default: false 24371 24372 --s3-encoding 24373 24374 The encoding for the backend. 24375 24376 See the encoding section in the overview for more info. 24377 24378 Properties: 24379 24380 - Config: encoding 24381 - Env Var: RCLONE_S3_ENCODING 24382 - Type: Encoding 24383 - Default: Slash,InvalidUtf8,Dot 24384 24385 --s3-memory-pool-flush-time 24386 24387 How often internal memory buffer pools will be flushed. (no longer used) 24388 24389 Properties: 24390 24391 - Config: memory_pool_flush_time 24392 - Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME 24393 - Type: Duration 24394 - Default: 1m0s 24395 24396 --s3-memory-pool-use-mmap 24397 24398 Whether to use mmap buffers in internal memory pool. (no longer used) 24399 24400 Properties: 24401 24402 - Config: memory_pool_use_mmap 24403 - Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP 24404 - Type: bool 24405 - Default: false 24406 24407 --s3-disable-http2 24408 24409 Disable usage of http2 for S3 backends. 24410 24411 There is currently an unsolved issue with the s3 (specifically minio) 24412 backend and HTTP/2. HTTP/2 is enabled by default for the s3 backend but 24413 can be disabled here. When the issue is solved this flag will be 24414 removed. 24415 24416 See: https://github.com/rclone/rclone/issues/4673, 24417 https://github.com/rclone/rclone/issues/3631 24418 24419 Properties: 24420 24421 - Config: disable_http2 24422 - Env Var: RCLONE_S3_DISABLE_HTTP2 24423 - Type: bool 24424 - Default: false 24425 24426 --s3-download-url 24427 24428 Custom endpoint for downloads. This is usually set to a CloudFront CDN 24429 URL as AWS S3 offers cheaper egress for data downloaded through the 24430 CloudFront network. 24431 24432 Properties: 24433 24434 - Config: download_url 24435 - Env Var: RCLONE_S3_DOWNLOAD_URL 24436 - Type: string 24437 - Required: false 24438 24439 --s3-directory-markers 24440 24441 Upload an empty object with a trailing slash when a new directory is 24442 created 24443 24444 Empty folders are unsupported for bucket based remotes, this option 24445 creates an empty object ending with "/", to persist the folder. 24446 24447 Properties: 24448 24449 - Config: directory_markers 24450 - Env Var: RCLONE_S3_DIRECTORY_MARKERS 24451 - Type: bool 24452 - Default: false 24453 24454 --s3-use-multipart-etag 24455 24456 Whether to use ETag in multipart uploads for verification 24457 24458 This should be true, false or left unset to use the default for the 24459 provider. 24460 24461 Properties: 24462 24463 - Config: use_multipart_etag 24464 - Env Var: RCLONE_S3_USE_MULTIPART_ETAG 24465 - Type: Tristate 24466 - Default: unset 24467 24468 --s3-use-presigned-request 24469 24470 Whether to use a presigned request or PutObject for single part uploads 24471 24472 If this is false rclone will use PutObject from the AWS SDK to upload an 24473 object. 24474 24475 Versions of rclone < 1.59 use presigned requests to upload a single part 24476 object and setting this flag to true will re-enable that functionality. 24477 This shouldn't be necessary except in exceptional circumstances or for 24478 testing. 24479 24480 Properties: 24481 24482 - Config: use_presigned_request 24483 - Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST 24484 - Type: bool 24485 - Default: false 24486 24487 --s3-versions 24488 24489 Include old versions in directory listings. 24490 24491 Properties: 24492 24493 - Config: versions 24494 - Env Var: RCLONE_S3_VERSIONS 24495 - Type: bool 24496 - Default: false 24497 24498 --s3-version-at 24499 24500 Show file versions as they were at the specified time. 24501 24502 The parameter should be a date, "2006-01-02", datetime "2006-01-02 24503 15:04:05" or a duration for that long ago, eg "100d" or "1h". 24504 24505 Note that when using this no file write operations are permitted, so you 24506 can't upload files or delete them. 24507 24508 See the time option docs for valid formats. 24509 24510 Properties: 24511 24512 - Config: version_at 24513 - Env Var: RCLONE_S3_VERSION_AT 24514 - Type: Time 24515 - Default: off 24516 24517 --s3-version-deleted 24518 24519 Show deleted file markers when using versions. 24520 24521 This shows deleted file markers in the listing when using versions. 24522 These will appear as 0 size files. The only operation which can be 24523 performed on them is deletion. 24524 24525 Deleting a delete marker will reveal the previous version. 24526 24527 Deleted files will always show with a timestamp. 24528 24529 Properties: 24530 24531 - Config: version_deleted 24532 - Env Var: RCLONE_S3_VERSION_DELETED 24533 - Type: bool 24534 - Default: false 24535 24536 --s3-decompress 24537 24538 If set this will decompress gzip encoded objects. 24539 24540 It is possible to upload objects to S3 with "Content-Encoding: gzip" 24541 set. Normally rclone will download these files as compressed objects. 24542 24543 If this flag is set then rclone will decompress these files with 24544 "Content-Encoding: gzip" as they are received. This means that rclone 24545 can't check the size and hash but the file contents will be 24546 decompressed. 24547 24548 Properties: 24549 24550 - Config: decompress 24551 - Env Var: RCLONE_S3_DECOMPRESS 24552 - Type: bool 24553 - Default: false 24554 24555 --s3-might-gzip 24556 24557 Set this if the backend might gzip objects. 24558 24559 Normally providers will not alter objects when they are downloaded. If 24560 an object was not uploaded with Content-Encoding: gzip then it won't be 24561 set on download. 24562 24563 However some providers may gzip objects even if they weren't uploaded 24564 with Content-Encoding: gzip (eg Cloudflare). 24565 24566 A symptom of this would be receiving errors like 24567 24568 ERROR corrupted on transfer: sizes differ NNN vs MMM 24569 24570 If you set this flag and rclone downloads an object with 24571 Content-Encoding: gzip set and chunked transfer encoding, then rclone 24572 will decompress the object on the fly. 24573 24574 If this is set to unset (the default) then rclone will choose according 24575 to the provider setting what to apply, but you can override rclone's 24576 choice here. 24577 24578 Properties: 24579 24580 - Config: might_gzip 24581 - Env Var: RCLONE_S3_MIGHT_GZIP 24582 - Type: Tristate 24583 - Default: unset 24584 24585 --s3-use-accept-encoding-gzip 24586 24587 Whether to send Accept-Encoding: gzip header. 24588 24589 By default, rclone will append Accept-Encoding: gzip to the request to 24590 download compressed objects whenever possible. 24591 24592 However some providers such as Google Cloud Storage may alter the HTTP 24593 headers, breaking the signature of the request. 24594 24595 A symptom of this would be receiving errors like 24596 24597 SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. 24598 24599 In this case, you might want to try disabling this option. 24600 24601 Properties: 24602 24603 - Config: use_accept_encoding_gzip 24604 - Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP 24605 - Type: Tristate 24606 - Default: unset 24607 24608 --s3-no-system-metadata 24609 24610 Suppress setting and reading of system metadata 24611 24612 Properties: 24613 24614 - Config: no_system_metadata 24615 - Env Var: RCLONE_S3_NO_SYSTEM_METADATA 24616 - Type: bool 24617 - Default: false 24618 24619 --s3-sts-endpoint 24620 24621 Endpoint for STS. 24622 24623 Leave blank if using AWS to use the default endpoint for the region. 24624 24625 Properties: 24626 24627 - Config: sts_endpoint 24628 - Env Var: RCLONE_S3_STS_ENDPOINT 24629 - Provider: AWS 24630 - Type: string 24631 - Required: false 24632 24633 --s3-use-already-exists 24634 24635 Set if rclone should report BucketAlreadyExists errors on bucket 24636 creation. 24637 24638 At some point during the evolution of the s3 protocol, AWS started 24639 returning an AlreadyOwnedByYou error when attempting to create a bucket 24640 that the user already owned, rather than a BucketAlreadyExists error. 24641 24642 Unfortunately exactly what has been implemented by s3 clones is a little 24643 inconsistent, some return AlreadyOwnedByYou, some return 24644 BucketAlreadyExists and some return no error at all. 24645 24646 This is important to rclone because it ensures the bucket exists by 24647 creating it on quite a lot of operations (unless --s3-no-check-bucket is 24648 used). 24649 24650 If rclone knows the provider can return AlreadyOwnedByYou or returns no 24651 error then it can report BucketAlreadyExists errors when the user 24652 attempts to create a bucket not owned by them. Otherwise rclone ignores 24653 the BucketAlreadyExists error which can lead to confusion. 24654 24655 This should be automatically set correctly for all providers rclone 24656 knows about - please make a bug report if not. 24657 24658 Properties: 24659 24660 - Config: use_already_exists 24661 - Env Var: RCLONE_S3_USE_ALREADY_EXISTS 24662 - Type: Tristate 24663 - Default: unset 24664 24665 --s3-use-multipart-uploads 24666 24667 Set if rclone should use multipart uploads. 24668 24669 You can change this if you want to disable the use of multipart uploads. 24670 This shouldn't be necessary in normal operation. 24671 24672 This should be automatically set correctly for all providers rclone 24673 knows about - please make a bug report if not. 24674 24675 Properties: 24676 24677 - Config: use_multipart_uploads 24678 - Env Var: RCLONE_S3_USE_MULTIPART_UPLOADS 24679 - Type: Tristate 24680 - Default: unset 24681 24682 --s3-description 24683 24684 Description of the remote 24685 24686 Properties: 24687 24688 - Config: description 24689 - Env Var: RCLONE_S3_DESCRIPTION 24690 - Type: string 24691 - Required: false 24692 24693 Metadata 24694 24695 User metadata is stored as x-amz-meta- keys. S3 metadata keys are case 24696 insensitive and are always returned in lower case. 24697 24698 Here are the possible system metadata items for the s3 backend. 24699 24700 ------------------------------------------------------------------------------------------------------------------ 24701 Name Help Type Example Read Only 24702 --------------------- --------------------- ----------- ------------------------------------- -------------------- 24703 btime Time of file birth RFC 3339 2006-01-02T15:04:05.999999999Z07:00 Y 24704 (creation) read from 24705 Last-Modified header 24706 24707 cache-control Cache-Control header string no-cache N 24708 24709 content-disposition Content-Disposition string inline N 24710 header 24711 24712 content-encoding Content-Encoding string gzip N 24713 header 24714 24715 content-language Content-Language string en-US N 24716 header 24717 24718 content-type Content-Type header string text/plain N 24719 24720 mtime Time of last RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N 24721 modification, read 24722 from rclone metadata 24723 24724 tier Tier of the object string GLACIER Y 24725 ------------------------------------------------------------------------------------------------------------------ 24726 24727 See the metadata docs for more info. 24728 24729 Backend commands 24730 24731 Here are the commands specific to the s3 backend. 24732 24733 Run them with 24734 24735 rclone backend COMMAND remote: 24736 24737 The help below will explain what arguments each command takes. 24738 24739 See the backend command for more info on how to pass options and 24740 arguments. 24741 24742 These can be run on a running backend using the rc command 24743 backend/command. 24744 24745 restore 24746 24747 Restore objects from GLACIER to normal storage 24748 24749 rclone backend restore remote: [options] [<arguments>+] 24750 24751 This command can be used to restore one or more objects from GLACIER to 24752 normal storage. 24753 24754 Usage Examples: 24755 24756 rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS 24757 rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS 24758 rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS 24759 24760 This flag also obeys the filters. Test first with --interactive/-i or 24761 --dry-run flags 24762 24763 rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1 24764 24765 All the objects shown will be marked for restore, then 24766 24767 rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1 24768 24769 It returns a list of status dictionaries with Remote and Status keys. 24770 The Status will be OK if it was successful or an error message if not. 24771 24772 [ 24773 { 24774 "Status": "OK", 24775 "Remote": "test.txt" 24776 }, 24777 { 24778 "Status": "OK", 24779 "Remote": "test/file4.txt" 24780 } 24781 ] 24782 24783 Options: 24784 24785 - "description": The optional description for the job. 24786 - "lifetime": Lifetime of the active copy in days 24787 - "priority": Priority of restore: Standard|Expedited|Bulk 24788 24789 restore-status 24790 24791 Show the restore status for objects being restored from GLACIER to 24792 normal storage 24793 24794 rclone backend restore-status remote: [options] [<arguments>+] 24795 24796 This command can be used to show the status for objects being restored 24797 from GLACIER to normal storage. 24798 24799 Usage Examples: 24800 24801 rclone backend restore-status s3:bucket/path/to/object 24802 rclone backend restore-status s3:bucket/path/to/directory 24803 rclone backend restore-status -o all s3:bucket/path/to/directory 24804 24805 This command does not obey the filters. 24806 24807 It returns a list of status dictionaries. 24808 24809 [ 24810 { 24811 "Remote": "file.txt", 24812 "VersionID": null, 24813 "RestoreStatus": { 24814 "IsRestoreInProgress": true, 24815 "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" 24816 }, 24817 "StorageClass": "GLACIER" 24818 }, 24819 { 24820 "Remote": "test.pdf", 24821 "VersionID": null, 24822 "RestoreStatus": { 24823 "IsRestoreInProgress": false, 24824 "RestoreExpiryDate": "2023-09-06T12:29:19+01:00" 24825 }, 24826 "StorageClass": "DEEP_ARCHIVE" 24827 } 24828 ] 24829 24830 Options: 24831 24832 - "all": if set then show all objects, not just ones with restore 24833 status 24834 24835 list-multipart-uploads 24836 24837 List the unfinished multipart uploads 24838 24839 rclone backend list-multipart-uploads remote: [options] [<arguments>+] 24840 24841 This command lists the unfinished multipart uploads in JSON format. 24842 24843 rclone backend list-multipart s3:bucket/path/to/object 24844 24845 It returns a dictionary of buckets with values as lists of unfinished 24846 multipart uploads. 24847 24848 You can call it with no bucket in which case it lists all bucket, with a 24849 bucket or with a bucket and path. 24850 24851 { 24852 "rclone": [ 24853 { 24854 "Initiated": "2020-06-26T14:20:36Z", 24855 "Initiator": { 24856 "DisplayName": "XXX", 24857 "ID": "arn:aws:iam::XXX:user/XXX" 24858 }, 24859 "Key": "KEY", 24860 "Owner": { 24861 "DisplayName": null, 24862 "ID": "XXX" 24863 }, 24864 "StorageClass": "STANDARD", 24865 "UploadId": "XXX" 24866 } 24867 ], 24868 "rclone-1000files": [], 24869 "rclone-dst": [] 24870 } 24871 24872 cleanup 24873 24874 Remove unfinished multipart uploads. 24875 24876 rclone backend cleanup remote: [options] [<arguments>+] 24877 24878 This command removes unfinished multipart uploads of age greater than 24879 max-age which defaults to 24 hours. 24880 24881 Note that you can use --interactive/-i or --dry-run with this command to 24882 see what it would do. 24883 24884 rclone backend cleanup s3:bucket/path/to/object 24885 rclone backend cleanup -o max-age=7w s3:bucket/path/to/object 24886 24887 Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc. 24888 24889 Options: 24890 24891 - "max-age": Max age of upload to delete 24892 24893 cleanup-hidden 24894 24895 Remove old versions of files. 24896 24897 rclone backend cleanup-hidden remote: [options] [<arguments>+] 24898 24899 This command removes any old hidden versions of files on a versions 24900 enabled bucket. 24901 24902 Note that you can use --interactive/-i or --dry-run with this command to 24903 see what it would do. 24904 24905 rclone backend cleanup-hidden s3:bucket/path/to/dir 24906 24907 versioning 24908 24909 Set/get versioning support for a bucket. 24910 24911 rclone backend versioning remote: [options] [<arguments>+] 24912 24913 This command sets versioning support if a parameter is passed and then 24914 returns the current versioning status for the bucket supplied. 24915 24916 rclone backend versioning s3:bucket # read status only 24917 rclone backend versioning s3:bucket Enabled 24918 rclone backend versioning s3:bucket Suspended 24919 24920 It may return "Enabled", "Suspended" or "Unversioned". Note that once 24921 versioning has been enabled the status can't be set back to 24922 "Unversioned". 24923 24924 set 24925 24926 Set command for updating the config parameters. 24927 24928 rclone backend set remote: [options] [<arguments>+] 24929 24930 This set command can be used to update the config parameters for a 24931 running s3 backend. 24932 24933 Usage Examples: 24934 24935 rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2] 24936 rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2] 24937 rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X 24938 24939 The option keys are named as they are in the config file. 24940 24941 This rebuilds the connection to the s3 backend when it is called with 24942 the new parameters. Only new parameters need be passed as the values 24943 will default to those currently in use. 24944 24945 It doesn't return anything. 24946 24947 Anonymous access to public buckets 24948 24949 If you want to use rclone to access a public bucket, configure with a 24950 blank access_key_id and secret_access_key. Your config should end up 24951 looking like this: 24952 24953 [anons3] 24954 type = s3 24955 provider = AWS 24956 env_auth = false 24957 access_key_id = 24958 secret_access_key = 24959 region = us-east-1 24960 endpoint = 24961 location_constraint = 24962 acl = private 24963 server_side_encryption = 24964 storage_class = 24965 24966 Then use it as normal with the name of the public bucket, e.g. 24967 24968 rclone lsd anons3:1000genomes 24969 24970 You will be able to list and copy data but not upload it. 24971 24972 Providers 24973 24974 AWS S3 24975 24976 This is the provider used as main example and described in the 24977 configuration section above. 24978 24979 AWS Snowball Edge 24980 24981 AWS Snowball is a hardware appliance used for transferring bulk data 24982 back to AWS. Its main software interface is S3 object storage. 24983 24984 To use rclone with AWS Snowball Edge devices, configure as standard for 24985 an 'S3 Compatible Service'. 24986 24987 If using rclone pre v1.59 be sure to set upload_cutoff = 0 otherwise you 24988 will run into authentication header issues as the snowball device does 24989 not support query parameter based authentication. 24990 24991 With rclone v1.59 or later setting upload_cutoff should not be 24992 necessary. 24993 24994 eg. 24995 24996 [snowball] 24997 type = s3 24998 provider = Other 24999 access_key_id = YOUR_ACCESS_KEY 25000 secret_access_key = YOUR_SECRET_KEY 25001 endpoint = http://[IP of Snowball]:8080 25002 upload_cutoff = 0 25003 25004 Ceph 25005 25006 Ceph is an open-source, unified, distributed storage system designed for 25007 excellent performance, reliability and scalability. It has an S3 25008 compatible object storage interface. 25009 25010 To use rclone with Ceph, configure as above but leave the region blank 25011 and set the endpoint. You should end up with something like this in your 25012 config: 25013 25014 [ceph] 25015 type = s3 25016 provider = Ceph 25017 env_auth = false 25018 access_key_id = XXX 25019 secret_access_key = YYY 25020 region = 25021 endpoint = https://ceph.endpoint.example.com 25022 location_constraint = 25023 acl = 25024 server_side_encryption = 25025 storage_class = 25026 25027 If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a 25028 version of rclone before v1.59 then you may need to supply the parameter 25029 --s3-upload-cutoff 0 or put this in the config file as upload_cutoff 0 25030 to work around a bug which causes uploading of small files to fail. 25031 25032 Note also that Ceph sometimes puts / in the passwords it gives users. If 25033 you read the secret access key using the command line tools you will get 25034 a JSON blob with the / escaped as \/. Make sure you only write / in the 25035 secret access key. 25036 25037 Eg the dump from Ceph looks something like this (irrelevant keys 25038 removed). 25039 25040 { 25041 "user_id": "xxx", 25042 "display_name": "xxxx", 25043 "keys": [ 25044 { 25045 "user": "xxx", 25046 "access_key": "xxxxxx", 25047 "secret_key": "xxxxxx\/xxxx" 25048 } 25049 ], 25050 } 25051 25052 Because this is a json dump, it is encoding the / as \/, so if you use 25053 the secret key as xxxxxx/xxxx it will work fine. 25054 25055 Cloudflare R2 25056 25057 Cloudflare R2 Storage allows developers to store large amounts of 25058 unstructured data without the costly egress bandwidth fees associated 25059 with typical cloud storage services. 25060 25061 Here is an example of making a Cloudflare R2 configuration. First run: 25062 25063 rclone config 25064 25065 This will guide you through an interactive setup process. 25066 25067 Note that all buckets are private, and all are stored in the same "auto" 25068 region. It is necessary to use Cloudflare workers to share the content 25069 of a bucket publicly. 25070 25071 No remotes found, make a new one? 25072 n) New remote 25073 s) Set configuration password 25074 q) Quit config 25075 n/s/q> n 25076 name> r2 25077 Option Storage. 25078 Type of storage to configure. 25079 Choose a number from below, or type in your own value. 25080 ... 25081 XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi 25082 \ (s3) 25083 ... 25084 Storage> s3 25085 Option provider. 25086 Choose your S3 provider. 25087 Choose a number from below, or type in your own value. 25088 Press Enter to leave empty. 25089 ... 25090 XX / Cloudflare R2 Storage 25091 \ (Cloudflare) 25092 ... 25093 provider> Cloudflare 25094 Option env_auth. 25095 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 25096 Only applies if access_key_id and secret_access_key is blank. 25097 Choose a number from below, or type in your own boolean value (true or false). 25098 Press Enter for the default (false). 25099 1 / Enter AWS credentials in the next step. 25100 \ (false) 25101 2 / Get AWS credentials from the environment (env vars or IAM). 25102 \ (true) 25103 env_auth> 1 25104 Option access_key_id. 25105 AWS Access Key ID. 25106 Leave blank for anonymous access or runtime credentials. 25107 Enter a value. Press Enter to leave empty. 25108 access_key_id> ACCESS_KEY 25109 Option secret_access_key. 25110 AWS Secret Access Key (password). 25111 Leave blank for anonymous access or runtime credentials. 25112 Enter a value. Press Enter to leave empty. 25113 secret_access_key> SECRET_ACCESS_KEY 25114 Option region. 25115 Region to connect to. 25116 Choose a number from below, or type in your own value. 25117 Press Enter to leave empty. 25118 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency. 25119 \ (auto) 25120 region> 1 25121 Option endpoint. 25122 Endpoint for S3 API. 25123 Required when using an S3 clone. 25124 Enter a value. Press Enter to leave empty. 25125 endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com 25126 Edit advanced config? 25127 y) Yes 25128 n) No (default) 25129 y/n> n 25130 -------------------- 25131 y) Yes this is OK (default) 25132 e) Edit this remote 25133 d) Delete this remote 25134 y/e/d> y 25135 25136 This will leave your config looking something like: 25137 25138 [r2] 25139 type = s3 25140 provider = Cloudflare 25141 access_key_id = ACCESS_KEY 25142 secret_access_key = SECRET_ACCESS_KEY 25143 region = auto 25144 endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com 25145 acl = private 25146 25147 Now run rclone lsf r2: to see your buckets and rclone lsf r2:bucket to 25148 look within a bucket. 25149 25150 Dreamhost 25151 25152 Dreamhost DreamObjects is an object storage system based on CEPH. 25153 25154 To use rclone with Dreamhost, configure as above but leave the region 25155 blank and set the endpoint. You should end up with something like this 25156 in your config: 25157 25158 [dreamobjects] 25159 type = s3 25160 provider = DreamHost 25161 env_auth = false 25162 access_key_id = your_access_key 25163 secret_access_key = your_secret_key 25164 region = 25165 endpoint = objects-us-west-1.dream.io 25166 location_constraint = 25167 acl = private 25168 server_side_encryption = 25169 storage_class = 25170 25171 Google Cloud Storage 25172 25173 GoogleCloudStorage is an S3-interoperable object storage service from 25174 Google Cloud Platform. 25175 25176 To connect to Google Cloud Storage you will need an access key and 25177 secret key. These can be retrieved by creating an HMAC key. 25178 25179 [gs] 25180 type = s3 25181 provider = GCS 25182 access_key_id = your_access_key 25183 secret_access_key = your_secret_key 25184 endpoint = https://storage.googleapis.com 25185 25186 Note that --s3-versions does not work with GCS when it needs to do 25187 directory paging. Rclone will return the error: 25188 25189 s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker 25190 25191 This is Google bug #312292516. 25192 25193 DigitalOcean Spaces 25194 25195 Spaces is an S3-interoperable object storage service from cloud provider 25196 DigitalOcean. 25197 25198 To connect to DigitalOcean Spaces you will need an access key and secret 25199 key. These can be retrieved on the "Applications & API" page of the 25200 DigitalOcean control panel. They will be needed when prompted by 25201 rclone config for your access_key_id and secret_access_key. 25202 25203 When prompted for a region or location_constraint, press enter to use 25204 the default value. The region must be included in the endpoint setting 25205 (e.g. nyc3.digitaloceanspaces.com). The default values can be used for 25206 other settings. 25207 25208 Going through the whole process of creating a new remote by running 25209 rclone config, each prompt should be answered as shown below: 25210 25211 Storage> s3 25212 env_auth> 1 25213 access_key_id> YOUR_ACCESS_KEY 25214 secret_access_key> YOUR_SECRET_KEY 25215 region> 25216 endpoint> nyc3.digitaloceanspaces.com 25217 location_constraint> 25218 acl> 25219 storage_class> 25220 25221 The resulting configuration file should look like: 25222 25223 [spaces] 25224 type = s3 25225 provider = DigitalOcean 25226 env_auth = false 25227 access_key_id = YOUR_ACCESS_KEY 25228 secret_access_key = YOUR_SECRET_KEY 25229 region = 25230 endpoint = nyc3.digitaloceanspaces.com 25231 location_constraint = 25232 acl = 25233 server_side_encryption = 25234 storage_class = 25235 25236 Once configured, you can create a new Space and begin copying files. For 25237 example: 25238 25239 rclone mkdir spaces:my-new-space 25240 rclone copy /path/to/files spaces:my-new-space 25241 25242 Huawei OBS 25243 25244 Object Storage Service (OBS) provides stable, secure, efficient, and 25245 easy-to-use cloud storage that lets you store virtually any volume of 25246 unstructured data in any format and access it from anywhere. 25247 25248 OBS provides an S3 interface, you can copy and modify the following 25249 configuration and add it to your rclone configuration file. 25250 25251 [obs] 25252 type = s3 25253 provider = HuaweiOBS 25254 access_key_id = your-access-key-id 25255 secret_access_key = your-secret-access-key 25256 region = af-south-1 25257 endpoint = obs.af-south-1.myhuaweicloud.com 25258 acl = private 25259 25260 Or you can also configure via the interactive command line: 25261 25262 No remotes found, make a new one? 25263 n) New remote 25264 s) Set configuration password 25265 q) Quit config 25266 n/s/q> n 25267 name> obs 25268 Option Storage. 25269 Type of storage to configure. 25270 Choose a number from below, or type in your own value. 25271 [snip] 25272 XX / Amazon S3 Compliant Storage Providers including AWS, ... 25273 \ (s3) 25274 [snip] 25275 Storage> s3 25276 Option provider. 25277 Choose your S3 provider. 25278 Choose a number from below, or type in your own value. 25279 Press Enter to leave empty. 25280 [snip] 25281 9 / Huawei Object Storage Service 25282 \ (HuaweiOBS) 25283 [snip] 25284 provider> 9 25285 Option env_auth. 25286 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 25287 Only applies if access_key_id and secret_access_key is blank. 25288 Choose a number from below, or type in your own boolean value (true or false). 25289 Press Enter for the default (false). 25290 1 / Enter AWS credentials in the next step. 25291 \ (false) 25292 2 / Get AWS credentials from the environment (env vars or IAM). 25293 \ (true) 25294 env_auth> 1 25295 Option access_key_id. 25296 AWS Access Key ID. 25297 Leave blank for anonymous access or runtime credentials. 25298 Enter a value. Press Enter to leave empty. 25299 access_key_id> your-access-key-id 25300 Option secret_access_key. 25301 AWS Secret Access Key (password). 25302 Leave blank for anonymous access or runtime credentials. 25303 Enter a value. Press Enter to leave empty. 25304 secret_access_key> your-secret-access-key 25305 Option region. 25306 Region to connect to. 25307 Choose a number from below, or type in your own value. 25308 Press Enter to leave empty. 25309 1 / AF-Johannesburg 25310 \ (af-south-1) 25311 2 / AP-Bangkok 25312 \ (ap-southeast-2) 25313 [snip] 25314 region> 1 25315 Option endpoint. 25316 Endpoint for OBS API. 25317 Choose a number from below, or type in your own value. 25318 Press Enter to leave empty. 25319 1 / AF-Johannesburg 25320 \ (obs.af-south-1.myhuaweicloud.com) 25321 2 / AP-Bangkok 25322 \ (obs.ap-southeast-2.myhuaweicloud.com) 25323 [snip] 25324 endpoint> 1 25325 Option acl. 25326 Canned ACL used when creating buckets and storing or copying objects. 25327 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 25328 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 25329 Note that this ACL is applied when server-side copying objects as S3 25330 doesn't copy the ACL from the source but rather writes a fresh one. 25331 Choose a number from below, or type in your own value. 25332 Press Enter to leave empty. 25333 / Owner gets FULL_CONTROL. 25334 1 | No one else has access rights (default). 25335 \ (private) 25336 [snip] 25337 acl> 1 25338 Edit advanced config? 25339 y) Yes 25340 n) No (default) 25341 y/n> 25342 -------------------- 25343 [obs] 25344 type = s3 25345 provider = HuaweiOBS 25346 access_key_id = your-access-key-id 25347 secret_access_key = your-secret-access-key 25348 region = af-south-1 25349 endpoint = obs.af-south-1.myhuaweicloud.com 25350 acl = private 25351 -------------------- 25352 y) Yes this is OK (default) 25353 e) Edit this remote 25354 d) Delete this remote 25355 y/e/d> y 25356 Current remotes: 25357 25358 Name Type 25359 ==== ==== 25360 obs s3 25361 25362 e) Edit existing remote 25363 n) New remote 25364 d) Delete remote 25365 r) Rename remote 25366 c) Copy remote 25367 s) Set configuration password 25368 q) Quit config 25369 e/n/d/r/c/s/q> q 25370 25371 IBM COS (S3) 25372 25373 Information stored with IBM Cloud Object Storage is encrypted and 25374 dispersed across multiple geographic locations, and accessed through an 25375 implementation of the S3 API. This service makes use of the distributed 25376 storage technologies provided by IBM’s Cloud Object Storage System 25377 (formerly Cleversafe). For more information visit: 25378 (http://www.ibm.com/cloud/object-storage) 25379 25380 To configure access to IBM COS S3, follow the steps below: 25381 25382 1. Run rclone config and select n for a new remote. 25383 25384 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults 25385 No remotes found, make a new one? 25386 n) New remote 25387 s) Set configuration password 25388 q) Quit config 25389 n/s/q> n 25390 25391 2. Enter the name for the configuration 25392 25393 name> <YOUR NAME> 25394 25395 3. Select "s3" storage. 25396 25397 Choose a number from below, or type in your own value 25398 [snip] 25399 XX / Amazon S3 Compliant Storage Providers including AWS, ... 25400 \ "s3" 25401 [snip] 25402 Storage> s3 25403 25404 4. Select IBM COS as the S3 Storage Provider. 25405 25406 Choose the S3 provider. 25407 Choose a number from below, or type in your own value 25408 1 / Choose this option to configure Storage to AWS S3 25409 \ "AWS" 25410 2 / Choose this option to configure Storage to Ceph Systems 25411 \ "Ceph" 25412 3 / Choose this option to configure Storage to Dreamhost 25413 \ "Dreamhost" 25414 4 / Choose this option to the configure Storage to IBM COS S3 25415 \ "IBMCOS" 25416 5 / Choose this option to the configure Storage to Minio 25417 \ "Minio" 25418 Provider>4 25419 25420 5. Enter the Access Key and Secret. 25421 25422 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 25423 access_key_id> <> 25424 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 25425 secret_access_key> <> 25426 25427 6. Specify the endpoint for IBM COS. For Public IBM COS, choose from 25428 the option below. For On Premise IBM COS, enter an endpoint address. 25429 25430 Endpoint for IBM COS S3 API. 25431 Specify if using an IBM COS On Premise. 25432 Choose a number from below, or type in your own value 25433 1 / US Cross Region Endpoint 25434 \ "s3-api.us-geo.objectstorage.softlayer.net" 25435 2 / US Cross Region Dallas Endpoint 25436 \ "s3-api.dal.us-geo.objectstorage.softlayer.net" 25437 3 / US Cross Region Washington DC Endpoint 25438 \ "s3-api.wdc-us-geo.objectstorage.softlayer.net" 25439 4 / US Cross Region San Jose Endpoint 25440 \ "s3-api.sjc-us-geo.objectstorage.softlayer.net" 25441 5 / US Cross Region Private Endpoint 25442 \ "s3-api.us-geo.objectstorage.service.networklayer.com" 25443 6 / US Cross Region Dallas Private Endpoint 25444 \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com" 25445 7 / US Cross Region Washington DC Private Endpoint 25446 \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com" 25447 8 / US Cross Region San Jose Private Endpoint 25448 \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com" 25449 9 / US Region East Endpoint 25450 \ "s3.us-east.objectstorage.softlayer.net" 25451 10 / US Region East Private Endpoint 25452 \ "s3.us-east.objectstorage.service.networklayer.com" 25453 11 / US Region South Endpoint 25454 [snip] 25455 34 / Toronto Single Site Private Endpoint 25456 \ "s3.tor01.objectstorage.service.networklayer.com" 25457 endpoint>1 25458 25459 7. Specify a IBM COS Location Constraint. The location constraint must 25460 match endpoint when using IBM Cloud Public. For on-prem COS, do not 25461 make a selection from this list, hit enter 25462 25463 1 / US Cross Region Standard 25464 \ "us-standard" 25465 2 / US Cross Region Vault 25466 \ "us-vault" 25467 3 / US Cross Region Cold 25468 \ "us-cold" 25469 4 / US Cross Region Flex 25470 \ "us-flex" 25471 5 / US East Region Standard 25472 \ "us-east-standard" 25473 6 / US East Region Vault 25474 \ "us-east-vault" 25475 7 / US East Region Cold 25476 \ "us-east-cold" 25477 8 / US East Region Flex 25478 \ "us-east-flex" 25479 9 / US South Region Standard 25480 \ "us-south-standard" 25481 10 / US South Region Vault 25482 \ "us-south-vault" 25483 [snip] 25484 32 / Toronto Flex 25485 \ "tor01-flex" 25486 location_constraint>1 25487 25488 9. Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and 25489 "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise 25490 COS supports all the canned ACLs. 25491 25492 Canned ACL used when creating buckets and/or storing objects in S3. 25493 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 25494 Choose a number from below, or type in your own value 25495 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS 25496 \ "private" 25497 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS 25498 \ "public-read" 25499 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS 25500 \ "public-read-write" 25501 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS 25502 \ "authenticated-read" 25503 acl> 1 25504 25505 12. Review the displayed configuration and accept to save the "remote" 25506 then quit. The config file should look like this 25507 25508 [xxx] 25509 type = s3 25510 Provider = IBMCOS 25511 access_key_id = xxx 25512 secret_access_key = yyy 25513 endpoint = s3-api.us-geo.objectstorage.softlayer.net 25514 location_constraint = us-standard 25515 acl = private 25516 25517 13. Execute rclone commands 25518 25519 1) Create a bucket. 25520 rclone mkdir IBM-COS-XREGION:newbucket 25521 2) List available buckets. 25522 rclone lsd IBM-COS-XREGION: 25523 -1 2017-11-08 21:16:22 -1 test 25524 -1 2018-02-14 20:16:39 -1 newbucket 25525 3) List contents of a bucket. 25526 rclone ls IBM-COS-XREGION:newbucket 25527 18685952 test.exe 25528 4) Copy a file from local to remote. 25529 rclone copy /Users/file.txt IBM-COS-XREGION:newbucket 25530 5) Copy a file from remote to local. 25531 rclone copy IBM-COS-XREGION:newbucket/file.txt . 25532 6) Delete a file on remote. 25533 rclone delete IBM-COS-XREGION:newbucket/file.txt 25534 25535 IDrive e2 25536 25537 Here is an example of making an IDrive e2 configuration. First run: 25538 25539 rclone config 25540 25541 This will guide you through an interactive setup process. 25542 25543 No remotes found, make a new one? 25544 n) New remote 25545 s) Set configuration password 25546 q) Quit config 25547 n/s/q> n 25548 25549 Enter name for new remote. 25550 name> e2 25551 25552 Option Storage. 25553 Type of storage to configure. 25554 Choose a number from below, or type in your own value. 25555 [snip] 25556 XX / Amazon S3 Compliant Storage Providers including AWS, ... 25557 \ (s3) 25558 [snip] 25559 Storage> s3 25560 25561 Option provider. 25562 Choose your S3 provider. 25563 Choose a number from below, or type in your own value. 25564 Press Enter to leave empty. 25565 [snip] 25566 XX / IDrive e2 25567 \ (IDrive) 25568 [snip] 25569 provider> IDrive 25570 25571 Option env_auth. 25572 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 25573 Only applies if access_key_id and secret_access_key is blank. 25574 Choose a number from below, or type in your own boolean value (true or false). 25575 Press Enter for the default (false). 25576 1 / Enter AWS credentials in the next step. 25577 \ (false) 25578 2 / Get AWS credentials from the environment (env vars or IAM). 25579 \ (true) 25580 env_auth> 25581 25582 Option access_key_id. 25583 AWS Access Key ID. 25584 Leave blank for anonymous access or runtime credentials. 25585 Enter a value. Press Enter to leave empty. 25586 access_key_id> YOUR_ACCESS_KEY 25587 25588 Option secret_access_key. 25589 AWS Secret Access Key (password). 25590 Leave blank for anonymous access or runtime credentials. 25591 Enter a value. Press Enter to leave empty. 25592 secret_access_key> YOUR_SECRET_KEY 25593 25594 Option acl. 25595 Canned ACL used when creating buckets and storing or copying objects. 25596 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 25597 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 25598 Note that this ACL is applied when server-side copying objects as S3 25599 doesn't copy the ACL from the source but rather writes a fresh one. 25600 Choose a number from below, or type in your own value. 25601 Press Enter to leave empty. 25602 / Owner gets FULL_CONTROL. 25603 1 | No one else has access rights (default). 25604 \ (private) 25605 / Owner gets FULL_CONTROL. 25606 2 | The AllUsers group gets READ access. 25607 \ (public-read) 25608 / Owner gets FULL_CONTROL. 25609 3 | The AllUsers group gets READ and WRITE access. 25610 | Granting this on a bucket is generally not recommended. 25611 \ (public-read-write) 25612 / Owner gets FULL_CONTROL. 25613 4 | The AuthenticatedUsers group gets READ access. 25614 \ (authenticated-read) 25615 / Object owner gets FULL_CONTROL. 25616 5 | Bucket owner gets READ access. 25617 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 25618 \ (bucket-owner-read) 25619 / Both the object owner and the bucket owner get FULL_CONTROL over the object. 25620 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 25621 \ (bucket-owner-full-control) 25622 acl> 25623 25624 Edit advanced config? 25625 y) Yes 25626 n) No (default) 25627 y/n> 25628 25629 Configuration complete. 25630 Options: 25631 - type: s3 25632 - provider: IDrive 25633 - access_key_id: YOUR_ACCESS_KEY 25634 - secret_access_key: YOUR_SECRET_KEY 25635 - endpoint: q9d9.la12.idrivee2-5.com 25636 Keep this "e2" remote? 25637 y) Yes this is OK (default) 25638 e) Edit this remote 25639 d) Delete this remote 25640 y/e/d> y 25641 25642 IONOS Cloud 25643 25644 IONOS S3 Object Storage is a service offered by IONOS for storing and 25645 accessing unstructured data. To connect to the service, you will need an 25646 access key and a secret key. These can be found in the Data Center 25647 Designer, by selecting Manager resources > Object Storage Key Manager. 25648 25649 Here is an example of a configuration. First, run rclone config. This 25650 will walk you through an interactive setup process. Type n to add the 25651 new remote, and then enter a name: 25652 25653 Enter name for new remote. 25654 name> ionos-fra 25655 25656 Type s3 to choose the connection type: 25657 25658 Option Storage. 25659 Type of storage to configure. 25660 Choose a number from below, or type in your own value. 25661 [snip] 25662 XX / Amazon S3 Compliant Storage Providers including AWS, ... 25663 \ (s3) 25664 [snip] 25665 Storage> s3 25666 25667 Type IONOS: 25668 25669 Option provider. 25670 Choose your S3 provider. 25671 Choose a number from below, or type in your own value. 25672 Press Enter to leave empty. 25673 [snip] 25674 XX / IONOS Cloud 25675 \ (IONOS) 25676 [snip] 25677 provider> IONOS 25678 25679 Press Enter to choose the default option 25680 Enter AWS credentials in the next step: 25681 25682 Option env_auth. 25683 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 25684 Only applies if access_key_id and secret_access_key is blank. 25685 Choose a number from below, or type in your own boolean value (true or false). 25686 Press Enter for the default (false). 25687 1 / Enter AWS credentials in the next step. 25688 \ (false) 25689 2 / Get AWS credentials from the environment (env vars or IAM). 25690 \ (true) 25691 env_auth> 25692 25693 Enter your Access Key and Secret key. These can be retrieved in the Data 25694 Center Designer, click on the menu “Manager resources” / "Object Storage 25695 Key Manager". 25696 25697 Option access_key_id. 25698 AWS Access Key ID. 25699 Leave blank for anonymous access or runtime credentials. 25700 Enter a value. Press Enter to leave empty. 25701 access_key_id> YOUR_ACCESS_KEY 25702 25703 Option secret_access_key. 25704 AWS Secret Access Key (password). 25705 Leave blank for anonymous access or runtime credentials. 25706 Enter a value. Press Enter to leave empty. 25707 secret_access_key> YOUR_SECRET_KEY 25708 25709 Choose the region where your bucket is located: 25710 25711 Option region. 25712 Region where your bucket will be created and your data stored. 25713 Choose a number from below, or type in your own value. 25714 Press Enter to leave empty. 25715 1 / Frankfurt, Germany 25716 \ (de) 25717 2 / Berlin, Germany 25718 \ (eu-central-2) 25719 3 / Logrono, Spain 25720 \ (eu-south-2) 25721 region> 2 25722 25723 Choose the endpoint from the same region: 25724 25725 Option endpoint. 25726 Endpoint for IONOS S3 Object Storage. 25727 Specify the endpoint from the same region. 25728 Choose a number from below, or type in your own value. 25729 Press Enter to leave empty. 25730 1 / Frankfurt, Germany 25731 \ (s3-eu-central-1.ionoscloud.com) 25732 2 / Berlin, Germany 25733 \ (s3-eu-central-2.ionoscloud.com) 25734 3 / Logrono, Spain 25735 \ (s3-eu-south-2.ionoscloud.com) 25736 endpoint> 1 25737 25738 Press Enter to choose the default option or choose the desired ACL 25739 setting: 25740 25741 Option acl. 25742 Canned ACL used when creating buckets and storing or copying objects. 25743 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 25744 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 25745 Note that this ACL is applied when server-side copying objects as S3 25746 doesn't copy the ACL from the source but rather writes a fresh one. 25747 Choose a number from below, or type in your own value. 25748 Press Enter to leave empty. 25749 / Owner gets FULL_CONTROL. 25750 1 | No one else has access rights (default). 25751 \ (private) 25752 / Owner gets FULL_CONTROL. 25753 [snip] 25754 acl> 25755 25756 Press Enter to skip the advanced config: 25757 25758 Edit advanced config? 25759 y) Yes 25760 n) No (default) 25761 y/n> 25762 25763 Press Enter to save the configuration, and then q to quit the 25764 configuration process: 25765 25766 Configuration complete. 25767 Options: 25768 - type: s3 25769 - provider: IONOS 25770 - access_key_id: YOUR_ACCESS_KEY 25771 - secret_access_key: YOUR_SECRET_KEY 25772 - endpoint: s3-eu-central-1.ionoscloud.com 25773 Keep this "ionos-fra" remote? 25774 y) Yes this is OK (default) 25775 e) Edit this remote 25776 d) Delete this remote 25777 y/e/d> y 25778 25779 Done! Now you can try some commands (for macOS, use ./rclone instead of 25780 rclone). 25781 25782 1) Create a bucket (the name must be unique within the whole IONOS S3) 25783 25784 rclone mkdir ionos-fra:my-bucket 25785 25786 2) List available buckets 25787 25788 rclone lsd ionos-fra: 25789 25790 4) Copy a file from local to remote 25791 25792 rclone copy /Users/file.txt ionos-fra:my-bucket 25793 25794 3) List contents of a bucket 25795 25796 rclone ls ionos-fra:my-bucket 25797 25798 5) Copy a file from remote to local 25799 25800 rclone copy ionos-fra:my-bucket/file.txt 25801 25802 Minio 25803 25804 Minio is an object storage server built for cloud application developers 25805 and devops. 25806 25807 It is very easy to install and provides an S3 compatible server which 25808 can be used by rclone. 25809 25810 To use it, install Minio following the instructions here. 25811 25812 When it configures itself Minio will print something like this 25813 25814 Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000 25815 AccessKey: USWUXHGYZQYFYFFIT3RE 25816 SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 25817 Region: us-east-1 25818 SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis 25819 25820 Browser Access: 25821 http://192.168.1.106:9000 http://172.23.0.1:9000 25822 25823 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide 25824 $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 25825 25826 Object API (Amazon S3 compatible): 25827 Go: https://docs.minio.io/docs/golang-client-quickstart-guide 25828 Java: https://docs.minio.io/docs/java-client-quickstart-guide 25829 Python: https://docs.minio.io/docs/python-client-quickstart-guide 25830 JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide 25831 .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide 25832 25833 Drive Capacity: 26 GiB Free, 165 GiB Total 25834 25835 These details need to go into rclone config like this. Note that it is 25836 important to put the region in as stated above. 25837 25838 env_auth> 1 25839 access_key_id> USWUXHGYZQYFYFFIT3RE 25840 secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 25841 region> us-east-1 25842 endpoint> http://192.168.1.106:9000 25843 location_constraint> 25844 server_side_encryption> 25845 25846 Which makes the config file look like this 25847 25848 [minio] 25849 type = s3 25850 provider = Minio 25851 env_auth = false 25852 access_key_id = USWUXHGYZQYFYFFIT3RE 25853 secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03 25854 region = us-east-1 25855 endpoint = http://192.168.1.106:9000 25856 location_constraint = 25857 server_side_encryption = 25858 25859 So once set up, for example, to copy files into a bucket 25860 25861 rclone copy /path/to/files minio:bucket 25862 25863 Qiniu Cloud Object Storage (Kodo) 25864 25865 Qiniu Cloud Object Storage (Kodo), a completely independent-researched 25866 core technology which is proven by repeated customer experience has 25867 occupied absolute leading market leader position. Kodo can be widely 25868 applied to mass data management. 25869 25870 To configure access to Qiniu Kodo, follow the steps below: 25871 25872 1. Run rclone config and select n for a new remote. 25873 25874 rclone config 25875 No remotes found, make a new one? 25876 n) New remote 25877 s) Set configuration password 25878 q) Quit config 25879 n/s/q> n 25880 25881 2. Give the name of the configuration. For example, name it 'qiniu'. 25882 25883 name> qiniu 25884 25885 3. Select s3 storage. 25886 25887 Choose a number from below, or type in your own value 25888 [snip] 25889 XX / Amazon S3 Compliant Storage Providers including AWS, ... 25890 \ (s3) 25891 [snip] 25892 Storage> s3 25893 25894 4. Select Qiniu provider. 25895 25896 Choose a number from below, or type in your own value 25897 1 / Amazon Web Services (AWS) S3 25898 \ "AWS" 25899 [snip] 25900 22 / Qiniu Object Storage (Kodo) 25901 \ (Qiniu) 25902 [snip] 25903 provider> Qiniu 25904 25905 5. Enter your SecretId and SecretKey of Qiniu Kodo. 25906 25907 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 25908 Only applies if access_key_id and secret_access_key is blank. 25909 Enter a boolean value (true or false). Press Enter for the default ("false"). 25910 Choose a number from below, or type in your own value 25911 1 / Enter AWS credentials in the next step 25912 \ "false" 25913 2 / Get AWS credentials from the environment (env vars or IAM) 25914 \ "true" 25915 env_auth> 1 25916 AWS Access Key ID. 25917 Leave blank for anonymous access or runtime credentials. 25918 Enter a string value. Press Enter for the default (""). 25919 access_key_id> AKIDxxxxxxxxxx 25920 AWS Secret Access Key (password) 25921 Leave blank for anonymous access or runtime credentials. 25922 Enter a string value. Press Enter for the default (""). 25923 secret_access_key> xxxxxxxxxxx 25924 25925 6. Select endpoint for Qiniu Kodo. This is the standard endpoint for 25926 different region. 25927 25928 / The default endpoint - a good choice if you are unsure. 25929 1 | East China Region 1. 25930 | Needs location constraint cn-east-1. 25931 \ (cn-east-1) 25932 / East China Region 2. 25933 2 | Needs location constraint cn-east-2. 25934 \ (cn-east-2) 25935 / North China Region 1. 25936 3 | Needs location constraint cn-north-1. 25937 \ (cn-north-1) 25938 / South China Region 1. 25939 4 | Needs location constraint cn-south-1. 25940 \ (cn-south-1) 25941 / North America Region. 25942 5 | Needs location constraint us-north-1. 25943 \ (us-north-1) 25944 / Southeast Asia Region 1. 25945 6 | Needs location constraint ap-southeast-1. 25946 \ (ap-southeast-1) 25947 / Northeast Asia Region 1. 25948 7 | Needs location constraint ap-northeast-1. 25949 \ (ap-northeast-1) 25950 [snip] 25951 endpoint> 1 25952 25953 Option endpoint. 25954 Endpoint for Qiniu Object Storage. 25955 Choose a number from below, or type in your own value. 25956 Press Enter to leave empty. 25957 1 / East China Endpoint 1 25958 \ (s3-cn-east-1.qiniucs.com) 25959 2 / East China Endpoint 2 25960 \ (s3-cn-east-2.qiniucs.com) 25961 3 / North China Endpoint 1 25962 \ (s3-cn-north-1.qiniucs.com) 25963 4 / South China Endpoint 1 25964 \ (s3-cn-south-1.qiniucs.com) 25965 5 / North America Endpoint 1 25966 \ (s3-us-north-1.qiniucs.com) 25967 6 / Southeast Asia Endpoint 1 25968 \ (s3-ap-southeast-1.qiniucs.com) 25969 7 / Northeast Asia Endpoint 1 25970 \ (s3-ap-northeast-1.qiniucs.com) 25971 endpoint> 1 25972 25973 Option location_constraint. 25974 Location constraint - must be set to match the Region. 25975 Used when creating buckets only. 25976 Choose a number from below, or type in your own value. 25977 Press Enter to leave empty. 25978 1 / East China Region 1 25979 \ (cn-east-1) 25980 2 / East China Region 2 25981 \ (cn-east-2) 25982 3 / North China Region 1 25983 \ (cn-north-1) 25984 4 / South China Region 1 25985 \ (cn-south-1) 25986 5 / North America Region 1 25987 \ (us-north-1) 25988 6 / Southeast Asia Region 1 25989 \ (ap-southeast-1) 25990 7 / Northeast Asia Region 1 25991 \ (ap-northeast-1) 25992 location_constraint> 1 25993 25994 7. Choose acl and storage class. 25995 25996 Note that this ACL is applied when server-side copying objects as S3 25997 doesn't copy the ACL from the source but rather writes a fresh one. 25998 Enter a string value. Press Enter for the default (""). 25999 Choose a number from below, or type in your own value 26000 / Owner gets FULL_CONTROL. 26001 1 | No one else has access rights (default). 26002 \ (private) 26003 / Owner gets FULL_CONTROL. 26004 2 | The AllUsers group gets READ access. 26005 \ (public-read) 26006 [snip] 26007 acl> 2 26008 The storage class to use when storing new objects in Tencent COS. 26009 Enter a string value. Press Enter for the default (""). 26010 Choose a number from below, or type in your own value 26011 1 / Standard storage class 26012 \ (STANDARD) 26013 2 / Infrequent access storage mode 26014 \ (LINE) 26015 3 / Archive storage mode 26016 \ (GLACIER) 26017 4 / Deep archive storage mode 26018 \ (DEEP_ARCHIVE) 26019 [snip] 26020 storage_class> 1 26021 Edit advanced config? (y/n) 26022 y) Yes 26023 n) No (default) 26024 y/n> n 26025 Remote config 26026 -------------------- 26027 [qiniu] 26028 - type: s3 26029 - provider: Qiniu 26030 - access_key_id: xxx 26031 - secret_access_key: xxx 26032 - region: cn-east-1 26033 - endpoint: s3-cn-east-1.qiniucs.com 26034 - location_constraint: cn-east-1 26035 - acl: public-read 26036 - storage_class: STANDARD 26037 -------------------- 26038 y) Yes this is OK (default) 26039 e) Edit this remote 26040 d) Delete this remote 26041 y/e/d> y 26042 Current remotes: 26043 26044 Name Type 26045 ==== ==== 26046 qiniu s3 26047 26048 RackCorp 26049 26050 RackCorp Object Storage is an S3 compatible object storage platform from 26051 your friendly cloud provider RackCorp. The service is fast, reliable, 26052 well priced and located in many strategic locations unserviced by 26053 others, to ensure you can maintain data sovereignty. 26054 26055 Before you can use RackCorp Object Storage, you'll need to "sign up" for 26056 an account on our "portal". Next you can create an access key, a 26057 secret key and buckets, in your location of choice with ease. These 26058 details are required for the next steps of configuration, when 26059 rclone config asks for your access_key_id and secret_access_key. 26060 26061 Your config should end up looking a bit like this: 26062 26063 [RCS3-demo-config] 26064 type = s3 26065 provider = RackCorp 26066 env_auth = true 26067 access_key_id = YOURACCESSKEY 26068 secret_access_key = YOURSECRETACCESSKEY 26069 region = au-nsw 26070 endpoint = s3.rackcorp.com 26071 location_constraint = au-nsw 26072 26073 Rclone Serve S3 26074 26075 Rclone can serve any remote over the S3 protocol. For details see the 26076 rclone serve s3 documentation. 26077 26078 For example, to serve remote:path over s3, run the server like this: 26079 26080 rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path 26081 26082 This will be compatible with an rclone remote which is defined like 26083 this: 26084 26085 [serves3] 26086 type = s3 26087 provider = Rclone 26088 endpoint = http://127.0.0.1:8080/ 26089 access_key_id = ACCESS_KEY_ID 26090 secret_access_key = SECRET_ACCESS_KEY 26091 use_multipart_uploads = false 26092 26093 Note that setting disable_multipart_uploads = true is to work around a 26094 bug which will be fixed in due course. 26095 26096 Scaleway 26097 26098 Scaleway The Object Storage platform allows you to store anything from 26099 backups, logs and web assets to documents and photos. Files can be 26100 dropped from the Scaleway console or transferred through our API and CLI 26101 or using any S3-compatible tool. 26102 26103 Scaleway provides an S3 interface which can be configured for use with 26104 rclone like this: 26105 26106 [scaleway] 26107 type = s3 26108 provider = Scaleway 26109 env_auth = false 26110 endpoint = s3.nl-ams.scw.cloud 26111 access_key_id = SCWXXXXXXXXXXXXXX 26112 secret_access_key = 1111111-2222-3333-44444-55555555555555 26113 region = nl-ams 26114 location_constraint = 26115 acl = private 26116 server_side_encryption = 26117 storage_class = 26118 26119 C14 Cold Storage is the low-cost S3 Glacier alternative from Scaleway 26120 and it works the same way as on S3 by accepting the "GLACIER" 26121 storage_class. So you can configure your remote with the 26122 storage_class = GLACIER option to upload directly to C14. Don't forget 26123 that in this state you can't read files back after, you will need to 26124 restore them to "STANDARD" storage_class first before being able to read 26125 them (see "restore" section above) 26126 26127 Seagate Lyve Cloud 26128 26129 Seagate Lyve Cloud is an S3 compatible object storage platform from 26130 Seagate intended for enterprise use. 26131 26132 Here is a config run through for a remote called remote - you may choose 26133 a different name of course. Note that to create an access key and secret 26134 key you will need to create a service account first. 26135 26136 $ rclone config 26137 No remotes found, make a new one? 26138 n) New remote 26139 s) Set configuration password 26140 q) Quit config 26141 n/s/q> n 26142 name> remote 26143 26144 Choose s3 backend 26145 26146 Type of storage to configure. 26147 Choose a number from below, or type in your own value. 26148 [snip] 26149 XX / Amazon S3 Compliant Storage Providers including AWS, ... 26150 \ (s3) 26151 [snip] 26152 Storage> s3 26153 26154 Choose LyveCloud as S3 provider 26155 26156 Choose your S3 provider. 26157 Choose a number from below, or type in your own value. 26158 Press Enter to leave empty. 26159 [snip] 26160 XX / Seagate Lyve Cloud 26161 \ (LyveCloud) 26162 [snip] 26163 provider> LyveCloud 26164 26165 Take the default (just press enter) to enter access key and secret in 26166 the config file. 26167 26168 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 26169 Only applies if access_key_id and secret_access_key is blank. 26170 Choose a number from below, or type in your own boolean value (true or false). 26171 Press Enter for the default (false). 26172 1 / Enter AWS credentials in the next step. 26173 \ (false) 26174 2 / Get AWS credentials from the environment (env vars or IAM). 26175 \ (true) 26176 env_auth> 26177 26178 AWS Access Key ID. 26179 Leave blank for anonymous access or runtime credentials. 26180 Enter a value. Press Enter to leave empty. 26181 access_key_id> XXX 26182 26183 AWS Secret Access Key (password). 26184 Leave blank for anonymous access or runtime credentials. 26185 Enter a value. Press Enter to leave empty. 26186 secret_access_key> YYY 26187 26188 Leave region blank 26189 26190 Region to connect to. 26191 Leave blank if you are using an S3 clone and you don't have a region. 26192 Choose a number from below, or type in your own value. 26193 Press Enter to leave empty. 26194 / Use this if unsure. 26195 1 | Will use v4 signatures and an empty region. 26196 \ () 26197 / Use this only if v4 signatures don't work. 26198 2 | E.g. pre Jewel/v10 CEPH. 26199 \ (other-v2-signature) 26200 region> 26201 26202 Choose an endpoint from the list 26203 26204 Endpoint for S3 API. 26205 Required when using an S3 clone. 26206 Choose a number from below, or type in your own value. 26207 Press Enter to leave empty. 26208 1 / Seagate Lyve Cloud US East 1 (Virginia) 26209 \ (s3.us-east-1.lyvecloud.seagate.com) 26210 2 / Seagate Lyve Cloud US West 1 (California) 26211 \ (s3.us-west-1.lyvecloud.seagate.com) 26212 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore) 26213 \ (s3.ap-southeast-1.lyvecloud.seagate.com) 26214 endpoint> 1 26215 26216 Leave location constraint blank 26217 26218 Location constraint - must be set to match the Region. 26219 Leave blank if not sure. Used when creating buckets only. 26220 Enter a value. Press Enter to leave empty. 26221 location_constraint> 26222 26223 Choose default ACL (private). 26224 26225 Canned ACL used when creating buckets and storing or copying objects. 26226 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 26227 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 26228 Note that this ACL is applied when server-side copying objects as S3 26229 doesn't copy the ACL from the source but rather writes a fresh one. 26230 Choose a number from below, or type in your own value. 26231 Press Enter to leave empty. 26232 / Owner gets FULL_CONTROL. 26233 1 | No one else has access rights (default). 26234 \ (private) 26235 [snip] 26236 acl> 26237 26238 And the config file should end up looking like this: 26239 26240 [remote] 26241 type = s3 26242 provider = LyveCloud 26243 access_key_id = XXX 26244 secret_access_key = YYY 26245 endpoint = s3.us-east-1.lyvecloud.seagate.com 26246 26247 SeaweedFS 26248 26249 SeaweedFS is a distributed storage system for blobs, objects, files, and 26250 data lake, with O(1) disk seek and a scalable file metadata store. It 26251 has an S3 compatible object storage interface. SeaweedFS can also act as 26252 a gateway to remote S3 compatible object store to cache data and 26253 metadata with asynchronous write back, for fast local speed and minimize 26254 access cost. 26255 26256 Assuming the SeaweedFS are configured with weed shell as such: 26257 26258 > s3.bucket.create -name foo 26259 > s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply 26260 { 26261 "identities": [ 26262 { 26263 "name": "me", 26264 "credentials": [ 26265 { 26266 "accessKey": "any", 26267 "secretKey": "any" 26268 } 26269 ], 26270 "actions": [ 26271 "Read:foo", 26272 "Write:foo", 26273 "List:foo", 26274 "Tagging:foo", 26275 "Admin:foo" 26276 ] 26277 } 26278 ] 26279 } 26280 26281 To use rclone with SeaweedFS, above configuration should end up with 26282 something like this in your config: 26283 26284 [seaweedfs_s3] 26285 type = s3 26286 provider = SeaweedFS 26287 access_key_id = any 26288 secret_access_key = any 26289 endpoint = localhost:8333 26290 26291 So once set up, for example to copy files into a bucket 26292 26293 rclone copy /path/to/files seaweedfs_s3:foo 26294 26295 Wasabi 26296 26297 Wasabi is a cloud-based object storage service for a broad range of 26298 applications and use cases. Wasabi is designed for individuals and 26299 organizations that require a high-performance, reliable, and secure data 26300 storage infrastructure at minimal cost. 26301 26302 Wasabi provides an S3 interface which can be configured for use with 26303 rclone like this. 26304 26305 No remotes found, make a new one? 26306 n) New remote 26307 s) Set configuration password 26308 n/s> n 26309 name> wasabi 26310 Type of storage to configure. 26311 Choose a number from below, or type in your own value 26312 [snip] 26313 XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara) 26314 \ "s3" 26315 [snip] 26316 Storage> s3 26317 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 26318 Choose a number from below, or type in your own value 26319 1 / Enter AWS credentials in the next step 26320 \ "false" 26321 2 / Get AWS credentials from the environment (env vars or IAM) 26322 \ "true" 26323 env_auth> 1 26324 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 26325 access_key_id> YOURACCESSKEY 26326 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 26327 secret_access_key> YOURSECRETACCESSKEY 26328 Region to connect to. 26329 Choose a number from below, or type in your own value 26330 / The default endpoint - a good choice if you are unsure. 26331 1 | US Region, Northern Virginia, or Pacific Northwest. 26332 | Leave location constraint empty. 26333 \ "us-east-1" 26334 [snip] 26335 region> us-east-1 26336 Endpoint for S3 API. 26337 Leave blank if using AWS to use the default endpoint for the region. 26338 Specify if using an S3 clone such as Ceph. 26339 endpoint> s3.wasabisys.com 26340 Location constraint - must be set to match the Region. Used when creating buckets only. 26341 Choose a number from below, or type in your own value 26342 1 / Empty for US Region, Northern Virginia, or Pacific Northwest. 26343 \ "" 26344 [snip] 26345 location_constraint> 26346 Canned ACL used when creating buckets and/or storing objects in S3. 26347 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 26348 Choose a number from below, or type in your own value 26349 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 26350 \ "private" 26351 [snip] 26352 acl> 26353 The server-side encryption algorithm used when storing this object in S3. 26354 Choose a number from below, or type in your own value 26355 1 / None 26356 \ "" 26357 2 / AES256 26358 \ "AES256" 26359 server_side_encryption> 26360 The storage class to use when storing objects in S3. 26361 Choose a number from below, or type in your own value 26362 1 / Default 26363 \ "" 26364 2 / Standard storage class 26365 \ "STANDARD" 26366 3 / Reduced redundancy storage class 26367 \ "REDUCED_REDUNDANCY" 26368 4 / Standard Infrequent Access storage class 26369 \ "STANDARD_IA" 26370 storage_class> 26371 Remote config 26372 -------------------- 26373 [wasabi] 26374 env_auth = false 26375 access_key_id = YOURACCESSKEY 26376 secret_access_key = YOURSECRETACCESSKEY 26377 region = us-east-1 26378 endpoint = s3.wasabisys.com 26379 location_constraint = 26380 acl = 26381 server_side_encryption = 26382 storage_class = 26383 -------------------- 26384 y) Yes this is OK 26385 e) Edit this remote 26386 d) Delete this remote 26387 y/e/d> y 26388 26389 This will leave the config file looking like this. 26390 26391 [wasabi] 26392 type = s3 26393 provider = Wasabi 26394 env_auth = false 26395 access_key_id = YOURACCESSKEY 26396 secret_access_key = YOURSECRETACCESSKEY 26397 region = 26398 endpoint = s3.wasabisys.com 26399 location_constraint = 26400 acl = 26401 server_side_encryption = 26402 storage_class = 26403 26404 Alibaba OSS 26405 26406 Here is an example of making an Alibaba Cloud (Aliyun) OSS 26407 configuration. First run: 26408 26409 rclone config 26410 26411 This will guide you through an interactive setup process. 26412 26413 No remotes found, make a new one? 26414 n) New remote 26415 s) Set configuration password 26416 q) Quit config 26417 n/s/q> n 26418 name> oss 26419 Type of storage to configure. 26420 Enter a string value. Press Enter for the default (""). 26421 Choose a number from below, or type in your own value 26422 [snip] 26423 XX / Amazon S3 Compliant Storage Providers including AWS, ... 26424 \ "s3" 26425 [snip] 26426 Storage> s3 26427 Choose your S3 provider. 26428 Enter a string value. Press Enter for the default (""). 26429 Choose a number from below, or type in your own value 26430 1 / Amazon Web Services (AWS) S3 26431 \ "AWS" 26432 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun 26433 \ "Alibaba" 26434 3 / Ceph Object Storage 26435 \ "Ceph" 26436 [snip] 26437 provider> Alibaba 26438 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 26439 Only applies if access_key_id and secret_access_key is blank. 26440 Enter a boolean value (true or false). Press Enter for the default ("false"). 26441 Choose a number from below, or type in your own value 26442 1 / Enter AWS credentials in the next step 26443 \ "false" 26444 2 / Get AWS credentials from the environment (env vars or IAM) 26445 \ "true" 26446 env_auth> 1 26447 AWS Access Key ID. 26448 Leave blank for anonymous access or runtime credentials. 26449 Enter a string value. Press Enter for the default (""). 26450 access_key_id> accesskeyid 26451 AWS Secret Access Key (password) 26452 Leave blank for anonymous access or runtime credentials. 26453 Enter a string value. Press Enter for the default (""). 26454 secret_access_key> secretaccesskey 26455 Endpoint for OSS API. 26456 Enter a string value. Press Enter for the default (""). 26457 Choose a number from below, or type in your own value 26458 1 / East China 1 (Hangzhou) 26459 \ "oss-cn-hangzhou.aliyuncs.com" 26460 2 / East China 2 (Shanghai) 26461 \ "oss-cn-shanghai.aliyuncs.com" 26462 3 / North China 1 (Qingdao) 26463 \ "oss-cn-qingdao.aliyuncs.com" 26464 [snip] 26465 endpoint> 1 26466 Canned ACL used when creating buckets and storing or copying objects. 26467 26468 Note that this ACL is applied when server-side copying objects as S3 26469 doesn't copy the ACL from the source but rather writes a fresh one. 26470 Enter a string value. Press Enter for the default (""). 26471 Choose a number from below, or type in your own value 26472 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 26473 \ "private" 26474 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. 26475 \ "public-read" 26476 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 26477 [snip] 26478 acl> 1 26479 The storage class to use when storing new objects in OSS. 26480 Enter a string value. Press Enter for the default (""). 26481 Choose a number from below, or type in your own value 26482 1 / Default 26483 \ "" 26484 2 / Standard storage class 26485 \ "STANDARD" 26486 3 / Archive storage mode. 26487 \ "GLACIER" 26488 4 / Infrequent access storage mode. 26489 \ "STANDARD_IA" 26490 storage_class> 1 26491 Edit advanced config? (y/n) 26492 y) Yes 26493 n) No 26494 y/n> n 26495 Remote config 26496 -------------------- 26497 [oss] 26498 type = s3 26499 provider = Alibaba 26500 env_auth = false 26501 access_key_id = accesskeyid 26502 secret_access_key = secretaccesskey 26503 endpoint = oss-cn-hangzhou.aliyuncs.com 26504 acl = private 26505 storage_class = Standard 26506 -------------------- 26507 y) Yes this is OK 26508 e) Edit this remote 26509 d) Delete this remote 26510 y/e/d> y 26511 26512 China Mobile Ecloud Elastic Object Storage (EOS) 26513 26514 Here is an example of making an China Mobile Ecloud Elastic Object 26515 Storage (EOS) configuration. First run: 26516 26517 rclone config 26518 26519 This will guide you through an interactive setup process. 26520 26521 No remotes found, make a new one? 26522 n) New remote 26523 s) Set configuration password 26524 q) Quit config 26525 n/s/q> n 26526 name> ChinaMobile 26527 Option Storage. 26528 Type of storage to configure. 26529 Choose a number from below, or type in your own value. 26530 ... 26531 XX / Amazon S3 Compliant Storage Providers including AWS, ... 26532 \ (s3) 26533 ... 26534 Storage> s3 26535 Option provider. 26536 Choose your S3 provider. 26537 Choose a number from below, or type in your own value. 26538 Press Enter to leave empty. 26539 ... 26540 4 / China Mobile Ecloud Elastic Object Storage (EOS) 26541 \ (ChinaMobile) 26542 ... 26543 provider> ChinaMobile 26544 Option env_auth. 26545 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 26546 Only applies if access_key_id and secret_access_key is blank. 26547 Choose a number from below, or type in your own boolean value (true or false). 26548 Press Enter for the default (false). 26549 1 / Enter AWS credentials in the next step. 26550 \ (false) 26551 2 / Get AWS credentials from the environment (env vars or IAM). 26552 \ (true) 26553 env_auth> 26554 Option access_key_id. 26555 AWS Access Key ID. 26556 Leave blank for anonymous access or runtime credentials. 26557 Enter a value. Press Enter to leave empty. 26558 access_key_id> accesskeyid 26559 Option secret_access_key. 26560 AWS Secret Access Key (password). 26561 Leave blank for anonymous access or runtime credentials. 26562 Enter a value. Press Enter to leave empty. 26563 secret_access_key> secretaccesskey 26564 Option endpoint. 26565 Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API. 26566 Choose a number from below, or type in your own value. 26567 Press Enter to leave empty. 26568 / The default endpoint - a good choice if you are unsure. 26569 1 | East China (Suzhou) 26570 \ (eos-wuxi-1.cmecloud.cn) 26571 2 / East China (Jinan) 26572 \ (eos-jinan-1.cmecloud.cn) 26573 3 / East China (Hangzhou) 26574 \ (eos-ningbo-1.cmecloud.cn) 26575 4 / East China (Shanghai-1) 26576 \ (eos-shanghai-1.cmecloud.cn) 26577 5 / Central China (Zhengzhou) 26578 \ (eos-zhengzhou-1.cmecloud.cn) 26579 6 / Central China (Changsha-1) 26580 \ (eos-hunan-1.cmecloud.cn) 26581 7 / Central China (Changsha-2) 26582 \ (eos-zhuzhou-1.cmecloud.cn) 26583 8 / South China (Guangzhou-2) 26584 \ (eos-guangzhou-1.cmecloud.cn) 26585 9 / South China (Guangzhou-3) 26586 \ (eos-dongguan-1.cmecloud.cn) 26587 10 / North China (Beijing-1) 26588 \ (eos-beijing-1.cmecloud.cn) 26589 11 / North China (Beijing-2) 26590 \ (eos-beijing-2.cmecloud.cn) 26591 12 / North China (Beijing-3) 26592 \ (eos-beijing-4.cmecloud.cn) 26593 13 / North China (Huhehaote) 26594 \ (eos-huhehaote-1.cmecloud.cn) 26595 14 / Southwest China (Chengdu) 26596 \ (eos-chengdu-1.cmecloud.cn) 26597 15 / Southwest China (Chongqing) 26598 \ (eos-chongqing-1.cmecloud.cn) 26599 16 / Southwest China (Guiyang) 26600 \ (eos-guiyang-1.cmecloud.cn) 26601 17 / Nouthwest China (Xian) 26602 \ (eos-xian-1.cmecloud.cn) 26603 18 / Yunnan China (Kunming) 26604 \ (eos-yunnan.cmecloud.cn) 26605 19 / Yunnan China (Kunming-2) 26606 \ (eos-yunnan-2.cmecloud.cn) 26607 20 / Tianjin China (Tianjin) 26608 \ (eos-tianjin-1.cmecloud.cn) 26609 21 / Jilin China (Changchun) 26610 \ (eos-jilin-1.cmecloud.cn) 26611 22 / Hubei China (Xiangyan) 26612 \ (eos-hubei-1.cmecloud.cn) 26613 23 / Jiangxi China (Nanchang) 26614 \ (eos-jiangxi-1.cmecloud.cn) 26615 24 / Gansu China (Lanzhou) 26616 \ (eos-gansu-1.cmecloud.cn) 26617 25 / Shanxi China (Taiyuan) 26618 \ (eos-shanxi-1.cmecloud.cn) 26619 26 / Liaoning China (Shenyang) 26620 \ (eos-liaoning-1.cmecloud.cn) 26621 27 / Hebei China (Shijiazhuang) 26622 \ (eos-hebei-1.cmecloud.cn) 26623 28 / Fujian China (Xiamen) 26624 \ (eos-fujian-1.cmecloud.cn) 26625 29 / Guangxi China (Nanning) 26626 \ (eos-guangxi-1.cmecloud.cn) 26627 30 / Anhui China (Huainan) 26628 \ (eos-anhui-1.cmecloud.cn) 26629 endpoint> 1 26630 Option location_constraint. 26631 Location constraint - must match endpoint. 26632 Used when creating buckets only. 26633 Choose a number from below, or type in your own value. 26634 Press Enter to leave empty. 26635 1 / East China (Suzhou) 26636 \ (wuxi1) 26637 2 / East China (Jinan) 26638 \ (jinan1) 26639 3 / East China (Hangzhou) 26640 \ (ningbo1) 26641 4 / East China (Shanghai-1) 26642 \ (shanghai1) 26643 5 / Central China (Zhengzhou) 26644 \ (zhengzhou1) 26645 6 / Central China (Changsha-1) 26646 \ (hunan1) 26647 7 / Central China (Changsha-2) 26648 \ (zhuzhou1) 26649 8 / South China (Guangzhou-2) 26650 \ (guangzhou1) 26651 9 / South China (Guangzhou-3) 26652 \ (dongguan1) 26653 10 / North China (Beijing-1) 26654 \ (beijing1) 26655 11 / North China (Beijing-2) 26656 \ (beijing2) 26657 12 / North China (Beijing-3) 26658 \ (beijing4) 26659 13 / North China (Huhehaote) 26660 \ (huhehaote1) 26661 14 / Southwest China (Chengdu) 26662 \ (chengdu1) 26663 15 / Southwest China (Chongqing) 26664 \ (chongqing1) 26665 16 / Southwest China (Guiyang) 26666 \ (guiyang1) 26667 17 / Nouthwest China (Xian) 26668 \ (xian1) 26669 18 / Yunnan China (Kunming) 26670 \ (yunnan) 26671 19 / Yunnan China (Kunming-2) 26672 \ (yunnan2) 26673 20 / Tianjin China (Tianjin) 26674 \ (tianjin1) 26675 21 / Jilin China (Changchun) 26676 \ (jilin1) 26677 22 / Hubei China (Xiangyan) 26678 \ (hubei1) 26679 23 / Jiangxi China (Nanchang) 26680 \ (jiangxi1) 26681 24 / Gansu China (Lanzhou) 26682 \ (gansu1) 26683 25 / Shanxi China (Taiyuan) 26684 \ (shanxi1) 26685 26 / Liaoning China (Shenyang) 26686 \ (liaoning1) 26687 27 / Hebei China (Shijiazhuang) 26688 \ (hebei1) 26689 28 / Fujian China (Xiamen) 26690 \ (fujian1) 26691 29 / Guangxi China (Nanning) 26692 \ (guangxi1) 26693 30 / Anhui China (Huainan) 26694 \ (anhui1) 26695 location_constraint> 1 26696 Option acl. 26697 Canned ACL used when creating buckets and storing or copying objects. 26698 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 26699 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 26700 Note that this ACL is applied when server-side copying objects as S3 26701 doesn't copy the ACL from the source but rather writes a fresh one. 26702 Choose a number from below, or type in your own value. 26703 Press Enter to leave empty. 26704 / Owner gets FULL_CONTROL. 26705 1 | No one else has access rights (default). 26706 \ (private) 26707 / Owner gets FULL_CONTROL. 26708 2 | The AllUsers group gets READ access. 26709 \ (public-read) 26710 / Owner gets FULL_CONTROL. 26711 3 | The AllUsers group gets READ and WRITE access. 26712 | Granting this on a bucket is generally not recommended. 26713 \ (public-read-write) 26714 / Owner gets FULL_CONTROL. 26715 4 | The AuthenticatedUsers group gets READ access. 26716 \ (authenticated-read) 26717 / Object owner gets FULL_CONTROL. 26718 acl> private 26719 Option server_side_encryption. 26720 The server-side encryption algorithm used when storing this object in S3. 26721 Choose a number from below, or type in your own value. 26722 Press Enter to leave empty. 26723 1 / None 26724 \ () 26725 2 / AES256 26726 \ (AES256) 26727 server_side_encryption> 26728 Option storage_class. 26729 The storage class to use when storing new objects in ChinaMobile. 26730 Choose a number from below, or type in your own value. 26731 Press Enter to leave empty. 26732 1 / Default 26733 \ () 26734 2 / Standard storage class 26735 \ (STANDARD) 26736 3 / Archive storage mode 26737 \ (GLACIER) 26738 4 / Infrequent access storage mode 26739 \ (STANDARD_IA) 26740 storage_class> 26741 Edit advanced config? 26742 y) Yes 26743 n) No (default) 26744 y/n> n 26745 -------------------- 26746 [ChinaMobile] 26747 type = s3 26748 provider = ChinaMobile 26749 access_key_id = accesskeyid 26750 secret_access_key = secretaccesskey 26751 endpoint = eos-wuxi-1.cmecloud.cn 26752 location_constraint = wuxi1 26753 acl = private 26754 -------------------- 26755 y) Yes this is OK (default) 26756 e) Edit this remote 26757 d) Delete this remote 26758 y/e/d> y 26759 26760 Leviia Cloud Object Storage 26761 26762 Leviia Object Storage, backup and secure your data in a 100% French 26763 cloud, independent of GAFAM.. 26764 26765 To configure access to Leviia, follow the steps below: 26766 26767 1. Run rclone config and select n for a new remote. 26768 26769 rclone config 26770 No remotes found, make a new one? 26771 n) New remote 26772 s) Set configuration password 26773 q) Quit config 26774 n/s/q> n 26775 26776 2. Give the name of the configuration. For example, name it 'leviia'. 26777 26778 name> leviia 26779 26780 3. Select s3 storage. 26781 26782 Choose a number from below, or type in your own value 26783 [snip] 26784 XX / Amazon S3 Compliant Storage Providers including AWS, ... 26785 \ (s3) 26786 [snip] 26787 Storage> s3 26788 26789 4. Select Leviia provider. 26790 26791 Choose a number from below, or type in your own value 26792 1 / Amazon Web Services (AWS) S3 26793 \ "AWS" 26794 [snip] 26795 15 / Leviia Object Storage 26796 \ (Leviia) 26797 [snip] 26798 provider> Leviia 26799 26800 5. Enter your SecretId and SecretKey of Leviia. 26801 26802 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 26803 Only applies if access_key_id and secret_access_key is blank. 26804 Enter a boolean value (true or false). Press Enter for the default ("false"). 26805 Choose a number from below, or type in your own value 26806 1 / Enter AWS credentials in the next step 26807 \ "false" 26808 2 / Get AWS credentials from the environment (env vars or IAM) 26809 \ "true" 26810 env_auth> 1 26811 AWS Access Key ID. 26812 Leave blank for anonymous access or runtime credentials. 26813 Enter a string value. Press Enter for the default (""). 26814 access_key_id> ZnIx.xxxxxxxxxxxxxxx 26815 AWS Secret Access Key (password) 26816 Leave blank for anonymous access or runtime credentials. 26817 Enter a string value. Press Enter for the default (""). 26818 secret_access_key> xxxxxxxxxxx 26819 26820 6. Select endpoint for Leviia. 26821 26822 / The default endpoint 26823 1 | Leviia. 26824 \ (s3.leviia.com) 26825 [snip] 26826 endpoint> 1 26827 26828 7. Choose acl. 26829 26830 Note that this ACL is applied when server-side copying objects as S3 26831 doesn't copy the ACL from the source but rather writes a fresh one. 26832 Enter a string value. Press Enter for the default (""). 26833 Choose a number from below, or type in your own value 26834 / Owner gets FULL_CONTROL. 26835 1 | No one else has access rights (default). 26836 \ (private) 26837 / Owner gets FULL_CONTROL. 26838 2 | The AllUsers group gets READ access. 26839 \ (public-read) 26840 [snip] 26841 acl> 1 26842 Edit advanced config? (y/n) 26843 y) Yes 26844 n) No (default) 26845 y/n> n 26846 Remote config 26847 -------------------- 26848 [leviia] 26849 - type: s3 26850 - provider: Leviia 26851 - access_key_id: ZnIx.xxxxxxx 26852 - secret_access_key: xxxxxxxx 26853 - endpoint: s3.leviia.com 26854 - acl: private 26855 -------------------- 26856 y) Yes this is OK (default) 26857 e) Edit this remote 26858 d) Delete this remote 26859 y/e/d> y 26860 Current remotes: 26861 26862 Name Type 26863 ==== ==== 26864 leviia s3 26865 26866 Liara 26867 26868 Here is an example of making a Liara Object Storage configuration. First 26869 run: 26870 26871 rclone config 26872 26873 This will guide you through an interactive setup process. 26874 26875 No remotes found, make a new one? 26876 n) New remote 26877 s) Set configuration password 26878 n/s> n 26879 name> Liara 26880 Type of storage to configure. 26881 Choose a number from below, or type in your own value 26882 [snip] 26883 XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio) 26884 \ "s3" 26885 [snip] 26886 Storage> s3 26887 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 26888 Choose a number from below, or type in your own value 26889 1 / Enter AWS credentials in the next step 26890 \ "false" 26891 2 / Get AWS credentials from the environment (env vars or IAM) 26892 \ "true" 26893 env_auth> 1 26894 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 26895 access_key_id> YOURACCESSKEY 26896 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 26897 secret_access_key> YOURSECRETACCESSKEY 26898 Region to connect to. 26899 Choose a number from below, or type in your own value 26900 / The default endpoint 26901 1 | US Region, Northern Virginia, or Pacific Northwest. 26902 | Leave location constraint empty. 26903 \ "us-east-1" 26904 [snip] 26905 region> 26906 Endpoint for S3 API. 26907 Leave blank if using Liara to use the default endpoint for the region. 26908 Specify if using an S3 clone such as Ceph. 26909 endpoint> storage.iran.liara.space 26910 Canned ACL used when creating buckets and/or storing objects in S3. 26911 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 26912 Choose a number from below, or type in your own value 26913 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 26914 \ "private" 26915 [snip] 26916 acl> 26917 The server-side encryption algorithm used when storing this object in S3. 26918 Choose a number from below, or type in your own value 26919 1 / None 26920 \ "" 26921 2 / AES256 26922 \ "AES256" 26923 server_side_encryption> 26924 The storage class to use when storing objects in S3. 26925 Choose a number from below, or type in your own value 26926 1 / Default 26927 \ "" 26928 2 / Standard storage class 26929 \ "STANDARD" 26930 storage_class> 26931 Remote config 26932 -------------------- 26933 [Liara] 26934 env_auth = false 26935 access_key_id = YOURACCESSKEY 26936 secret_access_key = YOURSECRETACCESSKEY 26937 endpoint = storage.iran.liara.space 26938 location_constraint = 26939 acl = 26940 server_side_encryption = 26941 storage_class = 26942 -------------------- 26943 y) Yes this is OK 26944 e) Edit this remote 26945 d) Delete this remote 26946 y/e/d> y 26947 26948 This will leave the config file looking like this. 26949 26950 [Liara] 26951 type = s3 26952 provider = Liara 26953 env_auth = false 26954 access_key_id = YOURACCESSKEY 26955 secret_access_key = YOURSECRETACCESSKEY 26956 region = 26957 endpoint = storage.iran.liara.space 26958 location_constraint = 26959 acl = 26960 server_side_encryption = 26961 storage_class = 26962 26963 Linode 26964 26965 Here is an example of making a Linode Object Storage configuration. 26966 First run: 26967 26968 rclone config 26969 26970 This will guide you through an interactive setup process. 26971 26972 No remotes found, make a new one? 26973 n) New remote 26974 s) Set configuration password 26975 q) Quit config 26976 n/s/q> n 26977 26978 Enter name for new remote. 26979 name> linode 26980 26981 Option Storage. 26982 Type of storage to configure. 26983 Choose a number from below, or type in your own value. 26984 [snip] 26985 XX / Amazon S3 Compliant Storage Providers including AWS, ...Linode, ...and others 26986 \ (s3) 26987 [snip] 26988 Storage> s3 26989 26990 Option provider. 26991 Choose your S3 provider. 26992 Choose a number from below, or type in your own value. 26993 Press Enter to leave empty. 26994 [snip] 26995 XX / Linode Object Storage 26996 \ (Linode) 26997 [snip] 26998 provider> Linode 26999 27000 Option env_auth. 27001 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 27002 Only applies if access_key_id and secret_access_key is blank. 27003 Choose a number from below, or type in your own boolean value (true or false). 27004 Press Enter for the default (false). 27005 1 / Enter AWS credentials in the next step. 27006 \ (false) 27007 2 / Get AWS credentials from the environment (env vars or IAM). 27008 \ (true) 27009 env_auth> 27010 27011 Option access_key_id. 27012 AWS Access Key ID. 27013 Leave blank for anonymous access or runtime credentials. 27014 Enter a value. Press Enter to leave empty. 27015 access_key_id> ACCESS_KEY 27016 27017 Option secret_access_key. 27018 AWS Secret Access Key (password). 27019 Leave blank for anonymous access or runtime credentials. 27020 Enter a value. Press Enter to leave empty. 27021 secret_access_key> SECRET_ACCESS_KEY 27022 27023 Option endpoint. 27024 Endpoint for Linode Object Storage API. 27025 Choose a number from below, or type in your own value. 27026 Press Enter to leave empty. 27027 1 / Atlanta, GA (USA), us-southeast-1 27028 \ (us-southeast-1.linodeobjects.com) 27029 2 / Chicago, IL (USA), us-ord-1 27030 \ (us-ord-1.linodeobjects.com) 27031 3 / Frankfurt (Germany), eu-central-1 27032 \ (eu-central-1.linodeobjects.com) 27033 4 / Milan (Italy), it-mil-1 27034 \ (it-mil-1.linodeobjects.com) 27035 5 / Newark, NJ (USA), us-east-1 27036 \ (us-east-1.linodeobjects.com) 27037 6 / Paris (France), fr-par-1 27038 \ (fr-par-1.linodeobjects.com) 27039 7 / Seattle, WA (USA), us-sea-1 27040 \ (us-sea-1.linodeobjects.com) 27041 8 / Singapore ap-south-1 27042 \ (ap-south-1.linodeobjects.com) 27043 9 / Stockholm (Sweden), se-sto-1 27044 \ (se-sto-1.linodeobjects.com) 27045 10 / Washington, DC, (USA), us-iad-1 27046 \ (us-iad-1.linodeobjects.com) 27047 endpoint> 3 27048 27049 Option acl. 27050 Canned ACL used when creating buckets and storing or copying objects. 27051 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 27052 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 27053 Note that this ACL is applied when server-side copying objects as S3 27054 doesn't copy the ACL from the source but rather writes a fresh one. 27055 If the acl is an empty string then no X-Amz-Acl: header is added and 27056 the default (private) will be used. 27057 Choose a number from below, or type in your own value. 27058 Press Enter to leave empty. 27059 / Owner gets FULL_CONTROL. 27060 1 | No one else has access rights (default). 27061 \ (private) 27062 [snip] 27063 acl> 27064 27065 Edit advanced config? 27066 y) Yes 27067 n) No (default) 27068 y/n> n 27069 27070 Configuration complete. 27071 Options: 27072 - type: s3 27073 - provider: Linode 27074 - access_key_id: ACCESS_KEY 27075 - secret_access_key: SECRET_ACCESS_KEY 27076 - endpoint: eu-central-1.linodeobjects.com 27077 Keep this "linode" remote? 27078 y) Yes this is OK (default) 27079 e) Edit this remote 27080 d) Delete this remote 27081 y/e/d> y 27082 27083 This will leave the config file looking like this. 27084 27085 [linode] 27086 type = s3 27087 provider = Linode 27088 access_key_id = ACCESS_KEY 27089 secret_access_key = SECRET_ACCESS_KEY 27090 endpoint = eu-central-1.linodeobjects.com 27091 27092 ArvanCloud 27093 27094 ArvanCloud ArvanCloud Object Storage goes beyond the limited traditional 27095 file storage. It gives you access to backup and archived files and 27096 allows sharing. Files like profile image in the app, images sent by 27097 users or scanned documents can be stored securely and easily in our 27098 Object Storage service. 27099 27100 ArvanCloud provides an S3 interface which can be configured for use with 27101 rclone like this. 27102 27103 No remotes found, make a new one? 27104 n) New remote 27105 s) Set configuration password 27106 n/s> n 27107 name> ArvanCloud 27108 Type of storage to configure. 27109 Choose a number from below, or type in your own value 27110 [snip] 27111 XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio) 27112 \ "s3" 27113 [snip] 27114 Storage> s3 27115 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. 27116 Choose a number from below, or type in your own value 27117 1 / Enter AWS credentials in the next step 27118 \ "false" 27119 2 / Get AWS credentials from the environment (env vars or IAM) 27120 \ "true" 27121 env_auth> 1 27122 AWS Access Key ID - leave blank for anonymous access or runtime credentials. 27123 access_key_id> YOURACCESSKEY 27124 AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 27125 secret_access_key> YOURSECRETACCESSKEY 27126 Region to connect to. 27127 Choose a number from below, or type in your own value 27128 / The default endpoint - a good choice if you are unsure. 27129 1 | US Region, Northern Virginia, or Pacific Northwest. 27130 | Leave location constraint empty. 27131 \ "us-east-1" 27132 [snip] 27133 region> 27134 Endpoint for S3 API. 27135 Leave blank if using ArvanCloud to use the default endpoint for the region. 27136 Specify if using an S3 clone such as Ceph. 27137 endpoint> s3.arvanstorage.com 27138 Location constraint - must be set to match the Region. Used when creating buckets only. 27139 Choose a number from below, or type in your own value 27140 1 / Empty for Iran-Tehran Region. 27141 \ "" 27142 [snip] 27143 location_constraint> 27144 Canned ACL used when creating buckets and/or storing objects in S3. 27145 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 27146 Choose a number from below, or type in your own value 27147 1 / Owner gets FULL_CONTROL. No one else has access rights (default). 27148 \ "private" 27149 [snip] 27150 acl> 27151 The server-side encryption algorithm used when storing this object in S3. 27152 Choose a number from below, or type in your own value 27153 1 / None 27154 \ "" 27155 2 / AES256 27156 \ "AES256" 27157 server_side_encryption> 27158 The storage class to use when storing objects in S3. 27159 Choose a number from below, or type in your own value 27160 1 / Default 27161 \ "" 27162 2 / Standard storage class 27163 \ "STANDARD" 27164 storage_class> 27165 Remote config 27166 -------------------- 27167 [ArvanCloud] 27168 env_auth = false 27169 access_key_id = YOURACCESSKEY 27170 secret_access_key = YOURSECRETACCESSKEY 27171 region = ir-thr-at1 27172 endpoint = s3.arvanstorage.com 27173 location_constraint = 27174 acl = 27175 server_side_encryption = 27176 storage_class = 27177 -------------------- 27178 y) Yes this is OK 27179 e) Edit this remote 27180 d) Delete this remote 27181 y/e/d> y 27182 27183 This will leave the config file looking like this. 27184 27185 [ArvanCloud] 27186 type = s3 27187 provider = ArvanCloud 27188 env_auth = false 27189 access_key_id = YOURACCESSKEY 27190 secret_access_key = YOURSECRETACCESSKEY 27191 region = 27192 endpoint = s3.arvanstorage.com 27193 location_constraint = 27194 acl = 27195 server_side_encryption = 27196 storage_class = 27197 27198 Tencent COS 27199 27200 Tencent Cloud Object Storage (COS) is a distributed storage service 27201 offered by Tencent Cloud for unstructured data. It is secure, stable, 27202 massive, convenient, low-delay and low-cost. 27203 27204 To configure access to Tencent COS, follow the steps below: 27205 27206 1. Run rclone config and select n for a new remote. 27207 27208 rclone config 27209 No remotes found, make a new one? 27210 n) New remote 27211 s) Set configuration password 27212 q) Quit config 27213 n/s/q> n 27214 27215 2. Give the name of the configuration. For example, name it 'cos'. 27216 27217 name> cos 27218 27219 3. Select s3 storage. 27220 27221 Choose a number from below, or type in your own value 27222 [snip] 27223 XX / Amazon S3 Compliant Storage Providers including AWS, ... 27224 \ "s3" 27225 [snip] 27226 Storage> s3 27227 27228 4. Select TencentCOS provider. 27229 27230 Choose a number from below, or type in your own value 27231 1 / Amazon Web Services (AWS) S3 27232 \ "AWS" 27233 [snip] 27234 11 / Tencent Cloud Object Storage (COS) 27235 \ "TencentCOS" 27236 [snip] 27237 provider> TencentCOS 27238 27239 5. Enter your SecretId and SecretKey of Tencent Cloud. 27240 27241 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 27242 Only applies if access_key_id and secret_access_key is blank. 27243 Enter a boolean value (true or false). Press Enter for the default ("false"). 27244 Choose a number from below, or type in your own value 27245 1 / Enter AWS credentials in the next step 27246 \ "false" 27247 2 / Get AWS credentials from the environment (env vars or IAM) 27248 \ "true" 27249 env_auth> 1 27250 AWS Access Key ID. 27251 Leave blank for anonymous access or runtime credentials. 27252 Enter a string value. Press Enter for the default (""). 27253 access_key_id> AKIDxxxxxxxxxx 27254 AWS Secret Access Key (password) 27255 Leave blank for anonymous access or runtime credentials. 27256 Enter a string value. Press Enter for the default (""). 27257 secret_access_key> xxxxxxxxxxx 27258 27259 6. Select endpoint for Tencent COS. This is the standard endpoint for 27260 different region. 27261 27262 1 / Beijing Region. 27263 \ "cos.ap-beijing.myqcloud.com" 27264 2 / Nanjing Region. 27265 \ "cos.ap-nanjing.myqcloud.com" 27266 3 / Shanghai Region. 27267 \ "cos.ap-shanghai.myqcloud.com" 27268 4 / Guangzhou Region. 27269 \ "cos.ap-guangzhou.myqcloud.com" 27270 [snip] 27271 endpoint> 4 27272 27273 7. Choose acl and storage class. 27274 27275 Note that this ACL is applied when server-side copying objects as S3 27276 doesn't copy the ACL from the source but rather writes a fresh one. 27277 Enter a string value. Press Enter for the default (""). 27278 Choose a number from below, or type in your own value 27279 1 / Owner gets Full_CONTROL. No one else has access rights (default). 27280 \ "default" 27281 [snip] 27282 acl> 1 27283 The storage class to use when storing new objects in Tencent COS. 27284 Enter a string value. Press Enter for the default (""). 27285 Choose a number from below, or type in your own value 27286 1 / Default 27287 \ "" 27288 [snip] 27289 storage_class> 1 27290 Edit advanced config? (y/n) 27291 y) Yes 27292 n) No (default) 27293 y/n> n 27294 Remote config 27295 -------------------- 27296 [cos] 27297 type = s3 27298 provider = TencentCOS 27299 env_auth = false 27300 access_key_id = xxx 27301 secret_access_key = xxx 27302 endpoint = cos.ap-guangzhou.myqcloud.com 27303 acl = default 27304 -------------------- 27305 y) Yes this is OK (default) 27306 e) Edit this remote 27307 d) Delete this remote 27308 y/e/d> y 27309 Current remotes: 27310 27311 Name Type 27312 ==== ==== 27313 cos s3 27314 27315 Netease NOS 27316 27317 For Netease NOS configure as per the configurator rclone config setting 27318 the provider Netease. This will automatically set 27319 force_path_style = false which is necessary for it to run properly. 27320 27321 Petabox 27322 27323 Here is an example of making a Petabox configuration. First run: 27324 27325 rclone config 27326 27327 This will guide you through an interactive setup process. 27328 27329 No remotes found, make a new one? 27330 n) New remote 27331 s) Set configuration password 27332 n/s> n 27333 27334 Enter name for new remote. 27335 name> My Petabox Storage 27336 27337 Option Storage. 27338 Type of storage to configure. 27339 Choose a number from below, or type in your own value. 27340 [snip] 27341 XX / Amazon S3 Compliant Storage Providers including AWS, ... 27342 \ "s3" 27343 [snip] 27344 Storage> s3 27345 27346 Option provider. 27347 Choose your S3 provider. 27348 Choose a number from below, or type in your own value. 27349 Press Enter to leave empty. 27350 [snip] 27351 XX / Petabox Object Storage 27352 \ (Petabox) 27353 [snip] 27354 provider> Petabox 27355 27356 Option env_auth. 27357 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 27358 Only applies if access_key_id and secret_access_key is blank. 27359 Choose a number from below, or type in your own boolean value (true or false). 27360 Press Enter for the default (false). 27361 1 / Enter AWS credentials in the next step. 27362 \ (false) 27363 2 / Get AWS credentials from the environment (env vars or IAM). 27364 \ (true) 27365 env_auth> 1 27366 27367 Option access_key_id. 27368 AWS Access Key ID. 27369 Leave blank for anonymous access or runtime credentials. 27370 Enter a value. Press Enter to leave empty. 27371 access_key_id> YOUR_ACCESS_KEY_ID 27372 27373 Option secret_access_key. 27374 AWS Secret Access Key (password). 27375 Leave blank for anonymous access or runtime credentials. 27376 Enter a value. Press Enter to leave empty. 27377 secret_access_key> YOUR_SECRET_ACCESS_KEY 27378 27379 Option region. 27380 Region where your bucket will be created and your data stored. 27381 Choose a number from below, or type in your own value. 27382 Press Enter to leave empty. 27383 1 / US East (N. Virginia) 27384 \ (us-east-1) 27385 2 / Europe (Frankfurt) 27386 \ (eu-central-1) 27387 3 / Asia Pacific (Singapore) 27388 \ (ap-southeast-1) 27389 4 / Middle East (Bahrain) 27390 \ (me-south-1) 27391 5 / South America (São Paulo) 27392 \ (sa-east-1) 27393 region> 1 27394 27395 Option endpoint. 27396 Endpoint for Petabox S3 Object Storage. 27397 Specify the endpoint from the same region. 27398 Choose a number from below, or type in your own value. 27399 1 / US East (N. Virginia) 27400 \ (s3.petabox.io) 27401 2 / US East (N. Virginia) 27402 \ (s3.us-east-1.petabox.io) 27403 3 / Europe (Frankfurt) 27404 \ (s3.eu-central-1.petabox.io) 27405 4 / Asia Pacific (Singapore) 27406 \ (s3.ap-southeast-1.petabox.io) 27407 5 / Middle East (Bahrain) 27408 \ (s3.me-south-1.petabox.io) 27409 6 / South America (São Paulo) 27410 \ (s3.sa-east-1.petabox.io) 27411 endpoint> 1 27412 27413 Option acl. 27414 Canned ACL used when creating buckets and storing or copying objects. 27415 This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. 27416 For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl 27417 Note that this ACL is applied when server-side copying objects as S3 27418 doesn't copy the ACL from the source but rather writes a fresh one. 27419 If the acl is an empty string then no X-Amz-Acl: header is added and 27420 the default (private) will be used. 27421 Choose a number from below, or type in your own value. 27422 Press Enter to leave empty. 27423 / Owner gets FULL_CONTROL. 27424 1 | No one else has access rights (default). 27425 \ (private) 27426 / Owner gets FULL_CONTROL. 27427 2 | The AllUsers group gets READ access. 27428 \ (public-read) 27429 / Owner gets FULL_CONTROL. 27430 3 | The AllUsers group gets READ and WRITE access. 27431 | Granting this on a bucket is generally not recommended. 27432 \ (public-read-write) 27433 / Owner gets FULL_CONTROL. 27434 4 | The AuthenticatedUsers group gets READ access. 27435 \ (authenticated-read) 27436 / Object owner gets FULL_CONTROL. 27437 5 | Bucket owner gets READ access. 27438 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 27439 \ (bucket-owner-read) 27440 / Both the object owner and the bucket owner get FULL_CONTROL over the object. 27441 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. 27442 \ (bucket-owner-full-control) 27443 acl> 1 27444 27445 Edit advanced config? 27446 y) Yes 27447 n) No (default) 27448 y/n> No 27449 27450 Configuration complete. 27451 Options: 27452 - type: s3 27453 - provider: Petabox 27454 - access_key_id: YOUR_ACCESS_KEY_ID 27455 - secret_access_key: YOUR_SECRET_ACCESS_KEY 27456 - region: us-east-1 27457 - endpoint: s3.petabox.io 27458 Keep this "My Petabox Storage" remote? 27459 y) Yes this is OK (default) 27460 e) Edit this remote 27461 d) Delete this remote 27462 y/e/d> y 27463 27464 This will leave the config file looking like this. 27465 27466 [My Petabox Storage] 27467 type = s3 27468 provider = Petabox 27469 access_key_id = YOUR_ACCESS_KEY_ID 27470 secret_access_key = YOUR_SECRET_ACCESS_KEY 27471 region = us-east-1 27472 endpoint = s3.petabox.io 27473 27474 Storj 27475 27476 Storj is a decentralized cloud storage which can be used through its 27477 native protocol or an S3 compatible gateway. 27478 27479 The S3 compatible gateway is configured using rclone config with a type 27480 of s3 and with a provider name of Storj. Here is an example run of the 27481 configurator. 27482 27483 Type of storage to configure. 27484 Storage> s3 27485 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 27486 Only applies if access_key_id and secret_access_key is blank. 27487 Choose a number from below, or type in your own boolean value (true or false). 27488 Press Enter for the default (false). 27489 1 / Enter AWS credentials in the next step. 27490 \ (false) 27491 2 / Get AWS credentials from the environment (env vars or IAM). 27492 \ (true) 27493 env_auth> 1 27494 Option access_key_id. 27495 AWS Access Key ID. 27496 Leave blank for anonymous access or runtime credentials. 27497 Enter a value. Press Enter to leave empty. 27498 access_key_id> XXXX (as shown when creating the access grant) 27499 Option secret_access_key. 27500 AWS Secret Access Key (password). 27501 Leave blank for anonymous access or runtime credentials. 27502 Enter a value. Press Enter to leave empty. 27503 secret_access_key> XXXX (as shown when creating the access grant) 27504 Option endpoint. 27505 Endpoint of the Shared Gateway. 27506 Choose a number from below, or type in your own value. 27507 Press Enter to leave empty. 27508 1 / EU1 Shared Gateway 27509 \ (gateway.eu1.storjshare.io) 27510 2 / US1 Shared Gateway 27511 \ (gateway.us1.storjshare.io) 27512 3 / Asia-Pacific Shared Gateway 27513 \ (gateway.ap1.storjshare.io) 27514 endpoint> 1 (as shown when creating the access grant) 27515 Edit advanced config? 27516 y) Yes 27517 n) No (default) 27518 y/n> n 27519 27520 Note that s3 credentials are generated when you create an access grant. 27521 27522 Backend quirks 27523 27524 - --chunk-size is forced to be 64 MiB or greater. This will use more 27525 memory than the default of 5 MiB. 27526 - Server side copy is disabled as it isn't currently supported in the 27527 gateway. 27528 - GetTier and SetTier are not supported. 27529 27530 Backend bugs 27531 27532 Due to issue #39 uploading multipart files via the S3 gateway causes 27533 them to lose their metadata. For rclone's purpose this means that the 27534 modification time is not stored, nor is any MD5SUM (if one is available 27535 from the source). 27536 27537 This has the following consequences: 27538 27539 - Using rclone rcat will fail as the medatada doesn't match after 27540 upload 27541 - Uploading files with rclone mount will fail for the same reason 27542 - This can worked around by using --vfs-cache-mode writes or 27543 --vfs-cache-mode full or setting --s3-upload-cutoff large 27544 - Files uploaded via a multipart upload won't have their modtimes 27545 - This will mean that rclone sync will likely keep trying to 27546 upload files bigger than --s3-upload-cutoff 27547 - This can be worked around with --checksum or --size-only or 27548 setting --s3-upload-cutoff large 27549 - The maximum value for --s3-upload-cutoff is 5GiB though 27550 27551 One general purpose workaround is to set --s3-upload-cutoff 5G. This 27552 means that rclone will upload files smaller than 5GiB as single parts. 27553 Note that this can be set in the config file with upload_cutoff = 5G or 27554 configured in the advanced settings. If you regularly transfer files 27555 larger than 5G then using --checksum or --size-only in rclone sync is 27556 the recommended workaround. 27557 27558 Comparison with the native protocol 27559 27560 Use the the native protocol to take advantage of client-side encryption 27561 as well as to achieve the best possible download performance. Uploads 27562 will be erasure-coded locally, thus a 1gb upload will result in 2.68gb 27563 of data being uploaded to storage nodes across the network. 27564 27565 Use this backend and the S3 compatible Hosted Gateway to increase upload 27566 performance and reduce the load on your systems and network. Uploads 27567 will be encrypted and erasure-coded server-side, thus a 1GB upload will 27568 result in only in 1GB of data being uploaded to storage nodes across the 27569 network. 27570 27571 For more detailed comparison please check the documentation of the storj 27572 backend. 27573 27574 Limitations 27575 27576 rclone about is not supported by the S3 backend. Backends without this 27577 capability cannot determine free space for an rclone mount or use policy 27578 mfs (most free space) as a member of an rclone union remote. 27579 27580 See List of backends that do not support rclone about and rclone about 27581 27582 Synology C2 Object Storage 27583 27584 Synology C2 Object Storage provides a secure, S3-compatible, and 27585 cost-effective cloud storage solution without API request, download 27586 fees, and deletion penalty. 27587 27588 The S3 compatible gateway is configured using rclone config with a type 27589 of s3 and with a provider name of Synology. Here is an example run of 27590 the configurator. 27591 27592 First run: 27593 27594 rclone config 27595 27596 This will guide you through an interactive setup process. 27597 27598 No remotes found, make a new one? 27599 n) New remote 27600 s) Set configuration password 27601 q) Quit config 27602 27603 n/s/q> n 27604 27605 Enter name for new remote.1 27606 name> syno 27607 27608 Type of storage to configure. 27609 Enter a string value. Press Enter for the default (""). 27610 Choose a number from below, or type in your own value 27611 27612 XX / Amazon S3 Compliant Storage Providers including AWS, ... 27613 \ "s3" 27614 27615 Storage> s3 27616 27617 Choose your S3 provider. 27618 Enter a string value. Press Enter for the default (""). 27619 Choose a number from below, or type in your own value 27620 24 / Synology C2 Object Storage 27621 \ (Synology) 27622 27623 provider> Synology 27624 27625 Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). 27626 Only applies if access_key_id and secret_access_key is blank. 27627 Enter a boolean value (true or false). Press Enter for the default ("false"). 27628 Choose a number from below, or type in your own value 27629 1 / Enter AWS credentials in the next step 27630 \ "false" 27631 2 / Get AWS credentials from the environment (env vars or IAM) 27632 \ "true" 27633 27634 env_auth> 1 27635 27636 AWS Access Key ID. 27637 Leave blank for anonymous access or runtime credentials. 27638 Enter a string value. Press Enter for the default (""). 27639 27640 access_key_id> accesskeyid 27641 27642 AWS Secret Access Key (password) 27643 Leave blank for anonymous access or runtime credentials. 27644 Enter a string value. Press Enter for the default (""). 27645 27646 secret_access_key> secretaccesskey 27647 27648 Region where your data stored. 27649 Choose a number from below, or type in your own value. 27650 Press Enter to leave empty. 27651 1 / Europe Region 1 27652 \ (eu-001) 27653 2 / Europe Region 2 27654 \ (eu-002) 27655 3 / US Region 1 27656 \ (us-001) 27657 4 / US Region 2 27658 \ (us-002) 27659 5 / Asia (Taiwan) 27660 \ (tw-001) 27661 27662 region > 1 27663 27664 Option endpoint. 27665 Endpoint for Synology C2 Object Storage API. 27666 Choose a number from below, or type in your own value. 27667 Press Enter to leave empty. 27668 1 / EU Endpoint 1 27669 \ (eu-001.s3.synologyc2.net) 27670 2 / US Endpoint 1 27671 \ (us-001.s3.synologyc2.net) 27672 3 / TW Endpoint 1 27673 \ (tw-001.s3.synologyc2.net) 27674 27675 endpoint> 1 27676 27677 Option location_constraint. 27678 Location constraint - must be set to match the Region. 27679 Leave blank if not sure. Used when creating buckets only. 27680 Enter a value. Press Enter to leave empty. 27681 location_constraint> 27682 27683 Edit advanced config? (y/n) 27684 y) Yes 27685 n) No 27686 y/n> y 27687 27688 Option no_check_bucket. 27689 If set, don't attempt to check the bucket exists or create it. 27690 This can be useful when trying to minimise the number of transactions 27691 rclone does if you know the bucket exists already. 27692 It can also be needed if the user you are using does not have bucket 27693 creation permissions. Before v1.52.0 this would have passed silently 27694 due to a bug. 27695 Enter a boolean value (true or false). Press Enter for the default (true). 27696 27697 no_check_bucket> true 27698 27699 Configuration complete. 27700 Options: 27701 - type: s3 27702 - provider: Synology 27703 - region: eu-001 27704 - endpoint: eu-001.s3.synologyc2.net 27705 - no_check_bucket: true 27706 Keep this "syno" remote? 27707 y) Yes this is OK (default) 27708 e) Edit this remote 27709 d) Delete this remote 27710 27711 y/e/d> y 27712 27713 # Backblaze B2 27714 27715 B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/). 27716 27717 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 27718 command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`. 27719 27720 ## Configuration 27721 27722 Here is an example of making a b2 configuration. First run 27723 27724 rclone config 27725 27726 This will guide you through an interactive setup process. To authenticate 27727 you will either need your Account ID (a short hex number) and Master 27728 Application Key (a long hex number) OR an Application Key, which is the 27729 recommended method. See below for further details on generating and using 27730 an Application Key. 27731 27732 No remotes found, make a new one? n) New remote q) Quit config n/q> n 27733 name> remote Type of storage to configure. Choose a number from below, 27734 or type in your own value [snip] XX / Backblaze B2 "b2" [snip] Storage> 27735 b2 Account ID or Application Key ID account> 123456789abc Application 27736 Key key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the 27737 service - leave blank normally. endpoint> Remote config 27738 -------------------- [remote] account = 123456789abc key = 27739 0123456789abcdef0123456789abcdef0123456789 endpoint = 27740 -------------------- y) Yes this is OK e) Edit this remote d) Delete 27741 this remote y/e/d> y 27742 27743 27744 This remote is called `remote` and can now be used like this 27745 27746 See all buckets 27747 27748 rclone lsd remote: 27749 27750 Create a new bucket 27751 27752 rclone mkdir remote:bucket 27753 27754 List the contents of a bucket 27755 27756 rclone ls remote:bucket 27757 27758 Sync `/home/local/directory` to the remote bucket, deleting any 27759 excess files in the bucket. 27760 27761 rclone sync --interactive /home/local/directory remote:bucket 27762 27763 ### Application Keys 27764 27765 B2 supports multiple [Application Keys for different access permission 27766 to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html). 27767 27768 You can use these with rclone too; you will need to use rclone version 1.43 27769 or later. 27770 27771 Follow Backblaze's docs to create an Application Key with the required 27772 permission and add the `applicationKeyId` as the `account` and the 27773 `Application Key` itself as the `key`. 27774 27775 Note that you must put the _applicationKeyId_ as the `account` – you 27776 can't use the master Account ID. If you try then B2 will return 401 27777 errors. 27778 27779 ### --fast-list 27780 27781 This remote supports `--fast-list` which allows you to use fewer 27782 transactions in exchange for more memory. See the [rclone 27783 docs](https://rclone.org/docs/#fast-list) for more details. 27784 27785 ### Modification times 27786 27787 The modification time is stored as metadata on the object as 27788 `X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01 27789 in the Backblaze standard. Other tools should be able to use this as 27790 a modified time. 27791 27792 Modified times are used in syncing and are fully supported. Note that 27793 if a modification time needs to be updated on an object then it will 27794 create a new version of the object. 27795 27796 ### Restricted filename characters 27797 27798 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 27799 the following characters are also replaced: 27800 27801 | Character | Value | Replacement | 27802 | --------- |:-----:|:-----------:| 27803 | \ | 0x5C | \ | 27804 27805 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 27806 as they can't be used in JSON strings. 27807 27808 Note that in 2020-05 Backblaze started allowing \ characters in file 27809 names. Rclone hasn't changed its encoding as this could cause syncs to 27810 re-transfer files. If you want rclone not to replace \ then see the 27811 `--b2-encoding` flag below and remove the `BackSlash` from the 27812 string. This can be set in the config. 27813 27814 ### SHA1 checksums 27815 27816 The SHA1 checksums of the files are checked on upload and download and 27817 will be used in the syncing process. 27818 27819 Large files (bigger than the limit in `--b2-upload-cutoff`) which are 27820 uploaded in chunks will store their SHA1 on the object as 27821 `X-Bz-Info-large_file_sha1` as recommended by Backblaze. 27822 27823 For a large file to be uploaded with an SHA1 checksum, the source 27824 needs to support SHA1 checksums. The local disk supports SHA1 27825 checksums so large file transfers from local disk will have an SHA1. 27826 See [the overview](https://rclone.org/overview/#features) for exactly which remotes 27827 support SHA1. 27828 27829 Sources which don't support SHA1, in particular `crypt` will upload 27830 large files without SHA1 checksums. This may be fixed in the future 27831 (see [#1767](https://github.com/rclone/rclone/issues/1767)). 27832 27833 Files sizes below `--b2-upload-cutoff` will always have an SHA1 27834 regardless of the source. 27835 27836 ### Transfers 27837 27838 Backblaze recommends that you do lots of transfers simultaneously for 27839 maximum speed. In tests from my SSD equipped laptop the optimum 27840 setting is about `--transfers 32` though higher numbers may be used 27841 for a slight speed improvement. The optimum number for you may vary 27842 depending on your hardware, how big the files are, how much you want 27843 to load your computer, etc. The default of `--transfers 4` is 27844 definitely too low for Backblaze B2 though. 27845 27846 Note that uploading big files (bigger than 200 MiB by default) will use 27847 a 96 MiB RAM buffer by default. There can be at most `--transfers` of 27848 these in use at any moment, so this sets the upper limit on the memory 27849 used. 27850 27851 ### Versions 27852 27853 When rclone uploads a new version of a file it creates a [new version 27854 of it](https://www.backblaze.com/b2/docs/file_versions.html). 27855 Likewise when you delete a file, the old version will be marked hidden 27856 and still be available. Conversely, you may opt in to a "hard delete" 27857 of files with the `--b2-hard-delete` flag which would permanently remove 27858 the file instead of hiding it. 27859 27860 Old versions of files, where available, are visible using the 27861 `--b2-versions` flag. 27862 27863 It is also possible to view a bucket as it was at a certain point in time, 27864 using the `--b2-version-at` flag. This will show the file versions as they 27865 were at that time, showing files that have been deleted afterwards, and 27866 hiding files that were created since. 27867 27868 If you wish to remove all the old versions then you can use the 27869 `rclone cleanup remote:bucket` command which will delete all the old 27870 versions of files, leaving the current ones intact. You can also 27871 supply a path and only old versions under that path will be deleted, 27872 e.g. `rclone cleanup remote:bucket/path/to/stuff`. 27873 27874 Note that `cleanup` will remove partially uploaded files from the bucket 27875 if they are more than a day old. 27876 27877 When you `purge` a bucket, the current and the old versions will be 27878 deleted then the bucket will be deleted. 27879 27880 However `delete` will cause the current versions of the files to 27881 become hidden old versions. 27882 27883 Here is a session showing the listing and retrieval of an old 27884 version followed by a `cleanup` of the old versions. 27885 27886 Show current version and all the versions with `--b2-versions` flag. 27887 27888 $ rclone -q ls b2:cleanup-test 9 one.txt 27889 27890 $ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 27891 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 27892 one-v2016-07-02-155621-000.txt 27893 27894 27895 Retrieve an old version 27896 27897 $ rclone -q --b2-versions copy 27898 b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp 27899 27900 $ ls -l /tmp/one-v2016-07-04-141003-000.txt -rw-rw-r-- 1 ncw ncw 16 Jul 27901 2 17:46 /tmp/one-v2016-07-04-141003-000.txt 27902 27903 27904 Clean up all the old versions and show that they've gone. 27905 27906 $ rclone -q cleanup b2:cleanup-test 27907 27908 $ rclone -q ls b2:cleanup-test 9 one.txt 27909 27910 $ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 27911 27912 27913 #### Versions naming caveat 27914 27915 When using `--b2-versions` flag rclone is relying on the file name 27916 to work out whether the objects are versions or not. Versions' names 27917 are created by inserting timestamp between file name and its extension. 27918 27919 9 file.txt 27920 8 file-v2023-07-17-161032-000.txt 27921 16 file-v2023-06-15-141003-000.txt 27922 27923 If there are real files present with the same names as versions, then 27924 behaviour of `--b2-versions` can be unpredictable. 27925 27926 ### Data usage 27927 27928 It is useful to know how many requests are sent to the server in different scenarios. 27929 27930 All copy commands send the following 4 requests: 27931 27932 /b2api/v1/b2_authorize_account /b2api/v1/b2_create_bucket 27933 /b2api/v1/b2_list_buckets /b2api/v1/b2_list_file_names 27934 27935 27936 The `b2_list_file_names` request will be sent once for every 1k files 27937 in the remote path, providing the checksum and modification time of 27938 the listed files. As of version 1.33 issue 27939 [#818](https://github.com/rclone/rclone/issues/818) causes extra requests 27940 to be sent when using B2 with Crypt. When a copy operation does not 27941 require any files to be uploaded, no more requests will be sent. 27942 27943 Uploading files that do not require chunking, will send 2 requests per 27944 file upload: 27945 27946 /b2api/v1/b2_get_upload_url /b2api/v1/b2_upload_file/ 27947 27948 27949 Uploading files requiring chunking, will send 2 requests (one each to 27950 start and finish the upload) and another 2 requests for each chunk: 27951 27952 /b2api/v1/b2_start_large_file /b2api/v1/b2_get_upload_part_url 27953 /b2api/v1/b2_upload_part/ /b2api/v1/b2_finish_large_file 27954 27955 27956 #### Versions 27957 27958 Versions can be viewed with the `--b2-versions` flag. When it is set 27959 rclone will show and act on older versions of files. For example 27960 27961 Listing without `--b2-versions` 27962 27963 $ rclone -q ls b2:cleanup-test 9 one.txt 27964 27965 27966 And with 27967 27968 $ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 27969 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 27970 one-v2016-07-02-155621-000.txt 27971 27972 27973 Showing that the current version is unchanged but older versions can 27974 be seen. These have the UTC date that they were uploaded to the 27975 server to the nearest millisecond appended to them. 27976 27977 Note that when using `--b2-versions` no file write operations are 27978 permitted, so you can't upload files or delete them. 27979 27980 ### B2 and rclone link 27981 27982 Rclone supports generating file share links for private B2 buckets. 27983 They can either be for a file for example: 27984 27985 ./rclone link B2:bucket/path/to/file.txt 27986 https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx 27987 27988 27989 or if run on a directory you will get: 27990 27991 ./rclone link B2:bucket/path 27992 https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx 27993 27994 27995 you can then use the authorization token (the part of the url from the 27996 `?Authorization=` on) on any file path under that directory. For example: 27997 27998 https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx 27999 https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx 28000 https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx 28001 28002 28003 28004 ### Standard options 28005 28006 Here are the Standard options specific to b2 (Backblaze B2). 28007 28008 #### --b2-account 28009 28010 Account ID or Application Key ID. 28011 28012 Properties: 28013 28014 - Config: account 28015 - Env Var: RCLONE_B2_ACCOUNT 28016 - Type: string 28017 - Required: true 28018 28019 #### --b2-key 28020 28021 Application Key. 28022 28023 Properties: 28024 28025 - Config: key 28026 - Env Var: RCLONE_B2_KEY 28027 - Type: string 28028 - Required: true 28029 28030 #### --b2-hard-delete 28031 28032 Permanently delete files on remote removal, otherwise hide files. 28033 28034 Properties: 28035 28036 - Config: hard_delete 28037 - Env Var: RCLONE_B2_HARD_DELETE 28038 - Type: bool 28039 - Default: false 28040 28041 ### Advanced options 28042 28043 Here are the Advanced options specific to b2 (Backblaze B2). 28044 28045 #### --b2-endpoint 28046 28047 Endpoint for the service. 28048 28049 Leave blank normally. 28050 28051 Properties: 28052 28053 - Config: endpoint 28054 - Env Var: RCLONE_B2_ENDPOINT 28055 - Type: string 28056 - Required: false 28057 28058 #### --b2-test-mode 28059 28060 A flag string for X-Bz-Test-Mode header for debugging. 28061 28062 This is for debugging purposes only. Setting it to one of the strings 28063 below will cause b2 to return specific errors: 28064 28065 * "fail_some_uploads" 28066 * "expire_some_account_authorization_tokens" 28067 * "force_cap_exceeded" 28068 28069 These will be set in the "X-Bz-Test-Mode" header which is documented 28070 in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html). 28071 28072 Properties: 28073 28074 - Config: test_mode 28075 - Env Var: RCLONE_B2_TEST_MODE 28076 - Type: string 28077 - Required: false 28078 28079 #### --b2-versions 28080 28081 Include old versions in directory listings. 28082 28083 Note that when using this no file write operations are permitted, 28084 so you can't upload files or delete them. 28085 28086 Properties: 28087 28088 - Config: versions 28089 - Env Var: RCLONE_B2_VERSIONS 28090 - Type: bool 28091 - Default: false 28092 28093 #### --b2-version-at 28094 28095 Show file versions as they were at the specified time. 28096 28097 Note that when using this no file write operations are permitted, 28098 so you can't upload files or delete them. 28099 28100 Properties: 28101 28102 - Config: version_at 28103 - Env Var: RCLONE_B2_VERSION_AT 28104 - Type: Time 28105 - Default: off 28106 28107 #### --b2-upload-cutoff 28108 28109 Cutoff for switching to chunked upload. 28110 28111 Files above this size will be uploaded in chunks of "--b2-chunk-size". 28112 28113 This value should be set no larger than 4.657 GiB (== 5 GB). 28114 28115 Properties: 28116 28117 - Config: upload_cutoff 28118 - Env Var: RCLONE_B2_UPLOAD_CUTOFF 28119 - Type: SizeSuffix 28120 - Default: 200Mi 28121 28122 #### --b2-copy-cutoff 28123 28124 Cutoff for switching to multipart copy. 28125 28126 Any files larger than this that need to be server-side copied will be 28127 copied in chunks of this size. 28128 28129 The minimum is 0 and the maximum is 4.6 GiB. 28130 28131 Properties: 28132 28133 - Config: copy_cutoff 28134 - Env Var: RCLONE_B2_COPY_CUTOFF 28135 - Type: SizeSuffix 28136 - Default: 4Gi 28137 28138 #### --b2-chunk-size 28139 28140 Upload chunk size. 28141 28142 When uploading large files, chunk the file into this size. 28143 28144 Must fit in memory. These chunks are buffered in memory and there 28145 might a maximum of "--transfers" chunks in progress at once. 28146 28147 5,000,000 Bytes is the minimum size. 28148 28149 Properties: 28150 28151 - Config: chunk_size 28152 - Env Var: RCLONE_B2_CHUNK_SIZE 28153 - Type: SizeSuffix 28154 - Default: 96Mi 28155 28156 #### --b2-upload-concurrency 28157 28158 Concurrency for multipart uploads. 28159 28160 This is the number of chunks of the same file that are uploaded 28161 concurrently. 28162 28163 Note that chunks are stored in memory and there may be up to 28164 "--transfers" * "--b2-upload-concurrency" chunks stored at once 28165 in memory. 28166 28167 Properties: 28168 28169 - Config: upload_concurrency 28170 - Env Var: RCLONE_B2_UPLOAD_CONCURRENCY 28171 - Type: int 28172 - Default: 4 28173 28174 #### --b2-disable-checksum 28175 28176 Disable checksums for large (> upload cutoff) files. 28177 28178 Normally rclone will calculate the SHA1 checksum of the input before 28179 uploading it so it can add it to metadata on the object. This is great 28180 for data integrity checking but can cause long delays for large files 28181 to start uploading. 28182 28183 Properties: 28184 28185 - Config: disable_checksum 28186 - Env Var: RCLONE_B2_DISABLE_CHECKSUM 28187 - Type: bool 28188 - Default: false 28189 28190 #### --b2-download-url 28191 28192 Custom endpoint for downloads. 28193 28194 This is usually set to a Cloudflare CDN URL as Backblaze offers 28195 free egress for data downloaded through the Cloudflare network. 28196 Rclone works with private buckets by sending an "Authorization" header. 28197 If the custom endpoint rewrites the requests for authentication, 28198 e.g., in Cloudflare Workers, this header needs to be handled properly. 28199 Leave blank if you want to use the endpoint provided by Backblaze. 28200 28201 The URL provided here SHOULD have the protocol and SHOULD NOT have 28202 a trailing slash or specify the /file/bucket subpath as rclone will 28203 request files with "{download_url}/file/{bucket_name}/{path}". 28204 28205 Example: 28206 > https://mysubdomain.mydomain.tld 28207 (No trailing "/", "file" or "bucket") 28208 28209 Properties: 28210 28211 - Config: download_url 28212 - Env Var: RCLONE_B2_DOWNLOAD_URL 28213 - Type: string 28214 - Required: false 28215 28216 #### --b2-download-auth-duration 28217 28218 Time before the public link authorization token will expire in s or suffix ms|s|m|h|d. 28219 28220 This is used in combination with "rclone link" for making files 28221 accessible to the public and sets the duration before the download 28222 authorization token will expire. 28223 28224 The minimum value is 1 second. The maximum value is one week. 28225 28226 Properties: 28227 28228 - Config: download_auth_duration 28229 - Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION 28230 - Type: Duration 28231 - Default: 1w 28232 28233 #### --b2-memory-pool-flush-time 28234 28235 How often internal memory buffer pools will be flushed. (no longer used) 28236 28237 Properties: 28238 28239 - Config: memory_pool_flush_time 28240 - Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME 28241 - Type: Duration 28242 - Default: 1m0s 28243 28244 #### --b2-memory-pool-use-mmap 28245 28246 Whether to use mmap buffers in internal memory pool. (no longer used) 28247 28248 Properties: 28249 28250 - Config: memory_pool_use_mmap 28251 - Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP 28252 - Type: bool 28253 - Default: false 28254 28255 #### --b2-lifecycle 28256 28257 Set the number of days deleted files should be kept when creating a bucket. 28258 28259 On bucket creation, this parameter is used to create a lifecycle rule 28260 for the entire bucket. 28261 28262 If lifecycle is 0 (the default) it does not create a lifecycle rule so 28263 the default B2 behaviour applies. This is to create versions of files 28264 on delete and overwrite and to keep them indefinitely. 28265 28266 If lifecycle is >0 then it creates a single rule setting the number of 28267 days before a file that is deleted or overwritten is deleted 28268 permanently. This is known as daysFromHidingToDeleting in the b2 docs. 28269 28270 The minimum value for this parameter is 1 day. 28271 28272 You can also enable hard_delete in the config also which will mean 28273 deletions won't cause versions but overwrites will still cause 28274 versions to be made. 28275 28276 See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket creation. 28277 28278 28279 Properties: 28280 28281 - Config: lifecycle 28282 - Env Var: RCLONE_B2_LIFECYCLE 28283 - Type: int 28284 - Default: 0 28285 28286 #### --b2-encoding 28287 28288 The encoding for the backend. 28289 28290 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 28291 28292 Properties: 28293 28294 - Config: encoding 28295 - Env Var: RCLONE_B2_ENCODING 28296 - Type: Encoding 28297 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 28298 28299 #### --b2-description 28300 28301 Description of the remote 28302 28303 Properties: 28304 28305 - Config: description 28306 - Env Var: RCLONE_B2_DESCRIPTION 28307 - Type: string 28308 - Required: false 28309 28310 ## Backend commands 28311 28312 Here are the commands specific to the b2 backend. 28313 28314 Run them with 28315 28316 rclone backend COMMAND remote: 28317 28318 The help below will explain what arguments each command takes. 28319 28320 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 28321 info on how to pass options and arguments. 28322 28323 These can be run on a running backend using the rc command 28324 [backend/command](https://rclone.org/rc/#backend-command). 28325 28326 ### lifecycle 28327 28328 Read or set the lifecycle for a bucket 28329 28330 rclone backend lifecycle remote: [options] [<arguments>+] 28331 28332 This command can be used to read or set the lifecycle for a bucket. 28333 28334 Usage Examples: 28335 28336 To show the current lifecycle rules: 28337 28338 rclone backend lifecycle b2:bucket 28339 28340 This will dump something like this showing the lifecycle rules. 28341 28342 [ 28343 { 28344 "daysFromHidingToDeleting": 1, 28345 "daysFromUploadingToHiding": null, 28346 "fileNamePrefix": "" 28347 } 28348 ] 28349 28350 If there are no lifecycle rules (the default) then it will just return []. 28351 28352 To reset the current lifecycle rules: 28353 28354 rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30 28355 rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1 28356 28357 This will run and then print the new lifecycle rules as above. 28358 28359 Rclone only lets you set lifecycles for the whole bucket with the 28360 fileNamePrefix = "". 28361 28362 You can't disable versioning with B2. The best you can do is to set 28363 the daysFromHidingToDeleting to 1 day. You can enable hard_delete in 28364 the config also which will mean deletions won't cause versions but 28365 overwrites will still cause versions to be made. 28366 28367 rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1 28368 28369 See: https://www.backblaze.com/docs/cloud-storage-lifecycle-rules 28370 28371 28372 Options: 28373 28374 - "daysFromHidingToDeleting": After a file has been hidden for this many days it is deleted. 0 is off. 28375 - "daysFromUploadingToHiding": This many days after uploading a file is hidden 28376 28377 28378 28379 ## Limitations 28380 28381 `rclone about` is not supported by the B2 backend. Backends without 28382 this capability cannot determine free space for an rclone mount or 28383 use policy `mfs` (most free space) as a member of an rclone union 28384 remote. 28385 28386 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 28387 28388 # Box 28389 28390 Paths are specified as `remote:path` 28391 28392 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 28393 28394 The initial setup for Box involves getting a token from Box which you 28395 can do either in your browser, or with a config.json downloaded from Box 28396 to use JWT authentication. `rclone config` walks you through it. 28397 28398 ## Configuration 28399 28400 Here is an example of how to make a remote called `remote`. First run: 28401 28402 rclone config 28403 28404 This will guide you through an interactive setup process: 28405 28406 No remotes found, make a new one? n) New remote s) Set configuration 28407 password q) Quit config n/s/q> n name> remote Type of storage to 28408 configure. Choose a number from below, or type in your own value [snip] 28409 XX / Box "box" [snip] Storage> box Box App Client Id - leave blank 28410 normally. client_id> Box App Client Secret - leave blank normally. 28411 client_secret> Box App config.json location Leave blank normally. Enter 28412 a string value. Press Enter for the default (""). box_config_file> Box 28413 App Primary Access Token Leave blank normally. Enter a string value. 28414 Press Enter for the default (""). access_token> 28415 28416 Enter a string value. Press Enter for the default ("user"). Choose a 28417 number from below, or type in your own value 1 / Rclone should act on 28418 behalf of a user "user" 2 / Rclone should act on behalf of a service 28419 account "enterprise" box_sub_type> Remote config Use web browser to 28420 automatically authenticate rclone with remote? * Say Y if the machine 28421 running rclone has a web browser you can use * Say N if running rclone 28422 on a (remote) machine without web browser access If not sure try Y. If Y 28423 failed, try N. y) Yes n) No y/n> y If your browser doesn't open 28424 automatically go to the following link: http://127.0.0.1:53682/auth Log 28425 in and authorize rclone for access Waiting for code... Got code 28426 -------------------- [remote] client_id = client_secret = token = 28427 {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} 28428 -------------------- y) Yes this is OK e) Edit this remote d) Delete 28429 this remote y/e/d> y 28430 28431 28432 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 28433 machine with no Internet browser available. 28434 28435 Note that rclone runs a webserver on your local machine to collect the 28436 token as returned from Box. This only runs from the moment it opens 28437 your browser to the moment you get back the verification code. This 28438 is on `http://127.0.0.1:53682/` and this it may require you to unblock 28439 it temporarily if you are running a host firewall. 28440 28441 Once configured you can then use `rclone` like this, 28442 28443 List directories in top level of your Box 28444 28445 rclone lsd remote: 28446 28447 List all the files in your Box 28448 28449 rclone ls remote: 28450 28451 To copy a local directory to an Box directory called backup 28452 28453 rclone copy /home/source remote:backup 28454 28455 ### Using rclone with an Enterprise account with SSO 28456 28457 If you have an "Enterprise" account type with Box with single sign on 28458 (SSO), you need to create a password to use Box with rclone. This can 28459 be done at your Enterprise Box account by going to Settings, "Account" 28460 Tab, and then set the password in the "Authentication" field. 28461 28462 Once you have done this, you can setup your Enterprise Box account 28463 using the same procedure detailed above in the, using the password you 28464 have just set. 28465 28466 ### Invalid refresh token 28467 28468 According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens): 28469 28470 > Each refresh_token is valid for one use in 60 days. 28471 28472 This means that if you 28473 28474 * Don't use the box remote for 60 days 28475 * Copy the config file with a box refresh token in and use it in two places 28476 * Get an error on a token refresh 28477 28478 then rclone will return an error which includes the text `Invalid 28479 refresh token`. 28480 28481 To fix this you will need to use oauth2 again to update the refresh 28482 token. You can use the methods in [the remote setup 28483 docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the 28484 config file method, you should not use that remote on the computer you 28485 did the authentication on. 28486 28487 Here is how to do it. 28488 28489 $ rclone config Current remotes: 28490 28491 Name Type ==== ==== remote box 28492 28493 e) Edit existing remote 28494 f) New remote 28495 g) Delete remote 28496 h) Rename remote 28497 i) Copy remote 28498 j) Set configuration password 28499 k) Quit config e/n/d/r/c/s/q> e Choose a number from below, or type in 28500 an existing value 1 > remote remote> remote -------------------- 28501 [remote] type = box token = 28502 {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} 28503 -------------------- Edit remote Value "client_id" = "" Edit? (y/n)> 28504 l) Yes 28505 m) No y/n> n Value "client_secret" = "" Edit? (y/n)> 28506 n) Yes 28507 o) No y/n> n Remote config Already have a token - refresh? 28508 p) Yes 28509 q) No y/n> y Use web browser to automatically authenticate rclone with 28510 remote? 28511 28512 - Say Y if the machine running rclone has a web browser you can use 28513 - Say N if running rclone on a (remote) machine without web browser 28514 access If not sure try Y. If Y failed, try N. 28515 28516 y) Yes 28517 z) No y/n> y If your browser doesn't open automatically go to the 28518 following link: http://127.0.0.1:53682/auth Log in and authorize 28519 rclone for access Waiting for code... Got code -------------------- 28520 [remote] type = box token = 28521 {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} 28522 -------------------- 28523 a) Yes this is OK 28524 b) Edit this remote 28525 c) Delete this remote y/e/d> y 28526 28527 28528 ### Modification times and hashes 28529 28530 Box allows modification times to be set on objects accurate to 1 28531 second. These will be used to detect whether objects need syncing or 28532 not. 28533 28534 Box supports SHA1 type hashes, so you can use the `--checksum` 28535 flag. 28536 28537 ### Restricted filename characters 28538 28539 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 28540 the following characters are also replaced: 28541 28542 | Character | Value | Replacement | 28543 | --------- |:-----:|:-----------:| 28544 | \ | 0x5C | \ | 28545 28546 File names can also not end with the following characters. 28547 These only get replaced if they are the last character in the name: 28548 28549 | Character | Value | Replacement | 28550 | --------- |:-----:|:-----------:| 28551 | SP | 0x20 | ␠ | 28552 28553 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 28554 as they can't be used in JSON strings. 28555 28556 ### Transfers 28557 28558 For files above 50 MiB rclone will use a chunked transfer. Rclone will 28559 upload up to `--transfers` chunks at the same time (shared among all 28560 the multipart uploads). Chunks are buffered in memory and are 28561 normally 8 MiB so increasing `--transfers` will increase memory use. 28562 28563 ### Deleting files 28564 28565 Depending on the enterprise settings for your user, the item will 28566 either be actually deleted from Box or moved to the trash. 28567 28568 Emptying the trash is supported via the rclone however cleanup command 28569 however this deletes every trashed file and folder individually so it 28570 may take a very long time. 28571 Emptying the trash via the WebUI does not have this limitation 28572 so it is advised to empty the trash via the WebUI. 28573 28574 ### Root folder ID 28575 28576 You can set the `root_folder_id` for rclone. This is the directory 28577 (identified by its `Folder ID`) that rclone considers to be the root 28578 of your Box drive. 28579 28580 Normally you will leave this blank and rclone will determine the 28581 correct root to use itself. 28582 28583 However you can set this to restrict rclone to a specific folder 28584 hierarchy. 28585 28586 In order to do this you will have to find the `Folder ID` of the 28587 directory you wish rclone to display. This will be the last segment 28588 of the URL when you open the relevant folder in the Box web 28589 interface. 28590 28591 So if the folder you want rclone to use has a URL which looks like 28592 `https://app.box.com/folder/11xxxxxxxxx8` 28593 in the browser, then you use `11xxxxxxxxx8` as 28594 the `root_folder_id` in the config. 28595 28596 28597 ### Standard options 28598 28599 Here are the Standard options specific to box (Box). 28600 28601 #### --box-client-id 28602 28603 OAuth Client Id. 28604 28605 Leave blank normally. 28606 28607 Properties: 28608 28609 - Config: client_id 28610 - Env Var: RCLONE_BOX_CLIENT_ID 28611 - Type: string 28612 - Required: false 28613 28614 #### --box-client-secret 28615 28616 OAuth Client Secret. 28617 28618 Leave blank normally. 28619 28620 Properties: 28621 28622 - Config: client_secret 28623 - Env Var: RCLONE_BOX_CLIENT_SECRET 28624 - Type: string 28625 - Required: false 28626 28627 #### --box-box-config-file 28628 28629 Box App config.json location 28630 28631 Leave blank normally. 28632 28633 Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`. 28634 28635 Properties: 28636 28637 - Config: box_config_file 28638 - Env Var: RCLONE_BOX_BOX_CONFIG_FILE 28639 - Type: string 28640 - Required: false 28641 28642 #### --box-access-token 28643 28644 Box App Primary Access Token 28645 28646 Leave blank normally. 28647 28648 Properties: 28649 28650 - Config: access_token 28651 - Env Var: RCLONE_BOX_ACCESS_TOKEN 28652 - Type: string 28653 - Required: false 28654 28655 #### --box-box-sub-type 28656 28657 28658 28659 Properties: 28660 28661 - Config: box_sub_type 28662 - Env Var: RCLONE_BOX_BOX_SUB_TYPE 28663 - Type: string 28664 - Default: "user" 28665 - Examples: 28666 - "user" 28667 - Rclone should act on behalf of a user. 28668 - "enterprise" 28669 - Rclone should act on behalf of a service account. 28670 28671 ### Advanced options 28672 28673 Here are the Advanced options specific to box (Box). 28674 28675 #### --box-token 28676 28677 OAuth Access Token as a JSON blob. 28678 28679 Properties: 28680 28681 - Config: token 28682 - Env Var: RCLONE_BOX_TOKEN 28683 - Type: string 28684 - Required: false 28685 28686 #### --box-auth-url 28687 28688 Auth server URL. 28689 28690 Leave blank to use the provider defaults. 28691 28692 Properties: 28693 28694 - Config: auth_url 28695 - Env Var: RCLONE_BOX_AUTH_URL 28696 - Type: string 28697 - Required: false 28698 28699 #### --box-token-url 28700 28701 Token server url. 28702 28703 Leave blank to use the provider defaults. 28704 28705 Properties: 28706 28707 - Config: token_url 28708 - Env Var: RCLONE_BOX_TOKEN_URL 28709 - Type: string 28710 - Required: false 28711 28712 #### --box-root-folder-id 28713 28714 Fill in for rclone to use a non root folder as its starting point. 28715 28716 Properties: 28717 28718 - Config: root_folder_id 28719 - Env Var: RCLONE_BOX_ROOT_FOLDER_ID 28720 - Type: string 28721 - Default: "0" 28722 28723 #### --box-upload-cutoff 28724 28725 Cutoff for switching to multipart upload (>= 50 MiB). 28726 28727 Properties: 28728 28729 - Config: upload_cutoff 28730 - Env Var: RCLONE_BOX_UPLOAD_CUTOFF 28731 - Type: SizeSuffix 28732 - Default: 50Mi 28733 28734 #### --box-commit-retries 28735 28736 Max number of times to try committing a multipart file. 28737 28738 Properties: 28739 28740 - Config: commit_retries 28741 - Env Var: RCLONE_BOX_COMMIT_RETRIES 28742 - Type: int 28743 - Default: 100 28744 28745 #### --box-list-chunk 28746 28747 Size of listing chunk 1-1000. 28748 28749 Properties: 28750 28751 - Config: list_chunk 28752 - Env Var: RCLONE_BOX_LIST_CHUNK 28753 - Type: int 28754 - Default: 1000 28755 28756 #### --box-owned-by 28757 28758 Only show items owned by the login (email address) passed in. 28759 28760 Properties: 28761 28762 - Config: owned_by 28763 - Env Var: RCLONE_BOX_OWNED_BY 28764 - Type: string 28765 - Required: false 28766 28767 #### --box-impersonate 28768 28769 Impersonate this user ID when using a service account. 28770 28771 Setting this flag allows rclone, when using a JWT service account, to 28772 act on behalf of another user by setting the as-user header. 28773 28774 The user ID is the Box identifier for a user. User IDs can found for 28775 any user via the GET /users endpoint, which is only available to 28776 admins, or by calling the GET /users/me endpoint with an authenticated 28777 user session. 28778 28779 See: https://developer.box.com/guides/authentication/jwt/as-user/ 28780 28781 28782 Properties: 28783 28784 - Config: impersonate 28785 - Env Var: RCLONE_BOX_IMPERSONATE 28786 - Type: string 28787 - Required: false 28788 28789 #### --box-encoding 28790 28791 The encoding for the backend. 28792 28793 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 28794 28795 Properties: 28796 28797 - Config: encoding 28798 - Env Var: RCLONE_BOX_ENCODING 28799 - Type: Encoding 28800 - Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot 28801 28802 #### --box-description 28803 28804 Description of the remote 28805 28806 Properties: 28807 28808 - Config: description 28809 - Env Var: RCLONE_BOX_DESCRIPTION 28810 - Type: string 28811 - Required: false 28812 28813 28814 28815 ## Limitations 28816 28817 Note that Box is case insensitive so you can't have a file called 28818 "Hello.doc" and one called "hello.doc". 28819 28820 Box file names can't have the `\` character in. rclone maps this to 28821 and from an identical looking unicode equivalent `\` (U+FF3C Fullwidth 28822 Reverse Solidus). 28823 28824 Box only supports filenames up to 255 characters in length. 28825 28826 Box has [API rate limits](https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits/) that sometimes reduce the speed of rclone. 28827 28828 `rclone about` is not supported by the Box backend. Backends without 28829 this capability cannot determine free space for an rclone mount or 28830 use policy `mfs` (most free space) as a member of an rclone union 28831 remote. 28832 28833 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 28834 28835 ## Get your own Box App ID 28836 28837 Here is how to create your own Box App ID for rclone: 28838 28839 1. Go to the [Box Developer Console](https://app.box.com/developers/console) 28840 and login, then click `My Apps` on the sidebar. Click `Create New App` 28841 and select `Custom App`. 28842 28843 2. In the first screen on the box that pops up, you can pretty much enter 28844 whatever you want. The `App Name` can be whatever. For `Purpose` choose 28845 automation to avoid having to fill out anything else. Click `Next`. 28846 28847 3. In the second screen of the creation screen, select 28848 `User Authentication (OAuth 2.0)`. Then click `Create App`. 28849 28850 4. You should now be on the `Configuration` tab of your new app. If not, 28851 click on it at the top of the webpage. Copy down `Client ID` 28852 and `Client Secret`, you'll need those for rclone. 28853 28854 5. Under "OAuth 2.0 Redirect URI", add `http://127.0.0.1:53682/` 28855 28856 6. For `Application Scopes`, select `Read all files and folders stored in Box` 28857 and `Write all files and folders stored in box` (assuming you want to do both). 28858 Leave others unchecked. Click `Save Changes` at the top right. 28859 28860 # Cache 28861 28862 The `cache` remote wraps another existing remote and stores file structure 28863 and its data for long running tasks like `rclone mount`. 28864 28865 ## Status 28866 28867 The cache backend code is working but it currently doesn't 28868 have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed. 28869 28870 The cache backend is due to be phased out in favour of the VFS caching 28871 layer eventually which is more tightly integrated into rclone. 28872 28873 Until this happens we recommend only using the cache backend if you 28874 find you can't work without it. There are many docs online describing 28875 the use of the cache backend to minimize API hits and by-and-large 28876 these are out of date and the cache backend isn't needed in those 28877 scenarios any more. 28878 28879 ## Configuration 28880 28881 To get started you just need to have an existing remote which can be configured 28882 with `cache`. 28883 28884 Here is an example of how to make a remote called `test-cache`. First run: 28885 28886 rclone config 28887 28888 This will guide you through an interactive setup process: 28889 28890 No remotes found, make a new one? n) New remote r) Rename remote c) Copy 28891 remote s) Set configuration password q) Quit config n/r/c/s/q> n name> 28892 test-cache Type of storage to configure. Choose a number from below, or 28893 type in your own value [snip] XX / Cache a remote "cache" [snip] 28894 Storage> cache Remote to cache. Normally should contain a ':' and a 28895 path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe 28896 "myremote:" (not recommended). remote> local:/test Optional: The URL of 28897 the Plex server plex_url> http://127.0.0.1:32400 Optional: The username 28898 of the Plex user plex_username> dummyusername Optional: The password of 28899 the Plex user y) Yes type in my own password g) Generate random password 28900 n) No leave this optional password blank y/g/n> y Enter the password: 28901 password: Confirm the password: password: The size of a chunk. Lower 28902 value good for slow connections but can affect seamless reading. 28903 Default: 5M Choose a number from below, or type in your own value 1 / 1 28904 MiB "1M" 2 / 5 MiB "5M" 3 / 10 MiB "10M" chunk_size> 2 How much time 28905 should object info (file size, file hashes, etc.) be stored in cache. 28906 Use a very high value if you don't plan on changing the source FS from 28907 outside the cache. Accepted units are: "s", "m", "h". Default: 5m Choose 28908 a number from below, or type in your own value 1 / 1 hour "1h" 2 / 24 28909 hours "24h" 3 / 24 hours "48h" info_age> 2 The maximum size of stored 28910 chunks. When the storage grows beyond this size, the oldest chunks will 28911 be deleted. Default: 10G Choose a number from below, or type in your own 28912 value 1 / 500 MiB "500M" 2 / 1 GiB "1G" 3 / 10 GiB "10G" 28913 chunk_total_size> 3 Remote config -------------------- [test-cache] 28914 remote = local:/test plex_url = http://127.0.0.1:32400 plex_username = 28915 dummyusername plex_password = *** ENCRYPTED *** chunk_size = 5M info_age 28916 = 48h chunk_total_size = 10G 28917 28918 28919 You can then use it like this, 28920 28921 List directories in top level of your drive 28922 28923 rclone lsd test-cache: 28924 28925 List all the files in your drive 28926 28927 rclone ls test-cache: 28928 28929 To start a cached mount 28930 28931 rclone mount --allow-other test-cache: /var/tmp/test-cache 28932 28933 ### Write Features ### 28934 28935 ### Offline uploading ### 28936 28937 In an effort to make writing through cache more reliable, the backend 28938 now supports this feature which can be activated by specifying a 28939 `cache-tmp-upload-path`. 28940 28941 A files goes through these states when using this feature: 28942 28943 1. An upload is started (usually by copying a file on the cache remote) 28944 2. When the copy to the temporary location is complete the file is part 28945 of the cached remote and looks and behaves like any other file (reading included) 28946 3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move` 28947 is used to move the file to the cloud provider 28948 4. Reading the file still works during the upload but most modifications on it will be prohibited 28949 5. Once the move is complete the file is unlocked for modifications as it 28950 becomes as any other regular file 28951 6. If the file is being read through `cache` when it's actually 28952 deleted from the temporary path then `cache` will simply swap the source 28953 to the cloud provider without interrupting the reading (small blip can happen though) 28954 28955 Files are uploaded in sequence and only one file is uploaded at a time. 28956 Uploads will be stored in a queue and be processed based on the order they were added. 28957 The queue and the temporary storage is persistent across restarts but 28958 can be cleared on startup with the `--cache-db-purge` flag. 28959 28960 ### Write Support ### 28961 28962 Writes are supported through `cache`. 28963 One caveat is that a mounted cache remote does not add any retry or fallback 28964 mechanism to the upload operation. This will depend on the implementation 28965 of the wrapped remote. Consider using `Offline uploading` for reliable writes. 28966 28967 One special case is covered with `cache-writes` which will cache the file 28968 data at the same time as the upload when it is enabled making it available 28969 from the cache store immediately once the upload is finished. 28970 28971 ### Read Features ### 28972 28973 #### Multiple connections #### 28974 28975 To counter the high latency between a local PC where rclone is running 28976 and cloud providers, the cache remote can split multiple requests to the 28977 cloud provider for smaller file chunks and combines them together locally 28978 where they can be available almost immediately before the reader usually 28979 needs them. 28980 28981 This is similar to buffering when media files are played online. Rclone 28982 will stay around the current marker but always try its best to stay ahead 28983 and prepare the data before. 28984 28985 #### Plex Integration #### 28986 28987 There is a direct integration with Plex which allows cache to detect during reading 28988 if the file is in playback or not. This helps cache to adapt how it queries 28989 the cloud provider depending on what is needed for. 28990 28991 Scans will have a minimum amount of workers (1) while in a confirmed playback cache 28992 will deploy the configured number of workers. 28993 28994 This integration opens the doorway to additional performance improvements 28995 which will be explored in the near future. 28996 28997 **Note:** If Plex options are not configured, `cache` will function with its 28998 configured options without adapting any of its settings. 28999 29000 How to enable? Run `rclone config` and add all the Plex options (endpoint, username 29001 and password) in your remote and it will be automatically enabled. 29002 29003 Affected settings: 29004 - `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times 29005 29006 ##### Certificate Validation ##### 29007 29008 When the Plex server is configured to only accept secure connections, it is 29009 possible to use `.plex.direct` URLs to ensure certificate validation succeeds. 29010 These URLs are used by Plex internally to connect to the Plex server securely. 29011 29012 The format for these URLs is the following: 29013 29014 `https://ip-with-dots-replaced.server-hash.plex.direct:32400/` 29015 29016 The `ip-with-dots-replaced` part can be any IPv4 address, where the dots 29017 have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`. 29018 29019 To get the `server-hash` part, the easiest way is to visit 29020 29021 https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token 29022 29023 This page will list all the available Plex servers for your account 29024 with at least one `.plex.direct` link for each. Copy one URL and replace 29025 the IP address with the desired address. This can be used as the 29026 `plex_url` value. 29027 29028 ### Known issues ### 29029 29030 #### Mount and --dir-cache-time #### 29031 29032 --dir-cache-time controls the first layer of directory caching which works at the mount layer. 29033 Being an independent caching mechanism from the `cache` backend, it will manage its own entries 29034 based on the configured time. 29035 29036 To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct 29037 one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are 29038 already configured in this way. 29039 29040 #### Windows support - Experimental #### 29041 29042 There are a couple of issues with Windows `mount` functionality that still require some investigations. 29043 It should be considered as experimental thus far as fixes come in for this OS. 29044 29045 Most of the issues seem to be related to the difference between filesystems 29046 on Linux flavors and Windows as cache is heavily dependent on them. 29047 29048 Any reports or feedback on how cache behaves on this OS is greatly appreciated. 29049 29050 - https://github.com/rclone/rclone/issues/1935 29051 - https://github.com/rclone/rclone/issues/1907 29052 - https://github.com/rclone/rclone/issues/1834 29053 29054 #### Risk of throttling #### 29055 29056 Future iterations of the cache backend will make use of the pooling functionality 29057 of the cloud provider to synchronize and at the same time make writing through it 29058 more tolerant to failures. 29059 29060 There are a couple of enhancements in track to add these but in the meantime 29061 there is a valid concern that the expiring cache listings can lead to cloud provider 29062 throttles or bans due to repeated queries on it for very large mounts. 29063 29064 Some recommendations: 29065 - don't use a very small interval for entry information (`--cache-info-age`) 29066 - while writes aren't yet optimised, you can still write through `cache` which gives you the advantage 29067 of adding the file in the cache at the same time if configured to do so. 29068 29069 Future enhancements: 29070 29071 - https://github.com/rclone/rclone/issues/1937 29072 - https://github.com/rclone/rclone/issues/1936 29073 29074 #### cache and crypt #### 29075 29076 One common scenario is to keep your data encrypted in the cloud provider 29077 using the `crypt` remote. `crypt` uses a similar technique to wrap around 29078 an existing remote and handles this translation in a seamless way. 29079 29080 There is an issue with wrapping the remotes in this order: 29081 **cloud remote** -> **crypt** -> **cache** 29082 29083 During testing, I experienced a lot of bans with the remotes in this order. 29084 I suspect it might be related to how crypt opens files on the cloud provider 29085 which makes it think we're downloading the full file instead of small chunks. 29086 Organizing the remotes in this order yields better results: 29087 **cloud remote** -> **cache** -> **crypt** 29088 29089 #### absolute remote paths #### 29090 29091 `cache` can not differentiate between relative and absolute paths for the wrapped remote. 29092 Any path given in the `remote` config setting and on the command line will be passed to 29093 the wrapped remote as is, but for storing the chunks on disk the path will be made 29094 relative by removing any leading `/` character. 29095 29096 This behavior is irrelevant for most backend types, but there are backends where a leading `/` 29097 changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are 29098 relative to the root of the SSH server and paths without are relative to the user home directory. 29099 As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent 29100 a different directory on the SSH server. 29101 29102 ### Cache and Remote Control (--rc) ### 29103 Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points: 29104 By default, the listener is disabled if you do not add the flag. 29105 29106 ### rc cache/expire 29107 Purge a remote from the cache backend. Supports either a directory or a file. 29108 It supports both encrypted and unencrypted file names if cache is wrapped by crypt. 29109 29110 Params: 29111 - **remote** = path to remote **(required)** 29112 - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_ 29113 29114 29115 ### Standard options 29116 29117 Here are the Standard options specific to cache (Cache a remote). 29118 29119 #### --cache-remote 29120 29121 Remote to cache. 29122 29123 Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", 29124 "myremote:bucket" or maybe "myremote:" (not recommended). 29125 29126 Properties: 29127 29128 - Config: remote 29129 - Env Var: RCLONE_CACHE_REMOTE 29130 - Type: string 29131 - Required: true 29132 29133 #### --cache-plex-url 29134 29135 The URL of the Plex server. 29136 29137 Properties: 29138 29139 - Config: plex_url 29140 - Env Var: RCLONE_CACHE_PLEX_URL 29141 - Type: string 29142 - Required: false 29143 29144 #### --cache-plex-username 29145 29146 The username of the Plex user. 29147 29148 Properties: 29149 29150 - Config: plex_username 29151 - Env Var: RCLONE_CACHE_PLEX_USERNAME 29152 - Type: string 29153 - Required: false 29154 29155 #### --cache-plex-password 29156 29157 The password of the Plex user. 29158 29159 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 29160 29161 Properties: 29162 29163 - Config: plex_password 29164 - Env Var: RCLONE_CACHE_PLEX_PASSWORD 29165 - Type: string 29166 - Required: false 29167 29168 #### --cache-chunk-size 29169 29170 The size of a chunk (partial file data). 29171 29172 Use lower numbers for slower connections. If the chunk size is 29173 changed, any downloaded chunks will be invalid and cache-chunk-path 29174 will need to be cleared or unexpected EOF errors will occur. 29175 29176 Properties: 29177 29178 - Config: chunk_size 29179 - Env Var: RCLONE_CACHE_CHUNK_SIZE 29180 - Type: SizeSuffix 29181 - Default: 5Mi 29182 - Examples: 29183 - "1M" 29184 - 1 MiB 29185 - "5M" 29186 - 5 MiB 29187 - "10M" 29188 - 10 MiB 29189 29190 #### --cache-info-age 29191 29192 How long to cache file structure information (directory listings, file size, times, etc.). 29193 If all write operations are done through the cache then you can safely make 29194 this value very large as the cache store will also be updated in real time. 29195 29196 Properties: 29197 29198 - Config: info_age 29199 - Env Var: RCLONE_CACHE_INFO_AGE 29200 - Type: Duration 29201 - Default: 6h0m0s 29202 - Examples: 29203 - "1h" 29204 - 1 hour 29205 - "24h" 29206 - 24 hours 29207 - "48h" 29208 - 48 hours 29209 29210 #### --cache-chunk-total-size 29211 29212 The total size that the chunks can take up on the local disk. 29213 29214 If the cache exceeds this value then it will start to delete the 29215 oldest chunks until it goes under this value. 29216 29217 Properties: 29218 29219 - Config: chunk_total_size 29220 - Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE 29221 - Type: SizeSuffix 29222 - Default: 10Gi 29223 - Examples: 29224 - "500M" 29225 - 500 MiB 29226 - "1G" 29227 - 1 GiB 29228 - "10G" 29229 - 10 GiB 29230 29231 ### Advanced options 29232 29233 Here are the Advanced options specific to cache (Cache a remote). 29234 29235 #### --cache-plex-token 29236 29237 The plex token for authentication - auto set normally. 29238 29239 Properties: 29240 29241 - Config: plex_token 29242 - Env Var: RCLONE_CACHE_PLEX_TOKEN 29243 - Type: string 29244 - Required: false 29245 29246 #### --cache-plex-insecure 29247 29248 Skip all certificate verification when connecting to the Plex server. 29249 29250 Properties: 29251 29252 - Config: plex_insecure 29253 - Env Var: RCLONE_CACHE_PLEX_INSECURE 29254 - Type: string 29255 - Required: false 29256 29257 #### --cache-db-path 29258 29259 Directory to store file structure metadata DB. 29260 29261 The remote name is used as the DB file name. 29262 29263 Properties: 29264 29265 - Config: db_path 29266 - Env Var: RCLONE_CACHE_DB_PATH 29267 - Type: string 29268 - Default: "$HOME/.cache/rclone/cache-backend" 29269 29270 #### --cache-chunk-path 29271 29272 Directory to cache chunk files. 29273 29274 Path to where partial file data (chunks) are stored locally. The remote 29275 name is appended to the final path. 29276 29277 This config follows the "--cache-db-path". If you specify a custom 29278 location for "--cache-db-path" and don't specify one for "--cache-chunk-path" 29279 then "--cache-chunk-path" will use the same path as "--cache-db-path". 29280 29281 Properties: 29282 29283 - Config: chunk_path 29284 - Env Var: RCLONE_CACHE_CHUNK_PATH 29285 - Type: string 29286 - Default: "$HOME/.cache/rclone/cache-backend" 29287 29288 #### --cache-db-purge 29289 29290 Clear all the cached data for this remote on start. 29291 29292 Properties: 29293 29294 - Config: db_purge 29295 - Env Var: RCLONE_CACHE_DB_PURGE 29296 - Type: bool 29297 - Default: false 29298 29299 #### --cache-chunk-clean-interval 29300 29301 How often should the cache perform cleanups of the chunk storage. 29302 29303 The default value should be ok for most people. If you find that the 29304 cache goes over "cache-chunk-total-size" too often then try to lower 29305 this value to force it to perform cleanups more often. 29306 29307 Properties: 29308 29309 - Config: chunk_clean_interval 29310 - Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL 29311 - Type: Duration 29312 - Default: 1m0s 29313 29314 #### --cache-read-retries 29315 29316 How many times to retry a read from a cache storage. 29317 29318 Since reading from a cache stream is independent from downloading file 29319 data, readers can get to a point where there's no more data in the 29320 cache. Most of the times this can indicate a connectivity issue if 29321 cache isn't able to provide file data anymore. 29322 29323 For really slow connections, increase this to a point where the stream is 29324 able to provide data but your experience will be very stuttering. 29325 29326 Properties: 29327 29328 - Config: read_retries 29329 - Env Var: RCLONE_CACHE_READ_RETRIES 29330 - Type: int 29331 - Default: 10 29332 29333 #### --cache-workers 29334 29335 How many workers should run in parallel to download chunks. 29336 29337 Higher values will mean more parallel processing (better CPU needed) 29338 and more concurrent requests on the cloud provider. This impacts 29339 several aspects like the cloud provider API limits, more stress on the 29340 hardware that rclone runs on but it also means that streams will be 29341 more fluid and data will be available much more faster to readers. 29342 29343 **Note**: If the optional Plex integration is enabled then this 29344 setting will adapt to the type of reading performed and the value 29345 specified here will be used as a maximum number of workers to use. 29346 29347 Properties: 29348 29349 - Config: workers 29350 - Env Var: RCLONE_CACHE_WORKERS 29351 - Type: int 29352 - Default: 4 29353 29354 #### --cache-chunk-no-memory 29355 29356 Disable the in-memory cache for storing chunks during streaming. 29357 29358 By default, cache will keep file data during streaming in RAM as well 29359 to provide it to readers as fast as possible. 29360 29361 This transient data is evicted as soon as it is read and the number of 29362 chunks stored doesn't exceed the number of workers. However, depending 29363 on other settings like "cache-chunk-size" and "cache-workers" this footprint 29364 can increase if there are parallel streams too (multiple files being read 29365 at the same time). 29366 29367 If the hardware permits it, use this feature to provide an overall better 29368 performance during streaming but it can also be disabled if RAM is not 29369 available on the local machine. 29370 29371 Properties: 29372 29373 - Config: chunk_no_memory 29374 - Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY 29375 - Type: bool 29376 - Default: false 29377 29378 #### --cache-rps 29379 29380 Limits the number of requests per second to the source FS (-1 to disable). 29381 29382 This setting places a hard limit on the number of requests per second 29383 that cache will be doing to the cloud provider remote and try to 29384 respect that value by setting waits between reads. 29385 29386 If you find that you're getting banned or limited on the cloud 29387 provider through cache and know that a smaller number of requests per 29388 second will allow you to work with it then you can use this setting 29389 for that. 29390 29391 A good balance of all the other settings should make this setting 29392 useless but it is available to set for more special cases. 29393 29394 **NOTE**: This will limit the number of requests during streams but 29395 other API calls to the cloud provider like directory listings will 29396 still pass. 29397 29398 Properties: 29399 29400 - Config: rps 29401 - Env Var: RCLONE_CACHE_RPS 29402 - Type: int 29403 - Default: -1 29404 29405 #### --cache-writes 29406 29407 Cache file data on writes through the FS. 29408 29409 If you need to read files immediately after you upload them through 29410 cache you can enable this flag to have their data stored in the 29411 cache store at the same time during upload. 29412 29413 Properties: 29414 29415 - Config: writes 29416 - Env Var: RCLONE_CACHE_WRITES 29417 - Type: bool 29418 - Default: false 29419 29420 #### --cache-tmp-upload-path 29421 29422 Directory to keep temporary files until they are uploaded. 29423 29424 This is the path where cache will use as a temporary storage for new 29425 files that need to be uploaded to the cloud provider. 29426 29427 Specifying a value will enable this feature. Without it, it is 29428 completely disabled and files will be uploaded directly to the cloud 29429 provider 29430 29431 Properties: 29432 29433 - Config: tmp_upload_path 29434 - Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH 29435 - Type: string 29436 - Required: false 29437 29438 #### --cache-tmp-wait-time 29439 29440 How long should files be stored in local cache before being uploaded. 29441 29442 This is the duration that a file must wait in the temporary location 29443 _cache-tmp-upload-path_ before it is selected for upload. 29444 29445 Note that only one file is uploaded at a time and it can take longer 29446 to start the upload if a queue formed for this purpose. 29447 29448 Properties: 29449 29450 - Config: tmp_wait_time 29451 - Env Var: RCLONE_CACHE_TMP_WAIT_TIME 29452 - Type: Duration 29453 - Default: 15s 29454 29455 #### --cache-db-wait-time 29456 29457 How long to wait for the DB to be available - 0 is unlimited. 29458 29459 Only one process can have the DB open at any one time, so rclone waits 29460 for this duration for the DB to become available before it gives an 29461 error. 29462 29463 If you set it to 0 then it will wait forever. 29464 29465 Properties: 29466 29467 - Config: db_wait_time 29468 - Env Var: RCLONE_CACHE_DB_WAIT_TIME 29469 - Type: Duration 29470 - Default: 1s 29471 29472 #### --cache-description 29473 29474 Description of the remote 29475 29476 Properties: 29477 29478 - Config: description 29479 - Env Var: RCLONE_CACHE_DESCRIPTION 29480 - Type: string 29481 - Required: false 29482 29483 ## Backend commands 29484 29485 Here are the commands specific to the cache backend. 29486 29487 Run them with 29488 29489 rclone backend COMMAND remote: 29490 29491 The help below will explain what arguments each command takes. 29492 29493 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 29494 info on how to pass options and arguments. 29495 29496 These can be run on a running backend using the rc command 29497 [backend/command](https://rclone.org/rc/#backend-command). 29498 29499 ### stats 29500 29501 Print stats on the cache backend in JSON format. 29502 29503 rclone backend stats remote: [options] [<arguments>+] 29504 29505 29506 29507 # Chunker 29508 29509 The `chunker` overlay transparently splits large files into smaller chunks 29510 during upload to wrapped remote and transparently assembles them back 29511 when the file is downloaded. This allows to effectively overcome size limits 29512 imposed by storage providers. 29513 29514 ## Configuration 29515 29516 To use it, first set up the underlying remote following the configuration 29517 instructions for that remote. You can also use a local pathname instead of 29518 a remote. 29519 29520 First check your chosen remote is working - we'll call it `remote:path` here. 29521 Note that anything inside `remote:path` will be chunked and anything outside 29522 won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift) 29523 then you should probably put the bucket in the remote `s3:bucket`. 29524 29525 Now configure `chunker` using `rclone config`. We will call this one `overlay` 29526 to separate it from the `remote` itself. 29527 29528 No remotes found, make a new one? n) New remote s) Set configuration 29529 password q) Quit config n/s/q> n name> overlay Type of storage to 29530 configure. Choose a number from below, or type in your own value [snip] 29531 XX / Transparently chunk/split large files "chunker" [snip] Storage> 29532 chunker Remote to chunk/unchunk. Normally should contain a ':' and a 29533 path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe 29534 "myremote:" (not recommended). Enter a string value. Press Enter for the 29535 default (""). remote> remote:path Files larger than chunk size will be 29536 split in chunks. Enter a size with suffix K,M,G,T. Press Enter for the 29537 default ("2G"). chunk_size> 100M Choose how chunker handles hash sums. 29538 All modes but "none" require metadata. Enter a string value. Press Enter 29539 for the default ("md5"). Choose a number from below, or type in your own 29540 value 1 / Pass any hash supported by wrapped remote for non-chunked 29541 files, return nothing otherwise "none" 2 / MD5 for composite files 29542 "md5" 3 / SHA1 for composite files "sha1" 4 / MD5 for all files 29543 "md5all" 5 / SHA1 for all files "sha1all" 6 / Copying a file to 29544 chunker will request MD5 from the source falling back to SHA1 if 29545 unsupported "md5quick" 7 / Similar to "md5quick" but prefers SHA1 over 29546 MD5 "sha1quick" hash_type> md5 Edit advanced config? (y/n) y) Yes n) No 29547 y/n> n Remote config -------------------- [overlay] type = chunker 29548 remote = remote:bucket chunk_size = 100M hash_type = md5 29549 -------------------- y) Yes this is OK e) Edit this remote d) Delete 29550 this remote y/e/d> y 29551 29552 29553 ### Specifying the remote 29554 29555 In normal use, make sure the remote has a `:` in. If you specify the remote 29556 without a `:` then rclone will use a local directory of that name. 29557 So if you use a remote of `/path/to/secret/files` then rclone will 29558 chunk stuff in that directory. If you use a remote of `name` then rclone 29559 will put files in a directory called `name` in the current directory. 29560 29561 29562 ### Chunking 29563 29564 When rclone starts a file upload, chunker checks the file size. If it 29565 doesn't exceed the configured chunk size, chunker will just pass the file 29566 to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut 29567 data in pieces with temporary names and stream them one by one, on the fly. 29568 Each data chunk will contain the specified number of bytes, except for the 29569 last one which may have less data. If file size is unknown in advance 29570 (this is called a streaming upload), chunker will internally create 29571 a temporary copy, record its size and repeat the above process. 29572 29573 When upload completes, temporary chunk files are finally renamed. 29574 This scheme guarantees that operations can be run in parallel and look 29575 from outside as atomic. 29576 A similar method with hidden temporary chunks is used for other operations 29577 (copy/move/rename, etc.). If an operation fails, hidden chunks are normally 29578 destroyed, and the target composite file stays intact. 29579 29580 When a composite file download is requested, chunker transparently 29581 assembles it by concatenating data chunks in order. As the split is trivial 29582 one could even manually concatenate data chunks together to obtain the 29583 original content. 29584 29585 When the `list` rclone command scans a directory on wrapped remote, 29586 the potential chunk files are accounted for, grouped and assembled into 29587 composite directory entries. Any temporary chunks are hidden. 29588 29589 List and other commands can sometimes come across composite files with 29590 missing or invalid chunks, e.g. shadowed by like-named directory or 29591 another file. This usually means that wrapped file system has been directly 29592 tampered with or damaged. If chunker detects a missing chunk it will 29593 by default print warning, skip the whole incomplete group of chunks but 29594 proceed with current command. 29595 You can set the `--chunker-fail-hard` flag to have commands abort with 29596 error message in such cases. 29597 29598 **Caveat**: As it is now, chunker will always create a temporary file in the 29599 backend and then rename it, even if the file is below the chunk threshold. 29600 This will result in unnecessary API calls and can severely restrict throughput 29601 when handling transfers primarily composed of small files on some backends (e.g. Box). 29602 A workaround to this issue is to use chunker only for files above the chunk threshold 29603 via `--min-size` and then perform a separate call without chunker on the remaining 29604 files. 29605 29606 29607 #### Chunk names 29608 29609 The default chunk name format is `*.rclone_chunk.###`, hence by default 29610 chunk names are `BIG_FILE_NAME.rclone_chunk.001`, 29611 `BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format 29612 using the `name_format` configuration file option. The format uses asterisk 29613 `*` as a placeholder for the base file name and one or more consecutive 29614 hash characters `#` as a placeholder for sequential chunk number. 29615 There must be one and only one asterisk. The number of consecutive hash 29616 characters defines the minimum length of a string representing a chunk number. 29617 If decimal chunk number has less digits than the number of hashes, it is 29618 left-padded by zeros. If the decimal string is longer, it is left intact. 29619 By default numbering starts from 1 but there is another option that allows 29620 user to start from 0, e.g. for compatibility with legacy software. 29621 29622 For example, if name format is `big_*-##.part` and original file name is 29623 `data.txt` and numbering starts from 0, then the first chunk will be named 29624 `big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part` 29625 and the 302nd chunk will become `big_data.txt-301.part`. 29626 29627 Note that `list` assembles composite directory entries only when chunk names 29628 match the configured format and treats non-conforming file names as normal 29629 non-chunked files. 29630 29631 When using `norename` transactions, chunk names will additionally have a unique 29632 file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`. 29633 29634 29635 ### Metadata 29636 29637 Besides data chunks chunker will by default create metadata object for 29638 a composite file. The object is named after the original file. 29639 Chunker allows user to disable metadata completely (the `none` format). 29640 Note that metadata is normally not created for files smaller than the 29641 configured chunk size. This may change in future rclone releases. 29642 29643 #### Simple JSON metadata format 29644 29645 This is the default format. It supports hash sums and chunk validation 29646 for composite files. Meta objects carry the following fields: 29647 29648 - `ver` - version of format, currently `1` 29649 - `size` - total size of composite file 29650 - `nchunks` - number of data chunks in file 29651 - `md5` - MD5 hashsum of composite file (if present) 29652 - `sha1` - SHA1 hashsum (if present) 29653 - `txn` - identifies current version of the file 29654 29655 There is no field for composite file name as it's simply equal to the name 29656 of meta object on the wrapped remote. Please refer to respective sections 29657 for details on hashsums and modified time handling. 29658 29659 #### No metadata 29660 29661 You can disable meta objects by setting the meta format option to `none`. 29662 In this mode chunker will scan directory for all files that follow 29663 configured chunk name format, group them by detecting chunks with the same 29664 base name and show group names as virtual composite files. 29665 This method is more prone to missing chunk errors (especially missing 29666 last chunk) than format with metadata enabled. 29667 29668 29669 ### Hashsums 29670 29671 Chunker supports hashsums only when a compatible metadata is present. 29672 Hence, if you choose metadata format of `none`, chunker will report hashsum 29673 as `UNSUPPORTED`. 29674 29675 Please note that by default metadata is stored only for composite files. 29676 If a file is smaller than configured chunk size, chunker will transparently 29677 redirect hash requests to wrapped remote, so support depends on that. 29678 You will see the empty string as a hashsum of requested type for small 29679 files if the wrapped remote doesn't support it. 29680 29681 Many storage backends support MD5 and SHA1 hash types, so does chunker. 29682 With chunker you can choose one or another but not both. 29683 MD5 is set by default as the most supported type. 29684 Since chunker keeps hashes for composite files and falls back to the 29685 wrapped remote hash for non-chunked ones, we advise you to choose the same 29686 hash type as supported by wrapped remote so that your file listings 29687 look coherent. 29688 29689 If your storage backend does not support MD5 or SHA1 but you need consistent 29690 file hashing, configure chunker with `md5all` or `sha1all`. These two modes 29691 guarantee given hash for all files. If wrapped remote doesn't support it, 29692 chunker will then add metadata to all files, even small. However, this can 29693 double the amount of small files in storage and incur additional service charges. 29694 You can even use chunker to force md5/sha1 support in any other remote 29695 at expense of sidecar meta objects by setting e.g. `hash_type=sha1all` 29696 to force hashsums and `chunk_size=1P` to effectively disable chunking. 29697 29698 Normally, when a file is copied to chunker controlled remote, chunker 29699 will ask the file source for compatible file hash and revert to on-the-fly 29700 calculation if none is found. This involves some CPU overhead but provides 29701 a guarantee that given hashsum is available. Also, chunker will reject 29702 a server-side copy or move operation if source and destination hashsum 29703 types are different resulting in the extra network bandwidth, too. 29704 In some rare cases this may be undesired, so chunker provides two optional 29705 choices: `sha1quick` and `md5quick`. If the source does not support primary 29706 hash type and the quick mode is enabled, chunker will try to fall back to 29707 the secondary type. This will save CPU and bandwidth but can result in empty 29708 hashsums at destination. Beware of consequences: the `sync` command will 29709 revert (sometimes silently) to time/size comparison if compatible hashsums 29710 between source and target are not found. 29711 29712 29713 ### Modification times 29714 29715 Chunker stores modification times using the wrapped remote so support 29716 depends on that. For a small non-chunked file the chunker overlay simply 29717 manipulates modification time of the wrapped remote file. 29718 For a composite file with metadata chunker will get and set 29719 modification time of the metadata object on the wrapped remote. 29720 If file is chunked but metadata format is `none` then chunker will 29721 use modification time of the first data chunk. 29722 29723 29724 ### Migrations 29725 29726 The idiomatic way to migrate to a different chunk size, hash type, transaction 29727 style or chunk naming scheme is to: 29728 29729 - Collect all your chunked files under a directory and have your 29730 chunker remote point to it. 29731 - Create another directory (most probably on the same cloud storage) 29732 and configure a new remote with desired metadata format, 29733 hash type, chunk naming etc. 29734 - Now run `rclone sync --interactive oldchunks: newchunks:` and all your data 29735 will be transparently converted in transfer. 29736 This may take some time, yet chunker will try server-side 29737 copy if possible. 29738 - After checking data integrity you may remove configuration section 29739 of the old remote. 29740 29741 If rclone gets killed during a long operation on a big composite file, 29742 hidden temporary chunks may stay in the directory. They will not be 29743 shown by the `list` command but will eat up your account quota. 29744 Please note that the `deletefile` command deletes only active 29745 chunks of a file. As a workaround, you can use remote of the wrapped 29746 file system to see them. 29747 An easy way to get rid of hidden garbage is to copy littered directory 29748 somewhere using the chunker remote and purge the original directory. 29749 The `copy` command will copy only active chunks while the `purge` will 29750 remove everything including garbage. 29751 29752 29753 ### Caveats and Limitations 29754 29755 Chunker requires wrapped remote to support server-side `move` (or `copy` + 29756 `delete`) operations, otherwise it will explicitly refuse to start. 29757 This is because it internally renames temporary chunk files to their final 29758 names when an operation completes successfully. 29759 29760 Chunker encodes chunk number in file name, so with default `name_format` 29761 setting it adds 17 characters. Also chunker adds 7 characters of temporary 29762 suffix during operations. Many file systems limit base file name without path 29763 by 255 characters. Using rclone's crypt remote as a base file system limits 29764 file name by 143 characters. Thus, maximum name length is 231 for most files 29765 and 119 for chunker-over-crypt. A user in need can change name format to 29766 e.g. `*.rcc##` and save 10 characters (provided at most 99 chunks per file). 29767 29768 Note that a move implemented using the copy-and-delete method may incur 29769 double charging with some cloud storage providers. 29770 29771 Chunker will not automatically rename existing chunks when you run 29772 `rclone config` on a live remote and change the chunk name format. 29773 Beware that in result of this some files which have been treated as chunks 29774 before the change can pop up in directory listings as normal files 29775 and vice versa. The same warning holds for the chunk size. 29776 If you desperately need to change critical chunking settings, you should 29777 run data migration as described above. 29778 29779 If wrapped remote is case insensitive, the chunker overlay will inherit 29780 that property (so you can't have a file called "Hello.doc" and "hello.doc" 29781 in the same directory). 29782 29783 Chunker included in rclone releases up to `v1.54` can sometimes fail to 29784 detect metadata produced by recent versions of rclone. We recommend users 29785 to keep rclone up-to-date to avoid data corruption. 29786 29787 Changing `transactions` is dangerous and requires explicit migration. 29788 29789 29790 ### Standard options 29791 29792 Here are the Standard options specific to chunker (Transparently chunk/split large files). 29793 29794 #### --chunker-remote 29795 29796 Remote to chunk/unchunk. 29797 29798 Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", 29799 "myremote:bucket" or maybe "myremote:" (not recommended). 29800 29801 Properties: 29802 29803 - Config: remote 29804 - Env Var: RCLONE_CHUNKER_REMOTE 29805 - Type: string 29806 - Required: true 29807 29808 #### --chunker-chunk-size 29809 29810 Files larger than chunk size will be split in chunks. 29811 29812 Properties: 29813 29814 - Config: chunk_size 29815 - Env Var: RCLONE_CHUNKER_CHUNK_SIZE 29816 - Type: SizeSuffix 29817 - Default: 2Gi 29818 29819 #### --chunker-hash-type 29820 29821 Choose how chunker handles hash sums. 29822 29823 All modes but "none" require metadata. 29824 29825 Properties: 29826 29827 - Config: hash_type 29828 - Env Var: RCLONE_CHUNKER_HASH_TYPE 29829 - Type: string 29830 - Default: "md5" 29831 - Examples: 29832 - "none" 29833 - Pass any hash supported by wrapped remote for non-chunked files. 29834 - Return nothing otherwise. 29835 - "md5" 29836 - MD5 for composite files. 29837 - "sha1" 29838 - SHA1 for composite files. 29839 - "md5all" 29840 - MD5 for all files. 29841 - "sha1all" 29842 - SHA1 for all files. 29843 - "md5quick" 29844 - Copying a file to chunker will request MD5 from the source. 29845 - Falling back to SHA1 if unsupported. 29846 - "sha1quick" 29847 - Similar to "md5quick" but prefers SHA1 over MD5. 29848 29849 ### Advanced options 29850 29851 Here are the Advanced options specific to chunker (Transparently chunk/split large files). 29852 29853 #### --chunker-name-format 29854 29855 String format of chunk file names. 29856 29857 The two placeholders are: base file name (*) and chunk number (#...). 29858 There must be one and only one asterisk and one or more consecutive hash characters. 29859 If chunk number has less digits than the number of hashes, it is left-padded by zeros. 29860 If there are more digits in the number, they are left as is. 29861 Possible chunk files are ignored if their name does not match given format. 29862 29863 Properties: 29864 29865 - Config: name_format 29866 - Env Var: RCLONE_CHUNKER_NAME_FORMAT 29867 - Type: string 29868 - Default: "*.rclone_chunk.###" 29869 29870 #### --chunker-start-from 29871 29872 Minimum valid chunk number. Usually 0 or 1. 29873 29874 By default chunk numbers start from 1. 29875 29876 Properties: 29877 29878 - Config: start_from 29879 - Env Var: RCLONE_CHUNKER_START_FROM 29880 - Type: int 29881 - Default: 1 29882 29883 #### --chunker-meta-format 29884 29885 Format of the metadata object or "none". 29886 29887 By default "simplejson". 29888 Metadata is a small JSON file named after the composite file. 29889 29890 Properties: 29891 29892 - Config: meta_format 29893 - Env Var: RCLONE_CHUNKER_META_FORMAT 29894 - Type: string 29895 - Default: "simplejson" 29896 - Examples: 29897 - "none" 29898 - Do not use metadata files at all. 29899 - Requires hash type "none". 29900 - "simplejson" 29901 - Simple JSON supports hash sums and chunk validation. 29902 - 29903 - It has the following fields: ver, size, nchunks, md5, sha1. 29904 29905 #### --chunker-fail-hard 29906 29907 Choose how chunker should handle files with missing or invalid chunks. 29908 29909 Properties: 29910 29911 - Config: fail_hard 29912 - Env Var: RCLONE_CHUNKER_FAIL_HARD 29913 - Type: bool 29914 - Default: false 29915 - Examples: 29916 - "true" 29917 - Report errors and abort current command. 29918 - "false" 29919 - Warn user, skip incomplete file and proceed. 29920 29921 #### --chunker-transactions 29922 29923 Choose how chunker should handle temporary files during transactions. 29924 29925 Properties: 29926 29927 - Config: transactions 29928 - Env Var: RCLONE_CHUNKER_TRANSACTIONS 29929 - Type: string 29930 - Default: "rename" 29931 - Examples: 29932 - "rename" 29933 - Rename temporary files after a successful transaction. 29934 - "norename" 29935 - Leave temporary file names and write transaction ID to metadata file. 29936 - Metadata is required for no rename transactions (meta format cannot be "none"). 29937 - If you are using norename transactions you should be careful not to downgrade Rclone 29938 - as older versions of Rclone don't support this transaction style and will misinterpret 29939 - files manipulated by norename transactions. 29940 - This method is EXPERIMENTAL, don't use on production systems. 29941 - "auto" 29942 - Rename or norename will be used depending on capabilities of the backend. 29943 - If meta format is set to "none", rename transactions will always be used. 29944 - This method is EXPERIMENTAL, don't use on production systems. 29945 29946 #### --chunker-description 29947 29948 Description of the remote 29949 29950 Properties: 29951 29952 - Config: description 29953 - Env Var: RCLONE_CHUNKER_DESCRIPTION 29954 - Type: string 29955 - Required: false 29956 29957 29958 29959 # Citrix ShareFile 29960 29961 [Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business. 29962 29963 ## Configuration 29964 29965 The initial setup for Citrix ShareFile involves getting a token from 29966 Citrix ShareFile which you can in your browser. `rclone config` walks you 29967 through it. 29968 29969 Here is an example of how to make a remote called `remote`. First run: 29970 29971 rclone config 29972 29973 This will guide you through an interactive setup process: 29974 29975 No remotes found, make a new one? n) New remote s) Set configuration 29976 password q) Quit config n/s/q> n name> remote Type of storage to 29977 configure. Enter a string value. Press Enter for the default (""). 29978 Choose a number from below, or type in your own value XX / Citrix 29979 Sharefile "sharefile" Storage> sharefile ** See help for sharefile 29980 backend at: https://rclone.org/sharefile/ ** 29981 29982 ID of the root folder 29983 29984 Leave blank to access "Personal Folders". You can use one of the 29985 standard values here or any folder ID (long hex number ID). Enter a 29986 string value. Press Enter for the default (""). Choose a number from 29987 below, or type in your own value 1 / Access the Personal Folders. 29988 (Default) "" 2 / Access the Favorites folder. "favorites" 3 / Access 29989 all the shared folders. "allshared" 4 / Access all the individual 29990 connectors. "connectors" 5 / Access the home, favorites, and shared 29991 folders as well as the connectors. "top" root_folder_id> Edit advanced 29992 config? (y/n) y) Yes n) No y/n> n Remote config Use web browser to 29993 automatically authenticate rclone with remote? * Say Y if the machine 29994 running rclone has a web browser you can use * Say N if running rclone 29995 on a (remote) machine without web browser access If not sure try Y. If Y 29996 failed, try N. y) Yes n) No y/n> y If your browser doesn't open 29997 automatically go to the following link: 29998 http://127.0.0.1:53682/auth?state=XXX Log in and authorize rclone for 29999 access Waiting for code... Got code -------------------- [remote] type = 30000 sharefile endpoint = https://XXX.sharefile.com token = 30001 {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} 30002 -------------------- y) Yes this is OK e) Edit this remote d) Delete 30003 this remote y/e/d> y 30004 30005 30006 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 30007 machine with no Internet browser available. 30008 30009 Note that rclone runs a webserver on your local machine to collect the 30010 token as returned from Citrix ShareFile. This only runs from the moment it opens 30011 your browser to the moment you get back the verification code. This 30012 is on `http://127.0.0.1:53682/` and this it may require you to unblock 30013 it temporarily if you are running a host firewall. 30014 30015 Once configured you can then use `rclone` like this, 30016 30017 List directories in top level of your ShareFile 30018 30019 rclone lsd remote: 30020 30021 List all the files in your ShareFile 30022 30023 rclone ls remote: 30024 30025 To copy a local directory to an ShareFile directory called backup 30026 30027 rclone copy /home/source remote:backup 30028 30029 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 30030 30031 ### Modification times and hashes 30032 30033 ShareFile allows modification times to be set on objects accurate to 1 30034 second. These will be used to detect whether objects need syncing or 30035 not. 30036 30037 ShareFile supports MD5 type hashes, so you can use the `--checksum` 30038 flag. 30039 30040 ### Transfers 30041 30042 For files above 128 MiB rclone will use a chunked transfer. Rclone will 30043 upload up to `--transfers` chunks at the same time (shared among all 30044 the multipart uploads). Chunks are buffered in memory and are 30045 normally 64 MiB so increasing `--transfers` will increase memory use. 30046 30047 ### Restricted filename characters 30048 30049 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 30050 the following characters are also replaced: 30051 30052 | Character | Value | Replacement | 30053 | --------- |:-----:|:-----------:| 30054 | \\ | 0x5C | \ | 30055 | * | 0x2A | * | 30056 | < | 0x3C | < | 30057 | > | 0x3E | > | 30058 | ? | 0x3F | ? | 30059 | : | 0x3A | : | 30060 | \| | 0x7C | | | 30061 | " | 0x22 | " | 30062 30063 File names can also not start or end with the following characters. 30064 These only get replaced if they are the first or last character in the 30065 name: 30066 30067 | Character | Value | Replacement | 30068 | --------- |:-----:|:-----------:| 30069 | SP | 0x20 | ␠ | 30070 | . | 0x2E | . | 30071 30072 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 30073 as they can't be used in JSON strings. 30074 30075 30076 ### Standard options 30077 30078 Here are the Standard options specific to sharefile (Citrix Sharefile). 30079 30080 #### --sharefile-client-id 30081 30082 OAuth Client Id. 30083 30084 Leave blank normally. 30085 30086 Properties: 30087 30088 - Config: client_id 30089 - Env Var: RCLONE_SHAREFILE_CLIENT_ID 30090 - Type: string 30091 - Required: false 30092 30093 #### --sharefile-client-secret 30094 30095 OAuth Client Secret. 30096 30097 Leave blank normally. 30098 30099 Properties: 30100 30101 - Config: client_secret 30102 - Env Var: RCLONE_SHAREFILE_CLIENT_SECRET 30103 - Type: string 30104 - Required: false 30105 30106 #### --sharefile-root-folder-id 30107 30108 ID of the root folder. 30109 30110 Leave blank to access "Personal Folders". You can use one of the 30111 standard values here or any folder ID (long hex number ID). 30112 30113 Properties: 30114 30115 - Config: root_folder_id 30116 - Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID 30117 - Type: string 30118 - Required: false 30119 - Examples: 30120 - "" 30121 - Access the Personal Folders (default). 30122 - "favorites" 30123 - Access the Favorites folder. 30124 - "allshared" 30125 - Access all the shared folders. 30126 - "connectors" 30127 - Access all the individual connectors. 30128 - "top" 30129 - Access the home, favorites, and shared folders as well as the connectors. 30130 30131 ### Advanced options 30132 30133 Here are the Advanced options specific to sharefile (Citrix Sharefile). 30134 30135 #### --sharefile-token 30136 30137 OAuth Access Token as a JSON blob. 30138 30139 Properties: 30140 30141 - Config: token 30142 - Env Var: RCLONE_SHAREFILE_TOKEN 30143 - Type: string 30144 - Required: false 30145 30146 #### --sharefile-auth-url 30147 30148 Auth server URL. 30149 30150 Leave blank to use the provider defaults. 30151 30152 Properties: 30153 30154 - Config: auth_url 30155 - Env Var: RCLONE_SHAREFILE_AUTH_URL 30156 - Type: string 30157 - Required: false 30158 30159 #### --sharefile-token-url 30160 30161 Token server url. 30162 30163 Leave blank to use the provider defaults. 30164 30165 Properties: 30166 30167 - Config: token_url 30168 - Env Var: RCLONE_SHAREFILE_TOKEN_URL 30169 - Type: string 30170 - Required: false 30171 30172 #### --sharefile-upload-cutoff 30173 30174 Cutoff for switching to multipart upload. 30175 30176 Properties: 30177 30178 - Config: upload_cutoff 30179 - Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF 30180 - Type: SizeSuffix 30181 - Default: 128Mi 30182 30183 #### --sharefile-chunk-size 30184 30185 Upload chunk size. 30186 30187 Must a power of 2 >= 256k. 30188 30189 Making this larger will improve performance, but note that each chunk 30190 is buffered in memory one per transfer. 30191 30192 Reducing this will reduce memory usage but decrease performance. 30193 30194 Properties: 30195 30196 - Config: chunk_size 30197 - Env Var: RCLONE_SHAREFILE_CHUNK_SIZE 30198 - Type: SizeSuffix 30199 - Default: 64Mi 30200 30201 #### --sharefile-endpoint 30202 30203 Endpoint for API calls. 30204 30205 This is usually auto discovered as part of the oauth process, but can 30206 be set manually to something like: https://XXX.sharefile.com 30207 30208 30209 Properties: 30210 30211 - Config: endpoint 30212 - Env Var: RCLONE_SHAREFILE_ENDPOINT 30213 - Type: string 30214 - Required: false 30215 30216 #### --sharefile-encoding 30217 30218 The encoding for the backend. 30219 30220 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 30221 30222 Properties: 30223 30224 - Config: encoding 30225 - Env Var: RCLONE_SHAREFILE_ENCODING 30226 - Type: Encoding 30227 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot 30228 30229 #### --sharefile-description 30230 30231 Description of the remote 30232 30233 Properties: 30234 30235 - Config: description 30236 - Env Var: RCLONE_SHAREFILE_DESCRIPTION 30237 - Type: string 30238 - Required: false 30239 30240 30241 ## Limitations 30242 30243 Note that ShareFile is case insensitive so you can't have a file called 30244 "Hello.doc" and one called "hello.doc". 30245 30246 ShareFile only supports filenames up to 256 characters in length. 30247 30248 `rclone about` is not supported by the Citrix ShareFile backend. Backends without 30249 this capability cannot determine free space for an rclone mount or 30250 use policy `mfs` (most free space) as a member of an rclone union 30251 remote. 30252 30253 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 30254 30255 # Crypt 30256 30257 Rclone `crypt` remotes encrypt and decrypt other remotes. 30258 30259 A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/) 30260 directly, but instead wraps another remote, which in turn accesses 30261 the storage system. This is similar to how [alias](https://rclone.org/alias/), 30262 [union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/) 30263 and a few others work. It makes the usage very flexible, as you can 30264 add a layer, in this case an encryption layer, on top of any other 30265 backend, even in multiple layers. Rclone's functionality 30266 can be used as with any other remote, for example you can 30267 [mount](https://rclone.org/commands/rclone_mount/) a crypt remote. 30268 30269 Accessing a storage system through a crypt remote realizes client-side 30270 encryption, which makes it safe to keep your data in a location you do 30271 not trust will not get compromised. 30272 When working against the `crypt` remote, rclone will automatically 30273 encrypt (before uploading) and decrypt (after downloading) on your local 30274 system as needed on the fly, leaving the data encrypted at rest in the 30275 wrapped remote. If you access the storage system using an application 30276 other than rclone, or access the wrapped remote directly using rclone, 30277 there will not be any encryption/decryption: Downloading existing content 30278 will just give you the encrypted (scrambled) format, and anything you 30279 upload will *not* become encrypted. 30280 30281 The encryption is a secret-key encryption (also called symmetric key encryption) 30282 algorithm, where a password (or pass phrase) is used to generate real encryption key. 30283 The password can be supplied by user, or you may chose to let rclone 30284 generate one. It will be stored in the configuration file, in a lightly obscured form. 30285 If you are in an environment where you are not able to keep your configuration 30286 secured, you should add 30287 [configuration encryption](https://rclone.org/docs/#configuration-encryption) 30288 as protection. As long as you have this configuration file, you will be able to 30289 decrypt your data. Without the configuration file, as long as you remember 30290 the password (or keep it in a safe place), you can re-create the configuration 30291 and gain access to the existing data. You may also configure a corresponding 30292 remote in a different installation to access the same data. 30293 See below for guidance to [changing password](#changing-password). 30294 30295 Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)), 30296 to permute the encryption key so that the same string may be encrypted in 30297 different ways. When configuring the crypt remote it is optional to enter a salt, 30298 or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string. 30299 Normally in cryptography, the salt is stored together with the encrypted content, 30300 and do not have to be memorized by the user. This is not the case in rclone, 30301 because rclone does not store any additional information on the remotes. Use of 30302 custom salt is effectively a second password that must be memorized. 30303 30304 [File content](#file-encryption) encryption is performed using 30305 [NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox), 30306 based on XSalsa20 cipher and Poly1305 for integrity. 30307 [Names](#name-encryption) (file- and directory names) are also encrypted 30308 by default, but this has some implications and is therefore 30309 possible to be turned off. 30310 30311 ## Configuration 30312 30313 Here is an example of how to make a remote called `secret`. 30314 30315 To use `crypt`, first set up the underlying remote. Follow the 30316 `rclone config` instructions for the specific backend. 30317 30318 Before configuring the crypt remote, check the underlying remote is 30319 working. In this example the underlying remote is called `remote`. 30320 We will configure a path `path` within this remote to contain the 30321 encrypted content. Anything inside `remote:path` will be encrypted 30322 and anything outside will not. 30323 30324 Configure `crypt` using `rclone config`. In this example the `crypt` 30325 remote is called `secret`, to differentiate it from the underlying 30326 `remote`. 30327 30328 When you are done you can use the crypt remote named `secret` just 30329 as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`, 30330 and rclone will encrypt and decrypt as needed on the fly. 30331 If you access the wrapped remote `remote:path` directly you will bypass 30332 the encryption, and anything you read will be in encrypted form, and 30333 anything you write will be unencrypted. To avoid issues it is best to 30334 configure a dedicated path for encrypted content, and access it 30335 exclusively through a crypt remote. 30336 30337 No remotes found, make a new one? n) New remote s) Set configuration 30338 password q) Quit config n/s/q> n name> secret Type of storage to 30339 configure. Enter a string value. Press Enter for the default (""). 30340 Choose a number from below, or type in your own value [snip] XX / 30341 Encrypt/Decrypt a remote "crypt" [snip] Storage> crypt ** See help for 30342 crypt backend at: https://rclone.org/crypt/ ** 30343 30344 Remote to encrypt/decrypt. Normally should contain a ':' and a path, eg 30345 "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not 30346 recommended). Enter a string value. Press Enter for the default (""). 30347 remote> remote:path How to encrypt the filenames. Enter a string value. 30348 Press Enter for the default ("standard"). Choose a number from below, or 30349 type in your own value. / Encrypt the filenames. 1 | See the docs for 30350 the details. "standard" 2 / Very simple filename obfuscation. 30351 "obfuscate" / Don't encrypt the file names. 3 | Adds a ".bin" extension 30352 only. "off" filename_encryption> Option to either encrypt directory 30353 names or leave them intact. 30354 30355 NB If filename_encryption is "off" then this option will do nothing. 30356 Enter a boolean value (true or false). Press Enter for the default 30357 ("true"). Choose a number from below, or type in your own value 1 / 30358 Encrypt directory names. "true" 2 / Don't encrypt directory names, 30359 leave them intact. "false" directory_name_encryption> Password or pass 30360 phrase for encryption. y) Yes type in my own password g) Generate random 30361 password y/g> y Enter the password: password: Confirm the password: 30362 password: Password or pass phrase for salt. Optional but recommended. 30363 Should be different to the previous password. y) Yes type in my own 30364 password g) Generate random password n) No leave this optional password 30365 blank (default) y/g/n> g Password strength in bits. 64 is just about 30366 memorable 128 is secure 1024 is the maximum Bits> 128 Your password is: 30367 JAsJvRcgR-_veXNfy_sGmQ Use this password? Please note that an obscured 30368 version of this password (and not the password itself) will be stored 30369 under your configuration file, so keep this generated password in a safe 30370 place. y) Yes (default) n) No y/n> Edit advanced config? (y/n) y) Yes n) 30371 No (default) y/n> Remote config -------------------- [secret] type = 30372 crypt remote = remote:path password = *** ENCRYPTED password2 = 30373 ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit 30374 this remote d) Delete this remote y/e/d> 30375 30376 30377 **Important** The crypt password stored in `rclone.conf` is lightly 30378 obscured. That only protects it from cursory inspection. It is not 30379 secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified. 30380 30381 A long passphrase is recommended, or `rclone config` can generate a 30382 random one. 30383 30384 The obscured password is created using AES-CTR with a static key. The 30385 salt is stored verbatim at the beginning of the obscured password. This 30386 static key is shared between all versions of rclone. 30387 30388 If you reconfigure rclone with the same passwords/passphrases 30389 elsewhere it will be compatible, but the obscured version will be different 30390 due to the different salt. 30391 30392 Rclone does not encrypt 30393 30394 * file length - this can be calculated within 16 bytes 30395 * modification time - used for syncing 30396 30397 ### Specifying the remote 30398 30399 When configuring the remote to encrypt/decrypt, you may specify any 30400 string that rclone accepts as a source/destination of other commands. 30401 30402 The primary use case is to specify the path into an already configured 30403 remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that 30404 data in a remote untrusted location can be stored encrypted. 30405 30406 You may also specify a local filesystem path, such as 30407 `/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating 30408 a crypt remote pointing to such a local filesystem path, you can 30409 use rclone as a utility for pure local file encryption, for example 30410 to keep encrypted files on a removable USB drive. 30411 30412 **Note**: A string which do not contain a `:` will by rclone be treated 30413 as a relative path in the local filesystem. For example, if you enter 30414 the name `remote` without the trailing `:`, it will be treated as 30415 a subdirectory of the current directory with name "remote". 30416 30417 If a path `remote:path/to/dir` is specified, rclone stores encrypted 30418 files in `path/to/dir` on the remote. With file name encryption, files 30419 saved to `secret:subdir/subfile` are stored in the unencrypted path 30420 `path/to/dir` but the `subdir/subpath` element is encrypted. 30421 30422 The path you specify does not have to exist, rclone will create 30423 it when needed. 30424 30425 If you intend to use the wrapped remote both directly for keeping 30426 unencrypted content, as well as through a crypt remote for encrypted 30427 content, it is recommended to point the crypt remote to a separate 30428 directory within the wrapped remote. If you use a bucket-based storage 30429 system (e.g. Swift, S3, Google Compute Storage, B2) it is generally 30430 advisable to wrap the crypt remote around a specific bucket (`s3:bucket`). 30431 If wrapping around the entire root of the storage (`s3:`), and use the 30432 optional file name encryption, rclone will encrypt the bucket name. 30433 30434 ### Changing password 30435 30436 Should the password, or the configuration file containing a lightly obscured 30437 form of the password, be compromised, you need to re-encrypt your data with 30438 a new password. Since rclone uses secret-key encryption, where the encryption 30439 key is generated directly from the password kept on the client, it is not 30440 possible to change the password/key of already encrypted content. Just changing 30441 the password configured for an existing crypt remote means you will no longer 30442 able to decrypt any of the previously encrypted content. The only possibility 30443 is to re-upload everything via a crypt remote configured with your new password. 30444 30445 Depending on the size of your data, your bandwidth, storage quota etc, there are 30446 different approaches you can take: 30447 - If you have everything in a different location, for example on your local system, 30448 you could remove all of the prior encrypted files, change the password for your 30449 configured crypt remote (or delete and re-create the crypt configuration), 30450 and then re-upload everything from the alternative location. 30451 - If you have enough space on the storage system you can create a new crypt 30452 remote pointing to a separate directory on the same backend, and then use 30453 rclone to copy everything from the original crypt remote to the new, 30454 effectively decrypting everything on the fly using the old password and 30455 re-encrypting using the new password. When done, delete the original crypt 30456 remote directory and finally the rclone crypt configuration with the old password. 30457 All data will be streamed from the storage system and back, so you will 30458 get half the bandwidth and be charged twice if you have upload and download quota 30459 on the storage system. 30460 30461 **Note**: A security problem related to the random password generator 30462 was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated 30463 by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2 30464 (released 2020-10-26) are not considered secure and should be changed. 30465 If you made up your own password, or used rclone version older than 1.49.0 or 30466 newer than 1.53.2 to generate it, you are *not* affected by this issue. 30467 See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more 30468 details, and a tool you can use to check if you are affected. 30469 30470 ### Example 30471 30472 Create the following file structure using "standard" file name 30473 encryption. 30474 30475 plaintext/ ├── file0.txt ├── file1.txt └── subdir ├── file2.txt ├── 30476 file3.txt └── subsubdir └── file4.txt 30477 30478 30479 Copy these to the remote, and list them 30480 30481 $ rclone -q copy plaintext secret: $ rclone -q ls secret: 7 file1.txt 6 30482 file0.txt 8 subdir/file2.txt 10 subdir/subsubdir/file4.txt 9 30483 subdir/file3.txt 30484 30485 30486 The crypt remote looks like 30487 30488 $ rclone -q ls remote:path 55 hagjclgavj2mbiqm6u6cnjjqcg 54 30489 v05749mltvv1tf4onltun46gls 57 30490 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 58 30491 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 30492 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 30493 30494 30495 The directory structure is preserved 30496 30497 $ rclone -q ls secret:subdir 8 file2.txt 9 file3.txt 10 30498 subsubdir/file4.txt 30499 30500 30501 Without file name encryption `.bin` extensions are added to underlying 30502 names. This prevents the cloud provider attempting to interpret file 30503 content. 30504 30505 $ rclone -q ls remote:path 54 file0.txt.bin 57 subdir/file3.txt.bin 56 30506 subdir/file2.txt.bin 58 subdir/subsubdir/file4.txt.bin 55 file1.txt.bin 30507 30508 30509 ### File name encryption modes 30510 30511 Off 30512 30513 * doesn't hide file names or directory structure 30514 * allows for longer file names (~246 characters) 30515 * can use sub paths and copy single files 30516 30517 Standard 30518 30519 * file names encrypted 30520 * file names can't be as long (~143 characters) 30521 * can use sub paths and copy single files 30522 * directory structure visible 30523 * identical files names will have identical uploaded names 30524 * can use shortcuts to shorten the directory recursion 30525 30526 Obfuscation 30527 30528 This is a simple "rotate" of the filename, with each file having a rot 30529 distance based on the filename. Rclone stores the distance at the 30530 beginning of the filename. A file called "hello" may become "53.jgnnq". 30531 30532 Obfuscation is not a strong encryption of filenames, but hinders 30533 automated scanning tools picking up on filename patterns. It is an 30534 intermediate between "off" and "standard" which allows for longer path 30535 segment names. 30536 30537 There is a possibility with some unicode based filenames that the 30538 obfuscation is weak and may map lower case characters to upper case 30539 equivalents. 30540 30541 Obfuscation cannot be relied upon for strong protection. 30542 30543 * file names very lightly obfuscated 30544 * file names can be longer than standard encryption 30545 * can use sub paths and copy single files 30546 * directory structure visible 30547 * identical files names will have identical uploaded names 30548 30549 Cloud storage systems have limits on file name length and 30550 total path length which rclone is more likely to breach using 30551 "Standard" file name encryption. Where file names are less than 156 30552 characters in length issues should not be encountered, irrespective of 30553 cloud storage provider. 30554 30555 An experimental advanced option `filename_encoding` is now provided to 30556 address this problem to a certain degree. 30557 For cloud storage systems with case sensitive file names (e.g. Google Drive), 30558 `base64` can be used to reduce file name length. 30559 For cloud storage systems using UTF-16 to store file names internally 30560 (e.g. OneDrive, Dropbox, Box), `base32768` can be used to drastically reduce 30561 file name length. 30562 30563 An alternative, future rclone file name encryption mode may tolerate 30564 backend provider path length limits. 30565 30566 ### Directory name encryption 30567 30568 Crypt offers the option of encrypting dir names or leaving them intact. 30569 There are two options: 30570 30571 True 30572 30573 Encrypts the whole file path including directory names 30574 Example: 30575 `1/12/123.txt` is encrypted to 30576 `p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0` 30577 30578 False 30579 30580 Only encrypts file names, skips directory names 30581 Example: 30582 `1/12/123.txt` is encrypted to 30583 `1/12/qgm4avr35m5loi1th53ato71v0` 30584 30585 30586 ### Modification times and hashes 30587 30588 Crypt stores modification times using the underlying remote so support 30589 depends on that. 30590 30591 Hashes are not stored for crypt. However the data integrity is 30592 protected by an extremely strong crypto authenticator. 30593 30594 Use the `rclone cryptcheck` command to check the 30595 integrity of an encrypted remote instead of `rclone check` which can't 30596 check the checksums properly. 30597 30598 30599 ### Standard options 30600 30601 Here are the Standard options specific to crypt (Encrypt/Decrypt a remote). 30602 30603 #### --crypt-remote 30604 30605 Remote to encrypt/decrypt. 30606 30607 Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", 30608 "myremote:bucket" or maybe "myremote:" (not recommended). 30609 30610 Properties: 30611 30612 - Config: remote 30613 - Env Var: RCLONE_CRYPT_REMOTE 30614 - Type: string 30615 - Required: true 30616 30617 #### --crypt-filename-encryption 30618 30619 How to encrypt the filenames. 30620 30621 Properties: 30622 30623 - Config: filename_encryption 30624 - Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION 30625 - Type: string 30626 - Default: "standard" 30627 - Examples: 30628 - "standard" 30629 - Encrypt the filenames. 30630 - See the docs for the details. 30631 - "obfuscate" 30632 - Very simple filename obfuscation. 30633 - "off" 30634 - Don't encrypt the file names. 30635 - Adds a ".bin", or "suffix" extension only. 30636 30637 #### --crypt-directory-name-encryption 30638 30639 Option to either encrypt directory names or leave them intact. 30640 30641 NB If filename_encryption is "off" then this option will do nothing. 30642 30643 Properties: 30644 30645 - Config: directory_name_encryption 30646 - Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION 30647 - Type: bool 30648 - Default: true 30649 - Examples: 30650 - "true" 30651 - Encrypt directory names. 30652 - "false" 30653 - Don't encrypt directory names, leave them intact. 30654 30655 #### --crypt-password 30656 30657 Password or pass phrase for encryption. 30658 30659 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 30660 30661 Properties: 30662 30663 - Config: password 30664 - Env Var: RCLONE_CRYPT_PASSWORD 30665 - Type: string 30666 - Required: true 30667 30668 #### --crypt-password2 30669 30670 Password or pass phrase for salt. 30671 30672 Optional but recommended. 30673 Should be different to the previous password. 30674 30675 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 30676 30677 Properties: 30678 30679 - Config: password2 30680 - Env Var: RCLONE_CRYPT_PASSWORD2 30681 - Type: string 30682 - Required: false 30683 30684 ### Advanced options 30685 30686 Here are the Advanced options specific to crypt (Encrypt/Decrypt a remote). 30687 30688 #### --crypt-server-side-across-configs 30689 30690 Deprecated: use --server-side-across-configs instead. 30691 30692 Allow server-side operations (e.g. copy) to work across different crypt configs. 30693 30694 Normally this option is not what you want, but if you have two crypts 30695 pointing to the same backend you can use it. 30696 30697 This can be used, for example, to change file name encryption type 30698 without re-uploading all the data. Just make two crypt backends 30699 pointing to two different directories with the single changed 30700 parameter and use rclone move to move the files between the crypt 30701 remotes. 30702 30703 Properties: 30704 30705 - Config: server_side_across_configs 30706 - Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS 30707 - Type: bool 30708 - Default: false 30709 30710 #### --crypt-show-mapping 30711 30712 For all files listed show how the names encrypt. 30713 30714 If this flag is set then for each file that the remote is asked to 30715 list, it will log (at level INFO) a line stating the decrypted file 30716 name and the encrypted file name. 30717 30718 This is so you can work out which encrypted names are which decrypted 30719 names just in case you need to do something with the encrypted file 30720 names, or for debugging purposes. 30721 30722 Properties: 30723 30724 - Config: show_mapping 30725 - Env Var: RCLONE_CRYPT_SHOW_MAPPING 30726 - Type: bool 30727 - Default: false 30728 30729 #### --crypt-no-data-encryption 30730 30731 Option to either encrypt file data or leave it unencrypted. 30732 30733 Properties: 30734 30735 - Config: no_data_encryption 30736 - Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION 30737 - Type: bool 30738 - Default: false 30739 - Examples: 30740 - "true" 30741 - Don't encrypt file data, leave it unencrypted. 30742 - "false" 30743 - Encrypt file data. 30744 30745 #### --crypt-pass-bad-blocks 30746 30747 If set this will pass bad blocks through as all 0. 30748 30749 This should not be set in normal operation, it should only be set if 30750 trying to recover an encrypted file with errors and it is desired to 30751 recover as much of the file as possible. 30752 30753 Properties: 30754 30755 - Config: pass_bad_blocks 30756 - Env Var: RCLONE_CRYPT_PASS_BAD_BLOCKS 30757 - Type: bool 30758 - Default: false 30759 30760 #### --crypt-strict-names 30761 30762 If set, this will raise an error when crypt comes across a filename that can't be decrypted. 30763 30764 (By default, rclone will just log a NOTICE and continue as normal.) 30765 This can happen if encrypted and unencrypted files are stored in the same 30766 directory (which is not recommended.) It may also indicate a more serious 30767 problem that should be investigated. 30768 30769 Properties: 30770 30771 - Config: strict_names 30772 - Env Var: RCLONE_CRYPT_STRICT_NAMES 30773 - Type: bool 30774 - Default: false 30775 30776 #### --crypt-filename-encoding 30777 30778 How to encode the encrypted filename to text string. 30779 30780 This option could help with shortening the encrypted filename. The 30781 suitable option would depend on the way your remote count the filename 30782 length and if it's case sensitive. 30783 30784 Properties: 30785 30786 - Config: filename_encoding 30787 - Env Var: RCLONE_CRYPT_FILENAME_ENCODING 30788 - Type: string 30789 - Default: "base32" 30790 - Examples: 30791 - "base32" 30792 - Encode using base32. Suitable for all remote. 30793 - "base64" 30794 - Encode using base64. Suitable for case sensitive remote. 30795 - "base32768" 30796 - Encode using base32768. Suitable if your remote counts UTF-16 or 30797 - Unicode codepoint instead of UTF-8 byte length. (Eg. Onedrive, Dropbox) 30798 30799 #### --crypt-suffix 30800 30801 If this is set it will override the default suffix of ".bin". 30802 30803 Setting suffix to "none" will result in an empty suffix. This may be useful 30804 when the path length is critical. 30805 30806 Properties: 30807 30808 - Config: suffix 30809 - Env Var: RCLONE_CRYPT_SUFFIX 30810 - Type: string 30811 - Default: ".bin" 30812 30813 #### --crypt-description 30814 30815 Description of the remote 30816 30817 Properties: 30818 30819 - Config: description 30820 - Env Var: RCLONE_CRYPT_DESCRIPTION 30821 - Type: string 30822 - Required: false 30823 30824 ### Metadata 30825 30826 Any metadata supported by the underlying remote is read and written. 30827 30828 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 30829 30830 ## Backend commands 30831 30832 Here are the commands specific to the crypt backend. 30833 30834 Run them with 30835 30836 rclone backend COMMAND remote: 30837 30838 The help below will explain what arguments each command takes. 30839 30840 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 30841 info on how to pass options and arguments. 30842 30843 These can be run on a running backend using the rc command 30844 [backend/command](https://rclone.org/rc/#backend-command). 30845 30846 ### encode 30847 30848 Encode the given filename(s) 30849 30850 rclone backend encode remote: [options] [<arguments>+] 30851 30852 This encodes the filenames given as arguments returning a list of 30853 strings of the encoded results. 30854 30855 Usage Example: 30856 30857 rclone backend encode crypt: file1 [file2...] 30858 rclone rc backend/command command=encode fs=crypt: file1 [file2...] 30859 30860 30861 ### decode 30862 30863 Decode the given filename(s) 30864 30865 rclone backend decode remote: [options] [<arguments>+] 30866 30867 This decodes the filenames given as arguments returning a list of 30868 strings of the decoded results. It will return an error if any of the 30869 inputs are invalid. 30870 30871 Usage Example: 30872 30873 rclone backend decode crypt: encryptedfile1 [encryptedfile2...] 30874 rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...] 30875 30876 30877 30878 30879 ## Backing up an encrypted remote 30880 30881 If you wish to backup an encrypted remote, it is recommended that you use 30882 `rclone sync` on the encrypted files, and make sure the passwords are 30883 the same in the new encrypted remote. 30884 30885 This will have the following advantages 30886 30887 * `rclone sync` will check the checksums while copying 30888 * you can use `rclone check` between the encrypted remotes 30889 * you don't decrypt and encrypt unnecessarily 30890 30891 For example, let's say you have your original remote at `remote:` with 30892 the encrypted version at `eremote:` with path `remote:crypt`. You 30893 would then set up the new remote `remote2:` and then the encrypted 30894 version `eremote2:` with path `remote2:crypt` using the same passwords 30895 as `eremote:`. 30896 30897 To sync the two remotes you would do 30898 30899 rclone sync --interactive remote:crypt remote2:crypt 30900 30901 And to check the integrity you would do 30902 30903 rclone check remote:crypt remote2:crypt 30904 30905 ## File formats 30906 30907 ### File encryption 30908 30909 Files are encrypted 1:1 source file to destination object. The file 30910 has a header and is divided into chunks. 30911 30912 #### Header 30913 30914 * 8 bytes magic string `RCLONE\x00\x00` 30915 * 24 bytes Nonce (IV) 30916 30917 The initial nonce is generated from the operating systems crypto 30918 strong random number generator. The nonce is incremented for each 30919 chunk read making sure each nonce is unique for each block written. 30920 The chance of a nonce being reused is minuscule. If you wrote an 30921 exabyte of data (10¹⁸ bytes) you would have a probability of 30922 approximately 2×10⁻³² of re-using a nonce. 30923 30924 #### Chunk 30925 30926 Each chunk will contain 64 KiB of data, except for the last one which 30927 may have less data. The data chunk is in standard NaCl SecretBox 30928 format. SecretBox uses XSalsa20 and Poly1305 to encrypt and 30929 authenticate messages. 30930 30931 Each chunk contains: 30932 30933 * 16 Bytes of Poly1305 authenticator 30934 * 1 - 65536 bytes XSalsa20 encrypted data 30935 30936 64k chunk size was chosen as the best performing chunk size (the 30937 authenticator takes too much time below this and the performance drops 30938 off due to cache effects above this). Note that these chunks are 30939 buffered in memory so they can't be too big. 30940 30941 This uses a 32 byte (256 bit key) key derived from the user password. 30942 30943 #### Examples 30944 30945 1 byte file will encrypt to 30946 30947 * 32 bytes header 30948 * 17 bytes data chunk 30949 30950 49 bytes total 30951 30952 1 MiB (1048576 bytes) file will encrypt to 30953 30954 * 32 bytes header 30955 * 16 chunks of 65568 bytes 30956 30957 1049120 bytes total (a 0.05% overhead). This is the overhead for big 30958 files. 30959 30960 ### Name encryption 30961 30962 File names are encrypted segment by segment - the path is broken up 30963 into `/` separated strings and these are encrypted individually. 30964 30965 File segments are padded using PKCS#7 to a multiple of 16 bytes 30966 before encryption. 30967 30968 They are then encrypted with EME using AES with 256 bit key. EME 30969 (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003 30970 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway. 30971 30972 This makes for deterministic encryption which is what we want - the 30973 same filename must encrypt to the same thing otherwise we can't find 30974 it on the cloud storage system. 30975 30976 This means that 30977 30978 * filenames with the same name will encrypt the same 30979 * filenames which start the same won't have a common prefix 30980 30981 This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of 30982 which are derived from the user password. 30983 30984 After encryption they are written out using a modified version of 30985 standard `base32` encoding as described in RFC4648. The standard 30986 encoding is modified in two ways: 30987 30988 * it becomes lower case (no-one likes upper case filenames!) 30989 * we strip the padding character `=` 30990 30991 `base32` is used rather than the more efficient `base64` so rclone can be 30992 used on case insensitive remotes (e.g. Windows, Box, Dropbox, Onedrive etc). 30993 30994 ### Key derivation 30995 30996 Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an 30997 optional user supplied salt (password2) to derive the 32+32+16 = 80 30998 bytes of key material required. If the user doesn't supply a salt 30999 then rclone uses an internal one. 31000 31001 `scrypt` makes it impractical to mount a dictionary attack on rclone 31002 encrypted data. For full protection against this you should always use 31003 a salt. 31004 31005 ## SEE ALSO 31006 31007 * [rclone cryptdecode](https://rclone.org/commands/rclone_cryptdecode/) - Show forward/reverse mapping of encrypted filenames 31008 31009 # Compress 31010 31011 ## Warning 31012 31013 This remote is currently **experimental**. Things may break and data may be lost. Anything you do with this remote is 31014 at your own risk. Please understand the risks associated with using experimental code and don't use this remote in 31015 critical applications. 31016 31017 The `Compress` remote adds compression to another remote. It is best used with remotes containing 31018 many large compressible files. 31019 31020 ## Configuration 31021 31022 To use this remote, all you need to do is specify another remote and a compression mode to use: 31023 31024 Current remotes: 31025 31026 Name Type ==== ==== remote_to_press sometype 31027 31028 e) Edit existing remote $ rclone config 31029 f) New remote 31030 g) Delete remote 31031 h) Rename remote 31032 i) Copy remote 31033 j) Set configuration password 31034 k) Quit config e/n/d/r/c/s/q> n name> compress ... 8 / Compress a 31035 remote "compress" ... Storage> compress ** See help for compress 31036 backend at: https://rclone.org/compress/ ** 31037 31038 Remote to compress. Enter a string value. Press Enter for the default 31039 (""). remote> remote_to_press:subdir Compression mode. Enter a string 31040 value. Press Enter for the default ("gzip"). Choose a number from below, 31041 or type in your own value 1 / Gzip compression balanced for speed and 31042 compression strength. "gzip" compression_mode> gzip Edit advanced 31043 config? (y/n) y) Yes n) No (default) y/n> n Remote config 31044 -------------------- [compress] type = compress remote = 31045 remote_to_press:subdir compression_mode = gzip -------------------- y) 31046 Yes this is OK (default) e) Edit this remote d) Delete this remote 31047 y/e/d> y 31048 31049 31050 ### Compression Modes 31051 31052 Currently only gzip compression is supported. It provides a decent balance between speed and size and is well 31053 supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no 31054 compression and 9 is strongest compression. 31055 31056 ### File types 31057 31058 If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to 31059 the compression algorithm you chose. These files are standard files that can be opened by various archive programs, 31060 but they have some hidden metadata that allows them to be used by rclone. 31061 While you may download and decompress these files at will, do **not** manually delete or rename files. Files without 31062 correct metadata files will not be recognized by rclone. 31063 31064 ### File names 31065 31066 The compressed files will be named `*.###########.gz` where `*` is the base file and the `#` part is base64 encoded 31067 size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend. 31068 31069 31070 ### Standard options 31071 31072 Here are the Standard options specific to compress (Compress a remote). 31073 31074 #### --compress-remote 31075 31076 Remote to compress. 31077 31078 Properties: 31079 31080 - Config: remote 31081 - Env Var: RCLONE_COMPRESS_REMOTE 31082 - Type: string 31083 - Required: true 31084 31085 #### --compress-mode 31086 31087 Compression mode. 31088 31089 Properties: 31090 31091 - Config: mode 31092 - Env Var: RCLONE_COMPRESS_MODE 31093 - Type: string 31094 - Default: "gzip" 31095 - Examples: 31096 - "gzip" 31097 - Standard gzip compression with fastest parameters. 31098 31099 ### Advanced options 31100 31101 Here are the Advanced options specific to compress (Compress a remote). 31102 31103 #### --compress-level 31104 31105 GZIP compression level (-2 to 9). 31106 31107 Generally -1 (default, equivalent to 5) is recommended. 31108 Levels 1 to 9 increase compression at the cost of speed. Going past 6 31109 generally offers very little return. 31110 31111 Level -2 uses Huffman encoding only. Only use if you know what you 31112 are doing. 31113 Level 0 turns off compression. 31114 31115 Properties: 31116 31117 - Config: level 31118 - Env Var: RCLONE_COMPRESS_LEVEL 31119 - Type: int 31120 - Default: -1 31121 31122 #### --compress-ram-cache-limit 31123 31124 Some remotes don't allow the upload of files with unknown size. 31125 In this case the compressed file will need to be cached to determine 31126 it's size. 31127 31128 Files smaller than this limit will be cached in RAM, files larger than 31129 this limit will be cached on disk. 31130 31131 Properties: 31132 31133 - Config: ram_cache_limit 31134 - Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT 31135 - Type: SizeSuffix 31136 - Default: 20Mi 31137 31138 #### --compress-description 31139 31140 Description of the remote 31141 31142 Properties: 31143 31144 - Config: description 31145 - Env Var: RCLONE_COMPRESS_DESCRIPTION 31146 - Type: string 31147 - Required: false 31148 31149 ### Metadata 31150 31151 Any metadata supported by the underlying remote is read and written. 31152 31153 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 31154 31155 31156 31157 # Combine 31158 31159 The `combine` backend joins remotes together into a single directory 31160 tree. 31161 31162 For example you might have a remote for images on one provider: 31163 31164 $ rclone tree s3:imagesbucket / ├── image1.jpg └── image2.jpg 31165 31166 31167 And a remote for files on another: 31168 31169 $ rclone tree drive:important/files / ├── file1.txt └── file2.txt 31170 31171 31172 The `combine` backend can join these together into a synthetic 31173 directory structure like this: 31174 31175 $ rclone tree combined: / ├── files │ ├── file1.txt │ └── file2.txt └── 31176 images ├── image1.jpg └── image2.jpg 31177 31178 31179 You'd do this by specifying an `upstreams` parameter in the config 31180 like this 31181 31182 upstreams = images=s3:imagesbucket files=drive:important/files 31183 31184 During the initial setup with `rclone config` you will specify the 31185 upstreams remotes as a space separated list. The upstream remotes can 31186 either be a local paths or other remotes. 31187 31188 ## Configuration 31189 31190 Here is an example of how to make a combine called `remote` for the 31191 example above. First run: 31192 31193 rclone config 31194 31195 This will guide you through an interactive setup process: 31196 31197 No remotes found, make a new one? n) New remote s) Set configuration 31198 password q) Quit config n/s/q> n name> remote Option Storage. Type of 31199 storage to configure. Choose a number from below, or type in your own 31200 value. ... XX / Combine several remotes into one (combine) ... Storage> 31201 combine Option upstreams. Upstreams for combining These should be in the 31202 form dir=remote:path dir2=remote2:path Where before the = is specified 31203 the root directory and after is the remote to put there. Embedded spaces 31204 can be added using quotes "dir=remote:path with space" 31205 "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> 31206 images=s3:imagesbucket files=drive:important/files -------------------- 31207 [remote] type = combine upstreams = images=s3:imagesbucket 31208 files=drive:important/files -------------------- y) Yes this is OK 31209 (default) e) Edit this remote d) Delete this remote y/e/d> y 31210 31211 31212 ### Configuring for Google Drive Shared Drives 31213 31214 Rclone has a convenience feature for making a combine backend for all 31215 the shared drives you have access to. 31216 31217 Assuming your main (non shared drive) Google drive remote is called 31218 `drive:` you would run 31219 31220 rclone backend -o config drives drive: 31221 31222 This would produce something like this: 31223 31224 [My Drive] 31225 type = alias 31226 remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=: 31227 31228 [Test Drive] 31229 type = alias 31230 remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=: 31231 31232 [AllDrives] 31233 type = combine 31234 upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:" 31235 31236 If you then add that config to your config file (find it with `rclone 31237 config file`) then you can access all the shared drives in one place 31238 with the `AllDrives:` remote. 31239 31240 See [the Google Drive docs](https://rclone.org/drive/#drives) for full info. 31241 31242 31243 ### Standard options 31244 31245 Here are the Standard options specific to combine (Combine several remotes into one). 31246 31247 #### --combine-upstreams 31248 31249 Upstreams for combining 31250 31251 These should be in the form 31252 31253 dir=remote:path dir2=remote2:path 31254 31255 Where before the = is specified the root directory and after is the remote to 31256 put there. 31257 31258 Embedded spaces can be added using quotes 31259 31260 "dir=remote:path with space" "dir2=remote2:path with space" 31261 31262 31263 31264 Properties: 31265 31266 - Config: upstreams 31267 - Env Var: RCLONE_COMBINE_UPSTREAMS 31268 - Type: SpaceSepList 31269 - Default: 31270 31271 ### Advanced options 31272 31273 Here are the Advanced options specific to combine (Combine several remotes into one). 31274 31275 #### --combine-description 31276 31277 Description of the remote 31278 31279 Properties: 31280 31281 - Config: description 31282 - Env Var: RCLONE_COMBINE_DESCRIPTION 31283 - Type: string 31284 - Required: false 31285 31286 ### Metadata 31287 31288 Any metadata supported by the underlying remote is read and written. 31289 31290 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 31291 31292 31293 31294 # Dropbox 31295 31296 Paths are specified as `remote:path` 31297 31298 Dropbox paths may be as deep as required, e.g. 31299 `remote:directory/subdirectory`. 31300 31301 ## Configuration 31302 31303 The initial setup for dropbox involves getting a token from Dropbox 31304 which you need to do in your browser. `rclone config` walks you 31305 through it. 31306 31307 Here is an example of how to make a remote called `remote`. First run: 31308 31309 rclone config 31310 31311 This will guide you through an interactive setup process: 31312 31313 n) New remote 31314 o) Delete remote 31315 p) Quit config e/n/d/q> n name> remote Type of storage to configure. 31316 Choose a number from below, or type in your own value [snip] XX / 31317 Dropbox "dropbox" [snip] Storage> dropbox Dropbox App Key - leave 31318 blank normally. app_key> Dropbox App Secret - leave blank normally. 31319 app_secret> Remote config Please visit: 31320 https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code 31321 Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX 31322 -------------------- [remote] app_key = app_secret = token = 31323 XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 31324 -------------------- 31325 q) Yes this is OK 31326 r) Edit this remote 31327 s) Delete this remote y/e/d> y 31328 31329 31330 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 31331 machine with no Internet browser available. 31332 31333 Note that rclone runs a webserver on your local machine to collect the 31334 token as returned from Dropbox. This only 31335 runs from the moment it opens your browser to the moment you get back 31336 the verification code. This is on `http://127.0.0.1:53682/` and it 31337 may require you to unblock it temporarily if you are running a host 31338 firewall, or use manual mode. 31339 31340 You can then use it like this, 31341 31342 List directories in top level of your dropbox 31343 31344 rclone lsd remote: 31345 31346 List all the files in your dropbox 31347 31348 rclone ls remote: 31349 31350 To copy a local directory to a dropbox directory called backup 31351 31352 rclone copy /home/source remote:backup 31353 31354 ### Dropbox for business 31355 31356 Rclone supports Dropbox for business and Team Folders. 31357 31358 When using Dropbox for business `remote:` and `remote:path/to/file` 31359 will refer to your personal folder. 31360 31361 If you wish to see Team Folders you must use a leading `/` in the 31362 path, so `rclone lsd remote:/` will refer to the root and show you all 31363 Team Folders and your User Folder. 31364 31365 You can then use team folders like this `remote:/TeamFolder` and 31366 `remote:/TeamFolder/path/to/file`. 31367 31368 A leading `/` for a Dropbox personal account will do nothing, but it 31369 will take an extra HTTP transaction so it should be avoided. 31370 31371 ### Modification times and hashes 31372 31373 Dropbox supports modified times, but the only way to set a 31374 modification time is to re-upload the file. 31375 31376 This means that if you uploaded your data with an older version of 31377 rclone which didn't support the v2 API and modified times, rclone will 31378 decide to upload all your old data to fix the modification times. If 31379 you don't want this to happen use `--size-only` or `--checksum` flag 31380 to stop it. 31381 31382 Dropbox supports [its own hash 31383 type](https://www.dropbox.com/developers/reference/content-hash) which 31384 is checked for all transfers. 31385 31386 ### Restricted filename characters 31387 31388 | Character | Value | Replacement | 31389 | --------- |:-----:|:-----------:| 31390 | NUL | 0x00 | ␀ | 31391 | / | 0x2F | / | 31392 | DEL | 0x7F | ␡ | 31393 | \ | 0x5C | \ | 31394 31395 File names can also not end with the following characters. 31396 These only get replaced if they are the last character in the name: 31397 31398 | Character | Value | Replacement | 31399 | --------- |:-----:|:-----------:| 31400 | SP | 0x20 | ␠ | 31401 31402 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 31403 as they can't be used in JSON strings. 31404 31405 ### Batch mode uploads {#batch-mode} 31406 31407 Using batch mode uploads is very important for performance when using 31408 the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide) 31409 for more info. 31410 31411 There are 3 modes rclone can use for uploads. 31412 31413 #### --dropbox-batch-mode off 31414 31415 In this mode rclone will not use upload batching. This was the default 31416 before rclone v1.55. It has the disadvantage that it is very likely to 31417 encounter `too_many_requests` errors like this 31418 31419 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds. 31420 31421 When rclone receives these it has to wait for 15s or sometimes 300s 31422 before continuing which really slows down transfers. 31423 31424 This will happen especially if `--transfers` is large, so this mode 31425 isn't recommended except for compatibility or investigating problems. 31426 31427 #### --dropbox-batch-mode sync 31428 31429 In this mode rclone will batch up uploads to the size specified by 31430 `--dropbox-batch-size` and commit them together. 31431 31432 Using this mode means you can use a much higher `--transfers` 31433 parameter (32 or 64 works fine) without receiving `too_many_requests` 31434 errors. 31435 31436 This mode ensures full data integrity. 31437 31438 Note that there may be a pause when quitting rclone while rclone 31439 finishes up the last batch using this mode. 31440 31441 #### --dropbox-batch-mode async 31442 31443 In this mode rclone will batch up uploads to the size specified by 31444 `--dropbox-batch-size` and commit them together. 31445 31446 However it will not wait for the status of the batch to be returned to 31447 the caller. This means rclone can use a much bigger batch size (much 31448 bigger than `--transfers`), at the cost of not being able to check the 31449 status of the upload. 31450 31451 This provides the maximum possible upload speed especially with lots 31452 of small files, however rclone can't check the file got uploaded 31453 properly using this mode. 31454 31455 If you are using this mode then using "rclone check" after the 31456 transfer completes is recommended. Or you could do an initial transfer 31457 with `--dropbox-batch-mode async` then do a final transfer with 31458 `--dropbox-batch-mode sync` (the default). 31459 31460 Note that there may be a pause when quitting rclone while rclone 31461 finishes up the last batch using this mode. 31462 31463 31464 31465 ### Standard options 31466 31467 Here are the Standard options specific to dropbox (Dropbox). 31468 31469 #### --dropbox-client-id 31470 31471 OAuth Client Id. 31472 31473 Leave blank normally. 31474 31475 Properties: 31476 31477 - Config: client_id 31478 - Env Var: RCLONE_DROPBOX_CLIENT_ID 31479 - Type: string 31480 - Required: false 31481 31482 #### --dropbox-client-secret 31483 31484 OAuth Client Secret. 31485 31486 Leave blank normally. 31487 31488 Properties: 31489 31490 - Config: client_secret 31491 - Env Var: RCLONE_DROPBOX_CLIENT_SECRET 31492 - Type: string 31493 - Required: false 31494 31495 ### Advanced options 31496 31497 Here are the Advanced options specific to dropbox (Dropbox). 31498 31499 #### --dropbox-token 31500 31501 OAuth Access Token as a JSON blob. 31502 31503 Properties: 31504 31505 - Config: token 31506 - Env Var: RCLONE_DROPBOX_TOKEN 31507 - Type: string 31508 - Required: false 31509 31510 #### --dropbox-auth-url 31511 31512 Auth server URL. 31513 31514 Leave blank to use the provider defaults. 31515 31516 Properties: 31517 31518 - Config: auth_url 31519 - Env Var: RCLONE_DROPBOX_AUTH_URL 31520 - Type: string 31521 - Required: false 31522 31523 #### --dropbox-token-url 31524 31525 Token server url. 31526 31527 Leave blank to use the provider defaults. 31528 31529 Properties: 31530 31531 - Config: token_url 31532 - Env Var: RCLONE_DROPBOX_TOKEN_URL 31533 - Type: string 31534 - Required: false 31535 31536 #### --dropbox-chunk-size 31537 31538 Upload chunk size (< 150Mi). 31539 31540 Any files larger than this will be uploaded in chunks of this size. 31541 31542 Note that chunks are buffered in memory (one at a time) so rclone can 31543 deal with retries. Setting this larger will increase the speed 31544 slightly (at most 10% for 128 MiB in tests) at the cost of using more 31545 memory. It can be set smaller if you are tight on memory. 31546 31547 Properties: 31548 31549 - Config: chunk_size 31550 - Env Var: RCLONE_DROPBOX_CHUNK_SIZE 31551 - Type: SizeSuffix 31552 - Default: 48Mi 31553 31554 #### --dropbox-impersonate 31555 31556 Impersonate this user when using a business account. 31557 31558 Note that if you want to use impersonate, you should make sure this 31559 flag is set when running "rclone config" as this will cause rclone to 31560 request the "members.read" scope which it won't normally. This is 31561 needed to lookup a members email address into the internal ID that 31562 dropbox uses in the API. 31563 31564 Using the "members.read" scope will require a Dropbox Team Admin 31565 to approve during the OAuth flow. 31566 31567 You will have to use your own App (setting your own client_id and 31568 client_secret) to use this option as currently rclone's default set of 31569 permissions doesn't include "members.read". This can be added once 31570 v1.55 or later is in use everywhere. 31571 31572 31573 Properties: 31574 31575 - Config: impersonate 31576 - Env Var: RCLONE_DROPBOX_IMPERSONATE 31577 - Type: string 31578 - Required: false 31579 31580 #### --dropbox-shared-files 31581 31582 Instructs rclone to work on individual shared files. 31583 31584 In this mode rclone's features are extremely limited - only list (ls, lsl, etc.) 31585 operations and read operations (e.g. downloading) are supported in this mode. 31586 All other operations will be disabled. 31587 31588 Properties: 31589 31590 - Config: shared_files 31591 - Env Var: RCLONE_DROPBOX_SHARED_FILES 31592 - Type: bool 31593 - Default: false 31594 31595 #### --dropbox-shared-folders 31596 31597 Instructs rclone to work on shared folders. 31598 31599 When this flag is used with no path only the List operation is supported and 31600 all available shared folders will be listed. If you specify a path the first part 31601 will be interpreted as the name of shared folder. Rclone will then try to mount this 31602 shared to the root namespace. On success shared folder rclone proceeds normally. 31603 The shared folder is now pretty much a normal folder and all normal operations 31604 are supported. 31605 31606 Note that we don't unmount the shared folder afterwards so the 31607 --dropbox-shared-folders can be omitted after the first use of a particular 31608 shared folder. 31609 31610 Properties: 31611 31612 - Config: shared_folders 31613 - Env Var: RCLONE_DROPBOX_SHARED_FOLDERS 31614 - Type: bool 31615 - Default: false 31616 31617 #### --dropbox-pacer-min-sleep 31618 31619 Minimum time to sleep between API calls. 31620 31621 Properties: 31622 31623 - Config: pacer_min_sleep 31624 - Env Var: RCLONE_DROPBOX_PACER_MIN_SLEEP 31625 - Type: Duration 31626 - Default: 10ms 31627 31628 #### --dropbox-encoding 31629 31630 The encoding for the backend. 31631 31632 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 31633 31634 Properties: 31635 31636 - Config: encoding 31637 - Env Var: RCLONE_DROPBOX_ENCODING 31638 - Type: Encoding 31639 - Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot 31640 31641 #### --dropbox-batch-mode 31642 31643 Upload file batching sync|async|off. 31644 31645 This sets the batch mode used by rclone. 31646 31647 For full info see [the main docs](https://rclone.org/dropbox/#batch-mode) 31648 31649 This has 3 possible values 31650 31651 - off - no batching 31652 - sync - batch uploads and check completion (default) 31653 - async - batch upload and don't check completion 31654 31655 Rclone will close any outstanding batches when it exits which may make 31656 a delay on quit. 31657 31658 31659 Properties: 31660 31661 - Config: batch_mode 31662 - Env Var: RCLONE_DROPBOX_BATCH_MODE 31663 - Type: string 31664 - Default: "sync" 31665 31666 #### --dropbox-batch-size 31667 31668 Max number of files in upload batch. 31669 31670 This sets the batch size of files to upload. It has to be less than 1000. 31671 31672 By default this is 0 which means rclone which calculate the batch size 31673 depending on the setting of batch_mode. 31674 31675 - batch_mode: async - default batch_size is 100 31676 - batch_mode: sync - default batch_size is the same as --transfers 31677 - batch_mode: off - not in use 31678 31679 Rclone will close any outstanding batches when it exits which may make 31680 a delay on quit. 31681 31682 Setting this is a great idea if you are uploading lots of small files 31683 as it will make them a lot quicker. You can use --transfers 32 to 31684 maximise throughput. 31685 31686 31687 Properties: 31688 31689 - Config: batch_size 31690 - Env Var: RCLONE_DROPBOX_BATCH_SIZE 31691 - Type: int 31692 - Default: 0 31693 31694 #### --dropbox-batch-timeout 31695 31696 Max time to allow an idle upload batch before uploading. 31697 31698 If an upload batch is idle for more than this long then it will be 31699 uploaded. 31700 31701 The default for this is 0 which means rclone will choose a sensible 31702 default based on the batch_mode in use. 31703 31704 - batch_mode: async - default batch_timeout is 10s 31705 - batch_mode: sync - default batch_timeout is 500ms 31706 - batch_mode: off - not in use 31707 31708 31709 Properties: 31710 31711 - Config: batch_timeout 31712 - Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT 31713 - Type: Duration 31714 - Default: 0s 31715 31716 #### --dropbox-batch-commit-timeout 31717 31718 Max time to wait for a batch to finish committing 31719 31720 Properties: 31721 31722 - Config: batch_commit_timeout 31723 - Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT 31724 - Type: Duration 31725 - Default: 10m0s 31726 31727 #### --dropbox-description 31728 31729 Description of the remote 31730 31731 Properties: 31732 31733 - Config: description 31734 - Env Var: RCLONE_DROPBOX_DESCRIPTION 31735 - Type: string 31736 - Required: false 31737 31738 31739 31740 ## Limitations 31741 31742 Note that Dropbox is case insensitive so you can't have a file called 31743 "Hello.doc" and one called "hello.doc". 31744 31745 There are some file names such as `thumbs.db` which Dropbox can't 31746 store. There is a full list of them in the ["Ignored Files" section 31747 of this document](https://www.dropbox.com/en/help/145). Rclone will 31748 issue an error message `File name disallowed - not uploading` if it 31749 attempts to upload one of those file names, but the sync won't fail. 31750 31751 Some errors may occur if you try to sync copyright-protected files 31752 because Dropbox has its own [copyright detector](https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/) that 31753 prevents this sort of file being downloaded. This will return the error `ERROR : 31754 /path/to/your/file: Failed to copy: failed to open source object: 31755 path/restricted_content/.` 31756 31757 If you have more than 10,000 files in a directory then `rclone purge 31758 dropbox:dir` will return the error `Failed to purge: There are too 31759 many files involved in this operation`. As a work-around do an 31760 `rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`. 31761 31762 When using `rclone link` you'll need to set `--expire` if using a 31763 non-personal account otherwise the visibility may not be correct. 31764 (Note that `--expire` isn't supported on personal accounts). See the 31765 [forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the 31766 [dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75). 31767 31768 ## Get your own Dropbox App ID 31769 31770 When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users. 31771 31772 Here is how to create your own Dropbox App ID for rclone: 31773 31774 1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not 31775 to be the same account as the Dropbox you want to access) 31776 31777 2. Choose an API => Usually this should be `Dropbox API` 31778 31779 3. Choose the type of access you want to use => `Full Dropbox` or `App Folder`. If you want to use Team Folders, `Full Dropbox` is required ([see here](https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/How-to-create-team-folder-inside-my-app-s-folder/m-p/601005/highlight/true#M27911)). 31780 31781 4. Name your App. The app name is global, so you can't use `rclone` for example 31782 31783 5. Click the button `Create App` 31784 31785 6. Switch to the `Permissions` tab. Enable at least the following permissions: `account_info.read`, `files.metadata.write`, `files.content.write`, `files.content.read`, `sharing.write`. The `files.metadata.read` and `sharing.read` checkboxes will be marked too. Click `Submit` 31786 31787 7. Switch to the `Settings` tab. Fill `OAuth2 - Redirect URIs` as `http://localhost:53682/` and click on `Add` 31788 31789 8. Find the `App key` and `App secret` values on the `Settings` tab. Use these values in rclone config to add a new remote or edit an existing remote. The `App key` setting corresponds to `client_id` in rclone config, the `App secret` corresponds to `client_secret` 31790 31791 # Enterprise File Fabric 31792 31793 This backend supports [Storage Made Easy's Enterprise File 31794 Fabric™](https://storagemadeeasy.com/about/) which provides a software 31795 solution to integrate and unify File and Object Storage accessible 31796 through a global file system. 31797 31798 ## Configuration 31799 31800 The initial setup for the Enterprise File Fabric backend involves 31801 getting a token from the Enterprise File Fabric which you need to 31802 do in your browser. `rclone config` walks you through it. 31803 31804 Here is an example of how to make a remote called `remote`. First run: 31805 31806 rclone config 31807 31808 This will guide you through an interactive setup process: 31809 31810 No remotes found, make a new one? n) New remote s) Set configuration 31811 password q) Quit config n/s/q> n name> remote Type of storage to 31812 configure. Enter a string value. Press Enter for the default (""). 31813 Choose a number from below, or type in your own value [snip] XX / 31814 Enterprise File Fabric "filefabric" [snip] Storage> filefabric ** See 31815 help for filefabric backend at: https://rclone.org/filefabric/ ** 31816 31817 URL of the Enterprise File Fabric to connect to Enter a string value. 31818 Press Enter for the default (""). Choose a number from below, or type in 31819 your own value 1 / Storage Made Easy US "https://storagemadeeasy.com" 2 31820 / Storage Made Easy EU "https://eu.storagemadeeasy.com" 3 / Connect to 31821 your Enterprise File Fabric "https://yourfabric.smestorage.com" url> 31822 https://yourfabric.smestorage.com/ ID of the root folder Leave blank 31823 normally. 31824 31825 Fill in to make rclone start with directory of a given ID. 31826 31827 Enter a string value. Press Enter for the default (""). root_folder_id> 31828 Permanent Authentication Token 31829 31830 A Permanent Authentication Token can be created in the Enterprise File 31831 Fabric, on the users Dashboard under Security, there is an entry you'll 31832 see called "My Authentication Tokens". Click the Manage button to create 31833 one. 31834 31835 These tokens are normally valid for several years. 31836 31837 For more info see: 31838 https://docs.storagemadeeasy.com/organisationcloud/api-tokens 31839 31840 Enter a string value. Press Enter for the default (""). permanent_token> 31841 xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx Edit advanced config? (y/n) y) Yes n) 31842 No (default) y/n> n Remote config -------------------- [remote] type = 31843 filefabric url = https://yourfabric.smestorage.com/ permanent_token = 31844 xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx -------------------- y) Yes this is OK 31845 (default) e) Edit this remote d) Delete this remote y/e/d> y 31846 31847 31848 Once configured you can then use `rclone` like this, 31849 31850 List directories in top level of your Enterprise File Fabric 31851 31852 rclone lsd remote: 31853 31854 List all the files in your Enterprise File Fabric 31855 31856 rclone ls remote: 31857 31858 To copy a local directory to an Enterprise File Fabric directory called backup 31859 31860 rclone copy /home/source remote:backup 31861 31862 ### Modification times and hashes 31863 31864 The Enterprise File Fabric allows modification times to be set on 31865 files accurate to 1 second. These will be used to detect whether 31866 objects need syncing or not. 31867 31868 The Enterprise File Fabric does not support any data hashes at this time. 31869 31870 ### Restricted filename characters 31871 31872 The [default restricted characters set](https://rclone.org/overview/#restricted-characters) 31873 will be replaced. 31874 31875 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 31876 as they can't be used in JSON strings. 31877 31878 ### Empty files 31879 31880 Empty files aren't supported by the Enterprise File Fabric. Rclone will therefore 31881 upload an empty file as a single space with a mime type of 31882 `application/vnd.rclone.empty.file` and files with that mime type are 31883 treated as empty. 31884 31885 ### Root folder ID ### 31886 31887 You can set the `root_folder_id` for rclone. This is the directory 31888 (identified by its `Folder ID`) that rclone considers to be the root 31889 of your Enterprise File Fabric. 31890 31891 Normally you will leave this blank and rclone will determine the 31892 correct root to use itself. 31893 31894 However you can set this to restrict rclone to a specific folder 31895 hierarchy. 31896 31897 In order to do this you will have to find the `Folder ID` of the 31898 directory you wish rclone to display. These aren't displayed in the 31899 web interface, but you can use `rclone lsf` to find them, for example 31900 31901 $ rclone lsf --dirs-only -Fip --csv filefabric: 120673758,Burnt PDFs/ 31902 120673759,My Quick Uploads/ 120673755,My Syncs/ 120673756,My backups/ 31903 120673757,My contacts/ 120673761,S3 Storage/ 31904 31905 31906 The ID for "S3 Storage" would be `120673761`. 31907 31908 31909 ### Standard options 31910 31911 Here are the Standard options specific to filefabric (Enterprise File Fabric). 31912 31913 #### --filefabric-url 31914 31915 URL of the Enterprise File Fabric to connect to. 31916 31917 Properties: 31918 31919 - Config: url 31920 - Env Var: RCLONE_FILEFABRIC_URL 31921 - Type: string 31922 - Required: true 31923 - Examples: 31924 - "https://storagemadeeasy.com" 31925 - Storage Made Easy US 31926 - "https://eu.storagemadeeasy.com" 31927 - Storage Made Easy EU 31928 - "https://yourfabric.smestorage.com" 31929 - Connect to your Enterprise File Fabric 31930 31931 #### --filefabric-root-folder-id 31932 31933 ID of the root folder. 31934 31935 Leave blank normally. 31936 31937 Fill in to make rclone start with directory of a given ID. 31938 31939 31940 Properties: 31941 31942 - Config: root_folder_id 31943 - Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID 31944 - Type: string 31945 - Required: false 31946 31947 #### --filefabric-permanent-token 31948 31949 Permanent Authentication Token. 31950 31951 A Permanent Authentication Token can be created in the Enterprise File 31952 Fabric, on the users Dashboard under Security, there is an entry 31953 you'll see called "My Authentication Tokens". Click the Manage button 31954 to create one. 31955 31956 These tokens are normally valid for several years. 31957 31958 For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens 31959 31960 31961 Properties: 31962 31963 - Config: permanent_token 31964 - Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN 31965 - Type: string 31966 - Required: false 31967 31968 ### Advanced options 31969 31970 Here are the Advanced options specific to filefabric (Enterprise File Fabric). 31971 31972 #### --filefabric-token 31973 31974 Session Token. 31975 31976 This is a session token which rclone caches in the config file. It is 31977 usually valid for 1 hour. 31978 31979 Don't set this value - rclone will set it automatically. 31980 31981 31982 Properties: 31983 31984 - Config: token 31985 - Env Var: RCLONE_FILEFABRIC_TOKEN 31986 - Type: string 31987 - Required: false 31988 31989 #### --filefabric-token-expiry 31990 31991 Token expiry time. 31992 31993 Don't set this value - rclone will set it automatically. 31994 31995 31996 Properties: 31997 31998 - Config: token_expiry 31999 - Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY 32000 - Type: string 32001 - Required: false 32002 32003 #### --filefabric-version 32004 32005 Version read from the file fabric. 32006 32007 Don't set this value - rclone will set it automatically. 32008 32009 32010 Properties: 32011 32012 - Config: version 32013 - Env Var: RCLONE_FILEFABRIC_VERSION 32014 - Type: string 32015 - Required: false 32016 32017 #### --filefabric-encoding 32018 32019 The encoding for the backend. 32020 32021 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 32022 32023 Properties: 32024 32025 - Config: encoding 32026 - Env Var: RCLONE_FILEFABRIC_ENCODING 32027 - Type: Encoding 32028 - Default: Slash,Del,Ctl,InvalidUtf8,Dot 32029 32030 #### --filefabric-description 32031 32032 Description of the remote 32033 32034 Properties: 32035 32036 - Config: description 32037 - Env Var: RCLONE_FILEFABRIC_DESCRIPTION 32038 - Type: string 32039 - Required: false 32040 32041 32042 32043 # FTP 32044 32045 FTP is the File Transfer Protocol. Rclone FTP support is provided using the 32046 [github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp) 32047 package. 32048 32049 [Limitations of Rclone's FTP backend](#limitations) 32050 32051 Paths are specified as `remote:path`. If the path does not begin with 32052 a `/` it is relative to the home directory of the user. An empty path 32053 `remote:` refers to the user's home directory. 32054 32055 ## Configuration 32056 32057 To create an FTP configuration named `remote`, run 32058 32059 rclone config 32060 32061 Rclone config guides you through an interactive setup process. A minimal 32062 rclone FTP remote definition only requires host, username and password. 32063 For an anonymous FTP server, see [below](#anonymous-ftp). 32064 32065 No remotes found, make a new one? n) New remote r) Rename remote c) Copy 32066 remote s) Set configuration password q) Quit config n/r/c/s/q> n name> 32067 remote Type of storage to configure. Enter a string value. Press Enter 32068 for the default (""). Choose a number from below, or type in your own 32069 value [snip] XX / FTP "ftp" [snip] Storage> ftp ** See help for ftp 32070 backend at: https://rclone.org/ftp/ ** 32071 32072 FTP host to connect to Enter a string value. Press Enter for the default 32073 (""). Choose a number from below, or type in your own value 1 / Connect 32074 to ftp.example.com "ftp.example.com" host> ftp.example.com FTP username 32075 Enter a string value. Press Enter for the default ("$USER"). user> FTP 32076 port number Enter a signed integer. Press Enter for the default (21). 32077 port> FTP password y) Yes type in my own password g) Generate random 32078 password y/g> y Enter the password: password: Confirm the password: 32079 password: Use FTP over TLS (Implicit) Enter a boolean value (true or 32080 false). Press Enter for the default ("false"). tls> Use FTP over TLS 32081 (Explicit) Enter a boolean value (true or false). Press Enter for the 32082 default ("false"). explicit_tls> Remote config -------------------- 32083 [remote] type = ftp host = ftp.example.com pass = *** ENCRYPTED *** 32084 -------------------- y) Yes this is OK e) Edit this remote d) Delete 32085 this remote y/e/d> y 32086 32087 32088 To see all directories in the home directory of `remote` 32089 32090 rclone lsd remote: 32091 32092 Make a new directory 32093 32094 rclone mkdir remote:path/to/directory 32095 32096 List the contents of a directory 32097 32098 rclone ls remote:path/to/directory 32099 32100 Sync `/home/local/directory` to the remote directory, deleting any 32101 excess files in the directory. 32102 32103 rclone sync --interactive /home/local/directory remote:directory 32104 32105 ### Anonymous FTP 32106 32107 When connecting to a FTP server that allows anonymous login, you can use the 32108 special "anonymous" username. Traditionally, this user account accepts any 32109 string as a password, although it is common to use either the password 32110 "anonymous" or "guest". Some servers require the use of a valid e-mail 32111 address as password. 32112 32113 Using [on-the-fly](#backend-path-to-dir) or 32114 [connection string](https://rclone.org/docs/#connection-strings) remotes makes it easy to access 32115 such servers, without requiring any configuration in advance. The following 32116 are examples of that: 32117 32118 rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=$(rclone obscure dummy) 32119 rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=$(rclone obscure dummy): 32120 32121 The above examples work in Linux shells and in PowerShell, but not Windows 32122 Command Prompt. They execute the [rclone obscure](https://rclone.org/commands/rclone_obscure/) 32123 command to create a password string in the format required by the 32124 [pass](#ftp-pass) option. The following examples are exactly the same, except use 32125 an already obscured string representation of the same password "dummy", and 32126 therefore works even in Windows Command Prompt: 32127 32128 rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM 32129 rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM: 32130 32131 ### Implicit TLS 32132 32133 Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to 32134 be enabled in the FTP backend config for the remote, or with 32135 [`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and 32136 can be set with [`--ftp-port`](#ftp-port). 32137 32138 ### Restricted filename characters 32139 32140 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 32141 the following characters are also replaced: 32142 32143 File names cannot end with the following characters. Replacement is 32144 limited to the last character in a file name: 32145 32146 | Character | Value | Replacement | 32147 | --------- |:-----:|:-----------:| 32148 | SP | 0x20 | ␠ | 32149 32150 Not all FTP servers can have all characters in file names, for example: 32151 32152 | FTP Server| Forbidden characters | 32153 | --------- |:--------------------:| 32154 | proftpd | `*` | 32155 | pureftpd | `\ [ ]` | 32156 32157 This backend's interactive configuration wizard provides a selection of 32158 sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd. 32159 Just hit a selection number when prompted. 32160 32161 32162 ### Standard options 32163 32164 Here are the Standard options specific to ftp (FTP). 32165 32166 #### --ftp-host 32167 32168 FTP host to connect to. 32169 32170 E.g. "ftp.example.com". 32171 32172 Properties: 32173 32174 - Config: host 32175 - Env Var: RCLONE_FTP_HOST 32176 - Type: string 32177 - Required: true 32178 32179 #### --ftp-user 32180 32181 FTP username. 32182 32183 Properties: 32184 32185 - Config: user 32186 - Env Var: RCLONE_FTP_USER 32187 - Type: string 32188 - Default: "$USER" 32189 32190 #### --ftp-port 32191 32192 FTP port number. 32193 32194 Properties: 32195 32196 - Config: port 32197 - Env Var: RCLONE_FTP_PORT 32198 - Type: int 32199 - Default: 21 32200 32201 #### --ftp-pass 32202 32203 FTP password. 32204 32205 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 32206 32207 Properties: 32208 32209 - Config: pass 32210 - Env Var: RCLONE_FTP_PASS 32211 - Type: string 32212 - Required: false 32213 32214 #### --ftp-tls 32215 32216 Use Implicit FTPS (FTP over TLS). 32217 32218 When using implicit FTP over TLS the client connects using TLS 32219 right from the start which breaks compatibility with 32220 non-TLS-aware servers. This is usually served over port 990 rather 32221 than port 21. Cannot be used in combination with explicit FTPS. 32222 32223 Properties: 32224 32225 - Config: tls 32226 - Env Var: RCLONE_FTP_TLS 32227 - Type: bool 32228 - Default: false 32229 32230 #### --ftp-explicit-tls 32231 32232 Use Explicit FTPS (FTP over TLS). 32233 32234 When using explicit FTP over TLS the client explicitly requests 32235 security from the server in order to upgrade a plain text connection 32236 to an encrypted one. Cannot be used in combination with implicit FTPS. 32237 32238 Properties: 32239 32240 - Config: explicit_tls 32241 - Env Var: RCLONE_FTP_EXPLICIT_TLS 32242 - Type: bool 32243 - Default: false 32244 32245 ### Advanced options 32246 32247 Here are the Advanced options specific to ftp (FTP). 32248 32249 #### --ftp-concurrency 32250 32251 Maximum number of FTP simultaneous connections, 0 for unlimited. 32252 32253 Note that setting this is very likely to cause deadlocks so it should 32254 be used with care. 32255 32256 If you are doing a sync or copy then make sure concurrency is one more 32257 than the sum of `--transfers` and `--checkers`. 32258 32259 If you use `--check-first` then it just needs to be one more than the 32260 maximum of `--checkers` and `--transfers`. 32261 32262 So for `concurrency 3` you'd use `--checkers 2 --transfers 2 32263 --check-first` or `--checkers 1 --transfers 1`. 32264 32265 32266 32267 Properties: 32268 32269 - Config: concurrency 32270 - Env Var: RCLONE_FTP_CONCURRENCY 32271 - Type: int 32272 - Default: 0 32273 32274 #### --ftp-no-check-certificate 32275 32276 Do not verify the TLS certificate of the server. 32277 32278 Properties: 32279 32280 - Config: no_check_certificate 32281 - Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE 32282 - Type: bool 32283 - Default: false 32284 32285 #### --ftp-disable-epsv 32286 32287 Disable using EPSV even if server advertises support. 32288 32289 Properties: 32290 32291 - Config: disable_epsv 32292 - Env Var: RCLONE_FTP_DISABLE_EPSV 32293 - Type: bool 32294 - Default: false 32295 32296 #### --ftp-disable-mlsd 32297 32298 Disable using MLSD even if server advertises support. 32299 32300 Properties: 32301 32302 - Config: disable_mlsd 32303 - Env Var: RCLONE_FTP_DISABLE_MLSD 32304 - Type: bool 32305 - Default: false 32306 32307 #### --ftp-disable-utf8 32308 32309 Disable using UTF-8 even if server advertises support. 32310 32311 Properties: 32312 32313 - Config: disable_utf8 32314 - Env Var: RCLONE_FTP_DISABLE_UTF8 32315 - Type: bool 32316 - Default: false 32317 32318 #### --ftp-writing-mdtm 32319 32320 Use MDTM to set modification time (VsFtpd quirk) 32321 32322 Properties: 32323 32324 - Config: writing_mdtm 32325 - Env Var: RCLONE_FTP_WRITING_MDTM 32326 - Type: bool 32327 - Default: false 32328 32329 #### --ftp-force-list-hidden 32330 32331 Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD. 32332 32333 Properties: 32334 32335 - Config: force_list_hidden 32336 - Env Var: RCLONE_FTP_FORCE_LIST_HIDDEN 32337 - Type: bool 32338 - Default: false 32339 32340 #### --ftp-idle-timeout 32341 32342 Max time before closing idle connections. 32343 32344 If no connections have been returned to the connection pool in the time 32345 given, rclone will empty the connection pool. 32346 32347 Set to 0 to keep connections indefinitely. 32348 32349 32350 Properties: 32351 32352 - Config: idle_timeout 32353 - Env Var: RCLONE_FTP_IDLE_TIMEOUT 32354 - Type: Duration 32355 - Default: 1m0s 32356 32357 #### --ftp-close-timeout 32358 32359 Maximum time to wait for a response to close. 32360 32361 Properties: 32362 32363 - Config: close_timeout 32364 - Env Var: RCLONE_FTP_CLOSE_TIMEOUT 32365 - Type: Duration 32366 - Default: 1m0s 32367 32368 #### --ftp-tls-cache-size 32369 32370 Size of TLS session cache for all control and data connections. 32371 32372 TLS cache allows to resume TLS sessions and reuse PSK between connections. 32373 Increase if default size is not enough resulting in TLS resumption errors. 32374 Enabled by default. Use 0 to disable. 32375 32376 Properties: 32377 32378 - Config: tls_cache_size 32379 - Env Var: RCLONE_FTP_TLS_CACHE_SIZE 32380 - Type: int 32381 - Default: 32 32382 32383 #### --ftp-disable-tls13 32384 32385 Disable TLS 1.3 (workaround for FTP servers with buggy TLS) 32386 32387 Properties: 32388 32389 - Config: disable_tls13 32390 - Env Var: RCLONE_FTP_DISABLE_TLS13 32391 - Type: bool 32392 - Default: false 32393 32394 #### --ftp-shut-timeout 32395 32396 Maximum time to wait for data connection closing status. 32397 32398 Properties: 32399 32400 - Config: shut_timeout 32401 - Env Var: RCLONE_FTP_SHUT_TIMEOUT 32402 - Type: Duration 32403 - Default: 1m0s 32404 32405 #### --ftp-ask-password 32406 32407 Allow asking for FTP password when needed. 32408 32409 If this is set and no password is supplied then rclone will ask for a password 32410 32411 32412 Properties: 32413 32414 - Config: ask_password 32415 - Env Var: RCLONE_FTP_ASK_PASSWORD 32416 - Type: bool 32417 - Default: false 32418 32419 #### --ftp-socks-proxy 32420 32421 Socks 5 proxy host. 32422 32423 Supports the format user:pass@host:port, user@host:port, host:port. 32424 32425 Example: 32426 32427 myUser:myPass@localhost:9005 32428 32429 32430 Properties: 32431 32432 - Config: socks_proxy 32433 - Env Var: RCLONE_FTP_SOCKS_PROXY 32434 - Type: string 32435 - Required: false 32436 32437 #### --ftp-encoding 32438 32439 The encoding for the backend. 32440 32441 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 32442 32443 Properties: 32444 32445 - Config: encoding 32446 - Env Var: RCLONE_FTP_ENCODING 32447 - Type: Encoding 32448 - Default: Slash,Del,Ctl,RightSpace,Dot 32449 - Examples: 32450 - "Asterisk,Ctl,Dot,Slash" 32451 - ProFTPd can't handle '*' in file names 32452 - "BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket" 32453 - PureFTPd can't handle '[]' or '*' in file names 32454 - "Ctl,LeftPeriod,Slash" 32455 - VsFTPd can't handle file names starting with dot 32456 32457 #### --ftp-description 32458 32459 Description of the remote 32460 32461 Properties: 32462 32463 - Config: description 32464 - Env Var: RCLONE_FTP_DESCRIPTION 32465 - Type: string 32466 - Required: false 32467 32468 32469 32470 ## Limitations 32471 32472 FTP servers acting as rclone remotes must support `passive` mode. 32473 The mode cannot be configured as `passive` is the only supported one. 32474 Rclone's FTP implementation is not compatible with `active` mode 32475 as [the library it uses doesn't support it](https://github.com/jlaffaye/ftp/issues/29). 32476 This will likely never be supported due to security concerns. 32477 32478 Rclone's FTP backend does not support any checksums but can compare 32479 file sizes. 32480 32481 `rclone about` is not supported by the FTP backend. Backends without 32482 this capability cannot determine free space for an rclone mount or 32483 use policy `mfs` (most free space) as a member of an rclone union 32484 remote. 32485 32486 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 32487 32488 The implementation of : `--dump headers`, 32489 `--dump bodies`, `--dump auth` for debugging isn't the same as 32490 for rclone HTTP based backends - it has less fine grained control. 32491 32492 `--timeout` isn't supported (but `--contimeout` is). 32493 32494 `--bind` isn't supported. 32495 32496 Rclone's FTP backend could support server-side move but does not 32497 at present. 32498 32499 The `ftp_proxy` environment variable is not currently supported. 32500 32501 ### Modification times 32502 32503 File modification time (timestamps) is supported to 1 second resolution 32504 for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server. 32505 The `VsFTPd` server has non-standard implementation of time related protocol 32506 commands and needs a special configuration setting: `writing_mdtm = true`. 32507 32508 Support for precise file time with other FTP servers varies depending on what 32509 protocol extensions they advertise. If all the `MLSD`, `MDTM` and `MFTM` 32510 extensions are present, rclone will use them together to provide precise time. 32511 Otherwise the times you see on the FTP server through rclone are those of the 32512 last file upload. 32513 32514 You can use the following command to check whether rclone can use precise time 32515 with your FTP server: `rclone backend features your_ftp_remote:` (the trailing 32516 colon is important). Look for the number in the line tagged by `Precision` 32517 designating the remote time precision expressed as nanoseconds. A value of 32518 `1000000000` means that file time precision of 1 second is available. 32519 A value of `3153600000000000000` (or another large number) means "unsupported". 32520 32521 # Google Cloud Storage 32522 32523 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 32524 command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`. 32525 32526 ## Configuration 32527 32528 The initial setup for google cloud storage involves getting a token from Google Cloud Storage 32529 which you need to do in your browser. `rclone config` walks you 32530 through it. 32531 32532 Here is an example of how to make a remote called `remote`. First run: 32533 32534 rclone config 32535 32536 This will guide you through an interactive setup process: 32537 32538 n) New remote 32539 o) Delete remote 32540 p) Quit config e/n/d/q> n name> remote Type of storage to configure. 32541 Choose a number from below, or type in your own value [snip] XX / 32542 Google Cloud Storage (this is not Google Drive) "google cloud 32543 storage" [snip] Storage> google cloud storage Google Application 32544 Client Id - leave blank normally. client_id> Google Application 32545 Client Secret - leave blank normally. client_secret> Project number 32546 optional - needed only for list/create/delete buckets - see your 32547 developer console. project_number> 12345678 Service Account 32548 Credentials JSON file path - needed only if you want use SA instead 32549 of interactive login. service_account_file> Access Control List for 32550 new objects. Choose a number from below, or type in your own value 1 32551 / Object owner gets OWNER access, and all Authenticated Users get 32552 READER access. "authenticatedRead" 2 / Object owner gets OWNER 32553 access, and project team owners get OWNER access. 32554 "bucketOwnerFullControl" 3 / Object owner gets OWNER access, and 32555 project team owners get READER access. "bucketOwnerRead" 4 / Object 32556 owner gets OWNER access [default if left blank]. "private" 5 / 32557 Object owner gets OWNER access, and project team members get access 32558 according to their roles. "projectPrivate" 6 / Object owner gets 32559 OWNER access, and all Users get READER access. "publicRead" 32560 object_acl> 4 Access Control List for new buckets. Choose a number 32561 from below, or type in your own value 1 / Project team owners get 32562 OWNER access, and all Authenticated Users get READER access. 32563 "authenticatedRead" 2 / Project team owners get OWNER access 32564 [default if left blank]. "private" 3 / Project team members get 32565 access according to their roles. "projectPrivate" 4 / Project team 32566 owners get OWNER access, and all Users get READER access. 32567 "publicRead" 5 / Project team owners get OWNER access, and all 32568 Users get WRITER access. "publicReadWrite" bucket_acl> 2 Location 32569 for the newly created buckets. Choose a number from below, or type 32570 in your own value 1 / Empty for default location (US). "" 2 / 32571 Multi-regional location for Asia. "asia" 3 / Multi-regional 32572 location for Europe. "eu" 4 / Multi-regional location for United 32573 States. "us" 5 / Taiwan. "asia-east1" 6 / Tokyo. 32574 "asia-northeast1" 7 / Singapore. "asia-southeast1" 8 / Sydney. 32575 "australia-southeast1" 9 / Belgium. "europe-west1" 10 / London. 32576 "europe-west2" 11 / Iowa. "us-central1" 12 / South Carolina. 32577 "us-east1" 13 / Northern Virginia. "us-east4" 14 / Oregon. 32578 "us-west1" location> 12 The storage class to use when storing 32579 objects in Google Cloud Storage. Choose a number from below, or type 32580 in your own value 1 / Default "" 2 / Multi-regional storage class 32581 "MULTI_REGIONAL" 3 / Regional storage class "REGIONAL" 4 / 32582 Nearline storage class "NEARLINE" 5 / Coldline storage class 32583 "COLDLINE" 6 / Durable reduced availability storage class 32584 "DURABLE_REDUCED_AVAILABILITY" storage_class> 5 Remote config Use 32585 web browser to automatically authenticate rclone with remote? 32586 32587 - Say Y if the machine running rclone has a web browser you can use 32588 - Say N if running rclone on a (remote) machine without web browser 32589 access If not sure try Y. If Y failed, try N. 32590 32591 y) Yes 32592 z) No y/n> y If your browser doesn't open automatically go to the 32593 following link: http://127.0.0.1:53682/auth Log in and authorize 32594 rclone for access Waiting for code... Got code -------------------- 32595 [remote] type = google cloud storage client_id = client_secret = 32596 token = 32597 {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} 32598 project_number = 12345678 object_acl = private bucket_acl = private 32599 -------------------- 32600 a) Yes this is OK 32601 b) Edit this remote 32602 c) Delete this remote y/e/d> y 32603 32604 32605 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 32606 machine with no Internet browser available. 32607 32608 Note that rclone runs a webserver on your local machine to collect the 32609 token as returned from Google if using web browser to automatically 32610 authenticate. This only 32611 runs from the moment it opens your browser to the moment you get back 32612 the verification code. This is on `http://127.0.0.1:53682/` and this 32613 it may require you to unblock it temporarily if you are running a host 32614 firewall, or use manual mode. 32615 32616 This remote is called `remote` and can now be used like this 32617 32618 See all the buckets in your project 32619 32620 rclone lsd remote: 32621 32622 Make a new bucket 32623 32624 rclone mkdir remote:bucket 32625 32626 List the contents of a bucket 32627 32628 rclone ls remote:bucket 32629 32630 Sync `/home/local/directory` to the remote bucket, deleting any excess 32631 files in the bucket. 32632 32633 rclone sync --interactive /home/local/directory remote:bucket 32634 32635 ### Service Account support 32636 32637 You can set up rclone with Google Cloud Storage in an unattended mode, 32638 i.e. not tied to a specific end-user Google account. This is useful 32639 when you want to synchronise files onto machines that don't have 32640 actively logged-in users, for example build machines. 32641 32642 To get credentials for Google Cloud Platform 32643 [IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts), 32644 please head to the 32645 [Service Account](https://console.cloud.google.com/permissions/serviceaccounts) 32646 section of the Google Developer Console. Service Accounts behave just 32647 like normal `User` permissions in 32648 [Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control), 32649 so you can limit their access (e.g. make them read only). After 32650 creating an account, a JSON file containing the Service Account's 32651 credentials will be downloaded onto your machines. These credentials 32652 are what rclone will use for authentication. 32653 32654 To use a Service Account instead of OAuth2 token flow, enter the path 32655 to your Service Account credentials at the `service_account_file` 32656 prompt and rclone won't use the browser based authentication 32657 flow. If you'd rather stuff the contents of the credentials file into 32658 the rclone config file, you can set `service_account_credentials` with 32659 the actual contents of the file instead, or set the equivalent 32660 environment variable. 32661 32662 ### Anonymous Access 32663 32664 For downloads of objects that permit public access you can configure rclone 32665 to use anonymous access by setting `anonymous` to `true`. 32666 With unauthorized access you can't write or create files but only read or list 32667 those buckets and objects that have public read access. 32668 32669 ### Application Default Credentials 32670 32671 If no other source of credentials is provided, rclone will fall back 32672 to 32673 [Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials) 32674 this is useful both when you already have configured authentication 32675 for your developer account, or in production when running on a google 32676 compute host. Note that if running in docker, you may need to run 32677 additional commands on your google compute machine - 32678 [see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper). 32679 32680 Note that in the case application default credentials are used, there 32681 is no need to explicitly configure a project number. 32682 32683 ### --fast-list 32684 32685 This remote supports `--fast-list` which allows you to use fewer 32686 transactions in exchange for more memory. See the [rclone 32687 docs](https://rclone.org/docs/#fast-list) for more details. 32688 32689 ### Custom upload headers 32690 32691 You can set custom upload headers with the `--header-upload` 32692 flag. Google Cloud Storage supports the headers as described in the 32693 [working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata) 32694 32695 - Cache-Control 32696 - Content-Disposition 32697 - Content-Encoding 32698 - Content-Language 32699 - Content-Type 32700 - X-Goog-Storage-Class 32701 - X-Goog-Meta- 32702 32703 Eg `--header-upload "Content-Type text/potato"` 32704 32705 Note that the last of these is for setting custom metadata in the form 32706 `--header-upload "x-goog-meta-key: value"` 32707 32708 ### Modification times 32709 32710 Google Cloud Storage stores md5sum natively. 32711 Google's [gsutil](https://cloud.google.com/storage/docs/gsutil) tool stores modification time 32712 with one-second precision as `goog-reserved-file-mtime` in file metadata. 32713 32714 To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries. 32715 `mtime` uses RFC3339 format with one-nanosecond precision. 32716 `goog-reserved-file-mtime` uses Unix timestamp format with one-second precision. 32717 To get modification time from object metadata, rclone reads the metadata in the following order: `mtime`, `goog-reserved-file-mtime`, object updated time. 32718 32719 Note that rclone's default modify window is 1ns. 32720 Files uploaded by gsutil only contain timestamps with one-second precision. 32721 If you use rclone to sync files previously uploaded by gsutil, 32722 rclone will attempt to update modification time for all these files. 32723 To avoid these possibly unnecessary updates, use `--modify-window 1s`. 32724 32725 ### Restricted filename characters 32726 32727 | Character | Value | Replacement | 32728 | --------- |:-----:|:-----------:| 32729 | NUL | 0x00 | ␀ | 32730 | LF | 0x0A | ␊ | 32731 | CR | 0x0D | ␍ | 32732 | / | 0x2F | / | 32733 32734 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 32735 as they can't be used in JSON strings. 32736 32737 32738 ### Standard options 32739 32740 Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 32741 32742 #### --gcs-client-id 32743 32744 OAuth Client Id. 32745 32746 Leave blank normally. 32747 32748 Properties: 32749 32750 - Config: client_id 32751 - Env Var: RCLONE_GCS_CLIENT_ID 32752 - Type: string 32753 - Required: false 32754 32755 #### --gcs-client-secret 32756 32757 OAuth Client Secret. 32758 32759 Leave blank normally. 32760 32761 Properties: 32762 32763 - Config: client_secret 32764 - Env Var: RCLONE_GCS_CLIENT_SECRET 32765 - Type: string 32766 - Required: false 32767 32768 #### --gcs-project-number 32769 32770 Project number. 32771 32772 Optional - needed only for list/create/delete buckets - see your developer console. 32773 32774 Properties: 32775 32776 - Config: project_number 32777 - Env Var: RCLONE_GCS_PROJECT_NUMBER 32778 - Type: string 32779 - Required: false 32780 32781 #### --gcs-user-project 32782 32783 User project. 32784 32785 Optional - needed only for requester pays. 32786 32787 Properties: 32788 32789 - Config: user_project 32790 - Env Var: RCLONE_GCS_USER_PROJECT 32791 - Type: string 32792 - Required: false 32793 32794 #### --gcs-service-account-file 32795 32796 Service Account Credentials JSON file path. 32797 32798 Leave blank normally. 32799 Needed only if you want use SA instead of interactive login. 32800 32801 Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`. 32802 32803 Properties: 32804 32805 - Config: service_account_file 32806 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE 32807 - Type: string 32808 - Required: false 32809 32810 #### --gcs-service-account-credentials 32811 32812 Service Account Credentials JSON blob. 32813 32814 Leave blank normally. 32815 Needed only if you want use SA instead of interactive login. 32816 32817 Properties: 32818 32819 - Config: service_account_credentials 32820 - Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS 32821 - Type: string 32822 - Required: false 32823 32824 #### --gcs-anonymous 32825 32826 Access public buckets and objects without credentials. 32827 32828 Set to 'true' if you just want to download files and don't configure credentials. 32829 32830 Properties: 32831 32832 - Config: anonymous 32833 - Env Var: RCLONE_GCS_ANONYMOUS 32834 - Type: bool 32835 - Default: false 32836 32837 #### --gcs-object-acl 32838 32839 Access Control List for new objects. 32840 32841 Properties: 32842 32843 - Config: object_acl 32844 - Env Var: RCLONE_GCS_OBJECT_ACL 32845 - Type: string 32846 - Required: false 32847 - Examples: 32848 - "authenticatedRead" 32849 - Object owner gets OWNER access. 32850 - All Authenticated Users get READER access. 32851 - "bucketOwnerFullControl" 32852 - Object owner gets OWNER access. 32853 - Project team owners get OWNER access. 32854 - "bucketOwnerRead" 32855 - Object owner gets OWNER access. 32856 - Project team owners get READER access. 32857 - "private" 32858 - Object owner gets OWNER access. 32859 - Default if left blank. 32860 - "projectPrivate" 32861 - Object owner gets OWNER access. 32862 - Project team members get access according to their roles. 32863 - "publicRead" 32864 - Object owner gets OWNER access. 32865 - All Users get READER access. 32866 32867 #### --gcs-bucket-acl 32868 32869 Access Control List for new buckets. 32870 32871 Properties: 32872 32873 - Config: bucket_acl 32874 - Env Var: RCLONE_GCS_BUCKET_ACL 32875 - Type: string 32876 - Required: false 32877 - Examples: 32878 - "authenticatedRead" 32879 - Project team owners get OWNER access. 32880 - All Authenticated Users get READER access. 32881 - "private" 32882 - Project team owners get OWNER access. 32883 - Default if left blank. 32884 - "projectPrivate" 32885 - Project team members get access according to their roles. 32886 - "publicRead" 32887 - Project team owners get OWNER access. 32888 - All Users get READER access. 32889 - "publicReadWrite" 32890 - Project team owners get OWNER access. 32891 - All Users get WRITER access. 32892 32893 #### --gcs-bucket-policy-only 32894 32895 Access checks should use bucket-level IAM policies. 32896 32897 If you want to upload objects to a bucket with Bucket Policy Only set 32898 then you will need to set this. 32899 32900 When it is set, rclone: 32901 32902 - ignores ACLs set on buckets 32903 - ignores ACLs set on objects 32904 - creates buckets with Bucket Policy Only set 32905 32906 Docs: https://cloud.google.com/storage/docs/bucket-policy-only 32907 32908 32909 Properties: 32910 32911 - Config: bucket_policy_only 32912 - Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY 32913 - Type: bool 32914 - Default: false 32915 32916 #### --gcs-location 32917 32918 Location for the newly created buckets. 32919 32920 Properties: 32921 32922 - Config: location 32923 - Env Var: RCLONE_GCS_LOCATION 32924 - Type: string 32925 - Required: false 32926 - Examples: 32927 - "" 32928 - Empty for default location (US) 32929 - "asia" 32930 - Multi-regional location for Asia 32931 - "eu" 32932 - Multi-regional location for Europe 32933 - "us" 32934 - Multi-regional location for United States 32935 - "asia-east1" 32936 - Taiwan 32937 - "asia-east2" 32938 - Hong Kong 32939 - "asia-northeast1" 32940 - Tokyo 32941 - "asia-northeast2" 32942 - Osaka 32943 - "asia-northeast3" 32944 - Seoul 32945 - "asia-south1" 32946 - Mumbai 32947 - "asia-south2" 32948 - Delhi 32949 - "asia-southeast1" 32950 - Singapore 32951 - "asia-southeast2" 32952 - Jakarta 32953 - "australia-southeast1" 32954 - Sydney 32955 - "australia-southeast2" 32956 - Melbourne 32957 - "europe-north1" 32958 - Finland 32959 - "europe-west1" 32960 - Belgium 32961 - "europe-west2" 32962 - London 32963 - "europe-west3" 32964 - Frankfurt 32965 - "europe-west4" 32966 - Netherlands 32967 - "europe-west6" 32968 - Zürich 32969 - "europe-central2" 32970 - Warsaw 32971 - "us-central1" 32972 - Iowa 32973 - "us-east1" 32974 - South Carolina 32975 - "us-east4" 32976 - Northern Virginia 32977 - "us-west1" 32978 - Oregon 32979 - "us-west2" 32980 - California 32981 - "us-west3" 32982 - Salt Lake City 32983 - "us-west4" 32984 - Las Vegas 32985 - "northamerica-northeast1" 32986 - Montréal 32987 - "northamerica-northeast2" 32988 - Toronto 32989 - "southamerica-east1" 32990 - São Paulo 32991 - "southamerica-west1" 32992 - Santiago 32993 - "asia1" 32994 - Dual region: asia-northeast1 and asia-northeast2. 32995 - "eur4" 32996 - Dual region: europe-north1 and europe-west4. 32997 - "nam4" 32998 - Dual region: us-central1 and us-east1. 32999 33000 #### --gcs-storage-class 33001 33002 The storage class to use when storing objects in Google Cloud Storage. 33003 33004 Properties: 33005 33006 - Config: storage_class 33007 - Env Var: RCLONE_GCS_STORAGE_CLASS 33008 - Type: string 33009 - Required: false 33010 - Examples: 33011 - "" 33012 - Default 33013 - "MULTI_REGIONAL" 33014 - Multi-regional storage class 33015 - "REGIONAL" 33016 - Regional storage class 33017 - "NEARLINE" 33018 - Nearline storage class 33019 - "COLDLINE" 33020 - Coldline storage class 33021 - "ARCHIVE" 33022 - Archive storage class 33023 - "DURABLE_REDUCED_AVAILABILITY" 33024 - Durable reduced availability storage class 33025 33026 #### --gcs-env-auth 33027 33028 Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars). 33029 33030 Only applies if service_account_file and service_account_credentials is blank. 33031 33032 Properties: 33033 33034 - Config: env_auth 33035 - Env Var: RCLONE_GCS_ENV_AUTH 33036 - Type: bool 33037 - Default: false 33038 - Examples: 33039 - "false" 33040 - Enter credentials in the next step. 33041 - "true" 33042 - Get GCP IAM credentials from the environment (env vars or IAM). 33043 33044 ### Advanced options 33045 33046 Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)). 33047 33048 #### --gcs-token 33049 33050 OAuth Access Token as a JSON blob. 33051 33052 Properties: 33053 33054 - Config: token 33055 - Env Var: RCLONE_GCS_TOKEN 33056 - Type: string 33057 - Required: false 33058 33059 #### --gcs-auth-url 33060 33061 Auth server URL. 33062 33063 Leave blank to use the provider defaults. 33064 33065 Properties: 33066 33067 - Config: auth_url 33068 - Env Var: RCLONE_GCS_AUTH_URL 33069 - Type: string 33070 - Required: false 33071 33072 #### --gcs-token-url 33073 33074 Token server url. 33075 33076 Leave blank to use the provider defaults. 33077 33078 Properties: 33079 33080 - Config: token_url 33081 - Env Var: RCLONE_GCS_TOKEN_URL 33082 - Type: string 33083 - Required: false 33084 33085 #### --gcs-directory-markers 33086 33087 Upload an empty object with a trailing slash when a new directory is created 33088 33089 Empty folders are unsupported for bucket based remotes, this option creates an empty 33090 object ending with "/", to persist the folder. 33091 33092 33093 Properties: 33094 33095 - Config: directory_markers 33096 - Env Var: RCLONE_GCS_DIRECTORY_MARKERS 33097 - Type: bool 33098 - Default: false 33099 33100 #### --gcs-no-check-bucket 33101 33102 If set, don't attempt to check the bucket exists or create it. 33103 33104 This can be useful when trying to minimise the number of transactions 33105 rclone does if you know the bucket exists already. 33106 33107 33108 Properties: 33109 33110 - Config: no_check_bucket 33111 - Env Var: RCLONE_GCS_NO_CHECK_BUCKET 33112 - Type: bool 33113 - Default: false 33114 33115 #### --gcs-decompress 33116 33117 If set this will decompress gzip encoded objects. 33118 33119 It is possible to upload objects to GCS with "Content-Encoding: gzip" 33120 set. Normally rclone will download these files as compressed objects. 33121 33122 If this flag is set then rclone will decompress these files with 33123 "Content-Encoding: gzip" as they are received. This means that rclone 33124 can't check the size and hash but the file contents will be decompressed. 33125 33126 33127 Properties: 33128 33129 - Config: decompress 33130 - Env Var: RCLONE_GCS_DECOMPRESS 33131 - Type: bool 33132 - Default: false 33133 33134 #### --gcs-endpoint 33135 33136 Endpoint for the service. 33137 33138 Leave blank normally. 33139 33140 Properties: 33141 33142 - Config: endpoint 33143 - Env Var: RCLONE_GCS_ENDPOINT 33144 - Type: string 33145 - Required: false 33146 33147 #### --gcs-encoding 33148 33149 The encoding for the backend. 33150 33151 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 33152 33153 Properties: 33154 33155 - Config: encoding 33156 - Env Var: RCLONE_GCS_ENCODING 33157 - Type: Encoding 33158 - Default: Slash,CrLf,InvalidUtf8,Dot 33159 33160 #### --gcs-description 33161 33162 Description of the remote 33163 33164 Properties: 33165 33166 - Config: description 33167 - Env Var: RCLONE_GCS_DESCRIPTION 33168 - Type: string 33169 - Required: false 33170 33171 33172 33173 ## Limitations 33174 33175 `rclone about` is not supported by the Google Cloud Storage backend. Backends without 33176 this capability cannot determine free space for an rclone mount or 33177 use policy `mfs` (most free space) as a member of an rclone union 33178 remote. 33179 33180 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 33181 33182 # Google Drive 33183 33184 Paths are specified as `drive:path` 33185 33186 Drive paths may be as deep as required, e.g. `drive:directory/subdirectory`. 33187 33188 ## Configuration 33189 33190 The initial setup for drive involves getting a token from Google drive 33191 which you need to do in your browser. `rclone config` walks you 33192 through it. 33193 33194 Here is an example of how to make a remote called `remote`. First run: 33195 33196 rclone config 33197 33198 This will guide you through an interactive setup process: 33199 33200 No remotes found, make a new one? n) New remote r) Rename remote c) Copy 33201 remote s) Set configuration password q) Quit config n/r/c/s/q> n name> 33202 remote Type of storage to configure. Choose a number from below, or type 33203 in your own value [snip] XX / Google Drive "drive" [snip] Storage> 33204 drive Google Application Client Id - leave blank normally. client_id> 33205 Google Application Client Secret - leave blank normally. client_secret> 33206 Scope that rclone should use when requesting access from drive. Choose a 33207 number from below, or type in your own value 1 / Full access all files, 33208 excluding Application Data Folder. "drive" 2 / Read-only access to file 33209 metadata and file contents. "drive.readonly" / Access to files created 33210 by rclone only. 3 | These are visible in the drive website. | File 33211 authorization is revoked when the user deauthorizes the app. 33212 "drive.file" / Allows read and write access to the Application Data 33213 folder. 4 | This is not visible in the drive website. "drive.appfolder" 33214 / Allows read-only access to file metadata but 5 | does not allow any 33215 access to read or download file content. "drive.metadata.readonly" 33216 scope> 1 Service Account Credentials JSON file path - needed only if you 33217 want use SA instead of interactive login. service_account_file> Remote 33218 config Use web browser to automatically authenticate rclone with remote? 33219 * Say Y if the machine running rclone has a web browser you can use * 33220 Say N if running rclone on a (remote) machine without web browser access 33221 If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your 33222 browser doesn't open automatically go to the following link: 33223 http://127.0.0.1:53682/auth Log in and authorize rclone for access 33224 Waiting for code... Got code Configure this as a Shared Drive (Team 33225 Drive)? y) Yes n) No y/n> n -------------------- [remote] client_id = 33226 client_secret = scope = drive root_folder_id = service_account_file = 33227 token = 33228 {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} 33229 -------------------- y) Yes this is OK e) Edit this remote d) Delete 33230 this remote y/e/d> y 33231 33232 33233 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 33234 machine with no Internet browser available. 33235 33236 Note that rclone runs a webserver on your local machine to collect the 33237 token as returned from Google if using web browser to automatically 33238 authenticate. This only 33239 runs from the moment it opens your browser to the moment you get back 33240 the verification code. This is on `http://127.0.0.1:53682/` and it 33241 may require you to unblock it temporarily if you are running a host 33242 firewall, or use manual mode. 33243 33244 You can then use it like this, 33245 33246 List directories in top level of your drive 33247 33248 rclone lsd remote: 33249 33250 List all the files in your drive 33251 33252 rclone ls remote: 33253 33254 To copy a local directory to a drive directory called backup 33255 33256 rclone copy /home/source remote:backup 33257 33258 ### Scopes 33259 33260 Rclone allows you to select which scope you would like for rclone to 33261 use. This changes what type of token is granted to rclone. [The 33262 scopes are defined 33263 here](https://developers.google.com/drive/v3/web/about-auth). 33264 33265 A comma-separated list is allowed e.g. `drive.readonly,drive.file`. 33266 33267 The scope are 33268 33269 #### drive 33270 33271 This is the default scope and allows full access to all files, except 33272 for the Application Data Folder (see below). 33273 33274 Choose this one if you aren't sure. 33275 33276 #### drive.readonly 33277 33278 This allows read only access to all files. Files may be listed and 33279 downloaded but not uploaded, renamed or deleted. 33280 33281 #### drive.file 33282 33283 With this scope rclone can read/view/modify only those files and 33284 folders it creates. 33285 33286 So if you uploaded files to drive via the web interface (or any other 33287 means) they will not be visible to rclone. 33288 33289 This can be useful if you are using rclone to backup data and you want 33290 to be sure confidential data on your drive is not visible to rclone. 33291 33292 Files created with this scope are visible in the web interface. 33293 33294 #### drive.appfolder 33295 33296 This gives rclone its own private area to store files. Rclone will 33297 not be able to see any other files on your drive and you won't be able 33298 to see rclone's files from the web interface either. 33299 33300 #### drive.metadata.readonly 33301 33302 This allows read only access to file names only. It does not allow 33303 rclone to download or upload data, or rename or delete files or 33304 directories. 33305 33306 ### Root folder ID 33307 33308 This option has been moved to the advanced section. You can set the `root_folder_id` for rclone. This is the directory 33309 (identified by its `Folder ID`) that rclone considers to be the root 33310 of your drive. 33311 33312 Normally you will leave this blank and rclone will determine the 33313 correct root to use itself. 33314 33315 However you can set this to restrict rclone to a specific folder 33316 hierarchy or to access data within the "Computers" tab on the drive 33317 web interface (where files from Google's Backup and Sync desktop 33318 program go). 33319 33320 In order to do this you will have to find the `Folder ID` of the 33321 directory you wish rclone to display. This will be the last segment 33322 of the URL when you open the relevant folder in the drive web 33323 interface. 33324 33325 So if the folder you want rclone to use has a URL which looks like 33326 `https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` 33327 in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as 33328 the `root_folder_id` in the config. 33329 33330 **NB** folders under the "Computers" tab seem to be read only (drive 33331 gives a 500 error) when using rclone. 33332 33333 There doesn't appear to be an API to discover the folder IDs of the 33334 "Computers" tab - please contact us if you know otherwise! 33335 33336 Note also that rclone can't access any data under the "Backups" tab on 33337 the google drive web interface yet. 33338 33339 ### Service Account support 33340 33341 You can set up rclone with Google Drive in an unattended mode, 33342 i.e. not tied to a specific end-user Google account. This is useful 33343 when you want to synchronise files onto machines that don't have 33344 actively logged-in users, for example build machines. 33345 33346 To use a Service Account instead of OAuth2 token flow, enter the path 33347 to your Service Account credentials at the `service_account_file` 33348 prompt during `rclone config` and rclone won't use the browser based 33349 authentication flow. If you'd rather stuff the contents of the 33350 credentials file into the rclone config file, you can set 33351 `service_account_credentials` with the actual contents of the file 33352 instead, or set the equivalent environment variable. 33353 33354 #### Use case - Google Apps/G-suite account and individual Drive 33355 33356 Let's say that you are the administrator of a Google Apps (old) or 33357 G-suite account. 33358 The goal is to store data on an individual's Drive account, who IS 33359 a member of the domain. 33360 We'll call the domain **example.com**, and the user 33361 **foo@example.com**. 33362 33363 There's a few steps we need to go through to accomplish this: 33364 33365 ##### 1. Create a service account for example.com 33366 - To create a service account and obtain its credentials, go to the 33367 [Google Developer Console](https://console.developers.google.com). 33368 - You must have a project - create one if you don't. 33369 - Then go to "IAM & admin" -> "Service Accounts". 33370 - Use the "Create Service Account" button. Fill in "Service account name" 33371 and "Service account ID" with something that identifies your client. 33372 - Select "Create And Continue". Step 2 and 3 are optional. 33373 - These credentials are what rclone will use for authentication. 33374 If you ever need to remove access, press the "Delete service 33375 account key" button. 33376 33377 ##### 2. Allowing API access to example.com Google Drive 33378 - Go to example.com's admin console 33379 - Go into "Security" (or use the search bar) 33380 - Select "Show more" and then "Advanced settings" 33381 - Select "Manage API client access" in the "Authentication" section 33382 - In the "Client Name" field enter the service account's 33383 "Client ID" - this can be found in the Developer Console under 33384 "IAM & Admin" -> "Service Accounts", then "View Client ID" for 33385 the newly created service account. 33386 It is a ~21 character numerical string. 33387 - In the next field, "One or More API Scopes", enter 33388 `https://www.googleapis.com/auth/drive` 33389 to grant access to Google Drive specifically. 33390 33391 ##### 3. Configure rclone, assuming a new install 33392 33393 rclone config 33394 33395 n/s/q> n # New name>gdrive # Gdrive is an example name Storage> # Select 33396 the number shown for Google Drive client_id> # Can be left blank 33397 client_secret> # Can be left blank scope> # Select your scope, 1 for 33398 example root_folder_id> # Can be left blank service_account_file> 33399 /home/foo/myJSONfile.json # This is where the JSON file goes! y/n> # 33400 Auto config, n 33401 33402 33403 ##### 4. Verify that it's working 33404 - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup` 33405 - The arguments do: 33406 - `-v` - verbose logging 33407 - `--drive-impersonate foo@example.com` - this is what does 33408 the magic, pretending to be user foo. 33409 - `lsf` - list files in a parsing friendly way 33410 - `gdrive:backup` - use the remote called gdrive, work in 33411 the folder named backup. 33412 33413 Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead: 33414 - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1 33415 - use rclone without specifying the `--drive-impersonate` option, like this: 33416 `rclone -v lsf gdrive:backup` 33417 33418 33419 ### Shared drives (team drives) 33420 33421 If you want to configure the remote to point to a Google Shared Drive 33422 (previously known as Team Drives) then answer `y` to the question 33423 `Configure this as a Shared Drive (Team Drive)?`. 33424 33425 This will fetch the list of Shared Drives from google and allow you to 33426 configure which one you want to use. You can also type in a Shared 33427 Drive ID if you prefer. 33428 33429 For example: 33430 33431 Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> y 33432 Fetching Shared Drive list... Choose a number from below, or type in 33433 your own value 1 / Rclone Test "xxxxxxxxxxxxxxxxxxxx" 2 / Rclone Test 2 33434 "yyyyyyyyyyyyyyyyyyyy" 3 / Rclone Test 3 "zzzzzzzzzzzzzzzzzzzz" Enter 33435 a Shared Drive ID> 1 -------------------- [remote] client_id = 33436 client_secret = token = 33437 {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} 33438 team_drive = xxxxxxxxxxxxxxxxxxxx -------------------- y) Yes this is OK 33439 e) Edit this remote d) Delete this remote y/e/d> y 33440 33441 33442 ### --fast-list 33443 33444 This remote supports `--fast-list` which allows you to use fewer 33445 transactions in exchange for more memory. See the [rclone 33446 docs](https://rclone.org/docs/#fast-list) for more details. 33447 33448 It does this by combining multiple `list` calls into a single API request. 33449 33450 This works by combining many `'%s' in parents` filters into one expression. 33451 To list the contents of directories a, b and c, the following requests will be send by the regular `List` function: 33452 33453 trashed=false and 'a' in parents trashed=false and 'b' in parents 33454 trashed=false and 'c' in parents 33455 33456 These can now be combined into a single request: 33457 33458 trashed=false and ('a' in parents or 'b' in parents or 'c' in parents) 33459 33460 33461 The implementation of `ListR` will put up to 50 `parents` filters into one request. 33462 It will use the `--checkers` value to specify the number of requests to run in parallel. 33463 33464 In tests, these batch requests were up to 20x faster than the regular method. 33465 Running the following command against different sized folders gives: 33466 33467 rclone lsjson -vv -R --checkers=6 gdrive:folder 33468 33469 33470 small folder (220 directories, 700 files): 33471 33472 - without `--fast-list`: 38s 33473 - with `--fast-list`: 10s 33474 33475 large folder (10600 directories, 39000 files): 33476 33477 - without `--fast-list`: 22:05 min 33478 - with `--fast-list`: 58s 33479 33480 ### Modification times and hashes 33481 33482 Google drive stores modification times accurate to 1 ms. 33483 33484 Hash algorithms MD5, SHA1 and SHA256 are supported. Note, however, 33485 that a small fraction of files uploaded may not have SHA1 or SHA256 33486 hashes especially if they were uploaded before 2018. 33487 33488 ### Restricted filename characters 33489 33490 Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), 33491 as they can't be used in JSON strings. 33492 33493 In contrast to other backends, `/` can also be used in names and `.` 33494 or `..` are valid names. 33495 33496 ### Revisions 33497 33498 Google drive stores revisions of files. When you upload a change to 33499 an existing file to google drive using rclone it will create a new 33500 revision of that file. 33501 33502 Revisions follow the standard google policy which at time of writing 33503 was 33504 33505 * They are deleted after 30 days or 100 revisions (whatever comes first). 33506 * They do not count towards a user storage quota. 33507 33508 ### Deleting files 33509 33510 By default rclone will send all files to the trash when deleting 33511 files. If deleting them permanently is required then use the 33512 `--drive-use-trash=false` flag, or set the equivalent environment 33513 variable. 33514 33515 ### Shortcuts 33516 33517 In March 2020 Google introduced a new feature in Google Drive called 33518 [drive shortcuts](https://support.google.com/drive/answer/9700156) 33519 ([API](https://developers.google.com/drive/api/v3/shortcuts)). These 33520 will (by September 2020) [replace the ability for files or folders to 33521 be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models). 33522 33523 Shortcuts are files that link to other files on Google Drive somewhat 33524 like a symlink in unix, except they point to the underlying file data 33525 (e.g. the inode in unix terms) so they don't break if the source is 33526 renamed or moved about. 33527 33528 By default rclone treats these as follows. 33529 33530 For shortcuts pointing to files: 33531 33532 - When listing a file shortcut appears as the destination file. 33533 - When downloading the contents of the destination file is downloaded. 33534 - When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut. 33535 - When server-side moving (renaming) the shortcut is renamed, not the destination file. 33536 - When server-side copying the shortcut is copied, not the contents of the shortcut. (unless `--drive-copy-shortcut-content` is in use in which case the contents of the shortcut gets copied). 33537 - When deleting the shortcut is deleted not the linked file. 33538 - When setting the modification time, the modification time of the linked file will be set. 33539 33540 For shortcuts pointing to folders: 33541 33542 - When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders) 33543 - When downloading the contents of the linked folder and sub contents are downloaded 33544 - When uploading to a shortcut folder the file will be placed in the linked folder 33545 - When server-side moving (renaming) the shortcut is renamed, not the destination folder 33546 - When server-side copying the contents of the linked folder is copied, not the shortcut. 33547 - When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder. 33548 - **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted. 33549 33550 The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts. 33551 33552 Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag 33553 or the corresponding `skip_shortcuts` configuration setting. 33554 33555 ### Emptying trash 33556 33557 If you wish to empty your trash you can use the `rclone cleanup remote:` 33558 command which will permanently delete all your trashed files. This command 33559 does not take any path arguments. 33560 33561 Note that Google Drive takes some time (minutes to days) to empty the 33562 trash even though the command returns within a few seconds. No output 33563 is echoed, so there will be no confirmation even using -v or -vv. 33564 33565 ### Quota information 33566 33567 To view your current quota you can use the `rclone about remote:` 33568 command which will display your usage limit (quota), the usage in Google 33569 Drive, the size of all files in the Trash and the space used by other 33570 Google services such as Gmail. This command does not take any path 33571 arguments. 33572 33573 #### Import/Export of google documents 33574 33575 Google documents can be exported from and uploaded to Google Drive. 33576 33577 When rclone downloads a Google doc it chooses a format to download 33578 depending upon the `--drive-export-formats` setting. 33579 By default the export formats are `docx,xlsx,pptx,svg` which are a 33580 sensible default for an editable document. 33581 33582 When choosing a format, rclone runs down the list provided in order 33583 and chooses the first file format the doc can be exported as from the 33584 list. If the file can't be exported to a format on the formats list, 33585 then rclone will choose a format from the default list. 33586 33587 If you prefer an archive copy then you might use `--drive-export-formats 33588 pdf`, or if you prefer openoffice/libreoffice formats you might use 33589 `--drive-export-formats ods,odt,odp`. 33590 33591 Note that rclone adds the extension to the google doc, so if it is 33592 called `My Spreadsheet` on google docs, it will be exported as `My 33593 Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc. 33594 33595 When importing files into Google Drive, rclone will convert all 33596 files with an extension in `--drive-import-formats` to their 33597 associated document type. 33598 rclone will not convert any files by default, since the conversion 33599 is lossy process. 33600 33601 The conversion must result in a file with the same extension when 33602 the `--drive-export-formats` rules are applied to the uploaded document. 33603 33604 Here are some examples for allowed and prohibited conversions. 33605 33606 | export-formats | import-formats | Upload Ext | Document Ext | Allowed | 33607 | -------------- | -------------- | ---------- | ------------ | ------- | 33608 | odt | odt | odt | odt | Yes | 33609 | odt | docx,odt | odt | odt | Yes | 33610 | | docx | docx | docx | Yes | 33611 | | odt | odt | docx | No | 33612 | odt,docx | docx,odt | docx | odt | No | 33613 | docx,odt | docx,odt | docx | docx | Yes | 33614 | docx,odt | docx,odt | odt | docx | No | 33615 33616 This limitation can be disabled by specifying `--drive-allow-import-name-change`. 33617 When using this flag, rclone can convert multiple files types resulting 33618 in the same document type at once, e.g. with `--drive-import-formats docx,odt,txt`, 33619 all files having these extension would result in a document represented as a docx file. 33620 This brings the additional risk of overwriting a document, if multiple files 33621 have the same stem. Many rclone operations will not handle this name change 33622 in any way. They assume an equal name when copying files and might copy the 33623 file again or delete them when the name changes. 33624 33625 Here are the possible export extensions with their corresponding mime types. 33626 Most of these can also be used for importing, but there more that are not 33627 listed here. Some of these additional ones might only be available when 33628 the operating system provides the correct MIME type entries. 33629 33630 This list can be changed by Google Drive at any time and might not 33631 represent the currently available conversions. 33632 33633 | Extension | Mime Type | Description | 33634 | --------- |-----------| ------------| 33635 | bmp | image/bmp | Windows Bitmap format | 33636 | csv | text/csv | Standard CSV format for Spreadsheets | 33637 | doc | application/msword | Classic Word file | 33638 | docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document | 33639 | epub | application/epub+zip | E-book format | 33640 | html | text/html | An HTML Document | 33641 | jpg | image/jpeg | A JPEG Image File | 33642 | json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts | 33643 | odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation | 33644 | ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 33645 | ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet | 33646 | odt | application/vnd.oasis.opendocument.text | Openoffice Document | 33647 | pdf | application/pdf | Adobe PDF Format | 33648 | pjpeg | image/pjpeg | Progressive JPEG Image | 33649 | png | image/png | PNG Image Format| 33650 | pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint | 33651 | rtf | application/rtf | Rich Text Format | 33652 | svg | image/svg+xml | Scalable Vector Graphics Format | 33653 | tsv | text/tab-separated-values | Standard TSV format for spreadsheets | 33654 | txt | text/plain | Plain Text | 33655 | wmf | application/x-msmetafile | Windows Meta File | 33656 | xls | application/vnd.ms-excel | Classic Excel file | 33657 | xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet | 33658 | zip | application/zip | A ZIP file of HTML, Images CSS | 33659 33660 Google documents can also be exported as link files. These files will 33661 open a browser window for the Google Docs website of that document 33662 when opened. The link file extension has to be specified as a 33663 `--drive-export-formats` parameter. They will match all available 33664 Google Documents. 33665 33666 | Extension | Description | OS Support | 33667 | --------- | ----------- | ---------- | 33668 | desktop | freedesktop.org specified desktop entry | Linux | 33669 | link.html | An HTML Document with a redirect | All | 33670 | url | INI style link file | macOS, Windows | 33671 | webloc | macOS specific XML format | macOS | 33672 33673 33674 ### Standard options 33675 33676 Here are the Standard options specific to drive (Google Drive). 33677 33678 #### --drive-client-id 33679 33680 Google Application Client Id 33681 Setting your own is recommended. 33682 See https://rclone.org/drive/#making-your-own-client-id for how to create your own. 33683 If you leave this blank, it will use an internal key which is low performance. 33684 33685 Properties: 33686 33687 - Config: client_id 33688 - Env Var: RCLONE_DRIVE_CLIENT_ID 33689 - Type: string 33690 - Required: false 33691 33692 #### --drive-client-secret 33693 33694 OAuth Client Secret. 33695 33696 Leave blank normally. 33697 33698 Properties: 33699 33700 - Config: client_secret 33701 - Env Var: RCLONE_DRIVE_CLIENT_SECRET 33702 - Type: string 33703 - Required: false 33704 33705 #### --drive-scope 33706 33707 Comma separated list of scopes that rclone should use when requesting access from drive. 33708 33709 Properties: 33710 33711 - Config: scope 33712 - Env Var: RCLONE_DRIVE_SCOPE 33713 - Type: string 33714 - Required: false 33715 - Examples: 33716 - "drive" 33717 - Full access all files, excluding Application Data Folder. 33718 - "drive.readonly" 33719 - Read-only access to file metadata and file contents. 33720 - "drive.file" 33721 - Access to files created by rclone only. 33722 - These are visible in the drive website. 33723 - File authorization is revoked when the user deauthorizes the app. 33724 - "drive.appfolder" 33725 - Allows read and write access to the Application Data folder. 33726 - This is not visible in the drive website. 33727 - "drive.metadata.readonly" 33728 - Allows read-only access to file metadata but 33729 - does not allow any access to read or download file content. 33730 33731 #### --drive-service-account-file 33732 33733 Service Account Credentials JSON file path. 33734 33735 Leave blank normally. 33736 Needed only if you want use SA instead of interactive login. 33737 33738 Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`. 33739 33740 Properties: 33741 33742 - Config: service_account_file 33743 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE 33744 - Type: string 33745 - Required: false 33746 33747 #### --drive-alternate-export 33748 33749 Deprecated: No longer needed. 33750 33751 Properties: 33752 33753 - Config: alternate_export 33754 - Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT 33755 - Type: bool 33756 - Default: false 33757 33758 ### Advanced options 33759 33760 Here are the Advanced options specific to drive (Google Drive). 33761 33762 #### --drive-token 33763 33764 OAuth Access Token as a JSON blob. 33765 33766 Properties: 33767 33768 - Config: token 33769 - Env Var: RCLONE_DRIVE_TOKEN 33770 - Type: string 33771 - Required: false 33772 33773 #### --drive-auth-url 33774 33775 Auth server URL. 33776 33777 Leave blank to use the provider defaults. 33778 33779 Properties: 33780 33781 - Config: auth_url 33782 - Env Var: RCLONE_DRIVE_AUTH_URL 33783 - Type: string 33784 - Required: false 33785 33786 #### --drive-token-url 33787 33788 Token server url. 33789 33790 Leave blank to use the provider defaults. 33791 33792 Properties: 33793 33794 - Config: token_url 33795 - Env Var: RCLONE_DRIVE_TOKEN_URL 33796 - Type: string 33797 - Required: false 33798 33799 #### --drive-root-folder-id 33800 33801 ID of the root folder. 33802 Leave blank normally. 33803 33804 Fill in to access "Computers" folders (see docs), or for rclone to use 33805 a non root folder as its starting point. 33806 33807 33808 Properties: 33809 33810 - Config: root_folder_id 33811 - Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID 33812 - Type: string 33813 - Required: false 33814 33815 #### --drive-service-account-credentials 33816 33817 Service Account Credentials JSON blob. 33818 33819 Leave blank normally. 33820 Needed only if you want use SA instead of interactive login. 33821 33822 Properties: 33823 33824 - Config: service_account_credentials 33825 - Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS 33826 - Type: string 33827 - Required: false 33828 33829 #### --drive-team-drive 33830 33831 ID of the Shared Drive (Team Drive). 33832 33833 Properties: 33834 33835 - Config: team_drive 33836 - Env Var: RCLONE_DRIVE_TEAM_DRIVE 33837 - Type: string 33838 - Required: false 33839 33840 #### --drive-auth-owner-only 33841 33842 Only consider files owned by the authenticated user. 33843 33844 Properties: 33845 33846 - Config: auth_owner_only 33847 - Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY 33848 - Type: bool 33849 - Default: false 33850 33851 #### --drive-use-trash 33852 33853 Send files to the trash instead of deleting permanently. 33854 33855 Defaults to true, namely sending files to the trash. 33856 Use `--drive-use-trash=false` to delete files permanently instead. 33857 33858 Properties: 33859 33860 - Config: use_trash 33861 - Env Var: RCLONE_DRIVE_USE_TRASH 33862 - Type: bool 33863 - Default: true 33864 33865 #### --drive-copy-shortcut-content 33866 33867 Server side copy contents of shortcuts instead of the shortcut. 33868 33869 When doing server side copies, normally rclone will copy shortcuts as 33870 shortcuts. 33871 33872 If this flag is used then rclone will copy the contents of shortcuts 33873 rather than shortcuts themselves when doing server side copies. 33874 33875 Properties: 33876 33877 - Config: copy_shortcut_content 33878 - Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT 33879 - Type: bool 33880 - Default: false 33881 33882 #### --drive-skip-gdocs 33883 33884 Skip google documents in all listings. 33885 33886 If given, gdocs practically become invisible to rclone. 33887 33888 Properties: 33889 33890 - Config: skip_gdocs 33891 - Env Var: RCLONE_DRIVE_SKIP_GDOCS 33892 - Type: bool 33893 - Default: false 33894 33895 #### --drive-show-all-gdocs 33896 33897 Show all Google Docs including non-exportable ones in listings. 33898 33899 If you try a server side copy on a Google Form without this flag, you 33900 will get this error: 33901 33902 No export formats found for "application/vnd.google-apps.form" 33903 33904 However adding this flag will allow the form to be server side copied. 33905 33906 Note that rclone doesn't add extensions to the Google Docs file names 33907 in this mode. 33908 33909 Do **not** use this flag when trying to download Google Docs - rclone 33910 will fail to download them. 33911 33912 33913 Properties: 33914 33915 - Config: show_all_gdocs 33916 - Env Var: RCLONE_DRIVE_SHOW_ALL_GDOCS 33917 - Type: bool 33918 - Default: false 33919 33920 #### --drive-skip-checksum-gphotos 33921 33922 Skip checksums on Google photos and videos only. 33923 33924 Use this if you get checksum errors when transferring Google photos or 33925 videos. 33926 33927 Setting this flag will cause Google photos and videos to return a 33928 blank checksums. 33929 33930 Google photos are identified by being in the "photos" space. 33931 33932 Corrupted checksums are caused by Google modifying the image/video but 33933 not updating the checksum. 33934 33935 Properties: 33936 33937 - Config: skip_checksum_gphotos 33938 - Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS 33939 - Type: bool 33940 - Default: false 33941 33942 #### --drive-shared-with-me 33943 33944 Only show files that are shared with me. 33945 33946 Instructs rclone to operate on your "Shared with me" folder (where 33947 Google Drive lets you access the files and folders others have shared 33948 with you). 33949 33950 This works both with the "list" (lsd, lsl, etc.) and the "copy" 33951 commands (copy, sync, etc.), and with all other commands too. 33952 33953 Properties: 33954 33955 - Config: shared_with_me 33956 - Env Var: RCLONE_DRIVE_SHARED_WITH_ME 33957 - Type: bool 33958 - Default: false 33959 33960 #### --drive-trashed-only 33961 33962 Only show files that are in the trash. 33963 33964 This will show trashed files in their original directory structure. 33965 33966 Properties: 33967 33968 - Config: trashed_only 33969 - Env Var: RCLONE_DRIVE_TRASHED_ONLY 33970 - Type: bool 33971 - Default: false 33972 33973 #### --drive-starred-only 33974 33975 Only show files that are starred. 33976 33977 Properties: 33978 33979 - Config: starred_only 33980 - Env Var: RCLONE_DRIVE_STARRED_ONLY 33981 - Type: bool 33982 - Default: false 33983 33984 #### --drive-formats 33985 33986 Deprecated: See export_formats. 33987 33988 Properties: 33989 33990 - Config: formats 33991 - Env Var: RCLONE_DRIVE_FORMATS 33992 - Type: string 33993 - Required: false 33994 33995 #### --drive-export-formats 33996 33997 Comma separated list of preferred formats for downloading Google docs. 33998 33999 Properties: 34000 34001 - Config: export_formats 34002 - Env Var: RCLONE_DRIVE_EXPORT_FORMATS 34003 - Type: string 34004 - Default: "docx,xlsx,pptx,svg" 34005 34006 #### --drive-import-formats 34007 34008 Comma separated list of preferred formats for uploading Google docs. 34009 34010 Properties: 34011 34012 - Config: import_formats 34013 - Env Var: RCLONE_DRIVE_IMPORT_FORMATS 34014 - Type: string 34015 - Required: false 34016 34017 #### --drive-allow-import-name-change 34018 34019 Allow the filetype to change when uploading Google docs. 34020 34021 E.g. file.doc to file.docx. This will confuse sync and reupload every time. 34022 34023 Properties: 34024 34025 - Config: allow_import_name_change 34026 - Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE 34027 - Type: bool 34028 - Default: false 34029 34030 #### --drive-use-created-date 34031 34032 Use file created date instead of modified date. 34033 34034 Useful when downloading data and you want the creation date used in 34035 place of the last modified date. 34036 34037 **WARNING**: This flag may have some unexpected consequences. 34038 34039 When uploading to your drive all files will be overwritten unless they 34040 haven't been modified since their creation. And the inverse will occur 34041 while downloading. This side effect can be avoided by using the 34042 "--checksum" flag. 34043 34044 This feature was implemented to retain photos capture date as recorded 34045 by google photos. You will first need to check the "Create a Google 34046 Photos folder" option in your google drive settings. You can then copy 34047 or move the photos locally and use the date the image was taken 34048 (created) set as the modification date. 34049 34050 Properties: 34051 34052 - Config: use_created_date 34053 - Env Var: RCLONE_DRIVE_USE_CREATED_DATE 34054 - Type: bool 34055 - Default: false 34056 34057 #### --drive-use-shared-date 34058 34059 Use date file was shared instead of modified date. 34060 34061 Note that, as with "--drive-use-created-date", this flag may have 34062 unexpected consequences when uploading/downloading files. 34063 34064 If both this flag and "--drive-use-created-date" are set, the created 34065 date is used. 34066 34067 Properties: 34068 34069 - Config: use_shared_date 34070 - Env Var: RCLONE_DRIVE_USE_SHARED_DATE 34071 - Type: bool 34072 - Default: false 34073 34074 #### --drive-list-chunk 34075 34076 Size of listing chunk 100-1000, 0 to disable. 34077 34078 Properties: 34079 34080 - Config: list_chunk 34081 - Env Var: RCLONE_DRIVE_LIST_CHUNK 34082 - Type: int 34083 - Default: 1000 34084 34085 #### --drive-impersonate 34086 34087 Impersonate this user when using a service account. 34088 34089 Properties: 34090 34091 - Config: impersonate 34092 - Env Var: RCLONE_DRIVE_IMPERSONATE 34093 - Type: string 34094 - Required: false 34095 34096 #### --drive-upload-cutoff 34097 34098 Cutoff for switching to chunked upload. 34099 34100 Properties: 34101 34102 - Config: upload_cutoff 34103 - Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF 34104 - Type: SizeSuffix 34105 - Default: 8Mi 34106 34107 #### --drive-chunk-size 34108 34109 Upload chunk size. 34110 34111 Must a power of 2 >= 256k. 34112 34113 Making this larger will improve performance, but note that each chunk 34114 is buffered in memory one per transfer. 34115 34116 Reducing this will reduce memory usage but decrease performance. 34117 34118 Properties: 34119 34120 - Config: chunk_size 34121 - Env Var: RCLONE_DRIVE_CHUNK_SIZE 34122 - Type: SizeSuffix 34123 - Default: 8Mi 34124 34125 #### --drive-acknowledge-abuse 34126 34127 Set to allow files which return cannotDownloadAbusiveFile to be downloaded. 34128 34129 If downloading a file returns the error "This file has been identified 34130 as malware or spam and cannot be downloaded" with the error code 34131 "cannotDownloadAbusiveFile" then supply this flag to rclone to 34132 indicate you acknowledge the risks of downloading the file and rclone 34133 will download it anyway. 34134 34135 Note that if you are using service account it will need Manager 34136 permission (not Content Manager) to for this flag to work. If the SA 34137 does not have the right permission, Google will just ignore the flag. 34138 34139 Properties: 34140 34141 - Config: acknowledge_abuse 34142 - Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE 34143 - Type: bool 34144 - Default: false 34145 34146 #### --drive-keep-revision-forever 34147 34148 Keep new head revision of each file forever. 34149 34150 Properties: 34151 34152 - Config: keep_revision_forever 34153 - Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER 34154 - Type: bool 34155 - Default: false 34156 34157 #### --drive-size-as-quota 34158 34159 Show sizes as storage quota usage, not actual size. 34160 34161 Show the size of a file as the storage quota used. This is the 34162 current version plus any older versions that have been set to keep 34163 forever. 34164 34165 **WARNING**: This flag may have some unexpected consequences. 34166 34167 It is not recommended to set this flag in your config - the 34168 recommended usage is using the flag form --drive-size-as-quota when 34169 doing rclone ls/lsl/lsf/lsjson/etc only. 34170 34171 If you do use this flag for syncing (not recommended) then you will 34172 need to use --ignore size also. 34173 34174 Properties: 34175 34176 - Config: size_as_quota 34177 - Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA 34178 - Type: bool 34179 - Default: false 34180 34181 #### --drive-v2-download-min-size 34182 34183 If Object's are greater, use drive v2 API to download. 34184 34185 Properties: 34186 34187 - Config: v2_download_min_size 34188 - Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE 34189 - Type: SizeSuffix 34190 - Default: off 34191 34192 #### --drive-pacer-min-sleep 34193 34194 Minimum time to sleep between API calls. 34195 34196 Properties: 34197 34198 - Config: pacer_min_sleep 34199 - Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP 34200 - Type: Duration 34201 - Default: 100ms 34202 34203 #### --drive-pacer-burst 34204 34205 Number of API calls to allow without sleeping. 34206 34207 Properties: 34208 34209 - Config: pacer_burst 34210 - Env Var: RCLONE_DRIVE_PACER_BURST 34211 - Type: int 34212 - Default: 100 34213 34214 #### --drive-server-side-across-configs 34215 34216 Deprecated: use --server-side-across-configs instead. 34217 34218 Allow server-side operations (e.g. copy) to work across different drive configs. 34219 34220 This can be useful if you wish to do a server-side copy between two 34221 different Google drives. Note that this isn't enabled by default 34222 because it isn't easy to tell if it will work between any two 34223 configurations. 34224 34225 Properties: 34226 34227 - Config: server_side_across_configs 34228 - Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS 34229 - Type: bool 34230 - Default: false 34231 34232 #### --drive-disable-http2 34233 34234 Disable drive using http2. 34235 34236 There is currently an unsolved issue with the google drive backend and 34237 HTTP/2. HTTP/2 is therefore disabled by default for the drive backend 34238 but can be re-enabled here. When the issue is solved this flag will 34239 be removed. 34240 34241 See: https://github.com/rclone/rclone/issues/3631 34242 34243 34244 34245 Properties: 34246 34247 - Config: disable_http2 34248 - Env Var: RCLONE_DRIVE_DISABLE_HTTP2 34249 - Type: bool 34250 - Default: true 34251 34252 #### --drive-stop-on-upload-limit 34253 34254 Make upload limit errors be fatal. 34255 34256 At the time of writing it is only possible to upload 750 GiB of data to 34257 Google Drive a day (this is an undocumented limit). When this limit is 34258 reached Google Drive produces a slightly different error message. When 34259 this flag is set it causes these errors to be fatal. These will stop 34260 the in-progress sync. 34261 34262 Note that this detection is relying on error message strings which 34263 Google don't document so it may break in the future. 34264 34265 See: https://github.com/rclone/rclone/issues/3857 34266 34267 34268 Properties: 34269 34270 - Config: stop_on_upload_limit 34271 - Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT 34272 - Type: bool 34273 - Default: false 34274 34275 #### --drive-stop-on-download-limit 34276 34277 Make download limit errors be fatal. 34278 34279 At the time of writing it is only possible to download 10 TiB of data from 34280 Google Drive a day (this is an undocumented limit). When this limit is 34281 reached Google Drive produces a slightly different error message. When 34282 this flag is set it causes these errors to be fatal. These will stop 34283 the in-progress sync. 34284 34285 Note that this detection is relying on error message strings which 34286 Google don't document so it may break in the future. 34287 34288 34289 Properties: 34290 34291 - Config: stop_on_download_limit 34292 - Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT 34293 - Type: bool 34294 - Default: false 34295 34296 #### --drive-skip-shortcuts 34297 34298 If set skip shortcut files. 34299 34300 Normally rclone dereferences shortcut files making them appear as if 34301 they are the original file (see [the shortcuts section](#shortcuts)). 34302 If this flag is set then rclone will ignore shortcut files completely. 34303 34304 34305 Properties: 34306 34307 - Config: skip_shortcuts 34308 - Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS 34309 - Type: bool 34310 - Default: false 34311 34312 #### --drive-skip-dangling-shortcuts 34313 34314 If set skip dangling shortcut files. 34315 34316 If this is set then rclone will not show any dangling shortcuts in listings. 34317 34318 34319 Properties: 34320 34321 - Config: skip_dangling_shortcuts 34322 - Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS 34323 - Type: bool 34324 - Default: false 34325 34326 #### --drive-resource-key 34327 34328 Resource key for accessing a link-shared file. 34329 34330 If you need to access files shared with a link like this 34331 34332 https://drive.google.com/drive/folders/XXX?resourcekey=YYY&usp=sharing 34333 34334 Then you will need to use the first part "XXX" as the "root_folder_id" 34335 and the second part "YYY" as the "resource_key" otherwise you will get 34336 404 not found errors when trying to access the directory. 34337 34338 See: https://developers.google.com/drive/api/guides/resource-keys 34339 34340 This resource key requirement only applies to a subset of old files. 34341 34342 Note also that opening the folder once in the web interface (with the 34343 user you've authenticated rclone with) seems to be enough so that the 34344 resource key is not needed. 34345 34346 34347 Properties: 34348 34349 - Config: resource_key 34350 - Env Var: RCLONE_DRIVE_RESOURCE_KEY 34351 - Type: string 34352 - Required: false 34353 34354 #### --drive-fast-list-bug-fix 34355 34356 Work around a bug in Google Drive listing. 34357 34358 Normally rclone will work around a bug in Google Drive when using 34359 --fast-list (ListR) where the search "(A in parents) or (B in 34360 parents)" returns nothing sometimes. See #3114, #4289 and 34361 https://issuetracker.google.com/issues/149522397 34362 34363 Rclone detects this by finding no items in more than one directory 34364 when listing and retries them as lists of individual directories. 34365 34366 This means that if you have a lot of empty directories rclone will end 34367 up listing them all individually and this can take many more API 34368 calls. 34369 34370 This flag allows the work-around to be disabled. This is **not** 34371 recommended in normal use - only if you have a particular case you are 34372 having trouble with like many empty directories. 34373 34374 34375 Properties: 34376 34377 - Config: fast_list_bug_fix 34378 - Env Var: RCLONE_DRIVE_FAST_LIST_BUG_FIX 34379 - Type: bool 34380 - Default: true 34381 34382 #### --drive-metadata-owner 34383 34384 Control whether owner should be read or written in metadata. 34385 34386 Owner is a standard part of the file metadata so is easy to read. But it 34387 isn't always desirable to set the owner from the metadata. 34388 34389 Note that you can't set the owner on Shared Drives, and that setting 34390 ownership will generate an email to the new owner (this can't be 34391 disabled), and you can't transfer ownership to someone outside your 34392 organization. 34393 34394 34395 Properties: 34396 34397 - Config: metadata_owner 34398 - Env Var: RCLONE_DRIVE_METADATA_OWNER 34399 - Type: Bits 34400 - Default: read 34401 - Examples: 34402 - "off" 34403 - Do not read or write the value 34404 - "read" 34405 - Read the value only 34406 - "write" 34407 - Write the value only 34408 - "read,write" 34409 - Read and Write the value. 34410 34411 #### --drive-metadata-permissions 34412 34413 Control whether permissions should be read or written in metadata. 34414 34415 Reading permissions metadata from files can be done quickly, but it 34416 isn't always desirable to set the permissions from the metadata. 34417 34418 Note that rclone drops any inherited permissions on Shared Drives and 34419 any owner permission on My Drives as these are duplicated in the owner 34420 metadata. 34421 34422 34423 Properties: 34424 34425 - Config: metadata_permissions 34426 - Env Var: RCLONE_DRIVE_METADATA_PERMISSIONS 34427 - Type: Bits 34428 - Default: off 34429 - Examples: 34430 - "off" 34431 - Do not read or write the value 34432 - "read" 34433 - Read the value only 34434 - "write" 34435 - Write the value only 34436 - "read,write" 34437 - Read and Write the value. 34438 34439 #### --drive-metadata-labels 34440 34441 Control whether labels should be read or written in metadata. 34442 34443 Reading labels metadata from files takes an extra API transaction and 34444 will slow down listings. It isn't always desirable to set the labels 34445 from the metadata. 34446 34447 The format of labels is documented in the drive API documentation at 34448 https://developers.google.com/drive/api/reference/rest/v3/Label - 34449 rclone just provides a JSON dump of this format. 34450 34451 When setting labels, the label and fields must already exist - rclone 34452 will not create them. This means that if you are transferring labels 34453 from two different accounts you will have to create the labels in 34454 advance and use the metadata mapper to translate the IDs between the 34455 two accounts. 34456 34457 34458 Properties: 34459 34460 - Config: metadata_labels 34461 - Env Var: RCLONE_DRIVE_METADATA_LABELS 34462 - Type: Bits 34463 - Default: off 34464 - Examples: 34465 - "off" 34466 - Do not read or write the value 34467 - "read" 34468 - Read the value only 34469 - "write" 34470 - Write the value only 34471 - "read,write" 34472 - Read and Write the value. 34473 34474 #### --drive-encoding 34475 34476 The encoding for the backend. 34477 34478 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 34479 34480 Properties: 34481 34482 - Config: encoding 34483 - Env Var: RCLONE_DRIVE_ENCODING 34484 - Type: Encoding 34485 - Default: InvalidUtf8 34486 34487 #### --drive-env-auth 34488 34489 Get IAM credentials from runtime (environment variables or instance meta data if no env vars). 34490 34491 Only applies if service_account_file and service_account_credentials is blank. 34492 34493 Properties: 34494 34495 - Config: env_auth 34496 - Env Var: RCLONE_DRIVE_ENV_AUTH 34497 - Type: bool 34498 - Default: false 34499 - Examples: 34500 - "false" 34501 - Enter credentials in the next step. 34502 - "true" 34503 - Get GCP IAM credentials from the environment (env vars or IAM). 34504 34505 #### --drive-description 34506 34507 Description of the remote 34508 34509 Properties: 34510 34511 - Config: description 34512 - Env Var: RCLONE_DRIVE_DESCRIPTION 34513 - Type: string 34514 - Required: false 34515 34516 ### Metadata 34517 34518 User metadata is stored in the properties field of the drive object. 34519 34520 Metadata is supported on files and directories. 34521 34522 Here are the possible system metadata items for the drive backend. 34523 34524 | Name | Help | Type | Example | Read Only | 34525 |------|------|------|---------|-----------| 34526 | btime | Time of file birth (creation) with mS accuracy. Note that this is only writable on fresh uploads - it can't be written for updates. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N | 34527 | content-type | The MIME type of the file. | string | text/plain | N | 34528 | copy-requires-writer-permission | Whether the options to copy, print, or download this file, should be disabled for readers and commenters. | boolean | true | N | 34529 | description | A short description of the file. | string | Contract for signing | N | 34530 | folder-color-rgb | The color for a folder or a shortcut to a folder as an RGB hex string. | string | 881133 | N | 34531 | labels | Labels attached to this file in a JSON dump of Googled drive format. Enable with --drive-metadata-labels. | JSON | [] | N | 34532 | mtime | Time of last modification with mS accuracy. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N | 34533 | owner | The owner of the file. Usually an email address. Enable with --drive-metadata-owner. | string | user@example.com | N | 34534 | permissions | Permissions in a JSON dump of Google drive format. On shared drives these will only be present if they aren't inherited. Enable with --drive-metadata-permissions. | JSON | {} | N | 34535 | starred | Whether the user has starred the file. | boolean | false | N | 34536 | viewed-by-me | Whether the file has been viewed by this user. | boolean | true | **Y** | 34537 | writers-can-share | Whether users with only writer permission can modify the file's permissions. Not populated for items in shared drives. | boolean | false | N | 34538 34539 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 34540 34541 ## Backend commands 34542 34543 Here are the commands specific to the drive backend. 34544 34545 Run them with 34546 34547 rclone backend COMMAND remote: 34548 34549 The help below will explain what arguments each command takes. 34550 34551 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 34552 info on how to pass options and arguments. 34553 34554 These can be run on a running backend using the rc command 34555 [backend/command](https://rclone.org/rc/#backend-command). 34556 34557 ### get 34558 34559 Get command for fetching the drive config parameters 34560 34561 rclone backend get remote: [options] [<arguments>+] 34562 34563 This is a get command which will be used to fetch the various drive config parameters 34564 34565 Usage Examples: 34566 34567 rclone backend get drive: [-o service_account_file] [-o chunk_size] 34568 rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size] 34569 34570 34571 Options: 34572 34573 - "chunk_size": show the current upload chunk size 34574 - "service_account_file": show the current service account file 34575 34576 ### set 34577 34578 Set command for updating the drive config parameters 34579 34580 rclone backend set remote: [options] [<arguments>+] 34581 34582 This is a set command which will be used to update the various drive config parameters 34583 34584 Usage Examples: 34585 34586 rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864] 34587 rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864] 34588 34589 34590 Options: 34591 34592 - "chunk_size": update the current upload chunk size 34593 - "service_account_file": update the current service account file 34594 34595 ### shortcut 34596 34597 Create shortcuts from files or directories 34598 34599 rclone backend shortcut remote: [options] [<arguments>+] 34600 34601 This command creates shortcuts from files or directories. 34602 34603 Usage: 34604 34605 rclone backend shortcut drive: source_item destination_shortcut 34606 rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut 34607 34608 In the first example this creates a shortcut from the "source_item" 34609 which can be a file or a directory to the "destination_shortcut". The 34610 "source_item" and the "destination_shortcut" should be relative paths 34611 from "drive:" 34612 34613 In the second example this creates a shortcut from the "source_item" 34614 relative to "drive:" to the "destination_shortcut" relative to 34615 "drive2:". This may fail with a permission error if the user 34616 authenticated with "drive2:" can't read files from "drive:". 34617 34618 34619 Options: 34620 34621 - "target": optional target remote for the shortcut destination 34622 34623 ### drives 34624 34625 List the Shared Drives available to this account 34626 34627 rclone backend drives remote: [options] [<arguments>+] 34628 34629 This command lists the Shared Drives (Team Drives) available to this 34630 account. 34631 34632 Usage: 34633 34634 rclone backend [-o config] drives drive: 34635 34636 This will return a JSON list of objects like this 34637 34638 [ 34639 { 34640 "id": "0ABCDEF-01234567890", 34641 "kind": "drive#teamDrive", 34642 "name": "My Drive" 34643 }, 34644 { 34645 "id": "0ABCDEFabcdefghijkl", 34646 "kind": "drive#teamDrive", 34647 "name": "Test Drive" 34648 } 34649 ] 34650 34651 With the -o config parameter it will output the list in a format 34652 suitable for adding to a config file to make aliases for all the 34653 drives found and a combined drive. 34654 34655 [My Drive] 34656 type = alias 34657 remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=: 34658 34659 [Test Drive] 34660 type = alias 34661 remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=: 34662 34663 [AllDrives] 34664 type = combine 34665 upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:" 34666 34667 Adding this to the rclone config file will cause those team drives to 34668 be accessible with the aliases shown. Any illegal characters will be 34669 substituted with "_" and duplicate names will have numbers suffixed. 34670 It will also add a remote called AllDrives which shows all the shared 34671 drives combined into one directory tree. 34672 34673 34674 ### untrash 34675 34676 Untrash files and directories 34677 34678 rclone backend untrash remote: [options] [<arguments>+] 34679 34680 This command untrashes all the files and directories in the directory 34681 passed in recursively. 34682 34683 Usage: 34684 34685 This takes an optional directory to trash which make this easier to 34686 use via the API. 34687 34688 rclone backend untrash drive:directory 34689 rclone backend --interactive untrash drive:directory subdir 34690 34691 Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it. 34692 34693 Result: 34694 34695 { 34696 "Untrashed": 17, 34697 "Errors": 0 34698 } 34699 34700 34701 ### copyid 34702 34703 Copy files by ID 34704 34705 rclone backend copyid remote: [options] [<arguments>+] 34706 34707 This command copies files by ID 34708 34709 Usage: 34710 34711 rclone backend copyid drive: ID path 34712 rclone backend copyid drive: ID1 path1 ID2 path2 34713 34714 It copies the drive file with ID given to the path (an rclone path which 34715 will be passed internally to rclone copyto). The ID and path pairs can be 34716 repeated. 34717 34718 The path should end with a / to indicate copy the file as named to 34719 this directory. If it doesn't end with a / then the last path 34720 component will be used as the file name. 34721 34722 If the destination is a drive backend then server-side copying will be 34723 attempted if possible. 34724 34725 Use the --interactive/-i or --dry-run flag to see what would be copied before copying. 34726 34727 34728 ### exportformats 34729 34730 Dump the export formats for debug purposes 34731 34732 rclone backend exportformats remote: [options] [<arguments>+] 34733 34734 ### importformats 34735 34736 Dump the import formats for debug purposes 34737 34738 rclone backend importformats remote: [options] [<arguments>+] 34739 34740 34741 34742 ## Limitations 34743 34744 Drive has quite a lot of rate limiting. This causes rclone to be 34745 limited to transferring about 2 files per second only. Individual 34746 files may be transferred much faster at 100s of MiB/s but lots of 34747 small files can take a long time. 34748 34749 Server side copies are also subject to a separate rate limit. If you 34750 see User rate limit exceeded errors, wait at least 24 hours and retry. 34751 You can disable server-side copies with `--disable copy` to download 34752 and upload the files if you prefer. 34753 34754 ### Limitations of Google Docs 34755 34756 Google docs will appear as size -1 in `rclone ls`, `rclone ncdu` etc, 34757 and as size 0 in anything which uses the VFS layer, e.g. `rclone mount` 34758 and `rclone serve`. When calculating directory totals, e.g. in 34759 `rclone size` and `rclone ncdu`, they will be counted in as empty 34760 files. 34761 34762 This is because rclone can't find out the size of the Google docs 34763 without downloading them. 34764 34765 Google docs will transfer correctly with `rclone sync`, `rclone copy` 34766 etc as rclone knows to ignore the size when doing the transfer. 34767 34768 However an unfortunate consequence of this is that you may not be able 34769 to download Google docs using `rclone mount`. If it doesn't work you 34770 will get a 0 sized file. If you try again the doc may gain its 34771 correct size and be downloadable. Whether it will work on not depends 34772 on the application accessing the mount and the OS you are running - 34773 experiment to find out if it does work for you! 34774 34775 ### Duplicated files 34776 34777 Sometimes, for no reason I've been able to track down, drive will 34778 duplicate a file that rclone uploads. Drive unlike all the other 34779 remotes can have duplicated files. 34780 34781 Duplicated files cause problems with the syncing and you will see 34782 messages in the log about duplicates. 34783 34784 Use `rclone dedupe` to fix duplicated files. 34785 34786 Note that this isn't just a problem with rclone, even Google Photos on 34787 Android duplicates files on drive sometimes. 34788 34789 ### Rclone appears to be re-copying files it shouldn't 34790 34791 The most likely cause of this is the duplicated file issue above - run 34792 `rclone dedupe` and check your logs for duplicate object or directory 34793 messages. 34794 34795 This can also be caused by a delay/caching on google drive's end when 34796 comparing directory listings. Specifically with team drives used in 34797 combination with --fast-list. Files that were uploaded recently may 34798 not appear on the directory list sent to rclone when using --fast-list. 34799 34800 Waiting a moderate period of time between attempts (estimated to be 34801 approximately 1 hour) and/or not using --fast-list both seem to be 34802 effective in preventing the problem. 34803 34804 ### SHA1 or SHA256 hashes may be missing 34805 34806 All files have MD5 hashes, but a small fraction of files uploaded may 34807 not have SHA1 or SHA256 hashes especially if they were uploaded before 2018. 34808 34809 ## Making your own client_id 34810 34811 When you use rclone with Google drive in its default configuration you 34812 are using rclone's client_id. This is shared between all the rclone 34813 users. There is a global rate limit on the number of queries per 34814 second that each client_id can do set by Google. rclone already has a 34815 high quota and I will continue to make sure it is high enough by 34816 contacting Google. 34817 34818 It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower. 34819 34820 Here is how to create your own Google Drive client ID for rclone: 34821 34822 1. Log into the [Google API 34823 Console](https://console.developers.google.com/) with your Google 34824 account. It doesn't matter what Google account you use. (It need not 34825 be the same account as the Google Drive you want to access) 34826 34827 2. Select a project or create a new project. 34828 34829 3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the 34830 "Google Drive API". 34831 34832 4. Click "Credentials" in the left-side panel (not "Create 34833 credentials", which opens the wizard). 34834 34835 5. If you already configured an "Oauth Consent Screen", then skip 34836 to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button 34837 (near the top right corner of the right panel), then select "External" 34838 and click on "CREATE"; on the next screen, enter an "Application name" 34839 ("rclone" is OK); enter "User Support Email" (your own email is OK); 34840 enter "Developer Contact Email" (your own email is OK); then click on 34841 "Save" (all other data is optional). You will also have to add some scopes, 34842 including `.../auth/docs` and `.../auth/drive` in order to be able to edit, 34843 create and delete files with RClone. You may also want to include the 34844 `../auth/drive.metadata.readonly` scope. After adding scopes, click 34845 "Save and continue" to add test users. Be sure to add your own account to 34846 the test users. Once you've added yourself as a test user and saved the 34847 changes, click again on "Credentials" on the left panel to go back to 34848 the "Credentials" screen. 34849 34850 (PS: if you are a GSuite user, you could also select "Internal" instead 34851 of "External" above, but this will restrict API use to Google Workspace 34852 users in your organisation). 34853 34854 6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen, 34855 then select "OAuth client ID". 34856 34857 7. Choose an application type of "Desktop app" and click "Create". (the default name is fine) 34858 34859 8. It will show you a client ID and client secret. Make a note of these. 34860 34861 (If you selected "External" at Step 5 continue to Step 9. 34862 If you chose "Internal" you don't need to publish and can skip straight to 34863 Step 10 but your destination drive must be part of the same Google Workspace.) 34864 34865 9. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm. 34866 You will also want to add yourself as a test user. 34867 34868 10. Provide the noted client ID and client secret to rclone. 34869 34870 Be aware that, due to the "enhanced security" recently introduced by 34871 Google, you are theoretically expected to "submit your app for verification" 34872 and then wait a few weeks(!) for their response; in practice, you can go right 34873 ahead and use the client ID and client secret with rclone, the only issue will 34874 be a very scary confirmation screen shown when you connect via your browser 34875 for rclone to be able to get its token-id (but as this only happens during 34876 the remote configuration, it's not such a big deal). Keeping the application in 34877 "Testing" will work as well, but the limitation is that any grants will expire 34878 after a week, which can be annoying to refresh constantly. If, for whatever 34879 reason, a short grant time is not a problem, then keeping the application in 34880 testing mode would also be sufficient. 34881 34882 (Thanks to @balazer on github for these instructions.) 34883 34884 Sometimes, creation of an OAuth consent in Google API Console fails due to an error message 34885 “The request failed because changes to one of the field of the resource is not supported”. 34886 As a convenient workaround, the necessary Google Drive API key can be created on the 34887 [Python Quickstart](https://developers.google.com/drive/api/v3/quickstart/python) page. 34888 Just push the Enable the Drive API button to receive the Client ID and Secret. 34889 Note that it will automatically create a new project in the API Console. 34890 34891 # Google Photos 34892 34893 The rclone backend for [Google Photos](https://www.google.com/photos/about/) is 34894 a specialized backend for transferring photos and videos to and from 34895 Google Photos. 34896 34897 **NB** The Google Photos API which rclone uses has quite a few 34898 limitations, so please read the [limitations section](#limitations) 34899 carefully to make sure it is suitable for your use. 34900 34901 ## Configuration 34902 34903 The initial setup for google cloud storage involves getting a token from Google Photos 34904 which you need to do in your browser. `rclone config` walks you 34905 through it. 34906 34907 Here is an example of how to make a remote called `remote`. First run: 34908 34909 rclone config 34910 34911 This will guide you through an interactive setup process: 34912 34913 No remotes found, make a new one? n) New remote s) Set configuration 34914 password q) Quit config n/s/q> n name> remote Type of storage to 34915 configure. Enter a string value. Press Enter for the default (""). 34916 Choose a number from below, or type in your own value [snip] XX / Google 34917 Photos "google photos" [snip] Storage> google photos ** See help for 34918 google photos backend at: https://rclone.org/googlephotos/ ** 34919 34920 Google Application Client Id Leave blank normally. Enter a string value. 34921 Press Enter for the default (""). client_id> Google Application Client 34922 Secret Leave blank normally. Enter a string value. Press Enter for the 34923 default (""). client_secret> Set to make the Google Photos backend read 34924 only. 34925 34926 If you choose read only then rclone will only request read only access 34927 to your photos, otherwise rclone will request full access. Enter a 34928 boolean value (true or false). Press Enter for the default ("false"). 34929 read_only> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config 34930 Use web browser to automatically authenticate rclone with remote? * Say 34931 Y if the machine running rclone has a web browser you can use * Say N if 34932 running rclone on a (remote) machine without web browser access If not 34933 sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser 34934 doesn't open automatically go to the following link: 34935 http://127.0.0.1:53682/auth Log in and authorize rclone for access 34936 Waiting for code... Got code 34937 34938 *** IMPORTANT: All media items uploaded to Google Photos with rclone *** 34939 are stored in full resolution at original quality. These uploads *** 34940 will count towards storage in your Google Account. 34941 34942 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 34943 [remote] type = google photos token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"} 34944 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 34945 y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y ``` 34946 34947 See the remote setup docs for how to set it up on a machine with no Internet browser available. 34948 34949 Note that rclone runs a webserver on your local machine to collect the token as returned from Google if using web browser to automatically authenticate. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this may require you to unblock it temporarily if you are running a host firewall, or use manual mode. 34950 34951 This remote is called remote and can now be used like this 34952 34953 See all the albums in your photos 34954 34955 rclone lsd remote:album 34956 34957 Make a new album 34958 34959 rclone mkdir remote:album/newAlbum 34960 34961 List the contents of an album 34962 34963 rclone ls remote:album/newAlbum 34964 34965 Sync /home/local/images to the Google Photos, removing any excess files in the album. 34966 34967 rclone sync --interactive /home/local/image remote:album/newAlbum 34968 34969 ### Layout 34970 34971 As Google Photos is not a general purpose cloud storage system, the backend is laid out to help you navigate it. 34972 34973 The directories under media show different ways of categorizing the media. Each file will appear multiple times. So if you want to make a backup of your google photos you might choose to backup remote:media/by-month. (NB remote:media/by-day is rather slow at the moment so avoid for syncing.) 34974 34975 Note that all your photos and videos will appear somewhere under media, but they may not appear under album unless you've put them into albums. 34976 34977 / - upload - file1.jpg - file2.jpg - ... - media - all - file1.jpg - file2.jpg - ... - by-year - 2000 - file1.jpg - ... - 2001 - file2.jpg - ... - ... - by-month - 2000 - 2000-01 - file1.jpg - ... - 2000-02 - file2.jpg - ... - ... - by-day - 2000 - 2000-01-01 - file1.jpg - ... - 2000-01-02 - file2.jpg - ... - ... - album - album name - album name/sub - shared-album - album name - album name/sub - feature - favorites - file1.jpg - file2.jpg 34978 34979 There are two writable parts of the tree, the upload directory and sub directories of the album directory. 34980 34981 The upload directory is for uploading files you don't want to put into albums. This will be empty to start with and will contain the files you've uploaded for one rclone session only, becoming empty again when you restart rclone. The use case for this would be if you have a load of files you just want to once off dump into Google Photos. For repeated syncing, uploading to album will work better. 34982 34983 Directories within the album directory are also writeable and you may create new directories (albums) under album. If you copy files with a directory hierarchy in there then rclone will create albums with the / character in them. For example if you do 34984 34985 rclone copy /path/to/images remote:album/images 34986 34987 and the images directory contains 34988 34989 images - file1.jpg dir file2.jpg dir2 dir3 file3.jpg 34990 34991 Then rclone will create the following albums with the following files in 34992 34993 - images - file1.jpg - images/dir - file2.jpg - images/dir2/dir3 - file3.jpg 34994 34995 This means that you can use the album path pretty much like a normal filesystem and it is a good target for repeated syncing. 34996 34997 The shared-album directory shows albums shared with you or by you. This is similar to the Sharing tab in the Google Photos web interface. 34998 34999 ### Standard options 35000 35001 Here are the Standard options specific to google photos (Google Photos). 35002 35003 #### --gphotos-client-id 35004 35005 OAuth Client Id. 35006 35007 Leave blank normally. 35008 35009 Properties: 35010 35011 - Config: client_id - Env Var: RCLONE_GPHOTOS_CLIENT_ID - Type: string - Required: false 35012 35013 #### --gphotos-client-secret 35014 35015 OAuth Client Secret. 35016 35017 Leave blank normally. 35018 35019 Properties: 35020 35021 - Config: client_secret - Env Var: RCLONE_GPHOTOS_CLIENT_SECRET - Type: string - Required: false 35022 35023 #### --gphotos-read-only 35024 35025 Set to make the Google Photos backend read only. 35026 35027 If you choose read only then rclone will only request read only access to your photos, otherwise rclone will request full access. 35028 35029 Properties: 35030 35031 - Config: read_only - Env Var: RCLONE_GPHOTOS_READ_ONLY - Type: bool - Default: false 35032 35033 ### Advanced options 35034 35035 Here are the Advanced options specific to google photos (Google Photos). 35036 35037 #### --gphotos-token 35038 35039 OAuth Access Token as a JSON blob. 35040 35041 Properties: 35042 35043 - Config: token - Env Var: RCLONE_GPHOTOS_TOKEN - Type: string - Required: false 35044 35045 #### --gphotos-auth-url 35046 35047 Auth server URL. 35048 35049 Leave blank to use the provider defaults. 35050 35051 Properties: 35052 35053 - Config: auth_url - Env Var: RCLONE_GPHOTOS_AUTH_URL - Type: string - Required: false 35054 35055 #### --gphotos-token-url 35056 35057 Token server url. 35058 35059 Leave blank to use the provider defaults. 35060 35061 Properties: 35062 35063 - Config: token_url - Env Var: RCLONE_GPHOTOS_TOKEN_URL - Type: string - Required: false 35064 35065 #### --gphotos-read-size 35066 35067 Set to read the size of media items. 35068 35069 Normally rclone does not read the size of media items since this takes another transaction. This isn't necessary for syncing. However rclone mount needs to know the size of files in advance of reading them, so setting this flag when using rclone mount is recommended if you want to read the media. 35070 35071 Properties: 35072 35073 - Config: read_size - Env Var: RCLONE_GPHOTOS_READ_SIZE - Type: bool - Default: false 35074 35075 #### --gphotos-start-year 35076 35077 Year limits the photos to be downloaded to those which are uploaded after the given year. 35078 35079 Properties: 35080 35081 - Config: start_year - Env Var: RCLONE_GPHOTOS_START_YEAR - Type: int - Default: 2000 35082 35083 #### --gphotos-include-archived 35084 35085 Also view and download archived media. 35086 35087 By default, rclone does not request archived media. Thus, when syncing, archived media is not visible in directory listings or transferred. 35088 35089 Note that media in albums is always visible and synced, no matter their archive status. 35090 35091 With this flag, archived media are always visible in directory listings and transferred. 35092 35093 Without this flag, archived media will not be visible in directory listings and won't be transferred. 35094 35095 Properties: 35096 35097 - Config: include_archived - Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED - Type: bool - Default: false 35098 35099 #### --gphotos-encoding 35100 35101 The encoding for the backend. 35102 35103 See the encoding section in the overview for more info. 35104 35105 Properties: 35106 35107 - Config: encoding - Env Var: RCLONE_GPHOTOS_ENCODING - Type: Encoding - Default: Slash,CrLf,InvalidUtf8,Dot 35108 35109 #### --gphotos-batch-mode 35110 35111 Upload file batching sync|async|off. 35112 35113 This sets the batch mode used by rclone. 35114 35115 This has 3 possible values 35116 35117 - off - no batching - sync - batch uploads and check completion (default) - async - batch upload and don't check completion 35118 35119 Rclone will close any outstanding batches when it exits which may make a delay on quit. 35120 35121 Properties: 35122 35123 - Config: batch_mode - Env Var: RCLONE_GPHOTOS_BATCH_MODE - Type: string - Default: "sync" 35124 35125 #### --gphotos-batch-size 35126 35127 Max number of files in upload batch. 35128 35129 This sets the batch size of files to upload. It has to be less than 50. 35130 35131 By default this is 0 which means rclone which calculate the batch size depending on the setting of batch_mode. 35132 35133 - batch_mode: async - default batch_size is 50 - batch_mode: sync - default batch_size is the same as --transfers - batch_mode: off - not in use 35134 35135 Rclone will close any outstanding batches when it exits which may make a delay on quit. 35136 35137 Setting this is a great idea if you are uploading lots of small files as it will make them a lot quicker. You can use --transfers 32 to maximise throughput. 35138 35139 Properties: 35140 35141 - Config: batch_size - Env Var: RCLONE_GPHOTOS_BATCH_SIZE - Type: int - Default: 0 35142 35143 #### --gphotos-batch-timeout 35144 35145 Max time to allow an idle upload batch before uploading. 35146 35147 If an upload batch is idle for more than this long then it will be uploaded. 35148 35149 The default for this is 0 which means rclone will choose a sensible default based on the batch_mode in use. 35150 35151 - batch_mode: async - default batch_timeout is 10s - batch_mode: sync - default batch_timeout is 1s - batch_mode: off - not in use 35152 35153 Properties: 35154 35155 - Config: batch_timeout - Env Var: RCLONE_GPHOTOS_BATCH_TIMEOUT - Type: Duration - Default: 0s 35156 35157 #### --gphotos-batch-commit-timeout 35158 35159 Max time to wait for a batch to finish committing 35160 35161 Properties: 35162 35163 - Config: batch_commit_timeout - Env Var: RCLONE_GPHOTOS_BATCH_COMMIT_TIMEOUT - Type: Duration - Default: 10m0s 35164 35165 #### --gphotos-description 35166 35167 Description of the remote 35168 35169 Properties: 35170 35171 - Config: description - Env Var: RCLONE_GPHOTOS_DESCRIPTION - Type: string - Required: false 35172 35173 ## Limitations 35174 35175 Only images and videos can be uploaded. If you attempt to upload non videos or images or formats that Google Photos doesn't understand, rclone will upload the file, then Google Photos will give an error when it is put turned into a media item. 35176 35177 Note that all media items uploaded to Google Photos through the API are stored in full resolution at "original quality" and will count towards your storage quota in your Google Account. The API does not offer a way to upload in "high quality" mode.. 35178 35179 rclone about is not supported by the Google Photos backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote. 35180 35181 See List of backends that do not support rclone about See rclone about 35182 35183 ### Downloading Images 35184 35185 When Images are downloaded this strips EXIF location (according to the docs and my tests). This is a limitation of the Google Photos API and is covered by bug #112096115. 35186 35187 The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort 35188 35189 ### Downloading Videos 35190 35191 When videos are downloaded they are downloaded in a really compressed version of the video compared to downloading it via the Google Photos web interface. This is covered by bug #113672044. 35192 35193 ### Duplicates 35194 35195 If a file name is duplicated in a directory then rclone will add the file ID into its name. So two files called file.jpg would then appear as file {123456}.jpg and file {ABCDEF}.jpg (the actual IDs are a lot longer alas!). 35196 35197 If you upload the same image (with the same binary data) twice then Google Photos will deduplicate it. However it will retain the filename from the first upload which may confuse rclone. For example if you uploaded an image to upload then uploaded the same image to album/my_album the filename of the image in album/my_album will be what it was uploaded with initially, not what you uploaded it with to album. In practise this shouldn't cause 35198 too many problems. 35199 35200 ### Modification times 35201 35202 The date shown of media in Google Photos is the creation date as determined by the EXIF information, or the upload date if that is not known. 35203 35204 This is not changeable by rclone and is not the modification date of the media on local disk. This means that rclone cannot use the dates from Google Photos for syncing purposes. 35205 35206 ### Size 35207 35208 The Google Photos API does not return the size of media. This means that when syncing to Google Photos, rclone can only do a file existence check. 35209 35210 It is possible to read the size of the media, but this needs an extra HTTP HEAD request per media item so is very slow and uses up a lot of transactions. This can be enabled with the --gphotos-read-size option or the read_size = true config parameter. 35211 35212 If you want to use the backend with rclone mount you may need to enable this flag (depending on your OS and application using the photos) otherwise you may not be able to read media off the mount. You'll need to experiment to see if it works for you without the flag. 35213 35214 ### Albums 35215 35216 Rclone can only upload files to albums it created. This is a limitation of the Google Photos API. 35217 35218 Rclone can remove files it uploaded from albums it created only. 35219 35220 ### Deleting files 35221 35222 Rclone can remove files from albums it created, but note that the Google Photos API does not allow media to be deleted permanently so this media will still remain. See bug #109759781. 35223 35224 Rclone cannot delete files anywhere except under album. 35225 35226 ### Deleting albums 35227 35228 The Google Photos API does not support deleting albums - see bug #135714733. 35229 35230 # Hasher 35231 35232 Hasher is a special overlay backend to create remotes which handle checksums for other remotes. It's main functions include: - Emulate hash types unimplemented by backends - Cache checksums to help with slow hashing of large local or (S)FTP files - Warm up checksum cache from external SUM files 35233 35234 ## Getting started 35235 35236 To use Hasher, first set up the underlying remote following the configuration instructions for that remote. You can also use a local pathname instead of a remote. Check that your base remote is working. 35237 35238 Let's call the base remote myRemote:path here. Note that anything inside myRemote:path will be handled by hasher and anything outside won't. This means that if you are using a bucket based remote (S3, B2, Swift) then you should put the bucket in the remote s3:bucket. 35239 35240 Now proceed to interactive or manual configuration. 35241 35242 ### Interactive configuration 35243 35244 Run rclone config: ``` No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> Hasher1 Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Handle checksums for other remotes "hasher" [snip] Storage> hasher Remote to cache checksums for, like myremote:mypath. Enter a string value. Press Enter for the default (""). remote> myRemote:path Comma 35245 separated list of supported checksum types. Enter a string value. Press Enter for the default ("md5,sha1"). hashsums> md5 Maximum time to keep checksums in cache. 0 = no cache, off = cache forever. max_age> off Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config 35246 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 35247 35248 [Hasher1] type = hasher remote = myRemote:path hashsums = md5 max_age = 35249 off -------------------- y) Yes this is OK e) Edit this remote d) Delete 35250 this remote y/e/d> y 35251 35252 35253 ### Manual configuration 35254 35255 Run `rclone config path` to see the path of current active config file, 35256 usually `YOURHOME/.config/rclone/rclone.conf`. 35257 Open it in your favorite text editor, find section for the base remote 35258 and create new section for hasher like in the following examples: 35259 35260 [Hasher1] type = hasher remote = myRemote:path hashes = md5 max_age = 35261 off 35262 35263 [Hasher2] type = hasher remote = /local/path hashes = dropbox,sha1 35264 max_age = 24h 35265 35266 35267 Hasher takes basically the following parameters: 35268 - `remote` is required, 35269 - `hashes` is a comma separated list of supported checksums 35270 (by default `md5,sha1`), 35271 - `max_age` - maximum time to keep a checksum value in the cache, 35272 `0` will disable caching completely, 35273 `off` will cache "forever" (that is until the files get changed). 35274 35275 Make sure the `remote` has `:` (colon) in. If you specify the remote without 35276 a colon then rclone will use a local directory of that name. So if you use 35277 a remote of `/local/path` then rclone will handle hashes for that directory. 35278 If you use `remote = name` literally then rclone will put files 35279 **in a directory called `name` located under current directory**. 35280 35281 ## Usage 35282 35283 ### Basic operations 35284 35285 Now you can use it as `Hasher2:subdir/file` instead of base remote. 35286 Hasher will transparently update cache with new checksums when a file 35287 is fully read or overwritten, like: 35288 35289 rclone copy External:path/file Hasher:dest/path 35290 35291 rclone cat Hasher:path/to/file > /dev/null 35292 35293 35294 The way to refresh **all** cached checksums (even unsupported by the base backend) 35295 for a subtree is to **re-download** all files in the subtree. For example, 35296 use `hashsum --download` using **any** supported hashsum on the command line 35297 (we just care to re-read): 35298 35299 rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null 35300 35301 rclone backend dump Hasher:path/to/subtree 35302 35303 35304 You can print or drop hashsum cache using custom backend commands: 35305 35306 rclone backend dump Hasher:dir/subdir 35307 35308 rclone backend drop Hasher: 35309 35310 35311 ### Pre-Seed from a SUM File 35312 35313 Hasher supports two backend commands: generic SUM file `import` and faster 35314 but less consistent `stickyimport`. 35315 35316 rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM 35317 [--checkers 4] 35318 35319 35320 Instead of SHA1 it can be any hash supported by the remote. The last argument 35321 can point to either a local or an `other-remote:path` text file in SUM format. 35322 The command will parse the SUM file, then walk down the path given by the 35323 first argument, snapshot current fingerprints and fill in the cache entries 35324 correspondingly. 35325 - Paths in the SUM file are treated as relative to `hasher:dir/subdir`. 35326 - The command will **not** check that supplied values are correct. 35327 You **must know** what you are doing. 35328 - This is a one-time action. The SUM file will not get "attached" to the 35329 remote. Cache entries can still be overwritten later, should the object's 35330 fingerprint change. 35331 - The tree walk can take long depending on the tree size. You can increase 35332 `--checkers` to make it faster. Or use `stickyimport` if you don't care 35333 about fingerprints and consistency. 35334 35335 rclone backend stickyimport hasher:path/to/data sha1 35336 remote:/path/to/sum.sha1 35337 35338 35339 `stickyimport` is similar to `import` but works much faster because it 35340 does not need to stat existing files and skips initial tree walk. 35341 Instead of binding cache entries to file fingerprints it creates _sticky_ 35342 entries bound to the file name alone ignoring size, modification time etc. 35343 Such hash entries can be replaced only by `purge`, `delete`, `backend drop` 35344 or by full re-read/re-write of the files. 35345 35346 ## Configuration reference 35347 35348 35349 ### Standard options 35350 35351 Here are the Standard options specific to hasher (Better checksums for other remotes). 35352 35353 #### --hasher-remote 35354 35355 Remote to cache checksums for (e.g. myRemote:path). 35356 35357 Properties: 35358 35359 - Config: remote 35360 - Env Var: RCLONE_HASHER_REMOTE 35361 - Type: string 35362 - Required: true 35363 35364 #### --hasher-hashes 35365 35366 Comma separated list of supported checksum types. 35367 35368 Properties: 35369 35370 - Config: hashes 35371 - Env Var: RCLONE_HASHER_HASHES 35372 - Type: CommaSepList 35373 - Default: md5,sha1 35374 35375 #### --hasher-max-age 35376 35377 Maximum time to keep checksums in cache (0 = no cache, off = cache forever). 35378 35379 Properties: 35380 35381 - Config: max_age 35382 - Env Var: RCLONE_HASHER_MAX_AGE 35383 - Type: Duration 35384 - Default: off 35385 35386 ### Advanced options 35387 35388 Here are the Advanced options specific to hasher (Better checksums for other remotes). 35389 35390 #### --hasher-auto-size 35391 35392 Auto-update checksum for files smaller than this size (disabled by default). 35393 35394 Properties: 35395 35396 - Config: auto_size 35397 - Env Var: RCLONE_HASHER_AUTO_SIZE 35398 - Type: SizeSuffix 35399 - Default: 0 35400 35401 #### --hasher-description 35402 35403 Description of the remote 35404 35405 Properties: 35406 35407 - Config: description 35408 - Env Var: RCLONE_HASHER_DESCRIPTION 35409 - Type: string 35410 - Required: false 35411 35412 ### Metadata 35413 35414 Any metadata supported by the underlying remote is read and written. 35415 35416 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 35417 35418 ## Backend commands 35419 35420 Here are the commands specific to the hasher backend. 35421 35422 Run them with 35423 35424 rclone backend COMMAND remote: 35425 35426 The help below will explain what arguments each command takes. 35427 35428 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 35429 info on how to pass options and arguments. 35430 35431 These can be run on a running backend using the rc command 35432 [backend/command](https://rclone.org/rc/#backend-command). 35433 35434 ### drop 35435 35436 Drop cache 35437 35438 rclone backend drop remote: [options] [<arguments>+] 35439 35440 Completely drop checksum cache. 35441 Usage Example: 35442 rclone backend drop hasher: 35443 35444 35445 ### dump 35446 35447 Dump the database 35448 35449 rclone backend dump remote: [options] [<arguments>+] 35450 35451 Dump cache records covered by the current remote 35452 35453 ### fulldump 35454 35455 Full dump of the database 35456 35457 rclone backend fulldump remote: [options] [<arguments>+] 35458 35459 Dump all cache records in the database 35460 35461 ### import 35462 35463 Import a SUM file 35464 35465 rclone backend import remote: [options] [<arguments>+] 35466 35467 Amend hash cache from a SUM file and bind checksums to files by size/time. 35468 Usage Example: 35469 rclone backend import hasher:subdir md5 /path/to/sum.md5 35470 35471 35472 ### stickyimport 35473 35474 Perform fast import of a SUM file 35475 35476 rclone backend stickyimport remote: [options] [<arguments>+] 35477 35478 Fill hash cache from a SUM file without verifying file fingerprints. 35479 Usage Example: 35480 rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5 35481 35482 35483 35484 35485 ## Implementation details (advanced) 35486 35487 This section explains how various rclone operations work on a hasher remote. 35488 35489 **Disclaimer. This section describes current implementation which can 35490 change in future rclone versions!.** 35491 35492 ### Hashsum command 35493 35494 The `rclone hashsum` (or `md5sum` or `sha1sum`) command will: 35495 35496 1. if requested hash is supported by lower level, just pass it. 35497 2. if object size is below `auto_size` then download object and calculate 35498 _requested_ hashes on the fly. 35499 3. if unsupported and the size is big enough, build object `fingerprint` 35500 (including size, modtime if supported, first-found _other_ hash if any). 35501 4. if the strict match is found in cache for the requested remote, return 35502 the stored hash. 35503 5. if remote found but fingerprint mismatched, then purge the entry and 35504 proceed to step 6. 35505 6. if remote not found or had no requested hash type or after step 5: 35506 download object, calculate all _supported_ hashes on the fly and store 35507 in cache; return requested hash. 35508 35509 ### Other operations 35510 35511 - whenever a file is uploaded or downloaded **in full**, capture the stream 35512 to calculate all supported hashes on the fly and update database 35513 - server-side `move` will update keys of existing cache entries 35514 - `deletefile` will remove a single cache entry 35515 - `purge` will remove all cache entries under the purged path 35516 35517 Note that setting `max_age = 0` will disable checksum caching completely. 35518 35519 If you set `max_age = off`, checksums in cache will never age, unless you 35520 fully rewrite or delete the file. 35521 35522 ### Cache storage 35523 35524 Cached checksums are stored as `bolt` database files under rclone cache 35525 directory, usually `~/.cache/rclone/kv/`. Databases are maintained 35526 one per _base_ backend, named like `BaseRemote~hasher.bolt`. 35527 Checksums for multiple `alias`-es into a single base backend 35528 will be stored in the single database. All local paths are treated as 35529 aliases into the `local` backend (unless encrypted or chunked) and stored 35530 in `~/.cache/rclone/kv/local~hasher.bolt`. 35531 Databases can be shared between multiple rclone processes. 35532 35533 # HDFS 35534 35535 [HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) is a 35536 distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/) framework. 35537 35538 Paths are specified as `remote:` or `remote:path/to/dir`. 35539 35540 ## Configuration 35541 35542 Here is an example of how to make a remote called `remote`. First run: 35543 35544 rclone config 35545 35546 This will guide you through an interactive setup process: 35547 35548 No remotes found, make a new one? n) New remote s) Set configuration 35549 password q) Quit config n/s/q> n name> remote Type of storage to 35550 configure. Enter a string value. Press Enter for the default (""). 35551 Choose a number from below, or type in your own value [skip] XX / Hadoop 35552 distributed file system "hdfs" [skip] Storage> hdfs ** See help for 35553 hdfs backend at: https://rclone.org/hdfs/ ** 35554 35555 hadoop name node and port Enter a string value. Press Enter for the 35556 default (""). Choose a number from below, or type in your own value 1 / 35557 Connect to host namenode at port 8020 "namenode:8020" namenode> 35558 namenode.hadoop:8020 hadoop user name Enter a string value. Press Enter 35559 for the default (""). Choose a number from below, or type in your own 35560 value 1 / Connect to hdfs as root "root" username> root Edit advanced 35561 config? (y/n) y) Yes n) No (default) y/n> n Remote config 35562 -------------------- [remote] type = hdfs namenode = 35563 namenode.hadoop:8020 username = root -------------------- y) Yes this is 35564 OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current 35565 remotes: 35566 35567 Name Type ==== ==== hadoop hdfs 35568 35569 e) Edit existing remote 35570 f) New remote 35571 g) Delete remote 35572 h) Rename remote 35573 i) Copy remote 35574 j) Set configuration password 35575 k) Quit config e/n/d/r/c/s/q> q 35576 35577 35578 This remote is called `remote` and can now be used like this 35579 35580 See all the top level directories 35581 35582 rclone lsd remote: 35583 35584 List the contents of a directory 35585 35586 rclone ls remote:directory 35587 35588 Sync the remote `directory` to `/home/local/directory`, deleting any excess files. 35589 35590 rclone sync --interactive remote:directory /home/local/directory 35591 35592 ### Setting up your own HDFS instance for testing 35593 35594 You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) 35595 or use the docker image from the tests: 35596 35597 If you want to build the docker image 35598 35599 git clone https://github.com/rclone/rclone.git cd 35600 rclone/fstest/testserver/images/test-hdfs docker build --rm -t 35601 rclone/test-hdfs . 35602 35603 35604 Or you can just use the latest one pushed 35605 35606 docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 35607 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs 35608 35609 35610 **NB** it need few seconds to startup. 35611 35612 For this docker image the remote needs to be configured like this: 35613 35614 [remote] type = hdfs namenode = 127.0.0.1:8020 username = root 35615 35616 35617 You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use volumes, so all data 35618 uploaded will be lost.) 35619 35620 ### Modification times 35621 35622 Time accurate to 1 second is stored. 35623 35624 ### Checksum 35625 35626 No checksums are implemented. 35627 35628 ### Usage information 35629 35630 You can use the `rclone about remote:` command which will display filesystem size and current usage. 35631 35632 ### Restricted filename characters 35633 35634 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 35635 the following characters are also replaced: 35636 35637 | Character | Value | Replacement | 35638 | --------- |:-----:|:-----------:| 35639 | : | 0x3A | : | 35640 35641 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8). 35642 35643 35644 ### Standard options 35645 35646 Here are the Standard options specific to hdfs (Hadoop distributed file system). 35647 35648 #### --hdfs-namenode 35649 35650 Hadoop name nodes and ports. 35651 35652 E.g. "namenode-1:8020,namenode-2:8020,..." to connect to host namenodes at port 8020. 35653 35654 Properties: 35655 35656 - Config: namenode 35657 - Env Var: RCLONE_HDFS_NAMENODE 35658 - Type: CommaSepList 35659 - Default: 35660 35661 #### --hdfs-username 35662 35663 Hadoop user name. 35664 35665 Properties: 35666 35667 - Config: username 35668 - Env Var: RCLONE_HDFS_USERNAME 35669 - Type: string 35670 - Required: false 35671 - Examples: 35672 - "root" 35673 - Connect to hdfs as root. 35674 35675 ### Advanced options 35676 35677 Here are the Advanced options specific to hdfs (Hadoop distributed file system). 35678 35679 #### --hdfs-service-principal-name 35680 35681 Kerberos service principal name for the namenode. 35682 35683 Enables KERBEROS authentication. Specifies the Service Principal Name 35684 (SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\" 35685 for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'. 35686 35687 Properties: 35688 35689 - Config: service_principal_name 35690 - Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME 35691 - Type: string 35692 - Required: false 35693 35694 #### --hdfs-data-transfer-protection 35695 35696 Kerberos data transfer protection: authentication|integrity|privacy. 35697 35698 Specifies whether or not authentication, data signature integrity 35699 checks, and wire encryption are required when communicating with 35700 the datanodes. Possible values are 'authentication', 'integrity' 35701 and 'privacy'. Used only with KERBEROS enabled. 35702 35703 Properties: 35704 35705 - Config: data_transfer_protection 35706 - Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION 35707 - Type: string 35708 - Required: false 35709 - Examples: 35710 - "privacy" 35711 - Ensure authentication, integrity and encryption enabled. 35712 35713 #### --hdfs-encoding 35714 35715 The encoding for the backend. 35716 35717 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 35718 35719 Properties: 35720 35721 - Config: encoding 35722 - Env Var: RCLONE_HDFS_ENCODING 35723 - Type: Encoding 35724 - Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot 35725 35726 #### --hdfs-description 35727 35728 Description of the remote 35729 35730 Properties: 35731 35732 - Config: description 35733 - Env Var: RCLONE_HDFS_DESCRIPTION 35734 - Type: string 35735 - Required: false 35736 35737 35738 35739 ## Limitations 35740 35741 - No server-side `Move` or `DirMove`. 35742 - Checksums not implemented. 35743 35744 # HiDrive 35745 35746 Paths are specified as `remote:path` 35747 35748 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 35749 35750 The initial setup for hidrive involves getting a token from HiDrive 35751 which you need to do in your browser. 35752 `rclone config` walks you through it. 35753 35754 ## Configuration 35755 35756 Here is an example of how to make a remote called `remote`. First run: 35757 35758 rclone config 35759 35760 This will guide you through an interactive setup process: 35761 35762 No remotes found - make a new one n) New remote s) Set configuration 35763 password q) Quit config n/s/q> n name> remote Type of storage to 35764 configure. Choose a number from below, or type in your own value [snip] 35765 XX / HiDrive "hidrive" [snip] Storage> hidrive OAuth Client Id - Leave 35766 blank normally. client_id> OAuth Client Secret - Leave blank normally. 35767 client_secret> Access permissions that rclone should use when requesting 35768 access from HiDrive. Leave blank normally. scope_access> Edit advanced 35769 config? y/n> n Use web browser to automatically authenticate rclone with 35770 remote? * Say Y if the machine running rclone has a web browser you can 35771 use * Say N if running rclone on a (remote) machine without web browser 35772 access If not sure try Y. If Y failed, try N. y/n> y If your browser 35773 doesn't open automatically go to the following link: 35774 http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxx Log in and 35775 authorize rclone for access Waiting for code... Got code 35776 -------------------- [remote] type = hidrive token = 35777 {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} 35778 -------------------- y) Yes this is OK (default) e) Edit this remote d) 35779 Delete this remote y/e/d> y 35780 35781 35782 **You should be aware that OAuth-tokens can be used to access your account 35783 and hence should not be shared with other persons.** 35784 See the [below section](#keeping-your-tokens-safe) for more information. 35785 35786 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 35787 machine with no Internet browser available. 35788 35789 Note that rclone runs a webserver on your local machine to collect the 35790 token as returned from HiDrive. This only runs from the moment it opens 35791 your browser to the moment you get back the verification code. 35792 The webserver runs on `http://127.0.0.1:53682/`. 35793 If local port `53682` is protected by a firewall you may need to temporarily 35794 unblock the firewall to complete authorization. 35795 35796 Once configured you can then use `rclone` like this, 35797 35798 List directories in top level of your HiDrive root folder 35799 35800 rclone lsd remote: 35801 35802 List all the files in your HiDrive filesystem 35803 35804 rclone ls remote: 35805 35806 To copy a local directory to a HiDrive directory called backup 35807 35808 rclone copy /home/source remote:backup 35809 35810 ### Keeping your tokens safe 35811 35812 Any OAuth-tokens will be stored by rclone in the remote's configuration file as unencrypted text. 35813 Anyone can use a valid refresh-token to access your HiDrive filesystem without knowing your password. 35814 Therefore you should make sure no one else can access your configuration. 35815 35816 It is possible to encrypt rclone's configuration file. 35817 You can find information on securing your configuration file by viewing the [configuration encryption docs](https://rclone.org/docs/#configuration-encryption). 35818 35819 ### Invalid refresh token 35820 35821 As can be verified [here](https://developer.hidrive.com/basics-flows/), 35822 each `refresh_token` (for Native Applications) is valid for 60 days. 35823 If used to access HiDrivei, its validity will be automatically extended. 35824 35825 This means that if you 35826 35827 * Don't use the HiDrive remote for 60 days 35828 35829 then rclone will return an error which includes a text 35830 that implies the refresh token is *invalid* or *expired*. 35831 35832 To fix this you will need to authorize rclone to access your HiDrive account again. 35833 35834 Using 35835 35836 rclone config reconnect remote: 35837 35838 the process is very similar to the process of initial setup exemplified before. 35839 35840 ### Modification times and hashes 35841 35842 HiDrive allows modification times to be set on objects accurate to 1 second. 35843 35844 HiDrive supports [its own hash type](https://static.hidrive.com/dev/0001) 35845 which is used to verify the integrity of file contents after successful transfers. 35846 35847 ### Restricted filename characters 35848 35849 HiDrive cannot store files or folders that include 35850 `/` (0x2F) or null-bytes (0x00) in their name. 35851 Any other characters can be used in the names of files or folders. 35852 Additionally, files or folders cannot be named either of the following: `.` or `..` 35853 35854 Therefore rclone will automatically replace these characters, 35855 if files or folders are stored or accessed with such names. 35856 35857 You can read about how this filename encoding works in general 35858 [here](overview/#restricted-filenames). 35859 35860 Keep in mind that HiDrive only supports file or folder names 35861 with a length of 255 characters or less. 35862 35863 ### Transfers 35864 35865 HiDrive limits file sizes per single request to a maximum of 2 GiB. 35866 To allow storage of larger files and allow for better upload performance, 35867 the hidrive backend will use a chunked transfer for files larger than 96 MiB. 35868 Rclone will upload multiple parts/chunks of the file at the same time. 35869 Chunks in the process of being uploaded are buffered in memory, 35870 so you may want to restrict this behaviour on systems with limited resources. 35871 35872 You can customize this behaviour using the following options: 35873 35874 * `chunk_size`: size of file parts 35875 * `upload_cutoff`: files larger or equal to this in size will use a chunked transfer 35876 * `upload_concurrency`: number of file-parts to upload at the same time 35877 35878 See the below section about configuration options for more details. 35879 35880 ### Root folder 35881 35882 You can set the root folder for rclone. 35883 This is the directory that rclone considers to be the root of your HiDrive. 35884 35885 Usually, you will leave this blank, and rclone will use the root of the account. 35886 35887 However, you can set this to restrict rclone to a specific folder hierarchy. 35888 35889 This works by prepending the contents of the `root_prefix` option 35890 to any paths accessed by rclone. 35891 For example, the following two ways to access the home directory are equivalent: 35892 35893 rclone lsd --hidrive-root-prefix="/users/test/" remote:path 35894 35895 rclone lsd remote:/users/test/path 35896 35897 See the below section about configuration options for more details. 35898 35899 ### Directory member count 35900 35901 By default, rclone will know the number of directory members contained in a directory. 35902 For example, `rclone lsd` uses this information. 35903 35904 The acquisition of this information will result in additional time costs for HiDrive's API. 35905 When dealing with large directory structures, it may be desirable to circumvent this time cost, 35906 especially when this information is not explicitly needed. 35907 For this, the `disable_fetching_member_count` option can be used. 35908 35909 See the below section about configuration options for more details. 35910 35911 35912 ### Standard options 35913 35914 Here are the Standard options specific to hidrive (HiDrive). 35915 35916 #### --hidrive-client-id 35917 35918 OAuth Client Id. 35919 35920 Leave blank normally. 35921 35922 Properties: 35923 35924 - Config: client_id 35925 - Env Var: RCLONE_HIDRIVE_CLIENT_ID 35926 - Type: string 35927 - Required: false 35928 35929 #### --hidrive-client-secret 35930 35931 OAuth Client Secret. 35932 35933 Leave blank normally. 35934 35935 Properties: 35936 35937 - Config: client_secret 35938 - Env Var: RCLONE_HIDRIVE_CLIENT_SECRET 35939 - Type: string 35940 - Required: false 35941 35942 #### --hidrive-scope-access 35943 35944 Access permissions that rclone should use when requesting access from HiDrive. 35945 35946 Properties: 35947 35948 - Config: scope_access 35949 - Env Var: RCLONE_HIDRIVE_SCOPE_ACCESS 35950 - Type: string 35951 - Default: "rw" 35952 - Examples: 35953 - "rw" 35954 - Read and write access to resources. 35955 - "ro" 35956 - Read-only access to resources. 35957 35958 ### Advanced options 35959 35960 Here are the Advanced options specific to hidrive (HiDrive). 35961 35962 #### --hidrive-token 35963 35964 OAuth Access Token as a JSON blob. 35965 35966 Properties: 35967 35968 - Config: token 35969 - Env Var: RCLONE_HIDRIVE_TOKEN 35970 - Type: string 35971 - Required: false 35972 35973 #### --hidrive-auth-url 35974 35975 Auth server URL. 35976 35977 Leave blank to use the provider defaults. 35978 35979 Properties: 35980 35981 - Config: auth_url 35982 - Env Var: RCLONE_HIDRIVE_AUTH_URL 35983 - Type: string 35984 - Required: false 35985 35986 #### --hidrive-token-url 35987 35988 Token server url. 35989 35990 Leave blank to use the provider defaults. 35991 35992 Properties: 35993 35994 - Config: token_url 35995 - Env Var: RCLONE_HIDRIVE_TOKEN_URL 35996 - Type: string 35997 - Required: false 35998 35999 #### --hidrive-scope-role 36000 36001 User-level that rclone should use when requesting access from HiDrive. 36002 36003 Properties: 36004 36005 - Config: scope_role 36006 - Env Var: RCLONE_HIDRIVE_SCOPE_ROLE 36007 - Type: string 36008 - Default: "user" 36009 - Examples: 36010 - "user" 36011 - User-level access to management permissions. 36012 - This will be sufficient in most cases. 36013 - "admin" 36014 - Extensive access to management permissions. 36015 - "owner" 36016 - Full access to management permissions. 36017 36018 #### --hidrive-root-prefix 36019 36020 The root/parent folder for all paths. 36021 36022 Fill in to use the specified folder as the parent for all paths given to the remote. 36023 This way rclone can use any folder as its starting point. 36024 36025 Properties: 36026 36027 - Config: root_prefix 36028 - Env Var: RCLONE_HIDRIVE_ROOT_PREFIX 36029 - Type: string 36030 - Default: "/" 36031 - Examples: 36032 - "/" 36033 - The topmost directory accessible by rclone. 36034 - This will be equivalent with "root" if rclone uses a regular HiDrive user account. 36035 - "root" 36036 - The topmost directory of the HiDrive user account 36037 - "" 36038 - This specifies that there is no root-prefix for your paths. 36039 - When using this you will always need to specify paths to this remote with a valid parent e.g. "remote:/path/to/dir" or "remote:root/path/to/dir". 36040 36041 #### --hidrive-endpoint 36042 36043 Endpoint for the service. 36044 36045 This is the URL that API-calls will be made to. 36046 36047 Properties: 36048 36049 - Config: endpoint 36050 - Env Var: RCLONE_HIDRIVE_ENDPOINT 36051 - Type: string 36052 - Default: "https://api.hidrive.strato.com/2.1" 36053 36054 #### --hidrive-disable-fetching-member-count 36055 36056 Do not fetch number of objects in directories unless it is absolutely necessary. 36057 36058 Requests may be faster if the number of objects in subdirectories is not fetched. 36059 36060 Properties: 36061 36062 - Config: disable_fetching_member_count 36063 - Env Var: RCLONE_HIDRIVE_DISABLE_FETCHING_MEMBER_COUNT 36064 - Type: bool 36065 - Default: false 36066 36067 #### --hidrive-chunk-size 36068 36069 Chunksize for chunked uploads. 36070 36071 Any files larger than the configured cutoff (or files of unknown size) will be uploaded in chunks of this size. 36072 36073 The upper limit for this is 2147483647 bytes (about 2.000Gi). 36074 That is the maximum amount of bytes a single upload-operation will support. 36075 Setting this above the upper limit or to a negative value will cause uploads to fail. 36076 36077 Setting this to larger values may increase the upload speed at the cost of using more memory. 36078 It can be set to smaller values smaller to save on memory. 36079 36080 Properties: 36081 36082 - Config: chunk_size 36083 - Env Var: RCLONE_HIDRIVE_CHUNK_SIZE 36084 - Type: SizeSuffix 36085 - Default: 48Mi 36086 36087 #### --hidrive-upload-cutoff 36088 36089 Cutoff/Threshold for chunked uploads. 36090 36091 Any files larger than this will be uploaded in chunks of the configured chunksize. 36092 36093 The upper limit for this is 2147483647 bytes (about 2.000Gi). 36094 That is the maximum amount of bytes a single upload-operation will support. 36095 Setting this above the upper limit will cause uploads to fail. 36096 36097 Properties: 36098 36099 - Config: upload_cutoff 36100 - Env Var: RCLONE_HIDRIVE_UPLOAD_CUTOFF 36101 - Type: SizeSuffix 36102 - Default: 96Mi 36103 36104 #### --hidrive-upload-concurrency 36105 36106 Concurrency for chunked uploads. 36107 36108 This is the upper limit for how many transfers for the same file are running concurrently. 36109 Setting this above to a value smaller than 1 will cause uploads to deadlock. 36110 36111 If you are uploading small numbers of large files over high-speed links 36112 and these uploads do not fully utilize your bandwidth, then increasing 36113 this may help to speed up the transfers. 36114 36115 Properties: 36116 36117 - Config: upload_concurrency 36118 - Env Var: RCLONE_HIDRIVE_UPLOAD_CONCURRENCY 36119 - Type: int 36120 - Default: 4 36121 36122 #### --hidrive-encoding 36123 36124 The encoding for the backend. 36125 36126 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 36127 36128 Properties: 36129 36130 - Config: encoding 36131 - Env Var: RCLONE_HIDRIVE_ENCODING 36132 - Type: Encoding 36133 - Default: Slash,Dot 36134 36135 #### --hidrive-description 36136 36137 Description of the remote 36138 36139 Properties: 36140 36141 - Config: description 36142 - Env Var: RCLONE_HIDRIVE_DESCRIPTION 36143 - Type: string 36144 - Required: false 36145 36146 36147 36148 ## Limitations 36149 36150 ### Symbolic links 36151 36152 HiDrive is able to store symbolic links (*symlinks*) by design, 36153 for example, when unpacked from a zip archive. 36154 36155 There exists no direct mechanism to manage native symlinks in remotes. 36156 As such this implementation has chosen to ignore any native symlinks present in the remote. 36157 rclone will not be able to access or show any symlinks stored in the hidrive-remote. 36158 This means symlinks cannot be individually removed, copied, or moved, 36159 except when removing, copying, or moving the parent folder. 36160 36161 *This does not affect the `.rclonelink`-files 36162 that rclone uses to encode and store symbolic links.* 36163 36164 ### Sparse files 36165 36166 It is possible to store sparse files in HiDrive. 36167 36168 Note that copying a sparse file will expand the holes 36169 into null-byte (0x00) regions that will then consume disk space. 36170 Likewise, when downloading a sparse file, 36171 the resulting file will have null-byte regions in the place of file holes. 36172 36173 # HTTP 36174 36175 The HTTP remote is a read only remote for reading files of a 36176 webserver. The webserver should provide file listings which rclone 36177 will read and turn into a remote. This has been tested with common 36178 webservers such as Apache/Nginx/Caddy and will likely work with file 36179 listings from most web servers. (If it doesn't then please file an 36180 issue, or send a pull request!) 36181 36182 Paths are specified as `remote:` or `remote:path`. 36183 36184 The `remote:` represents the configured [url](#http-url), and any path following 36185 it will be resolved relative to this url, according to the URL standard. This 36186 means with remote url `https://beta.rclone.org/branch` and path `fix`, the 36187 resolved URL will be `https://beta.rclone.org/branch/fix`, while with path 36188 `/fix` the resolved URL will be `https://beta.rclone.org/fix` as the absolute 36189 path is resolved from the root of the domain. 36190 36191 If the path following the `remote:` ends with `/` it will be assumed to point 36192 to a directory. If the path does not end with `/`, then a HEAD request is sent 36193 and the response used to decide if it it is treated as a file or a directory 36194 (run with `-vv` to see details). When [--http-no-head](#http-no-head) is 36195 specified, a path without ending `/` is always assumed to be a file. If rclone 36196 incorrectly assumes the path is a file, the solution is to specify the path with 36197 ending `/`. When you know the path is a directory, ending it with `/` is always 36198 better as it avoids the initial HEAD request. 36199 36200 To just download a single file it is easier to use 36201 [copyurl](https://rclone.org/commands/rclone_copyurl/). 36202 36203 ## Configuration 36204 36205 Here is an example of how to make a remote called `remote`. First 36206 run: 36207 36208 rclone config 36209 36210 This will guide you through an interactive setup process: 36211 36212 No remotes found, make a new one? n) New remote s) Set configuration 36213 password q) Quit config n/s/q> n name> remote Type of storage to 36214 configure. Choose a number from below, or type in your own value [snip] 36215 XX / HTTP "http" [snip] Storage> http URL of http host to connect to 36216 Choose a number from below, or type in your own value 1 / Connect to 36217 example.com "https://example.com" url> https://beta.rclone.org Remote 36218 config -------------------- [remote] url = https://beta.rclone.org 36219 -------------------- y) Yes this is OK e) Edit this remote d) Delete 36220 this remote y/e/d> y Current remotes: 36221 36222 Name Type ==== ==== remote http 36223 36224 e) Edit existing remote 36225 f) New remote 36226 g) Delete remote 36227 h) Rename remote 36228 i) Copy remote 36229 j) Set configuration password 36230 k) Quit config e/n/d/r/c/s/q> q 36231 36232 36233 This remote is called `remote` and can now be used like this 36234 36235 See all the top level directories 36236 36237 rclone lsd remote: 36238 36239 List the contents of a directory 36240 36241 rclone ls remote:directory 36242 36243 Sync the remote `directory` to `/home/local/directory`, deleting any excess files. 36244 36245 rclone sync --interactive remote:directory /home/local/directory 36246 36247 ### Read only 36248 36249 This remote is read only - you can't upload files to an HTTP server. 36250 36251 ### Modification times 36252 36253 Most HTTP servers store time accurate to 1 second. 36254 36255 ### Checksum 36256 36257 No checksums are stored. 36258 36259 ### Usage without a config file 36260 36261 Since the http remote only has one config parameter it is easy to use 36262 without a config file: 36263 36264 rclone lsd --http-url https://beta.rclone.org :http: 36265 36266 or: 36267 36268 rclone lsd :http,url='https://beta.rclone.org': 36269 36270 36271 ### Standard options 36272 36273 Here are the Standard options specific to http (HTTP). 36274 36275 #### --http-url 36276 36277 URL of HTTP host to connect to. 36278 36279 E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password. 36280 36281 Properties: 36282 36283 - Config: url 36284 - Env Var: RCLONE_HTTP_URL 36285 - Type: string 36286 - Required: true 36287 36288 ### Advanced options 36289 36290 Here are the Advanced options specific to http (HTTP). 36291 36292 #### --http-headers 36293 36294 Set HTTP headers for all transactions. 36295 36296 Use this to set additional HTTP headers for all transactions. 36297 36298 The input format is comma separated list of key,value pairs. Standard 36299 [CSV encoding](https://godoc.org/encoding/csv) may be used. 36300 36301 For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'. 36302 36303 You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'. 36304 36305 Properties: 36306 36307 - Config: headers 36308 - Env Var: RCLONE_HTTP_HEADERS 36309 - Type: CommaSepList 36310 - Default: 36311 36312 #### --http-no-slash 36313 36314 Set this if the site doesn't end directories with /. 36315 36316 Use this if your target website does not use / on the end of 36317 directories. 36318 36319 A / on the end of a path is how rclone normally tells the difference 36320 between files and directories. If this flag is set, then rclone will 36321 treat all files with Content-Type: text/html as directories and read 36322 URLs from them rather than downloading them. 36323 36324 Note that this may cause rclone to confuse genuine HTML files with 36325 directories. 36326 36327 Properties: 36328 36329 - Config: no_slash 36330 - Env Var: RCLONE_HTTP_NO_SLASH 36331 - Type: bool 36332 - Default: false 36333 36334 #### --http-no-head 36335 36336 Don't use HEAD requests. 36337 36338 HEAD requests are mainly used to find file sizes in dir listing. 36339 If your site is being very slow to load then you can try this option. 36340 Normally rclone does a HEAD request for each potential file in a 36341 directory listing to: 36342 36343 - find its size 36344 - check it really exists 36345 - check to see if it is a directory 36346 36347 If you set this option, rclone will not do the HEAD request. This will mean 36348 that directory listings are much quicker, but rclone won't have the times or 36349 sizes of any files, and some files that don't exist may be in the listing. 36350 36351 Properties: 36352 36353 - Config: no_head 36354 - Env Var: RCLONE_HTTP_NO_HEAD 36355 - Type: bool 36356 - Default: false 36357 36358 #### --http-description 36359 36360 Description of the remote 36361 36362 Properties: 36363 36364 - Config: description 36365 - Env Var: RCLONE_HTTP_DESCRIPTION 36366 - Type: string 36367 - Required: false 36368 36369 ## Backend commands 36370 36371 Here are the commands specific to the http backend. 36372 36373 Run them with 36374 36375 rclone backend COMMAND remote: 36376 36377 The help below will explain what arguments each command takes. 36378 36379 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 36380 info on how to pass options and arguments. 36381 36382 These can be run on a running backend using the rc command 36383 [backend/command](https://rclone.org/rc/#backend-command). 36384 36385 ### set 36386 36387 Set command for updating the config parameters. 36388 36389 rclone backend set remote: [options] [<arguments>+] 36390 36391 This set command can be used to update the config parameters 36392 for a running http backend. 36393 36394 Usage Examples: 36395 36396 rclone backend set remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2] 36397 rclone rc backend/command command=set fs=remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2] 36398 rclone rc backend/command command=set fs=remote: -o url=https://example.com 36399 36400 The option keys are named as they are in the config file. 36401 36402 This rebuilds the connection to the http backend when it is called with 36403 the new parameters. Only new parameters need be passed as the values 36404 will default to those currently in use. 36405 36406 It doesn't return anything. 36407 36408 36409 36410 36411 ## Limitations 36412 36413 `rclone about` is not supported by the HTTP backend. Backends without 36414 this capability cannot determine free space for an rclone mount or 36415 use policy `mfs` (most free space) as a member of an rclone union 36416 remote. 36417 36418 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 36419 36420 # ImageKit 36421 This is a backend for the [ImageKit.io](https://imagekit.io/) storage service. 36422 36423 #### About ImageKit 36424 [ImageKit.io](https://imagekit.io/) provides real-time image and video optimizations, transformations, and CDN delivery. Over 1,000 businesses and 70,000 developers trust ImageKit with their images and videos on the web. 36425 36426 36427 #### Accounts & Pricing 36428 36429 To use this backend, you need to [create an account](https://imagekit.io/registration/) on ImageKit. Start with a free plan with generous usage limits. Then, as your requirements grow, upgrade to a plan that best fits your needs. See [the pricing details](https://imagekit.io/plans). 36430 36431 ## Configuration 36432 36433 Here is an example of making an imagekit configuration. 36434 36435 Firstly create a [ImageKit.io](https://imagekit.io/) account and choose a plan. 36436 36437 You will need to log in and get the `publicKey` and `privateKey` for your account from the developer section. 36438 36439 Now run 36440 36441 rclone config 36442 36443 36444 This will guide you through an interactive setup process: 36445 36446 No remotes found, make a new one? n) New remote s) Set configuration 36447 password q) Quit config n/s/q> n 36448 36449 Enter the name for the new remote. name> imagekit-media-library 36450 36451 Option Storage. Type of storage to configure. Choose a number from 36452 below, or type in your own value. [snip] XX / ImageKit.io (imagekit) 36453 [snip] Storage> imagekit 36454 36455 Option endpoint. You can find your ImageKit.io URL endpoint in your 36456 dashboard Enter a value. endpoint> https://ik.imagekit.io/imagekit_id 36457 36458 Option public_key. You can find your ImageKit.io public key in your 36459 dashboard Enter a value. public_key> public_**************************** 36460 36461 Option private_key. You can find your ImageKit.io private key in your 36462 dashboard Enter a value. private_key> 36463 private_**************************** 36464 36465 Edit advanced config? y) Yes n) No (default) y/n> n 36466 36467 Configuration complete. Options: - type: imagekit - endpoint: 36468 https://ik.imagekit.io/imagekit_id - public_key: 36469 public_**************************** - private_key: 36470 private_**************************** 36471 36472 Keep this "imagekit-media-library" remote? y) Yes this is OK (default) 36473 e) Edit this remote d) Delete this remote y/e/d> y 36474 36475 List directories in the top level of your Media Library 36476 36477 rclone lsd imagekit-media-library: 36478 36479 Make a new directory. 36480 36481 rclone mkdir imagekit-media-library:directory 36482 36483 List the contents of a directory. 36484 36485 rclone ls imagekit-media-library:directory 36486 36487 36488 ### Modified time and hashes 36489 36490 ImageKit does not support modification times or hashes yet. 36491 36492 ### Checksums 36493 36494 No checksums are supported. 36495 36496 36497 ### Standard options 36498 36499 Here are the Standard options specific to imagekit (ImageKit.io). 36500 36501 #### --imagekit-endpoint 36502 36503 You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 36504 36505 Properties: 36506 36507 - Config: endpoint 36508 - Env Var: RCLONE_IMAGEKIT_ENDPOINT 36509 - Type: string 36510 - Required: true 36511 36512 #### --imagekit-public-key 36513 36514 You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 36515 36516 Properties: 36517 36518 - Config: public_key 36519 - Env Var: RCLONE_IMAGEKIT_PUBLIC_KEY 36520 - Type: string 36521 - Required: true 36522 36523 #### --imagekit-private-key 36524 36525 You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys) 36526 36527 Properties: 36528 36529 - Config: private_key 36530 - Env Var: RCLONE_IMAGEKIT_PRIVATE_KEY 36531 - Type: string 36532 - Required: true 36533 36534 ### Advanced options 36535 36536 Here are the Advanced options specific to imagekit (ImageKit.io). 36537 36538 #### --imagekit-only-signed 36539 36540 If you have configured `Restrict unsigned image URLs` in your dashboard settings, set this to true. 36541 36542 Properties: 36543 36544 - Config: only_signed 36545 - Env Var: RCLONE_IMAGEKIT_ONLY_SIGNED 36546 - Type: bool 36547 - Default: false 36548 36549 #### --imagekit-versions 36550 36551 Include old versions in directory listings. 36552 36553 Properties: 36554 36555 - Config: versions 36556 - Env Var: RCLONE_IMAGEKIT_VERSIONS 36557 - Type: bool 36558 - Default: false 36559 36560 #### --imagekit-upload-tags 36561 36562 Tags to add to the uploaded files, e.g. "tag1,tag2". 36563 36564 Properties: 36565 36566 - Config: upload_tags 36567 - Env Var: RCLONE_IMAGEKIT_UPLOAD_TAGS 36568 - Type: string 36569 - Required: false 36570 36571 #### --imagekit-encoding 36572 36573 The encoding for the backend. 36574 36575 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 36576 36577 Properties: 36578 36579 - Config: encoding 36580 - Env Var: RCLONE_IMAGEKIT_ENCODING 36581 - Type: Encoding 36582 - Default: Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket 36583 36584 #### --imagekit-description 36585 36586 Description of the remote 36587 36588 Properties: 36589 36590 - Config: description 36591 - Env Var: RCLONE_IMAGEKIT_DESCRIPTION 36592 - Type: string 36593 - Required: false 36594 36595 ### Metadata 36596 36597 Any metadata supported by the underlying remote is read and written. 36598 36599 Here are the possible system metadata items for the imagekit backend. 36600 36601 | Name | Help | Type | Example | Read Only | 36602 |------|------|------|---------|-----------| 36603 | aws-tags | AI generated tags by AWS Rekognition associated with the image | string | tag1,tag2 | **Y** | 36604 | btime | Time of file birth (creation) read from Last-Modified header | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** | 36605 | custom-coordinates | Custom coordinates of the file | string | 0,0,100,100 | **Y** | 36606 | file-type | Type of the file | string | image | **Y** | 36607 | google-tags | AI generated tags by Google Cloud Vision associated with the image | string | tag1,tag2 | **Y** | 36608 | has-alpha | Whether the image has alpha channel or not | bool | | **Y** | 36609 | height | Height of the image or video in pixels | int | | **Y** | 36610 | is-private-file | Whether the file is private or not | bool | | **Y** | 36611 | size | Size of the object in bytes | int64 | | **Y** | 36612 | tags | Tags associated with the file | string | tag1,tag2 | **Y** | 36613 | width | Width of the image or video in pixels | int | | **Y** | 36614 36615 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 36616 36617 36618 36619 # Internet Archive 36620 36621 The Internet Archive backend utilizes Items on [archive.org](https://archive.org/) 36622 36623 Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses. 36624 36625 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 36626 command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`. 36627 36628 Unlike S3, listing up all items uploaded by you isn't supported. 36629 36630 Once you have made a remote, you can use it like this: 36631 36632 Make a new item 36633 36634 rclone mkdir remote:item 36635 36636 List the contents of a item 36637 36638 rclone ls remote:item 36639 36640 Sync `/home/local/directory` to the remote item, deleting any excess 36641 files in the item. 36642 36643 rclone sync --interactive /home/local/directory remote:item 36644 36645 ## Notes 36646 Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available. 36647 The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior. 36648 36649 You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key. 36650 By making it wait, rclone can do normal file comparison. 36651 Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue. 36652 36653 ## About metadata 36654 This backend supports setting, updating and reading metadata of each file. 36655 The metadata will appear as file metadata on Internet Archive. 36656 However, some fields are reserved by both Internet Archive and rclone. 36657 36658 The following are reserved by Internet Archive: 36659 - `name` 36660 - `source` 36661 - `size` 36662 - `md5` 36663 - `crc32` 36664 - `sha1` 36665 - `format` 36666 - `old_version` 36667 - `viruscheck` 36668 - `summation` 36669 36670 Trying to set values to these keys is ignored with a warning. 36671 Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime. 36672 36673 rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request. 36674 36675 If there are multiple values for a key, only the first one is returned. 36676 This is a limitation of rclone, that supports one value per one key. 36677 It can be triggered when you did a server-side copy. 36678 36679 Reading metadata will also provide custom (non-standard nor reserved) ones. 36680 36681 ## Filtering auto generated files 36682 36683 The Internet Archive automatically creates metadata files after 36684 upload. These can cause problems when doing an `rclone sync` as rclone 36685 will try, and fail, to delete them. These metadata files are not 36686 changeable, as they are created by the Internet Archive automatically. 36687 36688 These auto-created files can be excluded from the sync using [metadata 36689 filtering](https://rclone.org/filtering/#metadata). 36690 36691 rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata" 36692 36693 Which excludes from the sync any files which have the 36694 `source=metadata` or `format=Metadata` flags which are added to 36695 Internet Archive auto-created files. 36696 36697 ## Configuration 36698 36699 Here is an example of making an internetarchive configuration. 36700 Most applies to the other providers as well, any differences are described [below](#providers). 36701 36702 First run 36703 36704 rclone config 36705 36706 This will guide you through an interactive setup process. 36707 36708 No remotes found, make a new one? n) New remote s) Set configuration 36709 password q) Quit config n/s/q> n name> remote Option Storage. Type of 36710 storage to configure. Choose a number from below, or type in your own 36711 value. XX / InternetArchive Items (internetarchive) Storage> 36712 internetarchive Option access_key_id. IAS3 Access Key. Leave blank for 36713 anonymous access. You can find one here: 36714 https://archive.org/account/s3.php Enter a value. Press Enter to leave 36715 empty. access_key_id> XXXX Option secret_access_key. IAS3 Secret Key 36716 (password). Leave blank for anonymous access. Enter a value. Press Enter 36717 to leave empty. secret_access_key> XXXX Edit advanced config? y) Yes n) 36718 No (default) y/n> y Option endpoint. IAS3 Endpoint. Leave blank for 36719 default value. Enter a string value. Press Enter for the default 36720 (https://s3.us.archive.org). endpoint> Option front_endpoint. Host of 36721 InternetArchive Frontend. Leave blank for default value. Enter a string 36722 value. Press Enter for the default (https://archive.org). 36723 front_endpoint> Option disable_checksum. Don't store MD5 checksum with 36724 object metadata. Normally rclone will calculate the MD5 checksum of the 36725 input before uploading it so it can ask the server to check the object 36726 against checksum. This is great for data integrity checking but can 36727 cause long delays for large files to start uploading. Enter a boolean 36728 value (true or false). Press Enter for the default (true). 36729 disable_checksum> true Option encoding. The encoding for the backend. 36730 See the encoding section in the overview for more info. Enter a 36731 encoder.MultiEncoder value. Press Enter for the default 36732 (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot). encoding> Edit 36733 advanced config? y) Yes n) No (default) y/n> n -------------------- 36734 [remote] type = internetarchive access_key_id = XXXX secret_access_key = 36735 XXXX -------------------- y) Yes this is OK (default) e) Edit this 36736 remote d) Delete this remote y/e/d> y 36737 36738 36739 36740 ### Standard options 36741 36742 Here are the Standard options specific to internetarchive (Internet Archive). 36743 36744 #### --internetarchive-access-key-id 36745 36746 IAS3 Access Key. 36747 36748 Leave blank for anonymous access. 36749 You can find one here: https://archive.org/account/s3.php 36750 36751 Properties: 36752 36753 - Config: access_key_id 36754 - Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID 36755 - Type: string 36756 - Required: false 36757 36758 #### --internetarchive-secret-access-key 36759 36760 IAS3 Secret Key (password). 36761 36762 Leave blank for anonymous access. 36763 36764 Properties: 36765 36766 - Config: secret_access_key 36767 - Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY 36768 - Type: string 36769 - Required: false 36770 36771 ### Advanced options 36772 36773 Here are the Advanced options specific to internetarchive (Internet Archive). 36774 36775 #### --internetarchive-endpoint 36776 36777 IAS3 Endpoint. 36778 36779 Leave blank for default value. 36780 36781 Properties: 36782 36783 - Config: endpoint 36784 - Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT 36785 - Type: string 36786 - Default: "https://s3.us.archive.org" 36787 36788 #### --internetarchive-front-endpoint 36789 36790 Host of InternetArchive Frontend. 36791 36792 Leave blank for default value. 36793 36794 Properties: 36795 36796 - Config: front_endpoint 36797 - Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT 36798 - Type: string 36799 - Default: "https://archive.org" 36800 36801 #### --internetarchive-disable-checksum 36802 36803 Don't ask the server to test against MD5 checksum calculated by rclone. 36804 Normally rclone will calculate the MD5 checksum of the input before 36805 uploading it so it can ask the server to check the object against checksum. 36806 This is great for data integrity checking but can cause long delays for 36807 large files to start uploading. 36808 36809 Properties: 36810 36811 - Config: disable_checksum 36812 - Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM 36813 - Type: bool 36814 - Default: true 36815 36816 #### --internetarchive-wait-archive 36817 36818 Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish. 36819 Only enable if you need to be guaranteed to be reflected after write operations. 36820 0 to disable waiting. No errors to be thrown in case of timeout. 36821 36822 Properties: 36823 36824 - Config: wait_archive 36825 - Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE 36826 - Type: Duration 36827 - Default: 0s 36828 36829 #### --internetarchive-encoding 36830 36831 The encoding for the backend. 36832 36833 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 36834 36835 Properties: 36836 36837 - Config: encoding 36838 - Env Var: RCLONE_INTERNETARCHIVE_ENCODING 36839 - Type: Encoding 36840 - Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot 36841 36842 #### --internetarchive-description 36843 36844 Description of the remote 36845 36846 Properties: 36847 36848 - Config: description 36849 - Env Var: RCLONE_INTERNETARCHIVE_DESCRIPTION 36850 - Type: string 36851 - Required: false 36852 36853 ### Metadata 36854 36855 Metadata fields provided by Internet Archive. 36856 If there are multiple values for a key, only the first one is returned. 36857 This is a limitation of Rclone, that supports one value per one key. 36858 36859 Owner is able to add custom keys. Metadata feature grabs all the keys including them. 36860 36861 Here are the possible system metadata items for the internetarchive backend. 36862 36863 | Name | Help | Type | Example | Read Only | 36864 |------|------|------|---------|-----------| 36865 | crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** | 36866 | format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** | 36867 | md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** | 36868 | mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** | 36869 | name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** | 36870 | old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** | 36871 | rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N | 36872 | rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N | 36873 | rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N | 36874 | sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** | 36875 | size | File size in bytes | decimal number | 123456 | **Y** | 36876 | source | The source of the file | string | original | **Y** | 36877 | summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** | 36878 | viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** | 36879 36880 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 36881 36882 36883 36884 # Jottacloud 36885 36886 Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters 36887 in Norway. In addition to the official service at [jottacloud.com](https://www.jottacloud.com/), 36888 it also provides white-label solutions to different companies, such as: 36889 * Telia 36890 * Telia Cloud (cloud.telia.se) 36891 * Telia Sky (sky.telia.no) 36892 * Tele2 36893 * Tele2 Cloud (mittcloud.tele2.se) 36894 * Onlime 36895 * Onlime Cloud Storage (onlime.dk) 36896 * Elkjøp (with subsidiaries): 36897 * Elkjøp Cloud (cloud.elkjop.no) 36898 * Elgiganten Sweden (cloud.elgiganten.se) 36899 * Elgiganten Denmark (cloud.elgiganten.dk) 36900 * Giganti Cloud (cloud.gigantti.fi) 36901 * ELKO Cloud (cloud.elko.is) 36902 36903 Most of the white-label versions are supported by this backend, although may require different 36904 authentication setup - described below. 36905 36906 Paths are specified as `remote:path` 36907 36908 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 36909 36910 ## Authentication types 36911 36912 Some of the whitelabel versions uses a different authentication method than the official service, 36913 and you have to choose the correct one when setting up the remote. 36914 36915 ### Standard authentication 36916 36917 The standard authentication method used by the official service (jottacloud.com), as well as 36918 some of the whitelabel services, requires you to generate a single-use personal login token 36919 from the account security settings in the service's web interface. Log in to your account, 36920 go to "Settings" and then "Security", or use the direct link presented to you by rclone when 36921 configuring the remote: <https://www.jottacloud.com/web/secure>. Scroll down to the section 36922 "Personal login token", and click the "Generate" button. Note that if you are using a 36923 whitelabel service you probably can't use the direct link, you need to find the same page in 36924 their dedicated web interface, and also it may be in a different location than described above. 36925 36926 To access your account from multiple instances of rclone, you need to configure each of them 36927 with a separate personal login token. E.g. you create a Jottacloud remote with rclone in one 36928 location, and copy the configuration file to a second location where you also want to run 36929 rclone and access the same remote. Then you need to replace the token for one of them, using 36930 the [config reconnect](https://rclone.org/commands/rclone_config_reconnect/) command, which 36931 requires you to generate a new personal login token and supply as input. If you do not 36932 do this, the token may easily end up being invalidated, resulting in both instances failing 36933 with an error message something along the lines of: 36934 36935 oauth2: cannot fetch token: 400 Bad Request 36936 Response: {"error":"invalid_grant","error_description":"Stale token"} 36937 36938 When this happens, you need to replace the token as described above to be able to use your 36939 remote again. 36940 36941 All personal login tokens you have taken into use will be listed in the web interface under 36942 "My logged in devices", and from the right side of that list you can click the "X" button to 36943 revoke individual tokens. 36944 36945 ### Legacy authentication 36946 36947 If you are using one of the whitelabel versions (e.g. from Elkjøp) you may not have the option 36948 to generate a CLI token. In this case you'll have to use the legacy authentication. To do this select 36949 yes when the setup asks for legacy authentication and enter your username and password. 36950 The rest of the setup is identical to the default setup. 36951 36952 ### Telia Cloud authentication 36953 36954 Similar to other whitelabel versions Telia Cloud doesn't offer the option of creating a CLI token, and 36955 additionally uses a separate authentication flow where the username is generated internally. To setup 36956 rclone to use Telia Cloud, choose Telia Cloud authentication in the setup. The rest of the setup is 36957 identical to the default setup. 36958 36959 ### Tele2 Cloud authentication 36960 36961 As Tele2-Com Hem merger was completed this authentication can be used for former Com Hem Cloud and 36962 Tele2 Cloud customers as no support for creating a CLI token exists, and additionally uses a separate 36963 authentication flow where the username is generated internally. To setup rclone to use Tele2 Cloud, 36964 choose Tele2 Cloud authentication in the setup. The rest of the setup is identical to the default setup. 36965 36966 ### Onlime Cloud Storage authentication 36967 36968 Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but 36969 have recently set up their own hosting, transferring their customers from Jottacloud servers to their 36970 own ones. 36971 36972 This, of course, necessitates using their servers for authentication, but otherwise functionality and 36973 architecture seems equivalent to Jottacloud. 36974 36975 To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest 36976 of the setup is identical to the default setup. 36977 36978 ## Configuration 36979 36980 Here is an example of how to make a remote called `remote` with the default setup. First run: 36981 36982 rclone config 36983 36984 This will guide you through an interactive setup process: 36985 36986 No remotes found, make a new one? n) New remote s) Set configuration 36987 password q) Quit config n/s/q> n name> remote Option Storage. Type of 36988 storage to configure. Choose a number from below, or type in your own 36989 value. [snip] XX / Jottacloud (jottacloud) [snip] Storage> jottacloud 36990 Edit advanced config? y) Yes n) No (default) y/n> n Option config_type. 36991 Select authentication type. Choose a number from below, or type in an 36992 existing string value. Press Enter for the default (standard). / 36993 Standard authentication. 1 | Use this if you're a normal Jottacloud 36994 user. (standard) / Legacy authentication. 2 | This is only required for 36995 certain whitelabel versions of Jottacloud and not recommended for normal 36996 users. (legacy) / Telia Cloud authentication. 3 | Use this if you are 36997 using Telia Cloud. (telia) / Tele2 Cloud authentication. 4 | Use this 36998 if you are using Tele2 Cloud. (tele2) / Onlime Cloud authentication. 5 36999 | Use this if you are using Onlime Cloud. (onlime) config_type> 1 37000 Personal login token. Generate here: 37001 https://www.jottacloud.com/web/secure Login Token> Use a non-standard 37002 device/mountpoint? Choosing no, the default, will let you access the 37003 storage used for the archive section of the official Jottacloud client. 37004 If you instead want to access the sync or the backup section, for 37005 example, you must choose yes. y) Yes n) No (default) y/n> y Option 37006 config_device. The device to use. In standard setup the built-in Jotta 37007 device is used, which contains predefined mountpoints for archive, sync 37008 etc. All other devices are treated as backup devices by the official 37009 Jottacloud client. You may create a new by entering a unique name. 37010 Choose a number from below, or type in your own string value. Press 37011 Enter for the default (DESKTOP-3H31129). 1 > DESKTOP-3H31129 2 > Jotta 37012 config_device> 2 Option config_mountpoint. The mountpoint to use for the 37013 built-in device Jotta. The standard setup is to use the Archive 37014 mountpoint. Most other mountpoints have very limited support in rclone 37015 and should generally be avoided. Choose a number from below, or type in 37016 an existing string value. Press Enter for the default (Archive). 1 > 37017 Archive 2 > Shared 3 > Sync config_mountpoint> 1 -------------------- 37018 [remote] type = jottacloud configVersion = 1 client_id = jottacli 37019 client_secret = tokenURL = 37020 https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token 37021 token = {........} username = 2940e57271a93d987d6f8a21 device = Jotta 37022 mountpoint = Archive -------------------- y) Yes this is OK (default) e) 37023 Edit this remote d) Delete this remote y/e/d> y 37024 37025 37026 Once configured you can then use `rclone` like this, 37027 37028 List directories in top level of your Jottacloud 37029 37030 rclone lsd remote: 37031 37032 List all the files in your Jottacloud 37033 37034 rclone ls remote: 37035 37036 To copy a local directory to an Jottacloud directory called backup 37037 37038 rclone copy /home/source remote:backup 37039 37040 ### Devices and Mountpoints 37041 37042 The official Jottacloud client registers a device for each computer you install 37043 it on, and shows them in the backup section of the user interface. For each 37044 folder you select for backup it will create a mountpoint within this device. 37045 A built-in device called Jotta is special, and contains mountpoints Archive, 37046 Sync and some others, used for corresponding features in official clients. 37047 37048 With rclone you'll want to use the standard Jotta/Archive device/mountpoint in 37049 most cases. However, you may for example want to access files from the sync or 37050 backup functionality provided by the official clients, and rclone therefore 37051 provides the option to select other devices and mountpoints during config. 37052 37053 You are allowed to create new devices and mountpoints. All devices except the 37054 built-in Jotta device are treated as backup devices by official Jottacloud 37055 clients, and the mountpoints on them are individual backup sets. 37056 37057 With the built-in Jotta device, only existing, built-in, mountpoints can be 37058 selected. In addition to the mentioned Archive and Sync, it may contain 37059 several other mountpoints such as: Latest, Links, Shared and Trash. All of 37060 these are special mountpoints with a different internal representation than 37061 the "regular" mountpoints. Rclone will only to a very limited degree support 37062 them. Generally you should avoid these, unless you know what you are doing. 37063 37064 ### --fast-list 37065 37066 This backend supports `--fast-list` which allows you to use fewer 37067 transactions in exchange for more memory. See the [rclone 37068 docs](https://rclone.org/docs/#fast-list) for more details. 37069 37070 Note that the implementation in Jottacloud always uses only a single 37071 API request to get the entire list, so for large folders this could 37072 lead to long wait time before the first results are shown. 37073 37074 Note also that with rclone version 1.58 and newer, information about 37075 [MIME types](https://rclone.org/overview/#mime-type) and metadata item [utime](#metadata) 37076 are not available when using `--fast-list`. 37077 37078 ### Modification times and hashes 37079 37080 Jottacloud allows modification times to be set on objects accurate to 1 37081 second. These will be used to detect whether objects need syncing or 37082 not. 37083 37084 Jottacloud supports MD5 type hashes, so you can use the `--checksum` 37085 flag. 37086 37087 Note that Jottacloud requires the MD5 hash before upload so if the 37088 source does not have an MD5 checksum then the file will be cached 37089 temporarily on disk (in location given by 37090 [--temp-dir](https://rclone.org/docs/#temp-dir-dir)) before it is uploaded. 37091 Small files will be cached in memory - see the 37092 [--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag. 37093 When uploading from local disk the source checksum is always available, 37094 so this does not apply. Starting with rclone version 1.52 the same is 37095 true for encrypted remotes (in older versions the crypt backend would not 37096 calculate hashes for uploads from local disk, so the Jottacloud 37097 backend had to do it as described above). 37098 37099 ### Restricted filename characters 37100 37101 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 37102 the following characters are also replaced: 37103 37104 | Character | Value | Replacement | 37105 | --------- |:-----:|:-----------:| 37106 | " | 0x22 | " | 37107 | * | 0x2A | * | 37108 | : | 0x3A | : | 37109 | < | 0x3C | < | 37110 | > | 0x3E | > | 37111 | ? | 0x3F | ? | 37112 | \| | 0x7C | | | 37113 37114 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 37115 as they can't be used in XML strings. 37116 37117 ### Deleting files 37118 37119 By default, rclone will send all files to the trash when deleting files. They will be permanently 37120 deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately 37121 by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable. 37122 Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command. 37123 37124 ### Versions 37125 37126 Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it. 37127 Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website. 37128 37129 Versioning can be disabled by `--jottacloud-no-versions` option. This is achieved by deleting the remote file prior to uploading 37130 a new version. If the upload the fails no version of the file will be available in the remote. 37131 37132 ### Quota information 37133 37134 To view your current quota you can use the `rclone about remote:` 37135 command which will display your usage limit (unless it is unlimited) 37136 and the current usage. 37137 37138 37139 ### Standard options 37140 37141 Here are the Standard options specific to jottacloud (Jottacloud). 37142 37143 #### --jottacloud-client-id 37144 37145 OAuth Client Id. 37146 37147 Leave blank normally. 37148 37149 Properties: 37150 37151 - Config: client_id 37152 - Env Var: RCLONE_JOTTACLOUD_CLIENT_ID 37153 - Type: string 37154 - Required: false 37155 37156 #### --jottacloud-client-secret 37157 37158 OAuth Client Secret. 37159 37160 Leave blank normally. 37161 37162 Properties: 37163 37164 - Config: client_secret 37165 - Env Var: RCLONE_JOTTACLOUD_CLIENT_SECRET 37166 - Type: string 37167 - Required: false 37168 37169 ### Advanced options 37170 37171 Here are the Advanced options specific to jottacloud (Jottacloud). 37172 37173 #### --jottacloud-token 37174 37175 OAuth Access Token as a JSON blob. 37176 37177 Properties: 37178 37179 - Config: token 37180 - Env Var: RCLONE_JOTTACLOUD_TOKEN 37181 - Type: string 37182 - Required: false 37183 37184 #### --jottacloud-auth-url 37185 37186 Auth server URL. 37187 37188 Leave blank to use the provider defaults. 37189 37190 Properties: 37191 37192 - Config: auth_url 37193 - Env Var: RCLONE_JOTTACLOUD_AUTH_URL 37194 - Type: string 37195 - Required: false 37196 37197 #### --jottacloud-token-url 37198 37199 Token server url. 37200 37201 Leave blank to use the provider defaults. 37202 37203 Properties: 37204 37205 - Config: token_url 37206 - Env Var: RCLONE_JOTTACLOUD_TOKEN_URL 37207 - Type: string 37208 - Required: false 37209 37210 #### --jottacloud-md5-memory-limit 37211 37212 Files bigger than this will be cached on disk to calculate the MD5 if required. 37213 37214 Properties: 37215 37216 - Config: md5_memory_limit 37217 - Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT 37218 - Type: SizeSuffix 37219 - Default: 10Mi 37220 37221 #### --jottacloud-trashed-only 37222 37223 Only show files that are in the trash. 37224 37225 This will show trashed files in their original directory structure. 37226 37227 Properties: 37228 37229 - Config: trashed_only 37230 - Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY 37231 - Type: bool 37232 - Default: false 37233 37234 #### --jottacloud-hard-delete 37235 37236 Delete files permanently rather than putting them into the trash. 37237 37238 Properties: 37239 37240 - Config: hard_delete 37241 - Env Var: RCLONE_JOTTACLOUD_HARD_DELETE 37242 - Type: bool 37243 - Default: false 37244 37245 #### --jottacloud-upload-resume-limit 37246 37247 Files bigger than this can be resumed if the upload fail's. 37248 37249 Properties: 37250 37251 - Config: upload_resume_limit 37252 - Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT 37253 - Type: SizeSuffix 37254 - Default: 10Mi 37255 37256 #### --jottacloud-no-versions 37257 37258 Avoid server side versioning by deleting files and recreating files instead of overwriting them. 37259 37260 Properties: 37261 37262 - Config: no_versions 37263 - Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS 37264 - Type: bool 37265 - Default: false 37266 37267 #### --jottacloud-encoding 37268 37269 The encoding for the backend. 37270 37271 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 37272 37273 Properties: 37274 37275 - Config: encoding 37276 - Env Var: RCLONE_JOTTACLOUD_ENCODING 37277 - Type: Encoding 37278 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot 37279 37280 #### --jottacloud-description 37281 37282 Description of the remote 37283 37284 Properties: 37285 37286 - Config: description 37287 - Env Var: RCLONE_JOTTACLOUD_DESCRIPTION 37288 - Type: string 37289 - Required: false 37290 37291 ### Metadata 37292 37293 Jottacloud has limited support for metadata, currently an extended set of timestamps. 37294 37295 Here are the possible system metadata items for the jottacloud backend. 37296 37297 | Name | Help | Type | Example | Read Only | 37298 |------|------|------|---------|-----------| 37299 | btime | Time of file birth (creation), read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N | 37300 | content-type | MIME type, also known as media type | string | text/plain | **Y** | 37301 | mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N | 37302 | utime | Time of last upload, when current revision was created, generated by backend | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** | 37303 37304 See the [metadata](https://rclone.org/docs/#metadata) docs for more info. 37305 37306 37307 37308 ## Limitations 37309 37310 Note that Jottacloud is case insensitive so you can't have a file called 37311 "Hello.doc" and one called "hello.doc". 37312 37313 There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical 37314 looking unicode equivalent. For example if a file has a ? in it will be mapped to ? instead. 37315 37316 Jottacloud only supports filenames up to 255 characters in length. 37317 37318 ## Troubleshooting 37319 37320 Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove 37321 operations to previously deleted paths to fail. Emptying the trash should help in such cases. 37322 37323 # Koofr 37324 37325 Paths are specified as `remote:path` 37326 37327 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 37328 37329 ## Configuration 37330 37331 The initial setup for Koofr involves creating an application password for 37332 rclone. You can do that by opening the Koofr 37333 [web application](https://app.koofr.net/app/admin/preferences/password), 37334 giving the password a nice name like `rclone` and clicking on generate. 37335 37336 Here is an example of how to make a remote called `koofr`. First run: 37337 37338 rclone config 37339 37340 This will guide you through an interactive setup process: 37341 37342 No remotes found, make a new one? n) New remote s) Set configuration 37343 password q) Quit config n/s/q> n name> koofr Option Storage. Type of 37344 storage to configure. Choose a number from below, or type in your own 37345 value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible 37346 storage providers (koofr) [snip] Storage> koofr Option provider. Choose 37347 your storage provider. Choose a number from below, or type in your own 37348 value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/ 37349 (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/ (digistorage) 3 37350 / Any other Koofr API compatible storage service (other) provider> 1 37351 Option user. Your user name. Enter a value. user> USERNAME Option 37352 password. Your password for rclone (generate one at 37353 https://app.koofr.net/app/admin/preferences/password). Choose an 37354 alternative below. y) Yes, type in my own password g) Generate random 37355 password y/g> y Enter the password: password: Confirm the password: 37356 password: Edit advanced config? y) Yes n) No (default) y/n> n Remote 37357 config -------------------- [koofr] type = koofr provider = koofr user = 37358 USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this 37359 is OK (default) e) Edit this remote d) Delete this remote y/e/d> y 37360 37361 37362 You can choose to edit advanced config in order to enter your own service URL 37363 if you use an on-premise or white label Koofr instance, or choose an alternative 37364 mount instead of your primary storage. 37365 37366 Once configured you can then use `rclone` like this, 37367 37368 List directories in top level of your Koofr 37369 37370 rclone lsd koofr: 37371 37372 List all the files in your Koofr 37373 37374 rclone ls koofr: 37375 37376 To copy a local directory to an Koofr directory called backup 37377 37378 rclone copy /home/source koofr:backup 37379 37380 ### Restricted filename characters 37381 37382 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 37383 the following characters are also replaced: 37384 37385 | Character | Value | Replacement | 37386 | --------- |:-----:|:-----------:| 37387 | \ | 0x5C | \ | 37388 37389 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 37390 as they can't be used in XML strings. 37391 37392 37393 ### Standard options 37394 37395 Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers). 37396 37397 #### --koofr-provider 37398 37399 Choose your storage provider. 37400 37401 Properties: 37402 37403 - Config: provider 37404 - Env Var: RCLONE_KOOFR_PROVIDER 37405 - Type: string 37406 - Required: false 37407 - Examples: 37408 - "koofr" 37409 - Koofr, https://app.koofr.net/ 37410 - "digistorage" 37411 - Digi Storage, https://storage.rcs-rds.ro/ 37412 - "other" 37413 - Any other Koofr API compatible storage service 37414 37415 #### --koofr-endpoint 37416 37417 The Koofr API endpoint to use. 37418 37419 Properties: 37420 37421 - Config: endpoint 37422 - Env Var: RCLONE_KOOFR_ENDPOINT 37423 - Provider: other 37424 - Type: string 37425 - Required: true 37426 37427 #### --koofr-user 37428 37429 Your user name. 37430 37431 Properties: 37432 37433 - Config: user 37434 - Env Var: RCLONE_KOOFR_USER 37435 - Type: string 37436 - Required: true 37437 37438 #### --koofr-password 37439 37440 Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password). 37441 37442 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 37443 37444 Properties: 37445 37446 - Config: password 37447 - Env Var: RCLONE_KOOFR_PASSWORD 37448 - Provider: koofr 37449 - Type: string 37450 - Required: true 37451 37452 ### Advanced options 37453 37454 Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers). 37455 37456 #### --koofr-mountid 37457 37458 Mount ID of the mount to use. 37459 37460 If omitted, the primary mount is used. 37461 37462 Properties: 37463 37464 - Config: mountid 37465 - Env Var: RCLONE_KOOFR_MOUNTID 37466 - Type: string 37467 - Required: false 37468 37469 #### --koofr-setmtime 37470 37471 Does the backend support setting modification time. 37472 37473 Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend. 37474 37475 Properties: 37476 37477 - Config: setmtime 37478 - Env Var: RCLONE_KOOFR_SETMTIME 37479 - Type: bool 37480 - Default: true 37481 37482 #### --koofr-encoding 37483 37484 The encoding for the backend. 37485 37486 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 37487 37488 Properties: 37489 37490 - Config: encoding 37491 - Env Var: RCLONE_KOOFR_ENCODING 37492 - Type: Encoding 37493 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 37494 37495 #### --koofr-description 37496 37497 Description of the remote 37498 37499 Properties: 37500 37501 - Config: description 37502 - Env Var: RCLONE_KOOFR_DESCRIPTION 37503 - Type: string 37504 - Required: false 37505 37506 37507 37508 ## Limitations 37509 37510 Note that Koofr is case insensitive so you can't have a file called 37511 "Hello.doc" and one called "hello.doc". 37512 37513 ## Providers 37514 37515 ### Koofr 37516 37517 This is the original [Koofr](https://koofr.eu) storage provider used as main example and described in the [configuration](#configuration) section above. 37518 37519 ### Digi Storage 37520 37521 [Digi Storage](https://www.digi.ro/servicii/online/digi-storage) is a cloud storage service run by [Digi.ro](https://www.digi.ro/) that 37522 provides a Koofr API. 37523 37524 Here is an example of how to make a remote called `ds`. First run: 37525 37526 rclone config 37527 37528 This will guide you through an interactive setup process: 37529 37530 No remotes found, make a new one? n) New remote s) Set configuration 37531 password q) Quit config n/s/q> n name> ds Option Storage. Type of 37532 storage to configure. Choose a number from below, or type in your own 37533 value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible 37534 storage providers (koofr) [snip] Storage> koofr Option provider. Choose 37535 your storage provider. Choose a number from below, or type in your own 37536 value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/ 37537 (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/ (digistorage) 3 37538 / Any other Koofr API compatible storage service (other) provider> 2 37539 Option user. Your user name. Enter a value. user> USERNAME Option 37540 password. Your password for rclone (generate one at 37541 https://storage.rcs-rds.ro/app/admin/preferences/password). Choose an 37542 alternative below. y) Yes, type in my own password g) Generate random 37543 password y/g> y Enter the password: password: Confirm the password: 37544 password: Edit advanced config? y) Yes n) No (default) y/n> n 37545 -------------------- [ds] type = koofr provider = digistorage user = 37546 USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this 37547 is OK (default) e) Edit this remote d) Delete this remote y/e/d> y 37548 37549 37550 ### Other 37551 37552 You may also want to use another, public or private storage provider that runs a Koofr API compatible service, by simply providing the base URL to connect to. 37553 37554 Here is an example of how to make a remote called `other`. First run: 37555 37556 rclone config 37557 37558 This will guide you through an interactive setup process: 37559 37560 No remotes found, make a new one? n) New remote s) Set configuration 37561 password q) Quit config n/s/q> n name> other Option Storage. Type of 37562 storage to configure. Choose a number from below, or type in your own 37563 value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible 37564 storage providers (koofr) [snip] Storage> koofr Option provider. Choose 37565 your storage provider. Choose a number from below, or type in your own 37566 value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/ 37567 (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/ (digistorage) 3 37568 / Any other Koofr API compatible storage service (other) provider> 3 37569 Option endpoint. The Koofr API endpoint to use. Enter a value. endpoint> 37570 https://koofr.other.org Option user. Your user name. Enter a value. 37571 user> USERNAME Option password. Your password for rclone (generate one 37572 at your service's settings page). Choose an alternative below. y) Yes, 37573 type in my own password g) Generate random password y/g> y Enter the 37574 password: password: Confirm the password: password: Edit advanced 37575 config? y) Yes n) No (default) y/n> n -------------------- [other] type 37576 = koofr provider = other endpoint = https://koofr.other.org user = 37577 USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this 37578 is OK (default) e) Edit this remote d) Delete this remote y/e/d> y 37579 37580 37581 # Linkbox 37582 37583 Linkbox is [a private cloud drive](https://linkbox.to/). 37584 37585 ## Configuration 37586 37587 Here is an example of making a remote for Linkbox. 37588 37589 First run: 37590 37591 rclone config 37592 37593 This will guide you through an interactive setup process: 37594 37595 No remotes found, make a new one? n) New remote s) Set configuration 37596 password q) Quit config n/s/q> n 37597 37598 Enter name for new remote. name> remote 37599 37600 Option Storage. Type of storage to configure. Choose a number from 37601 below, or type in your own value. XX / Linkbox (linkbox) Storage> XX 37602 37603 Option token. Token from https://www.linkbox.to/admin/account Enter a 37604 value. token> testFromCLToken 37605 37606 Configuration complete. Options: - type: linkbox - token: XXXXXXXXXXX 37607 Keep this "linkbox" remote? y) Yes this is OK (default) e) Edit this 37608 remote d) Delete this remote y/e/d> y 37609 37610 37611 37612 ### Standard options 37613 37614 Here are the Standard options specific to linkbox (Linkbox). 37615 37616 #### --linkbox-token 37617 37618 Token from https://www.linkbox.to/admin/account 37619 37620 Properties: 37621 37622 - Config: token 37623 - Env Var: RCLONE_LINKBOX_TOKEN 37624 - Type: string 37625 - Required: true 37626 37627 ### Advanced options 37628 37629 Here are the Advanced options specific to linkbox (Linkbox). 37630 37631 #### --linkbox-description 37632 37633 Description of the remote 37634 37635 Properties: 37636 37637 - Config: description 37638 - Env Var: RCLONE_LINKBOX_DESCRIPTION 37639 - Type: string 37640 - Required: false 37641 37642 37643 37644 ## Limitations 37645 37646 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 37647 as they can't be used in JSON strings. 37648 37649 # Mail.ru Cloud 37650 37651 [Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS. 37652 37653 ## Features highlights 37654 37655 - Paths may be as deep as required, e.g. `remote:directory/subdirectory` 37656 - Files have a `last modified time` property, directories don't 37657 - Deleted files are by default moved to the trash 37658 - Files and directories can be shared via public links 37659 - Partial uploads or streaming are not supported, file size must be known before upload 37660 - Maximum file size is limited to 2G for a free account, unlimited for paid accounts 37661 - Storage keeps hash for all files and performs transparent deduplication, 37662 the hash algorithm is a modified SHA1 37663 - If a particular file is already present in storage, one can quickly submit file hash 37664 instead of long file upload (this optimization is supported by rclone) 37665 37666 ## Configuration 37667 37668 Here is an example of making a mailru configuration. 37669 37670 First create a Mail.ru Cloud account and choose a tariff. 37671 37672 You will need to log in and create an app password for rclone. Rclone 37673 **will not work** with your normal username and password - it will 37674 give an error like `oauth2: server response missing access_token`. 37675 37676 - Click on your user icon in the top right 37677 - Go to Security / "Пароль и безопасность" 37678 - Click password for apps / "Пароли для внешних приложений" 37679 - Add the password - give it a name - eg "rclone" 37680 - Copy the password and use this password below - your normal login password won't work. 37681 37682 Now run 37683 37684 rclone config 37685 37686 This will guide you through an interactive setup process: 37687 37688 No remotes found, make a new one? n) New remote s) Set configuration 37689 password q) Quit config n/s/q> n name> remote Type of storage to 37690 configure. Type of storage to configure. Enter a string value. Press 37691 Enter for the default (""). Choose a number from below, or type in your 37692 own value [snip] XX / Mail.ru Cloud "mailru" [snip] Storage> mailru 37693 User name (usually email) Enter a string value. Press Enter for the 37694 default (""). user> username@mail.ru Password 37695 37696 This must be an app password - rclone will not work with your normal 37697 password. See the Configuration section in the docs for how to make an 37698 app password. y) Yes type in my own password g) Generate random password 37699 y/g> y Enter the password: password: Confirm the password: password: 37700 Skip full upload if there is another file with same data hash. This 37701 feature is called "speedup" or "put by hash". It is especially efficient 37702 in case of generally available files like popular books, video or audio 37703 clips [snip] Enter a boolean value (true or false). Press Enter for the 37704 default ("true"). Choose a number from below, or type in your own value 37705 1 / Enable "true" 2 / Disable "false" speedup_enable> 1 Edit advanced 37706 config? (y/n) y) Yes n) No y/n> n Remote config -------------------- 37707 [remote] type = mailru user = username@mail.ru pass = *** ENCRYPTED *** 37708 speedup_enable = true -------------------- y) Yes this is OK e) Edit 37709 this remote d) Delete this remote y/e/d> y 37710 37711 37712 Configuration of this backend does not require a local web browser. 37713 You can use the configured backend as shown below: 37714 37715 See top level directories 37716 37717 rclone lsd remote: 37718 37719 Make a new directory 37720 37721 rclone mkdir remote:directory 37722 37723 List the contents of a directory 37724 37725 rclone ls remote:directory 37726 37727 Sync `/home/local/directory` to the remote path, deleting any 37728 excess files in the path. 37729 37730 rclone sync --interactive /home/local/directory remote:directory 37731 37732 ### Modification times and hashes 37733 37734 Files support a modification time attribute with up to 1 second precision. 37735 Directories do not have a modification time, which is shown as "Jan 1 1970". 37736 37737 File hashes are supported, with a custom Mail.ru algorithm based on SHA1. 37738 If file size is less than or equal to the SHA1 block size (20 bytes), 37739 its hash is simply its data right-padded with zero bytes. 37740 Hashes of a larger file is computed as a SHA1 of the file data 37741 bytes concatenated with a decimal representation of the data length. 37742 37743 ### Emptying Trash 37744 37745 Removing a file or directory actually moves it to the trash, which is not 37746 visible to rclone but can be seen in a web browser. The trashed file 37747 still occupies part of total quota. If you wish to empty your trash 37748 and free some quota, you can use the `rclone cleanup remote:` command, 37749 which will permanently delete all your trashed files. 37750 This command does not take any path arguments. 37751 37752 ### Quota information 37753 37754 To view your current quota you can use the `rclone about remote:` 37755 command which will display your usage limit (quota) and the current usage. 37756 37757 ### Restricted filename characters 37758 37759 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 37760 the following characters are also replaced: 37761 37762 | Character | Value | Replacement | 37763 | --------- |:-----:|:-----------:| 37764 | " | 0x22 | " | 37765 | * | 0x2A | * | 37766 | : | 0x3A | : | 37767 | < | 0x3C | < | 37768 | > | 0x3E | > | 37769 | ? | 0x3F | ? | 37770 | \ | 0x5C | \ | 37771 | \| | 0x7C | | | 37772 37773 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 37774 as they can't be used in JSON strings. 37775 37776 37777 ### Standard options 37778 37779 Here are the Standard options specific to mailru (Mail.ru Cloud). 37780 37781 #### --mailru-client-id 37782 37783 OAuth Client Id. 37784 37785 Leave blank normally. 37786 37787 Properties: 37788 37789 - Config: client_id 37790 - Env Var: RCLONE_MAILRU_CLIENT_ID 37791 - Type: string 37792 - Required: false 37793 37794 #### --mailru-client-secret 37795 37796 OAuth Client Secret. 37797 37798 Leave blank normally. 37799 37800 Properties: 37801 37802 - Config: client_secret 37803 - Env Var: RCLONE_MAILRU_CLIENT_SECRET 37804 - Type: string 37805 - Required: false 37806 37807 #### --mailru-user 37808 37809 User name (usually email). 37810 37811 Properties: 37812 37813 - Config: user 37814 - Env Var: RCLONE_MAILRU_USER 37815 - Type: string 37816 - Required: true 37817 37818 #### --mailru-pass 37819 37820 Password. 37821 37822 This must be an app password - rclone will not work with your normal 37823 password. See the Configuration section in the docs for how to make an 37824 app password. 37825 37826 37827 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 37828 37829 Properties: 37830 37831 - Config: pass 37832 - Env Var: RCLONE_MAILRU_PASS 37833 - Type: string 37834 - Required: true 37835 37836 #### --mailru-speedup-enable 37837 37838 Skip full upload if there is another file with same data hash. 37839 37840 This feature is called "speedup" or "put by hash". It is especially efficient 37841 in case of generally available files like popular books, video or audio clips, 37842 because files are searched by hash in all accounts of all mailru users. 37843 It is meaningless and ineffective if source file is unique or encrypted. 37844 Please note that rclone may need local memory and disk space to calculate 37845 content hash in advance and decide whether full upload is required. 37846 Also, if rclone does not know file size in advance (e.g. in case of 37847 streaming or partial uploads), it will not even try this optimization. 37848 37849 Properties: 37850 37851 - Config: speedup_enable 37852 - Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE 37853 - Type: bool 37854 - Default: true 37855 - Examples: 37856 - "true" 37857 - Enable 37858 - "false" 37859 - Disable 37860 37861 ### Advanced options 37862 37863 Here are the Advanced options specific to mailru (Mail.ru Cloud). 37864 37865 #### --mailru-token 37866 37867 OAuth Access Token as a JSON blob. 37868 37869 Properties: 37870 37871 - Config: token 37872 - Env Var: RCLONE_MAILRU_TOKEN 37873 - Type: string 37874 - Required: false 37875 37876 #### --mailru-auth-url 37877 37878 Auth server URL. 37879 37880 Leave blank to use the provider defaults. 37881 37882 Properties: 37883 37884 - Config: auth_url 37885 - Env Var: RCLONE_MAILRU_AUTH_URL 37886 - Type: string 37887 - Required: false 37888 37889 #### --mailru-token-url 37890 37891 Token server url. 37892 37893 Leave blank to use the provider defaults. 37894 37895 Properties: 37896 37897 - Config: token_url 37898 - Env Var: RCLONE_MAILRU_TOKEN_URL 37899 - Type: string 37900 - Required: false 37901 37902 #### --mailru-speedup-file-patterns 37903 37904 Comma separated list of file name patterns eligible for speedup (put by hash). 37905 37906 Patterns are case insensitive and can contain '*' or '?' meta characters. 37907 37908 Properties: 37909 37910 - Config: speedup_file_patterns 37911 - Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS 37912 - Type: string 37913 - Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf" 37914 - Examples: 37915 - "" 37916 - Empty list completely disables speedup (put by hash). 37917 - "*" 37918 - All files will be attempted for speedup. 37919 - "*.mkv,*.avi,*.mp4,*.mp3" 37920 - Only common audio/video files will be tried for put by hash. 37921 - "*.zip,*.gz,*.rar,*.pdf" 37922 - Only common archives or PDF books will be tried for speedup. 37923 37924 #### --mailru-speedup-max-disk 37925 37926 This option allows you to disable speedup (put by hash) for large files. 37927 37928 Reason is that preliminary hashing can exhaust your RAM or disk space. 37929 37930 Properties: 37931 37932 - Config: speedup_max_disk 37933 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK 37934 - Type: SizeSuffix 37935 - Default: 3Gi 37936 - Examples: 37937 - "0" 37938 - Completely disable speedup (put by hash). 37939 - "1G" 37940 - Files larger than 1Gb will be uploaded directly. 37941 - "3G" 37942 - Choose this option if you have less than 3Gb free on local disk. 37943 37944 #### --mailru-speedup-max-memory 37945 37946 Files larger than the size given below will always be hashed on disk. 37947 37948 Properties: 37949 37950 - Config: speedup_max_memory 37951 - Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY 37952 - Type: SizeSuffix 37953 - Default: 32Mi 37954 - Examples: 37955 - "0" 37956 - Preliminary hashing will always be done in a temporary disk location. 37957 - "32M" 37958 - Do not dedicate more than 32Mb RAM for preliminary hashing. 37959 - "256M" 37960 - You have at most 256Mb RAM free for hash calculations. 37961 37962 #### --mailru-check-hash 37963 37964 What should copy do if file checksum is mismatched or invalid. 37965 37966 Properties: 37967 37968 - Config: check_hash 37969 - Env Var: RCLONE_MAILRU_CHECK_HASH 37970 - Type: bool 37971 - Default: true 37972 - Examples: 37973 - "true" 37974 - Fail with error. 37975 - "false" 37976 - Ignore and continue. 37977 37978 #### --mailru-user-agent 37979 37980 HTTP user agent used internally by client. 37981 37982 Defaults to "rclone/VERSION" or "--user-agent" provided on command line. 37983 37984 Properties: 37985 37986 - Config: user_agent 37987 - Env Var: RCLONE_MAILRU_USER_AGENT 37988 - Type: string 37989 - Required: false 37990 37991 #### --mailru-quirks 37992 37993 Comma separated list of internal maintenance flags. 37994 37995 This option must not be used by an ordinary user. It is intended only to 37996 facilitate remote troubleshooting of backend issues. Strict meaning of 37997 flags is not documented and not guaranteed to persist between releases. 37998 Quirks will be removed when the backend grows stable. 37999 Supported quirks: atomicmkdir binlist unknowndirs 38000 38001 Properties: 38002 38003 - Config: quirks 38004 - Env Var: RCLONE_MAILRU_QUIRKS 38005 - Type: string 38006 - Required: false 38007 38008 #### --mailru-encoding 38009 38010 The encoding for the backend. 38011 38012 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 38013 38014 Properties: 38015 38016 - Config: encoding 38017 - Env Var: RCLONE_MAILRU_ENCODING 38018 - Type: Encoding 38019 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot 38020 38021 #### --mailru-description 38022 38023 Description of the remote 38024 38025 Properties: 38026 38027 - Config: description 38028 - Env Var: RCLONE_MAILRU_DESCRIPTION 38029 - Type: string 38030 - Required: false 38031 38032 38033 38034 ## Limitations 38035 38036 File size limits depend on your account. A single file size is limited by 2G 38037 for a free account and unlimited for paid tariffs. Please refer to the Mail.ru 38038 site for the total uploaded size limits. 38039 38040 Note that Mailru is case insensitive so you can't have a file called 38041 "Hello.doc" and one called "hello.doc". 38042 38043 # Mega 38044 38045 [Mega](https://mega.nz/) is a cloud storage and file hosting service 38046 known for its security feature where all files are encrypted locally 38047 before they are uploaded. This prevents anyone (including employees of 38048 Mega) from accessing the files without knowledge of the key used for 38049 encryption. 38050 38051 This is an rclone backend for Mega which supports the file transfer 38052 features of Mega using the same client side encryption. 38053 38054 Paths are specified as `remote:path` 38055 38056 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 38057 38058 ## Configuration 38059 38060 Here is an example of how to make a remote called `remote`. First run: 38061 38062 rclone config 38063 38064 This will guide you through an interactive setup process: 38065 38066 No remotes found, make a new one? n) New remote s) Set configuration 38067 password q) Quit config n/s/q> n name> remote Type of storage to 38068 configure. Choose a number from below, or type in your own value [snip] 38069 XX / Mega "mega" [snip] Storage> mega User name user> you@example.com 38070 Password. y) Yes type in my own password g) Generate random password n) 38071 No leave this optional password blank y/g/n> y Enter the password: 38072 password: Confirm the password: password: Remote config 38073 -------------------- [remote] type = mega user = you@example.com pass = 38074 *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this 38075 remote d) Delete this remote y/e/d> y 38076 38077 38078 **NOTE:** The encryption keys need to have been already generated after a regular login 38079 via the browser, otherwise attempting to use the credentials in `rclone` will fail. 38080 38081 Once configured you can then use `rclone` like this, 38082 38083 List directories in top level of your Mega 38084 38085 rclone lsd remote: 38086 38087 List all the files in your Mega 38088 38089 rclone ls remote: 38090 38091 To copy a local directory to an Mega directory called backup 38092 38093 rclone copy /home/source remote:backup 38094 38095 ### Modification times and hashes 38096 38097 Mega does not support modification times or hashes yet. 38098 38099 ### Restricted filename characters 38100 38101 | Character | Value | Replacement | 38102 | --------- |:-----:|:-----------:| 38103 | NUL | 0x00 | ␀ | 38104 | / | 0x2F | / | 38105 38106 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 38107 as they can't be used in JSON strings. 38108 38109 ### Duplicated files 38110 38111 Mega can have two files with exactly the same name and path (unlike a 38112 normal file system). 38113 38114 Duplicated files cause problems with the syncing and you will see 38115 messages in the log about duplicates. 38116 38117 Use `rclone dedupe` to fix duplicated files. 38118 38119 ### Failure to log-in 38120 38121 #### Object not found 38122 38123 If you are connecting to your Mega remote for the first time, 38124 to test access and synchronization, you may receive an error such as 38125 38126 Failed to create file system for "my-mega-remote:": couldn't login: 38127 Object (typically, node or user) not found 38128 38129 38130 The diagnostic steps often recommended in the [rclone forum](https://forum.rclone.org/search?q=mega) 38131 start with the **MEGAcmd** utility. Note that this refers to 38132 the official C++ command from https://github.com/meganz/MEGAcmd 38133 and not the go language built command from t3rm1n4l/megacmd 38134 that is no longer maintained. 38135 38136 Follow the instructions for installing MEGAcmd and try accessing 38137 your remote as they recommend. You can establish whether or not 38138 you can log in using MEGAcmd, and obtain diagnostic information 38139 to help you, and search or work with others in the forum. 38140 38141 MEGA CMD> login me@example.com Password: Fetching nodes ... Loading 38142 transfers from local cache Login complete as me@example.com 38143 me@example.com:/$ 38144 38145 38146 Note that some have found issues with passwords containing special 38147 characters. If you can not log on with rclone, but MEGAcmd logs on 38148 just fine, then consider changing your password temporarily to 38149 pure alphanumeric characters, in case that helps. 38150 38151 38152 #### Repeated commands blocks access 38153 38154 Mega remotes seem to get blocked (reject logins) under "heavy use". 38155 We haven't worked out the exact blocking rules but it seems to be 38156 related to fast paced, successive rclone commands. 38157 38158 For example, executing this command 90 times in a row `rclone link 38159 remote:file` will cause the remote to become "blocked". This is not an 38160 abnormal situation, for example if you wish to get the public links of 38161 a directory with hundred of files... After more or less a week, the 38162 remote will remote accept rclone logins normally again. 38163 38164 You can mitigate this issue by mounting the remote it with `rclone 38165 mount`. This will log-in when mounting and a log-out when unmounting 38166 only. You can also run `rclone rcd` and then use `rclone rc` to run 38167 the commands over the API to avoid logging in each time. 38168 38169 Rclone does not currently close mega sessions (you can see them in the 38170 web interface), however closing the sessions does not solve the issue. 38171 38172 If you space rclone commands by 3 seconds it will avoid blocking the 38173 remote. We haven't identified the exact blocking rules, so perhaps one 38174 could execute the command 80 times without waiting and avoid blocking 38175 by waiting 3 seconds, then continuing... 38176 38177 Note that this has been observed by trial and error and might not be 38178 set in stone. 38179 38180 Other tools seem not to produce this blocking effect, as they use a 38181 different working approach (state-based, using sessionIDs instead of 38182 log-in) which isn't compatible with the current stateless rclone 38183 approach. 38184 38185 Note that once blocked, the use of other tools (such as megacmd) is 38186 not a sure workaround: following megacmd login times have been 38187 observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30 38188 min, 30min. Web access looks unaffected though. 38189 38190 Investigation is continuing in relation to workarounds based on 38191 timeouts, pacers, retrials and tpslimits - if you discover something 38192 relevant, please post on the forum. 38193 38194 So, if rclone was working nicely and suddenly you are unable to log-in 38195 and you are sure the user and the password are correct, likely you 38196 have got the remote blocked for a while. 38197 38198 38199 ### Standard options 38200 38201 Here are the Standard options specific to mega (Mega). 38202 38203 #### --mega-user 38204 38205 User name. 38206 38207 Properties: 38208 38209 - Config: user 38210 - Env Var: RCLONE_MEGA_USER 38211 - Type: string 38212 - Required: true 38213 38214 #### --mega-pass 38215 38216 Password. 38217 38218 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 38219 38220 Properties: 38221 38222 - Config: pass 38223 - Env Var: RCLONE_MEGA_PASS 38224 - Type: string 38225 - Required: true 38226 38227 ### Advanced options 38228 38229 Here are the Advanced options specific to mega (Mega). 38230 38231 #### --mega-debug 38232 38233 Output more debug from Mega. 38234 38235 If this flag is set (along with -vv) it will print further debugging 38236 information from the mega backend. 38237 38238 Properties: 38239 38240 - Config: debug 38241 - Env Var: RCLONE_MEGA_DEBUG 38242 - Type: bool 38243 - Default: false 38244 38245 #### --mega-hard-delete 38246 38247 Delete files permanently rather than putting them into the trash. 38248 38249 Normally the mega backend will put all deletions into the trash rather 38250 than permanently deleting them. If you specify this then rclone will 38251 permanently delete objects instead. 38252 38253 Properties: 38254 38255 - Config: hard_delete 38256 - Env Var: RCLONE_MEGA_HARD_DELETE 38257 - Type: bool 38258 - Default: false 38259 38260 #### --mega-use-https 38261 38262 Use HTTPS for transfers. 38263 38264 MEGA uses plain text HTTP connections by default. 38265 Some ISPs throttle HTTP connections, this causes transfers to become very slow. 38266 Enabling this will force MEGA to use HTTPS for all transfers. 38267 HTTPS is normally not necessary since all data is already encrypted anyway. 38268 Enabling it will increase CPU usage and add network overhead. 38269 38270 Properties: 38271 38272 - Config: use_https 38273 - Env Var: RCLONE_MEGA_USE_HTTPS 38274 - Type: bool 38275 - Default: false 38276 38277 #### --mega-encoding 38278 38279 The encoding for the backend. 38280 38281 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 38282 38283 Properties: 38284 38285 - Config: encoding 38286 - Env Var: RCLONE_MEGA_ENCODING 38287 - Type: Encoding 38288 - Default: Slash,InvalidUtf8,Dot 38289 38290 #### --mega-description 38291 38292 Description of the remote 38293 38294 Properties: 38295 38296 - Config: description 38297 - Env Var: RCLONE_MEGA_DESCRIPTION 38298 - Type: string 38299 - Required: false 38300 38301 38302 38303 ### Process `killed` 38304 38305 On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look [at this issue](https://forum.rclone.org/t/rclone-with-mega-appears-to-work-only-in-some-accounts/40233/4). 38306 38307 ## Limitations 38308 38309 This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource 38310 go library implementing the Mega API. There doesn't appear to be any 38311 documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code 38312 so there are likely quite a few errors still remaining in this library. 38313 38314 Mega allows duplicate files which may confuse rclone. 38315 38316 # Memory 38317 38318 The memory backend is an in RAM backend. It does not persist its 38319 data - use the local backend for that. 38320 38321 The memory backend behaves like a bucket-based remote (e.g. like 38322 s3). Because it has no parameters you can just use it with the 38323 `:memory:` remote name. 38324 38325 ## Configuration 38326 38327 You can configure it as a remote like this with `rclone config` too if 38328 you want to: 38329 38330 No remotes found, make a new one? n) New remote s) Set configuration 38331 password q) Quit config n/s/q> n name> remote Type of storage to 38332 configure. Enter a string value. Press Enter for the default (""). 38333 Choose a number from below, or type in your own value [snip] XX / Memory 38334 "memory" [snip] Storage> memory ** See help for memory backend at: 38335 https://rclone.org/memory/ ** 38336 38337 Remote config 38338 38339 --------------- 38340 [remote] 38341 type = memory 38342 --------------- 38343 38344 y) Yes this is OK (default) 38345 z) Edit this remote 38346 a) Delete this remote y/e/d> y 38347 38348 38349 Because the memory backend isn't persistent it is most useful for 38350 testing or with an rclone server or rclone mount, e.g. 38351 38352 rclone mount :memory: /mnt/tmp 38353 rclone serve webdav :memory: 38354 rclone serve sftp :memory: 38355 38356 ### Modification times and hashes 38357 38358 The memory backend supports MD5 hashes and modification times accurate to 1 nS. 38359 38360 ### Restricted filename characters 38361 38362 The memory backend replaces the [default restricted characters 38363 set](https://rclone.org/overview/#restricted-characters). 38364 38365 38366 ### Advanced options 38367 38368 Here are the Advanced options specific to memory (In memory object storage system.). 38369 38370 #### --memory-description 38371 38372 Description of the remote 38373 38374 Properties: 38375 38376 - Config: description 38377 - Env Var: RCLONE_MEMORY_DESCRIPTION 38378 - Type: string 38379 - Required: false 38380 38381 38382 38383 # Akamai NetStorage 38384 38385 Paths are specified as `remote:` 38386 You may put subdirectories in too, e.g. `remote:/path/to/dir`. 38387 If you have a CP code you can use that as the folder after the domain such as \<domain>\/\<cpcode>\/\<internal directories within cpcode>. 38388 38389 For example, this is commonly configured with or without a CP code: 38390 * **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/` 38391 * **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net` 38392 38393 38394 See all buckets 38395 rclone lsd remote: 38396 The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process. 38397 38398 ## Configuration 38399 38400 Here's an example of how to make a remote called `ns1`. 38401 38402 1. To begin the interactive configuration process, enter this command: 38403 38404 rclone config 38405 38406 38407 2. Type `n` to create a new remote. 38408 38409 n) New remote 38410 o) Delete remote 38411 p) Quit config e/n/d/q> n 38412 38413 38414 3. For this example, enter `ns1` when you reach the name> prompt. 38415 38416 name> ns1 38417 38418 38419 4. Enter `netstorage` as the type of storage to configure. 38420 38421 Type of storage to configure. Enter a string value. Press Enter for the 38422 default (""). Choose a number from below, or type in your own value XX / 38423 NetStorage "netstorage" Storage> netstorage 38424 38425 38426 5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes. 38427 38428 Enter a string value. Press Enter for the default (""). Choose a number 38429 from below, or type in your own value 1 / HTTP protocol "http" 2 / 38430 HTTPS protocol "https" protocol> 1 38431 38432 38433 6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `<domain>/<cpcode>/<content>/` 38434 38435 Enter a string value. Press Enter for the default (""). host> 38436 baseball-nsu.akamaihd.net/123456/content/ 38437 38438 38439 7. Set the netstorage account name 38440 38441 Enter a string value. Press Enter for the default (""). account> 38442 username 38443 38444 38445 8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret. 38446 Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption. 38447 38448 y) Yes type in my own password 38449 z) Generate random password y/g> y Enter the password: password: 38450 Confirm the password: password: 38451 38452 38453 9. View the summary and confirm your remote configuration. 38454 38455 [ns1] type = netstorage protocol = http host = 38456 baseball-nsu.akamaihd.net/123456/content/ account = username secret = 38457 *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) 38458 Edit this remote d) Delete this remote y/e/d> y 38459 38460 38461 This remote is called `ns1` and can now be used. 38462 38463 ## Example operations 38464 38465 Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/. 38466 38467 ### See contents of a directory in your project 38468 38469 rclone lsd ns1:/974012/testing/ 38470 38471 ### Sync the contents local with remote 38472 38473 rclone sync . ns1:/974012/testing/ 38474 38475 ### Upload local content to remote 38476 rclone copy notes.txt ns1:/974012/testing/ 38477 38478 ### Delete content on remote 38479 rclone delete ns1:/974012/testing/notes.txt 38480 38481 ### Move or copy content between CP codes. 38482 38483 Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes. 38484 38485 rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/ 38486 38487 ## Features 38488 38489 ### Symlink Support 38490 38491 The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote. 38492 38493 This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below. 38494 38495 Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink. 38496 38497 **Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote. 38498 38499 ### Implicit vs. Explicit Directories 38500 38501 With NetStorage, directories can exist in one of two forms: 38502 38503 1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group. 38504 2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file. 38505 38506 Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly. 38507 38508 ### `--fast-list` / ListR support 38509 38510 NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered. 38511 38512 * **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects. 38513 38514 * **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option. 38515 38516 There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster. 38517 38518 **Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output. 38519 38520 ### Purge 38521 38522 NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method. 38523 38524 **Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible. 38525 38526 38527 ### Standard options 38528 38529 Here are the Standard options specific to netstorage (Akamai NetStorage). 38530 38531 #### --netstorage-host 38532 38533 Domain+path of NetStorage host to connect to. 38534 38535 Format should be `<domain>/<internal folders>` 38536 38537 Properties: 38538 38539 - Config: host 38540 - Env Var: RCLONE_NETSTORAGE_HOST 38541 - Type: string 38542 - Required: true 38543 38544 #### --netstorage-account 38545 38546 Set the NetStorage account name 38547 38548 Properties: 38549 38550 - Config: account 38551 - Env Var: RCLONE_NETSTORAGE_ACCOUNT 38552 - Type: string 38553 - Required: true 38554 38555 #### --netstorage-secret 38556 38557 Set the NetStorage account secret/G2O key for authentication. 38558 38559 Please choose the 'y' option to set your own password then enter your secret. 38560 38561 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 38562 38563 Properties: 38564 38565 - Config: secret 38566 - Env Var: RCLONE_NETSTORAGE_SECRET 38567 - Type: string 38568 - Required: true 38569 38570 ### Advanced options 38571 38572 Here are the Advanced options specific to netstorage (Akamai NetStorage). 38573 38574 #### --netstorage-protocol 38575 38576 Select between HTTP or HTTPS protocol. 38577 38578 Most users should choose HTTPS, which is the default. 38579 HTTP is provided primarily for debugging purposes. 38580 38581 Properties: 38582 38583 - Config: protocol 38584 - Env Var: RCLONE_NETSTORAGE_PROTOCOL 38585 - Type: string 38586 - Default: "https" 38587 - Examples: 38588 - "http" 38589 - HTTP protocol 38590 - "https" 38591 - HTTPS protocol 38592 38593 #### --netstorage-description 38594 38595 Description of the remote 38596 38597 Properties: 38598 38599 - Config: description 38600 - Env Var: RCLONE_NETSTORAGE_DESCRIPTION 38601 - Type: string 38602 - Required: false 38603 38604 ## Backend commands 38605 38606 Here are the commands specific to the netstorage backend. 38607 38608 Run them with 38609 38610 rclone backend COMMAND remote: 38611 38612 The help below will explain what arguments each command takes. 38613 38614 See the [backend](https://rclone.org/commands/rclone_backend/) command for more 38615 info on how to pass options and arguments. 38616 38617 These can be run on a running backend using the rc command 38618 [backend/command](https://rclone.org/rc/#backend-command). 38619 38620 ### du 38621 38622 Return disk usage information for a specified directory 38623 38624 rclone backend du remote: [options] [<arguments>+] 38625 38626 The usage information returned, includes the targeted directory as well as all 38627 files stored in any sub-directories that may exist. 38628 38629 ### symlink 38630 38631 You can create a symbolic link in ObjectStore with the symlink action. 38632 38633 rclone backend symlink remote: [options] [<arguments>+] 38634 38635 The desired path location (including applicable sub-directories) ending in 38636 the object that will be the target of the symlink (for example, /links/mylink). 38637 Include the file extension for the object, if applicable. 38638 `rclone backend symlink <src> <path>` 38639 38640 38641 38642 # Microsoft Azure Blob Storage 38643 38644 Paths are specified as `remote:container` (or `remote:` for the `lsd` 38645 command.) You may put subdirectories in too, e.g. 38646 `remote:container/path/to/dir`. 38647 38648 ## Configuration 38649 38650 Here is an example of making a Microsoft Azure Blob Storage 38651 configuration. For a remote called `remote`. First run: 38652 38653 rclone config 38654 38655 This will guide you through an interactive setup process: 38656 38657 No remotes found, make a new one? n) New remote s) Set configuration 38658 password q) Quit config n/s/q> n name> remote Type of storage to 38659 configure. Choose a number from below, or type in your own value [snip] 38660 XX / Microsoft Azure Blob Storage "azureblob" [snip] Storage> azureblob 38661 Storage Account Name account> account_name Storage Account Key key> 38662 base64encodedkey== Endpoint for the service - leave blank normally. 38663 endpoint> Remote config -------------------- [remote] account = 38664 account_name key = base64encodedkey== endpoint = -------------------- y) 38665 Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y 38666 38667 38668 See all containers 38669 38670 rclone lsd remote: 38671 38672 Make a new container 38673 38674 rclone mkdir remote:container 38675 38676 List the contents of a container 38677 38678 rclone ls remote:container 38679 38680 Sync `/home/local/directory` to the remote container, deleting any excess 38681 files in the container. 38682 38683 rclone sync --interactive /home/local/directory remote:container 38684 38685 ### --fast-list 38686 38687 This remote supports `--fast-list` which allows you to use fewer 38688 transactions in exchange for more memory. See the [rclone 38689 docs](https://rclone.org/docs/#fast-list) for more details. 38690 38691 ### Modification times and hashes 38692 38693 The modification time is stored as metadata on the object with the 38694 `mtime` key. It is stored using RFC3339 Format time with nanosecond 38695 precision. The metadata is supplied during directory listings so 38696 there is no performance overhead to using it. 38697 38698 If you wish to use the Azure standard `LastModified` time stored on 38699 the object as the modified time, then use the `--use-server-modtime` 38700 flag. Note that rclone can't set `LastModified`, so using the 38701 `--update` flag when syncing is recommended if using 38702 `--use-server-modtime`. 38703 38704 MD5 hashes are stored with blobs. However blobs that were uploaded in 38705 chunks only have an MD5 if the source remote was capable of MD5 38706 hashes, e.g. the local disk. 38707 38708 ### Performance 38709 38710 When uploading large files, increasing the value of 38711 `--azureblob-upload-concurrency` will increase performance at the cost 38712 of using more memory. The default of 16 is set quite conservatively to 38713 use less memory. It maybe be necessary raise it to 64 or higher to 38714 fully utilize a 1 GBit/s link with a single file transfer. 38715 38716 ### Restricted filename characters 38717 38718 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 38719 the following characters are also replaced: 38720 38721 | Character | Value | Replacement | 38722 | --------- |:-----:|:-----------:| 38723 | / | 0x2F | / | 38724 | \ | 0x5C | \ | 38725 38726 File names can also not end with the following characters. 38727 These only get replaced if they are the last character in the name: 38728 38729 | Character | Value | Replacement | 38730 | --------- |:-----:|:-----------:| 38731 | . | 0x2E | . | 38732 38733 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 38734 as they can't be used in JSON strings. 38735 38736 ### Authentication {#authentication} 38737 38738 There are a number of ways of supplying credentials for Azure Blob 38739 Storage. Rclone tries them in the order of the sections below. 38740 38741 #### Env Auth 38742 38743 If the `env_auth` config parameter is `true` then rclone will pull 38744 credentials from the environment or runtime. 38745 38746 It tries these authentication methods in this order: 38747 38748 1. Environment Variables 38749 2. Managed Service Identity Credentials 38750 3. Azure CLI credentials (as used by the az tool) 38751 38752 These are described in the following sections 38753 38754 ##### Env Auth: 1. Environment Variables 38755 38756 If `env_auth` is set and environment variables are present rclone 38757 authenticates a service principal with a secret or certificate, or a 38758 user with a password, depending on which environment variable are set. 38759 It reads configuration from these variables, in the following order: 38760 38761 1. Service principal with client secret 38762 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 38763 - `AZURE_CLIENT_ID`: the service principal's client ID 38764 - `AZURE_CLIENT_SECRET`: one of the service principal's client secrets 38765 2. Service principal with certificate 38766 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 38767 - `AZURE_CLIENT_ID`: the service principal's client ID 38768 - `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key. 38769 - `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file. 38770 - `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 38771 3. User with username and password 38772 - `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations". 38773 - `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to 38774 - `AZURE_USERNAME`: a username (usually an email address) 38775 - `AZURE_PASSWORD`: the user's password 38776 4. Workload Identity 38777 - `AZURE_TENANT_ID`: Tenant to authenticate in. 38778 - `AZURE_CLIENT_ID`: Client ID of the application the user will authenticate to. 38779 - `AZURE_FEDERATED_TOKEN_FILE`: Path to projected service account token file. 38780 - `AZURE_AUTHORITY_HOST`: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com). 38781 38782 38783 ##### Env Auth: 2. Managed Service Identity Credentials 38784 38785 When using Managed Service Identity if the VM(SS) on which this 38786 program is running has a system-assigned identity, it will be used by 38787 default. If the resource has no system-assigned but exactly one 38788 user-assigned identity, the user-assigned identity will be used by 38789 default. 38790 38791 If the resource has multiple user-assigned identities you will need to 38792 unset `env_auth` and set `use_msi` instead. See the [`use_msi` 38793 section](#use_msi). 38794 38795 ##### Env Auth: 3. Azure CLI credentials (as used by the az tool) 38796 38797 Credentials created with the `az` tool can be picked up using `env_auth`. 38798 38799 For example if you were to login with a service principal like this: 38800 38801 az login --service-principal -u XXX -p XXX --tenant XXX 38802 38803 Then you could access rclone resources like this: 38804 38805 rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER 38806 38807 Or 38808 38809 rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER 38810 38811 Which is analogous to using the `az` tool: 38812 38813 az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login 38814 38815 #### Account and Shared Key 38816 38817 This is the most straight forward and least flexible way. Just fill 38818 in the `account` and `key` lines and leave the rest blank. 38819 38820 #### SAS URL 38821 38822 This can be an account level SAS URL or container level SAS URL. 38823 38824 To use it leave `account` and `key` blank and fill in `sas_url`. 38825 38826 An account level SAS URL or container level SAS URL can be obtained 38827 from the Azure portal or the Azure Storage Explorer. To get a 38828 container level SAS URL right click on a container in the Azure Blob 38829 explorer in the Azure portal. 38830 38831 If you use a container level SAS URL, rclone operations are permitted 38832 only on a particular container, e.g. 38833 38834 rclone ls azureblob:container 38835 38836 You can also list the single container from the root. This will only 38837 show the container specified by the SAS URL. 38838 38839 $ rclone lsd azureblob: 38840 container/ 38841 38842 Note that you can't see or access any other containers - this will 38843 fail 38844 38845 rclone ls azureblob:othercontainer 38846 38847 Container level SAS URLs are useful for temporarily allowing third 38848 parties access to a single container or putting credentials into an 38849 untrusted environment such as a CI build server. 38850 38851 #### Service principal with client secret 38852 38853 If these variables are set, rclone will authenticate with a service principal with a client secret. 38854 38855 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 38856 - `client_id`: the service principal's client ID 38857 - `client_secret`: one of the service principal's client secrets 38858 38859 The credentials can also be placed in a file using the 38860 `service_principal_file` configuration option. 38861 38862 #### Service principal with certificate 38863 38864 If these variables are set, rclone will authenticate with a service principal with certificate. 38865 38866 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 38867 - `client_id`: the service principal's client ID 38868 - `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key. 38869 - `client_certificate_password`: (optional) password for the certificate file. 38870 - `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 38871 38872 **NB** `client_certificate_password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 38873 38874 #### User with username and password 38875 38876 If these variables are set, rclone will authenticate with username and password. 38877 38878 - `tenant`: (optional) tenant to authenticate in. Defaults to "organizations". 38879 - `client_id`: client ID of the application the user will authenticate to 38880 - `username`: a username (usually an email address) 38881 - `password`: the user's password 38882 38883 Microsoft doesn't recommend this kind of authentication, because it's 38884 less secure than other authentication flows. This method is not 38885 interactive, so it isn't compatible with any form of multi-factor 38886 authentication, and the application must already have user or admin 38887 consent. This credential can only authenticate work and school 38888 accounts; it can't authenticate Microsoft accounts. 38889 38890 **NB** `password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 38891 38892 #### Managed Service Identity Credentials {#use_msi} 38893 38894 If `use_msi` is set then managed service identity credentials are 38895 used. This authentication only works when running in an Azure service. 38896 `env_auth` needs to be unset to use this. 38897 38898 However if you have multiple user identities to choose from these must 38899 be explicitly specified using exactly one of the `msi_object_id`, 38900 `msi_client_id`, or `msi_mi_res_id` parameters. 38901 38902 If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is 38903 set, this is is equivalent to using `env_auth`. 38904 38905 38906 ### Standard options 38907 38908 Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage). 38909 38910 #### --azureblob-account 38911 38912 Azure Storage Account Name. 38913 38914 Set this to the Azure Storage Account Name in use. 38915 38916 Leave blank to use SAS URL or Emulator, otherwise it needs to be set. 38917 38918 If this is blank and if env_auth is set it will be read from the 38919 environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible. 38920 38921 38922 Properties: 38923 38924 - Config: account 38925 - Env Var: RCLONE_AZUREBLOB_ACCOUNT 38926 - Type: string 38927 - Required: false 38928 38929 #### --azureblob-env-auth 38930 38931 Read credentials from runtime (environment variables, CLI or MSI). 38932 38933 See the [authentication docs](/azureblob#authentication) for full info. 38934 38935 Properties: 38936 38937 - Config: env_auth 38938 - Env Var: RCLONE_AZUREBLOB_ENV_AUTH 38939 - Type: bool 38940 - Default: false 38941 38942 #### --azureblob-key 38943 38944 Storage Account Shared Key. 38945 38946 Leave blank to use SAS URL or Emulator. 38947 38948 Properties: 38949 38950 - Config: key 38951 - Env Var: RCLONE_AZUREBLOB_KEY 38952 - Type: string 38953 - Required: false 38954 38955 #### --azureblob-sas-url 38956 38957 SAS URL for container level access only. 38958 38959 Leave blank if using account/key or Emulator. 38960 38961 Properties: 38962 38963 - Config: sas_url 38964 - Env Var: RCLONE_AZUREBLOB_SAS_URL 38965 - Type: string 38966 - Required: false 38967 38968 #### --azureblob-tenant 38969 38970 ID of the service principal's tenant. Also called its directory ID. 38971 38972 Set this if using 38973 - Service principal with client secret 38974 - Service principal with certificate 38975 - User with username and password 38976 38977 38978 Properties: 38979 38980 - Config: tenant 38981 - Env Var: RCLONE_AZUREBLOB_TENANT 38982 - Type: string 38983 - Required: false 38984 38985 #### --azureblob-client-id 38986 38987 The ID of the client in use. 38988 38989 Set this if using 38990 - Service principal with client secret 38991 - Service principal with certificate 38992 - User with username and password 38993 38994 38995 Properties: 38996 38997 - Config: client_id 38998 - Env Var: RCLONE_AZUREBLOB_CLIENT_ID 38999 - Type: string 39000 - Required: false 39001 39002 #### --azureblob-client-secret 39003 39004 One of the service principal's client secrets 39005 39006 Set this if using 39007 - Service principal with client secret 39008 39009 39010 Properties: 39011 39012 - Config: client_secret 39013 - Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET 39014 - Type: string 39015 - Required: false 39016 39017 #### --azureblob-client-certificate-path 39018 39019 Path to a PEM or PKCS12 certificate file including the private key. 39020 39021 Set this if using 39022 - Service principal with certificate 39023 39024 39025 Properties: 39026 39027 - Config: client_certificate_path 39028 - Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH 39029 - Type: string 39030 - Required: false 39031 39032 #### --azureblob-client-certificate-password 39033 39034 Password for the certificate file (optional). 39035 39036 Optionally set this if using 39037 - Service principal with certificate 39038 39039 And the certificate has a password. 39040 39041 39042 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39043 39044 Properties: 39045 39046 - Config: client_certificate_password 39047 - Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD 39048 - Type: string 39049 - Required: false 39050 39051 ### Advanced options 39052 39053 Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage). 39054 39055 #### --azureblob-client-send-certificate-chain 39056 39057 Send the certificate chain when using certificate auth. 39058 39059 Specifies whether an authentication request will include an x5c header 39060 to support subject name / issuer based authentication. When set to 39061 true, authentication requests include the x5c header. 39062 39063 Optionally set this if using 39064 - Service principal with certificate 39065 39066 39067 Properties: 39068 39069 - Config: client_send_certificate_chain 39070 - Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN 39071 - Type: bool 39072 - Default: false 39073 39074 #### --azureblob-username 39075 39076 User name (usually an email address) 39077 39078 Set this if using 39079 - User with username and password 39080 39081 39082 Properties: 39083 39084 - Config: username 39085 - Env Var: RCLONE_AZUREBLOB_USERNAME 39086 - Type: string 39087 - Required: false 39088 39089 #### --azureblob-password 39090 39091 The user's password 39092 39093 Set this if using 39094 - User with username and password 39095 39096 39097 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39098 39099 Properties: 39100 39101 - Config: password 39102 - Env Var: RCLONE_AZUREBLOB_PASSWORD 39103 - Type: string 39104 - Required: false 39105 39106 #### --azureblob-service-principal-file 39107 39108 Path to file containing credentials for use with a service principal. 39109 39110 Leave blank normally. Needed only if you want to use a service principal instead of interactive login. 39111 39112 $ az ad sp create-for-rbac --name "<name>" \ 39113 --role "Storage Blob Data Owner" \ 39114 --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \ 39115 > azure-principal.json 39116 39117 See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details. 39118 39119 It may be more convenient to put the credentials directly into the 39120 rclone config file under the `client_id`, `tenant` and `client_secret` 39121 keys instead of setting `service_principal_file`. 39122 39123 39124 Properties: 39125 39126 - Config: service_principal_file 39127 - Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE 39128 - Type: string 39129 - Required: false 39130 39131 #### --azureblob-use-msi 39132 39133 Use a managed service identity to authenticate (only works in Azure). 39134 39135 When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/) 39136 to authenticate to Azure Storage instead of a SAS token or account key. 39137 39138 If the VM(SS) on which this program is running has a system-assigned identity, it will 39139 be used by default. If the resource has no system-assigned but exactly one user-assigned identity, 39140 the user-assigned identity will be used by default. If the resource has multiple user-assigned 39141 identities, the identity to use must be explicitly specified using exactly one of the msi_object_id, 39142 msi_client_id, or msi_mi_res_id parameters. 39143 39144 Properties: 39145 39146 - Config: use_msi 39147 - Env Var: RCLONE_AZUREBLOB_USE_MSI 39148 - Type: bool 39149 - Default: false 39150 39151 #### --azureblob-msi-object-id 39152 39153 Object ID of the user-assigned MSI to use, if any. 39154 39155 Leave blank if msi_client_id or msi_mi_res_id specified. 39156 39157 Properties: 39158 39159 - Config: msi_object_id 39160 - Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID 39161 - Type: string 39162 - Required: false 39163 39164 #### --azureblob-msi-client-id 39165 39166 Object ID of the user-assigned MSI to use, if any. 39167 39168 Leave blank if msi_object_id or msi_mi_res_id specified. 39169 39170 Properties: 39171 39172 - Config: msi_client_id 39173 - Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID 39174 - Type: string 39175 - Required: false 39176 39177 #### --azureblob-msi-mi-res-id 39178 39179 Azure resource ID of the user-assigned MSI to use, if any. 39180 39181 Leave blank if msi_client_id or msi_object_id specified. 39182 39183 Properties: 39184 39185 - Config: msi_mi_res_id 39186 - Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID 39187 - Type: string 39188 - Required: false 39189 39190 #### --azureblob-use-emulator 39191 39192 Uses local storage emulator if provided as 'true'. 39193 39194 Leave blank if using real azure storage endpoint. 39195 39196 Properties: 39197 39198 - Config: use_emulator 39199 - Env Var: RCLONE_AZUREBLOB_USE_EMULATOR 39200 - Type: bool 39201 - Default: false 39202 39203 #### --azureblob-endpoint 39204 39205 Endpoint for the service. 39206 39207 Leave blank normally. 39208 39209 Properties: 39210 39211 - Config: endpoint 39212 - Env Var: RCLONE_AZUREBLOB_ENDPOINT 39213 - Type: string 39214 - Required: false 39215 39216 #### --azureblob-upload-cutoff 39217 39218 Cutoff for switching to chunked upload (<= 256 MiB) (deprecated). 39219 39220 Properties: 39221 39222 - Config: upload_cutoff 39223 - Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF 39224 - Type: string 39225 - Required: false 39226 39227 #### --azureblob-chunk-size 39228 39229 Upload chunk size. 39230 39231 Note that this is stored in memory and there may be up to 39232 "--transfers" * "--azureblob-upload-concurrency" chunks stored at once 39233 in memory. 39234 39235 Properties: 39236 39237 - Config: chunk_size 39238 - Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE 39239 - Type: SizeSuffix 39240 - Default: 4Mi 39241 39242 #### --azureblob-upload-concurrency 39243 39244 Concurrency for multipart uploads. 39245 39246 This is the number of chunks of the same file that are uploaded 39247 concurrently. 39248 39249 If you are uploading small numbers of large files over high-speed 39250 links and these uploads do not fully utilize your bandwidth, then 39251 increasing this may help to speed up the transfers. 39252 39253 In tests, upload speed increases almost linearly with upload 39254 concurrency. For example to fill a gigabit pipe it may be necessary to 39255 raise this to 64. Note that this will use more memory. 39256 39257 Note that chunks are stored in memory and there may be up to 39258 "--transfers" * "--azureblob-upload-concurrency" chunks stored at once 39259 in memory. 39260 39261 Properties: 39262 39263 - Config: upload_concurrency 39264 - Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY 39265 - Type: int 39266 - Default: 16 39267 39268 #### --azureblob-list-chunk 39269 39270 Size of blob list. 39271 39272 This sets the number of blobs requested in each listing chunk. Default 39273 is the maximum, 5000. "List blobs" requests are permitted 2 minutes 39274 per megabyte to complete. If an operation is taking longer than 2 39275 minutes per megabyte on average, it will time out ( 39276 [source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval) 39277 ). This can be used to limit the number of blobs items to return, to 39278 avoid the time out. 39279 39280 Properties: 39281 39282 - Config: list_chunk 39283 - Env Var: RCLONE_AZUREBLOB_LIST_CHUNK 39284 - Type: int 39285 - Default: 5000 39286 39287 #### --azureblob-access-tier 39288 39289 Access tier of blob: hot, cool, cold or archive. 39290 39291 Archived blobs can be restored by setting access tier to hot, cool or 39292 cold. Leave blank if you intend to use default access tier, which is 39293 set at account level 39294 39295 If there is no "access tier" specified, rclone doesn't apply any tier. 39296 rclone performs "Set Tier" operation on blobs while uploading, if objects 39297 are not modified, specifying "access tier" to new one will have no effect. 39298 If blobs are in "archive tier" at remote, trying to perform data transfer 39299 operations from remote will not be allowed. User should first restore by 39300 tiering blob to "Hot", "Cool" or "Cold". 39301 39302 Properties: 39303 39304 - Config: access_tier 39305 - Env Var: RCLONE_AZUREBLOB_ACCESS_TIER 39306 - Type: string 39307 - Required: false 39308 39309 #### --azureblob-archive-tier-delete 39310 39311 Delete archive tier blobs before overwriting. 39312 39313 Archive tier blobs cannot be updated. So without this flag, if you 39314 attempt to update an archive tier blob, then rclone will produce the 39315 error: 39316 39317 can't update archive tier blob without --azureblob-archive-tier-delete 39318 39319 With this flag set then before rclone attempts to overwrite an archive 39320 tier blob, it will delete the existing blob before uploading its 39321 replacement. This has the potential for data loss if the upload fails 39322 (unlike updating a normal blob) and also may cost more since deleting 39323 archive tier blobs early may be chargable. 39324 39325 39326 Properties: 39327 39328 - Config: archive_tier_delete 39329 - Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE 39330 - Type: bool 39331 - Default: false 39332 39333 #### --azureblob-disable-checksum 39334 39335 Don't store MD5 checksum with object metadata. 39336 39337 Normally rclone will calculate the MD5 checksum of the input before 39338 uploading it so it can add it to metadata on the object. This is great 39339 for data integrity checking but can cause long delays for large files 39340 to start uploading. 39341 39342 Properties: 39343 39344 - Config: disable_checksum 39345 - Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM 39346 - Type: bool 39347 - Default: false 39348 39349 #### --azureblob-memory-pool-flush-time 39350 39351 How often internal memory buffer pools will be flushed. (no longer used) 39352 39353 Properties: 39354 39355 - Config: memory_pool_flush_time 39356 - Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME 39357 - Type: Duration 39358 - Default: 1m0s 39359 39360 #### --azureblob-memory-pool-use-mmap 39361 39362 Whether to use mmap buffers in internal memory pool. (no longer used) 39363 39364 Properties: 39365 39366 - Config: memory_pool_use_mmap 39367 - Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP 39368 - Type: bool 39369 - Default: false 39370 39371 #### --azureblob-encoding 39372 39373 The encoding for the backend. 39374 39375 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 39376 39377 Properties: 39378 39379 - Config: encoding 39380 - Env Var: RCLONE_AZUREBLOB_ENCODING 39381 - Type: Encoding 39382 - Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8 39383 39384 #### --azureblob-public-access 39385 39386 Public access level of a container: blob or container. 39387 39388 Properties: 39389 39390 - Config: public_access 39391 - Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS 39392 - Type: string 39393 - Required: false 39394 - Examples: 39395 - "" 39396 - The container and its blobs can be accessed only with an authorized request. 39397 - It's a default value. 39398 - "blob" 39399 - Blob data within this container can be read via anonymous request. 39400 - "container" 39401 - Allow full public read access for container and blob data. 39402 39403 #### --azureblob-directory-markers 39404 39405 Upload an empty object with a trailing slash when a new directory is created 39406 39407 Empty folders are unsupported for bucket based remotes, this option 39408 creates an empty object ending with "/", to persist the folder. 39409 39410 This object also has the metadata "hdi_isfolder = true" to conform to 39411 the Microsoft standard. 39412 39413 39414 Properties: 39415 39416 - Config: directory_markers 39417 - Env Var: RCLONE_AZUREBLOB_DIRECTORY_MARKERS 39418 - Type: bool 39419 - Default: false 39420 39421 #### --azureblob-no-check-container 39422 39423 If set, don't attempt to check the container exists or create it. 39424 39425 This can be useful when trying to minimise the number of transactions 39426 rclone does if you know the container exists already. 39427 39428 39429 Properties: 39430 39431 - Config: no_check_container 39432 - Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER 39433 - Type: bool 39434 - Default: false 39435 39436 #### --azureblob-no-head-object 39437 39438 If set, do not do HEAD before GET when getting objects. 39439 39440 Properties: 39441 39442 - Config: no_head_object 39443 - Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT 39444 - Type: bool 39445 - Default: false 39446 39447 #### --azureblob-delete-snapshots 39448 39449 Set to specify how to deal with snapshots on blob deletion. 39450 39451 Properties: 39452 39453 - Config: delete_snapshots 39454 - Env Var: RCLONE_AZUREBLOB_DELETE_SNAPSHOTS 39455 - Type: string 39456 - Required: false 39457 - Choices: 39458 - "" 39459 - By default, the delete operation fails if a blob has snapshots 39460 - "include" 39461 - Specify 'include' to remove the root blob and all its snapshots 39462 - "only" 39463 - Specify 'only' to remove only the snapshots but keep the root blob. 39464 39465 #### --azureblob-description 39466 39467 Description of the remote 39468 39469 Properties: 39470 39471 - Config: description 39472 - Env Var: RCLONE_AZUREBLOB_DESCRIPTION 39473 - Type: string 39474 - Required: false 39475 39476 39477 39478 ### Custom upload headers 39479 39480 You can set custom upload headers with the `--header-upload` flag. 39481 39482 - Cache-Control 39483 - Content-Disposition 39484 - Content-Encoding 39485 - Content-Language 39486 - Content-Type 39487 39488 Eg `--header-upload "Content-Type: text/potato"` 39489 39490 ## Limitations 39491 39492 MD5 sums are only uploaded with chunked files if the source has an MD5 39493 sum. This will always be the case for a local to azure copy. 39494 39495 `rclone about` is not supported by the Microsoft Azure Blob storage backend. Backends without 39496 this capability cannot determine free space for an rclone mount or 39497 use policy `mfs` (most free space) as a member of an rclone union 39498 remote. 39499 39500 See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/) 39501 39502 ## Azure Storage Emulator Support 39503 39504 You can run rclone with the storage emulator (usually _azurite_). 39505 39506 To do this, just set up a new remote with `rclone config` following 39507 the instructions in the introduction and set `use_emulator` in the 39508 advanced settings as `true`. You do not need to provide a default 39509 account name nor an account key. But you can override them in the 39510 `account` and `key` options. (Prior to v1.61 they were hard coded to 39511 _azurite_'s `devstoreaccount1`.) 39512 39513 Also, if you want to access a storage emulator instance running on a 39514 different machine, you can override the `endpoint` parameter in the 39515 advanced settings, setting it to 39516 `http(s)://<host>:<port>/devstoreaccount1` 39517 (e.g. `http://10.254.2.5:10000/devstoreaccount1`). 39518 39519 # Microsoft Azure Files Storage 39520 39521 Paths are specified as `remote:` You may put subdirectories in too, 39522 e.g. `remote:path/to/dir`. 39523 39524 ## Configuration 39525 39526 Here is an example of making a Microsoft Azure Files Storage 39527 configuration. For a remote called `remote`. First run: 39528 39529 rclone config 39530 39531 This will guide you through an interactive setup process: 39532 39533 No remotes found, make a new one? n) New remote s) Set configuration 39534 password q) Quit config n/s/q> n name> remote Type of storage to 39535 configure. Choose a number from below, or type in your own value [snip] 39536 XX / Microsoft Azure Files Storage "azurefiles" [snip] 39537 39538 Option account. Azure Storage Account Name. Set this to the Azure 39539 Storage Account Name in use. Leave blank to use SAS URL or connection 39540 string, otherwise it needs to be set. If this is blank and if env_auth 39541 is set it will be read from the environment variable 39542 AZURE_STORAGE_ACCOUNT_NAME if possible. Enter a value. Press Enter to 39543 leave empty. account> account_name 39544 39545 Option share_name. Azure Files Share Name. This is required and is the 39546 name of the share to access. Enter a value. Press Enter to leave empty. 39547 share_name> share_name 39548 39549 Option env_auth. Read credentials from runtime (environment variables, 39550 CLI or MSI). See the authentication docs for full info. Enter a boolean 39551 value (true or false). Press Enter for the default (false). env_auth> 39552 39553 Option key. Storage Account Shared Key. Leave blank to use SAS URL or 39554 connection string. Enter a value. Press Enter to leave empty. key> 39555 base64encodedkey== 39556 39557 Option sas_url. SAS URL. Leave blank if using account/key or connection 39558 string. Enter a value. Press Enter to leave empty. sas_url> 39559 39560 Option connection_string. Azure Files Connection String. Enter a value. 39561 Press Enter to leave empty. connection_string> [snip] 39562 39563 Configuration complete. Options: - type: azurefiles - account: 39564 account_name - share_name: share_name - key: base64encodedkey== Keep 39565 this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) 39566 Delete this remote y/e/d> 39567 39568 39569 Once configured you can use rclone. 39570 39571 See all files in the top level: 39572 39573 rclone lsf remote: 39574 39575 Make a new directory in the root: 39576 39577 rclone mkdir remote:dir 39578 39579 Recursively List the contents: 39580 39581 rclone ls remote: 39582 39583 Sync `/home/local/directory` to the remote directory, deleting any 39584 excess files in the directory. 39585 39586 rclone sync --interactive /home/local/directory remote:dir 39587 39588 ### Modified time 39589 39590 The modified time is stored as Azure standard `LastModified` time on 39591 files 39592 39593 ### Performance 39594 39595 When uploading large files, increasing the value of 39596 `--azurefiles-upload-concurrency` will increase performance at the cost 39597 of using more memory. The default of 16 is set quite conservatively to 39598 use less memory. It maybe be necessary raise it to 64 or higher to 39599 fully utilize a 1 GBit/s link with a single file transfer. 39600 39601 ### Restricted filename characters 39602 39603 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 39604 the following characters are also replaced: 39605 39606 | Character | Value | Replacement | 39607 | --------- |:-----:|:-----------:| 39608 | " | 0x22 | " | 39609 | * | 0x2A | * | 39610 | : | 0x3A | : | 39611 | < | 0x3C | < | 39612 | > | 0x3E | > | 39613 | ? | 0x3F | ? | 39614 | \ | 0x5C | \ | 39615 | \| | 0x7C | | | 39616 39617 File names can also not end with the following characters. 39618 These only get replaced if they are the last character in the name: 39619 39620 | Character | Value | Replacement | 39621 | --------- |:-----:|:-----------:| 39622 | . | 0x2E | . | 39623 39624 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 39625 as they can't be used in JSON strings. 39626 39627 ### Hashes 39628 39629 MD5 hashes are stored with files. Not all files will have MD5 hashes 39630 as these have to be uploaded with the file. 39631 39632 ### Authentication {#authentication} 39633 39634 There are a number of ways of supplying credentials for Azure Files 39635 Storage. Rclone tries them in the order of the sections below. 39636 39637 #### Env Auth 39638 39639 If the `env_auth` config parameter is `true` then rclone will pull 39640 credentials from the environment or runtime. 39641 39642 It tries these authentication methods in this order: 39643 39644 1. Environment Variables 39645 2. Managed Service Identity Credentials 39646 3. Azure CLI credentials (as used by the az tool) 39647 39648 These are described in the following sections 39649 39650 ##### Env Auth: 1. Environment Variables 39651 39652 If `env_auth` is set and environment variables are present rclone 39653 authenticates a service principal with a secret or certificate, or a 39654 user with a password, depending on which environment variable are set. 39655 It reads configuration from these variables, in the following order: 39656 39657 1. Service principal with client secret 39658 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 39659 - `AZURE_CLIENT_ID`: the service principal's client ID 39660 - `AZURE_CLIENT_SECRET`: one of the service principal's client secrets 39661 2. Service principal with certificate 39662 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 39663 - `AZURE_CLIENT_ID`: the service principal's client ID 39664 - `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key. 39665 - `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file. 39666 - `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 39667 3. User with username and password 39668 - `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations". 39669 - `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to 39670 - `AZURE_USERNAME`: a username (usually an email address) 39671 - `AZURE_PASSWORD`: the user's password 39672 4. Workload Identity 39673 - `AZURE_TENANT_ID`: Tenant to authenticate in. 39674 - `AZURE_CLIENT_ID`: Client ID of the application the user will authenticate to. 39675 - `AZURE_FEDERATED_TOKEN_FILE`: Path to projected service account token file. 39676 - `AZURE_AUTHORITY_HOST`: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com). 39677 39678 39679 ##### Env Auth: 2. Managed Service Identity Credentials 39680 39681 When using Managed Service Identity if the VM(SS) on which this 39682 program is running has a system-assigned identity, it will be used by 39683 default. If the resource has no system-assigned but exactly one 39684 user-assigned identity, the user-assigned identity will be used by 39685 default. 39686 39687 If the resource has multiple user-assigned identities you will need to 39688 unset `env_auth` and set `use_msi` instead. See the [`use_msi` 39689 section](#use_msi). 39690 39691 ##### Env Auth: 3. Azure CLI credentials (as used by the az tool) 39692 39693 Credentials created with the `az` tool can be picked up using `env_auth`. 39694 39695 For example if you were to login with a service principal like this: 39696 39697 az login --service-principal -u XXX -p XXX --tenant XXX 39698 39699 Then you could access rclone resources like this: 39700 39701 rclone lsf :azurefiles,env_auth,account=ACCOUNT: 39702 39703 Or 39704 39705 rclone lsf --azurefiles-env-auth --azurefiles-account=ACCOUNT :azurefiles: 39706 39707 #### Account and Shared Key 39708 39709 This is the most straight forward and least flexible way. Just fill 39710 in the `account` and `key` lines and leave the rest blank. 39711 39712 #### SAS URL 39713 39714 To use it leave `account`, `key` and `connection_string` blank and fill in `sas_url`. 39715 39716 #### Connection String 39717 39718 To use it leave `account`, `key` and "sas_url" blank and fill in `connection_string`. 39719 39720 #### Service principal with client secret 39721 39722 If these variables are set, rclone will authenticate with a service principal with a client secret. 39723 39724 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 39725 - `client_id`: the service principal's client ID 39726 - `client_secret`: one of the service principal's client secrets 39727 39728 The credentials can also be placed in a file using the 39729 `service_principal_file` configuration option. 39730 39731 #### Service principal with certificate 39732 39733 If these variables are set, rclone will authenticate with a service principal with certificate. 39734 39735 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 39736 - `client_id`: the service principal's client ID 39737 - `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key. 39738 - `client_certificate_password`: (optional) password for the certificate file. 39739 - `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 39740 39741 **NB** `client_certificate_password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39742 39743 #### User with username and password 39744 39745 If these variables are set, rclone will authenticate with username and password. 39746 39747 - `tenant`: (optional) tenant to authenticate in. Defaults to "organizations". 39748 - `client_id`: client ID of the application the user will authenticate to 39749 - `username`: a username (usually an email address) 39750 - `password`: the user's password 39751 39752 Microsoft doesn't recommend this kind of authentication, because it's 39753 less secure than other authentication flows. This method is not 39754 interactive, so it isn't compatible with any form of multi-factor 39755 authentication, and the application must already have user or admin 39756 consent. This credential can only authenticate work and school 39757 accounts; it can't authenticate Microsoft accounts. 39758 39759 **NB** `password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39760 39761 #### Managed Service Identity Credentials {#use_msi} 39762 39763 If `use_msi` is set then managed service identity credentials are 39764 used. This authentication only works when running in an Azure service. 39765 `env_auth` needs to be unset to use this. 39766 39767 However if you have multiple user identities to choose from these must 39768 be explicitly specified using exactly one of the `msi_object_id`, 39769 `msi_client_id`, or `msi_mi_res_id` parameters. 39770 39771 If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is 39772 set, this is is equivalent to using `env_auth`. 39773 39774 39775 ### Standard options 39776 39777 Here are the Standard options specific to azurefiles (Microsoft Azure Files). 39778 39779 #### --azurefiles-account 39780 39781 Azure Storage Account Name. 39782 39783 Set this to the Azure Storage Account Name in use. 39784 39785 Leave blank to use SAS URL or connection string, otherwise it needs to be set. 39786 39787 If this is blank and if env_auth is set it will be read from the 39788 environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible. 39789 39790 39791 Properties: 39792 39793 - Config: account 39794 - Env Var: RCLONE_AZUREFILES_ACCOUNT 39795 - Type: string 39796 - Required: false 39797 39798 #### --azurefiles-share-name 39799 39800 Azure Files Share Name. 39801 39802 This is required and is the name of the share to access. 39803 39804 39805 Properties: 39806 39807 - Config: share_name 39808 - Env Var: RCLONE_AZUREFILES_SHARE_NAME 39809 - Type: string 39810 - Required: false 39811 39812 #### --azurefiles-env-auth 39813 39814 Read credentials from runtime (environment variables, CLI or MSI). 39815 39816 See the [authentication docs](/azurefiles#authentication) for full info. 39817 39818 Properties: 39819 39820 - Config: env_auth 39821 - Env Var: RCLONE_AZUREFILES_ENV_AUTH 39822 - Type: bool 39823 - Default: false 39824 39825 #### --azurefiles-key 39826 39827 Storage Account Shared Key. 39828 39829 Leave blank to use SAS URL or connection string. 39830 39831 Properties: 39832 39833 - Config: key 39834 - Env Var: RCLONE_AZUREFILES_KEY 39835 - Type: string 39836 - Required: false 39837 39838 #### --azurefiles-sas-url 39839 39840 SAS URL. 39841 39842 Leave blank if using account/key or connection string. 39843 39844 Properties: 39845 39846 - Config: sas_url 39847 - Env Var: RCLONE_AZUREFILES_SAS_URL 39848 - Type: string 39849 - Required: false 39850 39851 #### --azurefiles-connection-string 39852 39853 Azure Files Connection String. 39854 39855 Properties: 39856 39857 - Config: connection_string 39858 - Env Var: RCLONE_AZUREFILES_CONNECTION_STRING 39859 - Type: string 39860 - Required: false 39861 39862 #### --azurefiles-tenant 39863 39864 ID of the service principal's tenant. Also called its directory ID. 39865 39866 Set this if using 39867 - Service principal with client secret 39868 - Service principal with certificate 39869 - User with username and password 39870 39871 39872 Properties: 39873 39874 - Config: tenant 39875 - Env Var: RCLONE_AZUREFILES_TENANT 39876 - Type: string 39877 - Required: false 39878 39879 #### --azurefiles-client-id 39880 39881 The ID of the client in use. 39882 39883 Set this if using 39884 - Service principal with client secret 39885 - Service principal with certificate 39886 - User with username and password 39887 39888 39889 Properties: 39890 39891 - Config: client_id 39892 - Env Var: RCLONE_AZUREFILES_CLIENT_ID 39893 - Type: string 39894 - Required: false 39895 39896 #### --azurefiles-client-secret 39897 39898 One of the service principal's client secrets 39899 39900 Set this if using 39901 - Service principal with client secret 39902 39903 39904 Properties: 39905 39906 - Config: client_secret 39907 - Env Var: RCLONE_AZUREFILES_CLIENT_SECRET 39908 - Type: string 39909 - Required: false 39910 39911 #### --azurefiles-client-certificate-path 39912 39913 Path to a PEM or PKCS12 certificate file including the private key. 39914 39915 Set this if using 39916 - Service principal with certificate 39917 39918 39919 Properties: 39920 39921 - Config: client_certificate_path 39922 - Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PATH 39923 - Type: string 39924 - Required: false 39925 39926 #### --azurefiles-client-certificate-password 39927 39928 Password for the certificate file (optional). 39929 39930 Optionally set this if using 39931 - Service principal with certificate 39932 39933 And the certificate has a password. 39934 39935 39936 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39937 39938 Properties: 39939 39940 - Config: client_certificate_password 39941 - Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PASSWORD 39942 - Type: string 39943 - Required: false 39944 39945 ### Advanced options 39946 39947 Here are the Advanced options specific to azurefiles (Microsoft Azure Files). 39948 39949 #### --azurefiles-client-send-certificate-chain 39950 39951 Send the certificate chain when using certificate auth. 39952 39953 Specifies whether an authentication request will include an x5c header 39954 to support subject name / issuer based authentication. When set to 39955 true, authentication requests include the x5c header. 39956 39957 Optionally set this if using 39958 - Service principal with certificate 39959 39960 39961 Properties: 39962 39963 - Config: client_send_certificate_chain 39964 - Env Var: RCLONE_AZUREFILES_CLIENT_SEND_CERTIFICATE_CHAIN 39965 - Type: bool 39966 - Default: false 39967 39968 #### --azurefiles-username 39969 39970 User name (usually an email address) 39971 39972 Set this if using 39973 - User with username and password 39974 39975 39976 Properties: 39977 39978 - Config: username 39979 - Env Var: RCLONE_AZUREFILES_USERNAME 39980 - Type: string 39981 - Required: false 39982 39983 #### --azurefiles-password 39984 39985 The user's password 39986 39987 Set this if using 39988 - User with username and password 39989 39990 39991 **NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/). 39992 39993 Properties: 39994 39995 - Config: password 39996 - Env Var: RCLONE_AZUREFILES_PASSWORD 39997 - Type: string 39998 - Required: false 39999 40000 #### --azurefiles-service-principal-file 40001 40002 Path to file containing credentials for use with a service principal. 40003 40004 Leave blank normally. Needed only if you want to use a service principal instead of interactive login. 40005 40006 $ az ad sp create-for-rbac --name "<name>" \ 40007 --role "Storage Files Data Owner" \ 40008 --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \ 40009 > azure-principal.json 40010 40011 See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details. 40012 40013 **NB** this section needs updating for Azure Files - pull requests appreciated! 40014 40015 It may be more convenient to put the credentials directly into the 40016 rclone config file under the `client_id`, `tenant` and `client_secret` 40017 keys instead of setting `service_principal_file`. 40018 40019 40020 Properties: 40021 40022 - Config: service_principal_file 40023 - Env Var: RCLONE_AZUREFILES_SERVICE_PRINCIPAL_FILE 40024 - Type: string 40025 - Required: false 40026 40027 #### --azurefiles-use-msi 40028 40029 Use a managed service identity to authenticate (only works in Azure). 40030 40031 When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/) 40032 to authenticate to Azure Storage instead of a SAS token or account key. 40033 40034 If the VM(SS) on which this program is running has a system-assigned identity, it will 40035 be used by default. If the resource has no system-assigned but exactly one user-assigned identity, 40036 the user-assigned identity will be used by default. If the resource has multiple user-assigned 40037 identities, the identity to use must be explicitly specified using exactly one of the msi_object_id, 40038 msi_client_id, or msi_mi_res_id parameters. 40039 40040 Properties: 40041 40042 - Config: use_msi 40043 - Env Var: RCLONE_AZUREFILES_USE_MSI 40044 - Type: bool 40045 - Default: false 40046 40047 #### --azurefiles-msi-object-id 40048 40049 Object ID of the user-assigned MSI to use, if any. 40050 40051 Leave blank if msi_client_id or msi_mi_res_id specified. 40052 40053 Properties: 40054 40055 - Config: msi_object_id 40056 - Env Var: RCLONE_AZUREFILES_MSI_OBJECT_ID 40057 - Type: string 40058 - Required: false 40059 40060 #### --azurefiles-msi-client-id 40061 40062 Object ID of the user-assigned MSI to use, if any. 40063 40064 Leave blank if msi_object_id or msi_mi_res_id specified. 40065 40066 Properties: 40067 40068 - Config: msi_client_id 40069 - Env Var: RCLONE_AZUREFILES_MSI_CLIENT_ID 40070 - Type: string 40071 - Required: false 40072 40073 #### --azurefiles-msi-mi-res-id 40074 40075 Azure resource ID of the user-assigned MSI to use, if any. 40076 40077 Leave blank if msi_client_id or msi_object_id specified. 40078 40079 Properties: 40080 40081 - Config: msi_mi_res_id 40082 - Env Var: RCLONE_AZUREFILES_MSI_MI_RES_ID 40083 - Type: string 40084 - Required: false 40085 40086 #### --azurefiles-endpoint 40087 40088 Endpoint for the service. 40089 40090 Leave blank normally. 40091 40092 Properties: 40093 40094 - Config: endpoint 40095 - Env Var: RCLONE_AZUREFILES_ENDPOINT 40096 - Type: string 40097 - Required: false 40098 40099 #### --azurefiles-chunk-size 40100 40101 Upload chunk size. 40102 40103 Note that this is stored in memory and there may be up to 40104 "--transfers" * "--azurefile-upload-concurrency" chunks stored at once 40105 in memory. 40106 40107 Properties: 40108 40109 - Config: chunk_size 40110 - Env Var: RCLONE_AZUREFILES_CHUNK_SIZE 40111 - Type: SizeSuffix 40112 - Default: 4Mi 40113 40114 #### --azurefiles-upload-concurrency 40115 40116 Concurrency for multipart uploads. 40117 40118 This is the number of chunks of the same file that are uploaded 40119 concurrently. 40120 40121 If you are uploading small numbers of large files over high-speed 40122 links and these uploads do not fully utilize your bandwidth, then 40123 increasing this may help to speed up the transfers. 40124 40125 Note that chunks are stored in memory and there may be up to 40126 "--transfers" * "--azurefile-upload-concurrency" chunks stored at once 40127 in memory. 40128 40129 Properties: 40130 40131 - Config: upload_concurrency 40132 - Env Var: RCLONE_AZUREFILES_UPLOAD_CONCURRENCY 40133 - Type: int 40134 - Default: 16 40135 40136 #### --azurefiles-max-stream-size 40137 40138 Max size for streamed files. 40139 40140 Azure files needs to know in advance how big the file will be. When 40141 rclone doesn't know it uses this value instead. 40142 40143 This will be used when rclone is streaming data, the most common uses are: 40144 40145 - Uploading files with `--vfs-cache-mode off` with `rclone mount` 40146 - Using `rclone rcat` 40147 - Copying files with unknown length 40148 40149 You will need this much free space in the share as the file will be this size temporarily. 40150 40151 40152 Properties: 40153 40154 - Config: max_stream_size 40155 - Env Var: RCLONE_AZUREFILES_MAX_STREAM_SIZE 40156 - Type: SizeSuffix 40157 - Default: 10Gi 40158 40159 #### --azurefiles-encoding 40160 40161 The encoding for the backend. 40162 40163 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 40164 40165 Properties: 40166 40167 - Config: encoding 40168 - Env Var: RCLONE_AZUREFILES_ENCODING 40169 - Type: Encoding 40170 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot 40171 40172 #### --azurefiles-description 40173 40174 Description of the remote 40175 40176 Properties: 40177 40178 - Config: description 40179 - Env Var: RCLONE_AZUREFILES_DESCRIPTION 40180 - Type: string 40181 - Required: false 40182 40183 40184 40185 ### Custom upload headers 40186 40187 You can set custom upload headers with the `--header-upload` flag. 40188 40189 - Cache-Control 40190 - Content-Disposition 40191 - Content-Encoding 40192 - Content-Language 40193 - Content-Type 40194 40195 Eg `--header-upload "Content-Type: text/potato"` 40196 40197 ## Limitations 40198 40199 MD5 sums are only uploaded with chunked files if the source has an MD5 40200 sum. This will always be the case for a local to azure copy. 40201 40202 # Microsoft OneDrive 40203 40204 Paths are specified as `remote:path` 40205 40206 Paths may be as deep as required, e.g. `remote:directory/subdirectory`. 40207 40208 ## Configuration 40209 40210 The initial setup for OneDrive involves getting a token from 40211 Microsoft which you need to do in your browser. `rclone config` walks 40212 you through it. 40213 40214 Here is an example of how to make a remote called `remote`. First run: 40215 40216 rclone config 40217 40218 This will guide you through an interactive setup process: 40219 40220 e) Edit existing remote 40221 f) New remote 40222 g) Delete remote 40223 h) Rename remote 40224 i) Copy remote 40225 j) Set configuration password 40226 k) Quit config e/n/d/r/c/s/q> n name> remote Type of storage to 40227 configure. Enter a string value. Press Enter for the default (""). 40228 Choose a number from below, or type in your own value [snip] XX / 40229 Microsoft OneDrive "onedrive" [snip] Storage> onedrive Microsoft 40230 App Client Id Leave blank normally. Enter a string value. Press 40231 Enter for the default (""). client_id> Microsoft App Client Secret 40232 Leave blank normally. Enter a string value. Press Enter for the 40233 default (""). client_secret> Edit advanced config? (y/n) 40234 l) Yes 40235 m) No y/n> n Remote config Use web browser to automatically 40236 authenticate rclone with remote? 40237 40238 - Say Y if the machine running rclone has a web browser you can use 40239 - Say N if running rclone on a (remote) machine without web browser 40240 access If not sure try Y. If Y failed, try N. 40241 40242 y) Yes 40243 z) No y/n> y If your browser doesn't open automatically go to the 40244 following link: http://127.0.0.1:53682/auth Log in and authorize 40245 rclone for access Waiting for code... Got code Choose a number from 40246 below, or type in an existing value 1 / OneDrive Personal or 40247 Business "onedrive" 2 / Sharepoint site "sharepoint" 3 / Type in 40248 driveID "driveid" 4 / Type in SiteID "siteid" 5 / Search a 40249 Sharepoint site "search" Your choice> 1 Found 1 drives, please 40250 select the one you want to use: 0: OneDrive (business) 40251 id=b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk Chose 40252 drive to use:> 0 Found drive 'root' of type 'business', URL: 40253 https://org-my.sharepoint.com/personal/you/Documents Is that okay? 40254 a) Yes 40255 b) No y/n> y -------------------- [remote] type = onedrive token = 40256 {"access_token":"youraccesstoken","token_type":"Bearer","refresh_token":"yourrefreshtoken","expiry":"2018-08-26T22:39:52.486512262+08:00"} 40257 drive_id = 40258 b!Eqwertyuiopasdfghjklzxcvbnm-7mnbvcxzlkjhgfdsapoiuytrewqk 40259 drive_type = business -------------------- 40260 c) Yes this is OK 40261 d) Edit this remote 40262 e) Delete this remote y/e/d> y 40263 40264 40265 See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a 40266 machine with no Internet browser available. 40267 40268 Note that rclone runs a webserver on your local machine to collect the 40269 token as returned from Microsoft. This only runs from the moment it 40270 opens your browser to the moment you get back the verification 40271 code. This is on `http://127.0.0.1:53682/` and this it may require 40272 you to unblock it temporarily if you are running a host firewall. 40273 40274 Once configured you can then use `rclone` like this, 40275 40276 List directories in top level of your OneDrive 40277 40278 rclone lsd remote: 40279 40280 List all the files in your OneDrive 40281 40282 rclone ls remote: 40283 40284 To copy a local directory to an OneDrive directory called backup 40285 40286 rclone copy /home/source remote:backup 40287 40288 ### Getting your own Client ID and Key 40289 40290 rclone uses a default Client ID when talking to OneDrive, unless a custom `client_id` is specified in the config. 40291 The default Client ID and Key are shared by all rclone users when performing requests. 40292 40293 You may choose to create and use your own Client ID, in case the default one does not work well for you. 40294 For example, you might see throttling. 40295 40296 #### Creating Client ID for OneDrive Personal 40297 40298 To create your own Client ID, please follow these steps: 40299 40300 1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade and then click `New registration`. 40301 2. Enter a name for your app, choose account type `Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)`, select `Web` in `Redirect URI`, then type (do not copy and paste) `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use. 40302 3. Under `manage` select `Certificates & secrets`, click `New client secret`. Enter a description (can be anything) and set `Expires` to 24 months. Copy and keep that secret _Value_ for later use (you _won't_ be able to see this value afterwards). 40303 4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`. 40304 5. Search and select the following permissions: `Files.Read`, `Files.ReadWrite`, `Files.Read.All`, `Files.ReadWrite.All`, `offline_access`, `User.Read` and `Sites.Read.All` (if custom access scopes are configured, select the permissions accordingly). Once selected click `Add permissions` at the bottom. 40305 40306 Now the application is complete. Run `rclone config` to create or edit a OneDrive remote. 40307 Supply the app ID and password as Client ID and Secret, respectively. rclone will walk you through the remaining steps. 40308 40309 The access_scopes option allows you to configure the permissions requested by rclone. 40310 See [Microsoft Docs](https://docs.microsoft.com/en-us/graph/permissions-reference#files-permissions) for more information about the different scopes. 40311 40312 The `Sites.Read.All` permission is required if you need to [search SharePoint sites when configuring the remote](https://github.com/rclone/rclone/pull/5883). However, if that permission is not assigned, you need to exclude `Sites.Read.All` from your access scopes or set `disable_site_permission` option to true in the advanced options. 40313 40314 #### Creating Client ID for OneDrive Business 40315 40316 The steps for OneDrive Personal may or may not work for OneDrive Business, depending on the security settings of the organization. 40317 A common error is that the publisher of the App is not verified. 40318 40319 You may try to [verify you account](https://docs.microsoft.com/en-us/azure/active-directory/develop/publisher-verification-overview), or try to limit the App to your organization only, as shown below. 40320 40321 1. Make sure to create the App with your business account. 40322 2. Follow the steps above to create an App. However, we need a different account type here: `Accounts in this organizational directory only (*** - Single tenant)`. Note that you can also change the account type after creating the App. 40323 3. Find the [tenant ID](https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-how-to-find-tenant) of your organization. 40324 4. In the rclone config, set `auth_url` to `https://login.microsoftonline.com/YOUR_TENANT_ID/oauth2/v2.0/authorize`. 40325 5. In the rclone config, set `token_url` to `https://login.microsoftonline.com/YOUR_TENANT_ID/oauth2/v2.0/token`. 40326 40327 Note: If you have a special region, you may need a different host in step 4 and 5. Here are [some hints](https://github.com/rclone/rclone/blob/bc23bf11db1c78c6ebbf8ea538fbebf7058b4176/backend/onedrive/onedrive.go#L86). 40328 40329 40330 ### Modification times and hashes 40331 40332 OneDrive allows modification times to be set on objects accurate to 1 40333 second. These will be used to detect whether objects need syncing or 40334 not. 40335 40336 OneDrive Personal, OneDrive for Business and Sharepoint Server support 40337 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 40338 40339 Before rclone 1.62 the default hash for Onedrive Personal was `SHA1`. 40340 For rclone 1.62 and above the default for all Onedrive backends is 40341 `QuickXorHash`. 40342 40343 Starting from July 2023 `SHA1` support is being phased out in Onedrive 40344 Personal in favour of `QuickXorHash`. If necessary the 40345 `--onedrive-hash-type` flag (or `hash_type` config option) can be used 40346 to select `SHA1` during the transition period if this is important 40347 your workflow. 40348 40349 For all types of OneDrive you can use the `--checksum` flag. 40350 40351 ### --fast-list 40352 40353 This remote supports `--fast-list` which allows you to use fewer 40354 transactions in exchange for more memory. See the [rclone 40355 docs](https://rclone.org/docs/#fast-list) for more details. 40356 40357 This must be enabled with the `--onedrive-delta` flag (or `delta = 40358 true` in the config file) as it can cause performance degradation. 40359 40360 It does this by using the delta listing facilities of OneDrive which 40361 returns all the files in the remote very efficiently. This is much 40362 more efficient than listing directories recursively and is Microsoft's 40363 recommended way of reading all the file information from a drive. 40364 40365 This can be useful with `rclone mount` and [rclone rc vfs/refresh 40366 recursive=true](https://rclone.org/rc/#vfs-refresh)) to very quickly fill the mount with 40367 information about all the files. 40368 40369 The API used for the recursive listing (`ListR`) only supports listing 40370 from the root of the drive. This will become increasingly inefficient 40371 the further away you get from the root as rclone will have to discard 40372 files outside of the directory you are using. 40373 40374 Some commands (like `rclone lsf -R`) will use `ListR` by default - you 40375 can turn this off with `--disable ListR` if you need to. 40376 40377 ### Restricted filename characters 40378 40379 In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters) 40380 the following characters are also replaced: 40381 40382 | Character | Value | Replacement | 40383 | --------- |:-----:|:-----------:| 40384 | " | 0x22 | " | 40385 | * | 0x2A | * | 40386 | : | 0x3A | : | 40387 | < | 0x3C | < | 40388 | > | 0x3E | > | 40389 | ? | 0x3F | ? | 40390 | \ | 0x5C | \ | 40391 | \| | 0x7C | | | 40392 40393 File names can also not end with the following characters. 40394 These only get replaced if they are the last character in the name: 40395 40396 | Character | Value | Replacement | 40397 | --------- |:-----:|:-----------:| 40398 | SP | 0x20 | ␠ | 40399 | . | 0x2E | . | 40400 40401 File names can also not begin with the following characters. 40402 These only get replaced if they are the first character in the name: 40403 40404 | Character | Value | Replacement | 40405 | --------- |:-----:|:-----------:| 40406 | SP | 0x20 | ␠ | 40407 | ~ | 0x7E | ~ | 40408 40409 Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8), 40410 as they can't be used in JSON strings. 40411 40412 ### Deleting files 40413 40414 Any files you delete with rclone will end up in the trash. Microsoft 40415 doesn't provide an API to permanently delete files, nor to empty the 40416 trash, so you will have to do that with one of Microsoft's apps or via 40417 the OneDrive website. 40418 40419 40420 ### Standard options 40421 40422 Here are the Standard options specific to onedrive (Microsoft OneDrive). 40423 40424 #### --onedrive-client-id 40425 40426 OAuth Client Id. 40427 40428 Leave blank normally. 40429 40430 Properties: 40431 40432 - Config: client_id 40433 - Env Var: RCLONE_ONEDRIVE_CLIENT_ID 40434 - Type: string 40435 - Required: false 40436 40437 #### --onedrive-client-secret 40438 40439 OAuth Client Secret. 40440 40441 Leave blank normally. 40442 40443 Properties: 40444 40445 - Config: client_secret 40446 - Env Var: RCLONE_ONEDRIVE_CLIENT_SECRET 40447 - Type: string 40448 - Required: false 40449 40450 #### --onedrive-region 40451 40452 Choose national cloud region for OneDrive. 40453 40454 Properties: 40455 40456 - Config: region 40457 - Env Var: RCLONE_ONEDRIVE_REGION 40458 - Type: string 40459 - Default: "global" 40460 - Examples: 40461 - "global" 40462 - Microsoft Cloud Global 40463 - "us" 40464 - Microsoft Cloud for US Government 40465 - "de" 40466 - Microsoft Cloud Germany 40467 - "cn" 40468 - Azure and Office 365 operated by Vnet Group in China 40469 40470 ### Advanced options 40471 40472 Here are the Advanced options specific to onedrive (Microsoft OneDrive). 40473 40474 #### --onedrive-token 40475 40476 OAuth Access Token as a JSON blob. 40477 40478 Properties: 40479 40480 - Config: token 40481 - Env Var: RCLONE_ONEDRIVE_TOKEN 40482 - Type: string 40483 - Required: false 40484 40485 #### --onedrive-auth-url 40486 40487 Auth server URL. 40488 40489 Leave blank to use the provider defaults. 40490 40491 Properties: 40492 40493 - Config: auth_url 40494 - Env Var: RCLONE_ONEDRIVE_AUTH_URL 40495 - Type: string 40496 - Required: false 40497 40498 #### --onedrive-token-url 40499 40500 Token server url. 40501 40502 Leave blank to use the provider defaults. 40503 40504 Properties: 40505 40506 - Config: token_url 40507 - Env Var: RCLONE_ONEDRIVE_TOKEN_URL 40508 - Type: string 40509 - Required: false 40510 40511 #### --onedrive-chunk-size 40512 40513 Chunk size to upload files with - must be multiple of 320k (327,680 bytes). 40514 40515 Above this size files will be chunked - must be multiple of 320k (327,680 bytes) and 40516 should not exceed 250M (262,144,000 bytes) else you may encounter \"Microsoft.SharePoint.Client.InvalidClientQueryException: The request message is too big.\" 40517 Note that the chunks will be buffered into memory. 40518 40519 Properties: 40520 40521 - Config: chunk_size 40522 - Env Var: RCLONE_ONEDRIVE_CHUNK_SIZE 40523 - Type: SizeSuffix 40524 - Default: 10Mi 40525 40526 #### --onedrive-drive-id 40527 40528 The ID of the drive to use. 40529 40530 Properties: 40531 40532 - Config: drive_id 40533 - Env Var: RCLONE_ONEDRIVE_DRIVE_ID 40534 - Type: string 40535 - Required: false 40536 40537 #### --onedrive-drive-type 40538 40539 The type of the drive (personal | business | documentLibrary). 40540 40541 Properties: 40542 40543 - Config: drive_type 40544 - Env Var: RCLONE_ONEDRIVE_DRIVE_TYPE 40545 - Type: string 40546 - Required: false 40547 40548 #### --onedrive-root-folder-id 40549 40550 ID of the root folder. 40551 40552 This isn't normally needed, but in special circumstances you might 40553 know the folder ID that you wish to access but not be able to get 40554 there through a path traversal. 40555 40556 40557 Properties: 40558 40559 - Config: root_folder_id 40560 - Env Var: RCLONE_ONEDRIVE_ROOT_FOLDER_ID 40561 - Type: string 40562 - Required: false 40563 40564 #### --onedrive-access-scopes 40565 40566 Set scopes to be requested by rclone. 40567 40568 Choose or manually enter a custom space separated list with all scopes, that rclone should request. 40569 40570 40571 Properties: 40572 40573 - Config: access_scopes 40574 - Env Var: RCLONE_ONEDRIVE_ACCESS_SCOPES 40575 - Type: SpaceSepList 40576 - Default: Files.Read Files.ReadWrite Files.Read.All Files.ReadWrite.All Sites.Read.All offline_access 40577 - Examples: 40578 - "Files.Read Files.ReadWrite Files.Read.All Files.ReadWrite.All Sites.Read.All offline_access" 40579 - Read and write access to all resources 40580 - "Files.Read Files.Read.All Sites.Read.All offline_access" 40581 - Read only access to all resources 40582 - "Files.Read Files.ReadWrite Files.Read.All Files.ReadWrite.All offline_access" 40583 - Read and write access to all resources, without the ability to browse SharePoint sites. 40584 - Same as if disable_site_permission was set to true 40585 40586 #### --onedrive-disable-site-permission 40587 40588 Disable the request for Sites.Read.All permission. 40589 40590 If set to true, you will no longer be able to search for a SharePoint site when 40591 configuring drive ID, because rclone will not request Sites.Read.All permission. 40592 Set it to true if your organization didn't assign Sites.Read.All permission to the 40593 application, and your organization disallows users to consent app permission 40594 request on their own. 40595 40596 Properties: 40597 40598 - Config: disable_site_permission 40599 - Env Var: RCLONE_ONEDRIVE_DISABLE_SITE_PERMISSION 40600 - Type: bool 40601 - Default: false 40602 40603 #### --onedrive-expose-onenote-files 40604 40605 Set to make OneNote files show up in directory listings. 40606 40607 By default, rclone will hide OneNote files in directory listings because 40608 operations like "Open" and "Update" won't work on them. But this 40609 behaviour may also prevent you from deleting them. If you want to 40610 delete OneNote files or otherwise want them to show up in directory 40611 listing, set this option. 40612 40613 Properties: 40614 40615 - Config: expose_onenote_files 40616 - Env Var: RCLONE_ONEDRIVE_EXPOSE_ONENOTE_FILES 40617 - Type: bool 40618 - Default: false 40619 40620 #### --onedrive-server-side-across-configs 40621 40622 Deprecated: use --server-side-across-configs instead. 40623 40624 Allow server-side operations (e.g. copy) to work across different onedrive configs. 40625 40626 This will only work if you are copying between two OneDrive *Personal* drives AND 40627 the files to copy are already shared between them. In other cases, rclone will 40628 fall back to normal copy (which will be slightly slower). 40629 40630 Properties: 40631 40632 - Config: server_side_across_configs 40633 - Env Var: RCLONE_ONEDRIVE_SERVER_SIDE_ACROSS_CONFIGS 40634 - Type: bool 40635 - Default: false 40636 40637 #### --onedrive-list-chunk 40638 40639 Size of listing chunk. 40640 40641 Properties: 40642 40643 - Config: list_chunk 40644 - Env Var: RCLONE_ONEDRIVE_LIST_CHUNK 40645 - Type: int 40646 - Default: 1000 40647 40648 #### --onedrive-no-versions 40649 40650 Remove all versions on modifying operations. 40651 40652 Onedrive for business creates versions when rclone uploads new files 40653 overwriting an existing one and when it sets the modification time. 40654 40655 These versions take up space out of the quota. 40656 40657 This flag checks for versions after file upload and setting 40658 modification time and removes all but the last version. 40659 40660 **NB** Onedrive personal can't currently delete versions so don't use 40661 this flag there. 40662 40663 40664 Properties: 40665 40666 - Config: no_versions 40667 - Env Var: RCLONE_ONEDRIVE_NO_VERSIONS 40668 - Type: bool 40669 - Default: false 40670 40671 #### --onedrive-link-scope 40672 40673 Set the scope of the links created by the link command. 40674 40675 Properties: 40676 40677 - Config: link_scope 40678 - Env Var: RCLONE_ONEDRIVE_LINK_SCOPE 40679 - Type: string 40680 - Default: "anonymous" 40681 - Examples: 40682 - "anonymous" 40683 - Anyone with the link has access, without needing to sign in. 40684 - This may include people outside of your organization. 40685 - Anonymous link support may be disabled by an administrator. 40686 - "organization" 40687 - Anyone signed into your organization (tenant) can use the link to get access. 40688 - Only available in OneDrive for Business and SharePoint. 40689 40690 #### --onedrive-link-type 40691 40692 Set the type of the links created by the link command. 40693 40694 Properties: 40695 40696 - Config: link_type 40697 - Env Var: RCLONE_ONEDRIVE_LINK_TYPE 40698 - Type: string 40699 - Default: "view" 40700 - Examples: 40701 - "view" 40702 - Creates a read-only link to the item. 40703 - "edit" 40704 - Creates a read-write link to the item. 40705 - "embed" 40706 - Creates an embeddable link to the item. 40707 40708 #### --onedrive-link-password 40709 40710 Set the password for links created by the link command. 40711 40712 At the time of writing this only works with OneDrive personal paid accounts. 40713 40714 40715 Properties: 40716 40717 - Config: link_password 40718 - Env Var: RCLONE_ONEDRIVE_LINK_PASSWORD 40719 - Type: string 40720 - Required: false 40721 40722 #### --onedrive-hash-type 40723 40724 Specify the hash in use for the backend. 40725 40726 This specifies the hash type in use. If set to "auto" it will use the 40727 default hash which is QuickXorHash. 40728 40729 Before rclone 1.62 an SHA1 hash was used by default for Onedrive 40730 Personal. For 1.62 and later the default is to use a QuickXorHash for 40731 all onedrive types. If an SHA1 hash is desired then set this option 40732 accordingly. 40733 40734 From July 2023 QuickXorHash will be the only available hash for 40735 both OneDrive for Business and OneDriver Personal. 40736 40737 This can be set to "none" to not use any hashes. 40738 40739 If the hash requested does not exist on the object, it will be 40740 returned as an empty string which is treated as a missing hash by 40741 rclone. 40742 40743 40744 Properties: 40745 40746 - Config: hash_type 40747 - Env Var: RCLONE_ONEDRIVE_HASH_TYPE 40748 - Type: string 40749 - Default: "auto" 40750 - Examples: 40751 - "auto" 40752 - Rclone chooses the best hash 40753 - "quickxor" 40754 - QuickXor 40755 - "sha1" 40756 - SHA1 40757 - "sha256" 40758 - SHA256 40759 - "crc32" 40760 - CRC32 40761 - "none" 40762 - None - don't use any hashes 40763 40764 #### --onedrive-av-override 40765 40766 Allows download of files the server thinks has a virus. 40767 40768 The onedrive/sharepoint server may check files uploaded with an Anti 40769 Virus checker. If it detects any potential viruses or malware it will 40770 block download of the file. 40771 40772 In this case you will see a message like this 40773 40774 server reports this file is infected with a virus - use --onedrive-av-override to download anyway: Infected (name of virus): 403 Forbidden: 40775 40776 If you are 100% sure you want to download this file anyway then use 40777 the --onedrive-av-override flag, or av_override = true in the config 40778 file. 40779 40780 40781 Properties: 40782 40783 - Config: av_override 40784 - Env Var: RCLONE_ONEDRIVE_AV_OVERRIDE 40785 - Type: bool 40786 - Default: false 40787 40788 #### --onedrive-delta 40789 40790 If set rclone will use delta listing to implement recursive listings. 40791 40792 If this flag is set the onedrive backend will advertise `ListR` 40793 support for recursive listings. 40794 40795 Setting this flag speeds up these things greatly: 40796 40797 rclone lsf -R onedrive: 40798 rclone size onedrive: 40799 rclone rc vfs/refresh recursive=true 40800 40801 **However** the delta listing API **only** works at the root of the 40802 drive. If you use it not at the root then it recurses from the root 40803 and discards all the data that is not under the directory you asked 40804 for. So it will be correct but may not be very efficient. 40805 40806 This is why this flag is not set as the default. 40807 40808 As a rule of thumb if nearly all of your data is under rclone's root 40809 directory (the `root/directory` in `onedrive:root/directory`) then 40810 using this flag will be be a big performance win. If your data is 40811 mostly not under the root then using this flag will be a big 40812 performance loss. 40813 40814 It is recommended if you are mounting your onedrive at the root 40815 (or near the root when using crypt) and using rclone `rc vfs/refresh`. 40816 40817 40818 Properties: 40819 40820 - Config: delta 40821 - Env Var: RCLONE_ONEDRIVE_DELTA 40822 - Type: bool 40823 - Default: false 40824 40825 #### --onedrive-metadata-permissions 40826 40827 Control whether permissions should be read or written in metadata. 40828 40829 Reading permissions metadata from files can be done quickly, but it 40830 isn't always desirable to set the permissions from the metadata. 40831 40832 40833 Properties: 40834 40835 - Config: metadata_permissions 40836 - Env Var: RCLONE_ONEDRIVE_METADATA_PERMISSIONS 40837 - Type: Bits 40838 - Default: off 40839 - Examples: 40840 - "off" 40841 - Do not read or write the value 40842 - "read" 40843 - Read the value only 40844 - "write" 40845 - Write the value only 40846 - "read,write" 40847 - Read and Write the value. 40848 40849 #### --onedrive-encoding 40850 40851 The encoding for the backend. 40852 40853 See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info. 40854 40855 Properties: 40856 40857 - Config: encoding 40858 - Env Var: RCLONE_ONEDRIVE_ENCODING 40859 - Type: Encoding 40860 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot 40861 40862 #### --onedrive-description 40863 40864 Description of the remote 40865 40866 Properties: 40867 40868 - Config: description 40869 - Env Var: RCLONE_ONEDRIVE_DESCRIPTION 40870 - Type: string 40871 - Required: false 40872 40873 ### Metadata 40874 40875 OneDrive supports System Metadata (not User Metadata, as of this writing) for 40876 both files and directories. Much of the metadata is read-only, and there are some 40877 differences between OneDrive Personal and Business (see table below for 40878 details). 40879 40880 Permissions are also supported, if `--onedrive-metadata-permissions` is set. The 40881 accepted values for `--onedrive-metadata-permissions` are `read`, `write`, 40882 `read,write`, and `off` (the default). `write` supports adding new permissions, 40883 updating the "role" of existing permissions, and removing permissions. Updating 40884 and removing require the Permission ID to be known, so it is recommended to use 40885 `read,write` instead of `write` if you wish to update/remove permissions. 40886 40887 Permissions are read/written in JSON format using the same schema as the 40888 [OneDrive API](https://learn.microsoft.com/en-us/onedrive/developer/rest-api/resources/permission?view=odsp-graph-online), 40889 which differs slightly between OneDrive Personal and Business. 40890 40891 Example for OneDrive Personal: 40892 ```json 40893 [ 40894 { 40895 "id": "1234567890ABC!123", 40896 "grantedTo": { 40897 "user": { 40898 "id": "ryan@contoso.com" 40899 }, 40900 "application": {}, 40901 "device": {} 40902 }, 40903 "invitation": { 40904 "email": "ryan@contoso.com" 40905 }, 40906 "link": { 40907 "webUrl": "https://1drv.ms/t/s!1234567890ABC" 40908 }, 40909 "roles": [ 40910 "read" 40911 ], 40912 "shareId": "s!1234567890ABC" 40913 } 40914 ] 40915 40916 Example for OneDrive Business: 40917 40918 [ 40919 { 40920 "id": "48d31887-5fad-4d73-a9f5-3c356e68a038", 40921 "grantedToIdentities": [ 40922 { 40923 "user": { 40924 "displayName": "ryan@contoso.com" 40925 }, 40926 "application": {}, 40927 "device": {} 40928 } 40929 ], 40930 "link": { 40931 "type": "view", 40932 "scope": "users", 40933 "webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s" 40934 }, 40935 "roles": [ 40936 "read" 40937 ], 40938 "shareId": "u!LKj1lkdlals90j1nlkascl" 40939 }, 40940 { 40941 "id": "5D33DD65C6932946", 40942 "grantedTo": { 40943 "user": { 40944 "displayName": "John Doe", 40945 "id": "efee1b77-fb3b-4f65-99d6-274c11914d12" 40946 }, 40947 "application": {}, 40948 "device": {} 40949 }, 40950 "roles": [ 40951 "owner" 40952 ], 40953 "shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U" 40954 } 40955 ] 40956 40957 To write permissions, pass in a "permissions" metadata key using this 40958 same format. The --metadata-mapper tool can be very helpful for this. 40959 40960 When adding permissions, an email address can be provided in the User.ID 40961 or DisplayName properties of grantedTo or grantedToIdentities. 40962 Alternatively, an ObjectID can be provided in User.ID. At least one 40963 valid recipient must be provided in order to add a permission for a 40964 user. Creating a Public Link is also supported, if Link.Scope is set to 40965 "anonymous". 40966 40967 Example request to add a "read" permission: 40968 40969 [ 40970 { 40971 "id": "", 40972 "grantedTo": { 40973 "user": {}, 40974 "application": {}, 40975 "device": {} 40976 }, 40977 "grantedToIdentities": [ 40978 { 40979 "user": { 40980 "id": "ryan@contoso.com" 40981 }, 40982 "application": {}, 40983 "device": {} 40984 } 40985 ], 40986 "roles": [ 40987 "read" 40988 ] 40989 } 40990 ] 40991 40992 Note that adding a permission can fail if a conflicting permission 40993 already exists for the file/folder. 40994 40995 To update an existing permission, include both the Permission ID and the 40996 new roles to be assigned. roles is the only property that can be 40997 changed. 40998 40999 To remove permissions, pass in a blob containing only the permissions 41000 you wish to keep (which can be empty, to remove all.) 41001 41002 Note that both reading and writing permissions requires extra API calls, 41003 so if you don't need to read or write permissions it is recommended to 41004 omit --onedrive-metadata-permissions. 41005 41006 Metadata and permissions are supported for Folders (directories) as well 41007 as Files. Note that setting the mtime or btime on a Folder requires one 41008 extra API call on OneDrive Business only. 41009 41010 OneDrive does not currently support User Metadata. When writing 41011 metadata, only writeable system properties will be written -- any 41012 read-only or unrecognized keys passed in will be ignored. 41013 41014 TIP: to see the metadata and permissions for any file or folder, run: 41015 41016 rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read 41017 41018 Here are the possible system metadata items for the onedrive backend. 41019 41020 ------------------------------------------------------------------------------------------------------------------------------------------ 41021 Name Help Type Example Read Only 41022 ------------------------------- ---------------------------------- ----------- -------------------------------------- -------------------- 41023 btime Time of file birth (creation) with RFC 3339 2006-01-02T15:04:05Z N 41024 S accuracy (mS for OneDrive 41025 Personal). 41026 41027 content-type The MIME type of the file. string text/plain Y 41028 41029 created-by-display-name Display name of the user that string John Doe Y 41030 created the item. 41031 41032 created-by-id ID of the user that created the string 48d31887-5fad-4d73-a9f5-3c356e68a038 Y 41033 item. 41034 41035 description A short description of the file. string Contract for signing N 41036 Max 1024 characters. Only 41037 supported for OneDrive Personal. 41038 41039 id The unique identifier of the item string 01BYE5RZ6QN3ZWBTUFOFD3GSPGOHDJD36K Y 41040 within OneDrive. 41041 41042 last-modified-by-display-name Display name of the user that last string John Doe Y 41043 modified the item. 41044 41045 last-modified-by-id ID of the user that last modified string 48d31887-5fad-4d73-a9f5-3c356e68a038 Y 41046 the item. 41047 41048 malware-detected Whether OneDrive has detected that boolean true Y 41049 the item contains malware. 41050 41051 mtime Time of last modification with S RFC 3339 2006-01-02T15:04:05Z N 41052 accuracy (mS for OneDrive 41053 Personal). 41054 41055 package-type If present, indicates that this string oneNote Y 41056 item is a package instead of a 41057 folder or file. Packages are 41058 treated like files in some 41059 contexts and folders in others. 41060 41061 permissions Permissions in a JSON dump of JSON {} N 41062 OneDrive format. Enable with 41063 --onedrive-metadata-permissions. 41064 Properties: id, grantedTo, 41065 grantedToIdentities, invitation, 41066 inheritedFrom, link, roles, 41067 shareId 41068 41069 shared-by-id ID of the user that shared the string 48d31887-5fad-4d73-a9f5-3c356e68a038 Y 41070 item (if shared). 41071 41072 shared-owner-id ID of the owner of the shared item string 48d31887-5fad-4d73-a9f5-3c356e68a038 Y 41073 (if shared). 41074 41075 shared-scope If shared, indicates the scope of string users Y 41076 how the item is shared: anonymous, 41077 organization, or users. 41078 41079 shared-time Time when the item was shared, RFC 3339 2006-01-02T15:04:05Z Y 41080 with S accuracy (mS for OneDrive 41081 Personal). 41082 41083 utime Time of upload with S accuracy (mS RFC 3339 2006-01-02T15:04:05Z Y 41084 for OneDrive Personal). 41085 ------------------------------------------------------------------------------------------------------------------------------------------ 41086 41087 See the metadata docs for more info. 41088 41089 Limitations 41090 41091 If you don't use rclone for 90 days the refresh token will expire. This 41092 will result in authorization problems. This is easy to fix by running 41093 the rclone config reconnect remote: command to get a new token and 41094 refresh token. 41095 41096 Naming 41097 41098 Note that OneDrive is case insensitive so you can't have a file called 41099 "Hello.doc" and one called "hello.doc". 41100 41101 There are quite a few characters that can't be in OneDrive file names. 41102 These can't occur on Windows platforms, but on non-Windows platforms 41103 they are common. Rclone will map these names to and from an identical 41104 looking unicode equivalent. For example if a file has a ? in it will be 41105 mapped to ? instead. 41106 41107 File sizes 41108 41109 The largest allowed file size is 250 GiB for both OneDrive Personal and 41110 OneDrive for Business (Updated 13 Jan 2021). 41111 41112 Path length 41113 41114 The entire path, including the file name, must contain fewer than 400 41115 characters for OneDrive, OneDrive for Business and SharePoint Online. If 41116 you are encrypting file and folder names with rclone, you may want to 41117 pay attention to this limitation because the encrypted names are 41118 typically longer than the original ones. 41119 41120 Number of files 41121 41122 OneDrive seems to be OK with at least 50,000 files in a folder, but at 41123 100,000 rclone will get errors listing the directory like 41124 couldn’t list files: UnknownError:. See #2707 for more info. 41125 41126 An official document about the limitations for different types of 41127 OneDrive can be found here. 41128 41129 Versions 41130 41131 Every change in a file OneDrive causes the service to create a new 41132 version of the file. This counts against a users quota. For example 41133 changing the modification time of a file creates a second version, so 41134 the file apparently uses twice the space. 41135 41136 For example the copy command is affected by this as rclone copies the 41137 file and then afterwards sets the modification time to match the source 41138 file which uses another version. 41139 41140 You can use the rclone cleanup command (see below) to remove all old 41141 versions. 41142 41143 Or you can set the no_versions parameter to true and rclone will remove 41144 versions after operations which create new versions. This takes extra 41145 transactions so only enable it if you need it. 41146 41147 Note At the time of writing Onedrive Personal creates versions (but not 41148 for setting the modification time) but the API for removing them returns 41149 "API not found" so cleanup and no_versions should not be used on 41150 Onedrive Personal. 41151 41152 Disabling versioning 41153 41154 Starting October 2018, users will no longer be able to disable 41155 versioning by default. This is because Microsoft has brought an update 41156 to the mechanism. To change this new default setting, a PowerShell 41157 command is required to be run by a SharePoint admin. If you are an 41158 admin, you can run these commands in PowerShell to change that setting: 41159 41160 1. Install-Module -Name Microsoft.Online.SharePoint.PowerShell (in case 41161 you haven't installed this already) 41162 2. Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking 41163 3. Connect-SPOService -Url https://YOURSITE-admin.sharepoint.com -Credential YOU@YOURSITE.COM 41164 (replacing YOURSITE, YOU, YOURSITE.COM with the actual values; this 41165 will prompt for your credentials) 41166 4. Set-SPOTenant -EnableMinimumVersionRequirement $False 41167 5. Disconnect-SPOService (to disconnect from the server) 41168 41169 Below are the steps for normal users to disable versioning. If you don't 41170 see the "No Versioning" option, make sure the above requirements are 41171 met. 41172 41173 User Weropol has found a method to disable versioning on OneDrive 41174 41175 1. Open the settings menu by clicking on the gear symbol at the top of 41176 the OneDrive Business page. 41177 2. Click Site settings. 41178 3. Once on the Site settings page, navigate to Site Administration > 41179 Site libraries and lists. 41180 4. Click Customize "Documents". 41181 5. Click General Settings > Versioning Settings. 41182 6. Under Document Version History select the option No versioning. 41183 Note: This will disable the creation of new file versions, but will 41184 not remove any previous versions. Your documents are safe. 41185 7. Apply the changes by clicking OK. 41186 8. Use rclone to upload or modify files. (I also use the 41187 --no-update-modtime flag) 41188 9. Restore the versioning settings after using rclone. (Optional) 41189 41190 Cleanup 41191 41192 OneDrive supports rclone cleanup which causes rclone to look through 41193 every file under the path supplied and delete all version but the 41194 current version. Because this involves traversing all the files, then 41195 querying each file for versions it can be quite slow. Rclone does 41196 --checkers tests in parallel. The command also supports --interactive/i 41197 or --dry-run which is a great way to see what it would do. 41198 41199 rclone cleanup --interactive remote:path/subdir # interactively remove all old version for path/subdir 41200 rclone cleanup remote:path/subdir # unconditionally remove all old version for path/subdir 41201 41202 NB Onedrive personal can't currently delete versions 41203 41204 Troubleshooting 41205 41206 Excessive throttling or blocked on SharePoint 41207 41208 If you experience excessive throttling or is being blocked on SharePoint 41209 then it may help to set the user agent explicitly with a flag like this: 41210 --user-agent "ISV|rclone.org|rclone/v1.55.1" 41211 41212 The specific details can be found in the Microsoft document: Avoid 41213 getting throttled or blocked in SharePoint Online 41214 41215 Unexpected file size/hash differences on Sharepoint 41216 41217 It is a known issue that Sharepoint (not OneDrive or OneDrive for 41218 Business) silently modifies uploaded files, mainly Office files (.docx, 41219 .xlsx, etc.), causing file size and hash checks to fail. There are also 41220 other situations that will cause OneDrive to report inconsistent file 41221 sizes. To use rclone with such affected files on Sharepoint, you may 41222 disable these checks with the following command line arguments: 41223 41224 --ignore-checksum --ignore-size 41225 41226 Alternatively, if you have write access to the OneDrive files, it may be 41227 possible to fix this problem for certain files, by attempting the steps 41228 below. Open the web interface for OneDrive and find the affected files 41229 (which will be in the error messages/log for rclone). Simply click on 41230 each of these files, causing OneDrive to open them on the web. This will 41231 cause each file to be converted in place to a format that is 41232 functionally equivalent but which will no longer trigger the size 41233 discrepancy. Once all problematic files are converted you will no longer 41234 need the ignore options above. 41235 41236 Replacing/deleting existing files on Sharepoint gets "item not found" 41237 41238 It is a known issue that Sharepoint (not OneDrive or OneDrive for 41239 Business) may return "item not found" errors when users try to replace 41240 or delete uploaded files; this seems to mainly affect Office files 41241 (.docx, .xlsx, etc.) and web files (.html, .aspx, etc.). As a 41242 workaround, you may use the --backup-dir <BACKUP_DIR> command line 41243 argument so rclone moves the files to be replaced/deleted into a given 41244 backup directory (instead of directly replacing/deleting them). For 41245 example, to instruct rclone to move the files into the directory 41246 rclone-backup-dir on backend mysharepoint, you may use: 41247 41248 --backup-dir mysharepoint:rclone-backup-dir 41249 41250 access_denied (AADSTS65005) 41251 41252 Error: access_denied 41253 Code: AADSTS65005 41254 Description: Using application 'rclone' is currently not supported for your organization [YOUR_ORGANIZATION] because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of [YOUR_ORGANIZATION] before the application rclone can be provisioned. 41255 41256 This means that rclone can't use the OneDrive for Business API with your 41257 account. You can't do much about it, maybe write an email to your 41258 admins. 41259 41260 However, there are other ways to interact with your OneDrive account. 41261 Have a look at the WebDAV backend: https://rclone.org/webdav/#sharepoint 41262 41263 invalid_grant (AADSTS50076) 41264 41265 Error: invalid_grant 41266 Code: AADSTS50076 41267 Description: Due to a configuration change made by your administrator, or because you moved to a new location, you must use multi-factor authentication to access '...'. 41268 41269 If you see the error above after enabling multi-factor authentication 41270 for your account, you can fix it by refreshing your OAuth refresh token. 41271 To do that, run rclone config, and choose to edit your OneDrive backend. 41272 Then, you don't need to actually make any changes until you reach this 41273 question: Already have a token - refresh?. For this question, answer y 41274 and go through the process to refresh your token, just like the first 41275 time the backend is configured. After this, rclone should work again for 41276 this backend. 41277 41278 Invalid request when making public links 41279 41280 On Sharepoint and OneDrive for Business, rclone link may return an 41281 "Invalid request" error. A possible cause is that the organisation admin 41282 didn't allow public links to be made for the organisation/sharepoint 41283 library. To fix the permissions as an admin, take a look at the docs: 1, 41284 2. 41285 41286 Can not access Shared with me files 41287 41288 Shared with me files is not supported by rclone currently, but there is 41289 a workaround: 41290 41291 1. Visit https://onedrive.live.com 41292 2. Right click a item in Shared, then click Add shortcut to My files in 41293 the context [make_shortcut] 41294 3. The shortcut will appear in My files, you can access it with rclone, 41295 it behaves like a normal folder/file. [in_my_files] [rclone_mount] 41296 41297 Live Photos uploaded from iOS (small video clips in .heic files) 41298 41299 The iOS OneDrive app introduced upload and storage of Live Photos in 41300 2020. The usage and download of these uploaded Live Photos is 41301 unfortunately still work-in-progress and this introduces several issues 41302 when copying, synchronising and mounting – both in rclone and in the 41303 native OneDrive client on Windows. 41304 41305 The root cause can easily be seen if you locate one of your Live Photos 41306 in the OneDrive web interface. Then download the photo from the web 41307 interface. You will then see that the size of downloaded .heic file is 41308 smaller than the size displayed in the web interface. The downloaded 41309 file is smaller because it only contains a single frame (still photo) 41310 extracted from the Live Photo (movie) stored in OneDrive. 41311 41312 The different sizes will cause rclone copy/sync to repeatedly recopy 41313 unmodified photos something like this: 41314 41315 DEBUG : 20230203_123826234_iOS.heic: Sizes differ (src 4470314 vs dst 1298667) 41316 DEBUG : 20230203_123826234_iOS.heic: sha1 = fc2edde7863b7a7c93ca6771498ac797f8460750 OK 41317 INFO : 20230203_123826234_iOS.heic: Copied (replaced existing) 41318 41319 These recopies can be worked around by adding --ignore-size. Please note 41320 that this workaround only syncs the still-picture not the movie clip, 41321 and relies on modification dates being correctly updated on all files in 41322 all situations. 41323 41324 The different sizes will also cause rclone check to report size errors 41325 something like this: 41326 41327 ERROR : 20230203_123826234_iOS.heic: sizes differ 41328 41329 These check errors can be suppressed by adding --ignore-size. 41330 41331 The different sizes will also cause rclone mount to fail downloading 41332 with an error something like this: 41333 41334 ERROR : 20230203_123826234_iOS.heic: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF 41335 41336 or like this when using --cache-mode=full: 41337 41338 INFO : 20230203_123826234_iOS.heic: vfs cache: downloader: error count now 1: vfs reader: failed to write to cache file: 416 Requested Range Not Satisfiable: 41339 ERROR : 20230203_123826234_iOS.heic: vfs cache: failed to download: vfs reader: failed to write to cache file: 416 Requested Range Not Satisfiable: 41340 41341 OpenDrive 41342 41343 Paths are specified as remote:path 41344 41345 Paths may be as deep as required, e.g. remote:directory/subdirectory. 41346 41347 Configuration 41348 41349 Here is an example of how to make a remote called remote. First run: 41350 41351 rclone config 41352 41353 This will guide you through an interactive setup process: 41354 41355 n) New remote 41356 d) Delete remote 41357 q) Quit config 41358 e/n/d/q> n 41359 name> remote 41360 Type of storage to configure. 41361 Choose a number from below, or type in your own value 41362 [snip] 41363 XX / OpenDrive 41364 \ "opendrive" 41365 [snip] 41366 Storage> opendrive 41367 Username 41368 username> 41369 Password 41370 y) Yes type in my own password 41371 g) Generate random password 41372 y/g> y 41373 Enter the password: 41374 password: 41375 Confirm the password: 41376 password: 41377 -------------------- 41378 [remote] 41379 username = 41380 password = *** ENCRYPTED *** 41381 -------------------- 41382 y) Yes this is OK 41383 e) Edit this remote 41384 d) Delete this remote 41385 y/e/d> y 41386 41387 List directories in top level of your OpenDrive 41388 41389 rclone lsd remote: 41390 41391 List all the files in your OpenDrive 41392 41393 rclone ls remote: 41394 41395 To copy a local directory to an OpenDrive directory called backup 41396 41397 rclone copy /home/source remote:backup 41398 41399 Modification times and hashes 41400 41401 OpenDrive allows modification times to be set on objects accurate to 1 41402 second. These will be used to detect whether objects need syncing or 41403 not. 41404 41405 The MD5 hash algorithm is supported. 41406 41407 Restricted filename characters 41408 41409 Character Value Replacement 41410 ----------- ------- ------------- 41411 NUL 0x00 ␀ 41412 / 0x2F / 41413 " 0x22 " 41414 * 0x2A * 41415 : 0x3A : 41416 < 0x3C < 41417 > 0x3E > 41418 ? 0x3F ? 41419 \ 0x5C \ 41420 | 0x7C | 41421 41422 File names can also not begin or end with the following characters. 41423 These only get replaced if they are the first or last character in the 41424 name: 41425 41426 Character Value Replacement 41427 ----------- ------- ------------- 41428 SP 0x20 ␠ 41429 HT 0x09 ␉ 41430 LF 0x0A ␊ 41431 VT 0x0B ␋ 41432 CR 0x0D ␍ 41433 41434 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 41435 strings. 41436 41437 Standard options 41438 41439 Here are the Standard options specific to opendrive (OpenDrive). 41440 41441 --opendrive-username 41442 41443 Username. 41444 41445 Properties: 41446 41447 - Config: username 41448 - Env Var: RCLONE_OPENDRIVE_USERNAME 41449 - Type: string 41450 - Required: true 41451 41452 --opendrive-password 41453 41454 Password. 41455 41456 NB Input to this must be obscured - see rclone obscure. 41457 41458 Properties: 41459 41460 - Config: password 41461 - Env Var: RCLONE_OPENDRIVE_PASSWORD 41462 - Type: string 41463 - Required: true 41464 41465 Advanced options 41466 41467 Here are the Advanced options specific to opendrive (OpenDrive). 41468 41469 --opendrive-encoding 41470 41471 The encoding for the backend. 41472 41473 See the encoding section in the overview for more info. 41474 41475 Properties: 41476 41477 - Config: encoding 41478 - Env Var: RCLONE_OPENDRIVE_ENCODING 41479 - Type: Encoding 41480 - Default: 41481 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot 41482 41483 --opendrive-chunk-size 41484 41485 Files will be uploaded in chunks this size. 41486 41487 Note that these chunks are buffered in memory so increasing them will 41488 increase memory use. 41489 41490 Properties: 41491 41492 - Config: chunk_size 41493 - Env Var: RCLONE_OPENDRIVE_CHUNK_SIZE 41494 - Type: SizeSuffix 41495 - Default: 10Mi 41496 41497 --opendrive-description 41498 41499 Description of the remote 41500 41501 Properties: 41502 41503 - Config: description 41504 - Env Var: RCLONE_OPENDRIVE_DESCRIPTION 41505 - Type: string 41506 - Required: false 41507 41508 Limitations 41509 41510 Note that OpenDrive is case insensitive so you can't have a file called 41511 "Hello.doc" and one called "hello.doc". 41512 41513 There are quite a few characters that can't be in OpenDrive file names. 41514 These can't occur on Windows platforms, but on non-Windows platforms 41515 they are common. Rclone will map these names to and from an identical 41516 looking unicode equivalent. For example if a file has a ? in it will be 41517 mapped to ? instead. 41518 41519 rclone about is not supported by the OpenDrive backend. Backends without 41520 this capability cannot determine free space for an rclone mount or use 41521 policy mfs (most free space) as a member of an rclone union remote. 41522 41523 See List of backends that do not support rclone about and rclone about 41524 41525 Oracle Object Storage 41526 41527 - Oracle Object Storage Overview 41528 - Oracle Object Storage FAQ 41529 - Oracle Object Storage Limits 41530 41531 Paths are specified as remote:bucket (or remote: for the lsd command.) 41532 You may put subdirectories in too, e.g. remote:bucket/path/to/dir. 41533 41534 Sample command to transfer local artifacts to remote:bucket in oracle 41535 object storage: 41536 41537 rclone -vvv --progress --stats-one-line --max-stats-groups 10 --log-format date,time,UTC,longfile --fast-list --buffer-size 256Mi --oos-no-check-bucket --oos-upload-cutoff 10Mi --multi-thread-cutoff 16Mi --multi-thread-streams 3000 --transfers 3000 --checkers 64 --retries 2 --oos-chunk-size 10Mi --oos-upload-concurrency 10000 --oos-attempt-resume-upload --oos-leave-parts-on-error sync ./artifacts remote:bucket -vv 41538 41539 Configuration 41540 41541 Here is an example of making an oracle object storage configuration. 41542 rclone config walks you through it. 41543 41544 Here is an example of how to make a remote called remote. First run: 41545 41546 rclone config 41547 41548 This will guide you through an interactive setup process: 41549 41550 n) New remote 41551 d) Delete remote 41552 r) Rename remote 41553 c) Copy remote 41554 s) Set configuration password 41555 q) Quit config 41556 e/n/d/r/c/s/q> n 41557 41558 Enter name for new remote. 41559 name> remote 41560 41561 Option Storage. 41562 Type of storage to configure. 41563 Choose a number from below, or type in your own value. 41564 [snip] 41565 XX / Oracle Cloud Infrastructure Object Storage 41566 \ (oracleobjectstorage) 41567 Storage> oracleobjectstorage 41568 41569 Option provider. 41570 Choose your Auth Provider 41571 Choose a number from below, or type in your own string value. 41572 Press Enter for the default (env_auth). 41573 1 / automatically pickup the credentials from runtime(env), first one to provide auth wins 41574 \ (env_auth) 41575 / use an OCI user and an API key for authentication. 41576 2 | you’ll need to put in a config file your tenancy OCID, user OCID, region, the path, fingerprint to an API key. 41577 | https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm 41578 \ (user_principal_auth) 41579 / use instance principals to authorize an instance to make API calls. 41580 3 | each instance has its own identity, and authenticates using the certificates that are read from instance metadata. 41581 | https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm 41582 \ (instance_principal_auth) 41583 / use workload identity to grant Kubernetes pods policy-driven access to Oracle Cloud 41584 4 | Infrastructure (OCI) resources using OCI Identity and Access Management (IAM). 41585 | https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm 41586 \ (workload_identity_auth) 41587 5 / use resource principals to make API calls 41588 \ (resource_principal_auth) 41589 6 / no credentials needed, this is typically for reading public buckets 41590 \ (no_auth) 41591 provider> 2 41592 41593 Option namespace. 41594 Object storage namespace 41595 Enter a value. 41596 namespace> idbamagbg734 41597 41598 Option compartment. 41599 Object storage compartment OCID 41600 Enter a value. 41601 compartment> ocid1.compartment.oc1..aaaaaaaapufkxc7ame3sthry5i7ujrwfc7ejnthhu6bhanm5oqfjpyasjkba 41602 41603 Option region. 41604 Object storage Region 41605 Enter a value. 41606 region> us-ashburn-1 41607 41608 Option endpoint. 41609 Endpoint for Object storage API. 41610 Leave blank to use the default endpoint for the region. 41611 Enter a value. Press Enter to leave empty. 41612 endpoint> 41613 41614 Option config_file. 41615 Full Path to OCI config file 41616 Choose a number from below, or type in your own string value. 41617 Press Enter for the default (~/.oci/config). 41618 1 / oci configuration file location 41619 \ (~/.oci/config) 41620 config_file> /etc/oci/dev.conf 41621 41622 Option config_profile. 41623 Profile name inside OCI config file 41624 Choose a number from below, or type in your own string value. 41625 Press Enter for the default (Default). 41626 1 / Use the default profile 41627 \ (Default) 41628 config_profile> Test 41629 41630 Edit advanced config? 41631 y) Yes 41632 n) No (default) 41633 y/n> n 41634 41635 Configuration complete. 41636 Options: 41637 - type: oracleobjectstorage 41638 - namespace: idbamagbg734 41639 - compartment: ocid1.compartment.oc1..aaaaaaaapufkxc7ame3sthry5i7ujrwfc7ejnthhu6bhanm5oqfjpyasjkba 41640 - region: us-ashburn-1 41641 - provider: user_principal_auth 41642 - config_file: /etc/oci/dev.conf 41643 - config_profile: Test 41644 Keep this "remote" remote? 41645 y) Yes this is OK (default) 41646 e) Edit this remote 41647 d) Delete this remote 41648 y/e/d> y 41649 41650 See all buckets 41651 41652 rclone lsd remote: 41653 41654 Create a new bucket 41655 41656 rclone mkdir remote:bucket 41657 41658 List the contents of a bucket 41659 41660 rclone ls remote:bucket 41661 rclone ls remote:bucket --max-depth 1 41662 41663 Authentication Providers 41664 41665 OCI has various authentication methods. To learn more about 41666 authentication methods please refer oci authentication methods These 41667 choices can be specified in the rclone config file. 41668 41669 Rclone supports the following OCI authentication provider. 41670 41671 User Principal 41672 Instance Principal 41673 Resource Principal 41674 Workload Identity 41675 No authentication 41676 41677 User Principal 41678 41679 Sample rclone config file for Authentication Provider User Principal: 41680 41681 [oos] 41682 type = oracleobjectstorage 41683 namespace = id<redacted>34 41684 compartment = ocid1.compartment.oc1..aa<redacted>ba 41685 region = us-ashburn-1 41686 provider = user_principal_auth 41687 config_file = /home/opc/.oci/config 41688 config_profile = Default 41689 41690 Advantages: - One can use this method from any server within OCI or 41691 on-premises or from other cloud provider. 41692 41693 Considerations: - you need to configure user’s privileges / policy to 41694 allow access to object storage - Overhead of managing users and keys. - 41695 If the user is deleted, the config file will no longer work and may 41696 cause automation regressions that use the user's credentials. 41697 41698 Instance Principal 41699 41700 An OCI compute instance can be authorized to use rclone by using it's 41701 identity and certificates as an instance principal. With this approach 41702 no credentials have to be stored and managed. 41703 41704 Sample rclone configuration file for Authentication Provider Instance 41705 Principal: 41706 41707 [opc@rclone ~]$ cat ~/.config/rclone/rclone.conf 41708 [oos] 41709 type = oracleobjectstorage 41710 namespace = id<redacted>fn 41711 compartment = ocid1.compartment.oc1..aa<redacted>k7a 41712 region = us-ashburn-1 41713 provider = instance_principal_auth 41714 41715 Advantages: 41716 41717 - With instance principals, you don't need to configure user 41718 credentials and transfer/ save it to disk in your compute instances 41719 or rotate the credentials. 41720 - You don’t need to deal with users and keys. 41721 - Greatly helps in automation as you don't have to manage access keys, 41722 user private keys, storing them in vault, using kms etc. 41723 41724 Considerations: 41725 41726 - You need to configure a dynamic group having this instance as member 41727 and add policy to read object storage to that dynamic group. 41728 - Everyone who has access to this machine can execute the CLI 41729 commands. 41730 - It is applicable for oci compute instances only. It cannot be used 41731 on external instance or resources. 41732 41733 Resource Principal 41734 41735 Resource principal auth is very similar to instance principal auth but 41736 used for resources that are not compute instances such as serverless 41737 functions. To use resource principal ensure Rclone process is started 41738 with these environment variables set in its process. 41739 41740 export OCI_RESOURCE_PRINCIPAL_VERSION=2.2 41741 export OCI_RESOURCE_PRINCIPAL_REGION=us-ashburn-1 41742 export OCI_RESOURCE_PRINCIPAL_PRIVATE_PEM=/usr/share/model-server/key.pem 41743 export OCI_RESOURCE_PRINCIPAL_RPST=/usr/share/model-server/security_token 41744 41745 Sample rclone configuration file for Authentication Provider Resource 41746 Principal: 41747 41748 [oos] 41749 type = oracleobjectstorage 41750 namespace = id<redacted>34 41751 compartment = ocid1.compartment.oc1..aa<redacted>ba 41752 region = us-ashburn-1 41753 provider = resource_principal_auth 41754 41755 Workload Identity 41756 41757 Workload Identity auth may be used when running Rclone from Kubernetes 41758 pod on a Container Engine for Kubernetes (OKE) cluster. For more details 41759 on configuring Workload Identity, see Granting Workloads Access to OCI 41760 Resources. To use workload identity, ensure Rclone is started with these 41761 environment variables set in its process. 41762 41763 export OCI_RESOURCE_PRINCIPAL_VERSION=2.2 41764 export OCI_RESOURCE_PRINCIPAL_REGION=us-ashburn-1 41765 41766 No authentication 41767 41768 Public buckets do not require any authentication mechanism to read 41769 objects. Sample rclone configuration file for No authentication: 41770 41771 [oos] 41772 type = oracleobjectstorage 41773 namespace = id<redacted>34 41774 compartment = ocid1.compartment.oc1..aa<redacted>ba 41775 region = us-ashburn-1 41776 provider = no_auth 41777 41778 Modification times and hashes 41779 41780 The modification time is stored as metadata on the object as 41781 opc-meta-mtime as floating point since the epoch, accurate to 1 ns. 41782 41783 If the modification time needs to be updated rclone will attempt to 41784 perform a server side copy to update the modification if the object can 41785 be copied in a single part. In the case the object is larger than 5Gb, 41786 the object will be uploaded rather than copied. 41787 41788 Note that reading this from the object takes an additional HEAD request 41789 as the metadata isn't returned in object listings. 41790 41791 The MD5 hash algorithm is supported. 41792 41793 Multipart uploads 41794 41795 rclone supports multipart uploads with OOS which means that it can 41796 upload files bigger than 5 GiB. 41797 41798 Note that files uploaded both with multipart upload and through crypt 41799 remotes do not have MD5 sums. 41800 41801 rclone switches from single part uploads to multipart uploads at the 41802 point specified by --oos-upload-cutoff. This can be a maximum of 5 GiB 41803 and a minimum of 0 (ie always upload multipart files). 41804 41805 The chunk sizes used in the multipart upload are specified by 41806 --oos-chunk-size and the number of chunks uploaded concurrently is 41807 specified by --oos-upload-concurrency. 41808 41809 Multipart uploads will use --transfers * --oos-upload-concurrency * 41810 --oos-chunk-size extra memory. Single part uploads to not use extra 41811 memory. 41812 41813 Single part transfers can be faster than multipart transfers or slower 41814 depending on your latency from oos - the more latency, the more likely 41815 single part transfers will be faster. 41816 41817 Increasing --oos-upload-concurrency will increase throughput (8 would be 41818 a sensible value) and increasing --oos-chunk-size also increases 41819 throughput (16M would be sensible). Increasing either of these will use 41820 more memory. The default values are high enough to gain most of the 41821 possible performance without using too much memory. 41822 41823 Standard options 41824 41825 Here are the Standard options specific to oracleobjectstorage (Oracle 41826 Cloud Infrastructure Object Storage). 41827 41828 --oos-provider 41829 41830 Choose your Auth Provider 41831 41832 Properties: 41833 41834 - Config: provider 41835 - Env Var: RCLONE_OOS_PROVIDER 41836 - Type: string 41837 - Default: "env_auth" 41838 - Examples: 41839 - "env_auth" 41840 - automatically pickup the credentials from runtime(env), 41841 first one to provide auth wins 41842 - "user_principal_auth" 41843 - use an OCI user and an API key for authentication. 41844 - you’ll need to put in a config file your tenancy OCID, user 41845 OCID, region, the path, fingerprint to an API key. 41846 - https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm 41847 - "instance_principal_auth" 41848 - use instance principals to authorize an instance to make API 41849 calls. 41850 - each instance has its own identity, and authenticates using 41851 the certificates that are read from instance metadata. 41852 - https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm 41853 - "workload_identity_auth" 41854 - use workload identity to grant OCI Container Engine for 41855 Kubernetes workloads policy-driven access to OCI resources 41856 using OCI Identity and Access Management (IAM). 41857 - https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm 41858 - "resource_principal_auth" 41859 - use resource principals to make API calls 41860 - "no_auth" 41861 - no credentials needed, this is typically for reading public 41862 buckets 41863 41864 --oos-namespace 41865 41866 Object storage namespace 41867 41868 Properties: 41869 41870 - Config: namespace 41871 - Env Var: RCLONE_OOS_NAMESPACE 41872 - Type: string 41873 - Required: true 41874 41875 --oos-compartment 41876 41877 Object storage compartment OCID 41878 41879 Properties: 41880 41881 - Config: compartment 41882 - Env Var: RCLONE_OOS_COMPARTMENT 41883 - Provider: !no_auth 41884 - Type: string 41885 - Required: true 41886 41887 --oos-region 41888 41889 Object storage Region 41890 41891 Properties: 41892 41893 - Config: region 41894 - Env Var: RCLONE_OOS_REGION 41895 - Type: string 41896 - Required: true 41897 41898 --oos-endpoint 41899 41900 Endpoint for Object storage API. 41901 41902 Leave blank to use the default endpoint for the region. 41903 41904 Properties: 41905 41906 - Config: endpoint 41907 - Env Var: RCLONE_OOS_ENDPOINT 41908 - Type: string 41909 - Required: false 41910 41911 --oos-config-file 41912 41913 Path to OCI config file 41914 41915 Properties: 41916 41917 - Config: config_file 41918 - Env Var: RCLONE_OOS_CONFIG_FILE 41919 - Provider: user_principal_auth 41920 - Type: string 41921 - Default: "~/.oci/config" 41922 - Examples: 41923 - "~/.oci/config" 41924 - oci configuration file location 41925 41926 --oos-config-profile 41927 41928 Profile name inside the oci config file 41929 41930 Properties: 41931 41932 - Config: config_profile 41933 - Env Var: RCLONE_OOS_CONFIG_PROFILE 41934 - Provider: user_principal_auth 41935 - Type: string 41936 - Default: "Default" 41937 - Examples: 41938 - "Default" 41939 - Use the default profile 41940 41941 Advanced options 41942 41943 Here are the Advanced options specific to oracleobjectstorage (Oracle 41944 Cloud Infrastructure Object Storage). 41945 41946 --oos-storage-tier 41947 41948 The storage class to use when storing new objects in storage. 41949 https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/understandingstoragetiers.htm 41950 41951 Properties: 41952 41953 - Config: storage_tier 41954 - Env Var: RCLONE_OOS_STORAGE_TIER 41955 - Type: string 41956 - Default: "Standard" 41957 - Examples: 41958 - "Standard" 41959 - Standard storage tier, this is the default tier 41960 - "InfrequentAccess" 41961 - InfrequentAccess storage tier 41962 - "Archive" 41963 - Archive storage tier 41964 41965 --oos-upload-cutoff 41966 41967 Cutoff for switching to chunked upload. 41968 41969 Any files larger than this will be uploaded in chunks of chunk_size. The 41970 minimum is 0 and the maximum is 5 GiB. 41971 41972 Properties: 41973 41974 - Config: upload_cutoff 41975 - Env Var: RCLONE_OOS_UPLOAD_CUTOFF 41976 - Type: SizeSuffix 41977 - Default: 200Mi 41978 41979 --oos-chunk-size 41980 41981 Chunk size to use for uploading. 41982 41983 When uploading files larger than upload_cutoff or files with unknown 41984 size (e.g. from "rclone rcat" or uploaded with "rclone mount" they will 41985 be uploaded as multipart uploads using this chunk size. 41986 41987 Note that "upload_concurrency" chunks of this size are buffered in 41988 memory per transfer. 41989 41990 If you are transferring large files over high-speed links and you have 41991 enough memory, then increasing this will speed up the transfers. 41992 41993 Rclone will automatically increase the chunk size when uploading a large 41994 file of known size to stay below the 10,000 chunks limit. 41995 41996 Files of unknown size are uploaded with the configured chunk_size. Since 41997 the default chunk size is 5 MiB and there can be at most 10,000 chunks, 41998 this means that by default the maximum size of a file you can stream 41999 upload is 48 GiB. If you wish to stream upload larger files then you 42000 will need to increase chunk_size. 42001 42002 Increasing the chunk size decreases the accuracy of the progress 42003 statistics displayed with "-P" flag. 42004 42005 Properties: 42006 42007 - Config: chunk_size 42008 - Env Var: RCLONE_OOS_CHUNK_SIZE 42009 - Type: SizeSuffix 42010 - Default: 5Mi 42011 42012 --oos-max-upload-parts 42013 42014 Maximum number of parts in a multipart upload. 42015 42016 This option defines the maximum number of multipart chunks to use when 42017 doing a multipart upload. 42018 42019 OCI has max parts limit of 10,000 chunks. 42020 42021 Rclone will automatically increase the chunk size when uploading a large 42022 file of a known size to stay below this number of chunks limit. 42023 42024 Properties: 42025 42026 - Config: max_upload_parts 42027 - Env Var: RCLONE_OOS_MAX_UPLOAD_PARTS 42028 - Type: int 42029 - Default: 10000 42030 42031 --oos-upload-concurrency 42032 42033 Concurrency for multipart uploads. 42034 42035 This is the number of chunks of the same file that are uploaded 42036 concurrently. 42037 42038 If you are uploading small numbers of large files over high-speed links 42039 and these uploads do not fully utilize your bandwidth, then increasing 42040 this may help to speed up the transfers. 42041 42042 Properties: 42043 42044 - Config: upload_concurrency 42045 - Env Var: RCLONE_OOS_UPLOAD_CONCURRENCY 42046 - Type: int 42047 - Default: 10 42048 42049 --oos-copy-cutoff 42050 42051 Cutoff for switching to multipart copy. 42052 42053 Any files larger than this that need to be server-side copied will be 42054 copied in chunks of this size. 42055 42056 The minimum is 0 and the maximum is 5 GiB. 42057 42058 Properties: 42059 42060 - Config: copy_cutoff 42061 - Env Var: RCLONE_OOS_COPY_CUTOFF 42062 - Type: SizeSuffix 42063 - Default: 4.656Gi 42064 42065 --oos-copy-timeout 42066 42067 Timeout for copy. 42068 42069 Copy is an asynchronous operation, specify timeout to wait for copy to 42070 succeed 42071 42072 Properties: 42073 42074 - Config: copy_timeout 42075 - Env Var: RCLONE_OOS_COPY_TIMEOUT 42076 - Type: Duration 42077 - Default: 1m0s 42078 42079 --oos-disable-checksum 42080 42081 Don't store MD5 checksum with object metadata. 42082 42083 Normally rclone will calculate the MD5 checksum of the input before 42084 uploading it so it can add it to metadata on the object. This is great 42085 for data integrity checking but can cause long delays for large files to 42086 start uploading. 42087 42088 Properties: 42089 42090 - Config: disable_checksum 42091 - Env Var: RCLONE_OOS_DISABLE_CHECKSUM 42092 - Type: bool 42093 - Default: false 42094 42095 --oos-encoding 42096 42097 The encoding for the backend. 42098 42099 See the encoding section in the overview for more info. 42100 42101 Properties: 42102 42103 - Config: encoding 42104 - Env Var: RCLONE_OOS_ENCODING 42105 - Type: Encoding 42106 - Default: Slash,InvalidUtf8,Dot 42107 42108 --oos-leave-parts-on-error 42109 42110 If true avoid calling abort upload on a failure, leaving all 42111 successfully uploaded parts for manual recovery. 42112 42113 It should be set to true for resuming uploads across different sessions. 42114 42115 WARNING: Storing parts of an incomplete multipart upload counts towards 42116 space usage on object storage and will add additional costs if not 42117 cleaned up. 42118 42119 Properties: 42120 42121 - Config: leave_parts_on_error 42122 - Env Var: RCLONE_OOS_LEAVE_PARTS_ON_ERROR 42123 - Type: bool 42124 - Default: false 42125 42126 --oos-attempt-resume-upload 42127 42128 If true attempt to resume previously started multipart upload for the 42129 object. This will be helpful to speed up multipart transfers by resuming 42130 uploads from past session. 42131 42132 WARNING: If chunk size differs in resumed session from past incomplete 42133 session, then the resumed multipart upload is aborted and a new 42134 multipart upload is started with the new chunk size. 42135 42136 The flag leave_parts_on_error must be true to resume and optimize to 42137 skip parts that were already uploaded successfully. 42138 42139 Properties: 42140 42141 - Config: attempt_resume_upload 42142 - Env Var: RCLONE_OOS_ATTEMPT_RESUME_UPLOAD 42143 - Type: bool 42144 - Default: false 42145 42146 --oos-no-check-bucket 42147 42148 If set, don't attempt to check the bucket exists or create it. 42149 42150 This can be useful when trying to minimise the number of transactions 42151 rclone does if you know the bucket exists already. 42152 42153 It can also be needed if the user you are using does not have bucket 42154 creation permissions. 42155 42156 Properties: 42157 42158 - Config: no_check_bucket 42159 - Env Var: RCLONE_OOS_NO_CHECK_BUCKET 42160 - Type: bool 42161 - Default: false 42162 42163 --oos-sse-customer-key-file 42164 42165 To use SSE-C, a file containing the base64-encoded string of the AES-256 42166 encryption key associated with the object. Please note only one of 42167 sse_customer_key_file|sse_customer_key|sse_kms_key_id is needed.' 42168 42169 Properties: 42170 42171 - Config: sse_customer_key_file 42172 - Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY_FILE 42173 - Type: string 42174 - Required: false 42175 - Examples: 42176 - "" 42177 - None 42178 42179 --oos-sse-customer-key 42180 42181 To use SSE-C, the optional header that specifies the base64-encoded 42182 256-bit encryption key to use to encrypt or decrypt the data. Please 42183 note only one of sse_customer_key_file|sse_customer_key|sse_kms_key_id 42184 is needed. For more information, see Using Your Own Keys for Server-Side 42185 Encryption 42186 (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm) 42187 42188 Properties: 42189 42190 - Config: sse_customer_key 42191 - Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY 42192 - Type: string 42193 - Required: false 42194 - Examples: 42195 - "" 42196 - None 42197 42198 --oos-sse-customer-key-sha256 42199 42200 If using SSE-C, The optional header that specifies the base64-encoded 42201 SHA256 hash of the encryption key. This value is used to check the 42202 integrity of the encryption key. see Using Your Own Keys for Server-Side 42203 Encryption 42204 (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm). 42205 42206 Properties: 42207 42208 - Config: sse_customer_key_sha256 42209 - Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY_SHA256 42210 - Type: string 42211 - Required: false 42212 - Examples: 42213 - "" 42214 - None 42215 42216 --oos-sse-kms-key-id 42217 42218 if using your own master key in vault, this header specifies the OCID 42219 (https://docs.cloud.oracle.com/Content/General/Concepts/identifiers.htm) 42220 of a master encryption key used to call the Key Management service to 42221 generate a data encryption key or to encrypt or decrypt a data 42222 encryption key. Please note only one of 42223 sse_customer_key_file|sse_customer_key|sse_kms_key_id is needed. 42224 42225 Properties: 42226 42227 - Config: sse_kms_key_id 42228 - Env Var: RCLONE_OOS_SSE_KMS_KEY_ID 42229 - Type: string 42230 - Required: false 42231 - Examples: 42232 - "" 42233 - None 42234 42235 --oos-sse-customer-algorithm 42236 42237 If using SSE-C, the optional header that specifies "AES256" as the 42238 encryption algorithm. Object Storage supports "AES256" as the encryption 42239 algorithm. For more information, see Using Your Own Keys for Server-Side 42240 Encryption 42241 (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm). 42242 42243 Properties: 42244 42245 - Config: sse_customer_algorithm 42246 - Env Var: RCLONE_OOS_SSE_CUSTOMER_ALGORITHM 42247 - Type: string 42248 - Required: false 42249 - Examples: 42250 - "" 42251 - None 42252 - "AES256" 42253 - AES256 42254 42255 --oos-description 42256 42257 Description of the remote 42258 42259 Properties: 42260 42261 - Config: description 42262 - Env Var: RCLONE_OOS_DESCRIPTION 42263 - Type: string 42264 - Required: false 42265 42266 Backend commands 42267 42268 Here are the commands specific to the oracleobjectstorage backend. 42269 42270 Run them with 42271 42272 rclone backend COMMAND remote: 42273 42274 The help below will explain what arguments each command takes. 42275 42276 See the backend command for more info on how to pass options and 42277 arguments. 42278 42279 These can be run on a running backend using the rc command 42280 backend/command. 42281 42282 rename 42283 42284 change the name of an object 42285 42286 rclone backend rename remote: [options] [<arguments>+] 42287 42288 This command can be used to rename a object. 42289 42290 Usage Examples: 42291 42292 rclone backend rename oos:bucket relative-object-path-under-bucket object-new-name 42293 42294 list-multipart-uploads 42295 42296 List the unfinished multipart uploads 42297 42298 rclone backend list-multipart-uploads remote: [options] [<arguments>+] 42299 42300 This command lists the unfinished multipart uploads in JSON format. 42301 42302 rclone backend list-multipart-uploads oos:bucket/path/to/object 42303 42304 It returns a dictionary of buckets with values as lists of unfinished 42305 multipart uploads. 42306 42307 You can call it with no bucket in which case it lists all bucket, with a 42308 bucket or with a bucket and path. 42309 42310 { 42311 "test-bucket": [ 42312 { 42313 "namespace": "test-namespace", 42314 "bucket": "test-bucket", 42315 "object": "600m.bin", 42316 "uploadId": "51dd8114-52a4-b2f2-c42f-5291f05eb3c8", 42317 "timeCreated": "2022-07-29T06:21:16.595Z", 42318 "storageTier": "Standard" 42319 } 42320 ] 42321 42322 cleanup 42323 42324 Remove unfinished multipart uploads. 42325 42326 rclone backend cleanup remote: [options] [<arguments>+] 42327 42328 This command removes unfinished multipart uploads of age greater than 42329 max-age which defaults to 24 hours. 42330 42331 Note that you can use --interactive/-i or --dry-run with this command to 42332 see what it would do. 42333 42334 rclone backend cleanup oos:bucket/path/to/object 42335 rclone backend cleanup -o max-age=7w oos:bucket/path/to/object 42336 42337 Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc. 42338 42339 Options: 42340 42341 - "max-age": Max age of upload to delete 42342 42343 restore 42344 42345 Restore objects from Archive to Standard storage 42346 42347 rclone backend restore remote: [options] [<arguments>+] 42348 42349 This command can be used to restore one or more objects from Archive to 42350 Standard storage. 42351 42352 Usage Examples: 42353 42354 rclone backend restore oos:bucket/path/to/directory -o hours=HOURS 42355 rclone backend restore oos:bucket -o hours=HOURS 42356 42357 This flag also obeys the filters. Test first with --interactive/-i or 42358 --dry-run flags 42359 42360 rclone --interactive backend restore --include "*.txt" oos:bucket/path -o hours=72 42361 42362 All the objects shown will be marked for restore, then 42363 42364 rclone backend restore --include "*.txt" oos:bucket/path -o hours=72 42365 42366 It returns a list of status dictionaries with Object Name and Status 42367 keys. The Status will be "RESTORED"" if it was successful or an error message 42368 if not. 42369 42370 [ 42371 { 42372 "Object": "test.txt" 42373 "Status": "RESTORED", 42374 }, 42375 { 42376 "Object": "test/file4.txt" 42377 "Status": "RESTORED", 42378 } 42379 ] 42380 42381 Options: 42382 42383 - "hours": The number of hours for which this object will be restored. 42384 Default is 24 hrs. 42385 42386 Tutorials 42387 42388 Mounting Buckets 42389 42390 QingStor 42391 42392 Paths are specified as remote:bucket (or remote: for the lsd command.) 42393 You may put subdirectories in too, e.g. remote:bucket/path/to/dir. 42394 42395 Configuration 42396 42397 Here is an example of making an QingStor configuration. First run 42398 42399 rclone config 42400 42401 This will guide you through an interactive setup process. 42402 42403 No remotes found, make a new one? 42404 n) New remote 42405 r) Rename remote 42406 c) Copy remote 42407 s) Set configuration password 42408 q) Quit config 42409 n/r/c/s/q> n 42410 name> remote 42411 Type of storage to configure. 42412 Choose a number from below, or type in your own value 42413 [snip] 42414 XX / QingStor Object Storage 42415 \ "qingstor" 42416 [snip] 42417 Storage> qingstor 42418 Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank. 42419 Choose a number from below, or type in your own value 42420 1 / Enter QingStor credentials in the next step 42421 \ "false" 42422 2 / Get QingStor credentials from the environment (env vars or IAM) 42423 \ "true" 42424 env_auth> 1 42425 QingStor Access Key ID - leave blank for anonymous access or runtime credentials. 42426 access_key_id> access_key 42427 QingStor Secret Access Key (password) - leave blank for anonymous access or runtime credentials. 42428 secret_access_key> secret_key 42429 Enter an endpoint URL to connection QingStor API. 42430 Leave blank will use the default value "https://qingstor.com:443" 42431 endpoint> 42432 Zone connect to. Default is "pek3a". 42433 Choose a number from below, or type in your own value 42434 / The Beijing (China) Three Zone 42435 1 | Needs location constraint pek3a. 42436 \ "pek3a" 42437 / The Shanghai (China) First Zone 42438 2 | Needs location constraint sh1a. 42439 \ "sh1a" 42440 zone> 1 42441 Number of connection retry. 42442 Leave blank will use the default value "3". 42443 connection_retries> 42444 Remote config 42445 -------------------- 42446 [remote] 42447 env_auth = false 42448 access_key_id = access_key 42449 secret_access_key = secret_key 42450 endpoint = 42451 zone = pek3a 42452 connection_retries = 42453 -------------------- 42454 y) Yes this is OK 42455 e) Edit this remote 42456 d) Delete this remote 42457 y/e/d> y 42458 42459 This remote is called remote and can now be used like this 42460 42461 See all buckets 42462 42463 rclone lsd remote: 42464 42465 Make a new bucket 42466 42467 rclone mkdir remote:bucket 42468 42469 List the contents of a bucket 42470 42471 rclone ls remote:bucket 42472 42473 Sync /home/local/directory to the remote bucket, deleting any excess 42474 files in the bucket. 42475 42476 rclone sync --interactive /home/local/directory remote:bucket 42477 42478 --fast-list 42479 42480 This remote supports --fast-list which allows you to use fewer 42481 transactions in exchange for more memory. See the rclone docs for more 42482 details. 42483 42484 Multipart uploads 42485 42486 rclone supports multipart uploads with QingStor which means that it can 42487 upload files bigger than 5 GiB. Note that files uploaded with multipart 42488 upload don't have an MD5SUM. 42489 42490 Note that incomplete multipart uploads older than 24 hours can be 42491 removed with rclone cleanup remote:bucket just for one bucket 42492 rclone cleanup remote: for all buckets. QingStor does not ever remove 42493 incomplete multipart uploads so it may be necessary to run this from 42494 time to time. 42495 42496 Buckets and Zone 42497 42498 With QingStor you can list buckets (rclone lsd) using any zone, but you 42499 can only access the content of a bucket from the zone it was created in. 42500 If you attempt to access a bucket from the wrong zone, you will get an 42501 error, incorrect zone, the bucket is not in 'XXX' zone. 42502 42503 Authentication 42504 42505 There are two ways to supply rclone with a set of QingStor credentials. 42506 In order of precedence: 42507 42508 - Directly in the rclone configuration file (as configured by 42509 rclone config) 42510 - set access_key_id and secret_access_key 42511 - Runtime configuration: 42512 - set env_auth to true in the config file 42513 - Exporting the following environment variables before running 42514 rclone 42515 - Access Key ID: QS_ACCESS_KEY_ID or QS_ACCESS_KEY 42516 - Secret Access Key: QS_SECRET_ACCESS_KEY or QS_SECRET_KEY 42517 42518 Restricted filename characters 42519 42520 The control characters 0x00-0x1F and / are replaced as in the default 42521 restricted characters set. Note that 0x7F is not replaced. 42522 42523 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 42524 strings. 42525 42526 Standard options 42527 42528 Here are the Standard options specific to qingstor (QingCloud Object 42529 Storage). 42530 42531 --qingstor-env-auth 42532 42533 Get QingStor credentials from runtime. 42534 42535 Only applies if access_key_id and secret_access_key is blank. 42536 42537 Properties: 42538 42539 - Config: env_auth 42540 - Env Var: RCLONE_QINGSTOR_ENV_AUTH 42541 - Type: bool 42542 - Default: false 42543 - Examples: 42544 - "false" 42545 - Enter QingStor credentials in the next step. 42546 - "true" 42547 - Get QingStor credentials from the environment (env vars or 42548 IAM). 42549 42550 --qingstor-access-key-id 42551 42552 QingStor Access Key ID. 42553 42554 Leave blank for anonymous access or runtime credentials. 42555 42556 Properties: 42557 42558 - Config: access_key_id 42559 - Env Var: RCLONE_QINGSTOR_ACCESS_KEY_ID 42560 - Type: string 42561 - Required: false 42562 42563 --qingstor-secret-access-key 42564 42565 QingStor Secret Access Key (password). 42566 42567 Leave blank for anonymous access or runtime credentials. 42568 42569 Properties: 42570 42571 - Config: secret_access_key 42572 - Env Var: RCLONE_QINGSTOR_SECRET_ACCESS_KEY 42573 - Type: string 42574 - Required: false 42575 42576 --qingstor-endpoint 42577 42578 Enter an endpoint URL to connection QingStor API. 42579 42580 Leave blank will use the default value "https://qingstor.com:443". 42581 42582 Properties: 42583 42584 - Config: endpoint 42585 - Env Var: RCLONE_QINGSTOR_ENDPOINT 42586 - Type: string 42587 - Required: false 42588 42589 --qingstor-zone 42590 42591 Zone to connect to. 42592 42593 Default is "pek3a". 42594 42595 Properties: 42596 42597 - Config: zone 42598 - Env Var: RCLONE_QINGSTOR_ZONE 42599 - Type: string 42600 - Required: false 42601 - Examples: 42602 - "pek3a" 42603 - The Beijing (China) Three Zone. 42604 - Needs location constraint pek3a. 42605 - "sh1a" 42606 - The Shanghai (China) First Zone. 42607 - Needs location constraint sh1a. 42608 - "gd2a" 42609 - The Guangdong (China) Second Zone. 42610 - Needs location constraint gd2a. 42611 42612 Advanced options 42613 42614 Here are the Advanced options specific to qingstor (QingCloud Object 42615 Storage). 42616 42617 --qingstor-connection-retries 42618 42619 Number of connection retries. 42620 42621 Properties: 42622 42623 - Config: connection_retries 42624 - Env Var: RCLONE_QINGSTOR_CONNECTION_RETRIES 42625 - Type: int 42626 - Default: 3 42627 42628 --qingstor-upload-cutoff 42629 42630 Cutoff for switching to chunked upload. 42631 42632 Any files larger than this will be uploaded in chunks of chunk_size. The 42633 minimum is 0 and the maximum is 5 GiB. 42634 42635 Properties: 42636 42637 - Config: upload_cutoff 42638 - Env Var: RCLONE_QINGSTOR_UPLOAD_CUTOFF 42639 - Type: SizeSuffix 42640 - Default: 200Mi 42641 42642 --qingstor-chunk-size 42643 42644 Chunk size to use for uploading. 42645 42646 When uploading files larger than upload_cutoff they will be uploaded as 42647 multipart uploads using this chunk size. 42648 42649 Note that "--qingstor-upload-concurrency" chunks of this size are 42650 buffered in memory per transfer. 42651 42652 If you are transferring large files over high-speed links and you have 42653 enough memory, then increasing this will speed up the transfers. 42654 42655 Properties: 42656 42657 - Config: chunk_size 42658 - Env Var: RCLONE_QINGSTOR_CHUNK_SIZE 42659 - Type: SizeSuffix 42660 - Default: 4Mi 42661 42662 --qingstor-upload-concurrency 42663 42664 Concurrency for multipart uploads. 42665 42666 This is the number of chunks of the same file that are uploaded 42667 concurrently. 42668 42669 NB if you set this to > 1 then the checksums of multipart uploads become 42670 corrupted (the uploads themselves are not corrupted though). 42671 42672 If you are uploading small numbers of large files over high-speed links 42673 and these uploads do not fully utilize your bandwidth, then increasing 42674 this may help to speed up the transfers. 42675 42676 Properties: 42677 42678 - Config: upload_concurrency 42679 - Env Var: RCLONE_QINGSTOR_UPLOAD_CONCURRENCY 42680 - Type: int 42681 - Default: 1 42682 42683 --qingstor-encoding 42684 42685 The encoding for the backend. 42686 42687 See the encoding section in the overview for more info. 42688 42689 Properties: 42690 42691 - Config: encoding 42692 - Env Var: RCLONE_QINGSTOR_ENCODING 42693 - Type: Encoding 42694 - Default: Slash,Ctl,InvalidUtf8 42695 42696 --qingstor-description 42697 42698 Description of the remote 42699 42700 Properties: 42701 42702 - Config: description 42703 - Env Var: RCLONE_QINGSTOR_DESCRIPTION 42704 - Type: string 42705 - Required: false 42706 42707 Limitations 42708 42709 rclone about is not supported by the qingstor backend. Backends without 42710 this capability cannot determine free space for an rclone mount or use 42711 policy mfs (most free space) as a member of an rclone union remote. 42712 42713 See List of backends that do not support rclone about and rclone about 42714 42715 Quatrix 42716 42717 Quatrix by Maytech is Quatrix Secure Compliant File Sharing | Maytech. 42718 42719 Paths are specified as remote:path 42720 42721 Paths may be as deep as required, e.g., remote:directory/subdirectory. 42722 42723 The initial setup for Quatrix involves getting an API Key from Quatrix. 42724 You can get the API key in the user's profile at 42725 https://<account>/profile/api-keys or with the help of the API - 42726 https://docs.maytech.net/quatrix/quatrix-api/api-explorer#/API-Key/post_api_key_create. 42727 42728 See complete Swagger documentation for Quatrix - 42729 https://docs.maytech.net/quatrix/quatrix-api/api-explorer 42730 42731 Configuration 42732 42733 Here is an example of how to make a remote called remote. First run: 42734 42735 rclone config 42736 42737 This will guide you through an interactive setup process: 42738 42739 No remotes found, make a new one? 42740 n) New remote 42741 s) Set configuration password 42742 q) Quit config 42743 n/s/q> n 42744 name> remote 42745 Type of storage to configure. 42746 Choose a number from below, or type in your own value 42747 [snip] 42748 XX / Quatrix by Maytech 42749 \ "quatrix" 42750 [snip] 42751 Storage> quatrix 42752 API key for accessing Quatrix account. 42753 api_key> your_api_key 42754 Host name of Quatrix account. 42755 host> example.quatrix.it 42756 42757 -------------------- 42758 [remote] 42759 api_key = your_api_key 42760 host = example.quatrix.it 42761 -------------------- 42762 y) Yes this is OK 42763 e) Edit this remote 42764 d) Delete this remote 42765 y/e/d> y 42766 42767 Once configured you can then use rclone like this, 42768 42769 List directories in top level of your Quatrix 42770 42771 rclone lsd remote: 42772 42773 List all the files in your Quatrix 42774 42775 rclone ls remote: 42776 42777 To copy a local directory to an Quatrix directory called backup 42778 42779 rclone copy /home/source remote:backup 42780 42781 API key validity 42782 42783 API Key is created with no expiration date. It will be valid until you 42784 delete or deactivate it in your account. After disabling, the API Key 42785 can be enabled back. If the API Key was deleted and a new key was 42786 created, you can update it in rclone config. The same happens if the 42787 hostname was changed. 42788 42789 $ rclone config 42790 Current remotes: 42791 42792 Name Type 42793 ==== ==== 42794 remote quatrix 42795 42796 e) Edit existing remote 42797 n) New remote 42798 d) Delete remote 42799 r) Rename remote 42800 c) Copy remote 42801 s) Set configuration password 42802 q) Quit config 42803 e/n/d/r/c/s/q> e 42804 Choose a number from below, or type in an existing value 42805 1 > remote 42806 remote> remote 42807 -------------------- 42808 [remote] 42809 type = quatrix 42810 host = some_host.quatrix.it 42811 api_key = your_api_key 42812 -------------------- 42813 Edit remote 42814 Option api_key. 42815 API key for accessing Quatrix account 42816 Enter a string value. Press Enter for the default (your_api_key) 42817 api_key> 42818 Option host. 42819 Host name of Quatrix account 42820 Enter a string value. Press Enter for the default (some_host.quatrix.it). 42821 42822 -------------------- 42823 [remote] 42824 type = quatrix 42825 host = some_host.quatrix.it 42826 api_key = your_api_key 42827 -------------------- 42828 y) Yes this is OK 42829 e) Edit this remote 42830 d) Delete this remote 42831 y/e/d> y 42832 42833 Modification times and hashes 42834 42835 Quatrix allows modification times to be set on objects accurate to 1 42836 microsecond. These will be used to detect whether objects need syncing 42837 or not. 42838 42839 Quatrix does not support hashes, so you cannot use the --checksum flag. 42840 42841 Restricted filename characters 42842 42843 File names in Quatrix are case sensitive and have limitations like the 42844 maximum length of a filename is 255, and the minimum length is 1. A file 42845 name cannot be equal to . or .. nor contain / , \ or non-printable 42846 ascii. 42847 42848 Transfers 42849 42850 For files above 50 MiB rclone will use a chunked transfer. Rclone will 42851 upload up to --transfers chunks at the same time (shared among all 42852 multipart uploads). Chunks are buffered in memory, and the minimal chunk 42853 size is 10_000_000 bytes by default, and it can be changed in the 42854 advanced configuration, so increasing --transfers will increase the 42855 memory use. The chunk size has a maximum size limit, which is set to 42856 100_000_000 bytes by default and can be changed in the advanced 42857 configuration. The size of the uploaded chunk will dynamically change 42858 depending on the upload speed. The total memory use equals the number of 42859 transfers multiplied by the minimal chunk size. In case there's free 42860 memory allocated for the upload (which equals the difference of 42861 maximal_summary_chunk_size and minimal_chunk_size * transfers), the 42862 chunk size may increase in case of high upload speed. As well as it can 42863 decrease in case of upload speed problems. If no free memory is 42864 available, all chunks will equal minimal_chunk_size. 42865 42866 Deleting files 42867 42868 Files you delete with rclone will end up in Trash and be stored there 42869 for 30 days. Quatrix also provides an API to permanently delete files 42870 and an API to empty the Trash so that you can remove files permanently 42871 from your account. 42872 42873 Standard options 42874 42875 Here are the Standard options specific to quatrix (Quatrix by Maytech). 42876 42877 --quatrix-api-key 42878 42879 API key for accessing Quatrix account 42880 42881 Properties: 42882 42883 - Config: api_key 42884 - Env Var: RCLONE_QUATRIX_API_KEY 42885 - Type: string 42886 - Required: true 42887 42888 --quatrix-host 42889 42890 Host name of Quatrix account 42891 42892 Properties: 42893 42894 - Config: host 42895 - Env Var: RCLONE_QUATRIX_HOST 42896 - Type: string 42897 - Required: true 42898 42899 Advanced options 42900 42901 Here are the Advanced options specific to quatrix (Quatrix by Maytech). 42902 42903 --quatrix-encoding 42904 42905 The encoding for the backend. 42906 42907 See the encoding section in the overview for more info. 42908 42909 Properties: 42910 42911 - Config: encoding 42912 - Env Var: RCLONE_QUATRIX_ENCODING 42913 - Type: Encoding 42914 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 42915 42916 --quatrix-effective-upload-time 42917 42918 Wanted upload time for one chunk 42919 42920 Properties: 42921 42922 - Config: effective_upload_time 42923 - Env Var: RCLONE_QUATRIX_EFFECTIVE_UPLOAD_TIME 42924 - Type: string 42925 - Default: "4s" 42926 42927 --quatrix-minimal-chunk-size 42928 42929 The minimal size for one chunk 42930 42931 Properties: 42932 42933 - Config: minimal_chunk_size 42934 - Env Var: RCLONE_QUATRIX_MINIMAL_CHUNK_SIZE 42935 - Type: SizeSuffix 42936 - Default: 9.537Mi 42937 42938 --quatrix-maximal-summary-chunk-size 42939 42940 The maximal summary for all chunks. It should not be less than 42941 'transfers'*'minimal_chunk_size' 42942 42943 Properties: 42944 42945 - Config: maximal_summary_chunk_size 42946 - Env Var: RCLONE_QUATRIX_MAXIMAL_SUMMARY_CHUNK_SIZE 42947 - Type: SizeSuffix 42948 - Default: 95.367Mi 42949 42950 --quatrix-hard-delete 42951 42952 Delete files permanently rather than putting them into the trash 42953 42954 Properties: 42955 42956 - Config: hard_delete 42957 - Env Var: RCLONE_QUATRIX_HARD_DELETE 42958 - Type: bool 42959 - Default: false 42960 42961 --quatrix-skip-project-folders 42962 42963 Skip project folders in operations 42964 42965 Properties: 42966 42967 - Config: skip_project_folders 42968 - Env Var: RCLONE_QUATRIX_SKIP_PROJECT_FOLDERS 42969 - Type: bool 42970 - Default: false 42971 42972 --quatrix-description 42973 42974 Description of the remote 42975 42976 Properties: 42977 42978 - Config: description 42979 - Env Var: RCLONE_QUATRIX_DESCRIPTION 42980 - Type: string 42981 - Required: false 42982 42983 Storage usage 42984 42985 The storage usage in Quatrix is restricted to the account during the 42986 purchase. You can restrict any user with a smaller storage limit. The 42987 account limit is applied if the user has no custom storage limit. Once 42988 you've reached the limit, the upload of files will fail. This can be 42989 fixed by freeing up the space or increasing the quota. 42990 42991 Server-side operations 42992 42993 Quatrix supports server-side operations (copy and move). In case of 42994 conflict, files are overwritten during server-side operation. 42995 42996 Sia 42997 42998 Sia (sia.tech) is a decentralized cloud storage platform based on the 42999 blockchain technology. With rclone you can use it like any other remote 43000 filesystem or mount Sia folders locally. The technology behind it 43001 involves a number of new concepts such as Siacoins and Wallet, 43002 Blockchain and Consensus, Renting and Hosting, and so on. If you are new 43003 to it, you'd better first familiarize yourself using their excellent 43004 support documentation. 43005 43006 Introduction 43007 43008 Before you can use rclone with Sia, you will need to have a running copy 43009 of Sia-UI or siad (the Sia daemon) locally on your computer or on local 43010 network (e.g. a NAS). Please follow the Get started guide and install 43011 one. 43012 43013 rclone interacts with Sia network by talking to the Sia daemon via HTTP 43014 API which is usually available on port 9980. By default you will run the 43015 daemon locally on the same computer so it's safe to leave the API 43016 password blank (the API URL will be http://127.0.0.1:9980 making 43017 external access impossible). 43018 43019 However, if you want to access Sia daemon running on another node, for 43020 example due to memory constraints or because you want to share single 43021 daemon between several rclone and Sia-UI instances, you'll need to make 43022 a few more provisions: - Ensure you have Sia daemon installed directly 43023 or in a docker container because Sia-UI does not support this mode 43024 natively. - Run it on externally accessible port, for example provide 43025 --api-addr :9980 and --disable-api-security arguments on the daemon 43026 command line. - Enforce API password for the siad daemon via environment 43027 variable SIA_API_PASSWORD or text file named apipassword in the daemon 43028 directory. - Set rclone backend option api_password taking it from above 43029 locations. 43030 43031 Notes: 1. If your wallet is locked, rclone cannot unlock it 43032 automatically. You should either unlock it in advance by using Sia-UI or 43033 via command line siac wallet unlock. Alternatively you can make siad 43034 unlock your wallet automatically upon startup by running it with 43035 environment variable SIA_WALLET_PASSWORD. 2. If siad cannot find the 43036 SIA_API_PASSWORD variable or the apipassword file in the SIA_DIR 43037 directory, it will generate a random password and store in the text file 43038 named apipassword under YOUR_HOME/.sia/ directory on Unix or 43039 C:\Users\YOUR_HOME\AppData\Local\Sia\apipassword on Windows. Remember 43040 this when you configure password in rclone. 3. The only way to use siad 43041 without API password is to run it on localhost with command line 43042 argument --authorize-api=false, but this is insecure and strongly 43043 discouraged. 43044 43045 Configuration 43046 43047 Here is an example of how to make a sia remote called mySia. First, run: 43048 43049 rclone config 43050 43051 This will guide you through an interactive setup process: 43052 43053 No remotes found, make a new one? 43054 n) New remote 43055 s) Set configuration password 43056 q) Quit config 43057 n/s/q> n 43058 name> mySia 43059 Type of storage to configure. 43060 Enter a string value. Press Enter for the default (""). 43061 Choose a number from below, or type in your own value 43062 ... 43063 29 / Sia Decentralized Cloud 43064 \ "sia" 43065 ... 43066 Storage> sia 43067 Sia daemon API URL, like http://sia.daemon.host:9980. 43068 Note that siad must run with --disable-api-security to open API port for other hosts (not recommended). 43069 Keep default if Sia daemon runs on localhost. 43070 Enter a string value. Press Enter for the default ("http://127.0.0.1:9980"). 43071 api_url> http://127.0.0.1:9980 43072 Sia Daemon API Password. 43073 Can be found in the apipassword file located in HOME/.sia/ or in the daemon directory. 43074 y) Yes type in my own password 43075 g) Generate random password 43076 n) No leave this optional password blank (default) 43077 y/g/n> y 43078 Enter the password: 43079 password: 43080 Confirm the password: 43081 password: 43082 Edit advanced config? 43083 y) Yes 43084 n) No (default) 43085 y/n> n 43086 -------------------- 43087 [mySia] 43088 type = sia 43089 api_url = http://127.0.0.1:9980 43090 api_password = *** ENCRYPTED *** 43091 -------------------- 43092 y) Yes this is OK (default) 43093 e) Edit this remote 43094 d) Delete this remote 43095 y/e/d> y 43096 43097 Once configured, you can then use rclone like this: 43098 43099 - List directories in top level of your Sia storage 43100 43101 rclone lsd mySia: 43102 43103 - List all the files in your Sia storage 43104 43105 rclone ls mySia: 43106 43107 - Upload a local directory to the Sia directory called backup 43108 43109 rclone copy /home/source mySia:backup 43110 43111 Standard options 43112 43113 Here are the Standard options specific to sia (Sia Decentralized Cloud). 43114 43115 --sia-api-url 43116 43117 Sia daemon API URL, like http://sia.daemon.host:9980. 43118 43119 Note that siad must run with --disable-api-security to open API port for 43120 other hosts (not recommended). Keep default if Sia daemon runs on 43121 localhost. 43122 43123 Properties: 43124 43125 - Config: api_url 43126 - Env Var: RCLONE_SIA_API_URL 43127 - Type: string 43128 - Default: "http://127.0.0.1:9980" 43129 43130 --sia-api-password 43131 43132 Sia Daemon API Password. 43133 43134 Can be found in the apipassword file located in HOME/.sia/ or in the 43135 daemon directory. 43136 43137 NB Input to this must be obscured - see rclone obscure. 43138 43139 Properties: 43140 43141 - Config: api_password 43142 - Env Var: RCLONE_SIA_API_PASSWORD 43143 - Type: string 43144 - Required: false 43145 43146 Advanced options 43147 43148 Here are the Advanced options specific to sia (Sia Decentralized Cloud). 43149 43150 --sia-user-agent 43151 43152 Siad User Agent 43153 43154 Sia daemon requires the 'Sia-Agent' user agent by default for security 43155 43156 Properties: 43157 43158 - Config: user_agent 43159 - Env Var: RCLONE_SIA_USER_AGENT 43160 - Type: string 43161 - Default: "Sia-Agent" 43162 43163 --sia-encoding 43164 43165 The encoding for the backend. 43166 43167 See the encoding section in the overview for more info. 43168 43169 Properties: 43170 43171 - Config: encoding 43172 - Env Var: RCLONE_SIA_ENCODING 43173 - Type: Encoding 43174 - Default: Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot 43175 43176 --sia-description 43177 43178 Description of the remote 43179 43180 Properties: 43181 43182 - Config: description 43183 - Env Var: RCLONE_SIA_DESCRIPTION 43184 - Type: string 43185 - Required: false 43186 43187 Limitations 43188 43189 - Modification times not supported 43190 - Checksums not supported 43191 - rclone about not supported 43192 - rclone can work only with Siad or Sia-UI at the moment, the SkyNet 43193 daemon is not supported yet. 43194 - Sia does not allow control characters or symbols like question and 43195 pound signs in file names. rclone will transparently encode them for 43196 you, but you'd better be aware 43197 43198 Swift 43199 43200 Swift refers to OpenStack Object Storage. Commercial implementations of 43201 that being: 43202 43203 - Rackspace Cloud Files 43204 - Memset Memstore 43205 - OVH Object Storage 43206 - Oracle Cloud Storage 43207 - Blomp Cloud Storage 43208 - IBM Bluemix Cloud ObjectStorage Swift 43209 43210 Paths are specified as remote:container (or remote: for the lsd 43211 command.) You may put subdirectories in too, e.g. 43212 remote:container/path/to/dir. 43213 43214 Configuration 43215 43216 Here is an example of making a swift configuration. First run 43217 43218 rclone config 43219 43220 This will guide you through an interactive setup process. 43221 43222 No remotes found, make a new one? 43223 n) New remote 43224 s) Set configuration password 43225 q) Quit config 43226 n/s/q> n 43227 name> remote 43228 Type of storage to configure. 43229 Choose a number from below, or type in your own value 43230 [snip] 43231 XX / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH) 43232 \ "swift" 43233 [snip] 43234 Storage> swift 43235 Get swift credentials from environment variables in standard OpenStack form. 43236 Choose a number from below, or type in your own value 43237 1 / Enter swift credentials in the next step 43238 \ "false" 43239 2 / Get swift credentials from environment vars. Leave other fields blank if using this. 43240 \ "true" 43241 env_auth> true 43242 User name to log in (OS_USERNAME). 43243 user> 43244 API key or password (OS_PASSWORD). 43245 key> 43246 Authentication URL for server (OS_AUTH_URL). 43247 Choose a number from below, or type in your own value 43248 1 / Rackspace US 43249 \ "https://auth.api.rackspacecloud.com/v1.0" 43250 2 / Rackspace UK 43251 \ "https://lon.auth.api.rackspacecloud.com/v1.0" 43252 3 / Rackspace v2 43253 \ "https://identity.api.rackspacecloud.com/v2.0" 43254 4 / Memset Memstore UK 43255 \ "https://auth.storage.memset.com/v1.0" 43256 5 / Memset Memstore UK v2 43257 \ "https://auth.storage.memset.com/v2.0" 43258 6 / OVH 43259 \ "https://auth.cloud.ovh.net/v3" 43260 7 / Blomp Cloud Storage 43261 \ "https://authenticate.ain.net" 43262 auth> 43263 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 43264 user_id> 43265 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 43266 domain> 43267 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 43268 tenant> 43269 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 43270 tenant_id> 43271 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 43272 tenant_domain> 43273 Region name - optional (OS_REGION_NAME) 43274 region> 43275 Storage URL - optional (OS_STORAGE_URL) 43276 storage_url> 43277 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 43278 auth_token> 43279 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 43280 auth_version> 43281 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 43282 Choose a number from below, or type in your own value 43283 1 / Public (default, choose this if not sure) 43284 \ "public" 43285 2 / Internal (use internal service net) 43286 \ "internal" 43287 3 / Admin 43288 \ "admin" 43289 endpoint_type> 43290 Remote config 43291 -------------------- 43292 [test] 43293 env_auth = true 43294 user = 43295 key = 43296 auth = 43297 user_id = 43298 domain = 43299 tenant = 43300 tenant_id = 43301 tenant_domain = 43302 region = 43303 storage_url = 43304 auth_token = 43305 auth_version = 43306 endpoint_type = 43307 -------------------- 43308 y) Yes this is OK 43309 e) Edit this remote 43310 d) Delete this remote 43311 y/e/d> y 43312 43313 This remote is called remote and can now be used like this 43314 43315 See all containers 43316 43317 rclone lsd remote: 43318 43319 Make a new container 43320 43321 rclone mkdir remote:container 43322 43323 List the contents of a container 43324 43325 rclone ls remote:container 43326 43327 Sync /home/local/directory to the remote container, deleting any excess 43328 files in the container. 43329 43330 rclone sync --interactive /home/local/directory remote:container 43331 43332 Configuration from an OpenStack credentials file 43333 43334 An OpenStack credentials file typically looks something something like 43335 this (without the comments) 43336 43337 export OS_AUTH_URL=https://a.provider.net/v2.0 43338 export OS_TENANT_ID=ffffffffffffffffffffffffffffffff 43339 export OS_TENANT_NAME="1234567890123456" 43340 export OS_USERNAME="123abc567xy" 43341 echo "Please enter your OpenStack Password: " 43342 read -sr OS_PASSWORD_INPUT 43343 export OS_PASSWORD=$OS_PASSWORD_INPUT 43344 export OS_REGION_NAME="SBG1" 43345 if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi 43346 43347 The config file needs to look something like this where $OS_USERNAME 43348 represents the value of the OS_USERNAME variable - 123abc567xy in the 43349 example above. 43350 43351 [remote] 43352 type = swift 43353 user = $OS_USERNAME 43354 key = $OS_PASSWORD 43355 auth = $OS_AUTH_URL 43356 tenant = $OS_TENANT_NAME 43357 43358 Note that you may (or may not) need to set region too - try without 43359 first. 43360 43361 Configuration from the environment 43362 43363 If you prefer you can configure rclone to use swift using a standard set 43364 of OpenStack environment variables. 43365 43366 When you run through the config, make sure you choose true for env_auth 43367 and leave everything else blank. 43368 43369 rclone will then set any empty config parameters from the environment 43370 using standard OpenStack environment variables. There is a list of the 43371 variables in the docs for the swift library. 43372 43373 Using an alternate authentication method 43374 43375 If your OpenStack installation uses a non-standard authentication method 43376 that might not be yet supported by rclone or the underlying swift 43377 library, you can authenticate externally (e.g. calling manually the 43378 openstack commands to get a token). Then, you just need to pass the two 43379 configuration variables auth_token and storage_url. If they are both 43380 provided, the other variables are ignored. rclone will not try to 43381 authenticate but instead assume it is already authenticated and use 43382 these two variables to access the OpenStack installation. 43383 43384 Using rclone without a config file 43385 43386 You can use rclone with swift without a config file, if desired, like 43387 this: 43388 43389 source openstack-credentials-file 43390 export RCLONE_CONFIG_MYREMOTE_TYPE=swift 43391 export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true 43392 rclone lsd myremote: 43393 43394 --fast-list 43395 43396 This remote supports --fast-list which allows you to use fewer 43397 transactions in exchange for more memory. See the rclone docs for more 43398 details. 43399 43400 --update and --use-server-modtime 43401 43402 As noted below, the modified time is stored on metadata on the object. 43403 It is used by default for all operations that require checking the time 43404 a file was last updated. It allows rclone to treat the remote more like 43405 a true filesystem, but it is inefficient because it requires an extra 43406 API call to retrieve the metadata. 43407 43408 For many operations, the time the object was last uploaded to the remote 43409 is sufficient to determine if it is "dirty". By using --update along 43410 with --use-server-modtime, you can avoid the extra API call and simply 43411 upload files whose local modtime is newer than the time it was last 43412 uploaded. 43413 43414 Modification times and hashes 43415 43416 The modified time is stored as metadata on the object as 43417 X-Object-Meta-Mtime as floating point since the epoch accurate to 1 ns. 43418 43419 This is a de facto standard (used in the official python-swiftclient 43420 amongst others) for storing the modification time for an object. 43421 43422 The MD5 hash algorithm is supported. 43423 43424 Restricted filename characters 43425 43426 Character Value Replacement 43427 ----------- ------- ------------- 43428 NUL 0x00 ␀ 43429 / 0x2F / 43430 43431 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 43432 strings. 43433 43434 Standard options 43435 43436 Here are the Standard options specific to swift (OpenStack Swift 43437 (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)). 43438 43439 --swift-env-auth 43440 43441 Get swift credentials from environment variables in standard OpenStack 43442 form. 43443 43444 Properties: 43445 43446 - Config: env_auth 43447 - Env Var: RCLONE_SWIFT_ENV_AUTH 43448 - Type: bool 43449 - Default: false 43450 - Examples: 43451 - "false" 43452 - Enter swift credentials in the next step. 43453 - "true" 43454 - Get swift credentials from environment vars. 43455 - Leave other fields blank if using this. 43456 43457 --swift-user 43458 43459 User name to log in (OS_USERNAME). 43460 43461 Properties: 43462 43463 - Config: user 43464 - Env Var: RCLONE_SWIFT_USER 43465 - Type: string 43466 - Required: false 43467 43468 --swift-key 43469 43470 API key or password (OS_PASSWORD). 43471 43472 Properties: 43473 43474 - Config: key 43475 - Env Var: RCLONE_SWIFT_KEY 43476 - Type: string 43477 - Required: false 43478 43479 --swift-auth 43480 43481 Authentication URL for server (OS_AUTH_URL). 43482 43483 Properties: 43484 43485 - Config: auth 43486 - Env Var: RCLONE_SWIFT_AUTH 43487 - Type: string 43488 - Required: false 43489 - Examples: 43490 - "https://auth.api.rackspacecloud.com/v1.0" 43491 - Rackspace US 43492 - "https://lon.auth.api.rackspacecloud.com/v1.0" 43493 - Rackspace UK 43494 - "https://identity.api.rackspacecloud.com/v2.0" 43495 - Rackspace v2 43496 - "https://auth.storage.memset.com/v1.0" 43497 - Memset Memstore UK 43498 - "https://auth.storage.memset.com/v2.0" 43499 - Memset Memstore UK v2 43500 - "https://auth.cloud.ovh.net/v3" 43501 - OVH 43502 - "https://authenticate.ain.net" 43503 - Blomp Cloud Storage 43504 43505 --swift-user-id 43506 43507 User ID to log in - optional - most swift systems use user and leave 43508 this blank (v3 auth) (OS_USER_ID). 43509 43510 Properties: 43511 43512 - Config: user_id 43513 - Env Var: RCLONE_SWIFT_USER_ID 43514 - Type: string 43515 - Required: false 43516 43517 --swift-domain 43518 43519 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 43520 43521 Properties: 43522 43523 - Config: domain 43524 - Env Var: RCLONE_SWIFT_DOMAIN 43525 - Type: string 43526 - Required: false 43527 43528 --swift-tenant 43529 43530 Tenant name - optional for v1 auth, this or tenant_id required otherwise 43531 (OS_TENANT_NAME or OS_PROJECT_NAME). 43532 43533 Properties: 43534 43535 - Config: tenant 43536 - Env Var: RCLONE_SWIFT_TENANT 43537 - Type: string 43538 - Required: false 43539 43540 --swift-tenant-id 43541 43542 Tenant ID - optional for v1 auth, this or tenant required otherwise 43543 (OS_TENANT_ID). 43544 43545 Properties: 43546 43547 - Config: tenant_id 43548 - Env Var: RCLONE_SWIFT_TENANT_ID 43549 - Type: string 43550 - Required: false 43551 43552 --swift-tenant-domain 43553 43554 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME). 43555 43556 Properties: 43557 43558 - Config: tenant_domain 43559 - Env Var: RCLONE_SWIFT_TENANT_DOMAIN 43560 - Type: string 43561 - Required: false 43562 43563 --swift-region 43564 43565 Region name - optional (OS_REGION_NAME). 43566 43567 Properties: 43568 43569 - Config: region 43570 - Env Var: RCLONE_SWIFT_REGION 43571 - Type: string 43572 - Required: false 43573 43574 --swift-storage-url 43575 43576 Storage URL - optional (OS_STORAGE_URL). 43577 43578 Properties: 43579 43580 - Config: storage_url 43581 - Env Var: RCLONE_SWIFT_STORAGE_URL 43582 - Type: string 43583 - Required: false 43584 43585 --swift-auth-token 43586 43587 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN). 43588 43589 Properties: 43590 43591 - Config: auth_token 43592 - Env Var: RCLONE_SWIFT_AUTH_TOKEN 43593 - Type: string 43594 - Required: false 43595 43596 --swift-application-credential-id 43597 43598 Application Credential ID (OS_APPLICATION_CREDENTIAL_ID). 43599 43600 Properties: 43601 43602 - Config: application_credential_id 43603 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID 43604 - Type: string 43605 - Required: false 43606 43607 --swift-application-credential-name 43608 43609 Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME). 43610 43611 Properties: 43612 43613 - Config: application_credential_name 43614 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME 43615 - Type: string 43616 - Required: false 43617 43618 --swift-application-credential-secret 43619 43620 Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET). 43621 43622 Properties: 43623 43624 - Config: application_credential_secret 43625 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET 43626 - Type: string 43627 - Required: false 43628 43629 --swift-auth-version 43630 43631 AuthVersion - optional - set to (1,2,3) if your auth URL has no version 43632 (ST_AUTH_VERSION). 43633 43634 Properties: 43635 43636 - Config: auth_version 43637 - Env Var: RCLONE_SWIFT_AUTH_VERSION 43638 - Type: int 43639 - Default: 0 43640 43641 --swift-endpoint-type 43642 43643 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE). 43644 43645 Properties: 43646 43647 - Config: endpoint_type 43648 - Env Var: RCLONE_SWIFT_ENDPOINT_TYPE 43649 - Type: string 43650 - Default: "public" 43651 - Examples: 43652 - "public" 43653 - Public (default, choose this if not sure) 43654 - "internal" 43655 - Internal (use internal service net) 43656 - "admin" 43657 - Admin 43658 43659 --swift-storage-policy 43660 43661 The storage policy to use when creating a new container. 43662 43663 This applies the specified storage policy when creating a new container. 43664 The policy cannot be changed afterwards. The allowed configuration 43665 values and their meaning depend on your Swift storage provider. 43666 43667 Properties: 43668 43669 - Config: storage_policy 43670 - Env Var: RCLONE_SWIFT_STORAGE_POLICY 43671 - Type: string 43672 - Required: false 43673 - Examples: 43674 - "" 43675 - Default 43676 - "pcs" 43677 - OVH Public Cloud Storage 43678 - "pca" 43679 - OVH Public Cloud Archive 43680 43681 Advanced options 43682 43683 Here are the Advanced options specific to swift (OpenStack Swift 43684 (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)). 43685 43686 --swift-leave-parts-on-error 43687 43688 If true avoid calling abort upload on a failure. 43689 43690 It should be set to true for resuming uploads across different sessions. 43691 43692 Properties: 43693 43694 - Config: leave_parts_on_error 43695 - Env Var: RCLONE_SWIFT_LEAVE_PARTS_ON_ERROR 43696 - Type: bool 43697 - Default: false 43698 43699 --swift-chunk-size 43700 43701 Above this size files will be chunked into a _segments container. 43702 43703 Above this size files will be chunked into a _segments container. The 43704 default for this is 5 GiB which is its maximum value. 43705 43706 Properties: 43707 43708 - Config: chunk_size 43709 - Env Var: RCLONE_SWIFT_CHUNK_SIZE 43710 - Type: SizeSuffix 43711 - Default: 5Gi 43712 43713 --swift-no-chunk 43714 43715 Don't chunk files during streaming upload. 43716 43717 When doing streaming uploads (e.g. using rcat or mount) setting this 43718 flag will cause the swift backend to not upload chunked files. 43719 43720 This will limit the maximum upload size to 5 GiB. However non chunked 43721 files are easier to deal with and have an MD5SUM. 43722 43723 Rclone will still chunk files bigger than chunk_size when doing normal 43724 copy operations. 43725 43726 Properties: 43727 43728 - Config: no_chunk 43729 - Env Var: RCLONE_SWIFT_NO_CHUNK 43730 - Type: bool 43731 - Default: false 43732 43733 --swift-no-large-objects 43734 43735 Disable support for static and dynamic large objects 43736 43737 Swift cannot transparently store files bigger than 5 GiB. There are two 43738 schemes for doing that, static or dynamic large objects, and the API 43739 does not allow rclone to determine whether a file is a static or dynamic 43740 large object without doing a HEAD on the object. Since these need to be 43741 treated differently, this means rclone has to issue HEAD requests for 43742 objects for example when reading checksums. 43743 43744 When no_large_objects is set, rclone will assume that there are no 43745 static or dynamic large objects stored. This means it can stop doing the 43746 extra HEAD calls which in turn increases performance greatly especially 43747 when doing a swift to swift transfer with --checksum set. 43748 43749 Setting this option implies no_chunk and also that no files will be 43750 uploaded in chunks, so files bigger than 5 GiB will just fail on upload. 43751 43752 If you set this option and there are static or dynamic large objects, 43753 then this will give incorrect hashes for them. Downloads will succeed, 43754 but other operations such as Remove and Copy will fail. 43755 43756 Properties: 43757 43758 - Config: no_large_objects 43759 - Env Var: RCLONE_SWIFT_NO_LARGE_OBJECTS 43760 - Type: bool 43761 - Default: false 43762 43763 --swift-encoding 43764 43765 The encoding for the backend. 43766 43767 See the encoding section in the overview for more info. 43768 43769 Properties: 43770 43771 - Config: encoding 43772 - Env Var: RCLONE_SWIFT_ENCODING 43773 - Type: Encoding 43774 - Default: Slash,InvalidUtf8 43775 43776 --swift-description 43777 43778 Description of the remote 43779 43780 Properties: 43781 43782 - Config: description 43783 - Env Var: RCLONE_SWIFT_DESCRIPTION 43784 - Type: string 43785 - Required: false 43786 43787 Limitations 43788 43789 The Swift API doesn't return a correct MD5SUM for segmented files 43790 (Dynamic or Static Large Objects) so rclone won't check or use the 43791 MD5SUM for these. 43792 43793 Troubleshooting 43794 43795 Rclone gives Failed to create file system for "remote:": Bad Request 43796 43797 Due to an oddity of the underlying swift library, it gives a "Bad 43798 Request" error rather than a more sensible error when the authentication 43799 fails for Swift. 43800 43801 So this most likely means your username / password is wrong. You can 43802 investigate further with the --dump-bodies flag. 43803 43804 This may also be caused by specifying the region when you shouldn't have 43805 (e.g. OVH). 43806 43807 Rclone gives Failed to create file system: Response didn't have storage url and auth token 43808 43809 This is most likely caused by forgetting to specify your tenant when 43810 setting up a swift remote. 43811 43812 OVH Cloud Archive 43813 43814 To use rclone with OVH cloud archive, first use rclone config to set up 43815 a swift backend with OVH, choosing pca as the storage_policy. 43816 43817 Uploading Objects 43818 43819 Uploading objects to OVH cloud archive is no different to object 43820 storage, you just simply run the command you like (move, copy or sync) 43821 to upload the objects. Once uploaded the objects will show in a "Frozen" 43822 state within the OVH control panel. 43823 43824 Retrieving Objects 43825 43826 To retrieve objects use rclone copy as normal. If the objects are in a 43827 frozen state then rclone will ask for them all to be unfrozen and it 43828 will wait at the end of the output with a message like the following: 43829 43830 2019/03/23 13:06:33 NOTICE: Received retry after error - sleeping until 2019-03-23T13:16:33.481657164+01:00 (9m59.99985121s) 43831 43832 Rclone will wait for the time specified then retry the copy. 43833 43834 pCloud 43835 43836 Paths are specified as remote:path 43837 43838 Paths may be as deep as required, e.g. remote:directory/subdirectory. 43839 43840 Configuration 43841 43842 The initial setup for pCloud involves getting a token from pCloud which 43843 you need to do in your browser. rclone config walks you through it. 43844 43845 Here is an example of how to make a remote called remote. First run: 43846 43847 rclone config 43848 43849 This will guide you through an interactive setup process: 43850 43851 No remotes found, make a new one? 43852 n) New remote 43853 s) Set configuration password 43854 q) Quit config 43855 n/s/q> n 43856 name> remote 43857 Type of storage to configure. 43858 Choose a number from below, or type in your own value 43859 [snip] 43860 XX / Pcloud 43861 \ "pcloud" 43862 [snip] 43863 Storage> pcloud 43864 Pcloud App Client Id - leave blank normally. 43865 client_id> 43866 Pcloud App Client Secret - leave blank normally. 43867 client_secret> 43868 Remote config 43869 Use web browser to automatically authenticate rclone with remote? 43870 * Say Y if the machine running rclone has a web browser you can use 43871 * Say N if running rclone on a (remote) machine without web browser access 43872 If not sure try Y. If Y failed, try N. 43873 y) Yes 43874 n) No 43875 y/n> y 43876 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 43877 Log in and authorize rclone for access 43878 Waiting for code... 43879 Got code 43880 -------------------- 43881 [remote] 43882 client_id = 43883 client_secret = 43884 token = {"access_token":"XXX","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"} 43885 -------------------- 43886 y) Yes this is OK 43887 e) Edit this remote 43888 d) Delete this remote 43889 y/e/d> y 43890 43891 See the remote setup docs for how to set it up on a machine with no 43892 Internet browser available. 43893 43894 Note that rclone runs a webserver on your local machine to collect the 43895 token as returned from pCloud. This only runs from the moment it opens 43896 your browser to the moment you get back the verification code. This is 43897 on http://127.0.0.1:53682/ and this it may require you to unblock it 43898 temporarily if you are running a host firewall. 43899 43900 Once configured you can then use rclone like this, 43901 43902 List directories in top level of your pCloud 43903 43904 rclone lsd remote: 43905 43906 List all the files in your pCloud 43907 43908 rclone ls remote: 43909 43910 To copy a local directory to a pCloud directory called backup 43911 43912 rclone copy /home/source remote:backup 43913 43914 Modification times and hashes 43915 43916 pCloud allows modification times to be set on objects accurate to 1 43917 second. These will be used to detect whether objects need syncing or 43918 not. In order to set a Modification time pCloud requires the object be 43919 re-uploaded. 43920 43921 pCloud supports MD5 and SHA1 hashes in the US region, and SHA1 and 43922 SHA256 hashes in the EU region, so you can use the --checksum flag. 43923 43924 Restricted filename characters 43925 43926 In addition to the default restricted characters set the following 43927 characters are also replaced: 43928 43929 Character Value Replacement 43930 ----------- ------- ------------- 43931 \ 0x5C \ 43932 43933 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 43934 strings. 43935 43936 Deleting files 43937 43938 Deleted files will be moved to the trash. Your subscription level will 43939 determine how long items stay in the trash. rclone cleanup can be used 43940 to empty the trash. 43941 43942 Emptying the trash 43943 43944 Due to an API limitation, the rclone cleanup command will only work if 43945 you set your username and password in the advanced options for this 43946 backend. Since we generally want to avoid storing user passwords in the 43947 rclone config file, we advise you to only set this up if you need the 43948 rclone cleanup command to work. 43949 43950 Root folder ID 43951 43952 You can set the root_folder_id for rclone. This is the directory 43953 (identified by its Folder ID) that rclone considers to be the root of 43954 your pCloud drive. 43955 43956 Normally you will leave this blank and rclone will determine the correct 43957 root to use itself. 43958 43959 However you can set this to restrict rclone to a specific folder 43960 hierarchy. 43961 43962 In order to do this you will have to find the Folder ID of the directory 43963 you wish rclone to display. This will be the folder field of the URL 43964 when you open the relevant folder in the pCloud web interface. 43965 43966 So if the folder you want rclone to use has a URL which looks like 43967 https://my.pcloud.com/#page=filemanager&folder=5xxxxxxxx8&tpl=foldergrid 43968 in the browser, then you use 5xxxxxxxx8 as the root_folder_id in the 43969 config. 43970 43971 Standard options 43972 43973 Here are the Standard options specific to pcloud (Pcloud). 43974 43975 --pcloud-client-id 43976 43977 OAuth Client Id. 43978 43979 Leave blank normally. 43980 43981 Properties: 43982 43983 - Config: client_id 43984 - Env Var: RCLONE_PCLOUD_CLIENT_ID 43985 - Type: string 43986 - Required: false 43987 43988 --pcloud-client-secret 43989 43990 OAuth Client Secret. 43991 43992 Leave blank normally. 43993 43994 Properties: 43995 43996 - Config: client_secret 43997 - Env Var: RCLONE_PCLOUD_CLIENT_SECRET 43998 - Type: string 43999 - Required: false 44000 44001 Advanced options 44002 44003 Here are the Advanced options specific to pcloud (Pcloud). 44004 44005 --pcloud-token 44006 44007 OAuth Access Token as a JSON blob. 44008 44009 Properties: 44010 44011 - Config: token 44012 - Env Var: RCLONE_PCLOUD_TOKEN 44013 - Type: string 44014 - Required: false 44015 44016 --pcloud-auth-url 44017 44018 Auth server URL. 44019 44020 Leave blank to use the provider defaults. 44021 44022 Properties: 44023 44024 - Config: auth_url 44025 - Env Var: RCLONE_PCLOUD_AUTH_URL 44026 - Type: string 44027 - Required: false 44028 44029 --pcloud-token-url 44030 44031 Token server url. 44032 44033 Leave blank to use the provider defaults. 44034 44035 Properties: 44036 44037 - Config: token_url 44038 - Env Var: RCLONE_PCLOUD_TOKEN_URL 44039 - Type: string 44040 - Required: false 44041 44042 --pcloud-encoding 44043 44044 The encoding for the backend. 44045 44046 See the encoding section in the overview for more info. 44047 44048 Properties: 44049 44050 - Config: encoding 44051 - Env Var: RCLONE_PCLOUD_ENCODING 44052 - Type: Encoding 44053 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 44054 44055 --pcloud-root-folder-id 44056 44057 Fill in for rclone to use a non root folder as its starting point. 44058 44059 Properties: 44060 44061 - Config: root_folder_id 44062 - Env Var: RCLONE_PCLOUD_ROOT_FOLDER_ID 44063 - Type: string 44064 - Default: "d0" 44065 44066 --pcloud-hostname 44067 44068 Hostname to connect to. 44069 44070 This is normally set when rclone initially does the oauth connection, 44071 however you will need to set it by hand if you are using remote config 44072 with rclone authorize. 44073 44074 Properties: 44075 44076 - Config: hostname 44077 - Env Var: RCLONE_PCLOUD_HOSTNAME 44078 - Type: string 44079 - Default: "api.pcloud.com" 44080 - Examples: 44081 - "api.pcloud.com" 44082 - Original/US region 44083 - "eapi.pcloud.com" 44084 - EU region 44085 44086 --pcloud-username 44087 44088 Your pcloud username. 44089 44090 This is only required when you want to use the cleanup command. Due to a 44091 bug in the pcloud API the required API does not support OAuth 44092 authentication so we have to rely on user password authentication for 44093 it. 44094 44095 Properties: 44096 44097 - Config: username 44098 - Env Var: RCLONE_PCLOUD_USERNAME 44099 - Type: string 44100 - Required: false 44101 44102 --pcloud-password 44103 44104 Your pcloud password. 44105 44106 NB Input to this must be obscured - see rclone obscure. 44107 44108 Properties: 44109 44110 - Config: password 44111 - Env Var: RCLONE_PCLOUD_PASSWORD 44112 - Type: string 44113 - Required: false 44114 44115 --pcloud-description 44116 44117 Description of the remote 44118 44119 Properties: 44120 44121 - Config: description 44122 - Env Var: RCLONE_PCLOUD_DESCRIPTION 44123 - Type: string 44124 - Required: false 44125 44126 PikPak 44127 44128 PikPak is a private cloud drive. 44129 44130 Paths are specified as remote:path, and may be as deep as required, e.g. 44131 remote:directory/subdirectory. 44132 44133 Configuration 44134 44135 Here is an example of making a remote for PikPak. 44136 44137 First run: 44138 44139 rclone config 44140 44141 This will guide you through an interactive setup process: 44142 44143 No remotes found, make a new one? 44144 n) New remote 44145 s) Set configuration password 44146 q) Quit config 44147 n/s/q> n 44148 44149 Enter name for new remote. 44150 name> remote 44151 44152 Option Storage. 44153 Type of storage to configure. 44154 Choose a number from below, or type in your own value. 44155 XX / PikPak 44156 \ (pikpak) 44157 Storage> XX 44158 44159 Option user. 44160 Pikpak username. 44161 Enter a value. 44162 user> USERNAME 44163 44164 Option pass. 44165 Pikpak password. 44166 Choose an alternative below. 44167 y) Yes, type in my own password 44168 g) Generate random password 44169 y/g> y 44170 Enter the password: 44171 password: 44172 Confirm the password: 44173 password: 44174 44175 Edit advanced config? 44176 y) Yes 44177 n) No (default) 44178 y/n> 44179 44180 Configuration complete. 44181 Options: 44182 - type: pikpak 44183 - user: USERNAME 44184 - pass: *** ENCRYPTED *** 44185 - token: {"access_token":"eyJ...","token_type":"Bearer","refresh_token":"os...","expiry":"2023-01-26T18:54:32.170582647+09:00"} 44186 Keep this "remote" remote? 44187 y) Yes this is OK (default) 44188 e) Edit this remote 44189 d) Delete this remote 44190 y/e/d> y 44191 44192 Modification times and hashes 44193 44194 PikPak keeps modification times on objects, and updates them when 44195 uploading objects, but it does not support changing only the 44196 modification time 44197 44198 The MD5 hash algorithm is supported. 44199 44200 Standard options 44201 44202 Here are the Standard options specific to pikpak (PikPak). 44203 44204 --pikpak-user 44205 44206 Pikpak username. 44207 44208 Properties: 44209 44210 - Config: user 44211 - Env Var: RCLONE_PIKPAK_USER 44212 - Type: string 44213 - Required: true 44214 44215 --pikpak-pass 44216 44217 Pikpak password. 44218 44219 NB Input to this must be obscured - see rclone obscure. 44220 44221 Properties: 44222 44223 - Config: pass 44224 - Env Var: RCLONE_PIKPAK_PASS 44225 - Type: string 44226 - Required: true 44227 44228 Advanced options 44229 44230 Here are the Advanced options specific to pikpak (PikPak). 44231 44232 --pikpak-client-id 44233 44234 OAuth Client Id. 44235 44236 Leave blank normally. 44237 44238 Properties: 44239 44240 - Config: client_id 44241 - Env Var: RCLONE_PIKPAK_CLIENT_ID 44242 - Type: string 44243 - Required: false 44244 44245 --pikpak-client-secret 44246 44247 OAuth Client Secret. 44248 44249 Leave blank normally. 44250 44251 Properties: 44252 44253 - Config: client_secret 44254 - Env Var: RCLONE_PIKPAK_CLIENT_SECRET 44255 - Type: string 44256 - Required: false 44257 44258 --pikpak-token 44259 44260 OAuth Access Token as a JSON blob. 44261 44262 Properties: 44263 44264 - Config: token 44265 - Env Var: RCLONE_PIKPAK_TOKEN 44266 - Type: string 44267 - Required: false 44268 44269 --pikpak-auth-url 44270 44271 Auth server URL. 44272 44273 Leave blank to use the provider defaults. 44274 44275 Properties: 44276 44277 - Config: auth_url 44278 - Env Var: RCLONE_PIKPAK_AUTH_URL 44279 - Type: string 44280 - Required: false 44281 44282 --pikpak-token-url 44283 44284 Token server url. 44285 44286 Leave blank to use the provider defaults. 44287 44288 Properties: 44289 44290 - Config: token_url 44291 - Env Var: RCLONE_PIKPAK_TOKEN_URL 44292 - Type: string 44293 - Required: false 44294 44295 --pikpak-root-folder-id 44296 44297 ID of the root folder. Leave blank normally. 44298 44299 Fill in for rclone to use a non root folder as its starting point. 44300 44301 Properties: 44302 44303 - Config: root_folder_id 44304 - Env Var: RCLONE_PIKPAK_ROOT_FOLDER_ID 44305 - Type: string 44306 - Required: false 44307 44308 --pikpak-use-trash 44309 44310 Send files to the trash instead of deleting permanently. 44311 44312 Defaults to true, namely sending files to the trash. Use 44313 --pikpak-use-trash=false to delete files permanently instead. 44314 44315 Properties: 44316 44317 - Config: use_trash 44318 - Env Var: RCLONE_PIKPAK_USE_TRASH 44319 - Type: bool 44320 - Default: true 44321 44322 --pikpak-trashed-only 44323 44324 Only show files that are in the trash. 44325 44326 This will show trashed files in their original directory structure. 44327 44328 Properties: 44329 44330 - Config: trashed_only 44331 - Env Var: RCLONE_PIKPAK_TRASHED_ONLY 44332 - Type: bool 44333 - Default: false 44334 44335 --pikpak-hash-memory-limit 44336 44337 Files bigger than this will be cached on disk to calculate hash if 44338 required. 44339 44340 Properties: 44341 44342 - Config: hash_memory_limit 44343 - Env Var: RCLONE_PIKPAK_HASH_MEMORY_LIMIT 44344 - Type: SizeSuffix 44345 - Default: 10Mi 44346 44347 --pikpak-encoding 44348 44349 The encoding for the backend. 44350 44351 See the encoding section in the overview for more info. 44352 44353 Properties: 44354 44355 - Config: encoding 44356 - Env Var: RCLONE_PIKPAK_ENCODING 44357 - Type: Encoding 44358 - Default: 44359 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot 44360 44361 --pikpak-description 44362 44363 Description of the remote 44364 44365 Properties: 44366 44367 - Config: description 44368 - Env Var: RCLONE_PIKPAK_DESCRIPTION 44369 - Type: string 44370 - Required: false 44371 44372 Backend commands 44373 44374 Here are the commands specific to the pikpak backend. 44375 44376 Run them with 44377 44378 rclone backend COMMAND remote: 44379 44380 The help below will explain what arguments each command takes. 44381 44382 See the backend command for more info on how to pass options and 44383 arguments. 44384 44385 These can be run on a running backend using the rc command 44386 backend/command. 44387 44388 addurl 44389 44390 Add offline download task for url 44391 44392 rclone backend addurl remote: [options] [<arguments>+] 44393 44394 This command adds offline download task for url. 44395 44396 Usage: 44397 44398 rclone backend addurl pikpak:dirpath url 44399 44400 Downloads will be stored in 'dirpath'. If 'dirpath' is invalid, download 44401 will fallback to default 'My Pack' folder. 44402 44403 decompress 44404 44405 Request decompress of a file/files in a folder 44406 44407 rclone backend decompress remote: [options] [<arguments>+] 44408 44409 This command requests decompress of file/files in a folder. 44410 44411 Usage: 44412 44413 rclone backend decompress pikpak:dirpath {filename} -o password=password 44414 rclone backend decompress pikpak:dirpath {filename} -o delete-src-file 44415 44416 An optional argument 'filename' can be specified for a file located in 44417 'pikpak:dirpath'. You may want to pass '-o password=password' for a 44418 password-protected files. Also, pass '-o delete-src-file' to delete 44419 source files after decompression finished. 44420 44421 Result: 44422 44423 { 44424 "Decompressed": 17, 44425 "SourceDeleted": 0, 44426 "Errors": 0 44427 } 44428 44429 Limitations 44430 44431 Hashes may be empty 44432 44433 PikPak supports MD5 hash, but sometimes given empty especially for 44434 user-uploaded files. 44435 44436 Deleted files still visible with trashed-only 44437 44438 Deleted files will still be visible with --pikpak-trashed-only even 44439 after the trash emptied. This goes away after few days. 44440 44441 premiumize.me 44442 44443 Paths are specified as remote:path 44444 44445 Paths may be as deep as required, e.g. remote:directory/subdirectory. 44446 44447 Configuration 44448 44449 The initial setup for premiumize.me involves getting a token from 44450 premiumize.me which you need to do in your browser. rclone config walks 44451 you through it. 44452 44453 Here is an example of how to make a remote called remote. First run: 44454 44455 rclone config 44456 44457 This will guide you through an interactive setup process: 44458 44459 No remotes found, make a new one? 44460 n) New remote 44461 s) Set configuration password 44462 q) Quit config 44463 n/s/q> n 44464 name> remote 44465 Type of storage to configure. 44466 Enter a string value. Press Enter for the default (""). 44467 Choose a number from below, or type in your own value 44468 [snip] 44469 XX / premiumize.me 44470 \ "premiumizeme" 44471 [snip] 44472 Storage> premiumizeme 44473 ** See help for premiumizeme backend at: https://rclone.org/premiumizeme/ ** 44474 44475 Remote config 44476 Use web browser to automatically authenticate rclone with remote? 44477 * Say Y if the machine running rclone has a web browser you can use 44478 * Say N if running rclone on a (remote) machine without web browser access 44479 If not sure try Y. If Y failed, try N. 44480 y) Yes 44481 n) No 44482 y/n> y 44483 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 44484 Log in and authorize rclone for access 44485 Waiting for code... 44486 Got code 44487 -------------------- 44488 [remote] 44489 type = premiumizeme 44490 token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2029-08-07T18:44:15.548915378+01:00"} 44491 -------------------- 44492 y) Yes this is OK 44493 e) Edit this remote 44494 d) Delete this remote 44495 y/e/d> 44496 44497 See the remote setup docs for how to set it up on a machine with no 44498 Internet browser available. 44499 44500 Note that rclone runs a webserver on your local machine to collect the 44501 token as returned from premiumize.me. This only runs from the moment it 44502 opens your browser to the moment you get back the verification code. 44503 This is on http://127.0.0.1:53682/ and this it may require you to 44504 unblock it temporarily if you are running a host firewall. 44505 44506 Once configured you can then use rclone like this, 44507 44508 List directories in top level of your premiumize.me 44509 44510 rclone lsd remote: 44511 44512 List all the files in your premiumize.me 44513 44514 rclone ls remote: 44515 44516 To copy a local directory to an premiumize.me directory called backup 44517 44518 rclone copy /home/source remote:backup 44519 44520 Modification times and hashes 44521 44522 premiumize.me does not support modification times or hashes, therefore 44523 syncing will default to --size-only checking. Note that using --update 44524 will work. 44525 44526 Restricted filename characters 44527 44528 In addition to the default restricted characters set the following 44529 characters are also replaced: 44530 44531 Character Value Replacement 44532 ----------- ------- ------------- 44533 \ 0x5C \ 44534 " 0x22 " 44535 44536 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 44537 strings. 44538 44539 Standard options 44540 44541 Here are the Standard options specific to premiumizeme (premiumize.me). 44542 44543 --premiumizeme-client-id 44544 44545 OAuth Client Id. 44546 44547 Leave blank normally. 44548 44549 Properties: 44550 44551 - Config: client_id 44552 - Env Var: RCLONE_PREMIUMIZEME_CLIENT_ID 44553 - Type: string 44554 - Required: false 44555 44556 --premiumizeme-client-secret 44557 44558 OAuth Client Secret. 44559 44560 Leave blank normally. 44561 44562 Properties: 44563 44564 - Config: client_secret 44565 - Env Var: RCLONE_PREMIUMIZEME_CLIENT_SECRET 44566 - Type: string 44567 - Required: false 44568 44569 --premiumizeme-api-key 44570 44571 API Key. 44572 44573 This is not normally used - use oauth instead. 44574 44575 Properties: 44576 44577 - Config: api_key 44578 - Env Var: RCLONE_PREMIUMIZEME_API_KEY 44579 - Type: string 44580 - Required: false 44581 44582 Advanced options 44583 44584 Here are the Advanced options specific to premiumizeme (premiumize.me). 44585 44586 --premiumizeme-token 44587 44588 OAuth Access Token as a JSON blob. 44589 44590 Properties: 44591 44592 - Config: token 44593 - Env Var: RCLONE_PREMIUMIZEME_TOKEN 44594 - Type: string 44595 - Required: false 44596 44597 --premiumizeme-auth-url 44598 44599 Auth server URL. 44600 44601 Leave blank to use the provider defaults. 44602 44603 Properties: 44604 44605 - Config: auth_url 44606 - Env Var: RCLONE_PREMIUMIZEME_AUTH_URL 44607 - Type: string 44608 - Required: false 44609 44610 --premiumizeme-token-url 44611 44612 Token server url. 44613 44614 Leave blank to use the provider defaults. 44615 44616 Properties: 44617 44618 - Config: token_url 44619 - Env Var: RCLONE_PREMIUMIZEME_TOKEN_URL 44620 - Type: string 44621 - Required: false 44622 44623 --premiumizeme-encoding 44624 44625 The encoding for the backend. 44626 44627 See the encoding section in the overview for more info. 44628 44629 Properties: 44630 44631 - Config: encoding 44632 - Env Var: RCLONE_PREMIUMIZEME_ENCODING 44633 - Type: Encoding 44634 - Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot 44635 44636 --premiumizeme-description 44637 44638 Description of the remote 44639 44640 Properties: 44641 44642 - Config: description 44643 - Env Var: RCLONE_PREMIUMIZEME_DESCRIPTION 44644 - Type: string 44645 - Required: false 44646 44647 Limitations 44648 44649 Note that premiumize.me is case insensitive so you can't have a file 44650 called "Hello.doc" and one called "hello.doc". 44651 44652 premiumize.me file names can't have the \ or " characters in. rclone 44653 maps these to and from an identical looking unicode equivalents \ and 44654 " 44655 44656 premiumize.me only supports filenames up to 255 characters in length. 44657 44658 Proton Drive 44659 44660 Proton Drive is an end-to-end encrypted Swiss vault for your files that 44661 protects your data. 44662 44663 This is an rclone backend for Proton Drive which supports the file 44664 transfer features of Proton Drive using the same client-side encryption. 44665 44666 Due to the fact that Proton Drive doesn't publish its API documentation, 44667 this backend is implemented with best efforts by reading the 44668 open-sourced client source code and observing the Proton Drive traffic 44669 in the browser. 44670 44671 NB This backend is currently in Beta. It is believed to be correct and 44672 all the integration tests pass. However the Proton Drive protocol has 44673 evolved over time there may be accounts it is not compatible with. 44674 Please post on the rclone forum if you find an incompatibility. 44675 44676 Paths are specified as remote:path 44677 44678 Paths may be as deep as required, e.g. remote:directory/subdirectory. 44679 44680 Configurations 44681 44682 Here is an example of how to make a remote called remote. First run: 44683 44684 rclone config 44685 44686 This will guide you through an interactive setup process: 44687 44688 No remotes found, make a new one? 44689 n) New remote 44690 s) Set configuration password 44691 q) Quit config 44692 n/s/q> n 44693 name> remote 44694 Type of storage to configure. 44695 Choose a number from below, or type in your own value 44696 [snip] 44697 XX / Proton Drive 44698 \ "Proton Drive" 44699 [snip] 44700 Storage> protondrive 44701 User name 44702 user> you@protonmail.com 44703 Password. 44704 y) Yes type in my own password 44705 g) Generate random password 44706 n) No leave this optional password blank 44707 y/g/n> y 44708 Enter the password: 44709 password: 44710 Confirm the password: 44711 password: 44712 Option 2fa. 44713 2FA code (if the account requires one) 44714 Enter a value. Press Enter to leave empty. 44715 2fa> 123456 44716 Remote config 44717 -------------------- 44718 [remote] 44719 type = protondrive 44720 user = you@protonmail.com 44721 pass = *** ENCRYPTED *** 44722 -------------------- 44723 y) Yes this is OK 44724 e) Edit this remote 44725 d) Delete this remote 44726 y/e/d> y 44727 44728 NOTE: The Proton Drive encryption keys need to have been already 44729 generated after a regular login via the browser, otherwise attempting to 44730 use the credentials in rclone will fail. 44731 44732 Once configured you can then use rclone like this, 44733 44734 List directories in top level of your Proton Drive 44735 44736 rclone lsd remote: 44737 44738 List all the files in your Proton Drive 44739 44740 rclone ls remote: 44741 44742 To copy a local directory to an Proton Drive directory called backup 44743 44744 rclone copy /home/source remote:backup 44745 44746 Modification times and hashes 44747 44748 Proton Drive Bridge does not support updating modification times yet. 44749 44750 The SHA1 hash algorithm is supported. 44751 44752 Restricted filename characters 44753 44754 Invalid UTF-8 bytes will be replaced, also left and right spaces will be 44755 removed (code reference) 44756 44757 Duplicated files 44758 44759 Proton Drive can not have two files with exactly the same name and path. 44760 If the conflict occurs, depending on the advanced config, the file might 44761 or might not be overwritten. 44762 44763 Mailbox password 44764 44765 Please set your mailbox password in the advanced config section. 44766 44767 Caching 44768 44769 The cache is currently built for the case when the rclone is the only 44770 instance performing operations to the mount point. The event system, 44771 which is the proton API system that provides visibility of what has 44772 changed on the drive, is yet to be implemented, so updates from other 44773 clients won’t be reflected in the cache. Thus, if there are concurrent 44774 clients accessing the same mount point, then we might have a problem 44775 with caching the stale data. 44776 44777 Standard options 44778 44779 Here are the Standard options specific to protondrive (Proton Drive). 44780 44781 --protondrive-username 44782 44783 The username of your proton account 44784 44785 Properties: 44786 44787 - Config: username 44788 - Env Var: RCLONE_PROTONDRIVE_USERNAME 44789 - Type: string 44790 - Required: true 44791 44792 --protondrive-password 44793 44794 The password of your proton account. 44795 44796 NB Input to this must be obscured - see rclone obscure. 44797 44798 Properties: 44799 44800 - Config: password 44801 - Env Var: RCLONE_PROTONDRIVE_PASSWORD 44802 - Type: string 44803 - Required: true 44804 44805 --protondrive-2fa 44806 44807 The 2FA code 44808 44809 The value can also be provided with --protondrive-2fa=000000 44810 44811 The 2FA code of your proton drive account if the account is set up with 44812 two-factor authentication 44813 44814 Properties: 44815 44816 - Config: 2fa 44817 - Env Var: RCLONE_PROTONDRIVE_2FA 44818 - Type: string 44819 - Required: false 44820 44821 Advanced options 44822 44823 Here are the Advanced options specific to protondrive (Proton Drive). 44824 44825 --protondrive-mailbox-password 44826 44827 The mailbox password of your two-password proton account. 44828 44829 For more information regarding the mailbox password, please check the 44830 following official knowledge base article: 44831 https://proton.me/support/the-difference-between-the-mailbox-password-and-login-password 44832 44833 NB Input to this must be obscured - see rclone obscure. 44834 44835 Properties: 44836 44837 - Config: mailbox_password 44838 - Env Var: RCLONE_PROTONDRIVE_MAILBOX_PASSWORD 44839 - Type: string 44840 - Required: false 44841 44842 --protondrive-client-uid 44843 44844 Client uid key (internal use only) 44845 44846 Properties: 44847 44848 - Config: client_uid 44849 - Env Var: RCLONE_PROTONDRIVE_CLIENT_UID 44850 - Type: string 44851 - Required: false 44852 44853 --protondrive-client-access-token 44854 44855 Client access token key (internal use only) 44856 44857 Properties: 44858 44859 - Config: client_access_token 44860 - Env Var: RCLONE_PROTONDRIVE_CLIENT_ACCESS_TOKEN 44861 - Type: string 44862 - Required: false 44863 44864 --protondrive-client-refresh-token 44865 44866 Client refresh token key (internal use only) 44867 44868 Properties: 44869 44870 - Config: client_refresh_token 44871 - Env Var: RCLONE_PROTONDRIVE_CLIENT_REFRESH_TOKEN 44872 - Type: string 44873 - Required: false 44874 44875 --protondrive-client-salted-key-pass 44876 44877 Client salted key pass key (internal use only) 44878 44879 Properties: 44880 44881 - Config: client_salted_key_pass 44882 - Env Var: RCLONE_PROTONDRIVE_CLIENT_SALTED_KEY_PASS 44883 - Type: string 44884 - Required: false 44885 44886 --protondrive-encoding 44887 44888 The encoding for the backend. 44889 44890 See the encoding section in the overview for more info. 44891 44892 Properties: 44893 44894 - Config: encoding 44895 - Env Var: RCLONE_PROTONDRIVE_ENCODING 44896 - Type: Encoding 44897 - Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot 44898 44899 --protondrive-original-file-size 44900 44901 Return the file size before encryption 44902 44903 The size of the encrypted file will be different from (bigger than) the 44904 original file size. Unless there is a reason to return the file size 44905 after encryption is performed, otherwise, set this option to true, as 44906 features like Open() which will need to be supplied with original 44907 content size, will fail to operate properly 44908 44909 Properties: 44910 44911 - Config: original_file_size 44912 - Env Var: RCLONE_PROTONDRIVE_ORIGINAL_FILE_SIZE 44913 - Type: bool 44914 - Default: true 44915 44916 --protondrive-app-version 44917 44918 The app version string 44919 44920 The app version string indicates the client that is currently performing 44921 the API request. This information is required and will be sent with 44922 every API request. 44923 44924 Properties: 44925 44926 - Config: app_version 44927 - Env Var: RCLONE_PROTONDRIVE_APP_VERSION 44928 - Type: string 44929 - Default: "macos-drive@1.0.0-alpha.1+rclone" 44930 44931 --protondrive-replace-existing-draft 44932 44933 Create a new revision when filename conflict is detected 44934 44935 When a file upload is cancelled or failed before completion, a draft 44936 will be created and the subsequent upload of the same file to the same 44937 location will be reported as a conflict. 44938 44939 The value can also be set by --protondrive-replace-existing-draft=true 44940 44941 If the option is set to true, the draft will be replaced and then the 44942 upload operation will restart. If there are other clients also uploading 44943 at the same file location at the same time, the behavior is currently 44944 unknown. Need to set to true for integration tests. If the option is set 44945 to false, an error "a draft exist - usually this means a file is being 44946 uploaded at another client, or, there was a failed upload attempt" will 44947 be returned, and no upload will happen. 44948 44949 Properties: 44950 44951 - Config: replace_existing_draft 44952 - Env Var: RCLONE_PROTONDRIVE_REPLACE_EXISTING_DRAFT 44953 - Type: bool 44954 - Default: false 44955 44956 --protondrive-enable-caching 44957 44958 Caches the files and folders metadata to reduce API calls 44959 44960 Notice: If you are mounting ProtonDrive as a VFS, please disable this 44961 feature, as the current implementation doesn't update or clear the cache 44962 when there are external changes. 44963 44964 The files and folders on ProtonDrive are represented as links with 44965 keyrings, which can be cached to improve performance and be friendly to 44966 the API server. 44967 44968 The cache is currently built for the case when the rclone is the only 44969 instance performing operations to the mount point. The event system, 44970 which is the proton API system that provides visibility of what has 44971 changed on the drive, is yet to be implemented, so updates from other 44972 clients won’t be reflected in the cache. Thus, if there are concurrent 44973 clients accessing the same mount point, then we might have a problem 44974 with caching the stale data. 44975 44976 Properties: 44977 44978 - Config: enable_caching 44979 - Env Var: RCLONE_PROTONDRIVE_ENABLE_CACHING 44980 - Type: bool 44981 - Default: true 44982 44983 --protondrive-description 44984 44985 Description of the remote 44986 44987 Properties: 44988 44989 - Config: description 44990 - Env Var: RCLONE_PROTONDRIVE_DESCRIPTION 44991 - Type: string 44992 - Required: false 44993 44994 Limitations 44995 44996 This backend uses the Proton-API-Bridge, which is based on 44997 go-proton-api, a fork of the official repo. 44998 44999 There is no official API documentation available from Proton Drive. But, 45000 thanks to Proton open sourcing proton-go-api and the web, iOS, and 45001 Android client codebases, we don't need to completely reverse engineer 45002 the APIs by observing the web client traffic! 45003 45004 proton-go-api provides the basic building blocks of API calls and error 45005 handling, such as 429 exponential back-off, but it is pretty much just a 45006 barebone interface to the Proton API. For example, the encryption and 45007 decryption of the Proton Drive file are not provided in this library. 45008 45009 The Proton-API-Bridge, attempts to bridge the gap, so rclone can be 45010 built on top of this quickly. This codebase handles the intricate tasks 45011 before and after calling Proton APIs, particularly the complex 45012 encryption scheme, allowing developers to implement features for other 45013 software on top of this codebase. There are likely quite a few errors in 45014 this library, as there isn't official documentation available. 45015 45016 put.io 45017 45018 Paths are specified as remote:path 45019 45020 put.io paths may be as deep as required, e.g. 45021 remote:directory/subdirectory. 45022 45023 Configuration 45024 45025 The initial setup for put.io involves getting a token from put.io which 45026 you need to do in your browser. rclone config walks you through it. 45027 45028 Here is an example of how to make a remote called remote. First run: 45029 45030 rclone config 45031 45032 This will guide you through an interactive setup process: 45033 45034 No remotes found, make a new one? 45035 n) New remote 45036 s) Set configuration password 45037 q) Quit config 45038 n/s/q> n 45039 name> putio 45040 Type of storage to configure. 45041 Enter a string value. Press Enter for the default (""). 45042 Choose a number from below, or type in your own value 45043 [snip] 45044 XX / Put.io 45045 \ "putio" 45046 [snip] 45047 Storage> putio 45048 ** See help for putio backend at: https://rclone.org/putio/ ** 45049 45050 Remote config 45051 Use web browser to automatically authenticate rclone with remote? 45052 * Say Y if the machine running rclone has a web browser you can use 45053 * Say N if running rclone on a (remote) machine without web browser access 45054 If not sure try Y. If Y failed, try N. 45055 y) Yes 45056 n) No 45057 y/n> y 45058 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 45059 Log in and authorize rclone for access 45060 Waiting for code... 45061 Got code 45062 -------------------- 45063 [putio] 45064 type = putio 45065 token = {"access_token":"XXXXXXXX","expiry":"0001-01-01T00:00:00Z"} 45066 -------------------- 45067 y) Yes this is OK 45068 e) Edit this remote 45069 d) Delete this remote 45070 y/e/d> y 45071 Current remotes: 45072 45073 Name Type 45074 ==== ==== 45075 putio putio 45076 45077 e) Edit existing remote 45078 n) New remote 45079 d) Delete remote 45080 r) Rename remote 45081 c) Copy remote 45082 s) Set configuration password 45083 q) Quit config 45084 e/n/d/r/c/s/q> q 45085 45086 See the remote setup docs for how to set it up on a machine with no 45087 Internet browser available. 45088 45089 Note that rclone runs a webserver on your local machine to collect the 45090 token as returned from put.io if using web browser to automatically 45091 authenticate. This only runs from the moment it opens your browser to 45092 the moment you get back the verification code. This is on 45093 http://127.0.0.1:53682/ and this it may require you to unblock it 45094 temporarily if you are running a host firewall, or use manual mode. 45095 45096 You can then use it like this, 45097 45098 List directories in top level of your put.io 45099 45100 rclone lsd remote: 45101 45102 List all the files in your put.io 45103 45104 rclone ls remote: 45105 45106 To copy a local directory to a put.io directory called backup 45107 45108 rclone copy /home/source remote:backup 45109 45110 Restricted filename characters 45111 45112 In addition to the default restricted characters set the following 45113 characters are also replaced: 45114 45115 Character Value Replacement 45116 ----------- ------- ------------- 45117 \ 0x5C \ 45118 45119 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 45120 strings. 45121 45122 Standard options 45123 45124 Here are the Standard options specific to putio (Put.io). 45125 45126 --putio-client-id 45127 45128 OAuth Client Id. 45129 45130 Leave blank normally. 45131 45132 Properties: 45133 45134 - Config: client_id 45135 - Env Var: RCLONE_PUTIO_CLIENT_ID 45136 - Type: string 45137 - Required: false 45138 45139 --putio-client-secret 45140 45141 OAuth Client Secret. 45142 45143 Leave blank normally. 45144 45145 Properties: 45146 45147 - Config: client_secret 45148 - Env Var: RCLONE_PUTIO_CLIENT_SECRET 45149 - Type: string 45150 - Required: false 45151 45152 Advanced options 45153 45154 Here are the Advanced options specific to putio (Put.io). 45155 45156 --putio-token 45157 45158 OAuth Access Token as a JSON blob. 45159 45160 Properties: 45161 45162 - Config: token 45163 - Env Var: RCLONE_PUTIO_TOKEN 45164 - Type: string 45165 - Required: false 45166 45167 --putio-auth-url 45168 45169 Auth server URL. 45170 45171 Leave blank to use the provider defaults. 45172 45173 Properties: 45174 45175 - Config: auth_url 45176 - Env Var: RCLONE_PUTIO_AUTH_URL 45177 - Type: string 45178 - Required: false 45179 45180 --putio-token-url 45181 45182 Token server url. 45183 45184 Leave blank to use the provider defaults. 45185 45186 Properties: 45187 45188 - Config: token_url 45189 - Env Var: RCLONE_PUTIO_TOKEN_URL 45190 - Type: string 45191 - Required: false 45192 45193 --putio-encoding 45194 45195 The encoding for the backend. 45196 45197 See the encoding section in the overview for more info. 45198 45199 Properties: 45200 45201 - Config: encoding 45202 - Env Var: RCLONE_PUTIO_ENCODING 45203 - Type: Encoding 45204 - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot 45205 45206 --putio-description 45207 45208 Description of the remote 45209 45210 Properties: 45211 45212 - Config: description 45213 - Env Var: RCLONE_PUTIO_DESCRIPTION 45214 - Type: string 45215 - Required: false 45216 45217 Limitations 45218 45219 put.io has rate limiting. When you hit a limit, rclone automatically 45220 retries after waiting the amount of time requested by the server. 45221 45222 If you want to avoid ever hitting these limits, you may use the 45223 --tpslimit flag with a low number. Note that the imposed limits may be 45224 different for different operations, and may change over time. 45225 45226 Proton Drive 45227 45228 Proton Drive is an end-to-end encrypted Swiss vault for your files that 45229 protects your data. 45230 45231 This is an rclone backend for Proton Drive which supports the file 45232 transfer features of Proton Drive using the same client-side encryption. 45233 45234 Due to the fact that Proton Drive doesn't publish its API documentation, 45235 this backend is implemented with best efforts by reading the 45236 open-sourced client source code and observing the Proton Drive traffic 45237 in the browser. 45238 45239 NB This backend is currently in Beta. It is believed to be correct and 45240 all the integration tests pass. However the Proton Drive protocol has 45241 evolved over time there may be accounts it is not compatible with. 45242 Please post on the rclone forum if you find an incompatibility. 45243 45244 Paths are specified as remote:path 45245 45246 Paths may be as deep as required, e.g. remote:directory/subdirectory. 45247 45248 Configurations 45249 45250 Here is an example of how to make a remote called remote. First run: 45251 45252 rclone config 45253 45254 This will guide you through an interactive setup process: 45255 45256 No remotes found, make a new one? 45257 n) New remote 45258 s) Set configuration password 45259 q) Quit config 45260 n/s/q> n 45261 name> remote 45262 Type of storage to configure. 45263 Choose a number from below, or type in your own value 45264 [snip] 45265 XX / Proton Drive 45266 \ "Proton Drive" 45267 [snip] 45268 Storage> protondrive 45269 User name 45270 user> you@protonmail.com 45271 Password. 45272 y) Yes type in my own password 45273 g) Generate random password 45274 n) No leave this optional password blank 45275 y/g/n> y 45276 Enter the password: 45277 password: 45278 Confirm the password: 45279 password: 45280 Option 2fa. 45281 2FA code (if the account requires one) 45282 Enter a value. Press Enter to leave empty. 45283 2fa> 123456 45284 Remote config 45285 -------------------- 45286 [remote] 45287 type = protondrive 45288 user = you@protonmail.com 45289 pass = *** ENCRYPTED *** 45290 -------------------- 45291 y) Yes this is OK 45292 e) Edit this remote 45293 d) Delete this remote 45294 y/e/d> y 45295 45296 NOTE: The Proton Drive encryption keys need to have been already 45297 generated after a regular login via the browser, otherwise attempting to 45298 use the credentials in rclone will fail. 45299 45300 Once configured you can then use rclone like this, 45301 45302 List directories in top level of your Proton Drive 45303 45304 rclone lsd remote: 45305 45306 List all the files in your Proton Drive 45307 45308 rclone ls remote: 45309 45310 To copy a local directory to an Proton Drive directory called backup 45311 45312 rclone copy /home/source remote:backup 45313 45314 Modification times and hashes 45315 45316 Proton Drive Bridge does not support updating modification times yet. 45317 45318 The SHA1 hash algorithm is supported. 45319 45320 Restricted filename characters 45321 45322 Invalid UTF-8 bytes will be replaced, also left and right spaces will be 45323 removed (code reference) 45324 45325 Duplicated files 45326 45327 Proton Drive can not have two files with exactly the same name and path. 45328 If the conflict occurs, depending on the advanced config, the file might 45329 or might not be overwritten. 45330 45331 Mailbox password 45332 45333 Please set your mailbox password in the advanced config section. 45334 45335 Caching 45336 45337 The cache is currently built for the case when the rclone is the only 45338 instance performing operations to the mount point. The event system, 45339 which is the proton API system that provides visibility of what has 45340 changed on the drive, is yet to be implemented, so updates from other 45341 clients won’t be reflected in the cache. Thus, if there are concurrent 45342 clients accessing the same mount point, then we might have a problem 45343 with caching the stale data. 45344 45345 Standard options 45346 45347 Here are the Standard options specific to protondrive (Proton Drive). 45348 45349 --protondrive-username 45350 45351 The username of your proton account 45352 45353 Properties: 45354 45355 - Config: username 45356 - Env Var: RCLONE_PROTONDRIVE_USERNAME 45357 - Type: string 45358 - Required: true 45359 45360 --protondrive-password 45361 45362 The password of your proton account. 45363 45364 NB Input to this must be obscured - see rclone obscure. 45365 45366 Properties: 45367 45368 - Config: password 45369 - Env Var: RCLONE_PROTONDRIVE_PASSWORD 45370 - Type: string 45371 - Required: true 45372 45373 --protondrive-2fa 45374 45375 The 2FA code 45376 45377 The value can also be provided with --protondrive-2fa=000000 45378 45379 The 2FA code of your proton drive account if the account is set up with 45380 two-factor authentication 45381 45382 Properties: 45383 45384 - Config: 2fa 45385 - Env Var: RCLONE_PROTONDRIVE_2FA 45386 - Type: string 45387 - Required: false 45388 45389 Advanced options 45390 45391 Here are the Advanced options specific to protondrive (Proton Drive). 45392 45393 --protondrive-mailbox-password 45394 45395 The mailbox password of your two-password proton account. 45396 45397 For more information regarding the mailbox password, please check the 45398 following official knowledge base article: 45399 https://proton.me/support/the-difference-between-the-mailbox-password-and-login-password 45400 45401 NB Input to this must be obscured - see rclone obscure. 45402 45403 Properties: 45404 45405 - Config: mailbox_password 45406 - Env Var: RCLONE_PROTONDRIVE_MAILBOX_PASSWORD 45407 - Type: string 45408 - Required: false 45409 45410 --protondrive-client-uid 45411 45412 Client uid key (internal use only) 45413 45414 Properties: 45415 45416 - Config: client_uid 45417 - Env Var: RCLONE_PROTONDRIVE_CLIENT_UID 45418 - Type: string 45419 - Required: false 45420 45421 --protondrive-client-access-token 45422 45423 Client access token key (internal use only) 45424 45425 Properties: 45426 45427 - Config: client_access_token 45428 - Env Var: RCLONE_PROTONDRIVE_CLIENT_ACCESS_TOKEN 45429 - Type: string 45430 - Required: false 45431 45432 --protondrive-client-refresh-token 45433 45434 Client refresh token key (internal use only) 45435 45436 Properties: 45437 45438 - Config: client_refresh_token 45439 - Env Var: RCLONE_PROTONDRIVE_CLIENT_REFRESH_TOKEN 45440 - Type: string 45441 - Required: false 45442 45443 --protondrive-client-salted-key-pass 45444 45445 Client salted key pass key (internal use only) 45446 45447 Properties: 45448 45449 - Config: client_salted_key_pass 45450 - Env Var: RCLONE_PROTONDRIVE_CLIENT_SALTED_KEY_PASS 45451 - Type: string 45452 - Required: false 45453 45454 --protondrive-encoding 45455 45456 The encoding for the backend. 45457 45458 See the encoding section in the overview for more info. 45459 45460 Properties: 45461 45462 - Config: encoding 45463 - Env Var: RCLONE_PROTONDRIVE_ENCODING 45464 - Type: Encoding 45465 - Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot 45466 45467 --protondrive-original-file-size 45468 45469 Return the file size before encryption 45470 45471 The size of the encrypted file will be different from (bigger than) the 45472 original file size. Unless there is a reason to return the file size 45473 after encryption is performed, otherwise, set this option to true, as 45474 features like Open() which will need to be supplied with original 45475 content size, will fail to operate properly 45476 45477 Properties: 45478 45479 - Config: original_file_size 45480 - Env Var: RCLONE_PROTONDRIVE_ORIGINAL_FILE_SIZE 45481 - Type: bool 45482 - Default: true 45483 45484 --protondrive-app-version 45485 45486 The app version string 45487 45488 The app version string indicates the client that is currently performing 45489 the API request. This information is required and will be sent with 45490 every API request. 45491 45492 Properties: 45493 45494 - Config: app_version 45495 - Env Var: RCLONE_PROTONDRIVE_APP_VERSION 45496 - Type: string 45497 - Default: "macos-drive@1.0.0-alpha.1+rclone" 45498 45499 --protondrive-replace-existing-draft 45500 45501 Create a new revision when filename conflict is detected 45502 45503 When a file upload is cancelled or failed before completion, a draft 45504 will be created and the subsequent upload of the same file to the same 45505 location will be reported as a conflict. 45506 45507 The value can also be set by --protondrive-replace-existing-draft=true 45508 45509 If the option is set to true, the draft will be replaced and then the 45510 upload operation will restart. If there are other clients also uploading 45511 at the same file location at the same time, the behavior is currently 45512 unknown. Need to set to true for integration tests. If the option is set 45513 to false, an error "a draft exist - usually this means a file is being 45514 uploaded at another client, or, there was a failed upload attempt" will 45515 be returned, and no upload will happen. 45516 45517 Properties: 45518 45519 - Config: replace_existing_draft 45520 - Env Var: RCLONE_PROTONDRIVE_REPLACE_EXISTING_DRAFT 45521 - Type: bool 45522 - Default: false 45523 45524 --protondrive-enable-caching 45525 45526 Caches the files and folders metadata to reduce API calls 45527 45528 Notice: If you are mounting ProtonDrive as a VFS, please disable this 45529 feature, as the current implementation doesn't update or clear the cache 45530 when there are external changes. 45531 45532 The files and folders on ProtonDrive are represented as links with 45533 keyrings, which can be cached to improve performance and be friendly to 45534 the API server. 45535 45536 The cache is currently built for the case when the rclone is the only 45537 instance performing operations to the mount point. The event system, 45538 which is the proton API system that provides visibility of what has 45539 changed on the drive, is yet to be implemented, so updates from other 45540 clients won’t be reflected in the cache. Thus, if there are concurrent 45541 clients accessing the same mount point, then we might have a problem 45542 with caching the stale data. 45543 45544 Properties: 45545 45546 - Config: enable_caching 45547 - Env Var: RCLONE_PROTONDRIVE_ENABLE_CACHING 45548 - Type: bool 45549 - Default: true 45550 45551 --protondrive-description 45552 45553 Description of the remote 45554 45555 Properties: 45556 45557 - Config: description 45558 - Env Var: RCLONE_PROTONDRIVE_DESCRIPTION 45559 - Type: string 45560 - Required: false 45561 45562 Limitations 45563 45564 This backend uses the Proton-API-Bridge, which is based on 45565 go-proton-api, a fork of the official repo. 45566 45567 There is no official API documentation available from Proton Drive. But, 45568 thanks to Proton open sourcing proton-go-api and the web, iOS, and 45569 Android client codebases, we don't need to completely reverse engineer 45570 the APIs by observing the web client traffic! 45571 45572 proton-go-api provides the basic building blocks of API calls and error 45573 handling, such as 429 exponential back-off, but it is pretty much just a 45574 barebone interface to the Proton API. For example, the encryption and 45575 decryption of the Proton Drive file are not provided in this library. 45576 45577 The Proton-API-Bridge, attempts to bridge the gap, so rclone can be 45578 built on top of this quickly. This codebase handles the intricate tasks 45579 before and after calling Proton APIs, particularly the complex 45580 encryption scheme, allowing developers to implement features for other 45581 software on top of this codebase. There are likely quite a few errors in 45582 this library, as there isn't official documentation available. 45583 45584 Seafile 45585 45586 This is a backend for the Seafile storage service: - It works with both 45587 the free community edition or the professional edition. - Seafile 45588 versions 6.x, 7.x, 8.x and 9.x are all supported. - Encrypted libraries 45589 are also supported. - It supports 2FA enabled users - Using a Library 45590 API Token is not supported 45591 45592 Configuration 45593 45594 There are two distinct modes you can setup your remote: - you point your 45595 remote to the root of the server, meaning you don't specify a library 45596 during the configuration: Paths are specified as remote:library. You may 45597 put subdirectories in too, e.g. remote:library/path/to/dir. - you point 45598 your remote to a specific library during the configuration: Paths are 45599 specified as remote:path/to/dir. This is the recommended mode when using 45600 encrypted libraries. (This mode is possibly slightly faster than the 45601 root mode) 45602 45603 Configuration in root mode 45604 45605 Here is an example of making a seafile configuration for a user with no 45606 two-factor authentication. First run 45607 45608 rclone config 45609 45610 This will guide you through an interactive setup process. To 45611 authenticate you will need the URL of your server, your email (or 45612 username) and your password. 45613 45614 No remotes found, make a new one? 45615 n) New remote 45616 s) Set configuration password 45617 q) Quit config 45618 n/s/q> n 45619 name> seafile 45620 Type of storage to configure. 45621 Enter a string value. Press Enter for the default (""). 45622 Choose a number from below, or type in your own value 45623 [snip] 45624 XX / Seafile 45625 \ "seafile" 45626 [snip] 45627 Storage> seafile 45628 ** See help for seafile backend at: https://rclone.org/seafile/ ** 45629 45630 URL of seafile host to connect to 45631 Enter a string value. Press Enter for the default (""). 45632 Choose a number from below, or type in your own value 45633 1 / Connect to cloud.seafile.com 45634 \ "https://cloud.seafile.com/" 45635 url> http://my.seafile.server/ 45636 User name (usually email address) 45637 Enter a string value. Press Enter for the default (""). 45638 user> me@example.com 45639 Password 45640 y) Yes type in my own password 45641 g) Generate random password 45642 n) No leave this optional password blank (default) 45643 y/g> y 45644 Enter the password: 45645 password: 45646 Confirm the password: 45647 password: 45648 Two-factor authentication ('true' if the account has 2FA enabled) 45649 Enter a boolean value (true or false). Press Enter for the default ("false"). 45650 2fa> false 45651 Name of the library. Leave blank to access all non-encrypted libraries. 45652 Enter a string value. Press Enter for the default (""). 45653 library> 45654 Library password (for encrypted libraries only). Leave blank if you pass it through the command line. 45655 y) Yes type in my own password 45656 g) Generate random password 45657 n) No leave this optional password blank (default) 45658 y/g/n> n 45659 Edit advanced config? (y/n) 45660 y) Yes 45661 n) No (default) 45662 y/n> n 45663 Remote config 45664 Two-factor authentication is not enabled on this account. 45665 -------------------- 45666 [seafile] 45667 type = seafile 45668 url = http://my.seafile.server/ 45669 user = me@example.com 45670 pass = *** ENCRYPTED *** 45671 2fa = false 45672 -------------------- 45673 y) Yes this is OK (default) 45674 e) Edit this remote 45675 d) Delete this remote 45676 y/e/d> y 45677 45678 This remote is called seafile. It's pointing to the root of your seafile 45679 server and can now be used like this: 45680 45681 See all libraries 45682 45683 rclone lsd seafile: 45684 45685 Create a new library 45686 45687 rclone mkdir seafile:library 45688 45689 List the contents of a library 45690 45691 rclone ls seafile:library 45692 45693 Sync /home/local/directory to the remote library, deleting any excess 45694 files in the library. 45695 45696 rclone sync --interactive /home/local/directory seafile:library 45697 45698 Configuration in library mode 45699 45700 Here's an example of a configuration in library mode with a user that 45701 has the two-factor authentication enabled. Your 2FA code will be asked 45702 at the end of the configuration, and will attempt to authenticate you: 45703 45704 No remotes found, make a new one? 45705 n) New remote 45706 s) Set configuration password 45707 q) Quit config 45708 n/s/q> n 45709 name> seafile 45710 Type of storage to configure. 45711 Enter a string value. Press Enter for the default (""). 45712 Choose a number from below, or type in your own value 45713 [snip] 45714 XX / Seafile 45715 \ "seafile" 45716 [snip] 45717 Storage> seafile 45718 ** See help for seafile backend at: https://rclone.org/seafile/ ** 45719 45720 URL of seafile host to connect to 45721 Enter a string value. Press Enter for the default (""). 45722 Choose a number from below, or type in your own value 45723 1 / Connect to cloud.seafile.com 45724 \ "https://cloud.seafile.com/" 45725 url> http://my.seafile.server/ 45726 User name (usually email address) 45727 Enter a string value. Press Enter for the default (""). 45728 user> me@example.com 45729 Password 45730 y) Yes type in my own password 45731 g) Generate random password 45732 n) No leave this optional password blank (default) 45733 y/g> y 45734 Enter the password: 45735 password: 45736 Confirm the password: 45737 password: 45738 Two-factor authentication ('true' if the account has 2FA enabled) 45739 Enter a boolean value (true or false). Press Enter for the default ("false"). 45740 2fa> true 45741 Name of the library. Leave blank to access all non-encrypted libraries. 45742 Enter a string value. Press Enter for the default (""). 45743 library> My Library 45744 Library password (for encrypted libraries only). Leave blank if you pass it through the command line. 45745 y) Yes type in my own password 45746 g) Generate random password 45747 n) No leave this optional password blank (default) 45748 y/g/n> n 45749 Edit advanced config? (y/n) 45750 y) Yes 45751 n) No (default) 45752 y/n> n 45753 Remote config 45754 Two-factor authentication: please enter your 2FA code 45755 2fa code> 123456 45756 Authenticating... 45757 Success! 45758 -------------------- 45759 [seafile] 45760 type = seafile 45761 url = http://my.seafile.server/ 45762 user = me@example.com 45763 pass = 45764 2fa = true 45765 library = My Library 45766 -------------------- 45767 y) Yes this is OK (default) 45768 e) Edit this remote 45769 d) Delete this remote 45770 y/e/d> y 45771 45772 You'll notice your password is blank in the configuration. It's because 45773 we only need the password to authenticate you once. 45774 45775 You specified My Library during the configuration. The root of the 45776 remote is pointing at the root of the library My Library: 45777 45778 See all files in the library: 45779 45780 rclone lsd seafile: 45781 45782 Create a new directory inside the library 45783 45784 rclone mkdir seafile:directory 45785 45786 List the contents of a directory 45787 45788 rclone ls seafile:directory 45789 45790 Sync /home/local/directory to the remote library, deleting any excess 45791 files in the library. 45792 45793 rclone sync --interactive /home/local/directory seafile: 45794 45795 --fast-list 45796 45797 Seafile version 7+ supports --fast-list which allows you to use fewer 45798 transactions in exchange for more memory. See the rclone docs for more 45799 details. Please note this is not supported on seafile server version 6.x 45800 45801 Restricted filename characters 45802 45803 In addition to the default restricted characters set the following 45804 characters are also replaced: 45805 45806 Character Value Replacement 45807 ----------- ------- ------------- 45808 / 0x2F / 45809 " 0x22 " 45810 \ 0x5C \ 45811 45812 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 45813 strings. 45814 45815 Seafile and rclone link 45816 45817 Rclone supports generating share links for non-encrypted libraries only. 45818 They can either be for a file or a directory: 45819 45820 rclone link seafile:seafile-tutorial.doc 45821 http://my.seafile.server/f/fdcd8a2f93f84b8b90f4/ 45822 45823 or if run on a directory you will get: 45824 45825 rclone link seafile:dir 45826 http://my.seafile.server/d/9ea2455f6f55478bbb0d/ 45827 45828 Please note a share link is unique for each file or directory. If you 45829 run a link command on a file/dir that has already been shared, you will 45830 get the exact same link. 45831 45832 Compatibility 45833 45834 It has been actively developed using the seafile docker image of these 45835 versions: - 6.3.4 community edition - 7.0.5 community edition - 7.1.3 45836 community edition - 9.0.10 community edition 45837 45838 Versions below 6.0 are not supported. Versions between 6.0 and 6.3 45839 haven't been tested and might not work properly. 45840 45841 Each new version of rclone is automatically tested against the latest 45842 docker image of the seafile community server. 45843 45844 Standard options 45845 45846 Here are the Standard options specific to seafile (seafile). 45847 45848 --seafile-url 45849 45850 URL of seafile host to connect to. 45851 45852 Properties: 45853 45854 - Config: url 45855 - Env Var: RCLONE_SEAFILE_URL 45856 - Type: string 45857 - Required: true 45858 - Examples: 45859 - "https://cloud.seafile.com/" 45860 - Connect to cloud.seafile.com. 45861 45862 --seafile-user 45863 45864 User name (usually email address). 45865 45866 Properties: 45867 45868 - Config: user 45869 - Env Var: RCLONE_SEAFILE_USER 45870 - Type: string 45871 - Required: true 45872 45873 --seafile-pass 45874 45875 Password. 45876 45877 NB Input to this must be obscured - see rclone obscure. 45878 45879 Properties: 45880 45881 - Config: pass 45882 - Env Var: RCLONE_SEAFILE_PASS 45883 - Type: string 45884 - Required: false 45885 45886 --seafile-2fa 45887 45888 Two-factor authentication ('true' if the account has 2FA enabled). 45889 45890 Properties: 45891 45892 - Config: 2fa 45893 - Env Var: RCLONE_SEAFILE_2FA 45894 - Type: bool 45895 - Default: false 45896 45897 --seafile-library 45898 45899 Name of the library. 45900 45901 Leave blank to access all non-encrypted libraries. 45902 45903 Properties: 45904 45905 - Config: library 45906 - Env Var: RCLONE_SEAFILE_LIBRARY 45907 - Type: string 45908 - Required: false 45909 45910 --seafile-library-key 45911 45912 Library password (for encrypted libraries only). 45913 45914 Leave blank if you pass it through the command line. 45915 45916 NB Input to this must be obscured - see rclone obscure. 45917 45918 Properties: 45919 45920 - Config: library_key 45921 - Env Var: RCLONE_SEAFILE_LIBRARY_KEY 45922 - Type: string 45923 - Required: false 45924 45925 --seafile-auth-token 45926 45927 Authentication token. 45928 45929 Properties: 45930 45931 - Config: auth_token 45932 - Env Var: RCLONE_SEAFILE_AUTH_TOKEN 45933 - Type: string 45934 - Required: false 45935 45936 Advanced options 45937 45938 Here are the Advanced options specific to seafile (seafile). 45939 45940 --seafile-create-library 45941 45942 Should rclone create a library if it doesn't exist. 45943 45944 Properties: 45945 45946 - Config: create_library 45947 - Env Var: RCLONE_SEAFILE_CREATE_LIBRARY 45948 - Type: bool 45949 - Default: false 45950 45951 --seafile-encoding 45952 45953 The encoding for the backend. 45954 45955 See the encoding section in the overview for more info. 45956 45957 Properties: 45958 45959 - Config: encoding 45960 - Env Var: RCLONE_SEAFILE_ENCODING 45961 - Type: Encoding 45962 - Default: Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8 45963 45964 --seafile-description 45965 45966 Description of the remote 45967 45968 Properties: 45969 45970 - Config: description 45971 - Env Var: RCLONE_SEAFILE_DESCRIPTION 45972 - Type: string 45973 - Required: false 45974 45975 SFTP 45976 45977 SFTP is the Secure (or SSH) File Transfer Protocol. 45978 45979 The SFTP backend can be used with a number of different providers: 45980 45981 - Hetzner Storage Box 45982 - rsync.net 45983 45984 SFTP runs over SSH v2 and is installed as standard with most modern SSH 45985 installations. 45986 45987 Paths are specified as remote:path. If the path does not begin with a / 45988 it is relative to the home directory of the user. An empty path remote: 45989 refers to the user's home directory. For example, rclone lsd remote: 45990 would list the home directory of the user configured in the rclone 45991 remote config (i.e /home/sftpuser). However, rclone lsd remote:/ would 45992 list the root directory for remote machine (i.e. /) 45993 45994 Note that some SFTP servers will need the leading / - Synology is a good 45995 example of this. rsync.net and Hetzner, on the other hand, requires 45996 users to OMIT the leading /. 45997 45998 Note that by default rclone will try to execute shell commands on the 45999 server, see shell access considerations. 46000 46001 Configuration 46002 46003 Here is an example of making an SFTP configuration. First run 46004 46005 rclone config 46006 46007 This will guide you through an interactive setup process. 46008 46009 No remotes found, make a new one? 46010 n) New remote 46011 s) Set configuration password 46012 q) Quit config 46013 n/s/q> n 46014 name> remote 46015 Type of storage to configure. 46016 Choose a number from below, or type in your own value 46017 [snip] 46018 XX / SSH/SFTP 46019 \ "sftp" 46020 [snip] 46021 Storage> sftp 46022 SSH host to connect to 46023 Choose a number from below, or type in your own value 46024 1 / Connect to example.com 46025 \ "example.com" 46026 host> example.com 46027 SSH username 46028 Enter a string value. Press Enter for the default ("$USER"). 46029 user> sftpuser 46030 SSH port number 46031 Enter a signed integer. Press Enter for the default (22). 46032 port> 46033 SSH password, leave blank to use ssh-agent. 46034 y) Yes type in my own password 46035 g) Generate random password 46036 n) No leave this optional password blank 46037 y/g/n> n 46038 Path to unencrypted PEM-encoded private key file, leave blank to use ssh-agent. 46039 key_file> 46040 Remote config 46041 -------------------- 46042 [remote] 46043 host = example.com 46044 user = sftpuser 46045 port = 46046 pass = 46047 key_file = 46048 -------------------- 46049 y) Yes this is OK 46050 e) Edit this remote 46051 d) Delete this remote 46052 y/e/d> y 46053 46054 This remote is called remote and can now be used like this: 46055 46056 See all directories in the home directory 46057 46058 rclone lsd remote: 46059 46060 See all directories in the root directory 46061 46062 rclone lsd remote:/ 46063 46064 Make a new directory 46065 46066 rclone mkdir remote:path/to/directory 46067 46068 List the contents of a directory 46069 46070 rclone ls remote:path/to/directory 46071 46072 Sync /home/local/directory to the remote directory, deleting any excess 46073 files in the directory. 46074 46075 rclone sync --interactive /home/local/directory remote:directory 46076 46077 Mount the remote path /srv/www-data/ to the local path /mnt/www-data 46078 46079 rclone mount remote:/srv/www-data/ /mnt/www-data 46080 46081 SSH Authentication 46082 46083 The SFTP remote supports three authentication methods: 46084 46085 - Password 46086 - Key file, including certificate signed keys 46087 - ssh-agent 46088 46089 Key files should be PEM-encoded private key files. For instance 46090 /home/$USER/.ssh/id_rsa. Only unencrypted OpenSSH or PEM encrypted files 46091 are supported. 46092 46093 The key file can be specified in either an external file (key_file) or 46094 contained within the rclone config file (key_pem). If using key_pem in 46095 the config file, the entry should be on a single line with new line ('' 46096 or '') separating lines. i.e. 46097 46098 key_pem = -----BEGIN RSA PRIVATE KEY-----\nMaMbaIXtE\n0gAMbMbaSsd\nMbaass\n-----END RSA PRIVATE KEY----- 46099 46100 This will generate it correctly for key_pem for use in the config: 46101 46102 awk '{printf "%s\\n", $0}' < ~/.ssh/id_rsa 46103 46104 If you don't specify pass, key_file, or key_pem or ask_password then 46105 rclone will attempt to contact an ssh-agent. You can also specify 46106 key_use_agent to force the usage of an ssh-agent. In this case key_file 46107 or key_pem can also be specified to force the usage of a specific key in 46108 the ssh-agent. 46109 46110 Using an ssh-agent is the only way to load encrypted OpenSSH keys at the 46111 moment. 46112 46113 If you set the ask_password option, rclone will prompt for a password 46114 when needed and no password has been configured. 46115 46116 Certificate-signed keys 46117 46118 With traditional key-based authentication, you configure your private 46119 key only, and the public key built into it will be used during the 46120 authentication process. 46121 46122 If you have a certificate you may use it to sign your public key, 46123 creating a separate SSH user certificate that should be used instead of 46124 the plain public key extracted from the private key. Then you must 46125 provide the path to the user certificate public key file in pubkey_file. 46126 46127 Note: This is not the traditional public key paired with your private 46128 key, typically saved as /home/$USER/.ssh/id_rsa.pub. Setting this path 46129 in pubkey_file will not work. 46130 46131 Example: 46132 46133 [remote] 46134 type = sftp 46135 host = example.com 46136 user = sftpuser 46137 key_file = ~/id_rsa 46138 pubkey_file = ~/id_rsa-cert.pub 46139 46140 If you concatenate a cert with a private key then you can specify the 46141 merged file in both places. 46142 46143 Note: the cert must come first in the file. e.g. 46144 46145 cat id_rsa-cert.pub id_rsa > merged_key 46146 46147 Host key validation 46148 46149 By default rclone will not check the server's host key for validation. 46150 This can allow an attacker to replace a server with their own and if you 46151 use password authentication then this can lead to that password being 46152 exposed. 46153 46154 Host key matching, using standard known_hosts files can be turned on by 46155 enabling the known_hosts_file option. This can point to the file 46156 maintained by OpenSSH or can point to a unique file. 46157 46158 e.g. using the OpenSSH known_hosts file: 46159 46160 [remote] 46161 type = sftp 46162 host = example.com 46163 user = sftpuser 46164 pass = 46165 known_hosts_file = ~/.ssh/known_hosts 46166 46167 Alternatively you can create your own known hosts file like this: 46168 46169 ssh-keyscan -t dsa,rsa,ecdsa,ed25519 example.com >> known_hosts 46170 46171 There are some limitations: 46172 46173 - rclone will not manage this file for you. If the key is missing or 46174 wrong then the connection will be refused. 46175 - If the server is set up for a certificate host key then the entry in 46176 the known_hosts file must be the @cert-authority entry for the CA 46177 46178 If the host key provided by the server does not match the one in the 46179 file (or is missing) then the connection will be aborted and an error 46180 returned such as 46181 46182 NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch 46183 46184 or 46185 46186 NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key is unknown 46187 46188 If you see an error such as 46189 46190 NewFs: couldn't connect SSH: ssh: handshake failed: ssh: no authorities for hostname: example.com:22 46191 46192 then it is likely the server has presented a CA signed host certificate 46193 and you will need to add the appropriate @cert-authority entry. 46194 46195 The known_hosts_file setting can be set during rclone config as an 46196 advanced option. 46197 46198 ssh-agent on macOS 46199 46200 Note that there seem to be various problems with using an ssh-agent on 46201 macOS due to recent changes in the OS. The most effective work-around 46202 seems to be to start an ssh-agent in each session, e.g. 46203 46204 eval `ssh-agent -s` && ssh-add -A 46205 46206 And then at the end of the session 46207 46208 eval `ssh-agent -k` 46209 46210 These commands can be used in scripts of course. 46211 46212 Shell access 46213 46214 Some functionality of the SFTP backend relies on remote shell access, 46215 and the possibility to execute commands. This includes checksum, and in 46216 some cases also about. The shell commands that must be executed may be 46217 different on different type of shells, and also quoting/escaping of file 46218 path arguments containing special characters may be different. Rclone 46219 therefore needs to know what type of shell it is, and if shell access is 46220 available at all. 46221 46222 Most servers run on some version of Unix, and then a basic Unix shell 46223 can be assumed, without further distinction. Windows 10, Server 2019, 46224 and later can also run a SSH server, which is a port of OpenSSH (see 46225 official installation guide). On a Windows server the shell handling is 46226 different: Although it can also be set up to use a Unix type shell, e.g. 46227 Cygwin bash, the default is to use Windows Command Prompt (cmd.exe), and 46228 PowerShell is a recommended alternative. All of these have behave 46229 differently, which rclone must handle. 46230 46231 Rclone tries to auto-detect what type of shell is used on the server, 46232 first time you access the SFTP remote. If a remote shell session is 46233 successfully created, it will look for indications that it is CMD or 46234 PowerShell, with fall-back to Unix if not something else is detected. If 46235 unable to even create a remote shell session, then shell command 46236 execution will be disabled entirely. The result is stored in the SFTP 46237 remote configuration, in option shell_type, so that the auto-detection 46238 only have to be performed once. If you manually set a value for this 46239 option before first run, the auto-detection will be skipped, and if you 46240 set a different value later this will override any existing. Value none 46241 can be set to avoid any attempts at executing shell commands, e.g. if 46242 this is not allowed on the server. 46243 46244 When the server is rclone serve sftp, the rclone SFTP remote will detect 46245 this as a Unix type shell - even if it is running on Windows. This 46246 server does not actually have a shell, but it accepts input commands 46247 matching the specific ones that the SFTP backend relies on for Unix 46248 shells, e.g. md5sum and df. Also it handles the string escape rules used 46249 for Unix shell. Treating it as a Unix type shell from a SFTP remote will 46250 therefore always be correct, and support all features. 46251 46252 Shell access considerations 46253 46254 The shell type auto-detection logic, described above, means that by 46255 default rclone will try to run a shell command the first time a new sftp 46256 remote is accessed. If you configure a sftp remote without a config 46257 file, e.g. an on the fly remote, rclone will have nowhere to store the 46258 result, and it will re-run the command on every access. To avoid this 46259 you should explicitly set the shell_type option to the correct value, or 46260 to none if you want to prevent rclone from executing any remote shell 46261 commands. 46262 46263 It is also important to note that, since the shell type decides how 46264 quoting and escaping of file paths used as command-line arguments are 46265 performed, configuring the wrong shell type may leave you exposed to 46266 command injection exploits. Make sure to confirm the auto-detected shell 46267 type, or explicitly set the shell type you know is correct, or disable 46268 shell access until you know. 46269 46270 Checksum 46271 46272 SFTP does not natively support checksums (file hash), but rclone is able 46273 to use checksumming if the same login has shell access, and can execute 46274 remote commands. If there is a command that can calculate compatible 46275 checksums on the remote system, Rclone can then be configured to execute 46276 this whenever a checksum is needed, and read back the results. Currently 46277 MD5 and SHA-1 are supported. 46278 46279 Normally this requires an external utility being available on the 46280 server. By default rclone will try commands md5sum, md5 and 46281 rclone md5sum for MD5 checksums, and the first one found usable will be 46282 picked. Same with sha1sum, sha1 and rclone sha1sum commands for SHA-1 46283 checksums. These utilities normally need to be in the remote's PATH to 46284 be found. 46285 46286 In some cases the shell itself is capable of calculating checksums. 46287 PowerShell is an example of such a shell. If rclone detects that the 46288 remote shell is PowerShell, which means it most probably is a Windows 46289 OpenSSH server, rclone will use a predefined script block to produce the 46290 checksums when no external checksum commands are found (see shell 46291 access). This assumes PowerShell version 4.0 or newer. 46292 46293 The options md5sum_command and sha1_command can be used to customize the 46294 command to be executed for calculation of checksums. You can for example 46295 set a specific path to where md5sum and sha1sum executables are located, 46296 or use them to specify some other tools that print checksums in 46297 compatible format. The value can include command-line arguments, or even 46298 shell script blocks as with PowerShell. Rclone has subcommands md5sum 46299 and sha1sum that use compatible format, which means if you have an 46300 rclone executable on the server it can be used. As mentioned above, they 46301 will be automatically picked up if found in PATH, but if not you can set 46302 something like /path/to/rclone md5sum as the value of option 46303 md5sum_command to make sure a specific executable is used. 46304 46305 Remote checksumming is recommended and enabled by default. First time 46306 rclone is using a SFTP remote, if options md5sum_command or sha1_command 46307 are not set, it will check if any of the default commands for each of 46308 them, as described above, can be used. The result will be saved in the 46309 remote configuration, so next time it will use the same. Value none will 46310 be set if none of the default commands could be used for a specific 46311 algorithm, and this algorithm will not be supported by the remote. 46312 46313 Disabling the checksumming may be required if you are connecting to SFTP 46314 servers which are not under your control, and to which the execution of 46315 remote shell commands is prohibited. Set the configuration option 46316 disable_hashcheck to true to disable checksumming entirely, or set 46317 shell_type to none to disable all functionality based on remote shell 46318 command execution. 46319 46320 Modification times and hashes 46321 46322 Modified times are stored on the server to 1 second precision. 46323 46324 Modified times are used in syncing and are fully supported. 46325 46326 Some SFTP servers disable setting/modifying the file modification time 46327 after upload (for example, certain configurations of ProFTPd with 46328 mod_sftp). If you are using one of these servers, you can set the option 46329 set_modtime = false in your RClone backend configuration to disable this 46330 behaviour. 46331 46332 About command 46333 46334 The about command returns the total space, free space, and used space on 46335 the remote for the disk of the specified path on the remote or, if not 46336 set, the disk of the root on the remote. 46337 46338 SFTP usually supports the about command, but it depends on the server. 46339 If the server implements the vendor-specific VFS statistics extension, 46340 which is normally the case with OpenSSH instances, it will be used. If 46341 not, but the same login has access to a Unix shell, where the df command 46342 is available (e.g. in the remote's PATH), then this will be used 46343 instead. If the server shell is PowerShell, probably with a Windows 46344 OpenSSH server, rclone will use a built-in shell command (see shell 46345 access). If none of the above is applicable, about will fail. 46346 46347 Standard options 46348 46349 Here are the Standard options specific to sftp (SSH/SFTP). 46350 46351 --sftp-host 46352 46353 SSH host to connect to. 46354 46355 E.g. "example.com". 46356 46357 Properties: 46358 46359 - Config: host 46360 - Env Var: RCLONE_SFTP_HOST 46361 - Type: string 46362 - Required: true 46363 46364 --sftp-user 46365 46366 SSH username. 46367 46368 Properties: 46369 46370 - Config: user 46371 - Env Var: RCLONE_SFTP_USER 46372 - Type: string 46373 - Default: "$USER" 46374 46375 --sftp-port 46376 46377 SSH port number. 46378 46379 Properties: 46380 46381 - Config: port 46382 - Env Var: RCLONE_SFTP_PORT 46383 - Type: int 46384 - Default: 22 46385 46386 --sftp-pass 46387 46388 SSH password, leave blank to use ssh-agent. 46389 46390 NB Input to this must be obscured - see rclone obscure. 46391 46392 Properties: 46393 46394 - Config: pass 46395 - Env Var: RCLONE_SFTP_PASS 46396 - Type: string 46397 - Required: false 46398 46399 --sftp-key-pem 46400 46401 Raw PEM-encoded private key. 46402 46403 If specified, will override key_file parameter. 46404 46405 Properties: 46406 46407 - Config: key_pem 46408 - Env Var: RCLONE_SFTP_KEY_PEM 46409 - Type: string 46410 - Required: false 46411 46412 --sftp-key-file 46413 46414 Path to PEM-encoded private key file. 46415 46416 Leave blank or set key-use-agent to use ssh-agent. 46417 46418 Leading ~ will be expanded in the file name as will environment 46419 variables such as ${RCLONE_CONFIG_DIR}. 46420 46421 Properties: 46422 46423 - Config: key_file 46424 - Env Var: RCLONE_SFTP_KEY_FILE 46425 - Type: string 46426 - Required: false 46427 46428 --sftp-key-file-pass 46429 46430 The passphrase to decrypt the PEM-encoded private key file. 46431 46432 Only PEM encrypted key files (old OpenSSH format) are supported. 46433 Encrypted keys in the new OpenSSH format can't be used. 46434 46435 NB Input to this must be obscured - see rclone obscure. 46436 46437 Properties: 46438 46439 - Config: key_file_pass 46440 - Env Var: RCLONE_SFTP_KEY_FILE_PASS 46441 - Type: string 46442 - Required: false 46443 46444 --sftp-pubkey-file 46445 46446 Optional path to public key file. 46447 46448 Set this if you have a signed certificate you want to use for 46449 authentication. 46450 46451 Leading ~ will be expanded in the file name as will environment 46452 variables such as ${RCLONE_CONFIG_DIR}. 46453 46454 Properties: 46455 46456 - Config: pubkey_file 46457 - Env Var: RCLONE_SFTP_PUBKEY_FILE 46458 - Type: string 46459 - Required: false 46460 46461 --sftp-key-use-agent 46462 46463 When set forces the usage of the ssh-agent. 46464 46465 When key-file is also set, the ".pub" file of the specified key-file is 46466 read and only the associated key is requested from the ssh-agent. This 46467 allows to avoid Too many authentication failures for *username* errors 46468 when the ssh-agent contains many keys. 46469 46470 Properties: 46471 46472 - Config: key_use_agent 46473 - Env Var: RCLONE_SFTP_KEY_USE_AGENT 46474 - Type: bool 46475 - Default: false 46476 46477 --sftp-use-insecure-cipher 46478 46479 Enable the use of insecure ciphers and key exchange methods. 46480 46481 This enables the use of the following insecure ciphers and key exchange 46482 methods: 46483 46484 - aes128-cbc 46485 - aes192-cbc 46486 - aes256-cbc 46487 - 3des-cbc 46488 - diffie-hellman-group-exchange-sha256 46489 - diffie-hellman-group-exchange-sha1 46490 46491 Those algorithms are insecure and may allow plaintext data to be 46492 recovered by an attacker. 46493 46494 This must be false if you use either ciphers or key_exchange advanced 46495 options. 46496 46497 Properties: 46498 46499 - Config: use_insecure_cipher 46500 - Env Var: RCLONE_SFTP_USE_INSECURE_CIPHER 46501 - Type: bool 46502 - Default: false 46503 - Examples: 46504 - "false" 46505 - Use default Cipher list. 46506 - "true" 46507 - Enables the use of the aes128-cbc cipher and 46508 diffie-hellman-group-exchange-sha256, 46509 diffie-hellman-group-exchange-sha1 key exchange. 46510 46511 --sftp-disable-hashcheck 46512 46513 Disable the execution of SSH commands to determine if remote file 46514 hashing is available. 46515 46516 Leave blank or set to false to enable hashing (recommended), set to true 46517 to disable hashing. 46518 46519 Properties: 46520 46521 - Config: disable_hashcheck 46522 - Env Var: RCLONE_SFTP_DISABLE_HASHCHECK 46523 - Type: bool 46524 - Default: false 46525 46526 --sftp-ssh 46527 46528 Path and arguments to external ssh binary. 46529 46530 Normally rclone will use its internal ssh library to connect to the SFTP 46531 server. However it does not implement all possible ssh options so it may 46532 be desirable to use an external ssh binary. 46533 46534 Rclone ignores all the internal config if you use this option and 46535 expects you to configure the ssh binary with the user/host/port and any 46536 other options you need. 46537 46538 Important The ssh command must log in without asking for a password so 46539 needs to be configured with keys or certificates. 46540 46541 Rclone will run the command supplied either with the additional 46542 arguments "-s sftp" to access the SFTP subsystem or with commands such 46543 as "md5sum /path/to/file" appended to read checksums. 46544 46545 Any arguments with spaces in should be surrounded by "double quotes". 46546 46547 An example setting might be: 46548 46549 ssh -o ServerAliveInterval=20 user@example.com 46550 46551 Note that when using an external ssh binary rclone makes a new ssh 46552 connection for every hash it calculates. 46553 46554 Properties: 46555 46556 - Config: ssh 46557 - Env Var: RCLONE_SFTP_SSH 46558 - Type: SpaceSepList 46559 - Default: 46560 46561 Advanced options 46562 46563 Here are the Advanced options specific to sftp (SSH/SFTP). 46564 46565 --sftp-known-hosts-file 46566 46567 Optional path to known_hosts file. 46568 46569 Set this value to enable server host key validation. 46570 46571 Leading ~ will be expanded in the file name as will environment 46572 variables such as ${RCLONE_CONFIG_DIR}. 46573 46574 Properties: 46575 46576 - Config: known_hosts_file 46577 - Env Var: RCLONE_SFTP_KNOWN_HOSTS_FILE 46578 - Type: string 46579 - Required: false 46580 - Examples: 46581 - "~/.ssh/known_hosts" 46582 - Use OpenSSH's known_hosts file. 46583 46584 --sftp-ask-password 46585 46586 Allow asking for SFTP password when needed. 46587 46588 If this is set and no password is supplied then rclone will: - ask for a 46589 password - not contact the ssh agent 46590 46591 Properties: 46592 46593 - Config: ask_password 46594 - Env Var: RCLONE_SFTP_ASK_PASSWORD 46595 - Type: bool 46596 - Default: false 46597 46598 --sftp-path-override 46599 46600 Override path used by SSH shell commands. 46601 46602 This allows checksum calculation when SFTP and SSH paths are different. 46603 This issue affects among others Synology NAS boxes. 46604 46605 E.g. if shared folders can be found in directories representing volumes: 46606 46607 rclone sync /home/local/directory remote:/directory --sftp-path-override /volume2/directory 46608 46609 E.g. if home directory can be found in a shared folder called "home": 46610 46611 rclone sync /home/local/directory remote:/home/directory --sftp-path-override /volume1/homes/USER/directory 46612 46613 To specify only the path to the SFTP remote's root, and allow rclone to 46614 add any relative subpaths automatically (including unwrapping/decrypting 46615 remotes as necessary), add the '@' character to the beginning of the 46616 path. 46617 46618 E.g. the first example above could be rewritten as: 46619 46620 rclone sync /home/local/directory remote:/directory --sftp-path-override @/volume2 46621 46622 Note that when using this method with Synology "home" folders, the full 46623 "/homes/USER" path should be specified instead of "/home". 46624 46625 E.g. the second example above should be rewritten as: 46626 46627 rclone sync /home/local/directory remote:/homes/USER/directory --sftp-path-override @/volume1 46628 46629 Properties: 46630 46631 - Config: path_override 46632 - Env Var: RCLONE_SFTP_PATH_OVERRIDE 46633 - Type: string 46634 - Required: false 46635 46636 --sftp-set-modtime 46637 46638 Set the modified time on the remote if set. 46639 46640 Properties: 46641 46642 - Config: set_modtime 46643 - Env Var: RCLONE_SFTP_SET_MODTIME 46644 - Type: bool 46645 - Default: true 46646 46647 --sftp-shell-type 46648 46649 The type of SSH shell on remote server, if any. 46650 46651 Leave blank for autodetect. 46652 46653 Properties: 46654 46655 - Config: shell_type 46656 - Env Var: RCLONE_SFTP_SHELL_TYPE 46657 - Type: string 46658 - Required: false 46659 - Examples: 46660 - "none" 46661 - No shell access 46662 - "unix" 46663 - Unix shell 46664 - "powershell" 46665 - PowerShell 46666 - "cmd" 46667 - Windows Command Prompt 46668 46669 --sftp-md5sum-command 46670 46671 The command used to read md5 hashes. 46672 46673 Leave blank for autodetect. 46674 46675 Properties: 46676 46677 - Config: md5sum_command 46678 - Env Var: RCLONE_SFTP_MD5SUM_COMMAND 46679 - Type: string 46680 - Required: false 46681 46682 --sftp-sha1sum-command 46683 46684 The command used to read sha1 hashes. 46685 46686 Leave blank for autodetect. 46687 46688 Properties: 46689 46690 - Config: sha1sum_command 46691 - Env Var: RCLONE_SFTP_SHA1SUM_COMMAND 46692 - Type: string 46693 - Required: false 46694 46695 --sftp-skip-links 46696 46697 Set to skip any symlinks and any other non regular files. 46698 46699 Properties: 46700 46701 - Config: skip_links 46702 - Env Var: RCLONE_SFTP_SKIP_LINKS 46703 - Type: bool 46704 - Default: false 46705 46706 --sftp-subsystem 46707 46708 Specifies the SSH2 subsystem on the remote host. 46709 46710 Properties: 46711 46712 - Config: subsystem 46713 - Env Var: RCLONE_SFTP_SUBSYSTEM 46714 - Type: string 46715 - Default: "sftp" 46716 46717 --sftp-server-command 46718 46719 Specifies the path or command to run a sftp server on the remote host. 46720 46721 The subsystem option is ignored when server_command is defined. 46722 46723 If adding server_command to the configuration file please note that it 46724 should not be enclosed in quotes, since that will make rclone fail. 46725 46726 A working example is: 46727 46728 [remote_name] 46729 type = sftp 46730 server_command = sudo /usr/libexec/openssh/sftp-server 46731 46732 Properties: 46733 46734 - Config: server_command 46735 - Env Var: RCLONE_SFTP_SERVER_COMMAND 46736 - Type: string 46737 - Required: false 46738 46739 --sftp-use-fstat 46740 46741 If set use fstat instead of stat. 46742 46743 Some servers limit the amount of open files and calling Stat after 46744 opening the file will throw an error from the server. Setting this flag 46745 will call Fstat instead of Stat which is called on an already open file 46746 handle. 46747 46748 It has been found that this helps with IBM Sterling SFTP servers which 46749 have "extractability" level set to 1 which means only 1 file can be 46750 opened at any given time. 46751 46752 Properties: 46753 46754 - Config: use_fstat 46755 - Env Var: RCLONE_SFTP_USE_FSTAT 46756 - Type: bool 46757 - Default: false 46758 46759 --sftp-disable-concurrent-reads 46760 46761 If set don't use concurrent reads. 46762 46763 Normally concurrent reads are safe to use and not using them will 46764 degrade performance, so this option is disabled by default. 46765 46766 Some servers limit the amount number of times a file can be downloaded. 46767 Using concurrent reads can trigger this limit, so if you have a server 46768 which returns 46769 46770 Failed to copy: file does not exist 46771 46772 Then you may need to enable this flag. 46773 46774 If concurrent reads are disabled, the use_fstat option is ignored. 46775 46776 Properties: 46777 46778 - Config: disable_concurrent_reads 46779 - Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS 46780 - Type: bool 46781 - Default: false 46782 46783 --sftp-disable-concurrent-writes 46784 46785 If set don't use concurrent writes. 46786 46787 Normally rclone uses concurrent writes to upload files. This improves 46788 the performance greatly, especially for distant servers. 46789 46790 This option disables concurrent writes should that be necessary. 46791 46792 Properties: 46793 46794 - Config: disable_concurrent_writes 46795 - Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_WRITES 46796 - Type: bool 46797 - Default: false 46798 46799 --sftp-idle-timeout 46800 46801 Max time before closing idle connections. 46802 46803 If no connections have been returned to the connection pool in the time 46804 given, rclone will empty the connection pool. 46805 46806 Set to 0 to keep connections indefinitely. 46807 46808 Properties: 46809 46810 - Config: idle_timeout 46811 - Env Var: RCLONE_SFTP_IDLE_TIMEOUT 46812 - Type: Duration 46813 - Default: 1m0s 46814 46815 --sftp-chunk-size 46816 46817 Upload and download chunk size. 46818 46819 This controls the maximum size of payload in SFTP protocol packets. The 46820 RFC limits this to 32768 bytes (32k), which is the default. However, a 46821 lot of servers support larger sizes, typically limited to a maximum 46822 total package size of 256k, and setting it larger will increase transfer 46823 speed dramatically on high latency links. This includes OpenSSH, and, 46824 for example, using the value of 255k works well, leaving plenty of room 46825 for overhead while still being within a total packet size of 256k. 46826 46827 Make sure to test thoroughly before using a value higher than 32k, and 46828 only use it if you always connect to the same server or after 46829 sufficiently broad testing. If you get errors such as "failed to send 46830 packet payload: EOF", lots of "connection lost", or "corrupted on 46831 transfer", when copying a larger file, try lowering the value. The 46832 server run by rclone serve sftp sends packets with standard 32k maximum 46833 payload so you must not set a different chunk_size when downloading 46834 files, but it accepts packets up to the 256k total size, so for uploads 46835 the chunk_size can be set as for the OpenSSH example above. 46836 46837 Properties: 46838 46839 - Config: chunk_size 46840 - Env Var: RCLONE_SFTP_CHUNK_SIZE 46841 - Type: SizeSuffix 46842 - Default: 32Ki 46843 46844 --sftp-concurrency 46845 46846 The maximum number of outstanding requests for one file 46847 46848 This controls the maximum number of outstanding requests for one file. 46849 Increasing it will increase throughput on high latency links at the cost 46850 of using more memory. 46851 46852 Properties: 46853 46854 - Config: concurrency 46855 - Env Var: RCLONE_SFTP_CONCURRENCY 46856 - Type: int 46857 - Default: 64 46858 46859 --sftp-set-env 46860 46861 Environment variables to pass to sftp and commands 46862 46863 Set environment variables in the form: 46864 46865 VAR=value 46866 46867 to be passed to the sftp client and to any commands run (eg md5sum). 46868 46869 Pass multiple variables space separated, eg 46870 46871 VAR1=value VAR2=value 46872 46873 and pass variables with spaces in quotes, eg 46874 46875 "VAR3=value with space" "VAR4=value with space" VAR5=nospacehere 46876 46877 Properties: 46878 46879 - Config: set_env 46880 - Env Var: RCLONE_SFTP_SET_ENV 46881 - Type: SpaceSepList 46882 - Default: 46883 46884 --sftp-ciphers 46885 46886 Space separated list of ciphers to be used for session encryption, 46887 ordered by preference. 46888 46889 At least one must match with server configuration. This can be checked 46890 for example using ssh -Q cipher. 46891 46892 This must not be set if use_insecure_cipher is true. 46893 46894 Example: 46895 46896 aes128-ctr aes192-ctr aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com 46897 46898 Properties: 46899 46900 - Config: ciphers 46901 - Env Var: RCLONE_SFTP_CIPHERS 46902 - Type: SpaceSepList 46903 - Default: 46904 46905 --sftp-key-exchange 46906 46907 Space separated list of key exchange algorithms, ordered by preference. 46908 46909 At least one must match with server configuration. This can be checked 46910 for example using ssh -Q kex. 46911 46912 This must not be set if use_insecure_cipher is true. 46913 46914 Example: 46915 46916 sntrup761x25519-sha512@openssh.com curve25519-sha256 curve25519-sha256@libssh.org ecdh-sha2-nistp256 46917 46918 Properties: 46919 46920 - Config: key_exchange 46921 - Env Var: RCLONE_SFTP_KEY_EXCHANGE 46922 - Type: SpaceSepList 46923 - Default: 46924 46925 --sftp-macs 46926 46927 Space separated list of MACs (message authentication code) algorithms, 46928 ordered by preference. 46929 46930 At least one must match with server configuration. This can be checked 46931 for example using ssh -Q mac. 46932 46933 Example: 46934 46935 umac-64-etm@openssh.com umac-128-etm@openssh.com hmac-sha2-256-etm@openssh.com 46936 46937 Properties: 46938 46939 - Config: macs 46940 - Env Var: RCLONE_SFTP_MACS 46941 - Type: SpaceSepList 46942 - Default: 46943 46944 --sftp-host-key-algorithms 46945 46946 Space separated list of host key algorithms, ordered by preference. 46947 46948 At least one must match with server configuration. This can be checked 46949 for example using ssh -Q HostKeyAlgorithms. 46950 46951 Note: This can affect the outcome of key negotiation with the server 46952 even if server host key validation is not enabled. 46953 46954 Example: 46955 46956 ssh-ed25519 ssh-rsa ssh-dss 46957 46958 Properties: 46959 46960 - Config: host_key_algorithms 46961 - Env Var: RCLONE_SFTP_HOST_KEY_ALGORITHMS 46962 - Type: SpaceSepList 46963 - Default: 46964 46965 --sftp-socks-proxy 46966 46967 Socks 5 proxy host. 46968 46969 Supports the format user:pass@host:port, user@host:port, host:port. 46970 46971 Example: 46972 46973 myUser:myPass@localhost:9005 46974 46975 Properties: 46976 46977 - Config: socks_proxy 46978 - Env Var: RCLONE_SFTP_SOCKS_PROXY 46979 - Type: string 46980 - Required: false 46981 46982 --sftp-copy-is-hardlink 46983 46984 Set to enable server side copies using hardlinks. 46985 46986 The SFTP protocol does not define a copy command so normally server side 46987 copies are not allowed with the sftp backend. 46988 46989 However the SFTP protocol does support hardlinking, and if you enable 46990 this flag then the sftp backend will support server side copies. These 46991 will be implemented by doing a hardlink from the source to the 46992 destination. 46993 46994 Not all sftp servers support this. 46995 46996 Note that hardlinking two files together will use no additional space as 46997 the source and the destination will be the same file. 46998 46999 This feature may be useful backups made with --copy-dest. 47000 47001 Properties: 47002 47003 - Config: copy_is_hardlink 47004 - Env Var: RCLONE_SFTP_COPY_IS_HARDLINK 47005 - Type: bool 47006 - Default: false 47007 47008 --sftp-description 47009 47010 Description of the remote 47011 47012 Properties: 47013 47014 - Config: description 47015 - Env Var: RCLONE_SFTP_DESCRIPTION 47016 - Type: string 47017 - Required: false 47018 47019 Limitations 47020 47021 On some SFTP servers (e.g. Synology) the paths are different for SSH and 47022 SFTP so the hashes can't be calculated properly. For them using 47023 disable_hashcheck is a good idea. 47024 47025 The only ssh agent supported under Windows is Putty's pageant. 47026 47027 The Go SSH library disables the use of the aes128-cbc cipher by default, 47028 due to security concerns. This can be re-enabled on a per-connection 47029 basis by setting the use_insecure_cipher setting in the configuration 47030 file to true. Further details on the insecurity of this cipher can be 47031 found in this paper. 47032 47033 SFTP isn't supported under plan9 until this issue is fixed. 47034 47035 Note that since SFTP isn't HTTP based the following flags don't work 47036 with it: --dump-headers, --dump-bodies, --dump-auth. 47037 47038 Note that --timeout and --contimeout are both supported. 47039 47040 rsync.net 47041 47042 rsync.net is supported through the SFTP backend. 47043 47044 See rsync.net's documentation of rclone examples. 47045 47046 Hetzner Storage Box 47047 47048 Hetzner Storage Boxes are supported through the SFTP backend on port 23. 47049 47050 See Hetzner's documentation for details 47051 47052 SMB 47053 47054 SMB is a communication protocol to share files over network. 47055 47056 This relies on go-smb2 library for communication with SMB protocol. 47057 47058 Paths are specified as remote:sharename (or remote: for the lsd 47059 command.) You may put subdirectories in too, e.g. 47060 remote:item/path/to/dir. 47061 47062 Notes 47063 47064 The first path segment must be the name of the share, which you entered 47065 when you started to share on Windows. On smbd, it's the section title in 47066 smb.conf (usually in /etc/samba/) file. You can find shares by querying 47067 the root if you're unsure (e.g. rclone lsd remote:). 47068 47069 You can't access to the shared printers from rclone, obviously. 47070 47071 You can't use Anonymous access for logging in. You have to use the guest 47072 user with an empty password instead. The rclone client tries to avoid 47073 8.3 names when uploading files by encoding trailing spaces and periods. 47074 Alternatively, the local backend on Windows can access SMB servers using 47075 UNC paths, by \\server\share. This doesn't apply to non-Windows OSes, 47076 such as Linux and macOS. 47077 47078 Configuration 47079 47080 Here is an example of making a SMB configuration. 47081 47082 First run 47083 47084 rclone config 47085 47086 This will guide you through an interactive setup process. 47087 47088 No remotes found, make a new one? 47089 n) New remote 47090 s) Set configuration password 47091 q) Quit config 47092 n/s/q> n 47093 name> remote 47094 Option Storage. 47095 Type of storage to configure. 47096 Choose a number from below, or type in your own value. 47097 XX / SMB / CIFS 47098 \ (smb) 47099 Storage> smb 47100 47101 Option host. 47102 Samba hostname to connect to. 47103 E.g. "example.com". 47104 Enter a value. 47105 host> localhost 47106 47107 Option user. 47108 Samba username. 47109 Enter a string value. Press Enter for the default (lesmi). 47110 user> guest 47111 47112 Option port. 47113 Samba port number. 47114 Enter a signed integer. Press Enter for the default (445). 47115 port> 47116 47117 Option pass. 47118 Samba password. 47119 Choose an alternative below. Press Enter for the default (n). 47120 y) Yes, type in my own password 47121 g) Generate random password 47122 n) No, leave this optional password blank (default) 47123 y/g/n> g 47124 Password strength in bits. 47125 64 is just about memorable 47126 128 is secure 47127 1024 is the maximum 47128 Bits> 64 47129 Your password is: XXXX 47130 Use this password? Please note that an obscured version of this 47131 password (and not the password itself) will be stored under your 47132 configuration file, so keep this generated password in a safe place. 47133 y) Yes (default) 47134 n) No 47135 y/n> y 47136 47137 Option domain. 47138 Domain name for NTLM authentication. 47139 Enter a string value. Press Enter for the default (WORKGROUP). 47140 domain> 47141 47142 Edit advanced config? 47143 y) Yes 47144 n) No (default) 47145 y/n> n 47146 47147 Configuration complete. 47148 Options: 47149 - type: samba 47150 - host: localhost 47151 - user: guest 47152 - pass: *** ENCRYPTED *** 47153 Keep this "remote" remote? 47154 y) Yes this is OK (default) 47155 e) Edit this remote 47156 d) Delete this remote 47157 y/e/d> d 47158 47159 Standard options 47160 47161 Here are the Standard options specific to smb (SMB / CIFS). 47162 47163 --smb-host 47164 47165 SMB server hostname to connect to. 47166 47167 E.g. "example.com". 47168 47169 Properties: 47170 47171 - Config: host 47172 - Env Var: RCLONE_SMB_HOST 47173 - Type: string 47174 - Required: true 47175 47176 --smb-user 47177 47178 SMB username. 47179 47180 Properties: 47181 47182 - Config: user 47183 - Env Var: RCLONE_SMB_USER 47184 - Type: string 47185 - Default: "$USER" 47186 47187 --smb-port 47188 47189 SMB port number. 47190 47191 Properties: 47192 47193 - Config: port 47194 - Env Var: RCLONE_SMB_PORT 47195 - Type: int 47196 - Default: 445 47197 47198 --smb-pass 47199 47200 SMB password. 47201 47202 NB Input to this must be obscured - see rclone obscure. 47203 47204 Properties: 47205 47206 - Config: pass 47207 - Env Var: RCLONE_SMB_PASS 47208 - Type: string 47209 - Required: false 47210 47211 --smb-domain 47212 47213 Domain name for NTLM authentication. 47214 47215 Properties: 47216 47217 - Config: domain 47218 - Env Var: RCLONE_SMB_DOMAIN 47219 - Type: string 47220 - Default: "WORKGROUP" 47221 47222 --smb-spn 47223 47224 Service principal name. 47225 47226 Rclone presents this name to the server. Some servers use this as 47227 further authentication, and it often needs to be set for clusters. For 47228 example: 47229 47230 cifs/remotehost:1020 47231 47232 Leave blank if not sure. 47233 47234 Properties: 47235 47236 - Config: spn 47237 - Env Var: RCLONE_SMB_SPN 47238 - Type: string 47239 - Required: false 47240 47241 Advanced options 47242 47243 Here are the Advanced options specific to smb (SMB / CIFS). 47244 47245 --smb-idle-timeout 47246 47247 Max time before closing idle connections. 47248 47249 If no connections have been returned to the connection pool in the time 47250 given, rclone will empty the connection pool. 47251 47252 Set to 0 to keep connections indefinitely. 47253 47254 Properties: 47255 47256 - Config: idle_timeout 47257 - Env Var: RCLONE_SMB_IDLE_TIMEOUT 47258 - Type: Duration 47259 - Default: 1m0s 47260 47261 --smb-hide-special-share 47262 47263 Hide special shares (e.g. print$) which users aren't supposed to access. 47264 47265 Properties: 47266 47267 - Config: hide_special_share 47268 - Env Var: RCLONE_SMB_HIDE_SPECIAL_SHARE 47269 - Type: bool 47270 - Default: true 47271 47272 --smb-case-insensitive 47273 47274 Whether the server is configured to be case-insensitive. 47275 47276 Always true on Windows shares. 47277 47278 Properties: 47279 47280 - Config: case_insensitive 47281 - Env Var: RCLONE_SMB_CASE_INSENSITIVE 47282 - Type: bool 47283 - Default: true 47284 47285 --smb-encoding 47286 47287 The encoding for the backend. 47288 47289 See the encoding section in the overview for more info. 47290 47291 Properties: 47292 47293 - Config: encoding 47294 - Env Var: RCLONE_SMB_ENCODING 47295 - Type: Encoding 47296 - Default: 47297 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 47298 47299 --smb-description 47300 47301 Description of the remote 47302 47303 Properties: 47304 47305 - Config: description 47306 - Env Var: RCLONE_SMB_DESCRIPTION 47307 - Type: string 47308 - Required: false 47309 47310 Storj 47311 47312 Storj is an encrypted, secure, and cost-effective object storage service 47313 that enables you to store, back up, and archive large amounts of data in 47314 a decentralized manner. 47315 47316 Backend options 47317 47318 Storj can be used both with this native backend and with the s3 backend 47319 using the Storj S3 compatible gateway (shared or private). 47320 47321 Use this backend to take advantage of client-side encryption as well as 47322 to achieve the best possible download performance. Uploads will be 47323 erasure-coded locally, thus a 1gb upload will result in 2.68gb of data 47324 being uploaded to storage nodes across the network. 47325 47326 Use the s3 backend and one of the S3 compatible Hosted Gateways to 47327 increase upload performance and reduce the load on your systems and 47328 network. Uploads will be encrypted and erasure-coded server-side, thus a 47329 1GB upload will result in only in 1GB of data being uploaded to storage 47330 nodes across the network. 47331 47332 Side by side comparison with more details: 47333 47334 - Characteristics: 47335 - Storj backend: Uses native RPC protocol, connects directly to 47336 the storage nodes which hosts the data. Requires more CPU 47337 resource of encoding/decoding and has network amplification 47338 (especially during the upload), uses lots of TCP connections 47339 - S3 backend: Uses S3 compatible HTTP Rest API via the shared 47340 gateways. There is no network amplification, but performance 47341 depends on the shared gateways and the secret encryption key is 47342 shared with the gateway. 47343 - Typical usage: 47344 - Storj backend: Server environments and desktops with enough 47345 resources, internet speed and connectivity - and applications 47346 where storjs client-side encryption is required. 47347 - S3 backend: Desktops and similar with limited resources, 47348 internet speed or connectivity. 47349 - Security: 47350 - Storj backend: strong. Private encryption key doesn't need to 47351 leave the local computer. 47352 - S3 backend: weaker. Private encryption key is shared with the 47353 authentication service of the hosted gateway, where it's stored 47354 encrypted. It can be stronger when combining with the rclone 47355 crypt backend. 47356 - Bandwidth usage (upload): 47357 - Storj backend: higher. As data is erasure coded on the client 47358 side both the original data and the parities should be uploaded. 47359 About ~2.7 times more data is required to be uploaded. Client 47360 may start to upload with even higher number of nodes (~3.7 times 47361 more) and abandon/stop the slow uploads. 47362 - S3 backend: normal. Only the raw data is uploaded, erasure 47363 coding happens on the gateway. 47364 - Bandwidth usage (download) 47365 - Storj backend: almost normal. Only the minimal number of data is 47366 required, but to avoid very slow data providers a few more 47367 sources are used and the slowest are ignored (max 1.2x 47368 overhead). 47369 - S3 backend: normal. Only the raw data is downloaded, erasure 47370 coding happens on the shared gateway. 47371 - CPU usage: 47372 - Storj backend: higher, but more predictable. Erasure code and 47373 encryption/decryption happens locally which requires significant 47374 CPU usage. 47375 - S3 backend: less. Erasure code and encryption/decryption happens 47376 on shared s3 gateways (and as is, it depends on the current load 47377 on the gateways) 47378 - TCP connection usage: 47379 - Storj backend: high. A direct connection is required to each of 47380 the Storj nodes resulting in 110 connections on upload and 35 on 47381 download per 64 MB segment. Not all the connections are actively 47382 used (slow ones are pruned), but they are all opened. Adjusting 47383 the max open file limit may be required. 47384 - S3 backend: normal. Only one connection per download/upload 47385 thread is required to the shared gateway. 47386 - Overall performance: 47387 - Storj backend: with enough resources (CPU and bandwidth) storj 47388 backend can provide even 2x better performance. Data is directly 47389 downloaded to / uploaded from to the client instead of the 47390 gateway. 47391 - S3 backend: Can be faster on edge devices where CPU and network 47392 bandwidth is limited as the shared S3 compatible gateways take 47393 care about the encrypting/decryption and erasure coding and no 47394 download/upload amplification. 47395 - Decentralization: 47396 - Storj backend: high. Data is downloaded directly from the 47397 distributed cloud of storage providers. 47398 - S3 backend: low. Requires a running S3 gateway (either 47399 self-hosted or Storj-hosted). 47400 - Limitations: 47401 - Storj backend: rclone checksum is not possible without download, 47402 as checksum metadata is not calculated during upload 47403 - S3 backend: secret encryption key is shared with the gateway 47404 47405 Configuration 47406 47407 To make a new Storj configuration you need one of the following: * 47408 Access Grant that someone else shared with you. * API Key of a Storj 47409 project you are a member of. 47410 47411 Here is an example of how to make a remote called remote. First run: 47412 47413 rclone config 47414 47415 This will guide you through an interactive setup process: 47416 47417 Setup with access grant 47418 47419 No remotes found, make a new one? 47420 n) New remote 47421 s) Set configuration password 47422 q) Quit config 47423 n/s/q> n 47424 name> remote 47425 Type of storage to configure. 47426 Enter a string value. Press Enter for the default (""). 47427 Choose a number from below, or type in your own value 47428 [snip] 47429 XX / Storj Decentralized Cloud Storage 47430 \ "storj" 47431 [snip] 47432 Storage> storj 47433 ** See help for storj backend at: https://rclone.org/storj/ ** 47434 47435 Choose an authentication method. 47436 Enter a string value. Press Enter for the default ("existing"). 47437 Choose a number from below, or type in your own value 47438 1 / Use an existing access grant. 47439 \ "existing" 47440 2 / Create a new access grant from satellite address, API key, and passphrase. 47441 \ "new" 47442 provider> existing 47443 Access Grant. 47444 Enter a string value. Press Enter for the default (""). 47445 access_grant> your-access-grant-received-by-someone-else 47446 Remote config 47447 -------------------- 47448 [remote] 47449 type = storj 47450 access_grant = your-access-grant-received-by-someone-else 47451 -------------------- 47452 y) Yes this is OK (default) 47453 e) Edit this remote 47454 d) Delete this remote 47455 y/e/d> y 47456 47457 Setup with API key and passphrase 47458 47459 No remotes found, make a new one? 47460 n) New remote 47461 s) Set configuration password 47462 q) Quit config 47463 n/s/q> n 47464 name> remote 47465 Type of storage to configure. 47466 Enter a string value. Press Enter for the default (""). 47467 Choose a number from below, or type in your own value 47468 [snip] 47469 XX / Storj Decentralized Cloud Storage 47470 \ "storj" 47471 [snip] 47472 Storage> storj 47473 ** See help for storj backend at: https://rclone.org/storj/ ** 47474 47475 Choose an authentication method. 47476 Enter a string value. Press Enter for the default ("existing"). 47477 Choose a number from below, or type in your own value 47478 1 / Use an existing access grant. 47479 \ "existing" 47480 2 / Create a new access grant from satellite address, API key, and passphrase. 47481 \ "new" 47482 provider> new 47483 Satellite Address. Custom satellite address should match the format: `<nodeid>@<address>:<port>`. 47484 Enter a string value. Press Enter for the default ("us1.storj.io"). 47485 Choose a number from below, or type in your own value 47486 1 / US1 47487 \ "us1.storj.io" 47488 2 / EU1 47489 \ "eu1.storj.io" 47490 3 / AP1 47491 \ "ap1.storj.io" 47492 satellite_address> 1 47493 API Key. 47494 Enter a string value. Press Enter for the default (""). 47495 api_key> your-api-key-for-your-storj-project 47496 Encryption Passphrase. To access existing objects enter passphrase used for uploading. 47497 Enter a string value. Press Enter for the default (""). 47498 passphrase> your-human-readable-encryption-passphrase 47499 Remote config 47500 -------------------- 47501 [remote] 47502 type = storj 47503 satellite_address = 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777 47504 api_key = your-api-key-for-your-storj-project 47505 passphrase = your-human-readable-encryption-passphrase 47506 access_grant = the-access-grant-generated-from-the-api-key-and-passphrase 47507 -------------------- 47508 y) Yes this is OK (default) 47509 e) Edit this remote 47510 d) Delete this remote 47511 y/e/d> y 47512 47513 Standard options 47514 47515 Here are the Standard options specific to storj (Storj Decentralized 47516 Cloud Storage). 47517 47518 --storj-provider 47519 47520 Choose an authentication method. 47521 47522 Properties: 47523 47524 - Config: provider 47525 - Env Var: RCLONE_STORJ_PROVIDER 47526 - Type: string 47527 - Default: "existing" 47528 - Examples: 47529 - "existing" 47530 - Use an existing access grant. 47531 - "new" 47532 - Create a new access grant from satellite address, API key, 47533 and passphrase. 47534 47535 --storj-access-grant 47536 47537 Access grant. 47538 47539 Properties: 47540 47541 - Config: access_grant 47542 - Env Var: RCLONE_STORJ_ACCESS_GRANT 47543 - Provider: existing 47544 - Type: string 47545 - Required: false 47546 47547 --storj-satellite-address 47548 47549 Satellite address. 47550 47551 Custom satellite address should match the format: 47552 <nodeid>@<address>:<port>. 47553 47554 Properties: 47555 47556 - Config: satellite_address 47557 - Env Var: RCLONE_STORJ_SATELLITE_ADDRESS 47558 - Provider: new 47559 - Type: string 47560 - Default: "us1.storj.io" 47561 - Examples: 47562 - "us1.storj.io" 47563 - US1 47564 - "eu1.storj.io" 47565 - EU1 47566 - "ap1.storj.io" 47567 - AP1 47568 47569 --storj-api-key 47570 47571 API key. 47572 47573 Properties: 47574 47575 - Config: api_key 47576 - Env Var: RCLONE_STORJ_API_KEY 47577 - Provider: new 47578 - Type: string 47579 - Required: false 47580 47581 --storj-passphrase 47582 47583 Encryption passphrase. 47584 47585 To access existing objects enter passphrase used for uploading. 47586 47587 Properties: 47588 47589 - Config: passphrase 47590 - Env Var: RCLONE_STORJ_PASSPHRASE 47591 - Provider: new 47592 - Type: string 47593 - Required: false 47594 47595 Advanced options 47596 47597 Here are the Advanced options specific to storj (Storj Decentralized 47598 Cloud Storage). 47599 47600 --storj-description 47601 47602 Description of the remote 47603 47604 Properties: 47605 47606 - Config: description 47607 - Env Var: RCLONE_STORJ_DESCRIPTION 47608 - Type: string 47609 - Required: false 47610 47611 Usage 47612 47613 Paths are specified as remote:bucket (or remote: for the lsf command.) 47614 You may put subdirectories in too, e.g. remote:bucket/path/to/dir. 47615 47616 Once configured you can then use rclone like this. 47617 47618 Create a new bucket 47619 47620 Use the mkdir command to create new bucket, e.g. bucket. 47621 47622 rclone mkdir remote:bucket 47623 47624 List all buckets 47625 47626 Use the lsf command to list all buckets. 47627 47628 rclone lsf remote: 47629 47630 Note the colon (:) character at the end of the command line. 47631 47632 Delete a bucket 47633 47634 Use the rmdir command to delete an empty bucket. 47635 47636 rclone rmdir remote:bucket 47637 47638 Use the purge command to delete a non-empty bucket with all its content. 47639 47640 rclone purge remote:bucket 47641 47642 Upload objects 47643 47644 Use the copy command to upload an object. 47645 47646 rclone copy --progress /home/local/directory/file.ext remote:bucket/path/to/dir/ 47647 47648 The --progress flag is for displaying progress information. Remove it if 47649 you don't need this information. 47650 47651 Use a folder in the local path to upload all its objects. 47652 47653 rclone copy --progress /home/local/directory/ remote:bucket/path/to/dir/ 47654 47655 Only modified files will be copied. 47656 47657 List objects 47658 47659 Use the ls command to list recursively all objects in a bucket. 47660 47661 rclone ls remote:bucket 47662 47663 Add the folder to the remote path to list recursively all objects in 47664 this folder. 47665 47666 rclone ls remote:bucket/path/to/dir/ 47667 47668 Use the lsf command to list non-recursively all objects in a bucket or a 47669 folder. 47670 47671 rclone lsf remote:bucket/path/to/dir/ 47672 47673 Download objects 47674 47675 Use the copy command to download an object. 47676 47677 rclone copy --progress remote:bucket/path/to/dir/file.ext /home/local/directory/ 47678 47679 The --progress flag is for displaying progress information. Remove it if 47680 you don't need this information. 47681 47682 Use a folder in the remote path to download all its objects. 47683 47684 rclone copy --progress remote:bucket/path/to/dir/ /home/local/directory/ 47685 47686 Delete objects 47687 47688 Use the deletefile command to delete a single object. 47689 47690 rclone deletefile remote:bucket/path/to/dir/file.ext 47691 47692 Use the delete command to delete all object in a folder. 47693 47694 rclone delete remote:bucket/path/to/dir/ 47695 47696 Print the total size of objects 47697 47698 Use the size command to print the total size of objects in a bucket or a 47699 folder. 47700 47701 rclone size remote:bucket/path/to/dir/ 47702 47703 Sync two Locations 47704 47705 Use the sync command to sync the source to the destination, changing the 47706 destination only, deleting any excess files. 47707 47708 rclone sync --interactive --progress /home/local/directory/ remote:bucket/path/to/dir/ 47709 47710 The --progress flag is for displaying progress information. Remove it if 47711 you don't need this information. 47712 47713 Since this can cause data loss, test first with the --dry-run flag to 47714 see exactly what would be copied and deleted. 47715 47716 The sync can be done also from Storj to the local file system. 47717 47718 rclone sync --interactive --progress remote:bucket/path/to/dir/ /home/local/directory/ 47719 47720 Or between two Storj buckets. 47721 47722 rclone sync --interactive --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/ 47723 47724 Or even between another cloud storage and Storj. 47725 47726 rclone sync --interactive --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/ 47727 47728 Limitations 47729 47730 rclone about is not supported by the rclone Storj backend. Backends 47731 without this capability cannot determine free space for an rclone mount 47732 or use policy mfs (most free space) as a member of an rclone union 47733 remote. 47734 47735 See List of backends that do not support rclone about and rclone about 47736 47737 Known issues 47738 47739 If you get errors like too many open files this usually happens when the 47740 default ulimit for system max open files is exceeded. Native Storj 47741 protocol opens a large number of TCP connections (each of which is 47742 counted as an open file). For a single upload stream you can expect 110 47743 TCP connections to be opened. For a single download stream you can 47744 expect 35. This batch of connections will be opened for every 64 MiB 47745 segment and you should also expect TCP connections to be reused. If you 47746 do many transfers you eventually open a connection to most storage nodes 47747 (thousands of nodes). 47748 47749 To fix these, please raise your system limits. You can do this issuing a 47750 ulimit -n 65536 just before you run rclone. To change the limits more 47751 permanently you can add this to your shell startup script, e.g. 47752 $HOME/.bashrc, or change the system-wide configuration, usually 47753 /etc/sysctl.conf and/or /etc/security/limits.conf, but please refer to 47754 your operating system manual. 47755 47756 SugarSync 47757 47758 SugarSync is a cloud service that enables active synchronization of 47759 files across computers and other devices for file backup, access, 47760 syncing, and sharing. 47761 47762 Configuration 47763 47764 The initial setup for SugarSync involves getting a token from SugarSync 47765 which you can do with rclone. rclone config walks you through it. 47766 47767 Here is an example of how to make a remote called remote. First run: 47768 47769 rclone config 47770 47771 This will guide you through an interactive setup process: 47772 47773 No remotes found, make a new one? 47774 n) New remote 47775 s) Set configuration password 47776 q) Quit config 47777 n/s/q> n 47778 name> remote 47779 Type of storage to configure. 47780 Enter a string value. Press Enter for the default (""). 47781 Choose a number from below, or type in your own value 47782 [snip] 47783 XX / Sugarsync 47784 \ "sugarsync" 47785 [snip] 47786 Storage> sugarsync 47787 ** See help for sugarsync backend at: https://rclone.org/sugarsync/ ** 47788 47789 Sugarsync App ID. 47790 Leave blank to use rclone's. 47791 Enter a string value. Press Enter for the default (""). 47792 app_id> 47793 Sugarsync Access Key ID. 47794 Leave blank to use rclone's. 47795 Enter a string value. Press Enter for the default (""). 47796 access_key_id> 47797 Sugarsync Private Access Key 47798 Leave blank to use rclone's. 47799 Enter a string value. Press Enter for the default (""). 47800 private_access_key> 47801 Permanently delete files if true 47802 otherwise put them in the deleted files. 47803 Enter a boolean value (true or false). Press Enter for the default ("false"). 47804 hard_delete> 47805 Edit advanced config? (y/n) 47806 y) Yes 47807 n) No (default) 47808 y/n> n 47809 Remote config 47810 Username (email address)> nick@craig-wood.com 47811 Your Sugarsync password is only required during setup and will not be stored. 47812 password: 47813 -------------------- 47814 [remote] 47815 type = sugarsync 47816 refresh_token = https://api.sugarsync.com/app-authorization/XXXXXXXXXXXXXXXXXX 47817 -------------------- 47818 y) Yes this is OK (default) 47819 e) Edit this remote 47820 d) Delete this remote 47821 y/e/d> y 47822 47823 Note that the config asks for your email and password but doesn't store 47824 them, it only uses them to get the initial token. 47825 47826 Once configured you can then use rclone like this, 47827 47828 List directories (sync folders) in top level of your SugarSync 47829 47830 rclone lsd remote: 47831 47832 List all the files in your SugarSync folder "Test" 47833 47834 rclone ls remote:Test 47835 47836 To copy a local directory to an SugarSync folder called backup 47837 47838 rclone copy /home/source remote:backup 47839 47840 Paths are specified as remote:path 47841 47842 Paths may be as deep as required, e.g. remote:directory/subdirectory. 47843 47844 NB you can't create files in the top level folder you have to create a 47845 folder, which rclone will create as a "Sync Folder" with SugarSync. 47846 47847 Modification times and hashes 47848 47849 SugarSync does not support modification times or hashes, therefore 47850 syncing will default to --size-only checking. Note that using --update 47851 will work as rclone can read the time files were uploaded. 47852 47853 Restricted filename characters 47854 47855 SugarSync replaces the default restricted characters set except for DEL. 47856 47857 Invalid UTF-8 bytes will also be replaced, as they can't be used in XML 47858 strings. 47859 47860 Deleting files 47861 47862 Deleted files will be moved to the "Deleted items" folder by default. 47863 47864 However you can supply the flag --sugarsync-hard-delete or set the 47865 config parameter hard_delete = true if you would like files to be 47866 deleted straight away. 47867 47868 Standard options 47869 47870 Here are the Standard options specific to sugarsync (Sugarsync). 47871 47872 --sugarsync-app-id 47873 47874 Sugarsync App ID. 47875 47876 Leave blank to use rclone's. 47877 47878 Properties: 47879 47880 - Config: app_id 47881 - Env Var: RCLONE_SUGARSYNC_APP_ID 47882 - Type: string 47883 - Required: false 47884 47885 --sugarsync-access-key-id 47886 47887 Sugarsync Access Key ID. 47888 47889 Leave blank to use rclone's. 47890 47891 Properties: 47892 47893 - Config: access_key_id 47894 - Env Var: RCLONE_SUGARSYNC_ACCESS_KEY_ID 47895 - Type: string 47896 - Required: false 47897 47898 --sugarsync-private-access-key 47899 47900 Sugarsync Private Access Key. 47901 47902 Leave blank to use rclone's. 47903 47904 Properties: 47905 47906 - Config: private_access_key 47907 - Env Var: RCLONE_SUGARSYNC_PRIVATE_ACCESS_KEY 47908 - Type: string 47909 - Required: false 47910 47911 --sugarsync-hard-delete 47912 47913 Permanently delete files if true otherwise put them in the deleted 47914 files. 47915 47916 Properties: 47917 47918 - Config: hard_delete 47919 - Env Var: RCLONE_SUGARSYNC_HARD_DELETE 47920 - Type: bool 47921 - Default: false 47922 47923 Advanced options 47924 47925 Here are the Advanced options specific to sugarsync (Sugarsync). 47926 47927 --sugarsync-refresh-token 47928 47929 Sugarsync refresh token. 47930 47931 Leave blank normally, will be auto configured by rclone. 47932 47933 Properties: 47934 47935 - Config: refresh_token 47936 - Env Var: RCLONE_SUGARSYNC_REFRESH_TOKEN 47937 - Type: string 47938 - Required: false 47939 47940 --sugarsync-authorization 47941 47942 Sugarsync authorization. 47943 47944 Leave blank normally, will be auto configured by rclone. 47945 47946 Properties: 47947 47948 - Config: authorization 47949 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION 47950 - Type: string 47951 - Required: false 47952 47953 --sugarsync-authorization-expiry 47954 47955 Sugarsync authorization expiry. 47956 47957 Leave blank normally, will be auto configured by rclone. 47958 47959 Properties: 47960 47961 - Config: authorization_expiry 47962 - Env Var: RCLONE_SUGARSYNC_AUTHORIZATION_EXPIRY 47963 - Type: string 47964 - Required: false 47965 47966 --sugarsync-user 47967 47968 Sugarsync user. 47969 47970 Leave blank normally, will be auto configured by rclone. 47971 47972 Properties: 47973 47974 - Config: user 47975 - Env Var: RCLONE_SUGARSYNC_USER 47976 - Type: string 47977 - Required: false 47978 47979 --sugarsync-root-id 47980 47981 Sugarsync root id. 47982 47983 Leave blank normally, will be auto configured by rclone. 47984 47985 Properties: 47986 47987 - Config: root_id 47988 - Env Var: RCLONE_SUGARSYNC_ROOT_ID 47989 - Type: string 47990 - Required: false 47991 47992 --sugarsync-deleted-id 47993 47994 Sugarsync deleted folder id. 47995 47996 Leave blank normally, will be auto configured by rclone. 47997 47998 Properties: 47999 48000 - Config: deleted_id 48001 - Env Var: RCLONE_SUGARSYNC_DELETED_ID 48002 - Type: string 48003 - Required: false 48004 48005 --sugarsync-encoding 48006 48007 The encoding for the backend. 48008 48009 See the encoding section in the overview for more info. 48010 48011 Properties: 48012 48013 - Config: encoding 48014 - Env Var: RCLONE_SUGARSYNC_ENCODING 48015 - Type: Encoding 48016 - Default: Slash,Ctl,InvalidUtf8,Dot 48017 48018 --sugarsync-description 48019 48020 Description of the remote 48021 48022 Properties: 48023 48024 - Config: description 48025 - Env Var: RCLONE_SUGARSYNC_DESCRIPTION 48026 - Type: string 48027 - Required: false 48028 48029 Limitations 48030 48031 rclone about is not supported by the SugarSync backend. Backends without 48032 this capability cannot determine free space for an rclone mount or use 48033 policy mfs (most free space) as a member of an rclone union remote. 48034 48035 See List of backends that do not support rclone about and rclone about 48036 48037 Tardigrade 48038 48039 The Tardigrade backend has been renamed to be the Storj backend. Old 48040 configuration files will continue to work. 48041 48042 Uptobox 48043 48044 This is a Backend for Uptobox file storage service. Uptobox is closer to 48045 a one-click hoster than a traditional cloud storage provider and 48046 therefore not suitable for long term storage. 48047 48048 Paths are specified as remote:path 48049 48050 Paths may be as deep as required, e.g. remote:directory/subdirectory. 48051 48052 Configuration 48053 48054 To configure an Uptobox backend you'll need your personal api token. 48055 You'll find it in your account settings 48056 48057 Here is an example of how to make a remote called remote with the 48058 default setup. First run: 48059 48060 rclone config 48061 48062 This will guide you through an interactive setup process: 48063 48064 Current remotes: 48065 48066 Name Type 48067 ==== ==== 48068 TestUptobox uptobox 48069 48070 e) Edit existing remote 48071 n) New remote 48072 d) Delete remote 48073 r) Rename remote 48074 c) Copy remote 48075 s) Set configuration password 48076 q) Quit config 48077 e/n/d/r/c/s/q> n 48078 name> uptobox 48079 Type of storage to configure. 48080 Enter a string value. Press Enter for the default (""). 48081 Choose a number from below, or type in your own value 48082 [...] 48083 37 / Uptobox 48084 \ "uptobox" 48085 [...] 48086 Storage> uptobox 48087 ** See help for uptobox backend at: https://rclone.org/uptobox/ ** 48088 48089 Your API Key, get it from https://uptobox.com/my_account 48090 Enter a string value. Press Enter for the default (""). 48091 api_key> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 48092 Edit advanced config? (y/n) 48093 y) Yes 48094 n) No (default) 48095 y/n> n 48096 Remote config 48097 -------------------- 48098 [uptobox] 48099 type = uptobox 48100 api_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 48101 -------------------- 48102 y) Yes this is OK (default) 48103 e) Edit this remote 48104 d) Delete this remote 48105 y/e/d> 48106 48107 Once configured you can then use rclone like this, 48108 48109 List directories in top level of your Uptobox 48110 48111 rclone lsd remote: 48112 48113 List all the files in your Uptobox 48114 48115 rclone ls remote: 48116 48117 To copy a local directory to an Uptobox directory called backup 48118 48119 rclone copy /home/source remote:backup 48120 48121 Modification times and hashes 48122 48123 Uptobox supports neither modified times nor checksums. All timestamps 48124 will read as that set by --default-time. 48125 48126 Restricted filename characters 48127 48128 In addition to the default restricted characters set the following 48129 characters are also replaced: 48130 48131 Character Value Replacement 48132 ----------- ------- ------------- 48133 " 0x22 " 48134 ` 0x41 ` 48135 48136 Invalid UTF-8 bytes will also be replaced, as they can't be used in XML 48137 strings. 48138 48139 Standard options 48140 48141 Here are the Standard options specific to uptobox (Uptobox). 48142 48143 --uptobox-access-token 48144 48145 Your access token. 48146 48147 Get it from https://uptobox.com/my_account. 48148 48149 Properties: 48150 48151 - Config: access_token 48152 - Env Var: RCLONE_UPTOBOX_ACCESS_TOKEN 48153 - Type: string 48154 - Required: false 48155 48156 Advanced options 48157 48158 Here are the Advanced options specific to uptobox (Uptobox). 48159 48160 --uptobox-private 48161 48162 Set to make uploaded files private 48163 48164 Properties: 48165 48166 - Config: private 48167 - Env Var: RCLONE_UPTOBOX_PRIVATE 48168 - Type: bool 48169 - Default: false 48170 48171 --uptobox-encoding 48172 48173 The encoding for the backend. 48174 48175 See the encoding section in the overview for more info. 48176 48177 Properties: 48178 48179 - Config: encoding 48180 - Env Var: RCLONE_UPTOBOX_ENCODING 48181 - Type: Encoding 48182 - Default: 48183 Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot 48184 48185 --uptobox-description 48186 48187 Description of the remote 48188 48189 Properties: 48190 48191 - Config: description 48192 - Env Var: RCLONE_UPTOBOX_DESCRIPTION 48193 - Type: string 48194 - Required: false 48195 48196 Limitations 48197 48198 Uptobox will delete inactive files that have not been accessed in 60 48199 days. 48200 48201 rclone about is not supported by this backend an overview of used space 48202 can however been seen in the uptobox web interface. 48203 48204 Union 48205 48206 The union backend joins several remotes together to make a single 48207 unified view of them. 48208 48209 During the initial setup with rclone config you will specify the 48210 upstream remotes as a space separated list. The upstream remotes can 48211 either be a local paths or other remotes. 48212 48213 The attributes :ro, :nc and :writeback can be attached to the end of the 48214 remote to tag the remote as read only, no create or writeback, e.g. 48215 remote:directory/subdirectory:ro or remote:directory/subdirectory:nc. 48216 48217 - :ro means files will only be read from here and never written 48218 - :nc means new files or directories won't be created here 48219 - :writeback means files found in different remotes will be written 48220 back here. See the writeback section for more info. 48221 48222 Subfolders can be used in upstream remotes. Assume a union remote named 48223 backup with the remotes mydrive:private/backup. Invoking 48224 rclone mkdir backup:desktop is exactly the same as invoking 48225 rclone mkdir mydrive:private/backup/desktop. 48226 48227 There is no special handling of paths containing .. segments. Invoking 48228 rclone mkdir backup:../desktop is exactly the same as invoking 48229 rclone mkdir mydrive:private/backup/../desktop. 48230 48231 Configuration 48232 48233 Here is an example of how to make a union called remote for local 48234 folders. First run: 48235 48236 rclone config 48237 48238 This will guide you through an interactive setup process: 48239 48240 No remotes found, make a new one? 48241 n) New remote 48242 s) Set configuration password 48243 q) Quit config 48244 n/s/q> n 48245 name> remote 48246 Type of storage to configure. 48247 Choose a number from below, or type in your own value 48248 [snip] 48249 XX / Union merges the contents of several remotes 48250 \ "union" 48251 [snip] 48252 Storage> union 48253 List of space separated upstreams. 48254 Can be 'upstreama:test/dir upstreamb:', '\"upstreama:test/space:ro dir\" upstreamb:', etc. 48255 Enter a string value. Press Enter for the default (""). 48256 upstreams> remote1:dir1 remote2:dir2 remote3:dir3 48257 Policy to choose upstream on ACTION class. 48258 Enter a string value. Press Enter for the default ("epall"). 48259 action_policy> 48260 Policy to choose upstream on CREATE class. 48261 Enter a string value. Press Enter for the default ("epmfs"). 48262 create_policy> 48263 Policy to choose upstream on SEARCH class. 48264 Enter a string value. Press Enter for the default ("ff"). 48265 search_policy> 48266 Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used. 48267 Enter a signed integer. Press Enter for the default ("120"). 48268 cache_time> 48269 Remote config 48270 -------------------- 48271 [remote] 48272 type = union 48273 upstreams = remote1:dir1 remote2:dir2 remote3:dir3 48274 -------------------- 48275 y) Yes this is OK 48276 e) Edit this remote 48277 d) Delete this remote 48278 y/e/d> y 48279 Current remotes: 48280 48281 Name Type 48282 ==== ==== 48283 remote union 48284 48285 e) Edit existing remote 48286 n) New remote 48287 d) Delete remote 48288 r) Rename remote 48289 c) Copy remote 48290 s) Set configuration password 48291 q) Quit config 48292 e/n/d/r/c/s/q> q 48293 48294 Once configured you can then use rclone like this, 48295 48296 List directories in top level in remote1:dir1, remote2:dir2 and 48297 remote3:dir3 48298 48299 rclone lsd remote: 48300 48301 List all the files in remote1:dir1, remote2:dir2 and remote3:dir3 48302 48303 rclone ls remote: 48304 48305 Copy another local directory to the union directory called source, which 48306 will be placed into remote3:dir3 48307 48308 rclone copy C:\source remote:source 48309 48310 Behavior / Policies 48311 48312 The behavior of union backend is inspired by trapexit/mergerfs. All 48313 functions are grouped into 3 categories: action, create and search. 48314 These functions and categories can be assigned a policy which dictates 48315 what file or directory is chosen when performing that behavior. Any 48316 policy can be assigned to a function or category though some may not be 48317 very useful in practice. For instance: rand (random) may be useful for 48318 file creation (create) but could lead to very odd behavior if used for 48319 delete if there were more than one copy of the file. 48320 48321 Function / Category classifications 48322 48323 ---------------------------------------------------------------------------- 48324 Category Description Functions 48325 ---------- --------------- ------------------------------------------------- 48326 action Writing move, rmdir, rmdirs, delete, purge and copy, sync 48327 Existing file (as destination when file exist) 48328 48329 create Create copy, sync (as destination when file not exist) 48330 non-existing 48331 file 48332 48333 search Reading and ls, lsd, lsl, cat, md5sum, sha1sum and copy, sync 48334 listing file (as source) 48335 48336 N/A size, about 48337 ---------------------------------------------------------------------------- 48338 48339 Path Preservation 48340 48341 Policies, as described below, are of two basic types. path preserving 48342 and non-path preserving. 48343 48344 All policies which start with ep (epff, eplfs, eplus, epmfs, eprand) are 48345 path preserving. ep stands for existing path. 48346 48347 A path preserving policy will only consider upstreams where the relative 48348 path being accessed already exists. 48349 48350 When using non-path preserving policies paths will be created in target 48351 upstreams as necessary. 48352 48353 Quota Relevant Policies 48354 48355 Some policies rely on quota information. These policies should be used 48356 only if your upstreams support the respective quota fields. 48357 48358 Policy Required Field 48359 ------------ ---------------- 48360 lfs, eplfs Free 48361 mfs, epmfs Free 48362 lus, eplus Used 48363 lno, eplno Objects 48364 48365 To check if your upstream supports the field, run 48366 rclone about remote: [flags] and see if the required field exists. 48367 48368 Filters 48369 48370 Policies basically search upstream remotes and create a list of files / 48371 paths for functions to work on. The policy is responsible for filtering 48372 and sorting. The policy type defines the sorting but filtering is mostly 48373 uniform as described below. 48374 48375 - No search policies filter. 48376 - All action policies will filter out remotes which are tagged as 48377 read-only. 48378 - All create policies will filter out remotes which are tagged 48379 read-only or no-create. 48380 48381 If all remotes are filtered an error will be returned. 48382 48383 Policy descriptions 48384 48385 The policies definition are inspired by trapexit/mergerfs but not 48386 exactly the same. Some policy definition could be different due to the 48387 much larger latency of remote file systems. 48388 48389 ----------------------------------------------------------------------- 48390 Policy Description 48391 ---------------- ------------------------------------------------------ 48392 all Search category: same as epall. Action category: same 48393 as epall. Create category: act on all upstreams. 48394 48395 epall (existing Search category: Given this order configured, act on 48396 path, all) the first one found where the relative path exists. 48397 Action category: apply to all found. Create category: 48398 act on all upstreams where the relative path exists. 48399 48400 epff (existing Act on the first one found, by the time upstreams 48401 path, first reply, where the relative path exists. 48402 found) 48403 48404 eplfs (existing Of all the upstreams on which the relative path exists 48405 path, least free choose the one with the least free space. 48406 space) 48407 48408 eplus (existing Of all the upstreams on which the relative path exists 48409 path, least used choose the one with the least used space. 48410 space) 48411 48412 eplno (existing Of all the upstreams on which the relative path exists 48413 path, least choose the one with the least number of objects. 48414 number of 48415 objects) 48416 48417 epmfs (existing Of all the upstreams on which the relative path exists 48418 path, most free choose the one with the most free space. 48419 space) 48420 48421 eprand (existing Calls epall and then randomizes. Returns only one 48422 path, random) upstream. 48423 48424 ff (first found) Search category: same as epff. Action category: same 48425 as epff. Create category: Act on the first one found 48426 by the time upstreams reply. 48427 48428 lfs (least free Search category: same as eplfs. Action category: same 48429 space) as eplfs. Create category: Pick the upstream with the 48430 least available free space. 48431 48432 lus (least used Search category: same as eplus. Action category: same 48433 space) as eplus. Create category: Pick the upstream with the 48434 least used space. 48435 48436 lno (least Search category: same as eplno. Action category: same 48437 number of as eplno. Create category: Pick the upstream with the 48438 objects) least number of objects. 48439 48440 mfs (most free Search category: same as epmfs. Action category: same 48441 space) as epmfs. Create category: Pick the upstream with the 48442 most available free space. 48443 48444 newest Pick the file / directory with the largest mtime. 48445 48446 rand (random) Calls all and then randomizes. Returns only one 48447 upstream. 48448 ----------------------------------------------------------------------- 48449 48450 Writeback 48451 48452 The tag :writeback on an upstream remote can be used to make a simple 48453 cache system like this: 48454 48455 [union] 48456 type = union 48457 action_policy = all 48458 create_policy = all 48459 search_policy = ff 48460 upstreams = /local:writeback remote:dir 48461 48462 When files are opened for read, if the file is in remote:dir but not 48463 /local then rclone will copy the file entirely into /local before 48464 returning a reference to the file in /local. The copy will be done with 48465 the equivalent of rclone copy so will use --multi-thread-streams if 48466 configured. Any copies will be logged with an INFO log. 48467 48468 When files are written, they will be written to both remote:dir and 48469 /local. 48470 48471 As many remotes as desired can be added to upstreams but there should 48472 only be one :writeback tag. 48473 48474 Rclone does not manage the :writeback remote in any way other than 48475 writing files back to it. So if you need to expire old files or manage 48476 the size then you will have to do this yourself. 48477 48478 Standard options 48479 48480 Here are the Standard options specific to union (Union merges the 48481 contents of several upstream fs). 48482 48483 --union-upstreams 48484 48485 List of space separated upstreams. 48486 48487 Can be 'upstreama:test/dir upstreamb:', '"upstreama:test/space:ro dir" 48488 upstreamb:', etc. 48489 48490 Properties: 48491 48492 - Config: upstreams 48493 - Env Var: RCLONE_UNION_UPSTREAMS 48494 - Type: string 48495 - Required: true 48496 48497 --union-action-policy 48498 48499 Policy to choose upstream on ACTION category. 48500 48501 Properties: 48502 48503 - Config: action_policy 48504 - Env Var: RCLONE_UNION_ACTION_POLICY 48505 - Type: string 48506 - Default: "epall" 48507 48508 --union-create-policy 48509 48510 Policy to choose upstream on CREATE category. 48511 48512 Properties: 48513 48514 - Config: create_policy 48515 - Env Var: RCLONE_UNION_CREATE_POLICY 48516 - Type: string 48517 - Default: "epmfs" 48518 48519 --union-search-policy 48520 48521 Policy to choose upstream on SEARCH category. 48522 48523 Properties: 48524 48525 - Config: search_policy 48526 - Env Var: RCLONE_UNION_SEARCH_POLICY 48527 - Type: string 48528 - Default: "ff" 48529 48530 --union-cache-time 48531 48532 Cache time of usage and free space (in seconds). 48533 48534 This option is only useful when a path preserving policy is used. 48535 48536 Properties: 48537 48538 - Config: cache_time 48539 - Env Var: RCLONE_UNION_CACHE_TIME 48540 - Type: int 48541 - Default: 120 48542 48543 Advanced options 48544 48545 Here are the Advanced options specific to union (Union merges the 48546 contents of several upstream fs). 48547 48548 --union-min-free-space 48549 48550 Minimum viable free space for lfs/eplfs policies. 48551 48552 If a remote has less than this much free space then it won't be 48553 considered for use in lfs or eplfs policies. 48554 48555 Properties: 48556 48557 - Config: min_free_space 48558 - Env Var: RCLONE_UNION_MIN_FREE_SPACE 48559 - Type: SizeSuffix 48560 - Default: 1Gi 48561 48562 --union-description 48563 48564 Description of the remote 48565 48566 Properties: 48567 48568 - Config: description 48569 - Env Var: RCLONE_UNION_DESCRIPTION 48570 - Type: string 48571 - Required: false 48572 48573 Metadata 48574 48575 Any metadata supported by the underlying remote is read and written. 48576 48577 See the metadata docs for more info. 48578 48579 WebDAV 48580 48581 Paths are specified as remote:path 48582 48583 Paths may be as deep as required, e.g. remote:directory/subdirectory. 48584 48585 Configuration 48586 48587 To configure the WebDAV remote you will need to have a URL for it, and a 48588 username and password. If you know what kind of system you are 48589 connecting to then rclone can enable extra features. 48590 48591 Here is an example of how to make a remote called remote. First run: 48592 48593 rclone config 48594 48595 This will guide you through an interactive setup process: 48596 48597 No remotes found, make a new one? 48598 n) New remote 48599 s) Set configuration password 48600 q) Quit config 48601 n/s/q> n 48602 name> remote 48603 Type of storage to configure. 48604 Choose a number from below, or type in your own value 48605 [snip] 48606 XX / WebDAV 48607 \ "webdav" 48608 [snip] 48609 Storage> webdav 48610 URL of http host to connect to 48611 Choose a number from below, or type in your own value 48612 1 / Connect to example.com 48613 \ "https://example.com" 48614 url> https://example.com/remote.php/webdav/ 48615 Name of the WebDAV site/service/software you are using 48616 Choose a number from below, or type in your own value 48617 1 / Fastmail Files 48618 \ (fastmail) 48619 2 / Nextcloud 48620 \ (nextcloud) 48621 3 / Owncloud 48622 \ (owncloud) 48623 4 / Sharepoint Online, authenticated by Microsoft account 48624 \ (sharepoint) 48625 5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises 48626 \ (sharepoint-ntlm) 48627 6 / rclone WebDAV server to serve a remote over HTTP via the WebDAV protocol 48628 \ (rclone) 48629 7 / Other site/service or software 48630 \ (other) 48631 vendor> 2 48632 User name 48633 user> user 48634 Password. 48635 y) Yes type in my own password 48636 g) Generate random password 48637 n) No leave this optional password blank 48638 y/g/n> y 48639 Enter the password: 48640 password: 48641 Confirm the password: 48642 password: 48643 Bearer token instead of user/pass (e.g. a Macaroon) 48644 bearer_token> 48645 Remote config 48646 -------------------- 48647 [remote] 48648 type = webdav 48649 url = https://example.com/remote.php/webdav/ 48650 vendor = nextcloud 48651 user = user 48652 pass = *** ENCRYPTED *** 48653 bearer_token = 48654 -------------------- 48655 y) Yes this is OK 48656 e) Edit this remote 48657 d) Delete this remote 48658 y/e/d> y 48659 48660 Once configured you can then use rclone like this, 48661 48662 List directories in top level of your WebDAV 48663 48664 rclone lsd remote: 48665 48666 List all the files in your WebDAV 48667 48668 rclone ls remote: 48669 48670 To copy a local directory to an WebDAV directory called backup 48671 48672 rclone copy /home/source remote:backup 48673 48674 Modification times and hashes 48675 48676 Plain WebDAV does not support modified times. However when used with 48677 Fastmail Files, Owncloud or Nextcloud rclone will support modified 48678 times. 48679 48680 Likewise plain WebDAV does not support hashes, however when used with 48681 Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 48682 hashes. Depending on the exact version of Owncloud or Nextcloud hashes 48683 may appear on all objects, or only on objects which had a hash uploaded 48684 with them. 48685 48686 Standard options 48687 48688 Here are the Standard options specific to webdav (WebDAV). 48689 48690 --webdav-url 48691 48692 URL of http host to connect to. 48693 48694 E.g. https://example.com. 48695 48696 Properties: 48697 48698 - Config: url 48699 - Env Var: RCLONE_WEBDAV_URL 48700 - Type: string 48701 - Required: true 48702 48703 --webdav-vendor 48704 48705 Name of the WebDAV site/service/software you are using. 48706 48707 Properties: 48708 48709 - Config: vendor 48710 - Env Var: RCLONE_WEBDAV_VENDOR 48711 - Type: string 48712 - Required: false 48713 - Examples: 48714 - "fastmail" 48715 - Fastmail Files 48716 - "nextcloud" 48717 - Nextcloud 48718 - "owncloud" 48719 - Owncloud 48720 - "sharepoint" 48721 - Sharepoint Online, authenticated by Microsoft account 48722 - "sharepoint-ntlm" 48723 - Sharepoint with NTLM authentication, usually self-hosted or 48724 on-premises 48725 - "rclone" 48726 - rclone WebDAV server to serve a remote over HTTP via the 48727 WebDAV protocol 48728 - "other" 48729 - Other site/service or software 48730 48731 --webdav-user 48732 48733 User name. 48734 48735 In case NTLM authentication is used, the username should be in the 48736 format 'Domain'. 48737 48738 Properties: 48739 48740 - Config: user 48741 - Env Var: RCLONE_WEBDAV_USER 48742 - Type: string 48743 - Required: false 48744 48745 --webdav-pass 48746 48747 Password. 48748 48749 NB Input to this must be obscured - see rclone obscure. 48750 48751 Properties: 48752 48753 - Config: pass 48754 - Env Var: RCLONE_WEBDAV_PASS 48755 - Type: string 48756 - Required: false 48757 48758 --webdav-bearer-token 48759 48760 Bearer token instead of user/pass (e.g. a Macaroon). 48761 48762 Properties: 48763 48764 - Config: bearer_token 48765 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN 48766 - Type: string 48767 - Required: false 48768 48769 Advanced options 48770 48771 Here are the Advanced options specific to webdav (WebDAV). 48772 48773 --webdav-bearer-token-command 48774 48775 Command to run to get a bearer token. 48776 48777 Properties: 48778 48779 - Config: bearer_token_command 48780 - Env Var: RCLONE_WEBDAV_BEARER_TOKEN_COMMAND 48781 - Type: string 48782 - Required: false 48783 48784 --webdav-encoding 48785 48786 The encoding for the backend. 48787 48788 See the encoding section in the overview for more info. 48789 48790 Default encoding is 48791 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8 48792 for sharepoint-ntlm or identity otherwise. 48793 48794 Properties: 48795 48796 - Config: encoding 48797 - Env Var: RCLONE_WEBDAV_ENCODING 48798 - Type: string 48799 - Required: false 48800 48801 --webdav-headers 48802 48803 Set HTTP headers for all transactions. 48804 48805 Use this to set additional HTTP headers for all transactions 48806 48807 The input format is comma separated list of key,value pairs. Standard 48808 CSV encoding may be used. 48809 48810 For example, to set a Cookie use 'Cookie,name=value', or 48811 '"Cookie","name=value"'. 48812 48813 You can set multiple headers, e.g. 48814 '"Cookie","name=value","Authorization","xxx"'. 48815 48816 Properties: 48817 48818 - Config: headers 48819 - Env Var: RCLONE_WEBDAV_HEADERS 48820 - Type: CommaSepList 48821 - Default: 48822 48823 --webdav-pacer-min-sleep 48824 48825 Minimum time to sleep between API calls. 48826 48827 Properties: 48828 48829 - Config: pacer_min_sleep 48830 - Env Var: RCLONE_WEBDAV_PACER_MIN_SLEEP 48831 - Type: Duration 48832 - Default: 10ms 48833 48834 --webdav-nextcloud-chunk-size 48835 48836 Nextcloud upload chunk size. 48837 48838 We recommend configuring your NextCloud instance to increase the max 48839 chunk size to 1 GB for better upload performances. See 48840 https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html#adjust-chunk-size-on-nextcloud-side 48841 48842 Set to 0 to disable chunked uploading. 48843 48844 Properties: 48845 48846 - Config: nextcloud_chunk_size 48847 - Env Var: RCLONE_WEBDAV_NEXTCLOUD_CHUNK_SIZE 48848 - Type: SizeSuffix 48849 - Default: 10Mi 48850 48851 --webdav-owncloud-exclude-shares 48852 48853 Exclude ownCloud shares 48854 48855 Properties: 48856 48857 - Config: owncloud_exclude_shares 48858 - Env Var: RCLONE_WEBDAV_OWNCLOUD_EXCLUDE_SHARES 48859 - Type: bool 48860 - Default: false 48861 48862 --webdav-description 48863 48864 Description of the remote 48865 48866 Properties: 48867 48868 - Config: description 48869 - Env Var: RCLONE_WEBDAV_DESCRIPTION 48870 - Type: string 48871 - Required: false 48872 48873 Provider notes 48874 48875 See below for notes on specific providers. 48876 48877 Fastmail Files 48878 48879 Use https://webdav.fastmail.com/ or a subdirectory as the URL, and your 48880 Fastmail email username@domain.tld as the username. Follow this 48881 documentation to create an app password with access to Files (WebDAV) 48882 and use this as the password. 48883 48884 Fastmail supports modified times using the X-OC-Mtime header. 48885 48886 Owncloud 48887 48888 Click on the settings cog in the bottom right of the page and this will 48889 show the WebDAV URL that rclone needs in the config step. It will look 48890 something like https://example.com/remote.php/webdav/. 48891 48892 Owncloud supports modified times using the X-OC-Mtime header. 48893 48894 Nextcloud 48895 48896 This is configured in an identical way to Owncloud. Note that Nextcloud 48897 initially did not support streaming of files (rcat) whereas Owncloud 48898 did, but this seems to be fixed as of 2020-11-27 (tested with rclone 48899 v1.53.1 and Nextcloud Server v19). 48900 48901 Sharepoint Online 48902 48903 Rclone can be used with Sharepoint provided by OneDrive for Business or 48904 Office365 Education Accounts. This feature is only needed for a few of 48905 these Accounts, mostly Office365 Education ones. These accounts are 48906 sometimes not verified by the domain owner github#1975 48907 48908 This means that these accounts can't be added using the official API 48909 (other Accounts should work with the "onedrive" option). However, it is 48910 possible to access them using webdav. 48911 48912 To use a sharepoint remote with rclone, add it like this: First, you 48913 need to get your remote's URL: 48914 48915 - Go here to open your OneDrive or to sign in 48916 - Now take a look at your address bar, the URL should look like this: 48917 https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/_layouts/15/onedrive.aspx 48918 48919 You'll only need this URL up to the email address. After that, you'll 48920 most likely want to add "/Documents". That subdirectory contains the 48921 actual data stored on your OneDrive. 48922 48923 Add the remote to rclone like this: Configure the url as 48924 https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents 48925 and use your normal account email and password for user and pass. If you 48926 have 2FA enabled, you have to generate an app password. Set the vendor 48927 to sharepoint. 48928 48929 Your config file should look like this: 48930 48931 [sharepoint] 48932 type = webdav 48933 url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents 48934 vendor = sharepoint 48935 user = YourEmailAddress 48936 pass = encryptedpassword 48937 48938 Sharepoint with NTLM Authentication 48939 48940 Use this option in case your (hosted) Sharepoint is not tied to OneDrive 48941 accounts and uses NTLM authentication. 48942 48943 To get the url configuration, similarly to the above, first navigate to 48944 the desired directory in your browser to get the URL, then strip 48945 everything after the name of the opened directory. 48946 48947 Example: If the URL is: 48948 https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx 48949 48950 The configuration to use would be: 48951 https://example.sharepoint.com/sites/12345/Documents 48952 48953 Set the vendor to sharepoint-ntlm. 48954 48955 NTLM uses domain and user name combination for authentication, set user 48956 to DOMAIN\username. 48957 48958 Your config file should look like this: 48959 48960 [sharepoint] 48961 type = webdav 48962 url = https://[YOUR-DOMAIN]/some-path-to/Documents 48963 vendor = sharepoint-ntlm 48964 user = DOMAIN\user 48965 pass = encryptedpassword 48966 48967 Required Flags for SharePoint 48968 48969 As SharePoint does some special things with uploaded documents, you 48970 won't be able to use the documents size or the documents hash to compare 48971 if a file has been changed since the upload / which file is newer. 48972 48973 For Rclone calls copying files (especially Office files such as .docx, 48974 .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should 48975 append these flags to ensure Rclone uses the "Last Modified" datetime 48976 property to compare your documents: 48977 48978 --ignore-size --ignore-checksum --update 48979 48980 Rclone 48981 48982 Use this option if you are hosting remotes over WebDAV provided by 48983 rclone. Read rclone serve webdav for more details. 48984 48985 rclone serve supports modified times using the X-OC-Mtime header. 48986 48987 dCache 48988 48989 dCache is a storage system that supports many protocols and 48990 authentication/authorisation schemes. For WebDAV clients, it allows 48991 users to authenticate with username and password (BASIC), X.509, 48992 Kerberos, and various bearer tokens, including Macaroons and 48993 OpenID-Connect access tokens. 48994 48995 Configure as normal using the other type. Don't enter a username or 48996 password, instead enter your Macaroon as the bearer_token. 48997 48998 The config will end up looking something like this. 48999 49000 [dcache] 49001 type = webdav 49002 url = https://dcache... 49003 vendor = other 49004 user = 49005 pass = 49006 bearer_token = your-macaroon 49007 49008 There is a script that obtains a Macaroon from a dCache WebDAV endpoint, 49009 and creates an rclone config file. 49010 49011 Macaroons may also be obtained from the dCacheView 49012 web-browser/JavaScript client that comes with dCache. 49013 49014 OpenID-Connect 49015 49016 dCache also supports authenticating with OpenID-Connect access tokens. 49017 OpenID-Connect is a protocol (based on OAuth 2.0) that allows services 49018 to identify users who have authenticated with some central service. 49019 49020 Support for OpenID-Connect in rclone is currently achieved using another 49021 software package called oidc-agent. This is a command-line tool that 49022 facilitates obtaining an access token. Once installed and configured, an 49023 access token is obtained by running the oidc-token command. The 49024 following example shows a (shortened) access token obtained from the XDC 49025 OIDC Provider. 49026 49027 paul@celebrimbor:~$ oidc-token XDC 49028 eyJraWQ[...]QFXDt0 49029 paul@celebrimbor:~$ 49030 49031 Note Before the oidc-token command will work, the refresh token must be 49032 loaded into the oidc agent. This is done with the oidc-add command 49033 (e.g., oidc-add XDC). This is typically done once per login session. 49034 Full details on this and how to register oidc-agent with your OIDC 49035 Provider are provided in the oidc-agent documentation. 49036 49037 The rclone bearer_token_command configuration option is used to fetch 49038 the access token from oidc-agent. 49039 49040 Configure as a normal WebDAV endpoint, using the 'other' vendor, leaving 49041 the username and password empty. When prompted, choose to edit the 49042 advanced config and enter the command to get a bearer token (e.g., 49043 oidc-agent XDC). 49044 49045 The following example config shows a WebDAV endpoint that uses 49046 oidc-agent to supply an access token from the XDC OIDC Provider. 49047 49048 [dcache] 49049 type = webdav 49050 url = https://dcache.example.org/ 49051 vendor = other 49052 bearer_token_command = oidc-token XDC 49053 49054 Yandex Disk 49055 49056 Yandex Disk is a cloud storage solution created by Yandex. 49057 49058 Configuration 49059 49060 Here is an example of making a yandex configuration. First run 49061 49062 rclone config 49063 49064 This will guide you through an interactive setup process: 49065 49066 No remotes found, make a new one? 49067 n) New remote 49068 s) Set configuration password 49069 n/s> n 49070 name> remote 49071 Type of storage to configure. 49072 Choose a number from below, or type in your own value 49073 [snip] 49074 XX / Yandex Disk 49075 \ "yandex" 49076 [snip] 49077 Storage> yandex 49078 Yandex Client Id - leave blank normally. 49079 client_id> 49080 Yandex Client Secret - leave blank normally. 49081 client_secret> 49082 Remote config 49083 Use web browser to automatically authenticate rclone with remote? 49084 * Say Y if the machine running rclone has a web browser you can use 49085 * Say N if running rclone on a (remote) machine without web browser access 49086 If not sure try Y. If Y failed, try N. 49087 y) Yes 49088 n) No 49089 y/n> y 49090 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth 49091 Log in and authorize rclone for access 49092 Waiting for code... 49093 Got code 49094 -------------------- 49095 [remote] 49096 client_id = 49097 client_secret = 49098 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"OAuth","expiry":"2016-12-29T12:27:11.362788025Z"} 49099 -------------------- 49100 y) Yes this is OK 49101 e) Edit this remote 49102 d) Delete this remote 49103 y/e/d> y 49104 49105 See the remote setup docs for how to set it up on a machine with no 49106 Internet browser available. 49107 49108 Note that rclone runs a webserver on your local machine to collect the 49109 token as returned from Yandex Disk. This only runs from the moment it 49110 opens your browser to the moment you get back the verification code. 49111 This is on http://127.0.0.1:53682/ and this it may require you to 49112 unblock it temporarily if you are running a host firewall. 49113 49114 Once configured you can then use rclone like this, 49115 49116 See top level directories 49117 49118 rclone lsd remote: 49119 49120 Make a new directory 49121 49122 rclone mkdir remote:directory 49123 49124 List the contents of a directory 49125 49126 rclone ls remote:directory 49127 49128 Sync /home/local/directory to the remote path, deleting any excess files 49129 in the path. 49130 49131 rclone sync --interactive /home/local/directory remote:directory 49132 49133 Yandex paths may be as deep as required, e.g. 49134 remote:directory/subdirectory. 49135 49136 Modification times and hashes 49137 49138 Modified times are supported and are stored accurate to 1 ns in custom 49139 metadata called rclone_modified in RFC3339 with nanoseconds format. 49140 49141 The MD5 hash algorithm is natively supported by Yandex Disk. 49142 49143 Emptying Trash 49144 49145 If you wish to empty your trash you can use the rclone cleanup remote: 49146 command which will permanently delete all your trashed files. This 49147 command does not take any path arguments. 49148 49149 Quota information 49150 49151 To view your current quota you can use the rclone about remote: command 49152 which will display your usage limit (quota) and the current usage. 49153 49154 Restricted filename characters 49155 49156 The default restricted characters set are replaced. 49157 49158 Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON 49159 strings. 49160 49161 Standard options 49162 49163 Here are the Standard options specific to yandex (Yandex Disk). 49164 49165 --yandex-client-id 49166 49167 OAuth Client Id. 49168 49169 Leave blank normally. 49170 49171 Properties: 49172 49173 - Config: client_id 49174 - Env Var: RCLONE_YANDEX_CLIENT_ID 49175 - Type: string 49176 - Required: false 49177 49178 --yandex-client-secret 49179 49180 OAuth Client Secret. 49181 49182 Leave blank normally. 49183 49184 Properties: 49185 49186 - Config: client_secret 49187 - Env Var: RCLONE_YANDEX_CLIENT_SECRET 49188 - Type: string 49189 - Required: false 49190 49191 Advanced options 49192 49193 Here are the Advanced options specific to yandex (Yandex Disk). 49194 49195 --yandex-token 49196 49197 OAuth Access Token as a JSON blob. 49198 49199 Properties: 49200 49201 - Config: token 49202 - Env Var: RCLONE_YANDEX_TOKEN 49203 - Type: string 49204 - Required: false 49205 49206 --yandex-auth-url 49207 49208 Auth server URL. 49209 49210 Leave blank to use the provider defaults. 49211 49212 Properties: 49213 49214 - Config: auth_url 49215 - Env Var: RCLONE_YANDEX_AUTH_URL 49216 - Type: string 49217 - Required: false 49218 49219 --yandex-token-url 49220 49221 Token server url. 49222 49223 Leave blank to use the provider defaults. 49224 49225 Properties: 49226 49227 - Config: token_url 49228 - Env Var: RCLONE_YANDEX_TOKEN_URL 49229 - Type: string 49230 - Required: false 49231 49232 --yandex-hard-delete 49233 49234 Delete files permanently rather than putting them into the trash. 49235 49236 Properties: 49237 49238 - Config: hard_delete 49239 - Env Var: RCLONE_YANDEX_HARD_DELETE 49240 - Type: bool 49241 - Default: false 49242 49243 --yandex-encoding 49244 49245 The encoding for the backend. 49246 49247 See the encoding section in the overview for more info. 49248 49249 Properties: 49250 49251 - Config: encoding 49252 - Env Var: RCLONE_YANDEX_ENCODING 49253 - Type: Encoding 49254 - Default: Slash,Del,Ctl,InvalidUtf8,Dot 49255 49256 --yandex-description 49257 49258 Description of the remote 49259 49260 Properties: 49261 49262 - Config: description 49263 - Env Var: RCLONE_YANDEX_DESCRIPTION 49264 - Type: string 49265 - Required: false 49266 49267 Limitations 49268 49269 When uploading very large files (bigger than about 5 GiB) you will need 49270 to increase the --timeout parameter. This is because Yandex pauses 49271 (perhaps to calculate the MD5SUM for the entire file) before returning 49272 confirmation that the file has been uploaded. The default handling of 49273 timeouts in rclone is to assume a 5 minute pause is an error and close 49274 the connection - you'll see net/http: timeout awaiting response headers 49275 errors in the logs if this is happening. Setting the timeout to twice 49276 the max size of file in GiB should be enough, so if you want to upload a 49277 30 GiB file set a timeout of 2 * 30 = 60m, that is --timeout 60m. 49278 49279 Having a Yandex Mail account is mandatory to use the Yandex.Disk 49280 subscription. Token generation will work without a mail account, but 49281 Rclone won't be able to complete any actions. 49282 49283 [403 - DiskUnsupportedUserAccountTypeError] User account type is not supported. 49284 49285 Zoho Workdrive 49286 49287 Zoho WorkDrive is a cloud storage solution created by Zoho. 49288 49289 Configuration 49290 49291 Here is an example of making a zoho configuration. First run 49292 49293 rclone config 49294 49295 This will guide you through an interactive setup process: 49296 49297 No remotes found, make a new one? 49298 n) New remote 49299 s) Set configuration password 49300 n/s> n 49301 name> remote 49302 Type of storage to configure. 49303 Enter a string value. Press Enter for the default (""). 49304 Choose a number from below, or type in your own value 49305 [snip] 49306 XX / Zoho 49307 \ "zoho" 49308 [snip] 49309 Storage> zoho 49310 ** See help for zoho backend at: https://rclone.org/zoho/ ** 49311 49312 OAuth Client Id 49313 Leave blank normally. 49314 Enter a string value. Press Enter for the default (""). 49315 client_id> 49316 OAuth Client Secret 49317 Leave blank normally. 49318 Enter a string value. Press Enter for the default (""). 49319 client_secret> 49320 Edit advanced config? (y/n) 49321 y) Yes 49322 n) No (default) 49323 y/n> n 49324 Remote config 49325 Use web browser to automatically authenticate rclone with remote? 49326 * Say Y if the machine running rclone has a web browser you can use 49327 * Say N if running rclone on a (remote) machine without web browser access 49328 If not sure try Y. If Y failed, try N. 49329 y) Yes (default) 49330 n) No 49331 y/n> 49332 If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=LVn0IHzxej1ZkmQw31d0wQ 49333 Log in and authorize rclone for access 49334 Waiting for code... 49335 Got code 49336 Choose a number from below, or type in your own value 49337 1 / MyTeam 49338 \ "4u28602177065ff22426787a6745dba8954eb" 49339 Enter a Team ID> 1 49340 Choose a number from below, or type in your own value 49341 1 / General 49342 \ "4u2869d2aa6fca04f4f2f896b6539243b85b1" 49343 Enter a Workspace ID> 1 49344 -------------------- 49345 [remote] 49346 type = zoho 49347 token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Zoho-oauthtoken","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-10-12T00:54:52.370275223+02:00"} 49348 root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 49349 -------------------- 49350 y) Yes this is OK (default) 49351 e) Edit this remote 49352 d) Delete this remote 49353 y/e/d> 49354 49355 See the remote setup docs for how to set it up on a machine with no 49356 Internet browser available. 49357 49358 Rclone runs a webserver on your local computer to collect the 49359 authorization token from Zoho Workdrive. This is only from the moment 49360 your browser is opened until the token is returned. The webserver runs 49361 on http://127.0.0.1:53682/. If local port 53682 is protected by a 49362 firewall you may need to temporarily unblock the firewall to complete 49363 authorization. 49364 49365 Once configured you can then use rclone like this, 49366 49367 See top level directories 49368 49369 rclone lsd remote: 49370 49371 Make a new directory 49372 49373 rclone mkdir remote:directory 49374 49375 List the contents of a directory 49376 49377 rclone ls remote:directory 49378 49379 Sync /home/local/directory to the remote path, deleting any excess files 49380 in the path. 49381 49382 rclone sync --interactive /home/local/directory remote:directory 49383 49384 Zoho paths may be as deep as required, eg remote:directory/subdirectory. 49385 49386 Modification times and hashes 49387 49388 Modified times are currently not supported for Zoho Workdrive 49389 49390 No hash algorithms are supported. 49391 49392 Usage information 49393 49394 To view your current quota you can use the rclone about remote: command 49395 which will display your current usage. 49396 49397 Restricted filename characters 49398 49399 Only control characters and invalid UTF-8 are replaced. In addition most 49400 Unicode full-width characters are not supported at all and will be 49401 removed from filenames during upload. 49402 49403 Standard options 49404 49405 Here are the Standard options specific to zoho (Zoho). 49406 49407 --zoho-client-id 49408 49409 OAuth Client Id. 49410 49411 Leave blank normally. 49412 49413 Properties: 49414 49415 - Config: client_id 49416 - Env Var: RCLONE_ZOHO_CLIENT_ID 49417 - Type: string 49418 - Required: false 49419 49420 --zoho-client-secret 49421 49422 OAuth Client Secret. 49423 49424 Leave blank normally. 49425 49426 Properties: 49427 49428 - Config: client_secret 49429 - Env Var: RCLONE_ZOHO_CLIENT_SECRET 49430 - Type: string 49431 - Required: false 49432 49433 --zoho-region 49434 49435 Zoho region to connect to. 49436 49437 You'll have to use the region your organization is registered in. If not 49438 sure use the same top level domain as you connect to in your browser. 49439 49440 Properties: 49441 49442 - Config: region 49443 - Env Var: RCLONE_ZOHO_REGION 49444 - Type: string 49445 - Required: false 49446 - Examples: 49447 - "com" 49448 - United states / Global 49449 - "eu" 49450 - Europe 49451 - "in" 49452 - India 49453 - "jp" 49454 - Japan 49455 - "com.cn" 49456 - China 49457 - "com.au" 49458 - Australia 49459 49460 Advanced options 49461 49462 Here are the Advanced options specific to zoho (Zoho). 49463 49464 --zoho-token 49465 49466 OAuth Access Token as a JSON blob. 49467 49468 Properties: 49469 49470 - Config: token 49471 - Env Var: RCLONE_ZOHO_TOKEN 49472 - Type: string 49473 - Required: false 49474 49475 --zoho-auth-url 49476 49477 Auth server URL. 49478 49479 Leave blank to use the provider defaults. 49480 49481 Properties: 49482 49483 - Config: auth_url 49484 - Env Var: RCLONE_ZOHO_AUTH_URL 49485 - Type: string 49486 - Required: false 49487 49488 --zoho-token-url 49489 49490 Token server url. 49491 49492 Leave blank to use the provider defaults. 49493 49494 Properties: 49495 49496 - Config: token_url 49497 - Env Var: RCLONE_ZOHO_TOKEN_URL 49498 - Type: string 49499 - Required: false 49500 49501 --zoho-encoding 49502 49503 The encoding for the backend. 49504 49505 See the encoding section in the overview for more info. 49506 49507 Properties: 49508 49509 - Config: encoding 49510 - Env Var: RCLONE_ZOHO_ENCODING 49511 - Type: Encoding 49512 - Default: Del,Ctl,InvalidUtf8 49513 49514 --zoho-description 49515 49516 Description of the remote 49517 49518 Properties: 49519 49520 - Config: description 49521 - Env Var: RCLONE_ZOHO_DESCRIPTION 49522 - Type: string 49523 - Required: false 49524 49525 Setting up your own client_id 49526 49527 For Zoho we advise you to set up your own client_id. To do so you have 49528 to complete the following steps. 49529 49530 1. Log in to the Zoho API Console 49531 49532 2. Create a new client of type "Server-based Application". The name and 49533 website don't matter, but you must add the redirect URL 49534 http://localhost:53682/. 49535 49536 3. Once the client is created, you can go to the settings tab and 49537 enable it in other regions. 49538 49539 The client id and client secret can now be used with rclone. 49540 49541 Local Filesystem 49542 49543 Local paths are specified as normal filesystem paths, e.g. 49544 /path/to/wherever, so 49545 49546 rclone sync --interactive /home/source /tmp/destination 49547 49548 Will sync /home/source to /tmp/destination. 49549 49550 Configuration 49551 49552 For consistencies sake one can also configure a remote of type local in 49553 the config file, and access the local filesystem using rclone remote 49554 paths, e.g. remote:path/to/wherever, but it is probably easier not to. 49555 49556 Modification times 49557 49558 Rclone reads and writes the modification times using an accuracy 49559 determined by the OS. Typically this is 1ns on Linux, 10 ns on Windows 49560 and 1 Second on OS X. 49561 49562 Filenames 49563 49564 Filenames should be encoded in UTF-8 on disk. This is the normal case 49565 for Windows and OS X. 49566 49567 There is a bit more uncertainty in the Linux world, but new 49568 distributions will have UTF-8 encoded files names. If you are using an 49569 old Linux filesystem with non UTF-8 file names (e.g. latin1) then you 49570 can use the convmv tool to convert the filesystem to UTF-8. This tool is 49571 available in most distributions' package managers. 49572 49573 If an invalid (non-UTF8) filename is read, the invalid characters will 49574 be replaced with a quoted representation of the invalid bytes. The name 49575 gro\xdf will be transferred as gro‛DF. rclone will emit a debug message 49576 in this case (use -v to see), e.g. 49577 49578 Local file system at .: Replacing invalid UTF-8 characters in "gro\xdf" 49579 49580 Restricted characters 49581 49582 With the local backend, restrictions on the characters that are usable 49583 in file or directory names depend on the operating system. To check what 49584 rclone will replace by default on your system, run 49585 rclone help flags local-encoding. 49586 49587 On non Windows platforms the following characters are replaced when 49588 handling file names. 49589 49590 Character Value Replacement 49591 ----------- ------- ------------- 49592 NUL 0x00 ␀ 49593 / 0x2F / 49594 49595 When running on Windows the following characters are replaced. This list 49596 is based on the Windows file naming conventions. 49597 49598 Character Value Replacement 49599 ----------- ------- ------------- 49600 NUL 0x00 ␀ 49601 SOH 0x01 ␁ 49602 STX 0x02 ␂ 49603 ETX 0x03 ␃ 49604 EOT 0x04 ␄ 49605 ENQ 0x05 ␅ 49606 ACK 0x06 ␆ 49607 BEL 0x07 ␇ 49608 BS 0x08 ␈ 49609 HT 0x09 ␉ 49610 LF 0x0A ␊ 49611 VT 0x0B ␋ 49612 FF 0x0C ␌ 49613 CR 0x0D ␍ 49614 SO 0x0E ␎ 49615 SI 0x0F ␏ 49616 DLE 0x10 ␐ 49617 DC1 0x11 ␑ 49618 DC2 0x12 ␒ 49619 DC3 0x13 ␓ 49620 DC4 0x14 ␔ 49621 NAK 0x15 ␕ 49622 SYN 0x16 ␖ 49623 ETB 0x17 ␗ 49624 CAN 0x18 ␘ 49625 EM 0x19 ␙ 49626 SUB 0x1A ␚ 49627 ESC 0x1B ␛ 49628 FS 0x1C ␜ 49629 GS 0x1D ␝ 49630 RS 0x1E ␞ 49631 US 0x1F ␟ 49632 / 0x2F / 49633 " 0x22 " 49634 * 0x2A * 49635 : 0x3A : 49636 < 0x3C < 49637 > 0x3E > 49638 ? 0x3F ? 49639 \ 0x5C \ 49640 | 0x7C | 49641 49642 File names on Windows can also not end with the following characters. 49643 These only get replaced if they are the last character in the name: 49644 49645 Character Value Replacement 49646 ----------- ------- ------------- 49647 SP 0x20 ␠ 49648 . 0x2E . 49649 49650 Invalid UTF-8 bytes will also be replaced, as they can't be converted to 49651 UTF-16. 49652 49653 Paths on Windows 49654 49655 On Windows there are many ways of specifying a path to a file system 49656 resource. Local paths can be absolute, like C:\path\to\wherever, or 49657 relative, like ..\wherever. Network paths in UNC format, \\server\share, 49658 are also supported. Path separator can be either \ (as in 49659 C:\path\to\wherever) or / (as in C:/path/to/wherever). Length of these 49660 paths are limited to 259 characters for files and 247 characters for 49661 directories, but there is an alternative extended-length path format 49662 increasing the limit to (approximately) 32,767 characters. This format 49663 requires absolute paths and the use of prefix \\?\, e.g. 49664 \\?\D:\some\very\long\path. For convenience rclone will automatically 49665 convert regular paths into the corresponding extended-length paths, so 49666 in most cases you do not have to worry about this (read more below). 49667 49668 Note that Windows supports using the same prefix \\?\ to specify path to 49669 volumes identified by their GUID, e.g. 49670 \\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path. This is not 49671 supported in rclone, due to an issue in go. 49672 49673 Long paths 49674 49675 Rclone handles long paths automatically, by converting all paths to 49676 extended-length path format, which allows paths up to 32,767 characters. 49677 49678 This conversion will ensure paths are absolute and prefix them with the 49679 \\?\. This is why you will see that your paths, for instance .\files is 49680 shown as path \\?\C:\files in the output, and \\server\share as 49681 \\?\UNC\server\share. 49682 49683 However, in rare cases this may cause problems with buggy file system 49684 drivers like EncFS. To disable UNC conversion globally, add this to your 49685 .rclone.conf file: 49686 49687 [local] 49688 nounc = true 49689 49690 If you want to selectively disable UNC, you can add it to a separate 49691 entry like this: 49692 49693 [nounc] 49694 type = local 49695 nounc = true 49696 49697 And use rclone like this: 49698 49699 rclone copy c:\src nounc:z:\dst 49700 49701 This will use UNC paths on c:\src but not on z:\dst. Of course this will 49702 cause problems if the absolute path length of a file exceeds 259 49703 characters on z, so only use this option if you have to. 49704 49705 Symlinks / Junction points 49706 49707 Normally rclone will ignore symlinks or junction points (which behave 49708 like symlinks under Windows). 49709 49710 If you supply --copy-links or -L then rclone will follow the symlink and 49711 copy the pointed to file or directory. Note that this flag is 49712 incompatible with --links / -l. 49713 49714 This flag applies to all commands. 49715 49716 For example, supposing you have a directory structure like this 49717 49718 $ tree /tmp/a 49719 /tmp/a 49720 ├── b -> ../b 49721 ├── expected -> ../expected 49722 ├── one 49723 └── two 49724 └── three 49725 49726 Then you can see the difference with and without the flag like this 49727 49728 $ rclone ls /tmp/a 49729 6 one 49730 6 two/three 49731 49732 and 49733 49734 $ rclone -L ls /tmp/a 49735 4174 expected 49736 6 one 49737 6 two/three 49738 6 b/two 49739 6 b/one 49740 49741 --links, -l 49742 49743 Normally rclone will ignore symlinks or junction points (which behave 49744 like symlinks under Windows). 49745 49746 If you supply this flag then rclone will copy symbolic links from the 49747 local storage, and store them as text files, with a '.rclonelink' suffix 49748 in the remote storage. 49749 49750 The text file will contain the target of the symbolic link (see 49751 example). 49752 49753 This flag applies to all commands. 49754 49755 For example, supposing you have a directory structure like this 49756 49757 $ tree /tmp/a 49758 /tmp/a 49759 ├── file1 -> ./file4 49760 └── file2 -> /home/user/file3 49761 49762 Copying the entire directory with '-l' 49763 49764 $ rclone copyto -l /tmp/a/file1 remote:/tmp/a/ 49765 49766 The remote files are created with a '.rclonelink' suffix 49767 49768 $ rclone ls remote:/tmp/a 49769 5 file1.rclonelink 49770 14 file2.rclonelink 49771 49772 The remote files will contain the target of the symbolic links 49773 49774 $ rclone cat remote:/tmp/a/file1.rclonelink 49775 ./file4 49776 49777 $ rclone cat remote:/tmp/a/file2.rclonelink 49778 /home/user/file3 49779 49780 Copying them back with '-l' 49781 49782 $ rclone copyto -l remote:/tmp/a/ /tmp/b/ 49783 49784 $ tree /tmp/b 49785 /tmp/b 49786 ├── file1 -> ./file4 49787 └── file2 -> /home/user/file3 49788 49789 However, if copied back without '-l' 49790 49791 $ rclone copyto remote:/tmp/a/ /tmp/b/ 49792 49793 $ tree /tmp/b 49794 /tmp/b 49795 ├── file1.rclonelink 49796 └── file2.rclonelink 49797 49798 Note that this flag is incompatible with -copy-links / -L. 49799 49800 Restricting filesystems with --one-file-system 49801 49802 Normally rclone will recurse through filesystems as mounted. 49803 49804 However if you set --one-file-system or -x this tells rclone to stay in 49805 the filesystem specified by the root and not to recurse into different 49806 file systems. 49807 49808 For example if you have a directory hierarchy like this 49809 49810 root 49811 ├── disk1 - disk1 mounted on the root 49812 │ └── file3 - stored on disk1 49813 ├── disk2 - disk2 mounted on the root 49814 │ └── file4 - stored on disk12 49815 ├── file1 - stored on the root disk 49816 └── file2 - stored on the root disk 49817 49818 Using rclone --one-file-system copy root remote: will only copy file1 49819 and file2. Eg 49820 49821 $ rclone -q --one-file-system ls root 49822 0 file1 49823 0 file2 49824 49825 $ rclone -q ls root 49826 0 disk1/file3 49827 0 disk2/file4 49828 0 file1 49829 0 file2 49830 49831 NB Rclone (like most unix tools such as du, rsync and tar) treats a bind 49832 mount to the same device as being on the same filesystem. 49833 49834 NB This flag is only available on Unix based systems. On systems where 49835 it isn't supported (e.g. Windows) it will be ignored. 49836 49837 Advanced options 49838 49839 Here are the Advanced options specific to local (Local Disk). 49840 49841 --local-nounc 49842 49843 Disable UNC (long path names) conversion on Windows. 49844 49845 Properties: 49846 49847 - Config: nounc 49848 - Env Var: RCLONE_LOCAL_NOUNC 49849 - Type: bool 49850 - Default: false 49851 - Examples: 49852 - "true" 49853 - Disables long file names. 49854 49855 --copy-links / -L 49856 49857 Follow symlinks and copy the pointed to item. 49858 49859 Properties: 49860 49861 - Config: copy_links 49862 - Env Var: RCLONE_LOCAL_COPY_LINKS 49863 - Type: bool 49864 - Default: false 49865 49866 --links / -l 49867 49868 Translate symlinks to/from regular files with a '.rclonelink' extension. 49869 49870 Properties: 49871 49872 - Config: links 49873 - Env Var: RCLONE_LOCAL_LINKS 49874 - Type: bool 49875 - Default: false 49876 49877 --skip-links 49878 49879 Don't warn about skipped symlinks. 49880 49881 This flag disables warning messages on skipped symlinks or junction 49882 points, as you explicitly acknowledge that they should be skipped. 49883 49884 Properties: 49885 49886 - Config: skip_links 49887 - Env Var: RCLONE_LOCAL_SKIP_LINKS 49888 - Type: bool 49889 - Default: false 49890 49891 --local-zero-size-links 49892 49893 Assume the Stat size of links is zero (and read them instead) 49894 (deprecated). 49895 49896 Rclone used to use the Stat size of links as the link size, but this 49897 fails in quite a few places: 49898 49899 - Windows 49900 - On some virtual filesystems (such ash LucidLink) 49901 - Android 49902 49903 So rclone now always reads the link. 49904 49905 Properties: 49906 49907 - Config: zero_size_links 49908 - Env Var: RCLONE_LOCAL_ZERO_SIZE_LINKS 49909 - Type: bool 49910 - Default: false 49911 49912 --local-unicode-normalization 49913 49914 Apply unicode NFC normalization to paths and filenames. 49915 49916 This flag can be used to normalize file names into unicode NFC form that 49917 are read from the local filesystem. 49918 49919 Rclone does not normally touch the encoding of file names it reads from 49920 the file system. 49921 49922 This can be useful when using macOS as it normally provides decomposed 49923 (NFD) unicode which in some language (eg Korean) doesn't display 49924 properly on some OSes. 49925 49926 Note that rclone compares filenames with unicode normalization in the 49927 sync routine so this flag shouldn't normally be used. 49928 49929 Properties: 49930 49931 - Config: unicode_normalization 49932 - Env Var: RCLONE_LOCAL_UNICODE_NORMALIZATION 49933 - Type: bool 49934 - Default: false 49935 49936 --local-no-check-updated 49937 49938 Don't check to see if the files change during upload. 49939 49940 Normally rclone checks the size and modification time of files as they 49941 are being uploaded and aborts with a message which starts "can't copy - 49942 source file is being updated" if the file changes during upload. 49943 49944 However on some file systems this modification time check may fail (e.g. 49945 Glusterfs #2206) so this check can be disabled with this flag. 49946 49947 If this flag is set, rclone will use its best efforts to transfer a file 49948 which is being updated. If the file is only having things appended to it 49949 (e.g. a log) then rclone will transfer the log file with the size it had 49950 the first time rclone saw it. 49951 49952 If the file is being modified throughout (not just appended to) then the 49953 transfer may fail with a hash check failure. 49954 49955 In detail, once the file has had stat() called on it for the first time 49956 we: 49957 49958 - Only transfer the size that stat gave 49959 - Only checksum the size that stat gave 49960 - Don't update the stat info for the file 49961 49962 NB do not use this flag on a Windows Volume Shadow (VSS). For some 49963 unknown reason, files in a VSS sometimes show different sizes from the 49964 directory listing (where the initial stat value comes from on Windows) 49965 and when stat is called on them directly. Other copy tools always use 49966 the direct stat value and setting this flag will disable that. 49967 49968 Properties: 49969 49970 - Config: no_check_updated 49971 - Env Var: RCLONE_LOCAL_NO_CHECK_UPDATED 49972 - Type: bool 49973 - Default: false 49974 49975 --one-file-system / -x 49976 49977 Don't cross filesystem boundaries (unix/macOS only). 49978 49979 Properties: 49980 49981 - Config: one_file_system 49982 - Env Var: RCLONE_LOCAL_ONE_FILE_SYSTEM 49983 - Type: bool 49984 - Default: false 49985 49986 --local-case-sensitive 49987 49988 Force the filesystem to report itself as case sensitive. 49989 49990 Normally the local backend declares itself as case insensitive on 49991 Windows/macOS and case sensitive for everything else. Use this flag to 49992 override the default choice. 49993 49994 Properties: 49995 49996 - Config: case_sensitive 49997 - Env Var: RCLONE_LOCAL_CASE_SENSITIVE 49998 - Type: bool 49999 - Default: false 50000 50001 --local-case-insensitive 50002 50003 Force the filesystem to report itself as case insensitive. 50004 50005 Normally the local backend declares itself as case insensitive on 50006 Windows/macOS and case sensitive for everything else. Use this flag to 50007 override the default choice. 50008 50009 Properties: 50010 50011 - Config: case_insensitive 50012 - Env Var: RCLONE_LOCAL_CASE_INSENSITIVE 50013 - Type: bool 50014 - Default: false 50015 50016 --local-no-preallocate 50017 50018 Disable preallocation of disk space for transferred files. 50019 50020 Preallocation of disk space helps prevent filesystem fragmentation. 50021 However, some virtual filesystem layers (such as Google Drive File 50022 Stream) may incorrectly set the actual file size equal to the 50023 preallocated space, causing checksum and file size checks to fail. Use 50024 this flag to disable preallocation. 50025 50026 Properties: 50027 50028 - Config: no_preallocate 50029 - Env Var: RCLONE_LOCAL_NO_PREALLOCATE 50030 - Type: bool 50031 - Default: false 50032 50033 --local-no-sparse 50034 50035 Disable sparse files for multi-thread downloads. 50036 50037 On Windows platforms rclone will make sparse files when doing 50038 multi-thread downloads. This avoids long pauses on large files where the 50039 OS zeros the file. However sparse files may be undesirable as they cause 50040 disk fragmentation and can be slow to work with. 50041 50042 Properties: 50043 50044 - Config: no_sparse 50045 - Env Var: RCLONE_LOCAL_NO_SPARSE 50046 - Type: bool 50047 - Default: false 50048 50049 --local-no-set-modtime 50050 50051 Disable setting modtime. 50052 50053 Normally rclone updates modification time of files after they are done 50054 uploading. This can cause permissions issues on Linux platforms when the 50055 user rclone is running as does not own the file uploaded, such as when 50056 copying to a CIFS mount owned by another user. If this option is 50057 enabled, rclone will no longer update the modtime after copying a file. 50058 50059 Properties: 50060 50061 - Config: no_set_modtime 50062 - Env Var: RCLONE_LOCAL_NO_SET_MODTIME 50063 - Type: bool 50064 - Default: false 50065 50066 --local-encoding 50067 50068 The encoding for the backend. 50069 50070 See the encoding section in the overview for more info. 50071 50072 Properties: 50073 50074 - Config: encoding 50075 - Env Var: RCLONE_LOCAL_ENCODING 50076 - Type: Encoding 50077 - Default: Slash,Dot 50078 50079 --local-description 50080 50081 Description of the remote 50082 50083 Properties: 50084 50085 - Config: description 50086 - Env Var: RCLONE_LOCAL_DESCRIPTION 50087 - Type: string 50088 - Required: false 50089 50090 Metadata 50091 50092 Depending on which OS is in use the local backend may return only some 50093 of the system metadata. Setting system metadata is supported on all OSes 50094 but setting user metadata is only supported on linux, freebsd, netbsd, 50095 macOS and Solaris. It is not supported on Windows yet (see 50096 pkg/attrs#47). 50097 50098 User metadata is stored as extended attributes (which may not be 50099 supported by all file systems) under the "user.*" prefix. 50100 50101 Metadata is supported on files and directories. 50102 50103 Here are the possible system metadata items for the local backend. 50104 50105 --------------------------------------------------------------------------------------------------- 50106 Name Help Type Example Read Only 50107 ----------- -------------- ------------- ------------------------------------- -------------------- 50108 atime Time of last RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N 50109 access 50110 50111 btime Time of file RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N 50112 birth 50113 (creation) 50114 50115 gid Group ID of decimal 500 N 50116 owner number 50117 50118 mode File type and octal, unix 0100664 N 50119 mode style 50120 50121 mtime Time of last RFC 3339 2006-01-02T15:04:05.999999999Z07:00 N 50122 modification 50123 50124 rdev Device ID (if hexadecimal 1abc N 50125 special file) 50126 50127 uid User ID of decimal 500 N 50128 owner number 50129 --------------------------------------------------------------------------------------------------- 50130 50131 See the metadata docs for more info. 50132 50133 Backend commands 50134 50135 Here are the commands specific to the local backend. 50136 50137 Run them with 50138 50139 rclone backend COMMAND remote: 50140 50141 The help below will explain what arguments each command takes. 50142 50143 See the backend command for more info on how to pass options and 50144 arguments. 50145 50146 These can be run on a running backend using the rc command 50147 backend/command. 50148 50149 noop 50150 50151 A null operation for testing backend commands 50152 50153 rclone backend noop remote: [options] [<arguments>+] 50154 50155 This is a test command which has some options you can try to change the 50156 output. 50157 50158 Options: 50159 50160 - "echo": echo the input arguments 50161 - "error": return an error based on option value 50162 50163 Changelog 50164 50165 v1.66.0 - 2024-03-10 50166 50167 See commits 50168 50169 - Major features 50170 - Rclone will now sync directory modification times if the backend 50171 supports it. 50172 - This can be disabled with --no-update-dir-modtime 50173 - See the overview and look for the D flags in the ModTime 50174 column to see which backends support it. 50175 - Rclone will now sync directory metadata if the backend supports 50176 it when -M/--metadata is in use. 50177 - See the overview and look for the D flags in the Metadata 50178 column to see which backends support it. 50179 - Bisync has received many updates see below for more details or 50180 bisync's changelog 50181 - Removed backends 50182 - amazonclouddrive: Remove Amazon Drive backend code and docs 50183 (Nick Craig-Wood) 50184 - New Features 50185 - backend 50186 - Add description field for all backends (Paul Stern) 50187 - build 50188 - Update to go1.22 and make go1.20 the minimum required 50189 version (Nick Craig-Wood) 50190 - Fix CVE-2024-24786 by upgrading google.golang.org/protobuf 50191 (Nick Craig-Wood) 50192 - check: Respect --no-unicode-normalization and --ignore-case-sync 50193 for --checkfile (nielash) 50194 - cmd: Much improved shell auto completion which reduces the size 50195 of the completion file and works faster (Nick Craig-Wood) 50196 - doc updates (albertony, ben-ba, Eli, emyarod, huajin tong, Jack 50197 Provance, kapitainsky, keongalvin, Nick Craig-Wood, nielash, 50198 rarspace01, rzitzer, Tera, Vincent Murphy) 50199 - fs: Add more detailed logging for file includes/excludes (Kyle 50200 Reynolds) 50201 - lsf 50202 - Add --time-format flag (nielash) 50203 - Make metadata appear for directories (Nick Craig-Wood) 50204 - lsjson: Make metadata appear for directories (Nick Craig-Wood) 50205 - rc 50206 - Add srcFs and dstFs to core/stats and core/transferred stats 50207 (Nick Craig-Wood) 50208 - Add operations/hashsum to the rc as rclone hashsum 50209 equivalent (Nick Craig-Wood) 50210 - Add config/paths to the rc as rclone config paths equivalent 50211 (Nick Craig-Wood) 50212 - sync 50213 - Optionally report list of synced paths to file (nielash) 50214 - Implement directory sync for mod times and metadata (Nick 50215 Craig-Wood) 50216 - Don't set directory modtimes if already set (nielash) 50217 - Don't sync directory modtimes from backends which don't have 50218 directories (Nick Craig-Wood) 50219 - Bug Fixes 50220 - backend 50221 - Make backends which use oauth implement the Shutdown and 50222 shutdown the oauth properly (rkonfj) 50223 - bisync 50224 - Handle unicode and case normalization consistently (nielash) 50225 - Partial uploads known issue on local/ftp/sftp has been 50226 resolved (unless using --inplace) (nielash) 50227 - Fixed handling of unicode normalization and case 50228 insensitivity, support for --fix-case, --ignore-case-sync, 50229 --no-unicode-normalization (nielash) 50230 - Bisync no longer fails to find the correct listing file when 50231 configs are overridden with backend-specific flags. 50232 (nielash) 50233 - nfsmount 50234 - Fix exit after external unmount (nielash) 50235 - Fix --volname being ignored (nielash) 50236 - operations 50237 - Fix renaming a file on macOS (nielash) 50238 - Fix case-insensitive moves in operations.Move (nielash) 50239 - Fix TestCaseInsensitiveMoveFileDryRun on chunker integration 50240 tests (nielash) 50241 - Fix TestMkdirModTime test (Nick Craig-Wood) 50242 - Fix TestSetDirModTime for backends with SetDirModTime but 50243 not Metadata (Nick Craig-Wood) 50244 - Fix typo in log messages (nielash) 50245 - serve nfs: Fix writing files via Finder on macOS (nielash) 50246 - serve restic: Fix error handling (Michael Eischer) 50247 - serve webdav: Fix --baseurl without leading / (Nick Craig-Wood) 50248 - stats: Fix race between ResetCounters and stopAverageLoop called 50249 from time.AfterFunc (Nick Craig-Wood) 50250 - sync 50251 - --fix-case flag to rename case insensitive dest (nielash) 50252 - Use operations.DirMove instead of sync.MoveDir for 50253 --fix-case (nielash) 50254 - systemd: Fix detection and switch to the coreos package 50255 everywhere rather than having 2 separate libraries (Anagh Kumar 50256 Baranwal) 50257 - Mount 50258 - Fix macOS not noticing errors with --daemon (Nick Craig-Wood) 50259 - Notice daemon dying much quicker (Nick Craig-Wood) 50260 - VFS 50261 - Fix unicode normalization on macOS (nielash) 50262 - Bisync 50263 - Copies and deletes are now handled in one operation instead of 50264 two (nielash) 50265 - --track-renames and --backup-dir are now supported (nielash) 50266 - Final listings are now generated from sync results, to avoid 50267 needing to re-list (nielash) 50268 - Bisync is now much more resilient to changes that happen during 50269 a bisync run, and far less prone to critical errors / undetected 50270 changes (nielash) 50271 - Bisync is now capable of rolling a file listing back in cases of 50272 uncertainty, essentially marking the file as needing to be 50273 rechecked next time. (nielash) 50274 - A few basic terminal colors are now supported, controllable with 50275 --color (AUTO|NEVER|ALWAYS) (nielash) 50276 - Initial listing snapshots of Path1 and Path2 are now generated 50277 concurrently, using the same "march" infrastructure as check and 50278 sync, for performance improvements and less risk of error. 50279 (nielash) 50280 - --resync is now much more efficient (especially for users of 50281 --create-empty-src-dirs) (nielash) 50282 - Google Docs (and other files of unknown size) are now supported 50283 (with the same options as in sync) (nielash) 50284 - Equality checks before a sync conflict rename now fall back to 50285 cryptcheck (when possible) or --download, (nielash) instead of 50286 of --size-only, when check is not available. 50287 - Bisync now fully supports comparing based on any combination of 50288 size, modtime, and checksum, lifting the prior restriction on 50289 backends without modtime support. (nielash) 50290 - Bisync now supports a "Graceful Shutdown" mode to cleanly cancel 50291 a run early without requiring --resync. (nielash) 50292 - New --recover flag allows robust recovery in the event of 50293 interruptions, without requiring --resync. (nielash) 50294 - A new --max-lock setting allows lock files to automatically 50295 renew and expire, for better automatic recovery when a run is 50296 interrupted. (nielash) 50297 - Bisync now supports auto-resolving sync conflicts and 50298 customizing rename behavior with new --conflict-resolve, 50299 --conflict-loser, and --conflict-suffix flags. (nielash) 50300 - A new --resync-mode flag allows more control over which version 50301 of a file gets kept during a --resync. (nielash) 50302 - Bisync now supports --retries and --retries-sleep (when 50303 --resilient is set.) (nielash) 50304 - Clarify file operation directions in dry-run logs (Kyle 50305 Reynolds) 50306 - Local 50307 - Fix cleanRootPath on Windows after go1.21.4 stdlib update 50308 (nielash) 50309 - Implement setting modification time on directories (nielash) 50310 - Implement modtime and metadata for directories (Nick Craig-Wood) 50311 - Fix setting of btime on directories on Windows (Nick Craig-Wood) 50312 - Delete backend implementation of Purge to speed up and make 50313 stats (Nick Craig-Wood) 50314 - Support metadata setting and mapping on server side Move (Nick 50315 Craig-Wood) 50316 - Cache 50317 - Implement setting modification time on directories (if supported 50318 by wrapped remote) (nielash) 50319 - Implement setting metadata on directories (Nick Craig-Wood) 50320 - Crypt 50321 - Implement setting modification time on directories (if supported 50322 by wrapped remote) (nielash) 50323 - Implement setting metadata on directories (Nick Craig-Wood) 50324 - Improve handling of undecryptable file names (nielash) 50325 - Add missing error check spotted by linter (Nick Craig-Wood) 50326 - Azure Blob 50327 - Implement --azureblob-delete-snapshots (Nick Craig-Wood) 50328 - B2 50329 - Clarify exactly what --b2-download-auth-duration does in the 50330 docs (Nick Craig-Wood) 50331 - Chunker 50332 - Implement setting modification time on directories (if supported 50333 by wrapped remote) (nielash) 50334 - Implement setting metadata on directories (Nick Craig-Wood) 50335 - Combine 50336 - Implement setting modification time on directories (if supported 50337 by wrapped remote) (nielash) 50338 - Implement setting metadata on directories (Nick Craig-Wood) 50339 - Fix directory metadata error on upstream root (nielash) 50340 - Fix directory move across upstreams (nielash) 50341 - Compress 50342 - Implement setting modification time on directories (if supported 50343 by wrapped remote) (nielash) 50344 - Implement setting metadata on directories (Nick Craig-Wood) 50345 - Drive 50346 - Implement setting modification time on directories (nielash) 50347 - Implement modtime and metadata setting for directories (Nick 50348 Craig-Wood) 50349 - Support metadata setting and mapping on server side Move,Copy 50350 (Nick Craig-Wood) 50351 - FTP 50352 - Fix mkdir with rsftp which is returning the wrong code (Nick 50353 Craig-Wood) 50354 - Hasher 50355 - Implement setting modification time on directories (if supported 50356 by wrapped remote) (nielash) 50357 - Implement setting metadata on directories (Nick Craig-Wood) 50358 - Fix error from trying to stop an already-stopped db (nielash) 50359 - Look for cached hash if passed hash unexpectedly blank (nielash) 50360 - Imagekit 50361 - Updated docs and web content (Harshit Budhraja) 50362 - Updated overview - supported operations (Harshit Budhraja) 50363 - Mega 50364 - Fix panic with go1.22 (Nick Craig-Wood) 50365 - Netstorage 50366 - Fix Root to return correct directory when pointing to a file 50367 (Nick Craig-Wood) 50368 - Onedrive 50369 - Add metadata support (nielash) 50370 - Opendrive 50371 - Fix moving file/folder within the same parent dir (nielash) 50372 - Oracle Object Storage 50373 - Support backend restore command (Nikhil Ahuja) 50374 - Support workload identity authentication for OKE (Anders 50375 Swanson) 50376 - Protondrive 50377 - Fix encoding of Root method (Nick Craig-Wood) 50378 - Quatrix 50379 - Fix Content-Range header (Volodymyr) 50380 - Add option to skip project folders (Oksana Zhykina) 50381 - Fix Root to return correct directory when pointing to a file 50382 (Nick Craig-Wood) 50383 - S3 50384 - Add --s3-version-deleted to show delete markers in listings when 50385 using versions. (Nick Craig-Wood) 50386 - Add IPv6 support with option --s3-use-dual-stack (Anthony 50387 Metzidis) 50388 - Copy parts in parallel when doing chunked server side copy (Nick 50389 Craig-Wood) 50390 - GCS provider: fix server side copy of files bigger than 5G (Nick 50391 Craig-Wood) 50392 - Support metadata setting and mapping on server side Copy (Nick 50393 Craig-Wood) 50394 - Seafile 50395 - Fix download/upload error when FILE_SERVER_ROOT is relative 50396 (DanielEgbers) 50397 - Fix Root to return correct directory when pointing to a file 50398 (Nick Craig-Wood) 50399 - SFTP 50400 - Implement setting modification time on directories (nielash) 50401 - Set directory modtimes update on write flag (Nick Craig-Wood) 50402 - Shorten wait delay for external ssh binaries now that we are 50403 using go1.20 (Nick Craig-Wood) 50404 - Swift 50405 - Avoid unnecessary container versioning check (Joe Cai) 50406 - Union 50407 - Implement setting modification time on directories (if supported 50408 by wrapped remote) (nielash) 50409 - Implement setting metadata on directories (Nick Craig-Wood) 50410 - WebDAV 50411 - Reduce priority of chunks upload log (Gabriel Ramos) 50412 - owncloud: Add config owncloud_exclude_shares which allows to 50413 exclude shared files and folders when listing remote resources 50414 (Thomas Müller) 50415 50416 v1.65.2 - 2024-01-24 50417 50418 See commits 50419 50420 - Bug Fixes 50421 - build: bump github.com/cloudflare/circl from 1.3.6 to 1.3.7 50422 (dependabot) 50423 - docs updates (Nick Craig-Wood, kapitainsky, nielash, Tera, 50424 Harshit Budhraja) 50425 - VFS 50426 - Fix stale data when using --vfs-cache-mode full (Nick 50427 Craig-Wood) 50428 - Azure Blob 50429 - IMPORTANT Fix data corruption bug - see #7590 (Nick Craig-Wood) 50430 50431 v1.65.1 - 2024-01-08 50432 50433 See commits 50434 50435 - Bug Fixes 50436 - build 50437 - Bump golang.org/x/crypto to fix ssh terrapin CVE-2023-48795 50438 (dependabot) 50439 - Update to go1.21.5 to fix Windows path problems (Nick 50440 Craig-Wood) 50441 - Fix docker build on arm/v6 (Nick Craig-Wood) 50442 - install.sh: fix harmless error message on install (Nick 50443 Craig-Wood) 50444 - accounting: fix stats to show server side transfers (Nick 50445 Craig-Wood) 50446 - doc fixes (albertony, ben-ba, Eli Orzitzer, emyarod, keongalvin, 50447 rarspace01) 50448 - nfsmount: Compile for all unix oses, add --sudo and fix 50449 error/option handling (Nick Craig-Wood) 50450 - operations: Fix files moved by rclone move not being counted as 50451 transfers (Nick Craig-Wood) 50452 - oauthutil: Avoid panic when *token and *ts.token are the same 50453 (rkonfj) 50454 - serve s3: Fix listing oddities (Nick Craig-Wood) 50455 - VFS 50456 - Note that --vfs-refresh runs in the background (Nick Craig-Wood) 50457 - Azurefiles 50458 - Fix storage base url (Oksana) 50459 - Crypt 50460 - Fix rclone move a file over itself deleting the file (Nick 50461 Craig-Wood) 50462 - Chunker 50463 - Fix rclone move a file over itself deleting the file (Nick 50464 Craig-Wood) 50465 - Compress 50466 - Fix rclone move a file over itself deleting the file (Nick 50467 Craig-Wood) 50468 - Dropbox 50469 - Fix used space on dropbox team accounts (Nick Craig-Wood) 50470 - FTP 50471 - Fix multi-thread copy (WeidiDeng) 50472 - Googlephotos 50473 - Fix nil pointer exception when batch failed (Nick Craig-Wood) 50474 - Hasher 50475 - Fix rclone move a file over itself deleting the file (Nick 50476 Craig-Wood) 50477 - Fix invalid memory address error when MaxAge == 0 (nielash) 50478 - Onedrive 50479 - Fix error listing: unknown object type <nil> (Nick Craig-Wood) 50480 - Fix "unauthenticated: Unauthenticated" errors when uploading 50481 (Nick Craig-Wood) 50482 - Oracleobjectstorage 50483 - Fix object storage endpoint for custom endpoints (Manoj Ghosh) 50484 - Multipart copy create bucket if it doesn't exist. (Manoj Ghosh) 50485 - Protondrive 50486 - Fix CVE-2023-45286 / GHSA-xwh9-gc39-5298 (Nick Craig-Wood) 50487 - S3 50488 - Fix crash if no UploadId in multipart upload (Nick Craig-Wood) 50489 - Smb 50490 - Fix shares not listed by updating go-smb2 (halms) 50491 - Union 50492 - Fix rclone move a file over itself deleting the file (Nick 50493 Craig-Wood) 50494 50495 v1.65.0 - 2023-11-26 50496 50497 See commits 50498 50499 - New backends 50500 - Azure Files (karan, moongdal, Nick Craig-Wood) 50501 - ImageKit (Abhinav Dhiman) 50502 - Linkbox (viktor, Nick Craig-Wood) 50503 - New commands 50504 - serve s3: Let rclone act as an S3 compatible server (Mikubill, 50505 Artur Neumann, Saw-jan, Nick Craig-Wood) 50506 - nfsmount: mount command to provide mount mechanism on macOS 50507 without FUSE (Saleh Dindar) 50508 - serve nfs: to serve a remote for use by nfsmount (Saleh Dindar) 50509 - New Features 50510 - install.sh: Clean up temp files in install script (Jacob Hands) 50511 - build 50512 - Update all dependencies (Nick Craig-Wood) 50513 - Refactor version info and icon resource handling on windows 50514 (albertony) 50515 - doc updates (albertony, alfish2000, asdffdsazqqq, Dimitri 50516 Papadopoulos, Herby Gillot, Joda Stößer, Manoj Ghosh, Nick 50517 Craig-Wood) 50518 - Implement --metadata-mapper to transform metatadata with a user 50519 supplied program (Nick Craig-Wood) 50520 - Add ChunkWriterDoesntSeek feature flag and set it for b2 (Nick 50521 Craig-Wood) 50522 - lib/http: Export basic go string functions for use in --template 50523 (Gabriel Espinoza) 50524 - makefile: Use POSIX compatible install arguments (Mina Galić) 50525 - operations 50526 - Use less memory when doing multithread uploads (Nick 50527 Craig-Wood) 50528 - Implement --partial-suffix to control extension of temporary 50529 file names (Volodymyr) 50530 - rc 50531 - Add operations/check to the rc API (Nick Craig-Wood) 50532 - Always report an error as JSON (Nick Craig-Wood) 50533 - Set Last-Modified header for files served by --rc-serve 50534 (Nikita Shoshin) 50535 - size: Dont show duplicate object count when less than 1k 50536 (albertony) 50537 - Bug Fixes 50538 - fshttp: Fix --contimeout being ignored (你知道未来吗) 50539 - march: Fix excessive parallelism when using --no-traverse (Nick 50540 Craig-Wood) 50541 - ncdu: Fix crash when re-entering changed directory after rescan 50542 (Nick Craig-Wood) 50543 - operations 50544 - Fix overwrite of destination when multi-thread transfer 50545 fails (Nick Craig-Wood) 50546 - Fix invalid UTF-8 when truncating file names when not using 50547 --inplace (Nick Craig-Wood) 50548 - serve dnla: Fix crash on graceful exit (wuxingzhong) 50549 - Mount 50550 - Disable mount for freebsd and alias cmount as mount on that 50551 platform (Nick Craig-Wood) 50552 - VFS 50553 - Add --vfs-refresh flag to read all the directories on start 50554 (Beyond Meat) 50555 - Implement Name() method in WriteFileHandle and ReadFileHandle 50556 (Saleh Dindar) 50557 - Add go-billy dependency and make sure vfs.Handle implements 50558 billy.File (Saleh Dindar) 50559 - Error out early if can't upload 0 length file (Nick Craig-Wood) 50560 - Local 50561 - Fix copying from Windows Volume Shadows (Nick Craig-Wood) 50562 - Azure Blob 50563 - Add support for cold tier (Ivan Yanitra) 50564 - B2 50565 - Implement "rclone backend lifecycle" to read and set bucket 50566 lifecycles (Nick Craig-Wood) 50567 - Implement --b2-lifecycle to control lifecycle when creating 50568 buckets (Nick Craig-Wood) 50569 - Fix listing all buckets when not needed (Nick Craig-Wood) 50570 - Fix multi-thread upload with copyto going to wrong name (Nick 50571 Craig-Wood) 50572 - Fix server side chunked copy when file size was exactly 50573 --b2-copy-cutoff (Nick Craig-Wood) 50574 - Fix streaming chunked files an exact multiple of chunk size 50575 (Nick Craig-Wood) 50576 - Box 50577 - Filter more EventIDs when polling (David Sze) 50578 - Add more logging for polling (David Sze) 50579 - Fix performance problem reading metadata for single files (Nick 50580 Craig-Wood) 50581 - Drive 50582 - Add read/write metadata support (Nick Craig-Wood) 50583 - Add support for SHA-1 and SHA-256 checksums (rinsuki) 50584 - Add --drive-show-all-gdocs to allow unexportable gdocs to be 50585 server side copied (Nick Craig-Wood) 50586 - Add a note that --drive-scope accepts comma-separated list of 50587 scopes (Keigo Imai) 50588 - Fix error updating created time metadata on existing object 50589 (Nick Craig-Wood) 50590 - Fix integration tests by enabling metadata support from the 50591 context (Nick Craig-Wood) 50592 - Dropbox 50593 - Factor batcher into lib/batcher (Nick Craig-Wood) 50594 - Fix missing encoding for rclone purge (Nick Craig-Wood) 50595 - Google Cloud Storage 50596 - Fix 400 Bad request errors when using multi-thread copy (Nick 50597 Craig-Wood) 50598 - Googlephotos 50599 - Implement batcher for uploads (Nick Craig-Wood) 50600 - Hdfs 50601 - Added support for list of namenodes in hdfs remote config 50602 (Tayo-pasedaRJ) 50603 - HTTP 50604 - Implement set backend command to update running backend (Nick 50605 Craig-Wood) 50606 - Enable methods used with WebDAV (Alen Šiljak) 50607 - Jottacloud 50608 - Add support for reading and writing metadata (albertony) 50609 - Onedrive 50610 - Implement ListR method which gives --fast-list support (Nick 50611 Craig-Wood) 50612 - This must be enabled with the --onedrive-delta flag 50613 - Quatrix 50614 - Add partial upload support (Oksana Zhykina) 50615 - Overwrite files on conflict during server-side move (Oksana 50616 Zhykina) 50617 - S3 50618 - Add Linode provider (Nick Craig-Wood) 50619 - Add docs on how to add a new provider (Nick Craig-Wood) 50620 - Fix no error being returned when creating a bucket we don't own 50621 (Nick Craig-Wood) 50622 - Emit a debug message if anonymous credentials are in use (Nick 50623 Craig-Wood) 50624 - Add --s3-disable-multipart-uploads flag (Nick Craig-Wood) 50625 - Detect looping when using gcs and versions (Nick Craig-Wood) 50626 - SFTP 50627 - Implement --sftp-copy-is-hardlink to server side copy as 50628 hardlink (Nick Craig-Wood) 50629 - Smb 50630 - Fix incorrect about size by switching to 50631 github.com/cloudsoda/go-smb2 fork (Nick Craig-Wood) 50632 - Fix modtime of multithread uploads by setting PartialUploads 50633 (Nick Craig-Wood) 50634 - WebDAV 50635 - Added an rclone vendor to work with rclone serve webdav (Adithya 50636 Kumar) 50637 50638 v1.64.2 - 2023-10-19 50639 50640 See commits 50641 50642 - Bug Fixes 50643 - selfupdate: Fix "invalid hashsum signature" error (Nick 50644 Craig-Wood) 50645 - build: Fix docker build running out of space (Nick Craig-Wood) 50646 50647 v1.64.1 - 2023-10-17 50648 50649 See commits 50650 50651 - Bug Fixes 50652 - cmd: Make --progress output logs in the same format as without 50653 (Nick Craig-Wood) 50654 - docs fixes (Dimitri Papadopoulos Orfanos, Herby Gillot, Manoj 50655 Ghosh, Nick Craig-Wood) 50656 - lsjson: Make sure we set the global metadata flag too (Nick 50657 Craig-Wood) 50658 - operations 50659 - Ensure concurrency is no greater than the number of chunks 50660 (Pat Patterson) 50661 - Fix OpenOptions ignored in copy if operation was a 50662 multiThreadCopy (Vitor Gomes) 50663 - Fix error message on delete to have file name (Nick 50664 Craig-Wood) 50665 - serve sftp: Return not supported error for not supported 50666 commands (Nick Craig-Wood) 50667 - build: Upgrade golang.org/x/net to v0.17.0 to fix HTTP/2 rapid 50668 reset (Nick Craig-Wood) 50669 - pacer: Fix b2 deadlock by defaulting max connections to 50670 unlimited (Nick Craig-Wood) 50671 - Mount 50672 - Fix automount not detecting drive is ready (Nick Craig-Wood) 50673 - VFS 50674 - Fix update dir modification time (Saleh Dindar) 50675 - Azure Blob 50676 - Fix "fatal error: concurrent map writes" (Nick Craig-Wood) 50677 - B2 50678 - Fix multipart upload: corrupted on transfer: sizes differ XXX vs 50679 0 (Nick Craig-Wood) 50680 - Fix locking window when getting mutipart upload URL (Nick 50681 Craig-Wood) 50682 - Fix server side copies greater than 4GB (Nick Craig-Wood) 50683 - Fix chunked streaming uploads (Nick Craig-Wood) 50684 - Reduce default --b2-upload-concurrency to 4 to reduce memory 50685 usage (Nick Craig-Wood) 50686 - Onedrive 50687 - Fix the configurator to allow /teams/ID in the config (Nick 50688 Craig-Wood) 50689 - Oracleobjectstorage 50690 - Fix OpenOptions being ignored in uploadMultipart with 50691 chunkWriter (Nick Craig-Wood) 50692 - S3 50693 - Fix slice bounds out of range error when listing (Nick 50694 Craig-Wood) 50695 - Fix OpenOptions being ignored in uploadMultipart with 50696 chunkWriter (Vitor Gomes) 50697 - Storj 50698 - Update storj.io/uplink to v1.12.0 (Kaloyan Raev) 50699 50700 v1.64.0 - 2023-09-11 50701 50702 See commits 50703 50704 - New backends 50705 - Proton Drive (Chun-Hung Tseng) 50706 - Quatrix (Oksana, Volodymyr Kit) 50707 - New S3 providers 50708 - Synology C2 (BakaWang) 50709 - Leviia (Benjamin) 50710 - New Jottacloud providers 50711 - Onlime (Fjodor42) 50712 - Telia Sky (NoLooseEnds) 50713 - Major changes 50714 - Multi-thread transfers (Vitor Gomes, Nick Craig-Wood, Manoj 50715 Ghosh, Edwin Mackenzie-Owen) 50716 - Multi-thread transfers are now available when transferring 50717 to: 50718 - local, s3, azureblob, b2, oracleobjectstorage and smb 50719 - This greatly improves transfer speed between two network 50720 sources. 50721 - In memory buffering has been unified between all backends 50722 and should share memory better. 50723 - See --multi-thread docs for more info 50724 - New commands 50725 - rclone config redacted support mechanism for showing redacted 50726 config (Nick Craig-Wood) 50727 - New Features 50728 - accounting 50729 - Show server side stats in own lines and not as bytes 50730 transferred (Nick Craig-Wood) 50731 - bisync 50732 - Add new --ignore-listing-checksum flag to distinguish from 50733 --ignore-checksum (nielash) 50734 - Add experimental --resilient mode to allow recovery from 50735 self-correctable errors (nielash) 50736 - Add support for --create-empty-src-dirs (nielash) 50737 - Dry runs no longer commit filter changes (nielash) 50738 - Enforce --check-access during --resync (nielash) 50739 - Apply filters correctly during deletes (nielash) 50740 - Equality check before renaming (leave identical files alone) 50741 (nielash) 50742 - Fix dryRun rc parameter being ignored (nielash) 50743 - build 50744 - Update to go1.21 and make go1.19 the minimum required 50745 version (Anagh Kumar Baranwal, Nick Craig-Wood) 50746 - Update dependencies (Nick Craig-Wood) 50747 - Add snap installation (hideo aoyama) 50748 - Change Winget Releaser job to ubuntu-latest (sitiom) 50749 - cmd: Refactor and use sysdnotify in more commands (eNV25) 50750 - config: Add --multi-thread-chunk-size flag (Vitor Gomes) 50751 - doc updates (antoinetran, Benjamin, Bjørn Smith, Dean Attali, 50752 gabriel-suela, James Braza, Justin Hellings, kapitainsky, Mahad, 50753 Masamune3210, Nick Craig-Wood, Nihaal Sangha, Niklas Hambüchen, 50754 Raymond Berger, r-ricci, Sawada Tsunayoshi, Tiago Boeing, 50755 Vladislav Vorobev) 50756 - fs 50757 - Use atomic types everywhere (Roberto Ricci) 50758 - When --max-transfer limit is reached exit with code (10) 50759 (kapitainsky) 50760 - Add rclone completion powershell - basic implementation only 50761 (Nick Craig-Wood) 50762 - http servers: Allow CORS to be set with --allow-origin flag 50763 (yuudi) 50764 - lib/rest: Remove unnecessary nil check (Eng Zer Jun) 50765 - ncdu: Add keybinding to rescan filesystem (eNV25) 50766 - rc 50767 - Add executeId to job listings (yuudi) 50768 - Add core/du to measure local disk usage (Nick Craig-Wood) 50769 - Add operations/settier to API (Drew Stinnett) 50770 - rclone test info: Add --check-base32768 flag to check can store 50771 all base32768 characters (Nick Craig-Wood) 50772 - rmdirs: Remove directories concurrently controlled by --checkers 50773 (Nick Craig-Wood) 50774 - Bug Fixes 50775 - accounting: Don't stop calculating average transfer speed until 50776 the operation is complete (Jacob Hands) 50777 - fs: Fix transferTime not being set in JSON logs (Jacob Hands) 50778 - fshttp: Fix --bind 0.0.0.0 allowing IPv6 and --bind ::0 allowing 50779 IPv4 (Nick Craig-Wood) 50780 - operations: Fix overlapping check on case insensitive file 50781 systems (Nick Craig-Wood) 50782 - serve dlna: Fix MIME type if backend can't identify it (Nick 50783 Craig-Wood) 50784 - serve ftp: Fix race condition when using the auth proxy (Nick 50785 Craig-Wood) 50786 - serve sftp: Fix hash calculations with --vfs-cache-mode full 50787 (Nick Craig-Wood) 50788 - serve webdav: Fix error: Expecting fs.Object or fs.Directory, 50789 got nil (Nick Craig-Wood) 50790 - sync: Fix lockup with --cutoff-mode=soft and --max-duration 50791 (Nick Craig-Wood) 50792 - Mount 50793 - fix: Mount parsing for linux (Anagh Kumar Baranwal) 50794 - VFS 50795 - Add --vfs-cache-min-free-space to control minimum free space on 50796 the disk containing the cache (Nick Craig-Wood) 50797 - Added cache cleaner for directories to reduce memory usage 50798 (Anagh Kumar Baranwal) 50799 - Update parent directory modtimes on vfs actions (David Pedersen) 50800 - Keep virtual directory status accurate and reduce deadlock 50801 potential (Anagh Kumar Baranwal) 50802 - Make sure struct field is aligned for atomic access (Roberto 50803 Ricci) 50804 - Local 50805 - Rmdir return an error if the path is not a dir (zjx20) 50806 - Azure Blob 50807 - Implement OpenChunkWriter and multi-thread uploads (Nick 50808 Craig-Wood) 50809 - Fix creation of directory markers (Nick Craig-Wood) 50810 - Fix purging with directory markers (Nick Craig-Wood) 50811 - B2 50812 - Implement OpenChunkWriter and multi-thread uploads (Nick 50813 Craig-Wood) 50814 - Fix rclone link when object path contains special characters 50815 (Alishan Ladhani) 50816 - Box 50817 - Add polling support (David Sze) 50818 - Add --box-impersonate to impersonate a user ID (Nick Craig-Wood) 50819 - Fix unhelpful decoding of error messages into decimal numbers 50820 (Nick Craig-Wood) 50821 - Chunker 50822 - Update documentation to mention issue with small files (Ricardo 50823 D'O. Albanus) 50824 - Compress 50825 - Fix ChangeNotify (Nick Craig-Wood) 50826 - Drive 50827 - Add --drive-fast-list-bug-fix to control ListR bug workaround 50828 (Nick Craig-Wood) 50829 - Fichier 50830 - Implement DirMove (Nick Craig-Wood) 50831 - Fix error code parsing (alexia) 50832 - FTP 50833 - Add socks_proxy support for SOCKS5 proxies (Zach) 50834 - Fix 425 "TLS session of data connection not resumed" errors 50835 (Nick Craig-Wood) 50836 - Hdfs 50837 - Retry "replication in progress" errors when uploading (Nick 50838 Craig-Wood) 50839 - Fix uploading to the wrong object on Update with overridden 50840 remote name (Nick Craig-Wood) 50841 - HTTP 50842 - CORS should not be sent if not set (yuudi) 50843 - Fix webdav OPTIONS response (yuudi) 50844 - Opendrive 50845 - Fix List on a just deleted and remade directory (Nick 50846 Craig-Wood) 50847 - Oracleobjectstorage 50848 - Use rclone's rate limiter in multipart transfers (Manoj Ghosh) 50849 - Implement OpenChunkWriter and multi-thread uploads (Manoj Ghosh) 50850 - S3 50851 - Refactor multipart upload to use OpenChunkWriter and ChunkWriter 50852 (Vitor Gomes) 50853 - Factor generic multipart upload into lib/multipart (Nick 50854 Craig-Wood) 50855 - Fix purging of root directory with --s3-directory-markers (Nick 50856 Craig-Wood) 50857 - Add rclone backend set command to update the running config 50858 (Nick Craig-Wood) 50859 - Add rclone backend restore-status command (Nick Craig-Wood) 50860 - SFTP 50861 - Stop uploads re-using the same ssh connection to improve 50862 performance (Nick Craig-Wood) 50863 - Add --sftp-ssh to specify an external ssh binary to use (Nick 50864 Craig-Wood) 50865 - Add socks_proxy support for SOCKS5 proxies (Zach) 50866 - Support dynamic --sftp-path-override (nielash) 50867 - Fix spurious warning when using --sftp-ssh (Nick Craig-Wood) 50868 - Smb 50869 - Implement multi-threaded writes for copies to smb (Edwin 50870 Mackenzie-Owen) 50871 - Storj 50872 - Performance improvement for large file uploads (Kaloyan Raev) 50873 - Swift 50874 - Fix HEADing 0-length objects when --swift-no-large-objects set 50875 (Julian Lepinski) 50876 - Union 50877 - Add :writback to act as a simple cache (Nick Craig-Wood) 50878 - WebDAV 50879 - Nextcloud: fix segment violation in low-level retry (Paul) 50880 - Zoho 50881 - Remove Range requests workarounds to fix integration tests (Nick 50882 Craig-Wood) 50883 50884 v1.63.1 - 2023-07-17 50885 50886 See commits 50887 50888 - Bug Fixes 50889 - build: Fix macos builds for versions < 12 (Anagh Kumar Baranwal) 50890 - dirtree: Fix performance with large directories of directories 50891 and --fast-list (Nick Craig-Wood) 50892 - operations 50893 - Fix deadlock when using lsd/ls with --progress (Nick 50894 Craig-Wood) 50895 - Fix .rclonelink files not being converted back to symlinks 50896 (Nick Craig-Wood) 50897 - doc fixes (Dean Attali, Mahad, Nick Craig-Wood, Sawada 50898 Tsunayoshi, Vladislav Vorobev) 50899 - Local 50900 - Fix partial directory read for corrupted filesystem (Nick 50901 Craig-Wood) 50902 - Box 50903 - Fix reconnect failing with HTTP 400 Bad Request (albertony) 50904 - Smb 50905 - Fix "Statfs failed: bucket or container name is needed" when 50906 mounting (Nick Craig-Wood) 50907 - WebDAV 50908 - Nextcloud: fix must use /dav/files/USER endpoint not /webdav 50909 error (Paul) 50910 - Nextcloud chunking: add more guidance for the user to check the 50911 config (darix) 50912 50913 v1.63.0 - 2023-06-30 50914 50915 See commits 50916 50917 - New backends 50918 - Pikpak (wiserain) 50919 - New S3 providers 50920 - petabox.io (Andrei Smirnov) 50921 - Google Cloud Storage (Anthony Pessy) 50922 - New WebDAV providers 50923 - Fastmail (Arnavion) 50924 - Major changes 50925 - Files will be copied to a temporary name ending in .partial when 50926 copying to local,ftp,sftp then renamed at the end of the 50927 transfer. (Janne Hellsten, Nick Craig-Wood) 50928 - This helps with data integrity as we don't delete the 50929 existing file until the new one is complete. 50930 - It can be disabled with the --inplace flag. 50931 - This behaviour will also happen if the backend is wrapped, 50932 for example sftp wrapped with crypt. 50933 - The s3, azureblob and gcs backends now support directory markers 50934 so empty directories are supported (Jānis Bebrītis, Nick 50935 Craig-Wood) 50936 - The --default-time flag now controls the unknown modification 50937 time of files/dirs (Nick Craig-Wood) 50938 - If a file or directory does not have a modification time 50939 rclone can read then rclone will display this fixed time 50940 instead. 50941 - For the old behaviour use --default-time 0s which will set 50942 this time to the time rclone started up. 50943 - New Features 50944 - build 50945 - Modernise linters in use and fixup all affected code 50946 (albertony) 50947 - Push docker beta to GHCR (GitHub container registry) 50948 (Richard Tweed) 50949 - cat: Add --separator option to cat command (Loren Gordon) 50950 - config 50951 - Do not remove/overwrite other files during config file save 50952 (albertony) 50953 - Do not overwrite config file symbolic link (albertony) 50954 - Stop config create making invalid config files (Nick 50955 Craig-Wood) 50956 - doc updates (Adam K, Aditya Basu, albertony, asdffdsazqqq, Damo, 50957 danielkrajnik, Dimitri Papadopoulos, dlitster, Drew Parsons, 50958 jumbi77, kapitainsky, mac-15, Mariusz Suchodolski, Nick 50959 Craig-Wood, NickIAm, Rintze Zelle, Stanislav Gromov, Tareq 50960 Sharafy, URenko, yuudi, Zach Kipp) 50961 - fs 50962 - Add size to JSON logs when moving or copying an object (Nick 50963 Craig-Wood) 50964 - Allow boolean features to be enabled with --disable !Feature 50965 (Nick Craig-Wood) 50966 - genautocomplete: Rename to completion with alias to the old name 50967 (Nick Craig-Wood) 50968 - librclone: Added example on using librclone with Go (alankrit) 50969 - lsjson: Make --stat more efficient (Nick Craig-Wood) 50970 - operations 50971 - Implement --multi-thread-write-buffer-size for speed 50972 improvements on downloads (Paulo Schreiner) 50973 - Reopen downloads on error when using check --download and 50974 cat (Nick Craig-Wood) 50975 - rc: config/listremotes includes remotes defined with environment 50976 variables (kapitainsky) 50977 - selfupdate: Obey --no-check-certificate flag (Nick Craig-Wood) 50978 - serve restic: Trigger systemd notify (Shyim) 50979 - serve webdav: Implement owncloud checksum and modtime extensions 50980 (WeidiDeng) 50981 - sync: --suffix-keep-extension preserve 2 part extensions like 50982 .tar.gz (Nick Craig-Wood) 50983 - Bug Fixes 50984 - accounting 50985 - Fix Prometheus metrics to be the same as core/stats (Nick 50986 Craig-Wood) 50987 - Bwlimit signal handler should always start (Sam Lai) 50988 - bisync: Fix maxDelete parameter being ignored via the rc (Nick 50989 Craig-Wood) 50990 - cmd/ncdu: Fix screen corruption when logging (eNV25) 50991 - filter: Fix deadlock with errors on --files-from (douchen) 50992 - fs 50993 - Fix interaction between --progress and --interactive (Nick 50994 Craig-Wood) 50995 - Fix infinite recursive call in pacer ModifyCalculator (fixes 50996 issue reported by the staticcheck linter) (albertony) 50997 - lib/atexit: Ensure OnError only calls cancel function once (Nick 50998 Craig-Wood) 50999 - lib/rest: Fix problems re-using HTTP connections (Nick 51000 Craig-Wood) 51001 - rc 51002 - Fix operations/stat with trailing / (Nick Craig-Wood) 51003 - Fix missing --rc flags (Nick Craig-Wood) 51004 - Fix output of Time values in options/get (Nick Craig-Wood) 51005 - serve dlna: Fix potential data race (Nick Craig-Wood) 51006 - version: Fix reported os/kernel version for windows (albertony) 51007 - Mount 51008 - Add --mount-case-insensitive to force the mount to be case 51009 insensitive (Nick Craig-Wood) 51010 - Removed unnecessary byte slice allocation for reads (Anagh Kumar 51011 Baranwal) 51012 - Clarify rclone mount error when installed via homebrew (Nick 51013 Craig-Wood) 51014 - Added _netdev to the example mount so it gets treated as a 51015 remote-fs rather than local-fs (Anagh Kumar Baranwal) 51016 - Mount2 51017 - Updated go-fuse version (Anagh Kumar Baranwal) 51018 - Fixed statfs (Anagh Kumar Baranwal) 51019 - Disable xattrs (Anagh Kumar Baranwal) 51020 - VFS 51021 - Add MkdirAll function to make a directory and all beneath (Nick 51022 Craig-Wood) 51023 - Fix reload: failed to add virtual dir entry: file does not exist 51024 (Nick Craig-Wood) 51025 - Fix writing to a read only directory creating spurious directory 51026 entries (WeidiDeng) 51027 - Fix potential data race (Nick Craig-Wood) 51028 - Fix backends being Shutdown too early when startup takes a long 51029 time (Nick Craig-Wood) 51030 - Local 51031 - Fix filtering of symlinks with -l/--links flag (Nick Craig-Wood) 51032 - Fix /path/to/file.rclonelink when -l/--links is in use (Nick 51033 Craig-Wood) 51034 - Fix crash with --metadata on Android (Nick Craig-Wood) 51035 - Cache 51036 - Fix backends shutting down when in use when used via the rc 51037 (Nick Craig-Wood) 51038 - Crypt 51039 - Add --crypt-suffix option to set a custom suffix for encrypted 51040 files (jladbrook) 51041 - Add --crypt-pass-bad-blocks to allow corrupted file output (Nick 51042 Craig-Wood) 51043 - Fix reading 0 length files (Nick Craig-Wood) 51044 - Try not to return "unexpected EOF" error (Nick Craig-Wood) 51045 - Reduce allocations (albertony) 51046 - Recommend Dropbox for base32768 encoding (Nick Craig-Wood) 51047 - Azure Blob 51048 - Empty directory markers (Nick Craig-Wood) 51049 - Support azure workload identities (Tareq Sharafy) 51050 - Fix azure blob uploads with multiple bits of metadata (Nick 51051 Craig-Wood) 51052 - Fix azurite compatibility by sending nil tier if set to empty 51053 string (Roel Arents) 51054 - Combine 51055 - Implement missing methods (Nick Craig-Wood) 51056 - Fix goroutine stack overflow on bad object (Nick Craig-Wood) 51057 - Drive 51058 - Add --drive-env-auth to get IAM credentials from runtime (Peter 51059 Brunner) 51060 - Update drive service account guide (Juang, Yi-Lin) 51061 - Fix change notify picking up files outside the root (Nick 51062 Craig-Wood) 51063 - Fix trailing slash mis-identificaton of folder as file (Nick 51064 Craig-Wood) 51065 - Fix incorrect remote after Update on object (Nick Craig-Wood) 51066 - Dropbox 51067 - Implement --dropbox-pacer-min-sleep flag (Nick Craig-Wood) 51068 - Fix the dropbox batcher stalling (Misty) 51069 - Fichier 51070 - Add --ficicher-cdn option to use the CDN for download (Nick 51071 Craig-Wood) 51072 - FTP 51073 - Lower log message priority when SetModTime is not supported to 51074 debug (Tobias Gion) 51075 - Fix "unsupported LIST line" errors on startup (Nick Craig-Wood) 51076 - Fix "501 Not a valid pathname." errors when creating directories 51077 (Nick Craig-Wood) 51078 - Google Cloud Storage 51079 - Empty directory markers (Jānis Bebrītis, Nick Craig-Wood) 51080 - Added --gcs-user-project needed for requester pays (Christopher 51081 Merry) 51082 - HTTP 51083 - Add client certificate user auth middleware. This can auth 51084 serve restic from the username in the client cert. (Peter Fern) 51085 - Jottacloud 51086 - Fix vfs writeback stuck in a failed upload loop with file 51087 versioning disabled (albertony) 51088 - Onedrive 51089 - Add --onedrive-av-override flag to download files flagged as 51090 virus (Nick Craig-Wood) 51091 - Fix quickxorhash on 32 bit architectures (Nick Craig-Wood) 51092 - Report any list errors during rclone cleanup (albertony) 51093 - Putio 51094 - Fix uploading to the wrong object on Update with overridden 51095 remote name (Nick Craig-Wood) 51096 - Fix modification times not being preserved for server side copy 51097 and move (Nick Craig-Wood) 51098 - Fix server side copy failures (400 errors) (Nick Craig-Wood) 51099 - S3 51100 - Empty directory markers (Jānis Bebrītis, Nick Craig-Wood) 51101 - Update Scaleway storage classes (Brian Starkey) 51102 - Fix --s3-versions on individual objects (Nick Craig-Wood) 51103 - Fix hang on aborting multipart upload with iDrive e2 (Nick 51104 Craig-Wood) 51105 - Fix missing "tier" metadata (Nick Craig-Wood) 51106 - Fix V3sign: add missing subresource delete (cc) 51107 - Fix Arvancloud Domain and region changes and alphabetise the 51108 provider (Ehsan Tadayon) 51109 - Fix Qiniu KODO quirks virtualHostStyle is false (zzq) 51110 - SFTP 51111 - Add --sftp-host-key-algorithms to allow specifying SSH host key 51112 algorithms (Joel) 51113 - Fix using --sftp-key-use-agent and --sftp-key-file together 51114 needing private key file (Arnav Singh) 51115 - Fix move to allow overwriting existing files (Nick Craig-Wood) 51116 - Don't stat directories before listing them (Nick Craig-Wood) 51117 - Don't check remote points to a file if it ends with / (Nick 51118 Craig-Wood) 51119 - Sharefile 51120 - Disable streamed transfers as they no longer work (Nick 51121 Craig-Wood) 51122 - Smb 51123 - Code cleanup to avoid overwriting ctx before first use (fixes 51124 issue reported by the staticcheck linter) (albertony) 51125 - Storj 51126 - Fix "uplink: too many requests" errors when uploading to the 51127 same file (Nick Craig-Wood) 51128 - Fix uploading to the wrong object on Update with overridden 51129 remote name (Nick Craig-Wood) 51130 - Swift 51131 - Ignore 404 error when deleting an object (Nick Craig-Wood) 51132 - Union 51133 - Implement missing methods (Nick Craig-Wood) 51134 - Allow errors to be unwrapped for inspection (Nick Craig-Wood) 51135 - Uptobox 51136 - Add --uptobox-private flag to make all uploaded files private 51137 (Nick Craig-Wood) 51138 - Fix improper regex (Aaron Gokaslan) 51139 - Fix Update returning the wrong object (Nick Craig-Wood) 51140 - Fix rmdir declaring that directories weren't empty (Nick 51141 Craig-Wood) 51142 - WebDAV 51143 - nextcloud: Add support for chunked uploads (Paul) 51144 - Set modtime using propset for owncloud and nextcloud (WeidiDeng) 51145 - Make pacer minSleep configurable with --webdav-pacer-min-sleep 51146 (ed) 51147 - Fix server side copy/move not overwriting (WeidiDeng) 51148 - Fix modtime on server side copy for owncloud and nextcloud (Nick 51149 Craig-Wood) 51150 - Yandex 51151 - Fix 400 Bad Request on transfer failure (Nick Craig-Wood) 51152 - Zoho 51153 - Fix downloads with Range: header returning the wrong data (Nick 51154 Craig-Wood) 51155 51156 v1.62.2 - 2023-03-16 51157 51158 See commits 51159 51160 - Bug Fixes 51161 - docker volume plugin: Add missing fuse3 dependency (Nick 51162 Craig-Wood) 51163 - docs: Fix size documentation (asdffdsazqqq) 51164 - FTP 51165 - Fix 426 errors on downloads with vsftpd (Lesmiscore) 51166 51167 v1.62.1 - 2023-03-15 51168 51169 See commits 51170 51171 - Bug Fixes 51172 - docker: Add missing fuse3 dependency (cycneuramus) 51173 - build: Update release docs to be more careful with the tag (Nick 51174 Craig-Wood) 51175 - build: Set Github release to draft while uploading binaries 51176 (Nick Craig-Wood) 51177 51178 v1.62.0 - 2023-03-14 51179 51180 See commits 51181 51182 - New Features 51183 - accounting: Make checkers show what they are doing (Nick 51184 Craig-Wood) 51185 - authorize: Add support for custom templates (Hunter Wittenborn) 51186 - build 51187 - Update to go1.20 (Nick Craig-Wood, Anagh Kumar Baranwal) 51188 - Add winget releaser workflow (Ryan Caezar Itang) 51189 - Add dependabot (Ryan Caezar Itang) 51190 - doc updates (albertony, Bryan Kaplan, Gerard Bosch, 51191 IMTheNachoMan, Justin Winokur, Manoj Ghosh, Nick Craig-Wood, Ole 51192 Frost, Peter Brunner, piyushgarg, Ryan Caezar Itang, Simmon Li, 51193 ToBeFree) 51194 - filter: Emit INFO message when can't work out directory filters 51195 (Nick Craig-Wood) 51196 - fs 51197 - Added multiple ca certificate support. (alankrit) 51198 - Add --max-delete-size a delete size threshold (Leandro 51199 Sacchet) 51200 - fspath: Allow the symbols @ and + in remote names (albertony) 51201 - lib/terminal: Enable windows console virtual terminal sequences 51202 processing (ANSI/VT100 colors) (albertony) 51203 - move: If --check-first and --order-by are set then delete with 51204 perfect ordering (Nick Craig-Wood) 51205 - serve http: Support --auth-proxy (Matthias Baur) 51206 - Bug Fixes 51207 - accounting 51208 - Avoid negative ETA values for very slow speeds (albertony) 51209 - Limit length of ETA string (albertony) 51210 - Show human readable elapsed time when longer than a day 51211 (albertony) 51212 - all: Apply codeql fixes (Aaron Gokaslan) 51213 - build 51214 - Fix condition for manual workflow run (albertony) 51215 - Fix building for ARMv5 and ARMv6 (albertony) 51216 - selfupdate: Consider ARM version 51217 - install.sh: fix ARMv6 download 51218 - version: Report ARM version 51219 - deletefile: Return error code 4 if file does not exist (Nick 51220 Craig-Wood) 51221 - docker: Fix volume plugin does not remount volume on docker 51222 restart (logopk) 51223 - fs: Fix race conditions in --max-delete and --max-delete-size 51224 (Nick Craig-Wood) 51225 - lib/oauthutil: Handle fatal errors better (Alex Chen) 51226 - mount2: Fix --allow-non-empty (Nick Craig-Wood) 51227 - operations: Fix concurrency: use --checkers unless transferring 51228 files (Nick Craig-Wood) 51229 - serve ftp: Fix timestamps older than 1 year in listings (Nick 51230 Craig-Wood) 51231 - sync: Fix concurrency: use --checkers unless transferring files 51232 (Nick Craig-Wood) 51233 - tree 51234 - Fix nil pointer exception on stat failure (Nick Craig-Wood) 51235 - Fix colored output on windows (albertony) 51236 - Fix display of files with illegal Windows file system names 51237 (Nick Craig-Wood) 51238 - Mount 51239 - Fix creating and renaming files on case insensitive backends 51240 (Nick Craig-Wood) 51241 - Do not treat \\?\ prefixed paths as network share paths on 51242 windows (albertony) 51243 - Fix check for empty mount point on Linux (Nick Craig-Wood) 51244 - Fix --allow-non-empty (Nick Craig-Wood) 51245 - Avoid incorrect or premature overlap check on windows 51246 (albertony) 51247 - Update to fuse3 after bazil.org/fuse update (Nick Craig-Wood) 51248 - VFS 51249 - Make uploaded files retain modtime with non-modtime backends 51250 (Nick Craig-Wood) 51251 - Fix incorrect modtime on fs which don't support setting modtime 51252 (Nick Craig-Wood) 51253 - Fix rename of directory containing files to be uploaded (Nick 51254 Craig-Wood) 51255 - Local 51256 - Fix %!w(<nil>) in "failed to read directory" error (Marks 51257 Polakovs) 51258 - Fix exclusion of dangling symlinks with -L/--copy-links (Nick 51259 Craig-Wood) 51260 - Crypt 51261 - Obey --ignore-checksum (Nick Craig-Wood) 51262 - Fix for unencrypted directory names on case insensitive remotes 51263 (Ole Frost) 51264 - Azure Blob 51265 - Remove workarounds for SDK bugs after v0.6.1 update (Nick 51266 Craig-Wood) 51267 - B2 51268 - Fix uploading files bigger than 1TiB (Nick Craig-Wood) 51269 - Drive 51270 - Note that --drive-acknowledge-abuse needs SA Manager permission 51271 (Nick Craig-Wood) 51272 - Make --drive-stop-on-upload-limit to respond to 51273 storageQuotaExceeded (Ninh Pham) 51274 - FTP 51275 - Retry 426 errors (Nick Craig-Wood) 51276 - Retry errors when initiating downloads (Nick Craig-Wood) 51277 - Revert to upstream github.com/jlaffaye/ftp now fix is merged 51278 (Nick Craig-Wood) 51279 - Google Cloud Storage 51280 - Add --gcs-env-auth to pick up IAM credentials from env/instance 51281 (Peter Brunner) 51282 - Mega 51283 - Add --mega-use-https flag (NodudeWasTaken) 51284 - Onedrive 51285 - Default onedrive personal to QuickXorHash as Microsoft is 51286 removing SHA1 (Nick Craig-Wood) 51287 - Add --onedrive-hash-type to change the hash in use (Nick 51288 Craig-Wood) 51289 - Improve speed of QuickXorHash (LXY) 51290 - Oracle Object Storage 51291 - Speed up operations by using S3 pacer and setting minsleep to 51292 10ms (Manoj Ghosh) 51293 - Expose the storage_tier option in config (Manoj Ghosh) 51294 - Bring your own encryption keys (Manoj Ghosh) 51295 - S3 51296 - Check multipart upload ETag when --s3-no-head is in use (Nick 51297 Craig-Wood) 51298 - Add --s3-sts-endpoint to specify STS endpoint (Nick Craig-Wood) 51299 - Fix incorrect tier support for StorJ and IDrive when pointing at 51300 a file (Ole Frost) 51301 - Fix AWS STS failing if --s3-endpoint is set (Nick Craig-Wood) 51302 - Make purge remove directory markers too (Nick Craig-Wood) 51303 - Seafile 51304 - Renew library password (Fred) 51305 - SFTP 51306 - Fix uploads being 65% slower than they should be with crypt 51307 (Nick Craig-Wood) 51308 - Smb 51309 - Allow SPN (service principal name) to be configured (Nick 51310 Craig-Wood) 51311 - Check smb connection is closed (happyxhw) 51312 - Storj 51313 - Implement rclone link (Kaloyan Raev) 51314 - Implement rclone purge (Kaloyan Raev) 51315 - Update satellite urls and labels (Kaloyan Raev) 51316 - WebDAV 51317 - Fix interop with davrods server (Nick Craig-Wood) 51318 51319 v1.61.1 - 2022-12-23 51320 51321 See commits 51322 51323 - Bug Fixes 51324 - docs: 51325 - Show only significant parts of version number in version 51326 introduced label (albertony) 51327 - Fix unescaped HTML (Nick Craig-Wood) 51328 - lib/http: Shutdown all servers on exit to remove unix socket 51329 (Nick Craig-Wood) 51330 - rc: Fix --rc-addr flag (which is an alternate for --url) (Anagh 51331 Kumar Baranwal) 51332 - serve restic 51333 - Don't serve via http if serving via --stdio (Nick 51334 Craig-Wood) 51335 - Fix immediate exit when not using stdio (Nick Craig-Wood) 51336 - serve webdav 51337 - Fix --baseurl handling after lib/http refactor (Nick 51338 Craig-Wood) 51339 - Fix running duplicate Serve call (Nick Craig-Wood) 51340 - Azure Blob 51341 - Fix "409 Public access is not permitted on this storage account" 51342 (Nick Craig-Wood) 51343 - S3 51344 - storj: Update endpoints (Kaloyan Raev) 51345 51346 v1.61.0 - 2022-12-20 51347 51348 See commits 51349 51350 - New backends 51351 - New S3 providers 51352 - Liara LOS (MohammadReza) 51353 - New Features 51354 - build: Add vulnerability testing using govulncheck (albertony) 51355 - cmd: Enable SIGINFO (Ctrl-T) handler on FreeBSD, NetBSD, OpenBSD 51356 and Dragonfly BSD (x3-apptech) 51357 - config: Add config/setpath for setting config path via 51358 rc/librclone (Nick Craig-Wood) 51359 - dedupe 51360 - Count Checks in the stats while scanning for duplicates 51361 (Nick Craig-Wood) 51362 - Make dedupe obey the filters (Nick Craig-Wood) 51363 - dlna: Properly attribute code used from 51364 https://github.com/anacrolix/dms (Nick Craig-Wood) 51365 - docs 51366 - Add minimum versions and status badges to backend and 51367 command docs (Nick Craig-Wood, albertony) 51368 - Remote names may not start or end with space (albertony) 51369 - filter: Add metadata filters --metadata-include/exclude/filter 51370 and friends (Nick Craig-Wood) 51371 - fs 51372 - Make all duration flags take y, M, w, d etc suffixes (Nick 51373 Craig-Wood) 51374 - Add global flag --color to control terminal colors (Kevin 51375 Verstaen) 51376 - fspath: Allow unicode numbers and letters in remote names 51377 (albertony) 51378 - lib/file: Improve error message for creating dir on non-existent 51379 network host on windows (albertony) 51380 - lib/http: Finish port of rclone servers to lib/http (Tom 51381 Mombourquette, Nick Craig-Wood) 51382 - lib/oauthutil: Improved usability of config flows needing web 51383 browser (Ole Frost) 51384 - ncdu 51385 - Add support for modification time (albertony) 51386 - Fallback to sort by name also for sort by average size 51387 (albertony) 51388 - Rework to use tcell directly instead of the termbox wrapper 51389 (eNV25) 51390 - rc: Add commands to set GC Percent & Memory Limit (go 1.19+) 51391 (Anagh Kumar Baranwal) 51392 - rcat: Preserve metadata when Copy falls back to Rcat (Nick 51393 Craig-Wood) 51394 - rcd: Refactor rclone rc server to use lib/http (Nick Craig-Wood) 51395 - rcserver: Avoid generating default credentials with htpasswd 51396 (Kamui) 51397 - restic: Refactor to use lib/http (Nolan Woods) 51398 - serve http: Support unix sockets and multiple listeners (Tom 51399 Mombourquette) 51400 - serve webdav: Refactor to use lib/http (Nick Craig-Wood) 51401 - test: Replace defer cleanup with t.Cleanup (Eng Zer Jun) 51402 - test memory: Read metadata if -M flag is specified (Nick 51403 Craig-Wood) 51404 - wasm: Comply with wasm_exec.js licence terms (Matthew Vernon) 51405 - Bug Fixes 51406 - build: Update golang.org/x/net/http2 to fix GO-2022-1144 (Nick 51407 Craig-Wood) 51408 - restic: Fix typo in docs 'remove' should be 'remote' 51409 (asdffdsazqqq) 51410 - serve dlna: Fix panic: Logger uninitialized. (Nick Craig-Wood) 51411 - Mount 51412 - Update cgofuse for FUSE-T support for mounting volumes on Mac 51413 (Nick Craig-Wood) 51414 - VFS 51415 - Windows: fix slow opening of exe files by not truncating files 51416 when not necessary (Nick Craig-Wood) 51417 - Fix IO Error opening a file with O_CREATE|O_RDONLY in 51418 --vfs-cache-mode not full (Nick Craig-Wood) 51419 - Crypt 51420 - Fix compress wrapping crypt giving upload errors (Nick 51421 Craig-Wood) 51422 - Azure Blob 51423 - Port to new SDK (Nick Craig-Wood) 51424 - Revamp authentication to include all methods and docs (Nick 51425 Craig-Wood) 51426 - Port old authentication methods to new SDK (Nick Craig-Wood, 51427 Brad Ackerman) 51428 - Thanks to Stonebranch for sponsoring this work. 51429 - Add --azureblob-no-check-container to assume container exists 51430 (Nick Craig-Wood) 51431 - Add --use-server-modtime support (Abdullah Saglam) 51432 - Add support for custom upload headers (rkettelerij) 51433 - Allow emulator account/key override (Roel Arents) 51434 - Support simple "environment credentials" (Nathaniel Wesley 51435 Filardo) 51436 - Ignore AuthorizationFailure when trying to create a create a 51437 container (Nick Craig-Wood) 51438 - Box 51439 - Added note on Box API rate limits (Ole Frost) 51440 - Drive 51441 - Handle shared drives with leading/trailing space in name 51442 (related to) (albertony) 51443 - FTP 51444 - Update help text of implicit/explicit TLS options to refer to 51445 FTPS instead of FTP (ycdtosa) 51446 - Improve performance to speed up --files-from and NewObject 51447 (Anthony Pessy) 51448 - HTTP 51449 - Parse GET responses when no_head is set (Arnie97) 51450 - Do not update object size based on Range requests (Arnie97) 51451 - Support Content-Range response header (Arnie97) 51452 - Onedrive 51453 - Document workaround for shared with me files (vanplus) 51454 - S3 51455 - Add Liara LOS to provider list (MohammadReza) 51456 - Add DigitalOcean Spaces regions sfo3, fra1, syd1 (Jack) 51457 - Avoid privileged GetBucketLocation to resolve s3 region (Anthony 51458 Pessy) 51459 - Stop setting object and bucket ACL to private if it is an empty 51460 string (Philip Harvey) 51461 - If bucket or object ACL is empty string then don't add 51462 X-Amz-Acl: header (Nick Craig-Wood) 51463 - Reduce memory consumption for s3 objects (Erik Agterdenbos) 51464 - Fix listing loop when using v2 listing on v1 server (Nick 51465 Craig-Wood) 51466 - Fix nil pointer exception when using Versions (Nick Craig-Wood) 51467 - Fix excess memory usage when using versions (Nick Craig-Wood) 51468 - Ignore versionIDs from uploads unless using --s3-versions or 51469 --s3-versions-at (Nick Craig-Wood) 51470 - SFTP 51471 - Add configuration options to set ssh Ciphers / MACs / 51472 KeyExchange (dgouju) 51473 - Auto-detect shell type for fish (albertony) 51474 - Fix NewObject with leading / (Nick Craig-Wood) 51475 - Smb 51476 - Fix issue where spurious dot directory is created (albertony) 51477 - Storj 51478 - Implement server side Copy (Kaloyan Raev) 51479 51480 v1.60.1 - 2022-11-17 51481 51482 See commits 51483 51484 - Bug Fixes 51485 - lib/cache: Fix alias backend shutting down too soon (Nick 51486 Craig-Wood) 51487 - wasm: Fix walltime link error by adding up-to-date wasm_exec.js 51488 (João Henrique Franco) 51489 - docs 51490 - Update faq.md with bisync (Samuel Johnson) 51491 - Corrected download links in windows install docs 51492 (coultonluke) 51493 - Add direct download link for windows arm64 (albertony) 51494 - Remove link to rclone slack as it is no longer supported 51495 (Nick Craig-Wood) 51496 - Faq: how to use a proxy server that requires a username and 51497 password (asdffdsazqqq) 51498 - Oracle-object-storage: doc fix (Manoj Ghosh) 51499 - Fix typo remove in rclone_serve_restic command (Joda Stößer) 51500 - Fix character that was incorrectly interpreted as markdown 51501 (Clément Notin) 51502 - VFS 51503 - Fix deadlock caused by cache cleaner and upload finishing (Nick 51504 Craig-Wood) 51505 - Local 51506 - Clean absolute paths (albertony) 51507 - Fix -L/--copy-links with filters missing directories (Nick 51508 Craig-Wood) 51509 - Mailru 51510 - Note that an app password is now needed (Nick Craig-Wood) 51511 - Allow timestamps to be before the epoch 1970-01-01 (Nick 51512 Craig-Wood) 51513 - S3 51514 - Add provider quirk --s3-might-gzip to fix corrupted on transfer: 51515 sizes differ (Nick Craig-Wood) 51516 - Allow Storj to server side copy since it seems to work now (Nick 51517 Craig-Wood) 51518 - Fix for unchecked err value in s3 listv2 (Aaron Gokaslan) 51519 - Add additional Wasabi locations (techknowlogick) 51520 - Smb 51521 - Fix Failed to sync: context canceled at the end of syncs (Nick 51522 Craig-Wood) 51523 - WebDAV 51524 - Fix Move/Copy/DirMove when using -server-side-across-configs 51525 (Nick Craig-Wood) 51526 51527 v1.60.0 - 2022-10-21 51528 51529 See commits 51530 51531 - New backends 51532 - Oracle object storage (Manoj Ghosh) 51533 - SMB / CIFS (Windows file sharing) (Lesmiscore) 51534 - New S3 providers 51535 - IONOS Cloud Storage (Dmitry Deniskin) 51536 - Qiniu KODO (Bachue Zhou) 51537 - New Features 51538 - build 51539 - Update to go1.19 and make go1.17 the minimum required 51540 version (Nick Craig-Wood) 51541 - Install.sh: fix arm-v7 download (Ole Frost) 51542 - fs: Warn the user when using an existing remote name without a 51543 colon (Nick Craig-Wood) 51544 - httplib: Add --xxx-min-tls-version option to select minimum TLS 51545 version for HTTP servers (Robert Newson) 51546 - librclone: Add PHP bindings and test program (Jordi Gonzalez 51547 Muñoz) 51548 - operations 51549 - Add --server-side-across-configs global flag for any backend 51550 (Nick Craig-Wood) 51551 - Optimise --copy-dest and --compare-dest (Nick Craig-Wood) 51552 - rc: add job/stopgroup to stop group (Evan Spensley) 51553 - serve dlna 51554 - Add --announce-interval to control SSDP Announce Interval 51555 (YanceyChiew) 51556 - Add --interface to Specify SSDP interface names line (Simon 51557 Bos) 51558 - Add support for more external subtitles (YanceyChiew) 51559 - Add verification of addresses (YanceyChiew) 51560 - sync: Optimise --copy-dest and --compare-dest (Nick Craig-Wood) 51561 - doc updates (albertony, Alexander Knorr, anonion, João Henrique 51562 Franco, Josh Soref, Lorenzo Milesi, Marco Molteni, Mark Trolley, 51563 Ole Frost, partev, Ryan Morey, Tom Mombourquette, YFdyh000) 51564 - Bug Fixes 51565 - filter 51566 - Fix incorrect filtering with UseFilter context flag and 51567 wrapping backends (Nick Craig-Wood) 51568 - Make sure we check --files-from when looking for a single 51569 file (Nick Craig-Wood) 51570 - rc 51571 - Fix mount/listmounts not returning the full Fs entered in 51572 mount/mount (Tom Mombourquette) 51573 - Handle external unmount when mounting (Isaac Aymerich) 51574 - Validate Daemon option is not set when mounting a volume via 51575 RC (Isaac Aymerich) 51576 - sync: Update docs and error messages to reflect fixes to overlap 51577 checks (Nick Naumann) 51578 - VFS 51579 - Reduce memory use by embedding sync.Cond (Nick Craig-Wood) 51580 - Reduce memory usage by re-ordering commonly used structures 51581 (Nick Craig-Wood) 51582 - Fix excess CPU used by VFS cache cleaner looping (Nick 51583 Craig-Wood) 51584 - Local 51585 - Obey file filters in listing to fix errors on excluded files 51586 (Nick Craig-Wood) 51587 - Fix "Failed to read metadata: function not implemented" on old 51588 Linux kernels (Nick Craig-Wood) 51589 - Compress 51590 - Fix crash due to nil metadata (Nick Craig-Wood) 51591 - Fix error handling to not use or return nil objects (Nick 51592 Craig-Wood) 51593 - Drive 51594 - Make --drive-stop-on-upload-limit obey quota exceeded error 51595 (Steve Kowalik) 51596 - FTP 51597 - Add --ftp-force-list-hidden option to show hidden items (Øyvind 51598 Heddeland Instefjord) 51599 - Fix hang when using ExplicitTLS to certain servers. (Nick 51600 Craig-Wood) 51601 - Google Cloud Storage 51602 - Add --gcs-endpoint flag and config parameter (Nick Craig-Wood) 51603 - Hubic 51604 - Remove backend as service has now shut down (Nick Craig-Wood) 51605 - Onedrive 51606 - Rename Onedrive(cn) 21Vianet to Vnet Group (Yen Hu) 51607 - Disable change notify in China region since it is not supported 51608 (Nick Craig-Wood) 51609 - S3 51610 - Implement --s3-versions flag to show old versions of objects if 51611 enabled (Nick Craig-Wood) 51612 - Implement --s3-version-at flag to show versions of objects at a 51613 particular time (Nick Craig-Wood) 51614 - Implement backend versioning command to get/set bucket 51615 versioning (Nick Craig-Wood) 51616 - Implement Purge to purge versions and backend cleanup-hidden 51617 (Nick Craig-Wood) 51618 - Add --s3-decompress flag to decompress gzip-encoded files (Nick 51619 Craig-Wood) 51620 - Add --s3-sse-customer-key-base64 to supply keys with binary data 51621 (Richard Bateman) 51622 - Try to keep the maximum precision in ModTime with 51623 --user-server-modtime (Nick Craig-Wood) 51624 - Drop binary metadata with an ERROR message as it can't be stored 51625 (Nick Craig-Wood) 51626 - Add --s3-no-system-metadata to suppress read and write of system 51627 metadata (Nick Craig-Wood) 51628 - SFTP 51629 - Fix directory creation races (Lesmiscore) 51630 - Swift 51631 - Add --swift-no-large-objects to reduce HEAD requests (Nick 51632 Craig-Wood) 51633 - Union 51634 - Propagate SlowHash feature to fix hasher interaction 51635 (Lesmiscore) 51636 51637 v1.59.2 - 2022-09-15 51638 51639 See commits 51640 51641 - Bug Fixes 51642 - config: Move locking to fix fatal error: concurrent map read and 51643 map write (Nick Craig-Wood) 51644 - Local 51645 - Disable xattr support if the filesystems indicates it is not 51646 supported (Nick Craig-Wood) 51647 - Azure Blob 51648 - Fix chunksize calculations producing too many parts (Nick 51649 Craig-Wood) 51650 - B2 51651 - Fix chunksize calculations producing too many parts (Nick 51652 Craig-Wood) 51653 - S3 51654 - Fix chunksize calculations producing too many parts (Nick 51655 Craig-Wood) 51656 51657 v1.59.1 - 2022-08-08 51658 51659 See commits 51660 51661 - Bug Fixes 51662 - accounting: Fix panic in core/stats-reset with unknown group 51663 (Nick Craig-Wood) 51664 - build: Fix android build after GitHub actions change (Nick 51665 Craig-Wood) 51666 - dlna: Fix SOAP action header parsing (Joram Schrijver) 51667 - docs: Fix links to mount command from install docs (albertony) 51668 - dropbox: Fix ChangeNotify was unable to decrypt errors (Nick 51669 Craig-Wood) 51670 - fs: Fix parsing of times and durations of the form "YYYY-MM-DD 51671 HH:MM:SS" (Nick Craig-Wood) 51672 - serve sftp: Fix checksum detection (Nick Craig-Wood) 51673 - sync: Add accidentally missed filter-sensitivity to --backup-dir 51674 option (Nick Naumann) 51675 - Combine 51676 - Fix docs showing remote= instead of upstreams= (Nick Craig-Wood) 51677 - Throw error if duplicate directory name is specified (Nick 51678 Craig-Wood) 51679 - Fix errors with backends shutting down while in use (Nick 51680 Craig-Wood) 51681 - Dropbox 51682 - Fix hang on quit with --dropbox-batch-mode off (Nick Craig-Wood) 51683 - Fix infinite loop on uploading a corrupted file (Nick 51684 Craig-Wood) 51685 - Internetarchive 51686 - Ignore checksums for files using the different method 51687 (Lesmiscore) 51688 - Handle hash symbol in the middle of filename (Lesmiscore) 51689 - Jottacloud 51690 - Fix working with whitelabel Elgiganten Cloud 51691 - Do not store username in config when using standard auth 51692 (albertony) 51693 - Mega 51694 - Fix nil pointer exception when bad node received (Nick 51695 Craig-Wood) 51696 - S3 51697 - Fix --s3-no-head panic: reflect: Elem of invalid type 51698 s3.PutObjectInput (Nick Craig-Wood) 51699 - SFTP 51700 - Fix issue with WS_FTP by working around failing RealPath 51701 (albertony) 51702 - Union 51703 - Fix duplicated files when using directories with leading / (Nick 51704 Craig-Wood) 51705 - Fix multiple files being uploaded when roots don't exist (Nick 51706 Craig-Wood) 51707 - Fix panic due to misalignment of struct field in 32 bit 51708 architectures (r-ricci) 51709 51710 v1.59.0 - 2022-07-09 51711 51712 See commits 51713 51714 - New backends 51715 - Combine multiple remotes in one directory tree (Nick Craig-Wood) 51716 - Hidrive (Ovidiu Victor Tatar) 51717 - Internet Archive (Lesmiscore (Naoya Ozaki)) 51718 - New S3 providers 51719 - ArvanCloud AOS (ehsantdy) 51720 - Cloudflare R2 (Nick Craig-Wood) 51721 - Huawei OBS (m00594701) 51722 - IDrive e2 (vyloy) 51723 - New commands 51724 - test makefile: Create a single file for testing (Nick 51725 Craig-Wood) 51726 - New Features 51727 - Metadata framework to read and write system and user metadata on 51728 backends (Nick Craig-Wood) 51729 - Implemented initially for local, s3 and internetarchive 51730 backends 51731 - --metadata/-M flag to control whether metadata is copied 51732 - --metadata-set flag to specify metadata for uploads 51733 - Thanks to Manz Solutions for sponsoring this work. 51734 - build 51735 - Update to go1.18 and make go1.16 the minimum required 51736 version (Nick Craig-Wood) 51737 - Update android go build to 1.18.x and NDK to 23.1.7779620 51738 (Nick Craig-Wood) 51739 - All windows binaries now no longer CGO (Nick Craig-Wood) 51740 - Add linux/arm/v6 to docker images (Nick Craig-Wood) 51741 - A huge number of fixes found with staticcheck (albertony) 51742 - Configurable version suffix independent of version number 51743 (albertony) 51744 - check: Implement --no-traverse and --no-unicode-normalization 51745 (Nick Craig-Wood) 51746 - config: Readability improvements (albertony) 51747 - copyurl: Add --header-filename to honor the HTTP header filename 51748 directive (J-P Treen) 51749 - filter: Allow multiple --exclude-if-present flags (albertony) 51750 - fshttp: Add --disable-http-keep-alives to disable HTTP Keep 51751 Alives (Nick Craig-Wood) 51752 - install.sh 51753 - Set the modes on the files and/or directories on macOS 51754 (Michael C Tiernan - MIT-Research Computing Project) 51755 - Pre verify sudo authorization -v before calling curl. 51756 (Michael C Tiernan - MIT-Research Computing Project) 51757 - lib/encoder: Add Semicolon encoding (Nick Craig-Wood) 51758 - lsf: Add metadata support with M flag (Nick Craig-Wood) 51759 - lsjson: Add --metadata/-M flag (Nick Craig-Wood) 51760 - ncdu 51761 - Implement multi selection (CrossR) 51762 - Replace termbox with tcell's termbox wrapper (eNV25) 51763 - Display correct path in delete confirmation dialog (Roberto 51764 Ricci) 51765 - operations 51766 - Speed up hash checking by aborting the other hash if first 51767 returns nothing (Nick Craig-Wood) 51768 - Use correct src/dst in some log messages (zzr93) 51769 - rcat: Check checksums by default like copy does (Nick 51770 Craig-Wood) 51771 - selfupdate: Replace deprecated x/crypto/openpgp package with 51772 ProtonMail/go-crypto (albertony) 51773 - serve ftp: Check --passive-port arguments are correct (Nick 51774 Craig-Wood) 51775 - size: Warn about inaccurate results when objects with unknown 51776 size (albertony) 51777 - sync: Overlap check is now filter-sensitive so --backup-dir can 51778 be in the root provided it is filtered (Nick) 51779 - test info: Check file name lengths using 1,2,3,4 byte unicode 51780 characters (Nick Craig-Wood) 51781 - test makefile(s): --sparse, --zero, --pattern, --ascii, 51782 --chargen flags to control file contents (Nick Craig-Wood) 51783 - Make sure we call the Shutdown method on backends (Martin 51784 Czygan) 51785 - Bug Fixes 51786 - accounting: Fix unknown length file transfers counting 3 51787 transfers each (buda) 51788 - ncdu: Fix issue where dir size is summed when file sizes are -1 51789 (albertony) 51790 - sync/copy/move 51791 - Fix --fast-list --create-empty-src-dirs and --exclude (Nick 51792 Craig-Wood) 51793 - Fix --max-duration and --cutoff-mode soft (Nick Craig-Wood) 51794 - Fix fs cache unpin (Martin Czygan) 51795 - Set proper exit code for errors that are not low-level retried 51796 (e.g. size/timestamp changing) (albertony) 51797 - Mount 51798 - Support windows/arm64 (may still be problems - see #5828) (Nick 51799 Craig-Wood) 51800 - Log IO errors at ERROR level (Nick Craig-Wood) 51801 - Ignore _netdev mount argument (Hugal31) 51802 - VFS 51803 - Add --vfs-fast-fingerprint for less accurate but faster 51804 fingerprints (Nick Craig-Wood) 51805 - Add --vfs-disk-space-total-size option to manually set the total 51806 disk space (Claudio Maradonna) 51807 - vfscache: Fix fatal error: sync: unlock of unlocked mutex error 51808 (Nick Craig-Wood) 51809 - Local 51810 - Fix parsing of --local-nounc flag (Nick Craig-Wood) 51811 - Add Metadata support (Nick Craig-Wood) 51812 - Crypt 51813 - Support metadata (Nick Craig-Wood) 51814 - Azure Blob 51815 - Calculate Chunksize/blocksize to stay below maxUploadParts 51816 (Leroy van Logchem) 51817 - Use chunksize lib to determine chunksize dynamically (Derek 51818 Battams) 51819 - Case insensitive access tier (Rob Pickerill) 51820 - Allow remote emulator (azurite) (Lorenzo Maiorfi) 51821 - B2 51822 - Add --b2-version-at flag to show file versions at time specified 51823 (SwazRGB) 51824 - Use chunksize lib to determine chunksize dynamically (Derek 51825 Battams) 51826 - Chunker 51827 - Mark as not supporting metadata (Nick Craig-Wood) 51828 - Compress 51829 - Support metadata (Nick Craig-Wood) 51830 - Drive 51831 - Make backend config -o config add a combined AllDrives: remote 51832 (Nick Craig-Wood) 51833 - Make --drive-shared-with-me work with shared drives (Nick 51834 Craig-Wood) 51835 - Add --drive-resource-key for accessing link-shared files (Nick 51836 Craig-Wood) 51837 - Add backend commands exportformats and importformats for 51838 debugging (Nick Craig-Wood) 51839 - Fix 404 errors on copy/server side copy objects from public 51840 folder (Nick Craig-Wood) 51841 - Update Internal OAuth consent screen docs (Phil Shackleton) 51842 - Moved root_folder_id to advanced section (Abhiraj) 51843 - Dropbox 51844 - Migrate from deprecated api (m8rge) 51845 - Add logs to show when poll interval limits are exceeded (Nick 51846 Craig-Wood) 51847 - Fix nil pointer exception on dropbox impersonate user not found 51848 (Nick Craig-Wood) 51849 - Fichier 51850 - Parse api error codes and them accordingly (buengese) 51851 - FTP 51852 - Add support for disable_utf8 option (Jason Zheng) 51853 - Revert to upstream github.com/jlaffaye/ftp from our fork (Nick 51854 Craig-Wood) 51855 - Google Cloud Storage 51856 - Add --gcs-no-check-bucket to minimise transactions and perms 51857 (Nick Gooding) 51858 - Add --gcs-decompress flag to decompress gzip-encoded files (Nick 51859 Craig-Wood) 51860 - by default these will be downloaded compressed (which 51861 previously failed) 51862 - Hasher 51863 - Support metadata (Nick Craig-Wood) 51864 - HTTP 51865 - Fix missing response when using custom auth handler (albertony) 51866 - Jottacloud 51867 - Add support for upload to custom device and mountpoint 51868 (albertony) 51869 - Always store username in config and use it to avoid initial API 51870 request (albertony) 51871 - Fix issue with server-side copy when destination is in trash 51872 (albertony) 51873 - Fix listing output of remote with special characters (albertony) 51874 - Mailru 51875 - Fix timeout by using int instead of time.Duration for keeping 51876 number of seconds (albertony) 51877 - Mega 51878 - Document using MEGAcmd to help with login failures (Art M. 51879 Gallagher) 51880 - Onedrive 51881 - Implement --poll-interval for onedrive (Hugo Laloge) 51882 - Add access scopes option (Sven Gerber) 51883 - Opendrive 51884 - Resolve lag and truncate bugs (Scott Grimes) 51885 - Pcloud 51886 - Fix about with no free space left (buengese) 51887 - Fix cleanup (buengese) 51888 - S3 51889 - Use PUT Object instead of presigned URLs to upload single part 51890 objects (Nick Craig-Wood) 51891 - Backend restore command to skip non-GLACIER objects (Vincent 51892 Murphy) 51893 - Use chunksize lib to determine chunksize dynamically (Derek 51894 Battams) 51895 - Retry RequestTimeout errors (Nick Craig-Wood) 51896 - Implement reading and writing of metadata (Nick Craig-Wood) 51897 - SFTP 51898 - Add support for about and hashsum on windows server (albertony) 51899 - Use vendor-specific VFS statistics extension for about if 51900 available (albertony) 51901 - Add --sftp-chunk-size to control packets sizes for high latency 51902 links (Nick Craig-Wood) 51903 - Add --sftp-concurrency to improve high latency transfers (Nick 51904 Craig-Wood) 51905 - Add --sftp-set-env option to set environment variables (Nick 51906 Craig-Wood) 51907 - Add Hetzner Storage Boxes to supported sftp backends (Anthrazz) 51908 - Storj 51909 - Fix put which lead to the file being unreadable when using mount 51910 (Erik van Velzen) 51911 - Union 51912 - Add min_free_space option for lfs/eplfs policies (Nick 51913 Craig-Wood) 51914 - Fix uploading files to union of all bucket based remotes (Nick 51915 Craig-Wood) 51916 - Fix get free space for remotes which don't support it (Nick 51917 Craig-Wood) 51918 - Fix eplus policy to select correct entry for existing files 51919 (Nick Craig-Wood) 51920 - Support metadata (Nick Craig-Wood) 51921 - Uptobox 51922 - Fix root path handling (buengese) 51923 - WebDAV 51924 - Add SharePoint in other specific regions support (Noah Hsu) 51925 - Yandex 51926 - Handle api error on server-side move (albertony) 51927 - Zoho 51928 - Add Japan and China regions (buengese) 51929 51930 v1.58.1 - 2022-04-29 51931 51932 See commits 51933 51934 - Bug Fixes 51935 - build: Update github.com/billziss-gh to github.com/winfsp (Nick 51936 Craig-Wood) 51937 - filter: Fix timezone of --min-age/-max-age from UTC to local as 51938 documented (Nick Craig-Wood) 51939 - rc/js: Correct RC method names (Sơn Trần-Nguyễn) 51940 - docs 51941 - Fix some links to command pages (albertony) 51942 - Add --multi-thread-streams note to --transfers. (Zsolt Ero) 51943 - Mount 51944 - Fix --devname and fusermount: unknown option 'fsname' when 51945 mounting via rc (Nick Craig-Wood) 51946 - VFS 51947 - Remove wording which suggests VFS is only for mounting (Nick 51948 Craig-Wood) 51949 - Dropbox 51950 - Fix retries of multipart uploads with incorrect_offset error 51951 (Nick Craig-Wood) 51952 - Google Cloud Storage 51953 - Use the s3 pacer to speed up transactions (Nick Craig-Wood) 51954 - pacer: Default the Google pacer to a burst of 100 to fix gcs 51955 pacing (Nick Craig-Wood) 51956 - Jottacloud 51957 - Fix scope in token request (albertony) 51958 - Netstorage 51959 - Fix unescaped HTML in documentation (Nick Craig-Wood) 51960 - Make levels of headings consistent (Nick Craig-Wood) 51961 - Add support contacts to netstorage doc (Nil Alexandrov) 51962 - Onedrive 51963 - Note that sharepoint also changes web files (.html, .aspx) (GH) 51964 - Putio 51965 - Handle rate limit errors (Berkan Teber) 51966 - Fix multithread download and other ranged requests (rafma0) 51967 - S3 51968 - Add ChinaMobile EOS to provider list (GuoXingbin) 51969 - Sync providers in config description with providers (Nick 51970 Craig-Wood) 51971 - SFTP 51972 - Fix OpenSSH 8.8+ RSA keys incompatibility (KARBOWSKI Piotr) 51973 - Note that Scaleway C14 is deprecating SFTP in favor of S3 51974 (Adrien Rey-Jarthon) 51975 - Storj 51976 - Fix bucket creation on Move (Nick Craig-Wood) 51977 - WebDAV 51978 - Don't override Referer if user sets it (Nick Craig-Wood) 51979 51980 v1.58.0 - 2022-03-18 51981 51982 See commits 51983 51984 - New backends 51985 - Akamai Netstorage (Nil Alexandrov) 51986 - Seagate Lyve, SeaweedFS, Storj, RackCorp via s3 backend 51987 - Storj (renamed from Tardigrade - your old config files will 51988 continue working) 51989 - New commands 51990 - bisync - experimental bidirectional cloud sync (Ivan Andreev, 51991 Chris Nelson) 51992 - New Features 51993 - build 51994 - Add windows/arm64 build (rclone mount not supported yet) 51995 (Nick Craig-Wood) 51996 - Raise minimum go version to go1.15 (Nick Craig-Wood) 51997 - config: Allow dot in remote names and improve config editing 51998 (albertony) 51999 - dedupe: Add quit as a choice in interactive mode (albertony) 52000 - dlna: Change icons to the newest ones. (Alain Nussbaumer) 52001 - filter: Add {{ regexp }} syntax to pattern matches (Nick 52002 Craig-Wood) 52003 - fshttp: Add prometheus metrics for HTTP status code (Michał 52004 Matczuk) 52005 - hashsum: Support creating hash from data received on stdin 52006 (albertony) 52007 - librclone 52008 - Allow empty string or null input instead of empty json 52009 object (albertony) 52010 - Add support for mount commands (albertony) 52011 - operations: Add server-side moves to stats (Ole Frost) 52012 - rc: Allow user to disable authentication for web gui (negative0) 52013 - tree: Remove obsolete --human replaced by global 52014 --human-readable (albertony) 52015 - version: Report correct friendly-name for newer Windows 10/11 52016 versions (albertony) 52017 - Bug Fixes 52018 - build 52019 - Fix ARM architecture version in .deb packages after nfpm 52020 change (Nick Craig-Wood) 52021 - Hard fork github.com/jlaffaye/ftp to fix 52022 go get github.com/rclone/rclone (Nick Craig-Wood) 52023 - oauthutil: Fix crash when webbrowser requests /robots.txt (Nick 52024 Craig-Wood) 52025 - operations: Fix goroutine leak in case of copy retry (Ankur 52026 Gupta) 52027 - rc: 52028 - Fix operations/publiclink default for expires parameter 52029 (Nick Craig-Wood) 52030 - Fix missing computation of transferQueueSize when summing up 52031 statistics group (Carlo Mion) 52032 - Fix missing StatsInfo fields in the computation of the group 52033 sum (Carlo Mion) 52034 - sync: Fix --max-duration so it doesn't retry when the duration 52035 is exceeded (Nick Craig-Wood) 52036 - touch: Fix issue where a directory is created instead of a file 52037 (albertony) 52038 - Mount 52039 - Add --devname to set the device name sent to FUSE for mount 52040 display (Nick Craig-Wood) 52041 - VFS 52042 - Add vfs/stats remote control to show statistics (Nick 52043 Craig-Wood) 52044 - Fix 52045 failed to _ensure cache internal error: downloaders is nil error 52046 (Nick Craig-Wood) 52047 - Fix handling of special characters in file names (Bumsu Hyeon) 52048 - Local 52049 - Fix hash invalidation which caused errors with local crypt mount 52050 (Nick Craig-Wood) 52051 - Crypt 52052 - Add base64 and base32768 filename encoding options (Max Sum, 52053 Sinan Tan) 52054 - Azure Blob 52055 - Implement --azureblob-upload-concurrency parameter to speed 52056 uploads (Nick Craig-Wood) 52057 - Remove 100MB upper limit on chunk_size as it is no longer needed 52058 (Nick Craig-Wood) 52059 - Raise --azureblob-upload-concurrency to 16 by default (Nick 52060 Craig-Wood) 52061 - Fix crash with SAS URL and no container (Nick Craig-Wood) 52062 - Compress 52063 - Fix crash if metadata upload failed (Nick Craig-Wood) 52064 - Fix memory leak (Nick Craig-Wood) 52065 - Drive 52066 - Added --drive-copy-shortcut-content (Abhiraj) 52067 - Disable OAuth OOB flow (copy a token) due to Google deprecation 52068 (Nick Craig-Wood) 52069 - See the deprecation note. 52070 - Add --drive-skip-dangling-shortcuts flag (Nick Craig-Wood) 52071 - When using a link type --drive-export-formats shows all doc 52072 types (Nick Craig-Wood) 52073 - Dropbox 52074 - Speed up directory listings by specifying 1000 items in a chunk 52075 (Nick Craig-Wood) 52076 - Save an API request when at the root (Nick Craig-Wood) 52077 - Fichier 52078 - Implemented About functionality (Gourav T) 52079 - FTP 52080 - Add --ftp-ask-password to prompt for password when needed (Borna 52081 Butkovic) 52082 - Google Cloud Storage 52083 - Add missing regions (Nick Craig-Wood) 52084 - Disable OAuth OOB flow (copy a token) due to Google deprecation 52085 (Nick Craig-Wood) 52086 - See the deprecation note. 52087 - Googlephotos 52088 - Disable OAuth OOB flow (copy a token) due to Google deprecation 52089 (Nick Craig-Wood) 52090 - See the deprecation note. 52091 - Hasher 52092 - Fix crash on object not found (Nick Craig-Wood) 52093 - Hdfs 52094 - Add file (Move) and directory move (DirMove) support (Andy 52095 Jackson) 52096 - HTTP 52097 - Improved recognition of URL pointing to a single file 52098 (albertony) 52099 - Jottacloud 52100 - Change API used by recursive list (ListR) (Kim) 52101 - Add support for Tele2 Cloud (Fredric Arklid) 52102 - Koofr 52103 - Add Digistorage service as a Koofr provider. (jaKa) 52104 - Mailru 52105 - Fix int32 overflow on arm32 (Ivan Andreev) 52106 - Onedrive 52107 - Add config option for oauth scope Sites.Read.All (Charlie Jiang) 52108 - Minor optimization of quickxorhash (Isaac Levy) 52109 - Add --onedrive-root-folder-id flag (Nick Craig-Wood) 52110 - Do not retry on 400 pathIsTooLong error (ctrl-q) 52111 - Pcloud 52112 - Add support for recursive list (ListR) (Niels van de Weem) 52113 - Fix pre-1970 time stamps (Nick Craig-Wood) 52114 - S3 52115 - Use ListObjectsV2 for faster listings (Felix Bünemann) 52116 - Fallback to ListObject v1 on unsupported providers (Nick 52117 Craig-Wood) 52118 - Use the ETag on multipart transfers to verify the transfer was 52119 OK (Nick Craig-Wood) 52120 - Add --s3-use-multipart-etag provider quirk to disable this 52121 on unsupported providers (Nick Craig-Wood) 52122 - New Providers 52123 - RackCorp object storage (bbabich) 52124 - Seagate Lyve Cloud storage (Nick Craig-Wood) 52125 - SeaweedFS (Chris Lu) 52126 - Storj Shared gateways (Márton Elek, Nick Craig-Wood) 52127 - Add Wasabi AP Northeast 2 endpoint info (lindwurm) 52128 - Add GLACIER_IR storage class (Yunhai Luo) 52129 - Document Content-MD5 workaround for object-lock enabled buckets 52130 (Paulo Martins) 52131 - Fix multipart upload with --no-head flag (Nick Craig-Wood) 52132 - Simplify content length processing in s3 with download url 52133 (Logeshwaran Murugesan) 52134 - SFTP 52135 - Add rclone to list of supported md5sum/sha1sum commands to look 52136 for (albertony) 52137 - Refactor so we only have one way of running remote commands 52138 (Nick Craig-Wood) 52139 - Fix timeout on hashing large files by sending keepalives (Nick 52140 Craig-Wood) 52141 - Fix unnecessary seeking when uploading and downloading files 52142 (Nick Craig-Wood) 52143 - Update docs on how to create known_hosts file (Nick Craig-Wood) 52144 - Storj 52145 - Rename tardigrade backend to storj backend (Nick Craig-Wood) 52146 - Implement server side Move for files (Nick Craig-Wood) 52147 - Update docs to explain differences between s3 and this backend 52148 (Elek, Márton) 52149 - Swift 52150 - Fix About so it shows info about the current container only 52151 (Nick Craig-Wood) 52152 - Union 52153 - Fix treatment of remotes with // in (Nick Craig-Wood) 52154 - Fix deadlock when one part of a multi-upload fails (Nick 52155 Craig-Wood) 52156 - Fix eplus policy returned nil (Vitor Arruda) 52157 - Yandex 52158 - Add permanent deletion support (deinferno) 52159 52160 v1.57.0 - 2021-11-01 52161 52162 See commits 52163 52164 - New backends 52165 - Sia: for Sia decentralized cloud (Ian Levesque, Matthew Sevey, 52166 Ivan Andreev) 52167 - Hasher: caches hashes and enable hashes for backends that don't 52168 support them (Ivan Andreev) 52169 - New commands 52170 - lsjson --stat: to get info about a single file/dir and 52171 operations/stat api (Nick Craig-Wood) 52172 - config paths: show configured paths (albertony) 52173 - New Features 52174 - about: Make human-readable output more consistent with other 52175 commands (albertony) 52176 - build 52177 - Use go1.17 for building and make go1.14 the minimum 52178 supported (Nick Craig-Wood) 52179 - Update Go to 1.16 and NDK to 22b for Android builds (x0b) 52180 - config 52181 - Support hyphen in remote name from environment variable 52182 (albertony) 52183 - Make temporary directory user-configurable (albertony) 52184 - Convert --cache-dir value to an absolute path (albertony) 52185 - Do not override MIME types from OS defaults (albertony) 52186 - docs 52187 - Toc styling and header levels cleanup (albertony) 52188 - Extend documentation on valid remote names (albertony) 52189 - Mention make for building and cmount tag for macos (Alex 52190 Chen) 52191 - ...and many more contributions to numerous to mention! 52192 - fs: Move with --ignore-existing will not delete skipped files 52193 (Nathan Collins) 52194 - hashsum 52195 - Treat hash values in sum file as case insensitive (Ivan 52196 Andreev) 52197 - Don't put ERROR or UNSUPPORTED in output (Ivan Andreev) 52198 - lib/encoder: Add encoding of square brackets (Ivan Andreev) 52199 - lib/file: Improve error message when attempting to create dir on 52200 nonexistent drive on windows (albertony) 52201 - lib/http: Factor password hash salt into options with default 52202 (Nolan Woods) 52203 - lib/kv: Add key-value database api (Ivan Andreev) 52204 - librclone 52205 - Add RcloneFreeString function (albertony) 52206 - Free strings in python example (albertony) 52207 - log: Optionally print pid in logs (Ivan Andreev) 52208 - ls: Introduce --human-readable global option to print 52209 human-readable sizes (albertony) 52210 - ncdu: Introduce key u to toggle human-readable (albertony) 52211 - operations: Add rmdirs -v output (Justin Winokur) 52212 - serve sftp 52213 - Generate an ECDSA server key as well as RSA (Nick 52214 Craig-Wood) 52215 - Generate an Ed25519 server key as well as ECDSA and RSA 52216 (albertony) 52217 - serve docker 52218 - Allow to customize proxy settings of docker plugin (Ivan 52219 Andreev) 52220 - Build docker plugin for multiple platforms (Thomas Stachl) 52221 - size: Include human-readable count (albertony) 52222 - touch: Add support for touching files in directory, with 52223 recursive option, filtering and --dry-run/-i (albertony) 52224 - tree: Option to print human-readable sizes removed in favor of 52225 global option (albertony) 52226 - Bug Fixes 52227 - lib/http 52228 - Fix bad username check in single auth secret provider (Nolan 52229 Woods) 52230 - Fix handling of SSL credentials (Nolan Woods) 52231 - serve ftp: Ensure modtime is passed as UTC always to fix 52232 timezone oddities (Nick Craig-Wood) 52233 - serve sftp: Fix generation of server keys on windows (albertony) 52234 - serve docker: Fix octal umask (Ivan Andreev) 52235 - Mount 52236 - Enable rclone to be run as mount helper direct from the fstab 52237 (Ivan Andreev) 52238 - Use procfs to validate mount on linux (Ivan Andreev) 52239 - Correctly daemonize for compatibility with automount (Ivan 52240 Andreev) 52241 - VFS 52242 - Ensure names used in cache path are legal on current OS 52243 (albertony) 52244 - Ignore ECLOSED when truncating file handles to fix intermittent 52245 bad file descriptor error (Nick Craig-Wood) 52246 - Local 52247 - Refactor default OS encoding out from local backend into shared 52248 encoder lib (albertony) 52249 - Crypt 52250 - Return wrapped object even with --crypt-no-data-encryption (Ivan 52251 Andreev) 52252 - Fix uploads with --crypt-no-data-encryption (Nick Craig-Wood) 52253 - Azure Blob 52254 - Add --azureblob-no-head-object (Tatsuya Noyori) 52255 - Box 52256 - Make listings of heavily used directories more reliable (Nick 52257 Craig-Wood) 52258 - When doing cleanup delete as much as possible (Nick Craig-Wood) 52259 - Add --box-list-chunk to control listing chunk size (Nick 52260 Craig-Wood) 52261 - Delete items in parallel in cleanup using --checkers threads 52262 (Nick Craig-Wood) 52263 - Add --box-owned-by to only show items owned by the login passed 52264 (Nick Craig-Wood) 52265 - Retry operation_blocked_temporary errors (Nick Craig-Wood) 52266 - Chunker 52267 - Md5all must create metadata if base hash is slow (Ivan Andreev) 52268 - Drive 52269 - Speed up directory listings by constraining the API listing 52270 using the current filters (fotile96, Ivan Andreev) 52271 - Fix buffering for single request upload for files smaller than 52272 --drive-upload-cutoff (YenForYang) 52273 - Add -o config option to backend drives to make config for all 52274 shared drives (Nick Craig-Wood) 52275 - Dropbox 52276 - Add --dropbox-batch-commit-timeout to control batch timeout 52277 (Nick Craig-Wood) 52278 - Filefabric 52279 - Make backoff exponential for error_background to fix errors 52280 (Nick Craig-Wood) 52281 - Fix directory move after API change (Nick Craig-Wood) 52282 - FTP 52283 - Enable tls session cache by default (Ivan Andreev) 52284 - Add option to disable tls13 (Ivan Andreev) 52285 - Fix timeout after long uploads (Ivan Andreev) 52286 - Add support for precise time (Ivan Andreev) 52287 - Enable CI for ProFtpd, PureFtpd, VsFtpd (Ivan Andreev) 52288 - Googlephotos 52289 - Use encoder for album names to fix albums with control 52290 characters (Parth Shukla) 52291 - Jottacloud 52292 - Implement SetModTime to support modtime-only changes (albertony) 52293 - Improved error handling with SetModTime and corrupt files in 52294 general (albertony) 52295 - Add support for UserInfo (rclone config userinfo) feature 52296 (albertony) 52297 - Return direct download link from rclone link command (albertony) 52298 - Koofr 52299 - Create direct share link (Dmitry Bogatov) 52300 - Pcloud 52301 - Add sha256 support (Ken Enrique Morel) 52302 - Premiumizeme 52303 - Fix directory listing after API changes (Nick Craig-Wood) 52304 - Fix server side move after API change (Nick Craig-Wood) 52305 - Fix server side directory move after API changes (Nick 52306 Craig-Wood) 52307 - S3 52308 - Add support to use CDN URL to download the file (Logeshwaran) 52309 - Add AWS Snowball Edge to providers examples (r0kk3rz) 52310 - Use a combination of SDK retries and rclone retries (Nick 52311 Craig-Wood) 52312 - Fix IAM Role for Service Account not working and other auth 52313 problems (Nick Craig-Wood) 52314 - Fix shared_credentials_file auth after reverting incorrect fix 52315 (Nick Craig-Wood) 52316 - Fix corrupted on transfer: sizes differ 0 vs xxxx with Ceph 52317 (Nick Craig-Wood) 52318 - Seafile 52319 - Fix error when not configured for 2fa (Fred) 52320 - SFTP 52321 - Fix timeout when doing MD5SUM of large file (Nick Craig-Wood) 52322 - Swift 52323 - Update OCI URL (David Liu) 52324 - Document OVH Cloud Archive (HNGamingUK) 52325 - Union 52326 - Fix rename not working with union of local disk and bucket based 52327 remote (Nick Craig-Wood) 52328 52329 v1.56.2 - 2021-10-01 52330 52331 See commits 52332 52333 - Bug Fixes 52334 - serve http: Re-add missing auth to http service (Nolan Woods) 52335 - build: Update golang.org/x/sys to fix crash on macOS when 52336 compiled with go1.17 (Herby Gillot) 52337 - FTP 52338 - Fix deadlock after failed update when concurrency=1 (Ivan 52339 Andreev) 52340 52341 v1.56.1 - 2021-09-19 52342 52343 See commits 52344 52345 - Bug Fixes 52346 - accounting: Fix maximum bwlimit by scaling scale max token 52347 bucket size (Nick Craig-Wood) 52348 - rc: Fix speed does not update in core/stats (negative0) 52349 - selfupdate: Fix --quiet option, not quite quiet (yedamo) 52350 - serve http: Fix serve http exiting directly after starting 52351 (Cnly) 52352 - build 52353 - Apply gofmt from golang 1.17 (Ivan Andreev) 52354 - Update Go to 1.16 and NDK to 22b for android/any (x0b) 52355 - Mount 52356 - Fix --daemon mode (Ivan Andreev) 52357 - VFS 52358 - Fix duplicates on rename (Nick Craig-Wood) 52359 - Fix crash when truncating a just uploaded object (Nick 52360 Craig-Wood) 52361 - Fix issue where empty dirs would build up in cache meta dir 52362 (albertony) 52363 - Drive 52364 - Fix instructions for auto config (Greg Sadetsky) 52365 - Fix lsf example without drive-impersonate (Greg Sadetsky) 52366 - Onedrive 52367 - Handle HTTP 400 better in PublicLink (Alex Chen) 52368 - Clarification of the process for creating custom client_id 52369 (Mariano Absatz) 52370 - Pcloud 52371 - Return an early error when Put is called with an unknown size 52372 (Nick Craig-Wood) 52373 - Try harder to delete a failed upload (Nick Craig-Wood) 52374 - S3 52375 - Add Wasabi's AP-Northeast endpoint info (hota) 52376 - Fix typo in s3 documentation (Greg Sadetsky) 52377 - Seafile 52378 - Fix 2fa config state machine (Fred) 52379 - SFTP 52380 - Remove spurious error message on --sftp-disable-concurrent-reads 52381 (Nick Craig-Wood) 52382 - Sugarsync 52383 - Fix initial connection after config re-arrangement (Nick 52384 Craig-Wood) 52385 52386 v1.56.0 - 2021-07-20 52387 52388 See commits 52389 52390 - New backends 52391 - Uptobox (buengese) 52392 - New commands 52393 - serve docker (Antoine GIRARD) (Ivan Andreev) 52394 - and accompanying docker volume plugin 52395 - checksum to check files against a file of checksums (Ivan 52396 Andreev) 52397 - this is also available as rclone md5sum -C etc 52398 - config touch: ensure config exists at configured location 52399 (albertony) 52400 - test changenotify: command to help debugging changenotify (Nick 52401 Craig-Wood) 52402 - Deprecations 52403 - dbhashsum: Remove command deprecated a year ago (Ivan Andreev) 52404 - cache: Deprecate cache backend (Ivan Andreev) 52405 - New Features 52406 - rework config system so it can be used non-interactively via cli 52407 and rc API. 52408 - See docs in config create 52409 - This is a very big change to all the backends so may cause 52410 breakages - please file bugs! 52411 - librclone - export the rclone RC as a C library (lewisxy) (Nick 52412 Craig-Wood) 52413 - Link a C-API rclone shared object into your project 52414 - Use the RC as an in memory interface 52415 - Python example supplied 52416 - Also supports Android and gomobile 52417 - fs 52418 - Add --disable-http2 for global http2 disable (Nick 52419 Craig-Wood) 52420 - Make --dump imply -vv (Alex Chen) 52421 - Use binary prefixes for size and rate units (albertony) 52422 - Use decimal prefixes for counts (albertony) 52423 - Add google search widget to rclone.org (Ivan Andreev) 52424 - accounting: Calculate rolling average speed (Haochen Tong) 52425 - atexit: Terminate with non-zero status after receiving signal 52426 (Michael Hanselmann) 52427 - build 52428 - Only run event-based workflow scripts under rclone repo with 52429 manual override (Mathieu Carbou) 52430 - Add Android build with gomobile (x0b) 52431 - check: Log the hash in use like cryptcheck does (Nick 52432 Craig-Wood) 52433 - version: Print os/version, kernel and bitness (Ivan Andreev) 52434 - config 52435 - Prevent use of Windows reserved names in config file name 52436 (albertony) 52437 - Create config file in windows appdata directory by default 52438 (albertony) 52439 - Treat any config file paths with filename notfound as 52440 memory-only config (albertony) 52441 - Delay load config file (albertony) 52442 - Replace defaultConfig with a thread-safe in-memory 52443 implementation (Chris Macklin) 52444 - Allow config create and friends to take key=value parameters 52445 (Nick Craig-Wood) 52446 - Fixed issues with flags/options set by environment vars. 52447 (Ole Frost) 52448 - fshttp: Implement graceful DSCP error handling (Tyson Moore) 52449 - lib/http - provides an abstraction for a central http server 52450 that services can bind routes to (Nolan Woods) 52451 - Add --template config and flags to serve/data (Nolan Woods) 52452 - Add default 404 handler (Nolan Woods) 52453 - link: Use "off" value for unset expiry (Nick Craig-Wood) 52454 - oauthutil: Raise fatal error if token expired without refresh 52455 token (Alex Chen) 52456 - rcat: Add --size flag for more efficient uploads of known size 52457 (Nazar Mishturak) 52458 - serve sftp: Add --stdio flag to serve via stdio (Tom) 52459 - sync: Don't warn about --no-traverse when --files-from is set 52460 (Nick Gaya) 52461 - test makefiles 52462 - Add --seed flag and make data generated repeatable (Nick 52463 Craig-Wood) 52464 - Add log levels and speed summary (Nick Craig-Wood) 52465 - Bug Fixes 52466 - accounting: Fix startTime of statsGroups.sum (Haochen Tong) 52467 - cmd/ncdu: Fix out of range panic in delete (buengese) 52468 - config 52469 - Fix issues with memory-only config file paths (albertony) 52470 - Fix in memory config not saving on the fly backend config 52471 (Nick Craig-Wood) 52472 - fshttp: Fix address parsing for DSCP (Tyson Moore) 52473 - ncdu: Update termbox-go library to fix crash (Nick Craig-Wood) 52474 - oauthutil: Fix old authorize result not recognised (Cnly) 52475 - operations: Don't update timestamps of files in --compare-dest 52476 (Nick Gaya) 52477 - selfupdate: fix archive name on macos (Ivan Andreev) 52478 - Mount 52479 - Refactor before adding serve docker (Antoine GIRARD) 52480 - VFS 52481 - Add cache reset for --vfs-cache-max-size handling at cache poll 52482 interval (Leo Luan) 52483 - Fix modtime changing when reading file into cache (Nick 52484 Craig-Wood) 52485 - Avoid unnecessary subdir in cache path (albertony) 52486 - Fix that umask option cannot be set as environment variable 52487 (albertony) 52488 - Do not print notice about missing poll-interval support when set 52489 to 0 (albertony) 52490 - Local 52491 - Always use readlink to read symlink size for better 52492 compatibility (Nick Craig-Wood) 52493 - Add --local-unicode-normalization (and remove 52494 --local-no-unicode-normalization) (Nick Craig-Wood) 52495 - Skip entries removed concurrently with List() (Ivan Andreev) 52496 - Crypt 52497 - Support timestamped filenames from --b2-versions (Dominik 52498 Mydlil) 52499 - B2 52500 - Don't include the bucket name in public link file prefixes 52501 (Jeffrey Tolar) 52502 - Fix versions and .files with no extension (Nick Craig-Wood) 52503 - Factor version handling into lib/version (Dominik Mydlil) 52504 - Box 52505 - Use upload preflight check to avoid listings in file uploads 52506 (Nick Craig-Wood) 52507 - Return errors instead of calling log.Fatal with them (Nick 52508 Craig-Wood) 52509 - Drive 52510 - Switch to the Drives API for looking up shared drives (Nick 52511 Craig-Wood) 52512 - Fix some google docs being treated as files (Nick Craig-Wood) 52513 - Dropbox 52514 - Add --dropbox-batch-mode flag to speed up uploading (Nick 52515 Craig-Wood) 52516 - Read the batch mode docs for more info 52517 - Set visibility in link sharing when --expire is set (Nick 52518 Craig-Wood) 52519 - Simplify chunked uploads (Alexey Ivanov) 52520 - Improve "own App IP" instructions (Ivan Andreev) 52521 - Fichier 52522 - Check if more than one upload link is returned (Nick Craig-Wood) 52523 - Support downloading password protected files and folders 52524 (Florian Penzkofer) 52525 - Make error messages report text from the API (Nick Craig-Wood) 52526 - Fix move of files in the same directory (Nick Craig-Wood) 52527 - Check that we actually got a download token and retry if we 52528 didn't (buengese) 52529 - Filefabric 52530 - Fix listing after change of from field from "int" to int. (Nick 52531 Craig-Wood) 52532 - FTP 52533 - Make upload error 250 indicate success (Nick Craig-Wood) 52534 - GCS 52535 - Make compatible with gsutil's mtime metadata (database64128) 52536 - Clean up time format constants (database64128) 52537 - Google Photos 52538 - Fix read only scope not being used properly (Nick Craig-Wood) 52539 - HTTP 52540 - Replace httplib with lib/http (Nolan Woods) 52541 - Clean up Bind to better use middleware (Nolan Woods) 52542 - Jottacloud 52543 - Fix legacy auth with state based config system (buengese) 52544 - Fix invalid url in output from link command (albertony) 52545 - Add no versions option (buengese) 52546 - Onedrive 52547 - Add list_chunk option (Nick Gaya) 52548 - Also report root error if unable to cancel multipart upload 52549 (Cnly) 52550 - Fix failed to configure: empty token found error (Nick 52551 Craig-Wood) 52552 - Make link return direct download link (Xuanchen Wu) 52553 - S3 52554 - Add --s3-no-head-object (Tatsuya Noyori) 52555 - Remove WebIdentityRoleProvider to fix crash on auth (Nick 52556 Craig-Wood) 52557 - Don't check to see if remote is object if it ends with / (Nick 52558 Craig-Wood) 52559 - Add SeaweedFS (Chris Lu) 52560 - Update Alibaba OSS endpoints (Chuan Zh) 52561 - SFTP 52562 - Fix performance regression by re-enabling concurrent writes 52563 (Nick Craig-Wood) 52564 - Expand tilde and environment variables in configured 52565 known_hosts_file (albertony) 52566 - Tardigrade 52567 - Upgrade to uplink v1.4.6 (Caleb Case) 52568 - Use negative offset (Caleb Case) 52569 - Add warning about too many open files (acsfer) 52570 - WebDAV 52571 - Fix sharepoint auth over http (Nick Craig-Wood) 52572 - Add headers option (Antoon Prins) 52573 52574 v1.55.1 - 2021-04-26 52575 52576 See commits 52577 52578 - Bug Fixes 52579 - selfupdate 52580 - Dont detect FUSE if build is static (Ivan Andreev) 52581 - Add build tag noselfupdate (Ivan Andreev) 52582 - sync: Fix incorrect error reported by graceful cutoff (Nick 52583 Craig-Wood) 52584 - install.sh: fix macOS arm64 download (Nick Craig-Wood) 52585 - build: Fix version numbers in android branch builds (Nick 52586 Craig-Wood) 52587 - docs 52588 - Contributing.md: update setup instructions for go1.16 (Nick 52589 Gaya) 52590 - WinFsp 2021 is out of beta (albertony) 52591 - Minor cleanup of space around code section (albertony) 52592 - Fixed some typos (albertony) 52593 - VFS 52594 - Fix a code path which allows dirty data to be removed causing 52595 data loss (Nick Craig-Wood) 52596 - Compress 52597 - Fix compressed name regexp (buengese) 52598 - Drive 52599 - Fix backend copyid of google doc to directory (Nick Craig-Wood) 52600 - Don't open browser when service account... (Ansh Mittal) 52601 - Dropbox 52602 - Add missing team_data.member scope for use with --impersonate 52603 (Nick Craig-Wood) 52604 - Fix About after scopes changes - rclone config reconnect needed 52605 (Nick Craig-Wood) 52606 - Fix Unable to decrypt returned paths from changeNotify (Nick 52607 Craig-Wood) 52608 - FTP 52609 - Fix implicit TLS (Ivan Andreev) 52610 - Onedrive 52611 - Work around for random "Unable to initialize RPS" errors 52612 (OleFrost) 52613 - SFTP 52614 - Revert sftp library to v1.12.0 from v1.13.0 to fix performance 52615 regression (Nick Craig-Wood) 52616 - Fix Update ReadFrom failed: failed to send packet: EOF errors 52617 (Nick Craig-Wood) 52618 - Zoho 52619 - Fix error when region isn't set (buengese) 52620 - Do not ask for mountpoint twice when using headless setup 52621 (buengese) 52622 52623 v1.55.0 - 2021-03-31 52624 52625 See commits 52626 52627 - New commands 52628 - selfupdate (Ivan Andreev) 52629 - Allows rclone to update itself in-place or via a package 52630 (using --package flag) 52631 - Reads cryptographically signed signatures for non beta 52632 releases 52633 - Works on all OSes. 52634 - test - these are test commands - use with care! 52635 - histogram - Makes a histogram of file name characters. 52636 - info - Discovers file name or other limitations for paths. 52637 - makefiles - Make a random file hierarchy for testing. 52638 - memory - Load all the objects at remote:path into memory and 52639 report memory stats. 52640 - New Features 52641 - Connection strings 52642 - Config parameters can now be passed as part of the remote 52643 name as a connection string. 52644 - For example, to do the equivalent of --drive-shared-with-me 52645 use drive,shared_with_me: 52646 - Make sure we don't save on the fly remote config to the 52647 config file (Nick Craig-Wood) 52648 - Make sure backends with additional config have a different 52649 name for caching (Nick Craig-Wood) 52650 - This work was sponsored by CERN, through the CS3MESH4EOSC 52651 Project. 52652 - CS3MESH4EOSC has received funding from the European 52653 Union’s Horizon 2020 52654 - research and innovation programme under Grant Agreement 52655 no. 863353. 52656 - build 52657 - Update go build version to go1.16 and raise minimum go 52658 version to go1.13 (Nick Craig-Wood) 52659 - Make a macOS ARM64 build to support Apple Silicon (Nick 52660 Craig-Wood) 52661 - Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood) 52662 - Use GO386=softfloat instead of deprecated GO386=387 for 386 52663 builds (Nick Craig-Wood) 52664 - Disable IOS builds for the time being (Nick Craig-Wood) 52665 - Androids builds made with up to date NDK (x0b) 52666 - Add an rclone user to the Docker image but don't use it by 52667 default (cynthia kwok) 52668 - dedupe: Make largest directory primary to minimize data moved 52669 (Saksham Khanna) 52670 - config 52671 - Wrap config library in an interface (Fionera) 52672 - Make config file system pluggable (Nick Craig-Wood) 52673 - --config "" or "/notfound" for in memory config only (Nick 52674 Craig-Wood) 52675 - Clear fs cache of stale entries when altering config (Nick 52676 Craig-Wood) 52677 - copyurl: Add option to print resulting auto-filename (albertony) 52678 - delete: Make --rmdirs obey the filters (Nick Craig-Wood) 52679 - docs - many fixes and reworks from edwardxml, albertony, pvalls, 52680 Ivan Andreev, Evan Harris, buengese, Alexey Tabakman 52681 - encoder/filename - add SCSU as tables (Klaus Post) 52682 - Add multiple paths support to --compare-dest and --copy-dest 52683 flag (K265) 52684 - filter: Make --exclude "dir/" equivalent to --exclude "dir/**" 52685 (Nick Craig-Wood) 52686 - fshttp: Add DSCP support with --dscp for QoS with differentiated 52687 services (Max Sum) 52688 - lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood) 52689 - lib/file 52690 - Make pre-allocate detect disk full errors and return them 52691 (Nick Craig-Wood) 52692 - Don't run preallocate concurrently (Nick Craig-Wood) 52693 - Retry preallocate on EINTR (Nick Craig-Wood) 52694 - operations: Made copy and sync operations obey a RetryAfterError 52695 (Ankur Gupta) 52696 - rc 52697 - Add string alternatives for setting options over the rc 52698 (Nick Craig-Wood) 52699 - Add options/local to see the options configured in the 52700 context (Nick Craig-Wood) 52701 - Add _config parameter to set global config for just this rc 52702 call (Nick Craig-Wood) 52703 - Implement passing filter config with _filter parameter (Nick 52704 Craig-Wood) 52705 - Add fscache/clear and fscache/entries to control the fs 52706 cache (Nick Craig-Wood) 52707 - Avoid +Inf value for speed in core/stats (albertony) 52708 - Add a full set of stats to core/stats (Nick Craig-Wood) 52709 - Allow fs= params to be a JSON blob (Nick Craig-Wood) 52710 - rcd: Added systemd notification during the rclone rcd command. 52711 (Naveen Honest Raj) 52712 - rmdirs: Make --rmdirs obey the filters (Nick Craig-Wood) 52713 - version: Show build tags and type of executable (Ivan Andreev) 52714 - Bug Fixes 52715 - install.sh: make it fail on download errors (Ivan Andreev) 52716 - Fix excessive retries missing --max-duration timeout (Nick 52717 Craig-Wood) 52718 - Fix crash when --low-level-retries=0 (Nick Craig-Wood) 52719 - Fix failed token refresh on mounts created via the rc (Nick 52720 Craig-Wood) 52721 - fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood) 52722 - lib/atexit 52723 - Unregister interrupt handler once it has fired so users can 52724 interrupt again (Nick Craig-Wood) 52725 - Fix occasional failure to unmount with CTRL-C (Nick 52726 Craig-Wood) 52727 - Fix deadlock calling Finalise while Run is running (Nick 52728 Craig-Wood) 52729 - lib/rest: Fix multipart uploads not stopping on context cancel 52730 (Nick Craig-Wood) 52731 - Mount 52732 - Allow mounting to root directory on windows (albertony) 52733 - Improved handling of relative paths on windows (albertony) 52734 - Fix unicode issues with accented characters on macOS (Nick 52735 Craig-Wood) 52736 - Docs: document the new FileSecurity option in WinFsp 2021 52737 (albertony) 52738 - Docs: add note about volume path syntax on windows (albertony) 52739 - Fix caching of old directories after renaming them (Nick 52740 Craig-Wood) 52741 - Update cgofuse to the latest version to bring in macfuse 4 fix 52742 (Nick Craig-Wood) 52743 - VFS 52744 - --vfs-used-is-size to report used space using recursive scan 52745 (tYYGH) 52746 - Don't set modification time if it was already correct (Nick 52747 Craig-Wood) 52748 - Fix Create causing windows explorer to truncate files on CTRL-C 52749 CTRL-V (Nick Craig-Wood) 52750 - Fix modtimes not updating when writing via cache (Nick 52751 Craig-Wood) 52752 - Fix modtimes changing by fractional seconds after upload (Nick 52753 Craig-Wood) 52754 - Fix modtime set if --vfs-cache-mode writes/full and no write 52755 (Nick Craig-Wood) 52756 - Rename files in cache and cancel uploads on directory rename 52757 (Nick Craig-Wood) 52758 - Fix directory renaming by renaming dirs cached in memory (Nick 52759 Craig-Wood) 52760 - Local 52761 - Add flag --local-no-preallocate (David Sze) 52762 - Make nounc an advanced option except on Windows (albertony) 52763 - Don't ignore preallocate disk full errors (Nick Craig-Wood) 52764 - Cache 52765 - Add --fs-cache-expire-duration to control the fs cache (Nick 52766 Craig-Wood) 52767 - Crypt 52768 - Add option to not encrypt data (Vesnyx) 52769 - Log hash ok on upload (albertony) 52770 - Azure Blob 52771 - Add container public access level support. (Manish Kumar) 52772 - B2 52773 - Fix HTML files downloaded via cloudflare (Nick Craig-Wood) 52774 - Box 52775 - Fix transfers getting stuck on token expiry after API change 52776 (Nick Craig-Wood) 52777 - Chunker 52778 - Partially implement no-rename transactions (Maxwell Calman) 52779 - Drive 52780 - Don't stop server side copy if couldn't read description (Nick 52781 Craig-Wood) 52782 - Pass context on to drive SDK - to help with cancellation (Nick 52783 Craig-Wood) 52784 - Dropbox 52785 - Add polling for changes support (Robert Thomas) 52786 - Make --timeout 0 work properly (Nick Craig-Wood) 52787 - Raise priority of rate limited message to INFO to make it more 52788 noticeable (Nick Craig-Wood) 52789 - Fichier 52790 - Implement copy & move (buengese) 52791 - Implement public link (buengese) 52792 - FTP 52793 - Implement Shutdown method (Nick Craig-Wood) 52794 - Close idle connections after --ftp-idle-timeout (1m by default) 52795 (Nick Craig-Wood) 52796 - Make --timeout 0 work properly (Nick Craig-Wood) 52797 - Add --ftp-close-timeout flag for use with awkward ftp servers 52798 (Nick Craig-Wood) 52799 - Retry connections and logins on 421 errors (Nick Craig-Wood) 52800 - Hdfs 52801 - Fix permissions for when directory is created (Lucas Messenger) 52802 - Onedrive 52803 - Make --timeout 0 work properly (Nick Craig-Wood) 52804 - S3 52805 - Fix --s3-profile which wasn't working (Nick Craig-Wood) 52806 - SFTP 52807 - Close idle connections after --sftp-idle-timeout (1m by default) 52808 (Nick Craig-Wood) 52809 - Fix "file not found" errors for read once servers (Nick 52810 Craig-Wood) 52811 - Fix SetModTime stat failed: object not found with 52812 --sftp-set-modtime=false (Nick Craig-Wood) 52813 - Swift 52814 - Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood) 52815 - Implement copying large objects (nguyenhuuluan434) 52816 - Union 52817 - Fix crash when using epff policy (Nick Craig-Wood) 52818 - Fix union attempting to update files on a read only file system 52819 (Nick Craig-Wood) 52820 - Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick 52821 Craig-Wood) 52822 - Fix initialisation broken in refactor (Nick Craig-Wood) 52823 - WebDAV 52824 - Add support for sharepoint with NTLM authentication (Rauno Ots) 52825 - Make sharepoint-ntlm docs more consistent (Alex Chen) 52826 - Improve terminology in sharepoint-ntlm docs (Ivan Andreev) 52827 - Disable HTTP/2 for NTLM authentication (georne) 52828 - Fix sharepoint-ntlm error 401 for parallel actions (Ivan 52829 Andreev) 52830 - Check that purged directory really exists (Ivan Andreev) 52831 - Yandex 52832 - Make --timeout 0 work properly (Nick Craig-Wood) 52833 - Zoho 52834 - Replace client id - you will need to rclone config reconnect 52835 after this (buengese) 52836 - Add forgotten setupRegion() to NewFs - this finally fixes 52837 regions other than EU (buengese) 52838 52839 v1.54.1 - 2021-03-08 52840 52841 See commits 52842 52843 - Bug Fixes 52844 - accounting: Fix --bwlimit when up or down is off (Nick 52845 Craig-Wood) 52846 - docs 52847 - Fix nesting of brackets and backticks in ftp docs 52848 (edwardxml) 52849 - Fix broken link in sftp page (edwardxml) 52850 - Fix typo in crypt.md (Romeo Kienzler) 52851 - Changelog: Correct link to digitalis.io (Alex JOST) 52852 - Replace #file-caching with #vfs-file-caching (Miron 52853 Veryanskiy) 52854 - Convert bogus example link to code (edwardxml) 52855 - Remove dead link from rc.md (edwardxml) 52856 - rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick 52857 Craig-Wood) 52858 - lsjson: Fix unterminated JSON in the presence of errors (Nick 52859 Craig-Wood) 52860 - Mount 52861 - Fix mount dropping on macOS by setting --daemon-timeout 10m 52862 (Nick Craig-Wood) 52863 - VFS 52864 - Document simultaneous usage with the same cache shouldn't be 52865 used (Nick Craig-Wood) 52866 - B2 52867 - Automatically raise upload cutoff to avoid spurious error (Nick 52868 Craig-Wood) 52869 - Fix failed to create file system with application key limited to 52870 a prefix (Nick Craig-Wood) 52871 - Drive 52872 - Refer to Shared Drives instead of Team Drives (Nick Craig-Wood) 52873 - Dropbox 52874 - Add scopes to oauth request and optionally "members.read" (Nick 52875 Craig-Wood) 52876 - S3 52877 - Fix failed to create file system with folder level permissions 52878 policy (Nick Craig-Wood) 52879 - Fix Wasabi HEAD requests returning stale data by using only 1 52880 transport (Nick Craig-Wood) 52881 - Fix shared_credentials_file auth (Dmitry Chepurovskiy) 52882 - Add --s3-no-head to reducing costs docs (Nick Craig-Wood) 52883 - Union 52884 - Fix mkdir at root with remote:/ (Nick Craig-Wood) 52885 - Zoho 52886 - Fix custom client id's (buengese) 52887 52888 v1.54.0 - 2021-02-02 52889 52890 See commits 52891 52892 - New backends 52893 - Compression remote (experimental) (buengese) 52894 - Enterprise File Fabric (Nick Craig-Wood) 52895 - This work was sponsored by Storage Made Easy 52896 - HDFS (Hadoop Distributed File System) (Yury Stankevich) 52897 - Zoho workdrive (buengese) 52898 - New Features 52899 - Deglobalise the config (Nick Craig-Wood) 52900 - Global config now read from the context 52901 - This will enable passing of global config via the rc 52902 - This work was sponsored by Digitalis 52903 - Add --bwlimit for upload and download (Nick Craig-Wood) 52904 - Obey bwlimit in http Transport for better limiting 52905 - Enhance systemd integration (Hekmon) 52906 - log level identification, manual activation with flag, 52907 automatic systemd launch detection 52908 - Don't compile systemd log integration for non unix systems 52909 (Benjamin Gustin) 52910 - Add a --download flag to md5sum/sha1sum/hashsum to force rclone 52911 to download and hash files locally (lostheli) 52912 - Add --progress-terminal-title to print ETA to terminal title 52913 (LaSombra) 52914 - Make backend env vars show in help as the defaults for backend 52915 flags (Nick Craig-Wood) 52916 - build 52917 - Raise minimum go version to go1.12 (Nick Craig-Wood) 52918 - dedupe 52919 - Add --by-hash to dedupe on content hash not file name (Nick 52920 Craig-Wood) 52921 - Add --dedupe-mode list to just list dupes, changing nothing 52922 (Nick Craig-Wood) 52923 - Add warning if used on a remote which can't have duplicate 52924 names (Nick Craig-Wood) 52925 - fs 52926 - Add Shutdown optional method for backends (Nick Craig-Wood) 52927 - When using --files-from check files concurrently (zhucan) 52928 - Accumulate stats when using --dry-run (Ingo Weiss) 52929 - Always show stats when using --dry-run or --interactive 52930 (Nick Craig-Wood) 52931 - Add support for flag --no-console on windows to hide the 52932 console window (albertony) 52933 - genautocomplete: Add support to output to stdout (Ingo) 52934 - ncdu 52935 - Highlight read errors instead of aborting (Claudio 52936 Bantaloukas) 52937 - Add sort by average size in directory (Adam Plánský) 52938 - Add toggle option for average s3ize in directory - key 'a' 52939 (Adam Plánský) 52940 - Add empty folder flag into ncdu browser (Adam Plánský) 52941 - Add ! (error) and . (unreadable) file flags to go with e 52942 (empty) (Nick Craig-Wood) 52943 - obscure: Make rclone obscure - ignore newline at end of line 52944 (Nick Craig-Wood) 52945 - operations 52946 - Add logs when need to upload files to set mod times (Nick 52947 Craig-Wood) 52948 - Move and copy log name of the destination object in verbose 52949 (Adam Plánský) 52950 - Add size if known to skipped items and JSON log (Nick 52951 Craig-Wood) 52952 - rc 52953 - Prefer actual listener address if using ":port" or "addr:0" 52954 only (Nick Craig-Wood) 52955 - Add listener for finished jobs (Aleksandar Jankovic) 52956 - serve ftp: Add options to enable TLS (Deepak Sah) 52957 - serve http/webdav: Redirect requests to the base url without the 52958 / (Nick Craig-Wood) 52959 - serve restic: Implement object cache (Nick Craig-Wood) 52960 - stats: Add counter for deleted directories (Nick Craig-Wood) 52961 - sync: Only print "There was nothing to transfer" if no errors 52962 (Nick Craig-Wood) 52963 - webui 52964 - Prompt user for updating webui if an update is available 52965 (Chaitanya Bankanhal) 52966 - Fix plugins initialization (negative0) 52967 - Bug Fixes 52968 - fs 52969 - Fix nil pointer on copy & move operations directly to remote 52970 (Anagh Kumar Baranwal) 52971 - Fix parsing of .. when joining remotes (Nick Craig-Wood) 52972 - log: Fix enabling systemd logging when using --log-file (Nick 52973 Craig-Wood) 52974 - check 52975 - Make the error count match up in the log message (Nick 52976 Craig-Wood) 52977 - move: Fix data loss when source and destination are the same 52978 object (Nick Craig-Wood) 52979 - operations 52980 - Fix --cutoff-mode hard not cutting off immediately (Nick 52981 Craig-Wood) 52982 - Fix --immutable error message (Nick Craig-Wood) 52983 - sync 52984 - Fix --cutoff-mode soft & cautious so it doesn't end the 52985 transfer early (Nick Craig-Wood) 52986 - Fix --immutable errors retrying many times (Nick Craig-Wood) 52987 - Docs 52988 - Many fixes and a rewrite of the filtering docs (edwardxml) 52989 - Many spelling and grammar fixes (Josh Soref) 52990 - Doc fixes for commands delete, purge, rmdir, rmdirs and mount 52991 (albertony) 52992 - And thanks to these people for many doc fixes too numerous to 52993 list 52994 - Ameer Dawood, Antoine GIRARD, Bob Bagwill, Christopher 52995 Stewart 52996 - CokeMine, David, Dov Murik, Durval Menezes, Evan Harris, 52997 gtorelly 52998 - Ilyess Bachiri, Janne Johansson, Kerry Su, Marcin Zelent, 52999 - Martin Michlmayr, Milly, Sơn Trần-Nguyễn 53000 - Mount 53001 - Update systemd status with cache stats (Hekmon) 53002 - Disable bazil/fuse based mount on macOS (Nick Craig-Wood) 53003 - Make rclone mount actually run rclone cmount under macOS 53004 (Nick Craig-Wood) 53005 - Implement mknod to make NFS file creation work (Nick Craig-Wood) 53006 - Make sure we don't call umount more than once (Nick Craig-Wood) 53007 - More user friendly mounting as network drive on windows 53008 (albertony) 53009 - Detect if uid or gid are set in same option string: -o 53010 uid=123,gid=456 (albertony) 53011 - Don't attempt to unmount if fs has been destroyed already (Nick 53012 Craig-Wood) 53013 - VFS 53014 - Fix virtual entries causing deleted files to still appear (Nick 53015 Craig-Wood) 53016 - Fix "file already exists" error for stale cache files (Nick 53017 Craig-Wood) 53018 - Fix file leaks with --vfs-cache-mode full and --buffer-size 0 53019 (Nick Craig-Wood) 53020 - Fix invalid cache path on windows when using :backend: as remote 53021 (albertony) 53022 - Local 53023 - Continue listing files/folders when a circular symlink is 53024 detected (Manish Gupta) 53025 - New flag --local-zero-size-links to fix sync on some virtual 53026 filesystems (Riccardo Iaconelli) 53027 - Azure Blob 53028 - Add support for service principals (James Lim) 53029 - Add support for managed identities (Brad Ackerman) 53030 - Add examples for access tier (Bob Pusateri) 53031 - Utilize the streaming capabilities from the SDK for multipart 53032 uploads (Denis Neuling) 53033 - Fix setting of mime types (Nick Craig-Wood) 53034 - Fix crash when listing outside a SAS URL's root (Nick 53035 Craig-Wood) 53036 - Delete archive tier blobs before update if 53037 --azureblob-archive-tier-delete (Nick Craig-Wood) 53038 - Fix crash on startup (Nick Craig-Wood) 53039 - Fix memory usage by upgrading the SDK to v0.13.0 and 53040 implementing a TransferManager (Nick Craig-Wood) 53041 - Require go1.14+ to compile due to SDK changes (Nick Craig-Wood) 53042 - B2 53043 - Make NewObject use less expensive API calls (Nick Craig-Wood) 53044 - This will improve --files-from and restic serve in 53045 particular 53046 - Fixed crash on an empty file name (lluuaapp) 53047 - Box 53048 - Fix NewObject for files that differ in case (Nick Craig-Wood) 53049 - Fix finding directories in a case insensitive way (Nick 53050 Craig-Wood) 53051 - Chunker 53052 - Skip long local hashing, hash in-transit (fixes) (Ivan Andreev) 53053 - Set Features ReadMimeType to false as Object.MimeType not 53054 supported (Nick Craig-Wood) 53055 - Fix case-insensitive NewObject, test metadata detection (Ivan 53056 Andreev) 53057 - Drive 53058 - Implement rclone backend copyid command for copying files by ID 53059 (Nick Craig-Wood) 53060 - Added flag --drive-stop-on-download-limit to stop transfers when 53061 the download limit is exceeded (Anagh Kumar Baranwal) 53062 - Implement CleanUp workaround for team drives (buengese) 53063 - Allow shortcut resolution and creation to be retried (Nick 53064 Craig-Wood) 53065 - Log that emptying the trash can take some time (Nick Craig-Wood) 53066 - Add xdg office icons to xdg desktop files (Pau 53067 Rodriguez-Estivill) 53068 - Dropbox 53069 - Add support for viewing shared files and folders (buengese) 53070 - Enable short lived access tokens (Nick Craig-Wood) 53071 - Implement IDer on Objects so rclone lsf etc can read the IDs 53072 (buengese) 53073 - Set Features ReadMimeType to false as Object.MimeType not 53074 supported (Nick Craig-Wood) 53075 - Make malformed_path errors from too long files not retriable 53076 (Nick Craig-Wood) 53077 - Test file name length before upload to fix upload loop (Nick 53078 Craig-Wood) 53079 - Fichier 53080 - Set Features ReadMimeType to true as Object.MimeType is 53081 supported (Nick Craig-Wood) 53082 - FTP 53083 - Add --ftp-disable-msld option to ignore MLSD for really old 53084 servers (Nick Craig-Wood) 53085 - Make --tpslimit apply (Nick Craig-Wood) 53086 - Google Cloud Storage 53087 - Storage class object header support (Laurens Janssen) 53088 - Fix anonymous client to use rclone's HTTP client (Nick 53089 Craig-Wood) 53090 - Fix 53091 Entry doesn't belong in directory "" (same as directory) - ignoring 53092 (Nick Craig-Wood) 53093 - Googlephotos 53094 - New flag --gphotos-include-archived to show archived photos as 53095 well (Nicolas Rueff) 53096 - Jottacloud 53097 - Don't erroneously report support for writing mime types 53098 (buengese) 53099 - Add support for Telia Cloud (Patrik Nordlén) 53100 - Mailru 53101 - Accept special folders eg camera-upload (Ivan Andreev) 53102 - Avoid prehashing of large local files (Ivan Andreev) 53103 - Fix uploads after recent changes on server (Ivan Andreev) 53104 - Fix range requests after June 2020 changes on server (Ivan 53105 Andreev) 53106 - Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev) 53107 - Remove deprecated protocol quirks (Ivan Andreev) 53108 - Memory 53109 - Fix setting of mime types (Nick Craig-Wood) 53110 - Onedrive 53111 - Add support for China region operated by 21vianet and other 53112 regional suppliers (NyaMisty) 53113 - Warn on gateway timeout errors (Nick Craig-Wood) 53114 - Fall back to normal copy if server-side copy unavailable (Alex 53115 Chen) 53116 - Fix server-side copy completely disabled on OneDrive for 53117 Business (Cnly) 53118 - (business only) workaround to replace existing file on 53119 server-side copy (Alex Chen) 53120 - Enhance link creation with expiry, scope, type and password 53121 (Nick Craig-Wood) 53122 - Remove % and # from the set of encoded characters (Alex Chen) 53123 - Support addressing site by server-relative URL (kice) 53124 - Opendrive 53125 - Fix finding directories in a case insensitive way (Nick 53126 Craig-Wood) 53127 - Pcloud 53128 - Fix setting of mime types (Nick Craig-Wood) 53129 - Premiumizeme 53130 - Fix finding directories in a case insensitive way (Nick 53131 Craig-Wood) 53132 - Qingstor 53133 - Fix error propagation in CleanUp (Nick Craig-Wood) 53134 - Fix rclone cleanup (Nick Craig-Wood) 53135 - S3 53136 - Added --s3-disable-http2 to disable http/2 (Anagh Kumar 53137 Baranwal) 53138 - Complete SSE-C implementation (Nick Craig-Wood) 53139 - Fix hashes on small files with AWS:KMS and SSE-C (Nick 53140 Craig-Wood) 53141 - Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C 53142 (Nick Craig-Wood) 53143 - Add --s3-no-head parameter to minimise transactions on upload 53144 (Nick Craig-Wood) 53145 - Update docs with a Reducing Costs section (Nick Craig-Wood) 53146 - Added error handling for error code 429 indicating too many 53147 requests (Anagh Kumar Baranwal) 53148 - Add requester pays option (kelv) 53149 - Fix copy multipart with v2 auth failing with 53150 'SignatureDoesNotMatch' (Louis Koo) 53151 - SFTP 53152 - Allow cert based auth via optional pubkey (Stephen Harris) 53153 - Allow user to optionally check server hosts key to add security 53154 (Stephen Harris) 53155 - Defer asking for user passwords until the SSH connection 53156 succeeds (Stephen Harris) 53157 - Remember entered password in AskPass mode (Stephen Harris) 53158 - Implement Shutdown method (Nick Craig-Wood) 53159 - Implement keyboard interactive authentication (Nick Craig-Wood) 53160 - Make --tpslimit apply (Nick Craig-Wood) 53161 - Implement --sftp-use-fstat for unusual SFTP servers (Nick 53162 Craig-Wood) 53163 - Sugarsync 53164 - Fix NewObject for files that differ in case (Nick Craig-Wood) 53165 - Fix finding directories in a case insensitive way (Nick 53166 Craig-Wood) 53167 - Swift 53168 - Fix deletion of parts of Static Large Object (SLO) (Nguyễn Hữu 53169 Luân) 53170 - Ensure partially uploaded large files are uploaded unless 53171 --swift-leave-parts-on-error (Nguyễn Hữu Luân) 53172 - Tardigrade 53173 - Upgrade to uplink v1.4.1 (Caleb Case) 53174 - WebDAV 53175 - Updated docs to show streaming to nextcloud is working (Durval 53176 Menezes) 53177 - Yandex 53178 - Set Features WriteMimeType to false as Yandex ignores mime types 53179 (Nick Craig-Wood) 53180 53181 v1.53.4 - 2021-01-20 53182 53183 See commits 53184 53185 - Bug Fixes 53186 - accounting: Fix data race in Transferred() (Maciej Zimnoch) 53187 - build 53188 - Stop tagged releases making a current beta (Nick Craig-Wood) 53189 - Upgrade docker buildx action (Matteo Pietro Dazzi) 53190 - Add -buildmode to cross-compile.go (Nick Craig-Wood) 53191 - Fix docker build by upgrading ilteoood/docker_buildx (Nick 53192 Craig-Wood) 53193 - Revert GitHub actions brew fix since this is now fixed (Nick 53194 Craig-Wood) 53195 - Fix brew install --cask syntax for macOS build (Nick 53196 Craig-Wood) 53197 - Update nfpm syntax to fix build of .deb/.rpm packages (Nick 53198 Craig-Wood) 53199 - Fix for Windows build errors (Ivan Andreev) 53200 - fs: Parseduration: fixed tests to use UTC time (Ankur Gupta) 53201 - fshttp: Prevent overlap of HTTP headers in logs (Nathan Collins) 53202 - rc 53203 - Fix core/command giving 500 internal error (Nick Craig-Wood) 53204 - Add Copy method to rc.Params (Nick Craig-Wood) 53205 - Fix 500 error when marshalling errors from core/command 53206 (Nick Craig-Wood) 53207 - plugins: Create plugins files only if webui is enabled. 53208 (negative0) 53209 - serve http: Fix serving files of unknown length (Nick 53210 Craig-Wood) 53211 - serve sftp: Fix authentication on one connection blocking others 53212 (Nick Craig-Wood) 53213 - Mount 53214 - Add optional brew tag to throw an error when using mount in the 53215 binaries installed via Homebrew (Anagh Kumar Baranwal) 53216 - Add "." and ".." to directories to match cmount and expectations 53217 (Nick Craig-Wood) 53218 - VFS 53219 - Make cache dir absolute before using it to fix path too long 53220 errors (Nick Craig-Wood) 53221 - Chunker 53222 - Improve detection of incompatible metadata (Ivan Andreev) 53223 - Google Cloud Storage 53224 - Fix server side copy of large objects (Nick Craig-Wood) 53225 - Jottacloud 53226 - Fix token renewer to fix long uploads (Nick Craig-Wood) 53227 - Fix token refresh failed: is not a regular file error (Nick 53228 Craig-Wood) 53229 - Pcloud 53230 - Only use SHA1 hashes in EU region (Nick Craig-Wood) 53231 - Sharefile 53232 - Undo Fix backend due to API swapping integers for strings (Nick 53233 Craig-Wood) 53234 - WebDAV 53235 - Fix Open Range requests to fix 4shared mount (Nick Craig-Wood) 53236 - Add "Depth: 0" to GET requests to fix bitrix (Nick Craig-Wood) 53237 53238 v1.53.3 - 2020-11-19 53239 53240 See commits 53241 53242 - Bug Fixes 53243 - random: Fix incorrect use of math/rand instead of crypto/rand 53244 CVE-2020-28924 (Nick Craig-Wood) 53245 - Passwords you have generated with rclone config may be 53246 insecure 53247 - See issue #4783 for more details and a checking tool 53248 - random: Seed math/rand in one place with crypto strong seed 53249 (Nick Craig-Wood) 53250 - VFS 53251 - Fix vfs/refresh calls with fs= parameter (Nick Craig-Wood) 53252 - Sharefile 53253 - Fix backend due to API swapping integers for strings (Nick 53254 Craig-Wood) 53255 53256 v1.53.2 - 2020-10-26 53257 53258 See commits 53259 53260 - Bug Fixes 53261 - accounting 53262 - Fix incorrect speed and transferTime in core/stats (Nick 53263 Craig-Wood) 53264 - Stabilize display order of transfers on Windows (Nick 53265 Craig-Wood) 53266 - operations 53267 - Fix use of --suffix without --backup-dir (Nick Craig-Wood) 53268 - Fix spurious "--checksum is in use but the source and 53269 destination have no hashes in common" (Nick Craig-Wood) 53270 - build 53271 - Work around GitHub actions brew problem (Nick Craig-Wood) 53272 - Stop using set-env and set-path in the GitHub actions (Nick 53273 Craig-Wood) 53274 - Mount 53275 - mount2: Fix the swapped UID / GID values (Russell Cattelan) 53276 - VFS 53277 - Detect and recover from a file being removed externally from the 53278 cache (Nick Craig-Wood) 53279 - Fix a deadlock vulnerability in downloaders.Close (Leo Luan) 53280 - Fix a race condition in retryFailedResets (Leo Luan) 53281 - Fix missed concurrency control between some item operations and 53282 reset (Leo Luan) 53283 - Add exponential backoff during ENOSPC retries (Leo Luan) 53284 - Add a missed update of used cache space (Leo Luan) 53285 - Fix --no-modtime to not attempt to set modtimes (as documented) 53286 (Nick Craig-Wood) 53287 - Local 53288 - Fix sizes and syncing with --links option on Windows (Nick 53289 Craig-Wood) 53290 - Chunker 53291 - Disable ListR to fix missing files on GDrive (workaround) (Ivan 53292 Andreev) 53293 - Fix upload over crypt (Ivan Andreev) 53294 - Fichier 53295 - Increase maximum file size from 100GB to 300GB (gyutw) 53296 - Jottacloud 53297 - Remove clientSecret from config when upgrading to token based 53298 authentication (buengese) 53299 - Avoid double url escaping of device/mountpoint (albertony) 53300 - Remove DirMove workaround as it's not required anymore - also 53301 (buengese) 53302 - Mailru 53303 - Fix uploads after recent changes on server (Ivan Andreev) 53304 - Fix range requests after june changes on server (Ivan Andreev) 53305 - Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev) 53306 - Onedrive 53307 - Fix disk usage for sharepoint (Nick Craig-Wood) 53308 - S3 53309 - Add missing regions for AWS (Anagh Kumar Baranwal) 53310 - Seafile 53311 - Fix accessing libraries > 2GB on 32 bit systems (Muffin King) 53312 - SFTP 53313 - Always convert the checksum to lower case (buengese) 53314 - Union 53315 - Create root directories if none exist (Nick Craig-Wood) 53316 53317 v1.53.1 - 2020-09-13 53318 53319 See commits 53320 53321 - Bug Fixes 53322 - accounting: Remove new line from end of --stats-one-line display 53323 (Nick Craig-Wood) 53324 - check 53325 - Add back missing --download flag (Nick Craig-Wood) 53326 - Fix docs (Nick Craig-Wood) 53327 - docs 53328 - Note --log-file does append (Nick Craig-Wood) 53329 - Add full stops for consistency in rclone --help (edwardxml) 53330 - Add Tencent COS to s3 provider list (wjielai) 53331 - Updated mount command to reflect that it requires Go 1.13 or 53332 newer (Evan Harris) 53333 - jottacloud: Mention that uploads from local disk will not 53334 need to cache files to disk for md5 calculation (albertony) 53335 - Fix formatting of rc docs page (Nick Craig-Wood) 53336 - build 53337 - Include vendor tar ball in release and fix startdev (Nick 53338 Craig-Wood) 53339 - Fix "Illegal instruction" error for ARMv6 builds (Nick 53340 Craig-Wood) 53341 - Fix architecture name in ARMv7 build (Nick Craig-Wood) 53342 - VFS 53343 - Fix spurious error "vfs cache: failed to _ensure cache EOF" 53344 (Nick Craig-Wood) 53345 - Log an ERROR if we fail to set the file to be sparse (Nick 53346 Craig-Wood) 53347 - Local 53348 - Log an ERROR if we fail to set the file to be sparse (Nick 53349 Craig-Wood) 53350 - Drive 53351 - Re-adds special oauth help text (Tim Gallant) 53352 - Opendrive 53353 - Do not retry 400 errors (Evan Harris) 53354 53355 v1.53.0 - 2020-09-02 53356 53357 See commits 53358 53359 - New Features 53360 - The VFS layer was heavily reworked for this release - see below 53361 for more details 53362 - Interactive mode -i/--interactive for destructive operations 53363 (fishbullet) 53364 - Add --bwlimit-file flag to limit speeds of individual file 53365 transfers (Nick Craig-Wood) 53366 - Transfers are sorted by start time in the stats and progress 53367 output (Max Sum) 53368 - Make sure backends expand ~ and environment vars in file names 53369 they use (Nick Craig-Wood) 53370 - Add --refresh-times flag to set modtimes on hashless backends 53371 (Nick Craig-Wood) 53372 - build 53373 - Remove vendor directory in favour of Go modules (Nick 53374 Craig-Wood) 53375 - Build with go1.15.x by default (Nick Craig-Wood) 53376 - Drop macOS 386 build as it is no longer supported by go1.15 53377 (Nick Craig-Wood) 53378 - Add ARMv7 to the supported builds (Nick Craig-Wood) 53379 - Enable rclone cmount on macOS (Nick Craig-Wood) 53380 - Make rclone build with gccgo (Nick Craig-Wood) 53381 - Make rclone build with wasm (Nick Craig-Wood) 53382 - Change beta numbering to be semver compatible (Nick 53383 Craig-Wood) 53384 - Add file properties and icon to Windows executable 53385 (albertony) 53386 - Add experimental interface for integrating rclone into 53387 browsers (Nick Craig-Wood) 53388 - lib: Add file name compression (Klaus Post) 53389 - rc 53390 - Allow installation and use of plugins and test plugins with 53391 rclone-webui (Chaitanya Bankanhal) 53392 - Add reverse proxy pluginsHandler for serving plugins 53393 (Chaitanya Bankanhal) 53394 - Add mount/listmounts option for listing current mounts 53395 (Chaitanya Bankanhal) 53396 - Add operations/uploadfile to upload a file through rc using 53397 encoding multipart/form-data (Chaitanya Bankanhal) 53398 - Add core/command to execute rclone terminal commands. 53399 (Chaitanya Bankanhal) 53400 - rclone check 53401 - Add reporting of filenames for same/missing/changed (Nick 53402 Craig-Wood) 53403 - Make check command obey --dry-run/-i/--interactive (Nick 53404 Craig-Wood) 53405 - Make check do --checkers files concurrently (Nick 53406 Craig-Wood) 53407 - Retry downloads if they fail when using the --download flag 53408 (Nick Craig-Wood) 53409 - Make it show stats by default (Nick Craig-Wood) 53410 - rclone obscure: Allow obscure command to accept password on 53411 STDIN (David Ibarra) 53412 - rclone config 53413 - Set RCLONE_CONFIG_DIR for use in config files and 53414 subprocesses (Nick Craig-Wood) 53415 - Reject remote names starting with a dash. (jtagcat) 53416 - rclone cryptcheck: Add reporting of filenames for 53417 same/missing/changed (Nick Craig-Wood) 53418 - rclone dedupe: Make it obey the --size-only flag for duplicate 53419 detection (Nick Craig-Wood) 53420 - rclone link: Add --expire and --unlink flags (Roman Kredentser) 53421 - rclone mkdir: Warn when using mkdir on remotes which can't have 53422 empty directories (Nick Craig-Wood) 53423 - rclone rc: Allow JSON parameters to simplify command line usage 53424 (Nick Craig-Wood) 53425 - rclone serve ftp 53426 - Don't compile on < go1.13 after dependency update (Nick 53427 Craig-Wood) 53428 - Add error message if auth proxy fails (Nick Craig-Wood) 53429 - Use refactored goftp.io/server library for binary shrink 53430 (Nick Craig-Wood) 53431 - rclone serve restic: Expose interfaces so that rclone can be 53432 used as a library from within restic (Jack) 53433 - rclone sync: Add --track-renames-strategy leaf (Nick Craig-Wood) 53434 - rclone touch: Add ability to set nanosecond resolution times 53435 (Nick Craig-Wood) 53436 - rclone tree: Remove -i shorthand for --noindent as it conflicts 53437 with -i/--interactive (Nick Craig-Wood) 53438 - Bug Fixes 53439 - accounting 53440 - Fix documentation for speed/speedAvg (Nick Craig-Wood) 53441 - Fix elapsed time not show actual time since beginning 53442 (Chaitanya Bankanhal) 53443 - Fix deadlock in stats printing (Nick Craig-Wood) 53444 - build 53445 - Fix file handle leak in GitHub release tool (Garrett Squire) 53446 - rclone check: Fix successful retries with --download counting 53447 errors (Nick Craig-Wood) 53448 - rclone dedupe: Fix logging to be easier to understand (Nick 53449 Craig-Wood) 53450 - Mount 53451 - Warn macOS users that mount implementation is changing (Nick 53452 Craig-Wood) 53453 - to test the new implementation use rclone cmount instead of 53454 rclone mount 53455 - this is because the library rclone uses has dropped macOS 53456 support 53457 - rc interface 53458 - Add call for unmount all (Chaitanya Bankanhal) 53459 - Make mount/mount remote control take vfsOpt option (Nick 53460 Craig-Wood) 53461 - Add mountOpt to mount/mount (Nick Craig-Wood) 53462 - Add VFS and Mount options to mount/listmounts (Nick 53463 Craig-Wood) 53464 - Catch panics in cgofuse initialization and turn into error 53465 messages (Nick Craig-Wood) 53466 - Always supply stat information in Readdir (Nick Craig-Wood) 53467 - Add support for reading unknown length files using direct IO 53468 (Windows) (Nick Craig-Wood) 53469 - Fix On Windows don't add -o uid/gid=-1 if user supplies 53470 -o uid/gid. (Nick Craig-Wood) 53471 - Fix macOS losing directory contents in cmount (Nick Craig-Wood) 53472 - Fix volume name broken in recent refactor (Nick Craig-Wood) 53473 - VFS 53474 - Implement partial reads for --vfs-cache-mode full (Nick 53475 Craig-Wood) 53476 - Add --vfs-writeback option to delay writes back to cloud storage 53477 (Nick Craig-Wood) 53478 - Add --vfs-read-ahead parameter for use with 53479 --vfs-cache-mode full (Nick Craig-Wood) 53480 - Restart pending uploads on restart of the cache (Nick 53481 Craig-Wood) 53482 - Support synchronous cache space recovery upon ENOSPC (Leo Luan) 53483 - Allow ReadAt and WriteAt to run concurrently with themselves 53484 (Nick Craig-Wood) 53485 - Change modtime of file before upload to current (Rob Calistri) 53486 - Recommend --vfs-cache-modes writes on backends which can't 53487 stream (Nick Craig-Wood) 53488 - Add an optional fs parameter to vfs rc methods (Nick Craig-Wood) 53489 - Fix errors when using > 260 char files in the cache in Windows 53490 (Nick Craig-Wood) 53491 - Fix renaming of items while they are being uploaded (Nick 53492 Craig-Wood) 53493 - Fix very high load caused by slow directory listings (Nick 53494 Craig-Wood) 53495 - Fix renamed files not being uploaded with 53496 --vfs-cache-mode minimal (Nick Craig-Wood) 53497 - Fix directory locking caused by slow directory listings (Nick 53498 Craig-Wood) 53499 - Fix saving from chrome without --vfs-cache-mode writes (Nick 53500 Craig-Wood) 53501 - Local 53502 - Add --local-no-updated to provide a consistent view of changing 53503 objects (Nick Craig-Wood) 53504 - Add --local-no-set-modtime option to prevent modtime changes 53505 (tyhuber1) 53506 - Fix race conditions updating and reading Object metadata (Nick 53507 Craig-Wood) 53508 - Cache 53509 - Make any created backends be cached to fix rc problems (Nick 53510 Craig-Wood) 53511 - Fix dedupe on caches wrapping drives (Nick Craig-Wood) 53512 - Crypt 53513 - Add --crypt-server-side-across-configs flag (Nick Craig-Wood) 53514 - Make any created backends be cached to fix rc problems (Nick 53515 Craig-Wood) 53516 - Alias 53517 - Make any created backends be cached to fix rc problems (Nick 53518 Craig-Wood) 53519 - Azure Blob 53520 - Don't compile on < go1.13 after dependency update (Nick 53521 Craig-Wood) 53522 - B2 53523 - Implement server-side copy for files > 5GB (Nick Craig-Wood) 53524 - Cancel in progress multipart uploads and copies on rclone exit 53525 (Nick Craig-Wood) 53526 - Note that b2's encoding now allows but rclone's hasn't changed 53527 (Nick Craig-Wood) 53528 - Fix transfers when using download_url (Nick Craig-Wood) 53529 - Box 53530 - Implement rclone cleanup (buengese) 53531 - Cancel in progress multipart uploads and copies on rclone exit 53532 (Nick Craig-Wood) 53533 - Allow authentication with access token (David) 53534 - Chunker 53535 - Make any created backends be cached to fix rc problems (Nick 53536 Craig-Wood) 53537 - Drive 53538 - Add rclone backend drives to list shared drives (teamdrives) 53539 (Nick Craig-Wood) 53540 - Implement rclone backend untrash (Nick Craig-Wood) 53541 - Work around drive bug which didn't set modtime of copied docs 53542 (Nick Craig-Wood) 53543 - Added --drive-starred-only to only show starred files (Jay 53544 McEntire) 53545 - Deprecate --drive-alternate-export as it is no longer needed 53546 (themylogin) 53547 - Fix duplication of Google docs on server-side copy (Nick 53548 Craig-Wood) 53549 - Fix "panic: send on closed channel" when recycling dir entries 53550 (Nick Craig-Wood) 53551 - Dropbox 53552 - Add copyright detector info in limitations section in the docs 53553 (Alex Guerrero) 53554 - Fix rclone link by removing expires parameter (Nick Craig-Wood) 53555 - Fichier 53556 - Detect Flood detected: IP Locked error and sleep for 30s (Nick 53557 Craig-Wood) 53558 - FTP 53559 - Add explicit TLS support (Heiko Bornholdt) 53560 - Add support for --dump bodies and --dump auth for debugging 53561 (Nick Craig-Wood) 53562 - Fix interoperation with pure-ftpd (Nick Craig-Wood) 53563 - Google Cloud Storage 53564 - Add support for anonymous access (Kai Lüke) 53565 - Jottacloud 53566 - Bring back legacy authentication for use with whitelabel 53567 versions (buengese) 53568 - Switch to new api root - also implement a very ugly workaround 53569 for the DirMove failures (buengese) 53570 - Onedrive 53571 - Rework cancel of multipart uploads on rclone exit (Nick 53572 Craig-Wood) 53573 - Implement rclone cleanup (Nick Craig-Wood) 53574 - Add --onedrive-no-versions flag to remove old versions (Nick 53575 Craig-Wood) 53576 - Pcloud 53577 - Implement rclone link for public link creation (buengese) 53578 - Qingstor 53579 - Cancel in progress multipart uploads on rclone exit (Nick 53580 Craig-Wood) 53581 - S3 53582 - Preserve metadata when doing multipart copy (Nick Craig-Wood) 53583 - Cancel in progress multipart uploads and copies on rclone exit 53584 (Nick Craig-Wood) 53585 - Add rclone link for public link sharing (Roman Kredentser) 53586 - Add rclone backend restore command to restore objects from 53587 GLACIER (Nick Craig-Wood) 53588 - Add rclone cleanup and rclone backend cleanup to clean 53589 unfinished multipart uploads (Nick Craig-Wood) 53590 - Add rclone backend list-multipart-uploads to list unfinished 53591 multipart uploads (Nick Craig-Wood) 53592 - Add --s3-max-upload-parts support (Kamil Trzciński) 53593 - Add --s3-no-check-bucket for minimising rclone transactions and 53594 perms (Nick Craig-Wood) 53595 - Add --s3-profile and --s3-shared-credentials-file options (Nick 53596 Craig-Wood) 53597 - Use regional s3 us-east-1 endpoint (David) 53598 - Add Scaleway provider (Vincent Feltz) 53599 - Update IBM COS endpoints (Egor Margineanu) 53600 - Reduce the default --s3-copy-cutoff to < 5GB for Backblaze S3 53601 compatibility (Nick Craig-Wood) 53602 - Fix detection of bucket existing (Nick Craig-Wood) 53603 - SFTP 53604 - Use the absolute path instead of the relative path for listing 53605 for improved compatibility (Nick Craig-Wood) 53606 - Add --sftp-subsystem and --sftp-server-command options (aus) 53607 - Swift 53608 - Fix dangling large objects breaking the listing (Nick 53609 Craig-Wood) 53610 - Fix purge not deleting directory markers (Nick Craig-Wood) 53611 - Fix update multipart object removing all of its own parts (Nick 53612 Craig-Wood) 53613 - Fix missing hash from object returned from upload (Nick 53614 Craig-Wood) 53615 - Tardigrade 53616 - Upgrade to uplink v1.2.0 (Kaloyan Raev) 53617 - Union 53618 - Fix writing with the all policy (Nick Craig-Wood) 53619 - WebDAV 53620 - Fix directory creation with 4shared (Nick Craig-Wood) 53621 53622 v1.52.3 - 2020-08-07 53623 53624 See commits 53625 53626 - Bug Fixes 53627 - docs 53628 - Disable smart typography (e.g. en-dash) in MANUAL.* and man 53629 page (Nick Craig-Wood) 53630 - Update install.md to reflect minimum Go version (Evan 53631 Harris) 53632 - Update install from source instructions (Nick Craig-Wood) 53633 - make_manual: Support SOURCE_DATE_EPOCH (Morten Linderud) 53634 - log: Fix --use-json-log going to stderr not --log-file on 53635 Windows (Nick Craig-Wood) 53636 - serve dlna: Fix file list on Samsung Series 6+ TVs (Matteo 53637 Pietro Dazzi) 53638 - sync: Fix deadlock with --track-renames-strategy modtime (Nick 53639 Craig-Wood) 53640 - Cache 53641 - Fix moveto/copyto remote:file remote:file2 (Nick Craig-Wood) 53642 - Drive 53643 - Stop using root_folder_id as a cache (Nick Craig-Wood) 53644 - Make dangling shortcuts appear in listings (Nick Craig-Wood) 53645 - Drop "Disabling ListR" messages down to debug (Nick Craig-Wood) 53646 - Workaround and policy for Google Drive API (Dmitry Ustalov) 53647 - FTP 53648 - Add note to docs about home vs root directory selection (Nick 53649 Craig-Wood) 53650 - Onedrive 53651 - Fix reverting to Copy when Move would have worked (Nick 53652 Craig-Wood) 53653 - Avoid comma rendered in URL in onedrive.md (Kevin) 53654 - Pcloud 53655 - Fix oauth on European region "eapi.pcloud.com" (Nick Craig-Wood) 53656 - S3 53657 - Fix bucket Region auto detection when Region unset in config 53658 (Nick Craig-Wood) 53659 53660 v1.52.2 - 2020-06-24 53661 53662 See commits 53663 53664 - Bug Fixes 53665 - build 53666 - Fix docker release build action (Nick Craig-Wood) 53667 - Fix custom timezone in Docker image (NoLooseEnds) 53668 - check: Fix misleading message which printed errors instead of 53669 differences (Nick Craig-Wood) 53670 - errors: Add WSAECONNREFUSED and more to the list of retriable 53671 Windows errors (Nick Craig-Wood) 53672 - rcd: Fix incorrect prometheus metrics (Gary Kim) 53673 - serve restic: Fix flags so they use environment variables (Nick 53674 Craig-Wood) 53675 - serve webdav: Fix flags so they use environment variables (Nick 53676 Craig-Wood) 53677 - sync: Fix --track-renames-strategy modtime (Nick Craig-Wood) 53678 - Drive 53679 - Fix not being able to delete a directory with a trashed shortcut 53680 (Nick Craig-Wood) 53681 - Fix creating a directory inside a shortcut (Nick Craig-Wood) 53682 - Fix --drive-impersonate with cached root_folder_id (Nick 53683 Craig-Wood) 53684 - SFTP 53685 - Fix SSH key PEM loading (Zac Rubin) 53686 - Swift 53687 - Speed up deletes by not retrying segment container deletes (Nick 53688 Craig-Wood) 53689 - Tardigrade 53690 - Upgrade to uplink v1.1.1 (Caleb Case) 53691 - WebDAV 53692 - Fix free/used display for rclone about/df for certain backends 53693 (Nick Craig-Wood) 53694 53695 v1.52.1 - 2020-06-10 53696 53697 See commits 53698 53699 - Bug Fixes 53700 - lib/file: Fix SetSparse on Windows 7 which fixes downloads of 53701 files > 250MB (Nick Craig-Wood) 53702 - build 53703 - Update go.mod to go1.14 to enable -mod=vendor build (Nick 53704 Craig-Wood) 53705 - Remove quicktest from Dockerfile (Nick Craig-Wood) 53706 - Build Docker images with GitHub actions (Matteo Pietro 53707 Dazzi) 53708 - Update Docker build workflows (Nick Craig-Wood) 53709 - Set user_allow_other in /etc/fuse.conf in the Docker image 53710 (Nick Craig-Wood) 53711 - Fix xgo build after go1.14 go.mod update (Nick Craig-Wood) 53712 - docs 53713 - Add link to source and modified time to footer of every page 53714 (Nick Craig-Wood) 53715 - Remove manually set dates and use git dates instead (Nick 53716 Craig-Wood) 53717 - Minor tense, punctuation, brevity and positivity changes for 53718 the home page (edwardxml) 53719 - Remove leading slash in page reference in footer when 53720 present (Nick Craig-Wood) 53721 - Note commands which need obscured input in the docs (Nick 53722 Craig-Wood) 53723 - obscure: Write more help as we are referencing it elsewhere 53724 (Nick Craig-Wood) 53725 - VFS 53726 - Fix OS vs Unix path confusion - fixes ChangeNotify on Windows 53727 (Nick Craig-Wood) 53728 - Drive 53729 - Fix missing items when listing using --fast-list / ListR (Nick 53730 Craig-Wood) 53731 - Putio 53732 - Fix panic on Object.Open (Cenk Alti) 53733 - S3 53734 - Fix upload of single files into buckets without create 53735 permission (Nick Craig-Wood) 53736 - Fix --header-upload (Nick Craig-Wood) 53737 - Tardigrade 53738 - Fix listing bug by upgrading to v1.0.7 53739 - Set UserAgent to rclone (Caleb Case) 53740 53741 v1.52.0 - 2020-05-27 53742 53743 Special thanks to Martin Michlmayr for proof reading and correcting all 53744 the docs and Edward Barker for helping re-write the front page. 53745 53746 See commits 53747 53748 - New backends 53749 - Tardigrade backend for use with storj.io (Caleb Case) 53750 - Union re-write to have multiple writable remotes (Max Sum) 53751 - Seafile for Seafile server (Fred @creativeprojects) 53752 - New commands 53753 - backend: command for backend-specific commands (see backends) 53754 (Nick Craig-Wood) 53755 - cachestats: Deprecate in favour of rclone backend stats cache: 53756 (Nick Craig-Wood) 53757 - dbhashsum: Deprecate in favour of rclone hashsum DropboxHash 53758 (Nick Craig-Wood) 53759 - New Features 53760 - Add --header-download and --header-upload flags for setting HTTP 53761 headers when uploading/downloading (Tim Gallant) 53762 - Add --header flag to add HTTP headers to every HTTP transaction 53763 (Nick Craig-Wood) 53764 - Add --check-first to do all checking before starting transfers 53765 (Nick Craig-Wood) 53766 - Add --track-renames-strategy for configurable matching criteria 53767 for --track-renames (Bernd Schoolmann) 53768 - Add --cutoff-mode hard,soft,cautious (Shing Kit Chan & Franklyn 53769 Tackitt) 53770 - Filter flags (e.g. --files-from -) can read from stdin 53771 (fishbullet) 53772 - Add --error-on-no-transfer option (Jon Fautley) 53773 - Implement --order-by xxx,mixed for copying some small and some 53774 big files (Nick Craig-Wood) 53775 - Allow --max-backlog to be negative meaning as large as possible 53776 (Nick Craig-Wood) 53777 - Added --no-unicode-normalization flag to allow Unicode filenames 53778 to remain unique (Ben Zenker) 53779 - Allow --min-age/--max-age to take a date as well as a duration 53780 (Nick Craig-Wood) 53781 - Add rename statistics for file and directory renames (Nick 53782 Craig-Wood) 53783 - Add statistics output to JSON log (reddi) 53784 - Make stats be printed on non-zero exit code (Nick Craig-Wood) 53785 - When running --password-command allow use of stdin (Sébastien 53786 Gross) 53787 - Stop empty strings being a valid remote path (Nick Craig-Wood) 53788 - accounting: support WriterTo for less memory copying (Nick 53789 Craig-Wood) 53790 - build 53791 - Update to use go1.14 for the build (Nick Craig-Wood) 53792 - Add -trimpath to release build for reproduceable builds 53793 (Nick Craig-Wood) 53794 - Remove GOOS and GOARCH from Dockerfile (Brandon Philips) 53795 - config 53796 - Fsync the config file after writing to save more reliably 53797 (Nick Craig-Wood) 53798 - Add --obscure and --no-obscure flags to config create/update 53799 (Nick Craig-Wood) 53800 - Make config show take remote: as well as remote (Nick 53801 Craig-Wood) 53802 - copyurl: Add --no-clobber flag (Denis) 53803 - delete: Added --rmdirs flag to delete directories as well (Kush) 53804 - filter: Added --files-from-raw flag (Ankur Gupta) 53805 - genautocomplete: Add support for fish shell (Matan Rosenberg) 53806 - log: Add support for syslog LOCAL facilities (Patryk Jakuszew) 53807 - lsjson: Add --hash-type parameter and use it in lsf to speed up 53808 hashing (Nick Craig-Wood) 53809 - rc 53810 - Add -o/--opt and -a/--arg for more structured input (Nick 53811 Craig-Wood) 53812 - Implement backend/command for running backend-specific 53813 commands remotely (Nick Craig-Wood) 53814 - Add mount/mount command for starting rclone mount via the 53815 API (Chaitanya) 53816 - rcd: Add Prometheus metrics support (Gary Kim) 53817 - serve http 53818 - Added a --template flag for user defined markup (calistri) 53819 - Add Last-Modified headers to files and directories (Nick 53820 Craig-Wood) 53821 - serve sftp: Add support for multiple host keys by repeating 53822 --key flag (Maxime Suret) 53823 - touch: Add --localtime flag to make --timestamp localtime not 53824 UTC (Nick Craig-Wood) 53825 - Bug Fixes 53826 - accounting 53827 - Restore "Max number of stats groups reached" log line 53828 (Michał Matczuk) 53829 - Correct exitcode on Transfer Limit Exceeded flag. (Anuar 53830 Serdaliyev) 53831 - Reset bytes read during copy retry (Ankur Gupta) 53832 - Fix race clearing stats (Nick Craig-Wood) 53833 - copy: Only create empty directories when they don't exist on the 53834 remote (Ishuah Kariuki) 53835 - dedupe: Stop dedupe deleting files with identical IDs (Nick 53836 Craig-Wood) 53837 - oauth 53838 - Use custom http client so that --no-check-certificate is 53839 honored by oauth token fetch (Mark Spieth) 53840 - Replace deprecated oauth2.NoContext (Lars Lehtonen) 53841 - operations 53842 - Fix setting the timestamp on Windows for multithread copy 53843 (Nick Craig-Wood) 53844 - Make rcat obey --ignore-checksum (Nick Craig-Wood) 53845 - Make --max-transfer more accurate (Nick Craig-Wood) 53846 - rc 53847 - Fix dropped error (Lars Lehtonen) 53848 - Fix misplaced http server config (Xiaoxing Ye) 53849 - Disable duplicate log (ElonH) 53850 - serve dlna 53851 - Cds: don't specify childCount at all when unknown (Dan 53852 Walters) 53853 - Cds: use modification time as date in dlna metadata (Dan 53854 Walters) 53855 - serve restic: Fix tests after restic project removed vendoring 53856 (Nick Craig-Wood) 53857 - sync 53858 - Fix incorrect "nothing to transfer" message using 53859 --delete-before (Nick Craig-Wood) 53860 - Only create empty directories when they don't exist on the 53861 remote (Ishuah Kariuki) 53862 - Mount 53863 - Add --async-read flag to disable asynchronous reads (Nick 53864 Craig-Wood) 53865 - Ignore --allow-root flag with a warning as it has been removed 53866 upstream (Nick Craig-Wood) 53867 - Warn if --allow-non-empty used on Windows and clarify docs (Nick 53868 Craig-Wood) 53869 - Constrain to go1.13 or above otherwise bazil.org/fuse fails to 53870 compile (Nick Craig-Wood) 53871 - Fix fail because of too long volume name (evileye) 53872 - Report 1PB free for unknown disk sizes (Nick Craig-Wood) 53873 - Map more rclone errors into file systems errors (Nick 53874 Craig-Wood) 53875 - Fix disappearing cwd problem (Nick Craig-Wood) 53876 - Use ReaddirPlus on Windows to improve directory listing 53877 performance (Nick Craig-Wood) 53878 - Send a hint as to whether the filesystem is case insensitive or 53879 not (Nick Craig-Wood) 53880 - Add rc command mount/types (Nick Craig-Wood) 53881 - Change maximum leaf name length to 1024 bytes (Nick Craig-Wood) 53882 - VFS 53883 - Add --vfs-read-wait and --vfs-write-wait flags to control time 53884 waiting for a sequential read/write (Nick Craig-Wood) 53885 - Change default --vfs-read-wait to 20ms (it was 5ms and not 53886 configurable) (Nick Craig-Wood) 53887 - Make df output more consistent on a rclone mount. (Yves G) 53888 - Report 1PB free for unknown disk sizes (Nick Craig-Wood) 53889 - Fix race condition caused by unlocked reading of Dir.path (Nick 53890 Craig-Wood) 53891 - Make File lock and Dir lock not overlap to avoid deadlock (Nick 53892 Craig-Wood) 53893 - Implement lock ordering between File and Dir to eliminate 53894 deadlocks (Nick Craig-Wood) 53895 - Factor the vfs cache into its own package (Nick Craig-Wood) 53896 - Pin the Fs in use in the Fs cache (Nick Craig-Wood) 53897 - Add SetSys() methods to Node to allow caching stuff on a node 53898 (Nick Craig-Wood) 53899 - Ignore file not found errors from Hash in Read.Release (Nick 53900 Craig-Wood) 53901 - Fix hang in read wait code (Nick Craig-Wood) 53902 - Local 53903 - Speed up multi thread downloads by using sparse files on Windows 53904 (Nick Craig-Wood) 53905 - Implement --local-no-sparse flag for disabling sparse files 53906 (Nick Craig-Wood) 53907 - Implement rclone backend noop for testing purposes (Nick 53908 Craig-Wood) 53909 - Fix "file not found" errors on post transfer Hash calculation 53910 (Nick Craig-Wood) 53911 - Cache 53912 - Implement rclone backend stats command (Nick Craig-Wood) 53913 - Fix Server Side Copy with Temp Upload (Brandon McNama) 53914 - Remove Unused Functions (Lars Lehtonen) 53915 - Disable race tests until bbolt is fixed (Nick Craig-Wood) 53916 - Move methods used for testing into test file (greatroar) 53917 - Add Pin and Unpin and canonicalised lookup (Nick Craig-Wood) 53918 - Use proper import path go.etcd.io/bbolt (Robert-André Mauchin) 53919 - Crypt 53920 - Calculate hashes for uploads from local disk (Nick Craig-Wood) 53921 - This allows encrypted Jottacloud uploads without using local 53922 disk 53923 - This means encrypted s3/b2 uploads will now have hashes 53924 - Added rclone backend decode/encode commands to replicate 53925 functionality of cryptdecode (Anagh Kumar Baranwal) 53926 - Get rid of the unused Cipher interface as it obfuscated the code 53927 (Nick Craig-Wood) 53928 - Azure Blob 53929 - Implement streaming of unknown sized files so rcat is now 53930 supported (Nick Craig-Wood) 53931 - Implement memory pooling to control memory use (Nick Craig-Wood) 53932 - Add --azureblob-disable-checksum flag (Nick Craig-Wood) 53933 - Retry InvalidBlobOrBlock error as it may indicate block 53934 concurrency problems (Nick Craig-Wood) 53935 - Remove unused Object.parseTimeString() (Lars Lehtonen) 53936 - Fix permission error on SAS URL limited to container (Nick 53937 Craig-Wood) 53938 - B2 53939 - Add support for --header-upload and --header-download (Tim 53940 Gallant) 53941 - Ignore directory markers at the root also (Nick Craig-Wood) 53942 - Force the case of the SHA1 to lowercase (Nick Craig-Wood) 53943 - Remove unused largeUpload.clearUploadURL() (Lars Lehtonen) 53944 - Box 53945 - Add support for --header-upload and --header-download (Tim 53946 Gallant) 53947 - Implement About to read size used (Nick Craig-Wood) 53948 - Add token renew function for jwt auth (David Bramwell) 53949 - Added support for interchangeable root folder for Box backend 53950 (Sunil Patra) 53951 - Remove unnecessary iat from jws claims (David) 53952 - Drive 53953 - Follow shortcuts by default, skip with --drive-skip-shortcuts 53954 (Nick Craig-Wood) 53955 - Implement rclone backend shortcut command for creating shortcuts 53956 (Nick Craig-Wood) 53957 - Added rclone backend command to change service_account_file and 53958 chunk_size (Anagh Kumar Baranwal) 53959 - Fix missing files when using --fast-list and 53960 --drive-shared-with-me (Nick Craig-Wood) 53961 - Fix duplicate items when using --drive-shared-with-me (Nick 53962 Craig-Wood) 53963 - Extend --drive-stop-on-upload-limit to respond to 53964 teamDriveFileLimitExceeded. (harry) 53965 - Don't delete files with multiple parents to avoid data loss 53966 (Nick Craig-Wood) 53967 - Server side copy docs use default description if empty (Nick 53968 Craig-Wood) 53969 - Dropbox 53970 - Make error insufficient space to be fatal (harry) 53971 - Add info about required redirect url (Elan Ruusamäe) 53972 - Fichier 53973 - Add support for --header-upload and --header-download (Tim 53974 Gallant) 53975 - Implement custom pacer to deal with the new rate limiting 53976 (buengese) 53977 - FTP 53978 - Fix lockup when using concurrency limit on failed connections 53979 (Nick Craig-Wood) 53980 - Fix lockup on failed upload when using concurrency limit (Nick 53981 Craig-Wood) 53982 - Fix lockup on Close failures when using concurrency limit (Nick 53983 Craig-Wood) 53984 - Work around pureftp sending spurious 150 messages (Nick 53985 Craig-Wood) 53986 - Google Cloud Storage 53987 - Add support for --header-upload and --header-download (Nick 53988 Craig-Wood) 53989 - Add ARCHIVE storage class to help (Adam Stroud) 53990 - Ignore directory markers at the root (Nick Craig-Wood) 53991 - Googlephotos 53992 - Make the start year configurable (Daven) 53993 - Add support for --header-upload and --header-download (Tim 53994 Gallant) 53995 - Create feature/favorites directory (Brandon Philips) 53996 - Fix "concurrent map write" error (Nick Craig-Wood) 53997 - Don't put an image in error message (Nick Craig-Wood) 53998 - HTTP 53999 - Improved directory listing with new template from Caddy project 54000 (calisro) 54001 - Jottacloud 54002 - Implement --jottacloud-trashed-only (buengese) 54003 - Add support for --header-upload and --header-download (Tim 54004 Gallant) 54005 - Use RawURLEncoding when decoding base64 encoded login token 54006 (buengese) 54007 - Implement cleanup (buengese) 54008 - Update docs regarding cleanup, removed remains from old auth, 54009 and added warning about special mountpoints. (albertony) 54010 - Mailru 54011 - Describe 2FA requirements (valery1707) 54012 - Onedrive 54013 - Implement --onedrive-server-side-across-configs (Nick 54014 Craig-Wood) 54015 - Add support for --header-upload and --header-download (Tim 54016 Gallant) 54017 - Fix occasional 416 errors on multipart uploads (Nick Craig-Wood) 54018 - Added maximum chunk size limit warning in the docs (Harry) 54019 - Fix missing drive on config (Nick Craig-Wood) 54020 - Make error quotaLimitReached to be fatal (harry) 54021 - Opendrive 54022 - Add support for --header-upload and --header-download (Tim 54023 Gallant) 54024 - Pcloud 54025 - Added support for interchangeable root folder for pCloud backend 54026 (Sunil Patra) 54027 - Add support for --header-upload and --header-download (Tim 54028 Gallant) 54029 - Fix initial config "Auth state doesn't match" message (Nick 54030 Craig-Wood) 54031 - Premiumizeme 54032 - Add support for --header-upload and --header-download (Tim 54033 Gallant) 54034 - Prune unused functions (Lars Lehtonen) 54035 - Putio 54036 - Add support for --header-upload and --header-download (Nick 54037 Craig-Wood) 54038 - Make downloading files use the rclone http Client (Nick 54039 Craig-Wood) 54040 - Fix parsing of remotes with leading and trailing / (Nick 54041 Craig-Wood) 54042 - Qingstor 54043 - Make rclone cleanup remove pending multipart uploads older than 54044 24h (Nick Craig-Wood) 54045 - Try harder to cancel failed multipart uploads (Nick Craig-Wood) 54046 - Prune multiUploader.list() (Lars Lehtonen) 54047 - Lint fix (Lars Lehtonen) 54048 - S3 54049 - Add support for --header-upload and --header-download (Tim 54050 Gallant) 54051 - Use memory pool for buffer allocations (Maciej Zimnoch) 54052 - Add SSE-C support for AWS, Ceph, and MinIO (Jack Anderson) 54053 - Fail fast multipart upload (Michał Matczuk) 54054 - Report errors on bucket creation (mkdir) correctly (Nick 54055 Craig-Wood) 54056 - Specify that Minio supports URL encoding in listings (Nick 54057 Craig-Wood) 54058 - Added 500 as retryErrorCode (Michał Matczuk) 54059 - Use --low-level-retries as the number of SDK retries (Aleksandar 54060 Janković) 54061 - Fix multipart abort context (Aleksandar Jankovic) 54062 - Replace deprecated session.New() with session.NewSession() (Lars 54063 Lehtonen) 54064 - Use the provided size parameter when allocating a new memory 54065 pool (Joachim Brandon LeBlanc) 54066 - Use rclone's low level retries instead of AWS SDK to fix listing 54067 retries (Nick Craig-Wood) 54068 - Ignore directory markers at the root also (Nick Craig-Wood) 54069 - Use single memory pool (Michał Matczuk) 54070 - Do not resize buf on put to memBuf (Michał Matczuk) 54071 - Improve docs for --s3-disable-checksum (Nick Craig-Wood) 54072 - Don't leak memory or tokens in edge cases for multipart upload 54073 (Nick Craig-Wood) 54074 - Seafile 54075 - Implement 2FA (Fred) 54076 - SFTP 54077 - Added --sftp-pem-key to support inline key files (calisro) 54078 - Fix post transfer copies failing with 0 size when using 54079 set_modtime=false (Nick Craig-Wood) 54080 - Sharefile 54081 - Add support for --header-upload and --header-download (Tim 54082 Gallant) 54083 - Sugarsync 54084 - Add support for --header-upload and --header-download (Tim 54085 Gallant) 54086 - Swift 54087 - Add support for --header-upload and --header-download (Nick 54088 Craig-Wood) 54089 - Fix cosmetic issue in error message (Martin Michlmayr) 54090 - Union 54091 - Implement multiple writable remotes (Max Sum) 54092 - Fix server-side copy (Max Sum) 54093 - Implement ListR (Max Sum) 54094 - Enable ListR when upstreams contain local (Max Sum) 54095 - WebDAV 54096 - Add support for --header-upload and --header-download (Tim 54097 Gallant) 54098 - Fix X-OC-Mtime header for Transip compatibility (Nick 54099 Craig-Wood) 54100 - Report full and consistent usage with about (Yves G) 54101 - Yandex 54102 - Add support for --header-upload and --header-download (Tim 54103 Gallant) 54104 54105 v1.51.0 - 2020-02-01 54106 54107 - New backends 54108 - Memory (Nick Craig-Wood) 54109 - Sugarsync (Nick Craig-Wood) 54110 - New Features 54111 - Adjust all backends to have --backend-encoding parameter (Nick 54112 Craig-Wood) 54113 - this enables the encoding for special characters to be 54114 adjusted or disabled 54115 - Add --max-duration flag to control the maximum duration of a 54116 transfer session (boosh) 54117 - Add --expect-continue-timeout flag, default 1s (Nick Craig-Wood) 54118 - Add --no-check-dest flag for copying without testing the 54119 destination (Nick Craig-Wood) 54120 - Implement --order-by flag to order transfers (Nick Craig-Wood) 54121 - accounting 54122 - Don't show entries in both transferring and checking (Nick 54123 Craig-Wood) 54124 - Add option to delete stats (Aleksandar Jankovic) 54125 - build 54126 - Compress the test builds with gzip (Nick Craig-Wood) 54127 - Implement a framework for starting test servers during tests 54128 (Nick Craig-Wood) 54129 - cmd: Always print elapsed time to tenth place seconds in 54130 progress (Gary Kim) 54131 - config 54132 - Add --password-command to allow dynamic config password 54133 (Damon Permezel) 54134 - Give config questions default values (Nick Craig-Wood) 54135 - Check a remote exists when creating a new one (Nick 54136 Craig-Wood) 54137 - copyurl: Add --stdout flag to write to stdout (Nick Craig-Wood) 54138 - dedupe: Implement keep smallest too (Nick Craig-Wood) 54139 - hashsum: Add flag --base64 flag (landall) 54140 - lsf: Speed up on s3/swift/etc by not reading mimetype by default 54141 (Nick Craig-Wood) 54142 - lsjson: Add --no-mimetype flag (Nick Craig-Wood) 54143 - rc: Add methods to turn on blocking and mutex profiling (Nick 54144 Craig-Wood) 54145 - rcd 54146 - Adding group parameter to stats (Chaitanya) 54147 - Move webgui apart; option to disable browser (Xiaoxing Ye) 54148 - serve sftp: Add support for public key with auth proxy (Paul 54149 Tinsley) 54150 - stats: Show deletes in stats and hide zero stats (anuar45) 54151 - Bug Fixes 54152 - accounting 54153 - Fix error counter counting multiple times (Ankur Gupta) 54154 - Fix error count shown as checks (Cnly) 54155 - Clear finished transfer in stats-reset (Maciej Zimnoch) 54156 - Added StatsInfo locking in statsGroups sum function (Michał 54157 Matczuk) 54158 - asyncreader: Fix EOF error (buengese) 54159 - check: Fix --one-way recursing more directories than it needs to 54160 (Nick Craig-Wood) 54161 - chunkedreader: Disable hash calculation for first segment (Nick 54162 Craig-Wood) 54163 - config 54164 - Do not open browser on headless on drive/gcs/google photos 54165 (Xiaoxing Ye) 54166 - SetValueAndSave ignore error if config section does not 54167 exist yet (buengese) 54168 - cmd: Fix completion with an encrypted config (Danil Semelenov) 54169 - dbhashsum: Stop it returning UNSUPPORTED on dropbox (Nick 54170 Craig-Wood) 54171 - dedupe: Add missing modes to help string (Nick Craig-Wood) 54172 - operations 54173 - Fix dedupe continuing on errors like 54174 insufficientFilePersimmon (SezalAgrawal) 54175 - Clear accounting before low level retry (Maciej Zimnoch) 54176 - Write debug message when hashes could not be checked (Ole 54177 Schütt) 54178 - Move interface assertion to tests to remove pflag dependency 54179 (Nick Craig-Wood) 54180 - Make NewOverrideObjectInfo public and factor uses (Nick 54181 Craig-Wood) 54182 - proxy: Replace use of bcrypt with sha256 (Nick Craig-Wood) 54183 - vendor 54184 - Update bazil.org/fuse to fix FreeBSD 12.1 (Nick Craig-Wood) 54185 - Update github.com/t3rm1n4l/go-mega to fix mega "illegal 54186 base64 data at input byte 22" (Nick Craig-Wood) 54187 - Update termbox-go to fix ncdu command on FreeBSD (Kuang-che 54188 Wu) 54189 - Update t3rm1n4l/go-mega - fixes mega: couldn't login: 54190 crypto/aes: invalid key size 0 (Nick Craig-Wood) 54191 - Mount 54192 - Enable async reads for a 20% speedup (Nick Craig-Wood) 54193 - Replace use of WriteAt with Write for cache mode >= writes and 54194 O_APPEND (Brett Dutro) 54195 - Make sure we call unmount when exiting (Nick Craig-Wood) 54196 - Don't build on go1.10 as bazil/fuse no longer supports it (Nick 54197 Craig-Wood) 54198 - When setting dates discard out of range dates (Nick Craig-Wood) 54199 - VFS 54200 - Add a newly created file straight into the directory (Nick 54201 Craig-Wood) 54202 - Only calculate one hash for reads for a speedup (Nick 54203 Craig-Wood) 54204 - Make ReadAt for non cached files work better with non-sequential 54205 reads (Nick Craig-Wood) 54206 - Fix edge cases when reading ModTime from file (Nick Craig-Wood) 54207 - Make sure existing files opened for write show correct size 54208 (Nick Craig-Wood) 54209 - Don't cache the path in RW file objects to fix renaming (Nick 54210 Craig-Wood) 54211 - Fix rename of open files when using the VFS cache (Nick 54212 Craig-Wood) 54213 - When renaming files in the cache, rename the cache item in 54214 memory too (Nick Craig-Wood) 54215 - Fix open file renaming on drive when using 54216 --vfs-cache-mode writes (Nick Craig-Wood) 54217 - Fix incorrect modtime for mv into mount with 54218 --vfs-cache-modes writes (Nick Craig-Wood) 54219 - On rename, rename in cache too if the file exists (Anagh Kumar 54220 Baranwal) 54221 - Local 54222 - Make source file being updated errors be NoLowLevelRetry errors 54223 (Nick Craig-Wood) 54224 - Fix update of hidden files on Windows (Nick Craig-Wood) 54225 - Cache 54226 - Follow move of upstream library github.com/coreos/bbolt 54227 github.com/etcd-io/bbolt (Nick Craig-Wood) 54228 - Fix fatal error: concurrent map writes (Nick Craig-Wood) 54229 - Crypt 54230 - Reorder the filename encryption options (Thomas Eales) 54231 - Correctly handle trailing dot (buengese) 54232 - Chunker 54233 - Reduce length of temporary suffix (Ivan Andreev) 54234 - Drive 54235 - Add --drive-stop-on-upload-limit flag to stop syncs when upload 54236 limit reached (Nick Craig-Wood) 54237 - Add --drive-use-shared-date to use date file was shared instead 54238 of modified date (Garry McNulty) 54239 - Make sure invalid auth for teamdrives always reports an error 54240 (Nick Craig-Wood) 54241 - Fix --fast-list when using appDataFolder (Nick Craig-Wood) 54242 - Use multipart resumable uploads for streaming and uploads in 54243 mount (Nick Craig-Wood) 54244 - Log an ERROR if an incomplete search is returned (Nick 54245 Craig-Wood) 54246 - Hide dangerous config from the configurator (Nick Craig-Wood) 54247 - Dropbox 54248 - Treat insufficient_space errors as non retriable errors (Nick 54249 Craig-Wood) 54250 - Jottacloud 54251 - Use new auth method used by official client (buengese) 54252 - Add URL to generate Login Token to config wizard (Nick 54253 Craig-Wood) 54254 - Add support whitelabel versions (buengese) 54255 - Koofr 54256 - Use rclone HTTP client. (jaKa) 54257 - Onedrive 54258 - Add Sites.Read.All permission (Benjamin Richter) 54259 - Add support "Retry-After" header (Motonori IWAMURO) 54260 - Opendrive 54261 - Implement --opendrive-chunk-size (Nick Craig-Wood) 54262 - S3 54263 - Re-implement multipart upload to fix memory issues (Nick 54264 Craig-Wood) 54265 - Add --s3-copy-cutoff for size to switch to multipart copy (Nick 54266 Craig-Wood) 54267 - Add new region Asia Pacific (Hong Kong) (Outvi V) 54268 - Reduce memory usage streaming files by reducing max stream 54269 upload size (Nick Craig-Wood) 54270 - Add --s3-list-chunk option for bucket listing (Thomas 54271 Kriechbaumer) 54272 - Force path style bucket access to off for AWS deprecation (Nick 54273 Craig-Wood) 54274 - Use AWS web identity role provider if available (Tennix) 54275 - Add StackPath Object Storage Support (Dave Koston) 54276 - Fix ExpiryWindow value (Aleksandar Jankovic) 54277 - Fix DisableChecksum condition (Aleksandar Janković) 54278 - Fix URL decoding of NextMarker (Nick Craig-Wood) 54279 - SFTP 54280 - Add --sftp-skip-links to skip symlinks and non regular files 54281 (Nick Craig-Wood) 54282 - Retry Creation of Connection (Sebastian Brandt) 54283 - Fix "failed to parse private key file: ssh: not an encrypted 54284 key" error (Nick Craig-Wood) 54285 - Open files for update write only to fix AWS SFTP interop (Nick 54286 Craig-Wood) 54287 - Swift 54288 - Reserve segments of dynamic large object when delete objects in 54289 container what was enabled versioning. (Nguyễn Hữu Luân) 54290 - Fix parsing of X-Object-Manifest (Nick Craig-Wood) 54291 - Update OVH API endpoint (unbelauscht) 54292 - WebDAV 54293 - Make nextcloud only upload SHA1 checksums (Nick Craig-Wood) 54294 - Fix case of "Bearer" in Authorization: header to agree with RFC 54295 (Nick Craig-Wood) 54296 - Add Referer header to fix problems with WAFs (Nick Craig-Wood) 54297 54298 v1.50.2 - 2019-11-19 54299 54300 - Bug Fixes 54301 - accounting: Fix memory leak on retries operations (Nick 54302 Craig-Wood) 54303 - Drive 54304 - Fix listing of the root directory with drive.files scope (Nick 54305 Craig-Wood) 54306 - Fix --drive-root-folder-id with team/shared drives (Nick 54307 Craig-Wood) 54308 54309 v1.50.1 - 2019-11-02 54310 54311 - Bug Fixes 54312 - hash: Fix accidentally changed hash names for DropboxHash and 54313 CRC-32 (Nick Craig-Wood) 54314 - fshttp: Fix error reporting on tpslimit token bucket errors 54315 (Nick Craig-Wood) 54316 - fshttp: Don't print token bucket errors on context cancelled 54317 (Nick Craig-Wood) 54318 - Local 54319 - Fix listings of . on Windows (Nick Craig-Wood) 54320 - Onedrive 54321 - Fix DirMove/Move after Onedrive change (Xiaoxing Ye) 54322 54323 v1.50.0 - 2019-10-26 54324 54325 - New backends 54326 - Citrix Sharefile (Nick Craig-Wood) 54327 - Chunker - an overlay backend to split files into smaller parts 54328 (Ivan Andreev) 54329 - Mail.ru Cloud (Ivan Andreev) 54330 - New Features 54331 - encodings (Fabian Möller & Nick Craig-Wood) 54332 - All backends now use file name encoding to ensure any file 54333 name can be written to any backend. 54334 - See the restricted file name docs for more info and the 54335 local backend docs. 54336 - Some file names may look different in rclone if you are 54337 using any control characters in names or unicode FULLWIDTH 54338 symbols. 54339 - build 54340 - Update to use go1.13 for the build (Nick Craig-Wood) 54341 - Drop support for go1.9 (Nick Craig-Wood) 54342 - Build rclone with GitHub actions (Nick Craig-Wood) 54343 - Convert python scripts to python3 (Nick Craig-Wood) 54344 - Swap Azure/go-ansiterm for mattn/go-colorable (Nick 54345 Craig-Wood) 54346 - Dockerfile fixes (Matei David) 54347 - Add plugin support for backends and commands (Richard Patel) 54348 - config 54349 - Use alternating Red/Green in config to make more obvious 54350 (Nick Craig-Wood) 54351 - contrib 54352 - Add sample DLNA server Docker Compose manifest. (pataquets) 54353 - Add sample WebDAV server Docker Compose manifest. 54354 (pataquets) 54355 - copyurl 54356 - Add --auto-filename flag for using file name from URL in 54357 destination path (Denis) 54358 - serve dlna: 54359 - Many compatibility improvements (Dan Walters) 54360 - Support for external srt subtitles (Dan Walters) 54361 - rc 54362 - Added command core/quit (Saksham Khanna) 54363 - Bug Fixes 54364 - sync 54365 - Make --update/-u not transfer files that haven't changed 54366 (Nick Craig-Wood) 54367 - Free objects after they come out of the transfer pipe to 54368 save memory (Nick Craig-Wood) 54369 - Fix --files-from without --no-traverse doing a recursive 54370 scan (Nick Craig-Wood) 54371 - operations 54372 - Fix accounting for server-side copies (Nick Craig-Wood) 54373 - Display 'All duplicates removed' only if dedupe successful 54374 (Sezal Agrawal) 54375 - Display 'Deleted X extra copies' only if dedupe successful 54376 (Sezal Agrawal) 54377 - accounting 54378 - Only allow up to 100 completed transfers in the accounting 54379 list to save memory (Nick Craig-Wood) 54380 - Cull the old time ranges when possible to save memory (Nick 54381 Craig-Wood) 54382 - Fix panic due to server-side copy fallback (Ivan Andreev) 54383 - Fix memory leak noticeable for transfers of large numbers of 54384 objects (Nick Craig-Wood) 54385 - Fix total duration calculation (Nick Craig-Wood) 54386 - cmd 54387 - Fix environment variables not setting command line flags 54388 (Nick Craig-Wood) 54389 - Make autocomplete compatible with bash's posix mode for 54390 macOS (Danil Semelenov) 54391 - Make --progress work in git bash on Windows (Nick 54392 Craig-Wood) 54393 - Fix 'compopt: command not found' on autocomplete on macOS 54394 (Danil Semelenov) 54395 - config 54396 - Fix setting of non top level flags from environment 54397 variables (Nick Craig-Wood) 54398 - Check config names more carefully and report errors (Nick 54399 Craig-Wood) 54400 - Remove error: can't use --size-only and --ignore-size 54401 together. (Nick Craig-Wood) 54402 - filter: Prevent mixing options when --files-from is in use 54403 (Michele Caci) 54404 - serve sftp: Fix crash on unsupported operations (e.g. Readlink) 54405 (Nick Craig-Wood) 54406 - Mount 54407 - Allow files of unknown size to be read properly (Nick 54408 Craig-Wood) 54409 - Skip tests on <= 2 CPUs to avoid lockup (Nick Craig-Wood) 54410 - Fix panic on File.Open (Nick Craig-Wood) 54411 - Fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD 54412 (Nick Craig-Wood) 54413 - Don't pass huge filenames (>4k) to FUSE as it can't cope (Nick 54414 Craig-Wood) 54415 - VFS 54416 - Add flag --vfs-case-insensitive for windows/macOS mounts (Ivan 54417 Andreev) 54418 - Make objects of unknown size readable through the VFS (Nick 54419 Craig-Wood) 54420 - Move writeback of dirty data out of close() method into its own 54421 method (FlushWrites) and remove close() call from Flush() (Brett 54422 Dutro) 54423 - Stop empty dirs disappearing when renamed on bucket-based 54424 remotes (Nick Craig-Wood) 54425 - Stop change notify polling clearing so much of the directory 54426 cache (Nick Craig-Wood) 54427 - Azure Blob 54428 - Disable logging to the Windows event log (Nick Craig-Wood) 54429 - B2 54430 - Remove unverified: prefix on sha1 to improve interop (e.g. with 54431 CyberDuck) (Nick Craig-Wood) 54432 - Box 54433 - Add options to get access token via JWT auth (David) 54434 - Drive 54435 - Disable HTTP/2 by default to work around INTERNAL_ERROR problems 54436 (Nick Craig-Wood) 54437 - Make sure that drive root ID is always canonical (Nick 54438 Craig-Wood) 54439 - Fix --drive-shared-with-me from the root with lsand --fast-list 54440 (Nick Craig-Wood) 54441 - Fix ChangeNotify polling for shared drives (Nick Craig-Wood) 54442 - Fix change notify polling when using appDataFolder (Nick 54443 Craig-Wood) 54444 - Dropbox 54445 - Make disallowed filenames errors not retry (Nick Craig-Wood) 54446 - Fix nil pointer exception on restricted files (Nick Craig-Wood) 54447 - Fichier 54448 - Fix accessing files > 2GB on 32 bit systems (Nick Craig-Wood) 54449 - FTP 54450 - Allow disabling EPSV mode (Jon Fautley) 54451 - HTTP 54452 - HEAD directory entries in parallel to speedup (Nick Craig-Wood) 54453 - Add --http-no-head to stop rclone doing HEAD in listings (Nick 54454 Craig-Wood) 54455 - Putio 54456 - Add ability to resume uploads (Cenk Alti) 54457 - S3 54458 - Fix signature v2_auth headers (Anthony Rusdi) 54459 - Fix encoding for control characters (Nick Craig-Wood) 54460 - Only ask for URL encoded directory listings if we need them on 54461 Ceph (Nick Craig-Wood) 54462 - Add option for multipart failure behaviour (Aleksandar Jankovic) 54463 - Support for multipart copy (庄天翼) 54464 - Fix nil pointer reference if no metadata returned for object 54465 (Nick Craig-Wood) 54466 - SFTP 54467 - Fix --sftp-ask-password trying to contact the ssh agent (Nick 54468 Craig-Wood) 54469 - Fix hashes of files with backslashes (Nick Craig-Wood) 54470 - Include more ciphers with --sftp-use-insecure-cipher (Carlos 54471 Ferreyra) 54472 - WebDAV 54473 - Parse and return Sharepoint error response (Henning Surmeier) 54474 54475 v1.49.5 - 2019-10-05 54476 54477 - Bug Fixes 54478 - Revert back to go1.12.x for the v1.49.x builds as go1.13.x was 54479 causing issues (Nick Craig-Wood) 54480 - Fix rpm packages by using master builds of nfpm (Nick 54481 Craig-Wood) 54482 - Fix macOS build after brew changes (Nick Craig-Wood) 54483 54484 v1.49.4 - 2019-09-29 54485 54486 - Bug Fixes 54487 - cmd/rcd: Address ZipSlip vulnerability (Richard Patel) 54488 - accounting: Fix file handle leak on errors (Nick Craig-Wood) 54489 - oauthutil: Fix security problem when running with two users on 54490 the same machine (Nick Craig-Wood) 54491 - FTP 54492 - Fix listing of an empty root returning: error dir not found 54493 (Nick Craig-Wood) 54494 - S3 54495 - Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get 54496 tier (Nick Craig-Wood) 54497 54498 v1.49.3 - 2019-09-15 54499 54500 - Bug Fixes 54501 - accounting 54502 - Fix total duration calculation (Aleksandar Jankovic) 54503 - Fix "file already closed" on transfer retries (Nick 54504 Craig-Wood) 54505 54506 v1.49.2 - 2019-09-08 54507 54508 - New Features 54509 - build: Add Docker workflow support (Alfonso Montero) 54510 - Bug Fixes 54511 - accounting: Fix locking in Transfer to avoid deadlock with 54512 --progress (Nick Craig-Wood) 54513 - docs: Fix template argument for mktemp in install.sh (Cnly) 54514 - operations: Fix -u/--update with google photos / files of 54515 unknown size (Nick Craig-Wood) 54516 - rc: Fix docs for config/create /update /password (Nick 54517 Craig-Wood) 54518 - Google Cloud Storage 54519 - Fix need for elevated permissions on SetModTime (Nick 54520 Craig-Wood) 54521 54522 v1.49.1 - 2019-08-28 54523 54524 - Bug Fixes 54525 - config: Fix generated passwords being stored as empty password 54526 (Nick Craig-Wood) 54527 - rcd: Added missing parameter for web-gui info logs. (Chaitanya) 54528 - Googlephotos 54529 - Fix crash on error response (Nick Craig-Wood) 54530 - Onedrive 54531 - Fix crash on error response (Nick Craig-Wood) 54532 54533 v1.49.0 - 2019-08-26 54534 54535 - New backends 54536 - 1fichier (Laura Hausmann) 54537 - Google Photos (Nick Craig-Wood) 54538 - Putio (Cenk Alti) 54539 - premiumize.me (Nick Craig-Wood) 54540 - New Features 54541 - Experimental web GUI (Chaitanya Bankanhal) 54542 - Implement --compare-dest & --copy-dest (yparitcher) 54543 - Implement --suffix without --backup-dir for backup to current 54544 dir (yparitcher) 54545 - config reconnect to re-login (re-run the oauth login) for the 54546 backend. (Nick Craig-Wood) 54547 - config userinfo to discover which user you are logged in as. 54548 (Nick Craig-Wood) 54549 - config disconnect to disconnect you (log out) from the backend. 54550 (Nick Craig-Wood) 54551 - Add --use-json-log for JSON logging (justinalin) 54552 - Add context propagation to rclone (Aleksandar Jankovic) 54553 - Reworking internal statistics interfaces so they work with rc 54554 jobs (Aleksandar Jankovic) 54555 - Add Higher units for ETA (AbelThar) 54556 - Update rclone logos to new design (Andreas Chlupka) 54557 - hash: Add CRC-32 support (Cenk Alti) 54558 - help showbackend: Fixed advanced option category when there are 54559 no standard options (buengese) 54560 - ncdu: Display/Copy to Clipboard Current Path (Gary Kim) 54561 - operations: 54562 - Run hashing operations in parallel (Nick Craig-Wood) 54563 - Don't calculate checksums when using --ignore-checksum (Nick 54564 Craig-Wood) 54565 - Check transfer hashes when using --size-only mode (Nick 54566 Craig-Wood) 54567 - Disable multi thread copy for local to local copies (Nick 54568 Craig-Wood) 54569 - Debug successful hashes as well as failures (Nick 54570 Craig-Wood) 54571 - rc 54572 - Add ability to stop async jobs (Aleksandar Jankovic) 54573 - Return current settings if core/bwlimit called without 54574 parameters (Nick Craig-Wood) 54575 - Rclone-WebUI integration with rclone (Chaitanya Bankanhal) 54576 - Added command line parameter to control the cross origin 54577 resource sharing (CORS) in the rcd. (Security Improvement) 54578 (Chaitanya Bankanhal) 54579 - Add anchor tags to the docs so links are consistent (Nick 54580 Craig-Wood) 54581 - Remove _async key from input parameters after parsing so 54582 later operations won't get confused (buengese) 54583 - Add call to clear stats (Aleksandar Jankovic) 54584 - rcd 54585 - Auto-login for web-gui (Chaitanya Bankanhal) 54586 - Implement --baseurl for rcd and web-gui (Chaitanya 54587 Bankanhal) 54588 - serve dlna 54589 - Only select interfaces which can multicast for SSDP (Nick 54590 Craig-Wood) 54591 - Add more builtin mime types to cover standard audio/video 54592 (Nick Craig-Wood) 54593 - Fix missing mime types on Android causing missing videos 54594 (Nick Craig-Wood) 54595 - serve ftp 54596 - Refactor to bring into line with other serve commands (Nick 54597 Craig-Wood) 54598 - Implement --auth-proxy (Nick Craig-Wood) 54599 - serve http: Implement --baseurl (Nick Craig-Wood) 54600 - serve restic: Implement --baseurl (Nick Craig-Wood) 54601 - serve sftp 54602 - Implement auth proxy (Nick Craig-Wood) 54603 - Fix detection of whether server is authorized (Nick 54604 Craig-Wood) 54605 - serve webdav 54606 - Implement --baseurl (Nick Craig-Wood) 54607 - Support --auth-proxy (Nick Craig-Wood) 54608 - Bug Fixes 54609 - Make "bad record MAC" a retriable error (Nick Craig-Wood) 54610 - copyurl: Fix copying files that return HTTP errors (Nick 54611 Craig-Wood) 54612 - march: Fix checking sub-directories when using --no-traverse 54613 (buengese) 54614 - rc 54615 - Fix unmarshalable http.AuthFn in options and put in test for 54616 marshalability (Nick Craig-Wood) 54617 - Move job expire flags to rc to fix initialization problem 54618 (Nick Craig-Wood) 54619 - Fix --loopback with rc/list and others (Nick Craig-Wood) 54620 - rcat: Fix slowdown on systems with multiple hashes (Nick 54621 Craig-Wood) 54622 - rcd: Fix permissions problems on cache directory with web gui 54623 download (Nick Craig-Wood) 54624 - Mount 54625 - Default --daemon-timeout to 15 minutes on macOS and FreeBSD 54626 (Nick Craig-Wood) 54627 - Update docs to show mounting from root OK for bucket-based (Nick 54628 Craig-Wood) 54629 - Remove nonseekable flag from write files (Nick Craig-Wood) 54630 - VFS 54631 - Make write without cache more efficient (Nick Craig-Wood) 54632 - Fix --vfs-cache-mode minimal and writes ignoring cached files 54633 (Nick Craig-Wood) 54634 - Local 54635 - Add --local-case-sensitive and --local-case-insensitive (Nick 54636 Craig-Wood) 54637 - Avoid polluting page cache when uploading local files to remote 54638 backends (Michał Matczuk) 54639 - Don't calculate any hashes by default (Nick Craig-Wood) 54640 - Fadvise run syscall on a dedicated go routine (Michał Matczuk) 54641 - Azure Blob 54642 - Azure Storage Emulator support (Sandeep) 54643 - Updated config help details to remove connection string 54644 references (Sandeep) 54645 - Make all operations work from the root (Nick Craig-Wood) 54646 - B2 54647 - Implement link sharing (yparitcher) 54648 - Enable server-side copy to copy between buckets (Nick 54649 Craig-Wood) 54650 - Make all operations work from the root (Nick Craig-Wood) 54651 - Drive 54652 - Fix server-side copy of big files (Nick Craig-Wood) 54653 - Update API for teamdrive use (Nick Craig-Wood) 54654 - Add error for purge with --drive-trashed-only (ginvine) 54655 - Fichier 54656 - Make FolderID int and adjust related code (buengese) 54657 - Google Cloud Storage 54658 - Reduce oauth scope requested as suggested by Google (Nick 54659 Craig-Wood) 54660 - Make all operations work from the root (Nick Craig-Wood) 54661 - HTTP 54662 - Add --http-headers flag for setting arbitrary headers (Nick 54663 Craig-Wood) 54664 - Jottacloud 54665 - Use new api for retrieving internal username (buengese) 54666 - Refactor configuration and minor cleanup (buengese) 54667 - Koofr 54668 - Support setting modification times on Koofr backend. (jaKa) 54669 - Opendrive 54670 - Refactor to use existing lib/rest facilities for uploads (Nick 54671 Craig-Wood) 54672 - Qingstor 54673 - Upgrade to v3 SDK and fix listing loop (Nick Craig-Wood) 54674 - Make all operations work from the root (Nick Craig-Wood) 54675 - S3 54676 - Add INTELLIGENT_TIERING storage class (Matti Niemenmaa) 54677 - Make all operations work from the root (Nick Craig-Wood) 54678 - SFTP 54679 - Add missing interface check and fix About (Nick Craig-Wood) 54680 - Completely ignore all modtime checks if SetModTime=false (Jon 54681 Fautley) 54682 - Support md5/sha1 with rsync.net (Nick Craig-Wood) 54683 - Save the md5/sha1 command in use to the config file for 54684 efficiency (Nick Craig-Wood) 54685 - Opt-in support for diffie-hellman-group-exchange-sha256 54686 diffie-hellman-group-exchange-sha1 (Yi FU) 54687 - Swift 54688 - Use FixRangeOption to fix 0 length files via the VFS (Nick 54689 Craig-Wood) 54690 - Fix upload when using no_chunk to return the correct size (Nick 54691 Craig-Wood) 54692 - Make all operations work from the root (Nick Craig-Wood) 54693 - Fix segments leak during failed large file uploads. 54694 (nguyenhuuluan434) 54695 - WebDAV 54696 - Add --webdav-bearer-token-command (Nick Craig-Wood) 54697 - Refresh token when it expires with --webdav-bearer-token-command 54698 (Nick Craig-Wood) 54699 - Add docs for using bearer_token_command with oidc-agent (Paul 54700 Millar) 54701 54702 v1.48.0 - 2019-06-15 54703 54704 - New commands 54705 - serve sftp: Serve an rclone remote over SFTP (Nick Craig-Wood) 54706 - New Features 54707 - Multi threaded downloads to local storage (Nick Craig-Wood) 54708 - controlled with --multi-thread-cutoff and 54709 --multi-thread-streams 54710 - Use rclone.conf from rclone executable directory to enable 54711 portable use (albertony) 54712 - Allow sync of a file and a directory with the same name 54713 (forgems) 54714 - this is common on bucket-based remotes, e.g. s3, gcs 54715 - Add --ignore-case-sync for forced case insensitivity (garry415) 54716 - Implement --stats-one-line-date and --stats-one-line-date-format 54717 (Peter Berbec) 54718 - Log an ERROR for all commands which exit with non-zero status 54719 (Nick Craig-Wood) 54720 - Use go-homedir to read the home directory more reliably (Nick 54721 Craig-Wood) 54722 - Enable creating encrypted config through external script 54723 invocation (Wojciech Smigielski) 54724 - build: Drop support for go1.8 (Nick Craig-Wood) 54725 - config: Make config create/update encrypt passwords where 54726 necessary (Nick Craig-Wood) 54727 - copyurl: Honor --no-check-certificate (Stefan Breunig) 54728 - install: Linux skip man pages if no mandb (didil) 54729 - lsf: Support showing the Tier of the object (Nick Craig-Wood) 54730 - lsjson 54731 - Added EncryptedPath to output (calisro) 54732 - Support showing the Tier of the object (Nick Craig-Wood) 54733 - Add IsBucket field for bucket-based remote listing of the 54734 root (Nick Craig-Wood) 54735 - rc 54736 - Add --loopback flag to run commands directly without a 54737 server (Nick Craig-Wood) 54738 - Add operations/fsinfo: Return information about the remote 54739 (Nick Craig-Wood) 54740 - Skip auth for OPTIONS request (Nick Craig-Wood) 54741 - cmd/providers: Add DefaultStr, ValueStr and Type fields 54742 (Nick Craig-Wood) 54743 - jobs: Make job expiry timeouts configurable (Aleksandar 54744 Jankovic) 54745 - serve dlna reworked and improved (Dan Walters) 54746 - serve ftp: add --ftp-public-ip flag to specify public IP 54747 (calistri) 54748 - serve restic: Add support for --private-repos in serve restic 54749 (Florian Apolloner) 54750 - serve webdav: Combine serve webdav and serve http (Gary Kim) 54751 - size: Ignore negative sizes when calculating total (Garry 54752 McNulty) 54753 - Bug Fixes 54754 - Make move and copy individual files obey --backup-dir (Nick 54755 Craig-Wood) 54756 - If --ignore-checksum is in effect, don't calculate checksum 54757 (Nick Craig-Wood) 54758 - moveto: Fix case-insensitive same remote move (Gary Kim) 54759 - rc: Fix serving bucket-based objects with --rc-serve (Nick 54760 Craig-Wood) 54761 - serve webdav: Fix serveDir not being updated with changes from 54762 webdav (Gary Kim) 54763 - Mount 54764 - Fix poll interval documentation (Animosity022) 54765 - VFS 54766 - Make WriteAt for non cached files work with non-sequential 54767 writes (Nick Craig-Wood) 54768 - Local 54769 - Only calculate the required hashes for big speedup (Nick 54770 Craig-Wood) 54771 - Log errors when listing instead of returning an error (Nick 54772 Craig-Wood) 54773 - Fix preallocate warning on Linux with ZFS (Nick Craig-Wood) 54774 - Crypt 54775 - Make rclone dedupe work through crypt (Nick Craig-Wood) 54776 - Fix wrapping of ChangeNotify to decrypt directories properly 54777 (Nick Craig-Wood) 54778 - Support PublicLink (rclone link) of underlying backend (Nick 54779 Craig-Wood) 54780 - Implement Optional methods SetTier, GetTier (Nick Craig-Wood) 54781 - B2 54782 - Implement server-side copy (Nick Craig-Wood) 54783 - Implement SetModTime (Nick Craig-Wood) 54784 - Drive 54785 - Fix move and copy from TeamDrive to GDrive (Fionera) 54786 - Add notes that cleanup works in the background on drive (Nick 54787 Craig-Wood) 54788 - Add --drive-server-side-across-configs to default back to old 54789 server-side copy semantics by default (Nick Craig-Wood) 54790 - Add --drive-size-as-quota to show storage quota usage for file 54791 size (Garry McNulty) 54792 - FTP 54793 - Add FTP List timeout (Jeff Quinn) 54794 - Add FTP over TLS support (Gary Kim) 54795 - Add --ftp-no-check-certificate option for FTPS (Gary Kim) 54796 - Google Cloud Storage 54797 - Fix upload errors when uploading pre 1970 files (Nick 54798 Craig-Wood) 54799 - Jottacloud 54800 - Add support for selecting device and mountpoint. (buengese) 54801 - Mega 54802 - Add cleanup support (Gary Kim) 54803 - Onedrive 54804 - More accurately check if root is found (Cnly) 54805 - S3 54806 - Support S3 Accelerated endpoints with 54807 --s3-use-accelerate-endpoint (Nick Craig-Wood) 54808 - Add config info for Wasabi's EU Central endpoint (Robert Marko) 54809 - Make SetModTime work for GLACIER while syncing (Philip Harvey) 54810 - SFTP 54811 - Add About support (Gary Kim) 54812 - Fix about parsing of df results so it can cope with -ve results 54813 (Nick Craig-Wood) 54814 - Send custom client version and debug server version (Nick 54815 Craig-Wood) 54816 - WebDAV 54817 - Retry on 423 Locked errors (Nick Craig-Wood) 54818 54819 v1.47.0 - 2019-04-13 54820 54821 - New backends 54822 - Backend for Koofr cloud storage service. (jaKa) 54823 - New Features 54824 - Resume downloads if the reader fails in copy (Nick Craig-Wood) 54825 - this means rclone will restart transfers if the source has 54826 an error 54827 - this is most useful for downloads or cloud to cloud copies 54828 - Use --fast-list for listing operations where it won't use more 54829 memory (Nick Craig-Wood) 54830 - this should speed up the following operations on remotes 54831 which support ListR 54832 - dedupe, serve restic lsf, ls, lsl, lsjson, lsd, md5sum, 54833 sha1sum, hashsum, size, delete, cat, settier 54834 - use --disable ListR to get old behaviour if required 54835 - Make --files-from traverse the destination unless --no-traverse 54836 is set (Nick Craig-Wood) 54837 - this fixes --files-from with Google drive and excessive API 54838 use in general. 54839 - Make server-side copy account bytes and obey --max-transfer 54840 (Nick Craig-Wood) 54841 - Add --create-empty-src-dirs flag and default to not creating 54842 empty dirs (ishuah) 54843 - Add client side TLS/SSL flags 54844 --ca-cert/--client-cert/--client-key (Nick Craig-Wood) 54845 - Implement --suffix-keep-extension for use with --suffix (Nick 54846 Craig-Wood) 54847 - build: 54848 - Switch to semver compliant version tags to be go modules 54849 compliant (Nick Craig-Wood) 54850 - Update to use go1.12.x for the build (Nick Craig-Wood) 54851 - serve dlna: Add connection manager service description to 54852 improve compatibility (Dan Walters) 54853 - lsf: Add 'e' format to show encrypted names and 'o' for original 54854 IDs (Nick Craig-Wood) 54855 - lsjson: Added --files-only and --dirs-only flags (calistri) 54856 - rc: Implement operations/publiclink the equivalent of 54857 rclone link (Nick Craig-Wood) 54858 - Bug Fixes 54859 - accounting: Fix total ETA when --stats-unit bits is in effect 54860 (Nick Craig-Wood) 54861 - Bash TAB completion 54862 - Use private custom func to fix clash between rclone and 54863 kubectl (Nick Craig-Wood) 54864 - Fix for remotes with underscores in their names (Six) 54865 - Fix completion of remotes (Florian Gamböck) 54866 - Fix autocompletion of remote paths with spaces (Danil 54867 Semelenov) 54868 - serve dlna: Fix root XML service descriptor (Dan Walters) 54869 - ncdu: Fix display corruption with Chinese characters (Nick 54870 Craig-Wood) 54871 - Add SIGTERM to signals which run the exit handlers on unix (Nick 54872 Craig-Wood) 54873 - rc: Reload filter when the options are set via the rc (Nick 54874 Craig-Wood) 54875 - VFS / Mount 54876 - Fix FreeBSD: Ignore Truncate if called with no readers and 54877 already the correct size (Nick Craig-Wood) 54878 - Read directory and check for a file before mkdir (Nick 54879 Craig-Wood) 54880 - Shorten the locking window for vfs/refresh (Nick Craig-Wood) 54881 - Azure Blob 54882 - Enable MD5 checksums when uploading files bigger than the 54883 "Cutoff" (Dr.Rx) 54884 - Fix SAS URL support (Nick Craig-Wood) 54885 - B2 54886 - Allow manual configuration of backblaze downloadUrl (Vince) 54887 - Ignore already_hidden error on remove (Nick Craig-Wood) 54888 - Ignore malformed src_last_modified_millis (Nick Craig-Wood) 54889 - Drive 54890 - Add --skip-checksum-gphotos to ignore incorrect checksums on 54891 Google Photos (Nick Craig-Wood) 54892 - Allow server-side move/copy between different remotes. (Fionera) 54893 - Add docs on team drives and --fast-list eventual consistency 54894 (Nestar47) 54895 - Fix imports of text files (Nick Craig-Wood) 54896 - Fix range requests on 0 length files (Nick Craig-Wood) 54897 - Fix creation of duplicates with server-side copy (Nick 54898 Craig-Wood) 54899 - Dropbox 54900 - Retry blank errors to fix long listings (Nick Craig-Wood) 54901 - FTP 54902 - Add --ftp-concurrency to limit maximum number of connections 54903 (Nick Craig-Wood) 54904 - Google Cloud Storage 54905 - Fall back to default application credentials (marcintustin) 54906 - Allow bucket policy only buckets (Nick Craig-Wood) 54907 - HTTP 54908 - Add --http-no-slash for websites with directories with no 54909 slashes (Nick Craig-Wood) 54910 - Remove duplicates from listings (Nick Craig-Wood) 54911 - Fix socket leak on 404 errors (Nick Craig-Wood) 54912 - Jottacloud 54913 - Fix token refresh (Sebastian Bünger) 54914 - Add device registration (Oliver Heyme) 54915 - Onedrive 54916 - Implement graceful cancel of multipart uploads if rclone is 54917 interrupted (Cnly) 54918 - Always add trailing colon to path when addressing items, (Cnly) 54919 - Return errors instead of panic for invalid uploads (Fabian 54920 Möller) 54921 - S3 54922 - Add support for "Glacier Deep Archive" storage class (Manu) 54923 - Update Dreamhost endpoint (Nick Craig-Wood) 54924 - Note incompatibility with CEPH Jewel (Nick Craig-Wood) 54925 - SFTP 54926 - Allow custom ssh client config (Alexandru Bumbacea) 54927 - Swift 54928 - Obey Retry-After to enable OVH restore from cold storage (Nick 54929 Craig-Wood) 54930 - Work around token expiry on CEPH (Nick Craig-Wood) 54931 - WebDAV 54932 - Allow IsCollection property to be integer or boolean (Nick 54933 Craig-Wood) 54934 - Fix race when creating directories (Nick Craig-Wood) 54935 - Fix About/df when reading the available/total returns 0 (Nick 54936 Craig-Wood) 54937 54938 v1.46 - 2019-02-09 54939 54940 - New backends 54941 - Support Alibaba Cloud (Aliyun) OSS via the s3 backend (Nick 54942 Craig-Wood) 54943 - New commands 54944 - serve dlna: serves a remove via DLNA for the local network 54945 (nicolov) 54946 - New Features 54947 - copy, move: Restore deprecated --no-traverse flag (Nick 54948 Craig-Wood) 54949 - This is useful for when transferring a small number of files 54950 into a large destination 54951 - genautocomplete: Add remote path completion for bash completion 54952 (Christopher Peterson & Danil Semelenov) 54953 - Buffer memory handling reworked to return memory to the OS 54954 better (Nick Craig-Wood) 54955 - Buffer recycling library to replace sync.Pool 54956 - Optionally use memory mapped memory for better memory 54957 shrinking 54958 - Enable with --use-mmap if having memory problems - not 54959 default yet 54960 - Parallelise reading of files specified by --files-from (Nick 54961 Craig-Wood) 54962 - check: Add stats showing total files matched. (Dario Guzik) 54963 - Allow rename/delete open files under Windows (Nick Craig-Wood) 54964 - lsjson: Use exactly the correct number of decimal places in the 54965 seconds (Nick Craig-Wood) 54966 - Add cookie support with cmdline switch --use-cookies for all 54967 HTTP based remotes (qip) 54968 - Warn if --checksum is set but there are no hashes available 54969 (Nick Craig-Wood) 54970 - Rework rate limiting (pacer) to be more accurate and allow 54971 bursting (Nick Craig-Wood) 54972 - Improve error reporting for too many/few arguments in commands 54973 (Nick Craig-Wood) 54974 - listremotes: Remove -l short flag as it conflicts with the new 54975 global flag (weetmuts) 54976 - Make http serving with auth generate INFO messages on auth fail 54977 (Nick Craig-Wood) 54978 - Bug Fixes 54979 - Fix layout of stats (Nick Craig-Wood) 54980 - Fix --progress crash under Windows Jenkins (Nick Craig-Wood) 54981 - Fix transfer of google/onedrive docs by calling Rcat in Copy 54982 when size is -1 (Cnly) 54983 - copyurl: Fix checking of --dry-run (Denis Skovpen) 54984 - Mount 54985 - Check that mountpoint and local directory to mount don't overlap 54986 (Nick Craig-Wood) 54987 - Fix mount size under 32 bit Windows (Nick Craig-Wood) 54988 - VFS 54989 - Implement renaming of directories for backends without DirMove 54990 (Nick Craig-Wood) 54991 - now all backends except b2 support renaming directories 54992 - Implement --vfs-cache-max-size to limit the total size of the 54993 cache (Nick Craig-Wood) 54994 - Add --dir-perms and --file-perms flags to set default 54995 permissions (Nick Craig-Wood) 54996 - Fix deadlock on concurrent operations on a directory (Nick 54997 Craig-Wood) 54998 - Fix deadlock between RWFileHandle.close and File.Remove (Nick 54999 Craig-Wood) 55000 - Fix renaming/deleting open files with cache mode "writes" under 55001 Windows (Nick Craig-Wood) 55002 - Fix panic on rename with --dry-run set (Nick Craig-Wood) 55003 - Fix vfs/refresh with recurse=true needing the --fast-list flag 55004 - Local 55005 - Add support for -l/--links (symbolic link translation) 55006 (yair@unicorn) 55007 - this works by showing links as link.rclonelink - see local 55008 backend docs for more info 55009 - this errors if used with -L/--copy-links 55010 - Fix renaming/deleting open files on Windows (Nick Craig-Wood) 55011 - Crypt 55012 - Check for maximum length before decrypting filename to fix panic 55013 (Garry McNulty) 55014 - Azure Blob 55015 - Allow building azureblob backend on *BSD (themylogin) 55016 - Use the rclone HTTP client to support --dump headers, 55017 --tpslimit, etc. (Nick Craig-Wood) 55018 - Use the s3 pacer for 0 delay in non error conditions (Nick 55019 Craig-Wood) 55020 - Ignore directory markers (Nick Craig-Wood) 55021 - Stop Mkdir attempting to create existing containers (Nick 55022 Craig-Wood) 55023 - B2 55024 - cleanup: will remove unfinished large files >24hrs old (Garry 55025 McNulty) 55026 - For a bucket limited application key check the bucket name (Nick 55027 Craig-Wood) 55028 - before this, rclone would use the authorised bucket 55029 regardless of what you put on the command line 55030 - Added --b2-disable-checksum flag (Wojciech Smigielski) 55031 - this enables large files to be uploaded without a SHA-1 hash 55032 for speed reasons 55033 - Drive 55034 - Set default pacer to 100ms for 10 tps (Nick Craig-Wood) 55035 - This fits the Google defaults much better and reduces the 55036 403 errors massively 55037 - Add --drive-pacer-min-sleep and --drive-pacer-burst to 55038 control the pacer 55039 - Improve ChangeNotify support for items with multiple parents 55040 (Fabian Möller) 55041 - Fix ListR for items with multiple parents - this fixes oddities 55042 with vfs/refresh (Fabian Möller) 55043 - Fix using --drive-impersonate and appfolders (Nick Craig-Wood) 55044 - Fix google docs in rclone mount for some (not all) applications 55045 (Nick Craig-Wood) 55046 - Dropbox 55047 - Retry-After support for Dropbox backend (Mathieu Carbou) 55048 - FTP 55049 - Wait for 60 seconds for a connection to Close then declare it 55050 dead (Nick Craig-Wood) 55051 - helps with indefinite hangs on some FTP servers 55052 - Google Cloud Storage 55053 - Update google cloud storage endpoints (weetmuts) 55054 - HTTP 55055 - Add an example with username and password which is supported but 55056 wasn't documented (Nick Craig-Wood) 55057 - Fix backend with --files-from and nonexistent files (Nick 55058 Craig-Wood) 55059 - Hubic 55060 - Make error message more informative if authentication fails 55061 (Nick Craig-Wood) 55062 - Jottacloud 55063 - Resume and deduplication support (Oliver Heyme) 55064 - Use token auth for all API requests Don't store password anymore 55065 (Sebastian Bünger) 55066 - Add support for 2-factor authentication (Sebastian Bünger) 55067 - Mega 55068 - Implement v2 account login which fixes logins for newer Mega 55069 accounts (Nick Craig-Wood) 55070 - Return error if an unknown length file is attempted to be 55071 uploaded (Nick Craig-Wood) 55072 - Add new error codes for better error reporting (Nick Craig-Wood) 55073 - Onedrive 55074 - Fix broken support for "shared with me" folders (Alex Chen) 55075 - Fix root ID not normalised (Cnly) 55076 - Return err instead of panic on unknown-sized uploads (Cnly) 55077 - Qingstor 55078 - Fix go routine leak on multipart upload errors (Nick Craig-Wood) 55079 - Add upload chunk size/concurrency/cutoff control (Nick 55080 Craig-Wood) 55081 - Default --qingstor-upload-concurrency to 1 to work around bug 55082 (Nick Craig-Wood) 55083 - S3 55084 - Implement --s3-upload-cutoff for single part uploads below this 55085 (Nick Craig-Wood) 55086 - Change --s3-upload-concurrency default to 4 to increase 55087 performance (Nick Craig-Wood) 55088 - Add --s3-bucket-acl to control bucket ACL (Nick Craig-Wood) 55089 - Auto detect region for buckets on operation failure (Nick 55090 Craig-Wood) 55091 - Add GLACIER storage class (William Cocker) 55092 - Add Scaleway to s3 documentation (Rémy Léone) 55093 - Add AWS endpoint eu-north-1 (weetmuts) 55094 - SFTP 55095 - Add support for PEM encrypted private keys (Fabian Möller) 55096 - Add option to force the usage of an ssh-agent (Fabian Möller) 55097 - Perform environment variable expansion on key-file (Fabian 55098 Möller) 55099 - Fix rmdir on Windows based servers (e.g. CrushFTP) (Nick 55100 Craig-Wood) 55101 - Fix rmdir deleting directory contents on some SFTP servers (Nick 55102 Craig-Wood) 55103 - Fix error on dangling symlinks (Nick Craig-Wood) 55104 - Swift 55105 - Add --swift-no-chunk to disable segmented uploads in rcat/mount 55106 (Nick Craig-Wood) 55107 - Introduce application credential auth support (kayrus) 55108 - Fix memory usage by slimming Object (Nick Craig-Wood) 55109 - Fix extra requests on upload (Nick Craig-Wood) 55110 - Fix reauth on big files (Nick Craig-Wood) 55111 - Union 55112 - Fix poll-interval not working (Nick Craig-Wood) 55113 - WebDAV 55114 - Support About which means rclone mount will show the correct 55115 disk size (Nick Craig-Wood) 55116 - Support MD5 and SHA1 hashes with Owncloud and Nextcloud (Nick 55117 Craig-Wood) 55118 - Fail soft on time parsing errors (Nick Craig-Wood) 55119 - Fix infinite loop on failed directory creation (Nick Craig-Wood) 55120 - Fix identification of directories for Bitrix Site Manager (Nick 55121 Craig-Wood) 55122 - Fix upload of 0 length files on some servers (Nick Craig-Wood) 55123 - Fix if MKCOL fails with 423 Locked assume the directory exists 55124 (Nick Craig-Wood) 55125 55126 v1.45 - 2018-11-24 55127 55128 - New backends 55129 - The Yandex backend was re-written - see below for details 55130 (Sebastian Bünger) 55131 - New commands 55132 - rcd: New command just to serve the remote control API (Nick 55133 Craig-Wood) 55134 - New Features 55135 - The remote control API (rc) was greatly expanded to allow full 55136 control over rclone (Nick Craig-Wood) 55137 - sensitive operations require authorization or the 55138 --rc-no-auth flag 55139 - config/* operations to configure rclone 55140 - options/* for reading/setting command line flags 55141 - operations/* for all low level operations, e.g. copy file, 55142 list directory 55143 - sync/* for sync, copy and move 55144 - --rc-files flag to serve files on the rc http server 55145 - this is for building web native GUIs for rclone 55146 - Optionally serving objects on the rc http server 55147 - Ensure rclone fails to start up if the --rc port is in use 55148 already 55149 - See the rc docs for more info 55150 - sync/copy/move 55151 - Make --files-from only read the objects specified and don't 55152 scan directories (Nick Craig-Wood) 55153 - This is a huge speed improvement for destinations with 55154 lots of files 55155 - filter: Add --ignore-case flag (Nick Craig-Wood) 55156 - ncdu: Add remove function ('d' key) (Henning Surmeier) 55157 - rc command 55158 - Add --json flag for structured JSON input (Nick Craig-Wood) 55159 - Add --user and --pass flags and interpret --rc-user, 55160 --rc-pass, --rc-addr (Nick Craig-Wood) 55161 - build 55162 - Require go1.8 or later for compilation (Nick Craig-Wood) 55163 - Enable softfloat on MIPS arch (Scott Edlund) 55164 - Integration test framework revamped with a better report and 55165 better retries (Nick Craig-Wood) 55166 - Bug Fixes 55167 - cmd: Make --progress update the stats correctly at the end (Nick 55168 Craig-Wood) 55169 - config: Create config directory on save if it is missing (Nick 55170 Craig-Wood) 55171 - dedupe: Check for existing filename before renaming a dupe file 55172 (ssaqua) 55173 - move: Don't create directories with --dry-run (Nick Craig-Wood) 55174 - operations: Fix Purge and Rmdirs when dir is not the root (Nick 55175 Craig-Wood) 55176 - serve http/webdav/restic: Ensure rclone exits if the port is in 55177 use (Nick Craig-Wood) 55178 - Mount 55179 - Make --volname work for Windows and macOS (Nick Craig-Wood) 55180 - Azure Blob 55181 - Avoid context deadline exceeded error by setting a large 55182 TryTimeout value (brused27) 55183 - Fix erroneous Rmdir error "directory not empty" (Nick 55184 Craig-Wood) 55185 - Wait for up to 60s to create a just deleted container (Nick 55186 Craig-Wood) 55187 - Dropbox 55188 - Add dropbox impersonate support (Jake Coggiano) 55189 - Jottacloud 55190 - Fix bug in --fast-list handing of empty folders (albertony) 55191 - Opendrive 55192 - Fix transfer of files with + and & in (Nick Craig-Wood) 55193 - Fix retries of upload chunks (Nick Craig-Wood) 55194 - S3 55195 - Set ACL for server-side copies to that provided by the user 55196 (Nick Craig-Wood) 55197 - Fix role_arn, credential_source, ... (Erik Swanson) 55198 - Add config info for Wasabi's US-West endpoint (Henry Ptasinski) 55199 - SFTP 55200 - Ensure file hash checking is really disabled (Jon Fautley) 55201 - Swift 55202 - Add pacer for retries to make swift more reliable (Nick 55203 Craig-Wood) 55204 - WebDAV 55205 - Add Content-Type to PUT requests (Nick Craig-Wood) 55206 - Fix config parsing so --webdav-user and --webdav-pass flags work 55207 (Nick Craig-Wood) 55208 - Add RFC3339 date format (Ralf Hemberger) 55209 - Yandex 55210 - The yandex backend was re-written (Sebastian Bünger) 55211 - This implements low level retries (Sebastian Bünger) 55212 - Copy, Move, DirMove, PublicLink and About optional 55213 interfaces (Sebastian Bünger) 55214 - Improved general error handling (Sebastian Bünger) 55215 - Removed ListR for now due to inconsistent behaviour 55216 (Sebastian Bünger) 55217 55218 v1.44 - 2018-10-15 55219 55220 - New commands 55221 - serve ftp: Add ftp server (Antoine GIRARD) 55222 - settier: perform storage tier changes on supported remotes 55223 (sandeepkru) 55224 - New Features 55225 - Reworked command line help 55226 - Make default help less verbose (Nick Craig-Wood) 55227 - Split flags up into global and backend flags (Nick 55228 Craig-Wood) 55229 - Implement specialised help for flags and backends (Nick 55230 Craig-Wood) 55231 - Show URL of backend help page when starting config (Nick 55232 Craig-Wood) 55233 - stats: Long names now split in center (Joanna Marek) 55234 - Add --log-format flag for more control over log output (dcpu) 55235 - rc: Add support for OPTIONS and basic CORS (frenos) 55236 - stats: show FatalErrors and NoRetryErrors in stats (Cédric 55237 Connes) 55238 - Bug Fixes 55239 - Fix -P not ending with a new line (Nick Craig-Wood) 55240 - config: don't create default config dir when user supplies 55241 --config (albertony) 55242 - Don't print non-ASCII characters with --progress on windows 55243 (Nick Craig-Wood) 55244 - Correct logs for excluded items (ssaqua) 55245 - Mount 55246 - Remove EXPERIMENTAL tags (Nick Craig-Wood) 55247 - VFS 55248 - Fix race condition detected by serve ftp tests (Nick Craig-Wood) 55249 - Add vfs/poll-interval rc command (Fabian Möller) 55250 - Enable rename for nearly all remotes using server-side Move or 55251 Copy (Nick Craig-Wood) 55252 - Reduce directory cache cleared by poll-interval (Fabian Möller) 55253 - Remove EXPERIMENTAL tags (Nick Craig-Wood) 55254 - Local 55255 - Skip bad symlinks in dir listing with -L enabled (Cédric Connes) 55256 - Preallocate files on Windows to reduce fragmentation (Nick 55257 Craig-Wood) 55258 - Preallocate files on linux with fallocate(2) (Nick Craig-Wood) 55259 - Cache 55260 - Add cache/fetch rc function (Fabian Möller) 55261 - Fix worker scale down (Fabian Möller) 55262 - Improve performance by not sending info requests for cached 55263 chunks (dcpu) 55264 - Fix error return value of cache/fetch rc method (Fabian Möller) 55265 - Documentation fix for cache-chunk-total-size (Anagh Kumar 55266 Baranwal) 55267 - Preserve leading / in wrapped remote path (Fabian Möller) 55268 - Add plex_insecure option to skip certificate validation (Fabian 55269 Möller) 55270 - Remove entries that no longer exist in the source (dcpu) 55271 - Crypt 55272 - Preserve leading / in wrapped remote path (Fabian Möller) 55273 - Alias 55274 - Fix handling of Windows network paths (Nick Craig-Wood) 55275 - Azure Blob 55276 - Add --azureblob-list-chunk parameter (Santiago Rodríguez) 55277 - Implemented settier command support on azureblob remote. 55278 (sandeepkru) 55279 - Work around SDK bug which causes errors for chunk-sized files 55280 (Nick Craig-Wood) 55281 - Box 55282 - Implement link sharing. (Sebastian Bünger) 55283 - Drive 55284 - Add --drive-import-formats - google docs can now be imported 55285 (Fabian Möller) 55286 - Rewrite mime type and extension handling (Fabian Möller) 55287 - Add document links (Fabian Möller) 55288 - Add support for multipart document extensions (Fabian 55289 Möller) 55290 - Add support for apps-script to json export (Fabian Möller) 55291 - Fix escaped chars in documents during list (Fabian Möller) 55292 - Add --drive-v2-download-min-size a workaround for slow downloads 55293 (Fabian Möller) 55294 - Improve directory notifications in ChangeNotify (Fabian Möller) 55295 - When listing team drives in config, continue on failure (Nick 55296 Craig-Wood) 55297 - FTP 55298 - Add a small pause after failed upload before deleting file (Nick 55299 Craig-Wood) 55300 - Google Cloud Storage 55301 - Fix service_account_file being ignored (Fabian Möller) 55302 - Jottacloud 55303 - Minor improvement in quota info (omit if unlimited) (albertony) 55304 - Add --fast-list support (albertony) 55305 - Add permanent delete support: --jottacloud-hard-delete 55306 (albertony) 55307 - Add link sharing support (albertony) 55308 - Fix handling of reserved characters. (Sebastian Bünger) 55309 - Fix socket leak on Object.Remove (Nick Craig-Wood) 55310 - Onedrive 55311 - Rework to support Microsoft Graph (Cnly) 55312 - NB this will require re-authenticating the remote 55313 - Removed upload cutoff and always do session uploads (Oliver 55314 Heyme) 55315 - Use single-part upload for empty files (Cnly) 55316 - Fix new fields not saved when editing old config (Alex Chen) 55317 - Fix sometimes special chars in filenames not replaced (Alex 55318 Chen) 55319 - Ignore OneNote files by default (Alex Chen) 55320 - Add link sharing support (jackyzy823) 55321 - S3 55322 - Use custom pacer, to retry operations when reasonable (Craig 55323 Miskell) 55324 - Use configured server-side-encryption and storage class options 55325 when calling CopyObject() (Paul Kohout) 55326 - Make --s3-v2-auth flag (Nick Craig-Wood) 55327 - Fix v2 auth on files with spaces (Nick Craig-Wood) 55328 - Union 55329 - Implement union backend which reads from multiple backends 55330 (Felix Brucker) 55331 - Implement optional interfaces (Move, DirMove, Copy, etc.) (Nick 55332 Craig-Wood) 55333 - Fix ChangeNotify to support multiple remotes (Fabian Möller) 55334 - Fix --backup-dir on union backend (Nick Craig-Wood) 55335 - WebDAV 55336 - Add another time format (Nick Craig-Wood) 55337 - Add a small pause after failed upload before deleting file (Nick 55338 Craig-Wood) 55339 - Add workaround for missing mtime (buergi) 55340 - Sharepoint: Renew cookies after 12hrs (Henning Surmeier) 55341 - Yandex 55342 - Remove redundant nil checks (teresy) 55343 55344 v1.43.1 - 2018-09-07 55345 55346 Point release to fix hubic and azureblob backends. 55347 55348 - Bug Fixes 55349 - ncdu: Return error instead of log.Fatal in Show (Fabian Möller) 55350 - cmd: Fix crash with --progress and --stats 0 (Nick Craig-Wood) 55351 - docs: Tidy website display (Anagh Kumar Baranwal) 55352 - Azure Blob: 55353 - Fix multi-part uploads. (sandeepkru) 55354 - Hubic 55355 - Fix uploads (Nick Craig-Wood) 55356 - Retry auth fetching if it fails to make hubic more reliable 55357 (Nick Craig-Wood) 55358 55359 v1.43 - 2018-09-01 55360 55361 - New backends 55362 - Jottacloud (Sebastian Bünger) 55363 - New commands 55364 - copyurl: copies a URL to a remote (Denis) 55365 - New Features 55366 - Reworked config for backends (Nick Craig-Wood) 55367 - All backend config can now be supplied by command line, env 55368 var or config file 55369 - Advanced section in the config wizard for the optional items 55370 - A large step towards rclone backends being usable in other 55371 go software 55372 - Allow on the fly remotes with :backend: syntax 55373 - Stats revamp 55374 - Add --progress/-P flag to show interactive progress (Nick 55375 Craig-Wood) 55376 - Show the total progress of the sync in the stats (Nick 55377 Craig-Wood) 55378 - Add --stats-one-line flag for single line stats (Nick 55379 Craig-Wood) 55380 - Added weekday schedule into --bwlimit (Mateusz) 55381 - lsjson: Add option to show the original object IDs (Fabian 55382 Möller) 55383 - serve webdav: Make Content-Type without reading the file and add 55384 --etag-hash (Nick Craig-Wood) 55385 - build 55386 - Build macOS with native compiler (Nick Craig-Wood) 55387 - Update to use go1.11 for the build (Nick Craig-Wood) 55388 - rc 55389 - Added core/stats to return the stats (reddi1) 55390 - version --check: Prints the current release and beta versions 55391 (Nick Craig-Wood) 55392 - Bug Fixes 55393 - accounting 55394 - Fix time to completion estimates (Nick Craig-Wood) 55395 - Fix moving average speed for file stats (Nick Craig-Wood) 55396 - config: Fix error reading password from piped input (Nick 55397 Craig-Wood) 55398 - move: Fix --delete-empty-src-dirs flag to delete all empty dirs 55399 on move (ishuah) 55400 - Mount 55401 - Implement --daemon-timeout flag for OSXFUSE (Nick Craig-Wood) 55402 - Fix mount --daemon not working with encrypted config (Alex Chen) 55403 - Clip the number of blocks to 2^32-1 on macOS - fixes borg backup 55404 (Nick Craig-Wood) 55405 - VFS 55406 - Enable vfs-read-chunk-size by default (Fabian Möller) 55407 - Add the vfs/refresh rc command (Fabian Möller) 55408 - Add non recursive mode to vfs/refresh rc command (Fabian Möller) 55409 - Try to seek buffer on read only files (Fabian Möller) 55410 - Local 55411 - Fix crash when deprecated --local-no-unicode-normalization is 55412 supplied (Nick Craig-Wood) 55413 - Fix mkdir error when trying to copy files to the root of a drive 55414 on windows (Nick Craig-Wood) 55415 - Cache 55416 - Fix nil pointer deref when using lsjson on cached directory 55417 (Nick Craig-Wood) 55418 - Fix nil pointer deref for occasional crash on playback (Nick 55419 Craig-Wood) 55420 - Crypt 55421 - Fix accounting when checking hashes on upload (Nick Craig-Wood) 55422 - Amazon Cloud Drive 55423 - Make very clear in the docs that rclone has no ACD keys (Nick 55424 Craig-Wood) 55425 - Azure Blob 55426 - Add connection string and SAS URL auth (Nick Craig-Wood) 55427 - List the container to see if it exists (Nick Craig-Wood) 55428 - Port new Azure Blob Storage SDK (sandeepkru) 55429 - Added blob tier, tier between Hot, Cool and Archive. 55430 (sandeepkru) 55431 - Remove leading / from paths (Nick Craig-Wood) 55432 - B2 55433 - Support Application Keys (Nick Craig-Wood) 55434 - Remove leading / from paths (Nick Craig-Wood) 55435 - Box 55436 - Fix upload of > 2GB files on 32 bit platforms (Nick Craig-Wood) 55437 - Make --box-commit-retries flag defaulting to 100 to fix large 55438 uploads (Nick Craig-Wood) 55439 - Drive 55440 - Add --drive-keep-revision-forever flag (lewapm) 55441 - Handle gdocs when filtering file names in list (Fabian Möller) 55442 - Support using --fast-list for large speedups (Fabian Möller) 55443 - FTP 55444 - Fix Put mkParentDir failed: 521 for BunnyCDN (Nick Craig-Wood) 55445 - Google Cloud Storage 55446 - Fix index out of range error with --fast-list (Nick Craig-Wood) 55447 - Jottacloud 55448 - Fix MD5 error check (Oliver Heyme) 55449 - Handle empty time values (Martin Polden) 55450 - Calculate missing MD5s (Oliver Heyme) 55451 - Docs, fixes and tests for MD5 calculation (Nick Craig-Wood) 55452 - Add optional MimeTyper interface. (Sebastian Bünger) 55453 - Implement optional About interface (for df support). (Sebastian 55454 Bünger) 55455 - Mega 55456 - Wait for events instead of arbitrary sleeping (Nick Craig-Wood) 55457 - Add --mega-hard-delete flag (Nick Craig-Wood) 55458 - Fix failed logins with upper case chars in email (Nick 55459 Craig-Wood) 55460 - Onedrive 55461 - Shared folder support (Yoni Jah) 55462 - Implement DirMove (Cnly) 55463 - Fix rmdir sometimes deleting directories with contents (Nick 55464 Craig-Wood) 55465 - Pcloud 55466 - Delete half uploaded files on upload error (Nick Craig-Wood) 55467 - Qingstor 55468 - Remove leading / from paths (Nick Craig-Wood) 55469 - S3 55470 - Fix index out of range error with --fast-list (Nick Craig-Wood) 55471 - Add --s3-force-path-style (Nick Craig-Wood) 55472 - Add support for KMS Key ID (bsteiss) 55473 - Remove leading / from paths (Nick Craig-Wood) 55474 - Swift 55475 - Add storage_policy (Ruben Vandamme) 55476 - Make it so just storage_url or auth_token can be overridden 55477 (Nick Craig-Wood) 55478 - Fix server-side copy bug for unusual file names (Nick 55479 Craig-Wood) 55480 - Remove leading / from paths (Nick Craig-Wood) 55481 - WebDAV 55482 - Ensure we call MKCOL with a URL with a trailing / for QNAP 55483 interop (Nick Craig-Wood) 55484 - If root ends with / then don't check if it is a file (Nick 55485 Craig-Wood) 55486 - Don't accept redirects when reading metadata (Nick Craig-Wood) 55487 - Add bearer token (Macaroon) support for dCache (Nick Craig-Wood) 55488 - Document dCache and Macaroons (Onno Zweers) 55489 - Sharepoint recursion with different depth (Henning) 55490 - Attempt to remove failed uploads (Nick Craig-Wood) 55491 - Yandex 55492 - Fix listing/deleting files in the root (Nick Craig-Wood) 55493 55494 v1.42 - 2018-06-16 55495 55496 - New backends 55497 - OpenDrive (Oliver Heyme, Jakub Karlicek, ncw) 55498 - New commands 55499 - deletefile command (Filip Bartodziej) 55500 - New Features 55501 - copy, move: Copy single files directly, don't use --files-from 55502 work-around 55503 - this makes them much more efficient 55504 - Implement --max-transfer flag to quit transferring at a limit 55505 - make exit code 8 for --max-transfer exceeded 55506 - copy: copy empty source directories to destination (Ishuah 55507 Kariuki) 55508 - check: Add --one-way flag (Kasper Byrdal Nielsen) 55509 - Add siginfo handler for macOS for ctrl-T stats (kubatasiemski) 55510 - rc 55511 - add core/gc to run a garbage collection on demand 55512 - enable go profiling by default on the --rc port 55513 - return error from remote on failure 55514 - lsf 55515 - Add --absolute flag to add a leading / onto path names 55516 - Add --csv flag for compliant CSV output 55517 - Add 'm' format specifier to show the MimeType 55518 - Implement 'i' format for showing object ID 55519 - lsjson 55520 - Add MimeType to the output 55521 - Add ID field to output to show Object ID 55522 - Add --retries-sleep flag (Benjamin Joseph Dag) 55523 - Oauth tidy up web page and error handling (Henning Surmeier) 55524 - Bug Fixes 55525 - Password prompt output with --log-file fixed for unix (Filip 55526 Bartodziej) 55527 - Calculate ModifyWindow each time on the fly to fix various 55528 problems (Stefan Breunig) 55529 - Mount 55530 - Only print "File.rename error" if there actually is an error 55531 (Stefan Breunig) 55532 - Delay rename if file has open writers instead of failing 55533 outright (Stefan Breunig) 55534 - Ensure atexit gets run on interrupt 55535 - macOS enhancements 55536 - Make --noappledouble --noapplexattr 55537 - Add --volname flag and remove special chars from it 55538 - Make Get/List/Set/Remove xattr return ENOSYS for efficiency 55539 - Make --daemon work for macOS without CGO 55540 - VFS 55541 - Add --vfs-read-chunk-size and --vfs-read-chunk-size-limit 55542 (Fabian Möller) 55543 - Fix ChangeNotify for new or changed folders (Fabian Möller) 55544 - Local 55545 - Fix symlink/junction point directory handling under Windows 55546 - NB you will need to add -L to your command line to copy 55547 files with reparse points 55548 - Cache 55549 - Add non cached dirs on notifications (Remus Bunduc) 55550 - Allow root to be expired from rc (Remus Bunduc) 55551 - Clean remaining empty folders from temp upload path (Remus 55552 Bunduc) 55553 - Cache lists using batch writes (Remus Bunduc) 55554 - Use secure websockets for HTTPS Plex addresses (John Clayton) 55555 - Reconnect plex websocket on failures (Remus Bunduc) 55556 - Fix panic when running without plex configs (Remus Bunduc) 55557 - Fix root folder caching (Remus Bunduc) 55558 - Crypt 55559 - Check the encrypted hash of files when uploading for extra data 55560 security 55561 - Dropbox 55562 - Make Dropbox for business folders accessible using an initial / 55563 in the path 55564 - Google Cloud Storage 55565 - Low level retry all operations if necessary 55566 - Google Drive 55567 - Add --drive-acknowledge-abuse to download flagged files 55568 - Add --drive-alternate-export to fix large doc export 55569 - Don't attempt to choose Team Drives when using rclone config 55570 create 55571 - Fix change list polling with team drives 55572 - Fix ChangeNotify for folders (Fabian Möller) 55573 - Fix about (and df on a mount) for team drives 55574 - Onedrive 55575 - Errorhandler for onedrive for business requests (Henning 55576 Surmeier) 55577 - S3 55578 - Adjust upload concurrency with --s3-upload-concurrency 55579 (themylogin) 55580 - Fix --s3-chunk-size which was always using the minimum 55581 - SFTP 55582 - Add --ssh-path-override flag (Piotr Oleszczyk) 55583 - Fix slow downloads for long latency connections 55584 - Webdav 55585 - Add workarounds for biz.mail.ru 55586 - Ignore Reason-Phrase in status line to fix 4shared (Rodrigo) 55587 - Better error message generation 55588 55589 v1.41 - 2018-04-28 55590 55591 - New backends 55592 - Mega support added 55593 - Webdav now supports SharePoint cookie authentication (hensur) 55594 - New commands 55595 - link: create public link to files and folders (Stefan Breunig) 55596 - about: gets quota info from a remote (a-roussos, ncw) 55597 - hashsum: a generic tool for any hash to produce md5sum like 55598 output 55599 - New Features 55600 - lsd: Add -R flag and fix and update docs for all ls commands 55601 - ncdu: added a "refresh" key - CTRL-L (Keith Goldfarb) 55602 - serve restic: Add append-only mode (Steve Kriss) 55603 - serve restic: Disallow overwriting files in append-only mode 55604 (Alexander Neumann) 55605 - serve restic: Print actual listener address (Matt Holt) 55606 - size: Add --json flag (Matthew Holt) 55607 - sync: implement --ignore-errors (Mateusz Pabian) 55608 - dedupe: Add dedupe largest functionality (Richard Yang) 55609 - fs: Extend SizeSuffix to include TB and PB for rclone about 55610 - fs: add --dump goroutines and --dump openfiles for debugging 55611 - rc: implement core/memstats to print internal memory usage info 55612 - rc: new call rc/pid (Michael P. Dubner) 55613 - Compile 55614 - Drop support for go1.6 55615 - Release 55616 - Fix make tarball (Chih-Hsuan Yen) 55617 - Bug Fixes 55618 - filter: fix --min-age and --max-age together check 55619 - fs: limit MaxIdleConns and MaxIdleConnsPerHost in transport 55620 - lsd,lsf: make sure all times we output are in local time 55621 - rc: fix setting bwlimit to unlimited 55622 - rc: take note of the --rc-addr flag too as per the docs 55623 - Mount 55624 - Use About to return the correct disk total/used/free (e.g. in 55625 df) 55626 - Set --attr-timeout default to 1s - fixes: 55627 - rclone using too much memory 55628 - rclone not serving files to samba 55629 - excessive time listing directories 55630 - Fix df -i (upstream fix) 55631 - VFS 55632 - Filter files . and .. from directory listing 55633 - Only make the VFS cache if --vfs-cache-mode > Off 55634 - Local 55635 - Add --local-no-check-updated to disable updated file checks 55636 - Retry remove on Windows sharing violation error 55637 - Cache 55638 - Flush the memory cache after close 55639 - Purge file data on notification 55640 - Always forget parent dir for notifications 55641 - Integrate with Plex websocket 55642 - Add rc cache/stats (seuffert) 55643 - Add info log on notification 55644 - Box 55645 - Fix failure reading large directories - parse file/directory 55646 size as float 55647 - Dropbox 55648 - Fix crypt+obfuscate on dropbox 55649 - Fix repeatedly uploading the same files 55650 - FTP 55651 - Work around strange response from box FTP server 55652 - More workarounds for FTP servers to fix mkParentDir error 55653 - Fix no error on listing nonexistent directory 55654 - Google Cloud Storage 55655 - Add service_account_credentials (Matt Holt) 55656 - Detect bucket presence by listing it - minimises permissions 55657 needed 55658 - Ignore zero length directory markers 55659 - Google Drive 55660 - Add service_account_credentials (Matt Holt) 55661 - Fix directory move leaving a hardlinked directory behind 55662 - Return proper google errors when Opening files 55663 - When initialized with a filepath, optional features used 55664 incorrect root path (Stefan Breunig) 55665 - HTTP 55666 - Fix sync for servers which don't return Content-Length in HEAD 55667 - Onedrive 55668 - Add QuickXorHash support for OneDrive for business 55669 - Fix socket leak in multipart session upload 55670 - S3 55671 - Look in S3 named profile files for credentials 55672 - Add --s3-disable-checksum to disable checksum uploading (Chris 55673 Redekop) 55674 - Hierarchical configuration support (Giri Badanahatti) 55675 - Add in config for all the supported S3 providers 55676 - Add One Zone Infrequent Access storage class (Craig Rachel) 55677 - Add --use-server-modtime support (Peter Baumgartner) 55678 - Add --s3-chunk-size option to control multipart uploads 55679 - Ignore zero length directory markers 55680 - SFTP 55681 - Update docs to match code, fix typos and clarify 55682 disable_hashcheck prompt (Michael G. Noll) 55683 - Update docs with Synology quirks 55684 - Fail soft with a debug on hash failure 55685 - Swift 55686 - Add --use-server-modtime support (Peter Baumgartner) 55687 - Webdav 55688 - Support SharePoint cookie authentication (hensur) 55689 - Strip leading and trailing / off root 55690 55691 v1.40 - 2018-03-19 55692 55693 - New backends 55694 - Alias backend to create aliases for existing remote names 55695 (Fabian Möller) 55696 - New commands 55697 - lsf: list for parsing purposes (Jakub Tasiemski) 55698 - by default this is a simple non recursive list of files and 55699 directories 55700 - it can be configured to add more info in an easy to parse 55701 way 55702 - serve restic: for serving a remote as a Restic REST endpoint 55703 - This enables restic to use any backends that rclone can 55704 access 55705 - Thanks Alexander Neumann for help, patches and review 55706 - rc: enable the remote control of a running rclone 55707 - The running rclone must be started with --rc and related 55708 flags. 55709 - Currently there is support for bwlimit, and flushing for 55710 mount and cache. 55711 - New Features 55712 - --max-delete flag to add a delete threshold (Bjørn Erik 55713 Pedersen) 55714 - All backends now support RangeOption for ranged Open 55715 - cat: Use RangeOption for limited fetches to make more 55716 efficient 55717 - cryptcheck: make reading of nonce more efficient with 55718 RangeOption 55719 - serve http/webdav/restic 55720 - support SSL/TLS 55721 - add --user --pass and --htpasswd for authentication 55722 - copy/move: detect file size change during copy/move and abort 55723 transfer (ishuah) 55724 - cryptdecode: added option to return encrypted file names. 55725 (ishuah) 55726 - lsjson: add --encrypted to show encrypted name (Jakub Tasiemski) 55727 - Add --stats-file-name-length to specify the printed file name 55728 length for stats (Will Gunn) 55729 - Compile 55730 - Code base was shuffled and factored 55731 - backends moved into a backend directory 55732 - large packages split up 55733 - See the CONTRIBUTING.md doc for info as to what lives where 55734 now 55735 - Update to using go1.10 as the default go version 55736 - Implement daily full integration tests 55737 - Release 55738 - Include a source tarball and sign it and the binaries 55739 - Sign the git tags as part of the release process 55740 - Add .deb and .rpm packages as part of the build 55741 - Make a beta release for all branches on the main repo (but not 55742 pull requests) 55743 - Bug Fixes 55744 - config: fixes errors on nonexistent config by loading config 55745 file only on first access 55746 - config: retry saving the config after failure (Mateusz) 55747 - sync: when using --backup-dir don't delete files if we can't set 55748 their modtime 55749 - this fixes odd behaviour with Dropbox and --backup-dir 55750 - fshttp: fix idle timeouts for HTTP connections 55751 - serve http: fix serving files with : in - fixes 55752 - Fix --exclude-if-present to ignore directories which it doesn't 55753 have permission for (Iakov Davydov) 55754 - Make accounting work properly with crypt and b2 55755 - remove --no-traverse flag because it is obsolete 55756 - Mount 55757 - Add --attr-timeout flag to control attribute caching in kernel 55758 - this now defaults to 0 which is correct but less efficient 55759 - see the mount docs for more info 55760 - Add --daemon flag to allow mount to run in the background 55761 (ishuah) 55762 - Fix: Return ENOSYS rather than EIO on attempted link 55763 - This fixes FileZilla accessing an rclone mount served over 55764 sftp. 55765 - Fix setting modtime twice 55766 - Mount tests now run on CI for Linux (mount & cmount)/Mac/Windows 55767 - Many bugs fixed in the VFS layer - see below 55768 - VFS 55769 - Many fixes for --vfs-cache-mode writes and above 55770 - Update cached copy if we know it has changed (fixes stale 55771 data) 55772 - Clean path names before using them in the cache 55773 - Disable cache cleaner if --vfs-cache-poll-interval=0 55774 - Fill and clean the cache immediately on startup 55775 - Fix Windows opening every file when it stats the file 55776 - Fix applying modtime for an open Write Handle 55777 - Fix creation of files when truncating 55778 - Write 0 bytes when flushing unwritten handles to avoid race 55779 conditions in FUSE 55780 - Downgrade "poll-interval is not supported" message to Info 55781 - Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 55782 - Local 55783 - Downgrade "invalid cross-device link: trying copy" to debug 55784 - Make DirMove return fs.ErrorCantDirMove to allow fallback to 55785 Copy for cross device 55786 - Fix race conditions updating the hashes 55787 - Cache 55788 - Add support for polling - cache will update when remote changes 55789 on supported backends 55790 - Reduce log level for Plex api 55791 - Fix dir cache issue 55792 - Implement --cache-db-wait-time flag 55793 - Improve efficiency with RangeOption and RangeSeek 55794 - Fix dirmove with temp fs enabled 55795 - Notify vfs when using temp fs 55796 - Offline uploading 55797 - Remote control support for path flushing 55798 - Amazon cloud drive 55799 - Rclone no longer has any working keys - disable integration 55800 tests 55801 - Implement DirChangeNotify to notify cache/vfs/mount of changes 55802 - Azureblob 55803 - Don't check for bucket/container presence if listing was OK 55804 - this makes rclone do one less request per invocation 55805 - Improve accounting for chunked uploads 55806 - Backblaze B2 55807 - Don't check for bucket/container presence if listing was OK 55808 - this makes rclone do one less request per invocation 55809 - Box 55810 - Improve accounting for chunked uploads 55811 - Dropbox 55812 - Fix custom oauth client parameters 55813 - Google Cloud Storage 55814 - Don't check for bucket/container presence if listing was OK 55815 - this makes rclone do one less request per invocation 55816 - Google Drive 55817 - Migrate to api v3 (Fabian Möller) 55818 - Add scope configuration and root folder selection 55819 - Add --drive-impersonate for service accounts 55820 - thanks to everyone who tested, explored and contributed docs 55821 - Add --drive-use-created-date to use created date as modified 55822 date (nbuchanan) 55823 - Request the export formats only when required 55824 - This makes rclone quicker when there are no google docs 55825 - Fix finding paths with latin1 chars (a workaround for a drive 55826 bug) 55827 - Fix copying of a single Google doc file 55828 - Fix --drive-auth-owner-only to look in all directories 55829 - HTTP 55830 - Fix handling of directories with & in 55831 - Onedrive 55832 - Removed upload cutoff and always do session uploads 55833 - this stops the creation of multiple versions on business 55834 onedrive 55835 - Overwrite object size value with real size when reading file. 55836 (Victor) 55837 - this fixes oddities when onedrive misreports the size of 55838 images 55839 - Pcloud 55840 - Remove unused chunked upload flag and code 55841 - Qingstor 55842 - Don't check for bucket/container presence if listing was OK 55843 - this makes rclone do one less request per invocation 55844 - S3 55845 - Support hashes for multipart files (Chris Redekop) 55846 - Initial support for IBM COS (S3) (Giri Badanahatti) 55847 - Update docs to discourage use of v2 auth with CEPH and others 55848 - Don't check for bucket/container presence if listing was OK 55849 - this makes rclone do one less request per invocation 55850 - Fix server-side copy and set modtime on files with + in 55851 - SFTP 55852 - Add option to disable remote hash check command execution (Jon 55853 Fautley) 55854 - Add --sftp-ask-password flag to prompt for password when needed 55855 (Leo R. Lundgren) 55856 - Add set_modtime configuration option 55857 - Fix following of symlinks 55858 - Fix reading config file outside of Fs setup 55859 - Fix reading $USER in username fallback not $HOME 55860 - Fix running under crontab - Use correct OS way of reading 55861 username 55862 - Swift 55863 - Fix refresh of authentication token 55864 - in v1.39 a bug was introduced which ignored new tokens - 55865 this fixes it 55866 - Fix extra HEAD transaction when uploading a new file 55867 - Don't check for bucket/container presence if listing was OK 55868 - this makes rclone do one less request per invocation 55869 - Webdav 55870 - Add new time formats to support mydrive.ch and others 55871 55872 v1.39 - 2017-12-23 55873 55874 - New backends 55875 - WebDAV 55876 - tested with nextcloud, owncloud, put.io and others! 55877 - Pcloud 55878 - cache - wraps a cache around other backends (Remus Bunduc) 55879 - useful in combination with mount 55880 - NB this feature is in beta so use with care 55881 - New commands 55882 - serve command with subcommands: 55883 - serve webdav: this implements a webdav server for any rclone 55884 remote. 55885 - serve http: command to serve a remote over HTTP 55886 - config: add sub commands for full config file management 55887 - create/delete/dump/edit/file/password/providers/show/update 55888 - touch: to create or update the timestamp of a file (Jakub 55889 Tasiemski) 55890 - New Features 55891 - curl install for rclone (Filip Bartodziej) 55892 - --stats now shows percentage, size, rate and ETA in condensed 55893 form (Ishuah Kariuki) 55894 - --exclude-if-present to exclude a directory if a file is present 55895 (Iakov Davydov) 55896 - rmdirs: add --leave-root flag (lewapm) 55897 - move: add --delete-empty-src-dirs flag to remove dirs after move 55898 (Ishuah Kariuki) 55899 - Add --dump flag, introduce --dump requests, responses and remove 55900 --dump-auth, --dump-filters 55901 - Obscure X-Auth-Token: from headers when dumping too 55902 - Document and implement exit codes for different failure modes 55903 (Ishuah Kariuki) 55904 - Compile 55905 - Bug Fixes 55906 - Retry lots more different types of errors to make multipart 55907 transfers more reliable 55908 - Save the config before asking for a token, fixes disappearing 55909 oauth config 55910 - Warn the user if --include and --exclude are used together 55911 (Ernest Borowski) 55912 - Fix duplicate files (e.g. on Google drive) causing spurious 55913 copies 55914 - Allow trailing and leading whitespace for passwords (Jason Rose) 55915 - ncdu: fix crashes on empty directories 55916 - rcat: fix goroutine leak 55917 - moveto/copyto: Fix to allow copying to the same name 55918 - Mount 55919 - --vfs-cache mode to make writes into mounts more reliable. 55920 - this requires caching files on the disk (see --cache-dir) 55921 - As this is a new feature, use with care 55922 - Use sdnotify to signal systemd the mount is ready (Fabian 55923 Möller) 55924 - Check if directory is not empty before mounting (Ernest 55925 Borowski) 55926 - Local 55927 - Add error message for cross file system moves 55928 - Fix equality check for times 55929 - Dropbox 55930 - Rework multipart upload 55931 - buffer the chunks when uploading large files so they can be 55932 retried 55933 - change default chunk size to 48MB now we are buffering them 55934 in memory 55935 - retry every error after the first chunk is done successfully 55936 - Fix error when renaming directories 55937 - Swift 55938 - Fix crash on bad authentication 55939 - Google Drive 55940 - Add service account support (Tim Cooijmans) 55941 - S3 55942 - Make it work properly with Digital Ocean Spaces (Andrew 55943 Starr-Bochicchio) 55944 - Fix crash if a bad listing is received 55945 - Add support for ECS task IAM roles (David Minor) 55946 - Backblaze B2 55947 - Fix multipart upload retries 55948 - Fix --hard-delete to make it work 100% of the time 55949 - Swift 55950 - Allow authentication with storage URL and auth key (Giovanni 55951 Pizzi) 55952 - Add new fields for swift configuration to support IBM Bluemix 55953 Swift (Pierre Carlson) 55954 - Add OS_TENANT_ID and OS_USER_ID to config 55955 - Allow configs with user id instead of user name 55956 - Check if swift segments container exists before creating (John 55957 Leach) 55958 - Fix memory leak in swift transfers (upstream fix) 55959 - SFTP 55960 - Add option to enable the use of aes128-cbc cipher (Jon Fautley) 55961 - Amazon cloud drive 55962 - Fix download of large files failing with "Only one auth 55963 mechanism allowed" 55964 - crypt 55965 - Option to encrypt directory names or leave them intact 55966 - Implement DirChangeNotify (Fabian Möller) 55967 - onedrive 55968 - Add option to choose resourceURL during setup of OneDrive 55969 Business account if more than one is available for user 55970 55971 v1.38 - 2017-09-30 55972 55973 - New backends 55974 - Azure Blob Storage (thanks Andrei Dragomir) 55975 - Box 55976 - Onedrive for Business (thanks Oliver Heyme) 55977 - QingStor from QingCloud (thanks wuyu) 55978 - New commands 55979 - rcat - read from standard input and stream upload 55980 - tree - shows a nicely formatted recursive listing 55981 - cryptdecode - decode encrypted file names (thanks ishuah) 55982 - config show - print the config file 55983 - config file - print the config file location 55984 - New Features 55985 - Empty directories are deleted on sync 55986 - dedupe - implement merging of duplicate directories 55987 - check and cryptcheck made more consistent and use less memory 55988 - cleanup for remaining remotes (thanks ishuah) 55989 - --immutable for ensuring that files don't change (thanks Jacob 55990 McNamee) 55991 - --user-agent option (thanks Alex McGrath Kraak) 55992 - --disable flag to disable optional features 55993 - --bind flag for choosing the local addr on outgoing connections 55994 - Support for zsh auto-completion (thanks bpicode) 55995 - Stop normalizing file names but do a normalized compare in sync 55996 - Compile 55997 - Update to using go1.9 as the default go version 55998 - Remove snapd build due to maintenance problems 55999 - Bug Fixes 56000 - Improve retriable error detection which makes multipart uploads 56001 better 56002 - Make check obey --ignore-size 56003 - Fix bwlimit toggle in conjunction with schedules (thanks 56004 cbruegg) 56005 - config ensures newly written config is on the same mount 56006 - Local 56007 - Revert to copy when moving file across file system boundaries 56008 - --skip-links to suppress symlink warnings (thanks Zhiming Wang) 56009 - Mount 56010 - Reuse rcat internals to support uploads from all remotes 56011 - Dropbox 56012 - Fix "entry doesn't belong in directory" error 56013 - Stop using deprecated API methods 56014 - Swift 56015 - Fix server-side copy to empty container with --fast-list 56016 - Google Drive 56017 - Change the default for --drive-use-trash to true 56018 - S3 56019 - Set session token when using STS (thanks Girish Ramakrishnan) 56020 - Glacier docs and error messages (thanks Jan Varho) 56021 - Read 1000 (not 1024) items in dir listings to fix Wasabi 56022 - Backblaze B2 56023 - Fix SHA1 mismatch when downloading files with no SHA1 56024 - Calculate missing hashes on the fly instead of spooling 56025 - --b2-hard-delete to permanently delete (not hide) files (thanks 56026 John Papandriopoulos) 56027 - Hubic 56028 - Fix creating containers - no longer have to use the default 56029 container 56030 - Swift 56031 - Optionally configure from a standard set of OpenStack 56032 environment vars 56033 - Add endpoint_type config 56034 - Google Cloud Storage 56035 - Fix bucket creation to work with limited permission users 56036 - SFTP 56037 - Implement connection pooling for multiple ssh connections 56038 - Limit new connections per second 56039 - Add support for MD5 and SHA1 hashes where available (thanks 56040 Christian Brüggemann) 56041 - HTTP 56042 - Fix URL encoding issues 56043 - Fix directories with : in 56044 - Fix panic with URL encoded content 56045 56046 v1.37 - 2017-07-22 56047 56048 - New backends 56049 - FTP - thanks to Antonio Messina 56050 - HTTP - thanks to Vasiliy Tolstov 56051 - New commands 56052 - rclone ncdu - for exploring a remote with a text based user 56053 interface. 56054 - rclone lsjson - for listing with a machine-readable output 56055 - rclone dbhashsum - to show Dropbox style hashes of files (local 56056 or Dropbox) 56057 - New Features 56058 - Implement --fast-list flag 56059 - This allows remotes to list recursively if they can 56060 - This uses less transactions (important if you pay for them) 56061 - This may or may not be quicker 56062 - This will use more memory as it has to hold the listing in 56063 memory 56064 - --old-sync-method deprecated - the remaining uses are 56065 covered by --fast-list 56066 - This involved a major re-write of all the listing code 56067 - Add --tpslimit and --tpslimit-burst to limit transactions per 56068 second 56069 - this is useful in conjunction with rclone mount to limit 56070 external apps 56071 - Add --stats-log-level so can see --stats without -v 56072 - Print password prompts to stderr - Hraban Luyat 56073 - Warn about duplicate files when syncing 56074 - Oauth improvements 56075 - allow auth_url and token_url to be set in the config file 56076 - Print redirection URI if using own credentials. 56077 - Don't Mkdir at the start of sync to save transactions 56078 - Compile 56079 - Update build to go1.8.3 56080 - Require go1.6 for building rclone 56081 - Compile 386 builds with "GO386=387" for maximum compatibility 56082 - Bug Fixes 56083 - Fix menu selection when no remotes 56084 - Config saving reworked to not kill the file if disk gets full 56085 - Don't delete remote if name does not change while renaming 56086 - moveto, copyto: report transfers and checks as per move and copy 56087 - Local 56088 - Add --local-no-unicode-normalization flag - Bob Potter 56089 - Mount 56090 - Now supported on Windows using cgofuse and WinFsp - thanks to 56091 Bill Zissimopoulos for much help 56092 - Compare checksums on upload/download via FUSE 56093 - Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM - 56094 Jérôme Vizcaino 56095 - On read only open of file, make open pending until first read 56096 - Make --read-only reject modify operations 56097 - Implement ModTime via FUSE for remotes that support it 56098 - Allow modTime to be changed even before all writers are closed 56099 - Fix panic on renames 56100 - Fix hang on errored upload 56101 - Crypt 56102 - Report the name:root as specified by the user 56103 - Add an "obfuscate" option for filename encryption - Stephen 56104 Harris 56105 - Amazon Drive 56106 - Fix initialization order for token renewer 56107 - Remove revoked credentials, allow oauth proxy config and update 56108 docs 56109 - B2 56110 - Reduce minimum chunk size to 5MB 56111 - Drive 56112 - Add team drive support 56113 - Reduce bandwidth by adding fields for partial responses - Martin 56114 Kristensen 56115 - Implement --drive-shared-with-me flag to view shared with me 56116 files - Danny Tsai 56117 - Add --drive-trashed-only to read only the files in the trash 56118 - Remove obsolete --drive-full-list 56119 - Add missing seek to start on retries of chunked uploads 56120 - Fix stats accounting for upload 56121 - Convert / in names to a unicode equivalent (/) 56122 - Poll for Google Drive changes when mounted 56123 - OneDrive 56124 - Fix the uploading of files with spaces 56125 - Fix initialization order for token renewer 56126 - Display speeds accurately when uploading - Yoni Jah 56127 - Swap to using http://localhost:53682/ as redirect URL - Michael 56128 Ledin 56129 - Retry on token expired error, reset upload body on retry - Yoni 56130 Jah 56131 - Google Cloud Storage 56132 - Add ability to specify location and storage class via config and 56133 command line - thanks gdm85 56134 - Create container if necessary on server-side copy 56135 - Increase directory listing chunk to 1000 to increase performance 56136 - Obtain a refresh token for GCS - Steven Lu 56137 - Yandex 56138 - Fix the name reported in log messages (was empty) 56139 - Correct error return for listing empty directory 56140 - Dropbox 56141 - Rewritten to use the v2 API 56142 - Now supports ModTime 56143 - Can only set by uploading the file again 56144 - If you uploaded with an old rclone, rclone may upload 56145 everything again 56146 - Use --size-only or --checksum to avoid this 56147 - Now supports the Dropbox content hashing scheme 56148 - Now supports low level retries 56149 - S3 56150 - Work around eventual consistency in bucket creation 56151 - Create container if necessary on server-side copy 56152 - Add us-east-2 (Ohio) and eu-west-2 (London) S3 regions - Zahiar 56153 Ahmed 56154 - Swift, Hubic 56155 - Fix zero length directory markers showing in the subdirectory 56156 listing 56157 - this caused lots of duplicate transfers 56158 - Fix paged directory listings 56159 - this caused duplicate directory errors 56160 - Create container if necessary on server-side copy 56161 - Increase directory listing chunk to 1000 to increase performance 56162 - Make sensible error if the user forgets the container 56163 - SFTP 56164 - Add support for using ssh key files 56165 - Fix under Windows 56166 - Fix ssh agent on Windows 56167 - Adapt to latest version of library - Igor Kharin 56168 56169 v1.36 - 2017-03-18 56170 56171 - New Features 56172 - SFTP remote (Jack Schmidt) 56173 - Re-implement sync routine to work a directory at a time reducing 56174 memory usage 56175 - Logging revamped to be more inline with rsync - now much 56176 quieter * -v only shows transfers * -vv is for full debug * 56177 --syslog to log to syslog on capable platforms 56178 - Implement --backup-dir and --suffix 56179 - Implement --track-renames (initial implementation by Bjørn Erik 56180 Pedersen) 56181 - Add time-based bandwidth limits (Lukas Loesche) 56182 - rclone cryptcheck: checks integrity of crypt remotes 56183 - Allow all config file variables and options to be set from 56184 environment variables 56185 - Add --buffer-size parameter to control buffer size for copy 56186 - Make --delete-after the default 56187 - Add --ignore-checksum flag (fixed by Hisham Zarka) 56188 - rclone check: Add --download flag to check all the data, not 56189 just hashes 56190 - rclone cat: add --head, --tail, --offset, --count and --discard 56191 - rclone config: when choosing from a list, allow the value to be 56192 entered too 56193 - rclone config: allow rename and copy of remotes 56194 - rclone obscure: for generating encrypted passwords for rclone's 56195 config (T.C. Ferguson) 56196 - Comply with XDG Base Directory specification (Dario Giovannetti) 56197 - this moves the default location of the config file in a 56198 backwards compatible way 56199 - Release changes 56200 - Ubuntu snap support (Dedsec1) 56201 - Compile with go 1.8 56202 - MIPS/Linux big and little endian support 56203 - Bug Fixes 56204 - Fix copyto copying things to the wrong place if the destination 56205 dir didn't exist 56206 - Fix parsing of remotes in moveto and copyto 56207 - Fix --delete-before deleting files on copy 56208 - Fix --files-from with an empty file copying everything 56209 - Fix sync: don't update mod times if --dry-run set 56210 - Fix MimeType propagation 56211 - Fix filters to add ** rules to directory rules 56212 - Local 56213 - Implement -L, --copy-links flag to allow rclone to follow 56214 symlinks 56215 - Open files in write only mode so rclone can write to an rclone 56216 mount 56217 - Fix unnormalised unicode causing problems reading directories 56218 - Fix interaction between -x flag and --max-depth 56219 - Mount 56220 - Implement proper directory handling (mkdir, rmdir, renaming) 56221 - Make include and exclude filters apply to mount 56222 - Implement read and write async buffers - control with 56223 --buffer-size 56224 - Fix fsync on for directories 56225 - Fix retry on network failure when reading off crypt 56226 - Crypt 56227 - Add --crypt-show-mapping to show encrypted file mapping 56228 - Fix crypt writer getting stuck in a loop 56229 - IMPORTANT this bug had the potential to cause data 56230 corruption when 56231 - reading data from a network based remote and 56232 - writing to a crypt on Google Drive 56233 - Use the cryptcheck command to validate your data if you are 56234 concerned 56235 - If syncing two crypt remotes, sync the unencrypted remote 56236 - Amazon Drive 56237 - Fix panics on Move (rename) 56238 - Fix panic on token expiry 56239 - B2 56240 - Fix inconsistent listings and rclone check 56241 - Fix uploading empty files with go1.8 56242 - Constrain memory usage when doing multipart uploads 56243 - Fix upload url not being refreshed properly 56244 - Drive 56245 - Fix Rmdir on directories with trashed files 56246 - Fix "Ignoring unknown object" when downloading 56247 - Add --drive-list-chunk 56248 - Add --drive-skip-gdocs (Károly Oláh) 56249 - OneDrive 56250 - Implement Move 56251 - Fix Copy 56252 - Fix overwrite detection in Copy 56253 - Fix waitForJob to parse errors correctly 56254 - Use token renewer to stop auth errors on long uploads 56255 - Fix uploading empty files with go1.8 56256 - Google Cloud Storage 56257 - Fix depth 1 directory listings 56258 - Yandex 56259 - Fix single level directory listing 56260 - Dropbox 56261 - Normalise the case for single level directory listings 56262 - Fix depth 1 listing 56263 - S3 56264 - Added ca-central-1 region (Jon Yergatian) 56265 56266 v1.35 - 2017-01-02 56267 56268 - New Features 56269 - moveto and copyto commands for choosing a destination name on 56270 copy/move 56271 - rmdirs command to recursively delete empty directories 56272 - Allow repeated --include/--exclude/--filter options 56273 - Only show transfer stats on commands which transfer stuff 56274 - show stats on any command using the --stats flag 56275 - Allow overlapping directories in move when server-side dir move 56276 is supported 56277 - Add --stats-unit option - thanks Scott McGillivray 56278 - Bug Fixes 56279 - Fix the config file being overwritten when two rclone instances 56280 are running 56281 - Make rclone lsd obey the filters properly 56282 - Fix compilation on mips 56283 - Fix not transferring files that don't differ in size 56284 - Fix panic on nil retry/fatal error 56285 - Mount 56286 - Retry reads on error - should help with reliability a lot 56287 - Report the modification times for directories from the remote 56288 - Add bandwidth accounting and limiting (fixes --bwlimit) 56289 - If --stats provided will show stats and which files are 56290 transferring 56291 - Support R/W files if truncate is set. 56292 - Implement statfs interface so df works 56293 - Note that write is now supported on Amazon Drive 56294 - Report number of blocks in a file - thanks Stefan Breunig 56295 - Crypt 56296 - Prevent the user pointing crypt at itself 56297 - Fix failed to authenticate decrypted block errors 56298 - these will now return the underlying unexpected EOF instead 56299 - Amazon Drive 56300 - Add support for server-side move and directory move - thanks 56301 Stefan Breunig 56302 - Fix nil pointer deref on size attribute 56303 - B2 56304 - Use new prefix and delimiter parameters in directory listings 56305 - This makes --max-depth 1 dir listings as used in mount much 56306 faster 56307 - Reauth the account while doing uploads too - should help with 56308 token expiry 56309 - Drive 56310 - Make DirMove more efficient and complain about moving the root 56311 - Create destination directory on Move() 56312 56313 v1.34 - 2016-11-06 56314 56315 - New Features 56316 - Stop single file and --files-from operations iterating through 56317 the source bucket. 56318 - Stop removing failed upload to cloud storage remotes 56319 - Make ContentType be preserved for cloud to cloud copies 56320 - Add support to toggle bandwidth limits via SIGUSR2 - thanks 56321 Marco Paganini 56322 - rclone check shows count of hashes that couldn't be checked 56323 - rclone listremotes command 56324 - Support linux/arm64 build - thanks Fredrik Fornwall 56325 - Remove Authorization: lines from --dump-headers output 56326 - Bug Fixes 56327 - Ignore files with control characters in the names 56328 - Fix rclone move command 56329 - Delete src files which already existed in dst 56330 - Fix deletion of src file when dst file older 56331 - Fix rclone check on encrypted file systems 56332 - Make failed uploads not count as "Transferred" 56333 - Make sure high level retries show with -q 56334 - Use a vendor directory with godep for repeatable builds 56335 - rclone mount - FUSE 56336 - Implement FUSE mount options 56337 - --no-modtime, --debug-fuse, --read-only, --allow-non-empty, 56338 --allow-root, --allow-other 56339 - --default-permissions, --write-back-cache, --max-read-ahead, 56340 --umask, --uid, --gid 56341 - Add --dir-cache-time to control caching of directory entries 56342 - Implement seek for files opened for read (useful for video 56343 players) 56344 - with -no-seek flag to disable 56345 - Fix crash on 32 bit ARM (alignment of 64 bit counter) 56346 - ...and many more internal fixes and improvements! 56347 - Crypt 56348 - Don't show encrypted password in configurator to stop confusion 56349 - Amazon Drive 56350 - New wait for upload option --acd-upload-wait-per-gb 56351 - upload timeouts scale by file size and can be disabled 56352 - Add 502 Bad Gateway to list of errors we retry 56353 - Fix overwriting a file with a zero length file 56354 - Fix ACD file size warning limit - thanks Felix Bünemann 56355 - Local 56356 - Unix: implement -x/--one-file-system to stay on a single file 56357 system 56358 - thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana 56359 - Windows: ignore the symlink bit on files 56360 - Windows: Ignore directory-based junction points 56361 - B2 56362 - Make sure each upload has at least one upload slot - fixes 56363 strange upload stats 56364 - Fix uploads when using crypt 56365 - Fix download of large files (sha1 mismatch) 56366 - Return error when we try to create a bucket which someone else 56367 owns 56368 - Update B2 docs with Data usage, and Crypt section - thanks 56369 Tomasz Mazur 56370 - S3 56371 - Command line and config file support for 56372 - Setting/overriding ACL - thanks Radek Šenfeld 56373 - Setting storage class - thanks Asko Tamm 56374 - Drive 56375 - Make exponential backoff work exactly as per Google 56376 specification 56377 - add .epub, .odp and .tsv as export formats. 56378 - Swift 56379 - Don't read metadata for directory marker objects 56380 56381 v1.33 - 2016-08-24 56382 56383 - New Features 56384 - Implement encryption 56385 - data encrypted in NACL secretbox format 56386 - with optional file name encryption 56387 - New commands 56388 - rclone mount - implements FUSE mounting of remotes 56389 (EXPERIMENTAL) 56390 - works on Linux, FreeBSD and OS X (need testers for the 56391 last 2!) 56392 - rclone cat - outputs remote file or files to the terminal 56393 - rclone genautocomplete - command to make a bash completion 56394 script for rclone 56395 - Editing a remote using rclone config now goes through the wizard 56396 - Compile with go 1.7 - this fixes rclone on macOS Sierra and on 56397 386 processors 56398 - Use cobra for sub commands and docs generation 56399 - drive 56400 - Document how to make your own client_id 56401 - s3 56402 - User-configurable Amazon S3 ACL (thanks Radek Šenfeld) 56403 - b2 56404 - Fix stats accounting for upload - no more jumping to 100% done 56405 - On cleanup delete hide marker if it is the current file 56406 - New B2 API endpoint (thanks Per Cederberg) 56407 - Set maximum backoff to 5 Minutes 56408 - onedrive 56409 - Fix URL escaping in file names - e.g. uploading files with + in 56410 them. 56411 - amazon cloud drive 56412 - Fix token expiry during large uploads 56413 - Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors 56414 - local 56415 - Fix filenames with invalid UTF-8 not being uploaded 56416 - Fix problem with some UTF-8 characters on OS X 56417 56418 v1.32 - 2016-07-13 56419 56420 - Backblaze B2 56421 - Fix upload of files large files not in root 56422 56423 v1.31 - 2016-07-13 56424 56425 - New Features 56426 - Reduce memory on sync by about 50% 56427 - Implement --no-traverse flag to stop copy traversing the 56428 destination remote. 56429 - This can be used to reduce memory usage down to the smallest 56430 possible. 56431 - Useful to copy a small number of files into a large 56432 destination folder. 56433 - Implement cleanup command for emptying trash / removing old 56434 versions of files 56435 - Currently B2 only 56436 - Single file handling improved 56437 - Now copied with --files-from 56438 - Automatically sets --no-traverse when copying a single file 56439 - Info on using installing with ansible - thanks Stefan Weichinger 56440 - Implement --no-update-modtime flag to stop rclone fixing the 56441 remote modified times. 56442 - Bug Fixes 56443 - Fix move command - stop it running for overlapping Fses - this 56444 was causing data loss. 56445 - Local 56446 - Fix incomplete hashes - this was causing problems for B2. 56447 - Amazon Drive 56448 - Rename Amazon Cloud Drive to Amazon Drive - no changes to config 56449 file needed. 56450 - Swift 56451 - Add support for non-default project domain - thanks Antonio 56452 Messina. 56453 - S3 56454 - Add instructions on how to use rclone with minio. 56455 - Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions. 56456 - Skip setting the modified time for objects > 5GB as it isn't 56457 possible. 56458 - Backblaze B2 56459 - Add --b2-versions flag so old versions can be listed and 56460 retrieved. 56461 - Treat 403 errors (e.g. cap exceeded) as fatal. 56462 - Implement cleanup command for deleting old file versions. 56463 - Make error handling compliant with B2 integrations notes. 56464 - Fix handling of token expiry. 56465 - Implement --b2-test-mode to set X-Bz-Test-Mode header. 56466 - Set cutoff for chunked upload to 200MB as per B2 guidelines. 56467 - Make upload multi-threaded. 56468 - Dropbox 56469 - Don't retry 461 errors. 56470 56471 v1.30 - 2016-06-18 56472 56473 - New Features 56474 - Directory listing code reworked for more features and better 56475 error reporting (thanks to Klaus Post for help). This enables 56476 - Directory include filtering for efficiency 56477 - --max-depth parameter 56478 - Better error reporting 56479 - More to come 56480 - Retry more errors 56481 - Add --ignore-size flag - for uploading images to onedrive 56482 - Log -v output to stdout by default 56483 - Display the transfer stats in more human-readable form 56484 - Make 0 size files specifiable with --max-size 0b 56485 - Add b suffix so we can specify bytes in --bwlimit, --min-size, 56486 etc. 56487 - Use "password:" instead of "password>" prompt - thanks Klaus 56488 Post and Leigh Klotz 56489 - Bug Fixes 56490 - Fix retry doing one too many retries 56491 - Local 56492 - Fix problems with OS X and UTF-8 characters 56493 - Amazon Drive 56494 - Check a file exists before uploading to help with 408 Conflict 56495 errors 56496 - Reauth on 401 errors - this has been causing a lot of problems 56497 - Work around spurious 403 errors 56498 - Restart directory listings on error 56499 - Google Drive 56500 - Check a file exists before uploading to help with duplicates 56501 - Fix retry of multipart uploads 56502 - Backblaze B2 56503 - Implement large file uploading 56504 - S3 56505 - Add AES256 server-side encryption for - thanks Justin R. Wilson 56506 - Google Cloud Storage 56507 - Make sure we don't use conflicting content types on upload 56508 - Add service account support - thanks Michal Witkowski 56509 - Swift 56510 - Add auth version parameter 56511 - Add domain option for openstack (v3 auth) - thanks Fabian Ruff 56512 56513 v1.29 - 2016-04-18 56514 56515 - New Features 56516 - Implement -I, --ignore-times for unconditional upload 56517 - Improve dedupecommand 56518 - Now removes identical copies without asking 56519 - Now obeys --dry-run 56520 - Implement --dedupe-mode for non interactive running 56521 - --dedupe-mode interactive - interactive the default. 56522 - --dedupe-mode skip - removes identical files then skips 56523 anything left. 56524 - --dedupe-mode first - removes identical files then keeps 56525 the first one. 56526 - --dedupe-mode newest - removes identical files then 56527 keeps the newest one. 56528 - --dedupe-mode oldest - removes identical files then 56529 keeps the oldest one. 56530 - --dedupe-mode rename - removes identical files then 56531 renames the rest to be different. 56532 - Bug fixes 56533 - Make rclone check obey the --size-only flag. 56534 - Use "application/octet-stream" if discovered mime type is 56535 invalid. 56536 - Fix missing "quit" option when there are no remotes. 56537 - Google Drive 56538 - Increase default chunk size to 8 MB - increases upload speed of 56539 big files 56540 - Speed up directory listings and make more reliable 56541 - Add missing retries for Move and DirMove - increases reliability 56542 - Preserve mime type on file update 56543 - Backblaze B2 56544 - Enable mod time syncing 56545 - This means that B2 will now check modification times 56546 - It will upload new files to update the modification times 56547 - (there isn't an API to just set the mod time.) 56548 - If you want the old behaviour use --size-only. 56549 - Update API to new version 56550 - Fix parsing of mod time when not in metadata 56551 - Swift/Hubic 56552 - Don't return an MD5SUM for static large objects 56553 - S3 56554 - Fix uploading files bigger than 50GB 56555 56556 v1.28 - 2016-03-01 56557 56558 - New Features 56559 - Configuration file encryption - thanks Klaus Post 56560 - Improve rclone config adding more help and making it easier to 56561 understand 56562 - Implement -u/--update so creation times can be used on all 56563 remotes 56564 - Implement --low-level-retries flag 56565 - Optionally disable gzip compression on downloads with 56566 --no-gzip-encoding 56567 - Bug fixes 56568 - Don't make directories if --dry-run set 56569 - Fix and document the move command 56570 - Fix redirecting stderr on unix-like OSes when using --log-file 56571 - Fix delete command to wait until all finished - fixes missing 56572 deletes. 56573 - Backblaze B2 56574 - Use one upload URL per go routine fixes 56575 more than one upload using auth token 56576 - Add pacing, retries and reauthentication - fixes token expiry 56577 problems 56578 - Upload without using a temporary file from local (and remotes 56579 which support SHA1) 56580 - Fix reading metadata for all files when it shouldn't have been 56581 - Drive 56582 - Fix listing drive documents at root 56583 - Disable copy and move for Google docs 56584 - Swift 56585 - Fix uploading of chunked files with non ASCII characters 56586 - Allow setting of storage_url in the config - thanks Xavier Lucas 56587 - S3 56588 - Allow IAM role and credentials from environment variables - 56589 thanks Brian Stengaard 56590 - Allow low privilege users to use S3 (check if directory exists 56591 during Mkdir) - thanks Jakub Gedeon 56592 - Amazon Drive 56593 - Retry on more things to make directory listings more reliable 56594 56595 v1.27 - 2016-01-31 56596 56597 - New Features 56598 - Easier headless configuration with rclone authorize 56599 - Add support for multiple hash types - we now check SHA1 as well 56600 as MD5 hashes. 56601 - delete command which does obey the filters (unlike purge) 56602 - dedupe command to deduplicate a remote. Useful with Google 56603 Drive. 56604 - Add --ignore-existing flag to skip all files that exist on 56605 destination. 56606 - Add --delete-before, --delete-during, --delete-after flags. 56607 - Add --memprofile flag to debug memory use. 56608 - Warn the user about files with same name but different case 56609 - Make --include rules add their implicit exclude * at the end of 56610 the filter list 56611 - Deprecate compiling with go1.3 56612 - Amazon Drive 56613 - Fix download of files > 10 GB 56614 - Fix directory traversal ("Next token is expired") for large 56615 directory listings 56616 - Remove 409 conflict from error codes we will retry - stops very 56617 long pauses 56618 - Backblaze B2 56619 - SHA1 hashes now checked by rclone core 56620 - Drive 56621 - Add --drive-auth-owner-only to only consider files owned by the 56622 user - thanks Björn Harrtell 56623 - Export Google documents 56624 - Dropbox 56625 - Make file exclusion error controllable with -q 56626 - Swift 56627 - Fix upload from unprivileged user. 56628 - S3 56629 - Fix updating of mod times of files with + in. 56630 - Local 56631 - Add local file system option to disable UNC on Windows. 56632 56633 v1.26 - 2016-01-02 56634 56635 - New Features 56636 - Yandex storage backend - thank you Dmitry Burdeev ("dibu") 56637 - Implement Backblaze B2 storage backend 56638 - Add --min-age and --max-age flags - thank you Adriano Aurélio 56639 Meirelles 56640 - Make ls/lsl/md5sum/size/check obey includes and excludes 56641 - Fixes 56642 - Fix crash in http logging 56643 - Upload releases to github too 56644 - Swift 56645 - Fix sync for chunked files 56646 - OneDrive 56647 - Re-enable server-side copy 56648 - Don't mask HTTP error codes with JSON decode error 56649 - S3 56650 - Fix corrupting Content-Type on mod time update (thanks Joseph 56651 Spurrier) 56652 56653 v1.25 - 2015-11-14 56654 56655 - New features 56656 - Implement Hubic storage system 56657 - Fixes 56658 - Fix deletion of some excluded files without --delete-excluded 56659 - This could have deleted files unexpectedly on sync 56660 - Always check first with --dry-run! 56661 - Swift 56662 - Stop SetModTime losing metadata (e.g. X-Object-Manifest) 56663 - This could have caused data loss for files > 5GB in size 56664 - Use ContentType from Object to avoid lookups in listings 56665 - OneDrive 56666 - disable server-side copy as it seems to be broken at Microsoft 56667 56668 v1.24 - 2015-11-07 56669 56670 - New features 56671 - Add support for Microsoft OneDrive 56672 - Add --no-check-certificate option to disable server certificate 56673 verification 56674 - Add async readahead buffer for faster transfer of big files 56675 - Fixes 56676 - Allow spaces in remotes and check remote names for validity at 56677 creation time 56678 - Allow '&' and disallow ':' in Windows filenames. 56679 - Swift 56680 - Ignore directory marker objects where appropriate - allows 56681 working with Hubic 56682 - Don't delete the container if fs wasn't at root 56683 - S3 56684 - Don't delete the bucket if fs wasn't at root 56685 - Google Cloud Storage 56686 - Don't delete the bucket if fs wasn't at root 56687 56688 v1.23 - 2015-10-03 56689 56690 - New features 56691 - Implement rclone size for measuring remotes 56692 - Fixes 56693 - Fix headless config for drive and gcs 56694 - Tell the user they should try again if the webserver method 56695 failed 56696 - Improve output of --dump-headers 56697 - S3 56698 - Allow anonymous access to public buckets 56699 - Swift 56700 - Stop chunked operations logging "Failed to read info: Object Not 56701 Found" 56702 - Use Content-Length on uploads for extra reliability 56703 56704 v1.22 - 2015-09-28 56705 56706 - Implement rsync like include and exclude flags 56707 - swift 56708 - Support files > 5GB - thanks Sergey Tolmachev 56709 56710 v1.21 - 2015-09-22 56711 56712 - New features 56713 - Display individual transfer progress 56714 - Make lsl output times in localtime 56715 - Fixes 56716 - Fix allowing user to override credentials again in Drive, GCS 56717 and ACD 56718 - Amazon Drive 56719 - Implement compliant pacing scheme 56720 - Google Drive 56721 - Make directory reads concurrent for increased speed. 56722 56723 v1.20 - 2015-09-15 56724 56725 - New features 56726 - Amazon Drive support 56727 - Oauth support redone - fix many bugs and improve usability 56728 - Use "golang.org/x/oauth2" as oauth library of choice 56729 - Improve oauth usability for smoother initial signup 56730 - drive, googlecloudstorage: optionally use auto config for 56731 the oauth token 56732 - Implement --dump-headers and --dump-bodies debug flags 56733 - Show multiple matched commands if abbreviation too short 56734 - Implement server-side move where possible 56735 - local 56736 - Always use UNC paths internally on Windows - fixes a lot of bugs 56737 - dropbox 56738 - force use of our custom transport which makes timeouts work 56739 - Thanks to Klaus Post for lots of help with this release 56740 56741 v1.19 - 2015-08-28 56742 56743 - New features 56744 - Server side copies for s3/swift/drive/dropbox/gcs 56745 - Move command - uses server-side copies if it can 56746 - Implement --retries flag - tries 3 times by default 56747 - Build for plan9/amd64 and solaris/amd64 too 56748 - Fixes 56749 - Make a current version download with a fixed URL for scripting 56750 - Ignore rmdir in limited fs rather than throwing error 56751 - dropbox 56752 - Increase chunk size to improve upload speeds massively 56753 - Issue an error message when trying to upload bad file name 56754 56755 v1.18 - 2015-08-17 56756 56757 - drive 56758 - Add --drive-use-trash flag so rclone trashes instead of deletes 56759 - Add "Forbidden to download" message for files with no 56760 downloadURL 56761 - dropbox 56762 - Remove datastore 56763 - This was deprecated and it caused a lot of problems 56764 - Modification times and MD5SUMs no longer stored 56765 - Fix uploading files > 2GB 56766 - s3 56767 - use official AWS SDK from github.com/aws/aws-sdk-go 56768 - NB will most likely require you to delete and recreate remote 56769 - enable multipart upload which enables files > 5GB 56770 - tested with Ceph / RadosGW / S3 emulation 56771 - many thanks to Sam Liston and Brian Haymore at the Utah Center 56772 for High Performance Computing for a Ceph test account 56773 - misc 56774 - Show errors when reading the config file 56775 - Do not print stats in quiet mode - thanks Leonid Shalupov 56776 - Add FAQ 56777 - Fix created directories not obeying umask 56778 - Linux installation instructions - thanks Shimon Doodkin 56779 56780 v1.17 - 2015-06-14 56781 56782 - dropbox: fix case insensitivity issues - thanks Leonid Shalupov 56783 56784 v1.16 - 2015-06-09 56785 56786 - Fix uploading big files which was causing timeouts or panics 56787 - Don't check md5sum after download with --size-only 56788 56789 v1.15 - 2015-06-06 56790 56791 - Add --checksum flag to only discard transfers by MD5SUM - thanks 56792 Alex Couper 56793 - Implement --size-only flag to sync on size not checksum & modtime 56794 - Expand docs and remove duplicated information 56795 - Document rclone's limitations with directories 56796 - dropbox: update docs about case insensitivity 56797 56798 v1.14 - 2015-05-21 56799 56800 - local: fix encoding of non utf-8 file names - fixes a duplicate file 56801 problem 56802 - drive: docs about rate limiting 56803 - google cloud storage: Fix compile after API change in 56804 "google.golang.org/api/storage/v1" 56805 56806 v1.13 - 2015-05-10 56807 56808 - Revise documentation (especially sync) 56809 - Implement --timeout and --conntimeout 56810 - s3: ignore etags from multipart uploads which aren't md5sums 56811 56812 v1.12 - 2015-03-15 56813 56814 - drive: Use chunked upload for files above a certain size 56815 - drive: add --drive-chunk-size and --drive-upload-cutoff parameters 56816 - drive: switch to insert from update when a failed copy deletes the 56817 upload 56818 - core: Log duplicate files if they are detected 56819 56820 v1.11 - 2015-03-04 56821 56822 - swift: add region parameter 56823 - drive: fix crash on failed to update remote mtime 56824 - In remote paths, change native directory separators to / 56825 - Add synchronization to ls/lsl/lsd output to stop corruptions 56826 - Ensure all stats/log messages to go stderr 56827 - Add --log-file flag to log everything (including panics) to file 56828 - Make it possible to disable stats printing with --stats=0 56829 - Implement --bwlimit to limit data transfer bandwidth 56830 56831 v1.10 - 2015-02-12 56832 56833 - s3: list an unlimited number of items 56834 - Fix getting stuck in the configurator 56835 56836 v1.09 - 2015-02-07 56837 56838 - windows: Stop drive letters (e.g. C:) getting mixed up with remotes 56839 (e.g. drive:) 56840 - local: Fix directory separators on Windows 56841 - drive: fix rate limit exceeded errors 56842 56843 v1.08 - 2015-02-04 56844 56845 - drive: fix subdirectory listing to not list entire drive 56846 - drive: Fix SetModTime 56847 - dropbox: adapt code to recent library changes 56848 56849 v1.07 - 2014-12-23 56850 56851 - google cloud storage: fix memory leak 56852 56853 v1.06 - 2014-12-12 56854 56855 - Fix "Couldn't find home directory" on OSX 56856 - swift: Add tenant parameter 56857 - Use new location of Google API packages 56858 56859 v1.05 - 2014-08-09 56860 56861 - Improved tests and consequently lots of minor fixes 56862 - core: Fix race detected by go race detector 56863 - core: Fixes after running errcheck 56864 - drive: reset root directory on Rmdir and Purge 56865 - fs: Document that Purger returns error on empty directory, test and 56866 fix 56867 - google cloud storage: fix ListDir on subdirectory 56868 - google cloud storage: re-read metadata in SetModTime 56869 - s3: make reading metadata more reliable to work around eventual 56870 consistency problems 56871 - s3: strip trailing / from ListDir() 56872 - swift: return directories without / in ListDir 56873 56874 v1.04 - 2014-07-21 56875 56876 - google cloud storage: Fix crash on Update 56877 56878 v1.03 - 2014-07-20 56879 56880 - swift, s3, dropbox: fix updated files being marked as corrupted 56881 - Make compile with go 1.1 again 56882 56883 v1.02 - 2014-07-19 56884 56885 - Implement Dropbox remote 56886 - Implement Google Cloud Storage remote 56887 - Verify Md5sums and Sizes after copies 56888 - Remove times from "ls" command - lists sizes only 56889 - Add add "lsl" - lists times and sizes 56890 - Add "md5sum" command 56891 56892 v1.01 - 2014-07-04 56893 56894 - drive: fix transfer of big files using up lots of memory 56895 56896 v1.00 - 2014-07-03 56897 56898 - drive: fix whole second dates 56899 56900 v0.99 - 2014-06-26 56901 56902 - Fix --dry-run not working 56903 - Make compatible with go 1.1 56904 56905 v0.98 - 2014-05-30 56906 56907 - s3: Treat missing Content-Length as 0 for some ceph installations 56908 - rclonetest: add file with a space in 56909 56910 v0.97 - 2014-05-05 56911 56912 - Implement copying of single files 56913 - s3 & swift: support paths inside containers/buckets 56914 56915 v0.96 - 2014-04-24 56916 56917 - drive: Fix multiple files of same name being created 56918 - drive: Use o.Update and fs.Put to optimise transfers 56919 - Add version number, -V and --version 56920 56921 v0.95 - 2014-03-28 56922 56923 - rclone.org: website, docs and graphics 56924 - drive: fix path parsing 56925 56926 v0.94 - 2014-03-27 56927 56928 - Change remote format one last time 56929 - GNU style flags 56930 56931 v0.93 - 2014-03-16 56932 56933 - drive: store token in config file 56934 - cross compile other versions 56935 - set strict permissions on config file 56936 56937 v0.92 - 2014-03-15 56938 56939 - Config fixes and --config option 56940 56941 v0.91 - 2014-03-15 56942 56943 - Make config file 56944 56945 v0.90 - 2013-06-27 56946 56947 - Project named rclone 56948 56949 v0.00 - 2012-11-18 56950 56951 - Project started 56952 56953 Bugs and Limitations 56954 56955 Limitations 56956 56957 Directory timestamps aren't preserved on some backends 56958 56959 As of v1.66, rclone supports syncing directory modtimes, if the backend 56960 supports it. Some backends do not support it -- see overview for a 56961 complete list. Additionally, note that empty directories are not synced 56962 by default (this can be enabled with --create-empty-src-dirs.) 56963 56964 Rclone struggles with millions of files in a directory/bucket 56965 56966 Currently rclone loads each directory/bucket entirely into memory before 56967 using it. Since each rclone object takes 0.5k-1k of memory this can take 56968 a very long time and use a large amount of memory. 56969 56970 Millions of files in a directory tends to occur on bucket-based remotes 56971 (e.g. S3 buckets) since those remotes do not segregate subdirectories 56972 within the bucket. 56973 56974 Bucket-based remotes and folders 56975 56976 Bucket-based remotes (e.g. S3/GCS/Swift/B2) do not have a concept of 56977 directories. Rclone therefore cannot create directories in them which 56978 means that empty directories on a bucket-based remote will tend to 56979 disappear. 56980 56981 Some software creates empty keys ending in / as directory markers. 56982 Rclone doesn't do this as it potentially creates more objects and costs 56983 more. This ability may be added in the future (probably via a 56984 flag/option). 56985 56986 Bugs 56987 56988 Bugs are stored in rclone's GitHub project: 56989 56990 - Reported bugs 56991 - Known issues 56992 56993 Frequently Asked Questions 56994 56995 Do all cloud storage systems support all rclone commands 56996 56997 Yes they do. All the rclone commands (e.g. sync, copy, etc.) will work 56998 on all the remote storage systems. 56999 57000 Can I copy the config from one machine to another 57001 57002 Sure! Rclone stores all of its config in a single file. If you want to 57003 find this file, run rclone config file which will tell you where it is. 57004 57005 See the remote setup docs for more info. 57006 57007 How do I configure rclone on a remote / headless box with no browser? 57008 57009 This has now been documented in its own remote setup page. 57010 57011 Can rclone sync directly from drive to s3 57012 57013 Rclone can sync between two remote cloud storage systems just fine. 57014 57015 Note that it effectively downloads the file and uploads it again, so the 57016 node running rclone would need to have lots of bandwidth. 57017 57018 The syncs would be incremental (on a file by file basis). 57019 57020 e.g. 57021 57022 rclone sync --interactive drive:Folder s3:bucket 57023 57024 Using rclone from multiple locations at the same time 57025 57026 You can use rclone from multiple places at the same time if you choose 57027 different subdirectory for the output, e.g. 57028 57029 Server A> rclone sync --interactive /tmp/whatever remote:ServerA 57030 Server B> rclone sync --interactive /tmp/whatever remote:ServerB 57031 57032 If you sync to the same directory then you should use rclone copy 57033 otherwise the two instances of rclone may delete each other's files, 57034 e.g. 57035 57036 Server A> rclone copy /tmp/whatever remote:Backup 57037 Server B> rclone copy /tmp/whatever remote:Backup 57038 57039 The file names you upload from Server A and Server B should be different 57040 in this case, otherwise some file systems (e.g. Drive) may make 57041 duplicates. 57042 57043 Why doesn't rclone support partial transfers / binary diffs like rsync? 57044 57045 Rclone stores each file you transfer as a native object on the remote 57046 cloud storage system. This means that you can see the files you upload 57047 as expected using alternative access methods (e.g. using the Google 57048 Drive web interface). There is a 1:1 mapping between files on your hard 57049 disk and objects created in the cloud storage system. 57050 57051 Cloud storage systems (at least none I've come across yet) don't support 57052 partially uploading an object. You can't take an existing object, and 57053 change some bytes in the middle of it. 57054 57055 It would be possible to make a sync system which stored binary diffs 57056 instead of whole objects like rclone does, but that would break the 1:1 57057 mapping of files on your hard disk to objects in the remote cloud 57058 storage system. 57059 57060 All the cloud storage systems support partial downloads of content, so 57061 it would be possible to make partial downloads work. However to make 57062 this work efficiently this would require storing a significant amount of 57063 metadata, which breaks the desired 1:1 mapping of files to objects. 57064 57065 Can rclone do bi-directional sync? 57066 57067 Yes, since rclone v1.58.0, bidirectional cloud sync is available. 57068 57069 Can I use rclone with an HTTP proxy? 57070 57071 Yes. rclone will follow the standard environment variables for proxies, 57072 similar to cURL and other programs. 57073 57074 In general the variables are called http_proxy (for services reached 57075 over http) and https_proxy (for services reached over https). Most 57076 public services will be using https, but you may wish to set both. 57077 57078 The content of the variable is protocol://server:port. The protocol 57079 value is the one used to talk to the proxy server, itself, and is 57080 commonly either http or socks5. 57081 57082 Slightly annoyingly, there is no standard for the name; some 57083 applications may use http_proxy but another one HTTP_PROXY. The Go 57084 libraries used by rclone will try both variations, but you may wish to 57085 set all possibilities. So, on Linux, you may end up with code similar to 57086 57087 export http_proxy=http://proxyserver:12345 57088 export https_proxy=$http_proxy 57089 export HTTP_PROXY=$http_proxy 57090 export HTTPS_PROXY=$http_proxy 57091 57092 Note: If the proxy server requires a username and password, then use 57093 57094 export http_proxy=http://username:password@proxyserver:12345 57095 export https_proxy=$http_proxy 57096 export HTTP_PROXY=$http_proxy 57097 export HTTPS_PROXY=$http_proxy 57098 57099 The NO_PROXY allows you to disable the proxy for specific hosts. Hosts 57100 must be comma separated, and can contain domains or parts. For instance 57101 "foo.com" also matches "bar.foo.com". 57102 57103 e.g. 57104 57105 export no_proxy=localhost,127.0.0.0/8,my.host.name 57106 export NO_PROXY=$no_proxy 57107 57108 Note that the FTP backend does not support ftp_proxy yet. 57109 57110 Rclone gives x509: failed to load system roots and no roots provided error 57111 57112 This means that rclone can't find the SSL root certificates. Likely you 57113 are running rclone on a NAS with a cut-down Linux OS, or possibly on 57114 Solaris. 57115 57116 Rclone (via the Go runtime) tries to load the root certificates from 57117 these places on Linux. 57118 57119 "/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc. 57120 "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 57121 "/etc/ssl/ca-bundle.pem", // OpenSUSE 57122 "/etc/pki/tls/cacert.pem", // OpenELEC 57123 57124 So doing something like this should fix the problem. It also sets the 57125 time which is important for SSL to work properly. 57126 57127 mkdir -p /etc/ssl/certs/ 57128 curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 57129 ntpclient -s -h pool.ntp.org 57130 57131 The two environment variables SSL_CERT_FILE and SSL_CERT_DIR, mentioned 57132 in the x509 package, provide an additional way to provide the SSL root 57133 certificates. 57134 57135 Note that you may need to add the --insecure option to the curl command 57136 line if it doesn't work without. 57137 57138 curl --insecure -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt 57139 57140 Rclone gives Failed to load config file: function not implemented error 57141 57142 Likely this means that you are running rclone on Linux version not 57143 supported by the go runtime, ie earlier than version 2.6.23. 57144 57145 See the system requirements section in the go install docs for full 57146 details. 57147 57148 All my uploaded docx/xlsx/pptx files appear as archive/zip 57149 57150 This is caused by uploading these files from a Windows computer which 57151 hasn't got the Microsoft Office suite installed. The easiest way to fix 57152 is to install the Word viewer and the Microsoft Office Compatibility 57153 Pack for Word, Excel, and PowerPoint 2007 and later versions' file 57154 formats 57155 57156 tcp lookup some.domain.com no such host 57157 57158 This happens when rclone cannot resolve a domain. Please check that your 57159 DNS setup is generally working, e.g. 57160 57161 # both should print a long list of possible IP addresses 57162 dig www.googleapis.com # resolve using your default DNS 57163 dig www.googleapis.com @8.8.8.8 # resolve with Google's DNS server 57164 57165 If you are using systemd-resolved (default on Arch Linux), ensure it is 57166 at version 233 or higher. Previous releases contain a bug which causes 57167 not all domains to be resolved properly. 57168 57169 The Go resolver decision can be influenced with the GODEBUG=netdns=... 57170 environment variable. This also allows to resolve certain issues with 57171 DNS resolution. On Windows or MacOS systems, try forcing use of the 57172 internal Go resolver by setting GODEBUG=netdns=go at runtime. On other 57173 systems (Linux, *BSD, etc) try forcing use of the system name resolver 57174 by setting GODEBUG=netdns=cgo (and recompile rclone from source with CGO 57175 enabled if necessary). See the name resolution section in the go docs. 57176 57177 Failed to start auth webserver on Windows 57178 57179 Error: config failed to refresh token: failed to start auth webserver: listen tcp 127.0.0.1:53682: bind: An attempt was made to access a socket in a way forbidden by its access permissions. 57180 ... 57181 yyyy/mm/dd hh:mm:ss Fatal error: config failed to refresh token: failed to start auth webserver: listen tcp 127.0.0.1:53682: bind: An attempt was made to access a socket in a way forbidden by its access permissions. 57182 57183 This is sometimes caused by the Host Network Service causing issues with 57184 opening the port on the host. 57185 57186 A simple solution may be restarting the Host Network Service with eg. 57187 Powershell 57188 57189 Restart-Service hns 57190 57191 The total size reported in the stats for a sync is wrong and keeps changing 57192 57193 It is likely you have more than 10,000 files that need to be synced. By 57194 default, rclone only gets 10,000 files ahead in a sync so as not to use 57195 up too much memory. You can change this default with the --max-backlog 57196 flag. 57197 57198 Rclone is using too much memory or appears to have a memory leak 57199 57200 Rclone is written in Go which uses a garbage collector. The default 57201 settings for the garbage collector mean that it runs when the heap size 57202 has doubled. 57203 57204 However it is possible to tune the garbage collector to use less memory 57205 by setting GOGC to a lower value, say export GOGC=20. This will make the 57206 garbage collector work harder, reducing memory size at the expense of 57207 CPU usage. 57208 57209 The most common cause of rclone using lots of memory is a single 57210 directory with thousands or millions of files in. Rclone has to load 57211 this entirely into memory as rclone objects. Each rclone object takes 57212 0.5k-1k of memory. 57213 57214 Rclone changes fullwidth Unicode punctuation marks in file names 57215 57216 For example: On a Windows system, you have a file with name Test:1.jpg, 57217 where : is the Unicode fullwidth colon symbol. When using rclone to 57218 copy this to your Google Drive, you will notice that the file gets 57219 renamed to Test:1.jpg, where : is the regular (halfwidth) colon. 57220 57221 The reason for such renames is the way rclone handles different 57222 restricted filenames on different cloud storage systems. It tries to 57223 avoid ambiguous file names as much and allow moving files between many 57224 cloud storage systems transparently, by replacing invalid characters 57225 with similar looking Unicode characters when transferring to one storage 57226 system, and replacing back again when transferring to a different 57227 storage system where the original characters are supported. When the 57228 same Unicode characters are intentionally used in file names, this 57229 replacement strategy leads to unwanted renames. Read more here. 57230 57231 License 57232 57233 This is free software under the terms of the MIT license (check the 57234 COPYING file included with the source code). 57235 57236 Copyright (C) 2019 by Nick Craig-Wood https://www.craig-wood.com/nick/ 57237 57238 Permission is hereby granted, free of charge, to any person obtaining a copy 57239 of this software and associated documentation files (the "Software"), to deal 57240 in the Software without restriction, including without limitation the rights 57241 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 57242 copies of the Software, and to permit persons to whom the Software is 57243 furnished to do so, subject to the following conditions: 57244 57245 The above copyright notice and this permission notice shall be included in 57246 all copies or substantial portions of the Software. 57247 57248 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 57249 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 57250 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 57251 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 57252 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 57253 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 57254 THE SOFTWARE. 57255 57256 Authors and contributors 57257 57258 Authors 57259 57260 - Nick Craig-Wood nick@craig-wood.com 57261 57262 Contributors 57263 57264 {{< rem 57265 email addresses removed from here need to be added to bin/.ignore-emails to make sure update-authors.py doesn't immediately put them back in again. 57266 >}} 57267 57268 - Alex Couper amcouper@gmail.com 57269 - Leonid Shalupov leonid@shalupov.com shalupov@diverse.org.ru 57270 - Shimon Doodkin helpmepro1@gmail.com 57271 - Colin Nicholson colin@colinn.com 57272 - Klaus Post klauspost@gmail.com 57273 - Sergey Tolmachev tolsi.ru@gmail.com 57274 - Adriano Aurélio Meirelles adriano@atinge.com 57275 - C. Bess cbess@users.noreply.github.com 57276 - Dmitry Burdeev dibu28@gmail.com 57277 - Joseph Spurrier github@josephspurrier.com 57278 - Björn Harrtell bjorn@wololo.org 57279 - Xavier Lucas xavier.lucas@corp.ovh.com 57280 - Werner Beroux werner@beroux.com 57281 - Brian Stengaard brian@stengaard.eu 57282 - Jakub Gedeon jgedeon@sofi.com 57283 - Jim Tittsler jwt@onjapan.net 57284 - Michal Witkowski michal@improbable.io 57285 - Fabian Ruff fabian.ruff@sap.com 57286 - Leigh Klotz klotz@quixey.com 57287 - Romain Lapray lapray.romain@gmail.com 57288 - Justin R. Wilson jrw972@gmail.com 57289 - Antonio Messina antonio.s.messina@gmail.com 57290 - Stefan G. Weichinger office@oops.co.at 57291 - Per Cederberg cederberg@gmail.com 57292 - Radek Šenfeld rush@logic.cz 57293 - Fredrik Fornwall fredrik@fornwall.net 57294 - Asko Tamm asko@deekit.net 57295 - xor-zz xor@gstocco.com 57296 - Tomasz Mazur tmazur90@gmail.com 57297 - Marco Paganini paganini@paganini.net 57298 - Felix Bünemann buenemann@louis.info 57299 - Durval Menezes jmrclone@durval.com 57300 - Luiz Carlos Rumbelsperger Viana maxd13_luiz_carlos@hotmail.com 57301 - Stefan Breunig stefan-github@yrden.de 57302 - Alishan Ladhani ali-l@users.noreply.github.com 57303 - 0xJAKE 0xJAKE@users.noreply.github.com 57304 - Thibault Molleman thibaultmol@users.noreply.github.com 57305 - Scott McGillivray scott.mcgillivray@gmail.com 57306 - Bjørn Erik Pedersen bjorn.erik.pedersen@gmail.com 57307 - Lukas Loesche lukas@mesosphere.io 57308 - emyarod emyarod@users.noreply.github.com 57309 - T.C. Ferguson tcf909@gmail.com 57310 - Brandur brandur@mutelight.org 57311 - Dario Giovannetti dev@dariogiovannetti.net 57312 - Károly Oláh okaresz@aol.com 57313 - Jon Yergatian jon@macfanatic.ca 57314 - Jack Schmidt github@mowsey.org 57315 - Dedsec1 Dedsec1@users.noreply.github.com 57316 - Hisham Zarka hzarka@gmail.com 57317 - Jérôme Vizcaino jerome.vizcaino@gmail.com 57318 - Mike Tesch mjt6129@rit.edu 57319 - Marvin Watson marvwatson@users.noreply.github.com 57320 - Danny Tsai danny8376@gmail.com 57321 - Yoni Jah yonjah+git@gmail.com yonjah+github@gmail.com 57322 - Stephen Harris github@spuddy.org sweharris@users.noreply.github.com 57323 - Ihor Dvoretskyi ihor.dvoretskyi@gmail.com 57324 - Jon Craton jncraton@gmail.com 57325 - Hraban Luyat hraban@0brg.net 57326 - Michael Ledin mledin89@gmail.com 57327 - Martin Kristensen me@azgul.com 57328 - Too Much IO toomuchio@users.noreply.github.com 57329 - Anisse Astier anisse@astier.eu 57330 - Zahiar Ahmed zahiar@live.com 57331 - Igor Kharin igorkharin@gmail.com 57332 - Bill Zissimopoulos billziss@navimatics.com 57333 - Bob Potter bobby.potter@gmail.com 57334 - Steven Lu tacticalazn@gmail.com 57335 - Sjur Fredriksen sjurtf@ifi.uio.no 57336 - Ruwbin hubus12345@gmail.com 57337 - Fabian Möller fabianm88@gmail.com f.moeller@nynex.de 57338 - Edward Q. Bridges github@eqbridges.com 57339 - Vasiliy Tolstov v.tolstov@selfip.ru 57340 - Harshavardhana harsha@minio.io 57341 - sainaen sainaen@gmail.com 57342 - gdm85 gdm85@users.noreply.github.com 57343 - Yaroslav Halchenko debian@onerussian.com 57344 - John Papandriopoulos jpap@users.noreply.github.com 57345 - Zhiming Wang zmwangx@gmail.com 57346 - Andy Pilate cubox@cubox.me 57347 - Oliver Heyme olihey@googlemail.com olihey@users.noreply.github.com 57348 de8olihe@lego.com 57349 - wuyu wuyu@yunify.com 57350 - Andrei Dragomir adragomi@adobe.com 57351 - Christian Brüggemann mail@cbruegg.com 57352 - Alex McGrath Kraak amkdude@gmail.com 57353 - bpicode bjoern.pirnay@googlemail.com 57354 - Daniel Jagszent daniel@jagszent.de 57355 - Josiah White thegenius2009@gmail.com 57356 - Ishuah Kariuki kariuki@ishuah.com ishuah91@gmail.com 57357 - Jan Varho jan@varho.org 57358 - Girish Ramakrishnan girish@cloudron.io 57359 - LingMan LingMan@users.noreply.github.com 57360 - Jacob McNamee jacobmcnamee@gmail.com 57361 - jersou jertux@gmail.com 57362 - thierry thierry@substantiel.fr 57363 - Simon Leinen simon.leinen@gmail.com ubuntu@s3-test.novalocal 57364 - Dan Dascalescu ddascalescu+github@gmail.com 57365 - Jason Rose jason@jro.io 57366 - Andrew Starr-Bochicchio a.starr.b@gmail.com 57367 - John Leach john@johnleach.co.uk 57368 - Corban Raun craun@instructure.com 57369 - Pierre Carlson mpcarl@us.ibm.com 57370 - Ernest Borowski er.borowski@gmail.com 57371 - Remus Bunduc remus.bunduc@gmail.com 57372 - Iakov Davydov iakov.davydov@unil.ch dav05.gith@myths.ru 57373 - Jakub Tasiemski tasiemski@gmail.com 57374 - David Minor dminor@saymedia.com 57375 - Tim Cooijmans cooijmans.tim@gmail.com 57376 - Laurence liuxy6@gmail.com 57377 - Giovanni Pizzi gio.piz@gmail.com 57378 - Filip Bartodziej filipbartodziej@gmail.com 57379 - Jon Fautley jon@dead.li 57380 - lewapm 32110057+lewapm@users.noreply.github.com 57381 - Yassine Imounachen yassine256@gmail.com 57382 - Chris Redekop chris-redekop@users.noreply.github.com 57383 chris.redekop@gmail.com 57384 - Jon Fautley jon@adenoid.appstal.co.uk 57385 - Will Gunn WillGunn@users.noreply.github.com 57386 - Lucas Bremgartner lucas@bremis.ch 57387 - Jody Frankowski jody.frankowski@gmail.com 57388 - Andreas Roussos arouss1980@gmail.com 57389 - nbuchanan nbuchanan@utah.gov 57390 - Durval Menezes rclone@durval.com 57391 - Victor vb-github@viblo.se 57392 - Mateusz pabian.mateusz@gmail.com 57393 - Daniel Loader spicypixel@gmail.com 57394 - David0rk davidork@gmail.com 57395 - Alexander Neumann alexander@bumpern.de 57396 - Giri Badanahatti gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local 57397 - Leo R. Lundgren leo@finalresort.org 57398 - wolfv wolfv6@users.noreply.github.com 57399 - Dave Pedu dave@davepedu.com 57400 - Stefan Lindblom lindblom@spotify.com 57401 - seuffert oliver@seuffert.biz 57402 - gbadanahatti 37121690+gbadanahatti@users.noreply.github.com 57403 - Keith Goldfarb barkofdelight@gmail.com 57404 - Steve Kriss steve@heptio.com 57405 - Chih-Hsuan Yen yan12125@gmail.com 57406 - Alexander Neumann fd0@users.noreply.github.com 57407 - Matt Holt mholt@users.noreply.github.com 57408 - Eri Bastos bastos.eri@gmail.com 57409 - Michael P. Dubner pywebmail@list.ru 57410 - Antoine GIRARD sapk@users.noreply.github.com 57411 - Mateusz Piotrowski mpp302@gmail.com 57412 - Animosity022 animosity22@users.noreply.github.com 57413 earl.texter@gmail.com 57414 - Peter Baumgartner pete@lincolnloop.com 57415 - Craig Rachel craig@craigrachel.com 57416 - Michael G. Noll miguno@users.noreply.github.com 57417 - hensur me@hensur.de 57418 - Oliver Heyme de8olihe@lego.com 57419 - Richard Yang richard@yenforyang.com 57420 - Piotr Oleszczyk piotr.oleszczyk@gmail.com 57421 - Rodrigo rodarima@gmail.com 57422 - NoLooseEnds NoLooseEnds@users.noreply.github.com 57423 - Jakub Karlicek jakub@karlicek.me 57424 - John Clayton john@codemonkeylabs.com 57425 - Kasper Byrdal Nielsen byrdal76@gmail.com 57426 - Benjamin Joseph Dag bjdag1234@users.noreply.github.com 57427 - themylogin themylogin@gmail.com 57428 - Onno Zweers onno.zweers@surfsara.nl 57429 - Jasper Lievisse Adriaanse jasper@humppa.nl 57430 - sandeepkru sandeep.ummadi@gmail.com 57431 sandeepkru@users.noreply.github.com 57432 - HerrH atomtigerzoo@users.noreply.github.com 57433 - Andrew 4030760+sparkyman215@users.noreply.github.com 57434 - dan smith XX1011@gmail.com 57435 - Oleg Kovalov iamolegkovalov@gmail.com 57436 - Ruben Vandamme github-com-00ff86@vandamme.email 57437 - Cnly minecnly@gmail.com 57438 - Andres Alvarez 1671935+kir4h@users.noreply.github.com 57439 - reddi1 xreddi@gmail.com 57440 - Matt Tucker matthewtckr@gmail.com 57441 - Sebastian Bünger buengese@gmail.com buengese@protonmail.com 57442 - Martin Polden mpolden@mpolden.no 57443 - Alex Chen Cnly@users.noreply.github.com 57444 - Denis deniskovpen@gmail.com 57445 - bsteiss 35940619+bsteiss@users.noreply.github.com 57446 - Cédric Connes cedric.connes@gmail.com 57447 - Dr. Tobias Quathamer toddy15@users.noreply.github.com 57448 - dcpu 42736967+dcpu@users.noreply.github.com 57449 - Sheldon Rupp me@shel.io 57450 - albertony 12441419+albertony@users.noreply.github.com 57451 - cron410 cron410@gmail.com 57452 - Anagh Kumar Baranwal 6824881+darthShadow@users.noreply.github.com 57453 - Felix Brucker felix@felixbrucker.com 57454 - Santiago Rodríguez scollazo@users.noreply.github.com 57455 - Craig Miskell craig.miskell@fluxfederation.com 57456 - Antoine GIRARD sapk@sapk.fr 57457 - Joanna Marek joanna.marek@u2i.com 57458 - frenos frenos@users.noreply.github.com 57459 - ssaqua ssaqua@users.noreply.github.com 57460 - xnaas me@xnaas.info 57461 - Frantisek Fuka fuka@fuxoft.cz 57462 - Paul Kohout pauljkohout@yahoo.com 57463 - dcpu 43330287+dcpu@users.noreply.github.com 57464 - jackyzy823 jackyzy823@gmail.com 57465 - David Haguenauer ml@kurokatta.org 57466 - teresy hi.teresy@gmail.com 57467 - buergi patbuergi@gmx.de 57468 - Florian Gamboeck mail@floga.de 57469 - Ralf Hemberger 10364191+rhemberger@users.noreply.github.com 57470 - Scott Edlund sedlund@users.noreply.github.com 57471 - Erik Swanson erik@retailnext.net 57472 - Jake Coggiano jake@stripe.com 57473 - brused27 brused27@noemailaddress 57474 - Peter Kaminski kaminski@istori.com 57475 - Henry Ptasinski henry@logout.com 57476 - Alexander kharkovalexander@gmail.com 57477 - Garry McNulty garrmcnu@gmail.com 57478 - Mathieu Carbou mathieu.carbou@gmail.com 57479 - Mark Otway mark@otway.com 57480 - William Cocker 37018962+WilliamCocker@users.noreply.github.com 57481 - François Leurent 131.js@cloudyks.org 57482 - Arkadius Stefanski arkste@gmail.com 57483 - Jay dev@jaygoel.com 57484 - andrea rota a@xelera.eu 57485 - nicolov nicolov@users.noreply.github.com 57486 - Matt Joiner anacrolix@gmail.com 57487 - Dario Guzik dario@guzik.com.ar 57488 - qip qip@users.noreply.github.com 57489 - yair@unicorn yair@unicorn 57490 - Matt Robinson brimstone@the.narro.ws 57491 - kayrus kay.diam@gmail.com 57492 - Rémy Léone remy.leone@gmail.com 57493 - Wojciech Smigielski wojciech.hieronim.smigielski@gmail.com 57494 - weetmuts oehrstroem@gmail.com 57495 - Jonathan vanillajonathan@users.noreply.github.com 57496 - James Carpenter orbsmiv@users.noreply.github.com 57497 - Vince vince0villamora@gmail.com 57498 - Nestar47 47841759+Nestar47@users.noreply.github.com 57499 - Six brbsix@gmail.com 57500 - Alexandru Bumbacea alexandru.bumbacea@booking.com 57501 - calisro robert.calistri@gmail.com 57502 - Dr.Rx david.rey@nventive.com 57503 - marcintustin marcintustin@users.noreply.github.com 57504 - jaKa Močnik jaka@koofr.net 57505 - Fionera fionera@fionera.de 57506 - Dan Walters dan@walters.io 57507 - Danil Semelenov sgtpep@users.noreply.github.com 57508 - xopez 28950736+xopez@users.noreply.github.com 57509 - Ben Boeckel mathstuf@gmail.com 57510 - Manu manu@snapdragon.cc 57511 - Kyle E. Mitchell kyle@kemitchell.com 57512 - Gary Kim gary@garykim.dev 57513 - Jon jonathn@github.com 57514 - Jeff Quinn jeffrey.quinn@bluevoyant.com 57515 - Peter Berbec peter@berbec.com 57516 - didil 1284255+didil@users.noreply.github.com 57517 - id01 gaviniboom@gmail.com 57518 - Robert Marko robimarko@gmail.com 57519 - Philip Harvey 32467456+pharveybattelle@users.noreply.github.com 57520 - JorisE JorisE@users.noreply.github.com 57521 - garry415 garry.415@gmail.com 57522 - forgems forgems@gmail.com 57523 - Florian Apolloner florian@apolloner.eu 57524 - Aleksandar Janković office@ajankovic.com 57525 ajankovic@users.noreply.github.com 57526 - Maran maran@protonmail.com 57527 - nguyenhuuluan434 nguyenhuuluan434@gmail.com 57528 - Laura Hausmann zotan@zotan.pw laura@hausmann.dev 57529 - yparitcher y@paritcher.com 57530 - AbelThar abela.tharen@gmail.com 57531 - Matti Niemenmaa matti.niemenmaa+git@iki.fi 57532 - Russell Davis russelldavis@users.noreply.github.com 57533 - Yi FU yi.fu@tink.se 57534 - Paul Millar paul.millar@desy.de 57535 - justinalin justinalin@qnap.com 57536 - EliEron subanimehd@gmail.com 57537 - justina777 chiahuei.lin@gmail.com 57538 - Chaitanya Bankanhal bchaitanya15@gmail.com 57539 - Michał Matczuk michal@scylladb.com 57540 - Macavirus macavirus@zoho.com 57541 - Abhinav Sharma abhi18av@outlook.com 57542 - ginvine 34869051+ginvine@users.noreply.github.com 57543 - Patrick Wang mail6543210@yahoo.com.tw 57544 - Cenk Alti cenkalti@gmail.com 57545 - Andreas Chlupka andy@chlupka.com 57546 - Alfonso Montero amontero@tinet.org 57547 - Ivan Andreev ivandeex@gmail.com 57548 - David Baumgold david@davidbaumgold.com 57549 - Lars Lehtonen lars.lehtonen@gmail.com 57550 - Matei David matei.david@gmail.com 57551 - David david.bramwell@endemolshine.com 57552 - Anthony Rusdi 33247310+antrusd@users.noreply.github.com 57553 - Richard Patel me@terorie.dev 57554 - 庄天翼 zty0826@gmail.com 57555 - SwitchJS dev@switchjs.com 57556 - Raphael PowershellNinja@users.noreply.github.com 57557 - Sezal Agrawal sezalagrawal@gmail.com 57558 - Tyler TylerNakamura@users.noreply.github.com 57559 - Brett Dutro brett.dutro@gmail.com 57560 - Vighnesh SK booterror99@gmail.com 57561 - Arijit Biswas dibbyo456@gmail.com 57562 - Michele Caci michele.caci@gmail.com 57563 - AlexandrBoltris ua2fgb@gmail.com 57564 - Bryce Larson blarson@saltstack.com 57565 - Carlos Ferreyra crypticmind@gmail.com 57566 - Saksham Khanna sakshamkhanna@outlook.com 57567 - dausruddin 5763466+dausruddin@users.noreply.github.com 57568 - zero-24 zero-24@users.noreply.github.com 57569 - Xiaoxing Ye ye@xiaoxing.us 57570 - Barry Muldrey barry@muldrey.net 57571 - Sebastian Brandt sebastian.brandt@friday.de 57572 - Marco Molteni marco.molteni@mailbox.org 57573 - Ankur Gupta 7876747+ankur0493@users.noreply.github.com 57574 - Maciej Zimnoch maciej@scylladb.com 57575 - anuar45 serdaliyev.anuar@gmail.com 57576 - Fernando ferferga@users.noreply.github.com 57577 - David Cole david.cole@sohonet.com 57578 - Wei He git@weispot.com 57579 - Outvi V 19144373+outloudvi@users.noreply.github.com 57580 - Thomas Kriechbaumer thomas@kriechbaumer.name 57581 - Tennix tennix@users.noreply.github.com 57582 - Ole Schütt ole@schuett.name 57583 - Kuang-che Wu kcwu@csie.org 57584 - Thomas Eales wingsuit@users.noreply.github.com 57585 - Paul Tinsley paul.tinsley@vitalsource.com 57586 - Felix Hungenberg git@shiftgeist.com 57587 - Benjamin Richter github@dev.telepath.de 57588 - landall cst_zf@qq.com 57589 - thestigma thestigma@gmail.com 57590 - jtagcat 38327267+jtagcat@users.noreply.github.com 57591 - Damon Permezel permezel@me.com 57592 - boosh boosh@users.noreply.github.com 57593 - unbelauscht 58393353+unbelauscht@users.noreply.github.com 57594 - Motonori IWAMURO vmi@nifty.com 57595 - Benjapol Worakan benwrk@live.com 57596 - Dave Koston dave.koston@stackpath.com 57597 - Durval Menezes DurvalMenezes@users.noreply.github.com 57598 - Tim Gallant me@timgallant.us 57599 - Frederick Zhang frederick888@tsundere.moe 57600 - valery1707 valery1707@gmail.com 57601 - Yves G theYinYeti@yalis.fr 57602 - Shing Kit Chan chanshingkit@gmail.com 57603 - Franklyn Tackitt franklyn@tackitt.net 57604 - Robert-André Mauchin zebob.m@gmail.com 57605 - evileye 48332831+ibiruai@users.noreply.github.com 57606 - Joachim Brandon LeBlanc brandon@leblanc.codes 57607 - Patryk Jakuszew patryk.jakuszew@gmail.com 57608 - fishbullet shindu666@gmail.com 57609 - greatroar <@> 57610 - Bernd Schoolmann mail@quexten.com 57611 - Elan Ruusamäe glen@pld-linux.org 57612 - Max Sum max@lolyculture.com 57613 - Mark Spieth mspieth@users.noreply.github.com 57614 - harry me@harry.plus 57615 - Samantha McVey samantham@posteo.net 57616 - Jack Anderson jack.anderson@metaswitch.com 57617 - Michael G draget@speciesm.net 57618 - Brandon Philips brandon@ifup.org 57619 - Daven dooven@users.noreply.github.com 57620 - Martin Stone martin@d7415.co.uk 57621 - David Bramwell 13053834+dbramwell@users.noreply.github.com 57622 - Sunil Patra snl_su@live.com 57623 - Adam Stroud adam.stroud@gmail.com 57624 - Kush kushsharma@users.noreply.github.com 57625 - Matan Rosenberg matan129@gmail.com 57626 - gitch1 63495046+gitch1@users.noreply.github.com 57627 - ElonH elonhhuang@gmail.com 57628 - Fred fred@creativeprojects.tech 57629 - Sébastien Gross renard@users.noreply.github.com 57630 - Maxime Suret 11944422+msuret@users.noreply.github.com 57631 - Caleb Case caleb@storj.io calebcase@gmail.com 57632 - Ben Zenker imbenzenker@gmail.com 57633 - Martin Michlmayr tbm@cyrius.com 57634 - Brandon McNama bmcnama@pagerduty.com 57635 - Daniel Slyman github@skylayer.eu 57636 - Alex Guerrero guerrero@users.noreply.github.com 57637 - Matteo Pietro Dazzi matteopietro.dazzi@gft.com 57638 - edwardxml 56691903+edwardxml@users.noreply.github.com 57639 - Roman Kredentser shareed2k@gmail.com 57640 - Kamil Trzciński ayufan@ayufan.eu 57641 - Zac Rubin z-0@users.noreply.github.com 57642 - Vincent Feltz psycho@feltzv.fr 57643 - Heiko Bornholdt bornholdt@informatik.uni-hamburg.de 57644 - Matteo Pietro Dazzi matteopietro.dazzi@gmail.com 57645 - jtagcat gitlab@c7.ee 57646 - Petri Salminen petri@salminen.dev 57647 - Tim Burke tim.burke@gmail.com 57648 - Kai Lüke kai@kinvolk.io 57649 - Garrett Squire github@garrettsquire.com 57650 - Evan Harris eharris@puremagic.com 57651 - Kevin keyam@microsoft.com 57652 - Morten Linderud morten@linderud.pw 57653 - Dmitry Ustalov dmitry.ustalov@gmail.com 57654 - Jack 196648+jdeng@users.noreply.github.com 57655 - kcris cristian.tarsoaga@gmail.com 57656 - tyhuber1 68970760+tyhuber1@users.noreply.github.com 57657 - David Ibarra david.ibarra@realty.com 57658 - Tim Gallant tim@lilt.com 57659 - Kaloyan Raev kaloyan@storj.io 57660 - Jay McEntire jay.mcentire@gmail.com 57661 - Leo Luan leoluan@us.ibm.com 57662 - aus 549081+aus@users.noreply.github.com 57663 - Aaron Gokaslan agokaslan@fb.com 57664 - Egor Margineanu egmar@users.noreply.github.com 57665 - Lucas Kanashiro lucas.kanashiro@canonical.com 57666 - WarpedPixel WarpedPixel@users.noreply.github.com 57667 - Sam Edwards sam@samedwards.ca 57668 - wjielai gouki0123@gmail.com 57669 - Muffin King jinxz_k@live.com 57670 - Christopher Stewart 6573710+1f47a@users.noreply.github.com 57671 - Russell Cattelan cattelan@digitalelves.com 57672 - gyutw 30371241+gyutw@users.noreply.github.com 57673 - Hekmon edouardhur@gmail.com 57674 - LaSombra lasombra@users.noreply.github.com 57675 - Dov Murik dov.murik@gmail.com 57676 - Ameer Dawood ameer1234567890@gmail.com 57677 - Dan Hipschman dan.hipschman@opendoor.com 57678 - Josh Soref jsoref@users.noreply.github.com 57679 - David david@staron.nl 57680 - Ingo ingo@hoffmann.cx 57681 - Adam Plánský adamplansky@users.noreply.github.com 57682 adamplansky@gmail.com 57683 - Manish Gupta manishgupta.ait@gmail.com 57684 - Deepak Sah sah.sslpu@gmail.com 57685 - Marcin Zelent marcin@zelent.net 57686 - zhucan zhucan.k8s@gmail.com 57687 - James Lim james.lim@samsara.com 57688 - Laurens Janssen BD69BM@insim.biz 57689 - Bob Bagwill bobbagwill@gmail.com 57690 - Nathan Collins colli372@msu.edu 57691 - lostheli 57692 - kelv kelvin@acks.org 57693 - Milly milly.ca@gmail.com 57694 - gtorelly gtorelly@gmail.com 57695 - Brad Ackerman brad@facefault.org 57696 - Mitsuo Heijo mitsuo.heijo@gmail.com 57697 - Claudio Bantaloukas rockdreamer@gmail.com 57698 - Benjamin Gustin gustin.ben@gmail.com 57699 - Ingo Weiss ingo@redhat.com 57700 - Kerry Su me@sshockwave.net 57701 - Ilyess Bachiri ilyess.bachiri@sonder.com 57702 - Yury Stankevich urykhy@gmail.com 57703 - kice wslikerqs@gmail.com 57704 - Denis Neuling denisneuling@gmail.com 57705 - Janne Johansson icepic.dz@gmail.com 57706 - Patrik Nordlén patriki@gmail.com 57707 - CokeMine aptx4561@gmail.com 57708 - Sơn Trần-Nguyễn github@sntran.com 57709 - lluuaapp 266615+lluuaapp@users.noreply.github.com 57710 - Zach Kipp kipp.zach@gmail.com 57711 - Riccardo Iaconelli riccardo@kde.org 57712 - Sakuragawa Misty gyc990326@gmail.com 57713 - Nicolas Rueff nicolas@rueff.fr 57714 - Pau Rodriguez-Estivill prodrigestivill@gmail.com 57715 - Bob Pusateri BobPusateri@users.noreply.github.com 57716 - Alex JOST 25005220+dimejo@users.noreply.github.com 57717 - Alexey Tabakman samosad.ru@gmail.com 57718 - David Sze sze.david@gmail.com 57719 - cynthia kwok cynthia.m.kwok@gmail.com 57720 - Miron Veryanskiy MironVeryanskiy@gmail.com 57721 - K265 k.265@qq.com 57722 - Vesnyx Vesnyx@users.noreply.github.com 57723 - Dmitry Chepurovskiy me@dm3ch.net 57724 - Rauno Ots rauno.ots@cgi.com 57725 - Georg Neugschwandtner georg.neugschwandtner@gmx.net 57726 - pvalls polvallsrue@gmail.com 57727 - Robert Thomas 31854736+wolveix@users.noreply.github.com 57728 - Romeo Kienzler romeo.kienzler@gmail.com 57729 - tYYGH tYYGH@users.noreply.github.com 57730 - georne 77802995+georne@users.noreply.github.com 57731 - Maxwell Calman mcalman@MacBook-Pro.local 57732 - Naveen Honest Raj naveendurai19@gmail.com 57733 - Lucas Messenger lmesseng@cisco.com 57734 - Manish Kumar krmanish260@gmail.com 57735 - x0b x0bdev@gmail.com 57736 - CERN through the CS3MESH4EOSC Project 57737 - Nick Gaya nicholasgaya+github@gmail.com 57738 - Ashok Gelal 401055+ashokgelal@users.noreply.github.com 57739 - Dominik Mydlil dominik.mydlil@outlook.com 57740 - Nazar Mishturak nazarmx@gmail.com 57741 - Ansh Mittal iamAnshMittal@gmail.com 57742 - noabody noabody@yahoo.com 57743 - OleFrost 82263101+olefrost@users.noreply.github.com 57744 - Kenny Parsons kennyparsons93@gmail.com 57745 - Jeffrey Tolar tolar.jeffrey@gmail.com 57746 - jtagcat git-514635f7@jtag.cat 57747 - Tatsuya Noyori 57748 63089076+public-tatsuya-noyori@users.noreply.github.com 57749 - lewisxy lewisxy@users.noreply.github.com 57750 - Nolan Woods nolan_w@sfu.ca 57751 - Gautam Kumar 25435568+gautamajay52@users.noreply.github.com 57752 - Chris Macklin chris.macklin@10xgenomics.com 57753 - Antoon Prins antoon.prins@surfsara.nl 57754 - Alexey Ivanov rbtz@dropbox.com 57755 - Serge Pouliquen sp31415@free.fr 57756 - acsfer carlos@reendex.com 57757 - Tom tom@tom-fitzhenry.me.uk 57758 - Tyson Moore tyson@tyson.me 57759 - database64128 free122448@hotmail.com 57760 - Chris Lu chrislusf@users.noreply.github.com 57761 - Reid Buzby reid@rethink.software 57762 - darrenrhs darrenrhs@gmail.com 57763 - Florian Penzkofer fp@nullptr.de 57764 - Xuanchen Wu 117010292@link.cuhk.edu.cn 57765 - partev petrosyan@gmail.com 57766 - Dmitry Sitnikov fo2@inbox.ru 57767 - Haochen Tong i@hexchain.org 57768 - Michael Hanselmann public@hansmi.ch 57769 - Chuan Zh zhchuan7@gmail.com 57770 - Antoine GIRARD antoine.girard@sapk.fr 57771 - Justin Winokur (Jwink3101) Jwink3101@users.noreply.github.com 57772 - Mariano Absatz (git) scm@baby.com.ar 57773 - Greg Sadetsky lepetitg@gmail.com 57774 - yedamo logindaveye@gmail.com 57775 - hota lindwurm.q@gmail.com 57776 - vinibali vinibali1@gmail.com 57777 - Ken Enrique Morel ken.morel.santana@gmail.com 57778 - Justin Hellings justin.hellings@gmail.com 57779 - Parth Shukla pparth@pparth.net 57780 - wzl wangzl31@outlook.com 57781 - HNGamingUK connor@earnshawhome.co.uk 57782 - Jonta 359397+Jonta@users.noreply.github.com 57783 - YenForYang YenForYang@users.noreply.github.com 57784 - SimJoSt / Joda Stößer git@simjo.st 57785 - Logeshwaran waranlogesh@gmail.com 57786 - Rajat Goel rajat@dropbox.com 57787 - r0kk3rz r0kk3rz@gmail.com 57788 - Matthew Sevey mjsevey@gmail.com 57789 - Filip Rysavy fil@siasky.net 57790 - Ian Levesque ian@ianlevesque.org 57791 - Thomas Stachl thomas@stachl.me 57792 - Dmitry Bogatov git#v1@kaction.cc 57793 - thomae 4493560+thomae@users.noreply.github.com 57794 - trevyn trevyn-git@protonmail.com 57795 - David Liu david.yx.liu@oracle.com 57796 - Chris Nelson stuff@cjnaz.com 57797 - Felix Bünemann felix.buenemann@gmail.com 57798 - Atílio Antônio atiliodadalto@hotmail.com 57799 - Carlo Mion mion00@gmail.com 57800 - Chris Lu chris.lu@gmail.com 57801 - Vitor Arruda vitor.pimenta.arruda@gmail.com 57802 - bbabich bbabich@datamossa.com 57803 - David dp.davide.palma@gmail.com 57804 - Borna Butkovic borna@favicode.net 57805 - Fredric Arklid fredric.arklid@consid.se 57806 - Andy Jackson Andrew.Jackson@bl.uk 57807 - Sinan Tan i@tinytangent.com 57808 - deinferno 14363193+deinferno@users.noreply.github.com 57809 - rsapkf rsapkfff@pm.me 57810 - Will Holtz wholtz@gmail.com 57811 - GGG KILLER gggkiller2@gmail.com 57812 - Logeshwaran Murugesan logeshwaran@testpress.in 57813 - Lu Wang coolwanglu@gmail.com 57814 - Bumsu Hyeon ksitht@gmail.com 57815 - Shmz Ozggrn 98463324+ShmzOzggrn@users.noreply.github.com 57816 - Kim kim@jotta.no 57817 - Niels van de Weem n.van.de.weem@smile.nl 57818 - Koopa codingkoopa@gmail.com 57819 - Yunhai Luo yunhai-luo@hotmail.com 57820 - Charlie Jiang w@chariri.moe 57821 - Alain Nussbaumer alain.nussbaumer@alleluia.ch 57822 - Vanessasaurus 814322+vsoch@users.noreply.github.com 57823 - Isaac Levy isaac.r.levy@gmail.com 57824 - Gourav T workflowautomation@protonmail.com 57825 - Paulo Martins paulo.pontes.m@gmail.com 57826 - viveknathani viveknathani2402@gmail.com 57827 - Eng Zer Jun engzerjun@gmail.com 57828 - Abhiraj abhiraj.official15@gmail.com 57829 - Márton Elek elek@apache.org elek@users.noreply.github.com 57830 - Vincent Murphy vdm@vdm.ie 57831 - ctrl-q 34975747+ctrl-q@users.noreply.github.com 57832 - Nil Alexandrov nalexand@akamai.com 57833 - GuoXingbin 101376330+guoxingbin@users.noreply.github.com 57834 - Berkan Teber berkan@berkanteber.com 57835 - Tobias Klauser tklauser@distanz.ch 57836 - KARBOWSKI Piotr piotr.karbowski@gmail.com 57837 - GH geeklihui@foxmail.com 57838 - rafma0 int.main@gmail.com 57839 - Adrien Rey-Jarthon jobs@adrienjarthon.com 57840 - Nick Gooding 73336146+nickgooding@users.noreply.github.com 57841 - Leroy van Logchem lr.vanlogchem@gmail.com 57842 - Zsolt Ero zsolt.ero@gmail.com 57843 - Lesmiscore nao20010128@gmail.com 57844 - ehsantdy ehsan.tadayon@arvancloud.com ehsantadayon85@gmail.com 57845 - SwazRGB 65694696+swazrgb@users.noreply.github.com 57846 - Mateusz Puczyński mati6095@gmail.com 57847 - Michael C Tiernan - MIT-Research Computing Project mtiernan@mit.edu 57848 - Kaspian 34658474+KaspianDev@users.noreply.github.com 57849 - Werner EvilOlaf@users.noreply.github.com 57850 - Hugal31 hugo.laloge@gmail.com 57851 - Christian Galo 36752715+cgalo5758@users.noreply.github.com 57852 - Erik van Velzen erik@evanv.nl 57853 - Derek Battams derek@battams.ca 57854 - Paul devnoname120@gmail.com 57855 - SimonLiu simonliu009@users.noreply.github.com 57856 - Hugo Laloge hla@lescompanions.com 57857 - Mr-Kanister 68117355+Mr-Kanister@users.noreply.github.com 57858 - Rob Pickerill r.pickerill@gmail.com 57859 - Andrey to.merge@gmail.com 57860 - Eric Wolf 19wolf@gmail.com 57861 - Nick nick.naumann@mailbox.tu-dresden.de 57862 - Jason Zheng jszheng17@gmail.com 57863 - Matthew Vernon mvernon@wikimedia.org 57864 - Noah Hsu i@nn.ci 57865 - m00594701 mengpengbo@huawei.com 57866 - Art M. Gallagher artmg50@gmail.com 57867 - Sven Gerber 49589423+svengerber@users.noreply.github.com 57868 - CrossR r.cross@lancaster.ac.uk 57869 - Maciej Radzikowski maciej@radzikowski.com.pl 57870 - Scott Grimes scott.grimes@spaciq.com 57871 - Phil Shackleton 71221528+philshacks@users.noreply.github.com 57872 - eNV25 env252525@gmail.com 57873 - Caleb inventor96@users.noreply.github.com 57874 - J-P Treen jp@wraptious.com 57875 - Martin Czygan 53705+miku@users.noreply.github.com 57876 - buda sandrojijavadze@protonmail.com 57877 - mirekphd 36706320+mirekphd@users.noreply.github.com 57878 - vyloy vyloy@qq.com 57879 - Anthrazz 25553648+Anthrazz@users.noreply.github.com 57880 - zzr93 34027824+zzr93@users.noreply.github.com 57881 - Paul Norman penorman@mac.com 57882 - Lorenzo Maiorfi maiorfi@gmail.com 57883 - Claudio Maradonna penguyman@stronzi.org 57884 - Ovidiu Victor Tatar ovi.tatar@googlemail.com 57885 - Evan Spensley epspensley@gmail.com 57886 - Yen Hu 61753151+0x59656e@users.noreply.github.com 57887 - Steve Kowalik steven@wedontsleep.org 57888 - Jordi Gonzalez Muñoz jordigonzm@gmail.com 57889 - Joram Schrijver i@joram.io 57890 - Mark Trolley marktrolley@gmail.com 57891 - João Henrique Franco joaohenrique.franco@gmail.com 57892 - anonion aman207@users.noreply.github.com 57893 - Ryan Morey 4590343+rmorey@users.noreply.github.com 57894 - Simon Bos simonbos9@gmail.com 57895 - YFdyh000 yfdyh000@gmail.com * Josh Soref 57896 2119212+jsoref@users.noreply.github.com 57897 - Øyvind Heddeland Instefjord instefjord@outlook.com 57898 - Dmitry Deniskin 110819396+ddeniskin@users.noreply.github.com 57899 - Alexander Knorr 106825+opexxx@users.noreply.github.com 57900 - Richard Bateman richard@batemansr.us 57901 - Dimitri Papadopoulos Orfanos 57902 3234522+DimitriPapadopoulos@users.noreply.github.com 57903 - Lorenzo Milesi lorenzo.milesi@yetopen.com 57904 - Isaac Aymerich isaac.aymerich@gmail.com 57905 - YanceyChiew 35898533+YanceyChiew@users.noreply.github.com 57906 - Manoj Ghosh msays2000@gmail.com 57907 - Bachue Zhou bachue.shu@gmail.com 57908 - Manoj Ghosh manoj.ghosh@oracle.com 57909 - Tom Mombourquette tom@devnode.com 57910 - Robert Newson rnewson@apache.org 57911 - Samuel Johnson esamueljohnson@gmail.com 57912 - coultonluke luke@luke.org.uk 57913 - Anthony Pessy anthony@cogniteev.com 57914 - Philip Harvey pharvey@battelleecology.org 57915 - dgouju dgouju@users.noreply.github.com 57916 - Clément Notin clement.notin@gmail.com 57917 - x3-apptech 66947598+x3-apptech@users.noreply.github.com 57918 - Arnie97 arnie97@gmail.com 57919 - Roel Arents 2691308+roelarents@users.noreply.github.com 57920 - Aaron Gokaslan aaronGokaslan@gmail.com 57921 - techknowlogick matti@mdranta.net 57922 - rkettelerij richard@mindloops.nl 57923 - Kamui fin-kamui@pm.me 57924 - asdffdsazqqq 90116442+asdffdsazqqq@users.noreply.github.com 57925 - Nathaniel Wesley Filardo nfilardo@microsoft.com 57926 - ycdtosa ycdtosa@users.noreply.github.com 57927 - Erik Agterdenbos agterdenbos@users.noreply.github.com 57928 - Kevin Verstaen 48050031+kverstae@users.noreply.github.com 57929 - MohammadReza mrvashian@gmail.com 57930 - vanplus 60313789+vanplus@users.noreply.github.com 57931 - Jack 16779171+jkpe@users.noreply.github.com 57932 - Abdullah Saglam abdullah.saglam@stonebranch.com 57933 - Marks Polakovs github@markspolakovs.me 57934 - piyushgarg piyushgarg80@gmail.com 57935 - Kaloyan Raev kaloyan-raev@users.noreply.github.com 57936 - IMTheNachoMan imthenachoman@gmail.com 57937 - alankrit alankrit@google.com 57938 - Bryan Kaplan <#@bryankaplan.com> 57939 - LXY 767763591@qq.com 57940 - Simmon Li (he/him) li.simmon@gmail.com 57941 - happyxhw 44490504+happyxhw@users.noreply.github.com 57942 - Simmon Li (he/him) hello@crespire.dev 57943 - Matthias Baur baurmatt@users.noreply.github.com 57944 - Hunter Wittenborn hunter@hunterwittenborn.com 57945 - logopk peter@kreuser.name 57946 - Gerard Bosch 30733556+gerardbosch@users.noreply.github.com 57947 - ToBeFree github@tfrei.de 57948 - NodudeWasTaken 75137537+NodudeWasTaken@users.noreply.github.com 57949 - Peter Brunner peter@lugoues.net 57950 - Ninh Pham dongian.rapclubkhtn@gmail.com 57951 - Ryan Caezar Itang sitiom@proton.me 57952 - Peter Brunner peter@psykhe.com 57953 - Leandro Sacchet leandro.sacchet@animati.com.br 57954 - dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 57955 - cycneuramus 56681631+cycneuramus@users.noreply.github.com 57956 - Arnavion me@arnavion.dev 57957 - Christopher Merry christopher.merry@mlb.com 57958 - Thibault Coupin thibault.coupin@gmail.com 57959 - Richard Tweed RichardoC@users.noreply.github.com 57960 - Zach Kipp Zacho2@users.noreply.github.com 57961 - yuudi 26199752+yuudi@users.noreply.github.com 57962 - NickIAm NickIAm@users.noreply.github.com 57963 - Juang, Yi-Lin frankyjuang@gmail.com 57964 - jumbi77 jumbi77@users.noreply.github.com 57965 - Aditya Basu ab.aditya.basu@gmail.com 57966 - ed s@ocv.me 57967 - Drew Parsons dparsons@emerall.com 57968 - Joel joelnb@users.noreply.github.com 57969 - wiserain mail275@gmail.com 57970 - Roel Arents roel.arents@kadaster.nl 57971 - Shyim github@shyim.de 57972 - Rintze Zelle 78232505+rzelle-lallemand@users.noreply.github.com 57973 - Damo damoclark@users.noreply.github.com 57974 - WeidiDeng weidi_deng@icloud.com 57975 - Brian Starkey stark3y@gmail.com 57976 - jladbrook jhladbrook@gmail.com 57977 - Loren Gordon lorengordon@users.noreply.github.com 57978 - dlitster davidlitster@gmail.com 57979 - Tobias Gion tobias@gion.io 57980 - Jānis Bebrītis janis.bebritis@wunder.io 57981 - Adam K github.com@ak.tidy.email 57982 - Andrei Smirnov smirnov.captain@gmail.com 57983 - Janne Hellsten jjhellst@gmail.com 57984 - cc 12904584+shvc@users.noreply.github.com 57985 - Tareq Sharafy tareq.sha@gmail.com 57986 - kapitainsky dariuszb@me.com 57987 - douchen playgoobug@gmail.com 57988 - Sam Lai 70988+slai@users.noreply.github.com 57989 - URenko 18209292+URenko@users.noreply.github.com 57990 - Stanislav Gromov kullfar@gmail.com 57991 - Paulo Schreiner paulo.schreiner@delivion.de 57992 - Mariusz Suchodolski mariusz@suchodol.ski 57993 - danielkrajnik dan94kra@gmail.com 57994 - Peter Fern github@0xc0dedbad.com 57995 - zzq i@zhangzqs.cn 57996 - mac-15 usman.ilamdin@phpstudios.com 57997 - Sawada Tsunayoshi 34431649+TsunayoshiSawada@users.noreply.github.com 57998 - Dean Attali daattali@gmail.com 57999 - Fjodor42 molgaard@gmail.com 58000 - BakaWang wa11579@hotmail.com 58001 - Mahad 56235065+Mahad-lab@users.noreply.github.com 58002 - Vladislav Vorobev x.miere@gmail.com 58003 - darix darix@users.noreply.github.com 58004 - Benjamin 36415086+bbenjamin-sys@users.noreply.github.com 58005 - Chun-Hung Tseng henrybear327@users.noreply.github.com 58006 - Ricardo D'O. Albanus rdalbanus@users.noreply.github.com 58007 - gabriel-suela gscsuela@gmail.com 58008 - Tiago Boeing contato@tiagoboeing.com 58009 - Edwin Mackenzie-Owen edwin.mowen@gmail.com 58010 - Niklas Hambüchen mail@nh2.me 58011 - yuudi yuudi@users.noreply.github.com 58012 - Zach github@prozach.org 58013 - nielash 31582349+nielash@users.noreply.github.com 58014 - Julian Lepinski lepinsk@users.noreply.github.com 58015 - Raymond Berger RayBB@users.noreply.github.com 58016 - Nihaal Sangha nihaal.git@gmail.com 58017 - Masamune3210 1053504+Masamune3210@users.noreply.github.com 58018 - James Braza jamesbraza@gmail.com 58019 - antoinetran antoinetran@users.noreply.github.com 58020 - alexia me@alexia.lol 58021 - nielash nielronash@gmail.com 58022 - Vitor Gomes vitor.gomes@delivion.de mail@vitorgomes.com 58023 - Jacob Hands jacob@gogit.io 58024 - hideo aoyama 100831251+boukendesho@users.noreply.github.com 58025 - Roberto Ricci io@r-ricci.it 58026 - Bjørn Smith bjornsmith@gmail.com 58027 - Alishan Ladhani 8869764+aladh@users.noreply.github.com 58028 - zjx20 zhoujianxiong2@gmail.com 58029 - Oksana 142890647+oks-maytech@users.noreply.github.com 58030 - Volodymyr Kit v.kit@maytech.net 58031 - David Pedersen limero@me.com 58032 - Drew Stinnett drew@drewlink.com 58033 - Pat Patterson pat@backblaze.com 58034 - Herby Gillot herby.gillot@gmail.com 58035 - Nikita Shoshin shoshin_nikita@fastmail.com 58036 - rinsuki 428rinsuki+git@gmail.com 58037 - Beyond Meat 51850644+beyondmeat@users.noreply.github.com 58038 - Saleh Dindar salh@fb.com 58039 - Volodymyr 142890760+vkit-maytech@users.noreply.github.com 58040 - Gabriel Espinoza 31670639+gspinoza@users.noreply.github.com 58041 - Keigo Imai keigo.imai@gmail.com 58042 - Ivan Yanitra iyanitra@tesla-consulting.com 58043 - alfish2000 alfish2000@gmail.com 58044 - wuxingzhong qq330332812@gmail.com 58045 - Adithya Kumar akumar42@protonmail.com 58046 - Tayo-pasedaRJ 138471223+Tayo-pasedaRJ@users.noreply.github.com 58047 - Peter Kreuser logo@kreuser.name 58048 - Piyush 58049 - fotile96 fotile96@users.noreply.github.com 58050 - Luc Ritchie luc.ritchie@gmail.com 58051 - cynful cynful@users.noreply.github.com 58052 - wjielai wjielai@tencent.com 58053 - Jack Deng jackdeng@gmail.com 58054 - Mikubill 31246794+Mikubill@users.noreply.github.com 58055 - Artur Neumann artur@jankaritech.com 58056 - Saw-jan saw.jan.grg3e@gmail.com 58057 - Oksana Zhykina o.zhykina@maytech.net 58058 - karan karan.gupta92@gmail.com 58059 - viktor viktor@yakovchuk.net 58060 - moongdal moongdal@tutanota.com 58061 - Mina Galić freebsd@igalic.co 58062 - Alen Šiljak dev@alensiljak.eu.org 58063 - 你知道未来吗 rkonfj@gmail.com 58064 - Abhinav Dhiman 8640877+ahnv@users.noreply.github.com 58065 - halms 7513146+halms@users.noreply.github.com 58066 - ben-ba benjamin.brauner@gmx.de 58067 - Eli Orzitzer e_orz@yahoo.com 58068 - Anthony Metzidis anthony.metzidis@gmail.com 58069 - emyarod afw5059@gmail.com 58070 - keongalvin keongalvin@gmail.com 58071 - rarspace01 rarspace01@users.noreply.github.com 58072 - Paul Stern paulstern45@gmail.com 58073 - Nikhil Ahuja nikhilahuja@live.com 58074 - Harshit Budhraja 52413945+harshit-budhraja@users.noreply.github.com 58075 - Tera 24725862+teraa@users.noreply.github.com 58076 - Kyle Reynolds kylereynoldsdev@gmail.com 58077 - Michael Eischer michael.eischer@gmx.de 58078 - Thomas Müller 1005065+DeepDiver1975@users.noreply.github.com 58079 - DanielEgbers 27849724+DanielEgbers@users.noreply.github.com 58080 - Jack Provance 49460795+njprov@users.noreply.github.com 58081 - Gabriel Ramos 109390599+gabrielramos02@users.noreply.github.com 58082 - Dan McArdle d@nmcardle.com 58083 - Joe Cai joe.cai@bigcommerce.com 58084 - Anders Swanson anders.swanson@oracle.com 58085 - huajin tong 137764712+thirdkeyword@users.noreply.github.com 58086 58087 Contact the rclone project 58088 58089 Forum 58090 58091 Forum for questions and general discussion: 58092 58093 - https://forum.rclone.org 58094 58095 Business support 58096 58097 For business support or sponsorship enquiries please see: 58098 58099 - https://rclone.com/ 58100 - sponsorship@rclone.com 58101 58102 GitHub repository 58103 58104 The project's repository is located at: 58105 58106 - https://github.com/rclone/rclone 58107 58108 There you can file bug reports or contribute with pull requests. 58109 58110 Twitter 58111 58112 You can also follow Nick on twitter for rclone announcements: 58113 58114 - [@njcw](https://twitter.com/njcw) 58115 58116 Email 58117 58118 Or if all else fails or you want to ask something private or 58119 confidential 58120 58121 - info@rclone.com 58122 58123 Please don't email requests for help to this address - those are better 58124 directed to the forum unless you'd like to sign up for business support.