github.com/10XDev/rclone@v1.52.3-0.20200626220027-16af9ab76b2a/docs/content/overview.md (about) 1 --- 2 title: "Overview of cloud storage systems" 3 description: "Overview of cloud storage systems" 4 type: page 5 --- 6 7 # Overview of cloud storage systems # 8 9 Each cloud storage system is slightly different. Rclone attempts to 10 provide a unified interface to them, but some underlying differences 11 show through. 12 13 ## Features ## 14 15 Here is an overview of the major features of each cloud storage system. 16 17 | Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | 18 | ---------------------------- |:-----------:|:-------:|:----------------:|:---------------:|:---------:| 19 | 1Fichier | Whirlpool | No | No | Yes | R | 20 | Amazon Drive | MD5 | No | Yes | No | R | 21 | Amazon S3 | MD5 | Yes | No | No | R/W | 22 | Backblaze B2 | SHA1 | Yes | No | No | R/W | 23 | Box | SHA1 | Yes | Yes | No | - | 24 | Citrix ShareFile | MD5 | Yes | Yes | No | - | 25 | Dropbox | DBHASH † | Yes | Yes | No | - | 26 | FTP | - | No | No | No | - | 27 | Google Cloud Storage | MD5 | Yes | No | No | R/W | 28 | Google Drive | MD5 | Yes | No | Yes | R/W | 29 | Google Photos | - | No | No | Yes | R | 30 | HTTP | - | No | No | No | R | 31 | Hubic | MD5 | Yes | No | No | R/W | 32 | Jottacloud | MD5 | Yes | Yes | No | R/W | 33 | Koofr | MD5 | No | Yes | No | - | 34 | Mail.ru Cloud | Mailru ‡‡‡ | Yes | Yes | No | - | 35 | Mega | - | No | No | Yes | - | 36 | Memory | MD5 | Yes | No | No | - | 37 | Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W | 38 | Microsoft OneDrive | SHA1 ‡‡ | Yes | Yes | No | R | 39 | OpenDrive | MD5 | Yes | Yes | No | - | 40 | OpenStack Swift | MD5 | Yes | No | No | R/W | 41 | pCloud | MD5, SHA1 | Yes | No | No | W | 42 | premiumize.me | - | No | Yes | No | R | 43 | put.io | CRC-32 | Yes | No | Yes | R | 44 | QingStor | MD5 | No | No | No | R/W | 45 | Seafile | - | No | No | No | - | 46 | SFTP | MD5, SHA1 ‡ | Yes | Depends | No | - | 47 | SugarSync | - | No | No | No | - | 48 | Tardigrade | - | Yes | No | No | - | 49 | WebDAV | MD5, SHA1 ††| Yes ††† | Depends | No | - | 50 | Yandex Disk | MD5 | Yes | No | No | R/W | 51 | The local filesystem | All | Yes | Depends | No | - | 52 53 ### Hash ### 54 55 The cloud storage system supports various hash types of the objects. 56 The hashes are used when transferring data as an integrity check and 57 can be specifically used with the `--checksum` flag in syncs and in 58 the `check` command. 59 60 To use the verify checksums when transferring between cloud storage 61 systems they must support a common hash type. 62 63 † Note that Dropbox supports [its own custom 64 hash](https://www.dropbox.com/developers/reference/content-hash). 65 This is an SHA256 sum of all the 4MB block SHA256s. 66 67 ‡ SFTP supports checksums if the same login has shell access and `md5sum` 68 or `sha1sum` as well as `echo` are in the remote's PATH. 69 70 †† WebDAV supports hashes when used with Owncloud and Nextcloud only. 71 72 ††† WebDAV supports modtimes when used with Owncloud and Nextcloud only. 73 74 ‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive 75 for business and SharePoint server support Microsoft's own 76 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 77 78 ‡‡‡ Mail.ru uses its own modified SHA1 hash 79 80 ### ModTime ### 81 82 The cloud storage system supports setting modification times on 83 objects. If it does then this enables a using the modification times 84 as part of the sync. If not then only the size will be checked by 85 default, though the MD5SUM can be checked with the `--checksum` flag. 86 87 All cloud storage systems support some kind of date on the object and 88 these will be set when transferring from the cloud storage system. 89 90 ### Case Insensitive ### 91 92 If a cloud storage systems is case sensitive then it is possible to 93 have two files which differ only in case, eg `file.txt` and 94 `FILE.txt`. If a cloud storage system is case insensitive then that 95 isn't possible. 96 97 This can cause problems when syncing between a case insensitive 98 system and a case sensitive system. The symptom of this is that no 99 matter how many times you run the sync it never completes fully. 100 101 The local filesystem and SFTP may or may not be case sensitive 102 depending on OS. 103 104 * Windows - usually case insensitive, though case is preserved 105 * OSX - usually case insensitive, though it is possible to format case sensitive 106 * Linux - usually case sensitive, but there are case insensitive file systems (eg FAT formatted USB keys) 107 108 Most of the time this doesn't cause any problems as people tend to 109 avoid files whose name differs only by case even on case sensitive 110 systems. 111 112 ### Duplicate files ### 113 114 If a cloud storage system allows duplicate files then it can have two 115 objects with the same name. 116 117 This confuses rclone greatly when syncing - use the `rclone dedupe` 118 command to rename or remove duplicates. 119 120 ### Restricted filenames ### 121 122 Some cloud storage systems might have restrictions on the characters 123 that are usable in file or directory names. 124 When `rclone` detects such a name during a file upload, it will 125 transparently replace the restricted characters with similar looking 126 Unicode characters. 127 128 This process is designed to avoid ambiguous file names as much as 129 possible and allow to move files between many cloud storage systems 130 transparently. 131 132 The name shown by `rclone` to the user or during log output will only 133 contain a minimal set of [replaced characters](#restricted-characters) 134 to ensure correct formatting and not necessarily the actual name used 135 on the cloud storage. 136 137 This transformation is reversed when downloading a file or parsing 138 `rclone` arguments. 139 For example, when uploading a file named `my file?.txt` to Onedrive 140 will be displayed as `my file?.txt` on the console, but stored as 141 `my file？.txt` (the `?` gets replaced by the similar looking `？` 142 character) to Onedrive. 143 The reverse transformation allows to read a file`unusual/name.txt` 144 from Google Drive, by passing the name `unusual／name.txt` (the `/` needs 145 to be replaced by the similar looking `／` character) on the command line. 146 147 #### Default restricted characters {#restricted-characters} 148 149 The table below shows the characters that are replaced by default. 150 151 When a replacement character is found in a filename, this character 152 will be escaped with the `‛` character to avoid ambiguous file names. 153 (e.g. a file named `␀.txt` would shown as `‛␀.txt`) 154 155 Each cloud storage backend can use a different set of characters, 156 which will be specified in the documentation for each backend. 157 158 | Character | Value | Replacement | 159 | --------- |:-----:|:-----------:| 160 | NUL | 0x00 | ␀ | 161 | SOH | 0x01 | ␁ | 162 | STX | 0x02 | ␂ | 163 | ETX | 0x03 | ␃ | 164 | EOT | 0x04 | ␄ | 165 | ENQ | 0x05 | ␅ | 166 | ACK | 0x06 | ␆ | 167 | BEL | 0x07 | ␇ | 168 | BS | 0x08 | ␈ | 169 | HT | 0x09 | ␉ | 170 | LF | 0x0A | ␊ | 171 | VT | 0x0B | ␋ | 172 | FF | 0x0C | ␌ | 173 | CR | 0x0D | ␍ | 174 | SO | 0x0E | ␎ | 175 | SI | 0x0F | ␏ | 176 | DLE | 0x10 | ␐ | 177 | DC1 | 0x11 | ␑ | 178 | DC2 | 0x12 | ␒ | 179 | DC3 | 0x13 | ␓ | 180 | DC4 | 0x14 | ␔ | 181 | NAK | 0x15 | ␕ | 182 | SYN | 0x16 | ␖ | 183 | ETB | 0x17 | ␗ | 184 | CAN | 0x18 | ␘ | 185 | EM | 0x19 | ␙ | 186 | SUB | 0x1A | ␚ | 187 | ESC | 0x1B | ␛ | 188 | FS | 0x1C | ␜ | 189 | GS | 0x1D | ␝ | 190 | RS | 0x1E | ␞ | 191 | US | 0x1F | ␟ | 192 | / | 0x2F | ／ | 193 | DEL | 0x7F | ␡ | 194 195 The default encoding will also encode these file names as they are 196 problematic with many cloud storage systems. 197 198 | File name | Replacement | 199 | --------- |:-----------:| 200 | . | ． | 201 | .. | ．． | 202 203 #### Invalid UTF-8 bytes {#invalid-utf8} 204 205 Some backends only support a sequence of well formed UTF-8 bytes 206 as file or directory names. 207 208 In this case all invalid UTF-8 bytes will be replaced with a quoted 209 representation of the byte value to allow uploading a file to such a 210 backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`. 211 212 A common source of invalid UTF-8 bytes are local filesystems, that store 213 names in a different encoding than UTF-8 or UTF-16, like latin1. See the 214 [local filenames](/local/#filenames) section for details. 215 216 #### Encoding option {#encoding} 217 218 Most backends have an encoding options, specified as a flag 219 `--backend-encoding` where `backend` is the name of the backend, or as 220 a config parameter `encoding` (you'll need to select the Advanced 221 config in `rclone config` to see it). 222 223 This will have default value which encodes and decodes characters in 224 such a way as to preserve the maximum number of characters (see 225 above). 226 227 However this can be incorrect in some scenarios, for example if you 228 have a Windows file system with characters such as `＊` and `？` that 229 you want to remain as those characters on the remote rather than being 230 translated to `*` and `?`. 231 232 The `--backend-encoding` flags allow you to change that. You can 233 disable the encoding completely with `--backend-encoding None` or set 234 `encoding = None` in the config file. 235 236 Encoding takes a comma separated list of encodings. You can see the 237 list of all available characters by passing an invalid value to this 238 flag, eg `--local-encoding "help"` and `rclone help flags encoding` 239 will show you the defaults for the backends. 240 241 | Encoding | Characters | 242 | --------- | ---------- | 243 | Asterisk | `*` | 244 | BackQuote | `` ` `` | 245 | BackSlash | `\` | 246 | Colon | `:` | 247 | CrLf | CR 0x0D, LF 0x0A | 248 | Ctl | All control characters 0x00-0x1F | 249 | Del | DEL 0x7F | 250 | Dollar | `$` | 251 | Dot | `.` | 252 | DoubleQuote | `"` | 253 | Hash | `#` | 254 | InvalidUtf8 | An invalid UTF-8 character (eg latin1) | 255 | LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string | 256 | LeftPeriod | `.` on the left of a string | 257 | LeftSpace | SPACE on the left of a string | 258 | LeftTilde | `~` on the left of a string | 259 | LtGt | `<`, `>` | 260 | None | No characters are encoded | 261 | Percent | `%` | 262 | Pipe | \| | 263 | Question | `?` | 264 | RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | 265 | RightPeriod | `.` on the right of a string | 266 | RightSpace | SPACE on the right of a string | 267 | SingleQuote | `'` | 268 | Slash | `/` | 269 270 To take a specific example, the FTP backend's default encoding is 271 272 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 273 274 However, let's say the FTP server is running on Windows and can't have 275 any of the invalid Windows characters in file names. You are backing 276 up Linux servers to this FTP server which do have those characters in 277 file names. So you would add the Windows set which are 278 279 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 280 281 to the existing ones, giving: 282 283 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 284 285 This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file. 286 287 Or let's say you have a Windows server but you want to preserve `＊` 288 and `？`, you would then have this as the encoding (the Windows 289 encoding minus `Asterisk` and `Question`). 290 291 Slash,LtGt,DoubleQuote,Colon,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 292 293 This can be specified using the `--local-encoding` flag or using an 294 `encoding` parameter in the config file. 295 296 ### MIME Type ### 297 298 MIME types (also known as media types) classify types of documents 299 using a simple text classification, eg `text/html` or 300 `application/pdf`. 301 302 Some cloud storage systems support reading (`R`) the MIME type of 303 objects and some support writing (`W`) the MIME type of objects. 304 305 The MIME type can be important if you are serving files directly to 306 HTTP from the storage system. 307 308 If you are copying from a remote which supports reading (`R`) to a 309 remote which supports writing (`W`) then rclone will preserve the MIME 310 types. Otherwise they will be guessed from the extension, or the 311 remote itself may assign the MIME type. 312 313 ## Optional Features ## 314 315 All the remotes support a basic set of features, but there are some 316 optional features supported by some remotes used to make some 317 operations more efficient. 318 319 | Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir | 320 | ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:| :------: | 321 | 1Fichier | No | No | No | No | No | No | No | No | No | Yes | 322 | Amazon Drive | Yes | No | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 323 | Amazon S3 | No | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 324 | Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 325 | Box | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | No | Yes | 326 | Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | 327 | Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | Yes | Yes | 328 | FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 329 | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 330 | Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 331 | Google Photos | No | No | No | No | No | No | No | No | No | No | 332 | HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 333 | Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 334 | Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 335 | Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 336 | Mega | Yes | No | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 337 | Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | 338 | Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 339 | Microsoft OneDrive | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | Yes | Yes | Yes | 340 | OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | 341 | OpenStack Swift | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 342 | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 343 | premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes | 344 | put.io | Yes | No | Yes | Yes | Yes | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 345 | QingStor | No | Yes | No | No | Yes | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 346 | Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 347 | SFTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 348 | SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | 349 | Tardigrade | Yes † | No | No | No | No | Yes | Yes | No | No | No | 350 | WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 351 | Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | 352 | The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 353 354 ### Purge ### 355 356 This deletes a directory quicker than just deleting all the files in 357 the directory. 358 359 † Note Swift, Hubic, and Tardigrade implement this in order to delete 360 directory markers but they don't actually have a quicker way of deleting 361 files other than deleting them individually. 362 363 ‡ StreamUpload is not supported with Nextcloud 364 365 ### Copy ### 366 367 Used when copying an object to and from the same remote. This known 368 as a server side copy so you can copy a file without downloading it 369 and uploading it again. It is used if you use `rclone copy` or 370 `rclone move` if the remote doesn't support `Move` directly. 371 372 If the server doesn't support `Copy` directly then for copy operations 373 the file is downloaded then re-uploaded. 374 375 ### Move ### 376 377 Used when moving/renaming an object on the same remote. This is known 378 as a server side move of a file. This is used in `rclone move` if the 379 server doesn't support `DirMove`. 380 381 If the server isn't capable of `Move` then rclone simulates it with 382 `Copy` then delete. If the server doesn't support `Copy` then rclone 383 will download the file and re-upload it. 384 385 ### DirMove ### 386 387 This is used to implement `rclone move` to move a directory if 388 possible. If it isn't then it will use `Move` on each file (which 389 falls back to `Copy` then download and upload - see `Move` section). 390 391 ### CleanUp ### 392 393 This is used for emptying the trash for a remote by `rclone cleanup`. 394 395 If the server can't do `CleanUp` then `rclone cleanup` will return an 396 error. 397 398 ### ListR ### 399 400 The remote supports a recursive list to list all the contents beneath 401 a directory quickly. This enables the `--fast-list` flag to work. 402 See the [rclone docs](/docs/#fast-list) for more details. 403 404 ### StreamUpload ### 405 406 Some remotes allow files to be uploaded without knowing the file size 407 in advance. This allows certain operations to work without spooling the 408 file to local disk first, e.g. `rclone rcat`. 409 410 ### LinkSharing ### 411 412 Sets the necessary permissions on a file or folder and prints a link 413 that allows others to access them, even if they don't have an account 414 on the particular cloud provider. 415 416 ### About ### 417 418 This is used to fetch quota information from the remote, like bytes 419 used/free/quota and bytes used in the trash. 420 421 This is also used to return the space used, available for `rclone mount`. 422 423 If the server can't do `About` then `rclone about` will return an 424 error. 425 426 ### EmptyDir ### 427 428 The remote supports empty directories. See [Limitations](/bugs/#limitations) 429 for details. Most Object/Bucket based remotes do not support this.