github.com/xhghs/rclone@v1.51.1-0.20200430155106-e186a28cced8/docs/content/overview.md (about) 1 --- 2 title: "Overview of cloud storage systems" 3 description: "Overview of cloud storage systems" 4 type: page 5 date: "2019-02-25" 6 --- 7 8 # Overview of cloud storage systems # 9 10 Each cloud storage system is slightly different. Rclone attempts to 11 provide a unified interface to them, but some underlying differences 12 show through. 13 14 ## Features ## 15 16 Here is an overview of the major features of each cloud storage system. 17 18 | Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | 19 | ---------------------------- |:-----------:|:-------:|:----------------:|:---------------:|:---------:| 20 | 1Fichier | Whirlpool | No | No | Yes | R | 21 | Amazon Drive | MD5 | No | Yes | No | R | 22 | Amazon S3 | MD5 | Yes | No | No | R/W | 23 | Backblaze B2 | SHA1 | Yes | No | No | R/W | 24 | Box | SHA1 | Yes | Yes | No | - | 25 | Citrix ShareFile | MD5 | Yes | Yes | No | - | 26 | Dropbox | DBHASH † | Yes | Yes | No | - | 27 | FTP | - | No | No | No | - | 28 | Google Cloud Storage | MD5 | Yes | No | No | R/W | 29 | Google Drive | MD5 | Yes | No | Yes | R/W | 30 | Google Photos | - | No | No | Yes | R | 31 | HTTP | - | No | No | No | R | 32 | Hubic | MD5 | Yes | No | No | R/W | 33 | Jottacloud | MD5 | Yes | Yes | No | R/W | 34 | Koofr | MD5 | No | Yes | No | - | 35 | Mail.ru Cloud | Mailru ‡‡‡ | Yes | Yes | No | - | 36 | Mega | - | No | No | Yes | - | 37 | Memory | MD5 | Yes | No | No | - | 38 | Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W | 39 | Microsoft OneDrive | SHA1 ‡‡ | Yes | Yes | No | R | 40 | OpenDrive | MD5 | Yes | Yes | No | - | 41 | Openstack Swift | MD5 | Yes | No | No | R/W | 42 | pCloud | MD5, SHA1 | Yes | No | No | W | 43 | premiumize.me | - | No | Yes | No | R | 44 | put.io | CRC-32 | Yes | No | Yes | R | 45 | QingStor | MD5 | No | No | No | R/W | 46 | SFTP | MD5, SHA1 ‡ | Yes | Depends | No | - | 47 | SugarSync | - | No | No | No | - | 48 | WebDAV | MD5, SHA1 ††| Yes ††† | Depends | No | - | 49 | Yandex Disk | MD5 | Yes | No | No | R/W | 50 | The local filesystem | All | Yes | Depends | No | - | 51 52 ### Hash ### 53 54 The cloud storage system supports various hash types of the objects. 55 The hashes are used when transferring data as an integrity check and 56 can be specifically used with the `--checksum` flag in syncs and in 57 the `check` command. 58 59 To use the verify checksums when transferring between cloud storage 60 systems they must support a common hash type. 61 62 † Note that Dropbox supports [its own custom 63 hash](https://www.dropbox.com/developers/reference/content-hash). 64 This is an SHA256 sum of all the 4MB block SHA256s. 65 66 ‡ SFTP supports checksums if the same login has shell access and `md5sum` 67 or `sha1sum` as well as `echo` are in the remote's PATH. 68 69 †† WebDAV supports hashes when used with Owncloud and Nextcloud only. 70 71 ††† WebDAV supports modtimes when used with Owncloud and Nextcloud only. 72 73 ‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive 74 for business and SharePoint server support Microsoft's own 75 [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 76 77 ‡‡‡ Mail.ru uses its own modified SHA1 hash 78 79 ### ModTime ### 80 81 The cloud storage system supports setting modification times on 82 objects. If it does then this enables a using the modification times 83 as part of the sync. If not then only the size will be checked by 84 default, though the MD5SUM can be checked with the `--checksum` flag. 85 86 All cloud storage systems support some kind of date on the object and 87 these will be set when transferring from the cloud storage system. 88 89 ### Case Insensitive ### 90 91 If a cloud storage systems is case sensitive then it is possible to 92 have two files which differ only in case, eg `file.txt` and 93 `FILE.txt`. If a cloud storage system is case insensitive then that 94 isn't possible. 95 96 This can cause problems when syncing between a case insensitive 97 system and a case sensitive system. The symptom of this is that no 98 matter how many times you run the sync it never completes fully. 99 100 The local filesystem and SFTP may or may not be case sensitive 101 depending on OS. 102 103 * Windows - usually case insensitive, though case is preserved 104 * OSX - usually case insensitive, though it is possible to format case sensitive 105 * Linux - usually case sensitive, but there are case insensitive file systems (eg FAT formatted USB keys) 106 107 Most of the time this doesn't cause any problems as people tend to 108 avoid files whose name differs only by case even on case sensitive 109 systems. 110 111 ### Duplicate files ### 112 113 If a cloud storage system allows duplicate files then it can have two 114 objects with the same name. 115 116 This confuses rclone greatly when syncing - use the `rclone dedupe` 117 command to rename or remove duplicates. 118 119 ### Restricted filenames ### 120 121 Some cloud storage systems might have restrictions on the characters 122 that are usable in file or directory names. 123 When `rclone` detects such a name during a file upload, it will 124 transparently replace the restricted characters with similar looking 125 Unicode characters. 126 127 This process is designed to avoid ambiguous file names as much as 128 possible and allow to move files between many cloud storage systems 129 transparently. 130 131 The name shown by `rclone` to the user or during log output will only 132 contain a minimal set of [replaced characters](#restricted-characters) 133 to ensure correct formatting and not necessarily the actual name used 134 on the cloud storage. 135 136 This transformation is reversed when downloading a file or parsing 137 `rclone` arguments. 138 For example, when uploading a file named `my file?.txt` to Onedrive 139 will be displayed as `my file?.txt` on the console, but stored as 140 `my file？.txt` (the `?` gets replaced by the similar looking `？` 141 character) to Onedrive. 142 The reverse transformation allows to read a file`unusual/name.txt` 143 from Google Drive, by passing the name `unusual／name.txt` (the `/` needs 144 to be replaced by the similar looking `／` character) on the command line. 145 146 #### Default restricted characters {#restricted-characters} 147 148 The table below shows the characters that are replaced by default. 149 150 When a replacement character is found in a filename, this character 151 will be escaped with the `‛` character to avoid ambiguous file names. 152 (e.g. a file named `␀.txt` would shown as `‛␀.txt`) 153 154 Each cloud storage backend can use a different set of characters, 155 which will be specified in the documentation for each backend. 156 157 | Character | Value | Replacement | 158 | --------- |:-----:|:-----------:| 159 | NUL | 0x00 | ␀ | 160 | SOH | 0x01 | ␁ | 161 | STX | 0x02 | ␂ | 162 | ETX | 0x03 | ␃ | 163 | EOT | 0x04 | ␄ | 164 | ENQ | 0x05 | ␅ | 165 | ACK | 0x06 | ␆ | 166 | BEL | 0x07 | ␇ | 167 | BS | 0x08 | ␈ | 168 | HT | 0x09 | ␉ | 169 | LF | 0x0A | ␊ | 170 | VT | 0x0B | ␋ | 171 | FF | 0x0C | ␌ | 172 | CR | 0x0D | ␍ | 173 | SO | 0x0E | ␎ | 174 | SI | 0x0F | ␏ | 175 | DLE | 0x10 | ␐ | 176 | DC1 | 0x11 | ␑ | 177 | DC2 | 0x12 | ␒ | 178 | DC3 | 0x13 | ␓ | 179 | DC4 | 0x14 | ␔ | 180 | NAK | 0x15 | ␕ | 181 | SYN | 0x16 | ␖ | 182 | ETB | 0x17 | ␗ | 183 | CAN | 0x18 | ␘ | 184 | EM | 0x19 | ␙ | 185 | SUB | 0x1A | ␚ | 186 | ESC | 0x1B | ␛ | 187 | FS | 0x1C | ␜ | 188 | GS | 0x1D | ␝ | 189 | RS | 0x1E | ␞ | 190 | US | 0x1F | ␟ | 191 | / | 0x2F | ／ | 192 | DEL | 0x7F | ␡ | 193 194 The default encoding will also encode these file names as they are 195 problematic with many cloud storage systems. 196 197 | File name | Replacement | 198 | --------- |:-----------:| 199 | . | ． | 200 | .. | ．． | 201 202 #### Invalid UTF-8 bytes {#invalid-utf8} 203 204 Some backends only support a sequence of well formed UTF-8 bytes 205 as file or directory names. 206 207 In this case all invalid UTF-8 bytes will be replaced with a quoted 208 representation of the byte value to allow uploading a file to such a 209 backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`. 210 211 A common source of invalid UTF-8 bytes are local filesystems, that store 212 names in a different encoding than UTF-8 or UTF-16, like latin1. See the 213 [local filenames](/local/#filenames) section for details. 214 215 #### Encoding option {#encoding} 216 217 Most backends have an encoding options, specified as a flag 218 `--backend-encoding` where `backend` is the name of the backend, or as 219 a config parameter `encoding` (you'll need to select the Advanced 220 config in `rclone config` to see it). 221 222 This will have default value which encodes and decodes characters in 223 such a way as to preserve the maximum number of characters (see 224 above). 225 226 However this can be incorrect in some scenarios, for example if you 227 have a Windows file system with characters such as `＊` and `？` that 228 you want to remain as those characters on the remote rather than being 229 translated to `*` and `?`. 230 231 The `--backend-encoding` flags allow you to change that. You can 232 disable the encoding completely with `--backend-encoding None` or set 233 `encoding = None` in the config file. 234 235 Encoding takes a comma separated list of encodings. You can see the 236 list of all available characters by passing an invalid value to this 237 flag, eg `--local-encoding "help"` and `rclone help flags encoding` 238 will show you the defaults for the backends. 239 240 | Encoding | Characters | 241 | --------- | ---------- | 242 | Asterisk | `*` | 243 | BackQuote | `` ` `` | 244 | BackSlash | `\` | 245 | Colon | `:` | 246 | CrLf | CR 0x0D, LF 0x0A | 247 | Ctl | All control characters 0x00-0x1F | 248 | Del | DEL 0x7F | 249 | Dollar | `$` | 250 | Dot | `.` | 251 | DoubleQuote | `"` | 252 | Hash | `#` | 253 | InvalidUtf8 | An invalid UTF-8 character (eg latin1) | 254 | LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string | 255 | LeftPeriod | `.` on the left of a string | 256 | LeftSpace | SPACE on the left of a string | 257 | LeftTilde | `~` on the left of a string | 258 | LtGt | `<`, `>` | 259 | None | No characters are encoded | 260 | Percent | `%` | 261 | Pipe | <code>\|</code> | 262 | Question | `?` | 263 | RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | 264 | RightPeriod | `.` on the right of a string | 265 | RightSpace | SPACE on the right of a string | 266 | SingleQuote | `'` | 267 | Slash | `/` | 268 269 To take a specific example, the FTP backend's default encoding is 270 271 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 272 273 However, let's say the FTP server is running on Windows and can't have 274 any of the invalid Windows characters in file names. You are backing 275 up Linux servers to this FTP server which do have those characters in 276 file names. So you would add the Windows set which are 277 278 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 279 280 to the existing ones, giving: 281 282 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 283 284 This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file. 285 286 Or let's say you have a Windows server but you want to preserve `＊` 287 and `？`, you would then have this as the encoding (the Windows 288 encoding minus `Asterisk` and `Question`). 289 290 Slash,LtGt,DoubleQuote,Colon,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 291 292 This can be specified using the `--local-encoding` flag or using an 293 `encoding` parameter in the config file. 294 295 ### MIME Type ### 296 297 MIME types (also known as media types) classify types of documents 298 using a simple text classification, eg `text/html` or 299 `application/pdf`. 300 301 Some cloud storage systems support reading (`R`) the MIME type of 302 objects and some support writing (`W`) the MIME type of objects. 303 304 The MIME type can be important if you are serving files directly to 305 HTTP from the storage system. 306 307 If you are copying from a remote which supports reading (`R`) to a 308 remote which supports writing (`W`) then rclone will preserve the MIME 309 types. Otherwise they will be guessed from the extension, or the 310 remote itself may assign the MIME type. 311 312 ## Optional Features ## 313 314 All the remotes support a basic set of features, but there are some 315 optional features supported by some remotes used to make some 316 operations more efficient. 317 318 | Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir | 319 | ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:| :------: | 320 | 1Fichier | No | No | No | No | No | No | No | No | No | Yes | 321 | Amazon Drive | Yes | No | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 322 | Amazon S3 | No | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 323 | Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 324 | Box | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | No | Yes | 325 | Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | 326 | Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | Yes | Yes | 327 | FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 328 | Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 329 | Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 330 | Google Photos | No | No | No | No | No | No | No | No | No | No | 331 | HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes | 332 | Hubic | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 333 | Jottacloud | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | 334 | Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 335 | Mega | Yes | No | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 336 | Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | 337 | Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 338 | Microsoft OneDrive | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | Yes | Yes | Yes | 339 | OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | 340 | Openstack Swift | Yes † | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No | 341 | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 342 | premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes | 343 | put.io | Yes | No | Yes | Yes | Yes | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 344 | QingStor | No | Yes | No | No | No | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No | 345 | SFTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 346 | SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | 347 | WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes | 348 | Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | 349 | The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 350 351 ### Purge ### 352 353 This deletes a directory quicker than just deleting all the files in 354 the directory. 355 356 † Note Swift and Hubic implement this in order to delete directory 357 markers but they don't actually have a quicker way of deleting files 358 other than deleting them individually. 359 360 ‡ StreamUpload is not supported with Nextcloud 361 362 ### Copy ### 363 364 Used when copying an object to and from the same remote. This known 365 as a server side copy so you can copy a file without downloading it 366 and uploading it again. It is used if you use `rclone copy` or 367 `rclone move` if the remote doesn't support `Move` directly. 368 369 If the server doesn't support `Copy` directly then for copy operations 370 the file is downloaded then re-uploaded. 371 372 ### Move ### 373 374 Used when moving/renaming an object on the same remote. This is known 375 as a server side move of a file. This is used in `rclone move` if the 376 server doesn't support `DirMove`. 377 378 If the server isn't capable of `Move` then rclone simulates it with 379 `Copy` then delete. If the server doesn't support `Copy` then rclone 380 will download the file and re-upload it. 381 382 ### DirMove ### 383 384 This is used to implement `rclone move` to move a directory if 385 possible. If it isn't then it will use `Move` on each file (which 386 falls back to `Copy` then download and upload - see `Move` section). 387 388 ### CleanUp ### 389 390 This is used for emptying the trash for a remote by `rclone cleanup`. 391 392 If the server can't do `CleanUp` then `rclone cleanup` will return an 393 error. 394 395 ### ListR ### 396 397 The remote supports a recursive list to list all the contents beneath 398 a directory quickly. This enables the `--fast-list` flag to work. 399 See the [rclone docs](/docs/#fast-list) for more details. 400 401 ### StreamUpload ### 402 403 Some remotes allow files to be uploaded without knowing the file size 404 in advance. This allows certain operations to work without spooling the 405 file to local disk first, e.g. `rclone rcat`. 406 407 ### LinkSharing ### 408 409 Sets the necessary permissions on a file or folder and prints a link 410 that allows others to access them, even if they don't have an account 411 on the particular cloud provider. 412 413 ### About ### 414 415 This is used to fetch quota information from the remote, like bytes 416 used/free/quota and bytes used in the trash. 417 418 This is also used to return the space used, available for `rclone mount`. 419 420 If the server can't do `About` then `rclone about` will return an 421 error. 422 423 ### EmptyDir ### 424 425 The remote supports empty directories. See [Limitations](/bugs/#limitations) 426 for details. Most Object/Bucket based remotes do not support this.