github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/docs/content/crypt.md (about) 1 --- 2 title: "Crypt" 3 description: "Encryption overlay remote" 4 date: "2016-07-28" 5 --- 6 7 <i class="fa fa-lock"></i>Crypt 8 ---------------------------------------- 9 10 The `crypt` remote encrypts and decrypts another remote. 11 12 To use it first set up the underlying remote following the config 13 instructions for that remote. You can also use a local pathname 14 instead of a remote which will encrypt and decrypt from that directory 15 which might be useful for encrypting onto a USB stick for example. 16 17 First check your chosen remote is working - we'll call it 18 `remote:path` in these docs. Note that anything inside `remote:path` 19 will be encrypted and anything outside won't. This means that if you 20 are using a bucket based remote (eg S3, B2, swift) then you should 21 probably put the bucket in the remote `s3:bucket`. If you just use 22 `s3:` then rclone will make encrypted bucket names too (if using file 23 name encryption) which may or may not be what you want. 24 25 Now configure `crypt` using `rclone config`. We will call this one 26 `secret` to differentiate it from the `remote`. 27 28 ``` 29 No remotes found - make a new one 30 n) New remote 31 s) Set configuration password 32 q) Quit config 33 n/s/q> n 34 name> secret 35 Type of storage to configure. 36 Choose a number from below, or type in your own value 37 1 / Amazon Drive 38 \ "amazon cloud drive" 39 2 / Amazon S3 (also Dreamhost, Ceph, Minio) 40 \ "s3" 41 3 / Backblaze B2 42 \ "b2" 43 4 / Dropbox 44 \ "dropbox" 45 5 / Encrypt/Decrypt a remote 46 \ "crypt" 47 6 / Google Cloud Storage (this is not Google Drive) 48 \ "google cloud storage" 49 7 / Google Drive 50 \ "drive" 51 8 / Hubic 52 \ "hubic" 53 9 / Local Disk 54 \ "local" 55 10 / Microsoft OneDrive 56 \ "onedrive" 57 11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 58 \ "swift" 59 12 / SSH/SFTP Connection 60 \ "sftp" 61 13 / Yandex Disk 62 \ "yandex" 63 Storage> 5 64 Remote to encrypt/decrypt. 65 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 66 "myremote:bucket" or maybe "myremote:" (not recommended). 67 remote> remote:path 68 How to encrypt the filenames. 69 Choose a number from below, or type in your own value 70 1 / Don't encrypt the file names. Adds a ".bin" extension only. 71 \ "off" 72 2 / Encrypt the filenames see the docs for the details. 73 \ "standard" 74 3 / Very simple filename obfuscation. 75 \ "obfuscate" 76 filename_encryption> 2 77 Option to either encrypt directory names or leave them intact. 78 Choose a number from below, or type in your own value 79 1 / Encrypt directory names. 80 \ "true" 81 2 / Don't encrypt directory names, leave them intact. 82 \ "false" 83 filename_encryption> 1 84 Password or pass phrase for encryption. 85 y) Yes type in my own password 86 g) Generate random password 87 y/g> y 88 Enter the password: 89 password: 90 Confirm the password: 91 password: 92 Password or pass phrase for salt. Optional but recommended. 93 Should be different to the previous password. 94 y) Yes type in my own password 95 g) Generate random password 96 n) No leave this optional password blank 97 y/g/n> g 98 Password strength in bits. 99 64 is just about memorable 100 128 is secure 101 1024 is the maximum 102 Bits> 128 103 Your password is: JAsJvRcgR-_veXNfy_sGmQ 104 Use this password? 105 y) Yes 106 n) No 107 y/n> y 108 Remote config 109 -------------------- 110 [secret] 111 remote = remote:path 112 filename_encryption = standard 113 password = *** ENCRYPTED *** 114 password2 = *** ENCRYPTED *** 115 -------------------- 116 y) Yes this is OK 117 e) Edit this remote 118 d) Delete this remote 119 y/e/d> y 120 ``` 121 122 **Important** The password is stored in the config file is lightly 123 obscured so it isn't immediately obvious what it is. It is in no way 124 secure unless you use config file encryption. 125 126 A long passphrase is recommended, or you can use a random one. Note 127 that if you reconfigure rclone with the same passwords/passphrases 128 elsewhere it will be compatible - all the secrets used are derived 129 from those two passwords/passphrases. 130 131 Note that rclone does not encrypt 132 133 * file length - this can be calcuated within 16 bytes 134 * modification time - used for syncing 135 136 ## Specifying the remote ## 137 138 In normal use, make sure the remote has a `:` in. If you specify the 139 remote without a `:` then rclone will use a local directory of that 140 name. So if you use a remote of `/path/to/secret/files` then rclone 141 will encrypt stuff to that directory. If you use a remote of `name` 142 then rclone will put files in a directory called `name` in the current 143 directory. 144 145 If you specify the remote as `remote:path/to/dir` then rclone will 146 store encrypted files in `path/to/dir` on the remote. If you are using 147 file name encryption, then when you save files to 148 `secret:subdir/subfile` this will store them in the unencrypted path 149 `path/to/dir` but the `subdir/subpath` bit will be encrypted. 150 151 Note that unless you want encrypted bucket names (which are difficult 152 to manage because you won't know what directory they represent in web 153 interfaces etc), you should probably specify a bucket, eg 154 `remote:secretbucket` when using bucket based remotes such as S3, 155 Swift, Hubic, B2, GCS. 156 157 ## Example ## 158 159 To test I made a little directory of files using "standard" file name 160 encryption. 161 162 ``` 163 plaintext/ 164 ├── file0.txt 165 ├── file1.txt 166 └── subdir 167 ├── file2.txt 168 ├── file3.txt 169 └── subsubdir 170 └── file4.txt 171 ``` 172 173 Copy these to the remote and list them back 174 175 ``` 176 $ rclone -q copy plaintext secret: 177 $ rclone -q ls secret: 178 7 file1.txt 179 6 file0.txt 180 8 subdir/file2.txt 181 10 subdir/subsubdir/file4.txt 182 9 subdir/file3.txt 183 ``` 184 185 Now see what that looked like when encrypted 186 187 ``` 188 $ rclone -q ls remote:path 189 55 hagjclgavj2mbiqm6u6cnjjqcg 190 54 v05749mltvv1tf4onltun46gls 191 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 192 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 193 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 194 ``` 195 196 Note that this retains the directory structure which means you can do this 197 198 ``` 199 $ rclone -q ls secret:subdir 200 8 file2.txt 201 9 file3.txt 202 10 subsubdir/file4.txt 203 ``` 204 205 If don't use file name encryption then the remote will look like this 206 - note the `.bin` extensions added to prevent the cloud provider 207 attempting to interpret the data. 208 209 ``` 210 $ rclone -q ls remote:path 211 54 file0.txt.bin 212 57 subdir/file3.txt.bin 213 56 subdir/file2.txt.bin 214 58 subdir/subsubdir/file4.txt.bin 215 55 file1.txt.bin 216 ``` 217 218 ### File name encryption modes ### 219 220 Here are some of the features of the file name encryption modes 221 222 Off 223 224 * doesn't hide file names or directory structure 225 * allows for longer file names (~246 characters) 226 * can use sub paths and copy single files 227 228 Standard 229 230 * file names encrypted 231 * file names can't be as long (~143 characters) 232 * can use sub paths and copy single files 233 * directory structure visible 234 * identical files names will have identical uploaded names 235 * can use shortcuts to shorten the directory recursion 236 237 Obfuscation 238 239 This is a simple "rotate" of the filename, with each file having a rot 240 distance based on the filename. We store the distance at the beginning 241 of the filename. So a file called "hello" may become "53.jgnnq" 242 243 This is not a strong encryption of filenames, but it may stop automated 244 scanning tools from picking up on filename patterns. As such it's an 245 intermediate between "off" and "standard". The advantage is that it 246 allows for longer path segment names. 247 248 There is a possibility with some unicode based filenames that the 249 obfuscation is weak and may map lower case characters to upper case 250 equivalents. You can not rely on this for strong protection. 251 252 * file names very lightly obfuscated 253 * file names can be longer than standard encryption 254 * can use sub paths and copy single files 255 * directory structure visible 256 * identical files names will have identical uploaded names 257 258 Cloud storage systems have various limits on file name length and 259 total path length which you are more likely to hit using "Standard" 260 file name encryption. If you keep your file names to below 156 261 characters in length then you should be OK on all providers. 262 263 There may be an even more secure file name encryption mode in the 264 future which will address the long file name problem. 265 266 ### Directory name encryption ### 267 Crypt offers the option of encrypting dir names or leaving them intact. 268 There are two options: 269 270 True 271 272 Encrypts the whole file path including directory names 273 Example: 274 `1/12/123.txt` is encrypted to 275 `p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0` 276 277 False 278 279 Only encrypts file names, skips directory names 280 Example: 281 `1/12/123.txt` is encrypted to 282 `1/12/qgm4avr35m5loi1th53ato71v0` 283 284 285 ### Modified time and hashes ### 286 287 Crypt stores modification times using the underlying remote so support 288 depends on that. 289 290 Hashes are not stored for crypt. However the data integrity is 291 protected by an extremely strong crypto authenticator. 292 293 Note that you should use the `rclone cryptcheck` command to check the 294 integrity of a crypted remote instead of `rclone check` which can't 295 check the checksums properly. 296 297 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/crypt/crypt.go then run make backenddocs --> 298 ### Standard Options 299 300 Here are the standard options specific to crypt (Encrypt/Decrypt a remote). 301 302 #### --crypt-remote 303 304 Remote to encrypt/decrypt. 305 Normally should contain a ':' and a path, eg "myremote:path/to/dir", 306 "myremote:bucket" or maybe "myremote:" (not recommended). 307 308 - Config: remote 309 - Env Var: RCLONE_CRYPT_REMOTE 310 - Type: string 311 - Default: "" 312 313 #### --crypt-filename-encryption 314 315 How to encrypt the filenames. 316 317 - Config: filename_encryption 318 - Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION 319 - Type: string 320 - Default: "standard" 321 - Examples: 322 - "off" 323 - Don't encrypt the file names. Adds a ".bin" extension only. 324 - "standard" 325 - Encrypt the filenames see the docs for the details. 326 - "obfuscate" 327 - Very simple filename obfuscation. 328 329 #### --crypt-directory-name-encryption 330 331 Option to either encrypt directory names or leave them intact. 332 333 - Config: directory_name_encryption 334 - Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION 335 - Type: bool 336 - Default: true 337 - Examples: 338 - "true" 339 - Encrypt directory names. 340 - "false" 341 - Don't encrypt directory names, leave them intact. 342 343 #### --crypt-password 344 345 Password or pass phrase for encryption. 346 347 - Config: password 348 - Env Var: RCLONE_CRYPT_PASSWORD 349 - Type: string 350 - Default: "" 351 352 #### --crypt-password2 353 354 Password or pass phrase for salt. Optional but recommended. 355 Should be different to the previous password. 356 357 - Config: password2 358 - Env Var: RCLONE_CRYPT_PASSWORD2 359 - Type: string 360 - Default: "" 361 362 ### Advanced Options 363 364 Here are the advanced options specific to crypt (Encrypt/Decrypt a remote). 365 366 #### --crypt-show-mapping 367 368 For all files listed show how the names encrypt. 369 370 If this flag is set then for each file that the remote is asked to 371 list, it will log (at level INFO) a line stating the decrypted file 372 name and the encrypted file name. 373 374 This is so you can work out which encrypted names are which decrypted 375 names just in case you need to do something with the encrypted file 376 names, or for debugging purposes. 377 378 - Config: show_mapping 379 - Env Var: RCLONE_CRYPT_SHOW_MAPPING 380 - Type: bool 381 - Default: false 382 383 <!--- autogenerated options stop --> 384 385 ## Backing up a crypted remote ## 386 387 If you wish to backup a crypted remote, it it recommended that you use 388 `rclone sync` on the encrypted files, and make sure the passwords are 389 the same in the new encrypted remote. 390 391 This will have the following advantages 392 393 * `rclone sync` will check the checksums while copying 394 * you can use `rclone check` between the encrypted remotes 395 * you don't decrypt and encrypt unnecessarily 396 397 For example, let's say you have your original remote at `remote:` with 398 the encrypted version at `eremote:` with path `remote:crypt`. You 399 would then set up the new remote `remote2:` and then the encrypted 400 version `eremote2:` with path `remote2:crypt` using the same passwords 401 as `eremote:`. 402 403 To sync the two remotes you would do 404 405 rclone sync remote:crypt remote2:crypt 406 407 And to check the integrity you would do 408 409 rclone check remote:crypt remote2:crypt 410 411 ## File formats ## 412 413 ### File encryption ### 414 415 Files are encrypted 1:1 source file to destination object. The file 416 has a header and is divided into chunks. 417 418 #### Header #### 419 420 * 8 bytes magic string `RCLONE\x00\x00` 421 * 24 bytes Nonce (IV) 422 423 The initial nonce is generated from the operating systems crypto 424 strong random number generator. The nonce is incremented for each 425 chunk read making sure each nonce is unique for each block written. 426 The chance of a nonce being re-used is minuscule. If you wrote an 427 exabyte of data (10¹⁸ bytes) you would have a probability of 428 approximately 2×10⁻³² of re-using a nonce. 429 430 #### Chunk #### 431 432 Each chunk will contain 64kB of data, except for the last one which 433 may have less data. The data chunk is in standard NACL secretbox 434 format. Secretbox uses XSalsa20 and Poly1305 to encrypt and 435 authenticate messages. 436 437 Each chunk contains: 438 439 * 16 Bytes of Poly1305 authenticator 440 * 1 - 65536 bytes XSalsa20 encrypted data 441 442 64k chunk size was chosen as the best performing chunk size (the 443 authenticator takes too much time below this and the performance drops 444 off due to cache effects above this). Note that these chunks are 445 buffered in memory so they can't be too big. 446 447 This uses a 32 byte (256 bit key) key derived from the user password. 448 449 #### Examples #### 450 451 1 byte file will encrypt to 452 453 * 32 bytes header 454 * 17 bytes data chunk 455 456 49 bytes total 457 458 1MB (1048576 bytes) file will encrypt to 459 460 * 32 bytes header 461 * 16 chunks of 65568 bytes 462 463 1049120 bytes total (a 0.05% overhead). This is the overhead for big 464 files. 465 466 ### Name encryption ### 467 468 File names are encrypted segment by segment - the path is broken up 469 into `/` separated strings and these are encrypted individually. 470 471 File segments are padded using using PKCS#7 to a multiple of 16 bytes 472 before encryption. 473 474 They are then encrypted with EME using AES with 256 bit key. EME 475 (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003 476 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway. 477 478 This makes for deterministic encryption which is what we want - the 479 same filename must encrypt to the same thing otherwise we can't find 480 it on the cloud storage system. 481 482 This means that 483 484 * filenames with the same name will encrypt the same 485 * filenames which start the same won't have a common prefix 486 487 This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of 488 which are derived from the user password. 489 490 After encryption they are written out using a modified version of 491 standard `base32` encoding as described in RFC4648. The standard 492 encoding is modified in two ways: 493 494 * it becomes lower case (no-one likes upper case filenames!) 495 * we strip the padding character `=` 496 497 `base32` is used rather than the more efficient `base64` so rclone can be 498 used on case insensitive remotes (eg Windows, Amazon Drive). 499 500 ### Key derivation ### 501 502 Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an 503 optional user supplied salt (password2) to derive the 32+32+16 = 80 504 bytes of key material required. If the user doesn't supply a salt 505 then rclone uses an internal one. 506 507 `scrypt` makes it impractical to mount a dictionary attack on rclone 508 encrypted data. For full protection against this you should always use 509 a salt.