github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/docs/content/b2.md (about) 1 --- 2 title: "B2" 3 description: "Backblaze B2" 4 date: "2016-10-25" 5 --- 6 7 <i class="fa fa-fire"></i>Backblaze B2 8 ---------------------------------------- 9 10 B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/). 11 12 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 13 command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`. 14 15 Here is an example of making a b2 configuration. First run 16 17 rclone config 18 19 This will guide you through an interactive setup process. To authenticate 20 you will either need your Account ID (a short hex number) and Master 21 Application Key (a long hex number) OR an Application Key, which is the 22 recommended method. See below for further details on generating and using 23 an Application Key. 24 25 ``` 26 No remotes found - make a new one 27 n) New remote 28 q) Quit config 29 n/q> n 30 name> remote 31 Type of storage to configure. 32 Choose a number from below, or type in your own value 33 1 / Amazon Drive 34 \ "amazon cloud drive" 35 2 / Amazon S3 (also Dreamhost, Ceph, Minio) 36 \ "s3" 37 3 / Backblaze B2 38 \ "b2" 39 4 / Dropbox 40 \ "dropbox" 41 5 / Encrypt/Decrypt a remote 42 \ "crypt" 43 6 / Google Cloud Storage (this is not Google Drive) 44 \ "google cloud storage" 45 7 / Google Drive 46 \ "drive" 47 8 / Hubic 48 \ "hubic" 49 9 / Local Disk 50 \ "local" 51 10 / Microsoft OneDrive 52 \ "onedrive" 53 11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 54 \ "swift" 55 12 / SSH/SFTP Connection 56 \ "sftp" 57 13 / Yandex Disk 58 \ "yandex" 59 Storage> 3 60 Account ID or Application Key ID 61 account> 123456789abc 62 Application Key 63 key> 0123456789abcdef0123456789abcdef0123456789 64 Endpoint for the service - leave blank normally. 65 endpoint> 66 Remote config 67 -------------------- 68 [remote] 69 account = 123456789abc 70 key = 0123456789abcdef0123456789abcdef0123456789 71 endpoint = 72 -------------------- 73 y) Yes this is OK 74 e) Edit this remote 75 d) Delete this remote 76 y/e/d> y 77 ``` 78 79 This remote is called `remote` and can now be used like this 80 81 See all buckets 82 83 rclone lsd remote: 84 85 Create a new bucket 86 87 rclone mkdir remote:bucket 88 89 List the contents of a bucket 90 91 rclone ls remote:bucket 92 93 Sync `/home/local/directory` to the remote bucket, deleting any 94 excess files in the bucket. 95 96 rclone sync /home/local/directory remote:bucket 97 98 ### Application Keys ### 99 100 B2 supports multiple [Application Keys for different access permission 101 to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html). 102 103 You can use these with rclone too; you will need to use rclone version 1.43 104 or later. 105 106 Follow Backblaze's docs to create an Application Key with the required 107 permission and add the `applicationKeyId` as the `account` and the 108 `Application Key` itself as the `key`. 109 110 Note that you must put the _applicationKeyId_ as the `account` – you 111 can't use the master Account ID. If you try then B2 will return 401 112 errors. 113 114 ### --fast-list ### 115 116 This remote supports `--fast-list` which allows you to use fewer 117 transactions in exchange for more memory. See the [rclone 118 docs](/docs/#fast-list) for more details. 119 120 ### Modified time ### 121 122 The modified time is stored as metadata on the object as 123 `X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01 124 in the Backblaze standard. Other tools should be able to use this as 125 a modified time. 126 127 Modified times are used in syncing and are fully supported. Note that 128 if a modification time needs to be updated on an object then it will 129 create a new version of the object. 130 131 ### SHA1 checksums ### 132 133 The SHA1 checksums of the files are checked on upload and download and 134 will be used in the syncing process. 135 136 Large files (bigger than the limit in `--b2-upload-cutoff`) which are 137 uploaded in chunks will store their SHA1 on the object as 138 `X-Bz-Info-large_file_sha1` as recommended by Backblaze. 139 140 For a large file to be uploaded with an SHA1 checksum, the source 141 needs to support SHA1 checksums. The local disk supports SHA1 142 checksums so large file transfers from local disk will have an SHA1. 143 See [the overview](/overview/#features) for exactly which remotes 144 support SHA1. 145 146 Sources which don't support SHA1, in particular `crypt` will upload 147 large files without SHA1 checksums. This may be fixed in the future 148 (see [#1767](https://github.com/ncw/rclone/issues/1767)). 149 150 Files sizes below `--b2-upload-cutoff` will always have an SHA1 151 regardless of the source. 152 153 ### Transfers ### 154 155 Backblaze recommends that you do lots of transfers simultaneously for 156 maximum speed. In tests from my SSD equipped laptop the optimum 157 setting is about `--transfers 32` though higher numbers may be used 158 for a slight speed improvement. The optimum number for you may vary 159 depending on your hardware, how big the files are, how much you want 160 to load your computer, etc. The default of `--transfers 4` is 161 definitely too low for Backblaze B2 though. 162 163 Note that uploading big files (bigger than 200 MB by default) will use 164 a 96 MB RAM buffer by default. There can be at most `--transfers` of 165 these in use at any moment, so this sets the upper limit on the memory 166 used. 167 168 ### Versions ### 169 170 When rclone uploads a new version of a file it creates a [new version 171 of it](https://www.backblaze.com/b2/docs/file_versions.html). 172 Likewise when you delete a file, the old version will be marked hidden 173 and still be available. Conversely, you may opt in to a "hard delete" 174 of files with the `--b2-hard-delete` flag which would permanently remove 175 the file instead of hiding it. 176 177 Old versions of files, where available, are visible using the 178 `--b2-versions` flag. 179 180 **NB** Note that `--b2-versions` does not work with crypt at the 181 moment [#1627](https://github.com/ncw/rclone/issues/1627). Using 182 [--backup-dir](/docs/#backup-dir-dir) with rclone is the recommended 183 way of working around this. 184 185 If you wish to remove all the old versions then you can use the 186 `rclone cleanup remote:bucket` command which will delete all the old 187 versions of files, leaving the current ones intact. You can also 188 supply a path and only old versions under that path will be deleted, 189 eg `rclone cleanup remote:bucket/path/to/stuff`. 190 191 Note that `cleanup` will remove partially uploaded files from the bucket 192 if they are more than a day old. 193 194 When you `purge` a bucket, the current and the old versions will be 195 deleted then the bucket will be deleted. 196 197 However `delete` will cause the current versions of the files to 198 become hidden old versions. 199 200 Here is a session showing the listing and retrieval of an old 201 version followed by a `cleanup` of the old versions. 202 203 Show current version and all the versions with `--b2-versions` flag. 204 205 ``` 206 $ rclone -q ls b2:cleanup-test 207 9 one.txt 208 209 $ rclone -q --b2-versions ls b2:cleanup-test 210 9 one.txt 211 8 one-v2016-07-04-141032-000.txt 212 16 one-v2016-07-04-141003-000.txt 213 15 one-v2016-07-02-155621-000.txt 214 ``` 215 216 Retrieve an old version 217 218 ``` 219 $ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp 220 221 $ ls -l /tmp/one-v2016-07-04-141003-000.txt 222 -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt 223 ``` 224 225 Clean up all the old versions and show that they've gone. 226 227 ``` 228 $ rclone -q cleanup b2:cleanup-test 229 230 $ rclone -q ls b2:cleanup-test 231 9 one.txt 232 233 $ rclone -q --b2-versions ls b2:cleanup-test 234 9 one.txt 235 ``` 236 237 ### Data usage ### 238 239 It is useful to know how many requests are sent to the server in different scenarios. 240 241 All copy commands send the following 4 requests: 242 243 ``` 244 /b2api/v1/b2_authorize_account 245 /b2api/v1/b2_create_bucket 246 /b2api/v1/b2_list_buckets 247 /b2api/v1/b2_list_file_names 248 ``` 249 250 The `b2_list_file_names` request will be sent once for every 1k files 251 in the remote path, providing the checksum and modification time of 252 the listed files. As of version 1.33 issue 253 [#818](https://github.com/ncw/rclone/issues/818) causes extra requests 254 to be sent when using B2 with Crypt. When a copy operation does not 255 require any files to be uploaded, no more requests will be sent. 256 257 Uploading files that do not require chunking, will send 2 requests per 258 file upload: 259 260 ``` 261 /b2api/v1/b2_get_upload_url 262 /b2api/v1/b2_upload_file/ 263 ``` 264 265 Uploading files requiring chunking, will send 2 requests (one each to 266 start and finish the upload) and another 2 requests for each chunk: 267 268 ``` 269 /b2api/v1/b2_start_large_file 270 /b2api/v1/b2_get_upload_part_url 271 /b2api/v1/b2_upload_part/ 272 /b2api/v1/b2_finish_large_file 273 ``` 274 275 #### Versions #### 276 277 Versions can be viewed with the `--b2-versions` flag. When it is set 278 rclone will show and act on older versions of files. For example 279 280 Listing without `--b2-versions` 281 282 ``` 283 $ rclone -q ls b2:cleanup-test 284 9 one.txt 285 ``` 286 287 And with 288 289 ``` 290 $ rclone -q --b2-versions ls b2:cleanup-test 291 9 one.txt 292 8 one-v2016-07-04-141032-000.txt 293 16 one-v2016-07-04-141003-000.txt 294 15 one-v2016-07-02-155621-000.txt 295 ``` 296 297 Showing that the current version is unchanged but older versions can 298 be seen. These have the UTC date that they were uploaded to the 299 server to the nearest millisecond appended to them. 300 301 Note that when using `--b2-versions` no file write operations are 302 permitted, so you can't upload files or delete them. 303 304 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/b2/b2.go then run make backenddocs --> 305 ### Standard Options 306 307 Here are the standard options specific to b2 (Backblaze B2). 308 309 #### --b2-account 310 311 Account ID or Application Key ID 312 313 - Config: account 314 - Env Var: RCLONE_B2_ACCOUNT 315 - Type: string 316 - Default: "" 317 318 #### --b2-key 319 320 Application Key 321 322 - Config: key 323 - Env Var: RCLONE_B2_KEY 324 - Type: string 325 - Default: "" 326 327 #### --b2-hard-delete 328 329 Permanently delete files on remote removal, otherwise hide files. 330 331 - Config: hard_delete 332 - Env Var: RCLONE_B2_HARD_DELETE 333 - Type: bool 334 - Default: false 335 336 ### Advanced Options 337 338 Here are the advanced options specific to b2 (Backblaze B2). 339 340 #### --b2-endpoint 341 342 Endpoint for the service. 343 Leave blank normally. 344 345 - Config: endpoint 346 - Env Var: RCLONE_B2_ENDPOINT 347 - Type: string 348 - Default: "" 349 350 #### --b2-test-mode 351 352 A flag string for X-Bz-Test-Mode header for debugging. 353 354 This is for debugging purposes only. Setting it to one of the strings 355 below will cause b2 to return specific errors: 356 357 * "fail_some_uploads" 358 * "expire_some_account_authorization_tokens" 359 * "force_cap_exceeded" 360 361 These will be set in the "X-Bz-Test-Mode" header which is documented 362 in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html). 363 364 - Config: test_mode 365 - Env Var: RCLONE_B2_TEST_MODE 366 - Type: string 367 - Default: "" 368 369 #### --b2-versions 370 371 Include old versions in directory listings. 372 Note that when using this no file write operations are permitted, 373 so you can't upload files or delete them. 374 375 - Config: versions 376 - Env Var: RCLONE_B2_VERSIONS 377 - Type: bool 378 - Default: false 379 380 #### --b2-upload-cutoff 381 382 Cutoff for switching to chunked upload. 383 384 Files above this size will be uploaded in chunks of "--b2-chunk-size". 385 386 This value should be set no larger than 4.657GiB (== 5GB). 387 388 - Config: upload_cutoff 389 - Env Var: RCLONE_B2_UPLOAD_CUTOFF 390 - Type: SizeSuffix 391 - Default: 200M 392 393 #### --b2-chunk-size 394 395 Upload chunk size. Must fit in memory. 396 397 When uploading large files, chunk the file into this size. Note that 398 these chunks are buffered in memory and there might a maximum of 399 "--transfers" chunks in progress at once. 5,000,000 Bytes is the 400 minimum size. 401 402 - Config: chunk_size 403 - Env Var: RCLONE_B2_CHUNK_SIZE 404 - Type: SizeSuffix 405 - Default: 96M 406 407 #### --b2-disable-checksum 408 409 Disable checksums for large (> upload cutoff) files 410 411 - Config: disable_checksum 412 - Env Var: RCLONE_B2_DISABLE_CHECKSUM 413 - Type: bool 414 - Default: false 415 416 #### --b2-download-url 417 418 Custom endpoint for downloads. 419 420 This is usually set to a Cloudflare CDN URL as Backblaze offers 421 free egress for data downloaded through the Cloudflare network. 422 Leave blank if you want to use the endpoint provided by Backblaze. 423 424 - Config: download_url 425 - Env Var: RCLONE_B2_DOWNLOAD_URL 426 - Type: string 427 - Default: "" 428 429 <!--- autogenerated options stop --> 430