github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/docs/content/swift.md (about) 1 --- 2 title: "Swift" 3 description: "Swift" 4 date: "2014-04-26" 5 --- 6 7 <i class="fa fa-space-shuttle"></i>Swift 8 ---------------------------------------- 9 10 Swift refers to [Openstack Object Storage](https://docs.openstack.org/swift/latest/). 11 Commercial implementations of that being: 12 13 * [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/) 14 * [Memset Memstore](https://www.memset.com/cloud/storage/) 15 * [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/) 16 * [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc) 17 * [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html) 18 19 Paths are specified as `remote:container` (or `remote:` for the `lsd` 20 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 21 22 Here is an example of making a swift configuration. First run 23 24 rclone config 25 26 This will guide you through an interactive setup process. 27 28 ``` 29 No remotes found - make a new one 30 n) New remote 31 s) Set configuration password 32 q) Quit config 33 n/s/q> n 34 name> remote 35 Type of storage to configure. 36 Choose a number from below, or type in your own value 37 1 / Amazon Drive 38 \ "amazon cloud drive" 39 2 / Amazon S3 (also Dreamhost, Ceph, Minio) 40 \ "s3" 41 3 / Backblaze B2 42 \ "b2" 43 4 / Box 44 \ "box" 45 5 / Cache a remote 46 \ "cache" 47 6 / Dropbox 48 \ "dropbox" 49 7 / Encrypt/Decrypt a remote 50 \ "crypt" 51 8 / FTP Connection 52 \ "ftp" 53 9 / Google Cloud Storage (this is not Google Drive) 54 \ "google cloud storage" 55 10 / Google Drive 56 \ "drive" 57 11 / Hubic 58 \ "hubic" 59 12 / Local Disk 60 \ "local" 61 13 / Microsoft Azure Blob Storage 62 \ "azureblob" 63 14 / Microsoft OneDrive 64 \ "onedrive" 65 15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 66 \ "swift" 67 16 / Pcloud 68 \ "pcloud" 69 17 / QingCloud Object Storage 70 \ "qingstor" 71 18 / SSH/SFTP Connection 72 \ "sftp" 73 19 / Webdav 74 \ "webdav" 75 20 / Yandex Disk 76 \ "yandex" 77 21 / http Connection 78 \ "http" 79 Storage> swift 80 Get swift credentials from environment variables in standard OpenStack form. 81 Choose a number from below, or type in your own value 82 1 / Enter swift credentials in the next step 83 \ "false" 84 2 / Get swift credentials from environment vars. Leave other fields blank if using this. 85 \ "true" 86 env_auth> true 87 User name to log in (OS_USERNAME). 88 user> 89 API key or password (OS_PASSWORD). 90 key> 91 Authentication URL for server (OS_AUTH_URL). 92 Choose a number from below, or type in your own value 93 1 / Rackspace US 94 \ "https://auth.api.rackspacecloud.com/v1.0" 95 2 / Rackspace UK 96 \ "https://lon.auth.api.rackspacecloud.com/v1.0" 97 3 / Rackspace v2 98 \ "https://identity.api.rackspacecloud.com/v2.0" 99 4 / Memset Memstore UK 100 \ "https://auth.storage.memset.com/v1.0" 101 5 / Memset Memstore UK v2 102 \ "https://auth.storage.memset.com/v2.0" 103 6 / OVH 104 \ "https://auth.cloud.ovh.net/v2.0" 105 auth> 106 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 107 user_id> 108 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 109 domain> 110 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 111 tenant> 112 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 113 tenant_id> 114 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 115 tenant_domain> 116 Region name - optional (OS_REGION_NAME) 117 region> 118 Storage URL - optional (OS_STORAGE_URL) 119 storage_url> 120 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 121 auth_token> 122 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 123 auth_version> 124 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 125 Choose a number from below, or type in your own value 126 1 / Public (default, choose this if not sure) 127 \ "public" 128 2 / Internal (use internal service net) 129 \ "internal" 130 3 / Admin 131 \ "admin" 132 endpoint_type> 133 Remote config 134 -------------------- 135 [test] 136 env_auth = true 137 user = 138 key = 139 auth = 140 user_id = 141 domain = 142 tenant = 143 tenant_id = 144 tenant_domain = 145 region = 146 storage_url = 147 auth_token = 148 auth_version = 149 endpoint_type = 150 -------------------- 151 y) Yes this is OK 152 e) Edit this remote 153 d) Delete this remote 154 y/e/d> y 155 ``` 156 157 This remote is called `remote` and can now be used like this 158 159 See all containers 160 161 rclone lsd remote: 162 163 Make a new container 164 165 rclone mkdir remote:container 166 167 List the contents of a container 168 169 rclone ls remote:container 170 171 Sync `/home/local/directory` to the remote container, deleting any 172 excess files in the container. 173 174 rclone sync /home/local/directory remote:container 175 176 ### Configuration from an OpenStack credentials file ### 177 178 An OpenStack credentials file typically looks something something 179 like this (without the comments) 180 181 ``` 182 export OS_AUTH_URL=https://a.provider.net/v2.0 183 export OS_TENANT_ID=ffffffffffffffffffffffffffffffff 184 export OS_TENANT_NAME="1234567890123456" 185 export OS_USERNAME="123abc567xy" 186 echo "Please enter your OpenStack Password: " 187 read -sr OS_PASSWORD_INPUT 188 export OS_PASSWORD=$OS_PASSWORD_INPUT 189 export OS_REGION_NAME="SBG1" 190 if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi 191 ``` 192 193 The config file needs to look something like this where `$OS_USERNAME` 194 represents the value of the `OS_USERNAME` variable - `123abc567xy` in 195 the example above. 196 197 ``` 198 [remote] 199 type = swift 200 user = $OS_USERNAME 201 key = $OS_PASSWORD 202 auth = $OS_AUTH_URL 203 tenant = $OS_TENANT_NAME 204 ``` 205 206 Note that you may (or may not) need to set `region` too - try without first. 207 208 ### Configuration from the environment ### 209 210 If you prefer you can configure rclone to use swift using a standard 211 set of OpenStack environment variables. 212 213 When you run through the config, make sure you choose `true` for 214 `env_auth` and leave everything else blank. 215 216 rclone will then set any empty config parameters from the environment 217 using standard OpenStack environment variables. There is [a list of 218 the 219 variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment) 220 in the docs for the swift library. 221 222 ### Using an alternate authentication method ### 223 224 If your OpenStack installation uses a non-standard authentication method 225 that might not be yet supported by rclone or the underlying swift library, 226 you can authenticate externally (e.g. calling manually the `openstack` 227 commands to get a token). Then, you just need to pass the two 228 configuration variables ``auth_token`` and ``storage_url``. 229 If they are both provided, the other variables are ignored. rclone will 230 not try to authenticate but instead assume it is already authenticated 231 and use these two variables to access the OpenStack installation. 232 233 #### Using rclone without a config file #### 234 235 You can use rclone with swift without a config file, if desired, like 236 this: 237 238 ``` 239 source openstack-credentials-file 240 export RCLONE_CONFIG_MYREMOTE_TYPE=swift 241 export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true 242 rclone lsd myremote: 243 ``` 244 245 ### --fast-list ### 246 247 This remote supports `--fast-list` which allows you to use fewer 248 transactions in exchange for more memory. See the [rclone 249 docs](/docs/#fast-list) for more details. 250 251 ### --update and --use-server-modtime ### 252 253 As noted below, the modified time is stored on metadata on the object. It is 254 used by default for all operations that require checking the time a file was 255 last updated. It allows rclone to treat the remote more like a true filesystem, 256 but it is inefficient because it requires an extra API call to retrieve the 257 metadata. 258 259 For many operations, the time the object was last uploaded to the remote is 260 sufficient to determine if it is "dirty". By using `--update` along with 261 `--use-server-modtime`, you can avoid the extra API call and simply upload 262 files whose local modtime is newer than the time it was last uploaded. 263 264 <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/swift/swift.go then run make backenddocs --> 265 ### Standard Options 266 267 Here are the standard options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 268 269 #### --swift-env-auth 270 271 Get swift credentials from environment variables in standard OpenStack form. 272 273 - Config: env_auth 274 - Env Var: RCLONE_SWIFT_ENV_AUTH 275 - Type: bool 276 - Default: false 277 - Examples: 278 - "false" 279 - Enter swift credentials in the next step 280 - "true" 281 - Get swift credentials from environment vars. Leave other fields blank if using this. 282 283 #### --swift-user 284 285 User name to log in (OS_USERNAME). 286 287 - Config: user 288 - Env Var: RCLONE_SWIFT_USER 289 - Type: string 290 - Default: "" 291 292 #### --swift-key 293 294 API key or password (OS_PASSWORD). 295 296 - Config: key 297 - Env Var: RCLONE_SWIFT_KEY 298 - Type: string 299 - Default: "" 300 301 #### --swift-auth 302 303 Authentication URL for server (OS_AUTH_URL). 304 305 - Config: auth 306 - Env Var: RCLONE_SWIFT_AUTH 307 - Type: string 308 - Default: "" 309 - Examples: 310 - "https://auth.api.rackspacecloud.com/v1.0" 311 - Rackspace US 312 - "https://lon.auth.api.rackspacecloud.com/v1.0" 313 - Rackspace UK 314 - "https://identity.api.rackspacecloud.com/v2.0" 315 - Rackspace v2 316 - "https://auth.storage.memset.com/v1.0" 317 - Memset Memstore UK 318 - "https://auth.storage.memset.com/v2.0" 319 - Memset Memstore UK v2 320 - "https://auth.cloud.ovh.net/v2.0" 321 - OVH 322 323 #### --swift-user-id 324 325 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 326 327 - Config: user_id 328 - Env Var: RCLONE_SWIFT_USER_ID 329 - Type: string 330 - Default: "" 331 332 #### --swift-domain 333 334 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 335 336 - Config: domain 337 - Env Var: RCLONE_SWIFT_DOMAIN 338 - Type: string 339 - Default: "" 340 341 #### --swift-tenant 342 343 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 344 345 - Config: tenant 346 - Env Var: RCLONE_SWIFT_TENANT 347 - Type: string 348 - Default: "" 349 350 #### --swift-tenant-id 351 352 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 353 354 - Config: tenant_id 355 - Env Var: RCLONE_SWIFT_TENANT_ID 356 - Type: string 357 - Default: "" 358 359 #### --swift-tenant-domain 360 361 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 362 363 - Config: tenant_domain 364 - Env Var: RCLONE_SWIFT_TENANT_DOMAIN 365 - Type: string 366 - Default: "" 367 368 #### --swift-region 369 370 Region name - optional (OS_REGION_NAME) 371 372 - Config: region 373 - Env Var: RCLONE_SWIFT_REGION 374 - Type: string 375 - Default: "" 376 377 #### --swift-storage-url 378 379 Storage URL - optional (OS_STORAGE_URL) 380 381 - Config: storage_url 382 - Env Var: RCLONE_SWIFT_STORAGE_URL 383 - Type: string 384 - Default: "" 385 386 #### --swift-auth-token 387 388 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 389 390 - Config: auth_token 391 - Env Var: RCLONE_SWIFT_AUTH_TOKEN 392 - Type: string 393 - Default: "" 394 395 #### --swift-application-credential-id 396 397 Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 398 399 - Config: application_credential_id 400 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID 401 - Type: string 402 - Default: "" 403 404 #### --swift-application-credential-name 405 406 Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 407 408 - Config: application_credential_name 409 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME 410 - Type: string 411 - Default: "" 412 413 #### --swift-application-credential-secret 414 415 Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 416 417 - Config: application_credential_secret 418 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET 419 - Type: string 420 - Default: "" 421 422 #### --swift-auth-version 423 424 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 425 426 - Config: auth_version 427 - Env Var: RCLONE_SWIFT_AUTH_VERSION 428 - Type: int 429 - Default: 0 430 431 #### --swift-endpoint-type 432 433 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 434 435 - Config: endpoint_type 436 - Env Var: RCLONE_SWIFT_ENDPOINT_TYPE 437 - Type: string 438 - Default: "public" 439 - Examples: 440 - "public" 441 - Public (default, choose this if not sure) 442 - "internal" 443 - Internal (use internal service net) 444 - "admin" 445 - Admin 446 447 #### --swift-storage-policy 448 449 The storage policy to use when creating a new container 450 451 This applies the specified storage policy when creating a new 452 container. The policy cannot be changed afterwards. The allowed 453 configuration values and their meaning depend on your Swift storage 454 provider. 455 456 - Config: storage_policy 457 - Env Var: RCLONE_SWIFT_STORAGE_POLICY 458 - Type: string 459 - Default: "" 460 - Examples: 461 - "" 462 - Default 463 - "pcs" 464 - OVH Public Cloud Storage 465 - "pca" 466 - OVH Public Cloud Archive 467 468 ### Advanced Options 469 470 Here are the advanced options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 471 472 #### --swift-chunk-size 473 474 Above this size files will be chunked into a _segments container. 475 476 Above this size files will be chunked into a _segments container. The 477 default for this is 5GB which is its maximum value. 478 479 - Config: chunk_size 480 - Env Var: RCLONE_SWIFT_CHUNK_SIZE 481 - Type: SizeSuffix 482 - Default: 5G 483 484 #### --swift-no-chunk 485 486 Don't chunk files during streaming upload. 487 488 When doing streaming uploads (eg using rcat or mount) setting this 489 flag will cause the swift backend to not upload chunked files. 490 491 This will limit the maximum upload size to 5GB. However non chunked 492 files are easier to deal with and have an MD5SUM. 493 494 Rclone will still chunk files bigger than chunk_size when doing normal 495 copy operations. 496 497 - Config: no_chunk 498 - Env Var: RCLONE_SWIFT_NO_CHUNK 499 - Type: bool 500 - Default: false 501 502 <!--- autogenerated options stop --> 503 504 ### Modified time ### 505 506 The modified time is stored as metadata on the object as 507 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 508 ns. 509 510 This is a defacto standard (used in the official python-swiftclient 511 amongst others) for storing the modification time for an object. 512 513 ### Limitations ### 514 515 The Swift API doesn't return a correct MD5SUM for segmented files 516 (Dynamic or Static Large Objects) so rclone won't check or use the 517 MD5SUM for these. 518 519 ### Troubleshooting ### 520 521 #### Rclone gives Failed to create file system for "remote:": Bad Request #### 522 523 Due to an oddity of the underlying swift library, it gives a "Bad 524 Request" error rather than a more sensible error when the 525 authentication fails for Swift. 526 527 So this most likely means your username / password is wrong. You can 528 investigate further with the `--dump-bodies` flag. 529 530 This may also be caused by specifying the region when you shouldn't 531 have (eg OVH). 532 533 #### Rclone gives Failed to create file system: Response didn't have storage storage url and auth token #### 534 535 This is most likely caused by forgetting to specify your tenant when 536 setting up a swift remote.