github.com/10XDev/rclone@v1.52.3-0.20200626220027-16af9ab76b2a/docs/content/swift.md (about) 1 --- 2 title: "Swift" 3 description: "Swift" 4 --- 5 6 {{< icon "fa fa-space-shuttle" >}}Swift 7 ---------------------------------------- 8 9 Swift refers to [OpenStack Object Storage](https://docs.openstack.org/swift/latest/). 10 Commercial implementations of that being: 11 12 * [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/) 13 * [Memset Memstore](https://www.memset.com/cloud/storage/) 14 * [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/) 15 * [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc) 16 * [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html) 17 18 Paths are specified as `remote:container` (or `remote:` for the `lsd` 19 command.) You may put subdirectories in too, eg `remote:container/path/to/dir`. 20 21 Here is an example of making a swift configuration. First run 22 23 rclone config 24 25 This will guide you through an interactive setup process. 26 27 ``` 28 No remotes found - make a new one 29 n) New remote 30 s) Set configuration password 31 q) Quit config 32 n/s/q> n 33 name> remote 34 Type of storage to configure. 35 Choose a number from below, or type in your own value 36 [snip] 37 XX / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 38 \ "swift" 39 [snip] 40 Storage> swift 41 Get swift credentials from environment variables in standard OpenStack form. 42 Choose a number from below, or type in your own value 43 1 / Enter swift credentials in the next step 44 \ "false" 45 2 / Get swift credentials from environment vars. Leave other fields blank if using this. 46 \ "true" 47 env_auth> true 48 User name to log in (OS_USERNAME). 49 user> 50 API key or password (OS_PASSWORD). 51 key> 52 Authentication URL for server (OS_AUTH_URL). 53 Choose a number from below, or type in your own value 54 1 / Rackspace US 55 \ "https://auth.api.rackspacecloud.com/v1.0" 56 2 / Rackspace UK 57 \ "https://lon.auth.api.rackspacecloud.com/v1.0" 58 3 / Rackspace v2 59 \ "https://identity.api.rackspacecloud.com/v2.0" 60 4 / Memset Memstore UK 61 \ "https://auth.storage.memset.com/v1.0" 62 5 / Memset Memstore UK v2 63 \ "https://auth.storage.memset.com/v2.0" 64 6 / OVH 65 \ "https://auth.cloud.ovh.net/v3" 66 auth> 67 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 68 user_id> 69 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 70 domain> 71 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 72 tenant> 73 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 74 tenant_id> 75 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 76 tenant_domain> 77 Region name - optional (OS_REGION_NAME) 78 region> 79 Storage URL - optional (OS_STORAGE_URL) 80 storage_url> 81 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 82 auth_token> 83 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 84 auth_version> 85 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 86 Choose a number from below, or type in your own value 87 1 / Public (default, choose this if not sure) 88 \ "public" 89 2 / Internal (use internal service net) 90 \ "internal" 91 3 / Admin 92 \ "admin" 93 endpoint_type> 94 Remote config 95 -------------------- 96 [test] 97 env_auth = true 98 user = 99 key = 100 auth = 101 user_id = 102 domain = 103 tenant = 104 tenant_id = 105 tenant_domain = 106 region = 107 storage_url = 108 auth_token = 109 auth_version = 110 endpoint_type = 111 -------------------- 112 y) Yes this is OK 113 e) Edit this remote 114 d) Delete this remote 115 y/e/d> y 116 ``` 117 118 This remote is called `remote` and can now be used like this 119 120 See all containers 121 122 rclone lsd remote: 123 124 Make a new container 125 126 rclone mkdir remote:container 127 128 List the contents of a container 129 130 rclone ls remote:container 131 132 Sync `/home/local/directory` to the remote container, deleting any 133 excess files in the container. 134 135 rclone sync /home/local/directory remote:container 136 137 ### Configuration from an OpenStack credentials file ### 138 139 An OpenStack credentials file typically looks something something 140 like this (without the comments) 141 142 ``` 143 export OS_AUTH_URL=https://a.provider.net/v2.0 144 export OS_TENANT_ID=ffffffffffffffffffffffffffffffff 145 export OS_TENANT_NAME="1234567890123456" 146 export OS_USERNAME="123abc567xy" 147 echo "Please enter your OpenStack Password: " 148 read -sr OS_PASSWORD_INPUT 149 export OS_PASSWORD=$OS_PASSWORD_INPUT 150 export OS_REGION_NAME="SBG1" 151 if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi 152 ``` 153 154 The config file needs to look something like this where `$OS_USERNAME` 155 represents the value of the `OS_USERNAME` variable - `123abc567xy` in 156 the example above. 157 158 ``` 159 [remote] 160 type = swift 161 user = $OS_USERNAME 162 key = $OS_PASSWORD 163 auth = $OS_AUTH_URL 164 tenant = $OS_TENANT_NAME 165 ``` 166 167 Note that you may (or may not) need to set `region` too - try without first. 168 169 ### Configuration from the environment ### 170 171 If you prefer you can configure rclone to use swift using a standard 172 set of OpenStack environment variables. 173 174 When you run through the config, make sure you choose `true` for 175 `env_auth` and leave everything else blank. 176 177 rclone will then set any empty config parameters from the environment 178 using standard OpenStack environment variables. There is [a list of 179 the 180 variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment) 181 in the docs for the swift library. 182 183 ### Using an alternate authentication method ### 184 185 If your OpenStack installation uses a non-standard authentication method 186 that might not be yet supported by rclone or the underlying swift library, 187 you can authenticate externally (e.g. calling manually the `openstack` 188 commands to get a token). Then, you just need to pass the two 189 configuration variables ``auth_token`` and ``storage_url``. 190 If they are both provided, the other variables are ignored. rclone will 191 not try to authenticate but instead assume it is already authenticated 192 and use these two variables to access the OpenStack installation. 193 194 #### Using rclone without a config file #### 195 196 You can use rclone with swift without a config file, if desired, like 197 this: 198 199 ``` 200 source openstack-credentials-file 201 export RCLONE_CONFIG_MYREMOTE_TYPE=swift 202 export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true 203 rclone lsd myremote: 204 ``` 205 206 ### --fast-list ### 207 208 This remote supports `--fast-list` which allows you to use fewer 209 transactions in exchange for more memory. See the [rclone 210 docs](/docs/#fast-list) for more details. 211 212 ### --update and --use-server-modtime ### 213 214 As noted below, the modified time is stored on metadata on the object. It is 215 used by default for all operations that require checking the time a file was 216 last updated. It allows rclone to treat the remote more like a true filesystem, 217 but it is inefficient because it requires an extra API call to retrieve the 218 metadata. 219 220 For many operations, the time the object was last uploaded to the remote is 221 sufficient to determine if it is "dirty". By using `--update` along with 222 `--use-server-modtime`, you can avoid the extra API call and simply upload 223 files whose local modtime is newer than the time it was last uploaded. 224 225 {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/swift/swift.go then run make backenddocs" >}} 226 ### Standard Options 227 228 Here are the standard options specific to swift (OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 229 230 #### --swift-env-auth 231 232 Get swift credentials from environment variables in standard OpenStack form. 233 234 - Config: env_auth 235 - Env Var: RCLONE_SWIFT_ENV_AUTH 236 - Type: bool 237 - Default: false 238 - Examples: 239 - "false" 240 - Enter swift credentials in the next step 241 - "true" 242 - Get swift credentials from environment vars. Leave other fields blank if using this. 243 244 #### --swift-user 245 246 User name to log in (OS_USERNAME). 247 248 - Config: user 249 - Env Var: RCLONE_SWIFT_USER 250 - Type: string 251 - Default: "" 252 253 #### --swift-key 254 255 API key or password (OS_PASSWORD). 256 257 - Config: key 258 - Env Var: RCLONE_SWIFT_KEY 259 - Type: string 260 - Default: "" 261 262 #### --swift-auth 263 264 Authentication URL for server (OS_AUTH_URL). 265 266 - Config: auth 267 - Env Var: RCLONE_SWIFT_AUTH 268 - Type: string 269 - Default: "" 270 - Examples: 271 - "https://auth.api.rackspacecloud.com/v1.0" 272 - Rackspace US 273 - "https://lon.auth.api.rackspacecloud.com/v1.0" 274 - Rackspace UK 275 - "https://identity.api.rackspacecloud.com/v2.0" 276 - Rackspace v2 277 - "https://auth.storage.memset.com/v1.0" 278 - Memset Memstore UK 279 - "https://auth.storage.memset.com/v2.0" 280 - Memset Memstore UK v2 281 - "https://auth.cloud.ovh.net/v3" 282 - OVH 283 284 #### --swift-user-id 285 286 User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). 287 288 - Config: user_id 289 - Env Var: RCLONE_SWIFT_USER_ID 290 - Type: string 291 - Default: "" 292 293 #### --swift-domain 294 295 User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME) 296 297 - Config: domain 298 - Env Var: RCLONE_SWIFT_DOMAIN 299 - Type: string 300 - Default: "" 301 302 #### --swift-tenant 303 304 Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME) 305 306 - Config: tenant 307 - Env Var: RCLONE_SWIFT_TENANT 308 - Type: string 309 - Default: "" 310 311 #### --swift-tenant-id 312 313 Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) 314 315 - Config: tenant_id 316 - Env Var: RCLONE_SWIFT_TENANT_ID 317 - Type: string 318 - Default: "" 319 320 #### --swift-tenant-domain 321 322 Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME) 323 324 - Config: tenant_domain 325 - Env Var: RCLONE_SWIFT_TENANT_DOMAIN 326 - Type: string 327 - Default: "" 328 329 #### --swift-region 330 331 Region name - optional (OS_REGION_NAME) 332 333 - Config: region 334 - Env Var: RCLONE_SWIFT_REGION 335 - Type: string 336 - Default: "" 337 338 #### --swift-storage-url 339 340 Storage URL - optional (OS_STORAGE_URL) 341 342 - Config: storage_url 343 - Env Var: RCLONE_SWIFT_STORAGE_URL 344 - Type: string 345 - Default: "" 346 347 #### --swift-auth-token 348 349 Auth Token from alternate authentication - optional (OS_AUTH_TOKEN) 350 351 - Config: auth_token 352 - Env Var: RCLONE_SWIFT_AUTH_TOKEN 353 - Type: string 354 - Default: "" 355 356 #### --swift-application-credential-id 357 358 Application Credential ID (OS_APPLICATION_CREDENTIAL_ID) 359 360 - Config: application_credential_id 361 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID 362 - Type: string 363 - Default: "" 364 365 #### --swift-application-credential-name 366 367 Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME) 368 369 - Config: application_credential_name 370 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME 371 - Type: string 372 - Default: "" 373 374 #### --swift-application-credential-secret 375 376 Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET) 377 378 - Config: application_credential_secret 379 - Env Var: RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET 380 - Type: string 381 - Default: "" 382 383 #### --swift-auth-version 384 385 AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION) 386 387 - Config: auth_version 388 - Env Var: RCLONE_SWIFT_AUTH_VERSION 389 - Type: int 390 - Default: 0 391 392 #### --swift-endpoint-type 393 394 Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) 395 396 - Config: endpoint_type 397 - Env Var: RCLONE_SWIFT_ENDPOINT_TYPE 398 - Type: string 399 - Default: "public" 400 - Examples: 401 - "public" 402 - Public (default, choose this if not sure) 403 - "internal" 404 - Internal (use internal service net) 405 - "admin" 406 - Admin 407 408 #### --swift-storage-policy 409 410 The storage policy to use when creating a new container 411 412 This applies the specified storage policy when creating a new 413 container. The policy cannot be changed afterwards. The allowed 414 configuration values and their meaning depend on your Swift storage 415 provider. 416 417 - Config: storage_policy 418 - Env Var: RCLONE_SWIFT_STORAGE_POLICY 419 - Type: string 420 - Default: "" 421 - Examples: 422 - "" 423 - Default 424 - "pcs" 425 - OVH Public Cloud Storage 426 - "pca" 427 - OVH Public Cloud Archive 428 429 ### Advanced Options 430 431 Here are the advanced options specific to swift (OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)). 432 433 #### --swift-chunk-size 434 435 Above this size files will be chunked into a _segments container. 436 437 Above this size files will be chunked into a _segments container. The 438 default for this is 5GB which is its maximum value. 439 440 - Config: chunk_size 441 - Env Var: RCLONE_SWIFT_CHUNK_SIZE 442 - Type: SizeSuffix 443 - Default: 5G 444 445 #### --swift-no-chunk 446 447 Don't chunk files during streaming upload. 448 449 When doing streaming uploads (eg using rcat or mount) setting this 450 flag will cause the swift backend to not upload chunked files. 451 452 This will limit the maximum upload size to 5GB. However non chunked 453 files are easier to deal with and have an MD5SUM. 454 455 Rclone will still chunk files bigger than chunk_size when doing normal 456 copy operations. 457 458 - Config: no_chunk 459 - Env Var: RCLONE_SWIFT_NO_CHUNK 460 - Type: bool 461 - Default: false 462 463 #### --swift-encoding 464 465 This sets the encoding for the backend. 466 467 See: the [encoding section in the overview](/overview/#encoding) for more info. 468 469 - Config: encoding 470 - Env Var: RCLONE_SWIFT_ENCODING 471 - Type: MultiEncoder 472 - Default: Slash,InvalidUtf8 473 474 {{< rem autogenerated options stop >}} 475 476 ### Modified time ### 477 478 The modified time is stored as metadata on the object as 479 `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1 480 ns. 481 482 This is a de facto standard (used in the official python-swiftclient 483 amongst others) for storing the modification time for an object. 484 485 ### Restricted filename characters 486 487 | Character | Value | Replacement | 488 | --------- |:-----:|:-----------:| 489 | NUL | 0x00 | ␀ | 490 | / | 0x2F | ／ | 491 492 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 493 as they can't be used in JSON strings. 494 495 ### Limitations ### 496 497 The Swift API doesn't return a correct MD5SUM for segmented files 498 (Dynamic or Static Large Objects) so rclone won't check or use the 499 MD5SUM for these. 500 501 ### Troubleshooting ### 502 503 #### Rclone gives Failed to create file system for "remote:": Bad Request #### 504 505 Due to an oddity of the underlying swift library, it gives a "Bad 506 Request" error rather than a more sensible error when the 507 authentication fails for Swift. 508 509 So this most likely means your username / password is wrong. You can 510 investigate further with the `--dump-bodies` flag. 511 512 This may also be caused by specifying the region when you shouldn't 513 have (eg OVH). 514 515 #### Rclone gives Failed to create file system: Response didn't have storage url and auth token #### 516 517 This is most likely caused by forgetting to specify your tenant when 518 setting up a swift remote.