github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/docs/content/swift.md (about)

     1  ---
     2  title: "Swift"
     3  description: "Swift"
     4  date: "2014-04-26"
     5  ---
     6  
     7  <i class="fa fa-space-shuttle"></i>Swift
     8  ----------------------------------------
     9  
    10  Swift refers to [Openstack Object Storage](https://docs.openstack.org/swift/latest/).
    11  Commercial implementations of that being:
    12  
    13    * [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/)
    14    * [Memset Memstore](https://www.memset.com/cloud/storage/)
    15    * [OVH Object Storage](https://www.ovh.co.uk/public-cloud/storage/object-storage/)
    16    * [Oracle Cloud Storage](https://cloud.oracle.com/storage-opc)
    17    * [IBM Bluemix Cloud ObjectStorage Swift](https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html)
    18  
    19  Paths are specified as `remote:container` (or `remote:` for the `lsd`
    20  command.)  You may put subdirectories in too, eg `remote:container/path/to/dir`.
    21  
    22  Here is an example of making a swift configuration.  First run
    23  
    24      rclone config
    25  
    26  This will guide you through an interactive setup process.
    27  
    28  ```
    29  No remotes found - make a new one
    30  n) New remote
    31  s) Set configuration password
    32  q) Quit config
    33  n/s/q> n
    34  name> remote
    35  Type of storage to configure.
    36  Choose a number from below, or type in your own value
    37   1 / Amazon Drive
    38     \ "amazon cloud drive"
    39   2 / Amazon S3 (also Dreamhost, Ceph, Minio)
    40     \ "s3"
    41   3 / Backblaze B2
    42     \ "b2"
    43   4 / Box
    44     \ "box"
    45   5 / Cache a remote
    46     \ "cache"
    47   6 / Dropbox
    48     \ "dropbox"
    49   7 / Encrypt/Decrypt a remote
    50     \ "crypt"
    51   8 / FTP Connection
    52     \ "ftp"
    53   9 / Google Cloud Storage (this is not Google Drive)
    54     \ "google cloud storage"
    55  10 / Google Drive
    56     \ "drive"
    57  11 / Hubic
    58     \ "hubic"
    59  12 / Local Disk
    60     \ "local"
    61  13 / Microsoft Azure Blob Storage
    62     \ "azureblob"
    63  14 / Microsoft OneDrive
    64     \ "onedrive"
    65  15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
    66     \ "swift"
    67  16 / Pcloud
    68     \ "pcloud"
    69  17 / QingCloud Object Storage
    70     \ "qingstor"
    71  18 / SSH/SFTP Connection
    72     \ "sftp"
    73  19 / Webdav
    74     \ "webdav"
    75  20 / Yandex Disk
    76     \ "yandex"
    77  21 / http Connection
    78     \ "http"
    79  Storage> swift
    80  Get swift credentials from environment variables in standard OpenStack form.
    81  Choose a number from below, or type in your own value
    82   1 / Enter swift credentials in the next step
    83     \ "false"
    84   2 / Get swift credentials from environment vars. Leave other fields blank if using this.
    85     \ "true"
    86  env_auth> true
    87  User name to log in (OS_USERNAME).
    88  user> 
    89  API key or password (OS_PASSWORD).
    90  key> 
    91  Authentication URL for server (OS_AUTH_URL).
    92  Choose a number from below, or type in your own value
    93   1 / Rackspace US
    94     \ "https://auth.api.rackspacecloud.com/v1.0"
    95   2 / Rackspace UK
    96     \ "https://lon.auth.api.rackspacecloud.com/v1.0"
    97   3 / Rackspace v2
    98     \ "https://identity.api.rackspacecloud.com/v2.0"
    99   4 / Memset Memstore UK
   100     \ "https://auth.storage.memset.com/v1.0"
   101   5 / Memset Memstore UK v2
   102     \ "https://auth.storage.memset.com/v2.0"
   103   6 / OVH
   104     \ "https://auth.cloud.ovh.net/v2.0"
   105  auth> 
   106  User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).
   107  user_id> 
   108  User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
   109  domain> 
   110  Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
   111  tenant> 
   112  Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
   113  tenant_id> 
   114  Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
   115  tenant_domain> 
   116  Region name - optional (OS_REGION_NAME)
   117  region> 
   118  Storage URL - optional (OS_STORAGE_URL)
   119  storage_url> 
   120  Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
   121  auth_token> 
   122  AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
   123  auth_version> 
   124  Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE)
   125  Choose a number from below, or type in your own value
   126   1 / Public (default, choose this if not sure)
   127     \ "public"
   128   2 / Internal (use internal service net)
   129     \ "internal"
   130   3 / Admin
   131     \ "admin"
   132  endpoint_type> 
   133  Remote config
   134  --------------------
   135  [test]
   136  env_auth = true
   137  user = 
   138  key = 
   139  auth = 
   140  user_id = 
   141  domain = 
   142  tenant = 
   143  tenant_id = 
   144  tenant_domain = 
   145  region = 
   146  storage_url = 
   147  auth_token = 
   148  auth_version = 
   149  endpoint_type = 
   150  --------------------
   151  y) Yes this is OK
   152  e) Edit this remote
   153  d) Delete this remote
   154  y/e/d> y
   155  ```
   156  
   157  This remote is called `remote` and can now be used like this
   158  
   159  See all containers
   160  
   161      rclone lsd remote:
   162  
   163  Make a new container
   164  
   165      rclone mkdir remote:container
   166  
   167  List the contents of a container
   168  
   169      rclone ls remote:container
   170  
   171  Sync `/home/local/directory` to the remote container, deleting any
   172  excess files in the container.
   173  
   174      rclone sync /home/local/directory remote:container
   175  
   176  ### Configuration from an OpenStack credentials file ###
   177  
   178  An OpenStack credentials file typically looks something something
   179  like this (without the comments)
   180  
   181  ```
   182  export OS_AUTH_URL=https://a.provider.net/v2.0
   183  export OS_TENANT_ID=ffffffffffffffffffffffffffffffff
   184  export OS_TENANT_NAME="1234567890123456"
   185  export OS_USERNAME="123abc567xy"
   186  echo "Please enter your OpenStack Password: "
   187  read -sr OS_PASSWORD_INPUT
   188  export OS_PASSWORD=$OS_PASSWORD_INPUT
   189  export OS_REGION_NAME="SBG1"
   190  if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
   191  ```
   192  
   193  The config file needs to look something like this where `$OS_USERNAME`
   194  represents the value of the `OS_USERNAME` variable - `123abc567xy` in
   195  the example above.
   196  
   197  ```
   198  [remote]
   199  type = swift
   200  user = $OS_USERNAME
   201  key = $OS_PASSWORD
   202  auth = $OS_AUTH_URL
   203  tenant = $OS_TENANT_NAME
   204  ```
   205  
   206  Note that you may (or may not) need to set `region` too - try without first.
   207  
   208  ### Configuration from the environment ###
   209  
   210  If you prefer you can configure rclone to use swift using a standard
   211  set of OpenStack environment variables.
   212  
   213  When you run through the config, make sure you choose `true` for
   214  `env_auth` and leave everything else blank.
   215  
   216  rclone will then set any empty config parameters from the environment
   217  using standard OpenStack environment variables.  There is [a list of
   218  the
   219  variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment)
   220  in the docs for the swift library.
   221  
   222  ### Using an alternate authentication method ###
   223  
   224  If your OpenStack installation uses a non-standard authentication method
   225  that might not be yet supported by rclone or the underlying swift library, 
   226  you can authenticate externally (e.g. calling manually the `openstack` 
   227  commands to get a token). Then, you just need to pass the two 
   228  configuration variables ``auth_token`` and ``storage_url``. 
   229  If they are both provided, the other variables are ignored. rclone will 
   230  not try to authenticate but instead assume it is already authenticated 
   231  and use these two variables to access the OpenStack installation.
   232  
   233  #### Using rclone without a config file ####
   234  
   235  You can use rclone with swift without a config file, if desired, like
   236  this:
   237  
   238  ```
   239  source openstack-credentials-file
   240  export RCLONE_CONFIG_MYREMOTE_TYPE=swift
   241  export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
   242  rclone lsd myremote:
   243  ```
   244  
   245  ### --fast-list ###
   246  
   247  This remote supports `--fast-list` which allows you to use fewer
   248  transactions in exchange for more memory. See the [rclone
   249  docs](/docs/#fast-list) for more details.
   250  
   251  ### --update and --use-server-modtime ###
   252  
   253  As noted below, the modified time is stored on metadata on the object. It is
   254  used by default for all operations that require checking the time a file was
   255  last updated. It allows rclone to treat the remote more like a true filesystem,
   256  but it is inefficient because it requires an extra API call to retrieve the
   257  metadata.
   258  
   259  For many operations, the time the object was last uploaded to the remote is
   260  sufficient to determine if it is "dirty". By using `--update` along with
   261  `--use-server-modtime`, you can avoid the extra API call and simply upload
   262  files whose local modtime is newer than the time it was last uploaded.
   263  
   264  <!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/swift/swift.go then run make backenddocs -->
   265  ### Standard Options
   266  
   267  Here are the standard options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)).
   268  
   269  #### --swift-env-auth
   270  
   271  Get swift credentials from environment variables in standard OpenStack form.
   272  
   273  - Config:      env_auth
   274  - Env Var:     RCLONE_SWIFT_ENV_AUTH
   275  - Type:        bool
   276  - Default:     false
   277  - Examples:
   278      - "false"
   279          - Enter swift credentials in the next step
   280      - "true"
   281          - Get swift credentials from environment vars. Leave other fields blank if using this.
   282  
   283  #### --swift-user
   284  
   285  User name to log in (OS_USERNAME).
   286  
   287  - Config:      user
   288  - Env Var:     RCLONE_SWIFT_USER
   289  - Type:        string
   290  - Default:     ""
   291  
   292  #### --swift-key
   293  
   294  API key or password (OS_PASSWORD).
   295  
   296  - Config:      key
   297  - Env Var:     RCLONE_SWIFT_KEY
   298  - Type:        string
   299  - Default:     ""
   300  
   301  #### --swift-auth
   302  
   303  Authentication URL for server (OS_AUTH_URL).
   304  
   305  - Config:      auth
   306  - Env Var:     RCLONE_SWIFT_AUTH
   307  - Type:        string
   308  - Default:     ""
   309  - Examples:
   310      - "https://auth.api.rackspacecloud.com/v1.0"
   311          - Rackspace US
   312      - "https://lon.auth.api.rackspacecloud.com/v1.0"
   313          - Rackspace UK
   314      - "https://identity.api.rackspacecloud.com/v2.0"
   315          - Rackspace v2
   316      - "https://auth.storage.memset.com/v1.0"
   317          - Memset Memstore UK
   318      - "https://auth.storage.memset.com/v2.0"
   319          - Memset Memstore UK v2
   320      - "https://auth.cloud.ovh.net/v2.0"
   321          - OVH
   322  
   323  #### --swift-user-id
   324  
   325  User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).
   326  
   327  - Config:      user_id
   328  - Env Var:     RCLONE_SWIFT_USER_ID
   329  - Type:        string
   330  - Default:     ""
   331  
   332  #### --swift-domain
   333  
   334  User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
   335  
   336  - Config:      domain
   337  - Env Var:     RCLONE_SWIFT_DOMAIN
   338  - Type:        string
   339  - Default:     ""
   340  
   341  #### --swift-tenant
   342  
   343  Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
   344  
   345  - Config:      tenant
   346  - Env Var:     RCLONE_SWIFT_TENANT
   347  - Type:        string
   348  - Default:     ""
   349  
   350  #### --swift-tenant-id
   351  
   352  Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
   353  
   354  - Config:      tenant_id
   355  - Env Var:     RCLONE_SWIFT_TENANT_ID
   356  - Type:        string
   357  - Default:     ""
   358  
   359  #### --swift-tenant-domain
   360  
   361  Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
   362  
   363  - Config:      tenant_domain
   364  - Env Var:     RCLONE_SWIFT_TENANT_DOMAIN
   365  - Type:        string
   366  - Default:     ""
   367  
   368  #### --swift-region
   369  
   370  Region name - optional (OS_REGION_NAME)
   371  
   372  - Config:      region
   373  - Env Var:     RCLONE_SWIFT_REGION
   374  - Type:        string
   375  - Default:     ""
   376  
   377  #### --swift-storage-url
   378  
   379  Storage URL - optional (OS_STORAGE_URL)
   380  
   381  - Config:      storage_url
   382  - Env Var:     RCLONE_SWIFT_STORAGE_URL
   383  - Type:        string
   384  - Default:     ""
   385  
   386  #### --swift-auth-token
   387  
   388  Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
   389  
   390  - Config:      auth_token
   391  - Env Var:     RCLONE_SWIFT_AUTH_TOKEN
   392  - Type:        string
   393  - Default:     ""
   394  
   395  #### --swift-application-credential-id
   396  
   397  Application Credential ID (OS_APPLICATION_CREDENTIAL_ID)
   398  
   399  - Config:      application_credential_id
   400  - Env Var:     RCLONE_SWIFT_APPLICATION_CREDENTIAL_ID
   401  - Type:        string
   402  - Default:     ""
   403  
   404  #### --swift-application-credential-name
   405  
   406  Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME)
   407  
   408  - Config:      application_credential_name
   409  - Env Var:     RCLONE_SWIFT_APPLICATION_CREDENTIAL_NAME
   410  - Type:        string
   411  - Default:     ""
   412  
   413  #### --swift-application-credential-secret
   414  
   415  Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET)
   416  
   417  - Config:      application_credential_secret
   418  - Env Var:     RCLONE_SWIFT_APPLICATION_CREDENTIAL_SECRET
   419  - Type:        string
   420  - Default:     ""
   421  
   422  #### --swift-auth-version
   423  
   424  AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
   425  
   426  - Config:      auth_version
   427  - Env Var:     RCLONE_SWIFT_AUTH_VERSION
   428  - Type:        int
   429  - Default:     0
   430  
   431  #### --swift-endpoint-type
   432  
   433  Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE)
   434  
   435  - Config:      endpoint_type
   436  - Env Var:     RCLONE_SWIFT_ENDPOINT_TYPE
   437  - Type:        string
   438  - Default:     "public"
   439  - Examples:
   440      - "public"
   441          - Public (default, choose this if not sure)
   442      - "internal"
   443          - Internal (use internal service net)
   444      - "admin"
   445          - Admin
   446  
   447  #### --swift-storage-policy
   448  
   449  The storage policy to use when creating a new container
   450  
   451  This applies the specified storage policy when creating a new
   452  container. The policy cannot be changed afterwards. The allowed
   453  configuration values and their meaning depend on your Swift storage
   454  provider.
   455  
   456  - Config:      storage_policy
   457  - Env Var:     RCLONE_SWIFT_STORAGE_POLICY
   458  - Type:        string
   459  - Default:     ""
   460  - Examples:
   461      - ""
   462          - Default
   463      - "pcs"
   464          - OVH Public Cloud Storage
   465      - "pca"
   466          - OVH Public Cloud Archive
   467  
   468  ### Advanced Options
   469  
   470  Here are the advanced options specific to swift (Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)).
   471  
   472  #### --swift-chunk-size
   473  
   474  Above this size files will be chunked into a _segments container.
   475  
   476  Above this size files will be chunked into a _segments container.  The
   477  default for this is 5GB which is its maximum value.
   478  
   479  - Config:      chunk_size
   480  - Env Var:     RCLONE_SWIFT_CHUNK_SIZE
   481  - Type:        SizeSuffix
   482  - Default:     5G
   483  
   484  #### --swift-no-chunk
   485  
   486  Don't chunk files during streaming upload.
   487  
   488  When doing streaming uploads (eg using rcat or mount) setting this
   489  flag will cause the swift backend to not upload chunked files.
   490  
   491  This will limit the maximum upload size to 5GB. However non chunked
   492  files are easier to deal with and have an MD5SUM.
   493  
   494  Rclone will still chunk files bigger than chunk_size when doing normal
   495  copy operations.
   496  
   497  - Config:      no_chunk
   498  - Env Var:     RCLONE_SWIFT_NO_CHUNK
   499  - Type:        bool
   500  - Default:     false
   501  
   502  <!--- autogenerated options stop -->
   503  
   504  ### Modified time ###
   505  
   506  The modified time is stored as metadata on the object as
   507  `X-Object-Meta-Mtime` as floating point since the epoch accurate to 1
   508  ns.
   509  
   510  This is a defacto standard (used in the official python-swiftclient
   511  amongst others) for storing the modification time for an object.
   512  
   513  ### Limitations ###
   514  
   515  The Swift API doesn't return a correct MD5SUM for segmented files
   516  (Dynamic or Static Large Objects) so rclone won't check or use the
   517  MD5SUM for these.
   518  
   519  ### Troubleshooting ###
   520  
   521  #### Rclone gives Failed to create file system for "remote:": Bad Request ####
   522  
   523  Due to an oddity of the underlying swift library, it gives a "Bad
   524  Request" error rather than a more sensible error when the
   525  authentication fails for Swift.
   526  
   527  So this most likely means your username / password is wrong.  You can
   528  investigate further with the `--dump-bodies` flag.
   529  
   530  This may also be caused by specifying the region when you shouldn't
   531  have (eg OVH).
   532  
   533  #### Rclone gives Failed to create file system: Response didn't have storage storage url and auth token ####
   534  
   535  This is most likely caused by forgetting to specify your tenant when
   536  setting up a swift remote.