github.com/rclone/rclone@v1.66.1-0.20240517100346-7b89735ae726/docs/content/azurefiles.md (about) 1 --- 2 title: "Microsoft Azure Files Storage" 3 description: "Rclone docs for Microsoft Azure Files Storage" 4 versionIntroduced: "v1.65" 5 --- 6 7 # {{< icon "fab fa-windows" >}} Microsoft Azure Files Storage 8 9 Paths are specified as `remote:` You may put subdirectories in too, 10 e.g. `remote:path/to/dir`. 11 12 ## Configuration 13 14 Here is an example of making a Microsoft Azure Files Storage 15 configuration. For a remote called `remote`. First run: 16 17 rclone config 18 19 This will guide you through an interactive setup process: 20 21 ``` 22 No remotes found, make a new one? 23 n) New remote 24 s) Set configuration password 25 q) Quit config 26 n/s/q> n 27 name> remote 28 Type of storage to configure. 29 Choose a number from below, or type in your own value 30 [snip] 31 XX / Microsoft Azure Files Storage 32 \ "azurefiles" 33 [snip] 34 35 Option account. 36 Azure Storage Account Name. 37 Set this to the Azure Storage Account Name in use. 38 Leave blank to use SAS URL or connection string, otherwise it needs to be set. 39 If this is blank and if env_auth is set it will be read from the 40 environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible. 41 Enter a value. Press Enter to leave empty. 42 account> account_name 43 44 Option share_name. 45 Azure Files Share Name. 46 This is required and is the name of the share to access. 47 Enter a value. Press Enter to leave empty. 48 share_name> share_name 49 50 Option env_auth. 51 Read credentials from runtime (environment variables, CLI or MSI). 52 See the [authentication docs](/azurefiles#authentication) for full info. 53 Enter a boolean value (true or false). Press Enter for the default (false). 54 env_auth> 55 56 Option key. 57 Storage Account Shared Key. 58 Leave blank to use SAS URL or connection string. 59 Enter a value. Press Enter to leave empty. 60 key> base64encodedkey== 61 62 Option sas_url. 63 SAS URL. 64 Leave blank if using account/key or connection string. 65 Enter a value. Press Enter to leave empty. 66 sas_url> 67 68 Option connection_string. 69 Azure Files Connection String. 70 Enter a value. Press Enter to leave empty. 71 connection_string> 72 [snip] 73 74 Configuration complete. 75 Options: 76 - type: azurefiles 77 - account: account_name 78 - share_name: share_name 79 - key: base64encodedkey== 80 Keep this "remote" remote? 81 y) Yes this is OK (default) 82 e) Edit this remote 83 d) Delete this remote 84 y/e/d> 85 ``` 86 87 Once configured you can use rclone. 88 89 See all files in the top level: 90 91 rclone lsf remote: 92 93 Make a new directory in the root: 94 95 rclone mkdir remote:dir 96 97 Recursively List the contents: 98 99 rclone ls remote: 100 101 Sync `/home/local/directory` to the remote directory, deleting any 102 excess files in the directory. 103 104 rclone sync --interactive /home/local/directory remote:dir 105 106 ### Modified time 107 108 The modified time is stored as Azure standard `LastModified` time on 109 files 110 111 ### Performance 112 113 When uploading large files, increasing the value of 114 `--azurefiles-upload-concurrency` will increase performance at the cost 115 of using more memory. The default of 16 is set quite conservatively to 116 use less memory. It maybe be necessary raise it to 64 or higher to 117 fully utilize a 1 GBit/s link with a single file transfer. 118 119 ### Restricted filename characters 120 121 In addition to the [default restricted characters set](/overview/#restricted-characters) 122 the following characters are also replaced: 123 124 | Character | Value | Replacement | 125 | --------- |:-----:|:-----------:| 126 | " | 0x22 | " | 127 | * | 0x2A | * | 128 | : | 0x3A | : | 129 | < | 0x3C | < | 130 | > | 0x3E | > | 131 | ? | 0x3F | ? | 132 | \ | 0x5C | \ | 133 | \| | 0x7C | | | 134 135 File names can also not end with the following characters. 136 These only get replaced if they are the last character in the name: 137 138 | Character | Value | Replacement | 139 | --------- |:-----:|:-----------:| 140 | . | 0x2E | . | 141 142 Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8), 143 as they can't be used in JSON strings. 144 145 ### Hashes 146 147 MD5 hashes are stored with files. Not all files will have MD5 hashes 148 as these have to be uploaded with the file. 149 150 ### Authentication {#authentication} 151 152 There are a number of ways of supplying credentials for Azure Files 153 Storage. Rclone tries them in the order of the sections below. 154 155 #### Env Auth 156 157 If the `env_auth` config parameter is `true` then rclone will pull 158 credentials from the environment or runtime. 159 160 It tries these authentication methods in this order: 161 162 1. Environment Variables 163 2. Managed Service Identity Credentials 164 3. Azure CLI credentials (as used by the az tool) 165 166 These are described in the following sections 167 168 ##### Env Auth: 1. Environment Variables 169 170 If `env_auth` is set and environment variables are present rclone 171 authenticates a service principal with a secret or certificate, or a 172 user with a password, depending on which environment variable are set. 173 It reads configuration from these variables, in the following order: 174 175 1. Service principal with client secret 176 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 177 - `AZURE_CLIENT_ID`: the service principal's client ID 178 - `AZURE_CLIENT_SECRET`: one of the service principal's client secrets 179 2. Service principal with certificate 180 - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID. 181 - `AZURE_CLIENT_ID`: the service principal's client ID 182 - `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key. 183 - `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file. 184 - `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 185 3. User with username and password 186 - `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations". 187 - `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to 188 - `AZURE_USERNAME`: a username (usually an email address) 189 - `AZURE_PASSWORD`: the user's password 190 4. Workload Identity 191 - `AZURE_TENANT_ID`: Tenant to authenticate in. 192 - `AZURE_CLIENT_ID`: Client ID of the application the user will authenticate to. 193 - `AZURE_FEDERATED_TOKEN_FILE`: Path to projected service account token file. 194 - `AZURE_AUTHORITY_HOST`: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com). 195 196 197 ##### Env Auth: 2. Managed Service Identity Credentials 198 199 When using Managed Service Identity if the VM(SS) on which this 200 program is running has a system-assigned identity, it will be used by 201 default. If the resource has no system-assigned but exactly one 202 user-assigned identity, the user-assigned identity will be used by 203 default. 204 205 If the resource has multiple user-assigned identities you will need to 206 unset `env_auth` and set `use_msi` instead. See the [`use_msi` 207 section](#use_msi). 208 209 ##### Env Auth: 3. Azure CLI credentials (as used by the az tool) 210 211 Credentials created with the `az` tool can be picked up using `env_auth`. 212 213 For example if you were to login with a service principal like this: 214 215 az login --service-principal -u XXX -p XXX --tenant XXX 216 217 Then you could access rclone resources like this: 218 219 rclone lsf :azurefiles,env_auth,account=ACCOUNT: 220 221 Or 222 223 rclone lsf --azurefiles-env-auth --azurefiles-account=ACCOUNT :azurefiles: 224 225 #### Account and Shared Key 226 227 This is the most straight forward and least flexible way. Just fill 228 in the `account` and `key` lines and leave the rest blank. 229 230 #### SAS URL 231 232 To use it leave `account`, `key` and `connection_string` blank and fill in `sas_url`. 233 234 #### Connection String 235 236 To use it leave `account`, `key` and "sas_url" blank and fill in `connection_string`. 237 238 #### Service principal with client secret 239 240 If these variables are set, rclone will authenticate with a service principal with a client secret. 241 242 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 243 - `client_id`: the service principal's client ID 244 - `client_secret`: one of the service principal's client secrets 245 246 The credentials can also be placed in a file using the 247 `service_principal_file` configuration option. 248 249 #### Service principal with certificate 250 251 If these variables are set, rclone will authenticate with a service principal with certificate. 252 253 - `tenant`: ID of the service principal's tenant. Also called its "directory" ID. 254 - `client_id`: the service principal's client ID 255 - `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key. 256 - `client_certificate_password`: (optional) password for the certificate file. 257 - `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header. 258 259 **NB** `client_certificate_password` must be obscured - see [rclone obscure](/commands/rclone_obscure/). 260 261 #### User with username and password 262 263 If these variables are set, rclone will authenticate with username and password. 264 265 - `tenant`: (optional) tenant to authenticate in. Defaults to "organizations". 266 - `client_id`: client ID of the application the user will authenticate to 267 - `username`: a username (usually an email address) 268 - `password`: the user's password 269 270 Microsoft doesn't recommend this kind of authentication, because it's 271 less secure than other authentication flows. This method is not 272 interactive, so it isn't compatible with any form of multi-factor 273 authentication, and the application must already have user or admin 274 consent. This credential can only authenticate work and school 275 accounts; it can't authenticate Microsoft accounts. 276 277 **NB** `password` must be obscured - see [rclone obscure](/commands/rclone_obscure/). 278 279 #### Managed Service Identity Credentials {#use_msi} 280 281 If `use_msi` is set then managed service identity credentials are 282 used. This authentication only works when running in an Azure service. 283 `env_auth` needs to be unset to use this. 284 285 However if you have multiple user identities to choose from these must 286 be explicitly specified using exactly one of the `msi_object_id`, 287 `msi_client_id`, or `msi_mi_res_id` parameters. 288 289 If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is 290 set, this is is equivalent to using `env_auth`. 291 292 {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/azurefiles/azurefiles.go then run make backenddocs" >}} 293 ### Standard options 294 295 Here are the Standard options specific to azurefiles (Microsoft Azure Files). 296 297 #### --azurefiles-account 298 299 Azure Storage Account Name. 300 301 Set this to the Azure Storage Account Name in use. 302 303 Leave blank to use SAS URL or connection string, otherwise it needs to be set. 304 305 If this is blank and if env_auth is set it will be read from the 306 environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible. 307 308 309 Properties: 310 311 - Config: account 312 - Env Var: RCLONE_AZUREFILES_ACCOUNT 313 - Type: string 314 - Required: false 315 316 #### --azurefiles-share-name 317 318 Azure Files Share Name. 319 320 This is required and is the name of the share to access. 321 322 323 Properties: 324 325 - Config: share_name 326 - Env Var: RCLONE_AZUREFILES_SHARE_NAME 327 - Type: string 328 - Required: false 329 330 #### --azurefiles-env-auth 331 332 Read credentials from runtime (environment variables, CLI or MSI). 333 334 See the [authentication docs](/azurefiles#authentication) for full info. 335 336 Properties: 337 338 - Config: env_auth 339 - Env Var: RCLONE_AZUREFILES_ENV_AUTH 340 - Type: bool 341 - Default: false 342 343 #### --azurefiles-key 344 345 Storage Account Shared Key. 346 347 Leave blank to use SAS URL or connection string. 348 349 Properties: 350 351 - Config: key 352 - Env Var: RCLONE_AZUREFILES_KEY 353 - Type: string 354 - Required: false 355 356 #### --azurefiles-sas-url 357 358 SAS URL. 359 360 Leave blank if using account/key or connection string. 361 362 Properties: 363 364 - Config: sas_url 365 - Env Var: RCLONE_AZUREFILES_SAS_URL 366 - Type: string 367 - Required: false 368 369 #### --azurefiles-connection-string 370 371 Azure Files Connection String. 372 373 Properties: 374 375 - Config: connection_string 376 - Env Var: RCLONE_AZUREFILES_CONNECTION_STRING 377 - Type: string 378 - Required: false 379 380 #### --azurefiles-tenant 381 382 ID of the service principal's tenant. Also called its directory ID. 383 384 Set this if using 385 - Service principal with client secret 386 - Service principal with certificate 387 - User with username and password 388 389 390 Properties: 391 392 - Config: tenant 393 - Env Var: RCLONE_AZUREFILES_TENANT 394 - Type: string 395 - Required: false 396 397 #### --azurefiles-client-id 398 399 The ID of the client in use. 400 401 Set this if using 402 - Service principal with client secret 403 - Service principal with certificate 404 - User with username and password 405 406 407 Properties: 408 409 - Config: client_id 410 - Env Var: RCLONE_AZUREFILES_CLIENT_ID 411 - Type: string 412 - Required: false 413 414 #### --azurefiles-client-secret 415 416 One of the service principal's client secrets 417 418 Set this if using 419 - Service principal with client secret 420 421 422 Properties: 423 424 - Config: client_secret 425 - Env Var: RCLONE_AZUREFILES_CLIENT_SECRET 426 - Type: string 427 - Required: false 428 429 #### --azurefiles-client-certificate-path 430 431 Path to a PEM or PKCS12 certificate file including the private key. 432 433 Set this if using 434 - Service principal with certificate 435 436 437 Properties: 438 439 - Config: client_certificate_path 440 - Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PATH 441 - Type: string 442 - Required: false 443 444 #### --azurefiles-client-certificate-password 445 446 Password for the certificate file (optional). 447 448 Optionally set this if using 449 - Service principal with certificate 450 451 And the certificate has a password. 452 453 454 **NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/). 455 456 Properties: 457 458 - Config: client_certificate_password 459 - Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PASSWORD 460 - Type: string 461 - Required: false 462 463 ### Advanced options 464 465 Here are the Advanced options specific to azurefiles (Microsoft Azure Files). 466 467 #### --azurefiles-client-send-certificate-chain 468 469 Send the certificate chain when using certificate auth. 470 471 Specifies whether an authentication request will include an x5c header 472 to support subject name / issuer based authentication. When set to 473 true, authentication requests include the x5c header. 474 475 Optionally set this if using 476 - Service principal with certificate 477 478 479 Properties: 480 481 - Config: client_send_certificate_chain 482 - Env Var: RCLONE_AZUREFILES_CLIENT_SEND_CERTIFICATE_CHAIN 483 - Type: bool 484 - Default: false 485 486 #### --azurefiles-username 487 488 User name (usually an email address) 489 490 Set this if using 491 - User with username and password 492 493 494 Properties: 495 496 - Config: username 497 - Env Var: RCLONE_AZUREFILES_USERNAME 498 - Type: string 499 - Required: false 500 501 #### --azurefiles-password 502 503 The user's password 504 505 Set this if using 506 - User with username and password 507 508 509 **NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/). 510 511 Properties: 512 513 - Config: password 514 - Env Var: RCLONE_AZUREFILES_PASSWORD 515 - Type: string 516 - Required: false 517 518 #### --azurefiles-service-principal-file 519 520 Path to file containing credentials for use with a service principal. 521 522 Leave blank normally. Needed only if you want to use a service principal instead of interactive login. 523 524 $ az ad sp create-for-rbac --name "<name>" \ 525 --role "Storage Files Data Owner" \ 526 --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \ 527 > azure-principal.json 528 529 See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details. 530 531 **NB** this section needs updating for Azure Files - pull requests appreciated! 532 533 It may be more convenient to put the credentials directly into the 534 rclone config file under the `client_id`, `tenant` and `client_secret` 535 keys instead of setting `service_principal_file`. 536 537 538 Properties: 539 540 - Config: service_principal_file 541 - Env Var: RCLONE_AZUREFILES_SERVICE_PRINCIPAL_FILE 542 - Type: string 543 - Required: false 544 545 #### --azurefiles-use-msi 546 547 Use a managed service identity to authenticate (only works in Azure). 548 549 When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/) 550 to authenticate to Azure Storage instead of a SAS token or account key. 551 552 If the VM(SS) on which this program is running has a system-assigned identity, it will 553 be used by default. If the resource has no system-assigned but exactly one user-assigned identity, 554 the user-assigned identity will be used by default. If the resource has multiple user-assigned 555 identities, the identity to use must be explicitly specified using exactly one of the msi_object_id, 556 msi_client_id, or msi_mi_res_id parameters. 557 558 Properties: 559 560 - Config: use_msi 561 - Env Var: RCLONE_AZUREFILES_USE_MSI 562 - Type: bool 563 - Default: false 564 565 #### --azurefiles-msi-object-id 566 567 Object ID of the user-assigned MSI to use, if any. 568 569 Leave blank if msi_client_id or msi_mi_res_id specified. 570 571 Properties: 572 573 - Config: msi_object_id 574 - Env Var: RCLONE_AZUREFILES_MSI_OBJECT_ID 575 - Type: string 576 - Required: false 577 578 #### --azurefiles-msi-client-id 579 580 Object ID of the user-assigned MSI to use, if any. 581 582 Leave blank if msi_object_id or msi_mi_res_id specified. 583 584 Properties: 585 586 - Config: msi_client_id 587 - Env Var: RCLONE_AZUREFILES_MSI_CLIENT_ID 588 - Type: string 589 - Required: false 590 591 #### --azurefiles-msi-mi-res-id 592 593 Azure resource ID of the user-assigned MSI to use, if any. 594 595 Leave blank if msi_client_id or msi_object_id specified. 596 597 Properties: 598 599 - Config: msi_mi_res_id 600 - Env Var: RCLONE_AZUREFILES_MSI_MI_RES_ID 601 - Type: string 602 - Required: false 603 604 #### --azurefiles-endpoint 605 606 Endpoint for the service. 607 608 Leave blank normally. 609 610 Properties: 611 612 - Config: endpoint 613 - Env Var: RCLONE_AZUREFILES_ENDPOINT 614 - Type: string 615 - Required: false 616 617 #### --azurefiles-chunk-size 618 619 Upload chunk size. 620 621 Note that this is stored in memory and there may be up to 622 "--transfers" * "--azurefile-upload-concurrency" chunks stored at once 623 in memory. 624 625 Properties: 626 627 - Config: chunk_size 628 - Env Var: RCLONE_AZUREFILES_CHUNK_SIZE 629 - Type: SizeSuffix 630 - Default: 4Mi 631 632 #### --azurefiles-upload-concurrency 633 634 Concurrency for multipart uploads. 635 636 This is the number of chunks of the same file that are uploaded 637 concurrently. 638 639 If you are uploading small numbers of large files over high-speed 640 links and these uploads do not fully utilize your bandwidth, then 641 increasing this may help to speed up the transfers. 642 643 Note that chunks are stored in memory and there may be up to 644 "--transfers" * "--azurefile-upload-concurrency" chunks stored at once 645 in memory. 646 647 Properties: 648 649 - Config: upload_concurrency 650 - Env Var: RCLONE_AZUREFILES_UPLOAD_CONCURRENCY 651 - Type: int 652 - Default: 16 653 654 #### --azurefiles-max-stream-size 655 656 Max size for streamed files. 657 658 Azure files needs to know in advance how big the file will be. When 659 rclone doesn't know it uses this value instead. 660 661 This will be used when rclone is streaming data, the most common uses are: 662 663 - Uploading files with `--vfs-cache-mode off` with `rclone mount` 664 - Using `rclone rcat` 665 - Copying files with unknown length 666 667 You will need this much free space in the share as the file will be this size temporarily. 668 669 670 Properties: 671 672 - Config: max_stream_size 673 - Env Var: RCLONE_AZUREFILES_MAX_STREAM_SIZE 674 - Type: SizeSuffix 675 - Default: 10Gi 676 677 #### --azurefiles-encoding 678 679 The encoding for the backend. 680 681 See the [encoding section in the overview](/overview/#encoding) for more info. 682 683 Properties: 684 685 - Config: encoding 686 - Env Var: RCLONE_AZUREFILES_ENCODING 687 - Type: Encoding 688 - Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot 689 690 #### --azurefiles-description 691 692 Description of the remote 693 694 Properties: 695 696 - Config: description 697 - Env Var: RCLONE_AZUREFILES_DESCRIPTION 698 - Type: string 699 - Required: false 700 701 {{< rem autogenerated options stop >}} 702 703 ### Custom upload headers 704 705 You can set custom upload headers with the `--header-upload` flag. 706 707 - Cache-Control 708 - Content-Disposition 709 - Content-Encoding 710 - Content-Language 711 - Content-Type 712 713 Eg `--header-upload "Content-Type: text/potato"` 714 715 ## Limitations 716 717 MD5 sums are only uploaded with chunked files if the source has an MD5 718 sum. This will always be the case for a local to azure copy.