github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/docs/cli/aws_profile_endpoint.md (about)

     1  ---
     2  layout: post
     3  title: AWS_PROFILE_ENDPOINT
     4  permalink: /docs/cli/aws_profile_endpoint
     5  redirect_from:
     6   - /cli/aws_profile_endpoint.md/
     7   - /docs/cli/aws_profile_endpoint.md/
     8  ---
     9  
    10  AIStore supports vendor-specific configuration on a per bucket basis. For instance, any bucket _backed up_ by an AWS S3 bucket (**) can be configured to use alternative:
    11  
    12  * named AWS profiles (with alternative credentials and/or AWS region)
    13  * s3 endpoints
    14  
    15  (**) Terminology-wise, when we say "s3 bucket" or "google cloud bucket" we in fact reference a bucket in an AIS cluster that is either:
    16  
    17  * (A) denoted with the respective `s3:` or `gs:` protocol schema, or
    18  * (B) is a differently named AIS (that is, `ais://`) bucket that has its `backend_bck` property referencing the s3 (or google cloud) bucket in question.
    19  
    20  For supported backends (that include, but are not limited, to AWS S3), see also:
    21  
    22  * [Backend Provider](/docs/bucket.md#backend-provider)
    23  * [Backend Bucket](/docs/bucket.md#backend-bucket)
    24  
    25  ## Table of Contents
    26  - [Viewing vendor-specific properties](#viewing-vendor-specific-properties)
    27  - [Environment variables](#environment-variables)
    28  - [Setting profile with alternative access/secret keys and/or region](#setting-profile-with-alternative-accesssecret-keys-andor-region)
    29  - [When bucket does not exist](#when-bucket-does-not-exist)
    30  - [Configuring custom AWS S3 endpoint](#configuring-custom-aws-s3-endpoint)
    31  
    32  ## Viewing vendor-specific properties
    33  
    34  While `ais show bucket` will show all properties (which is a lengthy list), the way to maybe focus on vendor-specific extension is to look for the section called "extra". For example:
    35  
    36  ```console
    37  $ ais show bucket s3://abc | grep extra
    38  extra.aws.cloud_region      us-east-2
    39  extra.aws.endpoint
    40  extra.aws.profile
    41  ```
    42  
    43  Notice that the bucket's region (`cloud_region` above) is automatically populated when AIS looks up the bucket in s3. But the other two varables are settable and can provide alternative credentials and/or access endpoint.
    44  
    45  ## Environment variables
    46  
    47  AIStore supports the well-known `S3_ENDPOINT` and `AWS_PROFILE` environment. While `S3_ENDPOINT` is often used to utilize AIS cluster as s3-providing service, configurable `AWS_PROFILE` specifies what's called a _named_ configuration profile:
    48  
    49  * [Using named AWS profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-using-profiles)
    50  
    51  The rule is simple:
    52  
    53  * `S3_ENDPOINT` and `AWS_PROFILE` are loaded once upon AIS node startup.
    54  * Bucket configuration takes **precedence** over the environment and can be changed at **any** time.
    55  
    56  ## Setting profile with alternative access/secret keys and/or region
    57  
    58  Assuming, on the one hand:
    59  
    60  ```console
    61  $ cat ~/.aws/config
    62  [default]
    63  region = us-east-2
    64  
    65  [profile prod]
    66  region = us-west-1
    67  ```
    68  and
    69  
    70  ```console
    71  $ cat ~/.aws/credentials
    72  [default]
    73  aws_access_key_id = foo
    74  aws_secret_access_key = bar
    75  
    76  [prod]
    77  aws_access_key_id = 123
    78  aws_secret_access_key = 456
    79  ```
    80  
    81  on the other, we can then go ahead and set the "prod" profile directly into the bucket:
    82  
    83  ```console
    84  $ ais bucket props set s3://abc extra.aws.profile prod
    85  "extra.aws.profile" set to: "prod" (was: "")
    86  ```
    87  
    88  and show resulting "extra.aws" configuration:
    89  
    90  ```console
    91  $ ais show bucket s3://abc | grep extra
    92  extra.aws.cloud_region      us-west-1
    93  extra.aws.endpoint
    94  extra.aws.profile           prod
    95  ```
    96  
    97  From this point on, all calls to read, write, list `s3://abc` and get/set its properties will use AWS "prod" profile (see above).
    98  
    99  ## When bucket does not exist
   100  
   101  But what if we need to set alternative profile (with alternative access and secret keys) on a bucket that does not yet exist in the cluster?
   102  
   103  That must be a fairly common situation, and the way to resolve it is to use `--skip-lookup` option:
   104  
   105  ```console
   106  $ ais create --help
   107  ...
   108  OPTIONS:
   109     --props value   bucket properties, e.g. --props="mirror.enabled=true mirror.copies=4 checksum.type=md5"
   110     --skip-lookup   add Cloud bucket to aistore without checking the bucket's accessibility and getting its Cloud properties
   111                     (usage must be limited to setting up bucket's aistore properties with alternative profile and/or endpoint)
   112  
   113  
   114  $ ais create s3://abc --skip-lookup
   115  "s3://abc" created
   116  ```
   117  
   118  Once this is done (**), we simply go ahead and run `ais bucket props set s3://abc extra.aws.profile` (as shown above). Assuming, the updated profile contains correct access keys, the bucket will then be fully available for reading, writing, listing, and all the rest operations.
   119  
   120  > (**) `ais create` command results in adding the bucket to aistore `BMD` - a protected, versioned, and replicated bucket metadata that is further used to update properties of any bucket in the cluster, including certainly the one that we have just added.
   121  
   122  ## Configuring custom AWS S3 endpoint
   123  
   124  When a bucket is hosted by an S3 compliant backend (such as, e.g., minio), we may want to specify an alternative S3 endpoint,
   125  so that AIS nodes use it when reading, writing, listing, and generally, performing all operations on remote S3 bucket(s).
   126  
   127  Globally, S3 endpoint can be overridden for _all_ S3 buckets via "S3_ENDPOINT" environment.
   128  
   129  If you decide to make the change, you may need to restart AIS cluster while making sure that "S3_ENDPOINT" is available for the AIS nodes
   130  when they are starting up.
   131  
   132  But it can be also be done - and will take precedence over the global setting - on a per-bucket basis.
   133  
   134  Here are some examples:
   135  
   136  ```console
   137  # Let's say, there exists a bucket called s3://abc:
   138  $ ais ls s3://abc
   139  NAME             SIZE
   140  README.md        8.96KiB
   141  ```
   142  
   143  First, we override empty the endpoint property in the bucket's configuration.
   144  To see that a non-empty value *applies* and works, we will use the default AWS S3 endpoint: `https://s3.amazonaws.com`
   145  
   146  ```console
   147  $ ais bucket props set s3://abc extra.aws.endpoint=s3.amazonaws.com
   148  Bucket "aws://abc": property "extra.aws.endpoint=s3.amazonaws.com", nothing to do
   149  $ ais ls s3://abc
   150  NAME             SIZE
   151  README.md        8.96KiB
   152  ```
   153  
   154  Second, set the endpoint=foo (or, it could be any other invalid value), and observe that the bucket becomes unreachable:
   155  
   156  ```console
   157  $ ais bucket props set s3://abc extra.aws.endpoint=foo
   158  Bucket props successfully updated
   159  "extra.aws.endpoint" set to: "foo" (was: "s3.amazonaws.com")
   160  
   161  $ ais ls s3://abc
   162  RequestError: send request failed: dial tcp: lookup abc.foo: no such host
   163  ```
   164  
   165  Finally, revert the endpoint back to empty, and check that the bucket is visible again:
   166  
   167  ```console
   168  $ ais bucket props set s3://abc extra.aws.endpoint=""
   169  Bucket props successfully updated
   170  "extra.aws.endpoint" set to: "" (was: "foo")
   171  
   172  $ ais ls s3://abc
   173  NAME             SIZE
   174  README.md        8.96KiB
   175  ```
   176  
   177  > Global `export S3_ENDPOINT=...` override is static and readonly. Use it with extreme caution as it applies to all buckets.
   178  
   179  > On the other hand, for any given `s3://bucket` its S3 endpoint can be set, unset, and otherwise changed at any time - at runtime. As shown above.
   180