github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/docs/development.md (about)

     1  ---
     2  layout: post
     3  title: DEVELOPMENT
     4  permalink: /docs/development
     5  redirect_from:
     6   - /development.md/
     7   - /docs/development.md/
     8  ---
     9  
    10  ## Table of Contents
    11  
    12  - [Debugging: build time](#debugging-build-time)
    13  - [Debugging: run time](#debugging-run-time)
    14  - [Using CLI to debug](#using-cli-to-debug)
    15  - [MsgPack](/docs/msgp.md)
    16  - [Useful scripts](#scripts)
    17    - [Clean deploy](#clean-deploy)
    18    - [Performance comparison](#performance-comparison)
    19  - [More](#more)
    20  
    21  ## Debugging: build time
    22  
    23  By default, the cluster is deployed in `production` mode with verbose logging and asserts disabled.
    24  
    25  To turn on the debug mode, deploy a cluster with `MODE=debug` env variable (eg. `MODE=debug make deploy`).
    26  A cluster deployed in `debug` mode will produce a log like this:
    27  
    28  ## Debugging: run time
    29  
    30  As of v3.18 (git tag 1.3.18), the following can be done at any point in time:
    31  
    32  ```commandline
    33  $ ais config cluster log.modules <TAB-TAB>
    34  
    35  transport    memsys       fs       ec         ios        backend      mirror     downloader   s3
    36  ais          cluster      reb      stats      xs         space        dsort      etl          none
    37  ```
    38  
    39  ```commandline
    40  $ ais config cluster log.modules ec xs
    41  
    42  log.level: "3 (modules: ec,xs)"
    43  ```
    44  
    45  The example (above) elevates verbosity level of two specific modules: EC (erasure coding) and xactions ([batch jobs](batch.md)).
    46  
    47  The change takes an effect immediately. But you can also change logging verbosity for the entire cluster - all modules:
    48  
    49  ```commandline
    50  $ ais config cluster log
    51  PROPERTY         VALUE
    52  log.level        3
    53  log.max_size     4MiB
    54  log.max_total    64MiB
    55  log.flush_time   40s
    56  log.stats_time   1m
    57  
    58  $ ais config cluster log.level 5
    59  PROPERTY         VALUE
    60  log.level        5
    61  log.max_size     4MiB
    62  log.max_total    64MiB
    63  log.flush_time   40s
    64  log.stats_time   1m
    65  ```
    66  
    67  **Caution**: `5` is the maximum (super-verbose) level - use for shorter intervals of time and always reset back to the default (`3`).
    68  
    69  **NOTE**: for module names, see `cmn/cos/log_modules.go`. Or, type `ais config cluster` or `ais config node`, and press `<TAB-TAB>`.
    70  
    71  ## Using CLI to debug
    72  
    73  Please refer [CLI: verbose mode](cli.md#verbose-errors).
    74  
    75  ## Scripts
    76  
    77  There is a growing number of scripts and useful commands that can be used in development.
    78  To see `make` options and usage examples, do:
    79  
    80  ```console
    81  $ make help
    82  ```
    83  
    84  ### Clean deploy
    85  
    86  ```
    87  ./clean_deploy.sh [--target-cnt TARGET_CNT] [--proxy-cnt PROXY_CNT] [--mountpath-cnt MOUNTPATH_CNT] [--https] [--deployment local|remote|all] [--remote-alias REMOTE_ALIAS] [--PROVIDER ...] [--debug PKG=LOG_LEVEL[,PKG=LOG_LEVEL]] [--loopback SIZE]
    88  ```
    89  
    90  Performs cleanup and then deploys a new instance of an AIS cluster.
    91  To make it even more convenient, consider setting up an alias:
    92  
    93  ```bash
    94  alias cais="bash ${GOPATH}/src/github.com/NVIDIA/aistore/scripts/clean-deploy --aws --gcp"
    95  ```
    96  
    97  #### Example: minimal remote cluster
    98  
    99  The command below can be conveniently used to develop with remote AIS clusters and/or test related functionality.
   100  
   101  Here we run two minimal - one gateway and one target - clusters: "local" (by default, at http://localhost:8080)
   102  and "remote", at http://127.0.0.1:11080.
   103  
   104  > Henceforth, the terms "local cluster" and "remote cluster" are used without quotes.
   105  
   106  The script will not only deploy the two clusters - it will also assign the remote one its user-specified alias
   107  and attach one cluster to another, thus forming a [global namespace](providers.md#remote-ais-cluster).
   108  
   109  ```console
   110  $ scripts/clean_deploy.sh --target-cnt 1 --proxy-cnt 1 --mountpath-cnt 4 --deployment all --remote-alias remais
   111  ```
   112  
   113  Here's another example that illustrates multi-node (6 + 6) cluster with storage targets utilizing loopback devices to simulate actual non-shared storage disks (one disk per target mountpath):
   114  
   115  ```console
   116  $ scripts/clean_deploy.sh --target-cnt 6 --proxy-cnt 6 --mountpath-cnt 4 --deployment all --loopback 123M --remote-alias remais --gcp --aws
   117  ```
   118  
   119  > Overall, this line above will create 4 loopbacks of total size 123M * 4 = 0.5GiB. It'll take maybe a few extra seconds but only at the very first run - subsequent cluster restarts will utilize already provisioned devices and other persistent configuration.
   120  
   121  From here, you can create and destroy buckets, read and write data, show buckets, objects and their respective properties -
   122  in short, perform all supported operations on the remote cluster - either directly, via `AIS_ENDPOINT` or indirectly,
   123  via the (attached) local cluster. For example:
   124  
   125  ```console
   126  # create bucket by "pointing" the CLI i(directly) to the remote cluster:
   127  $ AIS_ENDPOINT=http://127.0.0.1:11080 ais create ais://abc
   128  
   129  # PUT an object into remote cluster's bucket:
   130  $ AIS_ENDPOINT=http://127.0.0.1:11080 ais put README.md ais://abc
   131  
   132  # make sure that the local cluster can "see" remote buckets (and **notice** the usage of the remote alias):
   133  $ ais ls ais://@remais/abc
   134  
   135  # show properties of objects stored in the remote cluster's buckets, etc.
   136  $ ais object show ais://@remais/abc/README.md
   137  ```
   138  
   139  > Notice the bucket naming syntax: by convention, prefix `@` indicated remote cluster's UUIDs, and so
   140  `ais://@remais/abc` translates as "AIS backend provider, where remote cluster has alias `remais`".
   141  
   142  #### Example: 5 proxies and 5 targets with GCP backend
   143  
   144  The command below starts a cluster with 5 proxies and 5 targets with GCP cloud enabled.
   145  Remember to set `GOOGLE_APPLICATION_CREDENTIALS` env when using GCP cloud!
   146  
   147  ```console
   148  $ bash ./scripts/clean-deploy --gcp
   149  ```
   150  
   151  The example below deploys:
   152  - a simulated remote cluster with alias "remoteAIS"
   153  - 3 targets
   154  - 3 proxies
   155  - with AWS support
   156  
   157  ```console
   158  $ bash ./scripts/clean-deploy --deployment all --remote-alias remoteAIS --target-cnt 3 --proxy-cnt 3 --aws
   159  ```
   160  
   161  #### Options
   162  
   163  | Option | Description |
   164  | ------ | ----------- |
   165  | `--target-cnt` | Number of targets to start (default: 5) |
   166  | `--proxy-cnt` | Number of proxies to start (default: 5) |
   167  | `--mountpath-cnt` | Number of mountpaths to use (default: 5) |
   168  | `--PROVIDER` | Specifies the backend provider(s). Can be: `--aws`, `--azure`, `--gcp` |
   169  | `--loopback` | Loopback device size, e.g. 10G, 100M (default: 0). Zero size means: no loopbacks. The minimum size is 100M. |
   170  | `--deployment` | Choose which AIS cluster to deploy. `local` to deploy only one AIS cluster, `remote` to only start an AIS-behind-AIS cluster, and `all` to deploy both the local and remote clusters. |
   171  | `--remote-alias` | Alias to assign to the remote cluster |
   172  | `--https` | Start cluster with HTTPS enabled (**) |
   173  | `--debug PKG=LOG_LEVEL` | Change logging level of particular package(s) |
   174  
   175  (**) To use this option, you must have generated certificates in `$HOME` directory. Here is a script which can help you with that:
   176  ```console
   177  $ cd $HOME && openssl req -x509 -out localhost.crt -keyout localhost.key \
   178      -newkey rsa:2048 -nodes -sha256 \
   179      -subj '/CN=localhost' -extensions EXT -config <( \
   180       printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
   181  ```
   182  
   183  ### Performance comparison
   184  
   185  ```
   186  ./bench.sh cmp [OLD_COMMIT] [--dir DIRECTORY] [--bench BENCHMARK_NAME] [--post_checkout SCRIPT_NAME] [--verbose]
   187  ```
   188  
   189  Compares benchmark between the current commit and old commit provided in argument.
   190  
   191  This script is incredibly important because it allows catching regressions.
   192  It also can quickly provide an answer if the change that was made actually improved the performance.
   193  
   194  #### Example usage
   195  
   196  The command below will compare the benchmark(s) `BenchmarkRandom*` between the current commit and `f9a1536f...`.
   197  
   198  ```console
   199  $ bash ./scripts/bootstrap.sh bench cmp f9a1536f4c9af0d1ac84c200e68f2ba73676c487 --dir bench/tools/aisloader --bench BenchmarkRandom
   200  ```
   201  
   202  #### Options
   203  
   204  | Option | Description |
   205  | ------ | ----------- |
   206  | `--dir DIRECTORY` | Directory in which benchmark(s) should be run |
   207  | `--bench BENCHMARK_NAME` | Name or prefix of benchmark(s) to run |
   208  | `--post_checkout SCRIPT_NAME` | Script name which will executed after each `git checkout <commit>` (old and new commit) |
   209  | `--verbose` | Run benchmarks in verbose mode |
   210  
   211  ## More
   212  
   213  This [local-playground usage example](/deploy/dev/local/README.md) is a yet another brief introduction into setting up Go environment, provisioniong data drives for AIS deployment, and running a minimal AIS cluster - locally.