github.com/NVIDIA/aistore@v1.3.23-0.20240517131212-7df6609be51d/docs/cli/blob-downloader.md (about)

     1  ---
     2  layout: post
     3  title: Blob downloader
     4  permalink: /docs/cli/blob-downloader
     5  redirect_from:
     6   - /cli/blob-downloader.md/
     7   - /docs/cli/blob-downloader.md/
     8  ---
     9  
    10  AIS comes with built-in blob downloading facility that employs multiple concurrent readers to speed-up reading very large remote objects.
    11  
    12  In terms of its from/to operation, blob downloading can be compared to [prefetching](/docs/cli/object.md#prefetch-objects).
    13  
    14  More precisely, the list of "comparables" includes:
    15  * [prefetch](/docs/cli/object.md#prefetch-objects)
    16  * [copy remote bucket](/docs/cli/bucket.md#copy-bucket)
    17  * [copy multiple remote objects](/docs/cli/bucket.md#copy-multiple-objects)
    18  
    19  In all these cases, destination of the remote content is aistore cluster (and not the client requesting the operation).
    20  
    21  In CLI, `ais blob-download` will run a job, or multiple jobs, to download user-specified remote blob(s). Command line options and examples follow below.
    22  
    23  ## Options
    24  
    25  ```console
    26  $ ais blob-download --help
    27  NAME:
    28     ais blob-download - run a job to download large object(s) from remote storage to aistore cluster, e.g.:
    29      - 'blob-download s3://abc/largefile --chunk-size=2mb --progress'          - download one blob at a given chunk size
    30      - 'blob-download s3://abc --list "f1, f2, f3" --num-workers=4 --progress' - use 4 concurrent readers to download each of the 3 blobs
    31     Note: when _not_ using '--progress' option, run 'ais show job' to monitor.
    32  
    33  USAGE:
    34     ais blob-download [command options] BUCKET/OBJECT_NAME
    35  
    36  OPTIONS:
    37     --refresh value      interval for continuous monitoring;
    38                          valid time units: ns, us (or µs), ms, s (default), m, h
    39     --progress           show progress bar(s) and progress of execution in real time
    40     --list value         comma-separated list of object or file names, e.g.:
    41                          --list 'o1,o2,o3'
    42                          --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
    43                          or, when listing files and/or directories:
    44                          --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
    45     --chunk-size value   chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
    46     --num-workers value  number of concurrent blob-downloading workers (readers); system default when omitted or zero
    47     --wait               wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
    48     --timeout value      maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
    49                          valid time units: ns, us (or µs), ms, s (default), m, h
    50     --latest             check in-cluster metadata and, possibly, GET, download, prefetch, or copy the latest object version
    51                          from the associated remote bucket:
    52                          - provides operation-level control over object versioning (and version synchronization)
    53                            without requiring to change bucket configuration
    54                          - the latter can be done using 'ais bucket props set BUCKET versioning'
    55                          - see also: 'ais ls --check-versions', 'ais cp', 'ais prefetch', 'ais get'
    56     --help, -h           show help
    57  ```
    58  
    59  ## Usage example
    60  
    61  ```console
    62  $ ais blob-download s3://abc --list "file-2gb, file-100mb" --chunk-size=2mb --progress
    63  
    64  blob-download[Qxz3EClVN]
    65  blob-download[xvC3nClSF]
    66  s3://abc/file-2gb  513 MiB / 2 GiB      [==============>-----------------------------------------------] 24 %
    67  s3://abc/file-100mb 44.17 MiB / 100 MiB [==========================>-----------------------------------] 44 %
    68  ```