github.com/htcondor/osdf-client/v6@v6.13.0-rc1.0.20231009141709-766e7b4d1dc8/README.md (about)

     1  Open Science Data Federation (OSDF) Client
     2  ==========================================
     3  
     4  [![Version][github-release-shield]][github-release]
     5  ![Go Mod][go-mod-version]
     6  ![Builds][github-build]
     7  
     8  The OSDF Client downloads files from the [Open Science Data Federation](https://osdf.osg-htc.org/) through a series of caches.  The OSDF is used by multiple organizations to effeciently transfer input and output data.
     9  
    10  The client consists of two parts: a command-line tool named StashCP, and an [HTCondor](https://htcondor.org/) file transfer [plugin](https://htcondor.readthedocs.io/en/latest/admin-manual/setting-up-special-environments.html#enabling-the-transfer-of-files-specified-by-a-url).
    11  
    12  When configured, this plugin will allow the user to specify `transfer_input_files` with the `stash://` protocol which will be downloaded through the OSDF caches.  An example of a submit file:
    13  
    14      ...
    15      transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1
    16      ...
    17  
    18  Note: This repo is the continuation of [opensciencegrid/stashcp][github-previous-repo]; visit that repo for old issues and releases.
    19  
    20  Building
    21  --------
    22  
    23  Building is performed with the [goreleaser](https://goreleaser.com/) tool.  To build a snapshot (not release):
    24  
    25      $ goreleaser --clean --snapshot
    26  
    27  The binaries will be located in `./dist` directory.
    28  
    29  Testing and Usage
    30  -----------------
    31  
    32  Run this simple command to download a test file
    33  
    34      $ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./
    35  
    36  
    37  Configuration
    38  -------------
    39  `stashcp` is affected by the environment variables:
    40  
    41  | Environment Variable      | Description                                                                                                                                                                                                                                                         |
    42  | ----------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    43  | `OSG_DISABLE_PROXY_FALLBACK`      | Do not disable using proxies. By default, `stashcp` will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then `stashcp` will not fallback to no proxy if the proxy download fails. |
    44  | `STASHCP_MINIMUM_DOWNLOAD_SPEED`  | The lower limit a download will be cancelled, in bytes per second                                                                                                                                                                                                   |
    45  | `STASH_NAMESPACE_URL`             | The URL to download the namespace and cache information.  Default: https://topology.opensciencegrid.org/stashcache/namespaces                                                                                                                                                                                                 |
    46  | `OSDF_DIRECTOR_URL`               | The URL for an OSDF Director. If set, `stashcp` will get its list of geo-ordered caches from the Director. Default: Unset
    47  
    48  
    49  
    50  <!-- MARKDOWN LINKS & IMAGES -->
    51  [go-mod-version]: https://img.shields.io/github/go-mod/go-version/htcondor/osdf-client
    52  [github-build]: https://img.shields.io/github/actions/workflow/status/htcondor/osdf-client/release.yml
    53  [github-release-shield]: https://img.shields.io/github/v/release/htcondor/osdf-client
    54  [github-release]: https://github.com/htcondor/osdf-client/releases
    55  [github-previous-repo]: https://github.com/opensciencegrid/stashcp