github.com/htcondor/osdf-client/v6@v6.13.0-rc1.0.20231009141709-766e7b4d1dc8/README.md (about) 1 Open Science Data Federation (OSDF) Client 2 ========================================== 3 4 [![Version][github-release-shield]][github-release] 5 ![Go Mod][go-mod-version] 6 ![Builds][github-build] 7 8 The OSDF Client downloads files from the [Open Science Data Federation](https://osdf.osg-htc.org/) through a series of caches. The OSDF is used by multiple organizations to effeciently transfer input and output data. 9 10 The client consists of two parts: a command-line tool named StashCP, and an [HTCondor](https://htcondor.org/) file transfer [plugin](https://htcondor.readthedocs.io/en/latest/admin-manual/setting-up-special-environments.html#enabling-the-transfer-of-files-specified-by-a-url). 11 12 When configured, this plugin will allow the user to specify `transfer_input_files` with the `stash://` protocol which will be downloaded through the OSDF caches. An example of a submit file: 13 14 ... 15 transfer_input_files = stash:///osgconnect/public/dweitzel/blast/queries/query1 16 ... 17 18 Note: This repo is the continuation of [opensciencegrid/stashcp][github-previous-repo]; visit that repo for old issues and releases. 19 20 Building 21 -------- 22 23 Building is performed with the [goreleaser](https://goreleaser.com/) tool. To build a snapshot (not release): 24 25 $ goreleaser --clean --snapshot 26 27 The binaries will be located in `./dist` directory. 28 29 Testing and Usage 30 ----------------- 31 32 Run this simple command to download a test file 33 34 $ ./stashcp /osgconnect/public/dweitzel/blast/queries/query1 ./ 35 36 37 Configuration 38 ------------- 39 `stashcp` is affected by the environment variables: 40 41 | Environment Variable | Description | 42 | ----------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 43 | `OSG_DISABLE_PROXY_FALLBACK` | Do not disable using proxies. By default, `stashcp` will try to use an HTTP proxy when connecting to a cache. If this environment variable is set (no value necessary, only if it's set), then `stashcp` will not fallback to no proxy if the proxy download fails. | 44 | `STASHCP_MINIMUM_DOWNLOAD_SPEED` | The lower limit a download will be cancelled, in bytes per second | 45 | `STASH_NAMESPACE_URL` | The URL to download the namespace and cache information. Default: https://topology.opensciencegrid.org/stashcache/namespaces | 46 | `OSDF_DIRECTOR_URL` | The URL for an OSDF Director. If set, `stashcp` will get its list of geo-ordered caches from the Director. Default: Unset 47 48 49 50 <!-- MARKDOWN LINKS & IMAGES --> 51 [go-mod-version]: https://img.shields.io/github/go-mod/go-version/htcondor/osdf-client 52 [github-build]: https://img.shields.io/github/actions/workflow/status/htcondor/osdf-client/release.yml 53 [github-release-shield]: https://img.shields.io/github/v/release/htcondor/osdf-client 54 [github-release]: https://github.com/htcondor/osdf-client/releases 55 [github-previous-repo]: https://github.com/opensciencegrid/stashcp