github.com/remind101/go-getter@v0.0.0-20180809191950-4bda8fa99001/README.md (about) 1 # go-getter 2 3 [][travis] 4 [][appveyor] 5 [][godocs] 6 7 [travis]: http://travis-ci.org/hashicorp/go-getter 8 [godocs]: http://godoc.org/github.com/hashicorp/go-getter 9 [appveyor]: https://ci.appveyor.com/project/hashicorp/go-getter/branch/master 10 11 go-getter is a library for Go (golang) for downloading files or directories 12 from various sources using a URL as the primary form of input. 13 14 The power of this library is being flexible in being able to download 15 from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) 16 using a single string as input. This removes the burden of knowing how to 17 download from a variety of sources from the implementer. 18 19 The concept of a _detector_ automatically turns invalid URLs into proper 20 URLs. For example: "github.com/hashicorp/go-getter" would turn into a 21 Git URL. Or "./foo" would turn into a file URL. These are extensible. 22 23 This library is used by [Terraform](https://terraform.io) for 24 downloading modules and [Nomad](https://nomadproject.io) for downloading 25 binaries. 26 27 ## Installation and Usage 28 29 Package documentation can be found on 30 [GoDoc](http://godoc.org/github.com/hashicorp/go-getter). 31 32 Installation can be done with a normal `go get`: 33 34 ``` 35 $ go get github.com/hashicorp/go-getter 36 ``` 37 38 go-getter also has a command you can use to test URL strings: 39 40 ``` 41 $ go install github.com/hashicorp/go-getter/cmd/go-getter 42 ... 43 44 $ go-getter github.com/foo/bar ./foo 45 ... 46 ``` 47 48 The command is useful for verifying URL structures. 49 50 ## URL Format 51 52 go-getter uses a single string URL as input to download from a variety of 53 protocols. go-getter has various "tricks" with this URL to do certain things. 54 This section documents the URL format. 55 56 ### Supported Protocols and Detectors 57 58 **Protocols** are used to download files/directories using a specific 59 mechanism. Example protocols are Git and HTTP. 60 61 **Detectors** are used to transform a valid or invalid URL into another 62 URL if it matches a certain pattern. Example: "github.com/user/repo" is 63 automatically transformed into a fully valid Git URL. This allows go-getter 64 to be very user friendly. 65 66 go-getter out of the box supports the following protocols. Additional protocols 67 can be augmented at runtime by implementing the `Getter` interface. 68 69 * Local files 70 * Git 71 * Mercurial 72 * HTTP 73 * Amazon S3 74 75 In addition to the above protocols, go-getter has what are called "detectors." 76 These take a URL and attempt to automatically choose the best protocol for 77 it, which might involve even changing the protocol. The following detection 78 is built-in by default: 79 80 * File paths such as "./foo" are automatically changed to absolute 81 file URLs. 82 * GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically 83 changed to Git protocol over HTTP. 84 * BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically 85 changed to a Git or mercurial protocol using the BitBucket API. 86 87 ### Forced Protocol 88 89 In some cases, the protocol to use is ambiguous depending on the source 90 URL. For example, "http://github.com/mitchellh/vagrant.git" could reference 91 an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this 92 URL. 93 94 Forced protocol can be done by prefixing the URL with the protocol followed 95 by double colons. For example: `git::http://github.com/mitchellh/vagrant.git` 96 would download the given HTTP URL using the Git protocol. 97 98 Forced protocols will also override any detectors. 99 100 In the absense of a forced protocol, detectors may be run on the URL, transforming 101 the protocol anyways. The above example would've used the Git protocol either 102 way since the Git detector would've detected it was a GitHub URL. 103 104 ### Protocol-Specific Options 105 106 Each protocol can support protocol-specific options to configure that 107 protocol. For example, the `git` protocol supports specifying a `ref` 108 query parameter that tells it what ref to checkout for that Git 109 repository. 110 111 The options are specified as query parameters on the URL (or URL-like string) 112 given to go-getter. Using the Git example above, the URL below is a valid 113 input to go-getter: 114 115 github.com/hashicorp/go-getter?ref=abcd1234 116 117 The protocol-specific options are documented below the URL format 118 section. But because they are part of the URL, we point it out here so 119 you know they exist. 120 121 ### Subdirectories 122 123 If you want to download only a specific subdirectory from a downloaded 124 directory, you can specify a subdirectory after a double-slash `//`. 125 go-getter will first download the URL specified _before_ the double-slash 126 (as if you didn't specify a double-slash), but will then copy the 127 path after the double slash into the target directory. 128 129 For example, if you're downloading this GitHub repository, but you only 130 want to download the `test-fixtures` directory, you can do the following: 131 132 ``` 133 https://github.com/hashicorp/go-getter.git//test-fixtures 134 ``` 135 136 If you downloaded this to the `/tmp` directory, then the file 137 `/tmp/archive.gz` would exist. Notice that this file is in the `test-fixtures` 138 directory in this repository, but because we specified a subdirectory, 139 go-getter automatically copied only that directory contents. 140 141 Subdirectory paths may contain may also use filesystem glob patterns. 142 The path must match _exactly one_ entry or go-getter will return an error. 143 This is useful if you're not sure the exact directory name but it follows 144 a predictable naming structure. 145 146 For example, the following URL would also work: 147 148 ``` 149 https://github.com/hashicorp/go-getter.git//test-* 150 ``` 151 152 ### Checksumming 153 154 For file downloads of any protocol, go-getter can automatically verify 155 a checksum for you. Note that checksumming only works for downloading files, 156 not directories, but checksumming will work for any protocol. 157 158 To checksum a file, append a `checksum` query parameter to the URL. 159 The paramter value should be in the format of `type:value`, where 160 type is "md5", "sha1", "sha256", or "sha512". The "value" should be 161 the actual checksum value. go-getter will parse out this query parameter 162 automatically and use it to verify the checksum. An example URL 163 is shown below: 164 165 ``` 166 ./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21 167 ``` 168 169 The checksum query parameter is never sent to the backend protocol 170 implementation. It is used at a higher level by go-getter itself. 171 172 ### Unarchiving 173 174 go-getter will automatically unarchive files into a file or directory 175 based on the extension of the file being requested (over any protocol). 176 This works for both file and directory downloads. 177 178 go-getter looks for an `archive` query parameter to specify the format of 179 the archive. If this isn't specified, go-getter will use the extension of 180 the path to see if it appears archived. Unarchiving can be explicitly 181 disabled by setting the `archive` query parameter to `false`. 182 183 The following archive formats are supported: 184 185 * `tar.gz` and `tgz` 186 * `tar.bz2` and `tbz2` 187 * `tar.xz` and `txz` 188 * `zip` 189 * `gz` 190 * `bz2` 191 * `xz` 192 193 For example, an example URL is shown below: 194 195 ``` 196 ./foo.zip 197 ``` 198 199 This will automatically be inferred to be a ZIP file and will be extracted. 200 You can also be explicit about the archive type: 201 202 ``` 203 ./some/other/path?archive=zip 204 ``` 205 206 And finally, you can disable archiving completely: 207 208 ``` 209 ./some/path?archive=false 210 ``` 211 212 You can combine unarchiving with the other features of go-getter such 213 as checksumming. The special `archive` query parameter will be removed 214 from the URL before going to the final protocol downloader. 215 216 ## Protocol-Specific Options 217 218 This section documents the protocol-specific options that can be specified 219 for go-getter. These options should be appended to the input as normal query 220 parameters. Depending on the usage of go-getter, applications may provide 221 alternate ways of inputting options. For example, [Nomad](https://www.nomadproject.io) 222 provides a nice options block for specifying options rather than in the URL. 223 224 ## General (All Protocols) 225 226 The options below are available to all protocols: 227 228 * `archive` - The archive format to use to unarchive this file, or "" (empty 229 string) to disable unarchiving. For more details, see the complete section 230 on archive support above. 231 232 * `checksum` - Checksum to verify the downloaded file or archive. See 233 the entire section on checksumming above for format and more details. 234 235 * `filename` - When in file download mode, allows specifying the name of the 236 downloaded file on disk. Has no effect in directory mode. 237 238 ### Local Files (`file`) 239 240 None 241 242 ### Git (`git`) 243 244 * `ref` - The Git ref to checkout. This is a ref, so it can point to 245 a commit SHA, a branch name, etc. If it is a named ref such as a branch 246 name, go-getter will update it to the latest on each get. 247 248 * `sshkey` - An SSH private key to use during clones. The provided key must 249 be a base64-encoded string. For example, to generate a suitable `sshkey` 250 from a private key file on disk, you would run `base64 -w0 <file>`. 251 252 **Note**: Git 2.3+ is required to use this feature. 253 254 ### Mercurial (`hg`) 255 256 * `rev` - The Mercurial revision to checkout. 257 258 ### HTTP (`http`) 259 260 #### Basic Authentication 261 262 To use HTTP basic authentication with go-getter, simply prepend `username:password@` to the 263 hostname in the URL such as `https://Aladdin:OpenSesame@www.example.com/index.html`. All special 264 characters, including the username and password, must be URL encoded. 265 266 ### S3 (`s3`) 267 268 S3 takes various access configurations in the URL. Note that it will also 269 read these from standard AWS environment variables if they're set. S3 compliant servers like Minio 270 are also supported. If the query parameters are present, these take priority. 271 272 * `aws_access_key_id` - AWS access key. 273 * `aws_access_key_secret` - AWS access key secret. 274 * `aws_access_token` - AWS access token if this is being used. 275 276 #### Using IAM Instance Profiles with S3 277 278 If you use go-getter and want to use an EC2 IAM Instance Profile to avoid 279 using credentials, then just omit these and the profile, if available will 280 be used automatically. 281 282 ### Using S3 with Minio 283 If you use go-gitter for Minio support, you must consider the following: 284 285 * `aws_access_key_id` (required) - Minio access key. 286 * `aws_access_key_secret` (required) - Minio access key secret. 287 * `region` (optional - defaults to us-east-1) - Region identifier to use. 288 * `version` (optional - defaults to Minio default) - Configuration file format. 289 290 #### S3 Bucket Examples 291 292 S3 has several addressing schemes used to reference your bucket. These are 293 listed here: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro 294 295 Some examples for these addressing schemes: 296 - s3::https://s3.amazonaws.com/bucket/foo 297 - s3::https://s3-eu-west-1.amazonaws.com/bucket/foo 298 - bucket.s3.amazonaws.com/foo 299 - bucket.s3-eu-west-1.amazonaws.com/foo/bar 300 - "s3::http://127.0.0.1:9000/test-bucket/hello.txt?aws_access_key_id=KEYID&aws_access_key_secret=SECRETKEY®ion=us-east-2" 301