gopkg.in/cavaliercoder/grab.v2@v2.0.0/README.md (about) 1 # grab 2 3 [![GoDoc](https://godoc.org/github.com/cavaliercoder/grab?status.svg)](https://godoc.org/github.com/cavaliercoder/grab) [![Build Status](https://travis-ci.org/cavaliercoder/grab.svg?branch=master)](https://travis-ci.org/cavaliercoder/grab) [![Go Report Card](https://goreportcard.com/badge/github.com/cavaliercoder/grab)](https://goreportcard.com/report/github.com/cavaliercoder/grab) 4 5 *Downloading the internet, one goroutine at a time!* 6 7 $ go get github.com/cavaliercoder/grab 8 9 Grab is a Go package for downloading files from the internet with the following 10 rad features: 11 12 * Monitor download progress concurrently 13 * Auto-resume incomplete downloads 14 * Guess filename from content header or URL path 15 * Safely cancel downloads using context.Context 16 * Validate downloads using checksums 17 * Download batches of files concurrently 18 * Apply rate limiters 19 20 Requires Go v1.7+ 21 22 ## Example 23 24 The following example downloads a PDF copy of the free eBook, "An Introduction 25 to Programming in Go" into the current working directory. 26 27 ```go 28 resp, err := grab.Get(".", "http://www.golang-book.com/public/pdf/gobook.pdf") 29 if err != nil { 30 log.Fatal(err) 31 } 32 33 fmt.Println("Download saved to", resp.Filename) 34 ``` 35 36 The following, more complete example allows for more granular control and 37 periodically prints the download progress until it is complete. 38 39 The second time you run the example, it will auto-resume the previous download 40 and exit sooner. 41 42 ```go 43 package main 44 45 import ( 46 "fmt" 47 "os" 48 "time" 49 50 "github.com/cavaliercoder/grab" 51 ) 52 53 func main() { 54 // create client 55 client := grab.NewClient() 56 req, _ := grab.NewRequest(".", "http://www.golang-book.com/public/pdf/gobook.pdf") 57 58 // start download 59 fmt.Printf("Downloading %v...\n", req.URL()) 60 resp := client.Do(req) 61 fmt.Printf(" %v\n", resp.HTTPResponse.Status) 62 63 // start UI loop 64 t := time.NewTicker(500 * time.Millisecond) 65 defer t.Stop() 66 67 Loop: 68 for { 69 select { 70 case <-t.C: 71 fmt.Printf(" transferred %v / %v bytes (%.2f%%)\n", 72 resp.BytesComplete(), 73 resp.Size, 74 100*resp.Progress()) 75 76 case <-resp.Done: 77 // download is complete 78 break Loop 79 } 80 } 81 82 // check for errors 83 if err := resp.Err(); err != nil { 84 fmt.Fprintf(os.Stderr, "Download failed: %v\n", err) 85 os.Exit(1) 86 } 87 88 fmt.Printf("Download saved to ./%v \n", resp.Filename) 89 90 // Output: 91 // Downloading http://www.golang-book.com/public/pdf/gobook.pdf... 92 // 200 OK 93 // transferred 42970 / 2893557 bytes (1.49%) 94 // transferred 1207474 / 2893557 bytes (41.73%) 95 // transferred 2758210 / 2893557 bytes (95.32%) 96 // Download saved to ./gobook.pdf 97 } 98 ``` 99 100 ## Design trade-offs 101 102 The primary use case for Grab is to concurrently downloading thousands of large 103 files from remote file repositories where the remote files are immutable. 104 Examples include operating system package repositories or ISO libraries. 105 106 Grab aims to provide robust, sane defaults. These are usually determined using 107 the HTTP specifications, or by mimicking the behavior of common web clients like 108 cURL, wget and common web browsers. 109 110 Grab aims to be stateless. The only state that exists is the remote files you 111 wish to download and the local copy which may be completed, partially completed 112 or not yet created. The advantage to this is that the local file system is not 113 cluttered unnecessarily with addition state files (like a `.crdownload` file). 114 The disadvantage of this approach is that grab must make assumptions about the 115 local and remote state; specifically, that they have not been modified by 116 another program. 117 118 If the local or remote file are modified outside of grab, and you download the 119 file again with resuming enabled, the local file will likely become corrupted. 120 In this case, you might consider making remote files immutable, or disabling 121 resume. 122 123 Grab aims to enable best-in-class functionality for more complex features 124 through extensible interfaces, rather than reimplementation. For example, 125 you can provide your own Hash algorithm to compute file checksums, or your 126 own rate limiter implementation (with all the associated trade-offs) to rate 127 limit downloads.