github.com/isi-lincoln/grab@v2.0.1-0.20200331080741-9f014744ee41+incompatible/README.md (about)

     1  # grab
     2  
     3  [![GoDoc](https://godoc.org/github.com/cavaliercoder/grab?status.svg)](https://godoc.org/github.com/cavaliercoder/grab) [![Build Status](https://travis-ci.org/cavaliercoder/grab.svg?branch=master)](https://travis-ci.org/cavaliercoder/grab) [![Go Report Card](https://goreportcard.com/badge/github.com/cavaliercoder/grab)](https://goreportcard.com/report/github.com/cavaliercoder/grab)
     4  
     5  *Downloading the internet, one goroutine at a time!*
     6  
     7  	$ go get github.com/cavaliercoder/grab
     8  
     9  Grab is a Go package for downloading files from the internet with the following
    10  rad features:
    11  
    12  * Monitor download progress concurrently
    13  * Auto-resume incomplete downloads
    14  * Guess filename from content header or URL path
    15  * Safely cancel downloads using context.Context
    16  * Validate downloads using checksums
    17  * Download batches of files concurrently
    18  * Apply rate limiters
    19  
    20  Requires Go v1.7+
    21  
    22  ## Example
    23  
    24  The following example downloads a PDF copy of the free eBook, "An Introduction
    25  to Programming in Go" into the current working directory.
    26  
    27  ```go
    28  resp, err := grab.Get(".", "http://www.golang-book.com/public/pdf/gobook.pdf")
    29  if err != nil {
    30  	log.Fatal(err)
    31  }
    32  
    33  fmt.Println("Download saved to", resp.Filename)
    34  ```
    35  
    36  The following, more complete example allows for more granular control and
    37  periodically prints the download progress until it is complete.
    38  
    39  The second time you run the example, it will auto-resume the previous download
    40  and exit sooner.
    41  
    42  ```go
    43  package main
    44  
    45  import (
    46  	"fmt"
    47  	"os"
    48  	"time"
    49  
    50  	"github.com/cavaliercoder/grab"
    51  )
    52  
    53  func main() {
    54  	// create client
    55  	client := grab.NewClient()
    56  	req, _ := grab.NewRequest(".", "http://www.golang-book.com/public/pdf/gobook.pdf")
    57  
    58  	// start download
    59  	fmt.Printf("Downloading %v...\n", req.URL())
    60  	resp := client.Do(req)
    61  	fmt.Printf("  %v\n", resp.HTTPResponse.Status)
    62  
    63  	// start UI loop
    64  	t := time.NewTicker(500 * time.Millisecond)
    65  	defer t.Stop()
    66  
    67  Loop:
    68  	for {
    69  		select {
    70  		case <-t.C:
    71  			fmt.Printf("  transferred %v / %v bytes (%.2f%%)\n",
    72  				resp.BytesComplete(),
    73  				resp.Size(),
    74  				100*resp.Progress())
    75  
    76  		case <-resp.Done:
    77  			// download is complete
    78  			break Loop
    79  		}
    80  	}
    81  
    82  	// check for errors
    83  	if err := resp.Err(); err != nil {
    84  		fmt.Fprintf(os.Stderr, "Download failed: %v\n", err)
    85  		os.Exit(1)
    86  	}
    87  
    88  	fmt.Printf("Download saved to ./%v \n", resp.Filename)
    89  
    90  	// Output:
    91  	// Downloading http://www.golang-book.com/public/pdf/gobook.pdf...
    92  	//   200 OK
    93  	//   transferred 42970 / 2893557 bytes (1.49%)
    94  	//   transferred 1207474 / 2893557 bytes (41.73%)
    95  	//   transferred 2758210 / 2893557 bytes (95.32%)
    96  	// Download saved to ./gobook.pdf
    97  }
    98  ```
    99  
   100  ## Design trade-offs
   101  
   102  The primary use case for Grab is to concurrently downloading thousands of large
   103  files from remote file repositories where the remote files are immutable.
   104  Examples include operating system package repositories or ISO libraries.
   105  
   106  Grab aims to provide robust, sane defaults. These are usually determined using
   107  the HTTP specifications, or by mimicking the behavior of common web clients like
   108  cURL, wget and common web browsers.
   109  
   110  Grab aims to be stateless. The only state that exists is the remote files you
   111  wish to download and the local copy which may be completed, partially completed
   112  or not yet created. The advantage to this is that the local file system is not
   113  cluttered unnecessarily with addition state files (like a `.crdownload` file).
   114  The disadvantage of this approach is that grab must make assumptions about the
   115  local and remote state; specifically, that they have not been modified by
   116  another program.
   117  
   118  If the local or remote file are modified outside of grab, and you download the
   119  file again with resuming enabled, the local file will likely become corrupted.
   120  In this case, you might consider making remote files immutable, or disabling
   121  resume.
   122  
   123  Grab aims to enable best-in-class functionality for more complex features
   124  through extensible interfaces, rather than reimplementation. For example,
   125  you can provide your own Hash algorithm to compute file checksums, or your
   126  own rate limiter implementation (with all the associated trade-offs) to rate
   127  limit downloads.