github.com/willabides/benchdiff@v0.9.1/README.md (about)

     1  # benchdiff
     2  
     3  [![Go Reference](https://pkg.go.dev/badge/github.com/willabides/benchdiff.svg)](https://pkg.go.dev/github.com/willabides/benchdiff)
     4  [![ci](https://github.com/WillAbides/benchdiff/workflows/ci/badge.svg?branch=main&event=push)](https://github.com/WillAbides/benchdiff/actions?query=workflow%3Aci+branch%3Amain+event%3Apush)
     5  
     6  Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
     7  the delta.
     8  
     9  ## What it does
    10  
    11  Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference.
    12  
    13  It essentially combines multiple `git`, `go test -bench` and `benchstat` commands into one.
    14  
    15  These are the basic steps `benchdiff` performs:
    16  - Runs `go test -bench ...` on your current worktree and saves the results to a cache directory.
    17  - Creates a new worktree at the base ref
    18  - Runs `go test -bench ...` on the base worktree and saves the results to cache again.
    19  - Runs `benchstat` to compare the base and head results from cache.
    20  
    21  ## GitHub Action
    22  
    23  benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action
    24  
    25  ## Usage
    26  
    27  ### `benchdiff --help` output
    28  <!--- everything between the next line and the "end usage output" comment is generated by script/generate-readme --->
    29  <!--- start usage output --->
    30  ```
    31  Usage: benchdiff
    32  
    33  benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
    34  the delta.
    35  
    36  More documentation at https://github.com/willabides/benchdiff.
    37  
    38  Flags:
    39    -h, --help       Show context-sensitive help.
    40        --version    Output the benchdiff version and exit.
    41        --debug      write verbose output to stderr
    42  
    43    --base-ref="HEAD"    The git ref to be used as a baseline.
    44    --cooldown=100ms     How long to pause for cooldown between head and base runs.
    45    --force-base         Rerun benchmarks on the base reference even if the output already exists.
    46    --git-cmd="git"      The executable to use for git commands.
    47    --json               Format output as JSON.
    48    --on-degrade=0       Exit code when there is a statistically significant degradation in the
    49                         results.
    50    --tolerance=10.0     The minimum percent change before a result is considered degraded.
    51  
    52  benchmark command line
    53    --bench="."              Run only those benchmarks matching a regular expression. To run all
    54                             benchmarks, use '--bench .'.
    55    --benchmark-args=args    Override the default args to the go command. This may be a template.
    56                             See https://github.com/willabides/benchdiff for details."
    57    --benchmark-cmd="go"     The command to use for benchmarks.
    58    --benchmem               Memory allocation statistics for benchmarks.
    59    --benchtime=STRING       Run enough iterations of each benchmark to take t, specified as a
    60                             time.Duration (for example, --benchtime 1h30s). The default is 1 second
    61                             (1s). The special syntax Nx means to run the benchmark N times (for
    62                             example, -benchtime 100x).
    63    --count=10               Run each benchmark n times. If --cpu is set, run n times for each
    64                             GOMAXPROCS value.'
    65    --cpu=GOMAXPROCS,...     Specify a list of GOMAXPROCS values for which the benchmarks should be
    66                             executed. The default is the current value of GOMAXPROCS.
    67    --packages="./..."       Run benchmarks in these packages.
    68    --show-bench-cmdline     Instead of running benchmarks, output the command that would be used and
    69                             exit.
    70    --tags=STRING            Set the -tags flag on the go test command
    71    --warmup-count=INT       Run benchmarks with -count=n as a warmup
    72    --warmup-time=STRING     When warmups are run, set -benchtime=n
    73  
    74  benchstat options
    75    --alpha=0.05                 consider change significant if p < α
    76    --benchstat-output="text"    format for benchstat output (csv,html,markdown or text)
    77    --delta-test="utest"         significance test to apply to delta: utest, ttest, or none
    78    --geomean                    print the geometric mean of each file
    79    --norange                    suppress range columns (CSV and markdown only)
    80    --reverse-sort               reverse sort order
    81    --sort="none"                sort by order: delta, name, none
    82    --split="pkg,goos,goarch"    split benchmarks by labels
    83  
    84  benchmark result cache
    85    --cache-dir=STRING    Override the default directory where benchmark output is kept.
    86    --clear-cache         Remove benchdiff files from the cache dir.
    87    --show-cache-dir      Output the cache dir and exit.
    88  ```
    89  <!--- end usage output --->
    90  
    91  ### `--benchmark-args`
    92  
    93  The default should work for most cases.
    94  
    95  `--benchmark-args` is everything after "go" in the `go test` command that runs the benchmarks. You can use either 
    96  pass in the raw values or use a go template.
    97  
    98  Default value:
    99  
   100  <!--- start template --->
   101  ```
   102  test {{ .Packages }} -run '^$'
   103  {{- if .Bench }} -bench {{ .Bench }}{{end}}
   104  {{- if .Count }} -count {{ .Count }}{{end}}
   105  {{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}}
   106  {{- if .CPU }} -cpu {{ .CPU }}{{ end }}
   107  {{- if .Tags }} -tags "{{ .Tags }}"{{ end }}
   108  {{- if .Benchmem }} -benchmem{{ end }}
   109  ```
   110  <!--- end template --->
   111  
   112  ## Install
   113  
   114  ### go get
   115  
   116  `go get -u github.com/willabides/benchdiff/cmd/benchdiff`
   117  
   118  ### bindown
   119  
   120  Add a [bindown](https://github.com/willabides/bindown) dependency:
   121  
   122  ```
   123  $ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml
   124  $ bindown dependency add benchdiff benchdiff#benchdiff
   125  Please enter a value for required variable "version":	<latest version>
   126  ```