github.com/willabides/benchdiff@v0.9.1/README.md (about) 1 # benchdiff 2 3 [![Go Reference](https://pkg.go.dev/badge/github.com/willabides/benchdiff.svg)](https://pkg.go.dev/github.com/willabides/benchdiff) 4 [![ci](https://github.com/WillAbides/benchdiff/workflows/ci/badge.svg?branch=main&event=push)](https://github.com/WillAbides/benchdiff/actions?query=workflow%3Aci+branch%3Amain+event%3Apush) 5 6 Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show 7 the delta. 8 9 ## What it does 10 11 Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference. 12 13 It essentially combines multiple `git`, `go test -bench` and `benchstat` commands into one. 14 15 These are the basic steps `benchdiff` performs: 16 - Runs `go test -bench ...` on your current worktree and saves the results to a cache directory. 17 - Creates a new worktree at the base ref 18 - Runs `go test -bench ...` on the base worktree and saves the results to cache again. 19 - Runs `benchstat` to compare the base and head results from cache. 20 21 ## GitHub Action 22 23 benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action 24 25 ## Usage 26 27 ### `benchdiff --help` output 28 <!--- everything between the next line and the "end usage output" comment is generated by script/generate-readme ---> 29 <!--- start usage output ---> 30 ``` 31 Usage: benchdiff 32 33 benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show 34 the delta. 35 36 More documentation at https://github.com/willabides/benchdiff. 37 38 Flags: 39 -h, --help Show context-sensitive help. 40 --version Output the benchdiff version and exit. 41 --debug write verbose output to stderr 42 43 --base-ref="HEAD" The git ref to be used as a baseline. 44 --cooldown=100ms How long to pause for cooldown between head and base runs. 45 --force-base Rerun benchmarks on the base reference even if the output already exists. 46 --git-cmd="git" The executable to use for git commands. 47 --json Format output as JSON. 48 --on-degrade=0 Exit code when there is a statistically significant degradation in the 49 results. 50 --tolerance=10.0 The minimum percent change before a result is considered degraded. 51 52 benchmark command line 53 --bench="." Run only those benchmarks matching a regular expression. To run all 54 benchmarks, use '--bench .'. 55 --benchmark-args=args Override the default args to the go command. This may be a template. 56 See https://github.com/willabides/benchdiff for details." 57 --benchmark-cmd="go" The command to use for benchmarks. 58 --benchmem Memory allocation statistics for benchmarks. 59 --benchtime=STRING Run enough iterations of each benchmark to take t, specified as a 60 time.Duration (for example, --benchtime 1h30s). The default is 1 second 61 (1s). The special syntax Nx means to run the benchmark N times (for 62 example, -benchtime 100x). 63 --count=10 Run each benchmark n times. If --cpu is set, run n times for each 64 GOMAXPROCS value.' 65 --cpu=GOMAXPROCS,... Specify a list of GOMAXPROCS values for which the benchmarks should be 66 executed. The default is the current value of GOMAXPROCS. 67 --packages="./..." Run benchmarks in these packages. 68 --show-bench-cmdline Instead of running benchmarks, output the command that would be used and 69 exit. 70 --tags=STRING Set the -tags flag on the go test command 71 --warmup-count=INT Run benchmarks with -count=n as a warmup 72 --warmup-time=STRING When warmups are run, set -benchtime=n 73 74 benchstat options 75 --alpha=0.05 consider change significant if p < α 76 --benchstat-output="text" format for benchstat output (csv,html,markdown or text) 77 --delta-test="utest" significance test to apply to delta: utest, ttest, or none 78 --geomean print the geometric mean of each file 79 --norange suppress range columns (CSV and markdown only) 80 --reverse-sort reverse sort order 81 --sort="none" sort by order: delta, name, none 82 --split="pkg,goos,goarch" split benchmarks by labels 83 84 benchmark result cache 85 --cache-dir=STRING Override the default directory where benchmark output is kept. 86 --clear-cache Remove benchdiff files from the cache dir. 87 --show-cache-dir Output the cache dir and exit. 88 ``` 89 <!--- end usage output ---> 90 91 ### `--benchmark-args` 92 93 The default should work for most cases. 94 95 `--benchmark-args` is everything after "go" in the `go test` command that runs the benchmarks. You can use either 96 pass in the raw values or use a go template. 97 98 Default value: 99 100 <!--- start template ---> 101 ``` 102 test {{ .Packages }} -run '^$' 103 {{- if .Bench }} -bench {{ .Bench }}{{end}} 104 {{- if .Count }} -count {{ .Count }}{{end}} 105 {{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}} 106 {{- if .CPU }} -cpu {{ .CPU }}{{ end }} 107 {{- if .Tags }} -tags "{{ .Tags }}"{{ end }} 108 {{- if .Benchmem }} -benchmem{{ end }} 109 ``` 110 <!--- end template ---> 111 112 ## Install 113 114 ### go get 115 116 `go get -u github.com/willabides/benchdiff/cmd/benchdiff` 117 118 ### bindown 119 120 Add a [bindown](https://github.com/willabides/bindown) dependency: 121 122 ``` 123 $ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml 124 $ bindown dependency add benchdiff benchdiff#benchdiff 125 Please enter a value for required variable "version": <latest version> 126 ```