gotest.tools/gotestsum@v1.11.0/README.md (about) 1 # gotestsum 2 3 `gotestsum` runs tests using `go test -json`, prints formatted test output, and a summary of the test run. 4 It is designed to work well for both local development, and for automation like CI. 5 `gotestsum` is [used by](#who-uses-gotestsum) some of the most popular Go projects. 6 7 ## Install 8 9 Download a binary from [releases](https://github.com/gotestyourself/gotestsum/releases), or build from 10 source with `go install gotest.tools/gotestsum@latest`. To run without installing use 11 `go run gotest.tools/gotestsum@latest`. 12 13 ## Documentation 14 15 **Core features** 16 - Change the [test output format](#output-format), from compact to verbose with color highlighting. 17 - Print a [summary](#summary) of the test run after running all the tests. 18 - Use any [`go test` flag](#custom-go-test-command), 19 run a script with [`--raw-command`](#custom-go-test-command), 20 or [run a compiled test binary](#executing-a-compiled-test-binary). 21 22 **CI and Automation** 23 - [`--junitfile`](#junit-xml-output) - write a JUnit XML file for integration with CI systems. 24 - [`--jsonfile`](#json-file-output) - write all the [test2json](https://pkg.go.dev/cmd/test2json) input received by `gotestsum` to a file. The file 25 can be used as input to [`gotestsum tool slowest`](#finding-and-skipping-slow-tests), or as a way to 26 store the full verbose output of tests when less verbose output is printed to stdout using a compact [`--format`](#output-format). 27 - [`--rerun-fails`](#re-running-failed-tests) - run failed (possibly flaky) tests again to avoid re-running the 28 entire suite. Re-running individual tests can save significant time when working with flaky test suites. 29 30 **Local Development** 31 - [`--watch`](#run-tests-when-a-file-is-saved) - every time a `.go` file is saved run the tests for the package that changed. 32 - [`--post-run-command`](#post-run-command) - run a command after the tests, can be used for desktop notification of the test run. 33 - [`gotestsum tool slowest`](#finding-and-skipping-slow-tests) - find the slowest tests, or automatically update the source code of 34 the slowest tests to add a conditional `t.Skip` statements. This statement allows you to skip the slowest tests using `gotestsum -- -short ./...`. 35 36 37 ### Output Format 38 39 The `--format` flag or `GOTESTSUM_FORMAT` environment variable set the format that 40 is used to print the test names, and possibly test output, as the tests run. Most 41 outputs use color to highlight pass, fail, or skip. 42 43 The `--format-hivis` flag changes the icons used by `pkgname` formats to higher 44 visiblity unicode characters. 45 46 Commonly used formats (see `--help` for a full list): 47 48 * `dots` - print a character for each test. 49 * `pkgname` (default) - print a line for each package. 50 * `testname` - print a line for each test and package. 51 * `testdox` - print a sentence for each test using [gotestdox](https://github.com/bitfield/gotestdox). 52 * `standard-quiet` - the standard `go test` format. 53 * `standard-verbose` - the standard `go test -v` format. 54 55 Have an idea for a new format? 56 Please [share it on github](https://github.com/gotestyourself/gotestsum/issues/new)! 57 58 #### Demo 59 60 A demonstration of three `--format` options. 61 62 ![Demo](https://user-images.githubusercontent.com/442180/182284939-e08a0aa5-4504-4e30-9e88-207ef47f4537.gif) 63 <br /><sup>[Source](https://github.com/gotestyourself/gotestsum/tree/readme-demo/scripts)</sup> 64 65 ### Summary 66 67 Following the formatted output is a summary of the test run. The summary includes: 68 69 * The test output, and elapsed time, for any test that fails or is skipped. 70 * The build errors for any package that fails to build. 71 * A `DONE` line with a count of tests run, tests skipped, tests failed, package build errors, 72 and the elapsed time including time to build. 73 74 ``` 75 DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s 76 ``` 77 78 To hide parts of the summary use `--hide-summary section`. 79 80 81 **Example: hide skipped tests in the summary** 82 ``` 83 gotestsum --hide-summary=skipped 84 ``` 85 86 **Example: hide everything except the DONE line** 87 ``` 88 gotestsum --hide-summary=skipped,failed,errors,output 89 # or 90 gotestsum --hide-summary=all 91 ``` 92 93 **Example: hide test output in the summary, only print names of failed and skipped tests 94 and errors** 95 ``` 96 gotestsum --hide-summary=output 97 ``` 98 99 ### JUnit XML output 100 101 When the `--junitfile` flag or `GOTESTSUM_JUNITFILE` environment variable are set 102 to a file path, `gotestsum` will write a test report, in JUnit XML format, to the file. 103 This file can be used to integrate with CI systems. 104 105 ``` 106 gotestsum --junitfile unit-tests.xml 107 ``` 108 109 If the package names in the `testsuite.name` or `testcase.classname` fields do not 110 work with your CI system these values can be customized using the 111 `--junitfile-testsuite-name`, or `--junitfile-testcase-classname` flags. These flags 112 accept the following values: 113 114 * `short` - the base name of the package (the single term specified by the 115 package statement). 116 * `relative` - a package path relative to the root of the repository 117 * `full` - the full package path (default) 118 119 120 Note: If Go is not installed, or the `go` binary is not in `PATH`, the `GOVERSION` 121 environment variable can be set to remove the "failed to lookup go version for junit xml" 122 warning. 123 124 ### JSON file output 125 126 When the `--jsonfile` flag or `GOTESTSUM_JSONFILE` environment variable are set 127 to a file path, `gotestsum` will write a line-delimited JSON file with all the 128 [test2json](https://golang.org/cmd/test2json/#hdr-Output_Format) 129 output that was written by `go test -json`. This file can be used to compare test 130 runs, or find flaky tests. 131 132 ``` 133 gotestsum --jsonfile test-output.log 134 ``` 135 136 ### Post Run Command 137 138 The `--post-run-command` flag may be used to execute a command after the 139 test run has completed. The binary will be run with the following environment 140 variables set: 141 142 ``` 143 GOTESTSUM_ELAPSED # test run time in seconds (ex: 2.45s) 144 GOTESTSUM_FORMAT # gotestsum format (ex: pkgname) 145 GOTESTSUM_JSONFILE # path to the jsonfile, empty if no file path was given 146 GOTESTSUM_JUNITFILE # path to the junit.xml file, empty if no file path was given 147 TESTS_ERRORS # number of errors 148 TESTS_FAILED # number of failed tests 149 TESTS_SKIPPED # number of skipped tests 150 TESTS_TOTAL # number of tests run 151 ``` 152 153 To get more details about the test run, such as failure messages or the full list of failed 154 tests, run `gotestsum` with either a `--jsonfile` or `--junitfile` and parse the 155 file from the post-run-command. The 156 [gotestsum/testjson](https://pkg.go.dev/gotest.tools/gotestsum/testjson?tab=doc) 157 package may be used to parse the JSON file output. 158 159 **Example: desktop notifications** 160 161 First install the example notification command with `go get gotest.tools/gotestsum/contrib/notify`. 162 The command will be downloaded to `$GOPATH/bin` as `notify`. Note that this 163 example `notify` command only works on Linux with `notify-send` and on macOS with 164 [terminal-notifer](https://github.com/julienXX/terminal-notifier) installed. 165 166 On Linux, you need to have some "test-pass" and "test-fail" icons installed in your icon theme. 167 Some sample icons can be found in `contrib/notify`, and can be installed with `make install`. 168 169 On Windows, you can install [notify-send.exe](https://github.com/vaskovsky/notify-send) 170 but it does not support custom icons so will have to use the basic "info" and "error". 171 172 ``` 173 gotestsum --post-run-command notify 174 ``` 175 176 **Example: command with flags** 177 178 Possitional arguments or command line flags can be passed to the `--post-run-command` by 179 quoting the whole command. 180 181 ``` 182 gotestsum --post-run-command "notify me --date" 183 ``` 184 185 **Example: printing slowest tests** 186 187 The post-run command can be combined with other `gotestsum` commands and tools to provide 188 a more detailed summary. This example uses `gotestsum tool slowest` to print the 189 slowest 10 tests after the summary. 190 191 ``` 192 gotestsum \ 193 --jsonfile tmp.json.log \ 194 --post-run-command "bash -c ' 195 echo; echo Slowest tests; 196 gotestsum tool slowest --num 10 --jsonfile tmp.json.log'" 197 ``` 198 199 ### Re-running failed tests 200 201 When the `--rerun-fails` flag is set, `gotestsum` will re-run any failed tests. 202 The tests will be re-run until each passes once, or the number of attempts 203 exceeds the maximum attempts. Maximum attempts defaults to 2, and can be changed 204 with `--rerun-fails=n`. 205 206 To avoid re-running tests when there are real failures, the re-run will be 207 skipped when there are too many test failures. By default this value is 10, and 208 can be changed with `--rerun-fails-max-failures=n`. 209 210 Note that using `--rerun-fails` may require the use of other flags, depending on 211 how you specify args to `go test`: 212 213 * when used with `--raw-command` the re-run will pass additional arguments to 214 the command. The first arg is a `-test.run` flag with a regex that matches the test to re-run, 215 and second is the name of a go package. These additional args can be passed to `go test`, 216 or a test binary. 217 * when used with any `go test` args (anything after `--` on the command line), the list of 218 packages to test must be specified as a space separated list using the `--packages` arg. 219 220 **Example** 221 222 ``` 223 gotestsum --rerun-fails --packages="./..." -- -count=2 224 ``` 225 226 * if any of the `go test` args should be passed to the test binary, instead of 227 `go test` itself, the `-args` flag must be used to separate the two groups of 228 arguments. `-args` is a special flag that is understood by `go test` to indicate 229 that any following args should be passed directly to the test binary. 230 231 **Example** 232 233 ``` 234 gotestsum --rerun-fails --packages="./..." -- -count=2 -args -update-golden 235 ``` 236 237 238 ### Custom `go test` command 239 240 By default `gotestsum` runs tests using the command `go test -json ./...`. You 241 can change the command with positional arguments after a `--`. You can change just the 242 test directory value (which defaults to `./...`) by setting the `TEST_DIRECTORY` 243 environment variable. 244 245 You can use `--debug` to echo the command before it is run. 246 247 **Example: set build tags** 248 ``` 249 gotestsum -- -tags=integration ./... 250 ``` 251 252 **Example: run tests in a single package** 253 ``` 254 gotestsum -- ./io/http 255 ``` 256 257 **Example: enable coverage** 258 ``` 259 gotestsum -- -coverprofile=cover.out ./... 260 ``` 261 262 **Example: run a script instead of `go test`** 263 ``` 264 gotestsum --raw-command -- ./scripts/run_tests.sh 265 ``` 266 267 Note: when using `--raw-command`, the script must follow a few rules about 268 stdout and stderr output: 269 270 * The stdout produced by the script must only contain the `test2json` output, or 271 `gotestsum` will fail. If it isn't possible to change the script to avoid 272 non-JSON output, you can use `--ignore-non-json-output-lines` (added in version 1.7.0) 273 to ignore non-JSON lines and write them to `gotestsum`'s stderr instead. 274 * Any stderr produced by the script will be considered an error (this behaviour 275 is necessary because package build errors are only reported by writting to 276 stderr, not the `test2json` stdout). Any stderr produced by tests is not 277 considered an error (it will be in the `test2json` stdout). 278 279 **Example: accept intput from stdin** 280 ``` 281 cat out.json | gotestsum --raw-command -- cat 282 ``` 283 284 **Example: run tests with profiling enabled** 285 286 Using a `profile.sh` script like this: 287 288 ```sh 289 #!/usr/bin/env bash 290 set -eu 291 292 for pkg in $(go list "$@"); do 293 dir="$(go list -f '{{ .Dir }}' $pkg)" 294 go test -json -cpuprofile="$dir/cpu.profile" "$pkg" 295 done 296 ``` 297 298 You can run: 299 ``` 300 gotestsum --raw-command ./profile.sh ./... 301 ``` 302 303 **Example: using `TEST_DIRECTORY`** 304 ``` 305 TEST_DIRECTORY=./io/http gotestsum 306 ``` 307 308 ### Executing a compiled test binary 309 310 `gotestsum` supports executing a compiled test binary (created with `go test -c`) by running 311 it as a custom command. 312 313 The `-json` flag is handled by `go test` itself, it is not available when using a 314 compiled test binary, so `go tool test2json` must be used to get the output 315 that `gotestsum` expects. 316 317 **Example: running `./binary.test`** 318 319 ``` 320 gotestsum --raw-command -- go tool test2json -t -p pkgname ./binary.test -test.v 321 ``` 322 323 `pkgname` is the name of the package being tested, it will show up in the test 324 output. `./binary.test` is the path to the compiled test binary. The `-test.v` 325 must be included so that `go tool test2json` receives all the output. 326 327 To execute a test binary without installing Go, see 328 [running without go](./.project/docs/running-without-go.md). 329 330 331 ### Finding and skipping slow tests 332 333 `gotestsum tool slowest` reads [test2json output][testjson], 334 from a file or stdin, and prints the names and elapsed time of slow tests. 335 The tests are sorted from slowest to fastest. 336 337 `gotestsum tool slowest` can also rewrite the source of tests slower than the 338 threshold, making it possible to optionally skip them. 339 340 The [test2json output][testjson] can be created with `gotestsum --jsonfile` or `go test -json`. 341 342 See `gotestsum tool slowest --help`. 343 344 **Example: printing a list of tests slower than 500 milliseconds** 345 346 ``` 347 $ gotestsum --format dots --jsonfile json.log 348 [.]····↷··↷· 349 $ gotestsum tool slowest --jsonfile json.log --threshold 500ms 350 gotest.tools/example TestSomething 1.34s 351 gotest.tools/example TestSomethingElse 810ms 352 ``` 353 354 **Example: skipping slow tests with `go test --short`** 355 356 Any test slower than 200 milliseconds will be modified to add: 357 358 ```go 359 if testing.Short() { 360 t.Skip("too slow for testing.Short") 361 } 362 ``` 363 364 ```sh 365 go test -json -short ./... | gotestsum tool slowest --skip-stmt "testing.Short" --threshold 200ms 366 ``` 367 368 Use `git diff` to see the file changes. 369 The next time tests are run using `--short` all the slow tests will be skipped. 370 371 [testjson]: https://golang.org/cmd/test2json/ 372 373 374 ### Run tests when a file is saved 375 376 When the `--watch` flag is set, `gotestsum` will watch directories using 377 [file system notifications](https://pkg.go.dev/github.com/fsnotify/fsnotify). 378 When a Go file in one of those directories is modified, `gotestsum` will run the 379 tests for the package that contains the changed file. By default all 380 directories under the current 381 directory with at least one `.go` file will be watched. 382 Use the `--packages` flag to specify a different list. 383 384 If `--watch` is used with a command line that includes the name of one or more 385 packages as command line arguments (ex: `gotestsum --watch -- ./...` or 386 `gotestsum --watch -- ./extrapkg`), the 387 tests in those packages will also be run when any file changes. 388 389 With the `--watch-chdir` flag, `gotestsum` will change the working directory 390 to the directory with the modified file before running tests. Changing the 391 directory is primarily useful when the project contains multiple Go modules. 392 Without this flag, `go test` will refuse to run tests for any package outside 393 of the main Go module. 394 395 While in watch mode, pressing some keys will perform an action: 396 397 * `r` will run tests for the previous event. 398 Added in version 1.6.1. 399 * `u` will run tests for the previous event, with the `-update` flag added. 400 Many [golden](https://gotest.tools/v3/golden) packages use this flag to automatically 401 update expected values of tests. 402 Added in version 1.8.1. 403 * `d` will run tests for the previous event using `dlv test`, allowing you to 404 debug a test failure using [delve]. A breakpoint will automatically be added at 405 the first line of any tests which failed in the previous run. Additional 406 breakpoints can be added with [`runtime.Breakpoint`](https://golang.org/pkg/runtime/#Breakpoint) 407 or by using the delve command prompt. 408 Added in version 1.6.1. 409 * `a` will run tests for all packages, by using `./...` as the package selector. 410 Added in version 1.7.0. 411 * `l` will scan the directory list again, and if there are any new directories 412 which contain a file with a `.go` extension, they will be added to the watch 413 list. 414 Added in version 1.7.0. 415 416 Note that [delve] must be installed in order to use debug (`d`). 417 418 [delve]: https://github.com/go-delve/delve 419 420 **Example: run tests for a package when any file in that package is saved** 421 ``` 422 gotestsum --watch --format testname 423 ``` 424 425 ## Who uses gotestsum? 426 427 The projects below use (or have used) gotestsum. 428 429 * [kubernetes](https://github.com/kubernetes/kubernetes/blob/master/hack/tools/tools.go) 430 * [moby](https://github.com/moby/moby/blob/master/hack/test/unit) (aka Docker) 431 * [etcd](https://github.com/etcd-io/etcd/blob/main/tools/mod/tools.go) 432 * [hashicorp/vault](https://github.com/hashicorp/vault/blob/main/tools/tools.go) 433 * [hashicorp/consul](https://github.com/hashicorp/consul/blob/main/.github/workflows/reusable-unit.yml) 434 * [prometheus](https://github.com/prometheus/prometheus/blob/main/Makefile.common) 435 * [minikube](https://github.com/kubernetes/minikube/blob/master/hack/jenkins/common.ps1) 436 * [influxdb](https://github.com/influxdata/influxdb/blob/master/scripts/ci/build-tests.sh) 437 * [pulumi](https://github.com/pulumi/pulumi/blob/master/.github/workflows/ci.yml) 438 * [grafana/k6](https://github.com/grafana/k6/issues/1986#issuecomment-996625874) 439 * [grafana/loki](https://github.com/grafana/loki/blob/main/loki-build-image/Dockerfile) 440 * [telegraf](https://github.com/influxdata/telegraf/blob/master/.circleci/config.yml) 441 * [containerd](https://github.com/containerd/containerd/blob/main/.cirrus.yml) 442 * [linkerd2](https://github.com/linkerd/linkerd2/blob/main/justfile) 443 * [elastic/go-elasticsearch](https://github.com/elastic/go-elasticsearch/blob/main/Makefile) 444 * [microsoft/hcsshim](https://github.com/microsoft/hcsshim/blob/main/.github/workflows/ci.yml) 445 * [pingcap/tidb](https://github.com/pingcap/tidb/blob/master/Makefile) 446 * [dex](https://github.com/dexidp/dex/blob/master/Makefile) 447 * [coder](https://github.com/coder/coder/blob/main/Makefile) 448 * [docker/cli](https://github.com/docker/cli/blob/master/Makefile) 449 450 Please open a GitHub issue or pull request to add or remove projects from this list. 451 452 ## Development 453 454 [![Godoc](https://godoc.org/gotest.tools/gotestsum?status.svg)](https://pkg.go.dev/gotest.tools/gotestsum?tab=subdirectories) 455 [![CircleCI](https://circleci.com/gh/gotestyourself/gotestsum/tree/main.svg?style=shield)](https://circleci.com/gh/gotestyourself/gotestsum/tree/main) 456 [![Go Recipes](https://raw.githubusercontent.com/nikolaydubina/go-recipes/main/badge.svg?raw=true)](https://github.com/nikolaydubina/go-recipes) 457 [![Go Reportcard](https://goreportcard.com/badge/gotest.tools/gotestsum)](https://goreportcard.com/report/gotest.tools/gotestsum) 458 459 460 Pull requests and bug reports are welcome! Please open an issue first for any 461 big changes. 462 463 ## Thanks 464 465 This package is heavily influenced by the [pytest](https://docs.pytest.org) test runner for `python`.