gotest.tools/gotestsum@v1.11.0/README.md (about)

     1  # gotestsum
     2  
     3  `gotestsum` runs tests using `go test -json`, prints formatted test output, and a summary of the test run.
     4  It is designed to work well for both local development, and for automation like CI.
     5  `gotestsum` is [used by](#who-uses-gotestsum) some of the most popular Go projects.
     6  
     7  ## Install
     8  
     9  Download a binary from [releases](https://github.com/gotestyourself/gotestsum/releases), or build from
    10  source with `go install gotest.tools/gotestsum@latest`. To run without installing use
    11  `go run gotest.tools/gotestsum@latest`.
    12  
    13  ## Documentation
    14  
    15  **Core features**
    16  - Change the [test output format](#output-format), from compact to verbose with color highlighting.
    17  - Print a [summary](#summary) of the test run after running all the tests.
    18  - Use any [`go test` flag](#custom-go-test-command),
    19    run a script with [`--raw-command`](#custom-go-test-command),
    20    or [run a compiled test binary](#executing-a-compiled-test-binary).
    21  
    22  **CI and Automation**
    23  - [`--junitfile`](#junit-xml-output) - write a JUnit XML file for integration with CI systems.
    24  - [`--jsonfile`](#json-file-output) - write all the [test2json](https://pkg.go.dev/cmd/test2json) input received by `gotestsum` to a file. The file
    25    can be used as input to [`gotestsum tool slowest`](#finding-and-skipping-slow-tests), or as a way to
    26    store the full verbose output of tests when less verbose output is printed to stdout using a compact [`--format`](#output-format).
    27  - [`--rerun-fails`](#re-running-failed-tests) - run failed (possibly flaky) tests again to avoid re-running the
    28    entire suite. Re-running individual tests can save significant time when working with flaky test suites.
    29  
    30  **Local Development**
    31  - [`--watch`](#run-tests-when-a-file-is-saved) - every time a `.go` file is saved run the tests for the package that changed.
    32  - [`--post-run-command`](#post-run-command) - run a command after the tests, can be used for desktop notification of the test run.
    33  - [`gotestsum tool slowest`](#finding-and-skipping-slow-tests) - find the slowest tests, or automatically update the source code of
    34    the slowest tests to add a conditional `t.Skip` statements. This statement allows you to skip the slowest tests using `gotestsum -- -short ./...`.
    35  
    36  
    37  ### Output Format
    38  
    39  The `--format` flag or `GOTESTSUM_FORMAT` environment variable set the format that
    40  is used to print the test names, and possibly test output, as the tests run. Most
    41  outputs use color to highlight pass, fail, or skip.
    42  
    43  The `--format-hivis` flag changes the icons used by `pkgname` formats to higher
    44  visiblity unicode characters.
    45  
    46  Commonly used formats (see `--help` for a full list):
    47  
    48   * `dots` - print a character for each test.
    49   * `pkgname` (default) - print a line for each package.
    50   * `testname` - print a line for each test and package.
    51   * `testdox` - print a sentence for each test using [gotestdox](https://github.com/bitfield/gotestdox).
    52   * `standard-quiet` - the standard `go test` format.
    53   * `standard-verbose` - the standard `go test -v` format.
    54  
    55  Have an idea for a new format?
    56  Please [share it on github](https://github.com/gotestyourself/gotestsum/issues/new)!
    57  
    58  #### Demo
    59  
    60  A demonstration of three `--format` options.
    61  
    62  ![Demo](https://user-images.githubusercontent.com/442180/182284939-e08a0aa5-4504-4e30-9e88-207ef47f4537.gif)
    63  <br /><sup>[Source](https://github.com/gotestyourself/gotestsum/tree/readme-demo/scripts)</sup>
    64  
    65  ### Summary
    66  
    67  Following the formatted output is a summary of the test run. The summary includes:
    68  
    69   * The test output, and elapsed time, for any test that fails or is skipped.
    70   * The build errors for any package that fails to build.
    71   * A `DONE` line with a count of tests run, tests skipped, tests failed, package build errors,
    72     and the elapsed time including time to build.
    73  
    74     ```
    75     DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s
    76     ```
    77  
    78  To hide parts of the summary use `--hide-summary section`.
    79  
    80  
    81  **Example: hide skipped tests in the summary**
    82  ```
    83  gotestsum --hide-summary=skipped
    84  ```
    85  
    86  **Example: hide everything except the DONE line**
    87  ```
    88  gotestsum --hide-summary=skipped,failed,errors,output
    89  # or
    90  gotestsum --hide-summary=all
    91  ```
    92  
    93  **Example: hide test output in the summary, only print names of failed and skipped tests
    94  and errors**
    95  ```
    96  gotestsum --hide-summary=output
    97  ```
    98  
    99  ### JUnit XML output
   100  
   101  When the `--junitfile` flag or `GOTESTSUM_JUNITFILE` environment variable are set
   102  to a file path, `gotestsum` will write a test report, in JUnit XML format, to the file.
   103  This file can be used to integrate with CI systems.
   104  
   105  ```
   106  gotestsum --junitfile unit-tests.xml
   107  ```
   108  
   109  If the package names in the `testsuite.name` or `testcase.classname` fields do not
   110  work with your CI system these values can be customized using the
   111  `--junitfile-testsuite-name`, or `--junitfile-testcase-classname` flags. These flags
   112  accept the following values:
   113  
   114  * `short` - the base name of the package (the single term specified by the
   115    package statement).
   116  * `relative` - a package path relative to the root of the repository
   117  * `full` - the full package path (default)
   118  
   119  
   120  Note: If Go is not installed, or the `go` binary is not in `PATH`, the `GOVERSION`
   121  environment variable can be set to remove the "failed to lookup go version for junit xml"
   122  warning.
   123  
   124  ### JSON file output
   125  
   126  When the `--jsonfile` flag or `GOTESTSUM_JSONFILE` environment variable are set
   127  to a file path, `gotestsum` will write a line-delimited JSON file with all the
   128  [test2json](https://golang.org/cmd/test2json/#hdr-Output_Format)
   129  output that was written by `go test -json`. This file can be used to compare test
   130  runs, or find flaky tests.
   131  
   132  ```
   133  gotestsum --jsonfile test-output.log
   134  ```
   135  
   136  ### Post Run Command
   137  
   138  The `--post-run-command` flag may be used to execute a command after the
   139  test run has completed. The binary will be run with the following environment
   140  variables set:
   141  
   142  ```
   143  GOTESTSUM_ELAPSED       # test run time in seconds (ex: 2.45s)
   144  GOTESTSUM_FORMAT        # gotestsum format (ex: pkgname)
   145  GOTESTSUM_JSONFILE      # path to the jsonfile, empty if no file path was given
   146  GOTESTSUM_JUNITFILE     # path to the junit.xml file, empty if no file path was given
   147  TESTS_ERRORS            # number of errors
   148  TESTS_FAILED            # number of failed tests
   149  TESTS_SKIPPED           # number of skipped tests
   150  TESTS_TOTAL             # number of tests run
   151  ```
   152  
   153  To get more details about the test run, such as failure messages or the full list of failed
   154  tests, run `gotestsum` with either a `--jsonfile` or `--junitfile` and parse the
   155  file from the post-run-command. The
   156  [gotestsum/testjson](https://pkg.go.dev/gotest.tools/gotestsum/testjson?tab=doc)
   157  package may be used to parse the JSON file output.
   158  
   159  **Example: desktop notifications**
   160  
   161  First install the example notification command with `go get gotest.tools/gotestsum/contrib/notify`.
   162  The command will be downloaded to `$GOPATH/bin` as `notify`. Note that this
   163  example `notify` command only works on Linux with `notify-send` and on macOS with
   164  [terminal-notifer](https://github.com/julienXX/terminal-notifier) installed.
   165  
   166  On Linux, you need to have some "test-pass" and "test-fail" icons installed in your icon theme.
   167  Some sample icons can be found in `contrib/notify`, and can be installed with `make install`.
   168  
   169  On Windows, you can install [notify-send.exe](https://github.com/vaskovsky/notify-send)
   170  but it does not support custom icons so will have to use the basic "info" and "error".
   171  
   172  ```
   173  gotestsum --post-run-command notify
   174  ```
   175  
   176  **Example: command with flags**
   177  
   178  Possitional arguments or command line flags can be passed to the `--post-run-command` by
   179  quoting the whole command.
   180  
   181  ```
   182  gotestsum --post-run-command "notify me --date"
   183  ```
   184  
   185  **Example: printing slowest tests**
   186  
   187  The post-run command can be combined with other `gotestsum` commands and tools to provide
   188  a more detailed summary. This example uses `gotestsum tool slowest` to print the
   189  slowest 10 tests after the summary.
   190  
   191  ```
   192  gotestsum \
   193    --jsonfile tmp.json.log \
   194    --post-run-command "bash -c '
   195      echo; echo Slowest tests;
   196      gotestsum tool slowest --num 10 --jsonfile tmp.json.log'"
   197  ```
   198  
   199  ### Re-running failed tests
   200  
   201  When the `--rerun-fails` flag is set, `gotestsum` will re-run any failed tests.
   202  The tests will be re-run until each passes once, or the number of attempts
   203  exceeds the maximum attempts. Maximum attempts defaults to 2, and can be changed
   204  with `--rerun-fails=n`.
   205  
   206  To avoid re-running tests when there are real failures, the re-run will be
   207  skipped when there are too many test failures. By default this value is 10, and
   208  can be changed with `--rerun-fails-max-failures=n`.
   209  
   210  Note that using `--rerun-fails` may require the use of other flags, depending on
   211  how you specify args to `go test`:
   212  
   213  * when used with `--raw-command` the re-run will pass additional arguments to
   214    the command. The first arg is a `-test.run` flag with a regex that matches the test to re-run,
   215    and second is the name of a go package. These additional args can be passed to `go test`,
   216    or a test binary.
   217  * when used with any `go test` args (anything after `--` on the command line), the list of
   218    packages to test must be specified as a space separated list using the `--packages` arg.
   219  
   220    **Example**
   221  
   222    ```
   223    gotestsum --rerun-fails --packages="./..." -- -count=2
   224    ```
   225  
   226  * if any of the `go test` args should be passed to the test binary, instead of
   227    `go test` itself, the `-args` flag must be used to separate the two groups of
   228    arguments. `-args` is a special flag that is understood by `go test` to indicate
   229    that any following args should be passed directly to the test binary.
   230  
   231    **Example**
   232  
   233    ```
   234    gotestsum --rerun-fails --packages="./..." -- -count=2 -args -update-golden
   235    ```
   236  
   237  
   238  ### Custom `go test` command
   239  
   240  By default `gotestsum` runs tests using the command `go test -json ./...`. You
   241  can change the command with positional arguments after a `--`. You can change just the
   242  test directory value (which defaults to `./...`) by setting the `TEST_DIRECTORY`
   243  environment variable.
   244  
   245  You can use `--debug` to echo the command before it is run.
   246  
   247  **Example: set build tags**
   248  ```
   249  gotestsum -- -tags=integration ./...
   250  ```
   251  
   252  **Example: run tests in a single package**
   253  ```
   254  gotestsum -- ./io/http
   255  ```
   256  
   257  **Example: enable coverage**
   258  ```
   259  gotestsum -- -coverprofile=cover.out ./...
   260  ```
   261  
   262  **Example: run a script instead of `go test`**
   263  ```
   264  gotestsum --raw-command -- ./scripts/run_tests.sh
   265  ```
   266  
   267  Note: when using `--raw-command`, the script must follow a few rules about
   268  stdout and stderr output:
   269  
   270  * The stdout produced by the script must only contain the `test2json` output, or
   271    `gotestsum` will fail. If it isn't possible to change the script to avoid
   272    non-JSON output, you can use `--ignore-non-json-output-lines` (added in version 1.7.0)
   273    to ignore non-JSON lines and write them to `gotestsum`'s stderr instead.
   274  * Any stderr produced by the script will be considered an error (this behaviour
   275    is necessary because package build errors are only reported by writting to
   276    stderr, not the `test2json` stdout). Any stderr produced by tests is not
   277    considered an error (it will be in the `test2json` stdout).
   278  
   279  **Example: accept intput from stdin**
   280  ```
   281  cat out.json | gotestsum --raw-command -- cat
   282  ```
   283  
   284  **Example: run tests with profiling enabled**
   285  
   286  Using a `profile.sh` script like this:
   287  
   288  ```sh
   289  #!/usr/bin/env bash
   290  set -eu
   291  
   292  for pkg in $(go list "$@"); do
   293      dir="$(go list -f '{{ .Dir }}' $pkg)"
   294      go test -json -cpuprofile="$dir/cpu.profile" "$pkg"
   295  done
   296  ```
   297  
   298  You can run:
   299  ```
   300  gotestsum --raw-command ./profile.sh ./...
   301  ```
   302  
   303  **Example: using `TEST_DIRECTORY`**
   304  ```
   305  TEST_DIRECTORY=./io/http gotestsum
   306  ```
   307  
   308  ### Executing a compiled test binary
   309  
   310  `gotestsum` supports executing a compiled test binary (created with `go test -c`) by running
   311  it as a custom command.
   312  
   313  The `-json` flag is handled by `go test` itself, it is not available when using a
   314  compiled test binary, so `go tool test2json` must be used to get the output
   315  that `gotestsum` expects.
   316  
   317  **Example: running `./binary.test`**
   318  
   319  ```
   320  gotestsum --raw-command -- go tool test2json -t -p pkgname ./binary.test -test.v
   321  ```
   322  
   323  `pkgname` is the name of the package being tested, it will show up in the test
   324  output. `./binary.test` is the path to the compiled test binary. The `-test.v`
   325  must be included so that `go tool test2json` receives all the output.
   326  
   327  To execute a test binary without installing Go, see
   328  [running without go](./.project/docs/running-without-go.md).
   329  
   330  
   331  ### Finding and skipping slow tests
   332  
   333  `gotestsum tool slowest` reads [test2json output][testjson],
   334  from a file or stdin, and prints the names and elapsed time of slow tests.
   335  The tests are sorted from slowest to fastest.
   336  
   337  `gotestsum tool slowest` can also rewrite the source of tests slower than the
   338  threshold, making it possible to optionally skip them.
   339  
   340  The [test2json output][testjson] can be created with `gotestsum --jsonfile` or `go test -json`.
   341  
   342  See `gotestsum tool slowest --help`.
   343  
   344  **Example: printing a list of tests slower than 500 milliseconds**
   345  
   346  ```
   347  $ gotestsum --format dots --jsonfile json.log
   348  [.]····↷··↷·
   349  $ gotestsum tool slowest --jsonfile json.log --threshold 500ms
   350  gotest.tools/example TestSomething 1.34s
   351  gotest.tools/example TestSomethingElse 810ms
   352  ```
   353  
   354  **Example: skipping slow tests with `go test --short`**
   355  
   356  Any test slower than 200 milliseconds will be modified to add:
   357  
   358  ```go
   359  if testing.Short() {
   360      t.Skip("too slow for testing.Short")
   361  }
   362  ```
   363  
   364  ```sh
   365  go test -json -short ./... | gotestsum tool slowest --skip-stmt "testing.Short" --threshold 200ms
   366  ```
   367  
   368  Use `git diff` to see the file changes.
   369  The next time tests are run using `--short` all the slow tests will be skipped.
   370  
   371  [testjson]: https://golang.org/cmd/test2json/
   372  
   373  
   374  ### Run tests when a file is saved 
   375  
   376  When the `--watch` flag is set, `gotestsum` will watch directories using
   377  [file system notifications](https://pkg.go.dev/github.com/fsnotify/fsnotify).
   378  When a Go file in one of those directories is modified, `gotestsum` will run the
   379  tests for the package that contains the changed file. By default all
   380  directories under the current
   381  directory with at least one `.go` file will be watched.
   382  Use the `--packages` flag to specify a different list.
   383  
   384  If `--watch` is used with a command line that includes the name of one or more
   385  packages as command line arguments (ex: `gotestsum --watch -- ./...` or
   386  `gotestsum --watch -- ./extrapkg`), the
   387  tests in those packages will also be run when any file changes.
   388  
   389  With the `--watch-chdir` flag, `gotestsum` will change the working directory
   390  to the directory with the modified file before running tests. Changing the
   391  directory is primarily useful when the project contains multiple Go modules.
   392  Without this flag, `go test` will refuse to run tests for any package outside
   393  of the main Go module.
   394  
   395  While in watch mode, pressing some keys will perform an action:
   396  
   397  * `r` will run tests for the previous event.
   398    Added in version 1.6.1.
   399  * `u` will run tests for the previous event, with the `-update` flag added.
   400    Many [golden](https://gotest.tools/v3/golden) packages use this flag to automatically
   401    update expected values of tests.
   402    Added in version 1.8.1.
   403  * `d` will run tests for the previous event using `dlv test`, allowing you to 
   404    debug a test failure using [delve]. A breakpoint will automatically be added at
   405    the first line of any tests which failed in the previous run. Additional
   406    breakpoints can be added with [`runtime.Breakpoint`](https://golang.org/pkg/runtime/#Breakpoint)
   407    or by using the delve command prompt.
   408    Added in version 1.6.1.
   409  * `a` will run tests for all packages, by using `./...` as the package selector.
   410    Added in version 1.7.0.
   411  * `l` will scan the directory list again, and if there are any new directories
   412    which contain a file with a `.go` extension, they will be added to the watch
   413    list.
   414    Added in version 1.7.0.
   415  
   416  Note that [delve] must be installed in order to use debug (`d`).
   417  
   418  [delve]: https://github.com/go-delve/delve
   419  
   420  **Example: run tests for a package when any file in that package is saved**
   421  ```
   422  gotestsum --watch --format testname
   423  ```
   424  
   425  ## Who uses gotestsum?
   426  
   427  The projects below use (or have used) gotestsum.
   428  
   429  * [kubernetes](https://github.com/kubernetes/kubernetes/blob/master/hack/tools/tools.go)
   430  * [moby](https://github.com/moby/moby/blob/master/hack/test/unit) (aka Docker)
   431  * [etcd](https://github.com/etcd-io/etcd/blob/main/tools/mod/tools.go)
   432  * [hashicorp/vault](https://github.com/hashicorp/vault/blob/main/tools/tools.go)
   433  * [hashicorp/consul](https://github.com/hashicorp/consul/blob/main/.github/workflows/reusable-unit.yml)
   434  * [prometheus](https://github.com/prometheus/prometheus/blob/main/Makefile.common)
   435  * [minikube](https://github.com/kubernetes/minikube/blob/master/hack/jenkins/common.ps1)
   436  * [influxdb](https://github.com/influxdata/influxdb/blob/master/scripts/ci/build-tests.sh)
   437  * [pulumi](https://github.com/pulumi/pulumi/blob/master/.github/workflows/ci.yml)
   438  * [grafana/k6](https://github.com/grafana/k6/issues/1986#issuecomment-996625874)
   439  * [grafana/loki](https://github.com/grafana/loki/blob/main/loki-build-image/Dockerfile)
   440  * [telegraf](https://github.com/influxdata/telegraf/blob/master/.circleci/config.yml)
   441  * [containerd](https://github.com/containerd/containerd/blob/main/.cirrus.yml)
   442  * [linkerd2](https://github.com/linkerd/linkerd2/blob/main/justfile)
   443  * [elastic/go-elasticsearch](https://github.com/elastic/go-elasticsearch/blob/main/Makefile)
   444  * [microsoft/hcsshim](https://github.com/microsoft/hcsshim/blob/main/.github/workflows/ci.yml)
   445  * [pingcap/tidb](https://github.com/pingcap/tidb/blob/master/Makefile)
   446  * [dex](https://github.com/dexidp/dex/blob/master/Makefile)
   447  * [coder](https://github.com/coder/coder/blob/main/Makefile)
   448  * [docker/cli](https://github.com/docker/cli/blob/master/Makefile)
   449  
   450  Please open a GitHub issue or pull request to add or remove projects from this list.
   451  
   452  ## Development
   453  
   454  [![Godoc](https://godoc.org/gotest.tools/gotestsum?status.svg)](https://pkg.go.dev/gotest.tools/gotestsum?tab=subdirectories)
   455  [![CircleCI](https://circleci.com/gh/gotestyourself/gotestsum/tree/main.svg?style=shield)](https://circleci.com/gh/gotestyourself/gotestsum/tree/main)
   456  [![Go Recipes](https://raw.githubusercontent.com/nikolaydubina/go-recipes/main/badge.svg?raw=true)](https://github.com/nikolaydubina/go-recipes)
   457  [![Go Reportcard](https://goreportcard.com/badge/gotest.tools/gotestsum)](https://goreportcard.com/report/gotest.tools/gotestsum)
   458  
   459  
   460  Pull requests and bug reports are welcome! Please open an issue first for any
   461  big changes.
   462  
   463  ## Thanks
   464  
   465  This package is heavily influenced by the [pytest](https://docs.pytest.org) test runner for `python`.