github.com/graybobo/golang.org-package-offline-cache@v0.0.0-20200626051047-6608995c132f/x/blog/content/cover.article (about)

     1  The cover story
     2  2 Dec 2013
     3  Tags: tools, coverage, testing
     4  
     5  Rob Pike
     6  
     7  * Introduction
     8  
     9  From the beginning of the project, Go was designed with tools in mind.
    10  Those tools include some of the most iconic pieces of Go technology such as
    11  the documentation presentation tool
    12  [[http://golang.org/cmd/godoc][godoc]],
    13  the code formatting tool
    14  [[http://golang.org/cmd/gofmt][gofmt]],
    15  and the API rewriter
    16  [[http://golang.org/cmd/fix][gofix]].
    17  Perhaps most important of all is the
    18  [[http://golang.org/cmd/go][`go` command]],
    19  the program that automatically installs, builds, and tests Go programs
    20  using nothing more than the source code as the build specification.
    21  
    22  The release of Go 1.2 introduces a new tool for test coverage that takes an
    23  unusual approach to the way it generates coverage statistics, an approach
    24  that builds on the technology laid down by godoc and friends.
    25  
    26  * Support for tools
    27  
    28  First, some background: What does it mean for a
    29  [[http://talks.golang.org/2012/splash.article#TOC_17.][language to support good tooling]]?
    30  It means that the language makes it easy to write good tools and that its ecosystem
    31  supports the construction of tools of all flavors.
    32  
    33  There are a number of properties of Go that make it suitable for tooling.
    34  For starters, Go has a regular syntax that is easy to parse.
    35  The grammar aims to be free of special cases that require complex machinery to analyze.
    36  
    37  Where possible, Go uses lexical and syntactic constructs to make semantic properties
    38  easy to understand.
    39  Examples include the use of upper-case letters to define exported names
    40  and the radically simplified scoping rules compared to other languages in the C tradition.
    41  
    42  Finally, the standard library comes with production-quality packages to lex and parse Go source code.
    43  They also include, more unusually, a production-quality package to pretty-print Go syntax trees.
    44  
    45  These packages in combination form the core of the gofmt tool, but the pretty-printer is worth singling out.
    46  Because it can take an arbitrary Go syntax tree and output standard-format, human-readable, correct
    47  code, it creates the possibility to build tools that transform the parse tree and output modified but
    48  correct and easy-to-read code.
    49  
    50  One example is the gofix tool, which automates the
    51  rewriting of code to use new language features or updated libraries.
    52  Gofix let us make fundamental changes to the language and libraries in the
    53  [[http://blog.golang.org/the-path-to-go-1][run-up to Go 1.0]],
    54  with the confidence that users could just run the tool to update their source to the newest version.
    55  
    56  Inside Google, we have used gofix to make sweeping changes in a huge code repository that would be almost
    57  unthinkable in the other languages we use.
    58  There's no need any more to support multiple versions of some API; we can use gofix to update
    59  the entire company in one operation.
    60  
    61  It's not just these big tools that these packages enable, of course.
    62  They also make it easy to write more modest programs such as IDE plugins, for instance.
    63  All these items build on each other, making the Go environment
    64  more productive by automating many tasks.
    65  
    66  * Test coverage
    67  
    68  Test coverage is a term that describes how much of a package's code is exercised by running the package's tests.
    69  If executing the test suite causes 80% of the package's source statements to be run, we say that the test coverage is 80%.
    70  
    71  The program that provides test coverage in Go 1.2 is the latest to exploit the tooling support in the Go ecosystem.
    72  
    73  The usual way to compute test coverage is to instrument the binary.
    74  For instance, the GNU [[http://gcc.gnu.org/onlinedocs/gcc/Gcov.html][gcov]] program sets breakpoints at branches
    75  executed by the binary.
    76  As each branch executes, the breakpoint is cleared and the target statements of the branch are marked as 'covered'.
    77  
    78  This approach is successful and widely used. An early test coverage tool for Go even worked the same way.
    79  But it has problems.
    80  It is difficult to implement, as analysis of the execution of binaries is challenging.
    81  It also requires a reliable way of tying the execution trace back to the source code, which can also be difficult,
    82  as any user of a source-level debugger can attest.
    83  Problems there include inaccurate debugging information and issues such as in-lined functions complicating
    84  the analysis.
    85  Most important, this approach is very non-portable.
    86  It needs to be done afresh for every architecture, and to some extent for every
    87  operating system since debugging support varies greatly from system to system.
    88  
    89  It does work, though, and for instance if you are a user of gccgo, the gcov tool can give you test coverage
    90  information.
    91  However If you're a user of gc, the more commonly used Go compiler suite, until Go 1.2 you were out of luck.
    92  
    93  * Test coverage for Go
    94  
    95  For the new test coverage tool for Go, we took a different approach that avoids dynamic debugging.
    96  The idea is simple: Rewrite the package's source code before compilation to add instrumentation,
    97  compile and run the modified source, and dump the statistics.
    98  The rewriting is easy to arrange because the `go` command controls the flow
    99  from source to test to execution.
   100  
   101  Here's an example. Say we have a simple, one-file package like this:
   102  
   103  .code cover/pkg.go
   104  
   105  and this test:
   106  
   107  .code cover/pkg_test.go
   108  
   109  To get the test coverage for the package,
   110  we run the test with coverage enabled by providing the `-cover` flag to `go` `test`:
   111  
   112  	% go test -cover
   113  	PASS
   114  	coverage: 42.9% of statements
   115  	ok  	size	0.026s
   116  	% 
   117  
   118  Notice that the coverage is 42.9%, which isn't very good.
   119  Before we ask how to raise that number, let's see how that was computed.
   120  
   121  When test coverage is enabled, `go` `test` runs the "cover" tool, a separate program included
   122  with the distribution, to rewrite the source code before compilation. Here's what the rewritten
   123  `Size` function looks like:
   124  
   125  .code cover/pkg.cover /func/,/^}/
   126  
   127  Each executable section of the program is annotated with an assignment statement that,
   128  when executed, records that that section ran.
   129  The counter is tied to the original source position of the statements it counts
   130  through a second read-only data structure that is also generated by the cover tool.
   131  When the test run completes, the counters are collected and the percentage is computed
   132  by seeing how many were set.
   133  
   134  Although that annotating assignment might look expensive, it compiles to a single "move" instruction.
   135  Its run-time overhead is therefore modest, adding only about 3% when running a typical (more realistic) test.
   136  That makes it reasonable to include test coverage as part of the standard development pipeline.
   137  
   138  * Viewing the results
   139  
   140  The test coverage for our example was poor.
   141  To discover why, we ask `go` `test` to write a "coverage profile" for us, a file that holds
   142  the collected statistics so we can study them in more detail.
   143  That's easy to do: use the `-coverprofile` flag to specify a file for the output:
   144  
   145  	% go test -coverprofile=coverage.out 
   146  	PASS
   147  	coverage: 42.9% of statements
   148  	ok  	size	0.030s
   149  	% 
   150  
   151  (The `-coverprofile` flag automatically sets `-cover` to enable coverage analysis.)
   152  The test runs just as before, but the results are saved in a file.
   153  To study them, we run the test coverage tool ourselves, without `go` `test`.
   154  As a start, we can ask for the coverage to be broken down by function,
   155  although that's not going to illuminate much in this case since there's
   156  only one function:
   157  
   158  	% go tool cover -func=coverage.out
   159  	size.go:	Size          42.9%
   160  	total:      (statements)  42.9%
   161  	% 
   162  
   163  A much more interesting way to see the data is to get an HTML presentation
   164  of the source code decorated with coverage information.
   165  This display is invoked by the `-html` flag:
   166  
   167  	$ go tool cover -html=coverage.out
   168  
   169  When this command is run, a browser window pops up, showing the covered (green),
   170  uncovered (red), and uninstrumented (grey) source.
   171  Here's a screen dump:
   172  
   173  .image cover/set.png
   174  
   175  With this presentation, it's obvious what's wrong: we neglected to test several
   176  of the cases!
   177  And we can see exactly which ones they are, which makes it easy to
   178  improve our test coverage.
   179  
   180  * Heat maps
   181  
   182  A big advantage of this source-level approach to test coverage is that it's
   183  easy to instrument the code in different ways.
   184  For instance, we can ask not only whether a statement has been executed,
   185  but how many times.
   186  
   187  The `go` `test` command accepts a `-covermode` flag to set the coverage mode
   188  to one of three settings:
   189  
   190  - set:    did each statement run?
   191  - count:  how many times did each statement run?
   192  - atomic: like count, but counts precisely in parallel programs
   193  
   194  The default is 'set', which we've already seen.
   195  The `atomic` setting is needed only when accurate counts are required
   196  when running parallel algorithms. It uses atomic operations from the
   197  [[http://golang.org/pkg/sync/atomic/][sync/atomic]] package,
   198  which can be quite expensive.
   199  For most purposes, though, the `count` mode works fine and, like
   200  the default `set` mode, is very cheap.
   201  
   202  Let's try counting statement execution for a standard package, the `fmt` formatting package.
   203  We run the test and write out a coverage profile so we can present the information
   204  nicely afterwards.
   205  
   206  	% go test -covermode=count -coverprofile=count.out fmt 
   207  	ok  	fmt	0.056s	coverage: 91.7% of statements
   208  	%
   209  
   210  That's a much better test coverage ratio than for our previous example.
   211  (The coverage ratio is not affected by the coverage mode.)
   212  We can display the function breakdown:
   213  
   214  	% go tool cover -func=count.out
   215  	fmt/format.go: init              100.0%
   216  	fmt/format.go: clearflags        100.0%
   217  	fmt/format.go: init              100.0%
   218  	fmt/format.go: computePadding     84.6%
   219  	fmt/format.go: writePadding      100.0%
   220  	fmt/format.go: pad               100.0%
   221  	...
   222  	fmt/scan.go:   advance            96.2%
   223  	fmt/scan.go:   doScanf            96.8%
   224  	total:         (statements)       91.7%
   225  
   226  The big payoff happens in the HTML output:
   227  
   228  	% go tool cover -html=count.out
   229  
   230  Here's what the `pad` function looks like in that presentation:
   231  
   232  .image cover/count.png
   233  
   234  Notice how the intensity of the green changes. Brighter-green
   235  statements have higher execution counts; less saturated greens
   236  represent lower execution counts.
   237  You can even hover the mouse over the statements to see the
   238  actual counts pop up in a tool tip.
   239  At the time of writing, the counts come out like this
   240  (we've moved the counts from the tool tips to beginning-of-line
   241  markers to make them easier to show):
   242  
   243  	2933    if !f.widPresent || f.wid == 0 {
   244  	2985        f.buf.Write(b)
   245  	2985        return
   246  	2985    }
   247  	  56    padding, left, right := f.computePadding(len(b))
   248  	  56    if left > 0 {
   249  	  37        f.writePadding(left, padding)
   250  	  37    }
   251  	  56    f.buf.Write(b)
   252  	  56    if right > 0 {
   253  	  13        f.writePadding(right, padding)
   254  	  13    }
   255  
   256  That's a lot of information about the execution of the function,
   257  information that might be useful in profiling.
   258  
   259  * Basic blocks
   260  
   261  You might have noticed that the counts in the previous example
   262  were not what you expected on the lines with closing braces.
   263  That's because, as always, test coverage is an inexact science.
   264  
   265  What's going on here is worth explaining, though. We'd like the
   266  coverage annotations to be demarcated by branches in the program,
   267  the way they are when the binary is instrumented in the traditional
   268  method.
   269  It's hard to do that by rewriting the source, though, since
   270  the branches don't appear explicitly in the source.
   271  
   272  What the coverage annotation does is instrument blocks, which
   273  are typically bounded by brace brackets.
   274  Getting this right in general is very hard.
   275  A consequence of the algorithm used is that the closing
   276  brace looks like it belongs to the block it closes, while the
   277  opening brace looks like it belongs outside the block.
   278  A more interesting consequence is that in an expression like
   279  
   280  	f() && g()
   281  
   282  there is no attempt to separately instrument the calls to `f` and `g`, Regardless of
   283  the facts it will always look like they both ran the same
   284  number of times, the number of times `f` ran.
   285  
   286  To be fair, even `gcov` has trouble here. That tool gets the
   287  instrumentation right but the presentation is line-based and
   288  can therefore miss some nuances.
   289  
   290  * The big picture
   291  
   292  That's the story about test coverage in Go 1.2.
   293  A new tool with an interesting implementation enables not only
   294  test coverage statistics, but easy-to-interpret presentations
   295  of them and even the possibility to extract profiling information.
   296  
   297  Testing is an important part of software development and test
   298  coverage a simple way to add discipline to your testing strategy.
   299  Go forth, test, and cover.