github.com/graybobo/golang.org-package-offline-cache@v0.0.0-20200626051047-6608995c132f/x/blog/content/cover.article (about) 1 The cover story 2 2 Dec 2013 3 Tags: tools, coverage, testing 4 5 Rob Pike 6 7 * Introduction 8 9 From the beginning of the project, Go was designed with tools in mind. 10 Those tools include some of the most iconic pieces of Go technology such as 11 the documentation presentation tool 12 [[http://golang.org/cmd/godoc][godoc]], 13 the code formatting tool 14 [[http://golang.org/cmd/gofmt][gofmt]], 15 and the API rewriter 16 [[http://golang.org/cmd/fix][gofix]]. 17 Perhaps most important of all is the 18 [[http://golang.org/cmd/go][`go` command]], 19 the program that automatically installs, builds, and tests Go programs 20 using nothing more than the source code as the build specification. 21 22 The release of Go 1.2 introduces a new tool for test coverage that takes an 23 unusual approach to the way it generates coverage statistics, an approach 24 that builds on the technology laid down by godoc and friends. 25 26 * Support for tools 27 28 First, some background: What does it mean for a 29 [[http://talks.golang.org/2012/splash.article#TOC_17.][language to support good tooling]]? 30 It means that the language makes it easy to write good tools and that its ecosystem 31 supports the construction of tools of all flavors. 32 33 There are a number of properties of Go that make it suitable for tooling. 34 For starters, Go has a regular syntax that is easy to parse. 35 The grammar aims to be free of special cases that require complex machinery to analyze. 36 37 Where possible, Go uses lexical and syntactic constructs to make semantic properties 38 easy to understand. 39 Examples include the use of upper-case letters to define exported names 40 and the radically simplified scoping rules compared to other languages in the C tradition. 41 42 Finally, the standard library comes with production-quality packages to lex and parse Go source code. 43 They also include, more unusually, a production-quality package to pretty-print Go syntax trees. 44 45 These packages in combination form the core of the gofmt tool, but the pretty-printer is worth singling out. 46 Because it can take an arbitrary Go syntax tree and output standard-format, human-readable, correct 47 code, it creates the possibility to build tools that transform the parse tree and output modified but 48 correct and easy-to-read code. 49 50 One example is the gofix tool, which automates the 51 rewriting of code to use new language features or updated libraries. 52 Gofix let us make fundamental changes to the language and libraries in the 53 [[http://blog.golang.org/the-path-to-go-1][run-up to Go 1.0]], 54 with the confidence that users could just run the tool to update their source to the newest version. 55 56 Inside Google, we have used gofix to make sweeping changes in a huge code repository that would be almost 57 unthinkable in the other languages we use. 58 There's no need any more to support multiple versions of some API; we can use gofix to update 59 the entire company in one operation. 60 61 It's not just these big tools that these packages enable, of course. 62 They also make it easy to write more modest programs such as IDE plugins, for instance. 63 All these items build on each other, making the Go environment 64 more productive by automating many tasks. 65 66 * Test coverage 67 68 Test coverage is a term that describes how much of a package's code is exercised by running the package's tests. 69 If executing the test suite causes 80% of the package's source statements to be run, we say that the test coverage is 80%. 70 71 The program that provides test coverage in Go 1.2 is the latest to exploit the tooling support in the Go ecosystem. 72 73 The usual way to compute test coverage is to instrument the binary. 74 For instance, the GNU [[http://gcc.gnu.org/onlinedocs/gcc/Gcov.html][gcov]] program sets breakpoints at branches 75 executed by the binary. 76 As each branch executes, the breakpoint is cleared and the target statements of the branch are marked as 'covered'. 77 78 This approach is successful and widely used. An early test coverage tool for Go even worked the same way. 79 But it has problems. 80 It is difficult to implement, as analysis of the execution of binaries is challenging. 81 It also requires a reliable way of tying the execution trace back to the source code, which can also be difficult, 82 as any user of a source-level debugger can attest. 83 Problems there include inaccurate debugging information and issues such as in-lined functions complicating 84 the analysis. 85 Most important, this approach is very non-portable. 86 It needs to be done afresh for every architecture, and to some extent for every 87 operating system since debugging support varies greatly from system to system. 88 89 It does work, though, and for instance if you are a user of gccgo, the gcov tool can give you test coverage 90 information. 91 However If you're a user of gc, the more commonly used Go compiler suite, until Go 1.2 you were out of luck. 92 93 * Test coverage for Go 94 95 For the new test coverage tool for Go, we took a different approach that avoids dynamic debugging. 96 The idea is simple: Rewrite the package's source code before compilation to add instrumentation, 97 compile and run the modified source, and dump the statistics. 98 The rewriting is easy to arrange because the `go` command controls the flow 99 from source to test to execution. 100 101 Here's an example. Say we have a simple, one-file package like this: 102 103 .code cover/pkg.go 104 105 and this test: 106 107 .code cover/pkg_test.go 108 109 To get the test coverage for the package, 110 we run the test with coverage enabled by providing the `-cover` flag to `go` `test`: 111 112 % go test -cover 113 PASS 114 coverage: 42.9% of statements 115 ok size 0.026s 116 % 117 118 Notice that the coverage is 42.9%, which isn't very good. 119 Before we ask how to raise that number, let's see how that was computed. 120 121 When test coverage is enabled, `go` `test` runs the "cover" tool, a separate program included 122 with the distribution, to rewrite the source code before compilation. Here's what the rewritten 123 `Size` function looks like: 124 125 .code cover/pkg.cover /func/,/^}/ 126 127 Each executable section of the program is annotated with an assignment statement that, 128 when executed, records that that section ran. 129 The counter is tied to the original source position of the statements it counts 130 through a second read-only data structure that is also generated by the cover tool. 131 When the test run completes, the counters are collected and the percentage is computed 132 by seeing how many were set. 133 134 Although that annotating assignment might look expensive, it compiles to a single "move" instruction. 135 Its run-time overhead is therefore modest, adding only about 3% when running a typical (more realistic) test. 136 That makes it reasonable to include test coverage as part of the standard development pipeline. 137 138 * Viewing the results 139 140 The test coverage for our example was poor. 141 To discover why, we ask `go` `test` to write a "coverage profile" for us, a file that holds 142 the collected statistics so we can study them in more detail. 143 That's easy to do: use the `-coverprofile` flag to specify a file for the output: 144 145 % go test -coverprofile=coverage.out 146 PASS 147 coverage: 42.9% of statements 148 ok size 0.030s 149 % 150 151 (The `-coverprofile` flag automatically sets `-cover` to enable coverage analysis.) 152 The test runs just as before, but the results are saved in a file. 153 To study them, we run the test coverage tool ourselves, without `go` `test`. 154 As a start, we can ask for the coverage to be broken down by function, 155 although that's not going to illuminate much in this case since there's 156 only one function: 157 158 % go tool cover -func=coverage.out 159 size.go: Size 42.9% 160 total: (statements) 42.9% 161 % 162 163 A much more interesting way to see the data is to get an HTML presentation 164 of the source code decorated with coverage information. 165 This display is invoked by the `-html` flag: 166 167 $ go tool cover -html=coverage.out 168 169 When this command is run, a browser window pops up, showing the covered (green), 170 uncovered (red), and uninstrumented (grey) source. 171 Here's a screen dump: 172 173 .image cover/set.png 174 175 With this presentation, it's obvious what's wrong: we neglected to test several 176 of the cases! 177 And we can see exactly which ones they are, which makes it easy to 178 improve our test coverage. 179 180 * Heat maps 181 182 A big advantage of this source-level approach to test coverage is that it's 183 easy to instrument the code in different ways. 184 For instance, we can ask not only whether a statement has been executed, 185 but how many times. 186 187 The `go` `test` command accepts a `-covermode` flag to set the coverage mode 188 to one of three settings: 189 190 - set: did each statement run? 191 - count: how many times did each statement run? 192 - atomic: like count, but counts precisely in parallel programs 193 194 The default is 'set', which we've already seen. 195 The `atomic` setting is needed only when accurate counts are required 196 when running parallel algorithms. It uses atomic operations from the 197 [[http://golang.org/pkg/sync/atomic/][sync/atomic]] package, 198 which can be quite expensive. 199 For most purposes, though, the `count` mode works fine and, like 200 the default `set` mode, is very cheap. 201 202 Let's try counting statement execution for a standard package, the `fmt` formatting package. 203 We run the test and write out a coverage profile so we can present the information 204 nicely afterwards. 205 206 % go test -covermode=count -coverprofile=count.out fmt 207 ok fmt 0.056s coverage: 91.7% of statements 208 % 209 210 That's a much better test coverage ratio than for our previous example. 211 (The coverage ratio is not affected by the coverage mode.) 212 We can display the function breakdown: 213 214 % go tool cover -func=count.out 215 fmt/format.go: init 100.0% 216 fmt/format.go: clearflags 100.0% 217 fmt/format.go: init 100.0% 218 fmt/format.go: computePadding 84.6% 219 fmt/format.go: writePadding 100.0% 220 fmt/format.go: pad 100.0% 221 ... 222 fmt/scan.go: advance 96.2% 223 fmt/scan.go: doScanf 96.8% 224 total: (statements) 91.7% 225 226 The big payoff happens in the HTML output: 227 228 % go tool cover -html=count.out 229 230 Here's what the `pad` function looks like in that presentation: 231 232 .image cover/count.png 233 234 Notice how the intensity of the green changes. Brighter-green 235 statements have higher execution counts; less saturated greens 236 represent lower execution counts. 237 You can even hover the mouse over the statements to see the 238 actual counts pop up in a tool tip. 239 At the time of writing, the counts come out like this 240 (we've moved the counts from the tool tips to beginning-of-line 241 markers to make them easier to show): 242 243 2933 if !f.widPresent || f.wid == 0 { 244 2985 f.buf.Write(b) 245 2985 return 246 2985 } 247 56 padding, left, right := f.computePadding(len(b)) 248 56 if left > 0 { 249 37 f.writePadding(left, padding) 250 37 } 251 56 f.buf.Write(b) 252 56 if right > 0 { 253 13 f.writePadding(right, padding) 254 13 } 255 256 That's a lot of information about the execution of the function, 257 information that might be useful in profiling. 258 259 * Basic blocks 260 261 You might have noticed that the counts in the previous example 262 were not what you expected on the lines with closing braces. 263 That's because, as always, test coverage is an inexact science. 264 265 What's going on here is worth explaining, though. We'd like the 266 coverage annotations to be demarcated by branches in the program, 267 the way they are when the binary is instrumented in the traditional 268 method. 269 It's hard to do that by rewriting the source, though, since 270 the branches don't appear explicitly in the source. 271 272 What the coverage annotation does is instrument blocks, which 273 are typically bounded by brace brackets. 274 Getting this right in general is very hard. 275 A consequence of the algorithm used is that the closing 276 brace looks like it belongs to the block it closes, while the 277 opening brace looks like it belongs outside the block. 278 A more interesting consequence is that in an expression like 279 280 f() && g() 281 282 there is no attempt to separately instrument the calls to `f` and `g`, Regardless of 283 the facts it will always look like they both ran the same 284 number of times, the number of times `f` ran. 285 286 To be fair, even `gcov` has trouble here. That tool gets the 287 instrumentation right but the presentation is line-based and 288 can therefore miss some nuances. 289 290 * The big picture 291 292 That's the story about test coverage in Go 1.2. 293 A new tool with an interesting implementation enables not only 294 test coverage statistics, but easy-to-interpret presentations 295 of them and even the possibility to extract profiling information. 296 297 Testing is an important part of software development and test 298 coverage a simple way to add discipline to your testing strategy. 299 Go forth, test, and cover.