k8s.io/test-infra@v0.0.0-20240520184403-27c6b4c223d8/pkg/benchmarkjunit/README.md (about) 1 # benchmarkjunit 2 3 `benchmarkjunit` is a go utility that runs go [`testing` package benchmarks] 4 with a command like `go test -v -bench=. <packages>` and translates the output 5 into JUnit XML. 6 7 ## Usage Examples 8 9 Run `go run ./pkg/benchmarkjunit --help` to see all the available flags. 10 11 ```shell 12 go run ./pkg/benchmarkjunit k8s.io/test-infra/experiment/dummybenchmarks 13 ``` 14 15 ```shell 16 go run ./pkg/benchmarkjunit ./experiment/dummybenchmarks/... 17 ``` 18 19 ```shell 20 go run ./pkg/benchmarkjunit \ 21 -o junit_benchmark.xml -l testlog.txt \ 22 --test-arg=-benchmem --bench=Core --pass-on-error \ 23 ./experiment/dummybenchmarks/... 24 ``` 25 26 With bazel: 27 ```shell 28 bazel run //pkg/benchmarkjunit -- k8s.io/test-infra/experiment/dummybenchmarks 29 ``` 30 31 ### Running as a ProwJob 32 33 `benchmarkjunit` can be used from a ProwJob to generate JUnit results for 34 consumption in Testgrid, Spyglass, Gubernator, etc. See the 35 [`prowjob_example.yaml`](/pkg/benchmarkjunit/prowjob_example.yaml) file. 36 37 ### Demo Job 38 39 The [`ci-test-infra-benchmark-demo`] periodic ProwJob provides a live example 40 of how to use `benchmarkjunit` and what the results look like. Examples: 41 - [Testgrid](https://testgrid.k8s.io/sig-testing-canaries#benchmark-demo&width=20&graph-metrics=avg%20op%20duration%20(ns%2Fop)&graph-metrics=test-duration-minutes&graph-metrics=op%20count&graph-metrics=MB%2Fs&graph-metrics=alloced%20B%2Fop&graph-metrics=allocs%2Fop), 42 - [Spyglass](https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-benchmark-demo/1124364619008184320), 43 - [Raw results](https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-test-infra-benchmark-demo/1124364619008184320/artifacts/). 44 45 ## Translation Example: 46 47 The following is the `go test` benchmark output and the corresponding JUnit XML 48 for the [`experiment/dummybenchmarks`](/experiment/dummybenchmarks) package. 49 This was generated by running 50 `go run ./pkg/benchmarkjunit ./experiment/dummybenchmarks -l testlog.txt`. 51 52 #### Benchmark output: 53 54 ``` 55 goos: linux 56 goarch: amd64 57 pkg: k8s.io/test-infra/experiment/dummybenchmarks 58 BenchmarkCoreSimple-12 5000000 231 ns/op 59 BenchmarkCoreAllocsAndBytes-12 10000000 232 ns/op 85.84 MB/s 112 B/op 1 allocs/op 60 BenchmarkCoreParallel-12 50000000 28.9 ns/op 61 BenchmarkCoreLog-12 10000000 224 ns/op 62 --- BENCH: BenchmarkCoreLog-12 63 dummy_test.go:53: About to DoTheThing() x1. 64 dummy_test.go:53: About to DoTheThing() x100. 65 dummy_test.go:53: About to DoTheThing() x10000. 66 dummy_test.go:53: About to DoTheThing() x1000000. 67 dummy_test.go:53: About to DoTheThing() x10000000. 68 --- SKIP: BenchmarkCoreSkip 69 dummy_test.go:60: This Benchmark is skipped. 70 --- SKIP: BenchmarkCoreSkipNow 71 --- FAIL: BenchmarkCoreError 72 dummy_test.go:68: Early Benchmark error. 73 dummy_test.go:53: About to DoTheThing() x1. 74 --- FAIL: BenchmarkCoreFatal 75 dummy_test.go:73: This Benchmark failed. 76 --- FAIL: BenchmarkCoreFailNow 77 BenchmarkCoreNestedShallow/simple-12 5000000 236 ns/op 78 BenchmarkCoreNestedShallow/parallel-12 50000000 28.9 ns/op 79 Benchmark-12 10000000 226 ns/op 80 BenchmarkReportAllocs-12 10000000 234 ns/op 112 B/op 1 allocs/op 81 BenchmarkSetBytes-12 5000000 240 ns/op 83.00 MB/s 82 BenchmarkNestedDeep/1_simple-12 5000000 228 ns/op 83 BenchmarkNestedDeep/1_parallel-12 50000000 30.5 ns/op 84 BenchmarkNestedDeep/3A_simple-12 10000000 225 ns/op 85 BenchmarkNestedDeep/3A-12 1 2453493929 ns/op 86 BenchmarkNestedDeep/3B_parallel-12 50000000 33.1 ns/op 87 BenchmarkNestedDeep/3B-12 1 1697677606 ns/op 88 BenchmarkNestedDeep/2-12 1 4152090400 ns/op 89 BenchmarkNestedDeep/1-12 1 7125087855 ns/op 90 FAIL 91 exit status 1 92 FAIL k8s.io/test-infra/experiment/dummybenchmarks 25.419s 93 ``` 94 95 #### JUnit XML: 96 97 ```xml 98 <testsuites> 99 <testsuite name="k8s.io/test-infra/experiment/dummybenchmarks" time="25.419" failures="3" tests="20"> 100 <testcase name="BenchmarkCoreSimple-12" time="1.155" classname="dummybenchmarks"> 101 <properties> 102 <property name="op count" value="5000000"></property> 103 <property name="avg op duration (ns/op)" value="231"></property> 104 </properties> 105 </testcase> 106 <testcase name="BenchmarkCoreAllocsAndBytes-12" time="2.32" classname="dummybenchmarks"> 107 <properties> 108 <property name="op count" value="10000000"></property> 109 <property name="avg op duration (ns/op)" value="232"></property> 110 <property name="MB/s" value="85.84"></property> 111 <property name="alloced B/op" value="112"></property> 112 <property name="allocs/op" value="1"></property> 113 </properties> 114 </testcase> 115 <testcase name="BenchmarkCoreParallel-12" time="1.445" classname="dummybenchmarks"> 116 <properties> 117 <property name="op count" value="50000000"></property> 118 <property name="avg op duration (ns/op)" value="28.9"></property> 119 </properties> 120 </testcase> 121 <testcase name="BenchmarkCoreLog-12" time="2.24" classname="dummybenchmarks"> 122 <system-out>dummy_test.go:53: About to DoTheThing() x1.
dummy_test.go:53: About to DoTheThing() x100.
dummy_test.go:53: About to DoTheThing() x10000.
dummy_test.go:53: About to DoTheThing() x1000000.
dummy_test.go:53: About to DoTheThing() x10000000.
</system-out> 123 <properties> 124 <property name="op count" value="10000000"></property> 125 <property name="avg op duration (ns/op)" value="224"></property> 126 </properties> 127 </testcase> 128 <testcase name="BenchmarkCoreSkip" time="0" classname="dummybenchmarks"> 129 <skipped>dummy_test.go:60: This Benchmark is skipped.
</skipped> 130 </testcase> 131 <testcase name="BenchmarkCoreSkipNow" time="0" classname="dummybenchmarks"> 132 <skipped></skipped> 133 </testcase> 134 <testcase name="BenchmarkCoreError" time="0" classname="dummybenchmarks"> 135 <failure>dummy_test.go:68: Early Benchmark error.
dummy_test.go:53: About to DoTheThing() x1.
</failure> 136 <properties> 137 <property name="categorized_fail" value="dummy_test.go:68: Early Benchmark error.
dummy_test.go:53: About to DoTheThing() x1.
"></property> 138 </properties> 139 </testcase> 140 <testcase name="BenchmarkCoreFatal" time="0" classname="dummybenchmarks"> 141 <failure>dummy_test.go:73: This Benchmark failed.
</failure> 142 <properties> 143 <property name="categorized_fail" value="dummy_test.go:73: This Benchmark failed.
"></property> 144 </properties> 145 </testcase> 146 <testcase name="BenchmarkCoreFailNow" time="0" classname="dummybenchmarks"> 147 <failure></failure> 148 </testcase> 149 <testcase name="BenchmarkCoreNestedShallow/simple-12" time="1.18" classname="dummybenchmarks"> 150 <properties> 151 <property name="op count" value="5000000"></property> 152 <property name="avg op duration (ns/op)" value="236"></property> 153 </properties> 154 </testcase> 155 <testcase name="BenchmarkCoreNestedShallow/parallel-12" time="1.445" classname="dummybenchmarks"> 156 <properties> 157 <property name="op count" value="50000000"></property> 158 <property name="avg op duration (ns/op)" value="28.9"></property> 159 </properties> 160 </testcase> 161 <testcase name="Benchmark-12" time="2.26" classname="dummybenchmarks"> 162 <properties> 163 <property name="op count" value="10000000"></property> 164 <property name="avg op duration (ns/op)" value="226"></property> 165 </properties> 166 </testcase> 167 <testcase name="BenchmarkReportAllocs-12" time="2.34" classname="dummybenchmarks"> 168 <properties> 169 <property name="op count" value="10000000"></property> 170 <property name="avg op duration (ns/op)" value="234"></property> 171 <property name="alloced B/op" value="112"></property> 172 <property name="allocs/op" value="1"></property> 173 </properties> 174 </testcase> 175 <testcase name="BenchmarkSetBytes-12" time="1.2" classname="dummybenchmarks"> 176 <properties> 177 <property name="op count" value="5000000"></property> 178 <property name="avg op duration (ns/op)" value="240"></property> 179 <property name="MB/s" value="83.00"></property> 180 </properties> 181 </testcase> 182 <testcase name="BenchmarkNestedDeep/1_simple-12" time="1.14" classname="dummybenchmarks"> 183 <properties> 184 <property name="op count" value="5000000"></property> 185 <property name="avg op duration (ns/op)" value="228"></property> 186 </properties> 187 </testcase> 188 <testcase name="BenchmarkNestedDeep/1_parallel-12" time="1.525" classname="dummybenchmarks"> 189 <properties> 190 <property name="op count" value="50000000"></property> 191 <property name="avg op duration (ns/op)" value="30.5"></property> 192 </properties> 193 </testcase> 194 <testcase name="BenchmarkNestedDeep/3A_simple-12" time="2.25" classname="dummybenchmarks"> 195 <properties> 196 <property name="op count" value="10000000"></property> 197 <property name="avg op duration (ns/op)" value="225"></property> 198 </properties> 199 </testcase> 200 <testcase name="BenchmarkNestedDeep/3A-12" time="2.453493929" classname="dummybenchmarks"> 201 <properties> 202 <property name="op count" value="1"></property> 203 <property name="avg op duration (ns/op)" value="2453493929"></property> 204 </properties> 205 </testcase> 206 <testcase name="BenchmarkNestedDeep/3B_parallel-12" time="1.655" classname="dummybenchmarks"> 207 <properties> 208 <property name="op count" value="50000000"></property> 209 <property name="avg op duration (ns/op)" value="33.1"></property> 210 </properties> 211 </testcase> 212 <testcase name="BenchmarkNestedDeep/3B-12" time="1.697677606" classname="dummybenchmarks"> 213 <properties> 214 <property name="op count" value="1"></property> 215 <property name="avg op duration (ns/op)" value="1697677606"></property> 216 </properties> 217 </testcase> 218 <testcase name="BenchmarkNestedDeep/2-12" time="4.1520904" classname="dummybenchmarks"> 219 <properties> 220 <property name="op count" value="1"></property> 221 <property name="avg op duration (ns/op)" value="4152090400"></property> 222 </properties> 223 </testcase> 224 <testcase name="BenchmarkNestedDeep/1-12" time="7.125087855" classname="dummybenchmarks"> 225 <properties> 226 <property name="op count" value="1"></property> 227 <property name="avg op duration (ns/op)" value="7125087855"></property> 228 </properties> 229 </testcase> 230 </testsuite> 231 </testsuites> 232 ``` 233 234 <!-- links --> 235 [`testing` package benchmarks]: https://golang.org/pkg/testing/#hdr-Benchmarks 236 [`ci-test-infra-benchmark-demo`]: https://github.com/kubernetes/test-infra/blob/1423204f375ee7cf99f2a8a61a213562039b2ac7/config/jobs/kubernetes/test-infra/test-infra-canaries.yaml#L33-L51