github.com/coreos/mantle@v0.13.0/runner-readme.md (about) 1 # Kola Runner 2 3 ## Table of Contents 4 5 * [General Function Overview](#general-function-overview) 6 * [Visual Function Flow](#visual-function-flow) 7 * [Detailed Workflow](#detailed-workflow) 8 * [Common Closures](#common-closures) 9 * [Logging](#logging) 10 11 The `kola` runner is very similar to the standard `go test` runner with some 12 modifications to allow tighter control over the run loop and extensibility 13 for logging (without parsing) & output control. 14 15 ## General Function Overview 16 17 ### kola/harness 18 19 #### kola/harness: RunTests 20 21 Responsible for filtering down the test list based on the given criteria, 22 creating the `harness/suite: Suite` object, and outputting the final result. 23 24 #### kola/harness: runTest 25 26 Creates the test cluster, runs the individual test function, and cleans up the 27 test cluster. 28 29 ### harness/suite 30 31 #### harness/suite: Run 32 33 Creates the output directory, the `test.tap` file and any profile related 34 files then calls `harness/suite: runTests`. 35 36 #### harness/suite: runTests 37 38 Sets up the `harness/harness: H` object and calls `harness/harness: tRunner` 39 with a closure to call `harness/harness: Run` for each test. 40 41 ### harness/harness 42 43 #### harness/harness: tRunner 44 45 Responsible for the timing, reporting, and execution of a closure. 46 47 #### harness/harness: Run 48 49 Handles the setup of child `harness/harness: H` objects, loggers, and the 50 running of closures as subtests. 51 52 ## Visual Function Flow 53 54 The first 4 steps handle filtering down the test list, creating the 55 clusters, and building the suite. The next 4 set up the reporting 56 structure of the test group and run the child tests. The following 2 get 57 ready to run each individual test. And the final step runs the actual 58 test function registered in the test. 59 60 ``` 61 cmd/kola/kola 62 | 63 v 64 kola/harness: RunTests 65 | 66 v 67 harness/suite: Run 68 | 69 v 70 harness/suite: runTests 71 | 72 v 73 harness/harness: tRunner 74 | 75 v 76 harness/harness: Run 77 | 78 v 79 harness/harness: tRunner 80 | 81 v 82 kola/harness: runTest 83 | 84 v 85 harness/harness: Run 86 | 87 v 88 harness/harness: tRunner 89 | 90 v 91 kola/register/register: Run 92 ``` 93 94 ## Detailed Workflow 95 96 1. The `kola` cmd calls into `kola/harness: RunTests` 97 2. `kola/harness: RunTests` calls `kola/harness: filterTests` to build a test 98 list filtered down by the given pattern & platform from all tests in 99 `kola/register/register: Tests` object. 100 3. `kola/harness: RunTests` checks if any of the tests do not exactly match 101 the given pattern and have either a `MinVersion` or `MaxVersion` tag, if so 102 then it will call `kola/harness: getClusterSemver` to spin up a machine on 103 the given platform to extract the OS semver. The tests will then be filtered 104 down again with the semver. Note that if a pattern matches an individual test 105 the `kola/harness: getClusterSemver` check will be skipped and the test will 106 be run without regard to the `MinVersion` or `MaxVersion` tags. 107 4. `kola/harness: RunTests` will then construct a `harness/suite: Options` 108 object and construct a `harness/test: Test` object containing the name of each 109 test and a closure (#1) calling `kola/harness: runTest`. 110 5. `kola/harness: RunTests` constructs a `harness/suite: Suite` object via 111 `harness/suite: NewSuite` using the `harness/suite: Options` and 112 `harness/test: Test` objects and proceeds to call the `harness/suite: Run` 113 function on the `harness/suite: Suite` object. 114 6. `harness/suite: Run` starts by creating or cleaning up the output directory 115 by calling the `harness/harness: CleanOutputDir` function. It then creates the 116 `test.tap` file inside of the output directory and prints a string to the file 117 containing `1..%d` where %d is the amount of tests being run. 118 7. `harness/suite: Run` then checks if the following options were selected and 119 if so creates the corresponding files in the output path: 120 121 | Option | Filename | 122 | -------------- | ---------- | 123 | MemProfile | mem.prof | 124 | BlockProfile | block.prof | 125 | CpuProfile | cpu.prof | 126 | ExecutionTrace | exec.trace | 127 128 8. `harness/suite: Run` then calls `harness/suite: runTests` passing 129 `os.Stdout` and the `tap io.Writer` object. 130 9. `harness/suite: runTests` starts by setting the `running` variable on the 131 `harness/suite: Suite` object, which is the count of running tests, to 1 and 132 creating the `harness/harness: H` object. 133 10. `harness/suite: runTests` then calls `harness/harness: tRunner` passing 134 the `harness/harness: H` object and a closure (#2) which loops each test in the 135 `harness/suite: Suite` object calling `harness/harness: Run` on each, passing 136 the name of the test, the `harness/test: Test` object, and a boolean pointer 137 set to false, followed by a goroutine call to receive from the signal channel 138 on the `harness/harness: H` object. 139 11. `harness/harness: tRunner` starts by creating a `context.WithCancel` 140 object, the result of `harness/harness: parentContext` is passed in which will 141 either be the context object of the `harness/harness: H` objects parent or 142 `context.Background()` if the object doesn't have a parent. 143 12. `harness/harness: tRunner` then defers a closure which will detect the 144 status of the test run, calculate the ending time, run any subtests, call 145 `harness/harness: report` (which will flush the test result to the parent 146 via the `harness/harness: flushToParent` function), and send `true` on the 147 `harness/harness: H` `signal` channel. 148 13. `harness/harness: tRunner` will then calculate the start time and call the 149 closure it received as an argument with the `harness/harness: H` variable as a 150 parameter, this will be the closure that was created in 151 `harness/suite: runTests` which will call `harness/harness: Run` for each test. 152 14. `harness/harness: Run` runs each function as a subtest of the 153 `harness/harness: H` object it is passed with the name passed. It starts by 154 marking the `hasSub` variable on the `harness/harness: H` object to true and 155 checking that the test name it received is a valid test via the 156 `harness/match: fullName` function. 157 15. `harness/harness: Run` will then create a new `harness/harness: H` object 158 which has the object it received as the parent and a `log` object. 159 16. `harness/harness: Run` then does a goroutine call on 160 `harness/harness: tRunner` passing in the new `harness/harness: H` object, 161 the closure function it was passed, which is the call to 162 `kola/harness: runTest`, and the boolean pointer it was passed. 163 17. `harness/harness: tRunner` will then run through and call 164 `kola/harness: runTest`. 165 18. `kola/harness: runTest` is the harness responsible for running a single 166 test grouping (test groupings tests. It will create the cluster that 167 will be used by the tests, validate that the machines spun up properly, 168 and then call `kola/register/register: Run` on the 169 `kola/register/register: Test` object, which is a function pointer which 170 accepts a `kola/cluster/cluster: TestCluster` object and is defined 171 inside of the individual test files. 172 173 ## Common Closures 174 175 1. `kola/harness: RunTests` 176 177 Accepts a `harness/harness: H` object and calls `kola/harness: runTest` 178 179 ``` 180 func(h *harness.H) { 181 runTest(h, []*register.Test{test}, pltfrm, false) 182 } 183 ``` 184 185 2. `harness/suite: runTests` 186 187 Accepts a `harness/harness: H` object. Loops each test in the 188 `harness/suite: Suite` object calling `harness/harness: Run`. This is being 189 pass as an argument to `harness/harness: tRunner`. `harness/harness:tRunner` 190 will time the the outer block and call `harness/harness: Run` which will run 191 the `test` function as a subtest. 192 193 For instance, `harness/harness: tRunner` will be called with the 194 `harness/harness: H` object representing the entire test run. It will then 195 execute this closure which will loop through every test and call 196 `harness/harness: Run` which will run each as a subtest for reporting purposes 197 inside of goroutines. 198 199 ``` 200 func(t *H) { 201 for name, test := range s.tests { 202 t.Run(name, test, util.BoolToPtr(false)) 203 } 204 // Run catching the signal rather than the tRunner as a separate 205 // goroutine to avoid adding a goroutine during the sequential 206 // phase as this pollutes the stacktrace output when aborting. 207 go func() { <-t.signal }() 208 } 209 ``` 210 211 ## Logging 212 213 The `kola` runner supports custom reporting via the 214 `harness/reporters: Reporter` interface. By default plain text will be output 215 into `stdout` and a JSON file will be produced inside of the `_kola_temp` run 216 log (e.x.: `_kola_temp/<platform>-latest/reports/report.json`). New output 217 formats can be added by creating a new struct which implements the 218 `harness/reporters: Reporter` interface and instantiating an object of said 219 reporter inside of the `harness: Options` object created in 220 `kola/harness: RunTests`. 221 222 [For example](https://github.com/coreos/mantle/blob/52407c3ae8cd0837511c665af2c7870393e024bb/kola/harness.go#L295-L297) this is how the JSON reporter is added.