k8s.io/kubernetes@v1.29.3/test/integration/scheduler_perf/README.md (about) 1 Scheduler Performance Test 2 ====== 3 4 Motivation 5 ------ 6 We already have a performance testing system -- Kubemark. However, Kubemark requires setting up and bootstrapping a whole cluster, which takes a lot of time. 7 8 We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. We have the following goals: 9 10 - Save time on testing 11 - The test and benchmark can be run in a single box. 12 We only set up components necessary to scheduling without booting up a cluster. 13 - Profiling runtime metrics to find out bottleneck 14 - Write scheduler integration test but focus on performance measurement. 15 Take advantage of go profiling tools and collect fine-grained metrics, 16 like cpu-profiling, memory-profiling and block-profiling. 17 - Reproduce test result easily 18 - We want to have a known place to do the performance related test for scheduler. 19 Developers should just run one script to collect all the information they need. 20 21 Currently the test suite has the following: 22 23 - benchmark 24 - make use of `go test -bench` and report nanosecond/op. 25 - schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small: 10 - 100. 26 27 28 How To Run 29 ------ 30 31 ## Benchmark tests 32 33 ```shell 34 # In Kubernetes root path 35 make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling" 36 ``` 37 38 The benchmark suite runs all the tests specified under config/performance-config.yaml. 39 By default, it runs all workloads that have the "performance" label. In the configuration, 40 labels can be added to a test case and/or individual workloads. Each workload also has 41 all labels of its test case. The `perf-scheduling-label-filter` command line flag can 42 be used to select workloads. It works like GitHub label filtering: the flag accepts 43 a comma-separated list of label names. Each label may have a `+` or `-` as prefix. Labels with 44 `+` or no prefix must be set for a workload for it to be run. `-` means that the label must not 45 be set. For example, this runs all performance benchmarks except those that are labeled 46 as "fast": 47 ```shell 48 make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -perf-scheduling-label-filter=performance,-fast" 49 ``` 50 51 Once the benchmark is finished, JSON file with metrics is available in the current directory (test/integration/scheduler_perf). Look for `BenchmarkPerfScheduling_benchmark_YYYY-MM-DDTHH:MM:SSZ.json`. 52 You can use `-data-items-dir` to generate the metrics file elsewhere. 53 54 In case you want to run a specific test in the suite, you can specify the test through `-bench` flag: 55 56 Also, bench time is explicitly set to 1ns (`-benchtime=1ns` flag) so each test is run only once. 57 Otherwise, the golang benchmark framework will try to run a test more than once in case it ran for less than 1s. 58 59 ```shell 60 # In Kubernetes root path 61 make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling/SchedulingBasic/5000Nodes/5000InitPods/1000PodsToSchedule" 62 ``` 63 64 To produce a cpu profile: 65 66 ```shell 67 # In Kubernetes root path 68 make test-integration WHAT=./test/integration/scheduler_perf KUBE_TIMEOUT="-timeout=3600s" ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -cpuprofile ~/cpu-profile.out" 69 ``` 70 71 ### How to configure benchmark tests 72 73 Configuration file located under `config/performance-config.yaml` contains a list of templates. 74 Each template allows to set: 75 - node manifest 76 - manifests for initial and testing pod 77 - number of nodes, number of initial and testing pods 78 - templates for PVs and PVCs 79 - feature gates 80 81 See `op` data type implementation in [scheduler_perf_test.go](scheduler_perf_test.go) 82 for available operations to build `WorkloadTemplate`. 83 84 Initial pods create a state of a cluster before the scheduler performance measurement can begin. 85 Testing pods are then subject to performance measurement. 86 87 The configuration file under `config/performance-config.yaml` contains a default list of templates to cover 88 various scenarios. In case you want to add your own, you can extend the list with new templates. 89 It's also possible to extend `op` data type, respectively its underlying data types 90 to extend configuration of possible test cases. 91 92 ### Logging 93 94 The default verbosity is 2 (the recommended value for production). -v can be 95 used to change this. The log format can be changed with 96 -logging-format=text|json. The default is to write into a log file (when using 97 the text format) or stderr (when using JSON). Together these options allow 98 simulating different real production configurations and to compare their 99 performance. 100 101 During interactive debugging sessions it is possible to enable per-test output 102 via -use-testing-log. 103 104 ## Integration tests 105 106 To run integration tests, use: 107 ``` 108 make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_ARGS=-use-testing-log 109 ``` 110 111 Integration testing uses the same `config/performance-config.yaml` as 112 benchmarking. By default, workloads labeled as `integration-test` are executed 113 as part of integration testing. `-test-scheduling-label-filter` can be used to 114 change that.