k8s.io/kubernetes@v1.29.3/test/integration/scheduler_perf/README.md (about)

     1  Scheduler Performance Test
     2  ======
     3  
     4  Motivation
     5  ------
     6  We already have a performance testing system -- Kubemark. However, Kubemark requires setting up and bootstrapping a whole cluster, which takes a lot of time.
     7  
     8  We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. We have the following goals:
     9  
    10  - Save time on testing
    11    - The test and benchmark can be run in a single box.
    12      We only set up components necessary to scheduling without booting up a cluster.
    13  - Profiling runtime metrics to find out bottleneck
    14    - Write scheduler integration test but focus on performance measurement.
    15      Take advantage of go profiling tools and collect fine-grained metrics,
    16      like cpu-profiling, memory-profiling and block-profiling.
    17  - Reproduce test result easily
    18    - We want to have a known place to do the performance related test for scheduler.
    19      Developers should just run one script to collect all the information they need.
    20  
    21  Currently the test suite has the following:
    22  
    23  - benchmark
    24    - make use of `go test -bench` and report nanosecond/op.
    25    - schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small: 10 - 100.
    26  
    27  
    28  How To Run
    29  ------
    30  
    31  ## Benchmark tests
    32  
    33  ```shell
    34  # In Kubernetes root path
    35  make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling"
    36  ```
    37  
    38  The benchmark suite runs all the tests specified under config/performance-config.yaml.
    39  By default, it runs all workloads that have the "performance" label. In the configuration,
    40  labels can be added to a test case and/or individual workloads. Each workload also has
    41  all labels of its test case. The `perf-scheduling-label-filter` command line flag can
    42  be used to select workloads. It works like GitHub label filtering: the flag accepts
    43  a comma-separated list of label names. Each label may have a `+` or `-` as prefix. Labels with
    44  `+` or no prefix must be set for a workload for it to be run. `-` means that the label must not
    45  be set. For example, this runs all performance benchmarks except those that are labeled
    46  as "fast":
    47  ```shell
    48  make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -perf-scheduling-label-filter=performance,-fast"
    49  ```
    50  
    51  Once the benchmark is finished, JSON file with metrics is available in the current directory (test/integration/scheduler_perf). Look for `BenchmarkPerfScheduling_benchmark_YYYY-MM-DDTHH:MM:SSZ.json`.
    52  You can use `-data-items-dir` to generate the metrics file elsewhere.
    53  
    54  In case you want to run a specific test in the suite, you can specify the test through `-bench` flag:
    55  
    56  Also, bench time is explicitly set to 1ns (`-benchtime=1ns` flag) so each test is run only once.
    57  Otherwise, the golang benchmark framework will try to run a test more than once in case it ran for less than 1s.
    58  
    59  ```shell
    60  # In Kubernetes root path
    61  make test-integration WHAT=./test/integration/scheduler_perf ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling/SchedulingBasic/5000Nodes/5000InitPods/1000PodsToSchedule"
    62  ```
    63  
    64  To produce a cpu profile:
    65  
    66  ```shell
    67  # In Kubernetes root path
    68  make test-integration WHAT=./test/integration/scheduler_perf KUBE_TIMEOUT="-timeout=3600s" ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -cpuprofile ~/cpu-profile.out"
    69  ```
    70  
    71  ### How to configure benchmark tests
    72  
    73  Configuration file located under `config/performance-config.yaml` contains a list of templates.
    74  Each template allows to set:
    75  - node manifest
    76  - manifests for initial and testing pod
    77  - number of nodes, number of initial and testing pods
    78  - templates for PVs and PVCs
    79  - feature gates
    80  
    81  See `op` data type implementation in [scheduler_perf_test.go](scheduler_perf_test.go) 
    82  for available operations to build `WorkloadTemplate`.
    83  
    84  Initial pods create a state of a cluster before the scheduler performance measurement can begin.
    85  Testing pods are then subject to performance measurement.
    86  
    87  The configuration file under `config/performance-config.yaml` contains a default list of templates to cover
    88  various scenarios. In case you want to add your own, you can extend the list with new templates.
    89  It's also possible to extend `op` data type, respectively its underlying data types
    90  to extend configuration of possible test cases.
    91  
    92  ### Logging
    93  
    94  The default verbosity is 2 (the recommended value for production). -v can be
    95  used to change this. The log format can be changed with
    96  -logging-format=text|json. The default is to write into a log file (when using
    97  the text format) or stderr (when using JSON). Together these options allow
    98  simulating different real production configurations and to compare their
    99  performance.
   100  
   101  During interactive debugging sessions it is possible to enable per-test output
   102  via -use-testing-log.
   103  
   104  ## Integration tests
   105  
   106  To run integration tests, use:
   107  ```
   108  make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_ARGS=-use-testing-log
   109  ```
   110  
   111  Integration testing uses the same `config/performance-config.yaml` as
   112  benchmarking. By default, workloads labeled as `integration-test` are executed
   113  as part of integration testing. `-test-scheduling-label-filter` can be used to
   114  change that.