k8s.io/perf-tests/clusterloader2@v0.0.0-20240304094227-64bdb12da87e/docs/test_suite_design.md (about)

     1  # Cluster loader - TestSuite
     2  
     3  ## Background
     4  
     5  Since 2019 cluster loader become main tool for kubernetes e2e performance tests.
     6  It's very convenient for running single test case, but currently it's impossible
     7  to run the same test with two different sets of parameters in a single job run.
     8  This forces developers to create separate job configs for almost each test.
     9  
    10  ## TestSuite
    11  
    12  
    13  ```
    14  type TestSuite []TestScenario
    15  
    16  type TestScenario struct {
    17  	// Identifier is a unique test scenario name across test suite.
    18  	Identifier string
    19  	// ConfigPath defines path to the config file that satisfy already
    20  	// existing cluster loader API.
    21  	ConfigPath string
    22  	// OverridePaths defines what override files should be applied
    23  	// to the config specified by the ConfigPath.
    24  	OverridePaths []string
    25  }
    26  ```
    27  
    28  Test scenario level overrides benefits:
    29  - Ability to use the same template with multiple overrides in a single job.
    30  - Less error prone - variables with same names across different configs
    31  could have been accidentally overwritten before.
    32  
    33  To make using same test configs in one test suite more user-friendly,
    34  the Identifier field was introduced. This identifier will be used to match
    35  run results to the corresponding test.
    36  
    37  ## Other
    38  
    39  Path to the test suite yaml file will be passed to cluster loader as a flag.
    40  After TestSuite is supported by cluster loader, passing single test config paths
    41  and overrides using flags will be disabled.