github.com/wmuizelaar/kpt@v0.0.0-20221018115725-bd564717b2ed/pkg/test/runner/README.md (about)

     1  This package is an end-to-end test runner for kpt fn commands. It
     2  can also be used to test kpt KRM functions.
     3  
     4  # How Does It Work
     5  
     6  After importing this package, you can use `runner.ScanTestCases` to
     7  recursively scan the test cases in a given directory. For each test
     8  case, call `runner.NewRunner` to create a test runner and then call
     9  `Run` to run the test. For example please see `e2e/fn_test.go`
    10  
    11  All test cases should be **independent** and should be able to run in
    12  fully parallel. They should be **self-contained** so the runner doesn't
    13  need any manual configuration.
    14  
    15  The configurations for a test case is located in the `.expected` directory
    16  in the test case directory. The runner will implicitly use following files
    17  in `.expected`:
    18  
    19  - `config.yaml`: This is the file which contains the configurations for the
    20    test case. It can have following fields:
    21    - `testType`: `eval` or `render`. It controls which `kpt fn` command will be
    22      used to run the test. Default: `render`.
    23    - `exitCode`: The expected exit code for the command. Default: 0.
    24    - `skip`: Runner will skip the test if `skip` is set to true. Default: false.
    25    - `sequential`: This test case should be run sequentially. Default: false.
    26    - `runtimes`: If the current runtime doesn't match any of the desired runtimes
    27      here, the test case will be skipped. Valid values are `docker` and `podman`.
    28      If unspecified, it will match any runtime.
    29    - `imagePullPolicy`: The image pull policy to be used. It can be set to one of
    30      `Always`, `IfNotPresent` and `Never`. Default value is inherited from the
    31      CLI flag.
    32    - `notIdempotent`: The functions and commands should be idempotent, but in
    33      some cases it's not doable. The runner will not run the command twice to
    34      check idempotent if this is set to true. Default: false.
    35    - `debug`: Debug means will the debug behavior be enabled. Default: false.
    36      Debug behaviors:
    37      - Keep the temporary directory used to run the test cases after test.
    38    - `stdOut`: The expected standard output from running the command. Default: "".
    39    - `stdErr`: The expected standard error from running the command. Default: "".
    40    - `StdErrRegEx`: A regular expression that is expected to match the standard error. Default: "".
    41    - `disableOutputTruncate`: Control should error output be truncated. Default:
    42      false.
    43    - Configurations only apply to `eval` tests:
    44      - `execPath`: A path to the executable file that will be run as function.
    45        Mutually exclusive with Image. The path should be separated by slash '/'
    46      - `image`: The image name for the function. Either `image` or `execPath`
    47        must be provided.
    48      - `args`: A map contains the arguments that will be passed into function.
    49        Args will be passed as 'key=value' format after the '--' in command.
    50      - `network`: Control the accessibility of network from the function
    51        container. Default: false.
    52      - `includeMetaResources`: Enable including meta resources, like Kptfile,
    53        in the function input. Default: false.
    54      - `fnConfig`: The path to the function config file. The path should be
    55        separated by slash '/'.
    56  - `diff.patch`: The expected `git diff` output after running the command.
    57    Default: "".
    58  - `results.yaml`: The expected result file after running the command.
    59    Default: "".
    60  - `setup.sh`: A **bash** script which will be run before the command if it exists.
    61  - `exec.sh`: A **bash** script which will be run if it exists and will replace the
    62    command (`kpt fn eval` or `kpt fn render`) that will be run according to
    63    `testType` in configurations. All configurations that used to control command
    64    behavior, like `disableOutputTruncate` and `args`, will be ignored.
    65  - `teardown.sh`: A **bash** script which will be run after the command and
    66    result comparison if it exists.
    67  
    68  # Expected Output
    69  
    70  The runner will compare the actual values of:
    71  
    72  - `stdout`
    73  - `stderr`
    74  - Exit Code
    75  - Diff
    76  - Results file