sigs.k8s.io/cluster-api@v1.7.1/docs/book/src/developer/e2e.md (about)

     1  # Developing E2E tests
     2  
     3  E2E tests are meant to verify the proper functioning of a Cluster API management
     4  cluster in an environment that resembles a real production environment.
     5  
     6  The following guidelines should be followed when developing E2E tests:
     7  
     8  - Use the [Cluster API test framework].
     9  - Define test spec reflecting real user workflow, e.g. [Cluster API quick start].
    10  - Unless you are testing provider specific features, ensure your test can run with
    11    different infrastructure providers (see [Writing Portable Tests](#writing-portable-e2e-tests)).
    12  
    13  The [Cluster API test framework] provides you a set of helper methods for getting your test in place
    14  quickly. The [test E2E package] provides examples of how this can be achieved and reusable
    15  test specs for the most common Cluster API use cases.
    16  
    17  ## Prerequisites
    18  
    19  Each E2E test requires a set of artifacts to be available:
    20  
    21  - Binaries & Docker images for Kubernetes, CNI, CRI & CSI
    22  - Manifests & Docker images for the Cluster API core components
    23  - Manifests & Docker images for the Cluster API infrastructure provider; in most cases
    24    machine images are also required (AMI, OVA etc.)
    25  - Credentials for the target infrastructure provider
    26  - Other support tools (e.g. kustomize, gsutil etc.)
    27  
    28  The Cluster API test framework provides support for building and retrieving the manifest
    29  files for Cluster API core components and for the Cluster API infrastructure provider
    30  (see [Setup](#setup)).
    31  
    32  For the remaining tasks you can find examples of
    33  how this can be implemented e.g. in [CAPA E2E tests] and [CAPG E2E tests].
    34  
    35  ## Setup
    36  
    37  In order to run E2E tests it is required to create a Kubernetes cluster with a
    38  complete set of Cluster API providers installed. Setting up those elements is
    39  usually implemented in a `BeforeSuite` function, and it consists of two steps:
    40  
    41  - Defining an E2E config file
    42  - Creating the management cluster and installing providers
    43  
    44  ### Defining an E2E config file
    45  
    46  The [E2E config file] provides a convenient and flexible way to define common tasks for
    47  setting up a management cluster.
    48  
    49  Using the config file it is possible to:
    50  
    51  - Define the list of providers to be installed in the management cluster. Most notably,
    52    for each provider it is possible to define:
    53    - One or more versions of the providers manifest (built from the sources, or pulled from a
    54      remote location).
    55    - A list of additional files to be added to the provider repository, to be used e.g.
    56      to provide `cluster-templates.yaml` files.
    57  - Define the list of variables to be used when doing `clusterctl init` or
    58    `clusterctl generate cluster`.
    59  - Define a list of intervals to be used in the test specs for defining timeouts for the
    60    wait and `Eventually` methods.
    61  - Define the list of images to be loaded in the management cluster (this is specific to
    62    management clusters based on kind).
    63  
    64  An [example E2E config file] can be found here.
    65  
    66  ### Creating the management cluster and installing providers
    67  
    68  In order to run Cluster API E2E tests, you need a Kubernetes cluster. The [NewKindClusterProvider] gives you a
    69  type that can be used to create a local kind cluster and pre-load images into it. Existing clusters can
    70  be used if available.
    71  
    72  Once you have a Kubernetes cluster, the [InitManagementClusterAndWatchControllerLogs method] provides a convenient
    73  way for installing providers.
    74  
    75  This method:
    76  - Runs `clusterctl init` using the above local repository.
    77  - Waits for the providers controllers to be running.
    78  - Creates log watchers for all the providers
    79  
    80  ## Writing test specs
    81  
    82  A typical test spec is a sequence of:
    83  
    84  - Creating a namespace to host in isolation all the test objects.
    85  - Creating objects in the management cluster, wait for the corresponding infrastructure to be provisioned.
    86  - Exec operations like e.g. changing the Kubernetes version or  `clusterctl move`, wait for the action to complete.
    87  - Delete objects in the management cluster, wait for the corresponding infrastructure to be terminated.
    88  
    89  ### Creating Namespaces
    90  
    91  The [CreateNamespaceAndWatchEvents method] provides a convenient way to create a namespace and setup
    92  watches for capturing namespaces events.
    93  
    94  ### Creating objects
    95  
    96  There are two possible approaches for creating objects in the management cluster:
    97  
    98  - Create object by object: create the `Cluster` object, then `AwsCluster`, `Machines`, `AwsMachines` etc.
    99  - Apply a `cluster-templates.yaml` file thus creating all the objects this file contains.
   100  
   101  The first approach leverages the [controller-runtime Client] and gives you full control, but it comes with
   102  some drawbacks as well, because this method does not directly reflect real user workflows, and most importantly,
   103  the resulting tests are not as reusable with other infrastructure providers. (See [writing portable tests](#writing-portable-e2e-tests)).
   104  
   105  We recommend using the [ClusterTemplate method] and the [Apply method] for creating objects in the cluster.
   106  This methods mimics the recommended user workflows, and it is based on `cluster-templates.yaml` files that can be
   107  provided via the [E2E config file], and thus easily swappable when changing the target infrastructure provider.
   108  
   109  <aside class="note">
   110  
   111  <h1>Tips</h1>
   112  
   113  If you need control over object creation but want to preserve portability, you can create many templates
   114  files each one creating only a small set of objects (instead of using a single template creating a full cluster).
   115  
   116  </aside>
   117  
   118  After creating objects in the cluster, use the existing methods in the [Cluster API test framework] to discover
   119  which object were created in the cluster so your code can adapt to different `cluster-templates.yaml` files.
   120  
   121  Once you have object references, the framework includes methods for waiting for the corresponding
   122  infrastructure to be provisioned, e.g. [WaitForClusterToProvision], [WaitForKubeadmControlPlaneMachinesToExist].
   123  
   124  ### Exec operations
   125  
   126  You can use [Cluster API test framework] methods to modify Cluster API objects, as a last option, use
   127  the [controller-runtime Client].
   128  
   129  The [Cluster API test framework] also includes methods for executing clusterctl operations, like e.g.
   130  the [ClusterTemplate method], the [ClusterctlMove method] etc.. In order to improve observability,
   131  each clusterctl operation creates a detailed log.
   132  
   133  After using clusterctl operations, you can rely on the `Get` and on the `Wait` methods
   134  defined in the [Cluster API test framework] to check if the operation completed successfully.
   135  
   136  ### Naming the test spec
   137  
   138  You can categorize the test with a custom label that can be used to filter a category of E2E tests to be run. Currently, the cluster-api codebase has [these labels](./testing.md#running-specific-tests) which are used to run a focused subset of tests.
   139  
   140  ## Tear down
   141  
   142  After a test completes/fails, it is required to:
   143  
   144  - Collect all the logs for the Cluster API controllers
   145  - Dump all the relevant Cluster API/Kubernetes objects
   146  - Cleanup all the infrastructure resources created during the test
   147  
   148  Those tasks are usually implemented in the `AfterSuite`, and again the [Cluster API test framework] provides
   149  you useful methods for those tasks.
   150  
   151  Please note that despite the fact that test specs are expected to delete objects in the management cluster and
   152  wait for the corresponding infrastructure to be terminated, it can happen that the test spec
   153  fails before starting object deletion or that objects deletion itself fails.
   154  
   155  As a consequence, when scheduling/running a test suite, it is required to ensure all the generated
   156  resources are cleaned up. In Kubernetes, this is implemented by the [boskos] project.
   157  
   158  ## Writing portable E2E tests
   159  
   160  A portable E2E test is a test that can run with different infrastructure providers by simply
   161  changing the test configuration file.
   162  
   163  The following recommendations should be followed to write portable E2E tests:
   164  
   165  - Create different [E2E config file], one for each target infrastructure provider,
   166    providing different sets of env variables and timeout intervals.
   167  - Use the [InitManagementClusterAndWatchControllerLogs method] for setting up the management cluster.
   168  - Use the [ClusterTemplate method] and the [Apply method]
   169    for creating objects in the cluster using `cluster-templates.yaml` files instead
   170    of hard coding object creation.
   171  - Use the `Get` methods defined in the [Cluster API test framework] to check objects
   172    being created, so your code can adapt to different `cluster-templates.yaml` files.
   173  - Never hard code the infrastructure provider name in your test spec.
   174    Instead, use the [InfrastructureProvider method] to get access to the
   175    name of the infrastructure provider defined in the [E2E config file].
   176  - Never hard code wait intervals in your test spec.
   177    Instead use the [GetIntervals method] to get access to the
   178    intervals defined in the [E2E config file].
   179  
   180  ## Cluster API conformance tests
   181  
   182  As of today there is no a well-defined suite of E2E tests that can be used as a
   183  baseline for Cluster API conformance.
   184  
   185  However, creating such a suite is something that can provide a huge value for the
   186  long term success of the project.
   187  
   188  The [test E2E package] provides examples of how this can be achieved by implementing a set of reusable
   189  test specs for the most common Cluster API use cases.
   190  
   191  <!-- links -->
   192  [Cluster API quick start]:  ../user/quick-start.md
   193  [Cluster API test framework]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc
   194  [Apply method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#Applier
   195  [CAPA E2E tests]: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/scripts/ci-e2e.sh
   196  [CAPG E2E tests]: https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/main/scripts/ci-e2e.sh
   197  [WaitForClusterToProvision]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#WaitForClusterToProvision
   198  [WaitForKubeadmControlPlaneMachinesToExist]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#WaitForKubeadmControlPlaneMachinesToExist
   199  [controller-runtime Client]: https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.5.2/pkg/client?tab=doc#Client
   200  [boskos]: https://git.k8s.io/test-infra/boskos
   201  [E2E config file]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig
   202  [example E2E config file]: https://github.com/kubernetes-sigs/cluster-api/blob/main/test/e2e/config/docker.yaml
   203  [NewKindClusterProvider]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/bootstrap?tab=doc#NewKindClusterProvider
   204  [InitManagementClusterAndWatchControllerLogs method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#InitManagementClusterAndWatchControllerLogs
   205  [ClusterTemplate method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#ConfigCluster
   206  [ClusterctlMove method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#Move
   207  [InfrastructureProvider method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig.InfrastructureProviders
   208  [GetIntervals method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework/clusterctl?tab=doc#E2EConfig.GetIntervals
   209  [test E2E package]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/e2e?tab=doc
   210  [CreateNamespaceAndWatchEvents method]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework?tab=doc#CreateNamespaceAndWatchEvents