sigs.k8s.io/gateway-api@v1.0.0/geps/gep-917.md (about)

     1  # GEP-917: Gateway API Conformance Testing
     2  
     3  * Issue: [#917](https://github.com/kubernetes-sigs/gateway-api/issues/917)
     4  * Status: Standard
     5  
     6  ## TLDR
     7  
     8  This GEP outlines the principles and overarching structure that Gateway API
     9  conformance tests will be built with.
    10  
    11  ## Goals
    12  
    13  - Record why we are doing conformance and what we hope to achieve with it
    14  - Record the success criteria for the conformance process and associated artifacts
    15  - Provide a set of guidelines for conformance test structure
    16  
    17  ## Non-Goals
    18  
    19  - Designing the conformance testing framework or implementation
    20  - Designing the process for implementations to prove they are conformant
    21  
    22  ## Introduction
    23  
    24  ### What is Conformance
    25  
    26  Conformance is the creation of a process that allows everyone, implementors and
    27  users alike, to check that an implementation conforms to the defined spec.
    28  
    29  For core Kubernetes, this also allows for the use of specific badges and branding.
    30  
    31  It usually includes some form of test harness that can produce a standard output,
    32  which can be submitted somewhere for validation. The place where the validations
    33  are held is then the canonical source of information about what implementations
    34  are conformant.
    35  
    36  ### Why do it for Gateway API?
    37  The Gateway API is a large, complex API with many use cases and implementations.
    38  Not all implementations support the same features, and some features have
    39  different required support levels.
    40  
    41  One of the primary goals of the Gateway API project is to make it safer and
    42  easier for end users to move their traffic configuration between implementations.
    43  Without some form of conformance to guarantee the same behavior between
    44  implementations, this is simply not achievable.
    45  
    46  Possibly a better way to say this is that we are looking to have Route resources
    47  portable between Gateways with a minimum of spec change.
    48  
    49  By creating a standard set of conformance tests and information, we can:
    50  
    51  - Make it easier for API consumers to understand what a particular API does
    52  - Make it possible for tooling to be constructed to check for portability
    53  between implementations
    54  
    55  Additionally, as the first project to develop an "official" set of CRDs, we have
    56  a responsibility to the community to build out a set of best practices for
    57  similar efforts in the future. Ensuring that whatever we build is reusable for
    58  other projects will help to lift everyone who works with CRDs.
    59  
    60  ### What do we need out of conformance for Gateway API?
    61  Must have:
    62  
    63  - Support for only implementing some of the Gateway API CRD resources. Some resources,
    64  like Gateway itself, are required for all implementations, but implementations
    65  may choose what Route resources they support. There will probably be some common
    66  sets of Route resources supported across similar implementations, but this proposal
    67  expects that we will make calls about what to call that common experience at a
    68  later date.
    69  - Support for fields or features that have Core, Extended, and
    70  ImplementationSpecific support. In particular, it must be possible for
    71  implementations to only support some subset of Extended fields, and to be able
    72  to use the framework for their own ImplementationSpecific features if required.
    73  - A testing suite that can validate that an implementation meets the conformance
    74  profiles it claims.
    75  - A way to retrieve conformance information in a machine-parseable way.
    76  
    77  ## Proposal
    78  
    79  ### Conformance Profiles
    80  
    81  The Gateway API project defines conformance purely in terms of what resources an
    82  implementation supports.
    83  To support a resource, an implementation must support all "Core" functionality
    84  defined by the resource. Support for "Extended" functionality will be indicated
    85  separately.
    86  
    87  All implementations must support all the Core functions on the following resources:
    88  
    89  - GatewayClass
    90  - Gateway
    91  - ReferenceGrant
    92  
    93  The following resources are optional to support, but have defined behavior if you
    94  do:
    95  
    96  - HTTPRoute
    97  - TLSRoute
    98  - TCPRoute
    99  - UDPRoute
   100  
   101  For all of these resources, we should aim to have the usual range of tests for
   102  both the happy and unhappy paths for various types of operations.
   103  
   104  The conformance is versioned - it tracks the required features for a specific
   105  version of the API, and must be included in and updated by a version bump in the
   106  bundle version of the Gateway API. (The _bundle version_ is the thing that we
   107  mark as a "release", that looks like `v0.4.0`, not `v1alpha2`). 
   108  
   109  This will enable an implementation to say that it supports a specific version
   110  of the Gateway API. This is again similar to upstream in that implementations need
   111  to submit conformance test results for each version of Kubernetes they support.
   112  
   113  Because the support is defined in terms of the resources that an implementation
   114  supports, the conformance is composable, and orthogonal for each object type.
   115  For example, it's valid to only support HTTPRoute and not TCPRoute, or TLSRoute
   116  and not HTTPRoute.
   117  
   118  ### Interaction with existing support levels
   119  
   120  Conformance definitions will ensure that an implementation can provide all the
   121  features currently marked as "Core" support in the API documentation.
   122  
   123  Fields marked "Extended" support will eventually have conformance tests that
   124  lock in the behavior of that feature, and there will be a mechanism for implementations
   125  to tell the testing framework what extended fields they support.
   126  
   127  ## Testing framework
   128  
   129  Conformance tests will initially be structured as a CLI that runs on existing
   130  Kubernetes clusters. This may be expanded to include a testing library that can
   131  be imported by implementations. This tool will output test results in a
   132  structured format that can be used to generate reports. Where possible it will
   133  share code with the existing ingress-controller-conformance test suite.
   134  
   135  ### Test Definitions
   136  
   137  Each set of tests will be built around a set of YAML manifests representing
   138  Gateway API resources. The tests will focus on how they expect a given
   139  controller to process those resources, configured via GatewayClass name. For
   140  example, tests may check to ensure that a controller correctly populates status,
   141  or that a request is appropriately routed to the desired backend with any
   142  relevant modifications.
   143  
   144  Each test definition will include the following:
   145  
   146  * Name
   147  * Description
   148  * Support Level
   149  
   150  In the future, this will be expanded to include:
   151  
   152  * Docs URL
   153  * Function to determine feature enablement
   154  
   155  ### Prepopulated Gateways
   156  
   157  Some implementations of Gateway API support prepopulated or unmanaged `Gateway`
   158  resources that refer to an underlying server that was created by other means
   159  (e.g. `Helm`) rather than provisioning and managing the server according to the
   160  `Gateway` resource. To ensure that our conformance tests can adequately cover
   161  these implementations, Gateways and any accompanying infrastructure may be
   162  preloaded into the cluster in which conformance tests will run. Importantly,
   163  Routes, ReferencePolicies, Services, or other resources used by the test MUST
   164  NOT be prepopulated. This approach will require a `--prepopulated-gateways` flag
   165  to be set, this will also be represented in the resulting conformance report.
   166  
   167  ## Certification process
   168  
   169  The Gateway API project will provide a process by which an implementation may
   170  submit the results of a run of the conformance test suite to a centralized,
   171  open repository, and once verified, these results will be used to maintain a
   172  canonical list of certified conformant implementations.
   173  
   174  Ideally, this process should be handled using similar methods to upstream
   175  Kubernetes, while also learning what we can from what the upstream conformance
   176  efforts wish they could improve.
   177  
   178  ## Alternatives Considered
   179  
   180  ### Tests Derived from Gateways
   181  
   182  It could be possible to write a testing framework that could derive the tests
   183  to run from Gateways that were present in a given cluster. For example, if a
   184  Gateway was present with an HTTP listener, it would run all tests that were
   185  possible with that type of listener.
   186  
   187  This approach could be especially helpful for implementations that do not yet
   188  support provisioning arbitrary Gateways. With the current proposal, these
   189  implementations would have to prepopulate a predefined set of Gateways for
   190  testing.
   191  
   192  Unfortunately, this approach also comes with some significant downsides:
   193  
   194  1. Implementations would still have to prepopulate Gateways.
   195  1. There are many possible inputs and outputs.
   196  1. We would need to test the testing framework to ensure that the correct tests
   197     were being run with a given input.
   198  1. The testing framework logic could become complicated if we wanted to avoid
   199     repeating tests that had already been run with other listeners.
   200  1. It may be more difficult to debug these tests if anything went wrong.
   201  1. Tests would need to be smart enough to recognize the side effects of any
   202     tests that had previously run. For example, multiple tests using the same
   203     Gateway could result in different status conditions depending on the order
   204     and/or combination that ran.
   205  
   206  
   207  ## References
   208  
   209  * [Gateway API Conformance Ideas](https://docs.google.com/document/d/18iECeKMp1OewSGISskv6Chfmjo9u2U0_iUH0jhPdKOk/edit#)
   210  * [Gateway API Conformance Requirements](https://docs.google.com/document/d/1QL-MpIVzqxe32Y2BZ_dYOB8zNsF9c4pnKEIB9ZLt118/edit)
   211  * [Gateway API Conformance Details](https://docs.google.com/document/d/1QL-MpIVzqxe32Y2BZ_dYOB8zNsF9c4pnKEIB9ZLt118/edit)