sigs.k8s.io/gateway-api@v1.0.0/site-src/concepts/conformance.md (about) 1 # Conformance 2 3 This API covers a broad set of features and use cases and has been implemented 4 widely. This combination of both a large feature set and variety of 5 implementations requires clear conformance definitions and tests to ensure the 6 API provides a consistent experience wherever it is used. 7 8 When considering Gateway API conformance, there are three important concepts: 9 10 ## 1. Release Channels 11 12 Within Gateway API, release channels are used to indicate the stability of a 13 field or resource. The "standard" channel of the API includes fields and 14 resources that have graduated to "beta". The "experimental" channel of the API 15 includes everything in the "standard" channel, along with experimental fields 16 and resources that may still be changed in breaking ways **or removed 17 altogether**. For more information on this concept, refer to our 18 [versioning](/concepts/versioning) documentation. 19 20 ## 2. Support Levels 21 22 Unfortunately some implementations of the API will not be able to support every 23 feature that has been defined. To address that, the API defines a corresponding 24 support level for each feature: 25 26 * **Core** features will be portable and we expect that there is a reasonable 27 roadmap for ALL implementations towards support of APIs in this category. 28 * **Extended** features are those that are portable but not universally 29 supported across implementations. Those implementations that support the 30 feature will have the same behavior and semantics. It is expected that some 31 number of roadmap features will eventually migrate into the Core. Extended 32 features will be part of the API types and schema. 33 * **Implementation-specific** features are those that are not portable and are 34 vendor-specific. Implementation-specific features will not have API types and 35 schema except via generic extension points. 36 37 Behavior and feature in the Core and Extended set will be defined and validated 38 via behavior-driven conformance tests. Implementation-specific features will not 39 be covered by conformance tests. 40 41 By including and standardizing Extended features in the API spec, we expect to 42 be able to converge on portable subsets of the API among implementations without 43 compromising overall API support. Lack of universal support will not be a 44 blocker towards developing portable feature sets. Standardizing on spec will 45 make it easier to eventually graduate to Core when support is widespread. 46 47 ### Overlapping Support Levels 48 49 It is possible for support levels to overlap for a specific field. When this 50 occurs, the minimum expressed support level should be interpreted. For example, 51 an identical struct may be embedded in two different places. In one of those 52 places, the struct is considered to have Core support while the other place only 53 includes Extended support. Fields within this struct may express separate Core 54 and Extended support levels, but those levels must not be interpreted as 55 exceeding the support level of the parent struct they are embedded in. 56 57 For a more concrete example, HTTPRoute includes Core support for filters defined 58 within a Rule and Extended support when defined within BackendRef. Those filters 59 may separately define support levels for each field. When interpreting 60 overlapping support levels, the minimum value should be interpreted. That means 61 if a field has a Core support level but is in a filter attached in a place with 62 Extended support, the interpreted support level must be Extended. 63 64 ## 3. Conformance Tests 65 66 Gateway API includes a set of conformance tests. These create a series of 67 Gateways and Routes with the specified GatewayClass, and test that the 68 implementation matches the API specification. 69 70 Each release contains a set of conformance tests, these will continue to 71 expand as the API evolves. Currently conformance tests cover the majority 72 of Core capabilities in the standard channel, in addition to some Extended 73 capabilities. 74 75 ### Running Tests 76 77 There are two main contrasting sets of conformance tests: 78 79 * Gateway related tests (can also be thought of as ingress tests) 80 * Service Mesh related tests 81 82 For `Gateway` tests you must enable the `Gateway` test feature, and then 83 opt-in to any other specific tests you want to run (e.g. `HTTPRoute`). For 84 Mesh related tests you must enable `Mesh`. 85 86 We'll cover each use case separately, but it's also possible to combine these 87 if your implementation implements both. There are also options which pertain 88 to the entire test suite regardless of which tests you're running. 89 90 #### Gateway Tests 91 92 By default `Gateway` oriented conformance tests will expect a GatewayClass 93 named `gateway-conformance` to be installed in the cluster, and tests will be 94 run against that. Most often, you'll use a different class, which can be 95 specified with the `-gateway-class` flag along with the corresponding test 96 command. Check your instance for the `gateway-class` name to use. You must 97 also enable `Gateway` support and test support for any `*Routes` your 98 implementation supports. 99 100 The following runs all the tests relevant to `Gateway`, `HTTPRoute`, and 101 `ReferenceGrant`: 102 103 ```shell 104 go test ./conformance/... -args \ 105 -gateway-class=my-gateway-class \ 106 -supported-features=Gateway,HTTPRoute 107 ``` 108 109 Other useful flags may be found in [conformance flags][cflags]. 110 111 [cflags]:https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/utils/flags/flags.go 112 113 #### Mesh Tests 114 115 Mesh tests can be run by simply enabling the `Mesh` feature: 116 117 ```shell 118 go test ./conformance/... -args -supported-features=Mesh 119 ``` 120 121 If your mesh also includes ingress support with an API such as `HTTPRoute`, you 122 can run the relevant tests in the same test run by enabling the `Gateway` 123 feature and any relevant API features, e.g: 124 125 ```shell 126 go test ./conformance/... -args -supported-features=Mesh,Gateway,HTTPRoute 127 ``` 128 129 #### Namespace Labels and Annotations 130 131 If labels are needed on namespaces used for testing, you can use the 132 `-namespace-labels` flag to pass one or more `name=value` labels to set on the 133 test namespaces. Likewise, `-namespace-annotations` can be used to specify 134 annotations to be applied to the test namespaces. For mesh testing, this flag 135 can be used if an implementation requires labels on namespaces that host mesh 136 workloads, for example, to enable sidecar injection. 137 138 As an example, when testing Linkerd, you might run 139 140 ```shell 141 go test ./conformance/... -args \ 142 ... 143 -namespace-annotations=linkerd.io/inject=enabled 144 ``` 145 146 so that the test namespaces are correctly injected into the mesh. 147 148 #### Excluding Tests 149 150 The `Gateway` and `ReferenceGrant` features are enabled by default. 151 You do not need to explicitly list them using the `-supported-features` flag. 152 However, if you don't want to run them, you will need to disable them using 153 the `-exempt-features` flag. For example, to run only the `Mesh` tests, 154 and nothing else: 155 156 ```shell 157 go test ./conformance/... -args \ 158 -supported-features=Mesh \ 159 -exempt-features=Gateway,ReferenceGrant 160 ``` 161 162 #### Suite Level Options 163 164 When running tests of any kind you may not want the test suite to cleanup the 165 test resources when it completes (i.e. so that you can inspect the cluster 166 state in the event of a failure). You can skip cleanup with: 167 168 ```shell 169 go test ./conformance/... -args -cleanup-base-resources=false 170 ``` 171 172 It may be helpful (particularly when working on implementing a specific 173 feature) to run a very specific test by name. This can be done using the 174 `ShortName` of that test: 175 176 ```shell 177 go test ./conformance/... --run TestConformance/<ShortName> 178 ``` 179 180 ## Contributing to Conformance 181 182 Many implementations run conformance tests as part of their full e2e test suite. 183 Contributing conformance tests means that implementations can share the 184 investment in test development and ensure that we're providing a consistent 185 experience. 186 187 All code related to conformance lives in the "/conformance" directory of the 188 project. Test definitions are in "/conformance/tests" with each test including 189 a pair of files. A YAML file contains the manifests to be applied as part of 190 running the test. A Go file contains code that confirms that an implementation 191 handles those manifests appropriately. 192 193 Issues related to conformance are [labeled with 194 "area/conformance"](https://github.com/kubernetes-sigs/gateway-api/issues?q=is%3Aissue+is%3Aopen+label%3Aarea%2Fconformance). 195 These often cover adding new tests to improve our test coverage or fixing flaws 196 or limitations in our existing tests.