sigs.k8s.io/gateway-api@v1.0.0/geps/gep-1911.md (about)

     1  # GEP-1911: Backend Protocol Selection
     2  
     3  * Issue: [#1911](https://github.com/kubernetes-sigs/gateway-api/issues/1911)
     4  * Status: Experimental
     5  
     6  (See status definitions [here](overview.md#status).)
     7  
     8  ## TLDR
     9  
    10  Not all implementations support automatic protocol selection. Even in some cases protocols are disabled without an explicit opt-in (eg. websockets with Contour & NGINX). Thus application developers need the ability to specify the protocol(s) that their application supports.
    11  
    12  ## Goals
    13  
    14  - Support protocols that can have a Gateway `*Route` resource as a frontend
    15  - Standardize Gateway API implementations on the protocols & constants defined by the Kubernetes [Standard Application Protocols (KEP-3726)][kep-3726]
    16  - Support backends with multiple protocols on the same port (ie. tcp/udp)
    17  
    18  ## Non-Goals
    19  
    20  - Backend TLS (covered in [GEP-1897](https://github.com/kubernetes-sigs/gateway-api/issues/1897))
    21  - Additional protocol specific configuration
    22  - Disabling Protocols
    23  
    24  ## Introduction
    25  
    26  Since Kubernetes 1.20 the [`core/v1.Service`][k8s-service] and [`core/v1.EndpointSlice`][k8s-endpointslices] resource has a stable `appProtocol` field. It's purpose is to allow end-users to specify an application protocol (L7) for each service port.
    27  
    28  Originally the use of this field in the Gateway API was rejected in [GEP-1282](/geps/gep-1282#non-goals):
    29  > v1.Service’s appProtocol field is not fit for purpose, because it is defined as accepting values either from the IANA Service Name registry, or domain-prefixed values and we need more flexibility than that. 
    30  
    31  Since then a Kubernetes enhancement proposal was created [KEP-3726][kep-3726] to repurpose `appProtocol` to include a convention for protocols that are not IANA service names. This would involve prefixing protocol names with `kubernetes.io/*`.
    32  
    33  Note: Kubernetes will automatically create `EndpointSlices` for `Services` that have a selector. [Custom `EndpointSlices`](https://kubernetes.io/docs/concepts/services-networking/service/#custom-endpointslices) can manually be created.
    34  
    35  ## API Semantics
    36  
    37  A Gateway implementation MUST recognize the Kubernetes Standard Application Protocols ([KEP-3726][kep-3726]) for specifying the protocol for a backend reference in a Gateway API `*Route` resource
    38  
    39  Thus when a `*Route` points to a Kubernetes Service, implementations SHOULD honor the appProtocol field if it
    40  is set for the target Service Port.
    41  
    42  
    43  At the moment there exists three defined constants:
    44  
    45  - `kubernetes.io/h2c` - HTTP/2 over cleartext as described in [RFC7540](https://www.rfc-editor.org/rfc/rfc7540)
    46  - `kubernetes.io/ws` - WebSocket over cleartext as described in [RFC6445](https://www.rfc-editor.org/rfc/rfc6455)
    47  - `kubernetes.io/wss` - WebSocket over TLS as described in [RFC6455](https://www.rfc-editor.org/rfc/rfc6455)
    48  
    49  ### New Protocols & Reserved Prefix
    50  
    51  To add support for a new protocol it should first become a Kubernetes Standard Application Protocol by updating the [KEP-3726][kep-3726]. [KEP-3726][kep-3726] also states the `appProtocol` field accepts a domain-prefixed implementation specific value. Thus, if the suggested protocol is not suited to have a `kubernetes.io/*` prefix, then the Gateway API MAY support the new protocol using it's own prefix `gateway.networking.k8s.io/*`.  Please make a PR to this GEP.
    52  
    53  For example we may want to add a sentinel `appProtocol` value that prevents Gateway implementations from discovering the protocol of the application. Instead they should just refer to the Service's `protocol` field. Such a constant was rejected upstream (https://github.com/kubernetes/enhancements/pull/4106) but as an example it could be defined in a future addition to this GEP as `gateway.networking.k8s.io/no-sniff`.
    54  
    55  ### Default Protocols
    56  
    57  If a Service `appProtocol` isn't specified an implementation MAY infer the backend protocol through its own means. Implementations MAY infer the protocol from the `Route` type referring to the backend Service.
    58  
    59  Absence of the `appProtocol` field does not imply the implementation should disable any features (eg. websocket upgrades).
    60  
    61  ### Multiple Protocols on the Same Port
    62  
    63  Only the Kubernetes `Service` `protocol` field supports multiple protocols on the same port. See the details in [KEP-1435][kep-1435].
    64  
    65  Implementations MAY support Kubernetes Service BackendRefs that are multiplexing TCP and UDP on the same port. Otherwise implementations MUST set *Route ResolvedRefs condition to False with the "UnsupportedProtocol" Reason with a clear message that multiplexing is not supported.
    66  
    67  Currently Kubernetes `Service` API only allows different `appProtocol` values for the same port when `protocol` fields differs. At this time there seems to be interest in changing `appProtocol` to be a list in order to facilitate this use-case.
    68  
    69  ### Supporting Protocols
    70  
    71  If a Route is not able to send traffic to the backend using the specified protocol then the backend is considered invalid. Implementations MUST set `ResolvedRefs` condition to `False` with the Reason `UnsupportedProtocol`.
    72  
    73  Implementations MAY support the following combinations below:
    74  
    75  ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
    76  -|-|-|-
    77  `TCP` | `kubernetes.io/h2c` | `GRPCRoute` | Yes [1]
    78  `TCP` | `kubernetes.io/h2c` | `HTTPRoute` | Yes
    79  `TCP` | `kubernetes.io/ws`  | `HTTPRoute` | Yes
    80  `TCP` | `kubernetes.io/wss` | `TLSRoute`  | Yes
    81  
    82  1. GRPC works over h2c - so a GRPCRoute should be able to connect to an h2c backend
    83  
    84  Implementations MAY support the following combinations below:
    85  
    86  ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
    87  -|-|-|-
    88  ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
    89  `TCP`  | `kubernetes.io/wss` | `HTTPRoute` | Conditional [1]
    90  
    91  1. Only if there is a corresponding `BackendTLSPolicy` - see [GEP-1897](/geps/gep-1897)
    92  
    93  ## Open Questions
    94  
    95  1. TLSRoute & UDP protocol
    96  
    97  TLS over UDP seems to be a thing via QUIC/HTTP3 [ref](https://www.smashingmagazine.com/2021/08/http3-core-concepts-part1/).
    98  Likewise there's also [DTLS](https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security). But it's unclear if Gateway's TLSRoute
    99  applies to an underlying UDP protocol.
   100  
   101  2. Websockets & HTTP/2/3
   102  
   103  Should we upstream new constants for websocket over [HTTP/2](https://www.rfc-editor.org/rfc/rfc8441.html) & [HTTP/3](https://www.rfc-editor.org/rfc/rfc9220.html) ? HTTP/3 makes things more complicated since its supports UDP as the underlying protocol.
   104  
   105  ## Alternatives
   106  
   107  ### Single Meta-resource 
   108  
   109  The first pass of this GEP proposed a new meta-resource [GEP-713](/geps/gep-713) called `BackendProtocol`.
   110  
   111  This allows end-users to specify a list of ports and a list of corresponding protocols that that single
   112  port supports.
   113  
   114  This was dropped in favour of supporting Kubernetes Standard Application Protocols.
   115  
   116  ### Multiple Protocol Meta-resources
   117  
   118  Rather than bundle protocol details into a single resource an alternative would be to create distinct meta resources.
   119  ie. `HTTP2Backend`, `GPRCBackend`, `WebsocketBackend`.
   120  
   121  The advantages of this approach are:
   122  
   123  - Easy to introduce new protocols
   124  - Definitions/types would be simpler
   125  
   126  The disadvantages of this approach are:
   127  
   128  - N resources for N protocols need to be created to describe a single backend
   129  - No easy mechanic to specify priority of protocols
   130  
   131  ### Adding Properties on Gateway Route Objects
   132  
   133  From [GEP-1282](/geps/gep-1282#tldr):
   134  > some types of configuration requested by users are more about defining functionality that describes capabilities of the backend more than the route you take to get to the backend.
   135  
   136  Backend protocol is specifying capabilities. This configuration is less about routing.
   137  
   138  ### Kubernetes Service - Expanding Protocol field
   139  
   140  The `protocol` field on a Kubernetes service is used to specify a L4 protocol over IP. This field isn't appropriate to describe protocols
   141  that operate at a higher 'application' level (eg. HTTP/GRPC etc.)
   142  
   143  ### Extending Kubernetes Service
   144  
   145  This is considered untenable due to the 'the turnaround time for those changes can be years.' ([ref-1282](/geps/gep-1282#non-goals))
   146  
   147  ### Unstructured Data/Special Values
   148  
   149  Unstructured data refers to using labels and annotations.
   150  
   151  From [GEP-1282](/geps/gep-1282#non-goals):
   152  > these are very sticky and hard to get rid of once you start using them.
   153  
   154  Special values refers to using special strings in existing Kubernetes Resources.
   155  For example Istio allows for protocol to be specified by prefixing the Kubernetes
   156  Service's port name with the protocol (ie. `http-`, `grpc-`). This approach is
   157  limiting as it doesn't allow for multiple protocols on the same port and future 
   158  configuration per protocol. One protocol per port may be relaxed in the future see 
   159  [KEP 1435][kep-1435]
   160  
   161  Additionally, annotations are not self-documenting unlike CRD fields which can display
   162  documentation via `kubectl explain`
   163  
   164  ## References
   165  
   166  - [GitHub Discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1244)
   167  - GEP-1282 - Describing Backend Properties
   168      - [GEP](/geps/gep-1282)
   169      - [Issue](https://github.com/kubernetes-sigs/gateway-api/issues/1911)
   170  - [GEP-713 - Metaresources](/geps/gep-713)
   171  - [Linkerd Protocol Detection](https://linkerd.io/2.12/features/protocol-detection/)
   172  - [Istio Protocol Selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/)
   173  - Contour Protocol Selection
   174      - [Websockets](https://projectcontour.io/docs/1.24/config/websockets/)
   175      - [GRPC](https://projectcontour.io/docs/1.24/guides/grpc/#httpproxy-configuration)
   176  - [AWS Gateway Protocol Selection](https://github.com/aws/aws-application-networking-k8s/blob/a277fb39449383f53cd7d1e5576b4fa190a1a853/config/crds/bases/application-networking.k8s.aws_targetgrouppolicies.yaml#L109)
   177  - [Google GKE AppProtocol Selection](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#https_tls_between_load_balancer_and_your_application)
   178  
   179  [k8s-service]: https://kubernetes.io/docs/concepts/services-networking/service/
   180  [k8s-endpointslices]: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
   181  [kep-3726]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3726-standard-application-protocols
   182  [kep-1435]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1435-mixed-protocol-lb