go.etcd.io/etcd@v3.3.27+incompatible/Documentation/op-guide/grpc_proxy.md (about) 1 --- 2 title: gRPC proxy 3 --- 4 5 The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests. 6 7 The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future. 8 9 ## Scalable watch API 10 11 The gRPC proxy coalesces multiple client watchers (`c-watchers`) on the same key or range into a single watcher (`s-watcher`) connected to an etcd server. The proxy broadcasts all events from the `s-watcher` to its `c-watchers`. 12 13 Assuming N clients watch the same key, one gRPC proxy can reduce the watch load on the etcd server from N to 1. Users can deploy multiple gRPC proxies to further distribute server load. 14 15 In the following example, three clients watch on key A. The gRPC proxy coalesces the three watchers, creating a single watcher attached to the etcd server. 16 17 ``` 18 +-------------+ 19 | etcd server | 20 +------+------+ 21 ^ watch key A (s-watcher) 22 | 23 +-------+-----+ 24 | gRPC proxy | <-------+ 25 | | | 26 ++-----+------+ |watch key A (c-watcher) 27 watch key A ^ ^ watch key A | 28 (c-watcher) | | (c-watcher) | 29 +-------+-+ ++--------+ +----+----+ 30 | client | | client | | client | 31 | | | | | | 32 +---------+ +---------+ +---------+ 33 ``` 34 35 ### Limitations 36 37 To effectively coalesce multiple client watchers into a single watcher, the gRPC proxy coalesces new `c-watchers` into an existing `s-watcher` when possible. This coalesced `s-watcher` may be out of sync with the etcd server due to network delays or buffered undelivered events. When the watch revision is unspecified, the gRPC proxy will not guarantee the `c-watcher` will start watching from the most recent store revision. For example, if a client watches from an etcd server with revision 1000, that watcher will begin at revision 1000. If a client watches from the gRPC proxy, may begin watching from revision 990. 38 39 Similar limitations apply to cancellation. When the watcher is cancelled, the etcd server’s revision may be greater than the cancellation response revision. 40 41 These two limitations should not cause problems for most use cases. In the future, there may be additional options to force the watcher to bypass the gRPC proxy for more accurate revision responses. 42 43 ## Scalable lease API 44 45 To keep its leases alive, a client must establish at least one gRPC stream to an etcd server for sending periodic heartbeats. If an etcd workload involves heavy lease activity spread over many clients, these streams may contribute to excessive CPU utilization. To reduce the total number of streams on the core cluster, the proxy supports lease stream coalescing. 46 47 Assuming N clients are updating leases, a single gRPC proxy reduces the stream load on the etcd server from N to 1. Deployments may have additional gRPC proxies to further distribute streams across multiple proxies. 48 49 In the following example, three clients update three independent leases (`L1`, `L2`, and `L3`). The gRPC proxy coalesces the three client lease streams (`c-streams`) into a single lease keep alive stream (`s-stream`) attached to an etcd server. The proxy forwards client-side lease heartbeats from the c-streams to the s-stream, then returns the responses to the corresponding c-streams. 50 51 ``` 52 +-------------+ 53 | etcd server | 54 +------+------+ 55 ^ 56 | heartbeat L1, L2, L3 57 | (s-stream) 58 v 59 +-------+-----+ 60 | gRPC proxy +<-----------+ 61 +---+------+--+ | heartbeat L3 62 ^ ^ | (c-stream) 63 heartbeat L1 | | heartbeat L2 | 64 (c-stream) v v (c-stream) v 65 +------+-+ +-+------+ +-----+--+ 66 | client | | client | | client | 67 +--------+ +--------+ +--------+ 68 ``` 69 70 ## Abusive clients protection 71 72 The gRPC proxy caches responses for requests when it does not break consistency requirements. This can protect the etcd server from abusive clients in tight for loops. 73 74 ## Start etcd gRPC proxy 75 76 Consider an etcd cluster with the following static endpoints: 77 78 |Name|Address|Hostname| 79 |------|---------|------------------| 80 |infra0|10.0.1.10|infra0.example.com| 81 |infra1|10.0.1.11|infra1.example.com| 82 |infra2|10.0.1.12|infra2.example.com| 83 84 Start the etcd gRPC proxy to use these static endpoints with the command: 85 86 ```bash 87 $ etcd grpc-proxy start --endpoints=infra0.example.com,infra1.example.com,infra2.example.com --listen-addr=127.0.0.1:2379 88 ``` 89 90 The etcd gRPC proxy starts and listens on port 2379. It forwards client requests to one of the three endpoints provided above. 91 92 Sending requests through the proxy: 93 94 ```bash 95 $ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 put foo bar 96 OK 97 $ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 get foo 98 foo 99 bar 100 ``` 101 102 ## Client endpoint synchronization and name resolution 103 104 The proxy supports registering its endpoints for discovery by writing to a user-defined endpoint. This serves two purposes. First, it allows clients to synchronize their endpoints against a set of proxy endpoints for high availability. Second, it is an endpoint provider for etcd [gRPC naming](../dev-guide/grpc_naming.md). 105 106 Register proxy(s) by providing a user-defined prefix: 107 108 ```bash 109 $ etcd grpc-proxy start --endpoints=localhost:2379 \ 110 --listen-addr=127.0.0.1:23790 \ 111 --advertise-client-url=127.0.0.1:23790 \ 112 --resolver-prefix="___grpc_proxy_endpoint" \ 113 --resolver-ttl=60 114 115 $ etcd grpc-proxy start --endpoints=localhost:2379 \ 116 --listen-addr=127.0.0.1:23791 \ 117 --advertise-client-url=127.0.0.1:23791 \ 118 --resolver-prefix="___grpc_proxy_endpoint" \ 119 --resolver-ttl=60 120 ``` 121 122 The proxy will list all its members for member list: 123 124 ```bash 125 ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23790 member list --write-out table 126 127 +----+---------+--------------------------------+------------+-----------------+ 128 | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | 129 +----+---------+--------------------------------+------------+-----------------+ 130 | 0 | started | Gyu-Hos-MBP.sfo.coreos.systems | | 127.0.0.1:23791 | 131 | 0 | started | Gyu-Hos-MBP.sfo.coreos.systems | | 127.0.0.1:23790 | 132 +----+---------+--------------------------------+------------+-----------------+ 133 ``` 134 135 This lets clients automatically discover proxy endpoints through Sync: 136 137 ```go 138 cli, err := clientv3.New(clientv3.Config{ 139 Endpoints: []string{"http://localhost:23790"}, 140 }) 141 if err != nil { 142 log.Fatal(err) 143 } 144 defer cli.Close() 145 146 // fetch registered grpc-proxy endpoints 147 if err := cli.Sync(context.Background()); err != nil { 148 log.Fatal(err) 149 } 150 ``` 151 152 Note that if a proxy is configured without a resolver prefix, 153 154 ```bash 155 $ etcd grpc-proxy start --endpoints=localhost:2379 \ 156 --listen-addr=127.0.0.1:23792 \ 157 --advertise-client-url=127.0.0.1:23792 158 ``` 159 160 The member list API to the grpc-proxy returns its own `advertise-client-url`: 161 162 ```bash 163 ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23792 member list --write-out table 164 165 +----+---------+--------------------------------+------------+-----------------+ 166 | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | 167 +----+---------+--------------------------------+------------+-----------------+ 168 | 0 | started | Gyu-Hos-MBP.sfo.coreos.systems | | 127.0.0.1:23792 | 169 +----+---------+--------------------------------+------------+-----------------+ 170 ``` 171 172 ## Namespacing 173 174 Suppose an application expects full control over the entire key space, but the etcd cluster is shared with other applications. To let all appications run without interfering with each other, the proxy can partition the etcd keyspace so clients appear to have access to the complete keyspace. When the proxy is given the flag `--namespace`, all client requests going into the proxy are translated to have a user-defined prefix on the keys. Accesses to the etcd cluster will be under the prefix and responses from the proxy will strip away the prefix; to the client, it appears as if there is no prefix at all. 175 176 To namespace a proxy, start it with `--namespace`: 177 178 ```bash 179 $ etcd grpc-proxy start --endpoints=localhost:2379 \ 180 --listen-addr=127.0.0.1:23790 \ 181 --namespace=my-prefix/ 182 ``` 183 184 Accesses to the proxy are now transparently prefixed on the etcd cluster: 185 186 ```bash 187 $ ETCDCTL_API=3 etcdctl --endpoints=localhost:23790 put my-key abc 188 # OK 189 $ ETCDCTL_API=3 etcdctl --endpoints=localhost:23790 get my-key 190 # my-key 191 # abc 192 $ ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 get my-prefix/my-key 193 # my-prefix/my-key 194 # abc 195 ``` 196 197 ## TLS termination 198 199 Terminate TLS from a secure etcd cluster with the gRPC proxy by serving an unencrypted local endpoint. 200 201 To try it out, start a single member etcd cluster with client https: 202 203 ```sh 204 $ etcd --listen-client-urls https://localhost:2379 --advertise-client-urls https://localhost:2379 --cert-file=peer.crt --key-file=peer.key --trusted-ca-file=ca.crt --client-cert-auth 205 ``` 206 207 Confirm the client port is serving https: 208 209 ```sh 210 # fails 211 $ ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 endpoint status 212 # works 213 $ ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cert=client.crt --key=client.key --cacert=ca.crt endpoint status 214 ``` 215 216 Next, start a gRPC proxy on `localhost:12379` by connecting to the etcd endpoint `https://localhost:2379` using the client certificates: 217 218 ```sh 219 $ etcd grpc-proxy start --endpoints=https://localhost:2379 --listen-addr localhost:12379 --cert client.crt --key client.key --cacert=ca.crt --insecure-skip-tls-verify & 220 ``` 221 222 Finally, test the TLS termination by putting a key into the proxy over http: 223 224 ```sh 225 $ ETCDCTL_API=3 etcdctl --endpoints=http://localhost:12379 put abc def 226 # OK 227 ``` 228 229 ## Metrics and Health 230 231 The gRPC proxy exposes `/health` and Prometheus `/metrics` endpoints for the etcd members defined by `--endpoints`. An alternative define an additional URL that will respond to both the `/metrics` and `/health` endpoints with the `--metrics-addr` flag. 232 233 ```bash 234 $ etcd grpc-proxy start \ 235 --endpoints https://localhost:2379 \ 236 --metrics-addr https://0.0.0.0:4443 \ 237 --listen-addr 127.0.0.1:23790 \ 238 --key client.key \ 239 --key-file proxy-server.key \ 240 --cert client.crt \ 241 --cert-file proxy-server.crt \ 242 --cacert ca.pem \ 243 --trusted-ca-file proxy-ca.pem 244 ``` 245 246 ### Known issue 247 248 The main interface of the proxy serves both HTTP2 and HTTP/1.1. If proxy is setup with TLS as show in the above example, when using a client such as cURL against the listening interface will require explicitly setting the protocol to HTTP/1.1 on the request to return `/metrics` or `/health`. By using the `--metrics-addr` flag the secondary interface will not have this requirement. 249 250 ```bash 251 $ curl --cacert proxy-ca.pem --key proxy-client.key --cert proxy-client.crt https://127.0.0.1:23790/metrics --http1.1 252 ```