github.com/tiancandevloper/helm@v2.17.0+incompatible/docs/tiller_ssl.md (about) 1 # Using SSL Between Helm and Tiller 2 3 This document explains how to create strong SSL/TLS connections between Helm and 4 Tiller. The emphasis here is on creating an internal CA, and using both the 5 cryptographic and identity functions of SSL. 6 7 > Support for TLS-based auth was introduced in Helm 2.3.0 8 9 Configuring SSL is considered an advanced topic, and knowledge of Helm and Tiller 10 is assumed. 11 12 ## Overview 13 14 The Tiller authentication model uses client-side SSL certificates. Tiller itself 15 verifies these certificates using a certificate authority. Likewise, the client 16 also verifies Tiller's identity by certificate authority. 17 18 There are numerous possible configurations for setting up certificates and authorities, 19 but the method we cover here will work for most situations. 20 21 > As of Helm 2.7.2, Tiller _requires_ that the client certificate be validated 22 > by its CA. In prior versions, Tiller used a weaker validation strategy that 23 > allowed self-signed certificates. 24 25 In this guide, we will show how to: 26 27 - Create a private CA that is used to issue certificates for Tiller clients and 28 servers. 29 - Create a certificate for Tiller 30 - Create a certificate for the Helm client 31 - Create a Tiller instance that uses the certificate 32 - Configure the Helm client to use the CA and client-side certificate 33 34 By the end of this guide, you should have a Tiller instance running that will 35 only accept connections from clients who can be authenticated by SSL certificate. 36 37 ## Generating Certificate Authorities and Certificates 38 39 One way to generate SSL CAs is via the `openssl` command line tool. There are many 40 guides and best practices documents available online. This explanation is focused 41 on getting ready within a small amount of time. For production configurations, 42 we urge readers to read [the official documentation](https://www.openssl.org) and 43 consult other resources. 44 45 There are other alternative ways to generating SSL CAs in addition to `openssl`, for example Terraform. They are not documented here but you can find links to these alternative means in [Related Projects and Documentation](https://helm.sh/docs/related/). 46 47 ### Generate a Certificate Authority 48 49 The simplest way to generate a certificate authority is to run two commands: 50 51 ```console 52 $ openssl genrsa -out ./ca.key.pem 4096 53 $ openssl req -key ca.key.pem -new -x509 -days 7300 -sha256 -out ca.cert.pem -extensions v3_ca 54 Enter pass phrase for ca.key.pem: 55 You are about to be asked to enter information that will be incorporated 56 into your certificate request. 57 What you are about to enter is what is called a Distinguished Name or a DN. 58 There are quite a few fields but you can leave some blank 59 For some fields there will be a default value, 60 If you enter '.', the field will be left blank. 61 ----- 62 Country Name (2 letter code) [AU]:US 63 State or Province Name (full name) [Some-State]:CO 64 Locality Name (eg, city) []:Boulder 65 Organization Name (eg, company) [Internet Widgits Pty Ltd]:tiller 66 Organizational Unit Name (eg, section) []: 67 Common Name (e.g. server FQDN or YOUR name) []:tiller 68 Email Address []:tiller@example.com 69 ``` 70 71 Note that the data input above is _sample data_. You should customize to your own 72 specifications. 73 74 The above will generate both a secret key and a CA. Note that these two files are 75 very important. The key in particular should be handled with particular care. 76 77 Often, you will want to generate an intermediate signing key. For the sake of brevity, 78 we will be signing keys with our root CA. 79 80 ### Generating Certificates 81 82 We will be generating two certificates, each representing a type of certificate: 83 84 - One certificate is for Tiller. You will want one of these _per tiller host_ that 85 you run. 86 - One certificate is for the user. You will want one of these _per helm user_. 87 88 Since the commands to generate these are the same, we'll be creating both at the 89 same time. The names will indicate their target. 90 91 First, the Tiller key: 92 93 ```console 94 $ openssl genrsa -out ./tiller.key.pem 4096 95 Generating RSA private key, 4096 bit long modulus 96 ..........................................................................................................................................................................................................................................................................................................................++ 97 ............................................................................++ 98 e is 65537 (0x10001) 99 Enter pass phrase for ./tiller.key.pem: 100 Verifying - Enter pass phrase for ./tiller.key.pem: 101 ``` 102 103 Next, generate the Helm client's key: 104 105 ```console 106 $ openssl genrsa -out ./helm.key.pem 4096 107 Generating RSA private key, 4096 bit long modulus 108 .....++ 109 ......................................................................................................................................................................................++ 110 e is 65537 (0x10001) 111 Enter pass phrase for ./helm.key.pem: 112 Verifying - Enter pass phrase for ./helm.key.pem: 113 ``` 114 115 Again, for production use you will generate one client certificate for each user. 116 117 Next we need to create certificates from these keys. For each certificate, this is 118 a two-step process of creating a CSR, and then creating the certificate. 119 120 ```console 121 $ openssl req -key tiller.key.pem -new -sha256 -out tiller.csr.pem 122 Enter pass phrase for tiller.key.pem: 123 You are about to be asked to enter information that will be incorporated 124 into your certificate request. 125 What you are about to enter is what is called a Distinguished Name or a DN. 126 There are quite a few fields but you can leave some blank 127 For some fields there will be a default value, 128 If you enter '.', the field will be left blank. 129 ----- 130 Country Name (2 letter code) [AU]:US 131 State or Province Name (full name) [Some-State]:CO 132 Locality Name (eg, city) []:Boulder 133 Organization Name (eg, company) [Internet Widgits Pty Ltd]:Tiller Server 134 Organizational Unit Name (eg, section) []: 135 Common Name (e.g. server FQDN or YOUR name) []:tiller-server 136 Email Address []: 137 138 Please enter the following 'extra' attributes 139 to be sent with your certificate request 140 A challenge password []: 141 An optional company name []: 142 ``` 143 144 And we repeat this step for the Helm client certificate: 145 146 ```console 147 $ openssl req -key helm.key.pem -new -sha256 -out helm.csr.pem 148 # Answer the questions with your client user's info 149 ``` 150 151 (In rare cases, we've had to add the `-nodes` flag when generating the request.) 152 153 Now we sign each of these CSRs with the CA certificate we created (adjust the days parameter to suit your requirements): 154 155 ```console 156 $ openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in tiller.csr.pem -out tiller.cert.pem -days 365 157 Signature ok 158 subject=/C=US/ST=CO/L=Boulder/O=Tiller Server/CN=tiller-server 159 Getting CA Private Key 160 Enter pass phrase for ca.key.pem: 161 ``` 162 163 And again for the client certificate: 164 165 ```console 166 $ openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in helm.csr.pem -out helm.cert.pem -days 365 167 ``` 168 169 At this point, the important files for us are these: 170 171 ``` 172 # The CA. Make sure the key is kept secret. 173 ca.cert.pem 174 ca.key.pem 175 # The Helm client files 176 helm.cert.pem 177 helm.key.pem 178 # The Tiller server files. 179 tiller.cert.pem 180 tiller.key.pem 181 ``` 182 183 Now we're ready to move on to the next steps. 184 185 ## Creating a Custom Tiller Installation 186 187 Helm includes full support for creating a deployment configured for SSL. By specifying 188 a few flags, the `helm init` command can create a new Tiller installation complete 189 with all of our SSL configuration. 190 191 To take a look at what this will generate, run this command: 192 193 ```console 194 $ helm init --dry-run --debug --tiller-tls --tiller-tls-cert ./tiller.cert.pem --tiller-tls-key ./tiller.key.pem --tiller-tls-verify --tls-ca-cert ca.cert.pem 195 ``` 196 197 The output will show you a Deployment, a Secret, and a Service. Your SSL information 198 will be preloaded into the Secret, which the Deployment will mount to pods as they 199 start up. 200 201 If you want to customize the manifest, you can save that output to a file and then 202 use `kubectl create` to load it into your cluster. 203 204 > We strongly recommend enabling RBAC on your cluster and adding [service accounts](rbac.md) 205 > with RBAC. 206 207 Otherwise, you can remove the `--dry-run` and `--debug` flags. We also recommend 208 putting Tiller in a non-system namespace (`--tiller-namespace=something`) and enable 209 a service account (`--service-account=somename`). But for this example we will stay 210 with the basics: 211 212 ```console 213 $ helm init --tiller-tls --tiller-tls-cert ./tiller.cert.pem --tiller-tls-key ./tiller.key.pem --tiller-tls-verify --tls-ca-cert ca.cert.pem 214 ``` 215 216 In a minute or two it should be ready. We can check Tiller like this: 217 218 ```console 219 $ kubectl -n kube-system get deployment 220 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 221 ... other stuff 222 tiller-deploy 1 1 1 1 2m 223 ``` 224 225 If there is a problem, you may want to use `kubectl get pods -n kube-system` to 226 find out what went wrong. With the SSL/TLS support, the most common problems all 227 have to do with improperly generated TLS certificates or accidentally swapping the 228 cert and the key. 229 230 At this point, you should get a _failure_ when you run basic Helm commands: 231 232 ```console 233 $ helm ls 234 Error: transport is closing 235 ``` 236 237 This is because your Helm client does not have the correct certificate to authenticate 238 to Tiller. 239 240 ## Configuring the Helm Client 241 242 The Tiller server is now running with TLS protection. It's time to configure the 243 Helm client to also perform TLS operations. 244 245 For a quick test, we can specify our configuration manually. We'll run a normal 246 Helm command (`helm ls`), but with SSL/TLS enabled. 247 248 ```console 249 helm ls --tls --tls-ca-cert ca.cert.pem --tls-cert helm.cert.pem --tls-key helm.key.pem 250 ``` 251 252 This configuration sends our client-side certificate to establish identity, uses 253 the client key for encryption, and uses the CA certificate to validate the remote 254 Tiller's identity. 255 256 Typing a line that is cumbersome, though. The shortcut is to move the key, 257 cert, and CA into `$HELM_HOME`: 258 259 ```console 260 $ cp ca.cert.pem $(helm home)/ca.pem 261 $ cp helm.cert.pem $(helm home)/cert.pem 262 $ cp helm.key.pem $(helm home)/key.pem 263 ``` 264 265 With this, you can simply run `helm ls --tls` to enable TLS. 266 267 ### Troubleshooting 268 269 *Running a command, I get `Error: transport is closing`* 270 271 This is almost always due to a configuration error in which the client is missing 272 a certificate (`--tls-cert`) or the certificate is bad. 273 274 *I'm using a certificate, but get `Error: remote error: tls: bad certificate`* 275 276 This means that Tiller's CA cannot verify your certificate. In the examples above, 277 we used a single CA to generate both the client and server certificates. In these 278 examples, the CA has _signed_ the client's certificate. We then load that CA 279 up to Tiller. So when the client certificate is sent to the server, Tiller 280 checks the client certificate against the CA. 281 282 *If I use `--tls-verify` on the client, I get `Error: x509: certificate is valid for tiller-server, not localhost`* 283 284 If you plan to use `--tls-verify` on the client, you will need to make sure that 285 the host name that Helm connects to matches the host name on the certificate. In 286 some cases this is awkward, since Helm will connect over localhost, or the FQDN is 287 not available for public resolution. 288 289 *If I use `--tls-verify` on the client, I get `Error: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs`* 290 291 By default, the Helm client connects to Tiller via tunnel (i.e. kube proxy) at 127.0.0.1. During the TLS handshake, 292 a target, usually provided as a hostname (e.g. example.com), is checked against the subject and subject alternative 293 names of the certificate (i.e. hostname verification). However, because of the tunnel, the target is an IP address. 294 Therefore, to validate the certificate, the IP address 127.0.0.1 must be listed as an IP subject alternative name 295 (IP SAN) in the Tiller certificate. 296 297 For example, to list 127.0.0.1 as an IP SAN when generating the Tiller certificate: 298 299 ```console 300 $ echo subjectAltName=IP:127.0.0.1 > extfile.cnf 301 $ openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in tiller.csr.pem -out tiller.cert.pem -days 365 -extfile extfile.cnf 302 ``` 303 304 Alternatively, you can override the expected hostname of the tiller certificate using the `--tls-hostname` flag. 305 306 *If I use `--tls-verify` on the client, I get `Error: x509: certificate has expired or is not yet valid`* 307 308 Your helm certificate has expired, you need to sign a new certificate using your private key and the CA (and consider increasing the number of days) 309 310 If your tiller certificate has expired, you'll need to sign a new certificate, base64 encode it and update the Tiller Secret: 311 `kubectl edit secret tiller-secret` 312 313 ## References 314 315 - https://github.com/denji/golang-tls 316 - https://www.openssl.org/docs/ 317 - https://jamielinux.com/docs/openssl-certificate-authority/sign-server-and-client-certificates.html