github.com/kubernetes-incubator/kube-aws@v0.16.4/docs/getting-started/step-2-render.md

# Configure your Kubernetes cluster on AWS

This is the second step of [running Kubernetes on AWS](README.md). Before we launch our cluster, let's define a few parameters that the cluster requires.

## Cluster parameters

### EC2 key pair

The keypair that will authenticate SSH access to your EC2 instances. The public half of this key pair will be configured on each Flatcar node.

After creating a key pair, you will use the name you gave the keys to configure the cluster. Key pairs are only available to EC2 instances in the same region. More info in the [EC2 Keypair docs](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html).

### KMS key

[Amazon KMS](http://docs.aws.amazon.com/kms/latest/developerguide/overview.html) keys are used to encrypt and decrypt cluster TLS assets. If you already have a KMS Key that you would like to use, you can skip creating a new key and provide the Arn string for your existing key.

You can create a KMS key in the [AWS console](http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html), or with the `aws` command line tool:

```sh
$ aws kms --region=<your-region> create-key --description="kube-aws assets"
{
    "KeyMetadata": {
        "CreationDate": 1458235139.724,
        "KeyState": "Enabled",
        "Arn": "arn:aws:kms:us-west-1:xxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx",
        "AWSAccountId": "xxxxxxxxxxxxx",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyId": "xxxxxxxxx",
        "Description": "kube-aws assets"
    }
}
```

You will use the `KeyMetadata.Arn` string to identify your KMS key in the init step.

### External DNS name

Select a DNS hostname where the cluster API will be accessible. Typically this hostname is available over the internet ("external"), so end users can connect from different networks. This hostname will be used to provision the TLS certificate for the API server, which encrypts traffic between your users and the API. Optionally, you can provide the certificates yourself, which is recommended for production clusters.

When the cluster is created, the cluster will expose the TLS-secured API on an internet-facing ELB. kube-aws can automatically create an ALIAS record for the ELB in an *existing* [Route 53][route53] hosted zone specified via the `--hosted-zone-id` flag. If you have a DNS zone hosted in Route 53, you can configure for it below.

You can also omit `--hosted-zone-id` by specifying the `--no-record-set` flag. However, then, you will need to create an Route53 ALIAS record or a CNAME record for the external DNS hostname you want to point to this ELB. You can find the public DNS name of the ELB after the cluster is created by invoking `kube-aws status`.

### S3 bucket

[Amazon S3](http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) buckets are used to transfer large stack templates generated by kube-aws to CloudFormation. If you already have a S3 bucket that you would like to use, you can skip creating a new bucket and provide the URI for your existing bucket.

You can create a S3 bucket in the [AWS console](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#create-bucket-intro), or with the `aws` command line tool.

The command varies among AWS regions.

For the us-east-1 region:

```sh
$ aws s3api --region=us-east-1 create-bucket --bucket <your-bucket-name>
{
    "Location": "/<your-bucket-name>"
}
```

For other regions:

```sh
$ aws s3api create-bucket --bucket my-bucket --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1
```

## Initialize an asset directory

Create a directory on your local machine to hold the generated assets:

```sh
$ mkdir my-cluster
$ cd my-cluster
```

Initialize the cluster CloudFormation stack with the KMS Arn, key pair name, and DNS name from the previous step:

```sh
$ kube-aws init \
--cluster-name=my-cluster-name \
--external-dns-name=my-cluster-endpoint \
--hosted-zone-id=hosted-zone-xxxxx \
--region=us-west-1 \
--availability-zone=us-west-1c \
--key-name=key-pair-name \
--kms-key-arn="arn:aws:kms:us-west-1:xxxxxxxxxx:key/xxxxxxxxxxxxxxxxxxx" \
--s3-uri=s3://my-kube-aws-assets-bucket
```

Here `us-west-1c` is used for parameter `--availability-zone`, but supported availability zone varies among AWS accounts.
Please check if `us-west-1c` is supported by `aws ec2 --region us-west-1 describe-availability-zones`, if not switch to other supported availability zone. (e.g., `us-west-1a`, or `us-west-1b`)

There will now be a `cluster.yaml` file in the asset directory. This is the main configuration file for your cluster.

### Render contents of the asset directory

#### TLS certificates

* In the simplest case, you can have kube-aws generate both your TLS identities and certificate authority for you.

  ```sh
  $ kube-aws render credentials --generate-ca
  ```

  This is not recommended for production, but is fine for development or testing purposes.

* It is recommended that you supply your own immediate certificate signing authority and let kube-aws take care of generating the cluster TLS credentials.

  ```sh
  $ kube-aws render credentials --ca-cert-path=/path/to/ca-cert.pem --ca-key-path=/path/to/ca-key.pem
  ```

  If the CA key is encrypted (which it should), you will be prompted for the key passphrase. Although not recommended, `KUBE_AWS_CA_KEY_PASSPHRASE` environment variable can be set to automate this process.

  For more information on operating your own CA, check out this [awesome guide](https://jamielinux.com/docs/openssl-certificate-authority/).

* In certain cases, such as users with advanced pre-existing PKI infrastructure, the operator may wish to pre-generate all cluster TLS assets. In this case, you can run `kube-aws render stack` and copy in your TLS assets into the `credentials/` folder before running `kube-aws apply`.

  ```sh

  ls -R credentials/
  admin-key.pem                   encryption-config.yaml          kiam-ca.pem                     service-account-key.pem
  admin.pem                       etcd-client-key.pem             kiam-server-key.pem             tokens.csv
  apiserver-aggregator-key.pem    etcd-client.pem                 kiam-server.pem                 worker-ca-key.pem
  apiserver-aggregator.pem        etcd-key.pem                    kube-controller-manager-key.pem worker-ca.pem
  apiserver-key.pem               etcd-trusted-ca.pem             kube-controller-manager.pem     worker-key.pem
  apiserver.pem                   etcd.pem                        kube-scheduler-key.pem          worker.pem
  ca-key.pem                      kiam-agent-key.pem              kube-scheduler.pem
  ca.pem                          kiam-agent.pem                  kubelet-tls-bootstrap-token
  ```

### Render cluster assets

The next command generates the default set of cluster assets in your asset directory.

  ```sh
  $ kube-aws render stack
  ```

Here's what the directory structure looks like:

```sh
$ tree
.
├── cluster.yaml
├── credentials
│   ├── admin-key.pem
│   ├── admin.pem
│   ├── apiserver-aggregator-key.pem
│   ├── apiserver-aggregator.pem
│   ├── apiserver-key.pem
│   ├── apiserver.pem
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── encryption-config.yaml
│   ├── etcd-client-key.pem
│   ├── etcd-client.pem
│   ├── etcd-key.pem
│   ├── etcd-trusted-ca.pem -> ca.pem
│   ├── etcd.pem
│   ├── kiam-agent-key.pem
│   ├── kiam-agent.pem
│   ├── kiam-ca.pem
│   ├── kiam-server-key.pem
│   ├── kiam-server.pem
│   ├── kube-controller-manager-key.pem
│   ├── kube-controller-manager.pem
│   ├── kube-scheduler-key.pem
│   ├── kube-scheduler.pem
│   ├── kubelet-tls-bootstrap-token
│   ├── service-account-key.pem
│   ├── tokens.csv
│   ├── worker-ca-key.pem -> ca-key.pem
│   ├── worker-ca.pem -> ca.pem
│   ├── worker-key.pem
│   └── worker.pem
├── kubeconfig
├── plugins
│   └── aws-iam-authenticator
│       ├── files
│       │   ├── authentication-token-webhook-config.yaml
│       │   ├── controller-kubeconfig.yaml
│       │   └── worker-kubeconfig.yaml
│       ├── manifests
│       │   ├── aws-auth-cm.yaml
│       │   └── daemonset.yaml
│       └── plugin.yaml
├── stack-templates
│   ├── control-plane.json.tmpl
│   ├── etcd.json.tmpl
│   ├── network.json.tmpl
│   ├── node-pool.json.tmpl
│   └── root.json.tmpl
└── userdata
    ├── cloud-config-controller
    ├── cloud-config-etcd
    └── cloud-config-worker
```

These assets (templates and credentials) are used to create, update and interact with your Kubernetes cluster.

At this point you should be ready to create your cluster. You can also now check the `my-cluster` asset directory into version control if you desire. The contents of this directory are your reproducible cluster assets. Please take care not to commit the `my-cluster/credentials` directory but rather to encrypt and/or put it to more secure storage, as it contains your TLS secrets and access tokens. If you're using git, the `credentials` directory will already be ignored for you.

**PRODUCTION NOTE**: the TLS keys and certificates generated by `kube-aws` should *not* be used to deploy a production Kubernetes cluster.
Each component certificate is only valid for 90 days, while the CA is valid for 365 days.
If deploying a production Kubernetes cluster, consider establishing PKI independently of this tool first. [Read more below.][tls-note]

**Did everything render correctly?**
If you are familiar with Flatcar and the AWS platform, you may want to include some additional customizations or optional features. Read on below to explore more.

[Yes, ready to launch the cluster][getting-started-step-3]

[View optional features &amp; customizations below](#customizations-to-your-cluster)

## Customizations to your cluster

You can now customize your cluster by editing asset files. Any changes to these files will require a `render` and `validate` workflow, covered below.

### Customize infrastructure

* **cluster.yaml**

  This is the configuration file for your cluster. It contains the configuration parameters that are templated into your userdata and CloudFormation stack.

  Some common customizations are:

  - change the number of workers
  - specify tags applied to all created resources
  - create cluster inside an existing VPC
  - change controller and worker volume sizes
  <br/><br/>

* **userdata/**

  * `cloud-config-worker`
  * `cloud-config-controller`

  This directory contains the [cloud-init](https://github.com/coreos/coreos-cloudinit) cloud-config userdata files. The Flatcar operating system supports automated provisioning via cloud-config files, which describe the various files, scripts and systemd actions necessary to produce a working cluster machine. These files are templated with your cluster configuration parameters and embedded into the CloudFormation stack template.

  Some common customizations are:

  - [mounting ephemeral disks][mount-disks]
  - [allow pods to mount RDB][rdb] or [iSCSI volumes][iscsi]
  - [allowing access to insecure container registries][insecure-registry]
  - [use host DNS configuration instead of a public DNS server][host-dns]
  - [changing your Flatcar auto-update settings][update]
  <br/><br/>

* **stack-template.json**

  This file describes the [AWS CloudFormation](https://aws.amazon.com/cloudformation/) stack which encompasses all the AWS resources associated with your cluster. This JSON document is templated with configuration parameters, we well as the encoded userdata files.

  Some common customizations are:

  - tweak AutoScaling rules and timing
  - instance IAM roles
  - customize security groups beyond the initial configuration
  <br/><br/>

* **credentials/**

  This directory contains both encryped and **unencrypted** TLS assets for your cluster, along with a pre-configured `kubeconfig` file which provides access to your cluster api via kubectl.

  You can also specify additional access tokens in `tokens.csv` as shown in the [official docs](https://kubernetes.io/docs/admin/authentication/#static-token-file).

[mount-disks]: https://coreos.com/os/docs/latest/mounting-storage.html
[insecure-registry]: https://coreos.com/os/docs/latest/registry-authentication.html#using-a-registry-without-ssl-configured
[update]: https://coreos.com/os/docs/latest/cloud-config.html#update

### Kubernetes Container Runtime

The kube-aws tool now optionally supports using rkt as the kubernetes container runtime. To configure rkt as the container runtime you must run with a Flatcar version >= `v1151.0.0` and configure the runtime flag.

Edit the `cluster.yaml` file:

```yaml
containerRuntime: rkt
releaseChannel: stable
```

Note that while using rkt as the runtime is now supported, it is still a new option as of the Kubernetes v1.4 release and has a few [known issues](http://kubernetes.io/docs/getting-started-guides/rkt/notes/).

### Calico network policy

The cluster can be optionally configured to use Calico to provide [network policy](http://kubernetes.io/docs/user-guide/networkpolicies/). These policies limit and control how different pods, namespaces, etc can communicate with each other. These rules can be managed after the cluster is launched, but the feature needs to be turned on beforehand.

Edit the `cluster.yaml` file:

```yaml
useCalico: true
```

### Route53 Host Record

`kube-aws` can create an ALIAS record for the controller's ELB in an existing Route53 hosted zone.

Just run `kube-aws init` with the flag `--hosted-zone-id` to specify the id of the hosted zone in which the record is created.

If you've run `kube-aws init` without the flag but with `--no-record-set`, edit the `cluster.yaml` file to add `loadBalancer.hostedZone.id` under the first item of `apiEndpoints` while setting `recordSetManaged` to `true` or removing it:

```yaml
apiEndpoints:
- name: default
  dNSName: kubernetes.staging.example.com
  loadBalancer:
    hostedZone:
      id: A12B3CDE4FG5HI

# DEPRECATED: use hostedZoneId instead
#hostedZone: staging.example.com

# DEPRECATED: use loadBalancer.hostedZone.id instead
#hostedZoneId: A12B3CDE4FG5HI

# DEPRECATED: use loadBalancer.createRecordSet instead
# This is even implied to be true when loadBalancer.hostedZone.id is specified
#createRecordSet: true

```

If `createRecordSet` is not set to true, the deployer will be responsible for making externalDNSName routable to the the ELB managing the controller nodes after the cluster is created.

### Multi-AZ Clusters

Kube-aws supports "spreading" a cluster across any number of Availability Zones in a given region.

__A word of caution about EBS and Persistent Volumes__: Any pods deployed to a Multi-AZ cluster must mount EBS volumes via [Persistent Volume Claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims). Specifying the ID of the EBS volume directly in the pod spec will not work consistently if nodes are spread across multiple zones.

Read more about Kubernetes Multi-AZ cluster support [here](https://kubernetes.io/docs/setup/best-practices/multiple-zones/).

#### A common pitfall when deploying multi-AZ clusters in combination with cluster-autoscaler

[cluster-autoscaler](https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler) is a tool that automatically adjusts the number of Kubernetes worker nodes when:

> * there is a pod that doesn’t have enough space to run in the cluster
> * some nodes in the cluster are so underutilized, for an extended period of time, that they can be deleted and their pods will be easily placed on some other, existing nodes.
> https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler

A common pitfall in deploying cluster-autoscaler to a multi-AZ cluster is that you have to instruct an Auto Scaling Group not to spread over multiple availability zones or cluster-autoscaler results in instability while scaling out the nodes i.e. it takes unnecessary much time to finally bring up a node in the insufficient zone.

> The autoscaling group should span 1 availability zone for the cluster autoscaler to work. If you want to distribute workloads evenly across zones, set up multiple ASGs, with a cluster autoscaler for each ASG. At the time of writing this, cluster autoscaler is unaware of availability zones and although autoscaling groups can contain instances in multiple availability zones when configured so, the cluster autoscaler can't reliably add nodes to desired zones. That's because AWS AutoScaling determines which zone to add nodes which is out of the control of the cluster autoscaler. For more information, see https://github.com/kubernetes/contrib/pull/1552#discussion_r75533090.
> https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#deployment-specification

Please read the following guides carefully and select the appropriate deployment according to your requirement regarding auto-scaling.

#### For production cluster not requiring cluster-autoscaler

If you don't need auto-scaling at all, or you need only the AWS-native auto-scaling i.e. the a combination of auto scaling groups, scaling policies, CloudWatch alarms,
theoretically, you can safely go this way.

Edit the `cluster.yaml` file to define multiple subnets, each with a different availability zone hence multi-AZ:

```yaml
 subnets:
   - availabilityZone: us-west-1a
     instanceCIDR: "10.0.0.0/24"
   - availabilityZone: us-west-1b
     instanceCIDR: "10.0.1.0/24"
```

This implies that you rely on AWS AutoScaling for selecting which subnet hence which availability zone to add a node when an auto-scaling group's `DesiredCapacity` is increased.

Please read [the AWS documentation for more details about AWS Auto Scaling](http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html).

### Certificates and Keys

`kube-aws render` begins by initializing the TLS infrastructure needed to securely operate Kubernetes. If you have your own key/certificate management system, you can overwrite the generated TLS assets after `kube-aws render`. More information on [Kubernetes certificate generation.][k8s-openssl]

When `kube-aws apply` creates the cluster stack, it will use whatever TLS assets it finds in the `credentials` folder at the time.

This includes the certificate authority, signed server certificates for the Kubernetes API server and workers, and a signed client certificate for administrative use.

* **APIServerCert, APIServerKey**

  The API server certificate will be valid for the value of `externalDNSName`, as well as a the DNS names used to route Kubernetes API requests inside the cluster.

  `kube-aws` does *not* manage a DNS zone for the cluster.
  This means that the deployer is responsible for ensuring the routability of the external DNS name to the public IP of the master node instance.

  The certificate and key granted to the kube-apiserver.
  This certificate will be presented to external clients of the Kubernetes cluster, so it should be valid for external DNS names, if necessary.

  Additionally, the certificate must have the following Subject Alternative Names (SANs).
  These IPs and DNS names are used within the cluster to route from applications to the Kubernetes API:

    - 127.0.0.1
    - 10.0.0.50
    - 10.3.0.1
    - kubernetes
    - kubernetes.default
    - kubernetes.default.svc
    - kubernetes.default.svc.cluster.local


* **WorkerCert, WorkerKey**

  The certificate and key granted to the kubelets on worker instances.
  The certificate is shared across all workers, so it must be valid for all worker hostnames.
  This is achievable with the Subject Alternative Name (SAN) `*.*.compute.internal`, or `*.ec2.internal` if using the us-east-1 AWS region.

* **CACert**

  The certificate authority's TLS certificate is used to sign other certificates in the cluster.

  These assets are stored unencrypted in your `credentials` folder, but are encyrpted using Amazon KMS before being embedded in the CloudFormation template.

  All keys and certs must be PEM-formatted and base64-encoded.

## Render and validate cluster assets

After you have completed your customizations, re-render your assets with the new settings:

```sh
$ kube-aws render credentials
$ kube-aws render stack
```

The `validate` command check the validity of your changes to the cloud-config userdata files and the CloudFormation stack description.

This is an important step to make sure your stack will launch successfully:

```sh
$ kube-aws validate
```

If your files are valid, you are ready to [launch your cluster][getting-started-step-3].

[getting-started-step-1]: step-1-configure.md
[getting-started-step-2]: step-2-render.md
[getting-started-step-3]: step-3-launch.md
[getting-started-step-4]: step-4-update.md
[getting-started-step-5]: step-5-add-node-pool.md
[getting-started-step-6]: step-6-configure-add-ons.md
[getting-started-step-7]: step-7-destroy.md
[k8s-openssl]: openssl.md
[tls-note]: #certificates-and-keys
[route53]: https://aws.amazon.com/route53/
[rdb]: https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubelet-wrapper.md#allow-pods-to-use-rbd-volumes
[iscsi]: https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubelet-wrapper.md#allow-pods-to-use-iscsi-mounts
[host-dns]: https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubelet-wrapper.md#use-the-hosts-dns-configuration
[node-pool]: kubernetes-on-aws-node-pool.md