github.com/enmand/kubernetes@v1.2.0-alpha.0/docs/getting-started-guides/scratch.md (about) 1 <!-- BEGIN MUNGE: UNVERSIONED_WARNING --> 2 3 <!-- BEGIN STRIP_FOR_RELEASE --> 4 5 <img src="http://kubernetes.io/img/warning.png" alt="WARNING" 6 width="25" height="25"> 7 <img src="http://kubernetes.io/img/warning.png" alt="WARNING" 8 width="25" height="25"> 9 <img src="http://kubernetes.io/img/warning.png" alt="WARNING" 10 width="25" height="25"> 11 <img src="http://kubernetes.io/img/warning.png" alt="WARNING" 12 width="25" height="25"> 13 <img src="http://kubernetes.io/img/warning.png" alt="WARNING" 14 width="25" height="25"> 15 16 <h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2> 17 18 If you are using a released version of Kubernetes, you should 19 refer to the docs that go with that version. 20 21 <strong> 22 The latest 1.0.x release of this document can be found 23 [here](http://releases.k8s.io/release-1.0/docs/getting-started-guides/scratch.md). 24 25 Documentation for other releases can be found at 26 [releases.k8s.io](http://releases.k8s.io). 27 </strong> 28 -- 29 30 <!-- END STRIP_FOR_RELEASE --> 31 32 <!-- END MUNGE: UNVERSIONED_WARNING --> 33 Getting started from Scratch 34 ---------------------------- 35 36 This guide is for people who want to craft a custom Kubernetes cluster. If you 37 can find an existing Getting Started Guide that meets your needs on [this 38 list](README.md), then we recommend using it, as you will be able to benefit 39 from the experience of others. However, if you have specific IaaS, networking, 40 configuration management, or operating system requirements not met by any of 41 those guides, then this guide will provide an outline of the steps you need to 42 take. Note that it requires considerably more effort than using one of the 43 pre-defined guides. 44 45 This guide is also useful for those wanting to understand at a high level some of the 46 steps that existing cluster setup scripts are making. 47 48 **Table of Contents** 49 50 <!-- BEGIN MUNGE: GENERATED_TOC --> 51 52 - [Designing and Preparing](#designing-and-preparing) 53 - [Learning](#learning) 54 - [Cloud Provider](#cloud-provider) 55 - [Nodes](#nodes) 56 - [Network](#network) 57 - [Cluster Naming](#cluster-naming) 58 - [Software Binaries](#software-binaries) 59 - [Downloading and Extracting Kubernetes Binaries](#downloading-and-extracting-kubernetes-binaries) 60 - [Selecting Images](#selecting-images) 61 - [Security Models](#security-models) 62 - [Preparing Certs](#preparing-certs) 63 - [Preparing Credentials](#preparing-credentials) 64 - [Configuring and Installing Base Software on Nodes](#configuring-and-installing-base-software-on-nodes) 65 - [Docker](#docker) 66 - [rkt](#rkt) 67 - [kubelet](#kubelet) 68 - [kube-proxy](#kube-proxy) 69 - [Networking](#networking) 70 - [Other](#other) 71 - [Using Configuration Management](#using-configuration-management) 72 - [Bootstrapping the Cluster](#bootstrapping-the-cluster) 73 - [etcd](#etcd) 74 - [Apiserver, Controller Manager, and Scheduler](#apiserver-controller-manager-and-scheduler) 75 - [Apiserver pod template](#apiserver-pod-template) 76 - [Cloud Providers](#cloud-providers) 77 - [Scheduler pod template](#scheduler-pod-template) 78 - [Controller Manager Template](#controller-manager-template) 79 - [Starting and Verifying Apiserver, Scheduler, and Controller Manager](#starting-and-verifying-apiserver-scheduler-and-controller-manager) 80 - [Logging](#logging) 81 - [Monitoring](#monitoring) 82 - [DNS](#dns) 83 - [Troubleshooting](#troubleshooting) 84 - [Running validate-cluster](#running-validate-cluster) 85 - [Inspect pods and services](#inspect-pods-and-services) 86 - [Try Examples](#try-examples) 87 - [Running the Conformance Test](#running-the-conformance-test) 88 - [Networking](#networking) 89 - [Getting Help](#getting-help) 90 91 <!-- END MUNGE: GENERATED_TOC --> 92 93 ## Designing and Preparing 94 95 ### Learning 96 97 1. You should be familiar with using Kubernetes already. We suggest you set 98 up a temporary cluster by following one of the other Getting Started Guides. 99 This will help you become familiar with the CLI ([kubectl](../user-guide/kubectl/kubectl.md)) and concepts ([pods](../user-guide/pods.md), [services](../user-guide/services.md), etc.) first. 100 1. You should have `kubectl` installed on your desktop. This will happen as a side 101 effect of completing one of the other Getting Started Guides. If not, follow the instructions 102 [here](../user-guide/prereqs.md). 103 104 ### Cloud Provider 105 106 Kubernetes has the concept of a Cloud Provider, which is a module which provides 107 an interface for managing TCP Load Balancers, Nodes (Instances) and Networking Routes. 108 The interface is defined in `pkg/cloudprovider/cloud.go`. It is possible to 109 create a custom cluster without implementing a cloud provider (for example if using 110 bare-metal), and not all parts of the interface need to be implemented, depending 111 on how flags are set on various components. 112 113 ### Nodes 114 115 - You can use virtual or physical machines. 116 - While you can build a cluster with 1 machine, in order to run all the examples and tests you 117 need at least 4 nodes. 118 - Many Getting-started-guides make a distinction between the master node and regular nodes. This 119 is not strictly necessary. 120 - Nodes will need to run some version of Linux with the x86_64 architecture. It may be possible 121 to run on other OSes and Architectures, but this guide does not try to assist with that. 122 - Apiserver and etcd together are fine on a machine with 1 core and 1GB RAM for clusters with 10s of nodes. 123 Larger or more active clusters may benefit from more cores. 124 - Other nodes can have any reasonable amount of memory and any number of cores. They need not 125 have identical configurations. 126 127 ### Network 128 129 Kubernetes has a distinctive [networking model](../admin/networking.md). 130 131 Kubernetes allocates an IP address to each pod. When creating a cluster, you 132 need to allocate a block of IPs for Kubernetes to use as Pod IPs. The simplest 133 approach is to allocate a different block of IPs to each node in the cluster as 134 the node is added. A process in one pod should be able to communicate with 135 another pod using the IP of the second pod. This connectivity can be 136 accomplished in two ways: 137 - Configure network to route Pod IPs 138 - Harder to setup from scratch. 139 - Google Compute Engine ([GCE](gce.md)) and [AWS](aws.md) guides use this approach. 140 - Need to make the Pod IPs routable by programming routers, switches, etc. 141 - Can be configured external to Kubernetes, or can implement in the "Routes" interface of a Cloud Provider module. 142 - Generally highest performance. 143 - Create an Overlay network 144 - Easier to setup 145 - Traffic is encapsulated, so per-pod IPs are routable. 146 - Examples: 147 - [Flannel](https://github.com/coreos/flannel) 148 - [Weave](http://weave.works/) 149 - [Open vSwitch (OVS)](http://openvswitch.org/) 150 - Does not require "Routes" portion of Cloud Provider module. 151 - Reduced performance (exactly how much depends on your solution). 152 153 You need to select an address range for the Pod IPs. 154 - Various approaches: 155 - GCE: each project has its own `10.0.0.0/8`. Carve off a `/16` for each 156 Kubernetes cluster from that space, which leaves room for several clusters. 157 Each node gets a further subdivision of this space. 158 - AWS: use one VPC for whole organization, carve off a chunk for each 159 cluster, or use different VPC for different clusters. 160 - IPv6 is not supported yet. 161 - Allocate one CIDR subnet for each node's PodIPs, or a single large CIDR 162 from which smaller CIDRs are automatically allocated to each node (if nodes 163 are dynamically added). 164 - You need max-pods-per-node * max-number-of-nodes IPs in total. A `/24` per 165 node supports 254 pods per machine and is a common choice. If IPs are 166 scarce, a `/26` (62 pods per machine) or even a `/27` (30 pods) may be sufficient. 167 - e.g. use `10.10.0.0/16` as the range for the cluster, with up to 256 nodes 168 using `10.10.0.0/24` through `10.10.255.0/24`, respectively. 169 - Need to make these routable or connect with overlay. 170 171 Kubernetes also allocates an IP to each [service](../user-guide/services.md). However, 172 service IPs do not necessarily need to be routable. The kube-proxy takes care 173 of translating Service IPs to Pod IPs before traffic leaves the node. You do 174 need to Allocate a block of IPs for services. Call this 175 `SERVICE_CLUSTER_IP_RANGE`. For example, you could set 176 `SERVICE_CLUSTER_IP_RANGE="10.0.0.0/16"`, allowing 65534 distinct services to 177 be active at once. Note that you can grow the end of this range, but you 178 cannot move it without disrupting the services and pods that already use it. 179 180 Also, you need to pick a static IP for master node. 181 - Call this `MASTER_IP`. 182 - Open any firewalls to allow access to the apiserver ports 80 and/or 443. 183 - Enable ipv4 forwarding sysctl, `net.ipv4.ip_forward = 1` 184 185 ### Cluster Naming 186 187 You should pick a name for your cluster. Pick a short name for each cluster 188 which is unique from future cluster names. This will be used in several ways: 189 - by kubectl to distinguish between various clusters you have access to. You will probably want a 190 second one sometime later, such as for testing new Kubernetes releases, running in a different 191 region of the world, etc. 192 - Kubernetes clusters can create cloud provider resources (e.g. AWS ELBs) and different clusters 193 need to distinguish which resources each created. Call this `CLUSTERNAME`. 194 195 ### Software Binaries 196 197 You will need binaries for: 198 - etcd 199 - A container runner, one of: 200 - docker 201 - rkt 202 - Kubernetes 203 - kubelet 204 - kube-proxy 205 - kube-apiserver 206 - kube-controller-manager 207 - kube-scheduler 208 209 #### Downloading and Extracting Kubernetes Binaries 210 211 A Kubernetes binary release includes all the Kubernetes binaries as well as the supported release of etcd. 212 You can use a Kubernetes binary release (recommended) or build your Kubernetes binaries following the instructions in the 213 [Developer Documentation](../devel/README.md). Only using a binary release is covered in this guide. 214 215 Download the [latest binary release](https://github.com/kubernetes/kubernetes/releases/latest) and unzip it. 216 Then locate `./kubernetes/server/kubernetes-server-linux-amd64.tar.gz` and unzip *that*. 217 Then, within the second set of unzipped files, locate `./kubernetes/server/bin`, which contains 218 all the necessary binaries. 219 220 #### Selecting Images 221 222 You will run docker, kubelet, and kube-proxy outside of a container, the same way you would run any system daemon, so 223 you just need the bare binaries. For etcd, kube-apiserver, kube-controller-manager, and kube-scheduler, 224 we recommend that you run these as containers, so you need an image to be built. 225 226 You have several choices for Kubernetes images: 227 - Use images hosted on Google Container Registry (GCR): 228 - e.g `gcr.io/google_containers/hyperkube:$TAG`, where `TAG` is the latest 229 release tag, which can be found on the [latest releases page](https://github.com/kubernetes/kubernetes/releases/latest). 230 - Ensure $TAG is the same tag as the release tag you are using for kubelet and kube-proxy. 231 - The [hyperkube](../../cmd/hyperkube/) binary is an all in one binary 232 - `hyperkube kubelet ...` runs the kublet, `hyperkube apiserver ...` runs an apiserver, etc. 233 - Build your own images. 234 - Useful if you are using a private registry. 235 - The release contains files such as `./kubernetes/server/bin/kube-apiserver.tar` which 236 can be converted into docker images using a command like 237 `docker load -i kube-apiserver.tar` 238 - You can verify if the image is loaded successfully with the right repository and tag using 239 command like `docker images` 240 241 For etcd, you can: 242 - Use images hosted on Google Container Registry (GCR), such as `gcr.io/google_containers/etcd:2.0.12` 243 - Use images hosted on [Docker Hub](https://hub.docker.com/search/?q=etcd) or [Quay.io](https://quay.io/repository/coreos/etcd), such as `quay.io/coreos/etcd:v2.2.0` 244 - Use etcd binary included in your OS distro. 245 - Build your own image 246 - You can do: `cd kubernetes/cluster/images/etcd; make` 247 248 We recommend that you use the etcd version which is provided in the Kubernetes binary distribution. The Kubernetes binaries in the release 249 were tested extensively with this version of etcd and not with any other version. 250 The recommended version number can also be found as the value of `ETCD_VERSION` in `kubernetes/cluster/images/etcd/Makefile`. 251 252 The remainder of the document assumes that the image identifiers have been chosen and stored in corresponding env vars. Examples (replace with latest tags and appropriate registry): 253 - `HYPERKUBE_IMAGE==gcr.io/google_containers/hyperkube:$TAG` 254 - `ETCD_IMAGE=gcr.io/google_containers/etcd:$ETCD_VERSION` 255 256 ### Security Models 257 258 There are two main options for security: 259 - Access the apiserver using HTTP. 260 - Use a firewall for security. 261 - This is easier to setup. 262 - Access the apiserver using HTTPS 263 - Use https with certs, and credentials for user. 264 - This is the recommended approach. 265 - Configuring certs can be tricky. 266 267 If following the HTTPS approach, you will need to prepare certs and credentials. 268 269 #### Preparing Certs 270 271 You need to prepare several certs: 272 - The master needs a cert to act as an HTTPS server. 273 - The kubelets optionally need certs to identify themselves as clients of the master, and when 274 serving its own API over HTTPS. 275 276 Unless you plan to have a real CA generate your certs, you will need to generate a root cert and use that to sign the master, kubelet, and kubectl certs. 277 - see function `create-certs` in `cluster/gce/util.sh` 278 - see also `cluster/saltbase/salt/generate-cert/make-ca-cert.sh` and 279 `cluster/saltbase/salt/generate-cert/make-cert.sh` 280 281 You will end up with the following files (we will use these variables later on) 282 - `CA_CERT` 283 - put in on node where apiserver runs, in e.g. `/srv/kubernetes/ca.crt`. 284 - `MASTER_CERT` 285 - signed by CA_CERT 286 - put in on node where apiserver runs, in e.g. `/srv/kubernetes/server.crt` 287 - `MASTER_KEY ` 288 - put in on node where apiserver runs, in e.g. `/srv/kubernetes/server.key` 289 - `KUBELET_CERT` 290 - optional 291 - `KUBELET_KEY` 292 - optional 293 294 #### Preparing Credentials 295 296 The admin user (and any users) need: 297 - a token or a password to identify them. 298 - tokens are just long alphanumeric strings, e.g. 32 chars. See 299 - `TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)` 300 301 Your tokens and passwords need to be stored in a file for the apiserver 302 to read. This guide uses `/var/lib/kube-apiserver/known_tokens.csv`. 303 The format for this file is described in the [authentication documentation](../admin/authentication.md). 304 305 For distributing credentials to clients, the convention in Kubernetes is to put the credentials 306 into a [kubeconfig file](../user-guide/kubeconfig-file.md). 307 308 The kubeconfig file for the administrator can be created as follows: 309 - If you have already used Kubernetes with a non-custom cluster (for example, used a Getting Started 310 Guide), you will already have a `$HOME/.kube/config` file. 311 - You need to add certs, keys, and the master IP to the kubeconfig file: 312 - If using the firewall-only security option, set the apiserver this way: 313 - `kubectl config set-cluster $CLUSTER_NAME --server=http://$MASTER_IP --insecure-skip-tls-verify=true` 314 - Otherwise, do this to set the apiserver ip, client certs, and user credentials. 315 - `kubectl config set-cluster $CLUSTER_NAME --certificate-authority=$CA_CERT --embed-certs=true --server=https://$MASTER_IP` 316 - `kubectl config set-credentials $USER --client-certificate=$CLI_CERT --client-key=$CLI_KEY --embed-certs=true --token=$TOKEN` 317 - Set your cluster as the default cluster to use: 318 - `kubectl config set-context $CONTEXT_NAME --cluster=$CLUSTER_NAME --user=$USER` 319 - `kubectl config use-context $CONTEXT_NAME` 320 321 Next, make a kubeconfig file for the kubelets and kube-proxy. There are a couple of options for how 322 many distinct files to make: 323 1. Use the same credential as the admin 324 - This is simplest to setup. 325 1. One token and kubeconfig file for all kubelets, one for all kube-proxy, one for admin. 326 - This mirrors what is done on GCE today 327 1. Different credentials for every kubelet, etc. 328 - We are working on this but all the pieces are not ready yet. 329 330 You can make the files by copying the `$HOME/.kube/config`, by following the code 331 in `cluster/gce/configure-vm.sh` or by using the following template: 332 333 ```yaml 334 apiVersion: v1 335 kind: Config 336 users: 337 - name: kubelet 338 user: 339 token: ${KUBELET_TOKEN} 340 clusters: 341 - name: local 342 cluster: 343 certificate-authority-data: ${CA_CERT_BASE64_ENCODED} 344 contexts: 345 - context: 346 cluster: local 347 user: kubelet 348 name: service-account-context 349 current-context: service-account-context 350 ``` 351 352 Put the kubeconfig(s) on every node. The examples later in this 353 guide assume that there are kubeconfigs in `/var/lib/kube-proxy/kubeconfig` and 354 `/var/lib/kubelet/kubeconfig`. 355 356 ## Configuring and Installing Base Software on Nodes 357 358 This section discusses how to configure machines to be Kubernetes nodes. 359 360 You should run three daemons on every node: 361 - docker or rkt 362 - kubelet 363 - kube-proxy 364 365 You will also need to do assorted other configuration on top of a 366 base OS install. 367 368 Tip: One possible starting point is to setup a cluster using an existing Getting 369 Started Guide. After getting a cluster running, you can then copy the init.d scripts or systemd unit files from that 370 cluster, and then modify them for use on your custom cluster. 371 372 ### Docker 373 374 The minimum required Docker version will vary as the kubelet version changes. The newest stable release is a good choice. Kubelet will log a warning and refuse to start pods if the version is too old, so pick a version and try it. 375 376 If you previously had Docker installed on a node without setting Kubernetes-specific 377 options, you may have a Docker-created bridge and iptables rules. You may want to remove these 378 as follows before proceeding to configure Docker for Kubernetes. 379 380 ```sh 381 iptables -t nat -F 382 ifconfig docker0 down 383 brctl delbr docker0 384 ``` 385 386 The way you configure docker will depend in whether you have chosen the routable-vip or overlay-network approaches for your network. 387 Some suggested docker options: 388 - create your own bridge for the per-node CIDR ranges, call it cbr0, and set `--bridge=cbr0` option on docker. 389 - set `--iptables=false` so docker will not manipulate iptables for host-ports (too coarse on older docker versions, may be fixed in newer versions) 390 so that kube-proxy can manage iptables instead of docker. 391 - `--ip-masq=false` 392 - if you have setup PodIPs to be routable, then you want this false, otherwise, docker will 393 rewrite the PodIP source-address to a NodeIP. 394 - some environments (e.g. GCE) still need you to masquerade out-bound traffic when it leaves the cloud environment. This is very environment specific. 395 - if you are using an overlay network, consult those instructions. 396 - `--mtu=` 397 - may be required when using Flannel, because of the extra packet size due to udp encapsulation 398 - `--insecure-registry $CLUSTER_SUBNET` 399 - to connect to a private registry, if you set one up, without using SSL. 400 401 You may want to increase the number of open files for docker: 402 - `DOCKER_NOFILE=1000000` 403 404 Where this config goes depends on your node OS. For example, GCE's Debian-based distro uses `/etc/default/docker`. 405 406 Ensure docker is working correctly on your system before proceeding with the rest of the 407 installation, by following examples given in the Docker documentation. 408 409 ### rkt 410 411 [rkt](https://github.com/coreos/rkt) is an alternative to Docker. You only need to install one of Docker or rkt. 412 The minimum version required is [v0.5.6](https://github.com/coreos/rkt/releases/tag/v0.5.6). 413 414 [systemd](http://www.freedesktop.org/wiki/Software/systemd/) is required on your node to run rkt. The 415 minimum version required to match rkt v0.5.6 is 416 [systemd 215](http://lists.freedesktop.org/archives/systemd-devel/2014-July/020903.html). 417 418 [rkt metadata service](https://github.com/coreos/rkt/blob/master/Documentation/networking.md) is also required 419 for rkt networking support. You can start rkt metadata service by using command like 420 `sudo systemd-run rkt metadata-service` 421 422 Then you need to configure your kubelet with flag: 423 - `--container-runtime=rkt` 424 425 ### kubelet 426 427 All nodes should run kubelet. See [Selecting Binaries](#selecting-binaries). 428 429 Arguments to consider: 430 - If following the HTTPS security approach: 431 - `--api-servers=https://$MASTER_IP` 432 - `--kubeconfig=/var/lib/kubelet/kubeconfig` 433 - Otherwise, if taking the firewall-based security approach 434 - `--api-servers=http://$MASTER_IP` 435 - `--config=/etc/kubernetes/manifests` 436 - `--cluster-dns=` to the address of the DNS server you will setup (see [Starting Addons](#starting-addons).) 437 - `--cluster-domain=` to the dns domain prefix to use for cluster DNS addresses. 438 - `--docker-root=` 439 - `--root-dir=` 440 - `--configure-cbr0=` (described above) 441 - `--register-node` (described in [Node](../admin/node.md) documentation.) 442 443 ### kube-proxy 444 445 All nodes should run kube-proxy. (Running kube-proxy on a "master" node is not 446 strictly required, but being consistent is easier.) Obtain a binary as described for 447 kubelet. 448 449 Arguments to consider: 450 - If following the HTTPS security approach: 451 - `--api-servers=https://$MASTER_IP` 452 - `--kubeconfig=/var/lib/kube-proxy/kubeconfig` 453 - Otherwise, if taking the firewall-based security approach 454 - `--api-servers=http://$MASTER_IP` 455 456 ### Networking 457 458 Each node needs to be allocated its own CIDR range for pod networking. 459 Call this `NODE_X_POD_CIDR`. 460 461 A bridge called `cbr0` needs to be created on each node. The bridge is explained 462 further in the [networking documentation](../admin/networking.md). The bridge itself 463 needs an address from `$NODE_X_POD_CIDR` - by convention the first IP. Call 464 this `NODE_X_BRIDGE_ADDR`. For example, if `NODE_X_POD_CIDR` is `10.0.0.0/16`, 465 then `NODE_X_BRIDGE_ADDR` is `10.0.0.1/16`. NOTE: this retains the `/16` suffix 466 because of how this is used later. 467 468 - Recommended, automatic approach: 469 1. Set `--configure-cbr0=true` option in kubelet init script and restart kubelet service. Kubelet will configure cbr0 automatically. 470 It will wait to do this until the node controller has set Node.Spec.PodCIDR. Since you have not setup apiserver and node controller 471 yet, the bridge will not be setup immediately. 472 - Alternate, manual approach: 473 1. Set `--configure-cbr0=false` on kubelet and restart. 474 1. Create a bridge 475 - e.g. `brctl addbr cbr0`. 476 1. Set appropriate MTU 477 - `ip link set dev cbr0 mtu 1460` (NOTE: the actual value of MTU will depend on your network environment) 478 1. Add the clusters network to the bridge (docker will go on other side of bridge). 479 - e.g. `ip addr add $NODE_X_BRIDGE_ADDR dev eth0` 480 1. Turn it on 481 - e.g. `ip link set dev cbr0 up` 482 483 If you have turned off Docker's IP masquerading to allow pods to talk to each 484 other, then you may need to do masquerading just for destination IPs outside 485 the cluster network. For example: 486 487 ```sh 488 iptables -w -t nat -A POSTROUTING -o eth0 -j MASQUERADE \! -d ${CLUSTER_SUBNET} 489 ``` 490 491 This will rewrite the source address from 492 the PodIP to the Node IP for traffic bound outside the cluster, and kernel 493 [connection tracking](http://www.iptables.info/en/connection-state.html) 494 will ensure that responses destined to the node still reach 495 the pod. 496 497 NOTE: This is environment specific. Some environments will not need 498 any masquerading at all. Others, such as GCE, will not allow pod IPs to send 499 traffic to the internet, but have no problem with them inside your GCE Project. 500 501 ### Other 502 503 - Enable auto-upgrades for your OS package manager, if desired. 504 - Configure log rotation for all node components (e.g. using [logrotate](http://linux.die.net/man/8/logrotate)). 505 - Setup liveness-monitoring (e.g. using [monit](http://linux.die.net/man/1/monit)). 506 - Setup volume plugin support (optional) 507 - Install any client binaries for optional volume types, such as `glusterfs-client` for GlusterFS 508 volumes. 509 510 ### Using Configuration Management 511 512 The previous steps all involved "conventional" system administration techniques for setting up 513 machines. You may want to use a Configuration Management system to automate the node configuration 514 process. There are examples of [Saltstack](../admin/salt.md), Ansible, Juju, and CoreOS Cloud Config in the 515 various Getting Started Guides. 516 517 ## Bootstrapping the Cluster 518 519 While the basic node services (kubelet, kube-proxy, docker) are typically started and managed using 520 traditional system administration/automation approaches, the remaining *master* components of Kubernetes are 521 all configured and managed *by Kubernetes*: 522 - their options are specified in a Pod spec (yaml or json) rather than an /etc/init.d file or 523 systemd unit. 524 - they are kept running by Kubernetes rather than by init. 525 526 ### etcd 527 528 You will need to run one or more instances of etcd. 529 - Recommended approach: run one etcd instance, with its log written to a directory backed 530 by durable storage (RAID, GCE PD) 531 - Alternative: run 3 or 5 etcd instances. 532 - Log can be written to non-durable storage because storage is replicated. 533 - run a single apiserver which connects to one of the etc nodes. 534 See [cluster-troubleshooting](../admin/cluster-troubleshooting.md) for more discussion on factors affecting cluster 535 availability. 536 537 To run an etcd instance: 538 539 1. copy `cluster/saltbase/salt/etcd/etcd.manifest` 540 1. make any modifications needed 541 1. start the pod by putting it into the kubelet manifest directory 542 543 ### Apiserver, Controller Manager, and Scheduler 544 545 The apiserver, controller manager, and scheduler will each run as a pod on the master node. 546 547 For each of these components, the steps to start them running are similar: 548 549 1. Start with a provided template for a pod. 550 1. Set the `HYPERKUBE_IMAGE` to the values chosen in [Selecting Images](#selecting-images). 551 1. Determine which flags are needed for your cluster, using the advice below each template. 552 1. Set the flags to be individual strings in the command array (e.g. $ARGN below) 553 1. Start the pod by putting the completed template into the kubelet manifest directory. 554 1. Verify that the pod is started. 555 556 #### Apiserver pod template 557 558 ```json 559 { 560 "kind": "Pod", 561 "apiVersion": "v1", 562 "metadata": { 563 "name": "kube-apiserver" 564 }, 565 "spec": { 566 "hostNetwork": true, 567 "containers": [ 568 { 569 "name": "kube-apiserver", 570 "image": "${HYPERKUBE_IMAGE}", 571 "command": [ 572 "/hyperkube", 573 "apiserver", 574 "$ARG1", 575 "$ARG2", 576 ... 577 "$ARGN" 578 ], 579 "ports": [ 580 { 581 "name": "https", 582 "hostPort": 443, 583 "containerPort": 443 584 }, 585 { 586 "name": "local", 587 "hostPort": 8080, 588 "containerPort": 8080 589 } 590 ], 591 "volumeMounts": [ 592 { 593 "name": "srvkube", 594 "mountPath": "/srv/kubernetes", 595 "readOnly": true 596 }, 597 { 598 "name": "etcssl", 599 "mountPath": "/etc/ssl", 600 "readOnly": true 601 } 602 ], 603 "livenessProbe": { 604 "httpGet": { 605 "path": "/healthz", 606 "port": 8080 607 }, 608 "initialDelaySeconds": 15, 609 "timeoutSeconds": 15 610 } 611 } 612 ], 613 "volumes": [ 614 { 615 "name": "srvkube", 616 "hostPath": { 617 "path": "/srv/kubernetes" 618 } 619 }, 620 { 621 "name": "etcssl", 622 "hostPath": { 623 "path": "/etc/ssl" 624 } 625 } 626 ] 627 } 628 } 629 ``` 630 631 Here are some apiserver flags you may need to set: 632 633 - `--cloud-provider=` see [cloud providers](#cloud-providers) 634 - `--cloud-config=` see [cloud providers](#cloud-providers) 635 - `--address=${MASTER_IP}` *or* `--bind-address=127.0.0.1` and `--address=127.0.0.1` if you want to run a proxy on the master node. 636 - `--cluster-name=$CLUSTER_NAME` 637 - `--service-cluster-ip-range=$SERVICE_CLUSTER_IP_RANGE` 638 - `--etcd-servers=http://127.0.0.1:4001` 639 - `--tls-cert-file=/srv/kubernetes/server.cert` 640 - `--tls-private-key-file=/srv/kubernetes/server.key` 641 - `--admission-control=$RECOMMENDED_LIST` 642 - See [admission controllers](../admin/admission-controllers.md) for recommended arguments. 643 - `--allow-privileged=true`, only if you trust your cluster user to run pods as root. 644 645 If you are following the firewall-only security approach, then use these arguments: 646 647 - `--token-auth-file=/dev/null` 648 - `--insecure-bind-address=$MASTER_IP` 649 - `--advertise-address=$MASTER_IP` 650 651 If you are using the HTTPS approach, then set: 652 - `--client-ca-file=/srv/kubernetes/ca.crt` 653 - `--token-auth-file=/srv/kubernetes/known_tokens.csv` 654 - `--basic-auth-file=/srv/kubernetes/basic_auth.csv` 655 656 This pod mounts several node file system directories using the `hostPath` volumes. Their purposes are: 657 - The `/etc/ssl` mount allows the apiserver to find the SSL root certs so it can 658 authenticate external services, such as a cloud provider. 659 - This is not required if you do not use a cloud provider (e.g. bare-metal). 660 - The `/srv/kubernetes` mount allows the apiserver to read certs and credentials stored on the 661 node disk. These could instead be stored on a persistent disk, such as a GCE PD, or baked into the image. 662 - Optionally, you may want to mount `/var/log` as well and redirect output there (not shown in template). 663 - Do this if you prefer your logs to be accessible from the root filesystem with tools like journalctl. 664 665 *TODO* document proxy-ssh setup. 666 667 ##### Cloud Providers 668 669 Apiserver supports several cloud providers. 670 671 - options for `--cloud-provider` flag are `aws`, `gce`, `mesos`, `openshift`, `ovirt`, `rackspace`, `vagrant`, or unset. 672 - unset used for e.g. bare metal setups. 673 - support for new IaaS is added by contributing code [here](../../pkg/cloudprovider/providers/) 674 675 Some cloud providers require a config file. If so, you need to put config file into apiserver image or mount through hostPath. 676 677 - `--cloud-config=` set if cloud provider requires a config file. 678 - Used by `aws`, `gce`, `mesos`, `openshift`, `ovirt` and `rackspace`. 679 - You must put config file into apiserver image or mount through hostPath. 680 - Cloud config file syntax is [Gcfg](https://code.google.com/p/gcfg/). 681 - AWS format defined by type [AWSCloudConfig](../../pkg/cloudprovider/providers/aws/aws.go) 682 - There is a similar type in the corresponding file for other cloud providers. 683 - GCE example: search for `gce.conf` in [this file](../../cluster/gce/configure-vm.sh) 684 685 #### Scheduler pod template 686 687 Complete this template for the scheduler pod: 688 689 ```json 690 691 { 692 "kind": "Pod", 693 "apiVersion": "v1", 694 "metadata": { 695 "name": "kube-scheduler" 696 }, 697 "spec": { 698 "hostNetwork": true, 699 "containers": [ 700 { 701 "name": "kube-scheduler", 702 "image": "$HYBERKUBE_IMAGE", 703 "command": [ 704 "/hyperkube", 705 "scheduler", 706 "--master=127.0.0.1:8080", 707 "$SCHEDULER_FLAG1", 708 ... 709 "$SCHEDULER_FLAGN" 710 ], 711 "livenessProbe": { 712 "httpGet": { 713 "host" : "127.0.0.1", 714 "path": "/healthz", 715 "port": 10251 716 }, 717 "initialDelaySeconds": 15, 718 "timeoutSeconds": 15 719 } 720 } 721 ] 722 } 723 } 724 725 ``` 726 727 Typically, no additional flags are required for the scheduler. 728 729 Optionally, you may want to mount `/var/log` as well and redirect output there. 730 731 #### Controller Manager Template 732 733 Template for controller manager pod: 734 735 ```json 736 737 { 738 "kind": "Pod", 739 "apiVersion": "v1", 740 "metadata": { 741 "name": "kube-controller-manager" 742 }, 743 "spec": { 744 "hostNetwork": true, 745 "containers": [ 746 { 747 "name": "kube-controller-manager", 748 "image": "$HYPERKUBE_IMAGE", 749 "command": [ 750 "/hyperkube", 751 "controller-manager", 752 "$CNTRLMNGR_FLAG1", 753 ... 754 "$CNTRLMNGR_FLAGN" 755 ], 756 "volumeMounts": [ 757 { 758 "name": "srvkube", 759 "mountPath": "/srv/kubernetes", 760 "readOnly": true 761 }, 762 { 763 "name": "etcssl", 764 "mountPath": "/etc/ssl", 765 "readOnly": true 766 } 767 ], 768 "livenessProbe": { 769 "httpGet": { 770 "host": "127.0.0.1", 771 "path": "/healthz", 772 "port": 10252 773 }, 774 "initialDelaySeconds": 15, 775 "timeoutSeconds": 15 776 } 777 } 778 ], 779 "volumes": [ 780 { 781 "name": "srvkube", 782 "hostPath": { 783 "path": "/srv/kubernetes" 784 } 785 }, 786 { 787 "name": "etcssl", 788 "hostPath": { 789 "path": "/etc/ssl" 790 } 791 } 792 ] 793 } 794 } 795 796 ``` 797 798 Flags to consider using with controller manager: 799 - `--cluster-name=$CLUSTER_NAME` 800 - `--cluster-cidr=` 801 - *TODO*: explain this flag. 802 - `--allocate-node-cidrs=` 803 - *TODO*: explain when you want controller to do this and when you want to do it another way. 804 - `--cloud-provider=` and `--cloud-config` as described in apiserver section. 805 - `--service-account-private-key-file=/srv/kubernetes/server.key`, used by the [service account](../user-guide/service-accounts.md) feature. 806 - `--master=127.0.0.1:8080` 807 808 #### Starting and Verifying Apiserver, Scheduler, and Controller Manager 809 810 Place each completed pod template into the kubelet config dir 811 (whatever `--config=` argument of kubelet is set to, typically 812 `/etc/kubernetes/manifests`). The order does not matter: scheduler and 813 controller manager will retry reaching the apiserver until it is up. 814 815 Use `ps` or `docker ps` to verify that each process has started. For example, verify that kubelet has started a container for the apiserver like this: 816 817 ```console 818 $ sudo docker ps | grep apiserver: 819 5783290746d5 gcr.io/google_containers/kube-apiserver:e36bf367342b5a80d7467fd7611ad873 "/bin/sh -c '/usr/lo'" 10 seconds ago Up 9 seconds k8s_kube-apiserver.feb145e7_kube-apiserver-kubernetes-master_default_eaebc600cf80dae59902b44225f2fc0a_225a4695 820 ``` 821 822 Then try to connect to the apiserver: 823 824 ```console 825 $ echo $(curl -s http://localhost:8080/healthz) 826 ok 827 $ curl -s http://localhost:8080/api 828 { 829 "versions": [ 830 "v1" 831 ] 832 } 833 ``` 834 835 If you have selected the `--register-node=true` option for kubelets, they will now begin self-registering with the apiserver. 836 You should soon be able to see all your nodes by running the `kubect get nodes` command. 837 Otherwise, you will need to manually create node objects. 838 839 ### Logging 840 841 **TODO** talk about starting Logging. 842 843 ### Monitoring 844 845 **TODO** talk about starting Logging. 846 847 ### DNS 848 849 **TODO** talk about starting DNS. 850 851 ## Troubleshooting 852 853 ### Running validate-cluster 854 855 **TODO** explain how to use `cluster/validate-cluster.sh` 856 857 ### Inspect pods and services 858 859 Try to run through the "Inspect your cluster" section in one of the other Getting Started Guides, such as [GCE](gce.md#inspect-your-cluster). 860 You should see some services. You should also see "mirror pods" for the apiserver, scheduler and controller-manager, plus any add-ons you started. 861 862 ### Try Examples 863 864 At this point you should be able to run through one of the basic examples, such as the [nginx example](../../examples/simple-nginx.md). 865 866 ### Running the Conformance Test 867 868 You may want to try to run the [Conformance test](http://releases.k8s.io/HEAD/hack/conformance-test.sh). Any failures may give a hint as to areas that need more attention. 869 870 ### Networking 871 872 The nodes must be able to connect to each other using their private IP. Verify this by 873 pinging or SSH-ing from one node to another. 874 875 ### Getting Help 876 877 If you run into trouble, please see the section on [troubleshooting](gce.md#troubleshooting), post to the 878 [google-containers group](https://groups.google.com/forum/#!forum/google-containers), or come ask questions on [Slack](../troubleshooting.md#slack). 879 880 881 <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> 882 [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/scratch.md?pixel)]() 883 <!-- END MUNGE: GENERATED_ANALYTICS -->