github.com/jmrodri/operator-sdk@v0.5.0/doc/user-guide.md (about) 1 # User Guide 2 3 This guide walks through an example of building a simple memcached-operator using the operator-sdk 4 CLI tool and controller-runtime library API. To learn how to use Ansible or Helm to create an 5 operator, see the [Ansible Operator User Guide][ansible_user_guide] or the [Helm Operator User 6 Guide][helm_user_guide]. The rest of this document will show how to program an operator in Go. 7 8 ## Prerequisites 9 10 - [dep][dep_tool] version v0.5.0+. 11 - [git][git_tool] 12 - [go][go_tool] version v1.10+. 13 - [docker][docker_tool] version 17.03+. 14 - [kubectl][kubectl_tool] version v1.11.0+. 15 - Access to a kubernetes v.1.11.0+ cluster. 16 17 **Note**: This guide uses [minikube][minikube_tool] version v0.25.0+ as the local kubernetes cluster and quay.io for the public registry. 18 19 ## Install the Operator SDK CLI 20 21 The Operator SDK has a CLI tool that helps the developer to create, build, and deploy a new operator project. 22 23 Checkout the desired release tag and install the SDK CLI tool: 24 25 ```sh 26 $ mkdir -p $GOPATH/src/github.com/operator-framework 27 $ cd $GOPATH/src/github.com/operator-framework 28 $ git clone https://github.com/operator-framework/operator-sdk 29 $ cd operator-sdk 30 $ git checkout master 31 $ make dep 32 $ make install 33 ``` 34 35 This installs the CLI binary `operator-sdk` at `$GOPATH/bin`. 36 37 ## Create a new project 38 39 Use the CLI to create a new memcached-operator project: 40 41 ```sh 42 $ mkdir -p $GOPATH/src/github.com/example-inc/ 43 $ cd $GOPATH/src/github.com/example-inc/ 44 $ operator-sdk new memcached-operator 45 $ cd memcached-operator 46 ``` 47 48 To learn about the project directory structure, see [project layout][layout_doc] doc. 49 50 #### Operator scope 51 52 A namespace-scoped operator (the default) watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions. However, there are use cases where a cluster-scoped operator may make sense. For example, the [cert-manager](https://github.com/jetstack/cert-manager) operator is often deployed with cluster-scoped permissions and watches so that it can manage issuing certificates for an entire cluster. 53 54 If you'd like to create your memcached-operator project to be cluster-scoped use the following `operator-sdk new` command instead: 55 ``` 56 $ operator-sdk new memcached-operator --cluster-scoped 57 ``` 58 59 Using `--cluster-scoped` will scaffold the new operator with the following modifications: 60 * `deploy/operator.yaml` - Set `WATCH_NAMESPACE=""` instead of setting it to the pod's namespace 61 * `deploy/role.yaml` - Use `ClusterRole` instead of `Role` 62 * `deploy/role_binding.yaml`: 63 * Use `ClusterRoleBinding` instead of `RoleBinding` 64 * Set the subject namespace to `REPLACE_NAMESPACE`. This must be changed to the namespace in which the operator is deployed. 65 66 ### Manager 67 The main program for the operator `cmd/manager/main.go` initializes and runs the [Manager][manager_go_doc]. 68 69 The Manager will automatically register the scheme for all custom resources defined under `pkg/apis/...` and run all controllers under `pkg/controller/...`. 70 71 The Manager can restrict the namespace that all controllers will watch for resources: 72 ```Go 73 mgr, err := manager.New(cfg, manager.Options{Namespace: namespace}) 74 ``` 75 By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty: 76 ```Go 77 mgr, err := manager.New(cfg, manager.Options{Namespace: ""}) 78 ``` 79 80 ## Add a new Custom Resource Definition 81 82 Add a new Custom Resource Definition(CRD) API called Memcached, with APIVersion `cache.example.com/v1alpha1` and Kind `Memcached`. 83 84 ```sh 85 $ operator-sdk add api --api-version=cache.example.com/v1alpha1 --kind=Memcached 86 ``` 87 88 This will scaffold the Memcached resource API under `pkg/apis/cache/v1alpha1/...`. 89 90 ### Define the spec and status 91 92 Modify the spec and status of the `Memcached` Custom Resource(CR) at `pkg/apis/cache/v1alpha1/memcached_types.go`: 93 94 ```Go 95 type MemcachedSpec struct { 96 // Size is the size of the memcached deployment 97 Size int32 `json:"size"` 98 } 99 type MemcachedStatus struct { 100 // Nodes are the names of the memcached pods 101 Nodes []string `json:"nodes"` 102 } 103 ``` 104 105 After modifying the `*_types.go` file always run the following command to update the generated code for that resource type: 106 107 ```sh 108 $ operator-sdk generate k8s 109 ``` 110 111 ## Add a new Controller 112 113 Add a new [Controller][controller-go-doc] to the project that will watch and reconcile the Memcached resource: 114 115 ```sh 116 $ operator-sdk add controller --api-version=cache.example.com/v1alpha1 --kind=Memcached 117 ``` 118 119 This will scaffold a new Controller implementation under `pkg/controller/memcached/...`. 120 121 For this example replace the generated Controller file `pkg/controller/memcached/memcached_controller.go` with the example [`memcached_controller.go`][memcached_controller] implementation. 122 123 The example Controller executes the following reconciliation logic for each `Memcached` CR: 124 - Create a memcached Deployment if it doesn't exist 125 - Ensure that the Deployment size is the same as specified by the `Memcached` CR spec 126 - Update the `Memcached` CR status using the status writer with the names of the memcached pods 127 128 The next two subsections explain how the Controller watches resources and how the reconcile loop is triggered. Skip to the [Build](#build-and-run-the-operator) section to see how to build and run the operator. 129 130 ### Resources watched by the Controller 131 132 Inspect the Controller implementation at `pkg/controller/memcached/memcached_controller.go` to see how the Controller watches resources. 133 134 The first watch is for the Memcached type as the primary resource. For each Add/Update/Delete event the reconcile loop will be sent a reconcile `Request` (a namespace/name key) for that Memcached object: 135 136 ```Go 137 err := c.Watch( 138 &source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{}) 139 ``` 140 141 The next watch is for Deployments but the event handler will map each event to a reconcile `Request` for the owner of the Deployment. Which in this case is the Memcached object for which the Deployment was created. This allows the controller to watch Deployments as a secondary resource. 142 143 ```Go 144 err := c.Watch(&source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForOwner{ 145 IsController: true, 146 OwnerType: &cachev1alpha1.Memcached{}, 147 }) 148 ``` 149 150 **// TODO:** Doc on eventhandler, arbitrary mapping between watched and reconciled resource. 151 152 **// TODO:** Doc on configuring a Controller: number of workers, predicates, watching channels, 153 154 ### Reconcile loop 155 156 Every Controller has a Reconciler object with a `Reconcile()` method that implements the reconcile loop. The reconcile loop is passed the [`Request`][request-go-doc] argument which is a Namespace/Name key used to lookup the primary resource object, Memcached, from the cache: 157 158 ```Go 159 func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) { 160 // Lookup the Memcached instance for this reconcile request 161 memcached := &cachev1alpha1.Memcached{} 162 err := r.client.Get(context.TODO(), request.NamespacedName, memcached) 163 ... 164 } 165 ``` 166 167 Based on the return values, [`Result`][result_go_doc] and error, the `Request` may be requeued and the reconcile loop may be triggered again: 168 169 ```Go 170 // Reconcile successful - don't requeue 171 return reconcile.Result{}, nil 172 // Reconcile failed due to error - requeue 173 return reconcile.Result{}, err 174 // Requeue for any reason other than error 175 return reconcile.Result{Requeue: true}, nil 176 ``` 177 178 You can set the `Result.RequeueAfter` to requeue the `Request` after a grace period as well: 179 ```Go 180 import "time" 181 182 // Reconcile for any reason than error after 5 seconds 183 return reconcile.Result{RequeueAfter: time.Second*5}, nil 184 ``` 185 186 **Note:** Returning `Result` with `RequeueAfter` set is how you can periodically reconcile a CR. 187 188 For a guide on Reconcilers, Clients, and interacting with resource Events, see the [Client API doc][doc_client_api]. 189 190 ## Build and run the operator 191 192 Before running the operator, the CRD must be registered with the Kubernetes apiserver: 193 194 ```sh 195 $ kubectl create -f deploy/crds/cache_v1alpha1_memcached_crd.yaml 196 ``` 197 198 Once this is done, there are two ways to run the operator: 199 200 - As a Deployment inside a Kubernetes cluster 201 - As Go program outside a cluster 202 203 ### 1. Run as a Deployment inside the cluster 204 205 Build the memcached-operator image and push it to a registry: 206 ``` 207 $ operator-sdk build quay.io/example/memcached-operator:v0.0.1 208 $ sed -i 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml 209 $ docker push quay.io/example/memcached-operator:v0.0.1 210 ``` 211 212 If you created your operator using `--cluster-scoped=true`, update the service account namespace in the generated `ClusterRoleBinding` to match where you are deploying your operator. 213 ``` 214 $ export OPERATOR_NAMESPACE=$(kubectl config view --minify -o jsonpath='{.contexts[0].context.namespace}') 215 $ sed -i "s|REPLACE_NAMESPACE|$OPERATOR_NAMESPACE|g" deploy/role_binding.yaml 216 ``` 217 218 **Note** 219 If you are performing these steps on OSX, use the following commands instead: 220 ``` 221 $ sed -i "" 's|REPLACE_IMAGE|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml 222 $ sed -i "" "s|REPLACE_NAMESPACE|$OPERATOR_NAMESPACE|g" deploy/role_binding.yaml 223 ``` 224 225 The Deployment manifest is generated at `deploy/operator.yaml`. Be sure to update the deployment image as shown above since the default is just a placeholder. 226 227 Setup RBAC and deploy the memcached-operator: 228 229 ```sh 230 $ kubectl create -f deploy/service_account.yaml 231 $ kubectl create -f deploy/role.yaml 232 $ kubectl create -f deploy/role_binding.yaml 233 $ kubectl create -f deploy/operator.yaml 234 ``` 235 236 Verify that the memcached-operator is up and running: 237 238 ```sh 239 $ kubectl get deployment 240 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 241 memcached-operator 1 1 1 1 1m 242 ``` 243 244 ### 2. Run locally outside the cluster 245 246 This method is preferred during development cycle to deploy and test faster. 247 248 Set the name of the operator in an environment variable: 249 250 ```sh 251 export OPERATOR_NAME=memcached-operator 252 ``` 253 254 Run the operator locally with the default kubernetes config file present at `$HOME/.kube/config`: 255 256 ```sh 257 $ operator-sdk up local --namespace=default 258 2018/09/30 23:10:11 Go Version: go1.10.2 259 2018/09/30 23:10:11 Go OS/Arch: darwin/amd64 260 2018/09/30 23:10:11 operator-sdk Version: 0.0.6+git 261 2018/09/30 23:10:12 Registering Components. 262 2018/09/30 23:10:12 Starting the Cmd. 263 ``` 264 265 You can use a specific kubeconfig via the flag `--kubeconfig=<path/to/kubeconfig>`. 266 267 ## Create a Memcached CR 268 269 Create the example `Memcached` CR that was generated at `deploy/crds/cache_v1alpha1_memcached_cr.yaml`: 270 271 ```sh 272 $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml 273 apiVersion: "cache.example.com/v1alpha1" 274 kind: "Memcached" 275 metadata: 276 name: "example-memcached" 277 spec: 278 size: 3 279 280 $ kubectl apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml 281 ``` 282 283 Ensure that the memcached-operator creates the deployment for the CR: 284 285 ```sh 286 $ kubectl get deployment 287 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 288 memcached-operator 1 1 1 1 2m 289 example-memcached 3 3 3 3 1m 290 ``` 291 292 Check the pods and CR status to confirm the status is updated with the memcached pod names: 293 294 ```sh 295 $ kubectl get pods 296 NAME READY STATUS RESTARTS AGE 297 example-memcached-6fd7c98d8-7dqdr 1/1 Running 0 1m 298 example-memcached-6fd7c98d8-g5k7v 1/1 Running 0 1m 299 example-memcached-6fd7c98d8-m7vn7 1/1 Running 0 1m 300 memcached-operator-7cc7cfdf86-vvjqk 1/1 Running 0 2m 301 ``` 302 303 ```sh 304 $ kubectl get memcached/example-memcached -o yaml 305 apiVersion: cache.example.com/v1alpha1 306 kind: Memcached 307 metadata: 308 clusterName: "" 309 creationTimestamp: 2018-03-31T22:51:08Z 310 generation: 0 311 name: example-memcached 312 namespace: default 313 resourceVersion: "245453" 314 selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/example-memcached 315 uid: 0026cc97-3536-11e8-bd83-0800274106a1 316 spec: 317 size: 3 318 status: 319 nodes: 320 - example-memcached-6fd7c98d8-7dqdr 321 - example-memcached-6fd7c98d8-g5k7v 322 - example-memcached-6fd7c98d8-m7vn7 323 ``` 324 325 ### Update the size 326 327 Change the `spec.size` field in the memcached CR from 3 to 4 and apply the change: 328 329 ```sh 330 $ cat deploy/crds/cache_v1alpha1_memcached_cr.yaml 331 apiVersion: "cache.example.com/v1alpha1" 332 kind: "Memcached" 333 metadata: 334 name: "example-memcached" 335 spec: 336 size: 4 337 338 $ kubectl apply -f deploy/crds/cache_v1alpha1_memcached_cr.yaml 339 ``` 340 341 Confirm that the operator changes the deployment size: 342 343 ```sh 344 $ kubectl get deployment 345 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 346 example-memcached 4 4 4 4 5m 347 ``` 348 349 ### Cleanup 350 351 Clean up the resources: 352 353 ```sh 354 $ kubectl delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml 355 $ kubectl delete -f deploy/operator.yaml 356 $ kubectl delete -f deploy/role_binding.yaml 357 $ kubectl delete -f deploy/role.yaml 358 $ kubectl delete -f deploy/service_account.yaml 359 ``` 360 361 ## Advanced Topics 362 363 ### Adding 3rd Party Resources To Your Operator 364 365 The operator's Manager supports the Core Kubernetes resource types as found in the client-go [scheme][scheme_package] package and will also register the schemes of all custom resource types defined in your project under `pkg/apis`. 366 ```Go 367 import ( 368 "github.com/example-inc/memcached-operator/pkg/apis" 369 ... 370 ) 371 // Setup Scheme for all resources 372 if err := apis.AddToScheme(mgr.GetScheme()); err != nil { 373 log.Error(err, "") 374 os.Exit(1) 375 } 376 ``` 377 378 To add a 3rd party resource to an operator, you must add it to the Manager's scheme. By creating an `AddToScheme` method or reusing one you can easily add a resource to your scheme. An [example][deployments_register] shows that you define a function and then use the [runtime][runtime_package] package to create a `SchemeBuilder`. 379 380 #### Register with the Manager's scheme 381 382 Call the `AddToScheme()` function for your 3rd party resource and pass it the Manager's scheme via `mgr.GetScheme()`. 383 384 Example: 385 ```go 386 import ( 387 .... 388 routev1 "github.com/openshift/api/route/v1" 389 ) 390 391 func main() { 392 .... 393 if err := routev1.AddToScheme(mgr.GetScheme()); err != nil { 394 log.Error(err, "") 395 os.Exit(1) 396 } 397 .... 398 } 399 ``` 400 401 After adding new import paths to your operator project, run `dep ensure` in the root of your project directory to fulfill these dependencies. 402 403 [memcached_handler]: ../example/memcached-operator/handler.go.tmpl 404 [memcached_controller]: ../example/memcached-operator/memcached_controller.go.tmpl 405 [layout_doc]:./project_layout.md 406 [ansible_user_guide]:./ansible/user-guide.md 407 [helm_user_guide]:./helm/user-guide.md 408 [dep_tool]:https://golang.github.io/dep/docs/installation.html 409 [git_tool]:https://git-scm.com/downloads 410 [go_tool]:https://golang.org/dl/ 411 [docker_tool]:https://docs.docker.com/install/ 412 [kubectl_tool]:https://kubernetes.io/docs/tasks/tools/install-kubectl/ 413 [minikube_tool]:https://github.com/kubernetes/minikube#installation 414 [scheme_package]:https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go 415 [deployments_register]: https://github.com/kubernetes/api/blob/master/apps/v1/register.go#L41 416 [doc_client_api]:./user/client.md 417 [runtime_package]: https://godoc.org/k8s.io/apimachinery/pkg/runtime 418 [manager_go_doc]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/manager#Manager 419 [controller-go-doc]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg#hdr-Controller 420 [request-go-doc]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/reconcile#Request 421 [result_go_doc]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/reconcile#Result