github.com/kubernetes-incubator/kube-aws@v0.16.4/proposals/plugins.md (about) 1 # Introduction 2 3 A draft design for [kube-aws plugins](https://github.com/kubernetes-incubator/kube-aws/issues/509). Plugins are initially defined as additions to a kube-aws generated 4 cluster that are not part of the core feature set of providing an easy to use, highly available, 5 best practice cluster on AWS. Plugins can also later be used for built-in functionality to provide a more modular deployment and ability to test. 6 7 # Initial Use Cases 8 9 1. A user wishes to deploy a component by default to their cluster instances outside of Kubernetes, e.g. a 10 monitoring agent. 11 1. A user wishes to deploy a component into their cluster by default that is not part of core 12 Kubernetes such as cluster wide services, addons, ingress controller or even persistent volumes. 13 Reasons can vary from company restrictions to always having this component form part of every 14 cluster. e.g. [the rescheduler](https://github.com/kubernetes-incubator/kube-aws/issues/118) 15 1. A user wishes to add [customised parts to the CloudFormation template](https://github.com/kubernetes-incubator/kube-aws/issues/96#issuecomment-263835808) 16 produced by kube-aws. Anything they add would need access to the same parameters and template 17 engine that already exists. 18 1. A user wishes to customise existing parts of the CloudFormation template produced by kube-aws 19 without necessarily having to diff all the files for changes with there sometimes being large 20 conflicts or huge updates due to kube-aws changes that can be difficult to digest while 21 being able to merge in the simple changes they need. Examples 22 [here](https://github.com/kubernetes-incubator/kube-aws/issues/340) and 23 [here](https://github.com/kubernetes-incubator/kube-aws/issues/338) 24 25 # Specific Examples 26 27 * Deployment outside of Kubernetes - Dex uses some additional API server parameter, Datadog agent still has more functionality if not using the containerised version 28 * Deployment to Kubernetes - rescheduler, kube2iam 29 * Add new parts to CloudFormation - bastion host 30 * Customisations to existing parts of CloudFormation - IAM roles/groups 31 32 # Plugin Principles 33 34 1. Plugin functionality should first be added in places where kube-aws needs it most, i.e. where 35 kube-aws receives more feature requests which do not fit with the core feature set. 36 1. kube-aws should dogfood the plugins system, i.e. use the plugins system where possible on existing functionality 37 1. Everything that is pluggable should be kept as modular as possible, from lists of IAM roles to kube API server parameters. 38 1. Where possible, checks will be added to kube-aws to ensure the integrity of the cluster via validation or disallowing particular changes. e.g. disallow overriding node pool init metadata 39 * Note, this could be gradually added rather than forming part of the first implementation 40 * Some of this will likely be provided by CloudFormation but kube-aws could possibly replicas the checks to save time for the stack create/update to fail 41 42 # Design 43 44 ## Plugin Types 45 46 * Built-in plugins versus ad-hoc user specific plugins 47 * Some plugins will be provided by kube-aws and some will be specific to a user 48 * Both will use the same mechanisms to hook into kube-aws/deploy etc 49 * Both types of plugins will be enabled and configured in the same way 50 * Built-in plugins versus contrib plugins 51 * kube-aws will accept contrib plugins for features considered to be applicable to many users 52 * Contrib plugins will be enabled in a similar way but specifically marked as not fully 53 battle tested; they will still be configured in the same way 54 * Plugins will be allowed to act across the whole cluster or act on a specific set of node pools, this should form part of the base configuration for which plugins to run 55 56 ## Stages 57 58 The workflow for kube-aws should consist of the following stages, each of which will be called on the plugins in turn. Each stage will complete across all plugins prior to proceeding to the next stage. All plugins will receive input based on the whole setup, not just their own. This will allow them to hook into parameters from other plugins or core logic. The stages are not intended to become kube-aws commands and should mostly be transparent to a kube-aws user. 59 60 1. initialise 61 * Perform any prerequisite setup required such as calls to external system. Configuration, presence of files etc can be validated. 62 * Input - configuration 63 * Output - configuration with any initialised values, internal kube-aws model with primary constructs for CloudFormation, Kubernetes YAML/JSON, Helm charts, systemd units, kube API server parameters and any other key areas that allow customisation. Some of these will be file references but some will be full kube-aws internal models. The available list of constructs will be added to in further implementations. 64 1. render 65 * Stack files are gathered and rendered locally, most plugins will perform a no-op here but some may choose to output additional CloudFormation stacks or perform additional validation. A core plugin could grab all the primary constructs and create the main CloudFormation or a kube API server plugin could validate none of the parameters it has set are conflicting. 66 * Input - output of `initialise` stage 67 * Output - CloudFormation file(s), other file reference(s) such as Helm Charts 68 1. deploy 69 * Take the CloudFormation and file(s), upload the files to S3 and deploy the template(s) to CloudFormation. This is an opportunity for plugins to perform waiting tasks and/or actions that need to be done on deploy/install, for example post to an endpoint created during deploy. 70 * Input - output of `render` stage, stack name 71 * Output - stack details such as stack name, endpoints, certs 72 1. validate 73 * Plugins can check they are functioning as expected which may depend on the configuration 74 * Input - output of `deploy` stage, configuration 75 * Output - validation errors 76 77 Validation errors can also be returned at any stage and can be: 78 1. fatal, kube-aws will stop processing further and log the problem 79 1. warning, kube-aws will continue processing but log a warning, can be used for deprecation 80 81 The output of the `initialise` stage is similar to `render stack` in kube-aws as of today with templates and a few other files. The output of the `render` stage is the fully formed CloudFormation without templating. Both of these could be stored as the user wishes and potentially version controlled. 82 83 ## Examples 84 85 ### Deployment Outside of Kubernetes 86 87 An example is a non-containerised monitoring agent or healthcheck. systemd units and other commands 88 could be output from plugins during the `initialise` stage and hence get included in the generated cluster. 89 90 ### Dex / Rescheduler / Add-on Installation 91 92 Dex installation requires changes to the kube API server parameters plus a deployment into a running Kubernetes cluster. 93 kube API server parameters will be one of the constructs available as an output from `initialise` and so can be added to from the Dex plugin. 94 The deployment can be performed either via Kubernetes YAML or Helm Chart both of which would be a file/directory reference. 95 96 ### Additional Worker IAM Permissions / kube2iam 97 98 Additional worker IAM permissions requires either additions to the existing IAM policies or new IAM policies 99 being attached to the worker instance role. This could be done in a few ways, for example `initialise` could 100 output the additional CloudFormation template snippet needed along with name/ref of the IAM policy. 101 The policies and policy document for the workers would be primarily managed by a core plugin that deals 102 with the worker permissions. It would take the output of `initialise` and render the final CloudFormation required. 103 104 The kube2iam add-on plugin Kubernetes YAML or a Helm Chart to be installed which is covered by the Dex 105 example above plus changes to the worker IAM permissions. 106 107 ### Bastion Host 108 109 A bastion host requires additional CloudFormation to deploy the instance into the correct part of the 110 VPC. It therefore requires knowledge of the VPC/network setup as input. This plugin would take the 111 configuration in `initialise` and output the additional CloudFormation based on the VPC settings. 112 113 ### CloudFormation Renderer 114 115 During the `initialise` stage a core CloudFormation render plugin could output the basic CloudFormation templates for the cluster. Then during the `render` stage the same core plugin could choose to take all the output across all plugins from `initialise` and render the final CloudFormation. 116 117 ### Kube API Server 118 119 A Kubernetes API server plugin could be responsible for main piece of cloud config that sets up the API server. It could then also perform additional validation during the `render` stage to ensure any plugins that have added to the kube API server parameters primary construct have not made the final state invalid. 120 121 # Implementation 122 123 * [Helm](https://github.com/kubernetes/helm) should be the main method for deploying additional components into the Kubernetes cluster 124 * How can kube-aws deal with plugin ordering? The ordering could be defined in a fixed YAML file and anything not present is always run last. 125 * What is the process to include and compile plugins? 126 * One idea is to use [go-plugin](https://github.com/hashicorp/go-plugin) since the native 1.8 plugins is not mature yet 127 * We need to protect sensitive information if uploading files, some Helm charts have secrets as config values so how will these be input? 128 129 ## Configuration 130 131 All plugins will be enabled and configured via the `cluster.yaml` plus a set of plugin specific 132 configuration files. However, as above, the whole configuration model will be passed to all plugins 133 so they can access parameters from another plugin if necessary. 134 135 ## Plugin Locations 136 137 * Built-in plugins can either be part of the kube-aws binary or downloadable from a secure source 138 if/when the user enables that plugin. The second is recommended. 139 * Built-in plugins will live under `kubernetes-incubator/kube-aws/plugins/<PLUGIN_NAME>` 140 * This location can be thought of as the built-in plugin repository 141 * User contributed plugins will live under `kubernetes-incubator/kube-aws/contrib/plugins/<PLUGIN_NAME>` 142 until they are considered core or deprecated. 143 * This location can be thought of as the contrib plugin repository kube-aws plugins 144 * User provided will live under `<USER_DIRECTORY>/plugins/<PLUGIN_NAME>` 145 * This location can be thought of as user provided repository 146 * A user can specify a set of directories for plugins to be read from, likely as a CLI argument