github.com/GoogleContainerTools/skaffold/v2@v2.13.2/docs-v2/design_proposals/new-render-and-deploy.md (about) 1 # New Skaffold Render and Deploy (v2) 2 3 * Author(s): [Yuwen Ma](https://github.com/yuwenma) 4 * Design Shepherd: 5 - [Vic Iglesias](https://github.com/viglesiasce) 6 - [Tejal Desai](https://github.com/tejal29) 7 - [Nick Kubala](https://github.com/nkubala) 8 9 * Last Updated: 2021-05-07 10 * Status: Approved/ Under implementation 11 12 ## Glossary 13 14 #### DRY configuration 15 16 Raw application configuration. 17 18 It means the configuration may use variants or base/overlays (kustomize structured layout) but not extended and can’t be understood by the cluster directly. The raw config is not validated or passed through any validation or other operation checks yet. 19 20 #### WET configuration 21 22 Manipulated application configuration. 23 24 It has been flattened (from the DRY structure) and processed through a series of validation, generation and transformation operations and is a pure kubernetes resource and can be applied to the cluster directly. 25 26 ## Objectives 27 28 `skaffold` v2 is aimed at redesigning the skaffold workflow to decouple these two orthogonal concerns: 29 30 1. Hydration (aka “rendering”) of configuration: [A DRY configuration](#DRY-configuration) is transformed, generated, and validated into [WET configs](#WET-configuration). 31 32 2. Deployment (aka “apply”) of configuration: The WET configuration is applied to the K8S control plane. 33 34 This `skaffold` v2 will use [`kpt`](https://github.com/GoogleContainerTools/kpt), which provides the canonical API(s) and toolchain for configuration packaging, hydration, and deployment, to enable the `skaffold` config generation, transformation and validation in a new *"render"* stage and replace the `skaffold` "deploy" stage with more accurate pruning. 35 36 ## Background 37 38 ### An overview of the skaffold workflow 39 40 `skaffold`, as a centralized tool for application’s local development as well as CI/CD workflow, leverages `kubectl`, `kustomize`, `helm` or/and `kpt` for config management and deployment. The workflow is composed of a series of steps as below. 41 42  43 44 Users are expected to configure one “deployer” in the skaffold.yaml deploy section. “Deployer” refers to the tools used to manage and deploy the configurations. Currently skaffold supports 4 "deployers": `kpt`, `kustomize`, `helm`, `kubectl`. Below table presents the main commands running in each step when a certain deployer is chosen (trivial flags and options are ignored). 45 46  47  48 49 ### Problems 50 51 From the above table and some user feedback (e.g. [3905](https://github.com/GoogleContainerTools/skaffold/issues/3905) [4856](https://github.com/GoogleContainerTools/skaffold/issues/4856) [4356](https://github.com/GoogleContainerTools/skaffold/issues/4356), we can summarize the skaffold deployment issues as below: 52 53 #### [User facing] Confusing scope of “deployer” 54 55 From the above table, there’s a misalignment between the "deployer" name and the tool’s actual "deployment" functionality. 56 57 For instance, `kustomize` "deployer" has no deployment operations (it relies on `kubectl` to deploy), but it is called a "deployer"; The `kustomize` CLI does have config validation functionalities (via `kustomize config`), but the `kustomize` "deployer" has a no-op in the validation stage (Step 3). This "deployer" behavior confuses customers who understand `kustomize` and want to use `kustomize` in `skaffold`. 58 59 #### [User facing] Unexpected `skaffold` usage compound 60 61 Though `skaffold` allows users to use different "deployers" together in a single workflow, a mixed usage of "deployers" normally means users are seeking workarounds to fill in the gaps due to the differentiation of the deployers’ scope and design principle (`kustomize` does not do in-place file changes and has no real "deploy" feature, `kubectl` deploys without good handling on prune, `helm` is package management tool and is imperative). Mixing the deployers under the hood in the deploy section gives the *wrong* signals that: 62 63 - the deployers should be used as a mix. This will exacerbate each deployer’s known problems and make the skaffold more brittle. 64 - `skaffold` should handle the issues caused by the conflicts/misalignment among the "deployers". 65 66 #### Limitations of current kpt deployer 67 68 `skaffold` has [a kpt "deployer"](https://skaffold.dev/docs/references/yaml/#deploy-kpt) which can support basic declarative config validation and hydration. This kpt deployer (stable version) has been released 2020 Dec, designed and implementated based on kpt version v0.34.0, which version does not have the declarative hydration in Kptfile file (new features in the [kpt v1 release](https://github.com/GoogleContainerTools/kpt/releases)). 69 70 Thus, the current skaffold kpt "deployer" has the following limitations: 71 72 1. `kpt` deployer defines its own `kpt fn` stdin/stdout pipe to build up a validation pipeline and will not be compatible with `kpt` v1. 73 Since the `kpt` deployer wraps the `kpt fn` pipeline inside the `skaffold`, failovers may not always be handled properly. 74 2. Skaffold deployers has no config resource path DAG detection, which will be properly handled by kpt v1. 75 3. `kpt` deployer unifies kustomize into the kpt pipeline by switching the kyaml “ResourceList” to k8s resource and switching back after running kustomize build. These extra steps are not declarative native and sacrifices the flexibility of declarative config management (e.g. the order of the hydration in a pipeline is hard-coded). 76 4. `kpt` deployer adds some additional steps to handle the kpt version incompatibility issues. E.g. both config.kubernetes.io/function and config.k8s.io/function annotations are supported, network flag is required to be configured in both skaffold.yaml and the declarative kpt function file, resources with config.kubernetes.io/local-config annotation is removed from sinkDir resource. 77 5. `kpt` deployer does not have a package management solution for skaffold. 78 79 ## Proposal 80 81 To obtain the Objectives and to fix the issues listed in the background, we propose the following design: 82 83 Create a new **rendering** functionality to manage the configuration hydration. This "rendering" splits the hydration functionality from the existing "deployment" step and is configurable via a new "render" section in the `skaffold.yaml`. Both rendering and deployment fully relies on `kpt` as the *backbone*: 84 85 - Leverage the kpt definition to define "transformation" and "validation" operations in `skaffold.yaml`. 86 - Run each operation declarative in the `kpt` managed container runtime. 87 - Support package management through `kpt` package and declarative setters. 88 - kpt launches together with skaffold as a builtin tool. 89 - Deprecate the `helm` operation with the `kpt` "helm-inflator" image. 90 - Deprecate the kustomize operation with the kpt `kustomize` image, and unify the `kpt` and `kustomize` operations into the `kpt` declarative pipeline. 91 - Strengthen the "deploy" functionality (or step) by removing the existing hydration operation (e.g. kustomize build) and using `kpt` to apply the config to the cluster (e.g. no deployers are specified). 92 93 94 ## Detailed Design 95 96 The new skaffold *render* uses the `kpt` ["kustomize"](https://github.com/GoogleContainerTools/kpt/issues/1447) function and "helm-inflator" function in lieu of `kustomize` and `helm`. So as `kustomize` resource and `helm` charts can fit into the `kpt` hydration and work together with other kpt validation and mutation functions in a pipeline. 97 98 The new skaffold *deploy* only focuses on WET (See glossary) configurations and uses `kpt live` to deploy the config to the cluster. Here, `kpt` uses ResourceGroup and three-way merge pruning, so it is guaranteed the applied configs are accurate and reliable. 99 100 ### An overview of the scope and internal operation changes 101 102  103 104 ### New "render" 105 106 Add a new render (has decided to name to `.manifests`, the following graphs still using ".render") in the `skaffold.yaml`. This section contains two modes to hydrate configuration: One is to point to a declarative Kptfile file via `.manifests.kpt` (a.k.a kpt-managed), the other is to write the hydration pipeline directly in the `skaffold.yaml` (a.k.a skaffold-managed). 107 108 #### kpt-managed mode 109 110 The kpt-managed mode is for customers who are already the `kpt` hydration users (using the "pipeline" in Kptfile) and understand [declarative application management](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md). Thus, there’s no need to ask them to re-write the similar config in skaffold.yaml. 111 112 113  114 115 #### skaffold-managed mode 116 117  118 119 The skaffold-managed hydration has three types of operations: `generate`, `transform` and `validate`. The dry config is manipulated through each operation in the order as they are written in the `skaffold.yaml`. 120 121 - **generate** defines the source of the configurations. It contains the raw manifests (kubernetes resources), `kustomize` resources (kustomization.yaml files and base/overlay resources) and/or `helm` charts as the configuration source. Besides, It provides the option to fetch and update the `kpt` packages as resources. 122 123 *Note*: `helm` is only presented to users as "helm charts". Under the hood, the helm charts are managed by the `kpt`. See kpt [helm-inflator](https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/examples/contrib/helm-inflator) example (the image prvoides minimum helm features, more work is needed before it can be used in skaffold). This allows `skaffold` to unify different config resources and pass them all through the kpt validation pipeline. What’s more, users are no longer required to install `kustomize`, `helm` or `kpt` and to fix the version conflicts. 124 125 - **transform** defines the operations to mutate the configuration. Different than the `kpt` transformers, `skaffold` only accepts a **whitelisted set** of transformers. This is to control the `skaffold dev` qualities and gives a reliable render experience. 126 127 - **validate** defines the operations to validate the configuration. Same as transform, only a whitelisted set of validators are accepted. E.g. `kubeval`, `conftest`. 128 129 130 ### "render" interface in `skaffold.yaml` 131 132 ``` yaml 133 134 apiVersion: skaffold/v3 135 kind: Config 136 metadata: 137 name: skaffold-config 138 build: 139 140 render: 141 # Optional. kpt-managed mode. The path to the Kptfile. If provided, the other "render" options are not allowed. 142 kpt: [string] 143 144 # This field defines the dry manifests from a variety of sources. 145 generate: 146 147 # Optional. A list of paths to the raw manifests or/and kustomize directory. 148 manifests: [array] 149 150 151 # Optional. A list of helm charts to fetch. This is done via kpt helm-inflater image under the hood. 152 helmCharts: 153 154 # Required. The helm chart path. 155 - path: 156 157 # Required. The paths to the values.yaml files. 158 valuesFile: [array] 159 160 # Optional. This field defines a set of transformation operations to run in series 161 transform: 162 163 # Required. The transformer name. Can only accept skaffold whitelisted tools. 164 - name: [string] 165 166 # Optional. This field defines a set of validator operations to run in series. 167 validate: 168 169 # Required. The validator image name. Can only accept skaffold whitelisted tool, e.g. kubeval 170 - name: [string] 171 172 173 # Optional. The path to the hydrated directory. 174 Output: [string] 175 176 # Optional. The flags passed to the `kpt fn` pipeline command directly. 177 flags: ... 178 179 deploy: ... 180 ``` 181 182 #### An [example](https://gist.github.com/yuwenma/90610081f0b38a6002fb0b0f8b8567a3) of skaffold-managed hydration 183 184 #### An [example](https://gist.github.com/yuwenma/dbadad4ce420b1ae8bb3a374b6ad9667) of kpt-managed hydration 185 186 ### New “deploy” 187 188 The `deploy` section in `skaffold.yaml` will be greatly simplified: 189 190 - `skaffold` decides to not support `helm` installation (deploy step) and only focuses on `helm` charts (render step). This means no helm deployer is needed. 191 - The skaffold team have solicited feedback on whether users require Helm’s release functionality and the signal has been that it is not required or used. We will start without it and if needed add back the functionality. 192 - Deployment via Helm is supported by many CI/CD and GitOps tools. Some users also wrap helm with other tools. So they aren't necessarily deploying directly via skaffold. 193 - Only `kpt live` is used to apply the WET configurations and users do not specify the deployer name in the deploy section. 194 195 #### "deploy" interface in `skaffold.yaml` 196 197 ```yaml 198 apiVersion: skaffold/v3 199 kind: Config 200 metadata: 201 name: skaffold-config 202 build: 203 … 204 render: 205 … 206 deploy: 207 208 # Optional. The path to the WET configuration. It not provided, using the output from the render step. 209 dir: [String] 210 211 # Optional. The kpt live inventory ID 212 inventoryID: [String] 213 214 # Optional. The kpt inventory namespace 215 inventoryNamespace: [String] 216 217 # Optional. The `kpt live apply` flags. 218 # Unchanged fields. 219 options: ... 220 ``` 221 222 ### `skaffold` commands and flags 223 224 | command | `skaffold.yaml` | comments 225 | :----------- | :-----------: |-----------: | 226 | `skaffold render` | `render.output` not given| By default, hydrate and show the WET config in stdout `kpt pipeline run --sink-mode=stdout`| 227 | `skaffold render` | `render.output: [PATH]`| hydrate and store the WET config to PATH ` kpt pipeline run --sink-mode=special-dir --output-dir=PATH` | 228 | `skaffold render --output [PATH2]`| `render.output: [PATH1]` | Hydrate and store the WET config in PATH2 (override `skaffold.yaml`) `kpt pipeline run --sink-mode=special-dir --output-dir=PATH2`| 229 | `skaffold deploy` |`deploy.dir` not given|*Render* and deploy 230 `kpt pipeline run --sink-mode=stdout | kpt live apply --`| 231 | `skaffold deploy` |`deploy.dir: [PATH]` |Only deploy `kpt live apply [PATH]`| 232 |`skaffold deploy --render-path [PATH2]`| `deploy.dir: [PATH1]`|Deploy and override PATH1 `kpt live apply [PATH2]`| 233 | `skaffold dev` | | Build, render and deploy| 234 235 ### Integration of `kpt` pipeline and `skaffold` render 236 237 Either *kpt-managed* Pipeline or *skaffold-managed* Pipeline is driven by a `kpt fn render` command (see below). This requires a `Kptfile` file. The *kpt-managed* approach finds the `Kptfile` from `.render.kpt`, and the *skaffold-managed* approach creates the `Kptfile` file on the fly. 238 239  240 241 This `Kptfile` is built as shown below. Since kpt considers all resources under the `Kptfile` directory as the source. `skaffold` will create a temporary directory `.kpt-hydrated` to cache the `kustomize` DRY configs, `helm` charts, and the raw manifests. Before running `kpt`, these resources are copied to the temporary directory, and `kpt` will hydrate the configurations in place. 242 243 ``` yaml 244 apiVersion: kpt.dev/v1alpha1 245 kind: Kptfile 246 metadata: 247 name: skaffold-kpt-builtin-pipeline 248 pipeline: 249 mutators: 250 251 # items directly defined in skaffold.yaml. It can only be the tools skaffold whitelisted. 252 - image: <render.transform.customize[0].name> 253 # specific arguments that could defer per whitelisted tool. 254 configPath/configPath: 255 256 - image: <render.transform.customize[1].name> 257 ... 258 259 260 validators: 261 262 # items directly defined in skaffold.yaml. It can only be the tools skaffold whitelisted. E.g. kubeval. 263 - image: <render.validators.customize[0].name> 264 # specific arguments that could defer per whitelisted tool. 265 configPath/configPath: 266 267 - image: <render.validators.customize[1].name> 268 ... 269 270 ``` 271 272 The following graph describes the commands and operations to generate the kpt sources. 273 274  275 276 ### Integration of `kpt` live and `skaffold` deploy 277 278 `kpt` relies on `ResourceGroup` to accurately deploy the WET configurations to the cluster. This requires the ResourceGroup CRD to be installed on the cluster side. 279 280 `skaffold dev` and `skaffold deploy` commands will detect and ask for user permissions before installing the `ResourceGroup` controller. It calls the following `kpt` commands. 281 282  283 284 *Note*: `skaffold` v2 will not provide the migration from `kpt` v0.X to `kpt` v1. Users can follow [this guide](https://googlecontainertools.github.io/kpt/reference/live/migrate/) to establish the `kpt` migration themselves. 285 286 ### Builtin `kpt` 287 288 `skaffold` will have `kpt` embedded via go bindata, which ship together with the skaffold binary to guarantee the versions match. 289 290 Currently, `skaffold` requires users to install `helm`, `kubectl`, `kustomize` and `kpt` and manage the version skew themselves. By making `kpt` and `skaffold` a bundle, users no longer need to install ANY other tools (`helm` and `kustomize` are used as `kpt` functions). 291 292 ## Migration Plan 293 294 ### `Helm` "deployer" 295 296 Customers can choose either to continue using `helm` deployer or migrate to `kpt` in both "render" and "deploy" stages. We provide users an easy approach to migrate `helm` "deployer" to the new skaffold. But if users continue using `helm`, we will not provide no further support for the new `kpt` features. 297 298 This guarantees the `skaffold` v2 is backward compatible with `helm` and users have a seamless experience when introduced to the new render UI. 299 300 #### The migration command 301 302 Users can run the existing skaffold `fix` command to migrate to the new skaffold. 303 304 This will wipe and recreate the `skaffold.yaml` as below. After the migration, if users make a `helm` config change in `skaffold.yaml`, skaffold dev (and other skaffold commands enabling file watcher) will recreate the following kpt resources. 305 306  307 308 #### Under the hood 309 310 Though the new UI looks similar except moving helm from "deploy" to "render", the actual implementation is very different. Instead of calling the `helm` CLI, the `skaffold-helm-inflator` functions are used to convert the `Helm` charts to a kpt [`ResourceList.items`]（https://googlecontainertools.github.io/kpt/guides/producer/functions/#resourcelistitems). This function is expected to digest all the `helm` arguments listed in the `skaffold.yaml` reference doc [v2beta16](https://skaffold.dev/docs/references/yaml/#deploy-helm). 311 312 313 #### Phase 1 314 315 The [helm-inflator](https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/examples/contrib/helm-inflator）) is used and it requires the helm config to be in the form of a ConfigMap resource. Note, this image does not fit into the kpt v1 Kptfile yet. 316 317 ##### Step 1. Create `.kpt-pipeline` directory and run `kpt pkg init .kpt-pipeline/` 318 319  320 321 ##### Step 2. Create `.kpt-pipeline/config-helm.yaml` file from `skaffold.yaml` `.render.generate.helm` 322 323  324 325 ##### Step 3. Use the helm-inflator to convert the helm charts 326 327  328 329 ##### Step 4. Store the converted `helm` resource in the temporary directory 330 331 #### Phase 2 332 333 Most steps are similar to phase 1 except in step 3. Instead of calling the `kpt fn eval` command, we can edit the pipeline in `.kpt-pipeline/Kptfile` as below. (parameters may change, defer to the final helm-inflator-2 image) 334 335  336 337 #### Phase 3 338 339 The `kpt` function format (ConfigMap) and *helm-inflator* images may change significantly in the future. More migration changes will happen defer to the final design of helm-inflator function and the `kpt` function configuration. 340 341 ### `kustomize` "deployer" 342 343 `kustomize` deployer should be deprecated in three phases. The goal is to provide the `skaffold` users a seamless experience on migrating to skaffold V2. 344 345 #### The migration command 346 347 Users can run the existing `skaffold fix` command to migrate to the new skaffold. 348 349 This will wipe and recreate the skaffold.yaml as below. 350 351  352 353 #### Under the hood 354 355 ##### Phase 1 356 Providing a full `kustomize` image requires a lot of design, implementation (tracked [1447](https://github.com/GoogleContainerTools/kpt/issues/1447)) and testing. Before the kustomize image is available, `skaffold` commands continue using the `kustomize` CLI to build the kustomize configs and pass them through the `kpt` validation and mutation pipeline. 357 358 ##### Step 1. Create `.kpt-pipeline/` directory and run `kpt pkg init .kpt-pipeline/.` 359 360 Same as “helm deployer” migration step 1. 361 362 ##### Step 2. Run `kustomize build` 363 364 Walk through the `.render.generate.manifests` and check if `kustomization.yaml` files exist. If so, run `kustomize build {render.generate.manifests}` (this may require a tmp dir if manifests contain multiple resources) and store the stdout to `.kpt-pipeline/kustomize-dest` 365 366 #### Phase 2 367 368 Once the `kustomize` kpt function (tracked [1447](https://github.com/GoogleContainerTools/kpt/issues/1447) is available, define the kustomize in Kptfile. Step 1-3 in phase 1 can be skipped. 369 370  371 372 #### Phase 3 373 374 Defer to the unified kpt/kustomize, the kustomize and kpt may be further aligned with each other. And more migration changes are needed.