github.com/argoproj/argo-cd/v2@v2.10.5/docs/proposals/2022-07-13-appset-progressive-rollout-strategy.md (about)

     1  ---
     2  title: ApplicationSet Progressive Rollout Strategy
     3  authors:
     4    - "@wmgroot"
     5    - "@cnmcavoy"
     6  sponsors:
     7    - indeed.com
     8  reviewers:
     9    - "@alexmt"
    10    - TBD
    11  approvers:
    12    - "@alexmt"
    13    - TBD
    14  
    15  creation-date: 2022-07-13
    16  last-updated: 2022-08-11
    17  ---
    18  
    19  # ApplicationSet Progressive Rollout Strategy
    20  
    21  ## Summary
    22  
    23  Enhance the ArgoCD ApplicationSet resource to embed a rollout strategy for a progressive application resource update after the ApplicationSet spec or Application templates are modified.
    24  Further discussion and interest has been communicated here: https://github.com/argoproj/argo-cd/issues/9437
    25  
    26  ## Motivation
    27  
    28  As cluster operators, we would like to make changes to ApplicationSets which may target multiple environments, pre-defined staging areas, or other configurations, and have these changes rolled out in a declarative, defined manner rather than all at once as ApplicationSets currently behave. A progressive ApplicationSet rollout would prevent mistakes in configuration from having a larger blast radius than intended and give cluster operators a chance to verify and have confidence in their changes.
    29  
    30  ### Goals
    31  
    32  Users are able to make a single change to ApplicationSet that is updated across the generated Applications in a controlled manner. When this enhancement is enabled, Applications are updated in a declaractive order, instead of simultaneously.
    33  
    34  ### Non-Goals
    35  
    36  Handling controlled rollouts for changes to a helm chart or raw manifests referenced by the Applications managed by the ApplicationSet. We understand this would be valuable, but we would like to implement the rollout implementation handling only changes to the ApplicationSet initially.
    37  
    38  ## Proposal
    39  
    40  This is where we get down to details of what the proposal is about.
    41  
    42  ### Use cases
    43  
    44  Add a list of detailed use cases this enhancement intends to take care of.
    45  
    46  #### Use case 1:
    47  As a user, I would like to declaratively control the rollout order of ApplicationSet changes to its generated Application resources.
    48  
    49  We propose adding a `RollingUpdate` and `RollingSync` strategy spec (taking inspiration from other controllers).
    50  
    51  The rolling update strategy deterministically chooses applications to update following a maxUpdate value. If maxUpdate is set to 1, then applications are updated one by one, proceeding each step only if the previous application syncs completed successfully. If set to more than 1, then applications are updated in parallel up to that number.
    52  Steps for the rolling update are defined by a list of matchExpression label selectors. Each step must finish updating before the next step advances. If steps are left undefined the application update order is deterministic.
    53  
    54  Complete ApplicationSet spec example.
    55  ```
    56  apiVersion: argoproj.io/v1alpha1
    57  kind: ApplicationSet
    58  metadata:
    59    name: guestbook
    60  spec:
    61    generators:
    62    - list:
    63        elements:
    64        - cluster: engineering-dev
    65          url: https://1.2.3.4
    66          env: dev
    67        - cluster: engineering-prod
    68          url: https://2.4.6.8
    69          env: prod
    70        - cluster: engineering-qa
    71          url: https://9.8.7.6/
    72          env: qa
    73    strategy:
    74      type: RollingUpdate
    75      rollingUpdate:
    76        steps:
    77          - matchExpressions:
    78              - key: env
    79                operator: In
    80                values:
    81                  - dev
    82            maxUpdate: 0 # if undefined or 0, all applications matched are updated together
    83          - matchExpressions:
    84              - key: env
    85                operator: In
    86                values:
    87                  - qa
    88          - matchExpressions:
    89              - key: env
    90                operator: In
    91                values:
    92                  - us-east-2
    93                  - eu-west-1
    94                  - ap-southeast-1
    95            maxUpdate: 1 # maxUpdate supports both integer and percentage string values
    96    template:
    97      metadata:
    98        name: '{{cluster}}-guestbook'
    99        labels:
   100          env: "{{env}}"                                # label can be provided explicitly from a list generator
   101          region: "{{metadata.labels.cluster/region}}"  # or pulled from labels on the argo cluster secrets
   102      spec:
   103        source:
   104          repoURL: https://github.com/infra-team/cluster-deployments.git
   105          targetRevision: HEAD
   106          path: guestbook/{{cluster}}
   107        destination:
   108          server: '{{url}}'
   109          namespace: guestbook
   110  ```
   111  
   112  In the above example, when the guestbook ApplicationSet is created or modified, the Application resources are each updated in the order defined in `strategy.rollingUpdate`. In this case, all generated Applications (applied or not) with a label that matches the expression `env: dev` are updated to match the template. All Applications in this step are updated in parallel, because the `maxUpdate` is set to zero. The rolling update strategy progresses after the first set of Applications has successfully progressed and become healthy again. Progress towards the next step starts only after the current step has completely finished, regardless of the `maxUpdate` value. The `maxUpdate` field only throttles the total number of matching Applications updating in the current step. After the first step completes, the ApplicationSet updates all Application resources with label `env: qa` at the same time, because `maxUpdate` is undefined. Finally, during the third step, the Application resources labeled `region: us-east-2`, `region: eu-west-1`, or `region: ap-southeast-1` are updated, one by one, as the `maxUpdate` for the final step is 1.
   113  
   114  An Application rollout is considered “complete” when the Application resource has been:
   115  - Synced successfully.
   116  - Moved into a “Progressing” state.
   117  - Moved out of a “Progressing” state and into a “Healthy” state.
   118  
   119  `RollingSync` operates using the same spec, but is a re-implementation of the https://github.com/Skyscanner/applicationset-progressive-sync tool. It notices that Applications become OutOfSync, and triggers sync operations on those Applications following the order declared in the Application strategy spec.
   120  
   121  
   122  #### Use case 2:
   123  As a user, I would like to continue to use the current simultaneous Application update behavior of the ApplicationSet controller.
   124  
   125  If no strategy is provided, we propose defaulting to an `AllAtOnce` strategy, which maintains the current behavior.
   126  
   127  
   128  ### Implementation Details/Notes/Constraints [optional]
   129  
   130  #### Initial ApplicationSet Creation
   131  Application resource creation from an ApplicationSet with a defined strategy looks much like the update process. When a brand new ApplicationSet is first created with a rollout strategy specified, the desired Application resource metadata labels are used to determine when each Application resource is created. Each Application created will be created in the order defined by the steps, if any, and advance to the next step only when a step completes successfully. The same applies if an ApplicationSet is modified to target a different set of destination clusters or namespaces, Applications are created or updated in the order defined by their desired state and the defined step order in the strategy.
   132  
   133  #### ApplicationSet Rollout Failure
   134  In the event that an ApplicationSet spec or template is modified and a target Application resource fails to “complete” a sync in any of the steps, the ApplicationSet rollout is stalled. The ApplicationSet resource will ensure the status field for “ApplicationSetUpToDate” is False. If the maxUpdate allows it, the ApplicationSet will continue updating Applications in the current step, but otherwise, no further changes will be propagated to Application resources by the ApplicationSet, and no steps will advance until each Application can successfully complete a sync. If the ApplicationSet is modified while still in the midst of an ApplicationSet rollout, stalled or otherwise, then the existing rollout is abandoned, the application resources are left in their present state, and the new rollout begins.
   135  
   136  #### "Pausing" Application Changes During Rollout
   137  To implement the “paused” functionality of Applications that are not yet ready to be updated, we have a few options.
   138  * Disable auto-sync.
   139  ** Potentially conflicts with user provided auto-sync settings.
   140  ** Provides the benefit of being able to see the full diff of the ApplicationSet change.
   141  * “Pause” the Application.
   142  ** Not Yet Implemented: https://github.com/argoproj/argo-cd/issues/4808
   143  * Prevent any updates at all to the live Applications via the rolling update strategy defined.
   144  ** This is likely the initial implementation method we'll target.
   145  
   146  #### Draft Pull Request
   147  This PR is now functional and ready for comment. We are actively working on unit tests and documentation.
   148  https://github.com/wmgroot/argo-cd/pull/1
   149  
   150  ### Security Considerations
   151  We do not believe this proposal results in any new security considerations for the ApplicationSet controller.
   152  
   153  ### Risks and Mitigations
   154  
   155  If this proposal is implemented, I believe the next logical step would be to solve the case where users would like to control rollout order for Application resources with a consistent specification, but changes being pushed to the upstream `source` of the Application. A common use case is an update to an unversioned "wrapper" helm chart that depends on a versioned upstream chart. The wrapper chart is often used to apply simple supplementary resources in a gitops pattern, such as company specific RBAC configuration, or ExternalSecrets configuration. These supplementary resources do not typically warrant publishing a versioned wrapper chart, making it difficult to implement changes to the chart's templates or value files and roll them out in an ordered way with the ApplicationSet changes discussed here.
   156  
   157  Implementing progressive rollout stragies to handled changes upstream of the generated Application source could be difficult, since the applicationset controller would need to intercept the sync operation of the Application to prevent the changes from syncing automatically.
   158  
   159  Added maintenance burden on the ArgoCD team is always a risk with the addition of new features.
   160  
   161  ### Upgrade / Downgrade Strategy
   162  
   163  We are introducing new fields to the ApplicationSet CRD, however no existing fields are being changed. We believe this means that a new ApplicationSet version is unnecessary, and that upgrading to the new spec with extra fields will be a clean operation.
   164  
   165  Downgrading would risk users receiving K8s API errors if they continue to try to apply the `strategy` field to a downgraded version of the ApplicationSet resource.
   166  Downgrading the controller while keeping the upgraded version of the CRD should cleanly downgrade/revert the behavior of the controller to the previous version without requiring users to adjust their existing ApplicationSet specs.
   167  
   168  ## Drawbacks
   169  
   170  The idea is to find the best form of an argument why this enhancement should _not_ be implemented.
   171  
   172  ## Alternatives
   173  
   174  One alternative we considered was to create an extra CRD specifically to govern the rollout process for an ApplicationSet. We ultimately decided against this approach because all other rollout strategy specs we looked at were implemented in the same CRD resource (K8s Deployments, Argo Rollouts, CAPI MachineDeployments, etc).
   175  
   176  Another alternative is to implement Application Dependencies through the application-controller instead. This is a far more complicated approach that requires implementing and maintaining an Application DAG.
   177  https://github.com/argoproj/argo-cd/issues/7437