github.com/argoproj/argo-cd/v2@v2.10.5/docs/proposals/applicationset-plugin-generator.md (about)

     1  ---
     2  title: applicationset-plugin-generator
     3  authors:
     4    - "@binboum"
     5    - "@scrocquesel"
     6  sponsors:
     7    - TBD
     8  reviewers:
     9    - TBD
    10  approvers:
    11    - "@alexmt"
    12    - TBD
    13  
    14  creation-date: 2022-03-21
    15  last-updated: 2022-03-21
    16  ---
    17  
    18  # ApplicationSet `plugin` generator
    19  
    20  Provide a generator that request its values through a RPC call.
    21  
    22  ## Summary
    23  
    24  ApplicationSet generators are useful for modeling templates using external data sources to deploy applications.
    25  
    26  Today, generators have been developed based on the needs of the community, and when a new need arises, it's necessary to modify the Appset codebase.
    27  
    28  The proposal here is to have a "plugin" generator that would allow extending the codebase according to specific needs, without having to modify it directly.
    29  
    30  ## Motivation
    31  
    32  Using the current generators, we sometimes encounter a need that arises, which may or may not be useful for the community. In such cases, several procedures need to be undertaken to make the modification, and sometimes it may be rejected because it's not in everyone's interest.
    33  
    34  The plugin approach also reduces the burden on community developers by externalizing feature requests into plugins that are outside the Appset controller's scope. From a security and scalability perspective, this can be advantageous.
    35  
    36  With this approach, it becomes possible to offer a catalog of plugins and encourage people with specific needs to develop standalone plugins that are independent of the controller's codebase.
    37  
    38  ### Goals
    39  
    40  Empowering community developers to develop and use plugins that extend the list of generators can be a significant advantage. It would be possible to offer a page listing plugins maintained by the community, which can help promote the development of a rich ecosystem of plugins for various use cases. This can enhance the overall user experience by providing more options for generating application templates.
    41  
    42  Additionally, allowing developers to create plugins and share them with the community can foster innovation and encourage experimentation with new features and functionalities. It can also reduce the workload on the Appset development team, enabling them to focus on core features and functionalities.
    43  
    44  Overall, giving autonomy to community developers through plugins is a practical way to enhance the Appset platform and provide more value to users.
    45  
    46  ### Non-Goals
    47  
    48  The concept of the plugin should not undermine the spirit of GitOps by externalizing data outside of Git. The goal is to be complementary in specific contexts.
    49  
    50  For example, when using one of the PullRequest generators, it's impossible to retrieve parameters related to the CI (only the commit hash is available), which limits the possibilities. By using a plugin, it's possible to retrieve the necessary parameters from a separate data source and use them to extend the functionality of the generator. This approach allows for greater flexibility and can help overcome limitations imposed by GitOps.
    51  
    52  Overall, the use of plugins should be considered as a way to enhance the capabilities of existing tools and processes rather than as a replacement for them. By leveraging plugins, developers can take advantage of the strengths of different tools and technologies, resulting in a more robust and flexible development process.
    53  
    54  ## Proposal
    55  
    56  ### Add a new `generator` plugin
    57  
    58  ```
    59  apiVersion: argoproj.io/v1alpha1
    60  kind: ApplicationSet
    61  metadata:
    62    name: fb-plugin
    63    namespace: argo-system
    64  spec:
    65    generators:
    66      - plugin:
    67          configMapRef: fb-plugin
    68          name: feature-branch-plugin
    69          params:
    70            repo: "my-repo"
    71            branch: "my-branch"
    72          requeueAfterSeconds: 10
    73    template:
    74  ...
    75  ```
    76  
    77  ### Add a configMap to configure the plugin
    78  
    79  The configMap name must match the configMapRef value in the plugin configuration. The configMap must be in the namespace of argo.
    80  
    81  ```
    82  apiVersion: v1
    83  kind: ConfigMap
    84  metadata:
    85    name: fb-plugin
    86    namespace: argo-system
    87  data:
    88    token: $plugin.myplugin.token # Alternatively $<some_K8S_secret>:plugin.myplugin.token
    89    baseUrl: http://myplugin.plugin.svc.cluster.local
    90  ```
    91  
    92  - token is used a a bearer token in the RPC request. It could be a [sensitive reference](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#sensitive-data-and-sso-client-secrets).
    93  
    94  ### Reconciliation logic
    95  
    96  Here is a diagram describing what the plugin generator should do to get the params to return:
    97  
    98  ```mermaid
    99  sequenceDiagram
   100      alt generator is plugin
   101      Generator->>K8S: Get configmap {configMapRef}
   102      K8S-->>Generator: (url,token)
   103      Generator->>Plugin endpoint: POST {url}/v1/generator.getParams<br/>Authorization: Bearer {token}<br/>Content-Type: application/json<br/>{params}
   104      Plugin endpoint-->>Generator: []map{string}interface{}
   105      end 
   106  ```
   107  
   108  
   109  ### Use cases
   110  
   111  #### Use case 1:
   112  As a user, I would like to enrich PullRequest generator params with digests of images generated by the pull request CI pipeline.
   113  
   114  I could define a generator matrix like
   115  
   116  ```yaml
   117    generators:
   118      - matrix:
   119          generators:
   120            - pullRequest:
   121                github:
   122                  owner: binboum
   123                  repo: argo-test
   124                  labels:
   125                  - preview-matrix
   126                  tokenRef:
   127                    secretName: github-secret
   128                    key: token
   129            - plugin:
   130                configMapRef: cm-plugin
   131                name: plugin-matrix
   132                params:
   133                  repo: "argo-test"
   134                  branch: "{{.branch}}"
   135  ```
   136  
   137  When pullRequest returns a new PR matching my labels, the plugin will be called with the branch name and would return a set of digests like
   138  
   139  ```json
   140  [
   141    {
   142      "digestFront": "xxxxxxxx",
   143      "digestBack": "xxxxxxxx",
   144    }
   145  ]
   146  ```
   147  
   148  Values can then be used in the template section :
   149  
   150  ```yaml
   151    template:
   152      metadata:
   153        name: "fb-matrix-{{.branch}}"
   154      spec:
   155        source:
   156          repoURL: "git@github.com:binboum/argo-test.git"
   157          targetRevision: "HEAD"
   158          path: charts/app-client
   159          helm:
   160            releaseName: feature-test-matrix-{{.branch}}
   161            valueFiles:
   162              - values.yaml
   163            values: |
   164              front:
   165                image: registry.my/argo-test/front:{{.branch}}@{{ .digestFront }}
   166              back:
   167                image: registry.my/argo-test/back:{{.branch}}@{{ .digestBack }}
   168        destination:
   169          server: https://kubernetes.default.svc
   170          namespace: "{{.branch}}"
   171  ```
   172  
   173  ### Detailed examples
   174  
   175  ### Security Considerations
   176  
   177  * Plugin server only has access to the params content. When deployed outside of the applicationset controller pod, operator must ensure the communication between applicationset controller and the plugin server is properly secured (https/network policy...). A few authentication mechanism are handled to help the plugin server authenticate the request.
   178  * For now, the response payload is considered trusted and returned params are used as-is upstream
   179  
   180  ### Risks and Mitigations
   181  
   182  TBD
   183  
   184  ### Upgrade / Downgrade Strategy
   185  
   186  On the evolution of the plugin, and calls :
   187  
   188  The RPC method is standardized with a versioning system, which allows for a version parameter to be included in the API call. This makes it possible to avoid breaking changes in case of architecture changes in the future.
   189  
   190  Thought that the contract interface with the plugin server is kept simple to reduce future changes and breaking changes
   191  
   192  ## Drawbacks
   193  
   194  No idea
   195  
   196  ## Alternatives
   197  
   198  1. A design similar to Argo Workflow executor plugin :
   199  
   200      ```
   201      generators:
   202      - plugin:
   203          hello: {}
   204      ```
   205  
   206      A set of ConfigMaps or a specific CRDs to express configuration of the plugin endpoint would be walk by ApplicationSet server. For each configuration, call the plugin endpoint with the content of plugin until one return a valid response.
   207      
   208      Reconciliation should be fast as fast as possible and trying out every endpoint to figure out which one is able to handle the plugin payload could induce a lot of delay.
   209     
   210      Configuration rely on implicit and weakly typed convention which make the usage of the plugin less self documented.
   211  
   212  2. Plugin server as defacto sidecars
   213  
   214      Some magic could have inject a container image for the plugin in the ApplicationSet controller in a similar way, Argo Workflow does when creating a pod to execute a job.
   215  
   216      Require an external controler or manual configuration. The plugin would not scale independently of the ApplicationSet controller.