github.com/argoproj/argo-cd/v2@v2.10.9/docs/operator-manual/applicationset/Generators-SCM-Provider.md (about)

     1  # SCM Provider Generator
     2  
     3  The SCM Provider generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover repositories within an organization. This fits well with GitOps layout patterns that split microservices across many repositories.
     4  
     5  ```yaml
     6  apiVersion: argoproj.io/v1alpha1
     7  kind: ApplicationSet
     8  metadata:
     9    name: myapps
    10  spec:
    11    generators:
    12    - scmProvider:
    13        # Which protocol to clone using.
    14        cloneProtocol: ssh
    15        # See below for provider specific options.
    16        github:
    17          # ...
    18  ```
    19  
    20  * `cloneProtocol`: Which protocol to use for the SCM URL. Default is provider-specific but ssh if possible. Not all providers necessarily support all protocols, see provider documentation below for available options.
    21  
    22  !!! note
    23      Know the security implications of using SCM generators. [Only admins may create ApplicationSets](./Security.md#only-admins-may-createupdatedelete-applicationsets)
    24      to avoid leaking Secrets, and [only admins may create repos/branches](./Security.md#templated-project-field) if the
    25      `project` field of an ApplicationSet with an SCM generator is templated, to avoid granting management of
    26      out-of-bounds resources.
    27  
    28  ## GitHub
    29  
    30  The GitHub mode uses the GitHub API to scan an organization in either github.com or GitHub Enterprise.
    31  
    32  ```yaml
    33  apiVersion: argoproj.io/v1alpha1
    34  kind: ApplicationSet
    35  metadata:
    36    name: myapps
    37  spec:
    38    generators:
    39    - scmProvider:
    40        github:
    41          # The GitHub organization to scan.
    42          organization: myorg
    43          # For GitHub Enterprise:
    44          api: https://git.example.com/
    45          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
    46          allBranches: true
    47          # Reference to a Secret containing an access token. (optional)
    48          tokenRef:
    49            secretName: github-token
    50            key: token
    51          # (optional) use a GitHub App to access the API instead of a PAT.
    52          appSecretName: gh-app-repo-creds
    53    template:
    54    # ...
    55  ```
    56  
    57  * `organization`: Required name of the GitHub organization to scan. If you have multiple organizations, use multiple generators.
    58  * `api`: If using GitHub Enterprise, the URL to access it.
    59  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
    60  * `tokenRef`: A `Secret` name and key containing the GitHub access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
    61  * `appSecretName`: A `Secret` name containing a GitHub App secret in [repo-creds format][repo-creds].
    62  
    63  [repo-creds]: ../declarative-setup.md#repository-credentials
    64  
    65  For label filtering, the repository topics are used.
    66  
    67  Available clone protocols are `ssh` and `https`.
    68  
    69  ## Gitlab
    70  
    71  The GitLab mode uses the GitLab API to scan and organization in either gitlab.com or self-hosted GitLab.
    72  
    73  ```yaml
    74  apiVersion: argoproj.io/v1alpha1
    75  kind: ApplicationSet
    76  metadata:
    77    name: myapps
    78  spec:
    79    generators:
    80    - scmProvider:
    81        gitlab:
    82          # The base GitLab group to scan.  You can either use the group id or the full namespaced path.
    83          group: "8675309"
    84          # For self-hosted GitLab:
    85          api: https://gitlab.example.com/
    86          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
    87          allBranches: true
    88          # If true, recurses through subgroups. If false, it searches only in the base group. Defaults to false.
    89          includeSubgroups: true
    90          # If true and includeSubgroups is also true, include Shared Projects, which is gitlab API default.
    91          # If false only search Projects under the same path. Defaults to true.
    92          includeSharedProjects: false
    93          # filter projects by topic. A single topic is supported by Gitlab API. Defaults to "" (all topics).
    94          topic: "my-topic"
    95          # Reference to a Secret containing an access token. (optional)
    96          tokenRef:
    97            secretName: gitlab-token
    98            key: token
    99          # If true, skips validating the SCM provider's TLS certificate - useful for self-signed certificates.
   100          insecure: false
   101    template:
   102    # ...
   103  ```
   104  
   105  * `group`: Required name of the base GitLab group to scan. If you have multiple base groups, use multiple generators.
   106  * `api`: If using self-hosted GitLab, the URL to access it.
   107  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   108  * `includeSubgroups`: By default (false) the controller will only search for repos directly in the base group. If this is true, it will recurse through all the subgroups searching for repos to scan.
   109  * `includeSharedProjects`: If true and includeSubgroups is also true, include Shared Projects, which is gitlab API default. If false only search Projects under the same path. In general most would want the behaviour when set to false. Defaults to true.
   110  * `topic`: filter projects by topic. A single topic is supported by Gitlab API. Defaults to "" (all topics).
   111  * `tokenRef`: A `Secret` name and key containing the GitLab access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
   112  * `insecure`: By default (false) - Skip checking the validity of the SCM's certificate - useful for self-signed TLS certificates.
   113  
   114  For label filtering, the repository topics are used.
   115  
   116  Available clone protocols are `ssh` and `https`.
   117  
   118  ### Self-signed TLS Certificates
   119  
   120  As a preferable alternative to setting `insecure` to true, you can configure self-signed TLS certificates for Gitlab.
   121  
   122  In order for a self-signed TLS certificate be used by an ApplicationSet's SCM / PR Gitlab Generator, the certificate needs to be mounted on the applicationset-controller. The path of the mounted certificate must be explicitly set using the environment variable `ARGOCD_APPLICATIONSET_CONTROLLER_SCM_ROOT_CA_PATH` or alternatively using parameter `--scm-root-ca-path`. The applicationset controller will read the mounted certificate to create the Gitlab client for SCM/PR Providers
   123  
   124  This can be achieved conveniently by setting `applicationsetcontroller.scm.root.ca.path` in the argocd-cmd-params-cm ConfigMap. Be sure to restart the ApplicationSet controller after setting this value.
   125  
   126  ## Gitea
   127  
   128  The Gitea mode uses the Gitea API to scan organizations in your instance
   129  
   130  ```yaml
   131  apiVersion: argoproj.io/v1alpha1
   132  kind: ApplicationSet
   133  metadata:
   134    name: myapps
   135  spec:
   136    generators:
   137    - scmProvider:
   138        gitea:
   139          # The Gitea owner to scan.
   140          owner: myorg
   141          # The Gitea instance url
   142          api: https://gitea.mydomain.com/
   143          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
   144          allBranches: true
   145          # Reference to a Secret containing an access token. (optional)
   146          tokenRef:
   147            secretName: gitea-token
   148            key: token
   149    template:
   150    # ...
   151  ```
   152  
   153  * `owner`: Required name of the Gitea organization to scan. If you have multiple organizations, use multiple generators.
   154  * `api`: The URL of the Gitea instance you are using.
   155  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   156  * `tokenRef`: A `Secret` name and key containing the Gitea access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
   157  * `insecure`: Allow for self-signed TLS certificates.
   158  
   159  This SCM provider does not yet support label filtering
   160  
   161  Available clone protocols are `ssh` and `https`.
   162  
   163  ## Bitbucket Server
   164  
   165  Use the Bitbucket Server API (1.0) to scan repos in a project. Note that Bitbucket Server is not to same as Bitbucket Cloud (API 2.0)
   166  
   167  ```yaml
   168  apiVersion: argoproj.io/v1alpha1
   169  kind: ApplicationSet
   170  metadata:
   171    name: myapps
   172  spec:
   173    generators:
   174    - scmProvider:
   175        bitbucketServer:
   176          project: myproject
   177          # URL of the Bitbucket Server. Required.
   178          api: https://mycompany.bitbucket.org
   179          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
   180          allBranches: true
   181          # Credentials for Basic authentication. Required for private repositories.
   182          basicAuth:
   183            # The username to authenticate with
   184            username: myuser
   185            # Reference to a Secret containing the password or personal access token.
   186            passwordRef:
   187              secretName: mypassword
   188              key: password
   189          # Support for filtering by labels is TODO. Bitbucket server labels are not supported for PRs, but they are for repos
   190    template:
   191    # ...
   192  ```
   193  
   194  * `project`: Required name of the Bitbucket project
   195  * `api`: Required URL to access the Bitbucket REST api.
   196  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   197  
   198  If you want to access a private repository, you must also provide the credentials for Basic auth (this is the only auth supported currently):
   199  * `username`: The username to authenticate with. It only needs read access to the relevant repo.
   200  * `passwordRef`: A `Secret` name and key containing the password or personal access token to use for requests.
   201  
   202  Available clone protocols are `ssh` and `https`.
   203  
   204  ## Azure DevOps
   205  
   206  Uses the Azure DevOps API to look up eligible repositories based on a team project within an Azure DevOps organization.
   207  The default Azure DevOps URL is `https://dev.azure.com`, but this can be overridden with the field `azureDevOps.api`.
   208  
   209  ```yaml
   210  apiVersion: argoproj.io/v1alpha1
   211  kind: ApplicationSet
   212  metadata:
   213    name: myapps
   214  spec:
   215    generators:
   216    - scmProvider:
   217        azureDevOps:
   218          # The Azure DevOps organization.
   219          organization: myorg
   220          # URL to Azure DevOps. Optional. Defaults to https://dev.azure.com.
   221          api: https://dev.azure.com
   222          # If true, scan every branch of eligible repositories. If false, check only the default branch of the eligible repositories. Defaults to false.
   223          allBranches: true
   224          # The team project within the specified Azure DevOps organization.
   225          teamProject: myProject
   226          # Reference to a Secret containing the Azure DevOps Personal Access Token (PAT) used for accessing Azure DevOps.
   227          accessTokenRef:
   228            secretName: azure-devops-scm
   229            key: accesstoken
   230    template:
   231    # ...
   232  ```
   233  
   234  * `organization`: Required. Name of the Azure DevOps organization.
   235  * `teamProject`: Required. The name of the team project within the specified `organization`.
   236  * `accessTokenRef`: Required. A `Secret` name and key containing the Azure DevOps Personal Access Token (PAT) to use for requests.
   237  * `api`: Optional. URL to Azure DevOps. If not set, `https://dev.azure.com` is used.
   238  * `allBranches`: Optional, default `false`. If `true`, scans every branch of eligible repositories. If `false`, check only the default branch of the eligible repositories.
   239  
   240  ## Bitbucket Cloud
   241  
   242  The Bitbucket mode uses the Bitbucket API V2 to scan a workspace in bitbucket.org.
   243  
   244  ```yaml
   245  apiVersion: argoproj.io/v1alpha1
   246  kind: ApplicationSet
   247  metadata:
   248    name: myapps
   249  spec:
   250    generators:
   251    - scmProvider:
   252        bitbucket:
   253          # The workspace id (slug).  
   254          owner: "example-owner"
   255          # The user to use for basic authentication with an app password.
   256          user: "example-user"
   257          # If true, scan every branch of every repository. If false, scan only the main branch. Defaults to false.
   258          allBranches: true
   259          # Reference to a Secret containing an app password.
   260          appPasswordRef:
   261            secretName: appPassword
   262            key: password
   263    template:
   264    # ...
   265  ```
   266  
   267  * `owner`: The workspace ID (slug) to use when looking up repositories.
   268  * `user`: The user to use for authentication to the Bitbucket API V2 at bitbucket.org.
   269  * `allBranches`: By default (false) the template will only be evaluated for the main branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   270  * `appPasswordRef`: A `Secret` name and key containing the bitbucket app password to use for requests.
   271  
   272  This SCM provider does not yet support label filtering
   273  
   274  Available clone protocols are `ssh` and `https`.
   275  
   276  ## AWS CodeCommit (Alpha)
   277  
   278  Uses AWS ResourceGroupsTagging and AWS CodeCommit APIs to scan repos across AWS accounts and regions.
   279  
   280  ```yaml
   281  apiVersion: argoproj.io/v1alpha1
   282  kind: ApplicationSet
   283  metadata:
   284    name: myapps
   285  spec:
   286    generators:
   287      - scmProvider:
   288          awsCodeCommit:
   289            # AWS region to scan repos.
   290            # default to the environmental region from ApplicationSet controller.
   291            region: us-east-1
   292            # AWS role to assume to scan repos.
   293            # default to the environmental role from ApplicationSet controller.
   294            role: arn:aws:iam::111111111111:role/argocd-application-set-discovery
   295            # If true, scan every branch of every repository. If false, scan only the main branch. Defaults to false.
   296            allBranches: true
   297            # AWS resource tags to filter repos with.
   298            # see https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html#resourcegrouptagging-GetResources-request-TagFilters for details
   299            # default to no tagFilters, to include all repos in the region.
   300            tagFilters:
   301              - key: organization
   302                value: platform-engineering
   303              - key: argo-ready
   304    template:
   305    # ...
   306  ```
   307  
   308  * `region`: (Optional) AWS region to scan repos. By default, use ApplicationSet controller's current region.
   309  * `role`: (Optional) AWS role to assume to scan repos. By default, use ApplicationSet controller's current role.
   310  * `allBranches`: (Optional) If `true`, scans every branch of eligible repositories. If `false`, check only the default branch of the eligible repositories. Default `false`.
   311  * `tagFilters`: (Optional) A list of tagFilters to filter AWS CodeCommit repos with. See [AWS ResourceGroupsTagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html#resourcegrouptagging-GetResources-request-TagFilters) for details. By default, no filter is included.
   312  
   313  This SCM provider does not support the following features
   314  
   315  * label filtering
   316  * `sha`, `short_sha` and `short_sha_7` template parameters
   317  
   318  Available clone protocols are `ssh`, `https` and `https-fips`.
   319  
   320  ### AWS IAM Permission Considerations
   321  
   322  In order to call AWS APIs to discover AWS CodeCommit repos, ApplicationSet controller must be configured with valid environmental AWS config, like current AWS region and AWS credentials.
   323  AWS config can be provided via all standard options, like Instance Metadata Service (IMDS), config file, environment variables, or IAM roles for service accounts (IRSA).
   324  
   325  Depending on whether `role` is provided in `awsCodeCommit` property, AWS IAM permission requirement is different.
   326  
   327  #### Discover AWS CodeCommit Repositories in the same AWS Account as ApplicationSet Controller
   328  
   329  Without specifying `role`, ApplicationSet controller will use its own AWS identity to scan AWS CodeCommit repos.
   330  This is suitable when you have a simple setup that all AWS CodeCommit repos reside in the same AWS account as your Argo CD.
   331  
   332  As the ApplicationSet controller AWS identity is used directly for repo discovery, it must be granted below AWS permissions.
   333  
   334  * `tag:GetResources`
   335  * `codecommit:ListRepositories`
   336  * `codecommit:GetRepository`
   337  * `codecommit:GetFolder`
   338  * `codecommit:ListBranches`
   339  
   340  #### Discover AWS CodeCommit Repositories across AWS Accounts and Regions
   341  
   342  By specifying `role`, ApplicationSet controller will first assume the `role`, and use it for repo discovery.
   343  This enables more complicated use cases to discover repos from different AWS accounts and regions.
   344  
   345  The ApplicationSet controller AWS identity should be granted permission to assume target AWS roles.
   346  
   347  * `sts:AssumeRole`
   348  
   349  All AWS roles must have repo discovery related permissions.
   350  
   351  * `tag:GetResources`
   352  * `codecommit:ListRepositories`
   353  * `codecommit:GetRepository`
   354  * `codecommit:GetFolder`
   355  * `codecommit:ListBranches`
   356  
   357  ## Filters
   358  
   359  Filters allow selecting which repositories to generate for. Each filter can declare one or more conditions, all of which must pass. If multiple filters are present, any can match for a repository to be included. If no filters are specified, all repositories will be processed.
   360  
   361  ```yaml
   362  apiVersion: argoproj.io/v1alpha1
   363  kind: ApplicationSet
   364  metadata:
   365    name: myapps
   366  spec:
   367    generators:
   368    - scmProvider:
   369        filters:
   370        # Include any repository starting with "myapp" AND including a Kustomize config AND labeled with "deploy-ok" ...
   371        - repositoryMatch: ^myapp
   372          pathsExist: [kubernetes/kustomization.yaml]
   373          labelMatch: deploy-ok
   374        # ... OR include any repository starting with "otherapp" AND a Helm folder and doesn't have file disabledrepo.txt.
   375        - repositoryMatch: ^otherapp
   376          pathsExist: [helm]
   377          pathsDoNotExist: [disabledrepo.txt]
   378    template:
   379    # ...
   380  ```
   381  
   382  * `repositoryMatch`: A regexp matched against the repository name.
   383  * `pathsExist`: An array of paths within the repository that must exist. Can be a file or directory.
   384  * `pathsDoNotExist`: An array of paths within the repository that must not exist. Can be a file or directory.
   385  * `labelMatch`: A regexp matched against repository labels. If any label matches, the repository is included.
   386  * `branchMatch`: A regexp matched against branch names.
   387  
   388  ## Template
   389  
   390  As with all generators, several parameters are generated for use within the `ApplicationSet` resource template.
   391  
   392  ```yaml
   393  apiVersion: argoproj.io/v1alpha1
   394  kind: ApplicationSet
   395  metadata:
   396    name: myapps
   397  spec:
   398    goTemplate: true
   399    goTemplateOptions: ["missingkey=error"]
   400    generators:
   401    - scmProvider:
   402      # ...
   403    template:
   404      metadata:
   405        name: '{{ .repository }}'
   406      spec:
   407        source:
   408          repoURL: '{{ .url }}'
   409          targetRevision: '{{ .branch }}'
   410          path: kubernetes/
   411        project: default
   412        destination:
   413          server: https://kubernetes.default.svc
   414          namespace: default
   415  ```
   416  
   417  * `organization`: The name of the organization the repository is in.
   418  * `repository`: The name of the repository.
   419  * `url`: The clone URL for the repository.
   420  * `branch`: The default branch of the repository.
   421  * `sha`: The Git commit SHA for the branch.
   422  * `short_sha`: The abbreviated Git commit SHA for the branch (8 chars or the length of the `sha` if it's shorter).
   423  * `short_sha_7`: The abbreviated Git commit SHA for the branch (7 chars or the length of the `sha` if it's shorter).
   424  * `labels`: A comma-separated list of repository labels in case of Gitea, repository topics in case of Gitlab and Github. Not supported by Bitbucket Cloud, Bitbucket Server, or Azure DevOps.
   425  * `branchNormalized`: The value of `branch` normalized to contain only lowercase alphanumeric characters, '-' or '.'.
   426  
   427  ## Pass additional key-value pairs via `values` field
   428  
   429  You may pass additional, arbitrary string key-value pairs via the `values` field of any SCM generator. Values added via the `values` field are added as `values.(field)`.
   430  
   431  In this example, a `name` parameter value is passed. It is interpolated from `organization` and `repository` to generate a different template name.
   432  ```yaml
   433  apiVersion: argoproj.io/v1alpha1
   434  kind: ApplicationSet
   435  metadata:
   436    name: myapps
   437  spec:
   438    goTemplate: true
   439    goTemplateOptions: ["missingkey=error"]
   440    generators:
   441    - scmProvider:
   442        bitbucketServer:
   443          project: myproject
   444          api: https://mycompany.bitbucket.org
   445          allBranches: true
   446          basicAuth:
   447            username: myuser
   448            passwordRef:
   449              secretName: mypassword
   450              key: password
   451        values:
   452          name: "{{.organization}}-{{.repository}}"
   453  
   454    template:
   455      metadata:
   456        name: '{{ .values.name }}'
   457      spec:
   458        source:
   459          repoURL: '{{ .url }}'
   460          targetRevision: '{{ .branch }}'
   461          path: kubernetes/
   462        project: default
   463        destination:
   464          server: https://kubernetes.default.svc
   465          namespace: default
   466  ```
   467  
   468  !!! note
   469      The `values.` prefix is always prepended to values provided via `generators.scmProvider.values` field. Ensure you include this prefix in the parameter name within the `template` when using it.
   470  
   471  In `values` we can also interpolate all fields set by the SCM generator as mentioned above.