github.com/argoproj/argo-cd/v3@v3.2.1/docs/operator-manual/applicationset/Generators-SCM-Provider.md (about)

     1  # SCM Provider Generator
     2  
     3  The SCM Provider generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover repositories within an organization. This fits well with GitOps layout patterns that split microservices across many repositories.
     4  
     5  ```yaml
     6  apiVersion: argoproj.io/v1alpha1
     7  kind: ApplicationSet
     8  metadata:
     9    name: myapps
    10  spec:
    11    generators:
    12    - scmProvider:
    13        # Which protocol to clone using.
    14        cloneProtocol: ssh
    15        # See below for provider specific options.
    16        github:
    17          # ...
    18  ```
    19  
    20  * `cloneProtocol`: Which protocol to use for the SCM URL. Default is provider-specific but ssh if possible. Not all providers necessarily support all protocols, see provider documentation below for available options.
    21  
    22  !!! note
    23      Know the security implications of using SCM generators. [Only admins may create ApplicationSets](./Security.md#only-admins-may-createupdatedelete-applicationsets)
    24      to avoid leaking Secrets, and [only admins may create repos/branches](./Security.md#templated-project-field) if the
    25      `project` field of an ApplicationSet with an SCM generator is templated, to avoid granting management of
    26      out-of-bounds resources.
    27  
    28  ## GitHub
    29  
    30  The GitHub mode uses the GitHub API to scan an organization in either github.com or GitHub Enterprise.
    31  
    32  ```yaml
    33  apiVersion: argoproj.io/v1alpha1
    34  kind: ApplicationSet
    35  metadata:
    36    name: myapps
    37  spec:
    38    generators:
    39    - scmProvider:
    40        github:
    41          # The GitHub organization to scan.
    42          organization: myorg
    43          # For GitHub Enterprise:
    44          api: https://git.example.com/
    45          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
    46          allBranches: true
    47          # Reference to a Secret containing an access token. (optional)
    48          tokenRef:
    49            secretName: github-token
    50            key: token
    51          # (optional) use a GitHub App to access the API instead of a PAT.
    52          appSecretName: gh-app-repo-creds
    53    template:
    54    # ...
    55  ```
    56  
    57  * `organization`: Required name of the GitHub organization to scan. If you have multiple organizations, use multiple generators.
    58  * `api`: If using GitHub Enterprise, the URL to access it.
    59  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
    60  * `tokenRef`: A `Secret` name and key containing the GitHub access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
    61  * `appSecretName`: A `Secret` name containing a GitHub App secret in [repo-creds format][repo-creds].
    62  
    63  [repo-creds]: ../declarative-setup.md#repository-credentials
    64  
    65  For label filtering, the repository topics are used.
    66  
    67  Available clone protocols are `ssh` and `https`.
    68  
    69  ## Gitlab
    70  
    71  The GitLab mode uses the GitLab API to scan and organization in either gitlab.com or self-hosted GitLab.
    72  
    73  ```yaml
    74  apiVersion: argoproj.io/v1alpha1
    75  kind: ApplicationSet
    76  metadata:
    77    name: myapps
    78  spec:
    79    generators:
    80    - scmProvider:
    81        gitlab:
    82          # The base GitLab group to scan.  You can either use the group id or the full namespaced path.
    83          group: "8675309"
    84          # For self-hosted GitLab:
    85          api: https://gitlab.example.com/
    86          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
    87          allBranches: true
    88          # If true, recurses through subgroups. If false, it searches only in the base group. Defaults to false.
    89          includeSubgroups: true
    90          # If true and includeSubgroups is also true, include Shared Projects, which is gitlab API default.
    91          # If false only search Projects under the same path. Defaults to true.
    92          includeSharedProjects: false
    93          # filter projects by topic. A single topic is supported by Gitlab API. Defaults to "" (all topics).
    94          topic: "my-topic"
    95          # Reference to a Secret containing an access token. (optional)
    96          tokenRef:
    97            secretName: gitlab-token
    98            key: token
    99          # If true, skips validating the SCM provider's TLS certificate - useful for self-signed certificates.
   100          insecure: false
   101          # Reference to a ConfigMap containing trusted CA certs - useful for self-signed certificates. (optional)
   102          caRef:
   103            configMapName: argocd-tls-certs-cm
   104            key: gitlab-ca
   105    template:
   106    # ...
   107  ```
   108  
   109  * `group`: Required name of the base GitLab group to scan. If you have multiple base groups, use multiple generators.
   110  * `api`: If using self-hosted GitLab, the URL to access it.
   111  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   112  * `includeSubgroups`: By default (false) the controller will only search for repos directly in the base group. If this is true, it will recurse through all the subgroups searching for repos to scan.
   113  * `includeSharedProjects`: If true and includeSubgroups is also true, include Shared Projects, which is gitlab API default. If false only search Projects under the same path. In general most would want the behaviour when set to false. Defaults to true.
   114  * `topic`: filter projects by topic. A single topic is supported by Gitlab API. Defaults to "" (all topics).
   115  * `tokenRef`: A `Secret` name and key containing the GitLab access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
   116  * `insecure`: By default (false) - Skip checking the validity of the SCM's certificate - useful for self-signed TLS certificates.
   117  * `caRef`: Optional `ConfigMap` name and key containing the GitLab certificates to trust - useful for self-signed TLS certificates. Possibly reference the ArgoCD CM holding the trusted certs.
   118  
   119  For label filtering, the repository topics are used.
   120  
   121  Available clone protocols are `ssh` and `https`.
   122  
   123  ### Self-signed TLS Certificates
   124  
   125  As a preferable alternative to setting `insecure` to true, you can configure self-signed TLS certificates for Gitlab.
   126  
   127  In order for a self-signed TLS certificate be used by an ApplicationSet's SCM / PR Gitlab Generator, the certificate needs to be mounted on the applicationset-controller. The path of the mounted certificate must be explicitly set using the environment variable `ARGOCD_APPLICATIONSET_CONTROLLER_SCM_ROOT_CA_PATH` or alternatively using parameter `--scm-root-ca-path`. The applicationset controller will read the mounted certificate to create the Gitlab client for SCM/PR Providers
   128  
   129  This can be achieved conveniently by setting `applicationsetcontroller.scm.root.ca.path` in the argocd-cmd-params-cm ConfigMap. Be sure to restart the ApplicationSet controller after setting this value.
   130  
   131  ## Gitea
   132  
   133  The Gitea mode uses the Gitea API to scan organizations in your instance
   134  
   135  ```yaml
   136  apiVersion: argoproj.io/v1alpha1
   137  kind: ApplicationSet
   138  metadata:
   139    name: myapps
   140  spec:
   141    generators:
   142    - scmProvider:
   143        gitea:
   144          # The Gitea owner to scan.
   145          owner: myorg
   146          # The Gitea instance url
   147          api: https://gitea.mydomain.com/
   148          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
   149          allBranches: true
   150          # Reference to a Secret containing an access token. (optional)
   151          tokenRef:
   152            secretName: gitea-token
   153            key: token
   154    template:
   155    # ...
   156  ```
   157  
   158  * `owner`: Required name of the Gitea organization to scan. If you have multiple organizations, use multiple generators.
   159  * `api`: The URL of the Gitea instance you are using.
   160  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   161  * `tokenRef`: A `Secret` name and key containing the Gitea access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
   162  * `insecure`: Allow for self-signed TLS certificates.
   163  
   164  This SCM provider does not yet support label filtering
   165  
   166  Available clone protocols are `ssh` and `https`.
   167  
   168  ## Bitbucket Server
   169  
   170  Use the Bitbucket Server API (1.0) to scan repos in a project. Note that Bitbucket Server is not to same as Bitbucket Cloud (API 2.0)
   171  
   172  ```yaml
   173  apiVersion: argoproj.io/v1alpha1
   174  kind: ApplicationSet
   175  metadata:
   176    name: myapps
   177  spec:
   178    generators:
   179    - scmProvider:
   180        bitbucketServer:
   181          project: myproject
   182          # URL of the Bitbucket Server. Required.
   183          api: https://mycompany.bitbucket.org
   184          # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
   185          allBranches: true
   186          # Credentials for Basic authentication (App Password). Either basicAuth or bearerToken
   187          # authentication is required to access private repositories
   188          basicAuth:
   189            # The username to authenticate with
   190            username: myuser
   191            # Reference to a Secret containing the password or personal access token.
   192            passwordRef:
   193              secretName: mypassword
   194              key: password
   195          # Credentials for Bearer Token (App Token) authentication. Either basicAuth or bearerToken
   196          # authentication is required to access private repositories
   197          bearerToken:
   198            # Reference to a Secret containing the bearer token.
   199            tokenRef:
   200              secretName: repotoken
   201              key: token
   202          # If true, skips validating the SCM provider's TLS certificate - useful for self-signed certificates.
   203          insecure: true
   204          # Reference to a ConfigMap containing trusted CA certs - useful for self-signed certificates. (optional)
   205          caRef:
   206            configMapName: argocd-tls-certs-cm
   207            key: bitbucket-ca
   208          # Support for filtering by labels is TODO. Bitbucket server labels are not supported for PRs, but they are for repos
   209    template:
   210    # ...
   211  ```
   212  
   213  * `project`: Required name of the Bitbucket project
   214  * `api`: Required URL to access the Bitbucket REST api.
   215  * `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   216  
   217  If you want to access a private repository, you must also provide the credentials for Basic auth (this is the only auth supported currently):
   218  * `username`: The username to authenticate with. It only needs read access to the relevant repo.
   219  * `passwordRef`: A `Secret` name and key containing the password or personal access token to use for requests.
   220  
   221  In case of Bitbucket App Token, go with `bearerToken` section.
   222  * `tokenRef`: A `Secret` name and key containing the app token to use for requests.
   223  
   224  In case self-signed BitBucket Server certificates, the following options can be usefully:
   225  * `insecure`: By default (false) - Skip checking the validity of the SCM's certificate - useful for self-signed TLS certificates.
   226  * `caRef`: Optional `ConfigMap` name and key containing the BitBucket server certificates to trust - useful for self-signed TLS certificates. Possibly reference the ArgoCD CM holding the trusted certs.
   227  
   228  Available clone protocols are `ssh` and `https`.
   229  
   230  ## Azure DevOps
   231  
   232  Uses the Azure DevOps API to look up eligible repositories based on a team project within an Azure DevOps organization.
   233  The default Azure DevOps URL is `https://dev.azure.com`, but this can be overridden with the field `azureDevOps.api`.
   234  
   235  ```yaml
   236  apiVersion: argoproj.io/v1alpha1
   237  kind: ApplicationSet
   238  metadata:
   239    name: myapps
   240  spec:
   241    generators:
   242    - scmProvider:
   243        azureDevOps:
   244          # The Azure DevOps organization.
   245          organization: myorg
   246          # URL to Azure DevOps. Optional. Defaults to https://dev.azure.com.
   247          api: https://dev.azure.com
   248          # If true, scan every branch of eligible repositories. If false, check only the default branch of the eligible repositories. Defaults to false.
   249          allBranches: true
   250          # The team project within the specified Azure DevOps organization.
   251          teamProject: myProject
   252          # Reference to a Secret containing the Azure DevOps Personal Access Token (PAT) used for accessing Azure DevOps.
   253          accessTokenRef:
   254            secretName: azure-devops-scm
   255            key: accesstoken
   256    template:
   257    # ...
   258  ```
   259  
   260  * `organization`: Required. Name of the Azure DevOps organization.
   261  * `teamProject`: Required. The name of the team project within the specified `organization`.
   262  * `accessTokenRef`: Required. A `Secret` name and key containing the Azure DevOps Personal Access Token (PAT) to use for requests.
   263  * `api`: Optional. URL to Azure DevOps. If not set, `https://dev.azure.com` is used.
   264  * `allBranches`: Optional, default `false`. If `true`, scans every branch of eligible repositories. If `false`, check only the default branch of the eligible repositories.
   265  
   266  ## Bitbucket Cloud
   267  
   268  The Bitbucket mode uses the Bitbucket API V2 to scan a workspace in bitbucket.org.
   269  
   270  ```yaml
   271  apiVersion: argoproj.io/v1alpha1
   272  kind: ApplicationSet
   273  metadata:
   274    name: myapps
   275  spec:
   276    generators:
   277    - scmProvider:
   278        bitbucket:
   279          # The workspace id (slug).  
   280          owner: "example-owner"
   281          # The user to use for basic authentication with an app password.
   282          user: "example-user"
   283          # If true, scan every branch of every repository. If false, scan only the main branch. Defaults to false.
   284          allBranches: true
   285          # Reference to a Secret containing an app password.
   286          appPasswordRef:
   287            secretName: appPassword
   288            key: password
   289    template:
   290    # ...
   291  ```
   292  
   293  * `owner`: The workspace ID (slug) to use when looking up repositories.
   294  * `user`: The user to use for authentication to the Bitbucket API V2 at bitbucket.org.
   295  * `allBranches`: By default (false) the template will only be evaluated for the main branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
   296  * `appPasswordRef`: A `Secret` name and key containing the bitbucket app password to use for requests.
   297  
   298  This SCM provider does not yet support label filtering
   299  
   300  Available clone protocols are `ssh` and `https`.
   301  
   302  ## AWS CodeCommit (Alpha)
   303  
   304  Uses AWS ResourceGroupsTagging and AWS CodeCommit APIs to scan repos across AWS accounts and regions.
   305  
   306  ```yaml
   307  apiVersion: argoproj.io/v1alpha1
   308  kind: ApplicationSet
   309  metadata:
   310    name: myapps
   311  spec:
   312    generators:
   313      - scmProvider:
   314          awsCodeCommit:
   315            # AWS region to scan repos.
   316            # default to the environmental region from ApplicationSet controller.
   317            region: us-east-1
   318            # AWS role to assume to scan repos.
   319            # default to the environmental role from ApplicationSet controller.
   320            role: arn:aws:iam::111111111111:role/argocd-application-set-discovery
   321            # If true, scan every branch of every repository. If false, scan only the main branch. Defaults to false.
   322            allBranches: true
   323            # AWS resource tags to filter repos with.
   324            # see https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html#resourcegrouptagging-GetResources-request-TagFilters for details
   325            # default to no tagFilters, to include all repos in the region.
   326            tagFilters:
   327              - key: organization
   328                value: platform-engineering
   329              - key: argo-ready
   330    template:
   331    # ...
   332  ```
   333  
   334  * `region`: (Optional) AWS region to scan repos. By default, use ApplicationSet controller's current region.
   335  * `role`: (Optional) AWS role to assume to scan repos. By default, use ApplicationSet controller's current role.
   336  * `allBranches`: (Optional) If `true`, scans every branch of eligible repositories. If `false`, check only the default branch of the eligible repositories. Default `false`.
   337  * `tagFilters`: (Optional) A list of tagFilters to filter AWS CodeCommit repos with. See [AWS ResourceGroupsTagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html#resourcegrouptagging-GetResources-request-TagFilters) for details. By default, no filter is included.
   338  
   339  This SCM provider does not support the following features
   340  
   341  * label filtering
   342  * `sha`, `short_sha` and `short_sha_7` template parameters
   343  
   344  Available clone protocols are `ssh`, `https` and `https-fips`.
   345  
   346  ### AWS IAM Permission Considerations
   347  
   348  In order to call AWS APIs to discover AWS CodeCommit repos, ApplicationSet controller must be configured with valid environmental AWS config, like current AWS region and AWS credentials.
   349  AWS config can be provided via all standard options, like Instance Metadata Service (IMDS), config file, environment variables, or IAM roles for service accounts (IRSA).
   350  
   351  Depending on whether `role` is provided in `awsCodeCommit` property, AWS IAM permission requirement is different.
   352  
   353  #### Discover AWS CodeCommit Repositories in the same AWS Account as ApplicationSet Controller
   354  
   355  Without specifying `role`, ApplicationSet controller will use its own AWS identity to scan AWS CodeCommit repos.
   356  This is suitable when you have a simple setup that all AWS CodeCommit repos reside in the same AWS account as your Argo CD.
   357  
   358  As the ApplicationSet controller AWS identity is used directly for repo discovery, it must be granted below AWS permissions.
   359  
   360  * `tag:GetResources`
   361  * `codecommit:ListRepositories`
   362  * `codecommit:GetRepository`
   363  * `codecommit:GetFolder`
   364  * `codecommit:ListBranches`
   365  
   366  #### Discover AWS CodeCommit Repositories across AWS Accounts and Regions
   367  
   368  By specifying `role`, ApplicationSet controller will first assume the `role`, and use it for repo discovery.
   369  This enables more complicated use cases to discover repos from different AWS accounts and regions.
   370  
   371  The ApplicationSet controller AWS identity should be granted permission to assume target AWS roles.
   372  
   373  * `sts:AssumeRole`
   374  
   375  All AWS roles must have repo discovery related permissions.
   376  
   377  * `tag:GetResources`
   378  * `codecommit:ListRepositories`
   379  * `codecommit:GetRepository`
   380  * `codecommit:GetFolder`
   381  * `codecommit:ListBranches`
   382  
   383  ## Filters
   384  
   385  Filters allow selecting which repositories to generate for. Each filter can declare one or more conditions, all of which must pass. If multiple filters are present, any can match for a repository to be included. If no filters are specified, all repositories will be processed.
   386  
   387  ```yaml
   388  apiVersion: argoproj.io/v1alpha1
   389  kind: ApplicationSet
   390  metadata:
   391    name: myapps
   392  spec:
   393    generators:
   394    - scmProvider:
   395        filters:
   396        # Include any repository starting with "myapp" AND including a Kustomize config AND labeled with "deploy-ok" ...
   397        - repositoryMatch: ^myapp
   398          pathsExist: [kubernetes/kustomization.yaml]
   399          labelMatch: deploy-ok
   400        # ... OR include any repository starting with "otherapp" AND a Helm folder and doesn't have file disabledrepo.txt.
   401        - repositoryMatch: ^otherapp
   402          pathsExist: [helm]
   403          pathsDoNotExist: [disabledrepo.txt]
   404    template:
   405    # ...
   406  ```
   407  
   408  * `repositoryMatch`: A regexp matched against the repository name.
   409  * `pathsExist`: An array of paths within the repository that must exist. Can be a file or directory.
   410  * `pathsDoNotExist`: An array of paths within the repository that must not exist. Can be a file or directory.
   411  * `labelMatch`: A regexp matched against repository labels. If any label matches, the repository is included.
   412  * `branchMatch`: A regexp matched against branch names.
   413  
   414  ## Template
   415  
   416  As with all generators, several parameters are generated for use within the `ApplicationSet` resource template.
   417  
   418  ```yaml
   419  apiVersion: argoproj.io/v1alpha1
   420  kind: ApplicationSet
   421  metadata:
   422    name: myapps
   423  spec:
   424    goTemplate: true
   425    goTemplateOptions: ["missingkey=error"]
   426    generators:
   427    - scmProvider:
   428      # ...
   429    template:
   430      metadata:
   431        name: '{{ .repository }}'
   432      spec:
   433        source:
   434          repoURL: '{{ .url }}'
   435          targetRevision: '{{ .branch }}'
   436          path: kubernetes/
   437        project: default
   438        destination:
   439          server: https://kubernetes.default.svc
   440          namespace: default
   441  ```
   442  
   443  * `organization`: The name of the organization the repository is in.
   444  * `repository`: The name of the repository.
   445  * `repository_id`: The id of the repository.
   446  * `url`: The clone URL for the repository.
   447  * `branch`: The default branch of the repository.
   448  * `sha`: The Git commit SHA for the branch.
   449  * `short_sha`: The abbreviated Git commit SHA for the branch (8 chars or the length of the `sha` if it's shorter).
   450  * `short_sha_7`: The abbreviated Git commit SHA for the branch (7 chars or the length of the `sha` if it's shorter).
   451  * `labels`: A comma-separated list of repository labels in case of Gitea, repository topics in case of Gitlab and Github. Not supported by Bitbucket Cloud, Bitbucket Server, or Azure DevOps.
   452  * `branchNormalized`: The value of `branch` normalized to contain only lowercase alphanumeric characters, '-' or '.'.
   453  
   454  ## Pass additional key-value pairs via `values` field
   455  
   456  You may pass additional, arbitrary string key-value pairs via the `values` field of any SCM generator. Values added via the `values` field are added as `values.(field)`.
   457  
   458  In this example, a `name` parameter value is passed. It is interpolated from `organization` and `repository` to generate a different template name.
   459  ```yaml
   460  apiVersion: argoproj.io/v1alpha1
   461  kind: ApplicationSet
   462  metadata:
   463    name: myapps
   464  spec:
   465    goTemplate: true
   466    goTemplateOptions: ["missingkey=error"]
   467    generators:
   468    - scmProvider:
   469        bitbucketServer:
   470          project: myproject
   471          api: https://mycompany.bitbucket.org
   472          allBranches: true
   473          basicAuth:
   474            username: myuser
   475            passwordRef:
   476              secretName: mypassword
   477              key: password
   478        values:
   479          name: "{{.organization}}-{{.repository}}"
   480  
   481    template:
   482      metadata:
   483        name: '{{ .values.name }}'
   484      spec:
   485        source:
   486          repoURL: '{{ .url }}'
   487          targetRevision: '{{ .branch }}'
   488          path: kubernetes/
   489        project: default
   490        destination:
   491          server: https://kubernetes.default.svc
   492          namespace: default
   493  ```
   494  
   495  !!! note
   496      The `values.` prefix is always prepended to values provided via `generators.scmProvider.values` field. Ensure you include this prefix in the parameter name within the `template` when using it.
   497  
   498  In `values` we can also interpolate all fields set by the SCM generator as mentioned above.