github.com/argoproj/argo-cd/v2@v2.10.9/docs/operator-manual/security.md (about) 1 # Security 2 3 Argo CD has undergone rigorous internal security reviews and penetration testing to satisfy [PCI 4 compliance](https://www.pcisecuritystandards.org) requirements. The following are some security 5 topics and implementation details of Argo CD. 6 7 ## Authentication 8 9 Authentication to Argo CD API server is performed exclusively using [JSON Web Tokens](https://jwt.io) 10 (JWTs). Username/password bearer tokens are not used for authentication. The JWT is obtained/managed 11 in one of the following ways: 12 13 1. For the local `admin` user, a username/password is exchanged for a JWT using the `/api/v1/session` 14 endpoint. This token is signed & issued by the Argo CD API server itself and it expires after 24 hours 15 (this token used not to expire, see [CVE-2021-26921](https://github.com/argoproj/argo-cd/security/advisories/GHSA-9h6w-j7w4-jr52)). 16 When the admin password is updated, all existing admin JWT tokens are immediately revoked. 17 The password is stored as a bcrypt hash in the [`argocd-secret`](https://github.com/argoproj/argo-cd/blob/master/manifests/base/config/argocd-secret.yaml) Secret. 18 19 2. For Single Sign-On users, the user completes an OAuth2 login flow to the configured OIDC identity 20 provider (either delegated through the bundled Dex provider, or directly to a self-managed OIDC 21 provider). This JWT is signed & issued by the IDP, and expiration and revocation is handled by 22 the provider. Dex tokens expire after 24 hours. 23 24 3. Automation tokens are generated for a project using the `/api/v1/projects/{project}/roles/{role}/token` 25 endpoint, and are signed & issued by Argo CD. These tokens are limited in scope and privilege, 26 and can only be used to manage application resources in the project which it belongs to. Project 27 JWTs have a configurable expiration and can be immediately revoked by deleting the JWT reference 28 ID from the project role. 29 30 ## Authorization 31 32 Authorization is performed by iterating the list of group membership in a user's JWT groups claims, 33 and comparing each group against the roles/rules in the [RBAC](../rbac) policy. Any matched rule 34 permits access to the API request. 35 36 ## TLS 37 38 All network communication is performed over TLS including service-to-service communication between 39 the three components (argocd-server, argocd-repo-server, argocd-application-controller). The Argo CD 40 API server can enforce the use of TLS 1.2 using the flag: `--tlsminversion 1.2`. 41 Communication with Redis is performed over plain HTTP by default. TLS can be setup with command line arguments. 42 43 ## Git & Helm Repositories 44 45 Git and helm repositories are managed by a stand-alone service, called the repo-server. The 46 repo-server does not carry any Kubernetes privileges and does not store credentials to any services 47 (including git). The repo-server is responsible for cloning repositories which have been permitted 48 and trusted by Argo CD operators, and generating Kubernetes manifests at a given path in the 49 repository. For performance and bandwidth efficiency, the repo-server maintains local clones of 50 these repositories so that subsequent commits to the repository are efficiently downloaded. 51 52 There are security considerations when configuring git repositories that Argo CD is permitted to 53 deploy from. In short, gaining unauthorized write access to a git repository trusted by Argo CD 54 will have serious security implications outlined below. 55 56 ### Unauthorized Deployments 57 58 Since Argo CD deploys the Kubernetes resources defined in git, an attacker with access to a trusted 59 git repo would be able to affect the Kubernetes resources which are deployed. For example, an 60 attacker could update the deployment manifest deploy malicious container images to the environment, 61 or delete resources in git causing them to be pruned in the live environment. 62 63 ### Tool command invocation 64 65 In addition to raw YAML, Argo CD natively supports two popular Kubernetes config management tools, 66 helm and kustomize. When rendering manifests, Argo CD executes these config management tools 67 (i.e. `helm template`, `kustomize build`) to generate the manifests. It is possible that an attacker 68 with write access to a trusted git repository may construct malicious helm charts or kustomizations 69 that attempt to read files out-of-tree. This includes adjacent git repos, as well as files on the 70 repo-server itself. Whether or not this is a risk to your organization depends on if the contents 71 in the git repos are sensitive in nature. By default, the repo-server itself does not contain 72 sensitive information, but might be configured with Config Management Plugins which do 73 (e.g. decryption keys). If such plugins are used, extreme care must be taken to ensure the 74 repository contents can be trusted at all times. 75 76 Optionally the built-in config management tools might be individually disabled. 77 If you know that your users will not need a certain config management tool, it's advisable 78 to disable that tool. 79 See [Tool Detection](../user-guide/tool_detection.md) for more information. 80 81 ### Remote bases and helm chart dependencies 82 83 Argo CD's repository allow-list only restricts the initial repository which is cloned. However, both 84 kustomize and helm contain features to reference and follow *additional* repositories 85 (e.g. kustomize remote bases, helm chart dependencies), of which might not be in the repository 86 allow-list. Argo CD operators must understand that users with write access to trusted git 87 repositories could reference other remote git repositories containing Kubernetes resources not 88 easily searchable or auditable in the configured git repositories. 89 90 ## Sensitive Information 91 92 ### Secrets 93 94 Argo CD never returns sensitive data from its API, and redacts all sensitive data in API payloads 95 and logs. This includes: 96 97 * cluster credentials 98 * Git credentials 99 * OAuth2 client secrets 100 * Kubernetes Secret values 101 102 ### External Cluster Credentials 103 104 To manage external clusters, Argo CD stores the credentials of the external cluster as a Kubernetes 105 Secret in the argocd namespace. This secret contains the K8s API bearer token associated with the 106 `argocd-manager` ServiceAccount created during `argocd cluster add`, along with connection options 107 to that API server (TLS configuration/certs, AWS role-arn, etc...). 108 The information is used to reconstruct a REST config and kubeconfig to the cluster used by Argo CD 109 services. 110 111 To rotate the bearer token used by Argo CD, the token can be deleted (e.g. using kubectl) which 112 causes Kubernetes to generate a new secret with a new bearer token. The new token can be re-inputted 113 to Argo CD by re-running `argocd cluster add`. Run the following commands against the *_managed_* 114 cluster: 115 116 ```bash 117 # run using a kubeconfig for the externally managed cluster 118 kubectl delete secret argocd-manager-token-XXXXXX -n kube-system 119 argocd cluster add CONTEXTNAME 120 ``` 121 122 !!! note 123 Kubernetes 1.24 [stopped automatically creating tokens for Service Accounts](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#no-really-you-must-read-this-before-you-upgrade). 124 [Starting in Argo CD 2.4](https://github.com/argoproj/argo-cd/pull/9546), `argocd cluster add` creates a 125 ServiceAccount _and_ a non-expiring Service Account token Secret when adding 1.24 clusters. In the future, Argo CD 126 will [add support for the Kubernetes TokenRequest API](https://github.com/argoproj/argo-cd/issues/9610) to avoid 127 using long-lived tokens. 128 129 To revoke Argo CD's access to a managed cluster, delete the RBAC artifacts against the *_managed_* 130 cluster, and remove the cluster entry from Argo CD: 131 132 ```bash 133 # run using a kubeconfig for the externally managed cluster 134 kubectl delete sa argocd-manager -n kube-system 135 kubectl delete clusterrole argocd-manager-role 136 kubectl delete clusterrolebinding argocd-manager-role-binding 137 argocd cluster rm https://your-kubernetes-cluster-addr 138 ``` 139 <!-- markdownlint-disable MD027 --> 140 > NOTE: for AWS EKS clusters, the [get-token](https://docs.aws.amazon.com/cli/latest/reference/eks/get-token.html) command 141 is used to authenticate to the external cluster, which uses IAM roles in lieu of locally stored 142 tokens, so token rotation is not needed, and revocation is handled through IAM. 143 <!-- markdownlint-enable MD027 --> 144 145 ## Cluster RBAC 146 147 By default, Argo CD uses a [clusteradmin level role](https://github.com/argoproj/argo-cd/blob/master/manifests/base/application-controller/argocd-application-controller-role.yaml) 148 in order to: 149 150 1. watch & operate on cluster state 151 2. deploy resources to the cluster 152 153 Although Argo CD requires cluster-wide **_read_** privileges to resources in the managed cluster to 154 function properly, it does not necessarily need full **_write_** privileges to the cluster. The 155 ClusterRole used by argocd-server and argocd-application-controller can be modified such 156 that write privileges are limited to only the namespaces and resources that you wish Argo CD to 157 manage. 158 159 To fine-tune privileges of externally managed clusters, edit the ClusterRole of the `argocd-manager-role` 160 161 ```bash 162 # run using a kubeconfig for the externally managed cluster 163 kubectl edit clusterrole argocd-manager-role 164 ``` 165 166 To fine-tune privileges which Argo CD has against its own cluster (i.e. `https://kubernetes.default.svc`), 167 edit the following cluster roles where Argo CD is running in: 168 169 ```bash 170 # run using a kubeconfig to the cluster Argo CD is running in 171 kubectl edit clusterrole argocd-server 172 kubectl edit clusterrole argocd-application-controller 173 ``` 174 175 !!! tip 176 If you want to deny Argo CD access to a kind of resource then add it as an [excluded resource](declarative-setup.md#resource-exclusion). 177 178 ## Auditing 179 180 As a GitOps deployment tool, the Git commit history provides a natural audit log of what changes 181 were made to application configuration, when they were made, and by whom. However, this audit log 182 only applies to what happened in Git and does not necessarily correlate one-to-one with events 183 that happen in a cluster. For example, User A could have made multiple commits to application 184 manifests, but User B could have just only synced those changes to the cluster sometime later. 185 186 To complement the Git revision history, Argo CD emits Kubernetes Events of application activity, 187 indicating the responsible actor when applicable. For example: 188 189 ```bash 190 $ kubectl get events 191 LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE 192 1m 1m 1 guestbook.157f7c5edd33aeac Application Normal ResourceCreated argocd-server admin created application 193 1m 1m 1 guestbook.157f7c5f0f747acf Application Normal ResourceUpdated argocd-application-controller Updated sync status: -> OutOfSync 194 1m 1m 1 guestbook.157f7c5f0fbebbff Application Normal ResourceUpdated argocd-application-controller Updated health status: -> Missing 195 1m 1m 1 guestbook.157f7c6069e14f4d Application Normal OperationStarted argocd-server admin initiated sync to HEAD (8a1cb4a02d3538e54907c827352f66f20c3d7b0d) 196 1m 1m 1 guestbook.157f7c60a55a81a8 Application Normal OperationCompleted argocd-application-controller Sync operation to 8a1cb4a02d3538e54907c827352f66f20c3d7b0d succeeded 197 1m 1m 1 guestbook.157f7c60af1ccae2 Application Normal ResourceUpdated argocd-application-controller Updated sync status: OutOfSync -> Synced 198 1m 1m 1 guestbook.157f7c60af5bc4f0 Application Normal ResourceUpdated argocd-application-controller Updated health status: Missing -> Progressing 199 1m 1m 1 guestbook.157f7c651990e848 Application Normal ResourceUpdated argocd-application-controller Updated health status: Progressing -> Healthy 200 ``` 201 202 These events can be then be persisted for longer periods of time using other tools as 203 [Event Exporter](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/event-exporter) or 204 [Event Router](https://github.com/heptiolabs/eventrouter). 205 206 ## WebHook Payloads 207 208 Payloads from webhook events are considered untrusted. Argo CD only examines the payload to infer 209 the involved applications of the webhook event (e.g. which repo was modified), then refreshes 210 the related application for reconciliation. This refresh is the same refresh which occurs regularly 211 at three minute intervals, just fast-tracked by the webhook event. 212 213 ## Logging 214 215 ### Security field 216 217 Security-related logs are tagged with a `security` field to make them easier to find, analyze, and report on. 218 219 | Level | Friendly Level | Description | Example | 220 |-------|----------------|---------------------------------------------------------------------------------------------------|---------------------------------------------| 221 | 1 | Low | Unexceptional, non-malicious events | Successful access | 222 | 2 | Medium | Could indicate malicious events, but has a high likelihood of being user/system error | Access denied | 223 | 3 | High | Likely malicious events but one that had no side effects or was blocked | Out of bounds symlinks in repo | 224 | 4 | Critical | Any malicious or exploitable event that had a side effect | Secrets being left behind on the filesystem | 225 | 5 | Emergency | Unmistakably malicious events that should NEVER occur accidentally and indicates an active attack | Brute forcing of accounts | 226 227 Where applicable, a `CWE` field is also added specifying the [Common Weakness Enumeration](https://cwe.mitre.org/index.html) number. 228 229 !!! warning 230 Please be aware that not all security logs are comprehensively tagged yet and these examples are not necessarily implemented. 231 232 ### API Logs 233 234 Argo CD logs payloads of most API requests except request that are considered sensitive, such as 235 `/cluster.ClusterService/Create`, `/session.SessionService/Create` etc. The full list of method 236 can be found in [server/server.go](https://github.com/argoproj/argo-cd/blob/abba8dddce8cd897ba23320e3715690f465b4a95/server/server.go#L516). 237 238 Argo CD does not log IP addresses of clients requesting API endpoints, since the API server is typically behind a proxy. Instead, it is recommended 239 to configure IP addresses logging in the proxy server that sits in front of the API server. 240 241 ## ApplicationSets 242 243 Argo CD's ApplicationSets feature has its own [security considerations](./applicationset/Security.md). Be aware of those 244 issues before using ApplicationSets. 245 246 ## Limiting Directory App Memory Usage 247 248 > >2.2.10, 2.1.16, >2.3.5 249 250 Directory-type Applications (those whose source is raw JSON or YAML files) can consume significant 251 [repo-server](architecture.md#repository-server) memory, depending on the size and structure of the YAML files. 252 253 To avoid over-using memory in the repo-server (potentially causing a crash and denial of service), set the 254 `reposerver.max.combined.directory.manifests.size` config option in [argocd-cmd-params-cm](argocd-cmd-params-cm.yaml). 255 256 This option limits the combined size of all JSON or YAML files in an individual app. Note that the in-memory 257 representation of a manifest may be as much as 300x the size of the manifest on disk. Also note that the limit is per 258 Application. If manifests are generated for multiple applications at once, memory usage will be higher. 259 260 **Example:** 261 262 Suppose your repo-server has a 10G memory limit, and you have ten Applications which use raw JSON or YAML files. To 263 calculate the max safe combined file size per Application, divide 10G by 300 * 10 Apps (300 being the worst-case memory 264 growth factor for the manifests). 265 266 ``` 267 10G / 300 * 10 = 3M 268 ``` 269 270 So a reasonably safe configuration for this setup would be a 3M limit per app. 271 272 ```yaml 273 apiVersion: v1 274 kind: ConfigMap 275 metadata: 276 name: argocd-cmd-params-cm 277 data: 278 reposerver.max.combined.directory.manifests.size: '3M' 279 ``` 280 281 The 300x ratio assumes a maliciously-crafted manifest file. If you only want to protect against accidental excessive 282 memory use, it is probably safe to use a smaller ratio. 283 284 Keep in mind that if a malicious user can create additional Applications, they can increase the total memory usage. 285 Grant [App creation privileges](rbac.md) carefully.