github.com/argoproj/argo-cd/v3@v3.2.1/docs/user-guide/diffing.md (about) 1 # Diffing Customization 2 3 It is possible for an application to be `OutOfSync` even immediately after a successful Sync operation. Some reasons for this might be: 4 5 - There is a bug in the manifest, where it contains extra/unknown fields from the actual K8s spec. These extra fields would get dropped when querying Kubernetes for the live state, 6 resulting in an `OutOfSync` status indicating a missing field was detected. 7 - The sync was performed (with pruning disabled), and there are resources which need to be deleted. 8 - A controller or [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) is altering the object after it was 9 submitted to Kubernetes so it differs from the one in Git. 10 - A Helm chart is using a template function such as [`randAlphaNum`](https://github.com/helm/charts/blob/master/stable/redis/templates/secret.yaml#L16), 11 which generates different data every time `helm template` is invoked. 12 - For Horizontal Pod Autoscaling (HPA) objects, the HPA controller is known to reorder `spec.metrics` 13 in a specific order. See [kubernetes issue #74099](https://github.com/kubernetes/kubernetes/issues/74099). 14 To work around this, you can order `spec.metrics` in Git in the same order that the controller 15 prefers. 16 17 In case it is impossible to fix the upstream issue, Argo CD allows you to optionally ignore differences of problematic resources. 18 The diffing customization can be configured for single or multiple application resources or at a system level. 19 20 ## Application Level Configuration 21 22 Argo CD allows ignoring differences at a specific JSON path, using [RFC6902 JSON patches](https://tools.ietf.org/html/rfc6902) and [JQ path expressions](<https://stedolan.github.io/jq/manual/#path(path_expression)>). It is also possible to ignore differences from fields owned by specific managers defined in `metadata.managedFields` in live resources. 23 24 The following sample application is configured to ignore differences in `spec.replicas` for all deployments: 25 26 ```yaml 27 spec: 28 ignoreDifferences: 29 - group: apps 30 kind: Deployment 31 jsonPointers: 32 - /spec/replicas 33 ``` 34 35 Note that the `group` field relates to the [Kubernetes API group](https://kubernetes.io/docs/reference/using-api/#api-groups) without the version. 36 The above customization could be narrowed to a resource with the specified name and optional namespace: 37 38 ```yaml 39 spec: 40 ignoreDifferences: 41 - group: apps 42 kind: Deployment 43 name: guestbook 44 namespace: default 45 jsonPointers: 46 - /spec/replicas 47 ``` 48 49 To ignore elements of a list, you can use JQ path expressions to identify list items based on item content: 50 51 ```yaml 52 spec: 53 ignoreDifferences: 54 - group: apps 55 kind: Deployment 56 jqPathExpressions: 57 - .spec.template.spec.initContainers[] | select(.name == "injected-init-container") 58 ``` 59 60 To ignore fields owned by specific managers defined in your live resources: 61 62 ```yaml 63 spec: 64 ignoreDifferences: 65 - group: '*' 66 kind: '*' 67 managedFieldsManagers: 68 - kube-controller-manager 69 ``` 70 71 The above configuration will ignore differences from all fields owned by `kube-controller-manager` for all resources belonging to this application. 72 73 If you have a slash `/` in your pointer path, you need to replace it with the `~1` character. For example: 74 75 ```yaml 76 spec: 77 ignoreDifferences: 78 - kind: Node 79 jsonPointers: 80 - /metadata/labels/node-role.kubernetes.io~1worker 81 ``` 82 83 ## System-Level Configuration 84 85 The comparison of resources with well-known issues can be customized at a system level. Ignored differences can be configured for a specified group and kind 86 in `resource.customizations` key of `argocd-cm` ConfigMap. Following is an example of a customization which ignores the `caBundle` field 87 of a `MutatingWebhookConfiguration` webhooks: 88 89 ```yaml 90 data: 91 resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: 92 | 93 jqPathExpressions: 94 - '.webhooks[]?.clientConfig.caBundle' 95 ``` 96 97 Resource customization can also be configured to ignore all differences made by a `managedField.manager` at the system level. The example below shows how to configure Argo CD to ignore changes made by `kube-controller-manager` in `Deployment` resources. 98 99 ```yaml 100 data: 101 resource.customizations.ignoreDifferences.apps_Deployment: | 102 managedFieldsManagers: 103 - kube-controller-manager 104 ``` 105 106 It is possible to configure `ignoreDifferences` to be applied to all resources in every Application managed by an Argo CD instance. In order to do so, resource customizations can be configured like in the example below: 107 108 ```yaml 109 data: 110 resource.customizations.ignoreDifferences.all: | 111 managedFieldsManagers: 112 - kube-controller-manager 113 jsonPointers: 114 - /spec/replicas 115 ``` 116 117 The `status` field of many resources is often stored in Git/Helm manifest and should be ignored during diffing. The `status` field is used by 118 Kubernetes controller to persist the current state of the resource and therefore cannot be applied as a desired configuration. 119 120 ```yaml 121 data: 122 resource.compareoptions: | 123 # disables status field diffing in specified resource types 124 # 'crd' - CustomResourceDefinitions 125 # 'all' - all resources (default) 126 # 'none' - disabled 127 ignoreResourceStatusField: all 128 ``` 129 130 If you rely on the status field being part of your desired state, although this is not recommended, the `ignoreResourceStatusField` setting can be used to configure this behavior. 131 132 !!! note 133 Since it is common for `CustomResourceDefinitions` to have their `status` committed to Git, consider using `crd` over `none`. 134 135 ### Ignoring RBAC changes made by AggregateRoles 136 137 If you are using [Aggregated ClusterRoles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) and don't want Argo CD to detect the `rules` changes as drift, you can set `resource.compareoptions.ignoreAggregatedRoles: true`. Then Argo CD will no longer detect these changes as an event that requires syncing. 138 139 ```yaml 140 apiVersion: v1 141 kind: ConfigMap 142 metadata: 143 name: argocd-cm 144 data: 145 resource.compareoptions: | 146 # disables status field diffing in specified resource types 147 ignoreAggregatedRoles: true 148 ``` 149 150 ## Known Kubernetes types in CRDs (Resource limits, Volume mounts etc) 151 152 Some CRDs are re-using data structures defined in the Kubernetes source base and therefore inheriting custom 153 JSON/YAML marshaling. Custom marshalers might serialize CRDs in a slightly different format that causes false 154 positives during drift detection. 155 156 A typical example is the `argoproj.io/Rollout` CRD that re-using `core/v1/PodSpec` data structure. Pod resource requests 157 might be reformatted by the custom marshaller of `IntOrString` data type: 158 159 from: 160 161 ```yaml 162 resources: 163 requests: 164 cpu: 100m 165 ``` 166 167 to: 168 169 ```yaml 170 resources: 171 requests: 172 cpu: 0.1 173 ``` 174 175 The solution is to specify which CRDs fields are using built-in Kubernetes types in the `resource.customizations` 176 section of `argocd-cm` ConfigMap: 177 178 ```yaml 179 apiVersion: v1 180 kind: ConfigMap 181 metadata: 182 name: argocd-cm 183 namespace: argocd 184 labels: 185 app.kubernetes.io/name: argocd-cm 186 app.kubernetes.io/part-of: argocd 187 data: 188 resource.customizations.knownTypeFields.argoproj.io_Rollout: | 189 - field: spec.template.spec 190 type: core/v1/PodSpec 191 ``` 192 193 The list of supported Kubernetes types is available in [diffing_known_types.txt](https://raw.githubusercontent.com/argoproj/argo-cd/master/util/argo/normalizers/diffing_known_types.txt) and additionally: 194 195 - `core/Quantity` 196 - `meta/v1/duration` 197 198 ### JQ Path expression timeout 199 200 By default, the evaluation of a JQPathExpression is limited to one second. If you encounter a "JQ patch execution timed out" error message due to a complex JQPathExpression that requires more time to evaluate, you can extend the timeout period by configuring the `ignore.normalizer.jq.timeout` setting within the `argocd-cmd-params-cm` ConfigMap. 201 202 ```yaml 203 apiVersion: v1 204 kind: ConfigMap 205 metadata: 206 name: argocd-cmd-params-cm 207 data: 208 ignore.normalizer.jq.timeout: '5s' 209 ```