🦑 Ignore differences in ArgoCD

A difference

Imagine the day you have your full gitops-process up and running and joyfully login to ArgoCD to see all running with green icons and then... there it is, a yellow icon indicating your app has drifted off from your gitops repository.

But you didn't change anything?

Analyze the difference

Luckily it's pretty easy to analyze the difference in an ArgoCD app. Just click on your application and the detail-view opens. In my case this came into my view:

And that explained it pretty quick! Sure I wanted to release a new version of the awesome-app. The code change which got pushed to the git repository triggered a new pipelinerun of the build-app pipeline - so far so good - but the new pipelinerun object build-app-xnhzw doesn't exist in the gitops repository!

A note on ArgoCD differences

In general, we can divide out-of-sync differences into two groups:

• differences in an object: That's the case if you have an object defined in a manifest and now some attributes get changed or added without any changes in your gitops repostory

• whole objects as differences: This is the case if someone adds new objects in your namespace where your app is located and managed by ArgoCD

With ArgoCD you can solve both cases just by changing a few manifests ;-)

Ignore differences in an object

If you want to ignore certain differences which may occur in a specific object then you can set an annotation in this object as described in the argocd-documentation:

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

It gets more interesting if you want to ignore certain attributes in all objects or in all objects of a certain kind of your app. ArgoCD also has a solution for this and this gets explained in their documentation. The example was a bit weired for me at first but after I tried it out it became clear to me how it can be used, here is an example how to ignore all imagepullsecrets of the serviceaccounts of your app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: awesome-app
  namespace: argocd-ns
spec:
  destination:
    namespace: awesome-ns
    server: 'https://kubernetes.default.svc'
  ignoreDifferences:
    - jsonPointers:
        - /imagePullSecrets
        - /secrets
      kind: ServiceAccount

If you add a name: attribue right under kind: ServiceAccount you can narrow the ignore down again to a specific sa.

Ignore objects as differences

This was much harder for me to find and at some point I thought this feature is missing at all.. Let's take a look at the screenshot I showed earlier:

ArgoCD tells me it's out of sync because of a PipelineRun object. If we click on it we see this detail difference view:

This means, the object is not known by ArgoCD at all! If we have autoprune enabled then ArgoCD would try to delete this object immediately which would be pretty bad for us because we want to get our new app built and the deletion cancels this all of a sudden.

The problem is that our pipeline is defined in our gitops-repository and ArgoCD automatically sets a label to the applied objects:

  labels:
    app.kubernetes.io/instance: awesome-app

If a pipelinerun gets created this run inherits the label

    app.kubernetes.io/instance: awesome-app

and because of this ArgoCD recognizes the pipelinerun as object which exists but is not present in our repository.

The ultimate solution of this problem is to ignore the whole object-kind (in my case the Tekton PipelineRun) at instance-level of our ArgoCD instance! If you have deployed ArgoCD with the awesome ArgoCD-Operator then just add resourceExclusions to your manifest of the instance:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: example-argocd
  namespace: argocd
spec:
  resourceExclusions: |
    - apiGroups:
      - tekton.dev
      kinds:
      - PipelineRun
      - TaskRun
      clusters:
      - "*"

If not then you can add resource.exclusions to your argocd-cm configmap as described in the argocd-docs.

Conclusion

That's it 🎉🎊! As you can see there are plenty of options to ignore certain types of differences, and from my point of view if you want to use a gitops-process to deploy apps there will be a situation where you need to ignore some tiny diffs - and it will be there soon!