GitOps with Argo CD and Helm: structure and image promotion

With GitOps, a Git repository describes the desired state of your environments, and a tool such as Argo CD makes sure the cluster matches that state. Sounds simple, but in practice, two questions come up quickly: How do you structure the repository for multiple environments? And how does a new Docker image automatically end up in the right environment? In this guide, we show how we solve both with Argo CD and Helm.

One environment per repository, branch or directory?

GitOps projects with Argo CD can be set up in three ways:

For most use cases, one environment per directory is the best compromise. That's what we focus on below.

The directory structure

Each environment gets its own directory in the repository:

.
├── argocd
│   ├── project.yaml
│   └── repo-secrets.yaml
├── Chart.yaml
├── environments
│   ├── asia
│   │   ├── dev
│   │   ├── test
│   │   └── prod
│   └── eu
│       ├── dev
│       ├── test
│       │   ├── argocd
│       │   │   └── application.yaml
│       │   └── values.yaml
│       └── prod
└── templates
    └── _helpers.tpl

If you work with Kustomize instead of Helm, overlays directories play the same role.

If certain environments, for example production, need to be separated from the rest for security reasons, move their directories to a second repository:

myapp-gitops-prod.git
myapp-gitops-dev-test.git

Several environments in one commit

With one environment per directory, you can change several environments in one commit, for example moving DEV and TEST to a new image at the same time. Each environment is its own application: if syncing one environment fails, the others are unaffected, and a rollback is a simple git revert. So in principle, nothing speaks against such commits.

If you want to prevent them for organisational reasons, this is only possible with a pre-receive hook that limits each commit to one directory. GitHub Enterprise, Bitbucket Data Center and GitLab support such hooks; GitHub in the cloud does not.

The Argo CD application with Helm

Helm does support subcharts, but in practice it is simpler to use a single Helm chart for the application and only adjust the values.yaml per environment. It lives in the respective environment's directory:

environments
└── eu
    └── prod
        ├── argocd
        │   └── application.yaml
        └── values.yaml

The Application manifest points to the chart in the root directory and to the environment's values.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-eu-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: myapp
  source:
    repoURL: git@github.com:my-org/myapp-gitops.git
    targetRevision: HEAD
    # the Helm chart lives in the root directory of the repository
    path: .
    helm:
      # relative to spec.source.path
      valueFiles:
        - environments/eu/prod/values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp-prod

Additional parameters via .argocd-source.yaml

During deployment, Argo CD checks whether the files .argocd-source.yaml and .argocd-source-<appname>.yaml exist in the spec.source.path directory. Their parameters override the values from values.yaml, with the application-specific file taking precedence:

helm:
  parameters:
    - name: image.tag
      value: 2025110713512263c634
      forceString: true

The Argo CD Image Updater, for example, uses these files to update the image tag. Note that they live in the spec.source.path directory, which in our example is the root directory, not the environment's directory.

Webhooks: update only the environment that changed

When GitHub triggers the Argo CD webhook, Argo CD checks all applications pointing to the repository by default. With the annotation argocd.argoproj.io/manifest-generate-paths, you limit this to the paths relevant to an application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-eu-prod
  annotations:
    argocd.argoproj.io/manifest-generate-paths: ./Chart.yaml;./templates;./environments/eu/prod;./.argocd-source-myapp-eu-prod.yaml

Now myapp-eu-prod is only updated if the chart, the environment or its .argocd-source file has changed.

Two pitfalls: if you use the annotation, you have to list the .argocd-source file explicitly. And the annotation only applies to the webhook. A sync via the UI or CLI ignores it and updates to the latest commit.

Rolling out new images automatically

That leaves the second question: how does a newly built image end up in the right environment? We use two approaches.

Approach 1: Argo CD Image Updater

The Argo CD Image Updater watches your container registry and updates the image tag as soon as a new image appears, either directly in the application or, more traceably, by committing back to the Git repository.

For this, it needs its own Argo CD account with an API token, RBAC permissions to read and update applications, read access to the registry and write access to the GitOps repository for writing back.

Note: the Image Updater's configuration has changed significantly between versions. Follow the documentation for exactly the version you use.

The Image Updater is a good fit if you want to pick up image tags according to fixed rules, such as "always the latest patch version".

Approach 2: promotion from the CI pipeline

Often you want more control: an image tagged *-dev should only update DEV, one tagged *-stable only PROD. Then you let the CI pipeline of the application repository notify the GitOps repository. On GitHub, this works with the repository_dispatch event.

1. Set up access. The application repository needs write access to exactly one repository: the GitOps repository. Use a fine-grained personal access token or a GitHub App for this, limited to that one repository with the permissions Contents: Read and write and Metadata: Read-only. Store the token and repository name as secrets, for example GITOPS_REPO and GITOPS_REPO_TOKEN.

2. Fire the event. After the pipeline has built the image and pushed it to the registry, it sends the event with the branch and image tag:

- name: Notify GitOps repository
  env:
    GH_TOKEN: ${{ secrets.GITOPS_REPO_TOKEN }}
  run: |
    gh api repos/${{ secrets.GITOPS_REPO }}/dispatches \
      -f event_type=update-gitops-repo \
      -f "client_payload[git_branch]=${GITHUB_REF_NAME}" \
      -f "client_payload[docker_image_tag]=${{ steps.publish.outputs.tag }}"

3. React in the GitOps repository. A workflow listens for the event, updates the matching values.yaml and commits the change. For mapping "which tag updates which environment", we built the GitHub Action dreitier/conditional-regex-search-and-replace-action:

name: Update environment from upstream image

on:
  repository_dispatch:
    types: [update-gitops-repo]

jobs:
  update-values:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: dreitier/conditional-regex-search-and-replace-action@main
        id: replace
        with:
          # -dev only updates dev, -stable only prod
          mappings: "docker_image_tag==.*-dev {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex {NEXT_MAPPING} docker_image_tag==.*-stable {THEN_UPDATE_FILES} **prod/values.yaml=docker_image_tag_regex&git_branch_regex"
          docker_image_tag: ${{ github.event.client_payload.docker_image_tag }}
          docker_image_tag_regex: 'imageTag: "(?<docker_image_tag>.*)"'
          git_branch: ${{ github.event.client_payload.git_branch }}
          git_branch_regex: 'git_branch: "(?<git_branch>.*)"'

      - name: Commit updated environment
        if: ${{ steps.replace.outputs.total_modified_files >= 1 }}
        uses: EndBug/add-and-commit@v9
        with:
          add: environments/*
          message: "Update to ${{ github.event.client_payload.docker_image_tag }} from ${{ github.event.client_payload.git_branch }}"

Argo CD then picks up the change automatically, or immediately if GitHub triggers the Argo CD webhook.

Conclusion

With one environment per directory, a shared Helm chart and targeted webhooks, your GitOps repository stays manageable even as the number of environments grows. Whether new images arrive via the Image Updater or via the CI pipeline depends on how much control you need over promotion.

Building a platform with Kubernetes and GitOps right now, or want an existing setup reviewed? That's exactly what we do in our DevOps and Kubernetes consulting.

Originally published on November 9, 2022.