Using environment-per-folder directory structure for GitOps projects with Argo CD and Helm

GitOps projects with Argo CD can be set up either as environment-per-repository, environment-per-branch or environment-per-directory. This article looks at environment-per-directory, as it is the best compromise for many use cases. The advantages and disadvantages of environment-per-branch have been discussed in detail elsewhere.

Directory structure for environment-per-folder strategy

A separate directory is created within the repository for each environment:

$ pwd
/myapp-gitops

$ git branch
* main

$ tree
.
├── argocd
│   ├── project.yaml
│   └── repo-secrets.yaml
├── Chart.yaml
├── environments
│   ├── asia
│   │   ├── dev
│   │   │   └── argocd
│   │   ├── prod
│   │   │   └── argocd
│   │   └── test
│   │       ├── argocd
│   │       │   └── application.yaml
│   │       └── values.yaml
│   └── eu
│       ├── dev
│       │   └── argocd
│       ├── prod
│       │   └── argocd
│       └── test
│           └── argocd
└── templates
    └── helpers.tpl

When using kustomize, matching overlays directories can be used. The environment-per-folder strategy allows easy and fast modification to multiple environments. If certain environments (e.g. PROD and TEST) should be separated for security reasons, the folders can each be moved to a second Git repository:

myapp-gitops-prod.git
myapp-gitops-dev-test.git

Rollbacks for changes to multiple environments in one commit

environment-per-folder allows multiple environments to be changed in one commit. For example, one commit could update both the DEV and TEST environments to a new Docker image. If Argo CD cannot update one of the Kubernetes service due to a failure, Argo CD uses the previous Git commit. So, in principle, there is no problem with updating multiple environments within one commit.

If you want to prevent changes to multiple environments for organizational reasons, this is only possible via a pre-receive hook. GitHub (Cloud) does not support pre-receive hooks, however GitHub Enterprise, BitBucket Server and GitLab do. You can use this pre-receive hook to limit changes to one directory.

Argo CD Application Kubernets manifest using Helm

Helm does have the feature of subcharts, however Argo CD cannot resolve dependencies when deploying. For this reason, it is simpler to use only the standard Helm chart of the application with the values.yaml parameterized in each case. The values.yaml is then located within the directory of the respective environment:

$ tree
.
├── environments
│   ├── eu
│   │   └── prod
│   │       ├── argocd
│   │       │   └── application.yaml
│   │       └── values.yaml

The application.yaml manifest then contains the path to the respective values.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-eu-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  labels:
    name: myapp-eu-prod 
spec:
  project: myapp-argo-project

  source:
    repoURL: git@github.com:my-orga/myapp-gitops.git
    targetRevision: HEAD
    # We are using the root directory of the Git repository to install the Helm chart
    path: .

    helm:
      # Helm values files for overriding values in the helm chart
      # The path is relative to the spec.source.path directory defined above
      valueFiles:
      - environments/prod/values.yaml

      version: v3

Additional parameters via .argocd-source.yaml and .argocd-source-<appname>.yaml

During deployment Argo CD checks if within spec.source.path (. in our case) the files

• .argocd-source.yaml
• and .argocd-source-<appname>.yaml are present.

are present. .argocd-source-<appname>.yaml is used by Argo CD Image Updater to automatically update the Docker image tag.

The structure of the two files is:

helm:
  parameters:
  - name: image.tag
    value: 2022110713512263c634
  forcestring: true

Entries from argocd-source-<appname>.yaml have higher priority than argocd-source.yaml. Existing entries in environments/prod/values.yaml will be overwritten by the parameters in both files.

Unfortunately, at the moment it is not possible to specify that .argocd-source-<appname>.yaml is located in environments/eu/prod. Instead, when using the image updater in the above example, the root directory is used.

Argo CD's WebHooks and environment-per-folder

As soon as GitHub triggers the Argo CD webhook, a sync of all applications takes place by default. To update only the changed environment at a time, the Manifest Path Annotation can be used.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  # ...
  labels:
    name: myapp-eu-prod 
  annotations:
    argocd.argoproj.io/manifest-generate-paths: ./Chart.yaml;./templates;./environments/eu/prod;./.argocd-source-myapp-eu-prod.yaml
spec:
  # ...

With this, when running the WebHook, only the myapp-eu-prod will be updated if the helmet chart, environment or the specific .argocd-source-<appname>.yaml has changed.