In a GitOps process, promoting new Docker images to different environments in Helm charts can be cumbersome. This article describes how new Docker image releases can be promoted to Helm charts for different environments. It leverages GitHub Actions to do this.
Introduction
When implementing GitOps processes, a normal requirement is that your Helm charts or kustomize YAMLs have to be updated with the new Docker image tag. There are various ways, how such a process can be implemented on the GitHub platform. We use GitHub's repository_dispatch
event for this.
Honor where credit is due: Rustam has inspired this article.
Let us set the scene:
- You have two dedicated GitHub repositories:
- One repository which contains the application sources. After a build succeeds, its GitHub workflow creates new Docker images and pushes them to a registry. This repository is the
${UPSTREAM_APPLICATION_SOURCE_REPOSITORY}
. - A second repository with your Helm charts and a folder-per-environment directory structure. This is the
${DOWNSTREAM_GITOPS_REPO}
.
- One repository which contains the application sources. After a build succeeds, its GitHub workflow creates new Docker images and pushes them to a registry. This repository is the
- Both Git repositories are located inside the same GitHub organization.
- Depending upon the created Docker image tag, only one of the environments must be updated. When a Docker image tag like
.*-stable
is pushed, we want to update theprod
environment. If a Docker image tag.*-dev
is pushed, thedev
environment must be updated.
The following directory structure can be found in the repository ${DOWNSTREAM_GITOPS_REPO}
:
.
├── Chart.yaml
└── environments
│──dev
│ └── values.yaml
└──prod
└── values.yaml
Notifying the downstream GitOps repository about new Docker images
If you have already used Jenkins, you might be familiar with Jenkins' concept of having downstream projects and how to trigger them. This concept is not available in GitHub. Luckily, GitHub provdes two events: repository_dispatch
and workflow_dispatch
. Those events can be dispatched by an upstream repository to trigger events in a downstream repository.
Creating a Personal Access Token
First of all, we need a Personal Access Token (PAT) inside the ${UPSTREAM_APPLICATION_SOURCE_REPOSITORY}
with limited access to the ${DOWNSTREAM_GITOPS_REPO}
.
Before October 2022, this would introduce large security issues: PATs could only be assigned to all repositories of a user or organization but not limited to a single repository. This has changed at the end of October 2022. GitHub released the Fine-grained tokens feature.
In your GitHub account, go to Settings > Developer Settings > Personal access tokens > Fine-grained tokens:
- As Resource owner, select the organization which both of the repositories belong to
- In Repository access > Only select repositories, select the
${DOWNSTREAM_GITOPS_REPO}
- For Repository permissions, select
- Contents > Access: Read and write
- Metadata > Read-only
Create the new token and store the string github_pat_...
(${PERSONAL_ACCESS_TOKEN}
) in your password manager.
Add the Personal Access Token to your upstream repository
In your ${UPSTREAM_APPLICATION_SOURCE_REPOSITORY}
go to Settings > Secrets > Actions and add the following three secrets:
Name | Secret |
---|---|
DOWNSTREAM_GITOPS_REPO_OWNER_AND_REPOSITORY |
Name of the downstream owner and repository ${DOWNSTREAM_GITOPS_REPO} , e.g. myorg/myapp |
DOWNSTREAM_GITOPS_REPO_PAT |
The recently created ${PERSONAL_ACCESS_TOKEN} |
After that, the Secrets > Actions overview should look like this:
Fire the repository_dispatch
event
After your ${UPSTREAM_APPLICATION_SOURCE_REPOSITORY}
has created and pushed a new Docker image, it has have to notify the downstream repository. For this, open up your GitHub Actions workflow's YAML file. In the following sample code, we use elgohr/Publish-Docker-Github-Action for pushing the recently created Docker image:
# ...
jobs:
deploy:
runs-on: ubuntu-latest
steps:
# ...
- name: Publish to Registry
id: publish_to_registry
uses: elgohr/Publish-Docker-Github-Action@v4
with:
name: myorg/myapp
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ghcr.io
snapshot: true
- name: Dispatch to downstream GitOps/infra repository
run: |
commitMsg=$(git log -n 1 --oneline --format=%s)
branchName="${GITHUB_REF##*/}"
curl -X POST https://api.github.com/repos/${{ secrets.DOWNSTREAM_GITOPS_REPO_OWNER_AND_REPOSITORY }}/dispatches \
-H 'Accept: application/vnd.github.everest-preview+json' \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Authorization: Bearer ${{ secrets.DOWNSTREAM_GITOPS_REPO_PAT }}" \
--data '{"event_type": "update-gitops-repo", "client_payload": { "git_branch": "'"$branchName"'", "docker_image_tag": "${{ steps.publish_to_registry.outputs.snapshot-tag }}", "commit_message": "'"$commitMsg"'" }}'
The curl
command calls the https://api.github.com/repos/${{ secrets.DOWNSTREAM_GITOPS_REPO_OWNER_AND_REPOSITORY }}/dispatches
endpoint and provides the required metadata (git_branch
, docker_image_tag
) to it. You can put more data in it as you require.
Also note that you can easily copy&paste the curl
command to your local terminal for testing purposes.
Update values.yaml
in your downstream GitOps repository
In your ${DOWNSTREAM_GITOPS_REPO_OWNER_AND_REPOSITORY}
repository, you have to create a new GitHub Actions workflow.
Paste in the following code:
name: Dispatch event from upstream application repository
on:
# listen to the repository_dispatch event with event_type `update-gitops-repo`
repository_dispatch:
types: [update-gitops-repo]
jobs:
update-values-yaml:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: dreitier/conditional-regex-search-and-replace-action@main
id: search_and_replace
with:
# define conditions that we only update the dev environment if .*-dev Docker image tag is present
mappings: "docker_image_tag==.*-dev {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex {NEXT_MAPPING} docker_image_tag==.*-prod {THEN_UPDATE_FILES} **prod/values.yaml=docker_image_tag_regex&git_branch_regex"
# extract the docker image tag from the payload the upstream has sent
docker_image_tag: "${{ github.event.client_payload.docker_image_tag }}"
docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""
# extract the Git branch the payload the upstream has sent
git_branch: "${{ github.event.client_payload.git_branch }}"
git_branch_regex: "git_branch: \\\"(?<git_brach>.*)\\\""
- name: Commit updated environment
# only do a commit if at least one file has been modified
if: ${{ steps.search_and_replace.outputs.total_modified_files >= 1 }}
uses: EndBug/add-and-commit@v7
with:
author_name: build@internal
author_email: build@internal
add: "environments/*"
message: "Branch ${{ github.event.client_payload.git_branch }} led to updated Docker image '${{ github.event.client_payload.docker_image_tag }}': ${{ github.event.client_payload.commit_message }}"
With
on:
# listen to the repository_dispatch event with event_type `update-gitops-repo`
repository_dispatch:
types: [update-gitops-repo]
the GitHub Action workflow is only executed, after the repository_dispatch
event with the specified event_type
has been fired.
Our dreitier/conditional-regex-search-and-replace-action GitHub action is the worker horse: With
mappings: "docker_image_tag==.*-dev {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex {NEXT_MAPPING} docker_image_tag==.*-prod {THEN_UPDATE_FILES} **prod/values.yaml=docker_image_tag_regex&git_branch_regex"
we specify that only the dev/values.yaml
should be updated, if docker_image_tag
ends with -dev
.
Please take a look at the documentation if you want to map more sophistic directory structures, e.g. with multiple environments and stages.
After you have committed the changes and a new Docker image has been pushed to the registry, the ${DOWNSTREAM_GITOPS_REPO}
gets notified and updates the values.yaml
files accordingly. They can then either picked up automatically by Argo CD or you can let GitHub execute a webhook to notify Argo CD proactively.