Setting up Argo CD Image Updater with Azure Container Registry

Knowledge base Continuous Delivery and Deployment Setting up Argo CD Image Updater with Azure Container Registry

This article makes you familiar with setting up Argo CD Image Updater in your own Argo CD environment. In this scenario, Azure Container Registry and GitHub are used as connected services.

Please note that Argo CD Image Updater will be merged into Argo CD in the near future.

Introduction

Jump over to argocd-image-updater releases and argo-cd releases to grab the latest binaries. For testing purposes the turn-around times are much lower with a local environment.

Creating a local Argo CD user for Image Updater

Create local account

Update your argocd-cm ConfigMap to add a new account with name image-updater:

$ kubectl edit configmaps argocd-cm -n argocd

data:
  accounts.image-updater: apiKey

If you are using Helm, your values.yaml has to look like this:

configs:
  cm:
    accounts.image-updater: apiKey

Create a new API token

You have to create a new API token in a JWT format for the recently created image-updater user:

# port-forwarding to let argo binary access Kubernetes
$ kubectl port-forward svc/argocd-server -n argocd 8080:443

# grab `admin` secret; alternatively use other credentials
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# log in into Argo CD
$ ./argo login 127.0.0.1:8080

# create a new token
$ ./argo account generate-token --account image-updater --id image-updater
# Output = ${IMAGE_UPDATER_JWT}

Note down the ${IMAGE_UPDATER_JWT} credentials.

Set up RBAC

You have to allow image-updater to query and update Argo CD's Application CRDs:

$ kubectl edit configmaps argocd-rbac-cm -n argocd

data:
  policy.csv: |
    # assign permissions to role:image-updater
    p, role:image-updater, applications, get, */*, allow
    p, role:image-updater, applications, update, */*, allow
    # grant account image-updater the role "image-updater"
    g, image-updater, role:image-updater
  policy.default: ""
  scopes: '[groups]'
kind: ConfigMap

First local run

At the moment, your Argo CD Applications are not configured. Start Image Updater with

$ export ARGOCD_TOKEN=${IMAGE_UPDATER_JWT}
$ ./argocd-image-updater run --once

The output will look like this

INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:INFO, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.default, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
INFO[0000] Finished cache warm-up, pre-loaded 0 meta data entries from 1 registries
INFO[0000] Starting image update cycle, considering 0 annotated application(s) for update
INFO[0000] Processing results: applications=0 images_considered=0 images_skipped=0 images_updated=0 errors=0
INFO[0000] Finished.

Update Application CRD to let Image Updater pick them up

Image Updater checks each Kubernetes Application resource. If the argocd-image-updater.argoproj.io annotation is present, that resource is considered by Image Updater. Pick one of your Application resources and add the annotations argocd-image-updater.argoproj.io/write-back-method and argocd-image-updater.argoproj.io/image-list:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # In the spec.source.repoURL, Image Updater must write back the latest pulled image tag
    argocd-image-updater.argoproj.io/write-back-method: git
    # /image-list ist the image's repository
    # `img-alias` is just a name which can be referenced later
    argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
# ...
spec:
  source:
    # ...
    repoURL: git@github.com:my-repository/my-artifact-gitops.git

Now start Image Updater and pass the namespace (--argocd-namespace parameter) in which the Argo CD Applications are located:

$ ./argocd-image-updater run --once --loglevel trace --argocd-namespace argocd

INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:TRACE, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
DEBU[0000] Successfully parsed commit message template
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
DEBU[0000] Creating in-cluster Kubernetes client
DEBU[0000] Using ArgoCD API credentials from environment ARGOCD_TOKEN
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.argocd, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm'  application=my-artifact-prod
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update             alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=my-container-registry.azurecr.io registry="https://my-container-registry.azurecr.io"
DEBU[0000] Inferred registry from prefix my-container-registry.azurecr.io to use API https://my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag  alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest                  image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found                     image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found               image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64  image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No pull-secret annotation found               image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
ERRO[0000] Could not get tags from registry: Get "https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.  alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
INFO[0000] Finished cache warm-up, pre-loaded 0 meta data entries from 2 registries
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm'  application=my-artifact-prod
INFO[0000] Starting image update cycle, considering 1 annotated application(s) for update
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update             alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag  alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest                  image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found                     image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found               image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64  image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No pull-secret annotation found               image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
ERRO[0006] Could not get tags from registry: Get "https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.  alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
INFO[0006] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1
INFO[0006] Finished.

In the output above, Image Updater fails with unauthorized: authentication required. This happens because Image Updater does not know, which credentials have to be used to authenticate against the Docker registry.

Configuring credentials for accessing the container registry

Image Updater offers various ways to access a Docker registry. In our case, we are using Azure Container Registry for storing the Docker Images. Those credentials have been previously configured with

$ kubectl -n my-artifact-prod create secret docker-registry acr-dreitier-my-artifact \
    --docker-server=dreitier.azurecr.io \
    --docker-username=${ACR_SP_RO_USERNAME} \
    --docker-password=${ACR_SP_RO_PASSWORD} \ 
    --docker-email=${EMAIL_ADDRESS}

${ACR_SP_RO_USERNAME} and ${ACR_SP_RO_USERNAME} are read-only credentials for the Azure Container Registry.

Now add the argocd-image-updater.argoproj.io/<imag>.pull-secret annotation to your Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # /image-list ist the image's repository
    # `img-alias` is just a name which can be referenced later
    argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
    # /${img-alias}.pull-secret  references the previously configured secret
    argocd-image-updater.argoproj.io/img-alias.pull-secret: pullsecret:my-artifact-prod/acr-dreitier-my-artifact

Please note that the annotation is named pull-secret but the prefix type is pullsecret. This is not a typo.

The next run of Image Updater will look like this:

INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:TRACE, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
DEBU[0000] Successfully parsed commit message template
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
DEBU[0000] Creating in-cluster Kubernetes client
DEBU[0000] Using ArgoCD API credentials from environment ARGOCD_TOKEN
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.argocd, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm'  application=my-artifact-prod
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update             alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=my-container-registry.azurecr.io registry="https://my-container-registry.azurecr.io"
DEBU[0000] Inferred registry from prefix my-container-registry.azurecr.io to use API https://my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest                  image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found                     image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found               image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64  image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Fetching credentials for registry https://my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
TRAC[0000] Getting manifest for image my-repository/my-artifact:20221104105424fb448a (operation 1/4)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore                   alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:20221107114352a4594b (operation 2/4)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore                   alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:2022110713512263c634 (operation 3/4)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore                   alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:latest (operation 4/4)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore                   alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/20221107114352a4594b
TRAC[0001] Manifest digest is 6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:69f99265994c00257bd8f9eade1b78e78466ecf0b29a0fa868a53cb09965a245  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/20221104105424fb448a
TRAC[0001] Manifest digest is 5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:6a3e085f46103e5542c41dd41d7e7164bd40ee43097ee4a364d95ed1ddc00597  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/2022110713512263c634
TRAC[0001] Manifest digest is 5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:da7b5d0f06c298ccf7161ebf67a71ebaa9ad07208f757470e1620551025d6a1e  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/latest
TRAC[0001] Manifest digest is 5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:da7b5d0f06c298ccf7161ebf67a71ebaa9ad07208f757470e1620551025d6a1e  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/6a/6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df/data?se=2022-11-10T11%3A07%3A38Z&sig=GKLqS2gQMSsEHAaVe4sy9%2BVDAUYzW8fiIHJnRycpYlY%3D&sp=r&spr=https&sr=b&sv=2016-05-31&regid=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/58/5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a/data?se=2022-11-10T11%3A07%3A38Z&sig=6zIfrKgDVtUybCPWwbKgAsubUEi8sEbnzWaX1BZXrQs%3D&sp=r&spr=https&sr=b&sv=2016-05-31&regid=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/5c/5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293/data?se=2022-11-10T11%3A07%3A38Z&sig=5%2B1GH7AUDO0yBtWwOjGaE8xEt5oHTpoibeOikts7UTQ%3D&sp=r&spr=https&sr=b&sv=2016-05-31&regid=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/5c/5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293/data?se=2022-11-10T11%3A07%3A38Z&sig=5%2B1GH7AUDO0yBtWwOjGaE8xEt5oHTpoibeOikts7UTQ%3D&sp=r&spr=https&sr=b&sv=2016-05-31&regid=8829767f21034a549d04966708c98cc3
TRAC[0001] Found date 2022-11-04 10:55:58.095180162 +0000 UTC  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 11:45:40.300617652 +0000 UTC  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 13:52:43.024595525 +0000 UTC  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 13:52:43.024595525 +0000 UTC  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] List of available tags found: [20221107114352a4594b 2022110713512263c634 latest 20221104105424fb448a]  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Finding out whether to consider 20221104105424fb448a for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider 20221107114352a4594b for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider 2022110713512263c634 for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider latest for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
DEBU[0001] found 4 from 4 tags eligible for consideration  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
INFO[0001] Setting new image to my-container-registry.azurecr.io/my-repository/my-artifact:latest  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] target parameters: image-spec= image-name=image.name, image-tag=image.tag  application=my-artifact-prod image=my-container-registry.azurecr.io/my-repository/my-artifact
INFO[0001] Successfully updated image 'my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634' to 'my-container-registry.azurecr.io/my-repository/my-artifact:latest', but pending spec update (dry run=true)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] Using commit message: build: automatic update of my-artifact-prod

updates image my-repository/my-artifact tag '2022110713512263c634' to 'latest'
INFO[0001] Dry run - not commiting 1 changes to application  application=my-artifact-prod
INFO[0001] Finished cache warm-up, pre-loaded 4 meta data entries from 2 registries
TRAC[0001] processing app 'my-artifact-prod' of type 'Helm'  application=my-artifact-prod
INFO[0001] Starting image update cycle, considering 1 annotated application(s) for update
DEBU[0001] Processing application my-artifact-prod
DEBU[0001] Considering this image for update             alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] Using no version constraint when looking for a new tag  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Found update strategy latest                  image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] No match annotation found                     image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] No ignore-tags annotation found               image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Using runtime platform constraint linux/amd64  image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Fetching credentials for registry https://my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
DEBU[0002] Cache hit for my-repository/my-artifact:20221104105424fb448a  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:20221107114352a4594b  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:2022110713512263c634  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:latest    alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0002] List of available tags found: [20221104105424fb448a 20221107114352a4594b 2022110713512263c634 latest]  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0002] Finding out whether to consider 20221104105424fb448a for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider 20221107114352a4594b for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider 2022110713512263c634 for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider latest for being updateable  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
DEBU[0002] found 4 from 4 tags eligible for consideration  image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
INFO[0002] Setting new image to my-container-registry.azurecr.io/my-repository/my-artifact:latest  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] target parameters: image-spec= image-name=image.name, image-tag=image.tag  application=my-artifact-prod image=my-container-registry.azurecr.io/my-repository/my-artifact
INFO[0002] Successfully updated image 'my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634' to 'my-container-registry.azurecr.io/my-repository/my-artifact:latest', but pending spec update (dry run=false)  alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Using commit message: build: automatic update of my-artifact-prod

updates image my-repository/my-artifact tag '2022110713512263c634' to 'latest'
INFO[0002] Committing 1 parameter update(s) for application my-artifact-prod  application=my-artifact-prod
INFO[0002] Starting configmap/secret informers
INFO[0003] Configmap/secret informer synced
INFO[0003] configmap informer cancelled
INFO[0003] Initializing git@github.com:my-repository/my-artifact-gitops.git to /tmp/git-my-artifact-prod1210370583
INFO[0003] rm -rf /tmp/git-my-artifact-prod1210370583   dir= execID=0fb01
INFO[0003] Trace                                         args="[rm -rf /tmp/git-my-artifact-prod1210370583]" dir= operation_name="exec rm" time_ms=4.302099999999999
INFO[0003] git fetch origin --tags --force               dir=/tmp/git-my-artifact-prod1210370583 execID=226b8
ERRO[0003] `git fetch origin --tags --force` failed exit status 128: No ED25519 host key is known for github.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.  execID=226b8
INFO[0003] Trace                                         args="[git fetch origin --tags --force]" dir=/tmp/git-my-artifact-prod1210370583 operation_name="exec git" time_ms=394.93489999999997
ERRO[0003] Could not update application spec: `git fetch origin --tags --force` failed exit status 128: No ED25519 host key is known for github.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.  application=my-artifact-prod
INFO[0003] Processing

Image Updater fails with No ED25519 host key is known for github.com and you have requested strict checking. Host key verification failed..

Configuring SSH access

This problem is very specific to Git repositories, when Image Updater accesses the repository over SSH. Git repositories provided over HTTPS are not affected.

Due to Image Updater's current implementation, the SSH option StrictHostKeyChecking=yes is always provided as command line parameter. There is no option to easily override that behaviour as the provided command line parameter has the highest precedence:

$ sudo strace -f -s 2048 ./argocd-image-updater --once --loglevel trace --argocd-namespace argocd --kubeconfig /home/ckl/.kube/config
# ...
2195  execve("/bin/sh", ["/bin/sh", "-c", "ssh -i /dev/shm/4050667513 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts \"$@\"", "ssh -i /dev/shm/4050667513 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts", "-o", "SendEnv=GIT_PROTOCOL", "git@github.com", "git-upload-pack 'my-repository/my-artifact-gitops.git'"], 0x7fffee30dd90 /* 18 vars */ <unfinished ...>
# ...

To fix this issue, add github.com's ssh-ed25519 as known host key to your environment. First, grap github.com's ED25519 host key:

ssh-keyscan -t ssh-ed25519 github.com
# output = ${KNWON_HOST_KEY}

Add the SSH host key in local environment

In a local environment, the key must be added to /app/config/ssh/ssh_known_hosts:

mkdir -p /app/config/ssh
echo ${KNWON_HOST_KEY} > /app/config/ssh/ssh_known_hosts

Adding the SSH host key in Kubernetes

When deployed to Kubernetes, you have to configure the argocd-ssh-known-hosts-cm ConfigMap:

$ kubectl edit configmaps argocd-ssh-known-hosts-cm -n argocd

# add ${KNWON_HOST_KEY}

Configuring the correct Git branch

After configuring the SSH host key, the next Image Updater run might fail with the error error: pathspec 'master' did not match any file(s) known to git:

INFO[0004] git checkout --force master                   dir=/tmp/git-my-artifact-prod4089267324 execID=e1fd2
ERRO[0004] `git checkout --force master` failed exit status 1: error: pathspec 'master' did not match any file(s) known to git  execID=e1fd2
INFO[0004] Trace                                         args="[git checkout --force master]" dir=/tmp/git-my-artifact-prod4089267324 operation_name="exec git" time_ms=6.1113
ERRO[0004] Could not update application spec: `git checkout --force master` failed exit status 1: error: pathspec 'master' did not match any file(s) known to git  application=my-artifact-prod
INFO[0004] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1
INFO[0004] Finished.

By default, Image Updater checks out the master branch. Add the argocd-image-updater.argoproj.io/git-branch annotation with your repositories primary branch to your Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # /image-list ist the image's repository
    # `img-alias` is just a name which can be referenced later
    argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
    # /${img-alias}.pull-secret 
    argocd-image-updater.argoproj.io/img-alias.pull-secret: pullsecret:my-artifact-prod/acr-dreitier-my-artifact
    # Used branch
    argocd-image-updater.argoproj.io/git-branch: main

After having set everything up, Argo CD Image Updater does now successfully run:

INFO[0010] Configmap/secret informer synced
INFO[0010] Initializing git@github.com:my-repository/my-artifact-gitops.git to /tmp/git-my-artifact-prod2883806823
INFO[0010] rm -rf /tmp/git-my-artifact-prod2883806823   dir= execID=5a0ef
INFO[0010] configmap informer cancelled
INFO[0010] Trace                                         args="[rm -rf /tmp/git-my-artifact-prod2883806823]" dir= operation_name="exec rm" time_ms=4.048
INFO[0010] git fetch origin --tags --force               dir=/tmp/git-my-artifact-prod2883806823 execID=52a5a
INFO[0011] secrets informer cancelled
INFO[0013] Trace                                         args="[git fetch origin --tags --force]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=3146.2313000000004
INFO[0013] git config user.name argocd-image-updater     dir=/tmp/git-my-artifact-prod2883806823 execID=885fa
INFO[0013] Trace                                         args="[git config user.name argocd-image-updater]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=4.8975
INFO[0013] git config user.email noreply@argoproj.io     dir=/tmp/git-my-artifact-prod2883806823 execID=e9be4
INFO[0013] Trace                                         args="[git config user.email noreply@argoproj.io]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=5.0155
TRAC[0013] targetRevision for update is 'main'           application=my-artifact-prod
INFO[0013] git checkout --force main                     dir=/tmp/git-my-artifact-prod2883806823 execID=bcffb
INFO[0013] Trace                                         args="[git checkout --force main]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=67.1502
INFO[0013] git clean -fdx                                dir=/tmp/git-my-artifact-prod2883806823 execID=4a84b
INFO[0013] Trace                                         args="[git clean -fdx]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=11.4397
DEBU[0013] Writing commit message to /tmp/image-updater-commit-msg1539850089  application=my-artifact-prod
INFO[0013] git commit -a -F /tmp/image-updater-commit-msg1539850089  dir=/tmp/git-my-artifact-prod2883806823 execID=f1eca
INFO[0013] Trace                                         args="[git commit -a -F /tmp/image-updater-commit-msg1539850089]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=80.8113
INFO[0013] git push origin main                          dir=/tmp/git-my-artifact-prod2883806823 execID=57f92
INFO[0016] Trace                                         args="[git push origin main]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=2071.5395
INFO[0016] Successfully updated the live application spec  application=my-artifact-prod
INFO[0016] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0
INFO[0016] Finished.