This article makes you familiar with setting up Argo CD Image Updater in your own Argo CD environment. In this scenario, Azure Container Registry and GitHub are used as connected services.
Please note that Argo CD Image Updater will be merged into Argo CD in the near future.
Introduction
Jump over to argocd-image-updater releases and argo-cd releases to grab the latest binaries. For testing purposes the turn-around times are much lower with a local environment.
Creating a local Argo CD user for Image Updater
Create local account
Update your argocd-cm
ConfigMap to add a new account with name image-updater
:
$ kubectl edit configmaps argocd-cm -n argocd
data:
accounts.image-updater: apiKey
If you are using Helm, your values.yaml
has to look like this:
configs:
cm:
accounts.image-updater: apiKey
Create a new API token
You have to create a new API token in a JWT format for the recently created image-updater
user:
# port-forwarding to let argo binary access Kubernetes
$ kubectl port-forward svc/argocd-server -n argocd 8080:443
# grab `admin` secret; alternatively use other credentials
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
# log in into Argo CD
$ ./argo login 127.0.0.1:8080
# create a new token
$ ./argo account generate-token --account image-updater --id image-updater
# Output = ${IMAGE_UPDATER_JWT}
Note down the ${IMAGE_UPDATER_JWT}
credentials.
Set up RBAC
You have to allow image-updater
to query and update Argo CD's Application CRDs:
$ kubectl edit configmaps argocd-rbac-cm -n argocd
data:
policy.csv: |
# assign permissions to role:image-updater
p, role:image-updater, applications, get, */*, allow
p, role:image-updater, applications, update, */*, allow
# grant account image-updater the role "image-updater"
g, image-updater, role:image-updater
policy.default: ""
scopes: '[groups]'
kind: ConfigMap
First local run
At the moment, your Argo CD Applications are not configured. Start Image Updater with
$ export ARGOCD_TOKEN=${IMAGE_UPDATER_JWT}
$ ./argocd-image-updater run --once
The output will look like this
INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:INFO, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.default, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
INFO[0000] Finished cache warm-up, pre-loaded 0 meta data entries from 1 registries
INFO[0000] Starting image update cycle, considering 0 annotated application(s) for update
INFO[0000] Processing results: applications=0 images_considered=0 images_skipped=0 images_updated=0 errors=0
INFO[0000] Finished.
Update Application CRD to let Image Updater pick them up
Image Updater checks each Kubernetes Application resource. If the argocd-image-updater.argoproj.io
annotation is present, that resource is considered by Image Updater.
Pick one of your Application resources and add the annotations argocd-image-updater.argoproj.io/write-back-method
and argocd-image-updater.argoproj.io/image-list
:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
# In the spec.source.repoURL, Image Updater must write back the latest pulled image tag
argocd-image-updater.argoproj.io/write-back-method: git
# /image-list ist the image's repository
# `img-alias` is just a name which can be referenced later
argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
# ...
spec:
source:
# ...
repoURL: git@github.com:my-repository/my-artifact-gitops.git
Now start Image Updater and pass the namespace (--argocd-namespace
parameter) in which the Argo CD Applications are located:
$ ./argocd-image-updater run --once --loglevel trace --argocd-namespace argocd
INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:TRACE, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
DEBU[0000] Successfully parsed commit message template
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
DEBU[0000] Creating in-cluster Kubernetes client
DEBU[0000] Using ArgoCD API credentials from environment ARGOCD_TOKEN
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.argocd, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm' application=my-artifact-prod
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second prefix=my-container-registry.azurecr.io registry="https://my-container-registry.azurecr.io"
DEBU[0000] Inferred registry from prefix my-container-registry.azurecr.io to use API https://my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64 image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No pull-secret annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
ERRO[0000] Could not get tags from registry: Get "https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
INFO[0000] Finished cache warm-up, pre-loaded 0 meta data entries from 2 registries
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm' application=my-artifact-prod
INFO[0000] Starting image update cycle, considering 1 annotated application(s) for update
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64 image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No pull-secret annotation found image_alias=my-artifact-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
ERRO[0006] Could not get tags from registry: Get "https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. alias=my-artifact-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
INFO[0006] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1
INFO[0006] Finished.
In the output above, Image Updater fails with unauthorized: authentication required
. This happens because Image Updater does not know, which credentials have to be used to authenticate against the Docker registry.
Configuring credentials for accessing the container registry
Image Updater offers various ways to access a Docker registry. In our case, we are using Azure Container Registry for storing the Docker Images. Those credentials have been previously configured with
$ kubectl -n my-artifact-prod create secret docker-registry acr-dreitier-my-artifact \
--docker-server=dreitier.azurecr.io \
--docker-username=${ACR_SP_RO_USERNAME} \
--docker-password=${ACR_SP_RO_PASSWORD} \
--docker-email=${EMAIL_ADDRESS}
${ACR_SP_RO_USERNAME}
and ${ACR_SP_RO_USERNAME}
are read-only credentials for the Azure Container Registry.
Now add the argocd-image-updater.argoproj.io/<imag>.pull-secret
annotation to your Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
# /image-list ist the image's repository
# `img-alias` is just a name which can be referenced later
argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
# /${img-alias}.pull-secret references the previously configured secret
argocd-image-updater.argoproj.io/img-alias.pull-secret: pullsecret:my-artifact-prod/acr-dreitier-my-artifact
Please note that the annotation is named pull-secret
but the prefix type is pullsecret
. This is not a typo.
The next run of Image Updater will look like this:
INFO[0000] argocd-image-updater v0.12.0+aee153d starting [loglevel:TRACE, interval:once, healthport:off]
WARN[0000] commit message template at /app/config/commit.template does not exist, using default
DEBU[0000] Successfully parsed commit message template
WARN[0000] Registry configuration at /app/config/registries.conf could not be read: stat /app/config/registries.conf: no such file or directory -- using default configuration
DEBU[0000] Creating in-cluster Kubernetes client
DEBU[0000] Using ArgoCD API credentials from environment ARGOCD_TOKEN
INFO[0000] ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.argocd, auth_token=true, insecure=false, grpc_web=false, plaintext=false]
INFO[0000] Starting metrics server on TCP port=8081
INFO[0000] Warming up image cache
TRAC[0000] processing app 'my-artifact-prod' of type 'Helm' application=my-artifact-prod
DEBU[0000] Processing application my-artifact-prod
DEBU[0000] Considering this image for update alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second prefix=my-container-registry.azurecr.io registry="https://my-container-registry.azurecr.io"
DEBU[0000] Inferred registry from prefix my-container-registry.azurecr.io to use API https://my-container-registry.azurecr.io
DEBU[0000] Using no version constraint when looking for a new tag alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Found update strategy latest image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No match annotation found image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] No ignore-tags annotation found image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Using runtime platform constraint linux/amd64 image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0000] Fetching credentials for registry https://my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
TRAC[0000] Getting manifest for image my-repository/my-artifact:20221104105424fb448a (operation 1/4) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:20221107114352a4594b (operation 2/4) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:2022110713512263c634 (operation 3/4) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Getting manifest for image my-repository/my-artifact:latest (operation 4/4) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] acquired metadata semaphore alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0000] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/20221107114352a4594b
TRAC[0001] Manifest digest is 6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:69f99265994c00257bd8f9eade1b78e78466ecf0b29a0fa868a53cb09965a245 alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/20221104105424fb448a
TRAC[0001] Manifest digest is 5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:6a3e085f46103e5542c41dd41d7e7164bd40ee43097ee4a364d95ed1ddc00597 alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/2022110713512263c634
TRAC[0001] Manifest digest is 5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293 alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:da7b5d0f06c298ccf7161ebf67a71ebaa9ad07208f757470e1620551025d6a1e alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/manifests/latest
TRAC[0001] Manifest digest is 5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293 alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] v2 SHA digest is sha256:da7b5d0f06c298ccf7161ebf67a71ebaa9ad07208f757470e1620551025d6a1e alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/blobs/sha256:5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/6a/6accd1a6ba21b94fc450c4b3f508c49be1a67a567dd81e423a88c0cf9caac3df/data?se=2022-11-10T11%3A07%3A38Z&sig=GKLqS2gQMSsEHAaVe4sy9%2BVDAUYzW8fiIHJnRycpYlY%3D&sp=r&spr=https&sr=b&sv=2016-05-31®id=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/58/5880e45507093f88ddd9e3d7d860bb0760103dad30b7b0a031d558e2483a1c9a/data?se=2022-11-10T11%3A07%3A38Z&sig=6zIfrKgDVtUybCPWwbKgAsubUEi8sEbnzWaX1BZXrQs%3D&sp=r&spr=https&sr=b&sv=2016-05-31®id=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/5c/5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293/data?se=2022-11-10T11%3A07%3A38Z&sig=5%2B1GH7AUDO0yBtWwOjGaE8xEt5oHTpoibeOikts7UTQ%3D&sp=r&spr=https&sr=b&sv=2016-05-31®id=8829767f21034a549d04966708c98cc3
TRAC[0001] Performing HTTP GET https://dewcmanaged49.blob.core.windows.net/8829767f21034a549d04966708c98cc3-h1a9mqlmwe//docker/registry/v2/blobs/sha256/5c/5c54a74a7f29ca82852a432856371871db8b76d36726c42b0976c0fbf222c293/data?se=2022-11-10T11%3A07%3A38Z&sig=5%2B1GH7AUDO0yBtWwOjGaE8xEt5oHTpoibeOikts7UTQ%3D&sp=r&spr=https&sr=b&sv=2016-05-31®id=8829767f21034a549d04966708c98cc3
TRAC[0001] Found date 2022-11-04 10:55:58.095180162 +0000 UTC alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 11:45:40.300617652 +0000 UTC alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 13:52:43.024595525 +0000 UTC alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] Found date 2022-11-07 13:52:43.024595525 +0000 UTC alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] released semaphore and terminated waitgroup
TRAC[0001] List of available tags found: [20221107114352a4594b 2022110713512263c634 latest 20221104105424fb448a] alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Finding out whether to consider 20221104105424fb448a for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider 20221107114352a4594b for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider 2022110713512263c634 for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0001] Finding out whether to consider latest for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
DEBU[0001] found 4 from 4 tags eligible for consideration image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
INFO[0001] Setting new image to my-container-registry.azurecr.io/my-repository/my-artifact:latest alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] target parameters: image-spec= image-name=image.name, image-tag=image.tag application=my-artifact-prod image=my-container-registry.azurecr.io/my-repository/my-artifact
INFO[0001] Successfully updated image 'my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634' to 'my-container-registry.azurecr.io/my-repository/my-artifact:latest', but pending spec update (dry run=true) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] Using commit message: build: automatic update of my-artifact-prod
updates image my-repository/my-artifact tag '2022110713512263c634' to 'latest'
INFO[0001] Dry run - not commiting 1 changes to application application=my-artifact-prod
INFO[0001] Finished cache warm-up, pre-loaded 4 meta data entries from 2 registries
TRAC[0001] processing app 'my-artifact-prod' of type 'Helm' application=my-artifact-prod
INFO[0001] Starting image update cycle, considering 1 annotated application(s) for update
DEBU[0001] Processing application my-artifact-prod
DEBU[0001] Considering this image for update alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0001] Using no version constraint when looking for a new tag alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0001] Found update strategy latest image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] No match annotation found image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] No ignore-tags annotation found image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Using runtime platform constraint linux/amd64 image_alias=img-alias image_name=my-container-registry.azurecr.io/my-repository/my-artifact registry_url=my-container-registry.azurecr.io
TRAC[0001] Fetching credentials for registry https://my-container-registry.azurecr.io
TRAC[0001] Performing HTTP GET https://my-container-registry.azurecr.io/v2/my-repository/my-artifact/tags/list
DEBU[0002] Cache hit for my-repository/my-artifact:20221104105424fb448a alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:20221107114352a4594b alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:2022110713512263c634 alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Cache hit for my-repository/my-artifact:latest alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0002] List of available tags found: [20221104105424fb448a 20221107114352a4594b 2022110713512263c634 latest] alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
TRAC[0002] Finding out whether to consider 20221104105424fb448a for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider 20221107114352a4594b for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider 2022110713512263c634 for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
TRAC[0002] Finding out whether to consider latest for being updateable image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
DEBU[0002] found 4 from 4 tags eligible for consideration image="my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634"
INFO[0002] Setting new image to my-container-registry.azurecr.io/my-repository/my-artifact:latest alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] target parameters: image-spec= image-name=image.name, image-tag=image.tag application=my-artifact-prod image=my-container-registry.azurecr.io/my-repository/my-artifact
INFO[0002] Successfully updated image 'my-container-registry.azurecr.io/my-repository/my-artifact:2022110713512263c634' to 'my-container-registry.azurecr.io/my-repository/my-artifact:latest', but pending spec update (dry run=false) alias=img-alias application=my-artifact-prod image_name=my-repository/my-artifact image_tag=2022110713512263c634 registry=my-container-registry.azurecr.io
DEBU[0002] Using commit message: build: automatic update of my-artifact-prod
updates image my-repository/my-artifact tag '2022110713512263c634' to 'latest'
INFO[0002] Committing 1 parameter update(s) for application my-artifact-prod application=my-artifact-prod
INFO[0002] Starting configmap/secret informers
INFO[0003] Configmap/secret informer synced
INFO[0003] configmap informer cancelled
INFO[0003] Initializing git@github.com:my-repository/my-artifact-gitops.git to /tmp/git-my-artifact-prod1210370583
INFO[0003] rm -rf /tmp/git-my-artifact-prod1210370583 dir= execID=0fb01
INFO[0003] Trace args="[rm -rf /tmp/git-my-artifact-prod1210370583]" dir= operation_name="exec rm" time_ms=4.302099999999999
INFO[0003] git fetch origin --tags --force dir=/tmp/git-my-artifact-prod1210370583 execID=226b8
ERRO[0003] `git fetch origin --tags --force` failed exit status 128: No ED25519 host key is known for github.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists. execID=226b8
INFO[0003] Trace args="[git fetch origin --tags --force]" dir=/tmp/git-my-artifact-prod1210370583 operation_name="exec git" time_ms=394.93489999999997
ERRO[0003] Could not update application spec: `git fetch origin --tags --force` failed exit status 128: No ED25519 host key is known for github.com and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists. application=my-artifact-prod
INFO[0003] Processing
Image Updater fails with No ED25519 host key is known for github.com and you have requested strict checking. Host key verification failed.
.
Configuring SSH access
This problem is very specific to Git repositories, when Image Updater accesses the repository over SSH. Git repositories provided over HTTPS are not affected.
Due to Image Updater's current implementation, the SSH option StrictHostKeyChecking=yes
is always provided as command line parameter. There is no option to easily override that behaviour as the provided command line parameter has the highest precedence:
$ sudo strace -f -s 2048 ./argocd-image-updater --once --loglevel trace --argocd-namespace argocd --kubeconfig /home/ckl/.kube/config
# ...
2195 execve("/bin/sh", ["/bin/sh", "-c", "ssh -i /dev/shm/4050667513 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts \"$@\"", "ssh -i /dev/shm/4050667513 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts", "-o", "SendEnv=GIT_PROTOCOL", "git@github.com", "git-upload-pack 'my-repository/my-artifact-gitops.git'"], 0x7fffee30dd90 /* 18 vars */ <unfinished ...>
# ...
To fix this issue, add github.com
's ssh-ed25519 as known host key to your environment. First, grap github.com's ED25519 host key:
ssh-keyscan -t ssh-ed25519 github.com
# output = ${KNWON_HOST_KEY}
Add the SSH host key in local environment
In a local environment, the key must be added to /app/config/ssh/ssh_known_hosts
:
mkdir -p /app/config/ssh
echo ${KNWON_HOST_KEY} > /app/config/ssh/ssh_known_hosts
Adding the SSH host key in Kubernetes
When deployed to Kubernetes, you have to configure the argocd-ssh-known-hosts-cm
ConfigMap:
$ kubectl edit configmaps argocd-ssh-known-hosts-cm -n argocd
# add ${KNWON_HOST_KEY}
Configuring the correct Git branch
After configuring the SSH host key, the next Image Updater run might fail with the error error: pathspec 'master' did not match any file(s) known to git
:
INFO[0004] git checkout --force master dir=/tmp/git-my-artifact-prod4089267324 execID=e1fd2
ERRO[0004] `git checkout --force master` failed exit status 1: error: pathspec 'master' did not match any file(s) known to git execID=e1fd2
INFO[0004] Trace args="[git checkout --force master]" dir=/tmp/git-my-artifact-prod4089267324 operation_name="exec git" time_ms=6.1113
ERRO[0004] Could not update application spec: `git checkout --force master` failed exit status 1: error: pathspec 'master' did not match any file(s) known to git application=my-artifact-prod
INFO[0004] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1
INFO[0004] Finished.
By default, Image Updater checks out the master
branch. Add the argocd-image-updater.argoproj.io/git-branch
annotation with your repositories primary branch to your Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
# /image-list ist the image's repository
# `img-alias` is just a name which can be referenced later
argocd-image-updater.argoproj.io/image-list: img-alias=my-container-registry.azurecr.io/my-repository/my-artifact
# /${img-alias}.pull-secret
argocd-image-updater.argoproj.io/img-alias.pull-secret: pullsecret:my-artifact-prod/acr-dreitier-my-artifact
# Used branch
argocd-image-updater.argoproj.io/git-branch: main
After having set everything up, Argo CD Image Updater does now successfully run:
INFO[0010] Configmap/secret informer synced
INFO[0010] Initializing git@github.com:my-repository/my-artifact-gitops.git to /tmp/git-my-artifact-prod2883806823
INFO[0010] rm -rf /tmp/git-my-artifact-prod2883806823 dir= execID=5a0ef
INFO[0010] configmap informer cancelled
INFO[0010] Trace args="[rm -rf /tmp/git-my-artifact-prod2883806823]" dir= operation_name="exec rm" time_ms=4.048
INFO[0010] git fetch origin --tags --force dir=/tmp/git-my-artifact-prod2883806823 execID=52a5a
INFO[0011] secrets informer cancelled
INFO[0013] Trace args="[git fetch origin --tags --force]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=3146.2313000000004
INFO[0013] git config user.name argocd-image-updater dir=/tmp/git-my-artifact-prod2883806823 execID=885fa
INFO[0013] Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=4.8975
INFO[0013] git config user.email noreply@argoproj.io dir=/tmp/git-my-artifact-prod2883806823 execID=e9be4
INFO[0013] Trace args="[git config user.email noreply@argoproj.io]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=5.0155
TRAC[0013] targetRevision for update is 'main' application=my-artifact-prod
INFO[0013] git checkout --force main dir=/tmp/git-my-artifact-prod2883806823 execID=bcffb
INFO[0013] Trace args="[git checkout --force main]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=67.1502
INFO[0013] git clean -fdx dir=/tmp/git-my-artifact-prod2883806823 execID=4a84b
INFO[0013] Trace args="[git clean -fdx]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=11.4397
DEBU[0013] Writing commit message to /tmp/image-updater-commit-msg1539850089 application=my-artifact-prod
INFO[0013] git commit -a -F /tmp/image-updater-commit-msg1539850089 dir=/tmp/git-my-artifact-prod2883806823 execID=f1eca
INFO[0013] Trace args="[git commit -a -F /tmp/image-updater-commit-msg1539850089]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=80.8113
INFO[0013] git push origin main dir=/tmp/git-my-artifact-prod2883806823 execID=57f92
INFO[0016] Trace args="[git push origin main]" dir=/tmp/git-my-artifact-prod2883806823 operation_name="exec git" time_ms=2071.5395
INFO[0016] Successfully updated the live application spec application=my-artifact-prod
INFO[0016] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0
INFO[0016] Finished.