Flux
How Flux deploys the polinetwork-cd repository to the K3s cluster.
What is Flux?
Flux is a GitOps tool: it runs inside the cluster, watches a Git repository and applies whatever it finds there, over and over. If someone changes a resource by hand, Flux puts it back the way Git says. It replaced the ArgoCD instance we had on AKS.
Ours is installed by Ansible through the
Flux Operator, and it syncs the
main branch of
polinetwork-cd, starting
from the clusters/k3s folder.
Merging to main in polinetwork-cd is deploying to production.
Never kubectl apply or kubectl edit a resource that Flux manages: Flux reverts
it at the next reconciliation.
Repository layout
📁 polinetworkorg/polinetwork-cd/
├─ 📁 ansible/ # Host configuration (see K3s Node)
├─ 📁 clusters/k3s/ # Entry point: one Flux Kustomization per component
│ ├─ infrastructure-*.yaml # Platform components
│ └─ 📁 apps/ # One file per app + kustomization.yaml listing them
├─ 📁 infrastructure/ # Manifests of the platform components
├─ 📁 apps/ # Manifests of the apps, one folder per namespace
│ ├─ 📁 backend/
│ ├─ 📁 web/
│ └─ ...
└─ 📁 tests/ # Manifest checks run in CIFolders at the root of the repository (bot-prod/, bot-rooms/, mariadb/, ...)
are leftovers from the AKS era, all disabled. Flux ignores them.
clusters/k3s
Each file here is a Flux
Kustomization
that points to one folder of infrastructure/ or apps/. It also sets:
dependsOn: what must be ready first. Apps wait for the secret stores, storage and Traefik;backendalso waits forpostgresandredis.prune: true: a resource removed from Git is deleted from the cluster.wait: true: the Kustomization is ready only when all its resources are healthy.interval: 30m: a full reconciliation at least every 30 minutes, even if Git didn't change.
clusters/k3s/apps/kustomization.yaml lists the apps Flux runs. An app file
that is not listed there does nothing.
Platform components (infrastructure/)
| Component | What it does |
|---|---|
security | Admission policies (no host namespaces, no pods as keyvault-reader) |
storage | local-path provisioner and the fast / standard storage classes |
traefik | Makes the bundled Traefik a ClusterIP service |
external-secrets | External Secrets Operator, installed with Helm |
secret-stores | The kv-pn-apps and kv-pn-infra ClusterSecretStores |
cloudflared | The Cloudflare Tunnel connector (2 replicas) |
image-automation | Detects new latest images on GHCR (see below) |
node-exporter | Node metrics for Prometheus |
flux-web | The Flux Web UI at flux.polinetwork.org |
Automatic image updates
Most of our apps are deployed from the latest tag of an image on
GHCR. A tag can point to a
different image over time, so Flux resolves it to a digest and deploys that
digest. When the digest changes, the app rolls out the new image.
The pieces, all in polinetwork-cd:
infrastructure/image-automation/image-providers.yaml: oneResourceSetInputProviderper image, which reads thelatesttag and its digest.infrastructure/image-automation/receiver.yaml: a FluxReceivercalled by the PoliNetworkOrg organization webhook onpackageevents, atflux-webhook.polinetwork.org. The webhook's HMAC signature is its only authentication. If the webhook doesn't fire, the providers poll every 6 hours anyway.clusters/k3s/apps/<app>.yaml: for apps with automatic updates this is aResourceSetinstead of a plain Kustomization. It generates the app's Kustomization with animagesoverride that pins the digest.
Apps that don't need automatic updates (databases, monitoring, ...) pin their image in the manifest, and you bump them with a PR.
Day-to-day
Web UI
flux.polinetwork.org shows every Kustomization, HelmRelease and ResourceSet, with its status, last applied revision and errors. You log in through auth.polinetwork.org; only the Direttivo is admitted. From the UI you can reconcile (sync now), suspend and resume any resource.
From the node
The flux CLI isn't installed on the node, but the Flux resources are regular
Kubernetes objects:
# Status of all the Kustomizations (apps and infrastructure)
kubectl get kustomizations -n flux-system
# Latest image resolved for each app
kubectl get resourcesetinputproviders -n flux-system
# Why is an app not ready?
kubectl describe kustomization <app> -n flux-system
# Sync now instead of waiting: fetch the latest commit, then apply it
kubectl annotate --overwrite gitrepository flux-system -n flux-system \
reconcile.fluxcd.io/requestedAt="$(date +%s)"
kubectl annotate --overwrite kustomization <app> -n flux-system \
reconcile.fluxcd.io/requestedAt="$(date +%s)"Suspending an app
To stop Flux from touching an app while you debug it by hand, suspend its Kustomization from the Web UI. Remember to resume it afterwards: a suspended app also stops receiving image updates.
PoliNetwork Docs