Version 1 was a push system. A state machine in a service, one CI job per step, SSH to the machines. The engine decided, the runner acted, and the cluster found out afterwards.
Then one of our clusters moved to Flux, and the obvious question was whether the pipeline survives GitOps at all. It does. What changed is smaller and stranger than “we replaced it”: exactly one hop inverted, and almost everything else stayed where it was.
The hop that inverted
In version 1 the last step was ssh and docker run. The runner reached into the machine.
Under GitOps nothing reaches in. The pipeline renders the cluster objects, opens a pull request against its own repo, merges it, and stops. A workflow publishes the merged tree as an OCI artifact. Inside the cluster, Flux follows that artifact and applies it.
flowchart LR
E["Rollout engine
(state machine)"] --> S["release step
on a CI runner"]
S --> R["render the objects"]
R --> PR["open a PR, wait for checks, merge"]
PR --> P["publish: OCI artifact"]
P -. "Flux pulls, on its own clock" .-> C["cluster applies"]
The dotted arrow is the whole point. The cluster never talks to the repo host, and nothing outside holds a write credential to the cluster. In the push model, CI needed a token that could change the cluster. In the pull model there is nothing to steal, because there is nothing to hold.
The honest version of “trigger Flux” is that you can’t. You merge; Flux notices on its next reconcile. So the pipeline’s step doesn’t finish at the merge — it waits for a successful publish whose commit contains that merge, then waits until the namespace reports the render’s own hash, then waits for the workload to be ready. Merging is not landing.
What the pipeline kept
It would be easy to read “GitOps” as “the pipeline is now a git push”. It isn’t, and the step list is the argument. One of our releases runs fifteen steps. The GitOps part — render, PR, merge, wait — is a fraction of one of them.
The other fourteen have no Kubernetes equivalent at all:
- the backend row in Postgres, which is what actually admits traffic
- internal keys distributed to the new stack
- the public hostname attached to the cluster’s tunnel
- a regression suite that lives in a third repository
- a one-hour soak on real traffic, then promote, then watch
Flux converges a directory. It cannot admit a request, because admission lives in a database and a CDN, not in the cluster. It has no notion of blue and green. And “revert the commit” is not a rollback once traffic has already moved somewhere that git has never heard of.
So the division came out like this, and I think it’s the durable part:
| owns | |
|---|---|
| Git | the desired shape of the workload |
| The cluster | how many of them there are |
| The registry | the bytes |
| The pipeline | the decision to change, and everything outside the cluster |
The seam nobody warns you about
The third row in that table is where we lost time.
If you put spec.replicas in git and also run an autoscaler, they fight. Flux’s controller applies with server-side apply and its own field manager, so it puts git’s number back on every reconcile. We watched an autoscaler’s 2 become git’s 1 again, on a schedule, forever.
The fix is not a smarter controller. It’s that an autoscaled workload must leave the count out of git entirely — the manifest declares no replicas, and the autoscaler is the only writer. Git holds the shape, the cluster holds the scale.
That sounds obvious written down. It is not obvious while you are staring at a workload that keeps resizing itself back.
Why the renderer replaces the whole directory
A detail that looks like style and isn’t: the release step deletes the backend’s directory and re-renders it from scratch, every time.
Four things depend on that.
- Deletion has to be expressible. If a render stops emitting an object, writing into the existing directory leaves the old file behind — and Flux, which prunes what it created, keeps applying something nobody wants any more.
- The empty diff is the idempotence check. Stage everything; if the diff is empty, the render already equals what’s live, and no pull request is opened at all.
- It makes the render hash honest. The commit title carries a hash of the render, and a later step waits until the cluster reports that same hash. Replacing the directory wholesale makes “this directory is this render” true by construction rather than by hope.
- Deploy and teardown become one function. Teardown is the same code with a different callback: remove the directory, let Flux prune what it held.
There’s a fifth reason hiding in the retry loop. On a conflict the step re-fetches the branch and re-renders from scratch. That is only safe because nothing in the directory is state worth preserving. The directory is derived, not authored — which is also why hand edits don’t survive, and why the tooling says so out loud when someone makes one.
One writer per file
Letting a bot merge its own pull requests sounds alarming until you see what fences it.
The bot may touch exactly one path prefix, enforced by a required check. People may not touch those files without an explicit override label. And the check runs the base branch’s copy of its own rule, so a pull request cannot edit the rule that judges it. The bot has no permission to change workflows either, so it can’t reach the check from the other side.
That’s the part I’d copy into any system where automation writes to a shared repo. Not “trust the bot” — remove the paths where trust would be required.
What this didn’t fix
Three honest gaps, a month in.
Not every cluster is pulling. We run three Kubernetes flavors. One is GitOps. One is still reached by SSH and kubectl. One is a managed control plane that CI pushes to through a bastion with a long-lived token. The pull model is clearly the better shape; it is not yet the only shape, and a system with two actuation models has two sets of failure modes.
Some workloads are still hand-typed. The LLM releases are rendered. The video lanes, which run the same engines for a different workload, are still hand-written YAML that a human edits. A renderer derives a file from the source of truth; a linter over hand-written files only finds yesterday’s typo. We started by writing the linter, then deleted it and wrote the renderer. That was the right call and it took a round trip to see.
Some config can’t travel with the thing it configures. A lane’s floor and burst live in a Prometheus recording rule, because they are operands inside a PromQL expression and PromQL has no named constants. That file sits outside the paths the release bot may write — so adding a lane is one pull request and setting its floor is another. Not fatal. Just a seam that no diagram shows you.
The takeaway
Flux didn’t replace the hand-written pipeline. It took one hop — the one where CI reached into a cluster — and turned it around, and in exchange it took over convergence, drift reversion and pruning, which the pipeline was never good at.
Everything the pipeline does outside the cluster is still the pipeline’s. If you’re weighing GitOps against an existing rollout system, that’s the question worth asking first: how much of your pipeline is actually “apply to the cluster”? For us it was one step out of fifteen.
The rest was always the hard part.
