Skip to content
Hogin Hogin
Go back

Flux 2.8: CEL-based health checks and what they change for release gating

9 мин чтения

Flux has long had a blind spot exactly where advanced ArgoCD setups with Argo Rollouts looked stronger: readiness checks after a rollout only understood “known” resource types. A custom CRD with its own readiness logic was effectively invisible to Flux. 2.8 closes that gap — healthCheckExprs lets you describe a health check as a CEL expression over an arbitrary status. Let’s look at what that changes in practice, and where its authority ends.

Table of contents

Open Table of contents

How Flux checked readiness before

kustomize-controller and helm-controller have always been able to wait for more than “manifest applied” — they wait for the resource to be actually ready, courtesy of kstatus, a library that knows the standard Kubernetes conventions: for a Deployment, ready means status.availableReplicas == spec.replicas; for a Job, status.succeeded > 0; for objects with status.conditions, the Ready: True condition.

The catch: this only works for types kstatus knows, or that follow the common conditions convention. As soon as a CRD shows up with its own, non-standard status schema — say, a database operator whose readiness is expressed as status.phase: Running, or a pair of fields like status.readyReplicas / status.lastJobStatus — Flux only sees that the resource applied, not whether it’s meaningfully ready. Reconciliation gets marked successful before the service can actually take traffic.

In practice this hurt rollout reliability. If a downstream Kustomization’s dependsOn pointed at such a CRD, the dependency was considered satisfied right after apply — Flux couldn’t tell “the operator is still scaling up” from “the operator can serve traffic.” The next step in the chain started too early, and the first failures showed up in front of users rather than in the reconcile loop. The only workaround used to be writing a wrapper operator that translates the custom status into standard conditions — an extra component for what’s essentially a one-liner check.

Health check without CEL vs. with CEL: kstatus only sees Ready on known types, a CEL expression reads arbitrary status fields

What healthCheckExprs adds

healthCheckExprs is a list of rules in a Kustomization or HelmRelease spec, each tied to an apiVersion/kind and describing three states as CEL expressions over that resource’s status:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps-db
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/db
  sourceRef:
    kind: GitRepository
    name: infra
  healthCheckExprs:
    - apiVersion: apps.acme.io/v1
      kind: AcmeOperator
      current: "status.readyReplicas >= 2 && status.lastJobStatus == 'Succeeded'"
      inProgress: "status.lastJobStatus == 'Running'"
      failed: "status.lastJobStatus == 'Failed'"

current — the resource is ready, reconciliation can be considered successful. inProgress — still rolling out, wait for the next poll. failed — an explicit error; Flux immediately marks the Kustomization unhealthy instead of waiting out the timeout. The CEL context is the resource object itself (status, and metadata if you need it), so the expression can key off anything in its status — not just whether conditions exists.

The CEL expression sees the whole object, not just status, so you can also key off metadata (a version annotation, say) — but in the overwhelming majority of cases the status fields your operator already writes are enough. The rule is simple: if a field shows up in kubectl get <crd> -o yaml, it’s available in the CEL expression under the same name.

The key practical difference from plain kstatus: before healthCheckExprs, a health check was binary based on whether standard fields existed at all — either a resource fit the convention, or Flux didn’t check it. Now you can encode arbitrary readiness logic: “at least 2 replicas available and the last migration Job succeeded” — an AND of two independent signals that kstatus could never combine on its own.

How this changes rollback and ordering

This is where the real effect kicks in. A Kustomization whose dependsOn points at another one won’t start applying until that dependency is Ready. Before, “Ready” for a CRD without a standard status effectively meant “applied” — so dependsOn didn’t actually guarantee much beyond known types. With healthCheckExprs, Ready starts meaning exactly what you described in current: the next Kustomization in the chain won’t move until the CEL check passes.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps-api
spec:
  dependsOn:
    - name: apps-db      # waits for current from apps-db's healthCheckExprs,
  # ...                  # not just the fact the manifest applied

If current doesn’t evaluate to true within timeout, the Kustomization is marked Ready: False, and anything depending on it via dependsOn stays blocked — the same “don’t roll forward until the previous step is confirmed” principle that Argo Rollouts implements with AnalysisTemplate on the ArgoCD side: the next rollout step doesn’t happen until the analysis metrics converge. The difference is what gets checked, and at what level.

A gate between Kustomizations: apps-api waits on current from apps-db's healthCheckExprs, not just the fact of apply

Where its authority ends: comparison with Flagger

It’s important not to confuse healthCheckExprs with progressive delivery. It’s a go/no-go check for one resource at one point in the reconcile loop: either current or not. It doesn’t gradually shift traffic, compare canary vs. stable metrics, or roll back based on SLOs — that’s still Flagger’s job in the Flux ecosystem, running a separate loop: 10% → 50% → 100% traffic with Prometheus metrics checked at each step, and an automatic rollback if the canary fails.

healthCheckExprs (Flux)Flagger canaryArgo Rollouts AnalysisTemplate
What it checksan arbitrary status field via CELmetrics (latency, error rate) over timemetrics / arbitrary checks over time
Granularityone resource, one snapshotgradual traffic shiftgradual traffic shift
ResultReady: True/Falsepromote / rollback the canarypromote / abort the rollout
Blocks whatthe next Kustomization via dependsOntraffic shift to the new versionthe next rollout step
Replaces canarynoyes, this is canaryyes, this is canary

In other words: healthCheckExprs closes the gap in gating — Flux can finally wait meaningfully on non-standard CRDs — but progressive delivery as a separate process with gradual traffic and metric-driven auto-rollback is still Flagger’s job, not Kustomization’s. The two aren’t competing, either — they sit at different layers: healthCheckExprs can be a precondition for Flagger even starting a canary rollout — Flux first confirms the database and migrations are ready, and only then does Flagger start shifting traffic to the new API version on top of it.

What you need to enable CEL health checks

You’ll need Flux 2.8+ in the cluster (flux check shows controller versions) and a CRD whose status you actually understand — check kubectl get <crd> <name> -o yaml and find the fields that genuinely reflect readiness.

Let’s take the HelmRelease from the one-cluster GitOps setup already covered on this blog and add a CEL check for a custom operator’s status:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: acme-operator
  namespace: flux-system
spec:
  interval: 10m
  chart:
    spec:
      chart: acme-operator
      sourceRef:
        kind: HelmRepository
        name: acme-charts
  healthCheckExprs:
    - apiVersion: apps.acme.io/v1
      kind: AcmeOperator
      current: "status.readyReplicas >= 2 && status.lastJobStatus == 'Succeeded'"
      failed: "status.lastJobStatus == 'Failed'"

Migrating an existing Kustomization to CEL

If you already have a working Kustomization without healthCheckExprs, adding it isn’t a breaking change: as long as the section is absent, Flux keeps using the standard kstatus, and behavior doesn’t change. Migration comes down to:

  1. Find CRDs whose readiness currently isn’t meaningfully checked — Flux only sees the apply. Usually these are operator resources with a non-standard status.
  2. Write current/failed/inProgress against the CRD’s real status fields, and check the CEL expression locally before committing — a typo in a field name won’t fail the apply, the health check will simply never become True.
  3. Add healthCheckExprs to the spec and commit — on the next reconcile Flux starts evaluating the new rule.
  4. If other Kustomizations depend on this one via dependsOn, verify the chain still reaches current in a reasonable time and doesn’t hit timeout.

How to verify it works

flux get kustomizations apps-db
# NAME      READY   MESSAGE
# apps-db   True    Health check passed

kubectl get kustomization apps-db -n flux-system -o jsonpath='{.status.conditions}'

If READY=False and the MESSAGE says Health check failed, either the failed expression fired or current didn’t evaluate to true within timeout. It’s worth briefly breaking the condition on purpose (drop readyReplicas, say) to confirm the Kustomization really does go Ready: False and dependents don’t apply — that proves the gate is actually working, not just sitting in the yaml.

Bottom line

healthCheckExprs fixes a specific, long-standing gap in Flux: a custom CRD’s readiness can now be described as an arbitrary CEL expression over its own status, instead of waiting for it to fit the kstatus convention. That makes dependsOn chains reliable for non-standard resources and brings parity with ArgoCD on smart gating. But it’s still a single-snapshot check, not progressive delivery — gradual traffic shifting with metrics and auto-rollback is still Flagger’s job, and if you need canary rather than “applied and working,” a CEL health check doesn’t replace a dedicated canary controller — it pairs well with one instead: the gate decides whether to move forward at all, and Flagger decides how to move forward safely.


Share this post:

Next Post
OBI: zero-code application tracing via eBPF, the Grafana Beyla successor