I upgraded Flux to v2.3.0. The bootstrap completed successfully — all six controllers were healthy. Then every Kustomization in the cluster started failing with: failed to convert new object: .spec.validation: field not declared in schema.
The upgrade worked. The cluster didn’t.
The Symptom
Kustomization/flux-system/my-app dry-run failed:
failed to convert new object:
.spec.validation: field not declared in schema
Every Kustomization that referenced .spec.validation broke. This field existed in the old Flux CRDs but was removed in the new version. The new CRDs were already applied — the bootstrap installed them — so now the existing Kustomization resources in the cluster were invalid against the new schema.
Why Bootstrap Didn’t Catch It
flux bootstrap does three things:
- Installs/updates Flux controllers
- Installs/updates CRDs
- Creates the
flux-systemKustomization
Steps 1-3 succeeded. The controllers started, CRDs were applied, flux-system Kustomization was healthy. But the downstream Kustomizations — the ones managing actual workloads — were written with the old CRD schema. Bootstrap doesn’t validate those.
The result: a green bootstrap with a broken cluster.
The Other Surprise
flux check also warned:
Kubernetes version v1.27.6 does not match >=1.28.0-0
Flux v2.3.0 requires Kubernetes 1.28+. The bootstrap didn’t block on this — it’s a warning, not an error. But some features silently fail on older Kubernetes versions.
The Fix
Two changes needed:
1. Update existing Kustomization manifests — remove .spec.validation (deprecated) and replace with the new equivalent if applicable. In most cases, the field could simply be deleted with no replacement.
2. Upgrade Kubernetes — bumped the cluster from 1.27 to 1.28 to meet Flux’s requirements.
After both changes, Flux reconciled all Kustomizations successfully.
How to Avoid This
Before upgrading Flux, check the release notes for removed CRD fields. Then grep your manifests:
grep -r "spec.validation\|spec.decryption\|spec.suspend" flux-system/
Any field that was deprecated in a previous version might be removed in the next. Flux doesn’t provide a --dry-run for upgrades that checks existing resources against new CRDs.
Takeaway
Flux bootstrap success doesn’t mean your cluster is healthy. The bootstrap only validates itself — not the downstream Kustomizations that manage your workloads. After any Flux upgrade, watch the Kustomization statuses (kubectl get ks -A), not just the controller pod health. And always check CRD schema changes in the release notes before upgrading.