openova/docs/CHART-AUTHORING.md

Chart Authoring Notes

Status: Authoritative. Audience: Anyone editing a products/<name>/chart/templates/*.yaml or platform/<name>/chart/templates/*.yaml resource that ships to a Flux- reconciled cluster.

This document captures sharp edges in the chart-authoring workflow that have already cost the project a real outage. Each section names a specific failure mode, a specific reproducer, and the canonical fix — in the same shape as docs/INVIOLABLE-PRINCIPLES.md. Read it before declaring "done" on any chart that mutates a long-lived resource.

---

Strategy flips on existing Deployments

What goes wrong

A chart manifest declares Deployment.spec.strategy.type: Recreate. The cluster already runs a Deployment of the same name that was created earlier with the default RollingUpdate strategy (so spec.strategy.rollingUpdate.maxSurge=25% and maxUnavailable=25% exist on the live object). Flux's kustomize-controller submits the new manifest via Server-Side Apply with the kustomize-controller field manager. The API server merges, then validates. Validation rejects with:

Deployment.apps "<name>" is invalid:
  spec.strategy.rollingUpdate: Forbidden:
    may not be specified when strategy `type` is 'Recreate'

The Flux Kustomization parks at Ready=False on every reconcile until an operator intervenes.

Why Server-Side Apply does this

SSA's contract is "set the fields you declare." It does NOT remove fields owned by other field managers. The pre-existing Deployment was created via kubectl apply (CSA), so the kubectl-client-side-apply field manager owns .spec.strategy.rollingUpdate.maxSurge and .spec.strategy.rollingUpdate.maxUnavailable. When kustomize- controller flips .spec.strategy.type to Recreate, those rolling- update fields stay on the object. The post-merge state has both type: Recreate AND rollingUpdate.* keys. The API validator forbids that combination. SSA cannot fix this on its own.

Why $patch: replace is NOT the answer

$patch: replace is a Strategic Merge Patch runtime directive. It does NOT belong in a chart's base resource. Reasons:

1. API strict-decoding rejects it on CREATE. kubectl create, kubectl apply to an empty namespace, and kubectl apply --server-side all return:

   strict decoding error: unknown field "spec.strategy.$patch"
   

   This BREAKS fresh installs — including every new Sovereign bootstrap.
2. Flux SSA rejects it. The kustomize-controller SSA path returns field not declared in schema on .spec.strategy.$patch.
3. It is a runtime directive, not a chart field. $patch: replace is processed at SMP merge time by SMP-aware mergers. kustomize build does NOT consume the directive when it appears in a base resource — it passes it through as if it were a normal YAML key. The downstream API call then fails as above.

The correct place for $patch: replace is inside a Kustomize patches: entry, where the kustomize binary processes it at build time and emits a clean output that contains no $patch key. That is not what fixes the strategy-flip problem either, because the build- time output is identical to declaring strategy.type: Recreate directly — it produces the same SSA failure.

The canonical fix

Annotate the Deployment with the Flux force annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalyst-api
  annotations:
    kustomize.toolkit.fluxcd.io/force: enabled
spec:
  replicas: 1
  strategy:
    type: Recreate
  # ...

When kustomize-controller's SSA dry-run fails with an Invalid response on this resource, the controller falls back to delete-and-recreate the SINGLE annotated resource (not the whole Kustomization). The recreated Deployment has no residual rollingUpdate.* fields — the regression cannot recur on the rebuilt object. The annotation lives in Git, version-controlled, applies on every reconcile.

This is not a "kubectl delete bandaid." Per INVIOLABLE-PRINCIPLES.md #3 (Follow the documented architecture, exactly — Flux is the ONLY GitOps reconciler) and #4 (Never hardcode — runtime configuration in Git, not in shell history): the remediation is declarative, scoped to the resource, and removed only by editing the chart.

When you may use this annotation

The Flux force annotation triggers delete + recreate on apply failure. Use it only on resources that:

• Already declare strategy.type: Recreate (so delete-and-recreate is the steady-state update path anyway), OR
• Carry no client traffic (a brief unavailability is acceptable), OR
• Are explicitly designed to lose in-process state on every roll.

Do NOT add the annotation to a resource whose default update mode is RollingUpdate and whose pods serve live traffic — you would be trading off availability against an outcome that better resource authoring (selectors, immutable-field migrations) could deliver.

Required test coverage

Every chart that flips Deployment.spec.strategy.type MUST be covered by a test fixture in tests/integration/strategy-flip.yaml (or its equivalent next to a similar regression). The test must:

1. Stage a Deployment with the OLD strategy at the same name.
2. Apply the NEW chart manifest.
3. Assert the apply succeeds via the documented apply path.
4. Assert the chart manifest carries the Flux force annotation.
5. Assert the chart manifest is also valid for fresh install (no inline $patch: replace or other strict-decoding-violating directives).

The current implementation lives at tests/integration/strategy-flip.sh and the CI workflow at .github/workflows/test-strategy-flip.yaml. Wire any new strategy- flip into both.

Reference incident

• Date: 2026-04-29
• Cluster: contabo-mkt
• Resource: catalyst/catalyst-api
• Symptom: Kustomization stuck Ready=False for hours; user unblocked manually with kubectl delete deploy catalyst-api -n catalyst. Flux re-created the Deployment from scratch on the next reconcile; the rollingUpdate.* fields were no longer present and the Kustomization went Ready=True.
• Root cause: chart's api-deployment.yaml declared strategy.type: Recreate; the live object had been created with default RollingUpdate; SSA preserved the rollingUpdate fields under the prior field manager.
• Durable fix: add kustomize.toolkit.fluxcd.io/force: enabled annotation to the chart manifest at products/catalyst/chart/templates/api-deployment.yaml.

---

Generalizing the lesson

Other chart fields that can collide on apply

The strategy-flip is one instance of a broader class: fields whose old value and new value cannot legally coexist, where the old value is owned by a non-Flux field manager. The same fix applies to each of them — annotate the resource with kustomize.toolkit.fluxcd.io/force: enabled and let Flux recover via delete-and-recreate when SSA dry-run fails.

Resource kind	Field that triggers an Invalid merge	Notes
Deployment	spec.strategy.type Recreate ↔ RollingUpdate	This document.
Deployment	spec.selector.matchLabels change	Selector is immutable post-create. Must recreate.
Service	spec.clusterIP (None ↔ value)	Immutable. Must recreate.
Service	spec.type ClusterIP ↔ NodePort ↔ LoadBalancer	Some transitions invalid; recreate is safe path.
PersistentVolumeClaim	spec.accessModes change after binding	Immutable post-bind. Recreate would lose data — DO NOT add force annotation; instead provision a new PVC under a new name and migrate.
StatefulSet	spec.serviceName, spec.selector	Immutable. Must recreate (which loses pod identity). Plan migrations carefully.
Job	spec.template.* after create	Immutable. Recreation is the only path.

For PVCs and StatefulSets specifically: NEVER add the Flux force annotation as a default. Data loss is the failure mode. The right move is a paired migration: provision the new resource under a new name, copy data, swap references, retire the old.

Authoring discipline

Before declaring "done" on any chart that touches a long-lived resource:

1. Run the chart's manifest through kubectl apply --dry-run=server against an EMPTY namespace. Must succeed (no $patch: in the spec, no fields the strict decoder rejects).
2. If the resource type appears in the table above, ALSO run kubectl apply --dry-run=server against a namespace where a PRIOR shape of the resource already exists. Must succeed under the user's documented apply path; if it fails, add the Flux force annotation AND the integration test.
3. Verify the chart's kustomization.yaml references all template files (catches the "I added a template but forgot to wire it" regression).
4. If the resource carries client traffic, document the recreate blast radius in the chart's leading comment — operators reading the chart need to know an apply may interrupt service.

Cross-references

• docs/INVIOLABLE-PRINCIPLES.md #3 — Follow the documented architecture, exactly. Flux is the ONLY GitOps reconciler; remediations live in IaC, not in shell history.
• docs/INVIOLABLE-PRINCIPLES.md #4 — Never hardcode. Runtime knobs live in Git as declarative resources, not as operator runbook steps.
• Flux docs: https://fluxcd.io/flux/components/kustomize/kustomizations/#force — official documentation of the kustomize.toolkit.fluxcd.io/force: enabled annotation.
• tests/integration/strategy-flip.sh — the runner that defends the Catalyst chart against this regression.
• tests/integration/strategy-flip.yaml — the bad-state fixture and assertion contract.
• .github/workflows/test-strategy-flip.yaml — CI wiring.