Launch failed instantly with "create workdir: mkdir /var/lib/catalyst:
permission denied". The catalyst-api Pod runs as UID 65534 with emptyDir
mounts only at /tmp and /home/nonroot — /var/lib was never writable, so
the provisioner.New() default for CATALYST_TOFU_WORKDIR
(/var/lib/catalyst/tofu) lost on the very first MkdirAll call.
Three coupled fixes:
- Set CATALYST_TOFU_WORKDIR=/tmp/catalyst/tofu so the per-deployment
workdir tree lands in the existing /tmp emptyDir.
- Bump cpu limit 100m → 1000m, memory limit 64Mi → 1Gi. tofu init pulls
~80MB hcloud + ~30MB dynadot provider plugins; tofu plan/apply hold
the state file in memory; 64Mi was always going to OOM on first init.
- Grow /tmp emptyDir sizeLimit 256Mi → 2Gi to fit the per-Sovereign
subdirectory tree (provider binaries + state + plan output).
Manifest-only change — Flux reconciles, kubectl rollout swaps the Pod,
no image rebuild required.
27527e4ca5