* fix(catalyst-api,bp-keycloak): handover 401 root-causes — Reloader annot + realm SA users array (#713)
Closes#713
Two distinct chart bugs surfaced live on otech62 (2026-05-03), both producing
401 on /auth/handover:
1. SOVEREIGN_FQDN race
api-deployment.yaml reads SOVEREIGN_FQDN from ConfigMap "sovereign-fqdn"
with optional:true. On Sovereigns, that ConfigMap is rendered by the
sovereign-tls Flux Kustomization concurrently with bp-catalyst-platform
HelmRelease. When the Pod starts first, valueFrom collapses to "" and
stays empty — audience check rejects every valid token as "invalid
audience". Fix: add Reloader annotations so the Pod rolls when the
ConfigMap (and the handover-jwt-public Secret) appears.
2. catalyst-api-server SA missing user-level realm-management role mappings
bp-keycloak realm import granted roles via clientScopeMappings — wrong
level. The actual service-account user had no clientRoles entry, so KC
rejected GET /users with 403 when catalyst-api tried to ensure the
operator user during handover. Fix: add explicit "users" array binding
service-account-catalyst-api-server to realm-management.{impersonation,
manage-users, view-users, query-users}.
* fix(catalyst-api,bp-reloader): tofu state on PVC + Reloader annotations strategy (#715)
Closes#715
Two architectural bugs surfaced live on otech64 (2026-05-03), both leading
to a healthy-looking Sovereign that the operator could not reach.
1. catalyst-api tofu workdir on emptyDir
CATALYST_TOFU_WORKDIR=/tmp/catalyst/tofu (emptyDir). When contabo's
catalyst-api Pod rolled mid-apply (the PR #714 deploy commit triggered
a rolling restart 3 minutes into otech64's tofu run), in-progress state
was lost. Tofu had created LB/network/server/services but not the
hcloud_load_balancer_target.control_plane resource yet — the cluster
came up at the k3s level but the public LB had no targets, returning
TLS handshake failure for every console.<sov> request.
Move CATALYST_TOFU_WORKDIR to /var/lib/catalyst/tofu (PVC-backed,
fsGroup=65534 already wires write access). tofu apply resumes from
where it left off after any Pod restart.
2. bp-reloader env-vars strategy
reloadStrategy=env-vars only injects checksum env vars for ConfigMaps
referenced via envFrom. Workloads using valueFrom: configMapKeyRef
(catalyst-api's SOVEREIGN_FQDN) are silently not reloaded — the
configmap.reloader.stakater.com/reload annotation added in PR #714
was a no-op under env-vars.
Switch to reloadStrategy=annotations. Reloader bumps a pod-template
annotation, triggering rollout regardless of how the CM/Secret is
referenced.
---------
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
9a58289786