Keycloak v26 dropped legacy 'requested_subject' token-exchange. The
auth_handover.go path still called kc.ImpersonateToken() which uses
that parameter, returning 400 'invalid_request'. PR #694 already
moved PIN-verify to local JWT minting via handoverSigner.SignCustomClaims;
apply the same pattern to /auth/handover.
Caught live on otech49 (2026-05-03):
ERROR auth_handover: ImpersonateToken failed
err=token endpoint 400: Parameter 'requested_subject' is not
supported for standard token exchange
Sovereign Keycloak still owns the canonical user record (created via
EnsureUser before token mint) — only the session-cookie minting
moves local. IdP brokering and federation paths are unaffected.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #692 moved the Sovereign-side JWK volume mount from
/var/lib/catalyst/handover-jwt-public.jwk (subPath, conflicted with
the catalyst-api PVC) to /etc/catalyst/handover-jwt-public/public.jwk
(directory mount). The chart sets CATALYST_HANDOVER_JWT_PUBLIC_KEY_PATH
to the new path, but the AuthHandover handler never read that env.
Result: auth_handover.go used the hardcoded default
/var/lib/catalyst/handover-jwt-public.jwk which no longer exists,
returning 401 'public key unavailable' on every handover.
Caught live on otech49 (2026-05-03):
ERROR auth_handover: load public key failed
err=read /var/lib/catalyst/handover-jwt-public.jwk: no such file
path=/var/lib/catalyst/handover-jwt-public.jwk
Fix:
- Resolution order: handler field -> env var -> default const
- Default const updated to the new path so cold-starts work without
the env var (defence in depth)
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Sovereign-side /auth/handover handler is the ENTRY POINT that
establishes the session. The operator's browser arrives with the
handover JWT in the URL query and zero cookies. Putting the route
inside the RequireSession middleware group rejects every handover
with 401 {error:unauthenticated} before AuthHandover ever runs.
Caught live on otech49 (2026-05-03):
GET /auth/handover?token=<valid-jwt> -> 401 in 43us (middleware
rejection, no body log line emitted).
This was working on otech48 only because catalyst-api there had no
Keycloak credentials wired (kc-sa-credentials Secret was missing) so
GetAuthConfig() returned nil and RequireSession became a passthrough.
Once PR #691 wired the credentials cleanly on otech49, the gate
activated and broke the handover.
Fix: register the route at the top-level mux outside the auth group,
mirroring the same pattern as /api/v1/deployments/{id}/kubeconfig
(cloud-init postback that also has no cookies). The handler's own
JWT validation IS the authentication.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Helm parses the entire file (including YAML comments) for template
directives BEFORE YAML parsing strips comments. Literal '{{ ... }}'
inside a # comment was treated as a template directive and failed
with 'unexpected <.> in operand' at line 419.
PR #698 introduced this in the explanatory comment for the
SOVEREIGN_FQDN ConfigMap workaround. Reword to avoid the literal
double-curlies — the comment still describes the constraint without
tripping the Helm parser.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #692 added an inline Helm-template `value:` for SOVEREIGN_FQDN in
api-deployment.yaml. That broke contabo-mkt's catalyst-platform Flux
Kustomization (path: ./products/catalyst/chart/templates) because Kustomize
parses raw YAML and Helm `{{ ... }}` is not valid YAML syntax. Live error
on contabo at adf8dc7d:
kustomize build failed: yaml: invalid map key:
map[string]interface {}{".Values.global.sovereignFQDN | default \"\" | quote":""}
Replace the Helm-template form with `valueFrom.configMapKeyRef.optional:
true` so the same template renders cleanly under both consumers:
- contabo-mkt (Kustomize): ConfigMap `sovereign-fqdn` doesn't exist →
optional ref → env stays empty → catalyst-api on contabo never validates
handover JWTs anyway (it's the SIGNER, not the validator). Correct.
- Sovereigns (Helm via bp-catalyst-platform OCI chart): on apply, the
sovereign-tls Kustomization renders `sovereign-fqdn-configmap.yaml` with
envsubst on ${SOVEREIGN_FQDN}, creating the ConfigMap with the per-
Sovereign FQDN. catalyst-api Pod resolves the ref → env populated →
audience check works.
This restores the bridge between the two consumers without forking the
template. The bp-catalyst-platform 1.2.5 → 1.2.7 bump publishes the new
chart; bootstrap-kit overlay pin updated.
Will be verified on otech49 (next provision after this lands).
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow-canvas): variable-width depth columns + ResizeObserver debounce (#669 round 3)
Round-2 UAT showed:
1. Dense bucket of 30+ siblings piled at the right edge while 60% of
canvas (left side) sat empty with one bubble per depth.
2. Sim "trying never stabilizing" during pane-transition animations.
Root cause #1: round-2 used a constant `perDepthX` for every depth.
With one-bubble depths next to a 30+ sibling depth, the dense bucket
got 80% × perDepthX (~128 px) of horizontal room and had to pile into
8+ sub-columns; sparse depths each got the same perDepthX (~160 px)
for a single bubble. Net: 60% canvas unused on the left, dense
cluster jammed at right.
Round-3 fix#1: variable-width depth columns. Each depth gets a slot
whose width tracks its bucket's natural extent at radius R:
sparse buckets need 2R + small gap; dense buckets need
(totalCols - 1) * (2R + COLLIDE_PADDING) to fit sub-columns
side-by-side. depthToX returns the centerline of slot[depth];
adjacent slots are separated by `gap = clamp(r*4, MIN, MAX)`. Total
layout width = sum(slots) + gaps.
Root cause #2: ResizeObserver fired on every animation frame during
the 220ms padding-right transition (pane open/close). Every fire
called setHostSize, which retriggered layoutMetrics → R changed by
1-2 px → all node targets shifted → sim re-seeded → never settled.
Round-3 fix#2: 180ms debounce on the observer + 8 px epsilon gate
(sub-pixel changes ignored entirely). Combined with snap-to-4 on R
and snap-to-8 on slot widths in layoutMetrics, the metrics now hold
constant during pane-transition animations and the sim converges
once.
Tests: bounded layout (17) + JobDetail (5) all green; tsc -b clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow-canvas): sqrt-aspect dense buckets + tight grid clamps (#669 round 4)
Round-3 still piled the dense bucket at the right edge. Distribution
test on the founder's exact screenshot shape (1+1+30) showed the dense
slot occupied only 28% of total X-extent — better than round-2 (~13%)
but not enough.
Round-4 fix:
1. layoutMetrics targets a sqrt-aspect-ratio for dense buckets:
targetRows = round(sqrt(count / 1.6))
30 leaves → 4 rows × 8 cols → ~700 px slot at R=40, occupying
>50% of total X-extent. The densest bucket's targetRows now sets
R via vertical-fit, so wide buckets actually claim X-room rather
than collapsing into thin tall columns.
2. gridTargets reads cols/rows from layoutMetrics.slotInfo instead
of recomputing — guarantees the per-tick clamp uses the same
sub-grid dimensions as the slot-width math.
3. Per-cell clamp window narrowed to ±(pitch/2 - R) so the bubble
edge can never reach a neighbour's centre. Old clamp used the
full pitch which let forceCollide push bubbles into a neighbour's
territory and then ratcheted them in — centres could collapse to
<2R apart.
Adds FlowCanvasOrganic.distribution.test.tsx replicating the founder's
UAT screenshot (depth 0: 1, depth 1: 1, depth 2: 30). Asserts:
- depth-0 X < depth-1 X < depth-2 X (left-to-right)
- dense leafSpan ≥ 30% of total layout extent
- no centre-to-centre distance < 2R
All tests green: distribution (2/2), bounded (17/17), JobDetail (5/5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The wizard surface is now anonymous-first. A visitor lands on
console.openova.io and runs the entire 7-step provisioning flow
without a session; auth fires only when they click Launch.
Frontend (catalyst-ui):
- Drop the wizardAuthGuard so the wizard route renders for anonymous
visitors. The existing zustand+persist store already keeps every
form field in localStorage with credential-hygiene partitioning
(Hetzner token, SSH private key, registrar token NEVER persisted),
so the guest-mode hydration on refresh works for free.
- New shared/lib/useSession hook polls /api/v1/whoami via React
Query; exposes signedIn / email / refetch / signOut.
- New widgets/auth/ProfileMenu in the wizard header — Sign in button
for anonymous, email-initial avatar with sign-out dropdown for
signed-in.
- New widgets/auth/PinSignInModal — two-stage email → 6-digit PIN
modal that POSTs /auth/pin/issue + /auth/pin/verify (issue #688).
Falls back to /auth/magic-link when the PIN endpoint is not
available, so this PR is shippable independent of #688's merge
order.
- StepReview Launch handler routes anonymous through the PIN modal;
on verify it stamps the verified email into orgEmail and POSTs
the deployment immediately.
- New /provision/* beforeLoad guard: anonymous → redirect to wizard
with a sessionStorage flash banner; signed-in cross-tenant gets
the canonical 404 from the API (no UI-side branch).
- New shared/lib/flashBanner — sessionStorage seam for the guard →
wizard banner hand-off.
Backend (catalyst-api):
- Add OwnerEmail to store.Record and handler.Deployment, stamped
from X-User-Email at CreateDeployment.
- New checkOwnership helper enforces 404 (NEVER 403) on cross-tenant
access — never leak existence of someone else's deployment via
the response code. Legacy records (OwnerEmail == "") pass through
with a warning so in-place upgrade does not lock operators out.
- Wired into GetDeployment, StreamLogs, GetDeploymentEvents,
WipeDeployment, GetKubeconfig, MintHandoverToken, ListJobs, and
GetJob. PutKubeconfig keeps its bearer-token auth (cloud-init
postback path).
Tests:
- Backend: deployments_owner_test.go covers legacy passthrough,
no-session passthrough, owner match (case-insensitive), the
load-bearing 404-not-403 cross-tenant assertion, and end-to-end
proof through GetDeployment + GetDeploymentEvents.
- Frontend: flashBanner round-trip + clear-on-read; useSession
signed-in / 401 / signOut paths; WizardLayout guest-mode
[Sign in] button + flash banner rendering.
Closes#689.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
otech48 incident (2026-05-03): all 37 bp-* HelmReleases on the Sovereign
cluster reached Ready=True, but the catalyst-api deployment record stayed
status=phase1-watching. Wizard's POST /mint-handover-token returned 409
not-handover-ready, blocking the auto-redirect to console.<sov>/auth/handover.
Root cause: helmwatch's terminate-on-all-done gate required len(observed) >=
MinBootstrapKitHRs. Chart shipped CATALYST_PHASE1_MIN_BOOTSTRAP_KIT_HRS=38,
but the actual bootstrap-kit cardinality had drifted to 37 — making the
gate permanently unsatisfiable. Watch ran until 60-minute WatchTimeout fired.
Fix: gate terminate-on-all-done on the informer's HasSynced signal instead
of the brittle count. After WaitForCacheSync returns the full bp-* set is
in the cache regardless of cardinality. MinBootstrapKitHRs stays as a
defence-in-depth floor (default lowered 11 → 1) for the empty-cache
footgun. Chart env CATALYST_PHASE1_MIN_BOOTSTRAP_KIT_HRS dropped to 1.
Implementation:
- helmwatch.Watcher: new informerSynced bool gate, set after
WaitForCacheSync. processEvent refuses to consider terminate-on-all-done
while informerSynced=false. After WaitForCacheSync, re-evaluate the
all-terminal check once on the synced cache (handles the rehydrate-
after-restart path where every HR is already Ready=True at attach).
- helmwatch.maybeEmitReadyTransition: emits the operator-visible
"All N blueprints reconciled. Sovereign ready for handover." SSE event
exactly once when the gate fires (idempotency guard against flicker
re-triggering the gate).
- handler.markPhase1Done: persistDeployment after status flip so the
on-disk JSON reflects status=ready before any wizard poll. Also
refuses to downgrade an already-adopted deployment if a late watcher
event tries to flap it.
- Tests: new transition_test.go with happy-path, idempotency, partial-
ready, realistic 37-HR convergence, and empty-cache scenarios. New
TestMarkPhase1Done_RefusesToDowngradeAdopted in phase1_watch_test.go.
Will be verified live on otech49 (next provision after this lands):
- Wizard auto-shows "Open your Sovereign Console" button within 30s of
all HRs reaching Ready
- No manual API calls or kubectl exec needed to flip status
- catalyst-api logs show "All 37 blueprints reconciled" event in SSE buffer
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the magic-link login flow on console.openova.io with a paste-friendly
6-digit numeric PIN, modelled on bank/Google verification screens. Founder
rejected magic links because they look like phishing (2026-05-03).
## Backend (products/catalyst/bootstrap/api)
- New handler/pinstore.go — sync.Mutex-guarded in-memory map keyed by email
with 10-minute TTL, 60-second per-email rate limit, 3-attempt lockout, and
a background goroutine that sweeps expired entries every minute.
PINs are NEVER persisted to disk per credential-hygiene rules.
- handler/auth.go rewritten:
* POST /api/v1/auth/pin/issue — body {email}. EnsureUser in openova realm,
generate 6-digit PIN with crypto/rand (NEVER math/rand), store, send
plaintext email with prominent "3 7 2 4 5 8" code and NO clickable URL,
return {ok, requestId, expiresInSec}. Rate-limit 60s.
* POST /api/v1/auth/pin/verify — body {email, pin, requestId}. Atomic
verify+decrement, on match mint self-signed session JWT (same handover
signer; KC 24.7 removed legacy token-exchange) and set HttpOnly Secure
SameSite=Lax cookie. Wrong: 401 with attemptsRemaining. Locked/expired:
410. Stable error codes: pin-invalid / pin-expired / attempts-exceeded /
email-required / pin-rate-limited.
- Routes wired in cmd/api/main.go. Legacy /auth/magic and /auth/callback
redirect to /login?error=flow_changed for stale bookmarks.
- Handler struct gets a pinStore field; openovaKC keycloakClient kept for
the EnsureUser call.
- Tests: auth_pin_test.go (14 tests covering happy path, all error codes,
SMTP rollback, rate limit, request-mismatch) + pinstore_test.go (12 tests
on the store invariants).
## Frontend (products/catalyst/bootstrap/ui)
- New PinInput6.tsx component — 6 inputs, inputmode=numeric, maxlength=1,
auto-advance focus, Backspace steps back, paste-anywhere splits clipboard
digits across boxes (extracts /\d/g), auto-submits on the 6th digit or
Enter. one-time-code autocomplete on box 0 for SMS prefill.
- LoginPage rewritten — single email field, "Send code" button, on success
navigates to /login/verify with email + requestId in the URL. PIN never
enters the URL.
- New VerifyPinPage — renders PinInput6, calls /pin/verify, on 401 shows
"Code incorrect, X attempts remaining", on 410 routes back to /login
with the error code, on 200 navigates to /wizard (or ?next=...).
- AuthCallbackPage stripped of magic-link code path; Catalyst-Zero branch
is now a 302 safety net for stale Keycloak redirect URIs.
- Router gets /login/verify route.
- 17 vitest cases on PinInput6 covering paste, typing, backspace, Enter,
pasting alphanumerics/long strings, controlled value, disabled state.
## DoD verification
- go test ./internal/handler/... -run "Pin|Handover|Auth" → PASS
(12 pinstore_test + 14 auth_pin_test + handover/auth tests)
- npm test src/components/PinInput6.test.tsx → 17 passed
- helm template products/catalyst/chart → renders without error
- Email body contains zero clickable URLs: TestSendPinEmail_NoMagicLinkURL
asserts ?token=, &token=, magic-link substrings absent
Closes#688
Co-authored-by: hatiyildiz <hatice@openova.io>
* fix(handover): provision Keycloak service-account credentials zero-touch (Phase-8b followup)
Sovereign-side catalyst-api needs Keycloak service-account credentials
to provision the operator's user during /auth/handover. Today the chart
references K8s Secret `catalyst-kc-sa-credentials` with keys addr/realm/
client-id/client-secret in the catalyst-system namespace — but no
zero-touch path materialised it. The dead SealedSecret template at
09a-keycloak-catalyst-api-secret.yaml had a different name AND different
keys (CATALYST_KC_*), used PLACEHOLDER_SEALED_VALUE markers no
provisioner replaced, and wasn't even listed in the bootstrap-kit
kustomization.
Symptom on otech48: GET /auth/handover?token=<valid-jwt> returns
"server misconfiguration: keycloak not configured"
(auth_handover.go:169).
Fix: bp-keycloak chart's configmap-sovereign-realm.yaml template now
emits the realm-import ConfigMap AND the catalyst-kc-sa-credentials
Secret in a single template scope so they share the same generated
client secret. Pattern mirrors platform/powerdns/chart/templates/
api-credentials-secret.yaml (canonical seam, ADR-0001 §11.3
anti-duplication).
Secret-value resolution order (first match wins):
1. operator-supplied .Values.catalystApiServerClientSecret
2. helm `lookup` of existing Secret in keycloak ns (idempotent)
3. fresh randAlphaNum 32 (zero-touch on first install)
The Secret carries the four keys exactly as the catalyst-api Pod's
secretKeyRef expects — addr / realm / client-id / client-secret —
with addr derived from gateway.host (https://auth.<sovereignFQDN>).
Reflector annotations auto-mirror the Secret to catalyst-system as
soon as that namespace materialises (bootstrap-kit slot 13).
The realm import already creates the catalyst-api-server client with
serviceAccountsEnabled + impersonation/manage-users/view-users/
query-users role mappings — so once Keycloak is Ready and the realm
imports, the SA is fully provisioned and the K8s Secret carries a
matching client secret. No post-install Job, no Admin-API script,
no out-of-band SealedSecret ceremony.
Cleanup: removes the dead 09a SealedSecret template (not in
kustomization, never produced a working Secret).
Bumps:
- bp-keycloak chart 1.3.0 -> 1.3.1
- clusters/_template/bootstrap-kit/09-keycloak.yaml HelmRelease
pin 1.3.0 -> 1.3.1
Existing per-Sovereign overlays (clusters/otech.omani.works/,
clusters/omantel.omani.works/) intentionally remain on 1.3.0 — fresh
otechN provisioning consumes _template at provision time.
Will be verified live on otech49 — handover end-to-end without ANY
manual Secret creation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(keycloak): bump blueprint.yaml spec.version to match chart 1.3.1
TestBootstrapKit_BlueprintCardsHaveRequiredFields/keycloak asserts
Chart.yaml.version == blueprint.yaml.spec.version. Forgot to bump
blueprint.yaml in the previous commit.
Note: 8 other blueprints (cert-manager, flux, crossplane, sealed-secrets,
spire, nats-jetstream, openbao, gitea) carry the same pre-existing
mismatch and the test fails on main too. Out of scope for this PR;
fixing the keycloak case to keep the new chart version internally
consistent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two handover bugs caught live on otech48 (2026-05-03):
1. Sovereign-side catalyst-api responded to GET /auth/handover with
"server misconfiguration: public key unavailable". Root cause: the
K8s Secret `catalyst-handover-jwt-public` (referenced by the chart's
optional Secret-volume) was never materialised on the Sovereign,
so the optional volume mount fell through and the JWK file was
absent inside the container. 1.2.0 wired the mount but no
provisioning step created the Secret. Fix mirrors the canonical
pattern from PR #543 (ghcr-pull) and PR #680 (harbor-robot-token):
cloud-init now writes the Secret manifest into catalyst-system NS
and runcmd applies it BEFORE flux-bootstrap, so the Secret exists
by the time bp-catalyst-platform reconciles. Also moves the chart
volume mount off the catalyst-api PVC (mountPath
/etc/catalyst/handover-jwt-public, no subPath) so a leftover empty
directory in the PVC from pre-#606 installs cannot collide with
the re-provisioned Secret mount.
2. /auth/handover validator rejected every valid JWT with 401
"invalid audience" because SOVEREIGN_FQDN was unset on Sovereigns
— the audience check collapsed to the literal "https://console."
prefix. The bp-catalyst-platform HelmRelease overlay was already
setting `global.sovereignFQDN` but the chart template never plumbed
it through to the Pod env. Added a SOVEREIGN_FQDN env reading
`.Values.global.sovereignFQDN` (default "" so Catalyst-Zero
installs, where catalyst-api is the SIGNER not the validator,
stay clean).
Bumps:
- bp-catalyst-platform 1.2.4 -> 1.2.5
- clusters/_template/bootstrap-kit/13-bp-catalyst-platform.yaml HelmRelease pin
Will be verified live on otech49 — fresh provision should reach
https://console.otech49.omani.works/auth/handover?token=... and
exchange to a Keycloak session WITHOUT manual Secret creation.
Issue #606 followup.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #686 added var.powerdns_api_key to variables.tf and referenced it as
${powerdns_api_key} in cloudinit-control-plane.tftpl, but missed wiring
it into the templatefile() vars dict in main.tf. Result on otech48:
Invalid value for "vars" parameter: vars map does not contain key
"powerdns_api_key", referenced at ./cloudinit-control-plane.tftpl:273
This commit closes the gap: powerdns_api_key now flows from var ->
templatefile vars -> cloud-init -> Secret manifest.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cilium-gateway): listener ports 80/443 → 30080/30443 + LB retarget
cilium-envoy refuses to bind privileged ports (80/443) on Sovereigns
even with all of:
- gatewayAPI.hostNetwork.enabled=true on the Cilium chart
- securityContext.privileged=true on the cilium-envoy DaemonSet
- securityContext.capabilities.add=[NET_BIND_SERVICE]
- envoy-keep-cap-netbindservice=true in cilium-config ConfigMap
- Gateway API CRDs at v1.3.0 (matching cilium 1.19.3 schema)
Repeatable error from cilium-envoy logs across otech45, otech46, otech47:
listener 'kube-system/cilium-gateway-cilium-gateway/listener' failed
to bind or apply socket options: cannot bind '0.0.0.0:80':
Permission denied
The bind() syscall is intercepted by cilium-agent's BPF socket-LB
program in a way that does not honour container capabilities. Even
PID 1 with CapEff=0x000001ffffffffff (all caps) and uid=0 gets
"Permission denied". Cilium 1.19.3 → 1.16.5 made no difference
(F1, PR #684 still ships — the version bump is sound for other
reasons; the listener bind is just a separate fix).
This commit moves the listeners to high ports (30080/30443) and lets
the Hetzner LB do the public-facing port translation:
HCLB :80 → CP node :30080 (cilium-gateway HTTP listener)
HCLB :443 → CP node :30443 (cilium-gateway HTTPS listener)
External users still hit `https://console.<sov>.omani.works/auth/handover`
on port 443; the high port is invisible. High-port bind succeeds
without NET_BIND_SERVICE because the kernel only gates ports below
`net.ipv4.ip_unprivileged_port_start` (default 1024).
Will be verified on otech48: the next fresh provision should serve
console.otech48/auth/handover end-to-end without the 502/timeout
chain seen on otech45–47.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(powerdns+catalyst-api): zero-touch contabo PowerDNS API key for Sovereign cert-manager
PR #681 followup. The new bp-cert-manager-powerdns-webhook (PR #681)
calls contabo's authoritative PowerDNS at pdns.openova.io to write
DNS-01 challenge TXT records for *.otech<N>.omani.works. That webhook
needs an X-API-Key Secret in the Sovereign's cert-manager namespace —
PR #681 didn't ship the materialization seam, so on otech43..otech47
the Secret was missing and the wildcard cert never issued.
This commit closes the seam from contabo to the Sovereign:
1. bp-powerdns chart 1.1.7 to 1.1.8: Reflector annotations on
openova-system/powerdns-api-credentials extended from "external-dns"
to "external-dns,catalyst" so contabo catalyst-api can mount the
API key.
2. bp-powerdns: api.basicAuth.enabled flips default true to false.
Layered Traefik basicAuth + PowerDNS X-API-Key was double auth that
blocked machine-to-machine API access from Sovereigns. The X-API-Key
contract is unchanged.
3. bp-catalyst-platform 1.2.3 to 1.2.4: api-deployment.yaml adds
CATALYST_POWERDNS_API_KEY env from powerdns-api-credentials/api-key
secret (optional=true so Sovereign-side catalyst-api Pods that don't
reflect this still start clean).
4. catalyst-api provisioner.go: new Provisioner.PowerDNSAPIKey field
reads from CATALYST_POWERDNS_API_KEY env at New(). Stamps onto every
Request before Validate(). Forwards as tofu var powerdns_api_key.
5. infra/hetzner/variables.tf: new var.powerdns_api_key (sensitive,
default "").
6. infra/hetzner/cloudinit-control-plane.tftpl: replaces the defunct
dynadot-api-credentials Secret block (PR #681 dropped
bp-cert-manager-dynadot-webhook) with a new
cert-manager/powerdns-api-credentials Secret block. runcmd applies
it BEFORE Flux reconciles bp-cert-manager-powerdns-webhook.
End-to-end seam mirrors PR #543 ghcr-pull and PR #680 harbor-robot-token.
Will be verified live on otech48 (next provision after this lands).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cilium-envoy refuses to bind privileged ports (80/443) on Sovereigns
even with all of:
- gatewayAPI.hostNetwork.enabled=true on the Cilium chart
- securityContext.privileged=true on the cilium-envoy DaemonSet
- securityContext.capabilities.add=[NET_BIND_SERVICE]
- envoy-keep-cap-netbindservice=true in cilium-config ConfigMap
- Gateway API CRDs at v1.3.0 (matching cilium 1.19.3 schema)
Repeatable error from cilium-envoy logs across otech45, otech46, otech47:
listener 'kube-system/cilium-gateway-cilium-gateway/listener' failed
to bind or apply socket options: cannot bind '0.0.0.0:80':
Permission denied
The bind() syscall is intercepted by cilium-agent's BPF socket-LB
program in a way that does not honour container capabilities. Even
PID 1 with CapEff=0x000001ffffffffff (all caps) and uid=0 gets
"Permission denied". Cilium 1.19.3 → 1.16.5 made no difference
(F1, PR #684 still ships — the version bump is sound for other
reasons; the listener bind is just a separate fix).
This commit moves the listeners to high ports (30080/30443) and lets
the Hetzner LB do the public-facing port translation:
HCLB :80 → CP node :30080 (cilium-gateway HTTP listener)
HCLB :443 → CP node :30443 (cilium-gateway HTTPS listener)
External users still hit `https://console.<sov>.omani.works/auth/handover`
on port 443; the high port is invisible. High-port bind succeeds
without NET_BIND_SERVICE because the kernel only gates ports below
`net.ipv4.ip_unprivileged_port_start` (default 1024).
Will be verified on otech48: the next fresh provision should serve
console.otech48/auth/handover end-to-end without the 502/timeout
chain seen on otech45–47.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1.16.x gateway-api hostNetwork mode is buggy on Sovereigns: cilium-envoy
NACKs listeners with "cannot bind '0.0.0.0:80': Permission denied" and
the loaded RDS for the Sovereign vhost only carries the default `/` route
to catalyst-ui — `/auth/*` and `/api/*` HTTPRoute matches defined in CEC
never reach envoy's live config. Result: console.<sov>/auth/handover?token=…
serves the React shell instead of the catalyst-api Go handler, defeating
the Phase-8b seamless handover. Caught live on otech46.
1.18+ ships the Gateway API implementation graduated from beta with the
hostNetwork bind path fixed; 1.19 is the current stable line (1.19.3).
Values shape verified backward-compatible across the keys we set:
gatewayAPI.hostNetwork.enabled, envoy.enabled, envoyConfig.enabled,
encryption.type=wireguard, encryption.nodeEncryption — all unchanged
between 1.16 and 1.19.
Bumps:
- bp-cilium chart 1.1.5 → 1.2.0 (minor — major upstream version jump)
- upstream cilium subchart 1.16.5 → 1.19.3
- blueprint.yaml spec.version 1.1.3 → 1.2.0 (was already drifted from
Chart.yaml; brings them back in sync per manifest-validation gate)
- clusters/_template/bootstrap-kit/01-cilium.yaml HelmRelease pin
1.1.5 → 1.2.0
Per-cluster overlays under clusters/<sovereign>/bootstrap-kit/ keep
their pinned versions until the operator opts in — fresh otechN
provisions render from _template/ and pick up 1.2.0 on first boot.
Will be verified live on the next fresh Sovereign provision (otech47+).
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(bp-cert-manager-powerdns-webhook): re-target to contabo PowerDNS, drop dynadot-webhook
Caught live on otech43-46: cert-manager DNS-01 challenges for
*.otechN.omani.works failed because the Sovereign-side webhook wrote
challenge TXT records to the Sovereign's local PowerDNS. omani.works is
delegated from Dynadot to ns1/2/3.openova.io which run on contabo's
central PowerDNS — the Sovereign's local PowerDNS is INVISIBLE on the
public DNS chain until pool-domain-manager seals the per-Sovereign NS
delegation. Let's Encrypt resolvers walk the public chain, query
contabo, get NXDOMAIN, the cert never issues. Manual workaround was
seeding challenge TXT directly in contabo PowerDNS.
This PR automates the right write path:
- bp-cert-manager-powerdns-webhook chart bumped to 1.0.4. Default
powerdns.host flips from "" (skip-render) to https://pdns.openova.io
(contabo's public PowerDNS API ingress, authoritative for omani.works).
- ClusterIssuer letsencrypt-dns01-prod-powerdns now usable with no
per-cluster powerdns.host override for the omani.works pool.
apiKeySecretRef.namespace clarified — upstream ignores it; the Secret
must live in cert-manager namespace (= ChallengeRequest.ResourceNamespace
for ClusterIssuers).
- bootstrap-kit slot 49 updated: drops bp-powerdns dependsOn (webhook
calls out-of-cluster contabo, not local PowerDNS), bumps chart version,
removes inline powerdns.host override (defaults are correct).
- bootstrap-kit slot 49b (bp-cert-manager-dynadot-webhook) DELETED
entirely — Dynadot is NOT the API-level authority for omani.works
subdomains, the dynadot webhook silently fails the same way the
Sovereign-local powerdns one did.
- clusters/_template/sovereign-tls/cilium-gateway-cert.yaml flips
issuerRef from letsencrypt-dns01-prod (was dynadot-backed) to
letsencrypt-dns01-prod-powerdns (the new contabo-backed issuer).
- bp-cert-manager chart: certManager.issuers.dns01.enabled defaults to
false (deprecated dynadot path). letsencrypt-http01-prod retained for
per-host certs. Cluster overlays MAY flip dns01.enabled=true for
non-omani.works pools where Dynadot IS the API-level authority.
- scripts/expected-bootstrap-deps.yaml: drops slot 49b, drops bp-powerdns
edge from slot 49.
- Documentation (README + blueprint.yaml + Chart.yaml description)
rewritten to reflect contabo retarget and lifecycle reasoning.
Credential plumbing (out of scope here, must be done in cloud-init):
- Every Sovereign needs a `powerdns-api-credentials` Secret in the
`cert-manager` namespace whose `api-key` value matches contabo's
PowerDNS API key. Same seeding pattern as `dynadot-api-credentials`
in infra/hetzner/cloudinit-control-plane.tftpl.
Caveat — basicAuth on contabo's PowerDNS API ingress: contabo currently
fronts pdns.openova.io with Traefik basicAuth (per
clusters/contabo-mkt/apps/powerdns/helmrelease.yaml). The upstream
zachomedia/cert-manager-webhook-pdns binary supports the X-API-Key
header but not HTTP Basic Auth out of the box. To make this end-to-end
green, contabo's basicAuth requirement must be relaxed (X-API-Key alone
provides the auth posture, and contabo's API endpoint is restricted to
operator IPs by other means OR the Sovereign's webhook needs an
Authorization header injected via the chart's powerdns.headers map
(plaintext password in the ClusterIssuer config — not ideal). This PR
ships the chart side; the basicAuth question is a follow-up on the
contabo side.
Verified locally:
- helm lint platform/cert-manager-powerdns-webhook/chart -> PASS
- helm template platform/cert-manager-powerdns-webhook/chart -> renders
- helm template ... --set clusterIssuer.enabled=true -> renders the
ClusterIssuer with host="https://pdns.openova.io" + correct apiKey
Secret reference.
- helm template platform/cert-manager/chart -> renders ONLY
letsencrypt-http01-prod (the dns01 dynadot issuer correctly gated off).
- scripts/check-bootstrap-deps.sh: net-zero new drift; my branch reduces
pre-existing errors from 3 to 2 (the dropped slot 49b removed the only
drift my branch was responsible for).
Closes follow-up to #373. Preconditions for handover URL TLS green
on otech43-46 lineage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(scripts): repair YAML structure in expected-bootstrap-deps.yaml
Two pre-existing drifts were blocking dependency-graph-audit CI:
1. Slot 5a (bp-reflector) was missing its closing list separator,
causing yq to merge the bp-nats-jetstream entry into the bp-reflector
map and effectively drop bp-reflector from the expected DAG.
Added explicit `- slot: 7` for bp-nats-jetstream and quoted "5a" so
yq treats it as a string slot (matches the convention with "49b").
2. bp-powerdns slot 11: actual bootstrap-kit declares dependsOn
bp-cnpg (live since otech28 — pdns-pg-app secret race) but the
expected DAG was missing this edge.
This is unblocks merging fix/cert-manager-powerdns-webhook-contabo (PR
above) — these drifts existed on main but weren't surfaced until the
last expected-deps edit forced a re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caught on otech41+; manual zone-seeding workaround was needed each
iteration. Closes#678.
## Root cause
PDM's reservation TTL is 10 minutes by default. Phase-0 (`tofu apply` on
Hetzner CP+LB + Flux bootstrap) routinely takes 8-12 minutes on a fresh
cluster, so by the time catalyst-api calls /commit the sweeper has
already deleted the reservation row. PDM returns 404 ("pool allocation
not found") and catalyst-api logged the error but kept going — the
Sovereign cluster came up live but `console.<sub>.omani.works` never
resolved because the child-zone records were never written.
Two further problems in the existing code:
1. /commit happened AFTER `close(dep.eventsCh)` and AFTER Phase 1
watch — the wizard SSE stream was already closed, so a
commit-time failure was invisible to the operator.
2. The client-side Commit only handled 200/202/404 — silently
mapped 410 (Gone, TTL expired) and 403 (token mismatch) to
a generic error.
## Fix
`pdm/client.go`:
- New sentinels `ErrExpired` (410) and `ErrTokenMismatch` (403).
- `CommitWithRetry`: 5 attempts with exponential backoff (1s → 16s
cap). On 404/410/403, calls a caller-supplied reserve closure to
obtain a fresh token, persists the new token via onRereserve
callback, and re-Commits — automatic recovery from TTL expiry,
no operator action.
- 7 unit tests covering 404→200, 410→200, 403→200, 5xx exhaustion,
5xx-then-recover, ctx-cancel-during-backoff, missing-reserve-
closure error path.
`handler/deployments.go`:
- Extracted `commitPDMWithRetry` and `releasePDMReservation` helpers.
- Moved the commit call to BEFORE Phase 1 watch starts (the LB IP
is the only data PDM needs; Phase-1 outcome doesn't change DNS
routing). Now the wizard SSE stream is still open when commit
runs, so each retry attempt + final outcome surfaces as an event.
- On final exhaustion, appends a human-actionable message to
`dep.Error` and persists, so the wizard FailureCard renders the
failure even though the cluster itself is live.
`handler/subdomains.go` + `subdomains_test.go`: pdmClient interface
adds CommitWithRetry; fakePDM in tests gets a matching shim that
delegates to the existing commit hook.
## Retry parameters
- 5 attempts total.
- Exponential backoff: 1s → 2s → 4s → 8s → 16s (capped).
- Per-attempt HTTP timeout: 15s (existing Client.HTTP timeout).
- Outer ctx timeout: 5 minutes (well above the worst-case 1+2+4+8+16+
per-attempt-HTTP).
- 404/410/403 do NOT sleep before re-Reserve (the row is gone, not
flapping) — they still count against MaxAttempts.
Co-authored-by: hatiyildiz <269457768+hatiyildiz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caught live on otech43–46 — manual placeholder Secret was being created
each iteration. RCA:
The catalyst-api Pod template references the `harbor-robot-token`
Secret via a REQUIRED (non-optional) secretKeyRef. On Sovereign
clusters that Secret was never materialised — only `ghcr-pull` had
the canonical cloud-init + Reflector auto-mirror seam (PR #543). The
chart's old comment said "Reflector mirrors from openova-harbor
namespace into catalyst" but `openova-harbor` doesn't exist on
Sovereigns; that namespace lives only on contabo where the central
Harbor source Secret is administered. Result: every fresh Sovereign's
catalyst-api Pod stuck in CreateContainerConfigError until the
operator hand-created a placeholder Secret.
The token VALUE was already arriving on the Sovereign — Tofu
var.harbor_robot_token is interpolated into
/etc/rancher/k3s/registries.yaml at cloud-init time so containerd
can authenticate against harbor.openova.io. We just never materialised
the same value as a Kubernetes Secret for catalyst-api to mount.
Permanent fix mirrors the canonical `ghcr-pull` seam:
1. infra/hetzner/cloudinit-control-plane.tftpl write_files block
emits /var/lib/catalyst/harbor-robot-token-secret.yaml — a
Secret in flux-system ns with auto-mirror Reflector annotations
(`reflection-auto-enabled: "true"`).
2. runcmd applies it BEFORE flux-bootstrap, so the Secret exists
before any Helm release reconciles.
3. bp-reflector (slot 05a, already deployed) propagates the Secret
into every namespace — including catalyst-system — on first
reconcile tick. catalyst-api's secretKeyRef resolves cleanly,
Pod starts.
4. Token rotation flows through `var.harbor_robot_token` →
re-render Tofu → re-apply cloud-init; Reflector propagates the
rotation to all mirrored copies on the next watch tick.
`harbor-robot-token` stays NOT optional in the chart: the architecture
mandate is every Sovereign image pull goes through harbor.openova.io;
falling through to docker.io is forbidden (anonymous rate-limit makes
a fresh Hetzner IP unbootable). A missing token must surface
immediately as Pod start failure, never silently mid-provision.
Bumps:
- bp-catalyst-platform 1.2.2 → 1.2.3 (chart-side change is a
comment-only update on the secretKeyRef explaining the new seam;
the Pod spec still references the same Secret name and key).
- clusters/_template/bootstrap-kit/13-bp-catalyst-platform.yaml
HelmRelease version pin → 1.2.3.
No bootstrap-kit dependency changes — bp-reflector's slot-05a position
is unchanged and was already a dependency for ghcr-pull. No
expected-bootstrap-deps.yaml edits needed.
Issue #557 follow-up. Closes the per-Sovereign manual workaround.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caught live on otech43–46: external-dns crashloops 10+ times on fresh
Sovereign before initial *v1.Pod sync completes. Default 30s timeout
insufficient when k3s apiserver is CPU-saturated.
Co-authored-by: hatiyildiz <hatiyildiz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The console.<sov>.omani.works hostname HTTPRoute caught everything
under PathPrefix '/' and sent it to catalyst-ui (the React shell).
But the handover JWT lands at /auth/handover, implemented by
catalyst-api (the Go backend). Result: React app saw /auth/handover,
had no client-side route for it, and the catch-all auth-guard
redirected to Keycloak's bare login screen — defeating Phase-8b
seamless auth. Founder caught it on otech46: 'still asking password'.
Add two rules BEFORE the catch-all:
/auth/* → catalyst-api:8080
/api/* → catalyst-api:8080
/ → catalyst-ui:80 (unchanged)
Chart bumped to 1.2.2.
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
The MintHandoverToken handler only read X-Forwarded-User /
X-Forwarded-Email — headers set by an upstream OIDC proxy. But on
Catalyst-Zero (console.openova.io) the auth path is magic-link →
Keycloak session cookie → catalyst-api's own auth.RequireSession
middleware, which sets X-User-Sub and X-User-Email instead.
Result: JWT carried sub='unknown' email='unknown'. Sovereign-side
handover handler couldn't pre-provision the operator account and
fell through to Keycloak's bare login screen — defeating the
Phase-8b seamless-auth promise (#20).
Caught live on otech46: founder navigated handover URL and saw
'Sovereign — Sign in to your account' instead of landing on the
Sovereign Console.
Fix: read X-User-Sub / X-User-Email FIRST, fall back to
X-Forwarded-* / X-Auth-Request-* for OIDC-proxy compatibility.
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
Live test on console.openova.io showed the canvas wrapper kept its
hardcoded dark navy radial gradient under [data-theme="light"] — the
LogPane reskinned, the bubble fills reskinned, but the `.flow-canvas-host`
backdrop stayed dark. Route the gradient through CSS variables with a
slate light-mode peer; same treatment for the border colour.
Also rename the inner SVG host's data-testid from `flow-canvas-host`
(name clash with FlowPage's outer .flow-canvas-host wrapper) to
`flow-canvas-svg-host` so test queries / Playwright probes don't get
the wrong element.
Refs #669, follow-up to #671/#672/#673.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sovereign-console): use DerivedJob.title not displayName/jobName (#669 follow-up)
Build-ui failed in CI on `tsc -b` (which `tsc --noEmit` doesn't catch
locally without strict project-references). DerivedJob from
src/pages/sovereign/jobs.ts uses `title`, not the flat-Job
`displayName`/`jobName` fields. Use `dj.title || dj.id` for the
global-log component-name prefix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow-canvas): MIN_HOST is a fallback, not a floor (#669 follow-up)
Live test on console.openova.io after PR #671 showed bubbles overlapping
by ~13 CSS px. Root cause: ResizeObserver clamped hostSize.w to
max(MIN_HOST_W=1200, contentRect.w=686). The SVG then rendered 1200
viewBox-units into 686 CSS px (0.57× downscale), shrinking bubble
diameters AND collapsing pairwise distances below the
NODE_RADIUS*2 + COLLIDE_PADDING (= 92 px) threshold.
Use the actual contentRect dimensions; only fall back to MIN_HOST
when the rect is 0×0 (degenerate first-paint). Now viewBox = host px
1:1 → bubble radius is exactly NODE_RADIUS CSS px and forceCollide's
pairwise spacing guarantee holds in screen space.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Cilium gateway-api L7LB nodePort chain was silently broken on
otech45: TCP to LB:443 succeeds, but TLS handshake never completes.
Root cause: Cilium 1.16.5's BPF L7LB Proxy Port (12869) doesn't match
what cilium-envoy actually listens on (verified via /proc/net/tcp on
the cilium-envoy pod — port 12869 not in listening sockets). The
nodePort indirection (31443→envoy:12869) is broken at the redirect
step.
Fix: bind cilium-envoy directly to the host's :80 and :443 via
gatewayAPI.hostNetwork.enabled=true. Hetzner LB forwards public
80→private:80 and 443→private:443 directly (no nodePort indirection).
Two coordinated changes:
1. platform/cilium/chart/values.yaml: gatewayAPI.hostNetwork.enabled=true
2. infra/hetzner/main.tf: LB destination_port = 80/443 (was 31080/31443)
bp-cilium chart bumped to 1.1.5.
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
Build-ui failed in CI on `tsc -b` (which `tsc --noEmit` doesn't catch
locally without strict project-references). DerivedJob from
src/pages/sovereign/jobs.ts uses `title`, not the flat-Job
`displayName`/`jobName` fields. Use `dj.title || dj.id` for the
global-log component-name prefix.
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
JobDetail page rewrite addressing five UX issues reported on the
running otech-N Sovereign console.
1. Flow canvas viewBox now tracks the host pixel rect via ResizeObserver
instead of being capped at 1200x700 with `preserveAspectRatio meet`.
Bubble radius (NODE_RADIUS=40) renders at 40 CSS px regardless of
host size; full-screening the canvas grows layout space along x for
the dependency chain instead of magnifying every bubble.
2. Removed the projection xScale/yScale compression that caused
overlap on wide clusters (positions scaled but not rendered radii,
defeating forceCollide). The per-tick clamp is now bounded by
hostSize.{w,h} so forceCollide protects pairwise distance end to
end, satisfying the founder's no-overlap rule.
3. Completed bubbles are now solid green (#16A34A) with a white tick
so done-vs-pending reads instantly. Was: dark fill + light-green
glyph that read identically to pending at a glance.
4. Status palette + log viewer surface now route through CSS variables
(--bubble-* and --log-viewer-*) with [data-theme=light] peers in
globals.css, so the canvas + ExecutionLogs reskin properly under
light theme. Was: hardcoded dark hex everywhere.
5. ExecutionLogs auto-tail uses scrollTo({behavior:smooth}) and each
incoming row plays a 180ms fade+rise animation. Reads as a real
tail -f stream.
6. JobDetail header collapsed: PortalShell renders the title; the
in-page strip keeps only Back, last-update timestamp and the status
chip. Removed the redundant subtitle line and the "Logs" reopen-pane
button (it overlapped the status chip when the pane was closed).
7. New: split-view toggle in LogPane. When on, body becomes 2 columns:
per-component on the left, provision-wide merged log stream on the
right. Global stream is built client-side by interleaving every
derived job's SSE step events by timestamp; updates live with the
reducer state.
Tests: src/test/setup.ts adds a ResizeObserver polyfill for jsdom.
JobDetail.test + FlowCanvasOrganic.bounded green; ExecutionLogs colour
test updated to assert on the CSS-variable wiring instead of the
resolved hex (jsdom doesn't load globals.css).
Closesopenova-io/openova#669
Co-authored-by: hatiyildiz <hatice@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous attempt referenced 'bp-gitea.labels' helper which doesn't
exist in this chart (bp-gitea has no _helpers.tpl, unlike bp-harbor).
Blueprint Release workflow's helm-template gate caught it:
template: bp-gitea/templates/database-secret-sync-job.yaml:53:8:
error calling include: template: no template 'bp-gitea.labels'
associated with template 'gotpl'
Fix: replace the 4 occurrences of 'include bp-gitea.labels' with
explicit catalyst.openova.io/blueprint + component labels. Same
shape, no helper dependency.
Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
