919514c added Helm template expressions (`{{ .Values.* }}`) into
products/catalyst/chart/templates/ingress.yaml + ui-deployment.yaml +
ui-configmap.yaml + values.yaml. These files are consumed by the
catalyst-platform Flux Kustomization on Catalyst-Zero (Contabo), which
goes through kustomize-controller — not helm-controller — so the
template expressions are NOT rendered.
Failure observed in production:
catalyst-platform kustomize build failed: updating name reference in
spec/ingressClassName field of Ingress.networking/console-sovereign:
path config error; no name field in node
The ingressClassName template expression broke kustomize's name-reference
resolver. The ConfigMap with Helm expressions in nginx config strings
would have left nginx unable to resolve upstreams at runtime.
Surgical revert:
- ingress.yaml, ui-deployment.yaml: back to pre-919514c plain YAML
- ui-configmap.yaml, values.yaml: deleted (had no plain-YAML predecessor)
The values-driven /sovereign nginx routing remains the right target
state — but the path forward is to convert catalyst-platform to a Flux
HelmRelease (helm-controller renders templates), not to mix Helm
templates into a kustomize-applied directory. Tracking ticket follows.
Per docs/INVIOLABLE-PRINCIPLES.md #4 (never hardcode), the catalyst-ui
nginx config now flows from values.yaml at chart-render time:
- routing.basePath (/sovereign) — also drives ingress strip-prefix
- routing.catalystApi.serviceDNS — in-cluster reverse-proxy target
- routing.catalystApi.port — upstream port
- dns.resolverIP — CoreDNS for proxy-time resolution (avoids stale
ClusterIP after catalyst-api restarts)
- ingress.host / ingress.priority / ingress.className
Files:
- products/catalyst/chart/values.yaml — new, documents every default
- products/catalyst/chart/templates/ui-configmap.yaml — new, nginx
reverse-proxies /api/* to catalyst-api Service DNS
- products/catalyst/chart/templates/ui-deployment.yaml — mounts the
ConfigMap at /etc/nginx/conf.d/default.conf
- products/catalyst/chart/templates/ingress.yaml — values-driven host
+ path + priority + class
- tests/e2e/sovereign-routing/* — Playwright smoke for the routing
Captured from stalled agent /tmp/agent-sovereign-route-finish — agent
stream watchdog timed out after the work was authored but before commit.
Issue #104: products/catalyst/chart/Chart.yaml had `name: catalyst-platform`
(missing the `bp-` prefix required by BLUEPRINT-AUTHORING.md §3) and no
`dependencies:` block. The Catalyst umbrella must depend on the 11 bootstrap-kit
leaf Blueprints so a single Flux HelmRelease at the umbrella OCI ref pulls in
the full Catalyst-Zero control plane.
Issue #107: bp-catalyst-platform was the missing 11th OCI artifact at
ghcr.io/openova-io. With this fix, blueprint-release.yaml will publish
ghcr.io/openova-io/bp-catalyst-platform:1.0.1 on push.
Changes:
- Rename chart to `bp-catalyst-platform`, bump version 1.0.0 -> 1.0.1
- Add `dependencies:` block listing all 11 leaves
(cilium, cert-manager, flux, crossplane, sealed-secrets, spire,
nats-jetstream, openbao, keycloak, gitea, external-dns), each
pinned to 1.0.0 at oci://ghcr.io/openova-io
- Workflow blueprint-release.yaml: read chart name from Chart.yaml `name:`
field instead of deriving `bp-<basename>` from the folder. The umbrella
folder is `catalyst` but the chart name is `bp-catalyst-platform` —
basename-derivation is wrong for any chart whose name doesn't equal
`bp-<folder>`. Removes the implicit `bp-` prefix in the push step;
Chart.yaml carries the full canonical name.
- Workflow: add `helm registry login ghcr.io` step before `helm dependency
build` so OCI-hosted leaf deps resolve. The pre-existing docker login
is for cosign/syft only; helm has its own auth store.
Disclosure (per INVIOLABLE-PRINCIPLES.md §8):
- bp-external-dns:1.0.0 is listed as a dependency but is not yet published;
platform/external-dns/ has README + policies but no chart/ dir (issue #109
scope). The umbrella build will fail on `helm dependency build` until #109
authors the chart and publishes bp-external-dns:1.0.0. The dependency is
declared anyway because the target-state contract per #104 is exactly 11
leaves — partial declaration would be a quality compromise (principle #2).
Verified leaf chart names (platform/<x>/chart/Chart.yaml, all `bp-<x>`):
cilium, cert-manager, flux, crossplane, sealed-secrets, spire,
nats-jetstream, openbao, keycloak, gitea — all match.
Verified published OCI tags (10/11 at ghcr.io/openova-io/bp-<name>:1.0.0).
The dynadot-api-credentials K8s secret in openova-system used to carry a
single `domain=openova.io` field. Per docs/INVIOLABLE-PRINCIPLES.md #4
("never hardcode") and the design constraint that adding a third pool
domain (e.g. acme.io) must NOT require a code change, the secret now
carries a `domains` field — a comma- or whitespace-separated list — and
the catalyst-api reads it at runtime via DYNADOT_MANAGED_DOMAINS.
Resolution order in dynadot.IsManagedDomain:
1. DYNADOT_MANAGED_DOMAINS env (canonical, multi-domain)
2. DYNADOT_DOMAIN env (legacy single-value, backward-compat)
3. Built-in defaults (openova.io, omani.works) — defensive fail-closed
fallback if the secret was not mounted.
The deployment manifest mounts both env vars from the secret with
optional=true, so existing clusters whose secret only carries the legacy
`domain` key keep working; migration is a one-key secret update with no
deployment edit required.
Closes#108.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per docs/PROVISIONING-PLAN.md and tickets [E] provisioner: bootstrap orchestrator. Adds the missing piece that turns a freshly-provisioned k3s cluster into a fully-functional Sovereign.
products/catalyst/bootstrap/api/internal/bootstrap/bootstrap.go:
- Step struct with Name/Phase/Install function
- Run() iterates DefaultSteps in dependency order, aborts on first error
- 11 install functions matching SOVEREIGN-PROVISIONING.md §3 Phase 0:
1. Cilium (CNI must come first — k3s started with --flannel-backend=none precisely so Cilium can take over)
2. cert-manager (CRDs + webhook ready before anything below issues TLS)
3. Flux (host-level GitOps)
4. Crossplane core + provider-hcloud (Phase 1 hand-off point per §4)
5. Sealed Secrets (transient bootstrap-only)
6. SPIRE server + agent (5-min SVID rotation)
7. NATS JetStream (3-node, control-plane event spine)
8. OpenBao (3-node Raft, region-local — no stretched cluster per SECURITY §5)
9. Keycloak (topology decided by Sovereign CRD spec.keycloakTopology)
10. Gitea (per-Sovereign Git server)
11. bp-catalyst-platform umbrella (registers Catalyst CRDs)
Each install pulls bp-<name>:<semver> from ghcr.io/openova-io/ via helm OCI install, with Catalyst-curated values overlay (cilium values inline shows kubeProxyReplacement+WireGuard mTLS+Hubble+Gateway API+Envoy).
products/catalyst/bootstrap/api/internal/bootstrap/exec.go:
- runHelm — exec helm CLI with kubeconfig flag, optional values from STDIN
- applyManifest — kubectl apply -f - with manifest from STDIN
- waitForDeployment — polls kubectl rollout status until Ready or timeout
- writeKubeconfig — temp file with mode 0600, returns cleanup func; never sets KUBECONFIG env var so concurrent provisioning runs don't race
Wired into hetzner.Provisioner.Provision: after fetchKubeconfig completes, bootstrap.Run installs the 11-component kit and emits per-step events to the wizard via the same SSE channel. Failures abort with a clear "step <name> failed" error.
Containerfile updates:
- Switch from FROM scratch to FROM alpine:3.20 (kubectl + helm need ca-certs + glibc-equivalents)
- Pin kubectl v1.31.4 (matches K3s install version) and helm v3.16.3
- adduser nonroot:65534 instead of bare USER 65534:65534
api-deployment.yaml updates:
- readOnlyRootFilesystem: false (helm cache + temp kubeconfigs need /tmp + /home/nonroot writable)
- emptyDir volumes for /tmp and /home/nonroot, sizeLimit 256Mi each
Closes [E] tickets: bootstrap orchestrator, k3s installation script (already in cloud-init), 11-component dependency order, helm/kubectl exec wrapper.
The 11 bp-<name> OCI artifacts must exist on ghcr.io before this installer can succeed. Group F charts ([F] tickets) will land them.
Per docs/PROVISIONING-PLAN.md and ticket [G] dns. Adds the missing pool-domain DNS automation: when a wizard user picks "OpenOva pool subdomain → omani.works → omantel", the provisioner now writes 6 A records via Dynadot's API so omantel.omani.works (and console./gitea./harbor./admin./api. underneath) all resolve to the new Hetzner load balancer.
New code:
products/catalyst/bootstrap/api/internal/dynadot/dynadot.go
- Client wraps Dynadot's REST API (set_dns2 with add_dns_to_current_setting=yes — never replace, always append, per the explicit "NEVER run exploratory set_dns2" warning in feedback_dynadot_dns.md)
- AddRecord — single-record append with subdomain+type+value+TTL
- AddSovereignRecords — canonical 6-record set: *.{sub}, console.{sub}, gitea.{sub}, harbor.{sub}, admin.{sub}, api.{sub} all → LB IP
- IsManagedDomain — returns true for openova.io and omani.works (the pool entries from the wizard's SOVEREIGN_POOL_DOMAINS list)
provisioner.go additions:
- ProvisionRequest gets SovereignDomainMode/SovereignPoolDomain/SovereignSubdomain fields
- DynadotAPIKey/DynadotAPISecret unmarshalled from "-" (handler injects from env at runtime; never round-tripped via wizard)
- New "dns" phase in Provision(): if pool-mode + managed domain → call dynadot.AddSovereignRecords; else emit a "BYO" message telling the customer to point their own CNAME at the LB IP
handler/handler.go:
- Handler now reads DYNADOT_API_KEY + DYNADOT_API_SECRET from environment
handler/deployments.go:
- CreateDeployment injects Dynadot credentials into req when SovereignDomainMode == "pool"
- BYO mode: provisioner runs without Dynadot; the success Result still includes LB IP so the wizard can show the customer the value to put in their CNAME
products/catalyst/chart/templates/api-deployment.yaml:
- catalyst-api Deployment env extended: DYNADOT_API_KEY + DYNADOT_API_SECRET sourced from the dynadot-api-credentials Secret (per project-memory: this secret already exists in openova-system namespace in Catalyst-Zero with account-scoped Dynadot credentials covering openova.io and omani.works)
Closes [G] tickets: dns multi-domain support, Dynadot client extension, A-record write during provisioning. Wildcard-A subdomain check (cross-checks against existing Sovereigns) tracked separately as [G] dns: implement subdomain reservation check.
Per docs/PROVISIONING-PLAN.md and tickets [B] sme-backend group. Migrates the 8 Go backend services from openova-private/services/ to openova/core/services/, plus the shared module they all depend on, plus the services-build CI workflow.
What moved:
- services/auth → core/services/auth (Go HTTP service for SME marketplace authentication)
- services/billing → core/services/billing (Go HTTP service for billing + voucher backend)
- services/catalog → core/services/catalog (Go HTTP service for App catalog)
- services/domain → core/services/domain (Go HTTP service for tenant domain mapping)
- services/gateway → core/services/gateway (Go HTTP gateway with rate limiting)
- services/notification → core/services/notification (Go HTTP service with email templates)
- services/provisioning → core/services/provisioning (Go HTTP service that commits tenant Application manifests via Gitea/GitHub API)
- services/tenant → core/services/tenant (Go HTTP service for tenant lifecycle)
- services/shared → core/services/shared (shared Go module: db, events, health, middleware, respond)
- 9 go.mod files updated: module github.com/openova-io/openova-private/services/<X> → github.com/openova-io/openova/core/services/<X>
- 9 go.sum and import paths similarly updated
- replace directives updated: openova-private/services/shared → openova/core/services/shared
- sme-services-build.yaml workflow → services-build.yaml in .github/workflows/, paths/context/image-base/deploy paths all repointed at core/services + ghcr.io/openova-io/openova/services-* + products/catalyst/chart/templates/sme-services
- All 8 manifests in products/catalyst/chart/templates/sme-services/ updated: image refs ghcr.io/openova-io/openova-private/sme-{X} → ghcr.io/openova-io/openova/services-{X}
- provisioning.yaml GITHUB_REPO env var: "openova-private" → "openova"
Closes [B] sme-backend (10 tickets).
After this commit, all 14 user-facing + backend Catalyst-Zero modules build from this public repo:
- 4 UIs: console, admin, marketplace, catalyst-ui
- 2 backends: marketplace-api, catalyst-api
- 8 SME services: auth, billing, catalog, domain, gateway, notification, provisioning, tenant
- 1 shared Go module
Note: 1 line in core/services/provisioning/main.go retains a literal default of "openova-private" for the GITHUB_REPO fallback when env var is unset; the K8s manifest sets GITHUB_REPO=openova explicitly so this path is never exercised in the deployed runtime, and the in-code default will be cleaned up in a follow-up.
Per docs/PROVISIONING-PLAN.md Phase 1. Catalyst-Zero (the running deployment on Contabo k3s, namespaces catalyst/sme/marketplace/website) source code now lives in this public repo. Cutover to public-repo CI builds happens in Phase 2.
What moved (from openova-private → openova):
- apps/console/ → core/console/ (Astro+Svelte UI)
- apps/admin/ → core/admin/ (Astro+Svelte UI, includes canonical voucher/billing/tenants admin surface)
- apps/marketplace/ → core/marketplace/ (Astro+Svelte UI, 5-step Plan→Apps→Addons→Checkout→Review flow)
- website/marketplace-api/ → core/marketplace-api/ (Go backend with handlers/, provisioner/, store/)
- clusters/contabo-mkt/apps/catalyst/ → products/catalyst/chart/templates/ (catalyst-{ui,api} K8s manifests)
- clusters/contabo-mkt/apps/sme/services/ → products/catalyst/chart/templates/sme-services/ (15 manifests)
- clusters/contabo-mkt/apps/marketplace-api/ → products/catalyst/chart/templates/marketplace-api/
- 5 CI workflows (catalyst-build, marketplace-api-build, sme-{admin,console,marketplace}-build) → .github/workflows/, renamed to drop "sme-" prefix
Image refs updated:
- ghcr.io/openova-io/openova-private/catalyst-{ui,api} → ghcr.io/openova-io/openova/catalyst-{ui,api}
- ghcr.io/openova-io/openova-private/sme-{admin,console,marketplace} → ghcr.io/openova-io/openova/{admin,console,marketplace}
- ghcr.io/openova-io/openova-private/marketplace-api → ghcr.io/openova-io/openova/marketplace-api
Workflow path updates:
- paths: 'apps/{X}/**' → 'core/{X}/**'
- context: apps/{X} → core/{X}
- deploy paths: clusters/contabo-mkt/apps/{X}/.../{X}.yaml → products/catalyst/chart/templates/.../{X}.yaml
- deploy commit: git add clusters/ → git add products/
Deferred to follow-up phase:
- 8 legacy SME backend services (auth, billing, catalog, domain, gateway, notification, provisioning, tenant) keep their ghcr.io/openova-io/openova-private/sme-* image refs because their source code in openova-private/services/ has not yet been migrated to public repo. Tracked via TODO in core/README.md migration history.
- sme-services-build.yaml NOT migrated (matches deferred services).
Documentation updates:
- core/README.md rewritten to describe what's actually in this directory now (4 deployed modules, not the old Go-monorepo placeholder design)
- products/catalyst/README.md created with migration status table
- products/catalyst/chart/Chart.yaml created (umbrella bp-catalyst-platform chart)
- docs/IMPLEMENTATION-STATUS.md §1 + §2.1 + §6 updated: console/admin/marketplace/marketplace-api/catalyst-{ui,api} all flipped from 📐 to 🚧 (deployed but not yet wired to unified Catalyst contract); openova Sovereign description rewritten to make Catalyst-Zero status explicit; omantel target updated to omantel.omani.works on Hetzner.
Verification:
- 99 source files copied (verified via git ls-files count)
- All image refs updated except the 8 deferred legacy SME backend services (verified via grep openova-private)
- Workflow naming reflects unified Catalyst (no more "sme-" prefix)
Phase 2 next: trigger public-repo CI builds, GHCR images published under openova/ namespace, Flux source on Catalyst-Zero repointed to this repo, rolling update of Contabo pods to new image SHAs. Catalyst-Zero becomes self-built from the public repo.
