# Catalyst-Zero Provisioning Plan **Status:** Authoritative working plan — **execution underway**. **Updated:** 2026-04-29. **Owner:** OpenOva engineering. **Parent issue:** [#43](https://github.com/openova-io/openova/issues/43). **Sub-tickets:** A–M groups, [#45–#175](https://github.com/openova-io/openova/issues?q=is%3Aopen+%5B+). Post-Group-M continuation tickets (#161, #162, #163, #167, #168, #169, #170, #171, #173, #174, #175) extend the plan with the per-Sovereign PowerDNS zone model, pool-domain-manager + registrar adapters, three-mode StepDomain (pool/byo-manual/byo-api), the wizard StepComponents redesign, and k8gb retirement. --- ## Execution status (live) | Group | Tickets | Status | Commits | |---|---|---|---| | A — Code consolidation | 9 | ✅ Done | 3c2f7e4 | | B — SME backend services | 10 | ✅ Source migrated; CI workflow live | 7646840 | | C — Cutover Catalyst-Zero | 8 | ✅ Flux is now reconciling Catalyst-Zero from `github.com/openova-io/openova` (public repo) — confirmed via `kubectl get gitrepository -A` returning `openova-public` source serving the catalyst-platform Kustomization | 9d93912, dc56854, bd967a7, 61de3da, 9fdfe07, 8c40984 (Group C cutover merge) | | D — Wizard | 10 | 🚧 Domain capture + Hetzner project ID added; AppsStep replacement pending | 854a063 | | E — Provisioner backend | 13 | 🚧 Real Hetzner client + bootstrap installer + Dynadot DNS landed; SSH kubeconfig fetch is stub | 915c467, db4f21a, 07b4bcf | | F — Bootstrap-kit Helm charts | 14 | ✅ All 12 G2 wrapper charts (original 11 + bp-powerdns #167) + blueprint-release CI live | 8c0f766, 0190c605 | | G — DNS multi-domain | 6 | ✅ Superseded by PowerDNS authoritative (#167) + pool-domain-manager (#163) + registrar adapters (#170) — Dynadot is now one of five registrar adapters inside PDM, not the authoritative DNS surface | db4f21a, 0190c605 (#167), 2854d652 (#163), 567d7e1f (#170) | | H — Franchise model | 7 | 🚧 docs/FRANCHISE-MODEL.md authored from existing admin impl; cross-Sovereign voucher deferred | this commit | | I — Wizard UX | 6 | 📐 SSE event log pane + step indicator pending | | | J — Hetzner infra | 6 | 🚧 cloud-init in repo; firewall + k3s flags wired into provisioner | 07b4bcf | | K — Documentation | 8 | 🚧 IMPLEMENTATION-STATUS + core/README + products/catalyst/README updated; component-count anchor refreshed 53 → 56 (spire + nats-jetstream + sealed-secrets factored in); reconcile-pass-1 (2026-04-29) refreshed canonical docs against PowerDNS/PDM/registrar-adapter ground truth | 3c2f7e4, 8c0f766, group-k-docs, reconcile-pass-1 | | L — Testing | 8 | 📐 Playwright + integration tests pending | | | M — End-to-end DoD | 9 | 📐 Awaiting Hetzner credentials from operator + first OCI-artifact CI runs to complete | | --- This document captures the agreed plan for consolidating the existing nova/console/admin/marketplace code into the public OpenOva Catalyst monorepo, deploying it as **Catalyst-Zero** (the first Catalyst Sovereign — running on Contabo, the chicken in the chicken-and-egg problem), and then provisioning the first **franchised Sovereign** on Hetzner via the wizard at `console.openova.io/sovereign`. This plan is the canonical reference. It supersedes any session-local conversation. Future Claude sessions or new contributors should read this first. --- ## 1. The chicken-and-egg problem and its resolution Catalyst is a Kubernetes-native control plane that provisions other Sovereigns. Provisioning a Sovereign requires a **provisioner service** (`catalyst-provisioner.openova.io` per `SOVEREIGN-PROVISIONING.md` §2). That provisioner has to **run somewhere**. It cannot run inside the Sovereign it is provisioning (chicken-and-egg). **Resolution:** the legacy nova/console/admin/marketplace stack currently running on **Contabo k3s** (in namespaces `catalyst`, `sme`, `marketplace`, `website`) is **Catalyst-Zero** — the first Sovereign. It exists today, has running pods today, and is the chicken from which the egg (the first Hetzner-hosted franchised Sovereign) gets provisioned. The work in this plan **consolidates** that existing code into the public repo, **redeploys** it as a public-repo build (CI from `github.com/openova-io/openova`), and then **uses it** to provision the first franchised Sovereign. There is no greenfield "build Catalyst from scratch" — the Sovereign already exists; we are aligning it to the canonical Catalyst contract. --- ## 2. Current state inventory (verified against live cluster + repos, 2026-04-28) ### 2.1 Code locations (today) | What | Where today | Where it must end up | |---|---|---| | Catalyst console UI (Astro+Svelte) | `openova-private/apps/console/` | `openova/core/console/` | | Catalyst admin UI (Astro+Svelte) | `openova-private/apps/admin/` | `openova/core/admin/` | | Catalyst marketplace UI (Astro+Svelte) | `openova-private/apps/marketplace/` | `openova/core/marketplace/` | | marketplace-api (Go backend) | `openova-private/website/marketplace-api/` | `openova/core/marketplace-api/` | | Catalyst-zero deployment chart | `openova-private/clusters/contabo-mkt/apps/catalyst/` | `openova/products/catalyst/chart/templates/` | | Vite scaffold for sovereign-wizard | `openova/products/catalyst/bootstrap/ui/` | merges into `openova/core/console/src/pages/sovereign/` | | CI workflows (6 of them) | `openova-private/.github/workflows/{catalyst-build,marketplace-api-build,sme-{admin,console,marketplace,services}-build}.yaml` | `openova/.github/workflows/` | | Voucher / billing / tenants admin surface | `openova-private/apps/admin/src/{components/BillingPage.svelte, lib/api.ts, pages/{billing,catalog,orders,tenants}.astro}` | `openova/core/admin/...` (carry forward unchanged) | ### 2.2 Live deployment on Contabo (verified via `kubectl get all -A`) | Namespace | Pods running | Notes | |---|---|---| | `catalyst` | catalyst-api + catalyst-ui | 39 days uptime | | `sme` | console + admin + marketplace | 5–6 days uptime | | `marketplace` | marketplace-api | 13 days uptime | | `website` | openova-website | live | These pods are Catalyst-Zero. They stay running through Phases 1–2; Phase 2 is a rolling-update cutover to public-repo image builds. ### 2.3 Existing 5-step wizard (the "Components (5)" page reference) The "Components (5)" the user referenced is the 5-step marketplace flow at `openova-private/apps/marketplace/src/components/`: ``` PlanStep → AppsStep → AddonsStep → CheckoutStep → ReviewStep ``` `AppsStep` is what gets replaced with the unified marketplace card grid (driven by the same `bp-` Blueprint surface every Catalyst Sovereign uses). ### 2.4 Voucher mechanism (already implemented) Lives in `openova-private/apps/admin/`: - `src/components/BillingPage.svelte` — voucher / billing UI - `src/lib/api.ts` — voucher API client - `src/pages/{billing,catalog,orders,tenants}.astro` — admin pages This is the **canonical** voucher implementation. Do not redesign. Read what's there, propagate to franchised Sovereigns, document in `docs/FRANCHISE-MODEL.md`. --- ## 3. Architectural agreements (from the design conversation, durable) These agreements survive any context compaction and apply to every phase of the work below. 1. **Catalyst-Zero is the existing Contabo deployment.** Not greenfield. The work is consolidate + cutover + extend, not rebuild. 2. **omani.works is the first Sovereign-provided subdomain pool** (registered to the OpenOva Dynadot account). User dynamically picks `omantel.omani.works` during provisioning. The wizard offers BYO domain (customer's own) or a Sovereign-pool subdomain (default). Multi-region setups are out of scope for the first run. 3. **Existing admin voucher implementation is the source of truth.** Do not propose new CRDs. Read the existing implementation, propagate it to franchised Sovereigns, document it. 4. **G2 quality only.** Catalyst-curated wrapper Helm charts at `platform//chart/` for every component in the bootstrap kit. No upstream-as-is shortcuts. No corner-cutting. The unified Blueprint contract from `BLUEPRINT-AUTHORING.md` §1 is the standard. 5. **No mocks. No iterations. No partial deliveries.** Waterfall — every phase produces real, deployed, working artifacts. 6. **All product code is public.** Per the build-minutes constraint, code moves to `openova/` (the public monorepo) before any further development. CI runs in the public repo from this point onward. 7. **The Vite scaffold at `products/catalyst/bootstrap/ui/`** merges into `core/console/src/pages/sovereign/`. It does not become its own deployable. 8. **Sovereign-provisioning wizard target URL: `console.openova.io/sovereign`.** Captured fields include domain (BYO or pool), Hetzner Cloud API token, Hetzner project ID, Hetzner region (runtime parameter, never hardcoded), plus the marketplace-style App selection. 9. **The Hetzner region is a runtime parameter chosen by the wizard user.** Never hardcoded anywhere in code. 10. **Dynadot is OpenOva's registrar of record for the pool domains.** The `dynadot-api-credentials` K8s secret in `openova-system` is account-scoped and covers `openova.io` plus `omani.works` (and any other domain in the same Dynadot account). Post-#167/#170 Dynadot is **not** authoritative DNS for any Sovereign zone — bp-powerdns is. Dynadot is one of five registrar adapters PDM uses to (a) keep the OpenOva pool domains' parent-zone NS records pointing at OpenOva PowerDNS and (b) honour `byo-api` Sovereigns whose customer happens to use Dynadot. --- ## 4. The 8-phase waterfall Each phase produces one or more commits to `openova/`. Each commit is real working code, not scaffold. No phase is skipped, abbreviated, or deferred. ### Phase 1 — Code consolidation (openova-private → openova) **What:** `git mv` the 4 apps (`console`, `admin`, `marketplace`, `marketplace-api`) from openova-private to `openova/core/`. Move 6 CI workflows to `openova/.github/workflows/`. Move Catalyst-Zero deployment manifests from `openova-private/clusters/contabo-mkt/apps/catalyst/` to `openova/products/catalyst/chart/templates/`. **Outputs:** - `openova/core/{console,admin,marketplace,marketplace-api}/` populated - `openova/.github/workflows/{catalyst-build,marketplace-api-build,sme-*-build}.yaml` - `openova/products/catalyst/chart/templates/{api-deployment,api-service,ui-deployment,ui-service,ingress}.yaml` - `openova/products/catalyst/chart/Chart.yaml` (new) - All import paths, image refs (`ghcr.io/openova-io/openova/{console,admin,marketplace,marketplace-api,catalyst-api,catalyst-ui}:`) updated - VALIDATION-LOG entry: Pass 105 **Commit message:** `feat(consolidation): move Catalyst-Zero apps + CI from openova-private to public monorepo` ### Phase 2 — Cutover Catalyst-Zero to public-repo build **What:** Trigger first public-repo CI run, get `:` images into GHCR, roll the existing Contabo deployment to the new images. Catalyst-Zero is now built from the public repo. Delete legacy paths from `openova-private` (preserved in git history). **Outputs:** - GHCR images at `ghcr.io/openova-io/openova/{console,admin,marketplace,marketplace-api,catalyst-api,catalyst-ui}:` - Contabo k3s pods rolled to new image SHAs - `openova-private` cleaned of legacy paths - VALIDATION-LOG entry: Pass 106 **Commit message:** `infra(cutover): Catalyst-Zero now built from public repo` **Acceptance:** `kubectl describe pod` on each rolled pod shows `image: ghcr.io/openova-io/openova/...`. Console at `console.openova.io` still loads. Brief rolling-update window (<60s). ### Phase 3 — Sovereign-provisioning wizard **What:** Build the wizard at `core/console/src/pages/sovereign/` using the Vite scaffold. Replace the legacy 5-step marketplace flow's `AppsStep` with a unified marketplace card grid (driven by `bp-` Blueprint surface). Add Sovereign-provisioning-specific fields: - Domain: BYO (customer's own domain) **or** pool selection (default `omani.works` → user picks subdomain like `omantel`, `acme-bank`, etc.) - Hetzner Cloud API token (capture, store via ESO into OpenBao, never log) - Hetzner project ID - Hetzner region (dropdown of valid Hetzner regions; runtime parameter) - Sovereign owner email (becomes initial sovereign-admin) - Initial App selection (the unified marketplace grid) **Outputs:** - `openova/core/console/src/pages/sovereign/index.astro` + sub-pages for each wizard step - `openova/core/console/src/components/sovereign/{DomainStep,HetznerStep,AppsStep-unified,ReviewStep}.svelte` - The legacy bootstrap Vite scaffold at `openova/products/catalyst/bootstrap/ui/` is merged in and the directory deleted (its content is now part of `core/console/`) - VALIDATION-LOG entry: Pass 107 **Commit message:** `feat(console): sovereign-provisioning wizard at /sovereign with domain + Hetzner inputs + unified marketplace App selection` ### Phase 4 — Provisioner backend **What:** Build the wizard's backend at [`products/catalyst/bootstrap/api/`](../products/catalyst/bootstrap/api/) (the Go service deployed as `catalyst-api` in the `catalyst` namespace on Catalyst-Zero). Real backend that takes wizard input → calls OpenTofu → returns Sovereign provisioning state via SSE. Per [`INVIOLABLE-PRINCIPLES.md`](INVIOLABLE-PRINCIPLES.md) #3, **no cloud APIs are called from Go directly** — OpenTofu owns Phase 0, Crossplane owns day-2, and Hetzner client code is reserved for read-only credential validation. **Outputs:** - [`products/catalyst/bootstrap/api/internal/provisioner/`](../products/catalyst/bootstrap/api/internal/provisioner/) — thin wrapper around `tofu` that writes `tofu.auto.tfvars.json` from validated wizard input, runs `tofu init && tofu plan && tofu apply -auto-approve`, streams stdout/stderr lines to the wizard via SSE - [`products/catalyst/bootstrap/api/internal/hetzner/`](../products/catalyst/bootstrap/api/internal/hetzner/) — read-only Hetzner client for credential validation (`POST /api/v1/credentials/validate`); never used to mutate cloud state - [`products/catalyst/bootstrap/api/internal/pdm/`](../products/catalyst/bootstrap/api/internal/pdm/) — PDM client (`/v1/reserve`, `/v1/commit`, `/v1/validate`) for pool-subdomain allocation and registrar-token validation - [`products/catalyst/bootstrap/api/internal/dynadot/`](../products/catalyst/bootstrap/api/internal/dynadot/) — Dynadot client (used as one registrar adapter inside PDM's adapter set, not for direct DNS writes from this service) - [`products/catalyst/bootstrap/api/internal/handler/`](../products/catalyst/bootstrap/api/internal/handler/) — REST handlers including `POST /api/v1/deployments`, `GET /api/v1/deployments/{id}/logs` (SSE), `POST /api/v1/deployments/{id}/phases/{phase}/retry`, `POST /api/v1/credentials/validate`, `POST /api/v1/subdomains/check`, `GET /api/v1/registrars` - [`infra/hetzner/main.tf`](../infra/hetzner/main.tf) — OpenTofu module (network, firewall, ssh-key, control-plane + worker servers, load balancer) - VALIDATION-LOG entry: Pass 108 **Commit message:** `feat(provisioner): real Hetzner Sovereign provisioning end-to-end` ### Phase 5 — Bootstrap kit Helm charts (G2 quality) **What:** Real Catalyst-curated wrapper Helm charts at `platform//chart/` for every bootstrap-kit component. Each chart wraps upstream OSS with Catalyst-specific values, includes a `blueprint.yaml` per the unified Blueprint contract from `BLUEPRINT-AUTHORING.md` §1, publishes a `bp-:` OCI artifact via CI fan-out. **Components (in dependency order):** 1. `platform/cilium/chart/` (CNI must come first) 2. `platform/cert-manager/chart/` 3. `platform/flux/chart/` (host-level) 4. `platform/crossplane/chart/` 5. `platform/sealed-secrets/chart/` (transient bootstrap-only) 6. `platform/spire/chart/` (the `platform/spire/` folder may need to be added — workload identity) 7. `platform/nats-jetstream/chart/` (the `platform/nats-jetstream/` folder may need to be added) 8. `platform/openbao/chart/` 9. `platform/keycloak/chart/` 10. `platform/gitea/chart/` 11. `products/catalyst/chart/` — the umbrella `bp-catalyst-platform` **Outputs:** - 11 directories with `Chart.yaml`, `values.yaml`, `templates/`, `blueprint.yaml`, optional `compositions/`, `policies/`, `overlays/` - 11 entries in `openova/.github/workflows/blueprint-release.yaml` (path-matrix CI fan-out) - 11 OCI artifacts published at `ghcr.io/openova-io/bp-:` after first CI run - One commit per chart (11 commits) — incremental review possible - VALIDATION-LOG entries: Pass 109 through Pass 119 **Commit messages:** `feat(bp-): G2 Catalyst-curated chart for per BLUEPRINT-AUTHORING contract` ### Phase 6 — DNS architecture: PowerDNS authoritative + PDM + registrar adapters **What:** The DNS architecture has two layers. **Authoritative DNS** lives on bp-powerdns (#167) — every Sovereign zone (pool: `omantel.omani.works`, BYO: `acme.bank.com`) gets its own PowerDNS zone with DNSSEC + lua-records. **Allocation + registrar control** lives on the pool-domain-manager service (#163), which exposes registrar adapters (#170) for byo-api flow: - **Pool subdomains** (e.g. `.omani.works`, `.openova.io`): PDM `/v1/reserve` checks availability, `/v1/commit` creates the per-Sovereign PowerDNS zone, writes the canonical 6-record set, and updates the parent zone's NS delegation via the OpenOva Dynadot registrar adapter. - **BYO with manual NS-flip** (`byo-manual`): wizard surfaces the OpenOva NS list; customer pastes them into their own registrar UI; catalyst-api polls until propagation; PDM `/v1/commit` then writes the canonical record set into the new PowerDNS zone (no parent-zone change from OpenOva). - **BYO with API NS-flip** (`byo-api`): customer picks their registrar from the supported list (Cloudflare, Namecheap, GoDaddy, OVH, Dynadot — #170), pastes a token; PDM `/v1/validate` confirms scope read-only; on commit, the matching registrar adapter flips the NS records to OpenOva's NS set. **Outputs:** - [`core/pool-domain-manager/`](../core/pool-domain-manager/) — Go service deployed at `pool-domain-manager` in `openova-system`, CNPG-backed `pdm-pg`. Modules: `internal/allocator`, `internal/pdns`, `internal/registrar`, `internal/dynadot`, `internal/reserved`, `internal/store`. CI: [`.github/workflows/pool-domain-manager-build.yaml`](../.github/workflows/pool-domain-manager-build.yaml). - [`platform/crossplane/compositions/composition-pool-allocation.yaml`](../platform/crossplane/compositions/composition-pool-allocation.yaml) + matching XRD — declarative Crossplane wrapper around PDM `/v1/reserve` so Sovereign provisioning runs through the canonical IaC path. - [`platform/powerdns/`](../platform/powerdns/) — bp-powerdns wrapper chart (Chart.yaml, values.yaml, blueprint.yaml, templates) with DNSSEC + lua-records on by default, dnsdist companion for rate-limiting. - VALIDATION-LOG entry: Pass 120 (component-count refresh + PDM landing). **Commit message:** `feat(dns): bp-powerdns + pool-domain-manager + registrar adapters for pool/byo flows` ### Phase 7 — Franchise model docs + voucher propagation **What:** Read existing voucher implementation in admin app. Write `docs/FRANCHISE-MODEL.md` documenting it as canonical. Ensure the new Sovereign at `omantel.omani.works` has its own admin surface (the same admin app, deployed inside the Sovereign) where omantel-admin can issue vouchers to omantel's tenants. Update `GLOSSARY.md` with `Voucher` and `Franchisee` definitions if not already present. **Outputs:** - `openova/docs/FRANCHISE-MODEL.md` — canonical doc - Updates to `GLOSSARY.md` if needed - Updates to `BUSINESS-STRATEGY.md` revenue model if needed - VALIDATION-LOG entry: Pass 121 **Commit message:** `docs(franchise): canonical franchise model + voucher propagation, sourced from existing admin impl` ### Phase 8 — End-to-end provisioning (live demo / DoD) **What:** From browser at `console.openova.io/sovereign`: 1. User logs in (Keycloak SSO) 2. Picks "New Sovereign" 3. Pastes Hetzner Cloud API token + project ID, picks region (any — runtime parameter) 4. Picks domain: pool → `omani.works` → user types `omantel` (creates `omantel.omani.works`) 5. Picks initial Apps (unified marketplace selection) 6. Click Provision 7. Watches SSE-driven progress for ~10 minutes 8. Provisioning completes; new Sovereign at `omantel.omani.works` is reachable 9. omantel-admin (initial sovereign-admin) logs into `console.omantel.omani.works` 10. omantel-admin issues 1 voucher 11. A fictional customer redeems the voucher at `omantel.omani.works/redeem?code=...` 12. Customer's Organization + Environment + first App is created on omantel.omani.works 13. Customer reaches their App's URL **Acceptance:** every step above works without intervention. No mocks, no manual steps beyond the browser clicks. **Outputs:** - VALIDATION-LOG entry: Pass 122 — DoD documented with screenshots / kubectl evidence - Optional: `docs/DEMO-RUNBOOK.md` for repeatability --- ## 5. What this plan does NOT change - The unified Application = Gitea Repo model (Pass 103) is preserved everywhere. The franchised Sovereign at omantel.omani.works will use the same model — one Gitea Org per Catalyst Organization, one Gitea Repo per Application. - The 5 conventional Gitea Orgs convention (`catalog`, `catalog-sovereign`, `` per Catalyst Organization, `system`) applies to the new Sovereign exactly as it does to Catalyst-Zero. - The component-count anchor (Pass 104 set 53; Pass 105 raised it to 56 with spire + nats-jetstream + sealed-secrets) holds. SeaweedFS unified S3 encapsulation, Guacamole in bp-relay, OpenBao independent-Raft per region — all preserved. - The audit procedure stays on-demand (no scheduled loops). The `audit-catalyst-docs` skill is the only validation entry point. --- ## 6. References - `docs/ARCHITECTURE.md` — target architecture (the design Catalyst-Zero is being aligned to) - `docs/SOVEREIGN-PROVISIONING.md` §3 Phase 0 — bootstrap kit dependency order (canonical reference for Phase 5 of this plan) - `docs/BLUEPRINT-AUTHORING.md` §1 — unified Blueprint shape (the contract Phase 5 charts must satisfy) - `docs/IMPLEMENTATION-STATUS.md` — gets updated incrementally as each phase lands (📐 → 🚧 → ✅) - `docs/AUDIT-PROCEDURE.md` — how to validate after each phase - `docs/VALIDATION-LOG.md` Pass 1–104 — historical record; Pass 105+ tracks this plan's execution --- *Part of [OpenOva](https://openova.io)*