04559e5c37
Reconcile Pass 1 — first holistic LLM-driven reconciliation pass per ~/.claude/skills/reconcile-catalyst-docs/SKILL.md. Skill triggered after the post-Group-M architectural batch (#161, #162, #163, #167, #168, #169, #170, #171, #173, #174, #175). Live ground truth verified against kubectl + ls platform/ + git log + GHCR + componentGroups.ts. Drift categories fixed: - A. Numerical: bp-powerdns 1.0.5 → 1.0.6; component-logos 63 → 62 (powerdns SVG missing, tracked under #173); bootstrap kit 11 → 12 with bp-powerdns added per #167. - B. Service: pool-domain-manager + 5 registrar adapters (Cloudflare/Namecheap/GoDaddy/OVH/Dynadot, #170) added to IMPLEMENTATION-STATUS, ARCHITECTURE, PLATFORM-TECH-STACK, GLOSSARY, and PROVISIONING-PLAN; bp-powerdns added to ARCHITECTURE bootstrap kit + Catalyst-on-Catalyst dependency tree. - C. Architectural: SOVEREIGN-PROVISIONING §3 + DEMO-RUNBOOK Step 4 + ORCHESTRATOR-STATE Step 6 rewritten from Dynadot-direct DNS writes to PowerDNS authoritative + PDM /v1/commit + registrar-adapter NS-flip; PROVISIONING-PLAN Phase 4 paths corrected to products/catalyst/bootstrap/api/ (per INVIOLABLE-PRINCIPLES #3 the Go provisioner does NOT call cloud APIs); Phase 6 retitled and rewritten for the new DNS architecture. - D. Process: RUNBOOK-PROVISIONING §2 wizard-step table + DEMO-RUNBOOK Step 2 wizard-step table updated to canonical 7-step ordering (Org → Domain → Topology → Provider → Credentials → Components → Review per WIZARD_STEPS in WizardLayout.tsx, post #169 + #174); the three-mode StepDomain (pool / byo-manual / byo-api per #169) and two-tab StepComponents (mandatory infra + apps per #161/#162/#175) now documented. - E. Cross-doc: Group G ✅ across PROVISIONING-PLAN + ORCHESTRATOR-STATE (superseded by #167+#163+#170, not by the original Dynadot-multi-domain plan); Group C ✅ in PROVISIONING-PLAN (Flux is reconciling from openova-public today); README Stack-at-a-glance DNS row expanded. - F. Stale terminology: 11-grep banned-terms scan clean — every k8gb residual is a legitimate "removed at #171, replaced by lua-records" reference. VALIDATION-LOG.md gains the Reconcile Pass 1 entry per skill spec. Reconcile-skill numbering is independent of the Audit-skill numbering (which continues at Pass 108+). Files: 13 docs + VALIDATION-LOG entry. Escalations: none.
512 lines
35 KiB
Markdown
512 lines
35 KiB
Markdown
# Catalyst Architecture
|
|
|
|
**Status:** Authoritative target architecture. **Updated:** 2026-04-29.
|
|
**Implementation:** Most of what this document describes is **design-stage** — see [`IMPLEMENTATION-STATUS.md`](IMPLEMENTATION-STATUS.md) for what exists in code today vs what is design. The DNS plane (bp-powerdns + pool-domain-manager + registrar adapters) is deployed today in `openova-system` on Catalyst-Zero.
|
|
|
|
This document describes the architecture of **Catalyst** — the OpenOva platform. For terminology, defer to [`GLOSSARY.md`](GLOSSARY.md). For naming, defer to [`NAMING-CONVENTION.md`](NAMING-CONVENTION.md). For current code state, defer to [`IMPLEMENTATION-STATUS.md`](IMPLEMENTATION-STATUS.md).
|
|
|
|
---
|
|
|
|
## 1. The platform in one paragraph
|
|
|
|
Catalyst is a self-sufficient Kubernetes-native control plane published as signed OCI Blueprints. A single deployed Catalyst is called a **Sovereign**. Inside a Sovereign, **Organizations** are the multi-tenancy unit. An Organization has **Environments** (`{org}-prod`, `{org}-dev`, etc.) where users install **Applications** from **Blueprints**. **Each Application is its own Gitea repo** (one App = one repo, uniformly across SME and corporate scale); branches `develop`/`staging`/`main` map to dev/stg/prod environments. One or more vclusters per Environment run lightweight Flux watching the appropriate branch across the Org's Application repos. Every state change flows through NATS JetStream, projects into per-Environment KV via the **projector** service, and reaches the console via SSE — so every UI surface sees the same picture, derived from Git (write side) and Kubernetes (runtime side) without fragmenting. Crossplane handles all non-Kubernetes resources. OpenBao + ESO + SPIRE handles secrets and workload identity. Keycloak handles user identity. **Same code runs in every Sovereign — whether it's run by us, by Omantel, or by Bank Dhofar.**
|
|
|
|
---
|
|
|
|
## 2. Two scales, one architecture
|
|
|
|
The model serves two distinct customer shapes through the **same code**:
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ SME-style Sovereign (e.g. omantel) │
|
|
│ │
|
|
│ Many small Organizations, mostly single-Environment │
|
|
│ Each Org gets its own minimal Keycloak (no HA) │
|
|
│ Self-service marketplace, next-next-next install │
|
|
│ Sovereign-admins are the SaaS provider's cloud team │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ Corporate-style Sovereign (e.g. bankdhofar) │
|
|
│ │
|
|
│ Few internal Organizations (core-banking, digital-channels…) │
|
|
│ One Sovereign-wide Keycloak (federates to corporate Azure AD) │
|
|
│ Rich governance: EnvironmentPolicy, soak gates, approvers │
|
|
│ Sovereign-admins are the bank's platform team │
|
|
│ Multi-region default; multi-Environment per Org default │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
The **only** runtime configuration difference is set at provisioning time:
|
|
|
|
```yaml
|
|
keycloakTopology: per-organization # SME default
|
|
# or
|
|
keycloakTopology: shared-sovereign # Corporate default
|
|
```
|
|
|
|
Everything else is identical in code.
|
|
|
|
---
|
|
|
|
## 3. Topology
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ Sovereign: omantel │
|
|
│ │
|
|
│ Management host cluster: hz-nbg-mgt-prod │
|
|
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Catalyst control plane (in catalyst-* namespaces) │ │
|
|
│ │ console marketplace admin catalog-svc projector │ │
|
|
│ │ provisioning environment-controller blueprint-controller │ │
|
|
│ │ billing │ │
|
|
│ │ gitea nats-jetstream openbao keycloak spire-server │ │
|
|
│ │ observability (Grafana stack) │ │
|
|
│ └────────────────────────────────────────────────────────────────────┘ │
|
|
│ Plus per-host-cluster infrastructure (Cilium, Flux, Crossplane, │
|
|
│ cert-manager, External-Secrets, Kyverno, Harbor, Reloader, Trivy, │
|
|
│ Falco, Sigstore, Syft+Grype, VPA, KEDA, External-DNS, PowerDNS, Coraza, │
|
|
│ SeaweedFS, Velero, failover-controller) — see PLATFORM-TECH-STACK §3. │
|
|
│ │
|
|
│ Workload host clusters: hz-fsn-rtz-prod, hz-hel-rtz-prod │
|
|
│ ┌──────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Per-Org vcluster (named {org}): │ │
|
|
│ │ muscatpharmacy acme-shop blue-pharmacy … │ │
|
|
│ │ each runs its own lightweight Flux pointed at the Environment │ │
|
|
│ │ Gitea repo │ │
|
|
│ └──────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ DMZ host clusters: hz-fsn-dmz-prod, hz-hel-dmz-prod │
|
|
│ Cilium Gateway, WAF (Coraza), PowerDNS authoritative + lua-records, │
|
|
│ dnsdist rate-limit, WireGuard endpoints │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
↕
|
|
Gitea (in management cluster) — 5 conventional Gitea Orgs
|
|
──────────────────────────────────────────────────────────
|
|
catalog/ ← public Blueprint mirror (read-only)
|
|
catalog-sovereign/ ← Sovereign-owner-curated private Blueprints (optional)
|
|
acme-pharmacy/ ← one Gitea Org per Catalyst Organization
|
|
├── shared-blueprints ← Org-private Blueprint authoring
|
|
├── store-frontend ← one Gitea Repo per Application
|
|
├── pharmacy-mail
|
|
├── consult-room
|
|
└── appointments
|
|
(branches develop/staging/main map
|
|
to dev/stg/prod environments)
|
|
kestrel-rx/ ← another Catalyst Organization
|
|
├── shared-blueprints
|
|
└── ...
|
|
system/ ← sovereign-admin scope
|
|
├── catalyst-config (CRs: Sovereign, Organization,
|
|
│ Environment, EnvironmentPolicy)
|
|
├── policy-bundle (Kyverno, Falco, RE Scorecard)
|
|
└── runbooks (auto-remediation)
|
|
...
|
|
```
|
|
|
|
**Sovereign self-sufficiency**: once a Sovereign is provisioned, it has its own Gitea, its own JetStream, its own OpenBao, its own Keycloak, its own Crossplane. It does not depend on any other Sovereign at runtime. OpenOva's `openova` Sovereign is in the picture only as the publisher of public Blueprints — and even those are mirrored locally, so the Sovereign keeps working if openova.io disappears.
|
|
|
|
---
|
|
|
|
## 4. Write side: Git → Flux → Kubernetes (+ Crossplane)
|
|
|
|
```
|
|
Console UI REST/GraphQL API
|
|
│ │
|
|
│ (Git push from any of these │
|
|
│ bypasses provisioning and goes │
|
|
│ straight to the App's repo; │
|
|
│ webhook + projector still fire) │
|
|
▼ ▼
|
|
┌──────────────────────────────────────────────────────────┐
|
|
│ provisioning service │
|
|
│ - validates configSchema against Blueprint │
|
|
│ - resolves dependency graph │
|
|
│ - creates one Gitea repo per Application │
|
|
│ - commits initial manifests to develop/staging/main │
|
|
└──────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────┐
|
|
│ Application Gitea repo: {org}/{app} │
|
|
│ (FQDN form per NAMING §11.2 — one repo per Application) │
|
|
│ ──────────────────────────────────────────────────────── │
|
|
│ branches: develop → dev env, staging → stg, main → prod │
|
|
│ ──────────────────────────────────────────────────────── │
|
|
│ kustomization.yaml ← root Flux Kustomization │
|
|
│ values.yaml ← base values │
|
|
│ overlays/ ← per-env overlays │
|
|
│ dev/values.yaml │
|
|
│ stg/values.yaml │
|
|
│ prod/values.yaml │
|
|
│ secrets/ ← ExternalSecret refs (no plain) │
|
|
│ CODEOWNERS ← team / approver list │
|
|
│ │
|
|
│ EnvironmentPolicy lives separately in the system Gitea │
|
|
│ Org: system/catalyst-config/policies/{org}-{env}-policy │
|
|
└──────────────────────────────────────────────────────────┘
|
|
│
|
|
▼ (Gitea webhook → projector → annotate)
|
|
┌──────────────────────────────────────────────────────────┐
|
|
│ Flux in vcluster {org} │
|
|
│ - N GitRepository sources, one per App repo │
|
|
│ - each watching the env-appropriate branch │
|
|
│ - kustomize-controller applies to per-App namespaces │
|
|
│ - helm-controller renders Helm-based Blueprints │
|
|
└──────────────────────────────────────────────────────────┘
|
|
│
|
|
┌──────────────────┴────────────────────┐
|
|
▼ ▼
|
|
K8s Application workloads Crossplane Claims
|
|
(Deployments, Services, (Hetzner servers, DNS records,
|
|
Pods, Secrets via ESO) S3 buckets, Cloudflare Workers)
|
|
│
|
|
▼
|
|
Crossplane Compositions
|
|
fan out to provider APIs
|
|
```
|
|
|
|
**Crossplane is the only IaC.** Users never write Crossplane Compositions in their Application configs. Blueprint authors do — when a Blueprint declares "needs an external Postgres," that becomes a Crossplane Claim under the hood. Advanced users (corporate sovereign-admins, OpenOva engineers) can author and contribute Crossplane Compositions as Blueprints. End users see "needs a database, pick existing or new" in the UI.
|
|
|
|
---
|
|
|
|
## 5. Read side: CQRS via JetStream → projector → console
|
|
|
|
```
|
|
┌────────────────────┐ ┌────────────────────┐ ┌──────────────────┐
|
|
│ k8s informers │ │ Flux events │ │ Gitea webhooks │
|
|
│ (one per vcluster) │ │ (per vcluster) │ │ (per Sovereign) │
|
|
└─────────┬──────────┘ └─────────┬──────────┘ └─────────┬────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌────────────────────────────────────────────────────────────────────┐
|
|
│ NATS JetStream │
|
|
│ Account isolation: one NATS Account per Organization. │
|
|
│ Subject prefix scoped per Environment (where <env> = {org}-{env_type}): │
|
|
│ ws.<env>.k8s.<obj-kind>.<ns>.<name> │
|
|
│ ws.<env>.flux.<kustomization> │
|
|
│ ws.<env>.git.<commit-hash> │
|
|
│ ws.<env>.crossplane.<resource> │
|
|
└────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼ durable consumer per env partition
|
|
┌────────────────────────────────────────────────────────────────────┐
|
|
│ projector │
|
|
│ - consumes events │
|
|
│ - rebuilds per-object state │
|
|
│ - writes to JetStream KV: ws-<env>-state/<kind>/<name> │
|
|
│ - fans out SSE to subscribed console clients │
|
|
│ - authorizes by JWT claim {environment, org, role} │
|
|
│ - serves REST/GraphQL snapshot read API │
|
|
└────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌────────────────────┐
|
|
│ Catalyst console │
|
|
└────────────────────┘
|
|
```
|
|
|
|
**One spine (JetStream), one read model (JetStream KV), one consumer (projector), one stream (SSE).**
|
|
|
|
The console **never talks to k8s API or Git directly.** This is the architectural lock that prevents the "App says installed in one tab, failed in another tab" class of bug. Both tabs read the same JetStream KV snapshot served by the same projector replica.
|
|
|
|
JetStream replaces the older Redpanda + Valkey pairing in the control plane: NATS is Apache 2.0 (no BSL risk), has native KV (fewer moving parts), and native multi-tenant Accounts (cleaner per-Org isolation). Application-layer event needs (e.g. TalentMesh's voice pipeline) remain free to choose Redpanda, Kafka, NATS, or anything else — that's an Application-level decision, not a control-plane one.
|
|
|
|
---
|
|
|
|
## 6. Identity and secrets
|
|
|
|
Two separate identity systems for two separate purposes:
|
|
|
|
| Subject | System | Lifetime | Purpose |
|
|
|---|---|---|---|
|
|
| **Workloads** (every Pod) | SPIFFE/SPIRE → SVID (mTLS cert) | 5 min, auto-rotated | Pod-to-Pod auth, Pod-to-OpenBao auth, Pod-to-NATS auth |
|
|
| **Users** (every human) | Keycloak → JWT | 15 min access / 30 day refresh | UI auth, API auth |
|
|
|
|
**Secrets** flow:
|
|
|
|
```
|
|
OpenBao (per-region, independent Raft cluster)
|
|
│
|
|
│ (workload requests via SPIFFE SVID)
|
|
▼
|
|
ESO ExternalSecret CR (in Git, references OpenBao path)
|
|
│
|
|
▼
|
|
K8s Secret (versioned, reloader watches for hash change)
|
|
│
|
|
▼
|
|
Pod (env var or mounted file)
|
|
```
|
|
|
|
**Multi-region**: each region runs its **own** 3-node Raft OpenBao cluster. **No stretched cluster.** Cross-region async perf replication for read availability and DR. A region failure does not require any other region to do anything.
|
|
|
|
**Keycloak** topology depends on Sovereign type:
|
|
- **SME-style** (`per-organization`): minimal single-replica Keycloak per Org, sized for hundreds of users. Embedded H2 or sqlite. Each Org's Keycloak is independent; failure does not affect other Orgs.
|
|
- **Corporate-style** (`shared-sovereign`): one HA Keycloak for the entire Sovereign, federating to the parent corporation's identity provider (Azure AD, Okta).
|
|
|
|
See [`SECURITY.md`](SECURITY.md) for full credential rotation and identity flow.
|
|
|
|
---
|
|
|
|
## 7. The user-facing surfaces
|
|
|
|
Three first-class surfaces. **No fourth.**
|
|
|
|
### 7.1 UI (the Catalyst console)
|
|
|
|
Default. Most users never leave it. Three depths the user can switch between:
|
|
|
|
- **Form view** — one Application page, fields driven by `configSchema`. Default for SME.
|
|
- **Advanced view** — same page with topology, secrets, observability, history, manifest tabs. Default for corporate.
|
|
- **IaC editor view** — in-browser Monaco editing the Application's Gitea repo with Blueprint-schema validation, live diff, commit-on-save. Toggle, not modal.
|
|
|
|
All three commit to the same Application Gitea repo (one repo per App, branches `develop`/`staging`/`main` mapping to dev/stg/prod). The **Application card** is the user's primary handle — see [`PERSONAS-AND-JOURNEYS.md`](PERSONAS-AND-JOURNEYS.md).
|
|
|
|
### 7.2 Git
|
|
|
|
Direct push or pull-request to the Application's Gitea repo (one repo per App), or to `shared-blueprints` for Org-private Blueprints, or to `catalog-sovereign` for Sovereign-curated private Blueprints.
|
|
|
|
Identical write semantics as the UI. Both end up as commits on the App's repo branches. EnvironmentPolicy (PR approvals, soak, change windows) applies regardless of the surface — the policy CR lives in the `system` Gitea Org and is matched by Org+env_type at projector enforcement time.
|
|
|
|
### 7.3 API (REST + GraphQL)
|
|
|
|
For **integrations**, not for primary IaC authoring.
|
|
|
|
Use cases:
|
|
- A bank's existing Backstage portal queries Catalyst to show Environments and Applications.
|
|
- A change-management tool (ServiceNow, JIRA) triggers Application installs based on a ticket.
|
|
- A monitoring/auditing tool exports state for compliance reports.
|
|
|
|
The API exposes the same operations the console performs. It is **not** an IaC authoring layer in the Terraform-cloud sense. We do not ship a Terraform provider, a Pulumi SDK, or any other "declare desired state through us" surface — the Application Gitea repo is that surface.
|
|
|
|
### 7.4 What's deliberately NOT a surface
|
|
|
|
- `kubectl` — useful for debugging inside one's own vcluster; never a configuration mechanism.
|
|
- A standalone CLI for production changes — Catalyst may expose a small read-only debug CLI in the future; not authoritative for installs/promotions.
|
|
- Terraform / Pulumi — Crossplane covers non-K8s; it is platform plumbing, not user-facing.
|
|
|
|
---
|
|
|
|
## 8. Promotion across Environments
|
|
|
|
Promotion is **not** a separate engine or a chain object. Because each Application is a single Gitea repo with branches mapping to env_types, promotion is the simple act of opening a PR from the lower-env branch to the higher-env branch (e.g. `staging` → `main` to promote stg → prod), plus a policy gating the destination branch.
|
|
|
|
```
|
|
Blueprint detail page in console:
|
|
|
|
bp-wordpress @ available 1.4.0
|
|
─────────────────────────────────────────────────
|
|
Applications using this Blueprint in your Org (4)
|
|
|
|
Application Environment Version Status
|
|
──────────────────────────────────────────────────────
|
|
marketing-site acme-dev 1.4.0 ● Running [Open]
|
|
marketing-site acme-stg 1.3.0 ● Running [Open]
|
|
marketing-site acme-prod 1.2.0 ● Running [Open]
|
|
blog acme-prod 1.2.0 ● Running [Open]
|
|
|
|
[ + Install in another Environment ]
|
|
[ Compare versions ]
|
|
```
|
|
|
|
From `marketing-site` in `acme-stg`, the user clicks "Promote to acme-prod". Catalyst opens a Gitea PR from the `staging` branch to the `main` branch in the **same** `marketing-site` Application repo. The destination Environment's `EnvironmentPolicy` CR (in the `system` Gitea Org, matched by `appliesTo.environments: [acme-prod]`) supplies the approvers, soak duration, change window, and RE-score gate that apply to the PR. On merge, the Flux instance in the `acme-prod` vcluster (which watches the `main` branch) reconciles. Done.
|
|
|
|
```yaml
|
|
# Lives at: system/catalyst-config/policies/acme-prod-policy.yaml
|
|
# (in the Sovereign-admin's `system` Gitea Org)
|
|
apiVersion: catalyst.openova.io/v1alpha1
|
|
kind: EnvironmentPolicy
|
|
metadata:
|
|
name: acme-prod-policy
|
|
namespace: catalyst-system
|
|
spec:
|
|
appliesTo:
|
|
environments: [acme-prod]
|
|
rules:
|
|
- kind: pr-required
|
|
approvers: [team-platform, team-security]
|
|
minApprovals: 2
|
|
- kind: re-score-gate
|
|
minScore: 80
|
|
severity: blocking
|
|
- kind: soak
|
|
sourceEnvironment: acme-stg
|
|
duration: 72h
|
|
- kind: change-window
|
|
cron: "0 14 * * 2,4" # Tue/Thu 14:00
|
|
duration: 2h
|
|
```
|
|
|
|
The policy lives in the Sovereign-admin's `system` Gitea Org and applies uniformly to every Application in the destination Environment, regardless of who initiated the change (UI, Git, API). The same CR shape is used for SME and corporate Sovereigns — only the field values differ (e.g. `minApprovals: 1` for SMEs with a single org-admin, `minApprovals: 2-3` for corporate teams).
|
|
|
|
---
|
|
|
|
## 9. Multi-Application linkage (the dependency tree)
|
|
|
|
A Blueprint can declare dependencies on other Blueprints:
|
|
|
|
```yaml
|
|
apiVersion: catalyst.openova.io/v1alpha1
|
|
kind: Blueprint
|
|
metadata:
|
|
name: bp-wordpress
|
|
version: 1.3.0
|
|
spec:
|
|
configSchema: …
|
|
depends:
|
|
- blueprint: bp-postgres
|
|
version: ^1.4
|
|
alias: db
|
|
when: "{{ .config.postgres.mode == 'embedded' }}"
|
|
values:
|
|
databases: ["{{ .application.name }}"]
|
|
```
|
|
|
|
When a User installs `marketing-site` from `bp-wordpress`:
|
|
|
|
1. **Catalog-svc** flattens the dependency tree.
|
|
2. **Console** asks: "WordPress requires Postgres. Use an existing Postgres Application or create a new dedicated one?" — querying projector for existing `bp-postgres` Applications in this Org.
|
|
3. **Provisioning service** composes an InstallPlan: either one Application (`marketing-site`) referencing an existing postgres Application, or two Applications (`marketing-site` + `marketing-site-postgres`).
|
|
4. **Gitea creates one or two repos** under the Org's Gitea Org (e.g. `acme-pharmacy/marketing-site` + `acme-pharmacy/marketing-site-postgres`), each with `develop`/`staging`/`main` branches and initial manifests.
|
|
5. **Flux** in the Org's vcluster picks up new `GitRepository` sources and reconciles in dependency order via cross-repo `Kustomization.dependsOn` edges.
|
|
|
|
Every Application is its own Gitea repo and its own Flux Kustomization. The dependency graph is materialized as `dependsOn` edges between Kustomizations (which are namespaced CRs in the vcluster, regardless of which Gitea repo each Kustomization was sourced from), computed at install time from the Blueprint's `depends` declaration.
|
|
|
|
---
|
|
|
|
## 10. Provisioning a Sovereign
|
|
|
|
```
|
|
Phase 0 Bootstrap (one-shot, runs from catalyst-provisioner.openova.io)
|
|
─────────────────────────────────────────────────────────────────────
|
|
1. OpenTofu provisions: VPC, host nodes, load balancers, object storage
|
|
on the target cloud provider (Hetzner / AWS / etc.). DNS is NOT
|
|
written here — it flows through the PowerDNS / pool-domain-manager
|
|
plane (see step 3 below + docs/PLATFORM-POWERDNS.md).
|
|
2. Bootstrap kit installs in order:
|
|
a. Cilium (CNI + Gateway API) ← network must come first
|
|
b. cert-manager ← TLS for everything below
|
|
c. Flux (host-level) ← GitOps engine
|
|
d. Crossplane + provider config ← cloud resource control plane
|
|
e. Sealed Secrets (transient, only for bootstrap secrets)
|
|
f. SPIRE server + agent ← workload identity
|
|
g. NATS JetStream cluster (3 nodes)
|
|
h. OpenBao cluster (3 nodes, region-local Raft)
|
|
i. Keycloak (per `keycloakTopology` choice)
|
|
j. Gitea (with public Blueprint mirror seeded)
|
|
k. PowerDNS (bp-powerdns) + dnsdist ← per-Sovereign authoritative
|
|
DNS zone, DNSSEC, lua-records
|
|
l. Catalyst control plane (umbrella Blueprint: bp-catalyst-platform)
|
|
3. Pool-domain-manager (running on the OpenOva-run Catalyst-Zero, NOT
|
|
on the new Sovereign) calls `/v1/commit`: creates the per-Sovereign
|
|
PowerDNS zone, writes the canonical 6-record set via the PowerDNS
|
|
REST API, and updates the parent-zone NS delegation via the matching
|
|
registrar adapter (Cloudflare / Namecheap / GoDaddy / OVH / Dynadot)
|
|
— see docs/SOVEREIGN-PROVISIONING.md §3 + docs/PLATFORM-POWERDNS.md.
|
|
|
|
Phase 1 Hand-off (~5 minutes after Phase 0 starts)
|
|
─────────────────────────────────────────────────────────────────────
|
|
Crossplane in the new Sovereign adopts management of further
|
|
infrastructure. OpenTofu state is archived. Bootstrap kit is no
|
|
longer in the runtime path.
|
|
|
|
Phase 2 Day-1 setup
|
|
─────────────────────────────────────────────────────────────────────
|
|
First sovereign-admin logs into the console; configures cert-manager
|
|
issuers, backup destinations, optional federation; onboards the first
|
|
Organization and creates its first Environment.
|
|
|
|
Phase 3 Steady-state operation
|
|
─────────────────────────────────────────────────────────────────────
|
|
Catalyst is fully autonomous. catalyst-provisioner.openova.io remains
|
|
online indefinitely as the entry point for future Sovereign
|
|
provisioning runs — but the existing Sovereign no longer depends on
|
|
it at runtime.
|
|
```
|
|
|
|
See [`SOVEREIGN-PROVISIONING.md`](SOVEREIGN-PROVISIONING.md) for the full procedure (this is the canonical reference for phase semantics).
|
|
|
|
---
|
|
|
|
## 11. Catalyst-on-Catalyst (dogfooding)
|
|
|
|
Every component in the Catalyst control plane is itself published as a Blueprint:
|
|
|
|
```
|
|
bp-catalyst-platform ← umbrella
|
|
├── depends: bp-catalyst-console
|
|
├── depends: bp-catalyst-marketplace
|
|
├── depends: bp-catalyst-admin
|
|
├── depends: bp-catalyst-catalog-svc
|
|
├── depends: bp-catalyst-projector
|
|
├── depends: bp-catalyst-provisioning
|
|
├── depends: bp-catalyst-environment-controller
|
|
├── depends: bp-catalyst-blueprint-controller
|
|
├── depends: bp-catalyst-billing
|
|
├── depends: bp-catalyst-gitea ← per-Sovereign Git server
|
|
├── depends: bp-catalyst-nats-jetstream ← event spine + KV
|
|
├── depends: bp-catalyst-openbao ← secret backend
|
|
├── depends: bp-catalyst-keycloak ← user identity
|
|
├── depends: bp-catalyst-spire ← workload identity
|
|
└── depends: bp-catalyst-observability ← OTel + Grafana stack
|
|
```
|
|
|
|
(Cilium, Flux, Crossplane, Cert-manager, Kyverno, Harbor, External-Secrets, Reloader, Falco, Sigstore, Syft+Grype, **PowerDNS** are **per-host-cluster infrastructure**, not Catalyst control-plane components — see [`PLATFORM-TECH-STACK.md`](PLATFORM-TECH-STACK.md) §1. They get installed once per host cluster, before Catalyst itself. The pool-domain-manager (PDM) is deployed on the OpenOva-run Catalyst-Zero only — it is part of the bootstrap surface, not the per-Sovereign control plane.)
|
|
|
|
Installing `bp-catalyst-platform` once gives you a working Sovereign. Same Blueprint installed on Hetzner = the openova Sovereign. Same Blueprint installed on AWS for a bank = that bank's Sovereign. Same Blueprint installed on Hetzner for a telco = the omantel Sovereign. **One artifact. Zero divergence.**
|
|
|
|
OpenOva's own customer Applications (Cortex, Fingate, Fabric, Relay, Specter, Axon) are similarly composite Blueprints that run **on top of** Catalyst — they are Applications inside the `openova-public` Environment of the openova Sovereign.
|
|
|
|
---
|
|
|
|
## 12. State-of-the-art principles applied
|
|
|
|
| Pattern | Where it lives in this design |
|
|
|---|---|
|
|
| **CQRS** | Write side: Git → Flux → K8s. Read side: catalog-svc + projector. |
|
|
| **GitOps as truth** | Every state change is a commit. Rollback = `git revert`. Audit = `git log`. |
|
|
| **Event sourcing** | NATS JetStream is the durable event log. Projector replays for recovery. |
|
|
| **CRD-driven control plane** | Sovereign, Organization, Environment, Application, Blueprint, EnvironmentPolicy, SecretPolicy, Runbook — all CRDs. Controllers reconcile. |
|
|
| **Multi-tenancy at OS layer** | vcluster per Organization per host cluster — isolated K8s API + control plane per Org. |
|
|
| **Crossplane for non-K8s** | All cloud-side resources via Compositions. Users never see Crossplane. |
|
|
| **OCI artifacts for software** | Blueprints are signed OCI manifests, cosigned, SBOMed. |
|
|
| **CloudEvents-shaped envelopes** | Standard event format on JetStream subjects. |
|
|
| **OpenTelemetry first-class** | All Catalyst services emit traces; every Blueprint inherits OTel by default. |
|
|
| **Policy as code** | Kyverno policies in Catalyst block out-of-policy commits and out-of-policy K8s resources. |
|
|
| **Supply chain security** | cosign signing, SLSA-3 build provenance, Syft+Grype SBOM, Trivy scans, Falco runtime. |
|
|
| **JSON Schema for config** | Console form is generated from Blueprint configSchema. No hand-written forms. |
|
|
| **Pull-based updates** | Each Sovereign mirrors the public Blueprint catalog on its own schedule. Air-gap-ready by construction. |
|
|
| **Workload identity** | SPIFFE/SPIRE SVIDs replace static service-account credentials end-to-end. |
|
|
| **Independent failure domains** | OpenBao Raft per region. vcluster per Org. Keycloak per Org (SME) or per Sovereign (corporate). |
|
|
|
|
---
|
|
|
|
## 13. Open Application Model influence
|
|
|
|
The Blueprint shape is influenced by — but not identical to — OAM:
|
|
|
|
| OAM term | Catalyst equivalent |
|
|
|---|---|
|
|
| Application | Blueprint with `card.category=composite` |
|
|
| Component | Blueprint (single-purpose) |
|
|
| Trait | Blueprint overlay (e.g. `overlays/small`, `overlays/medium`, `overlays/large`) |
|
|
| Scope | Environment + Placement |
|
|
|
|
We are not a strict OAM implementation. We borrow the layered composition idea but use Kubernetes-native primitives (Kustomize, Helm) rather than OAM-specific machinery — because Flux, Crossplane, and the K8s ecosystem are the runtime, and inventing a new layer adds no value.
|
|
|
|
---
|
|
|
|
## 14. Read further
|
|
|
|
- [`GLOSSARY.md`](GLOSSARY.md) — every term defined.
|
|
- [`NAMING-CONVENTION.md`](NAMING-CONVENTION.md) — every name's pattern.
|
|
- [`PERSONAS-AND-JOURNEYS.md`](PERSONAS-AND-JOURNEYS.md) — who uses each surface and how.
|
|
- [`SECURITY.md`](SECURITY.md) — identity, secrets, rotation in detail.
|
|
- [`SOVEREIGN-PROVISIONING.md`](SOVEREIGN-PROVISIONING.md) — bringing a Sovereign online.
|
|
- [`BLUEPRINT-AUTHORING.md`](BLUEPRINT-AUTHORING.md) — writing Blueprints (including Crossplane Compositions for advanced users).
|
|
- [`PLATFORM-TECH-STACK.md`](PLATFORM-TECH-STACK.md) — every component's role in Catalyst.
|
|
- [`SRE.md`](SRE.md) — operating a Sovereign.
|