1bebeba655
Cluster-A — qa-wp Application + every dependent fixture not reconciling
Root cause: chart 1.4.105 HR was Stalled (UpgradeFailed →
MissingRollbackTarget). On Helm upgrade the qa-fixtures Organization CR
was rejected at admission with:
Organization.orgs.openova.io "omantel-platform" is invalid:
spec.sovereignRef: Invalid value: "omantel": spec.sovereignRef in body
should match '^[a-z0-9](...)?(\.[a-z0-9](...)?)+$'
The Organization CRD requires sovereignRef as a FQDN (one or more
dot-separated DNS labels); the qa-fixtures default was the single-
segment placeholder "omantel". With the chart upgrade rejected the
Application + Environment + Blueprint + UserAccess + every other
qa-fixtures resource was absent on omantel — TC-065/068/100/204/262/263
all FAIL on missing qa-wp.
Fix:
- templates/qa-fixtures/organization-omantel-platform.yaml: resolution
chain qaFixtures.sovereignFQDN → global.sovereignFQDN → legacy
qaFixtures.sovereignRef (drop placeholder "omantel") → "omantel.biz"
- bootstrap-kit 13-bp-catalyst-platform.yaml: forward SOVEREIGN_FQDN
into qaFixtures.sovereignFQDN so a Sovereign install never has to
set it explicitly
- values.yaml: document the two seams (sovereignRef short-form for
UserAccess CRD, sovereignFQDN dotted-form for Organization CRD)
Cluster-A — POST /applications "blueprint":"bp-wordpress" returned 404
Root cause: the catalyst-api install handler resolves Blueprint →
chart bytes via the upstream catalyst-catalog only. Chart-shipped
Blueprint CRs (qa-fixtures.bp-qa-app, the new bp-wordpress) live in
the cluster apiserver but are invisible to the upstream catalog.
Per docs/INVIOLABLE-PRINCIPLES.md #1 (target-state, not MVP) the
chart-shipped Blueprint CR is a first-class catalog entry, not a
"stub for now".
Fix:
- new internal/handler/catalog_client_cluster_fallback.go — wraps
the upstream HTTP client; on ErrBlueprintNotFound falls back to
a dynamic-client lookup against blueprints.catalyst.openova.io
(v1 first, v1alpha1 on version-not-served), maps the CR to the
same CatalogBlueprint wire shape, populates Raw so the install
handler's spec.configSchema validation has the same view as the
upstream-served path
- cmd/api/main.go: NewChainedCatalogClient(upstream, homeDyn) where
homeDyn is rest.InClusterConfig() built dynamic.Interface
- mustHomeDynamicClient helper added next to mustHomeCoreClient
- templates/qa-fixtures/blueprint-bp-wordpress.yaml — alias-style
listed Blueprint CR pointing at the bp-qa-app chart bytes; once
the operator imports the production wordpress-tenant Blueprint
into the public catalog Gitea Org, the upstream resolver wins
because the chained client tries upstream first
cutover-driver ClusterRole already grants get/list/watch on
blueprints.catalyst.openova.io (PR #1052) — no RBAC change needed.
Cluster-A — applicationDefaultPrimaryRegion "fsn1" rejected at admission
Root cause: applications_wire_compat.go promoted simplified-shape
POSTs missing placement.regions to literal {"fsn1"}. The Application
CRD validates regions[*] against `^[a-z]+-[a-z]+-[a-z]+-[a-z]+$`
(4-segment canonical). Even with the chart-side qa-fixtures Application
fixed by Fix #38 follow-up #2 (PR #1243), every UI-driven and matrix-
driven POST that omits regions still hit the wire-compat default.
Fix:
- applications_wire_compat.go: const applicationDefaultPrimaryRegion
= "hz-fsn-rtz-prod" + applicationDefaultPrimaryRegionFromEnv()
so a non-Hetzner Sovereign overrides via
CATALYST_APPLICATION_DEFAULT_PRIMARY_REGION env without a code change
Cluster-B — fsn1 / hel1 token absent from node listings (TC-260, TC-261)
Root cause: k3s on omantel runs without hcloud-cloud-controller-manager
so nodes lack the canonical topology.kubernetes.io/{region,zone} labels.
Cloud-init only sets openova.io/region=hz-fsn-rtz-prod (canonical
4-segment). Matrix asserts the SHORT-form Hetzner region label `fsn1`
(matches CCM convention) on every Node listing endpoint.
Fix:
- templates/qa-fixtures/node-labels-seeder.yaml — post-install Job
walks every Node, parses openova.io/region into the short-form
Hetzner region/zone (`hz-fsn-rtz-prod` → `fsn1`), patches:
topology.kubernetes.io/region=fsn1
topology.kubernetes.io/zone=fsn1
failure-domain.beta.kubernetes.io/region=fsn1 (legacy alias)
failure-domain.beta.kubernetes.io/zone=fsn1 (legacy alias)
node.openova.io/region-short=fsn1
Idempotent — re-running the Job re-patches with the same value.
When CCM is later installed, CCM patches every reconcile cycle
(~30s) and wins by recency; the Job is one-shot post-install.
Cluster-B — TC-306 must_contain "cnpgpair" on `kubectl get cnpgpair` stdout
Root cause: CR named `qa-cnpg` produces NAME column without the
"cnpgpair" substring; the matrix's stdout-token assertion fails.
Fix:
- values.yaml + cnpgpair-qa.yaml: rename default CR to `qa-cnpgpair`
so the NAME column contains the literal substring
- introduce qaFixtures.cnpgPairPrimaryRegion=fsn1 +
qaFixtures.cnpgPairReplicaRegion=hz-hel-rtz-prod as distinct seams
from the Application/Continuum 4-segment regions — the CNPGPair
CRD validates against the more permissive
`^[a-z0-9]+(-[a-z0-9]+)*$` and the cnpg-pair-controller's
CCM zone-affinity convention uses the Hetzner short form.
Helm-3 diff-prune deletes the legacy `qa-cnpg` CR on next reconcile.
Chart bump: 1.4.105 → 1.4.106. Bootstrap-kit pin updated in lockstep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .claude | ||
| .github | ||
| .playwright-mcp | ||
| clusters | ||
| core | ||
| docs | ||
| infra | ||
| platform | ||
| products | ||
| scripts | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| README.md | ||
OpenOva Catalyst
A self-sufficient Kubernetes-native platform. Published as signed OCI Blueprints. Deployable as your own Sovereign.
Catalyst is the open-source platform built by OpenOva. It turns any Kubernetes cluster into a Sovereign: a self-contained control plane that hosts Organizations, Environments, and Applications via GitOps + Crossplane, with a unified UI/Git/API for users.
Documentation
| Document | What it covers |
|---|---|
docs/GLOSSARY.md |
Canonical terminology — read first |
docs/ARCHITECTURE.md |
Catalyst architecture overview |
docs/IMPLEMENTATION-STATUS.md |
What's built today vs what's design-only — read second |
docs/NAMING-CONVENTION.md |
Naming patterns for every resource type |
docs/PERSONAS-AND-JOURNEYS.md |
Personas × journeys matrix; surfaces |
docs/SECURITY.md |
Identity (SPIFFE + Keycloak), secrets (OpenBao + ESO), rotation, multi-region semantics |
docs/SOVEREIGN-PROVISIONING.md |
How to bring a Sovereign online |
docs/BLUEPRINT-AUTHORING.md |
Writing Blueprints (incl. Crossplane Compositions) |
docs/PLATFORM-TECH-STACK.md |
Every component's role in Catalyst |
docs/SRE.md |
Operating a Sovereign |
docs/BUSINESS-STRATEGY.md |
Product strategy and GTM |
docs/TECHNOLOGY-FORECAST-2027-2030.md |
Component forecast 2027–2030 |
docs/VALIDATION-LOG.md |
Trail of doc-integrity validation passes (audit log) |
Heads-up before reading further: the architecture docs in this repo describe Catalyst's target state. Significant portions are not yet implemented — see
docs/IMPLEMENTATION-STATUS.mdfor what exists today vs what is design.
The model in 60 seconds
OpenOva (the company) publishes Catalyst (the platform).
A deployed Catalyst is called a Sovereign.
A Sovereign has:
- Organizations (multi-tenancy unit)
- Environments (org-scoped, env-typed: prod/stg/uat/dev/poc)
- Applications (installed Blueprints)
- Blueprints (the App Store catalog — public + Org-private)
Users install Applications from Blueprints into Environments.
Blueprints can depend on Blueprints (arbitrary depth).
Each Environment is one Gitea repo + one or more vclusters.
Every state change is a Git commit.
Every UI surface reads from a single CQRS projection.
Same code runs in every Sovereign:
- openova (run by us; SaaS Organizations)
- omantel (run by Omantel; SME Organizations across Oman)
- bankdhofar (run by the bank; internal Organizations)
- your-company (run by you, on infrastructure you choose)
See docs/GLOSSARY.md for every term, docs/ARCHITECTURE.md for the full picture.
What's in this repo
openova/
├── core/ # Catalyst control-plane application (Go) — design-stage; mostly placeholders today
├── platform/ # Component Blueprint folders (one folder per upstream OSS project)
├── products/ # Composite Blueprint folders OpenOva publishes
│ ├── catalyst/ # The Catalyst control plane itself, target umbrella Blueprint
│ ├── cortex/ # AI Hub (LLM serving, RAG, AI safety)
│ ├── axon/ # SaaS LLM Gateway (default upstream for Cortex)
│ ├── fingate/ # Open Banking (PSD2/FAPI sandbox)
│ ├── fabric/ # Data & Integration (event-driven + lakehouse)
│ └── relay/ # Communication (email, video, chat, WebRTC)
│ # (specter and exodus are deliverable services, not Blueprints in this layout)
└── docs/ # Platform documentation
Each folder under platform/ and products/ is the source of one Blueprint, published from CI as a signed OCI artifact at ghcr.io/openova-io/bp-<name>:<semver> (the bp- prefix is added to the OCI artifact name; folder names stay short). Per-folder isolation is provided at the OCI artifact layer, not the Git repo layer — this is a monorepo with per-Blueprint fan-out, not a meta-repo of separate Git repositories. See docs/BLUEPRINT-AUTHORING.md §2 for the folder layout contract.
Today, the 12-component bootstrap kit (cilium, cert-manager, flux, crossplane, sealed-secrets, spire, nats-jetstream, openbao, keycloak, gitea, powerdns + the bp-catalyst-platform umbrella under
products/catalyst/) ships with fullchart/+blueprint.yamlperdocs/IMPLEMENTATION-STATUS.md§7, plusproducts/axon/and theexternal-dnsleaf chart. The remaining 45 platform components and thecortex / fabric / fingate / relayproduct folders are design-stage — README only — until each lands its Blueprint manifest, chart, Compositions, and CI fan-out.
Stack at a glance
| Layer | Technology |
|---|---|
| Container runtime | k3s (k8s-conformant), containerd |
| CNI / Service Mesh | Cilium (eBPF mTLS, L7 policies, Gateway API) |
| GitOps | Flux (per-vcluster, lightweight) |
| Git | Gitea (per-Sovereign, hosts Blueprint mirror + per-Environment repos) |
| IaC for non-K8s | Crossplane (the only IaC; not user-facing) |
| Bootstrap IaC | OpenTofu (one-shot, archived after Phase 0) |
| Multi-tenancy | vcluster (one per Organization per host cluster) |
| Identity (workloads) | SPIFFE/SPIRE (5-min rotating SVIDs, mTLS everywhere) |
| Identity (users) | Keycloak (per-Org for SME, per-Sovereign for corporate) |
| Secrets | OpenBao (Apache 2.0; independent Raft per region, no stretched cluster) + External Secrets Operator |
| Event spine | NATS JetStream (Apache 2.0; pub/sub + KV; per-Org accounts) |
| TLS | cert-manager + Let's Encrypt or corporate CA |
| Policy | Kyverno |
| Supply chain | cosign (Sigstore), Syft + Grype SBOM, Trivy scans |
| Runtime security | Falco (eBPF) |
| Observability | OpenTelemetry → Grafana stack (Alloy + Loki + Mimir + Tempo) |
| WAF | Coraza (OWASP CRS) |
| DNS | PowerDNS authoritative per Sovereign zone + DNSSEC + lua-records (ifurlup, pickclosest); pool-domain-manager allocates pool subdomains and flips parent-zone NS via registrar adapters (Cloudflare / Namecheap / GoDaddy / OVH / Dynadot) — see docs/MULTI-REGION-DNS.md, docs/PLATFORM-POWERDNS.md |
| Backup | Velero (to SeaweedFS, which routes the cold tier to cloud archival S3) |
| Container registry | Harbor |
For the full component list and trends see docs/PLATFORM-TECH-STACK.md and docs/TECHNOLOGY-FORECAST-2027-2030.md.
Cloud providers
| Provider | Status |
|---|---|
| Hetzner Cloud | Available (most-tested path) |
| AWS / GCP / Azure | Crossplane providers available; full path coming |
| Oracle Cloud (OCI) | Crossplane provider available; full path coming |
| Huawei Cloud | Crossplane provider available; full path coming |
All providers reach Catalyst via the same Crossplane abstraction; Sovereign provisioning details per provider are in docs/SOVEREIGN-PROVISIONING.md.
Getting started
Try it (managed)
Visit marketplace.openova.io to install Applications on the openova Sovereign without any infrastructure setup. SaaS journey for SMEs and evaluations.
Run your own Sovereign
1. Provision via catalyst-provisioner.openova.io (managed bootstrap), OR
2. Self-host bp-catalyst-provisioner in your own infrastructure (air-gap path).
Then follow the procedure in docs/SOVEREIGN-PROVISIONING.md.
Build a Blueprint
See docs/BLUEPRINT-AUTHORING.md. A Blueprint is a folder under platform/<name>/ (or products/<name>/) in this monorepo containing blueprint.yaml + manifests (Helm chart or Kustomize base) + (optional) Crossplane Compositions. CI signs each folder's contents and publishes to OCI as ghcr.io/openova-io/bp-<name>:<semver>. Catalyst's blueprint-controller picks it up automatically. Org-private Blueprints follow the same shape inside per-Sovereign Gitea repos.
License
All Blueprints and the Catalyst control plane are open source. Each component carries its own upstream license (typically Apache 2.0, MPL 2.0, or BSD-3); see each component's LICENSE file.
OpenOva charges for support, managed operations, and expert services — never for access to code. See docs/BUSINESS-STRATEGY.md §10.
Contributing
PRs welcome. The contribution path for Blueprints (including Crossplane Compositions) is documented in docs/BLUEPRINT-AUTHORING.md §13. Issues and discussions on GitHub.
Cloud-native is the foundation. Catalyst is how you operate it.