openova/docs/BLUEPRINT-AUTHORING.md
hatiyildiz 1ddd569789 fix(bp-*): observability toggles default false — break circular CRD dependency
Extends the v1.1.1 hardening that started with cilium / cert-manager /
crossplane to the remaining 8 bootstrap-kit + per-Sovereign Blueprints.
Every observability toggle in every Catalyst-curated Blueprint now ships
`false`/`null` by default; the operator opts in via a per-cluster values
overlay at clusters/<sovereign>/bootstrap-kit/* once
bp-kube-prometheus-stack reconciles.

Live failure mode that prompted this (omantel.omani.works 2026-04-29):
bp-cilium @ 1.1.0 defaulted hubble.relay/ui + prometheus.serviceMonitor
to true. The upstream Cilium 1.16.5 chart renders a
monitoring.coreos.com/v1 ServiceMonitor whose CRD ships with
kube-prometheus-stack — a tier-2 Application Blueprint that depends on
the bootstrap-kit (cilium first). Helm install fails on a fresh
Sovereign with "no matches for kind ServiceMonitor in version
monitoring.coreos.com/v1 — ensure CRDs are installed first" and every
downstream HelmRelease reports `dep is not ready`. The earlier
trustCRDsExist=true mitigation only suppresses Helm's render-time gate;
the apiserver still rejects the resource at install-time.

Per-Blueprint changes:
- bp-cilium: hubble.relay.enabled, hubble.ui.enabled → false;
  hubble.metrics.enabled → null (this is the exact value that disables
  the upstream metrics ServiceMonitor template branch — verified by
  reading cilium 1.16.5's _hubble.tpl); hubble.metrics.serviceMonitor
  .enabled → false. tests/observability-toggle.sh extended with Case 4
  (default render produces no hubble-relay / hubble-ui Deployments).
- bp-flux: flux2.prometheus.podMonitor.create → false.
- bp-sealed-secrets: sealed-secrets.metrics.serviceMonitor.enabled
  → false (explicit lock; upstream already defaults false).
- bp-spire: spire.global.spire.recommendations.enabled +
  recommendations.prometheus → false.
- bp-nats-jetstream: nats.promExporter.enabled +
  promExporter.podMonitor.enabled → false.
- bp-openbao: openbao.injector.metrics.enabled +
  openbao.serviceMonitor.enabled → false.
- bp-keycloak: keycloak.metrics.enabled + metrics.serviceMonitor.enabled
  + metrics.prometheusRule.enabled → false.
- bp-gitea: gitea.gitea.metrics.* and gitea.postgresql.metrics.*
  serviceMonitor + prometheusRule → false.
- bp-powerdns: powerdns.serviceMonitor.enabled + powerdns.metrics.enabled
  → false (forward-compatibility guard; current upstream
  pschichtel/powerdns 0.10.0 has no ServiceMonitor template, but a future
  upstream bump cannot silently regress).

Each chart ships a tests/observability-toggle.sh that asserts the rule
in three cases (default off / explicit on opt-in / explicit off) — runs
under blueprint-release.yaml's chart-test gate (added bdeb0f54 + the
existing wiring) before helm push. A regression that re-introduces a
hardcoded enabled: true in any chart fails CI before the OCI artifact
is published.

Versioning:
- All 11 leaf charts bumped 1.1.0 → 1.1.1.
- products/catalyst/chart (bp-catalyst-platform umbrella) deps updated
  to 1.1.1 across the board.
- clusters/_template/bootstrap-kit/03-flux through 10-gitea bumped to
  1.1.1; clusters/omantel.omani.works/bootstrap-kit/* mirror.

docs/BLUEPRINT-AUTHORING.md §11.2 table extended to enumerate every
toggle disabled across all 11 Blueprints. References
docs/INVIOLABLE-PRINCIPLES.md #4.

GATES (all green):
- helm dep build resolves cleanly post-change for every chart whose
  upstream is published (umbrella waits on per-leaf publish).
- helm lint clean on all 11 leaves.
- helm template . default render produces zero monitoring.coreos.com
  references on every leaf (verified locally).
- tests/observability-toggle.sh PASS on all 11 leaves.

Live verification: with v1.1.1 published the omantel.omani.works
HelmRelease can roll forward without a manual values patch — Flux picks
up the new chart digest automatically (semver: 1.x in OCIRepository).

Refs: issue #182.
2026-04-29 19:23:52 +02:00

29 KiB

Blueprint Authoring

Status: Authoritative target spec. Updated: 2026-04-27. Implementation: The Blueprint CRD, blueprint-controller, and CI fan-out described below are design-stage. See IMPLEMENTATION-STATUS.md. Today, platform/<name>/ and products/<name>/ folders contain only README.md.

How to author a Blueprint for Catalyst — the unified unit of installable software (replaces what was previously called "module" + "template"). Defer to GLOSSARY.md for terminology and ARCHITECTURE.md for the broader model.


1. What a Blueprint is

A Blueprint is:

  • A source location (one of three Gitea-Org-scoped places, all using identical Blueprint shape):
    • Public Blueprints: a directory under platform/<name>/ or products/<name>/ in the github.com/openova-io/openova monorepo (this repository). Per-Blueprint isolation is provided by CI fan-out — each folder publishes its own signed OCI artifact. Visible to every Sovereign via the catalog Gitea Org mirror.
    • Sovereign-curated private Blueprints: a Gitea Repo under the catalog-sovereign Gitea Org on a Sovereign (e.g. gitea.<location-code>.<sovereign-domain>/catalog-sovereign/bp-<name>/). Authored by the Sovereign owner, visible to every Catalyst Organization on that Sovereign without being public upstream. Use case: an SME-marketplace operator (like acme-telecom) curates bp-wordpress, bp-jitsi, bp-cal-com for their tenants.
    • Org-private Blueprints: a directory inside gitea.<location-code>.<sovereign-domain>/<org>/shared-blueprints/bp-<name>/ in that Organization's Gitea repo on its Sovereign (canonical Catalyst control-plane DNS form per NAMING-CONVENTION.md §5.1). Visible only within that Org.
  • A CRD manifest (blueprint.yaml) declaring its identity, configSchema, placementSchema, dependencies, and pointers to its manifests.
  • A set of manifests (Helm chart, Kustomize base + overlays, or raw YAML) that get applied when the Blueprint is installed as an Application.
  • A set of Crossplane Compositions (optional) for any non-Kubernetes resources the Blueprint provisions.
  • A CI pipeline that signs the artifact (cosign), generates an SBOM (Syft), publishes to OCI registry (ghcr.io/openova-io/bp-<name>:<semver>), and tags a release.

One Blueprint = one card in the marketplace (when visibility: listed).

Why monorepo for public Blueprints: a single repository is simpler to govern, gives one consistent CI pipeline shape across all components, and avoids the per-repo overhead of permissions, settings, and dependabot config. Per-Blueprint isolation is provided at the OCI artifact layer, not the Git repo layer — ghcr.io/openova-io/bp-<name>:<semver> artifacts are independently versioned, signed, and consumed.


2. Folder layout

A Blueprint folder lives at platform/<name>/ or products/<name>/ in the github.com/openova-io/openova monorepo. The CI pipeline at the monorepo root detects changes per folder and publishes per-Blueprint OCI artifacts.

platform/<name>/                 ← OR products/<name>/ for composite Blueprints
├── blueprint.yaml               ← the Blueprint CRD manifest
├── README.md                    ← what it does, links to docs
├── chart/                       ← Helm chart (preferred for typical apps)
│   ├── Chart.yaml
│   ├── values.yaml
│   └── templates/
│   OR
├── manifests/                   ← Kustomize base
│   ├── base/
│   │   ├── kustomization.yaml
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   └── overlays/
│       ├── small/
│       ├── medium/
│       └── large/
├── compositions/                ← (optional) Crossplane Compositions
│   ├── postgres-database.yaml
│   └── object-storage-bucket.yaml
├── card/                        ← marketplace presentation
│   ├── icon.svg
│   ├── screenshots/
│   └── description.md
└── tests/                       ← acceptance tests
    ├── integration.yaml         ← Litmus probe / Catalyst test harness
    └── upgrade.yaml

The CI workflow lives once at the monorepo root (.github/workflows/) and uses path-based matrix builds — every blueprint.yaml triggers its own pipeline:

# .github/workflows/blueprint-release.yaml (monorepo root, path-matrix)
on:
  push:
    tags: ['platform/*/v*', 'products/*/v*']    # tag form: platform/<name>/v1.2.3
  pull_request:
    paths: ['platform/**', 'products/**']

This shape is documented as the design contract; the workflow itself is not yet implemented (see IMPLEMENTATION-STATUS.md).


3. The Blueprint CRD

Annotated example for bp-wordpress:

apiVersion: catalyst.openova.io/v1alpha1
kind: Blueprint
metadata:
  name: bp-wordpress
  version: 1.3.0
spec:

  card:                                # presentation in marketplace
    title: WordPress
    tagline: Self-hosted CMS
    category: cms
    tags: [cms, blog, php]
    icon: ./card/icon.svg
    screenshots:
      - ./card/screenshots/admin.png
      - ./card/screenshots/post-editor.png
    license: GPL-2.0
    documentation: https://wordpress.org/documentation

  visibility: listed                   # listed | unlisted | private

  owner:
    team: apps                         # team responsible for upkeep
    contact: apps@openova.io

  configSchema:                        # JSON Schema; drives console form
    type: object
    required: [domain, adminEmail]
    properties:
      domain:
        type: string
        format: hostname
        description: Public domain for the site
      adminEmail:
        type: string
        format: email
      title:
        type: string
        default: "My WordPress site"
      replicas:
        type: integer
        default: 2
        minimum: 1
        maximum: 20
      postgres:
        type: object
        oneOf:
          - properties:
              mode: { const: embedded }
          - properties:
              mode: { const: external }
              ref:
                type: string
                description: Name of an existing bp-postgres Application

  placementSchema:                     # supported placement modes
    modes: [single-region, active-active, active-hotstandby]
    minRegions: 1
    maxRegions: 5

  depends:                             # dependency declarations
    - blueprint: bp-postgres
      version: ^1.4
      alias: db
      when: "{{ .config.postgres.mode == 'embedded' }}"
      values:
        databases: ["{{ .application.name }}"]
        size: medium

  manifests:                           # how to materialize on install
    source:
      kind: HelmChart
      ref: oci://ghcr.io/openova-io/bp-wordpress:1.3.0
    overlays:                          # vendor sizing variants
      small:
        replicas: 1
        postgres: { mode: embedded, size: small }
        backups: { schedule: weekly }
      medium:
        replicas: 2
        postgres: { mode: embedded, size: medium }
        backups: { schedule: daily }
      large:
        replicas: 5
        postgres: { mode: external }
        backups: { schedule: daily }
        pdb: true
        hpa: true

  upgrades:                            # supported upgrade paths
    from:
      - 1.2.x                          # safe automatic
      - 1.1.x                          # requires data migration
    blocks:
      - 1.0.x                          # no path; recreate

  rotation:                            # secrets this Blueprint owns
    - kind: oauth-client-secret
      name: wp-keycloak-client
      ttl: 90d

  observability:                       # what this Blueprint emits
    metrics: prometheus
    logs: stdout
    traces: otlp

4. configSchema design

The console form is generated from configSchema — never hand-written. JSON Schema features supported:

  • type, format, default, enum, minimum, maximum
  • oneOf / anyOf for branching (e.g. embedded vs external Postgres)
  • properties.x.description becomes form help text
  • dependencies for conditional fields
  • x-catalyst-ui-hint for non-trivial widgets:
    • password — masked input
    • domain-picker — autocomplete from existing Org domains
    • application-ref — picker over existing Apps in the Environment matching a Blueprint filter

Example with hint:

postgres:
  type: object
  properties:
    ref:
      type: string
      x-catalyst-ui-hint: application-ref
      x-catalyst-ui-filter:
        blueprint: bp-postgres
        environment: current

The console renders this as a dropdown of existing postgres Applications in the current Environment.


5. Dependencies

5.1 Hard dependencies

depends:
  - blueprint: bp-postgres
    version: ^1.4
    alias: db

Catalyst will install bp-postgres if not already present. The Blueprint may reference its dependency by alias in its manifests:

# in chart/templates/deployment.yaml
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: "{{ .Values.dependencies.db.connectionSecret }}"
        key: url

5.2 Conditional dependencies

depends:
  - blueprint: bp-postgres
    when: "{{ .config.postgres.mode == 'embedded' }}"
    alias: db

Skipped at install time if the predicate is false. Useful when the user can choose "embedded backing service" vs "use existing".

5.3 Reference dependencies

The user can choose external mode and reference an existing Application:

configSchema:
  postgres:
    oneOf:
      - properties:
          mode: { const: embedded }
      - properties:
          mode: { const: external }
          ref: { type: string }

When mode: external, the Blueprint's manifests resolve ref to a sibling Application in the same Environment, reads its connection details from the secret it exposes, and connects.


6. Placement and multi-region

placementSchema declares which Placement modes the Blueprint supports:

placementSchema:
  modes: [single-region, active-active, active-hotstandby]
  minRegions: 1
  maxRegions: 5

For active-active, the Blueprint must be designed for it:

  • Stateless services: trivial.
  • Stateful: the Blueprint declares the replication strategy in its manifests (e.g. CNPG WAL streaming, SeaweedFS bucket replication, Valkey REPLICAOF).

Catalyst's projector uses the Placement spec to fan out manifests across the right vclusters at install time.


7. Manifests

Three accepted source types:

manifests.source.kind When to use
HelmChart Most third-party apps with existing Helm charts.
Kustomize Small custom apps; full control over patches and overlays.
OAM (Future, not yet supported) — Open Application Model definitions.

For Helm: ref points at an OCI artifact; Catalyst's Flux helm-controller fetches and renders.

For Kustomize: the Blueprint repo's manifests/base/ is the base; each overlay in manifests/overlays/<size>/ is a Kustomize component layered on top. Catalyst's Flux kustomize-controller renders.


8. Crossplane Compositions

If the Blueprint requires non-Kubernetes resources (cloud DBs, DNS records, S3 buckets, etc.), it includes Crossplane Compositions in compositions/.

# compositions/postgres-database.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgres-database.bp-wordpress
spec:
  compositeTypeRef:
    apiVersion: compose.openova.io/v1alpha1   # shared XRD group across Blueprints
    kind: PostgresDatabase
  resources:
    - name: hetzner-postgres-instance
      base:
        apiVersion: db.hcloud.crossplane.io/v1alpha1
        kind: PostgresInstance
        spec:
          forProvider:
            location: { from: spec.region }
            tier: { from: spec.tier }

Crossplane is never user-facing. End users see "needs a database" in the form, not Crossplane Compositions. Advanced users who write Compositions are typically:

  • OpenOva engineers extending the public catalog.
  • Sovereign-admins authoring private Compositions for their Sovereign.
  • Corporate platform engineers contributing back upstream.

Compositions live in the Blueprint repo alongside the Helm chart / Kustomize manifests; CI signs and publishes them as part of the same OCI artifact.


9. Visibility

Value Where it appears Who can install it
listed Public marketplace card grid Everyone in the Sovereign
unlisted Not on cards; reachable by direct URL or search Anyone who knows the Blueprint name
private Visible only within the Org that owns the Blueprint repo Only that Org's users

Org-private Blueprints live in the Org's shared-blueprints Gitea repo, which only that Org's users have access to.


10. Versioning

  • Semver (MAJOR.MINOR.PATCH).
  • Each release publishes a signed OCI artifact at ghcr.io/openova-io/bp-<name>:<version> (where <name> is the folder name; the bp- prefix is added to the OCI artifact name to make it self-identifying as a Catalyst Blueprint).
  • The Blueprint declares which prior versions are upgrade-compatible (upgrades.from).
  • Customers pin to a version in their Application's kustomization.yaml. Upgrades are explicit (one-click in console, or a git push editing the version pin).

11. CI pipeline

Catalyst uses a single monorepo CI at the root of github.com/openova-io/openova (see §2 for the folder layout and path-matrix tag form). The same pipeline shape applies to every platform/<name>/ and products/<name>/ folder:

# .github/workflows/blueprint-release.yaml (monorepo root)
on:
  pull_request:
    paths: ['platform/**', 'products/**']        # runs validate on PR
  push:
    tags:
      - 'platform/*/v*'                          # tag form: platform/<name>/v1.2.3
      - 'products/*/v*'                          #          products/<name>/v1.2.3

jobs:
  validate:                                      # runs on every PR touching a Blueprint folder
    - detect changed Blueprint folders (path-matrix)
    - for each: lint blueprint.yaml against the Blueprint CRD schema
                lint Helm chart / Kustomize base
                dry-run install in a kind cluster
                run tests/integration.yaml
                run tests/upgrade.yaml against the previous version

  build-and-sign:                                # runs only on tag push
    - parse the tag → identify which Blueprint folder + version
    - render that folder's Helm chart / Kustomize → OCI artifact
    - syft generate SBOM (per Blueprint)
    - cosign sign artifact + SBOM
    - push to ghcr.io/openova-io/bp-<folder-name>:<version>
    - publish blueprint.yaml as the OCI manifest's metadata layer

So tagging platform/wordpress/v1.3.0 triggers a build of platform/wordpress/'s contents and publishes ghcr.io/openova-io/bp-wordpress:1.3.0. Other Blueprint folders are untouched. This is what "monorepo with per-Blueprint fan-out" means in practice.

Catalyst's blueprint-controller watches the GHCR catalog and registers new versions automatically — they appear in the marketplace within seconds of a successful release.


11.1 Umbrella shape (hard contract — CI-enforced)

Every Blueprint chart at platform/<name>/chart/ (and products/<name>/chart/ for composite Blueprints) MUST be an umbrella chart: it MUST declare its upstream chart(s) under dependencies: in Chart.yaml so helm dependency build pulls the upstream payload into the published OCI artifact.

Hollow charts — wrappers that carry only Catalyst overlay templates (NetworkPolicy, ClusterIssuer, ExternalSecret, ServiceMonitor) without an upstream subchart dependency — are forbidden. CI rejects them.

Why

Earlier this cycle, bp-cert-manager:1.0.0 shipped as a hollow chart: it carried only a ClusterIssuer template and no upstream cert-manager subchart bytes. Flux installed it on every Sovereign. Phase 1 broke on every Sovereign because cert-manager itself was never deployed — there was no controller, no CRDs, and the curated ClusterIssuer had nothing to register against. The artifact looked legitimate (right name, right version, signed, SBOM-attested) but the upstream payload was simply not there.

The fix is structural: the published OCI artifact's <chart_name>/charts/ directory MUST contain the upstream chart at the version pinned by Chart.yaml's dependencies: block.

What CI enforces

.github/workflows/blueprint-release.yaml runs four hollow-chart guards on every publish:

Stage Guard Failure mode caught
After helm dependency build Working-tree chart/charts/<dep>-<ver>.tgz (or unpacked chart/charts/<dep>/Chart.yaml) exists for every dependencies: entry. Missing/wrong repo URL, dependency-build silently skipped a dep.
After helm package tar -tzf listing of the produced .tgz contains <chart_name>/charts/<dep>-<ver>.tgz (or unpacked) for every dependencies: entry. .helmignore mishap, packaging-time stripping.
After helm push helm pull round-trips the artifact from GHCR; the pulled .tgz listing again contains every declared subchart. Registry-side path mangling, OCI manifest rewriting.
Always helm template smoke render with default values produces non-trivial output; rendered manifests uploaded as workflow artifact for forensics. Render-broken templates, schema violations, missing required values.

Any single guard failing fails the whole publish job. A hollow Blueprint can never reach a Sovereign through the sanctioned CI path.

Authoring rule

Every umbrella Chart.yaml declares the upstream chart(s) it wraps:

# platform/cilium/chart/Chart.yaml
apiVersion: v2
name: bp-cilium
version: 1.1.0
type: application

# Upstream chart pulled in as a Helm subchart so `helm dependency build`
# bundles it into the OCI artifact. Pinned upstream version matches
# platform/cilium/blueprint.yaml + values.yaml's
# `catalystBlueprint.upstream.version`.
dependencies:
  - name: cilium
    version: "1.16.5"
    repository: "https://helm.cilium.io"

Catalyst-curated overlay templates (NetworkPolicy, ServiceMonitor, ClusterIssuer, ExternalSecret) live under chart/templates/ alongside the dependency declaration. At install time Helm renders the upstream subchart and the Catalyst overlay — both ship from the same OCI artifact.

The version pinned in dependencies: MUST match the version recorded in platform/<name>/blueprint.yaml and the catalystBlueprint.upstream.version field in values.yaml. Operators bump all three together via PR + Blueprint release per Inviolable Principle #4 (no hardcoding).

Composite umbrellas (products/catalyst/chart/) follow the same rule: each leaf Blueprint they bundle is declared under dependencies:.

Verifying an existing artifact

helm pull oci://ghcr.io/openova-io/bp-cilium --version 1.1.0
tar -tzf bp-cilium-1.1.0.tgz | grep '^bp-cilium/charts/cilium/' | head

A non-empty result proves the upstream subchart is inside the OCI artifact.


11.2 Observability toggles must default false (hard contract — CI-enforced)

Every observability toggle in a Blueprint's chart/values.yamlserviceMonitor.enabled, metrics.enabled, prometheusRule.enabled, monitoring.enabled, tracing.enabled, prometheus.enabled and analogues — MUST default to false. The operator opts in via per-cluster values overlay AFTER the observability tier (kube-prometheus-stack / Grafana / Tempo) is reconciled.

This rule is a direct consequence of INVIOLABLE-PRINCIPLES.md #4 (never hardcode): a chart-level true is a hardcoded operational decision that assumes a runtime that does not yet exist.

Why

The CRDs that back ServiceMonitor / PrometheusRule (monitoring.coreos.com/v1) ship with kube-prometheus-stack — an Application-tier Blueprint that depends on the bootstrap-kit (Cilium first, then cert-manager, then Flux, etc.). If bp-cilium defaults cilium.prometheus.serviceMonitor.enabled: true, Helm renders a ServiceMonitor that the apiserver immediately rejects:

no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
— ensure CRDs are installed first

The apparent mitigation serviceMonitor.trustCRDsExist: true only suppresses Helm's render-time gate; the apiserver still rejects the resource at install-time. Result: bp-cilium's HelmRelease enters InstallFailed, every downstream bp-* HelmRelease (dependsOn: bp-cilium) reports dep is not ready, and the whole Sovereign bootstrap stalls. Verified failure mode on omantel.omani.works 2026-04-29 (issue #182).

The fix is structural: every observability knob is operator-tunable, lives in values.yaml, and ships false. The operator turns it on via the per-cluster overlay at clusters/<sovereign>/bootstrap-kit/<NN>-bp-<name>.yaml once the observability tier is reconciled — no rebuild of the Blueprint OCI artifact is required.

Canonical pattern

# platform/cilium/chart/values.yaml — DEFAULT OFF
cilium:
  prometheus:
    enabled: false
    serviceMonitor:
      enabled: false
# clusters/<sovereign>/bootstrap-kit/01-cilium.yaml — OPERATOR OPT-IN
spec:
  values:
    cilium:
      prometheus:
        enabled: true
        serviceMonitor:
          enabled: true

What CI enforces

.github/workflows/blueprint-release.yaml runs tests/observability-toggle.sh (when present under platform/<name>/chart/tests/) on every publish. The canonical script asserts:

Case Assertion
Default render (helm template no --set) Zero monitoring.coreos.com/v1 references AND zero kind: ServiceMonitor.
Opt-in render (--set <toggle>=true) Render succeeds AND produces a ServiceMonitor (proves the toggle is wired).
Explicit-off render (--set <toggle>=false) Render succeeds AND zero monitoring.coreos.com/v1 references.

Any case failing fails the publish job. A regression that re-introduces a hardcoded enabled: true cannot reach a Sovereign through the sanctioned CI path.

Authoring rule

When you wrap an upstream chart whose own values default an observability toggle true (e.g. cert-manager v1.16 prometheus.enabled: true historically), the Catalyst overlay MUST set it back to false in chart/values.yaml:

# platform/cert-manager/chart/values.yaml
cert-manager:
  prometheus:
    enabled: false        # Catalyst overrides upstream `true`
    servicemonitor:
      enabled: false

If a Blueprint exposes a more elaborate observability surface (e.g. a chart that ships its own PrometheusRule template gated by monitoring.alerts.enabled), default ALL such gates false. Add a row to tests/observability-toggle.sh for each non-trivial toggle.

Existing exemplars

Every bootstrap-kit Blueprint at v1.1.1+ ships every observability surface defaulted off. The table below is the complete audit (issue #182):

Blueprint Toggle Default Why
bp-cilium cilium.prometheus.enabled false Renders ServiceMonitor when true
bp-cilium cilium.prometheus.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor — CRD ships with kube-prometheus-stack
bp-cilium cilium.hubble.relay.enabled false Relay Deployment depends on hubble metrics scraping
bp-cilium cilium.hubble.ui.enabled false UI Deployment depends on relay
bp-cilium cilium.hubble.metrics.enabled null A populated list triggers an unconditional metrics ServiceMonitor render in the upstream chart
bp-cilium cilium.hubble.metrics.serviceMonitor.enabled false Belt-and-braces
bp-cert-manager cert-manager.prometheus.enabled false Upstream defaults true historically; we override
bp-cert-manager cert-manager.prometheus.servicemonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-flux flux2.prometheus.podMonitor.create false monitoring.coreos.com/v1 PodMonitor
bp-crossplane crossplane.metrics.enabled false Upstream emits prometheus.io/scrape annotation only — kept off for uniformity
bp-sealed-secrets sealed-secrets.metrics.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-spire spire.global.spire.recommendations.enabled false Cascades prometheus exporters into spire-server / spire-agent
bp-spire spire.global.spire.recommendations.prometheus false Belt-and-braces inside the recommendations bundle
bp-nats-jetstream nats.promExporter.enabled false Sidecar exporter container
bp-nats-jetstream nats.promExporter.podMonitor.enabled false monitoring.coreos.com/v1 PodMonitor
bp-openbao openbao.injector.metrics.enabled false injector metrics endpoint
bp-openbao openbao.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-keycloak keycloak.metrics.enabled false Statistics endpoint
bp-keycloak keycloak.metrics.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-keycloak keycloak.metrics.prometheusRule.enabled false monitoring.coreos.com/v1 PrometheusRule
bp-gitea gitea.gitea.metrics.enabled false Built-in /metrics endpoint
bp-gitea gitea.gitea.metrics.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-gitea gitea.postgresql.metrics.enabled false bitnami postgresql exporter sidecar
bp-gitea gitea.postgresql.metrics.serviceMonitor.enabled false monitoring.coreos.com/v1 ServiceMonitor
bp-gitea gitea.postgresql.metrics.prometheusRule.enabled false monitoring.coreos.com/v1 PrometheusRule
bp-powerdns powerdns.serviceMonitor.enabled false Forward-compatibility guard — current upstream pschichtel/powerdns 0.10.0 has no ServiceMonitor template, but a future upstream bump must not silently regress
bp-powerdns powerdns.metrics.enabled false Forward-compatibility guard

Operators flip these on at clusters/<sovereign>/bootstrap-kit/* once bp-kube-prometheus-stack (Application Blueprint) reconciles.


12. Authoring private Blueprints (in a customer Sovereign)

For corporate customers: the Org's platform team can author private Blueprints without involving OpenOva.

1. In the Catalyst console (Developer mode toggle on):
   Org context → Blueprint Studio → New Blueprint
2. Wizard offers two paths:
     a. Inherit from a public Blueprint (overlay path)
     b. Author from scratch (raw path)
3. Studio writes to gitea.<location-code>.<sovereign-domain>/<org>/shared-blueprints/bp-<name>.
4. On commit, CI runs (Gitea Actions inside the Sovereign).
5. blueprint-controller registers the new private Blueprint.
6. It appears in the Org's catalog as a private card.

Same flow works via direct git push to shared-blueprints. The console UI is convenience; Git is authoritative.


13. Contributing back to the public catalog

If an Org's private Blueprint would be useful to other customers, they can contribute it upstream:

1. Fork github.com/openova-io/openova
2. Add the Blueprint folder under platform/<name>/ or products/<name>/.
   Include blueprint.yaml + chart/ or manifests/ + (optional) compositions/ + tests/.
3. Open PR against main.
4. OpenOva engineers review for security, reusability, license, supply-chain (cosign,
   SBOM, dependency licenses, secret hygiene).
5. Merge → CI signs and publishes ghcr.io/openova-io/bp-<name>:<semver>.
6. blueprint-controller in every Sovereign's Catalyst picks it up on next mirror sync.

The contribution path applies equally to Crossplane Compositions, Helm charts, and full Blueprints. This is how the community grows the catalog.


14. Hard rules for Blueprint authors

Rule Why
All container images cosigned Supply-chain security; Kyverno admission policy denies unsigned.
All artifacts SBOMed Compliance (EU CRA, NIS2).
No plaintext secrets in chart values; use ExternalSecret references See SECURITY.md.
Workload identity via SPIFFE; no static service-account tokens See SECURITY.md §2.
Health endpoints standardized: /healthz (liveness) + /readyz (readiness) Catalyst observability assumes them.
Metrics on /metrics (Prometheus exposition) Catalyst Grafana stack scrapes them.
Logs to stdout, structured JSON Loki ingests them.
Traces via OTel Tempo ingests them.
app.kubernetes.io/* labels set on every resource Required for Catalyst projector to track.
Documentation in README.md, link from card.documentation User clicks "Docs" on the card.
Acceptance tests in tests/ CI runs them on every PR.
Upgrade tests against previous version Required to declare upgrade compatibility.

Cross-reference ARCHITECTURE.md for the runtime model and SECURITY.md for credential handling.