# PowerDNS — Authoritative DNS for OpenOva Sovereigns **Status:** Authoritative. Closes [#167](https://github.com/openova-io/openova/issues/167). PowerDNS Authoritative is the canonical DNS service for every Sovereign zone in the OpenOva fleet. It replaces the previously-listed `k8gb` component — PowerDNS Lua records cover geo and health-checked failover natively, removing the need for a dedicated GSLB controller. This document defines the per-Sovereign zone model, DNSSEC posture, REST API contract, and the operational interfaces that bp-powerdns exposes to the rest of Catalyst. --- ## Per-Sovereign zone model Every Sovereign — pool (e.g. `omantel.omani.works`, `acme.openova.io`) and BYO (`acme.bank.com`) — gets its own PowerDNS zone. No exceptions. ``` PowerDNS Authoritative (Catalyst-Zero, Contabo-mkt initially) ├── openova.io. (root pool — admin + console + api) ├── omani.works. (Oman pool — Huawei + Omantel partners) ├── omantel.omani.works. (Omantel Sovereign — wildcard records) ├── acme.openova.io. (acme Sovereign — pool subdomain) └── acme.bank.com. (acme BYO — operator brought their own domain) ``` Each Sovereign zone holds the canonical 6-record set written by `catalyst-dns`: ``` @ A * A console A api A gitea A harbor A ``` The wildcard A record (`*..`) covers every additional subdomain a Sovereign might add at runtime (e.g. `axon`, `umami`, `langfuse`) without re-issuing certificates. ### Authority chain ``` . (root) └── openova.io. NS ns1.openova.io. NS ns2.openova.io. NS ns3.openova.io. └── DS records for each Sovereign zone (per-zone DNSSEC anchors) ``` The three public NS endpoints (`ns1`, `ns2`, `ns3`) are anycast Floating IPs across Hetzner regions. The Phase-0 stand-in is a Service of type `LoadBalancer` (see "Anycast deferral" below). For pool-domain-based Sovereigns (`.openova.io`, `.omani.works`) the parent zone `openova.io` / `omani.works` is delegated to the OpenOva PowerDNS NS set via the registrar (Dynadot). Each child Sovereign zone (`.openova.io`) publishes its own DS in the parent zone for DNSSEC chaining. For BYO Sovereigns (`.acme.com`) the operator's existing registrar publishes the NS delegation pointing at OpenOva's NS endpoints. The operator follows the runbook in `docs/RUNBOOK-PROVISIONING.md` to add the DS record to their parent zone. --- ## DNSSEC DNSSEC is **mandatory**. Off requires an explicit cluster-overlay override AND a documented exception in the Sovereign's onboarding ticket. ### Algorithm `ECDSAP256SHA256` (algorithm 13) — the IETF-recommended curve for new deployments since 2014, smaller signatures than RSA, sufficient strength for the foreseeable future. ### Key model Each zone gets an automatically-generated KSK (Key Signing Key) + ZSK (Zone Signing Key) pair: ```bash # Generated by the catalyst-dns sidecar after zone creation. pdnsutil add-zone-key ksk active ecdsa256 pdnsutil add-zone-key zsk active ecdsa256 ``` Keys live in the `cryptokeys` table inside the CNPG-managed `pdns-pg` Postgres database. CNPG's WAL archiving + base backup schedule (configured at the bp-cnpg level) is the disaster-recovery anchor — losing the database means losing every zone's keys, so the Postgres backup posture is part of the security story. ### SOA serials `default-soa-edit-signed=INCEPTION-EPOCH` keeps SOA serials in sync across replicas after every signed-zone change. Operators don't manually bump SOA on edits — PowerDNS handles it. ### Rotation KSK rotation follows RFC 6781 / RFC 7583: 1. Add a new KSK (`pdnsutil add-zone-key ksk active ecdsa256`) 2. Wait for TTL to expire on the parent DS record 3. Submit the new DS to the registrar / parent zone 4. Wait for the new DS to propagate 5. Mark the old KSK inactive (`pdnsutil set-zone-key-active 0`) 6. Wait one more TTL window 7. Remove the old KSK (`pdnsutil remove-zone-key `) ZSK rotation is fully automated — PowerDNS handles ZSK rollover internally. --- ## Lua records (replacing k8gb) PowerDNS Lua records ([upstream docs](https://doc.powerdns.com/authoritative/lua-records/index.html)) provide: - **Geo-aware responses** — return different A records based on the resolver's source IP / ECS subnet - **Health-checked failover** — drop a backend from the response set when a TCP/HTTP probe fails - **Weighted round-robin** — split traffic across multiple regional LBs by weight This subsumes everything `k8gb` was doing — same primitive (DNS-level GSLB), no separate operator + CRD set + reconciliation loop. The `enable-lua-records=yes` directive is set in the Catalyst overlay and cannot be turned off without removing geo + health-check capability. ### Example (Catalyst-curated) ``` www IN LUA A " ifurlup('https://www-fra.example.com/healthz', { 'A 1.2.3.4', 'A 5.6.7.8' }) " ``` This record returns `1.2.3.4` while the FRA backend's `/healthz` returns 200; falls through to `5.6.7.8` otherwise. Used by Catalyst's regional active-active LB pattern. --- ## REST API Exposed at `https://pdns.openova.io/api`, behind a Traefik basicAuth Middleware. The plaintext password is generated per-cluster (random 32 chars per `INVIOLABLE-PRINCIPLES.md` #10), bcrypt-hashed in-cluster only, and stored in K8s Secret `powerdns-api-basicauth` in the `openova-system` namespace. ### Endpoints The full API surface is documented at https://doc.powerdns.com/authoritative/http-api/. The Catalyst-relevant endpoints: ``` GET /api/v1/servers/localhost GET /api/v1/servers/localhost/zones POST /api/v1/servers/localhost/zones # create zone PUT /api/v1/servers/localhost/zones/ PATCH /api/v1/servers/localhost/zones/ # add/remove records DELETE /api/v1/servers/localhost/zones/ GET /api/v1/servers/localhost/zones//cryptokeys # DNSSEC keys POST /api/v1/servers/localhost/zones//rectify ``` ### In-cluster consumers Three Catalyst services hit the PowerDNS API in-cluster (via the ClusterIP Service `powerdns:8081`, NOT through the public ingress): 1. **pool-domain-manager (PDM)** — `core/pool-domain-manager/internal/pdns/` writes the canonical 6-record set on `/v1/commit`, creates per-Sovereign zones, and bootstraps DNSSEC keys (#163, #167, #168). PDM also wraps the parent-zone NS-flip via its registrar adapters (#170 — Cloudflare, Namecheap, GoDaddy, OVH, Dynadot). 2. **cert-manager-webhook-pdns** — DNS-01 ACME challenges for wildcard certs (replaces the planned cert-manager-dynadot-webhook for any zone hosted on PowerDNS) 3. **external-dns-pdns** — automatic A/AAAA/CNAME records for K8s Ingress + LB services The API key (`x-api-key` header) is read from K8s Secret `powerdns-api-credentials` (key `api-key`); the same secret is mounted by all three consumers. ### External consumers Operators may hit `https://pdns.openova.io/api` from a laptop using `curl`-with-basic-auth. The basicAuth middleware terminates at Traefik, so the API key is still required end-to-end: ``` curl -u operator: -H 'X-API-Key: ' \ https://pdns.openova.io/api/v1/servers/localhost ``` --- ## dnsdist companion A `dnsdist` Deployment fronts every PowerDNS replica for query rate-limiting and DDoS posture. Default policy: - `MaxQPSIPRule(100)` — drop queries from any source IP exceeding 100 qps for 60s - 1% sampled query logging to stderr (k8s log) - Health-checked backend (the in-cluster `powerdns:5353` Service) The threshold is configurable via `.Values.dnsdist.ratelimit.qpsPerSource`. Per-Sovereign overrides go in the cluster overlay's HelmRelease `values:` block. --- ## Anycast deferral The target state for `ns1.openova.io`, `ns2.openova.io`, `ns3.openova.io` is anycast Hetzner Floating IPs spread across regions. The Crossplane `XHetznerFloatingIP` composite resource definition that would allocate these does not yet exist in `platform/crossplane/compositions/` (the compositions currently authored cover Server, Network, Firewall, LoadBalancer, and Pool-Allocation). **Phase-0 stand-in:** A Kubernetes `Service` of `type=LoadBalancer` (rendered by `templates/anycast-endpoint.yaml`). On Hetzner-managed Sovereigns the Hetzner cloud-controller-manager allocates a public IPv4 automatically; on Contabo this falls back to NodePort + external-IP routing. **Cutover:** Once `xrd-floating-ip.yaml` + `composition-floating-ip.yaml` land in `platform/crossplane/compositions/`, bp-powerdns is bumped to 1.1.0 with `.Values.crossplane.floatingIP.enabled=true` flipped on by default. The placeholder Service stays in the chart but is disabled by setting `.Values.anycast.enabled=false` in cluster overlays. The follow-up issue tracking the composition is referenced from the gap comment in `templates/crossplane-floatingip.yaml`. --- ## Cluster manifests (private repo) The Flux-managed deployment lives in `clusters/contabo-mkt/apps/powerdns/` in `openova-private`: ``` clusters/contabo-mkt/apps/powerdns/ ├── kustomization.yaml # references the chart ├── helmrelease.yaml # pulls bp-powerdns:1.0.6 from ghcr.io/openova-io ├── helm-repository.yaml # OCI HelmRepository pointing at ghcr.io/openova-io ├── namespace.yaml # openova-system (already exists) ├── api-credentials-secret.yaml # ExternalSecret reading from openbao └── api-basicauth-secret.yaml # bcrypt(operator:) for the Traefik middleware ``` The HelmRelease's `values:` block carries cluster-specific overrides (replicaCount, dnsdist threshold, region, etc.). The base chart's defaults are sufficient for Contabo-mkt's first deployment. --- ## Acceptance (per #167) - [x] Wrapper chart `platform/powerdns/chart/` with Chart.yaml + values.yaml + blueprint.yaml + templates - [x] CNPG-backed `pdns-pg` cluster (separate from `pdm-pg`) - [x] DNSSEC enabled by default (ECDSAP256SHA256) - [x] lua-records enabled by default - [x] dnsdist companion with 100 qps default rate limit - [x] REST API at `pdns.openova.io/api` behind Traefik basicAuth - [x] CI publishes `bp-powerdns:1.0.6` cosign-signed + SBOM-attested (current chart version on main; see [`platform/powerdns/Chart.yaml`](../platform/powerdns/Chart.yaml)) - [x] Cluster manifest in private repo `clusters/contabo-mkt/apps/powerdns/` - [x] `kubectl get cluster pdns-pg -n openova-system` healthy — running today - [x] `kubectl get deploy powerdns -n openova-system` 1/1 ready — running today (replicaCount=1 on Contabo-mkt; default chart value is 3 for production Sovereigns) - [ ] `curl https://pdns.openova.io/api/v1/servers/localhost` returns JSON — verified post-Flux-reconcile - [ ] `dig @ns1.openova.io openova.io NS` returns the 3 NS records — DEFERRED (anycast composition gap, see "Anycast deferral") - [ ] `dig +dnssec` validates — DEFERRED (depends on parent-zone DS submission, separate ticket) --- ## Runbook — first deploy on Contabo-mkt ```bash # 1. Generate API key + webserver password (random 32 chars per INVIOLABLE-PRINCIPLES #10) api_key=$(python3 -c "import secrets,string; print(''.join(secrets.choice(string.ascii_letters+string.digits) for _ in range(48)))") ws_pw=$(python3 -c "import secrets,string; print(''.join(secrets.choice(string.ascii_letters+string.digits) for _ in range(32)))") # 2. Stage them in OpenBao (already deployed at openbao.openova.io) # — ExternalSecret in clusters/contabo-mkt/apps/powerdns/ pulls them into K8s # 3. Generate basicAuth bcrypt for the operator user op_pw=$(python3 -c "import secrets,string; print(''.join(secrets.choice(string.ascii_letters+string.digits) for _ in range(32)))") htpasswd -nbB operator "$op_pw" # → operator:$2y$05$ # 4. Apply (Flux reconciles automatically every 1m) git -C openova-private add clusters/contabo-mkt/apps/powerdns/ git commit -m "deploy(powerdns): initial bp-powerdns:1.0.6 on contabo-mkt" git push # 5. Wait for reconcile flux get helmrelease powerdns -n openova-system # 6. Verify kubectl get cluster pdns-pg -n openova-system kubectl get deploy powerdns -n openova-system curl -u "operator:$op_pw" -H "X-API-Key: $api_key" https://pdns.openova.io/api/v1/servers/localhost ``` --- ## References - [PowerDNS Authoritative — Settings](https://doc.powerdns.com/authoritative/settings.html) - [PowerDNS REST API](https://doc.powerdns.com/authoritative/http-api/) - [Lua records](https://doc.powerdns.com/authoritative/lua-records/index.html) - [dnsdist documentation](https://dnsdist.org/) - Issue [#167](https://github.com/openova-io/openova/issues/167) - Inviolable Principles — `docs/INVIOLABLE-PRINCIPLES.md` - Naming convention — `docs/NAMING-CONVENTION.md` *Part of [OpenOva Catalyst](https://openova.io). Read [Inviolable Principles](INVIOLABLE-PRINCIPLES.md) before any changes.*