608db53a25
## Root cause (live on otech116 2026-05-05 14:38) After the #968 fix shipped (0.1.19), the cutover engine reached Step-7 (87%) successfully — Step-01..07 all completed. Then Step-08 (egress- block-test) caught 38/38 HelmRepositories had reverted to upstream: ``` external HelmRepositories still pointing at ghcr.io/openova-io: 38 OFFENDER flux-system/bp-cilium=oci://ghcr.io/openova-io ... (37 more) FAIL — at least one HelmRepository did not pivot ``` But Step-06's job logs say: ``` [helmrepository-patches] OK bp-cilium -> oci://harbor.otech116.omani.works/openova-io ... (37 more OK) ok=38 skip=0 fail=0 ``` So Step-06 thought it succeeded — and it had, momentarily. But then the bootstrap-kit Kustomization (which had successfully pivoted to local Gitea via Step-05) reconciled its YAML from local Gitea, where the YAML still declared `url: oci://ghcr.io/openova-io`. Within ~30s every kubectl patch was undone. The cutover engine then aborted at Step-8 verification. ## Fix Step-06 now runs in two phases: 1. **Live K8s patches** (existing behaviour) — flips spec.url on every HelmRepository immediately. Useful for the cluster between cutover and the next reconcile. 2. **NEW — Push YAML edit to local Gitea** — clones `openova/openova` from the local Gitea over basic-auth, sed-rewrites every `clusters/_template/bootstrap-kit/*.yaml` declaration of `url: oci://ghcr.io/openova-io` → `oci://harbor.<sov-fqdn>/openova-io`, commits with a clear message, pushes back. Subsequent reconciles see local Harbor as the steady-state. After the push, the script annotates `flux-system/openova` GitRepository to trigger immediate reconciliation so the new YAML lands without waiting for the polling interval. ## Image change Step-06 image bumped from `bitnami/kubectl:1.31.4` to `alpine/k8s:1.31.4` because the new phase needs both `kubectl` and `git` in one image (verified live on otech116 — both binaries present). ## Acceptance gate Test case 16 added to cutover-contract.sh — guards against future regressions that remove the `git clone`, the `git push origin main`, or the `clusters/_template/bootstrap-kit` target dir reference. ## Live verification Will fire on otech117 (next provision). Expected: - Step-06 logs `cloning gitea-http.gitea.../openova/openova.git` then `pushed to ...` - Step-08 verify PASSES (38/38 HelmRepositories pivoted in K8s + Gitea) - self-sovereign-cutover-status `cutoverComplete: "true"` - Egress block to ghcr.io safely activates Co-authored-by: e3mrah <ebaysal@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
71 lines
3.3 KiB
YAML
71 lines
3.3 KiB
YAML
apiVersion: v2
|
|
name: bp-self-sovereign-cutover
|
|
version: 0.1.20
|
|
description: |
|
|
Catalyst Self-Sovereignty Cutover Blueprint. Installs DORMANT — this
|
|
chart ships eight step ConfigMaps (PodSpec ConfigMaps, one per step),
|
|
the registry-pivot DaemonSet, the mutable cutover-status ConfigMap,
|
|
plus ServiceAccount/RBAC scoped to the cutover work. NO Helm
|
|
post-install hooks fire any of the steps; the catalyst-api cutover
|
|
endpoint (issue #792) reads each step ConfigMap by label selector,
|
|
stamps a real batch/v1 Job from the embedded PodSpec, and updates
|
|
the status ConfigMap as each step completes.
|
|
|
|
Trigger semantics (post-handover, per #790 scope-correction comment):
|
|
the operator clicks "Achieve True Sovereignty" in admin console (or
|
|
catalyst-api auto-fires after first successful operator login on the
|
|
new Sovereign). The handed-over Sovereign starts soft-tethered to
|
|
harbor.openova.io for image-pull rate-limit safety, then becomes
|
|
hard-independent only after this cutover sequence completes.
|
|
|
|
Step inventory (label app.kubernetes.io/component=cutover-step):
|
|
01 cutover-step-01-gitea-mirror mode=job
|
|
git push --mirror github.com/openova-io/openova into local Gitea.
|
|
02 cutover-step-02-harbor-projects mode=job
|
|
Create 7 proxy-cache projects in local Harbor.
|
|
03 cutover-step-03-harbor-prewarm mode=job
|
|
HEAD-pull every image referenced by the bootstrap-kit through
|
|
the proxy-cache.
|
|
04 cutover-step-04-registry-pivot mode=daemonset-wait
|
|
DaemonSet runs on every node + writes /etc/rancher/k3s/registries.yaml
|
|
v2 once the status ConfigMap key registriesYamlActive flips to v2.
|
|
05 cutover-step-05-flux-gitrepository-patch mode=job
|
|
Patch flux-system/openova GitRepository.url → local Gitea.
|
|
06 cutover-step-06-helmrepository-patches mode=job
|
|
Patch every OCI HelmRepository → oci://harbor.<sov-fqdn>/openova-io.
|
|
07 cutover-step-07-catalyst-api-env-patch mode=job
|
|
kubectl set env deploy/catalyst-api CATALYST_GITOPS_REPO_URL=...
|
|
08 cutover-step-08-egress-block-test mode=job
|
|
Apply CiliumNetworkPolicy denying egress to github.com / ghcr.io
|
|
/ harbor.openova.io for 10 min, assert all reconciles stay Ready.
|
|
|
|
Plus:
|
|
self-sovereign-cutover-status ConfigMap
|
|
Initial state machine state (cutoverComplete: "false" + per-step
|
|
empty fields). catalyst-api updates this ConfigMap as each step
|
|
advances.
|
|
registry-pivot DaemonSet
|
|
Always-on per-node reconcile loop; idempotent.
|
|
bp-self-sovereign-cutover-runner ServiceAccount + ClusterRole +
|
|
ClusterRoleBinding for the
|
|
stamped Jobs and the DaemonSet.
|
|
|
|
This Blueprint legitimately ships NO upstream subchart — the entire
|
|
payload is Catalyst-authored ConfigMaps, ServiceAccount, RBAC, and one
|
|
DaemonSet. The blueprint-release CI hollow-chart guard reads the
|
|
annotation below and skips the dependencies-required rule.
|
|
type: application
|
|
keywords:
|
|
- catalyst
|
|
- blueprint
|
|
- self-sovereignty
|
|
- cutover
|
|
- gitea
|
|
- harbor
|
|
maintainers:
|
|
- name: OpenOva Catalyst
|
|
email: catalyst@openova.io
|
|
|
|
annotations:
|
|
catalyst.openova.io/no-upstream: "true"
|