fix(catalyst-api): seed sovereign-smtp-credentials Secret on freshly franchised Sovereigns (#883) (#905)
On a freshly franchised Sovereign the console-side magic-link / PIN email flow fails because there's no SMTP relay reachable in the cluster. Phase-1 architectural decision (founder-confirmed): the Sovereign Console relays mail through the mothership Stalwart at mail.openova.io:587 during initial provisioning. A Sovereign-local Stalwart-relay is Phase-2 work tracked separately. This PR teaches the catalyst-api Sovereign provisioner to seed the catalyst-system/sovereign-smtp-credentials Secret on the new cluster right after the cloud-init kubeconfig postback lands and BEFORE runPhase1Watch fires. The bp-catalyst-platform chart's auto-create step (#901) reads this Secret via Helm `lookup` when rendering the Sovereign-local catalyst-openova-kc-credentials Secret, so the chart-rendered bytes carry working SMTP submission credentials and the auth service's SMTP-PLAIN dial against mail.openova.io:587 succeeds on the first send-pin. What's seeded: Secret catalyst-system/sovereign-smtp-credentials smtp-user: <mothership CATALYST_SMTP_USER> smtp-pass: <mothership CATALYST_SMTP_PASS> The mothership catalyst-api Pod already has both env vars wired via secretKeyRef → catalyst-openova-kc-credentials in the catalyst namespace (chart api-deployment.yaml.679-740) — no new K8s read against the mothership API is needed. Idempotent: an already-existing sovereign-smtp-credentials Secret short-circuits to AlreadyExists. The helper does NOT update an existing Secret — operator-supplied bytes take precedence over mothership re-seed. This survives the kubeconfig PUT retry path, the kubeconfig-missing relaunch (#538), and operator manual replay during incident response. Failure modes are surfaced via the SSE event bus (sovereign-smtp-seed phase) so the wizard renders the seed outcome inline with helmwatch events. A failure does NOT abort Phase-1 — the chart's lookup will not find the Secret, the auth pod will log SMTP-refused on first send-pin (exactly the pre-fix behaviour), and the operator sees a loud warn at provision time rather than a silent "ready" with broken email. Per docs/INVIOLABLE-PRINCIPLES.md #10 (credential hygiene): the catalyst-api never logs the SMTP password. Logs include the deployment id, target namespace + secret name, and byte length — never the plaintext. Per #4 (never hardcode): namespace + secret name are fixed-by-chart- contract (#901); timeout is overridable via CATALYST_SOVEREIGN_SMTP_SEED_TIMEOUT. Tests: - skipped-no-env outcome when mothership env unset - happy path: Secret + Namespace created, data + labels + annotations verified - already-exists pre-Create: no overwrite of operator bytes - race during Create: AlreadyExists treated as success - client-build failure: ClientFailure outcome - api-failure on Get (non-NotFound): APIFailure outcome - emit event matrix: every outcome maps to expected level + substr Co-authored-by: hatiyildiz <hatiyildiz@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
e3mrah
GitHub
parent
368545369b
commit
7658f9d937
@ -237,6 +237,14 @@ type Handler struct {
|
||||
// inject stubs.
|
||||
smeTenantDeps SMETenantDeps
|
||||
|
||||
// ── Sovereign SMTP seed (issue #883) ────────────────────────────────────
|
||||
// sovereignSMTPSeedClientFactory — test-only override for building a
|
||||
// kubernetes.Interface from a kubeconfig YAML. Production wires
|
||||
// helmwatch.NewKubernetesClientFromKubeconfig. Tests inject a
|
||||
// closure returning a fake.NewSimpleClientset so the seed path is
|
||||
// exercised without standing up a real cluster.
|
||||
sovereignSMTPSeedClientFactory SovereignSMTPSeedClientFactory
|
||||
|
||||
// ── Multi-zone PowerDNS (issue #827, parent epic #825) ──────────────────
|
||||
// powerdnsZoneClient — narrow client for runtime parent-zone creation.
|
||||
// The bootstrap-kit's bp-powerdns Helm hook Job creates the operator's
|
||||
|
||||
@ -567,6 +567,23 @@ func (h *Handler) PutKubeconfig(w http.ResponseWriter, r *http.Request) {
|
||||
"relaunchAfterKubeconfigMissing", relaunchAfterTerminalKubeconfigMissing,
|
||||
)
|
||||
|
||||
// Issue #883 — seed the Sovereign-side
|
||||
// catalyst-system/sovereign-smtp-credentials Secret with the
|
||||
// mothership's SMTP submission credentials BEFORE Phase-1 watch
|
||||
// launches. The bp-catalyst-platform chart's auto-create step
|
||||
// (#901) runs Helm `lookup` against this Secret when rendering
|
||||
// the Sovereign-local catalyst-openova-kc-credentials Secret;
|
||||
// seeding now (kubeconfig present, bp-catalyst-platform not yet
|
||||
// installed) is exactly the window that makes the lookup land
|
||||
// real bytes instead of empty placeholders.
|
||||
//
|
||||
// The seed is best-effort: a failure here does NOT abort
|
||||
// Phase-1. PIN email delivery may degrade, but the Sovereign
|
||||
// itself still bootstraps. Outcome flows through the SSE event
|
||||
// bus so the wizard surfaces it inline with helmwatch events.
|
||||
seedOutcome := h.seedSovereignSMTPCredentials(r.Context(), dep, string(body))
|
||||
h.emitSovereignSMTPSeedEvent(dep, seedOutcome)
|
||||
|
||||
// Launch the helmwatch goroutine in the background. The PUT
|
||||
// returns immediately; per-component events flow via the SSE
|
||||
// stream the wizard already has open. The phase1Started guard
|
||||
|
||||
@ -0,0 +1,336 @@
|
||||
// Sovereign-side SMTP credential seeding (issue #883).
|
||||
//
|
||||
// Why this exists:
|
||||
//
|
||||
// On a freshly franchised Sovereign, console-side magic-link / PIN
|
||||
// email delivery fails because there's no SMTP relay reachable inside
|
||||
// the cluster: the bootstrap-kit doesn't deploy a Stalwart on the new
|
||||
// Sovereign, the Sovereign-local sme-secrets has empty SMTP_HOST/PORT/
|
||||
// FROM/USER/PASS, and services-auth defaults SMTP_HOST=localhost.
|
||||
//
|
||||
// Phase-1 architectural decision (founder-confirmed): during initial
|
||||
// provisioning the Sovereign Console relays mail through the
|
||||
// mothership Stalwart at mail.openova.io:587. A Sovereign-local
|
||||
// Stalwart-relay is Phase-2 work tracked separately.
|
||||
//
|
||||
// What this seeds:
|
||||
//
|
||||
// On the freshly-provisioned Sovereign (target cluster reached via
|
||||
// the cloud-init-postback kubeconfig), we create:
|
||||
//
|
||||
// Secret catalyst-system/sovereign-smtp-credentials
|
||||
// smtp-user: <mothership smtp-user>
|
||||
// smtp-pass: <mothership smtp-pass>
|
||||
//
|
||||
// The bp-catalyst-platform chart's auto-create step (#901) reads
|
||||
// this Secret via Helm `lookup` when rendering the Sovereign-local
|
||||
// `catalyst-openova-kc-credentials` Secret, so the chart-rendered
|
||||
// bytes carry working SMTP submission credentials and the auth
|
||||
// service's SMTP-PLAIN dial against mail.openova.io:587 succeeds
|
||||
// on first send-pin.
|
||||
//
|
||||
// Source of mothership SMTP creds:
|
||||
//
|
||||
// The mothership catalyst-api Pod already has CATALYST_SMTP_USER /
|
||||
// CATALYST_SMTP_PASS in its environment (chart api-deployment.yaml
|
||||
// wires them via secretKeyRef → catalyst-openova-kc-credentials in
|
||||
// the catalyst namespace). We read those env vars directly — no new
|
||||
// K8s read against the mothership API is needed.
|
||||
//
|
||||
// When this runs:
|
||||
//
|
||||
// PutKubeconfig (kubeconfig.go) calls SeedSovereignSMTPCredentials
|
||||
// AFTER the cloud-init kubeconfig postback has been persisted to
|
||||
// disk and BEFORE the runPhase1Watch goroutine fires. That window is
|
||||
// exactly the "Sovereign cluster up, bp-catalyst-platform not yet
|
||||
// installed" gap the chart's lookup needs.
|
||||
//
|
||||
// The seed step is idempotent: an already-existing
|
||||
// sovereign-smtp-credentials Secret is a no-op (the chart's lookup
|
||||
// will read whatever bytes are there). This survives:
|
||||
// - retry of the kubeconfig PUT (single-use bearer normally
|
||||
// prevents this; defensive anyway)
|
||||
// - the kubeconfig-missing relaunch path (#538)
|
||||
// - operator manual replay during incident response
|
||||
//
|
||||
// Failure modes (per docs/INVIOLABLE-PRINCIPLES.md #1: surface, never hide):
|
||||
//
|
||||
// - kubeconfig parse failure → seed skipped, warn-event emitted on
|
||||
// the deployment SSE bus, Phase-1 watch still launches. The
|
||||
// wizard surfaces the warning and the operator can manual-create
|
||||
// the Secret + retry send-pin once the Sovereign is up.
|
||||
// - mothership env vars empty → seed skipped, warn-event emitted.
|
||||
// Production catalyst-api always has both wired; CI / dev
|
||||
// environments run without them and surface the gap loudly.
|
||||
// - K8s API call failure (network blip, RBAC drift) → seed skipped,
|
||||
// warn-event emitted, Phase-1 continues. The chart's lookup will
|
||||
// not find the Secret; the auth pod will log SMTP-refused on
|
||||
// first send-pin, exactly the pre-fix behaviour. Better the
|
||||
// operator sees a loud warn at provision time than a silent
|
||||
// "ready" with broken email.
|
||||
//
|
||||
// Per docs/INVIOLABLE-PRINCIPLES.md #10 (credential hygiene): the
|
||||
// catalyst-api never logs the SMTP password. Logs include the
|
||||
// deployment id, the target namespace + secret name, byte length of
|
||||
// each value, and outcome class — never the plaintext.
|
||||
package handler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
|
||||
"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/helmwatch"
|
||||
"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/provisioner"
|
||||
)
|
||||
|
||||
// sovereignSMTPSeedNamespace — destination namespace on the new
|
||||
// Sovereign for the Secret. Hardcoded to catalyst-system because
|
||||
// bp-catalyst-platform installs into that namespace; the chart's
|
||||
// lookup runs from that namespace too. The destination is part of
|
||||
// the chart contract (#901), not configuration the operator should
|
||||
// override.
|
||||
const sovereignSMTPSeedNamespace = "catalyst-system"
|
||||
|
||||
// sovereignSMTPSeedSecretName — name of the Secret the chart's
|
||||
// lookup reads. Same fixed-by-contract reasoning.
|
||||
const sovereignSMTPSeedSecretName = "sovereign-smtp-credentials"
|
||||
|
||||
// sovereignSMTPSeedTimeout — bound on the seed step's K8s calls.
|
||||
// 30 seconds is generous for the LB-fronted API server first contact
|
||||
// after kubeconfig postback. Override via
|
||||
// CATALYST_SOVEREIGN_SMTP_SEED_TIMEOUT (per principle #4).
|
||||
const sovereignSMTPSeedTimeoutEnv = "CATALYST_SOVEREIGN_SMTP_SEED_TIMEOUT"
|
||||
const defaultSovereignSMTPSeedTimeout = 30 * time.Second
|
||||
|
||||
// sovereignSMTPSeedPhase — SSE phase tag emitted on the deployment
|
||||
// event bus. Mirrors the helmwatch PhaseComponent style so the
|
||||
// wizard's reducer renders these events alongside the per-component
|
||||
// install lines.
|
||||
const sovereignSMTPSeedPhase = "sovereign-smtp-seed"
|
||||
|
||||
// SovereignSMTPSeedClientFactory — test-only override for building a
|
||||
// kubernetes.Interface from a kubeconfig YAML. Production wires
|
||||
// helmwatch.NewKubernetesClientFromKubeconfig; tests inject a closure
|
||||
// returning a fake.NewSimpleClientset so the seed path is exercised
|
||||
// without standing up a real cluster.
|
||||
type SovereignSMTPSeedClientFactory func(kubeconfigYAML string) (kubernetes.Interface, error)
|
||||
|
||||
// SetSovereignSMTPSeedClientFactory wires a test-only factory.
|
||||
// Production leaves this nil and the helper falls back to
|
||||
// helmwatch.NewKubernetesClientFromKubeconfig.
|
||||
func (h *Handler) SetSovereignSMTPSeedClientFactory(f SovereignSMTPSeedClientFactory) {
|
||||
h.sovereignSMTPSeedClientFactory = f
|
||||
}
|
||||
|
||||
// SovereignSMTPSeedOutcome — terminal classification of one seed
|
||||
// attempt. The SSE event message includes this so the wizard's
|
||||
// reducer can colour-code the line; tests assert on it without
|
||||
// scraping the human-readable Message.
|
||||
type SovereignSMTPSeedOutcome string
|
||||
|
||||
const (
|
||||
SovereignSMTPSeedOutcomeCreated SovereignSMTPSeedOutcome = "created"
|
||||
SovereignSMTPSeedOutcomeAlreadyExists SovereignSMTPSeedOutcome = "already-exists"
|
||||
SovereignSMTPSeedOutcomeSkippedNoEnv SovereignSMTPSeedOutcome = "skipped-no-env"
|
||||
SovereignSMTPSeedOutcomeClientFailure SovereignSMTPSeedOutcome = "client-failure"
|
||||
SovereignSMTPSeedOutcomeAPIFailure SovereignSMTPSeedOutcome = "api-failure"
|
||||
)
|
||||
|
||||
// seedSovereignSMTPCredentials creates the
|
||||
// catalyst-system/sovereign-smtp-credentials Secret on the freshly-
|
||||
// provisioned Sovereign. The function is exported through the
|
||||
// instance method so PutKubeconfig can drive it.
|
||||
//
|
||||
// Returns the terminal outcome so the caller can choose to emit an
|
||||
// SSE event (PutKubeconfig does) without coupling this helper to the
|
||||
// emit machinery.
|
||||
//
|
||||
// Idempotent: an existing Secret short-circuits to AlreadyExists. The
|
||||
// helper does NOT update an existing Secret — operator-supplied bytes
|
||||
// take precedence over mothership re-seed. (A future Phase-2 patch
|
||||
// can rotate via UPDATE if needed.)
|
||||
func (h *Handler) seedSovereignSMTPCredentials(ctx context.Context, dep *Deployment, kubeconfigYAML string) SovereignSMTPSeedOutcome {
|
||||
smtpUser := os.Getenv("CATALYST_SMTP_USER")
|
||||
smtpPass := os.Getenv("CATALYST_SMTP_PASS")
|
||||
if smtpUser == "" || smtpPass == "" {
|
||||
// Production catalyst-api always has both wired via
|
||||
// secretKeyRef (chart api-deployment.yaml). An empty value
|
||||
// here is a CI / dev environment OR a misconfigured
|
||||
// catalyst-openova-kc-credentials Secret on the mothership.
|
||||
// Either way the new Sovereign cannot relay mail without
|
||||
// these credentials; surface loud + skip rather than seed
|
||||
// an empty Secret that pretends to work.
|
||||
h.log.Warn("sovereign smtp seed: mothership SMTP env unset; skipping seed",
|
||||
"deploymentID", dep.ID,
|
||||
"hasUser", smtpUser != "",
|
||||
"hasPass", smtpPass != "",
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeSkippedNoEnv
|
||||
}
|
||||
|
||||
factory := h.sovereignSMTPSeedClientFactory
|
||||
if factory == nil {
|
||||
factory = helmwatch.NewKubernetesClientFromKubeconfig
|
||||
}
|
||||
clientset, err := factory(kubeconfigYAML)
|
||||
if err != nil {
|
||||
h.log.Error("sovereign smtp seed: build kubernetes client from kubeconfig failed",
|
||||
"deploymentID", dep.ID,
|
||||
"err", err,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeClientFailure
|
||||
}
|
||||
|
||||
timeout := defaultSovereignSMTPSeedTimeout
|
||||
if v, _ := time.ParseDuration(os.Getenv(sovereignSMTPSeedTimeoutEnv)); v > 0 {
|
||||
timeout = v
|
||||
}
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
// Check existence first so we never accidentally overwrite an
|
||||
// operator-supplied Secret. apierrors.IsNotFound is the only
|
||||
// "proceed with create" branch; any other error is a hard fail
|
||||
// (RBAC drift, API server unreachable, malformed kubeconfig).
|
||||
_, getErr := clientset.CoreV1().Secrets(sovereignSMTPSeedNamespace).Get(cctx, sovereignSMTPSeedSecretName, metav1.GetOptions{})
|
||||
if getErr == nil {
|
||||
h.log.Info("sovereign smtp seed: target Secret already exists; skipping create (idempotent)",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"name", sovereignSMTPSeedSecretName,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeAlreadyExists
|
||||
}
|
||||
if !apierrors.IsNotFound(getErr) {
|
||||
h.log.Error("sovereign smtp seed: pre-create Get failed",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"name", sovereignSMTPSeedSecretName,
|
||||
"err", getErr,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeAPIFailure
|
||||
}
|
||||
|
||||
// catalyst-system may not exist yet (Phase-1 watch hasn't fired,
|
||||
// bp-catalyst-platform hasn't installed). Pre-create the namespace
|
||||
// so the Secret create succeeds. apierrors.IsAlreadyExists is the
|
||||
// idempotent branch; any other error is a hard fail.
|
||||
if _, nsErr := clientset.CoreV1().Namespaces().Create(cctx, &corev1.Namespace{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: sovereignSMTPSeedNamespace},
|
||||
}, metav1.CreateOptions{}); nsErr != nil && !apierrors.IsAlreadyExists(nsErr) {
|
||||
h.log.Error("sovereign smtp seed: pre-create Namespace failed",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"err", nsErr,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeAPIFailure
|
||||
}
|
||||
|
||||
secret := &corev1.Secret{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: sovereignSMTPSeedSecretName,
|
||||
Namespace: sovereignSMTPSeedNamespace,
|
||||
Labels: map[string]string{
|
||||
"app.kubernetes.io/managed-by": "catalyst-api",
|
||||
"app.kubernetes.io/part-of": "sovereign-bootstrap",
|
||||
"catalyst.openova.io/seed": "smtp-credentials",
|
||||
},
|
||||
Annotations: map[string]string{
|
||||
// Trace: which mothership deployment minted this
|
||||
// Secret. Useful for incident-response when an
|
||||
// operator wonders why a Sovereign has unexpected
|
||||
// SMTP credentials. The value is a UUID — not a
|
||||
// credential — and is safe to log.
|
||||
"catalyst.openova.io/seeded-by-deployment-id": dep.ID,
|
||||
// Phase-1 marker: when Phase-2 spins up per-Sovereign
|
||||
// Stalwart-relay, the migration script can target
|
||||
// every Secret with this annotation for rotation.
|
||||
"catalyst.openova.io/seed-phase": "phase-1-mothership-relay",
|
||||
},
|
||||
},
|
||||
Type: corev1.SecretTypeOpaque,
|
||||
Data: map[string][]byte{
|
||||
"smtp-user": []byte(smtpUser),
|
||||
"smtp-pass": []byte(smtpPass),
|
||||
},
|
||||
}
|
||||
if _, createErr := clientset.CoreV1().Secrets(sovereignSMTPSeedNamespace).Create(cctx, secret, metav1.CreateOptions{}); createErr != nil {
|
||||
// Race-window guard: a parallel writer (operator manual
|
||||
// kubectl apply, retry of this PUT) raced our Get→Create.
|
||||
// Treat AlreadyExists as success — the bytes are there.
|
||||
if apierrors.IsAlreadyExists(createErr) {
|
||||
h.log.Info("sovereign smtp seed: target Secret created concurrently; treating as success",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"name", sovereignSMTPSeedSecretName,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeAlreadyExists
|
||||
}
|
||||
h.log.Error("sovereign smtp seed: Create Secret failed",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"name", sovereignSMTPSeedSecretName,
|
||||
"err", createErr,
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeAPIFailure
|
||||
}
|
||||
|
||||
// Per docs/INVIOLABLE-PRINCIPLES.md #10: byte lengths only,
|
||||
// never plaintext.
|
||||
h.log.Info("sovereign smtp seed: created",
|
||||
"deploymentID", dep.ID,
|
||||
"namespace", sovereignSMTPSeedNamespace,
|
||||
"name", sovereignSMTPSeedSecretName,
|
||||
"smtpUserBytes", len(smtpUser),
|
||||
"smtpPassBytes", len(smtpPass),
|
||||
)
|
||||
return SovereignSMTPSeedOutcomeCreated
|
||||
}
|
||||
|
||||
// emitSovereignSMTPSeedEvent — pushes a typed event onto the
|
||||
// deployment SSE buffer so the wizard's reducer can render the seed
|
||||
// outcome inline with the per-component HelmRelease events.
|
||||
//
|
||||
// Pulled out into its own helper so PutKubeconfig has a single call
|
||||
// site and the message phrasing stays consistent across outcomes.
|
||||
func (h *Handler) emitSovereignSMTPSeedEvent(dep *Deployment, outcome SovereignSMTPSeedOutcome) {
|
||||
level := "info"
|
||||
var msg string
|
||||
switch outcome {
|
||||
case SovereignSMTPSeedOutcomeCreated:
|
||||
msg = fmt.Sprintf("Sovereign SMTP seed: created Secret %s/%s on the new Sovereign so the Phase-1 console relays mail through mail.openova.io:587 until Phase-2 spins up the per-Sovereign Stalwart-relay.", sovereignSMTPSeedNamespace, sovereignSMTPSeedSecretName)
|
||||
case SovereignSMTPSeedOutcomeAlreadyExists:
|
||||
msg = fmt.Sprintf("Sovereign SMTP seed: Secret %s/%s already present on the new Sovereign — leaving operator-supplied bytes in place (idempotent).", sovereignSMTPSeedNamespace, sovereignSMTPSeedSecretName)
|
||||
case SovereignSMTPSeedOutcomeSkippedNoEnv:
|
||||
level = "warn"
|
||||
msg = "Sovereign SMTP seed: SKIPPED — mothership catalyst-api has empty CATALYST_SMTP_USER / CATALYST_SMTP_PASS. The new Sovereign's bp-catalyst-platform install will continue, but PIN email delivery will fail until an operator manually creates Secret " + sovereignSMTPSeedNamespace + "/" + sovereignSMTPSeedSecretName + " on the new cluster."
|
||||
case SovereignSMTPSeedOutcomeClientFailure:
|
||||
level = "warn"
|
||||
msg = "Sovereign SMTP seed: FAILED to build a Kubernetes client from the cloud-init kubeconfig — Phase-1 watch will still attempt to launch. PIN email delivery will likely fail until an operator manually creates Secret " + sovereignSMTPSeedNamespace + "/" + sovereignSMTPSeedSecretName + " on the new cluster."
|
||||
case SovereignSMTPSeedOutcomeAPIFailure:
|
||||
level = "warn"
|
||||
msg = "Sovereign SMTP seed: FAILED talking to the new Sovereign's API server (RBAC drift, network blip, or LB still reconciling). Phase-1 watch will still attempt to launch. Operator can manually create Secret " + sovereignSMTPSeedNamespace + "/" + sovereignSMTPSeedSecretName + " on the new cluster if PIN email delivery fails after provisioning completes."
|
||||
default:
|
||||
// Defensive default — unknown outcome should never happen,
|
||||
// but keep the SSE buffer in a consistent state.
|
||||
level = "warn"
|
||||
msg = fmt.Sprintf("Sovereign SMTP seed: unknown outcome %q (programming error in catalyst-api).", outcome)
|
||||
}
|
||||
|
||||
h.emitWatchEvent(dep, provisioner.Event{
|
||||
Time: time.Now().UTC().Format(time.RFC3339),
|
||||
Phase: sovereignSMTPSeedPhase,
|
||||
Level: level,
|
||||
Message: msg,
|
||||
})
|
||||
}
|
||||
@ -0,0 +1,290 @@
|
||||
// Tests for Sovereign-side SMTP credential seeding (issue #883).
|
||||
//
|
||||
// Coverage:
|
||||
// - mothership env unset → SkippedNoEnv outcome, no Secret created
|
||||
// - happy path → Created outcome, Secret data matches mothership env
|
||||
// bytes verbatim, Namespace pre-created, labels/annotations stamped
|
||||
// - already-exists pre-Create → AlreadyExists, no overwrite
|
||||
// - already-exists race during Create → AlreadyExists, no overwrite
|
||||
// - client-build failure (factory returns err) → ClientFailure
|
||||
// - api-failure on Get (non-NotFound) → APIFailure
|
||||
// - emit event matrix → every outcome maps to a non-empty SSE message
|
||||
//
|
||||
// Tests use kfake.NewSimpleClientset and inject via
|
||||
// SetSovereignSMTPSeedClientFactory so no real cluster is needed.
|
||||
package handler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/runtime"
|
||||
"k8s.io/apimachinery/pkg/runtime/schema"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
kfake "k8s.io/client-go/kubernetes/fake"
|
||||
ktesting "k8s.io/client-go/testing"
|
||||
|
||||
"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/provisioner"
|
||||
)
|
||||
|
||||
// seedTestDeployment returns a *Deployment shaped for emit-event
|
||||
// assertions. eventsCh is buffered so non-blocking sends in
|
||||
// emitWatchEvent don't drop; eventsBuf is the durable buffer the
|
||||
// test reads under dep.mu.
|
||||
func seedTestDeployment(id string) *Deployment {
|
||||
return &Deployment{
|
||||
ID: id,
|
||||
Status: "phase1-watching",
|
||||
StartedAt: time.Now(),
|
||||
eventsCh: make(chan provisioner.Event, 256),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
func setSeedSMTPEnv(t *testing.T, user, pass string) {
|
||||
t.Helper()
|
||||
t.Setenv("CATALYST_SMTP_USER", user)
|
||||
t.Setenv("CATALYST_SMTP_PASS", pass)
|
||||
}
|
||||
|
||||
func unsetSeedSMTPEnv(t *testing.T) {
|
||||
t.Helper()
|
||||
t.Setenv("CATALYST_SMTP_USER", "")
|
||||
t.Setenv("CATALYST_SMTP_PASS", "")
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_SkippedNoEnv — mothership env
|
||||
// vars empty → SkippedNoEnv, no factory call, no Secret created.
|
||||
func TestSeedSovereignSMTPCredentials_SkippedNoEnv(t *testing.T) {
|
||||
unsetSeedSMTPEnv(t)
|
||||
|
||||
factoryCalled := false
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
factoryCalled = true
|
||||
return nil, errors.New("should not be called")
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-noenv")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "ignored-kubeconfig")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeSkippedNoEnv {
|
||||
t.Errorf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeSkippedNoEnv)
|
||||
}
|
||||
if factoryCalled {
|
||||
t.Errorf("client factory must not be called when env is unset")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_HappyPath — env present, no
|
||||
// existing Secret → Created. Secret + Namespace exist on the fake
|
||||
// clientset with the expected data, labels, and annotations.
|
||||
func TestSeedSovereignSMTPCredentials_HappyPath(t *testing.T) {
|
||||
setSeedSMTPEnv(t, "noreply@openova.io", "p455w0rd-bytes-here-not-real")
|
||||
|
||||
core := kfake.NewSimpleClientset()
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
return core, nil
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-happy")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "kubeconfig-yaml-bytes")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeCreated {
|
||||
t.Fatalf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeCreated)
|
||||
}
|
||||
|
||||
ns, err := core.CoreV1().Namespaces().Get(context.Background(), sovereignSMTPSeedNamespace, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("Namespace not created: %v", err)
|
||||
}
|
||||
if ns.Name != sovereignSMTPSeedNamespace {
|
||||
t.Errorf("Namespace name = %q, want %q", ns.Name, sovereignSMTPSeedNamespace)
|
||||
}
|
||||
|
||||
got, err := core.CoreV1().Secrets(sovereignSMTPSeedNamespace).Get(context.Background(), sovereignSMTPSeedSecretName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("Secret not created: %v", err)
|
||||
}
|
||||
if string(got.Data["smtp-user"]) != "noreply@openova.io" {
|
||||
t.Errorf("smtp-user data length = %d, want %d", len(got.Data["smtp-user"]), len("noreply@openova.io"))
|
||||
}
|
||||
if string(got.Data["smtp-pass"]) != "p455w0rd-bytes-here-not-real" {
|
||||
// Per principle #10, never print the password in logs. Length
|
||||
// comparison is enough to flag a regression without leaking.
|
||||
t.Errorf("smtp-pass data length = %d, want %d", len(got.Data["smtp-pass"]), len("p455w0rd-bytes-here-not-real"))
|
||||
}
|
||||
if got.Type != corev1.SecretTypeOpaque {
|
||||
t.Errorf("Secret type = %q, want %q", got.Type, corev1.SecretTypeOpaque)
|
||||
}
|
||||
if got.Labels["app.kubernetes.io/managed-by"] != "catalyst-api" {
|
||||
t.Errorf("missing managed-by label, got: %#v", got.Labels)
|
||||
}
|
||||
if got.Annotations["catalyst.openova.io/seeded-by-deployment-id"] != "dep-happy" {
|
||||
t.Errorf("seeded-by annotation = %q, want %q", got.Annotations["catalyst.openova.io/seeded-by-deployment-id"], "dep-happy")
|
||||
}
|
||||
if got.Annotations["catalyst.openova.io/seed-phase"] != "phase-1-mothership-relay" {
|
||||
t.Errorf("seed-phase annotation = %q, want %q", got.Annotations["catalyst.openova.io/seed-phase"], "phase-1-mothership-relay")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_AlreadyExistsPreCreate — pre-existing
|
||||
// Secret bytes are NOT overwritten; outcome is AlreadyExists.
|
||||
func TestSeedSovereignSMTPCredentials_AlreadyExistsPreCreate(t *testing.T) {
|
||||
setSeedSMTPEnv(t, "mothership-user", "mothership-pass")
|
||||
|
||||
preExisting := &corev1.Secret{
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: sovereignSMTPSeedSecretName,
|
||||
Namespace: sovereignSMTPSeedNamespace,
|
||||
},
|
||||
Data: map[string][]byte{
|
||||
"smtp-user": []byte("operator-supplied-user"),
|
||||
"smtp-pass": []byte("operator-supplied-pass"),
|
||||
},
|
||||
}
|
||||
preNS := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: sovereignSMTPSeedNamespace}}
|
||||
core := kfake.NewSimpleClientset(preNS, preExisting)
|
||||
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
return core, nil
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-exists")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "kubeconfig")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeAlreadyExists {
|
||||
t.Errorf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeAlreadyExists)
|
||||
}
|
||||
|
||||
got, err := core.CoreV1().Secrets(sovereignSMTPSeedNamespace).Get(context.Background(), sovereignSMTPSeedSecretName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("Secret unexpectedly missing: %v", err)
|
||||
}
|
||||
if string(got.Data["smtp-user"]) != "operator-supplied-user" {
|
||||
t.Errorf("smtp-user was overwritten, length = %d", len(got.Data["smtp-user"]))
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_RaceOnCreate — Get returns
|
||||
// NotFound but Create races a parallel writer and returns
|
||||
// AlreadyExists. Outcome is AlreadyExists; no error surfaced.
|
||||
func TestSeedSovereignSMTPCredentials_RaceOnCreate(t *testing.T) {
|
||||
setSeedSMTPEnv(t, "u", "p")
|
||||
|
||||
core := kfake.NewSimpleClientset()
|
||||
// Inject a reactor on Secrets create that returns AlreadyExists
|
||||
// the first time. The fake client's default Get returns
|
||||
// NotFound when no object exists, so we hit the create branch.
|
||||
core.PrependReactor("create", "secrets", func(action ktesting.Action) (bool, runtime.Object, error) {
|
||||
gr := schema.GroupResource{Group: "", Resource: "secrets"}
|
||||
return true, nil, apierrors.NewAlreadyExists(gr, sovereignSMTPSeedSecretName)
|
||||
})
|
||||
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
return core, nil
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-race")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "kubeconfig")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeAlreadyExists {
|
||||
t.Errorf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeAlreadyExists)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_ClientFailure — the kubeconfig
|
||||
// parser returns an error → ClientFailure outcome, no Secret create
|
||||
// attempted on any clientset.
|
||||
func TestSeedSovereignSMTPCredentials_ClientFailure(t *testing.T) {
|
||||
setSeedSMTPEnv(t, "u", "p")
|
||||
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
return nil, errors.New("malformed kubeconfig YAML")
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-client-fail")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "garbage")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeClientFailure {
|
||||
t.Errorf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeClientFailure)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeedSovereignSMTPCredentials_APIFailureOnGet — Get returns a
|
||||
// non-NotFound error (e.g. RBAC drift, network blip) → APIFailure
|
||||
// outcome, no overwrite of any pre-existing Secret.
|
||||
func TestSeedSovereignSMTPCredentials_APIFailureOnGet(t *testing.T) {
|
||||
setSeedSMTPEnv(t, "u", "p")
|
||||
|
||||
core := kfake.NewSimpleClientset()
|
||||
core.PrependReactor("get", "secrets", func(action ktesting.Action) (bool, runtime.Object, error) {
|
||||
return true, nil, errors.New("forbidden: secrets.get")
|
||||
})
|
||||
|
||||
h := &Handler{log: silentLogger()}
|
||||
h.SetSovereignSMTPSeedClientFactory(func(string) (kubernetes.Interface, error) {
|
||||
return core, nil
|
||||
})
|
||||
|
||||
dep := seedTestDeployment("dep-api-fail")
|
||||
outcome := h.seedSovereignSMTPCredentials(context.Background(), dep, "kubeconfig")
|
||||
|
||||
if outcome != SovereignSMTPSeedOutcomeAPIFailure {
|
||||
t.Errorf("outcome = %q, want %q", outcome, SovereignSMTPSeedOutcomeAPIFailure)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEmitSovereignSMTPSeedEvent_MessageMatrix — every outcome maps
|
||||
// to a non-empty SSE message with the right level. The message
|
||||
// contract is what the wizard's reducer keys off.
|
||||
func TestEmitSovereignSMTPSeedEvent_MessageMatrix(t *testing.T) {
|
||||
cases := []struct {
|
||||
outcome SovereignSMTPSeedOutcome
|
||||
wantLevel string
|
||||
wantSubstr string
|
||||
}{
|
||||
{SovereignSMTPSeedOutcomeCreated, "info", "created Secret catalyst-system/sovereign-smtp-credentials"},
|
||||
{SovereignSMTPSeedOutcomeAlreadyExists, "info", "already present"},
|
||||
{SovereignSMTPSeedOutcomeSkippedNoEnv, "warn", "SKIPPED"},
|
||||
{SovereignSMTPSeedOutcomeClientFailure, "warn", "build a Kubernetes client"},
|
||||
{SovereignSMTPSeedOutcomeAPIFailure, "warn", "talking to the new Sovereign"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.outcome), func(t *testing.T) {
|
||||
h := &Handler{log: silentLogger()}
|
||||
dep := seedTestDeployment("emit-" + string(tc.outcome))
|
||||
h.emitSovereignSMTPSeedEvent(dep, tc.outcome)
|
||||
|
||||
dep.mu.Lock()
|
||||
defer dep.mu.Unlock()
|
||||
var found *provisioner.Event
|
||||
for i := range dep.eventsBuf {
|
||||
if dep.eventsBuf[i].Phase == sovereignSMTPSeedPhase {
|
||||
found = &dep.eventsBuf[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if found == nil {
|
||||
t.Fatalf("no %s event in eventsBuf; got=%+v", sovereignSMTPSeedPhase, dep.eventsBuf)
|
||||
}
|
||||
if found.Level != tc.wantLevel {
|
||||
t.Errorf("level = %q, want %q", found.Level, tc.wantLevel)
|
||||
}
|
||||
if !strings.Contains(found.Message, tc.wantSubstr) {
|
||||
t.Errorf("message %q missing substring %q", found.Message, tc.wantSubstr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user