fix(bp-stalwart-tenant): unbootable on fresh tenants — values shape, missing admin Secret, sec ctx (#898) (#904)

Three fixes that left bp-stalwart-tenant 0.1.0 unable to come up on a
freshly-franchised SME tenant. All surfaced on the otech103 alice
tenant during the Phase-1 DoD sweep.

1. Tenant-domain values shape (HelmRelease render error)

   The 0.1.0 chart referenced `.Values.domain.primary` in five
   templates. The live HR on otech103 had `values.domain:
   acme.omani.works` (a string), emitted by a pre-#897 catalyst-api
   build, so every reconcile died with:

     can't evaluate field primary in type interface {}

   Added `bp-stalwart-tenant.tenantDomain` + `tenantMode` helpers
   that resolve in priority order:

     1. `tenant.domain`        (forward-looking flat shape)
     2. `domain.primary`       (canonical post-#897 map shape)
     3. `domain` (string)      (legacy pre-#897 shape — back-compat)

   Returns "" smoke-render-safe; per-template gates skip when empty.

2. Missing stalwart-admin Secret

   deployment.yaml + mailbox-provision-job.yaml reference a Secret
   key `ADMIN_PASSWORD` on `.Values.admin.secretName`. The 0.1.0
   chart only emitted an ExternalSecret, and only when
   `admin.externalSecret.remoteRef.key` was non-empty (smoke-render
   concession). Fresh tenants land in CreateContainerConfigError.

   Added `templates/admin-secret.yaml` mirroring marketplace-api/
   secret.yaml (#887): random 32-char ADMIN_PASSWORD generated by
   sprig randAlphaNum, persisted across reconcile via lookup,
   helm.sh/resource-policy: keep so reinstall picks it back up.
   Auto-disabled when an authoritative ExternalSecret is wired —
   no double-bind between two controllers.

3. Pod sec ctx vs. upstream image's file capabilities

   `getcap docker.io/stalwartlabs/stalwart:v0.16.3 /usr/local/bin/
   stalwart` reports `cap_net_bind_service=ep`. The image creates
   user `stalwart` at UID 2000 and the binary IS the entrypoint
   (no demotion script). The 0.1.0 chart ran as UID 65534 with
   `drop: ALL` — kernel refuses to elevate file caps with empty
   bounding set, so exec failed with `operation not permitted`.

   Aligned to image's native UID 2000, kept `drop: ALL` and added
   `NET_BIND_SERVICE` explicitly. fsGroup 2000 ensures /opt/stalwart
   PVC is writable.

Other:
- Bumped Chart.yaml + blueprint.yaml to 0.1.1 (#817 alignment).
- configSchema in blueprint.yaml now permits the legacy + tenant
  shapes alongside the canonical map.
- mailboxProvisioner.setupJob.enabled defaults to false until the
  canonical stalwart-cli image is published (re-uses upstream
  stalwart container as fallback CLI host).

Acceptance: targeted at otech103 alice tenant
(sme-789ae512-bc0f-467c-a016-001f5496c403) where 0.1.0 reconciliation
fails with the value-shape error and the pod CrashLoops with `exec
... operation not permitted`. Verification on otech103 in #898.

Co-authored-by: hatiyildiz <hatiyildiz@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
e3mrah 2026-05-05 10:55:03 +04:00 committed by GitHub
parent cab0a30e4a
commit 368545369b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 290 additions and 33 deletions

View File

@ -6,7 +6,20 @@ metadata:
catalyst.openova.io/category: communication
catalyst.openova.io/section: pts-4-5-communication
spec:
version: 0.1.0
# 0.1.1 (#898) — three fixes from the otech103 fresh-tenant sweep:
# 1. `tenantDomain` helper accepts both `domain.primary` (canonical)
# and the legacy string-shape `domain: <fqdn>` emitted by
# pre-#897 catalyst-api builds, plus a forward-looking
# `tenant.domain` field (#898 brief).
# 2. auto-provisioned `stalwart-admin` Secret with random
# ADMIN_PASSWORD via lookup-persistence (no more
# CreateContainerConfigError on fresh tenants).
# 3. pod sec ctx aligned to the upstream image's UID 2000 +
# drop ALL caps then add NET_BIND_SERVICE so the binary's
# `cap_net_bind_service=ep` file-cap can elevate (was failing
# with `exec: operation not permitted` on every fresh tenant).
# Per #817 Chart.yaml version MUST equal blueprint.yaml spec.version.
version: 0.1.1
card:
title: Stalwart (per-tenant)
summary: |
@ -31,11 +44,17 @@ spec:
contact: catalyst@openova.io
configSchema:
type: object
required: [domain, keycloak]
required: [keycloak]
properties:
# `domain` is the canonical mail-domain shape (post-#897). The
# chart's `tenantDomain` helper additionally accepts a legacy
# string-shape `domain: <fqdn>` (pre-#897 catalyst-api builds)
# and a forward-looking `tenant.domain` block (#898). Either
# the chart-level `domain` block OR `tenant.domain` MUST resolve
# to a non-empty FQDN at runtime — the helper fails closed
# otherwise.
domain:
type: object
required: [primary]
properties:
primary:
type: string
@ -44,6 +63,19 @@ spec:
type: string
enum: [free-subdomain, byo]
default: free-subdomain
tenant:
type: object
description: |
Forward-looking tenant block (#898). When `tenant.domain` is
set it takes priority over the legacy `domain` block. Useful
for orchestrators that want a flat per-tenant values shape.
properties:
domain:
type: string
description: Mail domain (alternative to `domain.primary`).
mode:
type: string
enum: [free-subdomain, byo]
keycloak:
type: object
required: [realmURL]
@ -78,6 +110,17 @@ spec:
username:
type: string
default: admin
autoProvision:
type: object
description: |
Auto-provision the admin Secret with a random
ADMIN_PASSWORD when no ExternalSecret/SealedSecret is
wired (#898). Defaults to enabled so fresh tenants
boot without operator intervention.
properties:
enabled:
type: boolean
default: true
ingress:
type: object
properties:

View File

@ -1,6 +1,30 @@
apiVersion: v2
name: bp-stalwart-tenant
version: 0.1.0
# 0.1.1 (#898) — three fixes that left the 0.1.0 chart unbootable on
# fresh otech provisions:
# 1. accept legacy string-shape `domain: <fqdn>` values produced by
# pre-#897 catalyst-api builds (the live HR shape on otech103
# was `values.domain: acme.omani.works` which broke
# `.Values.domain.primary` indexing across config-configmap,
# deployment, dns-records-configmap, mailbox-provision-job, and
# the webmailHost helper). Resolved via a single `tenantDomain`
# helper in _helpers.tpl that handles both shapes (string OR
# `{primary,mode}` map).
# 2. auto-provision the `stalwart-admin` Secret with a
# randomly-generated ADMIN_PASSWORD via lookup-persistence
# (mirrors marketplace-api/secret.yaml in catalyst chart). The
# 0.1.0 chart referenced the Secret in deployment.yaml but never
# created it — every fresh tenant lands in
# CreateContainerConfigError until the operator hand-rolls one.
# 3. make the pod sec context compatible with the upstream
# stalwart binary's file capabilities (`getcap` on
# docker.io/stalwartlabs/stalwart:v0.16.3 reports
# `cap_net_bind_service=ep`). The 0.1.0 sec ctx ran as UID 65534
# (image-default is 2000) AND dropped ALL caps — kernel refuses
# to elevate file caps with empty bounding set, so exec failed
# with `operation not permitted`. Now: runAsUser 2000, drop ALL
# then add NET_BIND_SERVICE explicitly.
version: 0.1.1
appVersion: "0.16.3"
description: |
Catalyst Blueprint scratch chart for a per-SME (per-vcluster) dedicated

View File

@ -70,9 +70,63 @@ Surfaced by the unified-rbac console UI.
{{- printf "%s-dns-records-required" (include "bp-stalwart-tenant.fullname" .) | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Tenant primary mail domain — resolves the SME's mail FQDN regardless of
the values shape the orchestrator supplies. Three accepted shapes (in
priority order):
1. `tenant.domain: <string>` — newer per-tenant overlay schema
(#898 forward-looking convention).
2. `domain: { primary: <str>, current chart canonical schema
mode: ... }` (matches blueprint.yaml configSchema
and post-#897 catalyst-api emission).
3. `domain: <string>` — LEGACY shape emitted by pre-#897
catalyst-api builds (live on
otech103: `values.domain: acme.omani.works`).
Kept for back-compat so a tenant
pinned to chart 0.1.1 reconciles
cleanly even when the catalyst-api
image upgrade lags behind.
Returns "" (empty) when none of the three resolves, which is the
smoke-render-safe default (CI's blueprint-release.yaml renders with
empty values; per-template gates downstream skip on empty host).
Per docs/INVIOLABLE-PRINCIPLES.md #4 (never hardcode) — no fallback
to a literal domain anywhere in this file.
*/}}
{{- define "bp-stalwart-tenant.tenantDomain" -}}
{{- $tenant := .Values.tenant | default dict -}}
{{- $d := .Values.domain -}}
{{- if $tenant.domain -}}
{{- $tenant.domain -}}
{{- else if kindIs "map" $d -}}
{{- if $d.primary -}}{{- $d.primary -}}{{- end -}}
{{- else if kindIs "string" $d -}}
{{- $d -}}
{{- end -}}
{{- end -}}
{{/*
Tenant domain mode — `free-subdomain` (default) or `byo`. Mirrors the
shape resolution of tenantDomain so legacy string-shape `domain` falls
back to the values-yaml `domain.mode` default of "free-subdomain".
*/}}
{{- define "bp-stalwart-tenant.tenantMode" -}}
{{- $tenant := .Values.tenant | default dict -}}
{{- $d := .Values.domain -}}
{{- if $tenant.mode -}}
{{- $tenant.mode -}}
{{- else if and (kindIs "map" $d) $d.mode -}}
{{- $d.mode -}}
{{- else -}}
free-subdomain
{{- end -}}
{{- end -}}
{{/*
Webmail host. If `ingress.webmail.host` is explicitly set, use it. Otherwise
compose `mail.<domain.primary>`. Per docs/INVIOLABLE-PRINCIPLES.md #4 —
compose `mail.<tenant-domain>`. Per docs/INVIOLABLE-PRINCIPLES.md #4 —
no hardcoded base domain.
Returns empty string when both are unset (smoke-render-safe). The
@ -81,10 +135,11 @@ when host is empty, so a default-values render produces a syntactically
valid (but non-functional) manifest tree.
*/}}
{{- define "bp-stalwart-tenant.webmailHost" -}}
{{- $domain := include "bp-stalwart-tenant.tenantDomain" . -}}
{{- if .Values.ingress.webmail.host -}}
{{- .Values.ingress.webmail.host -}}
{{- else if .Values.domain.primary -}}
{{- printf "mail.%s" .Values.domain.primary -}}
{{- else if $domain -}}
{{- printf "mail.%s" $domain -}}
{{- end -}}
{{- end -}}

View File

@ -0,0 +1,95 @@
{{- /*
Auto-provision the Stalwart admin Secret (issue #898).
Why this template exists:
templates/deployment.yaml + templates/mailbox-provision-job.yaml
reference a secretKeyRef on `.Values.admin.secretName` (key
`ADMIN_PASSWORD`) via the `bp-stalwart-tenant.adminPasswordEnv`
helper. The 0.1.0 chart relied on EITHER an operator-supplied
SealedSecret OR an ExternalSecret rendered by
`admin-externalsecret.yaml` — but the ExternalSecret only renders
when `admin.externalSecret.remoteRef.key` is non-empty (smoke-render
concession), which is NEVER true on a fresh per-tenant install
(#898 evidence on otech103: alice tenant landed in
CreateContainerConfigError because no `stalwart-admin` Secret existed
in the tenant namespace).
This template ALWAYS materialises the Secret with a randomly-
generated ADMIN_PASSWORD when no externally-managed source is
active, so a freshly-franchised SME tenant gets a usable mail
server out of the box without operator hand-rolling. Mirrors the
established pattern in `products/catalyst/chart/templates/
marketplace-api/secret.yaml` (jwt-secret), which itself was added
for the same fresh-Sovereign breakage class (#887).
Render gates (skip auto-provisioning when an authoritative source
is in play):
1. `admin.autoProvision.enabled=false` — operator opts out (e.g.
they pre-create a SealedSecret named <secretName> in the tenant
namespace).
2. `admin.externalSecret.enabled=true` AND
`admin.externalSecret.remoteRef.key` non-empty — the
ExternalSecret in admin-externalsecret.yaml will own the Secret;
two controllers fighting over the same Secret is a known
anti-pattern (chart docs already warn).
Persistence across reconciles (load-bearing):
Per the gitea-admin-secret + sme-secrets pattern (issues #830/#859/
#887), the randomly-generated ADMIN_PASSWORD MUST survive
`helm upgrade` / Flux re-reconciliation. Without `lookup`, every
reconcile would emit a NEW password, immediately invalidating
Stalwart's running RocksDB-stored credential and locking the SME
admin out (RocksDB only accepts the bootstrap password on first
install — subsequent password changes go through the admin API).
Steady state: lookup finds the existing Secret, decodes the value,
and re-encodes verbatim.
First install (or `helm template`): lookup returns nil; `randAlphaNum`
generates a fresh 32-character value (per global CLAUDE.md credential
hygiene + INVIOLABLE-PRINCIPLES.md #10 — fully random, no dictionary
words, no human-memorizable patterns).
helm.sh/resource-policy: keep — `helm uninstall` doesn't drop the
bytes; a re-install picks up the same value via lookup.
Per docs/INVIOLABLE-PRINCIPLES.md #10: NO plaintext credentials in
this template. The random value is generated by sprig's randAlphaNum
inside the lookup-or-generate switch and lives ONLY in the resulting
Secret bytes. NEVER echo, print, or persist the generated value
outside the Secret manifest.
*/}}
{{- $a := .Values.admin -}}
{{- $auto := $a.autoProvision | default dict -}}
{{- $autoEnabled := dig "enabled" true $auto -}}
{{- $eso := $a.externalSecret | default dict -}}
{{- $esoRemote := $eso.remoteRef | default dict -}}
{{- $esoActive := and $eso.enabled $esoRemote.key -}}
{{- if and $autoEnabled (not $esoActive) -}}
{{- $secretName := $a.secretName -}}
{{- $namespace := .Release.Namespace -}}
{{- /* ---- Persistent random via lookup ---- */ -}}
{{- $existing := lookup "v1" "Secret" $namespace $secretName -}}
{{- $adminPassword := "" -}}
{{- if and $existing $existing.data (index $existing.data "ADMIN_PASSWORD") -}}
{{- $adminPassword = index $existing.data "ADMIN_PASSWORD" | b64dec -}}
{{- else -}}
{{- $adminPassword = randAlphaNum 32 -}}
{{- end -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ $secretName | quote }}
namespace: {{ $namespace | quote }}
labels:
{{- include "bp-stalwart-tenant.labels" . | nindent 4 }}
catalyst.openova.io/component: stalwart-admin
annotations:
# Survive helm uninstall — the Secret outlives the release. A
# subsequent helm install picks the bytes back up via lookup, so
# ADMIN_PASSWORD remains stable across reinstall (mirrors the
# gitea-admin-secret / marketplace-api-secrets pattern, #830/#887).
helm.sh/resource-policy: keep
type: Opaque
data:
ADMIN_PASSWORD: {{ $adminPassword | b64enc | quote }}
{{- end }}

View File

@ -9,6 +9,7 @@ matchers MUST use `==` not `=`. A single `=` is assignment and silently
never matches — see stalwart_expression_syntax.md. Every comparison in
this template uses `==`.
*/}}
{{- $tenantDomain := include "bp-stalwart-tenant.tenantDomain" . -}}
apiVersion: v1
kind: ConfigMap
metadata:
@ -95,7 +96,7 @@ data:
sign = [{if = "is_local_domain('*')", then = true}, {else = false}]
[signature.dkim]
domain = {{ .Values.domain.primary | quote }}
domain = {{ $tenantDomain | quote }}
selector = {{ .Values.dns.dkim.selector | quote }}
algorithm = {{ .Values.dns.dkim.algorithm | quote }}
canonicalization = "relaxed/relaxed"
@ -131,4 +132,4 @@ data:
# Local domains — the SME's primary domain (and any aliases the
# operator adds via webadmin runtime).
[config.local-domain]
{{ printf "%q" .Values.domain.primary }} = true
{{ printf "%q" $tenantDomain }} = true

View File

@ -70,7 +70,7 @@ spec:
name: {{ .Values.keycloak.clientSecretName | quote }}
key: OIDC_CLIENT_SECRET
- name: STALWART_DOMAIN
value: {{ .Values.domain.primary | quote }}
value: {{ include "bp-stalwart-tenant.tenantDomain" . | quote }}
- name: STALWART_HOSTNAME
value: {{ include "bp-stalwart-tenant.webmailHost" . | quote }}
resources:

View File

@ -15,6 +15,8 @@ template below ships the static records (MX, SPF, DMARC) and a
placeholder DKIM record line so the UI can render before DKIM is sealed.
*/}}
{{- if .Values.stalwart.enabled }}
{{- $tenantDomain := include "bp-stalwart-tenant.tenantDomain" . -}}
{{- $tenantMode := include "bp-stalwart-tenant.tenantMode" . -}}
apiVersion: v1
kind: ConfigMap
metadata:
@ -23,21 +25,21 @@ metadata:
{{- include "bp-stalwart-tenant.labels" . | nindent 4 }}
catalyst.openova.io/role: dns-records-required
data:
domain: {{ .Values.domain.primary | quote }}
mode: {{ .Values.domain.mode | quote }}
domain: {{ $tenantDomain | quote }}
mode: {{ $tenantMode | quote }}
records.yaml: |
# MX record — points the SME's domain at this Stalwart's mail Service.
# The hostname is the LoadBalancer-assigned address (filled in at
# runtime by the unified-rbac controller once the LB IP is known).
- kind: MX
name: {{ .Values.domain.primary | quote }}
name: {{ $tenantDomain | quote }}
priority: 10
value: {{ printf "mail.%s" .Values.domain.primary | quote }}
value: {{ printf "mail.%s" $tenantDomain | quote }}
# SPF — declare this Stalwart's IP as a permitted sender; close
# everything else off per .Values.dns.spf.policy.
- kind: TXT
name: {{ .Values.domain.primary | quote }}
name: {{ $tenantDomain | quote }}
value: {{ printf "v=spf1 mx %s" .Values.dns.spf.policy | quote }}
# DKIM public key — selector + algorithm rendered here; the actual
@ -45,11 +47,11 @@ data:
# once Stalwart has minted the key on first boot. Placeholder
# "<DKIM-PUBLIC-KEY>" is a sentinel the controller searches for.
- kind: TXT
name: {{ printf "%s._domainkey.%s" .Values.dns.dkim.selector .Values.domain.primary | quote }}
name: {{ printf "%s._domainkey.%s" .Values.dns.dkim.selector $tenantDomain | quote }}
value: "v=DKIM1; k=ed25519; p=<DKIM-PUBLIC-KEY>"
# DMARC.
- kind: TXT
name: {{ printf "_dmarc.%s" .Values.domain.primary | quote }}
value: {{ printf "v=DMARC1; p=%s; rua=mailto:%s" .Values.dns.dmarc.policy (default (printf "dmarc@%s" .Values.domain.primary) .Values.dns.dmarc.rua) | quote }}
name: {{ printf "_dmarc.%s" $tenantDomain | quote }}
value: {{ printf "v=DMARC1; p=%s; rua=mailto:%s" .Values.dns.dmarc.policy (default (printf "dmarc@%s" $tenantDomain) .Values.dns.dmarc.rua) | quote }}
{{- end }}

View File

@ -64,7 +64,7 @@ spec:
- name: ADMIN_USER
value: {{ .Values.admin.username | quote }}
- name: STALWART_DOMAIN
value: {{ .Values.domain.primary | quote }}
value: {{ include "bp-stalwart-tenant.tenantDomain" . | quote }}
command:
- /bin/sh
- -ec

View File

@ -52,17 +52,29 @@ stalwart:
cpu: 1
memory: 1Gi
# SecurityContext — non-root. Stalwart's official image runs as
# `stalwart-mail` (UID 65534 on the upstream image). Container
# restricts elevated capabilities; SMTP port 25 inside the container
# is bound by the upstream entrypoint without CAP_NET_BIND_SERVICE
# because Stalwart binds AFTER the entrypoint demotes (it uses a
# listener offset configurable via `server.listener.smtp.bind`).
# SecurityContext — non-root + NET_BIND_SERVICE (issue #898).
#
# Stalwart's upstream image (docker.io/stalwartlabs/stalwart:v0.16.3)
# creates an in-image `stalwart` user at UID 2000 and ships the
# binary at /usr/local/bin/stalwart with file capability
# `cap_net_bind_service=ep`. The binary needs that capability to
# bind 25/465/587/143/993 directly (no entrypoint demotion script
# in this image — the binary is the entrypoint).
#
# The 0.1.0 chart ran as UID 65534 with `drop: ALL` — kernel
# refuses to elevate file capabilities when the caller's bounding
# set is empty, so exec failed at startup with `operation not
# permitted` on every fresh tenant (otech103 evidence in #898).
#
# 0.1.1 fix: align with the image's native UID 2000, drop ALL
# then add ONLY NET_BIND_SERVICE. fsGroup 2000 ensures the PVC
# at /opt/stalwart is writable by the stalwart user without
# privileged chown.
podSecurityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
fsGroup: 65534
runAsUser: 2000
runAsGroup: 2000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
@ -70,6 +82,7 @@ stalwart:
readOnlyRootFilesystem: false
capabilities:
drop: ["ALL"]
add: ["NET_BIND_SERVICE"]
# Liveness/readiness via stalwart-cli healthcheck. The CLI authenticates
# against the local management API using the admin credentials the chart
@ -183,6 +196,20 @@ admin:
# initial superuser. Webmail SSO covers regular users (Keycloak); this
# account is for the rescue-shell admin path only.
username: "admin"
# Auto-provision render gate (#898). When true (default), the chart
# emits `templates/admin-secret.yaml` which materialises the
# `<secretName>` Secret with a random 32-char ADMIN_PASSWORD via
# lookup-persistence (mirrors gitea-admin-secret + marketplace-api-
# secrets, #830/#887). Set false to opt out — operator pre-creates
# a SealedSecret OR the ExternalSecret block below is wired with a
# non-empty remoteRef.key (auto-provision auto-disables in that
# case, no double-bind).
#
# Per docs/INVIOLABLE-PRINCIPLES.md #10 the generated value is
# written ONLY into the Secret bytes — NEVER echoed or persisted
# elsewhere. It survives helm upgrade / Flux reconcile via lookup.
autoProvision:
enabled: true
# ExternalSecret render gate — when true (default), the chart emits an
# ExternalSecret that pulls the admin password from OpenBao. Disable
# when the operator pre-creates the SealedSecret and wants no churn.
@ -295,13 +322,23 @@ ingress:
# works from t=0.
mailboxProvisioner:
# Run-once setup Job (admin principal + OIDC client config in
# Stalwart). ALWAYS true on a fresh install; idempotent on upgrade.
# Stalwart). The Job's image is operator-tunable (#898) — until the
# canonical `ghcr.io/openova-io/openova/stalwart-cli` image is
# published, the Job re-uses the upstream stalwart container which
# ships `stalwart-cli` AND `curl` for HTTP calls. Disabled by
# default in 0.1.1 to avoid CrashLoopBackOff on tenants where the
# canonical CLI image is unavailable; SME admin can opt in via
# per-tenant overlay once the image is published.
setupJob:
enabled: true
enabled: false
image:
repository: ghcr.io/openova-io/openova/stalwart-cli
tag: "0.16.3"
digest: ""
# Re-use the upstream Stalwart image which already ships curl +
# stalwart-cli. SHA-pinned via the same digest as the StatefulSet
# so the bytes trace back to a single source (#898 / #560 ADR-0001
# §11.5 — Sovereign-local Harbor proxy-cache rewrites at runtime).
repository: stalwartlabs/stalwart
tag: "v0.16.3"
digest: "sha256:5d75cff4e9c6d75e64636e9ef9674b1d877f8f6fb2e11ee8176fbad3faaa5289"
pullPolicy: IfNotPresent
# Job timeout — the Stalwart pod must be Ready before the Job's
# API calls succeed. 10 min is generous for cold starts on small