Container Catalog Design Decisions
Design Decision Registry (DDR) for container-catalog — every major design
choice, the alternatives considered, and why the chosen approach won. For
operational usage, see Container Catalog.
DDR-1: Historical RELEASE_SBOM writer survives the SBOM cutover as a write-if-absent seed
Decision: When the index function takes over SBOM-item writes, sync’s
build_release_sbom_item keeps its RELEASE_SBOM#{hex} construction,
written via an atomic write-if-absent conditional write
(put_content_versioned with attribute_not_exists(ver) — every
post-cutover writer stamps ver, so the condition blocks any rewrite of
normalized content); the historical branch of merge_release writes only
digest-keyed items; SBOM#{canonical} becomes index-exclusive.
Context: The SBOM cutover (W1 MR2) moves SBOM-item writes from sync to
index. Index only writes for digests current in TAGS; the SBOM backfill
enumerates scan blobs. Digests that were never scanned while current
therefore lose every writer. Production census (2026-11-11, 10 images,
10,697 history digests, vuln-item presence as the scan/blob proxy): 228
digests (2.1%) have an SBOM item but were never scanned — 225 from 2026-04,
before the scan pipeline wrote blobs, plus ~1/month of new misses (events
raced out of TAGS, dropped events). The registry tagging model (unique
sha256-{hex}/timestamp tags per build; ~a dozen moving plain version tags)
means every rebuild supersedes the previous digest out of TAGS, so this
population accrues slowly but indefinitely. The only code path that can
serve these digests is the catalog sync CLI’s historical sweep — the
production sync Lambda processes only each event’s release and never sweeps.
The sweep is an on-demand repair/repopulation tool (“Full DynamoDB
population”, --purge).
Alternatives: (a) Accept the gap — delete the RELEASE_SBOM construction outright; SBOM items become exclusively the index/backfill domain; the frozen 225 keep their existing items, new misses get history and details but no SBOM, documented as a known gap. (b) Keep an unconditional retained write — preserve today’s behavior for the historical branch unchanged. (c) Extend the backfill to synthesize RELEASE_SBOM from attestations for no-blob digests.
Rationale: (a) rejected because a repair tool with a measured blind spot
is an operational footgun: after an incident or data loss, a catalog sync
run would leave SBOM holes nothing else can fill (no blob → the backfill
enumerates blobs and skips them; no event → index never sees them). (b)
rejected because the sweep population is dominated by ex-current, blob-backed
digests that the SBOM backfill normalizes to scan-derived content; sync’s
writes are unconditional (batch_write; put_content_versioned has no
content comparison), so every CLI run would overwrite blob-derived items
back to attestation-derived content, silently undoing the backfill. (c)
rejected because the backfill is one-off by design — it cannot cover future
misses, only the current cohort. The write-if-absent seed keeps the repair
tool complete for exactly the population only it can reach, while the
conditional write makes it converge with the backfill instead of fighting
it: sync seeds once
(attestation-derived, today’s field spellings — consistent with the accepted
mixed-vocabulary story for no-blob digests), index and the backfill own the
final content for blob-backed digests, and a CLI run never rewrites an item
that already exists. Zero new IAM (one DDB read); ~20 lines retained plus
the guard, confined to a CLI-only code path.
DDR-2: Index invocation converges on the queue-decoupled pattern
Decision: The internal index function drops its SNS Release subscription
and consumes a queue of tag-release events; sync performs its DynamoDB
writes and then enqueues one event per release canonical, using the shared
TagReleaseEvent shape (the promoted ExternalTagReleaseEvent). The
function is shaped as the future unified index — the same per-event
processing structure as the external index, with the event shape and
blob-key prefixes already shared — so the external plane’s unification
(retiring external-index-lambda into this function) remains a small,
well-bounded change.
Context: The internal index was SNS-triggered like sync, which forced it to read TAGS to learn a release’s canonicals — introducing the sync-fast/index-slow race (TAGS-absent → retry → DLQ) and requiring per-image error aggregation inside one invocation. The external plane had already solved invocation decoupling: its sync resolves tags, writes TAGS, then enqueues per-digest events to an index queue whose batch-item-failure semantics provide per-record retry for free. Downstream of sync’s writes the two planes are the same pattern; only the chain’s head differs (the internal producer is the near-realtime bus event, the external one a 6-hour poll).
Alternatives: (a) Keep the SNS trigger and heal the TAGS race with retries — the race class stays, one slow image can block the record, and the two index functions keep divergent shapes. (b) Unify into a single shared deployment rather than two deployments of one function — rejected for blast-radius isolation: a bad deployment would take out both planes at once. (c) Defer the convergence to the external-plane MR — rejected: every week of divergence adds code that both index functions must carry, and the queue convergence was already decided for the internal plane on its own merits.
Rationale: Queue decoupling deletes the TAGS-race class by construction:
the event is emitted after the writes it depends on and carries everything
the index needs (canonicals, digest, architectures), so the index never
reads TAGS and canonical identity is re-resolved against the registry. SQS
batch-item failures replace hand-rolled per-image aggregation with the
platform’s own per-record retry. Sync redelivery is harmless — a duplicate
event hits the index’s dedup skip and flows into the recovery path, which is
the recovery’s designed purpose. The canonical re-verification stays
regardless of ordering: the moved-digest mid-scan race is real in any
invocation shape. Shaping the internal index as the unified function now
(shared event type, external’s processing structure, env-configured
prefixes) keeps MR4 small: point the external queue at the function and
delete external-index-lambda, plus four parameterizations the planes
still differ on — the DDB partition key (IMAGE# vs EXTERNAL#), the
pull-reference source (config.pull_registry vs the raw registry
reference), the arch source (manifest list via fetch_arch_attestations
vs the event’s architectures), and the Hermeto-merge gating
(classify_by_hermeto from DISTRO vs none). The event shape and the
blob-key prefixes are already zero-code.