Source Pipeline Tool

Source Pipeline Tool

Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.

Problem Statement

RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:

1. Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.

2. Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.

3. Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.

4. Unverified patches. Patches (.patch files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.

Tool Overview

A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.

                          ┌──────────────────────────────┐
                          │     source-pipeline tool     │
                          │      (container image)       │
                          │                              │
  ┌───────────┐           │  ┌───────┐    ┌───────────┐  │    ┌───────────────┐
  │  package  │           │  │ fetch │───▸│ transform │  │    │   tarballs    │
  │   dir     │──────────▸│  └───────┘    └─────┬─────┘  │───▸│   sources     │
  │ (spec,    │           │                     │        │    │   report      │
  │  patches, │           │  ┌───────────┐      │        │    └───────┬───────┘
  │  config)  │           │  │  verify   │◂─────┘        │            │
  │           │           │  └─────┬─────┘               │            ▼
  └───────────┘           │        │                     │    ┌───────────────┐
                          │  ┌─────▼─────┐               │    │   lookaside   │
  ┌───────────┐           │  │  enforce  │               │    │    cache      │
  │ pipeline  │           │  │  policy   │               │    └───────────────┘
  │   yaml    │──────────▸│  └─────┬─────┘               │
  │           │           │        │                     │
  └───────────┘           │  ┌─────▼─────┐               │
                          │  │   emit    │               │
                          │  └───────────┘               │
                          └──────────────────────────────┘

Design principles

• Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
• Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
• Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
• Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.

Container interface

podman run --rm \
  -v ./<package-dir>:/package:ro \
  -v ./pipeline.yaml:/pipeline.yaml:ro \
  -v ./gpg-keys:/gpg-keys:ro \
  -v ./output:/output \
  source-pipeline:latest \
  --version <new-version> \
  [--old-version <old-version>] \
  [--verify-patches /patches-from-upstream] \
  [--dry-run]

Inputs (mounted read-only):

• /package — the package directory (spec file, patches, existing sources file)
• /pipeline.yaml — the declarative pipeline definition
• /gpg-keys — centralized GPG keyring directory

Outputs (written to /output):

• Source tarballs (ready for lookaside upload)
• sources — updated sources manifest with checksums
• report.json — verification and policy results (pass/fail per check, patch classifications)

Exit codes:

• 0 — success, all checks passed
• 1 — error (download failure, tool error)
• 2 — policy violation (verification or constraint failure)

--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to /output. Exit codes and report.json behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing.

On any non-zero exit, report.json is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no commit is created, the package stays at its current version, and the automation continues to the next package. Transient failures (exit 1) self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention.

Pipeline Stages

1. Fetch

Downloads source artifacts directly from upstream.

• Parses Source: URLs from the spec file (resolving RPM macros like %{version}, %{name}, %{url})
• Downloads each source from the upstream URL
• Supports fetching from git repositories at a tag, branch, or commit
• For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)

2. Transform

Applies per-package source modifications.

• Vendor archive generation — runs ecosystem-specific tooling (e.g., go_vendor_archive for Go, npm pack for Node.js, cargo vendor for Rust)
• Vendor dependency pinning — modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix.
• Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
• UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
• Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
• Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a toolchain: section. The container ships defaults; per-package overrides ensure that build-ui, vendor, and run: steps use the correct toolchain without requiring separate container images per package

Known sharp edge: patch-list duplication

Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill point back here instead of re-telling the story.

Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ... followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain their own copy of “which patches touch this source tree” rather than reading it from the spec’s PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see Rebuilding Packages) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform’s install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation step, and the mismatch didn’t surface until the next automated version bump re-ran the transform from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact.

go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up requiring different versions of the same package — go build -mod=vendor rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s %check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

3. Verify

Validates integrity and authenticity of fetched sources.

• GPG signature verification — downloads signature files (.asc, .sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation.
• Checksum verification — compares against published checksums where available
• Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
• Re-publication detection — if a previously committed checksum exists in the sources file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an accepted-checksums entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.

4. Enforce policy

Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect) and catches violations in packages that don’t use vendor-pin.

• Vendor dependency constraints — ensures vendored dependencies meet version requirements (e.g., sanitize-html >= 2.17.5 for a CVE fix)
• Ecosystem-specific checks — Go module verification, npm audit, cargo audit
• License compliance — flags vendored dependencies with incompatible licenses
• Fail-closed: any policy violation exits with code 2

5. Emit

Produces final artifacts.

• Writes tarballs to the output directory
• Generates sources file in dist-git format (SHA512 (filename) = hash)
• Writes report.json with verification results, policy check results, and patch classification

Pipeline Schema

Per-package pipeline definitions are YAML files that declare the full source generation workflow.

Packages without a pipeline definition use a built-in default that fetches sources from the spec’s Source: URLs — no transforms, no verification, no policy.

Schema definition

# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
  # Macro substitutions applied before Source URL resolution
  macros:
    - name: go_patch                     # %global go_patch <new-value>
      value: "${VERSION_PATCH}"          # extracted from VERSION (e.g., 1.25.3 → 3)

    - name: k8s_ver                      # %global k8s_ver <new-value>
      value: "${VERSION}"

  # Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
  reset-release: true

# Source fetching
fetch:
  sources:
    # Fetch Source0 from the URL declared in the spec
    - spec-source: 0

    # Fetch Source1 (e.g., a GPG signature file)
    - spec-source: 1

    # Or fetch from an explicit URL (for sources not in the spec)
    - url: "https://example.com/extra-source-${VERSION}.tar.gz"

    # Or fetch from a git repository at a tag/commit
    - git:
        repo: "https://github.com/example/project"
        ref: "v${VERSION}"              # tag, branch, or commit hash
        include-history: false           # include .git dir in tarball (default: false)

  # Vendor archive generation (optional)
  vendor:
    ecosystem: go                        # go | npm | cargo | composer
    config: go-vendor-tools.toml         # ecosystem-specific config file in the package dir
    source-dir: .                        # directory to vendor from (default: extracted source root)
    submodules:                          # (optional) vendor multiple Go submodules independently
      - server
      - etcdctl
      - etcdutl

# Source transforms (optional, ordered)
transform:
  # Built-in transform types
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
        - "deps/ngtcp2/ngtcp2/crypto/"
      output: "node-v${VERSION}-stripped.tar.gz"

  # Pin vendored dependency versions before vendor archive generation.
  # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
  # bump specific dependencies, then re-resolves the dependency graph.
  # Runs before the vendor stage so the pinned versions are included in
  # the vendor archive. Re-applied on every source generation, so upstream
  # updates cannot silently revert a security fix.
  #
  # Version constraints are always minimum versions — the version field
  # means "at least this version." The tool translates to ecosystem-native
  # operations:
  #   Go:    go get <package>@v<version> (minimum version selection)
  #   npm:   npm install <package>@">= <version>"
  #   Cargo: set dependency requirement to ">= <version>" in Cargo.toml
  - vendor-pin:
      - package: golang.org/x/crypto
        ecosystem: go
        version: "0.31.0"
        reason: "CVE-2024-45337"

      - package: sanitize-html
        ecosystem: npm
        version: "2.17.5"
        reason: "CVE-2024-XXXXX"

      - package: tokio
        ecosystem: cargo
        version: "1.38.1"
        reason: "CVE-2024-YYYYY"

  # Custom command (escape hatch)
  - run: "./packaging/make-tarball.sh ${VERSION}"
    outputs:
      - "package-${VERSION}-stripped.tar.gz"

  # UI asset build
  - build-ui:
      ecosystem: npm                     # npm | yarn
      source-dir: "web/ui"
      output: "${PACKAGE}-${VERSION}-ui.tar.gz"

# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
  node: "20"                             # Node.js version for build-ui, npm vendor, and run: steps
  go: "1.23"                             # Go version for go vendor and run: steps
  rust: "1.80"                           # Rust version for cargo vendor and run: steps
  python: "3.12"                         # Python version for run: steps

# Integrity verification (optional)
verify:
  gpg:
    signature-source: 1                  # spec Source index containing the .asc/.sig
    keyring: "curl.gpg"                  # key name in the GPG keys directory
  checksums:
    url: "https://example.com/SHA256SUMS"
    algorithm: sha256

  # Override for upstream re-publications. Required when upstream re-publishes
  # a release artifact with different content under the same version. The
  # pipeline refuses to accept changed content automatically — a human must
  # add the new checksum here after investigation.
  accepted-checksums:
    - file: "example-1.2.3.tar.gz"
      sha512: "abc123..."
      reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"

# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
  vendor-constraints:
    - package: sanitize-html
      ecosystem: npm
      version: "2.17.5"
      reason: "CVE-2024-XXXXX"

    - package: golang.org/x/crypto
      ecosystem: go
      version: "0.31.0"
      reason: "CVE-2024-45337"

# Patch verification (optional)
patches:
  verify: true                           # diff-detect new/changed patches
  classify: true                         # attempt to classify by origin
  upstream-repo: "https://github.com/curl/curl"
  fail-on-unverified: false              # warn-only by default

  # Patch lifecycle rules (optional)
  # Declares version-scoped applicability for patches. The pipeline tool reads
  # these rules from patch headers (preferred) or from this YAML, and enforces
  # them during updates: patches outside their valid range are flagged or dropped.
  lifecycle:
    - file: "fix-memory-leak.patch"
      applies-to: "< 1.5.0"             # drop this patch at version 1.5.0+
      reason: "Fixed upstream in 1.5.0 (commit abc123)"
      action: drop                       # drop | warn (default: warn)

    - file: "cve-2024-45337.patch"
      applies-to: "< 0.31.0"
      reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
      action: drop

    - file: "distro-branding.patch"
      applies-to: "*"                    # carry forward unconditionally
      reason: "Distribution-specific branding, always required"

# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
  # Extract bundled dependency versions and splice into spec between markers
  - bundled-provides:
      modules-txt: "vendor/modules.txt"  # Go modules.txt path inside extracted source
      start-marker: "# --- bundled-deps.sh ---"
      end-marker: "# --- end bundled-deps.sh ---"

  # Run a custom command (escape hatch for complex metadata extraction)
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"

Variable substitution

The following variables are available in all string values:

Version constraint semantics

All version: fields in vendor-pin and policy.vendor-constraints use minimum-version semantics: the value means “at least this version.” The tool translates this into ecosystem-native operations:

• Go: go get <package>@v<version> (minimum version selection via MVS)
• npm: npm install <package>@">= <version>"
• Cargo: sets the dependency requirement to >= <version> in Cargo.toml

Default behavior (no pipeline YAML)

When no pipeline YAML is provided, the tool applies a built-in default:

1. Parse all Source: URLs from the spec
2. Download each from the upstream URL
3. Generate sources file with SHA512 checksums
4. No transforms, no verification, no policy enforcement

This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.

Patch Verification

When packages are updated from an upstream distribution, patch files (.patch) may arrive without inspection. The pipeline tool can optionally verify and manage patches.

Verification approach

The pipeline tool can optionally verify patches when invoked with --verify-patches:

1. Diff detection. Compare the set of .patch files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches.

2. Header parsing. Extract metadata from patch headers:

   • From: — author identity
   • Subject: — description
   • Commit hash references (e.g., From <hash>, cherry picked from commit <hash>)
   • Bug: / CVE: references

3. Classification. Attempt to classify each patch:

   • Upstream backport — references a commit hash that exists in the upstream repo
   • CVE fix — references a CVE identifier
   • Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
   • Distribution-specific — modifies paths, branding, or distribution-specific integration
   • Unclassified — cannot be automatically categorized

4. Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via git ls-remote or the upstream API).

5. Reporting. Output classification and verification results in report.json. Optionally fail on unverified patches (controlled by patches.fail-on-unverified in the pipeline YAML).

Patch lifecycle enforcement

Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.

The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML’s patches.lifecycle section.

Header-based declaration (preferred). Patch authors add structured keywords to the patch header:

From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---

YAML-based declaration (fallback). For patches from upstream distributions that cannot have headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch.

Enforcement behavior:

1. During an update to version ${VERSION}, the tool evaluates each patch’s applies-to range against the new version.
2. If a patch is outside its valid range:
   • action: drop — the patch file is deleted from the package directory and the corresponding Patch: declaration and %patch / %autopatch application directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in report.json.
   • action: warn (default) — the patch is flagged in report.json but not removed. The update proceeds.
3. If a patch has applies-to: *, it is always carried forward.
4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to applies-to: *, action: warn).

Carry-forward enforcement. The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked applies-to: * is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error.

Example report output

{
  "patches": {
    "new": [
      {
        "file": "fix-memory-leak.patch",
        "classification": "upstream-backport",
        "upstream_commit": "abc123def456",
        "verified": true
      }
    ],
    "removed": ["old-workaround.patch"],
    "unchanged": ["distro-paths.patch"],
    "lifecycle": [
      {
        "file": "cve-2024-45337.patch",
        "applies_to": "< 0.31.0",
        "current_version": "0.31.0",
        "action": "drop",
        "result": "removed — version 0.31.0 is outside applies-to range"
      },
      {
        "file": "cve-2024-99999.patch",
        "applies_to": "< 2.0.0",
        "current_version": "2.0.0",
        "action": "drop",
        "result": "flagged — patch is applied inside a conditional %if block; manual removal required"
      },
      {
        "file": "distro-branding.patch",
        "applies_to": "*",
        "action": "carry-forward",
        "result": "present"
      }
    ]
  }
}

Examples

Trivial package (curl)

No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you want GPG verification:

fetch:
  sources:
    - spec-source: 0
    - spec-source: 1

verify:
  gpg:
    signature-source: 1
    keyring: "curl.gpg"

Transformed package (Node.js)

Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:

spec-update:
  macros:
    - name: nodejs_define_version node
      value: "${VERSION}"
  reset-release: true

toolchain:
  node: "22"

fetch:
  sources:
    - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"

transform:
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
      output: "node-v${VERSION}-stripped.tar.gz"

verify:
  checksums:
    url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
    algorithm: sha256

post:
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"

Simple Go vendor (caddy)

A pattern shared by nats-server, oauth2-proxy, and similar Go projects:

fetch:
  sources:
    - git:
        repo: "https://github.com/caddyserver/caddy"
        ref: "v${VERSION}"

  vendor:
    ecosystem: go

Multi-submodule Go vendor (etcd)

fetch:
  sources:
    - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    submodules:
      - server
      - etcdctl
      - etcdutl

Code generation + Go vendor (opentelemetry-collector-contrib)

Built-in primitives handle fetch and vendor; the code generation step requires run::

fetch:
  sources:
    - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    source-dir: _build

transform:
  - run: |
      curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
      curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
      sha256sum -c --ignore-missing ocb_checksums.txt
      chmod +x ocb
      tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
      cd "opentelemetry-collector-releases-${VERSION}"
      if [ -f scripts/prepare-obi.sh ]; then
        bash scripts/prepare-obi.sh otelcol-contrib
      fi
      ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
    outputs:
      - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"

Git snapshot (libXtst)

A pattern shared by libX11, libXext, libXi, libXrender:

fetch:
  sources:
    - git:
        repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
        ref: "libXtst-${VERSION}"
        include-history: true

Built-in Primitives Coverage

An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:

Fully declarative (no run: needed) — ~30 script files (15 unique patterns):

Partially declarative (generic primitives + 1–2 run: steps):

Genuinely custom (run: required) — 7 packages:

The run: escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract.

Open Questions

1. Implementation approach. The pipeline YAML schema (ordered transform stages, run: escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:
   • Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
   • Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
   • Shared shell function library — the majority of scripts decompose into 3–4 operations (vendor_go, strip_tarball, fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands.
   • Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no run: blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.

Hummingbird Integration

This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.

Current Architecture

Two update systems

dist_git.py update flow:

1. Clones Fedora dist-git for the package
2. Checks version, Koji build status, pre-release filtering
3. Copies entire Fedora dist-git checkout into rpms/<package>/ via shutil.copytree — this includes the spec, patches, .gitignore, and sources file
4. For modified packages, performs a 3-way git merge to preserve local changes
5. Commits everything (spec, patches, sources file, metadata)

The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash). Actual tarballs are excluded by .gitignore and stored in a lookaside cache.

check_upstream_versions.py flow:

1. Queries release-monitoring.org for new upstream versions
2. Runs three hook phases per package:
   • update_spec — updates the spec’s Version/Release (default: specfile.update_version())
   • download_sources — downloads source tarballs (default: fetches from spec Source URLs)
   • post_update — additional steps (default: no-op)
3. Uploads downloaded tarballs to Hummingbird’s lookaside cache
4. Updates the sources file with new checksums
5. Commits

Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.

Three lookaside cache backends

Configured in mock/dist-git-client.ini:

The forked_from field in ci/package-overrides.yaml determines which cache a package uses at build time. Packages without forked_from default to Fedora’s cache.

How Konflux builds consume sources

Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):

• monorepo-subdir: rpms/<package> — locates the spec, patches, and sources file
• package-name: <name> — upstream name for lookaside URL construction
• forked-from: <url> — (optional) selects the lookaside backend
• dist-git-client-configdir: mock/ — (optional) points to the custom dist-git-client.ini

The pipeline uses dist-git-client to read the sources file and download tarballs from the appropriate cache, then runs mock to build RPMs.

Existing source generation patterns

Some independent packages already have tarball generation scripts:

• rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh — downloads upstream release, runs OCB to generate source code, vendors Go dependencies
• rpms/caddy/create-vendor-tarball.sh — downloads upstream tarball, generates Go vendor archive
• rpms/nodejs25/packaging/make-nodejs-tarball.sh — downloads upstream, strips bundled content

These scripts are wired into the update path via download_sources hooks in metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache.

Integration with dist_git.py update

The current flow copies everything from Fedora dist-git (including the sources file) via shutil.copytree. The pipeline tool inserts after this step:

Current:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Commit

Proposed:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Run source-pipeline tool             ← fetches from upstream, replaces sources file
  4. Upload tarballs to Hummingbird lookaside
  5. Commit                               ← sources file now points to Hummingbird lookaside

Implementation:

• After the shutil.copytree (or merge for modified packages), check if metadata/<package>.source-pipeline.yaml exists
• If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the sources file
• If it does not exist: run the built-in default (fetch from spec URLs, upload, replace sources)
• A --skip-pipeline flag allows falling back to the current behavior during migration
• Update ci/package-overrides.yaml to set forked_from to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)

Integration with check_upstream_versions.py

The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating per-package update behavior into a single *.source-pipeline.yaml file:

Current:
  1. update_spec hook/default             ← *.update-hooks.yaml
  2. download_sources hook/default        ← *.update-hooks.yaml
  3. post_update hook/default             ← *.update-hooks.yaml

Proposed:
  1. spec-update (pipeline YAML)          ← replaces update_spec hooks
  2. fetch + transform (pipeline YAML)    ← replaces download_sources hooks
  3. post (pipeline YAML)                 ← replaces post_update hooks

This reduces per-package metadata from three files (metadata/<package>.json, *.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for identity/tracking, *.source-pipeline.yaml for all update behavior).

Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed.

Integration with Konflux

Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via dist-git-client as it does today.

Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.

Hummingbird-specific configuration

Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.

GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.

The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.

Local development

Developers can run the tool directly:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  -v /tmp/output:/output \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0

# Outputs in /tmp/output/:
#   curl-8.21.0.tar.xz
#   curl-8.21.0.tar.xz.asc
#   sources
#   report.json

To preview without producing artifacts:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0 \
  --dry-run

# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.

Migration Path

Phase 1: Independent Go packages

Migrate the 22 independent packages that already fetch from upstream. Convert their existing create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions. Validates the tool against known-good packages.

Scope: 7 packages with existing scripts + 15 without.

Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code generation, but their vendor step is generic.

Phase 2: Trivial clean packages

Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.

Prerequisites:

• Audit to classify packages as trivial vs. transformed (HUM-4620)
• dist_git.py update integration complete (HUM-4621)

Per-package steps:

1. Run the pipeline tool, compare output against existing Fedora-sourced tarball
2. If identical: add forked_from: hummingbird to package-overrides.yaml
3. If different: flag for Phase 3

Phase 3: Transformed packages

Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:

• Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): strip-tarball
• Git snapshots (libX11 family×5, oniguruma, php-jsqueeze): fetch: {git: ...}
• Composer vendor (composer): vendor: {ecosystem: composer}

Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require run: blocks for genuinely custom logic.

Scope: determined by Phase 2 audit.

Phase 4: Policy enforcement

Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates.

Phase 5: Patch verification

Enable patches.verify and patches.classify for packages. Start with warn-only (fail-on-unverified: false), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages.

Operational Health

Escape hatch ratio

The primary health metric for the source pipeline is the escape hatch ratio: the percentage of pipeline YAML definitions that contain run: blocks (in transform or post sections) relative to the total number of pipeline YAML definitions.

A run: block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the same way: a package that cannot be expressed declaratively gets a run: block. This makes the ratio a single number that aggregates all pressures on the declarative model.

Interpretation:

When the ratio climbs, examine the run: blocks for repeated patterns. If three or more packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad – it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set.

CI enforcement

The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files, calculates the escape hatch ratio, and reports the result. It runs as part of make check.

• During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need run: blocks before new primitives are added.
• After Phase 3: enable --fail mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a run: block is added.