RPMs repository

• 1: RPM Pipeline
• 2: Konflux Resource Deployment
• 3: Source Pipeline Tool

./ci/build_rpms.sh <package_name>
# Results are written to: /tmp/konflux-build-<package_name>-*/results/

limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build <package_name>'

./ci/build_rpms.sh --shell-before setup

./ci/build_rpms.sh --shell-after setup

# Basic usage (binary rpm)
./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm <package_name>

# Include source RPM tests
./ci/run_tests_rpm.sh \
  --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \
  --src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \
  <package_name>

# Test all RPMs in the build output directory after build_rpms.sh
./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds/<package_name>/RPMS/*.rpm) \
  --repo-dir builds/<package_name>/RPMS/ \
  --src-rpm builds/<package_name>/SRPMS/*.src.rpm \
  <package_name>

TEST_IMAGE=quay.io/hummingbird/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm <package_name>

# If using an OCI artifact with RPMs
IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a

# Or run the script directly with built RPMs (bypassing the FMF wrapper)
./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm <package_name>

./ci/dist_git.py update-releases

./ci/dist_git.py import fedora/bash

./ci/dist_git.py import --branch f42 fedora/glibc
./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git

./ci/dist_git.py update

./ci/dist_git.py update bash

./ci/dist_git.py sync bash

./ci/dist_git.py set-upstream bash --track
./ci/dist_git.py set-upstream bash --no-track

./ci/check_upstream_versions.py check
./ci/check_upstream_versions.py list
./ci/check_upstream_versions.py list --json

# Long-running package with custom timeout
setup:
  timeout_hours: 12

# Package requiring larger build instances
llvm:
  timeout_hours: 12
  build_platforms:
    - "linux-d160-c8xlarge/arm64"
    - "linux-d160-c8xlarge/amd64"

make generate

1 - RPM Pipeline

Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.

Overview

The RPM pipeline consists of five main stages:

1. Spec Change - A package spec file is modified in rpms/<package>/
2. Merge Request & Validation - CI validates the change
3. Build - Konflux builds RPMs per architecture via mock in hermetic mode
4. Signing - Built RPMs are cryptographically signed
5. Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository

After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.

flowchart TD
    A["Spec Change<br/>rpms/&lt;package&gt;/"] --> B["Merge Request"]
    B --> C["CI Validation<br/>(check, tree_status)"]
    C --> D["Merge to main"]
    D --> E["Konflux RPM Build<br/>Tekton PipelineRun per package<br/>mock hermetic build per arch"]
    E --> F["RPM Signing<br/>Kerberos-based"]
    F --> G["Publish to Pulp<br/>packages.redhat.com<br/>/public-hummingbird/&lt;arch&gt;/"]
    G --> H["Container lockfile updates<br/>(see Image Pipeline)"]

Stage 1: Spec Change

An RPM change begins as a commit in rpms/<package>/. The change typically includes a modified .spec file (updated version, new patch, or Release bump) and any associated source files or patches.

How changes originate

See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.

Package metadata

Each package has a metadata file at metadata/<package>.json that tracks its relationship to upstream:

{
  "source": "https://src.fedoraproject.org/rpms/curl.git",
  "branch": "rawhide",
  "sha": "abc123...",
  "version": "8.12.1",
  "release": "1",
  "modification_status": "clean",
  "upstream_repo": "https://github.com/curl/curl"
}

Key fields:

See Package Metadata Fields for modification_status and release configuration, and Package Modification Tracking for managing modification status day to day.

Stage 2: Merge Request & Validation

Changes reach main via merge requests. Automated MRs are created using:

• ci/create_mr.sh - Single MR with configurable options
• ci/rebuild_multi_mr.sh - One MR per package with auto-merge enabled
• ci/dist_git_update_multi_mr.sh - Automated Fedora update MRs (scheduled CI job)
• ci/upstream_update_multi_mr.sh - Automated upstream version update MRs (scheduled CI job)

The scheduled upstream-version update job creates normal MRs for successful package updates. If a package update fails after a new version has been identified, it creates (or reuses) a draft diagnostic MR instead. The diagnostic MR includes the original error but no partial package changes, and must be resolved manually before a package update can be proposed.

CI validation

GitLab CI runs validation jobs on every merge request:

• check - Linting, type checking, and spec validation
• check_nevr_conflicts - Predicts the NEVR(A) each changed package’s spec would build (using rpmspec with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale main, e.g. two rebuild MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch rpms/**/*, and within that, only checks packages whose rpms/<package>/ directory itself changed – matching Konflux’s own build trigger – not packages whose only change is under metadata/, since a metadata-only change (e.g. ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally with make check-nevr-conflicts ARGS='<package>', or against real built RPMs with make check-nevr-conflicts ARGS='--rpms-dir builds/<package>' after ci/build_rpms.sh. See ci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages routed to a private_product are noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode some OK results can be for a package name a real build never actually produces (e.g. a base package with no top-level %files section, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount a CONFLICT found elsewhere in the same run.
• tree_status - Repository consistency checks (metadata integrity, spec file presence)
• Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)

Auto-approval for chore MRs

Automated update MRs (chore/* branches) follow an accelerated path:

1. MR is created with auto-merge enabled
2. Konflux builds the package and posts commit statuses
3. After a delay, the chore_mr_approval CI job auto-approves the MR
4. GitLab merges the MR once the pipeline succeeds

Stage 3: Build

After merging to main, Konflux builds RPMs automatically.

Build triggers

Each package has a Tekton PipelineRun definition generated from its directory in rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered .tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers a build when changes to that package’s directory are pushed to main. When a single push touches multiple packages, one PipelineRun is triggered per affected package directory. Merge request pushes also trigger builds for validation.

The setup package acts as a canary: changes to ci/ or mock/ directories also trigger a setup build on pull requests, validating build infrastructure changes before they reach main.

Build process

The build pipeline (build-rpm-package) executes for each target architecture:

1. Checks out the repository at the merge commit
2. Calculates build dependencies
3. Runs mock in hermetic mode (network-isolated with pre-fetched dependencies)
4. Produces RPM artifacts and build logs
5. Generates an SBOM (Software Bill of Materials)
6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry

Build architectures

Packages are built for the architectures specified in their PipelineRun definition. Most packages build for x86_64 and aarch64. Some packages also build for s390x and ppc64le.

Build output

Build artifacts are stored in the Konflux OCI registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main

Konflux resources

Build infrastructure is defined across two repositories:

• rpms repo (konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy
• infrastructure repo (kubernetes/rpms-main/) - Application, ReleasePlan, IntegrationTestScenario, ServiceAccount, and Secret resources

See Konflux Resource Deployment for details on how these resources are managed and deployed.

Testing

RPM packages are validated through integration tests that run on Testing Farm infrastructure.

Tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test triggering

Tests are only triggered on merge requests, not on main branch builds. For a package at rpms/<package>/, tests are triggered for all changes below that directory.

Integration Test Scenario

Tests are triggered via an IntegrationTestScenario resource defined in the infrastructure repository:

Configuration: infrastructure/kubernetes/rpms-main/10-integration-test-scenarios-testing-farm.yml.j2

The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:

Scenario parameters:

• COMPOSE: RHEL-9-Nightly - Test environment OS/compose
• PIPELINE_MODE: rpm - Configures RPM testing mode (vs. container)
• ARCH: x86_64,aarch64 - Architectures to test (creates separate test runs per arch)
• PASS_SNAPSHOT_TO_TF: false - Don’t pass full snapshot JSON (package info comes via IMAGE_NAME/IMAGE_URL instead)
• IMAGE_TAG: - Version of the tmt-via-testing-farm pipeline bundle

Key characteristics:

• Single scenario for all packages - Not per-package, runs for every package build
• Context-based triggering - Runs on pull_request snapshots only
• Package identification - Konflux automatically provides IMAGE_NAME (e.g., openssl-main) and IMAGE_URL (OCI image with built RPMs) as environment variables

FMF test plan

The root folder of the rpms repository is marked with a .fmf directory to identify it as an FMF metadata tree for tmt. The test plan in ci/run_tests_rpm.fmf sets up the test instance and runs default and package-specific tests via ci/run_tests_rpm.sh.

Test environment

These environment variables are available to the tmt test execution:

Stage 4: Signing

The Konflux Release Service signs built RPMs before publishing:

• Method: Kerberos-based signing via konflux-release-signing-prod@IPA.REDHAT.COM
• Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
• Signing image: quay.io/konflux-ci/signing:latest (verify pinned digest in ReleasePlanAdmission)

Stage 5: Publishing to Pulp

After signing, RPMs are uploaded to the Hummingbird Pulp repository.

Release pipeline

The push-rpms-to-pulp release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (releng/hummingbird-rpms-tech-preview-staging.yaml):

mapping:
  rpm-repositories:
    - name: x86_64
      repository_id: public-hummingbird-x86_64-rpms
    - name: aarch64
      repository_id: public-hummingbird-aarch64-rpms
    - name: src
      repository_id: public-hummingbird-source-rpms
    # See ReleasePlanAdmission for the full list of published architectures

Pulp repository

Published RPMs are available in the Pulp content index:

Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.

What Happens Next

Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.

Related Documentation

• Rebuilding Packages - No-change rebuilds, reverse dependency rebuilds, and patch backports
• Updating Dist-git Packages - Automated Fedora sync workflow
• Package Metadata Fields - modification_status and release configuration
• Package Modification Tracking - Managing modification status and version constraints
• Konflux Resource Deployment - How Konflux resources are defined and deployed
• Image Pipeline - Container image pipeline (continues from where this document ends)

2 - Konflux Resource Deployment

Konflux resources for RPM packages are split between two repositories:

• rpms: Per-package resources generated from package directories
• infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Legend:

• RPMs MR: Downstream pipeline triggered from rpms MR (deploys from MR commit)
• RPMs Push: Downstream pipeline triggered from rpms push to main
• Infra MR: Infrastructure MR pipeline (manual trigger)
• Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

RPMs Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

1. Ownership: Components are managed by the rpms repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle.
2. Dynamic generation: Generated from rpms/*/ directories via ci/generate_konflux_resources.sh, depending on package presence and configuration.
3. Timing: Must be deployed early during MR review so Konflux can build and test new packages.

ReleasePlanAdmission:

1. Static configuration: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern.
2. Main branch only: Deployed only on push to main to maintain deployment consistency.

EnterpriseContractPolicy:

1. Consistency: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern.

Infrastructure Repo

Application, ReleasePlan, and IntegrationTestScenario are defined in kubernetes/rpms-main/:

1. Independent iteration: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo.
2. Testable before merge: If defined in the rpms repo, these would only deploy after merging to main, making iteration difficult.
3. Static configuration: These resources don’t depend on per-package data.

ServiceAccount and Secret for releases are defined in kubernetes/setup-konflux/:

1. Security: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo.
2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo.

The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp.

3 - Source Pipeline Tool

Source Pipeline Tool

Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.

Problem Statement

RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:

1. Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.

2. Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.

3. Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.

4. Unverified patches. Patches (.patch files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.

Tool Overview

A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.

                          ┌──────────────────────────────┐
                          │     source-pipeline tool     │
                          │      (container image)       │
                          │                              │
  ┌───────────┐           │  ┌───────┐    ┌───────────┐  │    ┌───────────────┐
  │  package  │           │  │ fetch │───▸│ transform │  │    │   tarballs    │
  │   dir     │──────────▸│  └───────┘    └─────┬─────┘  │───▸│   sources     │
  │ (spec,    │           │                     │        │    │   report      │
  │  patches, │           │  ┌───────────┐      │        │    └───────┬───────┘
  │  config)  │           │  │  verify   │◂─────┘        │            │
  │           │           │  └─────┬─────┘               │            ▼
  └───────────┘           │        │                     │    ┌───────────────┐
                          │  ┌─────▼─────┐               │    │   lookaside   │
  ┌───────────┐           │  │  enforce  │               │    │    cache      │
  │ pipeline  │           │  │  policy   │               │    └───────────────┘
  │   yaml    │──────────▸│  └─────┬─────┘               │
  │           │           │        │                     │
  └───────────┘           │  ┌─────▼─────┐               │
                          │  │   emit    │               │
                          │  └───────────┘               │
                          └──────────────────────────────┘

Design principles

• Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
• Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
• Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
• Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.

Container interface

podman run --rm \
  -v ./<package-dir>:/package:ro \
  -v ./pipeline.yaml:/pipeline.yaml:ro \
  -v ./gpg-keys:/gpg-keys:ro \
  -v ./output:/output \
  source-pipeline:latest \
  --version <new-version> \
  [--old-version <old-version>] \
  [--verify-patches /patches-from-upstream] \
  [--dry-run]

Inputs (mounted read-only):

• /package — the package directory (spec file, patches, existing sources file)
• /pipeline.yaml — the declarative pipeline definition
• /gpg-keys — centralized GPG keyring directory

Outputs (written to /output):

• Source tarballs (ready for lookaside upload)
• sources — updated sources manifest with checksums
• report.json — verification and policy results (pass/fail per check, patch classifications)

Exit codes:

• 0 — success, all checks passed
• 1 — error (download failure, tool error)
• 2 — policy violation (verification or constraint failure)

--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to /output. Exit codes and report.json behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing.

On any non-zero exit, report.json is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no package update commit is created, the package stays at its current version, and the automation continues to the next package. In the scheduled upstream-version-update job, the wrapper also creates (or reuses) a draft diagnostic MR for a per-package update failure. That MR contains the full error and no partial package changes, so a maintainer can reproduce and resolve the problem safely. Transient failures (exit 1) may self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention.

Pipeline Stages

1. Fetch

Downloads source artifacts directly from upstream.

• Parses Source: URLs from the spec file (resolving RPM macros like %{version}, %{name}, %{url})
• Downloads each source from the upstream URL
• Supports fetching from git repositories at a tag, branch, or commit
• For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)

2. Transform

Applies per-package source modifications.

• Vendor archive generation — runs ecosystem-specific tooling (e.g., go_vendor_archive for Go, npm pack for Node.js, cargo vendor for Rust)
• Vendor dependency pinning — modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix.
• Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
• UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
• Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
• Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a toolchain: section. The container ships defaults; per-package overrides ensure that build-ui, vendor, and run: steps use the correct toolchain without requiring separate container images per package

Known sharp edge: patch-list duplication

Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill point back here instead of re-telling the story.

Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ... followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain their own copy of “which patches touch this source tree” rather than reading it from the spec’s PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see Rebuilding Packages) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform’s install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation step, and the mismatch didn’t surface until the next automated version bump re-ran the transform from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact.

go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up requiring different versions of the same package — go build -mod=vendor rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s %check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

3. Verify

Validates integrity and authenticity of fetched sources.

• GPG signature verification — downloads signature files (.asc, .sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation.
• Checksum verification — compares against published checksums where available
• Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
• Re-publication detection — if a previously committed checksum exists in the sources file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an accepted-checksums entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.

4. Enforce policy

Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect) and catches violations in packages that don’t use vendor-pin.

• Vendor dependency constraints — ensures vendored dependencies meet version requirements (e.g., sanitize-html >= 2.17.5 for a CVE fix)
• Ecosystem-specific checks — Go module verification, npm audit, cargo audit
• License compliance — flags vendored dependencies with incompatible licenses
• Fail-closed: any policy violation exits with code 2

5. Emit

Produces final artifacts.

• Writes tarballs to the output directory
• Generates sources file in dist-git format (SHA512 (filename) = hash)
• Writes report.json with verification results, policy check results, and patch classification

Pipeline Schema

Per-package pipeline definitions are YAML files that declare the full source generation workflow.

Packages without a pipeline definition use a built-in default that fetches sources from the spec’s Source: URLs — no transforms, no verification, no policy.

Schema definition

# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
  # Macro substitutions applied before Source URL resolution
  macros:
    - name: go_patch                     # %global go_patch <new-value>
      value: "${VERSION_PATCH}"          # extracted from VERSION (e.g., 1.25.3 → 3)

    - name: k8s_ver                      # %global k8s_ver <new-value>
      value: "${VERSION}"

  # Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
  reset-release: true

# Source fetching
fetch:
  sources:
    # Fetch Source0 from the URL declared in the spec
    - spec-source: 0

    # Fetch Source1 (e.g., a GPG signature file)
    - spec-source: 1

    # Or fetch from an explicit URL (for sources not in the spec)
    - url: "https://example.com/extra-source-${VERSION}.tar.gz"

    # Or fetch from a git repository at a tag/commit
    - git:
        repo: "https://github.com/example/project"
        ref: "v${VERSION}"              # tag, branch, or commit hash
        include-history: false           # include .git dir in tarball (default: false)

  # Vendor archive generation (optional)
  vendor:
    ecosystem: go                        # go | npm | cargo | composer
    config: go-vendor-tools.toml         # ecosystem-specific config file in the package dir
    source-dir: .                        # directory to vendor from (default: extracted source root)
    submodules:                          # (optional) vendor multiple Go submodules independently
      - server
      - etcdctl
      - etcdutl

# Source transforms (optional, ordered)
transform:
  # Built-in transform types
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
        - "deps/ngtcp2/ngtcp2/crypto/"
      output: "node-v${VERSION}-stripped.tar.gz"

  # Pin vendored dependency versions before vendor archive generation.
  # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
  # bump specific dependencies, then re-resolves the dependency graph.
  # Runs before the vendor stage so the pinned versions are included in
  # the vendor archive. Re-applied on every source generation, so upstream
  # updates cannot silently revert a security fix.
  #
  # Version constraints are always minimum versions — the version field
  # means "at least this version." The tool translates to ecosystem-native
  # operations:
  #   Go:    go get <package>@v<version> (minimum version selection)
  #   npm:   npm install <package>@">= <version>"
  #   Cargo: set dependency requirement to ">= <version>" in Cargo.toml
  - vendor-pin:
      - package: golang.org/x/crypto
        ecosystem: go
        version: "0.31.0"
        reason: "CVE-2024-45337"

      - package: sanitize-html
        ecosystem: npm
        version: "2.17.5"
        reason: "CVE-2024-XXXXX"

      - package: tokio
        ecosystem: cargo
        version: "1.38.1"
        reason: "CVE-2024-YYYYY"

  # Custom command (escape hatch)
  - run: "./packaging/make-tarball.sh ${VERSION}"
    outputs:
      - "package-${VERSION}-stripped.tar.gz"

  # UI asset build
  - build-ui:
      ecosystem: npm                     # npm | yarn
      source-dir: "web/ui"
      output: "${PACKAGE}-${VERSION}-ui.tar.gz"

# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
  node: "20"                             # Node.js version for build-ui, npm vendor, and run: steps
  go: "1.23"                             # Go version for go vendor and run: steps
  rust: "1.80"                           # Rust version for cargo vendor and run: steps
  python: "3.12"                         # Python version for run: steps

# Integrity verification (optional)
verify:
  gpg:
    signature-source: 1                  # spec Source index containing the .asc/.sig
    keyring: "curl.gpg"                  # key name in the GPG keys directory
  checksums:
    url: "https://example.com/SHA256SUMS"
    algorithm: sha256

  # Override for upstream re-publications. Required when upstream re-publishes
  # a release artifact with different content under the same version. The
  # pipeline refuses to accept changed content automatically — a human must
  # add the new checksum here after investigation.
  accepted-checksums:
    - file: "example-1.2.3.tar.gz"
      sha512: "abc123..."
      reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"

# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
  vendor-constraints:
    - package: sanitize-html
      ecosystem: npm
      version: "2.17.5"
      reason: "CVE-2024-XXXXX"

    - package: golang.org/x/crypto
      ecosystem: go
      version: "0.31.0"
      reason: "CVE-2024-45337"

# Patch verification (optional)
patches:
  verify: true                           # diff-detect new/changed patches
  classify: true                         # attempt to classify by origin
  upstream-repo: "https://github.com/curl/curl"
  fail-on-unverified: false              # warn-only by default

  # Patch lifecycle rules (optional)
  # Declares version-scoped applicability for patches. The pipeline tool reads
  # these rules from patch headers (preferred) or from this YAML, and enforces
  # them during updates: patches outside their valid range are flagged or dropped.
  lifecycle:
    - file: "fix-memory-leak.patch"
      applies-to: "< 1.5.0"             # drop this patch at version 1.5.0+
      reason: "Fixed upstream in 1.5.0 (commit abc123)"
      action: drop                       # drop | warn (default: warn)

    - file: "cve-2024-45337.patch"
      applies-to: "< 0.31.0"
      reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
      action: drop

    - file: "distro-branding.patch"
      applies-to: "*"                    # carry forward unconditionally
      reason: "Distribution-specific branding, always required"

# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
  # Extract bundled dependency versions and splice into spec between markers
  - bundled-provides:
      modules-txt: "vendor/modules.txt"  # Go modules.txt path inside extracted source
      start-marker: "# --- bundled-deps.sh ---"
      end-marker: "# --- end bundled-deps.sh ---"

  # Run a custom command (escape hatch for complex metadata extraction)
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"

Variable substitution

The following variables are available in all string values:

Version constraint semantics

All version: fields in vendor-pin and policy.vendor-constraints use minimum-version semantics: the value means “at least this version.” The tool translates this into ecosystem-native operations:

• Go: go get <package>@v<version> (minimum version selection via MVS)
• npm: npm install <package>@">= <version>"
• Cargo: sets the dependency requirement to >= <version> in Cargo.toml

Default behavior (no pipeline YAML)

When no pipeline YAML is provided, the tool applies a built-in default:

1. Parse all Source: URLs from the spec
2. Download each from the upstream URL
3. Generate sources file with SHA512 checksums
4. No transforms, no verification, no policy enforcement

This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.

Patch Verification

When packages are updated from an upstream distribution, patch files (.patch) may arrive without inspection. The pipeline tool can optionally verify and manage patches.

Verification approach

The pipeline tool can optionally verify patches when invoked with --verify-patches:

1. Diff detection. Compare the set of .patch files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches.

2. Header parsing. Extract metadata from patch headers:

   • From: — author identity
   • Subject: — description
   • Commit hash references (e.g., From <hash>, cherry picked from commit <hash>)
   • Bug: / CVE: references

3. Classification. Attempt to classify each patch:

   • Upstream backport — references a commit hash that exists in the upstream repo
   • CVE fix — references a CVE identifier
   • Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
   • Distribution-specific — modifies paths, branding, or distribution-specific integration
   • Unclassified — cannot be automatically categorized

4. Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via git ls-remote or the upstream API).

5. Reporting. Output classification and verification results in report.json. Optionally fail on unverified patches (controlled by patches.fail-on-unverified in the pipeline YAML).

Patch lifecycle enforcement

Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.

The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML’s patches.lifecycle section.

Header-based declaration (preferred). Patch authors add structured keywords to the patch header:

From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---

YAML-based declaration (fallback). For patches from upstream distributions that cannot have headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch.

Enforcement behavior:

1. During an update to version ${VERSION}, the tool evaluates each patch’s applies-to range against the new version.
2. If a patch is outside its valid range:
   • action: drop — the patch file is deleted from the package directory and the corresponding Patch: declaration and %patch / %autopatch application directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in report.json.
   • action: warn (default) — the patch is flagged in report.json but not removed. The update proceeds.
3. If a patch has applies-to: *, it is always carried forward.
4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to applies-to: *, action: warn).

Carry-forward enforcement. The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked applies-to: * is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error.

Example report output

{
  "patches": {
    "new": [
      {
        "file": "fix-memory-leak.patch",
        "classification": "upstream-backport",
        "upstream_commit": "abc123def456",
        "verified": true
      }
    ],
    "removed": ["old-workaround.patch"],
    "unchanged": ["distro-paths.patch"],
    "lifecycle": [
      {
        "file": "cve-2024-45337.patch",
        "applies_to": "< 0.31.0",
        "current_version": "0.31.0",
        "action": "drop",
        "result": "removed — version 0.31.0 is outside applies-to range"
      },
      {
        "file": "cve-2024-99999.patch",
        "applies_to": "< 2.0.0",
        "current_version": "2.0.0",
        "action": "drop",
        "result": "flagged — patch is applied inside a conditional %if block; manual removal required"
      },
      {
        "file": "distro-branding.patch",
        "applies_to": "*",
        "action": "carry-forward",
        "result": "present"
      }
    ]
  }
}

Examples

Trivial package (curl)

No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you want GPG verification:

fetch:
  sources:
    - spec-source: 0
    - spec-source: 1

verify:
  gpg:
    signature-source: 1
    keyring: "curl.gpg"

Transformed package (Node.js)

Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:

spec-update:
  macros:
    - name: nodejs_define_version node
      value: "${VERSION}"
  reset-release: true

toolchain:
  node: "22"

fetch:
  sources:
    - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"

transform:
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
      output: "node-v${VERSION}-stripped.tar.gz"

verify:
  checksums:
    url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
    algorithm: sha256

post:
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"

Simple Go vendor (caddy)

A pattern shared by nats-server, oauth2-proxy, and similar Go projects:

fetch:
  sources:
    - git:
        repo: "https://github.com/caddyserver/caddy"
        ref: "v${VERSION}"

  vendor:
    ecosystem: go

Multi-submodule Go vendor (etcd)

fetch:
  sources:
    - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    submodules:
      - server
      - etcdctl
      - etcdutl

Code generation + Go vendor (opentelemetry-collector-contrib)

Built-in primitives handle fetch and vendor; the code generation step requires run::

fetch:
  sources:
    - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    source-dir: _build

transform:
  - run: |
      curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
      curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
      sha256sum -c --ignore-missing ocb_checksums.txt
      chmod +x ocb
      tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
      cd "opentelemetry-collector-releases-${VERSION}"
      if [ -f scripts/prepare-obi.sh ]; then
        bash scripts/prepare-obi.sh otelcol-contrib
      fi
      ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
    outputs:
      - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"

Git snapshot (libXtst)

A pattern shared by libX11, libXext, libXi, libXrender:

fetch:
  sources:
    - git:
        repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
        ref: "libXtst-${VERSION}"
        include-history: true

Built-in Primitives Coverage

An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:

Fully declarative (no run: needed) — ~30 script files (15 unique patterns):

Partially declarative (generic primitives + 1–2 run: steps):

Genuinely custom (run: required) — 7 packages:

The run: escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract.

Open Questions

1. Implementation approach. The pipeline YAML schema (ordered transform stages, run: escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:
   • Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
   • Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
   • Shared shell function library — the majority of scripts decompose into 3–4 operations (vendor_go, strip_tarball, fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands.
   • Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no run: blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.

Hummingbird Integration

This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.

Current Architecture

Two update systems

dist_git.py update flow:

1. Clones Fedora dist-git for the package
2. Checks version, Koji build status, pre-release filtering
3. Copies entire Fedora dist-git checkout into rpms/<package>/ via shutil.copytree — this includes the spec, patches, .gitignore, and sources file
4. For modified packages, performs a 3-way git merge to preserve local changes
5. Commits everything (spec, patches, sources file, metadata)

The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash). Actual tarballs are excluded by .gitignore and stored in a lookaside cache.

check_upstream_versions.py flow:

1. Queries release-monitoring.org for new upstream versions
2. Runs three hook phases per package:
   • update_spec — updates the spec’s Version/Release (default: specfile.update_version())
   • download_sources — downloads source tarballs (default: fetches from spec Source URLs)
   • post_update — additional steps (default: no-op)
3. Uploads downloaded tarballs to Hummingbird’s lookaside cache
4. Updates the sources file with new checksums
5. Commits

Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.

Three lookaside cache backends

Configured in mock/dist-git-client.ini:

The forked_from field in ci/package-overrides.yaml determines which cache a package uses at build time. Packages without forked_from default to Fedora’s cache.

How Konflux builds consume sources

Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):

• monorepo-subdir: rpms/<package> — locates the spec, patches, and sources file
• package-name: <name> — upstream name for lookaside URL construction
• forked-from: <url> — (optional) selects the lookaside backend
• dist-git-client-configdir: mock/ — (optional) points to the custom dist-git-client.ini

The pipeline uses dist-git-client to read the sources file and download tarballs from the appropriate cache, then runs mock to build RPMs.

Existing source generation patterns

Some independent packages already have tarball generation scripts:

• rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh — downloads upstream release, runs OCB to generate source code, vendors Go dependencies
• rpms/caddy/create-vendor-tarball.sh — downloads upstream tarball, generates Go vendor archive
• rpms/nodejs25/packaging/make-nodejs-tarball.sh — downloads upstream, strips bundled content

These scripts are wired into the update path via download_sources hooks in metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache.

Integration with dist_git.py update

The current flow copies everything from Fedora dist-git (including the sources file) via shutil.copytree. The pipeline tool inserts after this step:

Current:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Commit

Proposed:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Run source-pipeline tool             ← fetches from upstream, replaces sources file
  4. Upload tarballs to Hummingbird lookaside
  5. Commit                               ← sources file now points to Hummingbird lookaside

Implementation:

• After the shutil.copytree (or merge for modified packages), check if metadata/<package>.source-pipeline.yaml exists
• If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the sources file
• If it does not exist: run the built-in default (fetch from spec URLs, upload, replace sources)
• A --skip-pipeline flag allows falling back to the current behavior during migration
• Update ci/package-overrides.yaml to set forked_from to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)

Integration with check_upstream_versions.py

The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating per-package update behavior into a single *.source-pipeline.yaml file:

Current:
  1. update_spec hook/default             ← *.update-hooks.yaml
  2. download_sources hook/default        ← *.update-hooks.yaml
  3. post_update hook/default             ← *.update-hooks.yaml

Proposed:
  1. spec-update (pipeline YAML)          ← replaces update_spec hooks
  2. fetch + transform (pipeline YAML)    ← replaces download_sources hooks
  3. post (pipeline YAML)                 ← replaces post_update hooks

This reduces per-package metadata from three files (metadata/<package>.json, *.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for identity/tracking, *.source-pipeline.yaml for all update behavior).

Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed.

Integration with Konflux

Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via dist-git-client as it does today.

Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.

Hummingbird-specific configuration

Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.

GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.

The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.

Local development

Developers can run the tool directly:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  -v /tmp/output:/output \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0

# Outputs in /tmp/output/:
#   curl-8.21.0.tar.xz
#   curl-8.21.0.tar.xz.asc
#   sources
#   report.json

To preview without producing artifacts:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0 \
  --dry-run

# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.

Migration Path

Phase 1: Independent Go packages

Migrate the 22 independent packages that already fetch from upstream. Convert their existing create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions. Validates the tool against known-good packages.

Scope: 7 packages with existing scripts + 15 without.

Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code generation, but their vendor step is generic.

Phase 2: Trivial clean packages

Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.

Prerequisites:

• Audit to classify packages as trivial vs. transformed (HUM-4620)
• dist_git.py update integration complete (HUM-4621)

Per-package steps:

1. Run the pipeline tool, compare output against existing Fedora-sourced tarball
2. If identical: add forked_from: hummingbird to package-overrides.yaml
3. If different: flag for Phase 3

Phase 3: Transformed packages

Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:

• Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): strip-tarball
• Git snapshots (libX11 family×5, oniguruma, php-jsqueeze): fetch: {git: ...}
• Composer vendor (composer): vendor: {ecosystem: composer}

Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require run: blocks for genuinely custom logic.

Scope: determined by Phase 2 audit.

Phase 4: Policy enforcement

Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates.

Phase 5: Patch verification

Enable patches.verify and patches.classify for packages. Start with warn-only (fail-on-unverified: false), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages.

Operational Health

Escape hatch ratio

The primary health metric for the source pipeline is the escape hatch ratio: the percentage of pipeline YAML definitions that contain run: blocks (in transform or post sections) relative to the total number of pipeline YAML definitions.

A run: block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the same way: a package that cannot be expressed declaratively gets a run: block. This makes the ratio a single number that aggregates all pressures on the declarative model.

Interpretation:

When the ratio climbs, examine the run: blocks for repeated patterns. If three or more packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad – it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set.

CI enforcement

The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files, calculates the escape hatch ratio, and reports the result. It runs as part of make check.

• During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need run: blocks before new primitives are added.
• After Phase 3: enable --fail mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a run: block is added.