Lot's of background information

• 1: RPMs repository
• 1.1: RPM Pipeline
• 1.2: Konflux Resource Deployment
• 1.3: Source Pipeline Tool
• 2: Agent-Friendly Documentation
• 3: Containers repository
• 3.1: Image Pipeline
• 3.2: Global Variables Reference
• 3.3: Konflux Resource Deployment
• 3.4: Security Labels and Metadata
• 3.5: Container Image Labels
• 3.6: CI Scripts
• 3.6.1: build_images.sh
• 3.6.2: run_tests_container.sh
• 3.6.3: run_tests_k8s.sh
• 3.6.4: retrigger_failed_checks.py
• 3.6.5: gitlab_sync.py
• 4: K8s Test Pipeline
• 4.1: Pipeline Design
• 4.2: Test Format
• 4.3: EaaS and Debugging
• 5: Tools repository
• 5.1: Message Bus Architecture
• 5.2: Alloy CloudWatch
• 5.3: Deployment Bot
• 5.4: Grafana CloudWatch
• 5.5: Hummingbird AI
• 5.6: Hummingbird Events Topic
• 5.7: Vertex AI Cost Metrics
• 5.8: SNS S3 Archiver
• 5.9: Hummingbird Tools
• 5.10: ProdSec RPM Catalog
• 5.11: Chainguard Comparison
• 5.12: CVE Analysis
• 5.13: Dashboard MR Linker
• 5.14: GitLab Event Forwarder
• 5.15: Hummingbird MR Human Tracker
• 5.16: MR Auto-Approver
• 5.17: CVE Analysis Reference
• 5.18: Jira Event Forwarder
• 5.19: Container Catalog
• 5.20: Hummingbird Agent
• 5.21: Hummingbird Agent Model Loop
• 5.22: Red Hat Catalog
• 5.23: Hummingbird Agent Design
• 5.24: Hummingbird Agent Design — Engine
• 5.25: Hummingbird Agent Evals
• 5.26: Documentation Quality Evaluation
• 5.27: Hummingbird Agent Design — Runtime
• 5.28: Hummingbird Agent Design — Decisions
• 5.29: Hummingbird Data Flow
• 5.30: Kubernetes Event Forwarder
• 5.31: Jira Image Requests
• 5.32: PAC Trigger
• 5.33: Hummingbird Status
• 5.34: Lambda S3 Cache
• 5.35: RPM CVE Count
• 5.36: Hummingbird Dashboard
• 5.37: Pipeline Timing
• 5.38: RPM Tracker
• 5.39: Work Queue Service
• 5.40: Work Queue Service Design
• 5.41: Work Queue Service Design Decisions
• 5.42: Service metrics
• 5.43: VEX Checker
• 5.44: Hummingbird CVE agent
• 5.45: Red Hat Catalog Environment Promotion
• 5.46: S3 Lookaside Cache
• 5.47: Docs Content Negotiation
• 5.48: Red Hat Catalog UAT Program
• 5.49: Hummingbird MR Collaboration
• 5.50: K8s Integration Tests

1 - RPMs repository

Thank you for your interest in contributing! This repository contains RPM packaging, minimal tests, and CI plumbing used by Hummingbird to build, lint, and validate RPMs in containers and Testing Farm.

This guide is adapted from the Hummingbird containers contribution guide and aligned with our workflows and tooling. See the original for broader context: Hummingbird Containers CONTRIBUTING.

Code of Conduct

Be respectful and constructive. By participating, you agree to uphold a professional and inclusive environment.

Repository layout

• rpms/<package>/ – RPM dist-gits (spec, sources), mostly auto-imported
• ci/ – helper scripts and default tests
  • build_rpms.sh – build RPMs using Mock in the Konflux-compatible container
  • run_tests_rpm.sh – run rpmlint and install/rebuild tests in a container
  • default-tests/tests-rpm.yml – default tests included for all packages
  • run_tests_rpm.fmf – tmt/Testing Farm entry point
• konflux-templates/ – Konflux/PAC resources
• mock/ – mock configuration
• test/rpms/<package>.yml – package-specific tests

Prerequisites

• Fedora or RHEL-like environment
• Podman
• rpmlint
• tmt (optional, for local TF-style runs)
• jq, python3, python3-yaml (used by run_tests_rpm.sh)

Build locally

Build a package’s SRPM and RPMs using the Konflux-aligned environment:

./ci/build_rpms.sh <package_name>
# Results are written to: /tmp/konflux-build-<package_name>-*/results/

Building in Lima VM

When building inside a Lima VM (macOS users), use the --build-dir flag to specify a directory on the VM’s native filesystem. This avoids permission issues with mock’s bootstrap process on macOS mounts:

limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build <package_name>'

The build directory must be on the VM’s native filesystem (not a macOS mount) to ensure Linux file ownership, permissions, and symlinks work correctly with /var/lib/mock.

Interactive repository debugging

To investigate package/dependency/installability issues, you can run an interactive shell in the same environment used by the package builds:

./ci/build_rpms.sh --shell-before setup

This will show you the mock command that would be used to build the setup package, and drop you into a shell in the same environment before executing the build, with dnf available. You can then run the command manually and inspect the environment, repositories, and package metadata.

If you want it to actually build the local package first, so that you can include it in your investigation, use --shell-after instead:

./ci/build_rpms.sh --shell-after setup

Test locally (containerized)

Run the default and package-specific tests against one or more built RPMs:

# Basic usage (binary rpm)
./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm <package_name>

# Include source RPM tests
./ci/run_tests_rpm.sh \
  --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \
  --src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \
  <package_name>

# Test all RPMs in the build output directory after build_rpms.sh
./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds/<package_name>/RPMS/*.rpm) \
  --repo-dir builds/<package_name>/RPMS/ \
  --src-rpm builds/<package_name>/SRPMS/*.src.rpm \
  <package_name>

Notes:

• Tests run inside a Podman container.
• The test image defaults to registry.access.redhat.com/hi/core-runtime:latest-builder. You can override via TEST_IMAGE:

TEST_IMAGE=registry.access.redhat.com/hi/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm <package_name>

• Container execution is performed as root with HOME=/root to avoid XDG state permission issues in dnf5.

Test in tmt/Testing Farm locally

You can drive the same FMF test locally with tmt. The FMF test expects an OCI artifact that contains RPMs, referenced via IMAGE_URL (Testing Farm sets this automatically). For local trials you can point to any compatible OCI artifact or skip the ORAS pull logic and directly call the script.

# If using an OCI artifact with RPMs
IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a

# Or run the script directly with built RPMs (bypassing the FMF wrapper)
./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm <package_name>

If you need longer time in Testing Farm, the FMF test includes duration, which you can adjust in ci/run_tests_rpm.fmf.

Dist-git Imports

The ci/dist_git.py tool imports Fedora/CentOS dist-git packages into the rpms/ directory. We expect most packages to not have (permanent) Hummingbird specific changes, so most of them will keep syncing with upstream dist-gits. In most cases that will be Fedora rawhide, but for some packages we may pick a different upstream, e.g. stable Fedora or even CentOS Stream.

The status of all imports is tracked in imports.json. Active Fedora releases are tracked in upstream-releases.json, which is updated from the Bodhi API.

Note that rawhide is automatically resolved to the highest numbered Fedora release at runtime and is not stored in the JSON file.

See ./ci/dist_git.py --help for all available options. Some examples:

• Update upstream-releases.json from Bodhi API (should be done periodically, e.g., when new Fedora versions are released):

./ci/dist_git.py update-releases

This fetches the latest Fedora releases from Bodhi. The rawhide branch automatically resolves to the highest numbered Fedora version (e.g., if f43 and f44 are available, rawhide uses f44).

• Import a new package. This requires specifying the dist-git URL (with fedora/ being a shortcut for the Fedora dist-git URL) and optionally a branch (default: rawhide):

./ci/dist_git.py import fedora/bash

./ci/dist_git.py import --branch f42 fedora/glibc
./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git

• Update all or a single package:

./ci/dist_git.py update

./ci/dist_git.py update bash

This only imports changes if these were actually built in Koji, to ensure we only import changes which are meant to be released. You can disable this check with --skip-build-check.

• Re-sync a package to upstream, discarding any local modifications. We use this after Fedora adopted our change, or it is no longer relevant:

./ci/dist_git.py sync bash

All of these commands automatically commit changes with descriptive commit messages including the upstream SHA. To avoid that, you can use the --dry-run option.

• Enable upstream version tracking for a package (used by check_upstream_versions.py check). These commands modify the metadata file but do not create a git commit:

./ci/dist_git.py set-upstream bash --track
./ci/dist_git.py set-upstream bash --no-track

• Check upstream version status. The check subcommand checks tracked packages; list shows all packages in a table:

./ci/check_upstream_versions.py check
./ci/check_upstream_versions.py list
./ci/check_upstream_versions.py list --json

Package-specific overrides

Per-package build configuration can be customized in ci/package-overrides.yaml. If a package is not listed, it uses default settings.

Available options:

Example configuration:

# Long-running package with custom timeout
setup:
  timeout_hours: 12

# Package requiring larger build instances
llvm:
  timeout_hours: 12
  build_platforms:
    - "linux-d160-c8xlarge/arm64"
    - "linux-d160-c8xlarge/amd64"

All available MPLs can be found in the Konflux multi-platform builds documentation.

After modifying overrides, regenerate the pipeline files:

make generate

This updates .tekton/rpms-on-push.yaml and .tekton/rpms-on-pull-request.yaml with the new configuration.

Branching and pull requests

• Create feature branches from the default branch.
• Keep edits focused and small; separate unrelated changes into separate PRs.
• Include meaningful commit messages (why + what). Reference related issues if applicable.
• PRs must pass CI (build + tests). Fix lint/test failures or mark known failures properly.

Commit message conventions

• First line: short imperative summary (≤ 72 chars)
• Body (optional): context, rationale, and user/ops impact
• Reference issues using standard notation (e.g. “Fixes: #123”)

Spec guidelines

• Don’t use %changelog for Hummingbird specific changes. It’s just a point of conflict with Fedora imports, commit messages are good enough.
• Hummingbird specific changes increase Release: number in steps of 0.1 (e.g. 2 → 2.1 → 2.2). This avoids colliding with Fedora’s Release number namespace and allows us to sync again to a future -3.

Packaging and testing guidelines

• Spec files should be reproducible and minimal.
• Prefer pinned container digests for CI images where feasible. If a tag and digest are both present (name:tag@sha256:<digest>), the digest is authoritative for content selection.
• Default tests in ci/default-tests/tests-rpm.yml must be fast, deterministic, and safe for all packages.
• Package-specific tests (rpms/<package>/tests-rpm.yml) can override or extend defaults. Keep them bounded in runtime and dependencies.
• When possible, capture flaky or environmental issues under known_issues in test YAML with clear matching patterns and descriptions.

Style

• Shell: bash with set -euo pipefail, readable variable names, and early returns where reasonable.
• YAML: consistent indentation and quoting; prefer explicitness over magic.
• Comments: add when necessary to explain non-obvious rationale; avoid restating the code.

Security and supply chain

• Avoid embedding credentials or secrets; use environment variables or CI secret stores.
• Prefer pulling images by digest for stability and repeatability in CI.
• Validate inputs and sanitize any paths used by scripts.

Resolving CVEs

When manually fixing a CVE (e.g., adding a patch or performing a manual update), a Fixes: CVE-YYYY-XXXX must be added to the MR description and/or a commit. This will indicate to the build system that a CVE fix has been submitted for review.

Reporting issues

Open an issue describing the problem, reproduction steps, and environment. Attach logs (build/test) when possible. If the failure is intermittent, call that out explicitly.

Licensing

Ensure files include appropriate licenses and that any third-party content is compatible with the project’s license.

Thank you

Your contributions make the Hummingbird RPMs better for everyone. We appreciate your time and feedback!

1.1 - RPM Pipeline

Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.

Overview

The RPM pipeline consists of five main stages:

1. Spec Change - A package spec file is modified in rpms/<package>/
2. Merge Request & Validation - CI validates the change
3. Build - Konflux builds RPMs per architecture via mock in hermetic mode
4. Signing - Built RPMs are cryptographically signed
5. Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository

After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.

flowchart TD
    A["Spec Change<br/>rpms/&lt;package&gt;/"] --> B["Merge Request"]
    B --> C["CI Validation<br/>(check, tree_status)"]
    C --> D["Merge to main"]
    D --> E["Konflux RPM Build<br/>Tekton PipelineRun per package<br/>mock hermetic build per arch"]
    E --> F["RPM Signing<br/>Kerberos-based"]
    F --> G["Publish to Pulp<br/>packages.redhat.com<br/>/public-hummingbird/&lt;arch&gt;/"]
    G --> H["Container lockfile updates<br/>(see Image Pipeline)"]

Stage 1: Spec Change

An RPM change begins as a commit in rpms/<package>/. The change typically includes a modified .spec file (updated version, new patch, or Release bump) and any associated source files or patches.

How changes originate

See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.

Package metadata

Each package has a metadata file at metadata/<package>.json that tracks its relationship to upstream:

{
  "source": "https://src.fedoraproject.org/rpms/curl.git",
  "branch": "rawhide",
  "sha": "abc123...",
  "version": "8.12.1",
  "release": "1",
  "modification_status": "clean",
  "upstream_repo": "https://github.com/curl/curl"
}

Key fields:

See Package Metadata Fields for modification_status and release configuration, and Package Modification Tracking for managing modification status day to day.

Stage 2: Merge Request & Validation

Changes reach main via merge requests. Automated MRs are created using:

• ci/create_mr.sh - Single MR with configurable options
• ci/rebuild_multi_mr.sh - One MR per package with auto-merge enabled
• ci/dist_git_update_multi_mr.sh - Automated Fedora update MRs (scheduled CI job)
• ci/upstream_update_multi_mr.sh - Automated upstream version update MRs (scheduled CI job)

The scheduled upstream-version update job creates normal MRs for successful package updates. If a package update fails after a new version has been identified, it creates (or reuses) a draft diagnostic MR instead. The diagnostic MR includes the original error but no partial package changes, and must be resolved manually before a package update can be proposed.

Both scheduled update jobs send a Slack completion message when SLACK_WEBHOOK_URL is configured: success sends a plain success message, including when upstream warnings were logged, while failure includes actionable package failures and links to MRs needing attention. Routine upstream warnings remain in the job log and summary, not in Slack.

These completion messages also act as a heartbeat. The notification runs in the job’s after_script, so a job that never starts, or a runner failure before after_script can run, may send no message. A missing expected message is a signal to check the schedule and runner availability; it is not an automatic missed-run alert.

CI validation

GitLab CI runs validation jobs on every merge request:

• check - Linting, type checking, and spec validation
• check_nevr_conflicts - Predicts the NEVR(A) each changed package’s spec would build (using rpmspec with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale main, e.g. two rebuild MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch rpms/**/*, and within that, only checks packages whose rpms/<package>/ directory itself changed – matching Konflux’s own build trigger – not packages whose only change is under metadata/, since a metadata-only change (e.g. ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally with make check-nevr-conflicts ARGS='<package>', or against real built RPMs with make check-nevr-conflicts ARGS='--rpms-dir builds/<package>' after ci/build_rpms.sh. See ci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages routed to a private_product are noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode some OK results can be for a package name a real build never actually produces (e.g. a base package with no top-level %files section, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount a CONFLICT found elsewhere in the same run.
• tree_status - Repository consistency checks (metadata integrity, spec file presence)
• Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)

Auto-approval for chore MRs

Automated update MRs (chore/* branches) follow an accelerated path:

1. MR is created with auto-merge enabled
2. Konflux builds the package and posts commit statuses
3. After a delay, the chore_mr_approval CI job auto-approves the MR
4. GitLab merges the MR once the pipeline succeeds

Stage 3: Build

After merging to main, Konflux builds RPMs automatically.

Build triggers

Each package has a Tekton PipelineRun definition generated from its directory in rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered .tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers a build when changes to that package’s directory are pushed to main. When a single push touches multiple packages, one PipelineRun is triggered per affected package directory. Merge request pushes also trigger builds for validation.

The setup package acts as a canary: changes to ci/ or mock/ directories also trigger a setup build on pull requests, validating build infrastructure changes before they reach main.

Build process

The build pipeline (build-rpm-package) executes for each target architecture:

1. Checks out the repository at the merge commit
2. Calculates build dependencies
3. Runs mock in hermetic mode (network-isolated with pre-fetched dependencies)
4. Produces RPM artifacts and build logs
5. Generates an SBOM (Software Bill of Materials)
6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry

Build architectures

Packages are built for the architectures specified in their PipelineRun definition. Packages build for x86_64, aarch64, and s390x by default. Per-package overrides can select other architectures or build platforms.

Build output

Build artifacts are stored in the Konflux OCI registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main

Konflux resources

Build infrastructure is defined across two repositories:

• rpms repo (konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy
• infrastructure repo (kubernetes/rpms-main/) - Application, ReleasePlan, IntegrationTestScenario, ServiceAccount, and Secret resources

See Konflux Resource Deployment for details on how these resources are managed and deployed.

Testing

RPM packages are validated through integration tests that run on Testing Farm infrastructure.

Tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test triggering

Tests are only triggered on merge requests, not on main branch builds. For a package at rpms/<package>/, tests are triggered for all changes below that directory.

Integration Test Scenario

Tests are triggered via an IntegrationTestScenario resource defined in the infrastructure repository:

Configuration: infrastructure/kubernetes/rpms-main/10-integration-test-scenarios-testing-farm.yml.j2

The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:

Scenario parameters:

• COMPOSE: RHEL-9-Nightly - Test environment OS/compose
• PIPELINE_MODE: rpm - Configures RPM testing mode (vs. container)
• ARCH: x86_64,aarch64 - Architectures to test (creates separate test runs per arch)
• PASS_SNAPSHOT_TO_TF: false - Don’t pass full snapshot JSON (package info comes via IMAGE_NAME/IMAGE_URL instead)
• IMAGE_TAG: - Version of the tmt-via-testing-farm pipeline bundle

Key characteristics:

• Single scenario for all packages - Not per-package, runs for every package build
• Context-based triggering - Runs on pull_request snapshots only
• Package identification - Konflux automatically provides IMAGE_NAME (e.g., openssl-main) and IMAGE_URL (OCI image with built RPMs) as environment variables

FMF test plan

The root folder of the rpms repository is marked with a .fmf directory to identify it as an FMF metadata tree for tmt. The test plan in ci/run_tests_rpm.fmf sets up the test instance and runs default and package-specific tests via ci/run_tests_rpm.sh.

The plan configures Podman to use public DNS servers 1.1.1.1 and 8.8.8.8 inside test containers. This selects the public Pulp endpoint to match customer access. Containers use their original CA trust stores.

Test environment

These environment variables are available to the tmt test execution:

Stage 4: Signing

The Konflux Release Service signs built RPMs before publishing:

• Method: Kerberos-based signing via konflux-release-signing-prod@IPA.REDHAT.COM
• Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
• Signing image: quay.io/konflux-ci/signing:latest (verify pinned digest in ReleasePlanAdmission)

Stage 5: Publishing to Pulp

After signing, RPMs are uploaded to the Hummingbird Pulp repository.

Release pipeline

The push-rpms-to-pulp release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (releng/hummingbird-rpms-tech-preview-staging.yaml):

mapping:
  rpm-repositories:
    - name: x86_64
      repository_id: public-hummingbird-x86_64-rpms
    - name: aarch64
      repository_id: public-hummingbird-aarch64-rpms
    - name: src
      repository_id: public-hummingbird-source-rpms
    # See ReleasePlanAdmission for the full list of published architectures

Pulp repository

Published RPMs are available in the Pulp content index:

Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.

What Happens Next

Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.

Related Documentation

• Rebuilding Packages - No-change rebuilds, reverse dependency rebuilds, and patch backports
• Updating Dist-git Packages - Automated Fedora sync workflow
• Package Metadata Fields - modification_status and release configuration
• Package Modification Tracking - Managing modification status and version constraints
• Konflux Resource Deployment - How Konflux resources are defined and deployed
• Image Pipeline - Container image pipeline (continues from where this document ends)

1.2 - Konflux Resource Deployment

Konflux resources for RPM packages are split between two repositories:

• rpms: Per-package resources generated from package directories
• infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Legend:

• RPMs MR: Downstream pipeline triggered from rpms MR (deploys from MR commit)
• RPMs Push: Downstream pipeline triggered from rpms push to main
• Infra MR: Infrastructure MR pipeline (manual trigger)
• Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

RPMs Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

1. Ownership: Components are managed by the rpms repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle.
2. Dynamic generation: Generated from rpms/*/ directories via ci/generate_konflux_resources.sh, depending on package presence and configuration.
3. Timing: Must be deployed early during MR review so Konflux can build and test new packages.

ReleasePlanAdmission:

1. Static configuration: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern.
2. Main branch only: Deployed only on push to main to maintain deployment consistency.

EnterpriseContractPolicy:

1. Consistency: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern.

Infrastructure Repo

Application, ReleasePlan, and IntegrationTestScenario are defined in kubernetes/rpms-main/:

1. Independent iteration: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo.
2. Testable before merge: If defined in the rpms repo, these would only deploy after merging to main, making iteration difficult.
3. Static configuration: These resources don’t depend on per-package data.

ServiceAccount and Secret for releases are defined in kubernetes/setup-konflux/:

1. Security: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo.
2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo.

The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp.

1.3 - Source Pipeline Tool

Source Pipeline Tool

Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.

Problem Statement

RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:

1. Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.

2. Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.

3. Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.

4. Unverified patches. Patches (.patch files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.

Tool Overview

A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.

                          ┌──────────────────────────────┐
                          │     source-pipeline tool     │
                          │      (container image)       │
                          │                              │
  ┌───────────┐           │  ┌───────┐    ┌───────────┐  │    ┌───────────────┐
  │  package  │           │  │ fetch │───▸│ transform │  │    │   tarballs    │
  │   dir     │──────────▸│  └───────┘    └─────┬─────┘  │───▸│   sources     │
  │ (spec,    │           │                     │        │    │   report      │
  │  patches, │           │  ┌───────────┐      │        │    └───────┬───────┘
  │  config)  │           │  │  verify   │◂─────┘        │            │
  │           │           │  └─────┬─────┘               │            ▼
  └───────────┘           │        │                     │    ┌───────────────┐
                          │  ┌─────▼─────┐               │    │   lookaside   │
  ┌───────────┐           │  │  enforce  │               │    │    cache      │
  │ pipeline  │           │  │  policy   │               │    └───────────────┘
  │   yaml    │──────────▸│  └─────┬─────┘               │
  │           │           │        │                     │
  └───────────┘           │  ┌─────▼─────┐               │
                          │  │   emit    │               │
                          │  └───────────┘               │
                          └──────────────────────────────┘

Design principles

• Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
• Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
• Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
• Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.

Container interface

podman run --rm \
  -v ./<package-dir>:/package:ro \
  -v ./pipeline.yaml:/pipeline.yaml:ro \
  -v ./gpg-keys:/gpg-keys:ro \
  -v ./output:/output \
  source-pipeline:latest \
  --version <new-version> \
  [--old-version <old-version>] \
  [--verify-patches /patches-from-upstream] \
  [--dry-run]

Inputs (mounted read-only):

• /package — the package directory (spec file, patches, existing sources file)
• /pipeline.yaml — the declarative pipeline definition
• /gpg-keys — centralized GPG keyring directory

Outputs (written to /output):

• Source tarballs (ready for lookaside upload)
• sources — updated sources manifest with checksums
• report.json — verification and policy results (pass/fail per check, patch classifications)

Exit codes:

• 0 — success, all checks passed
• 1 — error (download failure, tool error)
• 2 — policy violation (verification or constraint failure)

--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to /output. Exit codes and report.json behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing.

On any non-zero exit, report.json is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no package update commit is created, the package stays at its current version, and the automation continues to the next package. In the scheduled upstream-version-update job, the wrapper also creates (or reuses) a draft diagnostic MR for a per-package update failure. That MR contains the full error and no partial package changes, so a maintainer can reproduce and resolve the problem safely. Transient failures (exit 1) may self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention. When a later scheduled run successfully processes the same package, it closes and removes the still-empty diagnostic MR branch. A diagnostic MR with maintainer changes is left open for manual handling.

Pipeline Stages

1. Fetch

Downloads source artifacts directly from upstream.

• Parses Source: URLs from the spec file (resolving RPM macros like %{version}, %{name}, %{url})
• Downloads each source from the upstream URL
• Supports fetching from git repositories at a tag, branch, or commit
• For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)

2. Transform

Applies per-package source modifications.

• Vendor archive generation — runs ecosystem-specific tooling (e.g., go_vendor_archive for Go, npm pack for Node.js, cargo vendor for Rust)
• Vendor dependency pinning — modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix.
• Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
• UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
• Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
• Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a toolchain: section. The container ships defaults; per-package overrides ensure that build-ui, vendor, and run: steps use the correct toolchain without requiring separate container images per package

Known sharp edge: patch-list duplication

Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill point back here instead of re-telling the story.

Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ... followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain their own copy of “which patches touch this source tree” rather than reading it from the spec’s PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see Rebuilding Packages) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform’s install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation step, and the mismatch didn’t surface until the next automated version bump re-ran the transform from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact.

go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up requiring different versions of the same package — go build -mod=vendor rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s %check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

3. Verify

Validates integrity and authenticity of fetched sources.

• GPG signature verification — downloads signature files (.asc, .sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation.
• Checksum verification — compares against published checksums where available
• Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
• Re-publication detection — if a previously committed checksum exists in the sources file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an accepted-checksums entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.

4. Enforce policy

Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect) and catches violations in packages that don’t use vendor-pin.

• Vendor dependency constraints — ensures vendored dependencies meet version requirements (e.g., sanitize-html >= 2.17.5 for a CVE fix)
• Ecosystem-specific checks — Go module verification, npm audit, cargo audit
• License compliance — flags vendored dependencies with incompatible licenses
• Fail-closed: any policy violation exits with code 2

5. Emit

Produces final artifacts.

• Writes tarballs to the output directory
• Generates sources file in dist-git format (SHA512 (filename) = hash)
• Writes report.json with verification results, policy check results, and patch classification

Pipeline Schema

Per-package pipeline definitions are YAML files that declare the full source generation workflow.

Packages without a pipeline definition use a built-in default that fetches sources from the spec’s Source: URLs — no transforms, no verification, no policy.

Schema definition

# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
  # Macro substitutions applied before Source URL resolution
  macros:
    - name: go_patch                     # %global go_patch <new-value>
      value: "${VERSION_PATCH}"          # extracted from VERSION (e.g., 1.25.3 → 3)

    - name: k8s_ver                      # %global k8s_ver <new-value>
      value: "${VERSION}"

  # Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
  reset-release: true

# Source fetching
fetch:
  sources:
    # Fetch Source0 from the URL declared in the spec
    - spec-source: 0

    # Fetch Source1 (e.g., a GPG signature file)
    - spec-source: 1

    # Or fetch from an explicit URL (for sources not in the spec)
    - url: "https://example.com/extra-source-${VERSION}.tar.gz"

    # Or fetch from a git repository at a tag/commit
    - git:
        repo: "https://github.com/example/project"
        ref: "v${VERSION}"              # tag, branch, or commit hash
        include-history: false           # include .git dir in tarball (default: false)

  # Vendor archive generation (optional)
  vendor:
    ecosystem: go                        # go | npm | cargo | composer
    config: go-vendor-tools.toml         # ecosystem-specific config file in the package dir
    source-dir: .                        # directory to vendor from (default: extracted source root)
    submodules:                          # (optional) vendor multiple Go submodules independently
      - server
      - etcdctl
      - etcdutl

# Source transforms (optional, ordered)
transform:
  # Built-in transform types
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
        - "deps/ngtcp2/ngtcp2/crypto/"
      output: "node-v${VERSION}-stripped.tar.gz"

  # Pin vendored dependency versions before vendor archive generation.
  # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
  # bump specific dependencies, then re-resolves the dependency graph.
  # Runs before the vendor stage so the pinned versions are included in
  # the vendor archive. Re-applied on every source generation, so upstream
  # updates cannot silently revert a security fix.
  #
  # Version constraints are always minimum versions — the version field
  # means "at least this version." The tool translates to ecosystem-native
  # operations:
  #   Go:    go get <package>@v<version> (minimum version selection)
  #   npm:   npm install <package>@">= <version>"
  #   Cargo: set dependency requirement to ">= <version>" in Cargo.toml
  - vendor-pin:
      - package: golang.org/x/crypto
        ecosystem: go
        version: "0.31.0"
        reason: "CVE-2024-45337"

      - package: sanitize-html
        ecosystem: npm
        version: "2.17.5"
        reason: "CVE-2024-XXXXX"

      - package: tokio
        ecosystem: cargo
        version: "1.38.1"
        reason: "CVE-2024-YYYYY"

  # Custom command (escape hatch)
  - run: "./packaging/make-tarball.sh ${VERSION}"
    outputs:
      - "package-${VERSION}-stripped.tar.gz"

  # UI asset build
  - build-ui:
      ecosystem: npm                     # npm | yarn
      source-dir: "web/ui"
      output: "${PACKAGE}-${VERSION}-ui.tar.gz"

# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
  node: "20"                             # Node.js version for build-ui, npm vendor, and run: steps
  go: "1.23"                             # Go version for go vendor and run: steps
  rust: "1.80"                           # Rust version for cargo vendor and run: steps
  python: "3.12"                         # Python version for run: steps

# Integrity verification (optional)
verify:
  gpg:
    signature-source: 1                  # spec Source index containing the .asc/.sig
    keyring: "curl.gpg"                  # key name in the GPG keys directory
  checksums:
    url: "https://example.com/SHA256SUMS"
    algorithm: sha256

  # Override for upstream re-publications. Required when upstream re-publishes
  # a release artifact with different content under the same version. The
  # pipeline refuses to accept changed content automatically — a human must
  # add the new checksum here after investigation.
  accepted-checksums:
    - file: "example-1.2.3.tar.gz"
      sha512: "abc123..."
      reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"

# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
  vendor-constraints:
    - package: sanitize-html
      ecosystem: npm
      version: "2.17.5"
      reason: "CVE-2024-XXXXX"

    - package: golang.org/x/crypto
      ecosystem: go
      version: "0.31.0"
      reason: "CVE-2024-45337"

# Patch verification (optional)
patches:
  verify: true                           # diff-detect new/changed patches
  classify: true                         # attempt to classify by origin
  upstream-repo: "https://github.com/curl/curl"
  fail-on-unverified: false              # warn-only by default

  # Patch lifecycle rules (optional)
  # Declares version-scoped applicability for patches. The pipeline tool reads
  # these rules from patch headers (preferred) or from this YAML, and enforces
  # them during updates: patches outside their valid range are flagged or dropped.
  lifecycle:
    - file: "fix-memory-leak.patch"
      applies-to: "< 1.5.0"             # drop this patch at version 1.5.0+
      reason: "Fixed upstream in 1.5.0 (commit abc123)"
      action: drop                       # drop | warn (default: warn)

    - file: "cve-2024-45337.patch"
      applies-to: "< 0.31.0"
      reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
      action: drop

    - file: "distro-branding.patch"
      applies-to: "*"                    # carry forward unconditionally
      reason: "Distribution-specific branding, always required"

# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
  # Extract bundled dependency versions and splice into spec between markers
  - bundled-provides:
      modules-txt: "vendor/modules.txt"  # Go modules.txt path inside extracted source
      start-marker: "# --- bundled-deps.sh ---"
      end-marker: "# --- end bundled-deps.sh ---"

  # Run a custom command (escape hatch for complex metadata extraction)
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"

Variable substitution

The following variables are available in all string values:

Version constraint semantics

All version: fields in vendor-pin and policy.vendor-constraints use minimum-version semantics: the value means “at least this version.” The tool translates this into ecosystem-native operations:

• Go: go get <package>@v<version> (minimum version selection via MVS)
• npm: npm install <package>@">= <version>"
• Cargo: sets the dependency requirement to >= <version> in Cargo.toml

Default behavior (no pipeline YAML)

When no pipeline YAML is provided, the tool applies a built-in default:

1. Parse all Source: URLs from the spec
2. Download each from the upstream URL
3. Generate sources file with SHA512 checksums
4. No transforms, no verification, no policy enforcement

This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.

Patch Verification

When packages are updated from an upstream distribution, patch files (.patch) may arrive without inspection. The pipeline tool can optionally verify and manage patches.

Verification approach

The pipeline tool can optionally verify patches when invoked with --verify-patches:

1. Diff detection. Compare the set of .patch files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches.

2. Header parsing. Extract metadata from patch headers:

   • From: — author identity
   • Subject: — description
   • Commit hash references (e.g., From <hash>, cherry picked from commit <hash>)
   • Bug: / CVE: references

3. Classification. Attempt to classify each patch:

   • Upstream backport — references a commit hash that exists in the upstream repo
   • CVE fix — references a CVE identifier
   • Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
   • Distribution-specific — modifies paths, branding, or distribution-specific integration
   • Unclassified — cannot be automatically categorized

4. Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via git ls-remote or the upstream API).

5. Reporting. Output classification and verification results in report.json. Optionally fail on unverified patches (controlled by patches.fail-on-unverified in the pipeline YAML).

Patch lifecycle enforcement

Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.

The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML’s patches.lifecycle section.

Header-based declaration (preferred). Patch authors add structured keywords to the patch header:

From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---

YAML-based declaration (fallback). For patches from upstream distributions that cannot have headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch.

Enforcement behavior:

1. During an update to version ${VERSION}, the tool evaluates each patch’s applies-to range against the new version.
2. If a patch is outside its valid range:
   • action: drop — the patch file is deleted from the package directory and the corresponding Patch: declaration and %patch / %autopatch application directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in report.json.
   • action: warn (default) — the patch is flagged in report.json but not removed. The update proceeds.
3. If a patch has applies-to: *, it is always carried forward.
4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to applies-to: *, action: warn).

Carry-forward enforcement. The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked applies-to: * is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error.

Example report output

{
  "patches": {
    "new": [
      {
        "file": "fix-memory-leak.patch",
        "classification": "upstream-backport",
        "upstream_commit": "abc123def456",
        "verified": true
      }
    ],
    "removed": ["old-workaround.patch"],
    "unchanged": ["distro-paths.patch"],
    "lifecycle": [
      {
        "file": "cve-2024-45337.patch",
        "applies_to": "< 0.31.0",
        "current_version": "0.31.0",
        "action": "drop",
        "result": "removed — version 0.31.0 is outside applies-to range"
      },
      {
        "file": "cve-2024-99999.patch",
        "applies_to": "< 2.0.0",
        "current_version": "2.0.0",
        "action": "drop",
        "result": "flagged — patch is applied inside a conditional %if block; manual removal required"
      },
      {
        "file": "distro-branding.patch",
        "applies_to": "*",
        "action": "carry-forward",
        "result": "present"
      }
    ]
  }
}

Examples

Trivial package (curl)

No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you want GPG verification:

fetch:
  sources:
    - spec-source: 0
    - spec-source: 1

verify:
  gpg:
    signature-source: 1
    keyring: "curl.gpg"

Transformed package (Node.js)

Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:

spec-update:
  macros:
    - name: nodejs_define_version node
      value: "${VERSION}"
  reset-release: true

toolchain:
  node: "22"

fetch:
  sources:
    - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"

transform:
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
      output: "node-v${VERSION}-stripped.tar.gz"

verify:
  checksums:
    url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
    algorithm: sha256

post:
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"

Simple Go vendor (caddy)

A pattern shared by nats-server, oauth2-proxy, and similar Go projects:

fetch:
  sources:
    - git:
        repo: "https://github.com/caddyserver/caddy"
        ref: "v${VERSION}"

  vendor:
    ecosystem: go

Multi-submodule Go vendor (etcd)

fetch:
  sources:
    - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    submodules:
      - server
      - etcdctl
      - etcdutl

Code generation + Go vendor (opentelemetry-collector-contrib)

Built-in primitives handle fetch and vendor; the code generation step requires run::

fetch:
  sources:
    - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    source-dir: _build

transform:
  - run: |
      curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
      curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
      sha256sum -c --ignore-missing ocb_checksums.txt
      chmod +x ocb
      tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
      cd "opentelemetry-collector-releases-${VERSION}"
      if [ -f scripts/prepare-obi.sh ]; then
        bash scripts/prepare-obi.sh otelcol-contrib
      fi
      ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
    outputs:
      - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"

Git snapshot (libXtst)

A pattern shared by libX11, libXext, libXi, libXrender:

fetch:
  sources:
    - git:
        repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
        ref: "libXtst-${VERSION}"
        include-history: true

Built-in Primitives Coverage

An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:

Fully declarative (no run: needed) — ~30 script files (15 unique patterns):

Partially declarative (generic primitives + 1–2 run: steps):

Genuinely custom (run: required) — 7 packages:

The run: escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract.

Open Questions

1. Implementation approach. The pipeline YAML schema (ordered transform stages, run: escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:
   • Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
   • Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
   • Shared shell function library — the majority of scripts decompose into 3–4 operations (vendor_go, strip_tarball, fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands.
   • Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no run: blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.

Hummingbird Integration

This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.

Current Architecture

Two update systems

dist_git.py update flow:

1. Clones Fedora dist-git for the package
2. Checks version, Koji build status, pre-release filtering
3. Copies entire Fedora dist-git checkout into rpms/<package>/ via shutil.copytree — this includes the spec, patches, .gitignore, and sources file
4. For modified packages, performs a 3-way git merge to preserve local changes
5. Commits everything (spec, patches, sources file, metadata)

The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash). Actual tarballs are excluded by .gitignore and stored in a lookaside cache.

check_upstream_versions.py flow:

1. Queries release-monitoring.org for new upstream versions
2. Runs three hook phases per package:
   • update_spec — updates the spec’s Version/Release (default: specfile.update_version())
   • download_sources — downloads source tarballs (default: fetches from spec Source URLs)
   • post_update — additional steps (default: no-op)
3. Uploads downloaded tarballs to Hummingbird’s lookaside cache
4. Updates the sources file with new checksums
5. Commits

Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.

Three lookaside cache backends

Configured in mock/dist-git-client.ini:

The forked_from field in ci/package-overrides.yaml determines which cache a package uses at build time. Packages without forked_from default to Fedora’s cache.

How Konflux builds consume sources

Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):

• monorepo-subdir: rpms/<package> — locates the spec, patches, and sources file
• package-name: <name> — upstream name for lookaside URL construction
• forked-from: <url> — (optional) selects the lookaside backend
• dist-git-client-configdir: mock/ — (optional) points to the custom dist-git-client.ini

The pipeline uses dist-git-client to read the sources file and download tarballs from the appropriate cache, then runs mock to build RPMs.

Existing source generation patterns

Some independent packages already have tarball generation scripts:

• rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh — downloads upstream release, runs OCB to generate source code, vendors Go dependencies
• rpms/caddy/create-vendor-tarball.sh — downloads upstream tarball, generates Go vendor archive
• rpms/nodejs25/packaging/make-nodejs-tarball.sh — downloads upstream, strips bundled content

These scripts are wired into the update path via download_sources hooks in metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache.

Integration with dist_git.py update

The current flow copies everything from Fedora dist-git (including the sources file) via shutil.copytree. The pipeline tool inserts after this step:

Current:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Commit

Proposed:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Run source-pipeline tool             ← fetches from upstream, replaces sources file
  4. Upload tarballs to Hummingbird lookaside
  5. Commit                               ← sources file now points to Hummingbird lookaside

Implementation:

• After the shutil.copytree (or merge for modified packages), check if metadata/<package>.source-pipeline.yaml exists
• If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the sources file
• If it does not exist: run the built-in default (fetch from spec URLs, upload, replace sources)
• A --skip-pipeline flag allows falling back to the current behavior during migration
• Update ci/package-overrides.yaml to set forked_from to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)

Integration with check_upstream_versions.py

The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating per-package update behavior into a single *.source-pipeline.yaml file:

Current:
  1. update_spec hook/default             ← *.update-hooks.yaml
  2. download_sources hook/default        ← *.update-hooks.yaml
  3. post_update hook/default             ← *.update-hooks.yaml

Proposed:
  1. spec-update (pipeline YAML)          ← replaces update_spec hooks
  2. fetch + transform (pipeline YAML)    ← replaces download_sources hooks
  3. post (pipeline YAML)                 ← replaces post_update hooks

This reduces per-package metadata from three files (metadata/<package>.json, *.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for identity/tracking, *.source-pipeline.yaml for all update behavior).

Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed.

Integration with Konflux

Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via dist-git-client as it does today.

Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.

Hummingbird-specific configuration

Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.

GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.

The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.

Local development

Developers can run the tool directly:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  -v /tmp/output:/output \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0

# Outputs in /tmp/output/:
#   curl-8.21.0.tar.xz
#   curl-8.21.0.tar.xz.asc
#   sources
#   report.json

To preview without producing artifacts:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0 \
  --dry-run

# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.

Migration Path

Phase 1: Independent Go packages

Migrate the 22 independent packages that already fetch from upstream. Convert their existing create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions. Validates the tool against known-good packages.

Scope: 7 packages with existing scripts + 15 without.

Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code generation, but their vendor step is generic.

Phase 2: Trivial clean packages

Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.

Prerequisites:

• Audit to classify packages as trivial vs. transformed (HUM-4620)
• dist_git.py update integration complete (HUM-4621)

Per-package steps:

1. Run the pipeline tool, compare output against existing Fedora-sourced tarball
2. If identical: add forked_from: hummingbird to package-overrides.yaml
3. If different: flag for Phase 3

Phase 3: Transformed packages

Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:

• Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): strip-tarball
• Git snapshots (libX11 family×5, oniguruma, php-jsqueeze): fetch: {git: ...}
• Composer vendor (composer): vendor: {ecosystem: composer}

Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require run: blocks for genuinely custom logic.

Scope: determined by Phase 2 audit.

Phase 4: Policy enforcement

Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates.

Phase 5: Patch verification

Enable patches.verify and patches.classify for packages. Start with warn-only (fail-on-unverified: false), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages.

Operational Health

Escape hatch ratio

The primary health metric for the source pipeline is the escape hatch ratio: the percentage of pipeline YAML definitions that contain run: blocks (in transform or post sections) relative to the total number of pipeline YAML definitions.

A run: block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the same way: a package that cannot be expressed declaratively gets a run: block. This makes the ratio a single number that aggregates all pressures on the declarative model.

Interpretation:

When the ratio climbs, examine the run: blocks for repeated patterns. If three or more packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad – it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set.

CI enforcement

The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files, calculates the escape hatch ratio, and reports the result. It runs as part of make check.

• During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need run: blocks before new primitives are added.
• After Phase 3: enable --fail mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a run: block is added.

2 - Agent-Friendly Documentation

The documentation site provides several features that make it easier for AI agents and LLMs to discover, index, and consume its content.

Discovery and access methods

llms.txt

The site publishes an llms.txt file at the root, following the llmstxt.org specification. This file is a Markdown-formatted index of all documentation pages, grouped by section, with links to each page’s markdown URL. It serves as the primary entry point for agents discovering the site.

llms-full.txt

The site also publishes llms-full.txt – a full content dump of all documentation pages in a single file. This is intended for LLMs with large context windows that want to ingest the entire site in a single request without crawling individual pages.

Pages whose rendered content exceeds a size threshold (currently 50,000 characters) are replaced with a placeholder that includes the page title, description, and a link to the individual page’s markdown URL for on-demand fetching. This keeps the file at a manageable size while ensuring no content is completely inaccessible.

Per-page markdown output

Every page on the site is available in clean markdown format by appending /index.md to its URL. For example:

• HTML: https://hummingbird-project.io/docs/contributing/quickstart/
• Markdown: https://hummingbird-project.io/docs/contributing/quickstart/index.md

The markdown output resolves all shortcodes (such as file includes from other repositories) while preserving the content as markdown rather than converting to HTML. Each markdown page includes links to both llms.txt and llms-full.txt so agents can discover the site-wide indexes from any page.

Content negotiation

A CloudFront distribution sits between DNS and the site’s page host. A viewer-request CloudFront Function inspects the Accept header on every request: when the header includes text/markdown, the function rewrites the URI to serve the pre-built .md equivalent. Only HTML-like patterns (trailing slash, .html, bare paths) are rewritten; all other URIs pass through unchanged.

This means agents can use standard HTTP content negotiation on any page URL without needing to know the /index.md convention:

GET /docs/contributing/quickstart/ HTTP/1.1
Accept: text/markdown

Stable link aliases (/l/)

Pages referenced from external sources have stable short URLs under the /l/ prefix (e.g., /l/adding-images). These return HTTP 301 to the canonical page URL, so pages can move without breaking incoming links. The Writing Documentation guide describes how to add aliases.

The CloudFront viewer-request function recognizes /l/ paths and normalizes them before they reach the origin. Content negotiation (Accept header) is skipped for /l/ paths — instead, the URL form determines the response: /l/foo and /l/foo/ redirect to the HTML page, while /l/foo.md and /l/foo/index.md redirect to the markdown file. If no matching alias exists, the origin returns a 404.

Specifications and tools

The site aims to follow the emerging standards for agent-friendly documentation:

• llmstxt.org – the llms.txt specification
• Agent-Friendly Documentation Spec – broader specification for documentation sites that serve AI agents
• AFDocs – scoring tool that evaluates a site’s compliance with the Agent-Friendly Documentation Spec

Planned features

HTML llms-directive (HUM-6571)

A Docsy upgrade to v0.17.0 will embed a pointer to llms.txt inside the HTML <body> of every page, so agents landing on any HTML page can discover the LLMS index without prior knowledge of the site structure.

Documentation quality evaluation (HUM-6575)

A promptfoo-based evaluation harness will measure how accurately LLMs can answer questions about Hummingbird using the published documentation. This provides a feedback loop for identifying documentation pages that need improvement for agent consumption.

3 - Containers repository

Background documentation sourced from the containers repository.

3.1 - Image Pipeline

Explanation of the complete container image pipeline, from RPM dependency updates through source templates, generation, build, testing, release, and advisory publication.

Overview

The container image pipeline consists of six main stages:

1. Source Templates - Jinja2 templates in images/*/ define container images
2. Generation - Templates are rendered into Containerfiles and Konflux resources
3. Build - Konflux builds multi-architecture container images
4. Testing - Testing Farm runs integration tests via Konflux
5. Enterprise Contract Validation - Conforma validates policy compliance before release
6. Release - Images are published to registries and advisories are created

RPM packages are built and published separately via the RPM Pipeline. This document covers what happens after RPMs are available in the Hummingbird package repository.

RPM Dependency Updates

Container images pin exact RPM versions via lockfiles. When new RPMs are published to the Hummingbird Pulp repository, lockfiles must be regenerated for images to pick up the updates.

Lockfile structure

Each image variant has two RPM-related files:

The lockfile contains every RPM (direct and transitive dependencies) with its exact version, architecture, URL pointing to the Pulp repository, and checksum. During build, the locked RPMs are downloaded in hermetic mode — no network access to external repositories.

Automatic lockfile updates

MintMaker / Renovate — A custom Renovate instance (forked rpm-lockfile manager from redhat-exd-rebuilds/renovate) runs as a Kubernetes CronJob. It scans each rpms.in.yaml file individually, resolves dependencies against Pulp repositories, and opens auto-merge MRs — one per image group and distro (e.g., “Refresh RPM lockfiles for go-1-25/hummingbird”).

The underlying tool is rpm-lockfile-prototype, which wraps the DNF dependency solver. For a given rpms.in.yaml, it resolves all direct and transitive dependencies against the yum repositories defined in yum-repos/*.repo and outputs a complete rpms.lock.yaml with pinned versions, download URLs, and checksums. Locally, generate_rpms_lock.py adds hash-based optimization: it hashes the input file and skips regeneration when the hash matches the existing lockfile, avoiding expensive solver runs when nothing has changed.

Manual lockfile updates

To refresh the lockfile for a single image variant:

make generated/images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml FORCE_REFRESH=true

For example:

make generated/images/caddy/hummingbird/default/rpms/rpms.lock.yaml FORCE_REFRESH=true

To regenerate all lockfiles and open MRs for the changes:

# Regenerate all rpms.lock.yaml files
make all-host FORCE_REFRESH=true
# Open MRs for any changed lockfiles
ci/create_lockfile_update_mrs.sh

From lockfile MR to container build

When a lockfile MR merges to main, PipelinesAsCode triggers a container build (Stage 3). Each image variant has CEL expressions in its PipelineRun definition that match on file paths, so only images whose lockfiles changed get rebuilt.

Stage 1: Source Templates

Each container image is defined by templates in images/<image-name>/:

• properties.yml - Image configuration (packages, variants, tags, etc.)
• Containerfile.j2 - Jinja2 template for the container build
• README.md.j2 - Documentation template
• tests-container.yml - Integration test definitions

Templates use reusable macros from macros/*.yml.j2:

• setup_newroot() - Configures DNF and filesystem
• install_newroot() - Installs packages
• cleanup_newroot() - Cleans up files
• final_stage() - Creates scratch-based final image

Shared configuration is defined in images/variables.yml. See Global Variables Reference for details.

Stage 2: Generation

Templates are rendered into concrete artifacts that drive the pipeline:

• Containerfiles - Build instructions for each image variant
• README documentation - Image documentation for Quay.io
• Konflux resources - CI/CD pipeline definitions

Containerfile Generation

Containerfiles are generated from templates for each image variant using Make’s incremental build system:

make

This combines:

• Reusable macros from macros/
• Service-specific templates from images/*/Containerfile.j2
• Configuration from properties.yml (see Image Configuration Reference)
• Variables from images/variables.yml (see Global Variables Reference)
• RPM versions from rpms.lock.yaml files
• Git submodule information from .gitmodules

Output: generated/images/<image-name>/<distro>/<variant>/Containerfile, along with VERSION and TAGS files

The build system uses timestamp-based dependency tracking, so only changed files are regenerated.

README Generation

After Containerfiles are generated, README documentation is generated from README.md.j2 templates using macros from macros/readme.yml.j2. Each image template produces one README per target defined in images/variables.yml (Hummingbird, Rawhide, Red Hat).

Output: generated/images/<image-name>/hummingbird/README.md, generated/images/<image-name>/rawhide/README.md, generated/images/<image-name>/README.md

Konflux Resource Generation

Konflux CI/CD resources are generated from templates in konflux-templates/:

make

This generates:

• Components - Define what to build (one per image variant)
• ImageRepositories - Define where to push images
• ReleasePlanAdmissions - Define how to release images

Output: konflux-templates/rendered.yml

These resources must be deployed to Konflux before builds can run. See the Konflux Resource Deployment guide for how and when resources are deployed.

Stage 3: Build

Konflux builds container images automatically when changes are pushed to GitLab.

Build Triggers

• Merge Requests: Builds all changed images and triggers tests
• Main Branch: Builds all changed images (tests do not run on main)

Build Process

For each image variant:

1. Konflux Component watches the GitLab repository
2. On changes, Konflux triggers a build PipelineRun
3. The build uses the generated Containerfile from generated/images/<name>/<distro>/<variant>/Containerfile
4. Images are built for multiple architectures (x86_64 and aarch64)
5. Built images are pushed to the development registry

Build Output

Development images are pushed to the Red Hat User Workloads registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main

Merge request builds are tagged with:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main:on-mr-<MR_ID>-<COMMIT_SHA>

Examples:

• quay.io/redhat-user-workloads/hummingbird-tenant/curl--default--main
• quay.io/redhat-user-workloads/hummingbird-tenant/nginx--builder--main:on-mr-123-abc1234

SBOM Generation

Each per-architecture build also produces an SPDX 2.3 Software Bill of Materials (SBOM), attached as an OCI artifact to the image. The SBOM is assembled from two independent scans, merged into a single document:

flowchart LR
    syft["Syft\n(buildah-remote-oci-ta)"] --> mobster["Mobster\n(buildah-remote-oci-ta)"]
    hermeto["Hermeto\n(prefetch-dependencies-oci-ta)"] --> mobster
    mobster --> sbom["Per-arch SPDX SBOM"]
    sbom --> index["Mobster\n(build-image-index)"]
    index --> oci["Index SBOM\n(.sbom OCI artifact)"]

• Syft runs as the sbom-syft-generate step inside the buildah-remote-oci-ta Tekton task. It scans the RPM database of the built per-arch image and finds installed binary RPMs plus non-RPM packages (Go modules, pip packages, etc.). One SBOM is produced per architecture.
• Hermeto runs inside the prefetch-dependencies-oci-ta Tekton task. It records all build-time dependencies from lockfiles. A single Hermeto SBOM covers all architectures and source RPMs.
• Mobster runs as the prepare-sboms step inside buildah-remote-oci-ta, merging the Syft and Hermeto SBOMs into one SPDX document per architecture. A second Mobster invocation in the build-image-index task generates an index-level SBOM and attaches it as an OCI artifact.

See Security Labels and Metadata for the SBOM entry structure and how to access SBOMs from the registry.

Stage 4: Testing

Images are validated through two types of integration tests:

• Container tests (tests-container.yml) - Run with Podman and Docker via Testing Farm on RHEL-9-Nightly systems
• K8s tests (tests-k8s.yml) - Run in Konflux ephemeral Kubernetes namespaces

Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test Triggering

Tests are only triggered on merge requests, not on main branch builds.

For an image group at images/<group-name>/, tests are triggered for all changes below that directory, excluding documentation-only changes.

Any changes below .tekton/ or ci/ will trigger tests for the caddy image to ensure infrastructure changes work before merging.

Test Execution

For each image group, pipelines are created per variant. If there is more than one variant in an image group, additional group pipelines are created. See Konflux group snapshot documentation for details.

Tests are selected based on the variants field in test files. Additionally, global tests from ci/{variant}-tests/ are included. Group pipelines run tests that specify variants: [group], allowing validation across multiple variants.

Reverse Dependency Testing

When a base image like core-runtime changes, dependent images (rust, xcaddy, etc.) need to be tested to ensure compatibility. Reverse dependency testing is enabled by default. Images can opt-out by setting reverse_dependency_tests: false in their properties.yml. This is recommended for images like “curl” which are widely used and only have a small API which their own tests cover well enough.

For container tests, dependent images are rebuilt locally in the testing environment to prevent version skew and ensure dependent images are in sync with the repository status in the merge request under test.

Integration Test Scenarios

Integration tests are triggered via IntegrationTestScenario resources defined in the infrastructure repository:

Container Testing

Container tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Container Testing Flow

1. Developer opens MR modifying images/nginx/
2. GitLab CI pipeline and Konflux pipelines start in parallel
3. Konflux builds nginx image variants
4. IntegrationTestScenario triggers Testing Farm job
5. tmt discovers enabled child plans in ci/run_tests_container.fmf
6. Testing Farm provisions one RHEL-9-Nightly machine per enabled plan (per architecture)
7. tmt sets up the testing environment on each machine
8. tmt runs tests via ci/run_tests_container.sh

Container Test ITS Configuration

The container tests use the upstream Testing Farm pipeline for Konflux CI:

kind: IntegrationTestScenario
spec:
  contexts:
    - {name: pull_request}
  params:
    - {name: COMPOSE, value: RHEL-9-Nightly}
    - {name: ARCH, value: x86_64|aarch64}  # one for each
    - {name: CONTEXT, value: fips=enabled}  # containers-hummingbird only
    - {name: IMAGE_TAG, value: v3.2}
  resolverRef:
    resolver: bundles
    params:
      - {name: bundle, value: quay.io/testing-farm/tmt-via-testing-farm:$(params.IMAGE_TAG)}
      - {name: name, value: tmt-via-testing-farm}
      - {name: kind, value: pipeline}

FMF Test Plan

The root folder of the containers repository is marked with a .fmf directory to enable Testing Farm support. The test plan in ci/run_tests_container.fmf uses FMF virtual child plans to define two execution variants:

• /standard — inherits all parent configuration, always enabled
• /fips — inherits all parent configuration plus a FIPS prepare step; disabled by default, enabled via adjust when the fips=enabled context is set

The parent node defines shared configuration (prepare, discover, execute) inherited by both children. Because it has children, FMF treats it as a branch node and does not select it as a runnable plan. The containers-hummingbird ITS passes CONTEXT=fips=enabled, so both /standard and /fips run in parallel. Other applications (rawhide, ci-hummingbird, community-hummingbird) pass no CONTEXT, so only /standard runs.

Both child plans run the following steps:

1. Install podman for container testing
2. Install Docker and start the Docker daemon on the host for Docker integration tests
3. Fix git submodules until TFT-3991 is resolved
4. For regular pipelines:
   • Verify the image built by Konflux is reproducible using ci/test_rebuild.sh
   • If needed, build reverse dependency images locally via Buildah
   • Run Podman and Docker tests via ci/run_tests_container.sh --component-name
5. For group pipelines:
   • Run Podman and Docker group tests via ci/run_tests_container.sh --group-component-name

The /fips child additionally enables FIPS mode on the provisioned RHEL machine before all other prepare steps (via order: 0).

Both plans configure Podman, Buildah, and Docker to use public DNS servers 1.1.1.1 and 8.8.8.8 inside test containers. This selects the public Pulp endpoint to match customer access. Containers use their original CA trust stores. Podman and Buildah read /etc/containers/containers.conf.d/90-hummingbird-dns.conf. Docker uses the dns setting in /etc/docker/daemon.json. The host resolver and image CA bundles remain unchanged.

Container Test Environment

Testing Farm provides these environment variables to the test plan:

K8s Testing

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

K8s Testing Flow

1. Developer opens MR modifying an image in the containers repository (e.g., nginx)
2. GitLab CI pipeline and Konflux pipelines start in parallel
3. Konflux builds image variants
4. IntegrationTestScenario triggers K8s test pipeline
5. Pipeline checks for tests-k8s.yml; skips with SUCCESS if not found
6. Pipeline provisions ephemeral namespace via Konflux EaaS (tied to PipelineRun lifecycle)
7. Pipeline fetches source via Trusted Artifacts
8. Tests run via ci/run_tests_k8s.sh with kubeconfig for ephemeral namespace
9. Pipeline fails if any test reports non-SUCCESS, ensuring GitLab sees correct status

K8s Test ITS Configuration

The K8s tests use the k8s-test-pipeline:

kind: IntegrationTestScenario
spec:
  contexts:
    - {name: pull_request}
  resolverRef:
    resolver: bundles
    params:
      - {name: bundle, value: quay.io/hummingbird-ci/k8s-test-pipeline:latest}
      - {name: name, value: k8s-test}
      - {name: kind, value: pipeline}

Stage 5: Enterprise Contract Validation

Before images can be released, they must pass Enterprise Contract (also known as Conforma) policy validation. This ensures images meet security, compliance, and build quality standards. These checks can also be run locally against Konflux-built images.

Policy Validation

Enterprise Contract validates that:

• Images are built using trusted, verified Tekton tasks
• Builds are hermetic (network-isolated with pre-fetched dependencies)
• Required security tests have passed
• Images have proper metadata and labels
• Build artifacts meet supply chain security requirements

Policy Configuration

Policies are defined as EnterpriseContractPolicy resources in konflux-templates/macros/policy.yml.j2:

• containers-hummingbird / containers-rawhide - Strict policies for production images
• containers-community-hummingbird - Policy for community-supported images in the containers repository (same base exclusions as production policies)

These policies use the @redhat rule collection from the ec-release-policy.

Policy Exclusions

The following checks are excluded from the default @redhat policy set. When modifying exclusions, update the policy macro and this documentation.

Test Package

The test package verifies that each build was subjected to a set of tests and that those tests all passed.

Snyk SAST checks (test.required_tests_passed:sast-snyk-check, test.no_skipped_tests:sast-snyk-check, test.required_tests_passed:sast-snyk-check-oci-ta, test.no_skipped_tests:sast-snyk-check-oci-ta) are excluded because Hummingbird images are currently not supported by Snyk.

Red Hat certification preflight checks (test.no_failed_tests:ecosystem-cert-preflight-checks, test.no_erred_tests:ecosystem-cert-preflight-checks) are excluded because Hummingbird images are not yet published to the Red Hat certified container registry.

Informative test failures (test.no_failed_informative_tests) are excluded because these produce warnings for advisory purposes only and are explicitly non-blocking.

Deprecated image warnings (test.no_test_warnings:deprecated-image-check) are excluded because final images are built FROM scratch, and the builder image is updated via Renovate like all other images.

Trusted Task Package

The trusted_task package verifies that all Tekton Tasks involved in building the image are trusted by comparing Task references with a pre-defined list of trusted Tasks.

The trusted_task.current check warns when newer versions of tasks are available. This is excluded because we use stable pinned task versions and control upgrade timing via Renovate rather than requiring the latest version at all times.

RPM Repos Package

The rpm_repos package confirms that all RPM packages listed in SBOMs specify a known and permitted repository ID.

The rpm_repos.ids_known check is excluded because images use the internal hummingbird repository and Fedora repositories, which are not in the upstream known_rpm_repositories.yml list (that file only contains Red Hat official repositories).

RPM Signature Package

The rpm_signature package verifies that all RPMs in the image are signed by an allowed signing key.

The upstream default only allows the Red Hat release key 2 (199e2f91fd431d51). Rawhide, community, and experimental images contain Fedora-signed RPMs, so the containers-rawhide, containers-community-hummingbird, and containers-ci-hummingbird policies add Fedora signing keys via the additional_rpm_signature_keys macro parameter. The containers-hummingbird policy uses the default (Red Hat key only) because Hummingbird images install RPMs exclusively from the Hummingbird repository.

The Fedora key list in konflux-resources.yml.j2 should be kept in sync with the keys published at https://fedoraproject.org/security.

Labels Package

The labels package checks if the image has the expected labels set, including required and optional labels for Red Hat container certification.

Both labels.required_labels and labels.optional_labels are excluded because images currently only include basic labels (maintainer, license_terms, name, cpe) and version labels, not the full set of Red Hat certification labels (vendor, version, release, summary, description, url, etc.) required for Red Hat Ecosystem Catalog publishing.

Buildah Build Task Package

The buildah_build_task package verifies buildah build task parameters.

The buildah_build_task.privileged_nested_param check verifies that PRIVILEGED_NESTED is not set to true. This is excluded because images use the dnf-installroot helper from the builder image to build all containers, including the builder image itself. This script requires privileged operations (unshare, mount -t tmpfs, mount --bind for /proc and /dev/*) to set up the install root environment (see commit 3668af16).

Schedule Package

The schedule package verifies that releases conform to a given schedule, including weekday restrictions.

The schedule.weekday_restriction check is excluded to allow releases any day including weekends. The @redhat policy restricts weekend releases, but this project needs the ability to ship urgent CVE fixes immediately regardless of the day of week.

CVE Package

The cve package checks for blocking and non-blocking CVEs in container images.

The cve.cve_blockers check is excluded because blocking on known CVEs would prevent releasing images that fix other CVEs. A VEX feed could suppress false positives for RPM-level CVEs, but CVEs can also originate from other artifact types where VEX does not apply (see MR !2227).

Hermetic Task Package (CI-Only)

The hermetic_task package verifies that tasks were invoked with the proper parameters to perform a hermetic (network-isolated) execution.

All Konflux applications in the containers repository enforce the hermetic task check. All applications allow quay.io/hummingbird-ci/ as a base image source because images build FROM quay.io/hummingbird-ci/hummingbird-builder via setup_newroot.yml.j2.

Stage 6: Release

After images pass testing in merge requests and are merged to main, they are released to public registries.

Registry Organization

Images are published to different registries based on distro and purpose:

Production Registry: Red Hat supported Hummingbird images are published to registry.access.redhat.com/hi/ via the rh-advisories release pipeline and mirrored to quay.io/hummingbird/.

Community Registry (hummingbird-community): Publishes community-supported images (those with support_level: community in properties.yml). Examples include minio, minio-client, and os-minimal.

CI/tools Registry (hummingbird-ci): Publishes tooling and infrastructure images. This includes tools-repo images (built outside the containers repository) and experimental images from the containers repository (support_level: experimental), such as hummingbird-builder.

Release Process

1. Merge request passes all tests
2. Merge request is merged to main branch
3. Konflux builds images from main
4. ReleasePlanAdmission resources trigger the release pipeline
5. Images are signed via Cosign for supply chain attestation
6. Images are copied from the Konflux registry to target registries
7. Tags are applied based on properties.yml configuration
8. For production releases: images are registered in the Red Hat container catalog via Pyxis

Release Mechanism

Releases are configured via ReleasePlanAdmission (RPA) resources in the containers repo and ReleasePlan resources in the infrastructure repo:

• ReleasePlanAdmission - Defines per-registry release configuration (target registry, tags, visibility settings)
• ReleasePlan - Triggers the release pipeline for a specific application and registry

Each distro/registry combination has its own RPA.

Production release pipeline

The production release uses the rh-advisories pipeline from the release-service-catalog. The push-snapshot task pushes images from the development registry to:

• Production: registry.access.redhat.com/hi/<image>:<tags>

Configuration is defined in releng/hummingbird-containers-prod.yaml.

Image signing

Images are signed using Cosign as part of the release pipeline. Signing provides supply chain attestation, allowing consumers to verify image provenance.

Pyxis catalog registration

The release pipeline registers images in the Red Hat container catalog via the Pyxis API:

• Product: Red Hat Hardened Images (Product ID 1071)
• Metadata: version, categories, layer information, SBOM references
• Configuration: releng/pyxis-hummingbird.yaml

Pyxis registration makes images discoverable in the Red Hat Ecosystem Catalog and links them to security advisories.

Release Output

Released images are published to registries based on distro and support level:

registry.access.redhat.com/hi/<image-repository>:<image-tag>
quay.io/hummingbird/<image-repository>:<image-tag>
quay.io/hummingbird-community/<image-repository>:<image-tag>
quay.io/hummingbird-rawhide/<image-repository>:<image-tag>
quay.io/hummingbird-ci/<image-repository>:<image-tag>

Examples:

• registry.access.redhat.com/hi/curl:8 - Production Red Hat supported image
• quay.io/hummingbird/nodejs:20 - Red Hat supported Hummingbird image (mirror)
• quay.io/hummingbird-community/minio:latest - Community-supported image
• quay.io/hummingbird-ci/hummingbird-builder:latest - Hummingbird Builder image
• quay.io/hummingbird-rawhide/curl:latest - Rawhide image

Release Tags

Tags are extracted from Containerfile labels as defined in properties.yml:

• latest - Latest version of the image
• <major> - Major version (e.g., 20 for Node.js 20.x)
• <major>.<minor> - Major.minor version (e.g., 20.11)
• <full-version> - Complete version with release (e.g., 20.11.1-1.fc42)
• <YYYYMMDD> - UTC date when RPM lockfile content last changed, available via rpm_lockfile.content_date for any image that includes it in its tags. Only advances when the resolved RPM set changes.
• <timestamp> - Build timestamp (production releases only)

Non-default variants receive a -<variant> suffix (e.g., latest-builder).

Advisory Creation

When the production release pipeline runs, advisories are created:

1. The Release Service generates an Advisory YAML from ReleasePlan and ReleasePlanAdmission metadata
2. The advisory is pushed to the advisories repo on CEE GitLab
3. GitLab CI validates the advisory against the schema and enforces field-level permissions
4. On merge to main, the advisory is published via UMB and Kafka

Advisory types:

After publication, the Pyxis API links each image record to its advisory via image_advisory_id.

VEX Feed Update

After advisory publication, Red Hat SDEngine generates public vulnerability data:

• CSAF VEX documents (per-CVE) published at https://security.access.redhat.com/data/csaf/v2/vex-feed/
• CSAF Advisory documents (per-advisory) published at https://security.access.redhat.com/data/csaf/v2/advisories/

The hummingbird-vex-feed repository maintains CPE mappings that associate Hummingbird package repositories with cpe:/a:redhat:hummingbird:1.

Image Documentation

When a commit to main changes a README.md file, the content is automatically pushed to quay.io as the image description via the update_quay_description job.

References

• RPM Pipeline - How RPM spec changes flow through the build pipeline to the package repository
• Development Workflow - Practical guide for contributing
• Adding Images - Step-by-step guide for adding new images
• Testing Guide - How to run and write tests locally
• Image Configuration Reference - Complete properties.yml reference

3.2 - Global Variables Reference

Complete reference for global configuration settings that apply to all container images.

Overview

The images/variables.yml file contains global configuration that applies to all container images in the repository. These settings define project-wide defaults, security parameters, and variant-specific behavior.

Location: images/variables.yml (in the containers repository root)

Scope: All images inherit these settings unless overridden in image-specific properties.yml

Configuration Reference

cpe

• Type: String (CPE 2.2 formatted string)
• Default: "cpe:/a:redhat:hummingbird:1"
• Description: Common Platform Enumeration (CPE) identifier for all Hummingbird containers.
• Usage: Automatically added as a label to all generated container images and passed to inject-source-info during the build process.
• Format: CPE 2.2 URI format: cpe:/part:vendor:product:version

default_user

• Type: String (numeric UID)
• Default: "65532"
• Description: The default unprivileged user ID that all containers run as.
• Usage: Referenced by the {{ set_user() }} macro when user: default is specified (or when user: is omitted, as default is the default value).
• Value: 65532 is chosen as a high, non-conflicting UID that:
  • Avoids conflicts with system users (typically < 1000)
  • Avoids conflicts with regular users (typically 1000-60000)

default_distros

• Type: Array of strings
• Default: ["rawhide", "hummingbird"]
• Description: Defines the distribution variants (distros) for which images are built. Each distro represents a different base package source.
• Directory Structure: Images are organized as images/<name>/<distro>/<variant>/
• Common Values:
  • rawhide - Uses Fedora Rawhide repositories only
  • hummingbird - Uses both Fedora Rawhide and Hummingbird repositories
• Usage: Scripts iterate over distros and variants to find all image variants to build. Each distro/variant combination produces a separate container image.
• See Also: default_variant_repos for repository configuration per distro

default_variants

• Type: Array of strings
• Default: ["default", "builder"]
• Description: Defines the default set of variants to generate for each image within each distro when not explicitly specified in the image’s properties.yml.
• Directory Structure: Variants are subdirectories within each distro: images/<name>/<distro>/<variant>/
• Common Values:
  • default - The minimal runtime variant
  • builder - Extended variant with build tools and package managers
• Extending: Images can use additional_variants in properties.yml to add extra variants while keeping the defaults (preferred approach)
• Override: Images can specify variants in properties.yml to replace these defaults entirely (use when you need to exclude default variants)
• See Also: Image Configuration Reference - variants

default_rpm_packages

• Type: Object with variant names as keys, arrays of package names as values

• Default:

  default_rpm_packages:
    builder:
      - bash
      - dnf5
      - shadow-utils
  

• Description: Defines packages that are automatically included in specific variants across all images. These are added in addition to packages defined in image-specific properties.yml.

• Variants: Currently only builder is defined, but other variants can be added

• Usage: Use this to provide variant-specific packages that should be available in all images using that variant. For example, builder variants include package management tools for development workflows.

default_variant_repos

• Type: Object with distro names as keys, arrays of repository filenames as values

• Default:

  default_variant_repos:
    rawhide:
      - fedora-44.repo
    hummingbird:
      - fedora-44.repo
      - hummingbird.repo
  

• Description: Defines which yum repositories each distro uses by default for package installation and lockfile generation. The Rawhide distro uses the latest branched Fedora development repo, while Hummingbird uses a stable Fedora release plus the Hummingbird package repository.

• Repository Files: References files in the yum-repos/ directory

• Usage: This allows different distros to use different package sources. The hummingbird distro includes additional repositories that provide Hummingbird-specific packages.

• Override: Image-specific additional_repos in properties.yml are appended to the distro-specific repos

• See Also: Image Configuration Reference - additional_repos

oscap

• Type: Object

• Default:

  oscap:
    enabled: true
    profiles:
      cis: true
      stig:
        variants:
          - "*fips*"
    exclude_rules:
      - id: xccdf_org.ssgproject.content_rule_root_path_no_dot
        reason: "False positive: environmentvariable58 probe requires /proc
          which is unavailable in offline chroot scanning"
  

• Description: Global defaults for OpenSCAP compliance scanning. Scanning is enabled by default (enabled: true). Images can opt out by setting enabled: false in their properties.yml.

• Fields:

  • enabled - Whether compliance scanning is active (overridden per image)
  • profiles - Which compliance profiles run per variant. Each profile can be true (all variants), false (disabled), or {variants: [...]} with glob patterns. Default: CIS for all variants, STIG for FIPS variants.
  • exclude_rules - Global rule exclusions applied to all oscap-enabled images. Image-specific exclusions in properties.yml are concatenated with these (not replaced).

• Merge Behavior: When an image overrides oscap fields:

  • enabled (scalar) is replaced by the image value
  • profiles (dict) is recursively merged, so an image can override individual profiles without affecting others
  • exclude_rules (list) is concatenated, so image rules are added after global rules

• See Also: Image Configuration Reference - Compliance Scanning

readme_targets

• Type: Object with target names as keys, configuration objects as values

• Default: See the actual file

• Description: Defines target-specific configuration for generating multiple README files from a single template. Each target represents a different build/distribution channel (e.g., Hummingbird project vs Red Hat product).

• Target Configuration Fields:

  • filename - Output filename for this target’s README
  • registry - Container registry URL for image references
  • product_name - Full product name for documentation
  • product_name_short - Short product name for headings and titles
  • doc_base_url - Base URL for documentation links

• Usage: Templates access these values via {{ readme_targets[target].registry }} and similar expressions. The build system generates one README file per target from the same README.md.j2 template.

• See Also: Image Pipeline - README Generation

Next Steps

• Image Configuration Reference - Per-image configuration
• Image Variants - Understanding variant types
• Adding Images - How to add a new container image

3.3 - Konflux Resource Deployment

Konflux resources for container images are split between two repositories:

• containers: Per-image resources generated from properties.yml
• infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Legend:

• Containers MR: Downstream pipeline triggered from containers MR (deploys from MR commit)
• Containers Push: Downstream pipeline triggered from containers push to main
• Infra MR: Infrastructure MR pipeline (manual trigger)
• Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

Containers Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

1. Ownership: Components are managed by the containers repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from containers deploy changes, giving the containers repo full control over the component lifecycle.
2. Dynamic generation: Generated from images/*/properties.yml via ci/internal/generate_konflux_resources.sh, depending on per-image configuration (variants, tags, repository names).
3. Timing: Must be deployed early during MR review so Konflux can build and test new images.

ReleasePlanAdmission:

1. Dynamic generation: Includes per-image tag mappings extracted from properties.yml.
2. Main branch only: Prevents race conditions—deploying from a feature branch risks undoing other images’ onboarding. If branch A adds image-foo and branch B (created earlier) adds image-bar, deploying B’s RPA would remove image-foo from the release config.

EnterpriseContractPolicy:

1. Documentation co-location: Policy exclusions are documented in Image Pipeline. Keeping policy and documentation together eases maintenance.

Infrastructure Repo

Application, ReleasePlan, and IntegrationTestScenario are defined in kubernetes/containers-*/ directories (one per Konflux application):

1. Independent iteration: Changes are decoupled from containers repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the containers repo.
2. Testable before merge: If defined in the containers repo, these would only deploy after merging to main (like RPAs), making iteration difficult.
3. Static configuration: These resources don’t depend on per-image data from properties.yml.

ServiceAccount and Secret for releases are defined in kubernetes/release-quay-hummingbird*/ directories (one per Quay organization):

1. Security: Secret specifications (names, structure, credential references) should not be exposed in the containers repo.
2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the containers repo.

The service accounts are referenced by ReleasePlanAdmissions to authorize pushing images to each quay.io/hummingbird* organization.

See Also

• Image Pipeline – Build, test, and release stages

3.4 - Security Labels and Metadata

Hummingbird container images include labels, embedded metadata, and SBOMs that enable security scanners to perform container-first vulnerability reporting.

Overview

Security scanners need a way to determine which vulnerabilities are applicable to a container image. Three mechanisms provide this information:

1. Container Labels - OCI image labels (name, cpe) set in the Containerfile
2. Embedded Metadata - A labels.json file written to the container filesystem during build
3. Software Bill of Materials (SBOM) - An SPDX document listing all packages, attached as an OCI artifact alongside the image

Labels and embedded metadata identify the product; the SBOM identifies the packages within it. Labels and embedded metadata provide the same core information but serve different access paths: labels are accessible via container inspection tools, while embedded metadata is accessible to scanners with only filesystem access.

Applicability

The cpe label is only added to released Hummingbird distro images (no CPE identifier for non-product images). All other labels are added to all images.

The labels.json embedded metadata file contains all Containerfile LABEL values plus auto-computed fields.

SBOMs are attached to all released images (both Rawhide and Hummingbird).

Container Labels

name

• Type: String
• Format: <org>/<image> or <org>/<image>-<variant>
• Examples: hummingbird/nodejs-24, hummingbird/nodejs-24-builder, hummingbird/caddy
• Description: Canonical name for the container image. This is the name that appears in VEX (Vulnerability Exploitability eXchange) statements and is used by scanners to map vulnerabilities to specific images.

The canonical name is derived from the image directory name (which includes the version, e.g. nodejs-24) and the variant. For default variants, the name is hummingbird/<image> (e.g. hummingbird/nodejs-24). For non-default variants, the variant is appended with a hyphen (e.g. hummingbird/nodejs-24-builder). Note that the canonical name may differ from the registry repository path. For example, nodejs-24 and nodejs-20 are both pushed to the hummingbird/nodejs registry repository with different tags, but their canonical names are hummingbird/nodejs-24 and hummingbird/nodejs-20 respectively.

See container-image-labels.md for the full name label format across all image types.

cpe

• Type: String (CPE 2.2 formatted)
• Format: cpe:/a:redhat:hummingbird:1
• Description: Common Platform Enumeration (CPE) identifier for the container. CPE is a standardized naming scheme for software products that enables correlation with vulnerability databases.

The CPE value is defined globally in images/variables.yml and applied to all Hummingbird images. Containers with the same CPE are considered the same software product for vulnerability reporting purposes.

org.opencontainers.image.created

• Type: String (RFC3339 timestamp)
• Format: 2025-01-15T12:00:00Z
• Description: Creation timestamp of the container image. Used by scanners to determine if an image predates or postdates a vulnerability fix.

If SOURCE_DATE_EPOCH is set during the build, that timestamp is used instead of the actual build time, supporting reproducible builds.

Embedded Metadata (labels.json)

The labels.json file is written to /usr/share/buildinfo/labels.json inside the container filesystem by the inject-source-info script during build. It contains all Containerfile LABEL values plus auto-computed fields like architecture and org.opencontainers.image.created.

Schema

The file follows the embedded_metadata.v1 schema published by Red Hat Product Security.

Fields

The embedded_metadata.v1 schema defines the minimum fields required by security scanners:

The file also includes all other Containerfile LABEL values (e.g., description, summary, vendor, version). See container-image-labels.md for the complete label reference.

Example

{
  "name": "hummingbird/caddy",
  "cpe": "cpe:/a:redhat:hummingbird:1",
  "architecture": "amd64",
  "org.opencontainers.image.created": "2025-01-15T12:00:00Z"
}

Software Bill of Materials (SBOM)

Each released image has a per-architecture SPDX 2.3 SBOM attached as an OCI artifact. See SBOM Generation for how the production SBOMs are built from Syft, Hermeto, and Mobster.

Accessing SBOMs

SBOMs are stored as OCI artifacts alongside each per-architecture image. To download one, resolve the per-arch digest and use cosign:

IMAGE="registry.access.redhat.com/hi/caddy"
TAG="latest"
ARCH="amd64"

# Get per-architecture digests from the image index
skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq '.manifests[] | {digest, platform}'

# Download the SBOM for the chosen architecture
ARCH_DIGEST="$(skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq -r --arg arch "${ARCH}" \
    '.manifests[] | select(.platform.architecture == $arch) | .digest')"
cosign download sbom "${IMAGE}@${ARCH_DIGEST}" > sbom.json

Entry Sources

The merged SBOM contains entries from two tools:

A package installed in the final image may have entries from both Syft and Hermeto (with different metadata), since it was both a build dependency and is present at runtime.

Distinguishing Entry Sources

After the Mobster merge, Syft and Hermeto entries carry mutually exclusive markers:

As a general rule, any entry with an annotation containing the word “hermeto” (case-insensitive) originates from Hermeto (build-time provenance). All other entries originate from Syft (runtime image scan).

PURL Format

Syft and Hermeto use different PURL qualifier sets for the same package. Examples from caddy:latest (Hummingbird, amd64):

Syft (runtime):

pkg:rpm/hummingbird/caddy@2.10.2-1.hum1?arch=x86_64&distro=hummingbird-20251124&upstream=caddy-2.10.2-1.hum1.src.rpm

Hermeto (build, binary):

pkg:rpm/caddy@2.10.2-1.hum1?arch=x86_64&checksum=sha256:ca02a0...&repository_id=public-hummingbird-x86_64-rpms

Hermeto (build, source):

pkg:rpm/caddy@2.10.2-1.hum1?arch=src&checksum=sha256:42912d...&repository_id=public-hummingbird-source-rpms

Key differences:

• Distro namespace: Syft includes the distro in the PURL path (pkg:rpm/hummingbird/...). Hermeto omits it (pkg:rpm/...).
• EVR: Syft populates versionInfo with the full epoch:version-release. Hermeto sets versionInfo to the bare upstream version; the full EVR is only in the PURL @version.
• Architecture: Syft entries match the image architecture. Hermeto entries carry an arch= PURL qualifier that may be the image arch, a different arch (cross-arch), noarch, or src (source RPMs).
• Source RPM: Syft entries carry upstream=<srpm> in the PURL. Hermeto entries have separate arch=src entries instead.

Hermeto Annotation Format

Hermeto entries carry annotations with JSON-encoded metadata:

{
  "annotationDate": "2026-03-02T11:58:12Z",
  "annotationType": "OTHER",
  "annotator": "Tool: hermeto:jsonencoded",
  "comment": "{\"name\": \"hermeto:found_by\", \"value\": \"hermeto\"}"
}

Entry Breakdown Example

Typical counts for caddy:latest (Hummingbird, amd64); exact counts vary as image dependencies change:

How Scanners Use This Metadata

1. CPE Matching: Scanners use the cpe value to look up applicable VEX statements for the product
2. Name Matching: The name identifies which specific container the VEX statements apply to
3. Version Comparison: The creation timestamp enables comparison between the scanned image and fixed versions reported in VEX statements
4. Package Enumeration: Scanners use the SBOM to enumerate all packages in the image and correlate them with vulnerability databases via PURLs and CPEs

Related Files

See Also

• Container Image Labels – complete reference for all image labels across all categories
• Image Pipeline – SBOM Generation – how the production SBOMs are built from Syft, Hermeto, and Mobster

References

• Embedded Metadata Schema
• Container-First Vulnerability Reporting (Konflux feature specification)
• VEX (Vulnerability Exploitability eXchange)
• Syft – SBOM generation tool
• Hermeto – build-time dependency provenance tool
• Mobster – SBOM merging tool
• SPDX Specification

3.5 - Container Image Labels

Hummingbird container images carry labels from multiple standards and namespaces. This page is the single reference for every label.

Standards: C = Conforma (rule dataset), O = OCI Image Spec, H = Hummingbird project, S = Security schema

Labels

Notes

¹ Licenses

The com.redhat.license_terms label is not required by Conforma, OCI, or the security schema. It is a Red Hat convention present on all Red Hat container images (UBI, language runtimes, etc.), pointing to the UBI EULA.

The org.opencontainers.image.licenses label is intentionally not set. Per-package license data is derived from SBOMs stored as OCI artifacts alongside each image. A manually-set SPDX expression would be incomplete compared to the SBOM-derived data available in the image catalog.

² Name and identity

The name label identifies the canonical publishing repository. It uses the repository value from properties.yml, prefixed by the registry organization:

• Red Hat supported Hummingbird images: hi/<repository>
• Community images: hummingbird-community/<repository>
• Experimental images: hummingbird-ci/<repository>
• Rawhide images: hummingbird-rawhide/<repository>

All variants published to the same repository have the same name label. For example, the default and builder variants of Caddy both use hi/caddy.

The cpe label is only set for Hummingbird distro images (not Rawhide). See Security Labels and Metadata for details on the scanning workflow and labels.json.

³ Version labels

Three version granularities are derived from the image’s tags: version (full, e.g., 2.10.2), major-minor-version (e.g., 2.10), and major-version (e.g., 2). The full version is the OCI/Conforma org.opencontainers.image.version; the two coarser granularities are Hummingbird project labels used for tag aliasing. There is no io.hummingbird-project.version label — it would duplicate the OCI label.

⁴ Repository and stream

The io.hummingbird-project.repository and io.hummingbird-project.stream fields together form the release identity pair. Both are defined in properties.yml. See image-configuration-reference for field definitions.

The io.hummingbird-project.deprecated label is present with the value true on the final release of a deprecated stream. It is omitted for non-deprecated streams. The catalog uses the label to preserve the current deprecation state without changing older image digests.

⁵ Description, summary, and URL

The description, summary, and url fields are defined in properties.yml. Description and summary target different display contexts:

• summary: One-liner (~40-80 chars) for table/list views
• description: Short paragraph (~100-250 chars, 1-2 sentences) for card views and podman inspect

Style rules:

• Do not start summary or description with the image name
• Use >- YAML scalar for multi-line readability in properties.yml
• Avoid embedded double quotes and backslashes (no escaping in templates)

⁶ Vendor

All images use Red Hat, Inc. as vendor (the distributing entity). The OCI org.opencontainers.image.vendor uses Red Hat (without “, Inc.”) per OCI convention. Both values are set on all distros.

⁷ Release

The release label uses the commit timestamp as a Unix epoch (matching Red Hat convention). In Konflux this is set via the generate-labels pipeline task.

⁸ Variant labels

Each variant name is decomposed into a base specialization and cross-cutting modifiers (builder, fips). The naming convention is <base>[-fips][-builder], where modifier order does not matter. A bare modifier like builder has base default.

Examples: fpm-builder → base=fpm, builder=yes; fips-builder → base=default, builder=yes, fips=yes.

The .description label stores only the base description (e.g., “PHP FastCGI process manager”). Modifier display is handled by consumers using the .builder and .fips boolean labels. Base descriptions come from variant_descriptions in images/variables.yml (for default) and image-specific properties.yml (for bases like fpm, runtime).

⁹ Containerfile path

The io.hummingbird-project.containerfile label contains the path to the Containerfile source relative to the repository root (e.g., generated/images/caddy/hummingbird/default/Containerfile). A full URL to the source can be constructed from org.opencontainers.image.source + org.opencontainers.image.revision + this path.

Embedded Metadata (labels.json)

All labels are written to /usr/share/buildinfo/labels.json inside the container filesystem, providing filesystem-level access for security scanners. See Security Labels and Metadata for schema details.

Related Files

3.6 - CI Scripts

This section documents the CI scripts used in the Project Hummingbird container build and test pipeline.

These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline.

3.6.1 - build_images.sh

Purpose

Build container images using buildah with support for multiple architectures and container engines.

Usage

Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...]

OPTIONS:
    --verbose, -v        Enable verbose output during build
    --arch ARCHITECTURE  Specify target architecture (e.g., amd64, arm64, s390x)
    --engine ENGINE      Specify runtime engine for testing/export (podman or docker)
    --setup              Set up Docker-in-Docker environment before exporting
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --local-rpms-dir DIR Directory containing custom RPMs to use during build.
                         These have a higher priority and thus override the
                         standard repositories.
    --build-deps         Build all dependencies of specified groups, but NOT the
                         specified groups themselves. Collects forward, reverse,
                         and transitive dependencies automatically.
    --affected-tests     With --build-deps, select dependencies needed by affected tests
    --pull               Pull images from registry instead of building locally.
                         Fast alternative for local development. Uses published
                         images from quay.io.
    --dryrun             Show what would be built (or pulled with --pull) without
                         actually building/pulling.
    --help, -h           Show this help message

Examples

# Build single image group (all distro/variants)
ci/build_images.sh nginx
ci/build_images.sh nodejs-20

# Build specific distro/variant only
ci/build_images.sh nginx/rawhide/builder
ci/build_images.sh nodejs-20/hummingbird/default

# Build multiple image groups at once
ci/build_images.sh nginx curl git
ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder

# Build with options
ci/build_images.sh --verbose curl
ci/build_images.sh --arch arm64 nginx
ci/build_images.sh --engine docker nginx
ci/build_images.sh --engine podman --verbose nginx

# Build multiple image groups with options
ci/build_images.sh --verbose nginx curl git
ci/build_images.sh --arch arm64 nginx postgresql

# Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies from registry instead of building (faster for local dev)
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Pull a specific image from registry
ci/build_images.sh --pull nginx/rawhide/default

# Build from CI component name format (used in CI environments)
ci/build_images.sh --component-name curl--rawhide--default

# Build with custom RPMs (for testing modified packages)
ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder

Note: Some images require git submodules initialized (use git init --recurse-submodules when cloning or git submodule update --init if already cloned). When building for foreign architectures, make sure qemu-user-static is available.

Dependency Building

The --build-deps flag builds all dependencies needed for testing an image, but not the image itself. This is used in CI to ensure all required images are available before running tests.

Pulling vs Building Dependencies

For local test development, use --pull with --build-deps to pull pre-built images from the registry instead of building them locally. This is much faster and uses the same dependency collection logic:

# Slow: Build all dependencies locally
ci/build_images.sh --build-deps nginx

# Fast: Pull all dependencies from registry
ci/build_images.sh --build-deps --pull nginx

The --pull flag works without --build-deps too:

# Pull a specific image instead of building it
ci/build_images.sh --pull caddy/rawhide/default

How Dependency Collection Works

When you run ci/build_images.sh --build-deps <image>, it performs a 2-level expansion:

1. Level 1: Collects forward and reverse dependencies of the specified image

   • Forward dependencies: Images that the specified image’s tests depend on (detected by TEST_IMAGES[...] references in test files)
   • Reverse dependencies: Images that depend on the specified image (filtered by reverse_dependency_tests: true in properties.yml)

2. Level 2: Collects forward dependencies of the reverse dependencies

3. Stops: No further expansion (avoids infinite graph traversal)

All dependencies are automatically deduplicated to ensure each image is built exactly once.

Relationship to Testing

The reverse_dependency_tests property in properties.yml affects both building and testing:

• Build phase (--build-deps): Filters which reverse dependencies to build
• Test phase (ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps): Filters which reverse dependencies to test

Set reverse_dependency_tests: false for images like curl that are used pervasively but don’t need reverse dependency workflows.

Dependency Building Examples

# Build dependencies for dotnet-runtime-10-0
# Builds: dotnet-sdk-10-0 (forward dependency)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies for dotnet-runtime-10-0 (faster alternative)
# Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Build dependencies for core-runtime
# Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true)
# Plus their forward dependencies
ci/build_images.sh --build-deps core-runtime

# Use --dryrun to see what would be built/pulled without actually doing it
ci/build_images.sh --dryrun --build-deps core-runtime
ci/build_images.sh --dryrun --build-deps --pull core-runtime

Building with custom RPMs

The --local-rpms-dir option enables testing container images with custom-built RPM packages, for iterating on package changes before committing to the RPM repository.

Workflow

1. Build custom RPMs in the rpms repository (see its documentation for the complete workflow):

   cd ../rpms
   # modify a package
   ci/build_rpms.sh packagename
   # Built RPMs will be in builds/packagename/RPMS/
   

2. Build container image using the custom RPMs:

   cd ../containers
   ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder
   

3. Verify the custom package was installed:

   podman run --rm --entrypoint '' registry.access.redhat.com/hi/imagename:latest-builder rpm -qa
   

Builds for affected container tests

Use --build-deps --affected-tests to select dependencies needed by affected container checks across both engines. Testing Farm adds --pull to fetch published dependencies. Supplied Konflux image URL overrides take precedence. Omit --pull to validate unpublished dependency changes from the checkout.

ci/build_images.sh --build-deps --affected-tests --pull core-runtime/hummingbird/default

--affected-tests requires --build-deps. Without it, dependency building retains the full reverse suite for callers that need broader coverage.

3.6.2 - run_tests_container.sh

Purpose

Run tests for image groups using containerized test environment with support for both Docker and Podman engines.

Usage

Run ci/run_tests_container.sh --help for full usage information.

Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --hermetic           Use hermetic builds (--pull=never, only use prefetched/built images)
                         Default for CI. Without this, missing images are pulled on demand.
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --affected-tests     Run only affected dependent image checks, retaining globals
    --dryrun             Print selected tests without invoking container engines
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Engine Selection: Use --engine to specify the container engine (podman or docker). When using Docker, add --setup for automatic Docker-in-Docker environment setup.

Hermetic vs Non-Hermetic Testing:

• Local development (default): Without --hermetic, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first.
• CI environment: Use --hermetic to enforce that tests only use prefetched or locally-built images (via --pull=never). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built.

Building and Testing

For local development, you’ll typically want to build the images first, then test them:

# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups

The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.

When working on base images (like core-runtime) that other images depend on, build the base image and its dependencies before running reverse-dependency tests:

# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime

The -p/--pause option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging.

Testing with Specific Image Builds

To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT> environment variables (uppercase, hyphens/dots → underscores, __ separates parts):

# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
  ci/run_tests_container.sh tomcat-10/hummingbird/builder

Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.

Retry Behavior:

• Failed tests are checked against a list of retriable error patterns
• If a match is found, the test is automatically retried
• If the test still fails after all attempts, it’s reported as a normal failure

Examples

# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder

# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default

Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:

• Find images that depend on it by scanning for TEST_IMAGES[group/distro/variant] references
• Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting reverse_dependency_tests: false in properties.yml (e.g., curl, which is widely used but typically doesn’t need reverse dependency testing).

Affected dependency tests

Testing Farm uses --affected-tests for single-component jobs. Dependency pulls use ci/build_images.sh --build-deps --affected-tests --pull with the same selector. The requested component runs its complete suite. Dependent variants run only image checks that reference the changed component variant, plus all applicable global checks.

Use --dryrun to list runnable checks without invoking container engines. Use --include-reverse-deps without --affected-tests to run full dependent suites.

Run each suite with an isolated container engine. Cleanup preserves resources present at startup and removes resources created during the suite, including resources created concurrently by other processes.

3.6.3 - run_tests_k8s.sh

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

The --push-image flag:

• Compares local image digest with remote
• Skips push if image already exists with same digest
• Uses port-forward to registry-proxy for pushing
• Sets TEST_IMAGE to the internal registry reference

Prerequisites:

• oc login to the target cluster
• oc project <namespace> to set the target namespace
• Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Environment Variables

Tests have access to these environment variables:

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

# Build locally and push to internal registry
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

• --kubeconfig pointing to the ephemeral namespace kubeconfig
• --component-name or --group-component-name for component identification
• --output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.

3.6.4 - retrigger_failed_checks.py

Purpose

Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running.

Usage

Run ci/retrigger_failed_checks.py --help for full usage information.

Usage: ci/retrigger_failed_checks.py <MR_URL> [OPTIONS]

OPTIONS:
    -h, --help              show this help message and exit
    --dry-run               just print what comments would be posted
    --token-path, -t PATH   path to file containing GitLab API token

Authentication (in order of precedence):
  1. --token-path: Path to file containing GitLab API token (highest priority)
  2. GITLAB_TOKEN_PATH: Environment variable with path to token file
  3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback)

Behavior

The script will:

1. Extract failed pipeline runs from the commit statuses on the MR
2. Generate /retest {pipeline-name} commands for each failed run
3. Post the retest commands as MR comments (unless –dry-run)
4. Wait for the pipeline runs to start (unless –dry-run)

Examples

# Using --token-path option (highest priority)
ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN_PATH environment variable
GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN environment variable (legacy)
GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Dry-run mode (show what would be done, don't post or wait)
ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234

3.6.5 - gitlab_sync.py

Purpose

Sync a file from a public source URL to a target GitLab repository via merge request. The script always waits for the MR to be merged and for the post-merge pipeline to complete before returning. It is invoked by infrastructure CI jobs to keep external repos (pyxis-repo-configs, konflux-release-data) in sync with generated files from the source repos.

Usage

Run gitlab_sync.py --help for full usage information.

usage: gitlab_sync.py [-h] --source-url SOURCE_URL
                      --target-project TARGET_PROJECT
                      --target-file TARGET_FILE --sync-branch SYNC_BRANCH
                      --mr-title MR_TITLE --gitlab-url GITLAB_URL
                      [--merge-timeout MERGE_TIMEOUT]
                      [--post-merge-timeout POST_MERGE_TIMEOUT]
                      [--mr-description MR_DESCRIPTION] [--squash-on-merge]
                      [--dry-run]

Authentication:
  GITLAB_TOKEN    Environment variable with GitLab API token (required)

Monitoring:
  SENTRY_DSN      Sentry DSN for alerting on failures (optional)

How It Works

1. Fetch source from the public --source-url (with HTTP retries)
2. Determine comparison ref: use the sync branch if it exists, otherwise the project’s default branch
3. Compare source content against the comparison ref (stripped whitespace)
4. Early exit if content matches and the comparison ref is the default branch (nothing to deploy)
5. Commit the source to the sync branch using the GitLab Commits API with force: true (creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits.
6. Create or update MR targeting the default branch
7. Self-approve the MR (best-effort; continues if already approved)
8. Post takeover comment with CI_JOB_ID for preemption tracking
9. Wait for MR merge (polling every 30s, up to --merge-timeout)
10. Wait for post-merge pipeline to succeed (up to --post-merge-timeout)

Exit Codes

Preemption

When multiple CI jobs target the same sync branch, the script uses MR comments to coordinate. Each job posts a comment containing its CI_JOB_ID (monotonically increasing within a GitLab instance). Before each merge attempt, the script checks for comments with a higher job ID. If found, it yields by exiting with code 42.

The caller should handle exit 42 to stop processing (a newer job will handle all remaining syncs). When running outside CI (CI_JOB_ID not set), preemption is disabled.

Error Recovery

• Pipeline failure: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted.
• Merge conflict: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert.
• Post-merge pipeline failure: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA).
• Draft MR: Keeps polling without attempting to merge (allows manual intervention).

Examples

# Dry run: show diff without modifying anything
GITLAB_TOKEN=$TOKEN gitlab_sync.py \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --dry-run

# Sync Pyxis config (requires squash merge)
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --squash-on-merge

# Sync RPM RPA to konflux-release-data
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \
  --target-project "releng/konflux-release-data" \
  --target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \
  --sync-branch "hummingbird/sync-rpms-rpa" \
  --mr-title "Update hummingbird RPM ReleasePlanAdmission"

Preemption wrapper for sequential syncs in a CI job:

gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge
rc=$?
if [ "$rc" -eq 42 ]; then
  echo "Preempted by newer job, stopping"
  exit 0
elif [ "$rc" -ne 0 ]; then
  exit "$rc"
fi
# Proceed to next sync only on success
gitlab_sync.py --gitlab-url ... --source-url ...

Environment Variables

Development

The script and gitlab-ci tool image are maintained in the Hummingbird tools repository, not this repository. Make implementation and test changes there.

4 - K8s Test Pipeline

Background documentation sourced from the K8s test pipeline repository, covering pipeline design, test format, and EaaS debugging for Kubernetes integration tests.

4.1 - Pipeline Design

Task/Step Overview

Both check-for-tests and run-tests independently fetch source — each task gets its own emptyDir volume, so no state carries across tasks.

Design Guidelines

1. Filesystem over results

Pass data between steps via files on a shared emptyDir, not Tekton results. Tekton results have a 4KB termination message limit (tektoncd/pipeline#4060) and large $(params.*) substitutions hit ARG_MAX.

Each task mounts an emptyDir at /workdir. Steps communicate via files: snapshot.json, component, image_names, src/, kubeconfig, result, error. Zero $(params.*) references remain inside script: blocks — all param values are passed via env vars or written to /workdir.

2. Distinguish retryable from non-retryable errors

Use exit 1 for transient failures (network, registry) that benefit from retry. Use exit 0 + a sentinel file for permanent failures (bad data) to prevent wasted retries.

Non-retryable errors write Konflux-format JSON to /workdir/error and exit 0. Subsequent steps check for this sentinel and skip. The write-results step copies it to TEST_OUTPUT. Tasks default to retries: 2, but fail-pipeline-on-test-failure has retries: 0 — test failures should not be retried.

3. Fail fast

Every shell script enables set -euo pipefail so unexpected command failures surface immediately rather than cascading silently.

4. Separate data from status

Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration.

Konflux reads TEST_OUTPUT results as data but does not fail the pipeline based on them. The fail-pipeline-on-test-failure finally-task reads both TEST_OUTPUT values and exit 1 on non-SUCCESS. This is needed because GitLab PAC integration requires the pipeline itself to fail for MR status reporting.

5. Gate expensive work behind cheap checks

Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do.

provision-namespace and run-tests are gated on check-for-tests.results.HAS_TESTS == "true" via Tekton when expressions. If no tests-k8s.yml exists, the pipeline completes with just check-for-tests, skipping EaaS provisioning. This is the common case for components without K8s tests.

6. Make failures reproducible

Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally.

Stderr from cosign/jq/oras is not swallowed. The full run_tests_k8s.sh command line is logged.