RPM Pipeline

Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.

Overview

The RPM pipeline consists of five main stages:

1. Spec Change - A package spec file is modified in rpms/<package>/
2. Merge Request & Validation - CI validates the change
3. Build - Konflux builds RPMs per architecture via mock in hermetic mode
4. Signing - Built RPMs are cryptographically signed
5. Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository

After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.

flowchart TD
    A["Spec Change<br/>rpms/&lt;package&gt;/"] --> B["Merge Request"]
    B --> C["CI Validation<br/>(check, tree_status)"]
    C --> D["Merge to main"]
    D --> E["Konflux RPM Build<br/>Tekton PipelineRun per package<br/>mock hermetic build per arch"]
    E --> F["RPM Signing<br/>Kerberos-based"]
    F --> G["Publish to Pulp<br/>packages.redhat.com<br/>/public-hummingbird/&lt;arch&gt;/"]
    G --> H["Container lockfile updates<br/>(see Image Pipeline)"]

Stage 1: Spec Change

An RPM change begins as a commit in rpms/<package>/. The change typically includes a modified .spec file (updated version, new patch, or Release bump) and any associated source files or patches.

How changes originate

See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.

Package metadata

Each package has a metadata file at metadata/<package>.json that tracks its relationship to upstream:

{
  "source": "https://src.fedoraproject.org/rpms/curl.git",
  "branch": "rawhide",
  "sha": "abc123...",
  "version": "8.12.1",
  "release": "1",
  "modification_status": "clean",
  "upstream_repo": "https://github.com/curl/curl"
}

Key fields:

See Package Metadata Fields for modification_status and release configuration, and Package Modification Tracking for managing modification status day to day.

Stage 2: Merge Request & Validation

Changes reach main via merge requests. Automated MRs are created using:

• ci/create_mr.sh - Single MR with configurable options
• ci/rebuild_multi_mr.sh - One MR per package with auto-merge enabled
• ci/dist_git_update_multi_mr.sh - Automated Fedora update MRs (scheduled CI job)
• ci/upstream_update_multi_mr.sh - Automated upstream version update MRs (scheduled CI job)

CI validation

GitLab CI runs validation jobs on every merge request:

• check - Linting, type checking, and spec validation
• check_nevr_conflicts - Predicts the NEVR(A) each changed package’s spec would build (using rpmspec with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale main, e.g. two rebuild MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch rpms/**/*, and within that, only checks packages whose rpms/<package>/ directory itself changed – matching Konflux’s own build trigger – not packages whose only change is under metadata/, since a metadata-only change (e.g. ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally with make check-nevr-conflicts ARGS='<package>', or against real built RPMs with make check-nevr-conflicts ARGS='--rpms-dir builds/<package>' after ci/build_rpms.sh. See ci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages routed to a private_product are noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode some OK results can be for a package name a real build never actually produces (e.g. a base package with no top-level %files section, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount a CONFLICT found elsewhere in the same run.
• tree_status - Repository consistency checks (metadata integrity, spec file presence)
• Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)

Auto-approval for chore MRs

Automated update MRs (chore/* branches) follow an accelerated path:

1. MR is created with auto-merge enabled
2. Konflux builds the package and posts commit statuses
3. After a delay, the chore_mr_approval CI job auto-approves the MR
4. GitLab merges the MR once the pipeline succeeds

Stage 3: Build

After merging to main, Konflux builds RPMs automatically.

Build triggers

Each package has a Tekton PipelineRun definition generated from its directory in rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered .tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers a build when changes to that package’s directory are pushed to main. When a single push touches multiple packages, one PipelineRun is triggered per affected package directory. Merge request pushes also trigger builds for validation.

The setup package acts as a canary: changes to ci/ or mock/ directories also trigger a setup build on pull requests, validating build infrastructure changes before they reach main.

Build process

The build pipeline (build-rpm-package) executes for each target architecture:

1. Checks out the repository at the merge commit
2. Calculates build dependencies
3. Runs mock in hermetic mode (network-isolated with pre-fetched dependencies)
4. Produces RPM artifacts and build logs
5. Generates an SBOM (Software Bill of Materials)
6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry

Build architectures

Packages are built for the architectures specified in their PipelineRun definition. Most packages build for x86_64 and aarch64. Some packages also build for s390x and ppc64le.

Build output

Build artifacts are stored in the Konflux OCI registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main

Konflux resources

Build infrastructure is defined across two repositories:

• rpms repo (konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy
• infrastructure repo (kubernetes/rpms-main/) - Application, ReleasePlan, IntegrationTestScenario, ServiceAccount, and Secret resources

See Konflux Resource Deployment for details on how these resources are managed and deployed.

Testing

RPM packages are validated through integration tests that run on Testing Farm infrastructure.

Tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test triggering

Tests are only triggered on merge requests, not on main branch builds. For a package at rpms/<package>/, tests are triggered for all changes below that directory.

Integration Test Scenario

Tests are triggered via an IntegrationTestScenario resource defined in the infrastructure repository:

Configuration: infrastructure/kubernetes/rpms-main/10-integration-test-scenarios-testing-farm.yml.j2

The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:

Scenario parameters:

• COMPOSE: RHEL-9-Nightly - Test environment OS/compose
• PIPELINE_MODE: rpm - Configures RPM testing mode (vs. container)
• ARCH: x86_64,aarch64 - Architectures to test (creates separate test runs per arch)
• PASS_SNAPSHOT_TO_TF: false - Don’t pass full snapshot JSON (package info comes via IMAGE_NAME/IMAGE_URL instead)
• IMAGE_TAG: - Version of the tmt-via-testing-farm pipeline bundle

Key characteristics:

• Single scenario for all packages - Not per-package, runs for every package build
• Context-based triggering - Runs on pull_request snapshots only
• Package identification - Konflux automatically provides IMAGE_NAME (e.g., openssl-main) and IMAGE_URL (OCI image with built RPMs) as environment variables

FMF test plan

The root folder of the rpms repository is marked with a .fmf directory to identify it as an FMF metadata tree for tmt. The test plan in ci/run_tests_rpm.fmf sets up the test instance and runs default and package-specific tests via ci/run_tests_rpm.sh.

Test environment

These environment variables are available to the tmt test execution:

Stage 4: Signing

The Konflux Release Service signs built RPMs before publishing:

• Method: Kerberos-based signing via konflux-release-signing-prod@IPA.REDHAT.COM
• Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
• Signing image: quay.io/konflux-ci/signing:latest (verify pinned digest in ReleasePlanAdmission)

Stage 5: Publishing to Pulp

After signing, RPMs are uploaded to the Hummingbird Pulp repository.

Release pipeline

The push-rpms-to-pulp release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (releng/hummingbird-rpms-tech-preview-staging.yaml):

mapping:
  rpm-repositories:
    - name: x86_64
      repository_id: public-hummingbird-x86_64-rpms
    - name: aarch64
      repository_id: public-hummingbird-aarch64-rpms
    - name: src
      repository_id: public-hummingbird-source-rpms
    # See ReleasePlanAdmission for the full list of published architectures

Pulp repository

Published RPMs are available in the Pulp content index:

Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.

What Happens Next

Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.

Related Documentation

• Rebuilding Packages - No-change rebuilds, reverse dependency rebuilds, and patch backports
• Updating Dist-git Packages - Automated Fedora sync workflow
• Package Metadata Fields - modification_status and release configuration
• Package Modification Tracking - Managing modification status and version constraints
• Konflux Resource Deployment - How Konflux resources are defined and deployed
• Image Pipeline - Container image pipeline (continues from where this document ends)