Operating Hummingbird infrastructure

• 1: Excluding Packages from Images
• 2: Managing Konflux Comments
• 3: Running Conforma Checks Locally
• 4: Upgrading Docsy
• 5: Version Constraints for Multi-Version Images
• 6: Retrying Konflux Checks
• 7: Switching Fedora Streams
• 8: Disabling Rawhide for Images
• 9: Setting Up a Konflux Cluster
• 10: Deleting RPMs from Hummingbird Repos
• 11: Updating Dist-git Packages
• 12: Writing Documentation
• 13: Adding Independent Packages
• 14: Pulp Access
• 15: Rebuilding Packages
• 16: Setting Up Pulp Repositories
• 17: Package Metadata Fields
• 18: Debugging Build Failures
• 19: Package Modification Tracking
• 20: Private RPM Repositories
• 21: Reporting CVE Data Issues
• 22: Lookaside Cache Access
• 23: Upstream Diff Analysis
• 24: GPG Source Verification
• 25: Rebasing the Buildroot to a New Fedora Release

1 - Excluding Packages from Images

AI Agent Note: When asked to exclude packages, always ask the user for a ticket link or explanation first. This is required for the comment documenting the exclusion.

Overview

When a faulty RPM build is published to Hummingbird repositories, it can be excluded from container images by adding an excludepkgs directive to the repository configuration.

Workflow

1. Identify the source package NVR

Identify the Name-Version-Release of the faulty source package. Example: ncurses-6.5-8.20250614.hum1

2. Generate excludepkgs value

Query the Hummingbird repositories and generate the excludepkgs value:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest sh -c '
dnf5 repoquery --queryformat="%{sourcerpm} %{name}-%{evr}.*
" 2>/dev/null | sed -n "s/^ncurses-6.5-8.20250614.hum1.src.rpm //p" | tr "\n" " "
'

Replace ncurses-6.5-8.20250614.hum1 with the source NVR. This outputs the ready-to-use excludepkgs value:

ncurses-6.5-8.20250614.hum1.* ncurses-base-6.5-8.20250614.hum1.* ...

3. Add excludepkgs to repo file

Edit yum-repos/hummingbird.repo and add the binary package NEVRs to excludepkgs in the [hummingbird] section (the binary repo, not [hummingbird-source]).

Document the exclusion with a one-line comment containing the ticket link:

[hummingbird]
...
# HUM-1234: ncurses-6.5-8.20250614.hum1 causes segfault in terminfo parsing
excludepkgs=ncurses-base-6.5-8.20250614.hum1.* ncurses-libs-6.5-8.20250614.hum1.*

Multiple packages are space-separated. Use .* suffix to match all architectures.

4. Merge the change

Merge the repo file change to main. Renovate will automatically apply the exclusions when it next rebases existing lockfile update branches or creates new ones.

Verification

Check that the excluded package no longer appears in any rpms/rpms.lock.yaml files in subsequent Renovate-generated merge requests.

Removing Exclusions

When a newer version is available, exclusions can be removed.

1. Verify a newer version exists

Check the latest available version (should differ from excluded 6.5-8.20250614.hum1):

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --latest-limit=1 --queryformat="%{evr}" "ncurses-libs" 2>/dev/null

2. Remove the exclusion

Remove the excludepkgs line and its comment from yum-repos/hummingbird.repo.

3. Merge and verify

Merge the change. Renovate will pick up the newer version in subsequent lockfile updates.

2 - Managing Konflux Comments

When Konflux comments become excessive on merge requests, you can clean them up using the provided comment deletion tool.

Prerequisites

• COM_GITLAB_TOKEN: GitLab API token with api scope and Maintainer access

Usage

If Konflux comments become excessive on merge requests, you can clean them up using:

# Show help for the comment deletion tool
make delete-konflux-comments ARGS="--help"

# Example: Delete comments from a specific merge request
make delete-konflux-comments ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete only failed build/test comments (dry run)
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete all Konflux comments from a merge request
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

Comment Categories

The tool can target specific types of Konflux comments:

• failed: Only comments related to failed builds/tests
• all: All Konflux-related comments (use with --all flag)

Safety Features

• Dry Run Mode: Use --dry-run to preview what would be deleted without actually removing comments
• Selective Deletion: Target specific comment categories to avoid removing important information
• Confirmation: The tool will show what it plans to delete before taking action

Examples

# Set your GitLab token
export COM_GITLAB_TOKEN=your_token_here

# Preview what failed comments would be deleted
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete only failed build comments
make delete-konflux-comments ARGS="--comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete all Konflux comments (use with caution)
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

When to Use

Consider cleaning up Konflux comments when:

• Merge requests have accumulated many outdated failure comments
• Comments are making it difficult to find relevant discussion
• You want to start fresh after resolving systematic issues
• The comment thread has become cluttered with automated messages

Note: Use this tool judiciously, as it permanently removes comments that might contain useful debugging information.

Related Operations

• Retrying Konflux Checks - Retrigger failed CI pipeline runs

3 - Running Conforma Checks Locally

Overview

Conforma (Enterprise Contract) policy checks run automatically in the Konflux pipeline before release (see Image Pipeline — Stage 5). These checks can also be run locally against any Konflux-built image to validate compliance, test policy changes, or investigate failures.

Prerequisites

Install the Conforma CLI:

curl -sLO https://github.com/conforma/cli/releases/download/snapshot/ec_linux_amd64
chmod 755 ec_linux_amd64
mkdir -p ~/.local/bin
mv ec_linux_amd64 ~/.local/bin/ec

Running All Checks

The full set of Conforma checks (signatures, attestations, SLSA provenance, labels, etc.) can be run against any Konflux-built image using ec validate image with the Konflux signing key.

Public Key

The Konflux signing public key is committed at ci/key.pub. This is the same key stored in the Konflux member cluster at k8s://openshift-pipelines/public-key.

Obtaining the Image Reference

Combine the IMAGE_URL and IMAGE_DIGEST results from a successful build PipelineRun:

<IMAGE_URL>@<IMAGE_DIGEST>

Merge request build images follow the pattern:

quay.io/redhat-user-workloads/hummingbird-tenant/<component>:on-mr-<MR_ID>-<COMMIT_SHA>

Running the Validation

Create a policy file that matches the pipeline policy but with any modifications needed for testing. For example, to run the pipeline policy with label checks enabled (removing the labels.required_labels and labels.optional_labels exclusions):

cat > /tmp/policy.yaml << 'YAML'
sources:
  - name: Release Policies
    config:
      exclude:
        - test.required_tests_passed:sast-snyk-check
        - test.no_skipped_tests:sast-snyk-check
        - test.required_tests_passed:sast-snyk-check-oci-ta
        - test.no_skipped_tests:sast-snyk-check-oci-ta
        - test.no_failed_tests:ecosystem-cert-preflight-checks
        - test.no_erred_tests:ecosystem-cert-preflight-checks
        - test.no_failed_informative_tests
        - test.no_test_warnings:deprecated-image-check
        - trusted_task.current
        - rpm_repos.ids_known
        - buildah_build_task.privileged_nested_param
        - schedule.weekday_restriction
      include:
        - '@redhat'
    data:
      - github.com/release-engineering/rhtap-ec-policy//data
      - oci::quay.io/konflux-ci/tekton-catalog/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/konflux-vanguard/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/integration-service-catalog/data-acceptable-bundles:latest
    policy:
      - oci::quay.io/enterprise-contract/ec-release-policy:konflux
    ruleData:
      allowed_registry_prefixes:
        - "quay.io/hummingbird-rawhide/"
        - "quay.io/hummingbird-ci/"
        - "quay.io/hummingbird-community/"
        - "quay.io/hummingbird/"
        - "quay.io/redhat-user-workloads/"
YAML

Then validate:

ec validate image \
    --image "quay.io/redhat-user-workloads/hummingbird-tenant/<component>@sha256:<digest>" \
    --policy /tmp/policy.yaml \
    --public-key ci/key.pub \
    --ignore-rekor \
    --strict=false \
    --show-successes \
    --output text

Options

Running Label Checks Only

The label checks (labels.required_labels, labels.optional_labels) are the only Conforma rules that can be evaluated without signatures or attestations, because they only read OCI image metadata. This lighter-weight approach does not require the signing key and works against any registry image, including published (non-Konflux-built) images.

Setup

Fetch the policy rego files and upstream rule data (one-time):

ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

git clone --depth=1 \
    https://github.com/release-engineering/rhtap-ec-policy.git \
    /tmp/rhtap-ec-policy

Running Checks

ec opa eval \
    --data /tmp/ec-policy/policy/*/policy \
    --data /tmp/rhtap-ec-policy/data/rule_data.yml \
    --input <(echo '{"image":{"ref":"quay.io/hummingbird-rawhide/caddy:latest"}}') \
    '{"deny": data.labels.deny, "warn": data.labels.warn}' \
    --format pretty

This works against any image accessible from a container registry — both Konflux-built and published images. It uses the ec CLI’s embedded OPA engine with ec.oci.* built-in functions to read image config directly from the registry.

Note: The --data flag (not --bundle) must be used for the rego files. Loading the policy as a bundle creates an OPA root conflict that prevents the rule data from being resolved. The ec.oci.* built-in functions are only available in ec opa, not in standalone opa or conftest.

Interpreting Results

For ec validate image:

• Successes indicate passing policy rules (visible with --show-successes).
• Violations are failures that block release. Each includes a code (rule name) and msg.
• Warnings are non-blocking advisories.

For ec opa eval (label checks only):

• deny entries are failures. Each includes code, msg, and optionally effective_on (a date when the rule becomes enforced — future dates mean the rule is not yet active).
• warn entries are non-blocking warnings for optional labels.
• Empty arrays mean the image passes all label checks.

Updating Cached Data

The fetched policy and rule data are point-in-time snapshots. To pick up upstream changes:

# Re-fetch the policy rego files (for ec opa eval)
rm -rf /tmp/ec-policy
ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

# Update the rule data (for ec opa eval)
git -C /tmp/rhtap-ec-policy pull

The ec validate image command fetches policy and data bundles on each run based on the policy.yaml references, so no manual update is needed.

4 - Upgrading Docsy

Overview

Both Hummingbird documentation sites use Docsy as their Hugo theme, installed as a Hugo module. Docsy releases roughly quarterly. This runbook covers how to upgrade to a new Docsy version.

The public docs (documentation) own all layout overrides. The internal docs (infrastructure/internal-docs/) only have build scaffolding and pick up layouts from the public docs via Hugo module import.

Layout override inventory

We override three Docsy templates to inject the companion page banner and sidebar lock icons. Each override is a copy of the upstream template with a small patch applied.

layouts/docs/_td-content.html

Overrides Docsy’s shared layouts/_td-content.html for docs pages only.

Patch: One line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/docs/list.html

Overrides Docsy’s layouts/docs/list.html for docs section pages.

Patch: Same one line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/partials/sidebar-tree.html

Overrides Docsy’s layouts/_partials/sidebar-tree.html for the navigation sidebar.

Patch: One snippet inserted in two places (foldable and non-foldable branches), after the {{ $s.LinkTitle }} </span> and before </a>:

{{- if isset $s.Params "companion" }}{{ if not $s.Params.companion }} <i class="fa-solid fa-lock companion-icon internal-only" title="Internal only"></i>{{ end }}{{ end -}}

Other overrides (no forward-porting needed)

These files are fully custom and don’t track an upstream template:

Upgrade procedure

1. Read the release notes

Check the Docsy changelog and the release upgrade guide for the target version. Note any breaking changes that affect templates we override.

2. Update the Hugo module

From the documentation repo worktree:

# Update to new version (replace vX.Y.Z with the target version)
podman run --pull=newer -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy"

3. Update npm dependencies

Docsy v0.16.0+ sources Bootstrap and Font Awesome from npm:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod npm pack && npm install --no-package-lock"

4. Forward-port layout overrides

For each of the three tracked overrides:

1. Find the new upstream template in the Hugo module cache:

   find .hugo_cache -path "*docsy/theme@vX.Y.Z*" \
     -name "_td-content.html" -o \
     -name "sidebar-tree.html" -o \
     \( -name "list.html" -path "*docs*" \)
   

2. Compare the new upstream against our current override to see what changed upstream.

3. Start from the new upstream template and re-apply the patch documented in the inventory above.

4. For docs/list.html, keep all upstream hooks (especially {{ .Render "_td-content-after-header" -}}).

5. Verify dark mode and code highlighting

The site uses Docsy’s light/dark mode toggle and dark-aware code syntax highlighting. After upgrading, confirm these config and SCSS settings are still in place:

• config.yaml: params.ui.showLightDarkModeMenu: true
• config.yaml: markup.highlight.noClasses: false (required for CSS-based Chroma themes to work)
• assets/scss/_styles_project.scss: @import 'td/code-dark'; (loads light/dark Chroma themes: friendly for light, native for dark)

If a Docsy upgrade changes the dark mode mechanism, check the Look and Feel docs for updated instructions.

6. Build and validate

make build  # or: make build-host (if Hugo/Go are available locally)

The build should complete with no errors and no deprecation warnings.

Validation checklist

After upgrading, verify:

• Build passes without errors or deprecation warnings
• Companion banners render on docs pages with companion: true front matter
• Lock icons appear in the internal site sidebar for internal-only pages
• Home page cover image and search render correctly
• Light/dark mode toggle works and all page elements adapt (navbar, sidebar, footer, code blocks)
• Code blocks have dark background with readable syntax colors in dark mode
• Agent-support outputs generate, if enabled (.md URL variants, /llms.txt)
• Stable URL aliases (/l/...) still resolve

For the internal site, also build infrastructure/internal-docs/ after updating its config/internal/config.yaml and go.mod to the same Docsy version.

Updating the internal site

The internal site only needs config and module changes (no layout work):

1. Update internal-docs/config/internal/config.yaml: change the theme: list to match the public site.

2. Update the Hugo module and rebuild:

   cd internal-docs
   make container
   # Inside container:
   hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy
   exit
   make setup  # npm dependencies
   make build
   

Rollback

To revert to a previous Docsy version:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vPREVIOUS && hugo mod tidy"

Then restore the layout overrides from the previous commit (git checkout HEAD~1 -- layouts/).

5 - Version Constraints for Multi-Version Images

Overview

Version constraints prevent unwanted major or minor version updates for multi-version container images (e.g., python-3-11, nodejs-20). Constraints are defined in properties.yml and validated during make check by comparing VERSION files against defined patterns.

Adding Version Constraints

Edit the image’s properties.yml to add a version constraint:

---
main_package: python3.11
version_constraints: '3.11.*'  # Allow 3.11.X, block 3.12+ and 4.X

The constraint applies to all distros for the image:

---
main_package: dotnet-sdk-8.0
version_constraints: '8.*'  # Applies to rawhide, hummingbird, and any other distros

Constraint Patterns

Constraints use glob-style patterns:

• 3.11.* - Match any 3.11.X version (recommended for version families)
• 20.* - Match any 20.X version
• 8.* - Match any 8.X version
• 1.25.* - Match any 1.25.X version (for Go-style 3-part versions)

The pattern matches against the version from the VERSION file. For example, if the VERSION file contains 3.11.14, the constraint 3.11.* will match.

How Validation Works

1. Lockfile Generation: When make all runs, lockfiles are generated with resolved package versions
2. VERSION File Creation: generate_jinja2.py creates VERSION files from resolved package versions
3. Constraint Checking: When make check runs, check_version_constraints.py validates VERSION files against constraints
4. Failure: If a violation is detected, the build fails with a detailed error message
5. Renovate Integration: Renovate MRs that violate constraints fail CI and are blocked from automerge

Handling Constraint Violations

When a build fails due to a version constraint violation, you have several options:

Option 1: Accept the new version (create new image)

If the new major/minor version is acceptable:

1. Create a new image directory (e.g., images/python-3-14/)

2. Copy properties and configuration from the old version

3. Update the constraint in the new directory:

   version_constraints: '3.14.*'
   

4. Update repository field to group versions together

5. Consider adding latest tag to the new version if appropriate

Option 2: Block the version update (exclude package)

If the new version should be blocked entirely:

1. Add the package to yum-repos/*.repo excludepkgs
2. Follow the procedure in Excluding Packages from Images
3. Renovate will skip the excluded version in future updates
4. When the issue is resolved, remove the exclusion

Option 3: Update the constraint (allow version bump)

If the version bump is acceptable for the existing image:

1. Update the constraint in properties.yml:

   version_constraints: '3.12.*'  # Allow 3.12.X now
   

2. Consider if this changes the image’s purpose (major version change)

3. Update image documentation and tags accordingly

4. Consider renaming the image directory to reflect the new version

Examples

Python Multi-Version Images

# images/python-3-11/properties.yml
distros:
  - hummingbird
main_package: python3.11
repository: python
version_constraints: '3.11.*'

# images/python-3-13/properties.yml
distros:
  - hummingbird
main_package: python3.13
repository: python
version_constraints: '3.13.*'

Node.js Multi-Version Images

# images/nodejs-20/properties.yml
main_package: nodejs20
repository: nodejs
version_constraints: '20.*'

# images/nodejs-24/properties.yml
main_package: nodejs24
repository: nodejs
version_constraints: '24.*'

.NET SDK Multi-Version Images

# images/dotnet-sdk-8-0/properties.yml
main_package: dotnet-sdk-8.0
repository: dotnet-sdk
version_constraints: '8.*'

.NET Runtime Multi-Version Images

# images/dotnet-runtime-8-0/properties.yml
main_package: dotnet-runtime-8.0
repository: dotnet-runtime
version_constraints: '8.*'

ASP.NET Runtime Multi-Version Images

# images/aspnet-runtime-8-0/properties.yml
main_package: aspnetcore-runtime-8.0
repository: aspnet-runtime
version_constraints: '8.*'

OpenJDK Multi-Version Images

# images/openjdk-21/properties.yml
main_package: java-21-openjdk-headless
repository: openjdk
version_constraints: '21.*'

Go Multi-Version Images (3-part versions)

# images/go-1-25/properties.yml
main_package: golang1.25
repository: go
distros:
  - hummingbird
version_constraints: '1.25.*'  # Matches 1.25.0, 1.25.3, etc.

Troubleshooting

Constraint not working

• Verify package name matches exactly (case-sensitive)
• Ensure pattern syntax is correct (use * wildcard)
• Run make all to regenerate VERSION files before make check

False positive violations

• Check for typos in the constraint pattern
• Verify the pattern matches the version format (e.g., 3.11.* not 3.11*)
• Ensure you’re using glob patterns, not regex

Package renamed in repositories

If a package is renamed (e.g., python3.11 → python311):

1. Update main_package field
2. Update constraint with new package name (not needed - constraints check VERSION files)
3. Regenerate lockfiles and VERSION files with make all

Implementation Details

The version constraint system consists of:

1. properties.yml: Optional version_constraints field with a glob pattern string
2. ci/check_version_constraints.py: Python script that validates VERSION files
3. Makefile integration: Script runs as part of make check
4. CI integration: Failures block merge, including Renovate automerge

The validation process:

1. Reads properties.yml for each image
2. If version_constraints defined, finds all VERSION files across all distros
3. Applies the same constraint pattern to all distros for the image
4. Compares VERSION file content against glob pattern using Python’s fnmatch
5. Reports violations with file path, version, constraint, and properties file location
6. Exits with error code 1 if any violations found

See Also

• Image Configuration Reference - Complete properties.yml field reference
• Excluding Packages from Images - How to exclude problematic package versions
• Switching Fedora Streams - How to pin distro versions

6 - Retrying Konflux Checks

When Konflux CI checks fail on merge requests, you can automatically retrigger them using the provided retrigger script.

Prerequisites

• GITLAB_TOKEN: GitLab API access token with api scope and Maintainer access

Usage

For automated retriggers of failed Konflux pipeline runs:

# Retrigger only failed pipeline runs and wait for them to start
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Preview what would be retriggered without posting comments
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

How It Works

The retrigger script will:

1. Extract failed pipeline runs from the commit statuses on the merge request
2. Generate /retest pipeline-name commands for each failed run
3. Post the retest commands as merge request comments (unless --dry-run)
4. Wait for the pipeline runs to start (unless --dry-run)

Manual Retrigger Commands

You can also manually retrigger specific checks using slash commands in merge request comments:

/retest gitlab-ci--default--main-on-pull-request

To retrigger a specific pipeline run as listed in .tekton/images-on-pull-request.yaml.

See the Tekton Pipelines as Code documentation for more details on available commands.

Examples

# Set your GitLab token
export GITLAB_TOKEN=your_token_here

# Retrigger failed checks for a specific merge request
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

# Dry run to see what would be retriggered
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

The script will output which pipeline runs are being retriggered and wait for them to start before completing.

Related Operations

• Managing Konflux Comments - Clean up excessive comments on merge requests

7 - Switching Fedora Streams

AI Agent Note: When asked to switch Fedora streams, follow the workflow below. The repo file naming convention determines the GPG key automatically.

Overview

The rawhide distro can be pinned to a specific Fedora release branch instead of following the rolling Rawhide stream. This provides stability during the period when Rawhide moves to N+1 but the current stable release is still N-1.

For example, when Fedora 44 branches from Rawhide, the rawhide distro can be pinned to Fedora 44 packages while Rawhide itself starts tracking Fedora 45.

Repo ID naming convention

All Fedora repo files use a fedora-<stream> naming convention for their repo IDs, where <stream> matches the filename stem:

• fedora-43.repo uses [fedora-43], [fedora-43-source], [fedora-43-updates], etc.
• fedora-44.repo uses [fedora-44], [fedora-44-source], etc.
• fedora-rawhide.repo uses [fedora-rawhide], [fedora-rawhide-source], etc.

This convention ensures:

• No DNF repo ID conflicts when multiple repo files coexist in yum-repos/
• Predictable, greppable repo IDs in lockfiles
• The GPG key name can be derived automatically from the filename

Workflow

1. Create or update the repo file

Create a new file yum-repos/fedora-<version>.repo (e.g., fedora-45.repo).

The repo file must follow these conventions:

• Repo IDs match the filename stem: Use [fedora-45], [fedora-45-debuginfo], and [fedora-45-source] as section names
• Version-specific URLs: Point to the correct Fedora version’s repository (e.g., /development/45/ or /releases/45/)
• Version-specific GPG key: Reference the correct signing key (e.g., RPM-GPG-KEY-fedora-45-$basearch)

Use an existing repo file (e.g., yum-repos/fedora-44.repo) as a template.

Example for a development branch:

[fedora-45]
name=Fedora 45 - Developmental packages for the next Fedora release
baseurl=https://koji-s3-cache.hummingbird-project.io/download-ib01.fedoraproject.org/pub/fedora/linux/development/45/Everything/$basearch/os/
enabled=1
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-45-$basearch
skip_if_unavailable=False

2. Update variables.yml

Edit images/variables.yml and change the default_variant_repos.rawhide entry to reference the new repo file:

default_variant_repos:
  rawhide:
    - fedora-45.repo

3. Regenerate all derived files

Run regeneration to update rpms.in.yaml files, Containerfiles, and lockfiles:

make -j 16 FORCE=true

4. Verify GPG key derivation

The Jinja templates in macros/setup_newroot.yml.j2 automatically derive the GPG key name from the repo filename:

• fedora-45.repo becomes RPM-GPG-KEY-fedora-45-primary
• fedora-44.repo becomes RPM-GPG-KEY-fedora-44-primary
• fedora-rawhide.repo becomes RPM-GPG-KEY-fedora-rawhide-primary

No template changes are needed when switching versions. Verify by checking the generated Containerfile for the builder image contains the correct key:

grep rpmkeys images/hummingbird-builder/rawhide/default/Containerfile

Switching back to rolling Rawhide

To switch back to the rolling Rawhide stream, update variables.yml to reference fedora-rawhide.repo:

default_variant_repos:
  rawhide:
    - fedora-rawhide.repo

Then regenerate all derived files with make -j 16 FORCE=true.

8 - Disabling Rawhide for Images

AI Agent Note: When asked to disable Rawhide for images, follow the workflow below. Run make all after editing properties to clean up stale directories automatically.

Overview

When Rawhide is unstable (mass rebuild, broken critical packages, Fedora branching), Rawhide variants can be temporarily disabled for specific images or globally. This stops building and testing Rawhide variants while keeping Hummingbird variants unaffected.

Disabling Rawhide for a single image

1. Edit properties.yml

Add an explicit distros field to images/<name>/properties.yml that excludes rawhide:

distros:
  - hummingbird

This overrides the default distros (which include rawhide) for this image only.

2. Regenerate

make all

The stale Rawhide directories are removed automatically. Generated resources (.gitattributes, Konflux templates, Tekton pipelines) are updated to exclude the Rawhide variants.

3. Commit and merge

Commit the properties change together with the removed generated files.

Disabling Rawhide for all images

1. Edit variables.yml

Remove rawhide from default_distros in images/variables.yml:

default_distros:
  - hummingbird

2. Edit per-image overrides

Images that explicitly list distros including rawhide in their own properties.yml are not affected by the variables.yml change. Edit those files to remove rawhide as well.

3. Regenerate

make all

4. Commit and merge

Commit all properties changes together with the removed generated files.

Re-enabling Rawhide

Reverse the properties change (add rawhide back to default_distros or remove the per-image distros override), then regenerate:

make all

The Rawhide directories no longer exist, so Make creates all targets from scratch.

Verification

After running make all:

• git status shows Rawhide directories as deleted (when disabling) or new files (when re-enabling)
• .gitattributes no longer lists Rawhide paths for affected images
• konflux-templates/rendered.yml and .tekton/ pipelines no longer reference Rawhide variants for affected images

See Also

• Image Configuration Reference - distros field documentation
• Switching Fedora Streams - Pin Rawhide to a specific Fedora release branch

9 - Setting Up a Konflux Cluster

Setting Up the Cluster

1. Set up the new cluster in the infrastructure repository, mirroring the existing cluster configuration
2. In the Konflux UI, create a new Component for the Application with the same name as the git repository
3. Verify the Pipeline as Code Repository resource is created
4. Run two successive full pipelines in the infrastructure repository to deploy resources from konflux/rendered.yml and fix up owner references
5. Delete the initial Component created from the UI
6. Disable Konflux build status comments (requires elevated permissions):

kubectl patch \
  --namespace hummingbird-tenant \
  repository REPOSITORY_NAME \
  --type merge \
  --patch '{"spec":{"settings":{"gitlab":{"comment_strategy":"disable_all"}}}}'

Next Steps

• Retrying Konflux Checks - Retrigger failed CI pipeline runs
• Managing Konflux Comments - Clean up excessive comments

10 - Deleting RPMs from Hummingbird Repos

Overview

• This document should be used to remove RPMs from the Hummingbird repositories

Setup

• make sure you have followed the local development setup instructions in the infrastructure repository
• install pulp cli via dnf install pulp-cli
• setup the local pulp environment via:

cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

Steps

• Note the names of the rpms to delete. In this example, we’ll use systemd-stub
• Obtain the relevant metadata required to delete.
  • You need to execute this command once for each architecture.

pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.pulp_href)"'

or for prettier content,

pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.name) \(.version)-\(.release)  \(.arch)  \(.pulp_created)  \(.pulp_href)"'

Example output from basic command:

/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/

• Using the hrefs, construct your delete commands:

pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/

• Repeat for x64_64 and source repositories
• Verify deletion by navigating to https://console.redhat.com/api/pulp-content/public-hummingbird/
• Deletion is a serial operation that can be impacted by other automated jobs updating the repositories.

Related Operations

• Setting Up Pulp Repositories

11 - Updating Dist-git Packages

Packages are automatically updated from Fedora dist-git. Each update creates a separate MR that is automatically approved and merged when CI passes.

Testing Locally

# Dry-run (check all packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone

# Dry-run (check first 5 packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone --max-packages=5

# Check only a specific package (for testing/debugging)
./ci/dist_git_update_multi_mr.sh --clone --only-package=libgcrypt

# Check only clean packages (skip modified/independent)
./ci/dist_git_update_multi_mr.sh --clone --clean-only

# Check only modified packages (skip clean/independent)
./ci/dist_git_update_multi_mr.sh --clone --modified-only

# Create up to 3 test MRs (checks all packages, stops after finding 3 updates)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-updates=3 --create-mrs

# Check first 10 packages, create up to 3 MRs
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-packages=10 --max-updates=3 --create-mrs

# Create MRs only for clean packages (skip modified/independent)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --clean-only --create-mrs

Environment Variables

• CHORE_MR_GITLAB_TOKEN - GitLab token with write_repository scope (required for --create-mrs)
• CHORE_MR_APPROVAL_GITLAB_TOKEN - GitLab token with api scope for auto-approving MRs (used by CI)
• GITLAB_REMOTE_URL - Target repo (default: https://gitlab.com/redhat/hummingbird/rpms.git)

Flags

• --clone - Clone from GitLab to /tmp (safe for local testing, uses latest main branch)
• --max-packages=N - Check only the first N packages (limits input set)
• --max-updates=N - Stop after finding N updates (limits output MRs created)
• --create-mrs - Actually create MRs (requires token)
• --only-package=NAME - Check only the specified package (for testing/debugging specific packages)
• --clean-only - Skip packages with modification_status of ‘modified’ or ‘independent’, only process clean packages
• --modified-only - Skip packages with modification_status of ‘clean’ or ‘independent’, only process modified packages (mutually exclusive with --clean-only)

Using –clean-only

The --clean-only flag filters out packages marked as ‘modified’ or ‘independent’ before attempting updates. This is useful for:

1. Better failure detection - Exit code 1 indicates real update failures, not expected errors from modified/independent packages
2. Cleaner output - No error messages for packages that can’t be auto-updated by design
3. Efficient CI - Focus on packages that should update automatically
4. Performance - Avoids invoking dist_git.py for packages that will fail

Without --clean-only, the script attempts to update all packages. Modified/independent packages fail with:

ERROR: Cannot auto-update <package>
       Status: modified/independent
       Reason: <reason>
       Use 'sync' to force update or 'mark-modified --clean' to allow updates

These expected failures can mask genuine update issues. Using --clean-only prevents these false failures.

Using –modified-only

The --modified-only flag filters to only process packages marked as ‘modified’, skipping clean and independent packages. This is useful for checking the merge logic, as well as getting an overview of current merge conflicts.

Auto-Merge and Auto-Approval

MRs created by this script are configured to:

• Auto-merge when pipeline succeeds (set via merge_request.merge_when_pipeline_succeeds)
• Auto-approve after 10 minutes via the chore_mr_approval CI job (gives Konflux time to post commit statuses)

Version-Constrained Updates

Packages with track_upstream set to a version prefix in their metadata are version-constrained. This is used for versioned packages like golang1.26 that track a specific upstream version line.

How It Works

The track_upstream and release_monitoring_project_id metadata fields work together across two systems:

dist_git.py update (dist-git sync from Fedora):

When track_upstream is a version prefix (e.g., "1.26"), the update command uses prefix matching:

• 1.26 matches 1.26, 1.26.0, 1.26.3 (allowed)
• 1.26 does not match 1.27.0, 2.0 (skipped)

Skipped packages log a warning:

WARNING: Skipping golang1.26: upstream version 1.27.0 doesn't match tracked version 1.26

check_upstream_versions.py (release-monitoring.org checks):

When release_monitoring_project_id is a string, Anitya is queried using that name instead of the RPM package name. For example, golang1.26 with release_monitoring_project_id: "golang" queries Anitya for golang. When it is an integer, the v2 API is queried directly by project ID. When track_upstream is a version prefix, the list of upstream versions returned by Anitya is filtered to only those matching the prefix. This means check_upstream_versions.py check will report 1.26.5 as the latest version for golang1.26 even if Anitya reports 1.27.0 as the latest golang release.

When --update is used, the script updates the spec file, downloads new sources, and commits the result. Packages that need custom update logic can provide a hooks file at metadata/<package>.update-hooks.yaml to override the spec update, source download, or add a post-update step. See Package Modification Tracking for details.

Setting Up Version Constraints

# Constrain golang1.26 to only receive 1.26.x updates
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Remove the version constraint
./ci/dist_git.py set-upstream golang1.26 --no-track-version

Behavior

• dist_git.py: Version constraint is checked before pre-release and Koji build checks
• dist_git.py: The sync command bypasses the version constraint (explicit force operation)
• check_upstream_versions.py: release_monitoring_project_id determines the Anitya lookup (int for project ID, string for name, absent for RPM name); track_upstream filters versions when set to a prefix
• Batch operations continue processing other packages after skipping constrained ones

Pre-Release Version Filtering

By default, the update mechanism skips pre-release versions to prevent unstable packages from entering the repository automatically. Pre-release patterns include:

• Tilde notation: 5.3.0~rc1, 2.0~beta1, 1.0~alpha (RPM standard)
• Suffix notation: 5.3.0-rc1, 2.0.beta1, 3.0-alpha, 1.0.dev
• Development markers: 1.5git20240101, 2024.01.snapshot, 1.0dev

Manual Override

To explicitly update to a pre-release version:

# Update single package to pre-release version
./ci/dist_git.py update --allow-prerelease package-name --skip-build-check

# Batch update allowing pre-releases
./ci/dist_git.py update --allow-prerelease --skip-build-check

Pre-release filtering behavior

• Pre-release detection occurs before Koji build checks (saves API calls)
• Skipped packages log a warning with the detected pattern
• Batch updates continue processing other packages
• The sync command bypasses this check (explicit force operation)

12 - Writing Documentation

Overview

New documentation is always welcome! This guide explains how to contribute to the Project Hummingbird documentation.

Documentation Sites

The Project Hummingbird documentation is open by default and available at two sites:

• Public documentation: https://hummingbird-project.io/
• Internal documentation: https://internal-documentation.hummingbird-project.io/

Documentation Repositories

Documentation content comes from six repositories:

1. Containers repository - Documentation focused on container images (cross-cutting layout):

   • documentation/contributing/ - How to add and modify container images
   • documentation/background/ - Container image architecture and reference docs
   • documentation/ci-scripts/ - CI script documentation
   • documentation/operating/ - Operator runbooks for image maintenance
   • documentation/using/ - End-user guides (custom CA certificates, etc.)
   • README.md - User guide
   • CONTRIBUTING.md - Quickstart guide

2. RPMs repository - Documentation focused on RPM packages (cross-cutting layout):

   • documentation/background/ - RPM pipeline architecture and Konflux deployment
   • documentation/operating/ - Operator runbooks for package management
   • CONTRIBUTING.md - Contributing guide for RPM packages

3. Tools repository - Documentation for infrastructure tools and services (per-component layout):

   • documentation/ - One page per tool/service (flat structure)

4. K8s test pipeline repository - Documentation for Kubernetes integration testing (per-component layout):

   • documentation/ - Pipeline design, test format, and EaaS debugging

5. Public documentation repository - Site infrastructure and cross-cutting guides:

   • content/docs/using/ - End-user guides for using container images
   • content/docs/operating/ - Infrastructure operation guides (Konflux, GitLab, etc.)
   • content/docs/background/ - General architecture and concepts

6. Infrastructure repository - Internal-only documentation (cross-cutting layout):

   • documentation/operating/ - Internal operational docs (AWS, DNS, GitLab, etc.)
   • documentation/background/ - Internal background docs (error budgets, etc.)
   • documentation/presentations/ - Team presentations and talks

How They Work Together

The repositories are layered using Hugo Modules:

1. Public documentation imports specific directories from the containers, rpms, tools, and k8s-test-pipeline repositories
2. Internal documentation is built from the infrastructure repository’s internal-docs/ subtree. It imports the public documentation (which transitively includes the imported repository documentation) and mounts the infrastructure repo’s documentation/ directory as the Internal section

Hugo Module Mounts:

# In documentation/config.yaml
module:
  imports:
    - path: gitlab.com/redhat/hummingbird/containers
      mounts:
        - {source: 'README.md', target: content/snippets/readme.md}
        - {source: 'CONTRIBUTING.md', target: content/snippets/contributing.md}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/background', target: content/docs/background/containers, files: "*.md"}
        - {source: 'documentation/ci-scripts', target: content/docs/background/containers/ci-scripts, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/rpms
      mounts:
        - {source: 'CONTRIBUTING.md', target: content/snippets/rpms-contributing.md}
        - {source: 'documentation/background', target: content/docs/background/rpms, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/tools
      mounts:
        - {source: 'documentation', target: content/docs/background/tools, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
      mounts:
        - {source: 'documentation', target: content/docs/background/k8s-test-pipeline, files: "*.md"}

For README.md and CONTRIBUTING.md, front matter is provided via wrapper files.

This creates a unified documentation site from multiple repositories without duplication.

Documentation Layout Patterns

Source repositories use one of two layout patterns depending on their content:

Cross-cutting layout (containers, rpms): The repository documents cross-cutting concerns like contributing, operating, and background information. Documentation is organized into subdirectories matching the site sections:

• documentation/background/ - Explanation and reference material
• documentation/contributing/ - How to contribute
• documentation/operating/ - Operator runbooks
• documentation/using/ - End-user guides

Each subdirectory is mounted into the matching site section, so operating docs from different repositories appear together under Operating.

Per-component layout (tools, k8s-test-pipeline): The repository is a monorepo where each documentation page covers one tool or component. Documentation is a flat directory with one file per component:

• documentation/<component-name>.md

The entire directory is mounted under background/<repo-name>/ on the site.

Getting Started

Quick Start

Clone and start the documentation server:

git clone https://gitlab.com/redhat/hummingbird/documentation
cd documentation
make serve  # Available at http://localhost:1313/

For the internal documentation:

git clone https://gitlab.com/redhat/hummingbird/infrastructure
cd infrastructure/internal-docs
make serve  # Available at http://localhost:1314/

Recommended Setup

For the best development experience, set up all repositories with direnv:

mkdir -p ~/git/hummingbird
cd ~/git/hummingbird

# Clone all repositories
git clone https://gitlab.com/redhat/hummingbird/containers
git clone https://gitlab.com/redhat/hummingbird/rpms
git clone https://gitlab.com/redhat/hummingbird/tools
git clone https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
git clone https://gitlab.com/redhat/hummingbird/documentation
git clone https://gitlab.com/redhat/hummingbird/infrastructure

# Set up environment variables
cat > .envrc << 'EOF'
export CONTAINERS_REPO_PATH=$(pwd)/containers
export RPMS_REPO_PATH=$(pwd)/rpms
export TOOLS_REPO_PATH=$(pwd)/tools
export K8S_TEST_PIPELINE_REPO_PATH=$(pwd)/k8s-test-pipeline
export DOCUMENTATION_REPO_PATH=$(pwd)/documentation
export INFRASTRUCTURE_REPO_PATH=$(pwd)/infrastructure
EOF
direnv allow

This automatically uses your local repositories for imported content, making changes immediately visible in the documentation preview.

Adding Documentation

Add new documentation as markdown files inside the appropriate directory under content/. Every file should end in .md.

Preview your changes:

make serve  # Starts local server with live reload

Check for problems:

make check  # Runs linters and validation

Link Format

Documentation uses two types of internal links depending on the context:

Same-Directory Links and Cross-Section Links in the Documentation Repository

Use relative .md links for pages in the same directory, and for cross-section links in the documentation repository:

[Testing Guide](testing-images.md)
[Adding Images](adding-images.md)

Cross-Section and Cross-Repository Links

Use full URLs with /l/ aliases for links across repositories or across sections in all repositories:

[Image Pipeline][image-pipeline]
[Global Variables][global-vars]
[Retrying Checks][retrying-checks]

[image-pipeline]: https://hummingbird-project.io/l/image-pipeline
[global-vars]: https://hummingbird-project.io/l/global-variables-reference
[retrying-checks]: https://hummingbird-project.io/l/retrying-konflux-checks

Why full URLs?

• Work in standalone contexts (GitHub, text editors, when files are copied)
• Hugo’s link render hook automatically converts them to relative links in the rendered site
• Stable even when files are moved or reorganized
• Clear and explicit about which page is being referenced

Link Render Hook

The documentation site includes a custom link render hook (layouts/_default/_markup/render-link.html) that:

1. Detects https://hummingbird-project.io/l/... URLs
2. Looks up the page with the matching alias
3. Converts to a relative link in the rendered HTML
4. Keeps external links as-is with target="_blank"

This provides the best of both worlds: full URLs in markdown source files that work everywhere, and optimized relative links in the rendered site.

Stable Link Aliases

All cross-referenced pages should have stable /l/ aliases. These aliases serve two purposes:

1. Incoming links from external sources (documentation, scripts, merge request comments)
2. Cross-references within documentation (across sections or repositories)

Adding an Alias

Add an alias in the page front matter:

---
title: "Konflux Integration"
aliases: [/l/konflux-integration]
---

Now you can link to https://hummingbird-project.io/l/konflux-integration and it will continue working even if the page moves.

When to Add Aliases

Add /l/ aliases to pages that:

• Are referenced from other repositories (e.g., containers repo → docs repo)
• Are linked to from CI scripts or infrastructure
• Are shared in merge request comments or external documentation
• Are referenced across documentation sections

Naming Convention

• Use /l/ prefix for all stable links
• Keep names short, memorable, and descriptive
• Use kebab-case (lowercase with hyphens)
• No nested paths (flat structure only)

Submitting Changes

1. Fork the repository on GitLab
2. Clone your fork locally
3. Create a new branch for your changes
4. Make your changes and test them with make serve
5. Run make check to ensure there are no errors
6. Commit your changes with a descriptive commit message
7. Push to your fork
8. Create a merge request against the main repository

Documentation Structure

The public documentation is organized into several main sections:

• Using - Guides for users of Project Hummingbird container images
• Contributing - Guides for contributors to the project
• Operating - Guides for infrastructure operators
• Background - Technical background and architecture information

The internal documentation adds:

• Internal - Internal-only infrastructure documentation, organized into:
  • Operating - AWS, DNS, GitLab, OpenShift, and other service runbooks
  • Background - Error budgets, SLIs/SLOs
  • Presentations - Team talks and lightning talks

When adding new documentation, place it in the most appropriate section.

13 - Adding Independent Packages

Independent packages are packages that originate in the Hummingbird repository rather than being imported from Fedora dist-git.

Steps to Add an Independent Package

1. Create Package Directory

mkdir rpms/<package-name>
cd rpms/<package-name>

2. Add Package Files

Create the following files in the package directory:

• <package-name>.spec - RPM spec file
• sources - SHA512 checksums of source tarballs
• Source tarballs (.tar.gz, .tar.bz2, .tar.xz)
• Any patches or additional configuration files

3. Generate the sources File

The sources file must contain SHA512 checksums in BSD-style format:

sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

This produces the correct format:

SHA512 (oras-1.3.0.tar.gz) = fb871c0577f621f7e1f56a54f249f96c659b29c771dad529f9a9e838379b2d56bca9cef03a90071915a568e28e266326989e67359e589a87908c4fc077b14689
SHA512 (oras-1.3.0-vendor.tar.bz2) = fa23324bf3910c3dc050eb3c0ebfef3224168489679cf1195d668c656c6c53beaac94b4786e45c2cf3782f9437b0d166da9c18e608a3b474b04957ff1dec5226

Important: Use sha512sum --tag to generate the BSD-style format. Do not use the default GNU format.

4. Create Package Metadata

Create metadata/<package-name>.json:

{
  "modification_status": "independent",
  "release": "1",
  "upstream_repo": "https://github.com/example/project",
  "version": "1.3.0"
}

Fields:

• modification_status: Must be "independent" for packages not imported from Fedora (see Package Metadata Fields)
• upstream_repo: Canonical upstream git repository URL (required — CI enforces this). If no upstream repo exists, use https://src.fedoraproject.org/rpms/<name> as a fallback.
• version: Package version (must match spec file)
• release: Base release number without dist tag (typically 1 or 0.1; the .hum1 suffix comes from %{?dist} in the spec Release: line — see Package Metadata Fields)

5. Generate Konflux Resources

Generate the Konflux Component and ImageRepository resources:

make generate-host

Or directly:

python3 ci/generate_resources.py all

This updates:

• konflux-templates/rendered.yml
• .tekton/ pipeline files

6. Commit Changes

git add rpms/<package-name>/ metadata/<package-name>.json konflux-templates/ .tekton/
git commit -m "Add <package-name>-<version>-<release>

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"

Note: Source tarballs (.tar.gz, .tar.bz2, .tar.xz) are automatically ignored by .gitignore and should be uploaded to the lookaside cache instead.

Example: Adding oras

# 1. Create directory
mkdir rpms/oras
cd rpms/oras

# 2. Add spec file and sources
# ... create oras.spec ...

# 3. Generate sources file
sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

# 4. Create metadata
cat > ../../metadata/oras.json << 'EOF'
{
  "modification_status": "independent",
  "release": "1",
  "upstream_repo": "https://github.com/oras-project/oras",
  "version": "1.3.0"
}
EOF

# 5. Generate Konflux resources
make generate-host

# 6. Commit
git add rpms/oras/ metadata/oras.json konflux-templates/ .tekton/
git commit -m "Add oras-1.3.0-1.hum1"

See Also

• Package Metadata Fields - modification_status and release configuration
• Package Modification Tracking - Marking packages as modified vs clean
• Rebuilding Packages - Rebuilding existing packages
• Updating Dist-git Packages - Importing/updating from Fedora

14 - Pulp Access

Overview

Pulp is used to host and distribute Hummingbird RPM repositories. Access to the Pulp API requires a service account with credentials stored in the vault.

Existing credentials

The public Pulp service account credentials are stored in the vault at rhel-primitives/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE. Retrieve the cli.toml field and save it locally (see ci/pulp-setup/README.md for usage).

Creating a new Pulp service account

When you need a dedicated service account (e.g., for private product repositories):

1. Go to https://access.redhat.com/terms-based-registry/
2. Log in with your @redhat.com account
3. Click New Service Account
4. Enter a username like hummingbird-<qualifier>-pulp-bot
5. Enter a description like bot account for hummingbird <qualifier> pulp operations
6. Click Create
7. Capture the full username (in the format <id>|<username>) and the token

Configure the CLI

Create a cli.toml with the new credentials:

[cli]
base_url = "https://packages.redhat.com"
api_root = "/api/pulp/"
username = "<id>|hummingbird-<qualifier>-pulp-bot"
password = "<token>"
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
domain = "<target-domain>"

You can either save this to ~/.config/pulp/cli.toml or pass it explicitly with --config ./cli.toml when running scripts.

Store credentials in the vault

Store the credentials in the vault at apps/hummingbird with the key HUMMINGBIRD_<QUALIFIER>_PULP_BOT_CONFIG_FILE (e.g., HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE). Create two subkeys:

• cli.toml — the full config file contents
• value — the token value

Related

• Lookaside Cache Access — AWS credentials for source tarball uploads
• Private RPM Repositories — setting up private per-product Pulp repos
• ci/pulp-setup/README.md — Pulp domain and repository creation script

15 - Rebuilding Packages

AI Agent Note: When asked to rebuild packages, use the rebuild command: ./ci/dist_git.py rebuild <package> --reason "<reason>". For rebuilding reverse dependencies (e.g., “rebuild all Go packages”), use rebuild-rev-deps <package> --reason "<reason>". Always ask the user for a ticket link or explanation first to use as the reason. If the command fails with “uses macros in Release field”, see the “Packages requiring manual rebuild” section below for instructions.

Overview

This document covers three scenarios for triggering package builds:

1. No-change rebuild: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes).

2. Rebuilding reverse dependencies: Rebuild all packages that depend on a changed package (e.g., rebuild all Go packages when golang updates).

3. Backporting a patch: Add an upstream patch that hasn’t yet landed in Fedora to fast-track a fix or feature.

All scenarios use the .N release suffix pattern to ensure our builds sort higher than the upstream Fedora release while remaining lower than the next upstream version.

No-Change Rebuild

Using the rebuild command (recommended)

The rebuild command automates the Release field bump:

# Rebuild a single package
./ci/dist_git.py rebuild <package> --reason "<reason>"

# Rebuild multiple packages (one commit per package)
./ci/dist_git.py rebuild <package1> <package2> ... --reason "<reason>"

# Rebuild all packages (one commit per package)
./ci/dist_git.py rebuild --all --reason "<reason>"

# Rebuild all packages except specific ones (requires --all)
./ci/dist_git.py rebuild --all --exclude <pkg1>,<pkg2> --reason "<reason>"

Examples:

# Single package
./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes"

# Multiple packages
./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds"

# All packages (useful after toolchain updates)
./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15"

# All packages except a few already rebuilt earlier in the same rollout
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm --reason "toolchain update: GCC 15"

--exclude takes a comma-separated list of package names and only works together with --all — it is rejected when combined with an explicit package list, since you can simply omit the packages you don’t want to rebuild in that case.

The command:

• Automatically bumps the Release field using the .N suffix pattern
• Handles %autorelease by resolving and replacing with explicit values
• Preserves macros in the Release field (e.g., %{revision})
• Creates a properly formatted commit message (one per package with --all)
• Does not mark the package as modified (release-only changes are ephemeral)

Supports --dry-run to preview changes without committing:

./ci/dist_git.py --dry-run rebuild <package> --reason "test"

Creating MRs for rebuild commits

After creating rebuild commits locally, use rebuild_multi_mr.sh to push each commit as its own merge request (one MR per package, auto-merge enabled):

./ci/rebuild_multi_mr.sh

By default the script compares against origin/main. For local development, use --base to point at a different ref:

# Use local main branch as base (useful when origin/main is not up to date)
./ci/rebuild_multi_mr.sh --base main

# Use a specific commit SHA as base
./ci/rebuild_multi_mr.sh --base abc1234

# Preview what would be created without pushing
./ci/rebuild_multi_mr.sh --dry-run

# Limit to at most N MRs
./ci/rebuild_multi_mr.sh --max-updates=5

The script:

• Creates a chore/rebuild-{package} branch per commit and pushes it
• Titles each MR chore(rpms): Rebuild {package}: {reason}
• Enables auto-merge on all rebuild MRs
• Leaves the current branch untouched
• Skips branches that already exist on the remote (idempotent)
• Only processes commits whose message matches Rebuild {package}: {reason}; other commits in the range are silently skipped

Full workflow example:

# 1. Create the rebuild commits
./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update"

# 2. Preview the MRs that would be created
./ci/rebuild_multi_mr.sh --base main --dry-run

# 3. Create the MRs
./ci/rebuild_multi_mr.sh --base main

Packages requiring manual rebuild

Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file.

Macro indirection patterns

These packages define the Release field using a macro, where the macro itself contains %{?dist}. The rebuild command cannot detect or manipulate these without expanding all macros, which would break the macro system.

nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):

%{load:%{_sourcedir}/nodejs.srpm.macros}
%nodejs_define_version node 1:25.8.2-%{autorelease} -p
...
Release: %{node_release}

The %{node_release} macro is defined by an external macro system loaded from nodejs.srpm.macros. The release component is embedded in the version definition.

How to rebuild: Edit the %nodejs_define_version node line to bump the release component (e.g., change -%{autorelease} to -1.1 or increment existing .N).

kernel-headers:

%define specrelease 59%{?buildid}%{?dist}
...
Release: %{specrelease}

How to rebuild: Edit the %define specrelease line to add/increment the .N suffix before %{?buildid}:

%define specrelease 59.1%{?buildid}%{?dist}

krb5:

%global krb5_release 4%{?dist}
...
Release: %{krb5_release}

How to rebuild: Edit the %global krb5_release line to add/increment the .N suffix:

%global krb5_release 4.1%{?dist}

Why these can’t be automated

The rebuild command can handle:

• ✅ Simple numeric: Release: 5%{?dist}
• ✅ Macros ending with dist: Release: %{baserelease}%{?dist} (e.g., rpm, gcc)
• ✅ Complex macros with dist: Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist}
• ✅ Content after dist: Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}} (e.g., unbound)

The rebuild command cannot handle:

• ❌ Macros without %{?dist}: Release: %{node_release}
• ❌ Macros where dist is inside the macro definition: %{krb5_release} contains %{?dist}

This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM’s full macro parser.

Manual rebuild process

If you need to rebuild manually or the automated command doesn’t work for your use case, follow these steps:

1. Identify the package to rebuild

Identify the source package name and locate its spec file in rpms/<package>/<package>.spec.

If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --queryformat '%{SOURCERPM}' <binary-package> 2>/dev/null

Example: ncurses-libs-6.5-8.20250614.hum1 -> SRPM ncurses-6.5-8.20250614.hum1.src.rpm -> spec file at rpms/ncurses/ncurses.spec

2. Determine the Release bump pattern

The .N bump suffix must always appear immediately before %{?dist}. The %{?dist} suffix should always be the final component since it identifies the build environment.

For %autorelease, first resolve its value using rpmspec, then replace with the resolved value plus .1. In the Hummingbird monorepo, %autorelease always evaluates to 1.

Note: If the Release field is missing %{?dist} entirely or looks unusual (e.g., 1build1 instead of 1.1%{?dist}), flag this to the user for resolution. Check the git history to understand the original value:

git log -p -S "Release:" -- rpms/<package>/<package>.spec

This helps determine the correct fix when a previous bump was malformed.

3. Modify the spec file

Use sed to edit only the Release: line, avoiding any unintended whitespace changes that text editors may introduce:

sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms/<package>/<package>.spec

Verify the change with git diff before committing:

git diff rpms/<package>/<package>.spec

The diff should show only the Release line change:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

Important: Only modify the Release line. Do not introduce any other changes such as whitespace fixes or trailing newline modifications. If the diff shows additional changes, reset and retry with sed.

Important: Do not change the release field in metadata/<package>.json during local rebuilds or backports. That field is the current base release (Fedora/rawhide baseline, or a local base such as 0.1 when ahead of Fedora); see Package Metadata Fields.

4. Verify the bump is correct

Use rpm --eval to confirm the new release sorts higher than the original:

# Returns -1 if first < second (correct), 1 if first > second (wrong)
rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}'
# Expected output: -1

5. Commit the change

Use this commit message format:

Rebuild <package>: <reason>

<ticket link or explanation>

Example:

Rebuild ncurses: published multiple times with different hashes

HUM-1234

6. Verify the commit

After committing, verify only the Release line was changed:

git show --stat HEAD

Expected output should show exactly 1 insertion and 1 deletion:

 rpms/<package>/<package>.spec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

If the commit shows more changes, amend or reset and redo the change using sed.

Important notes about rebuilds

Modification status

Rebuilds do not change a package’s modification_status. Spec Release: bumps are ephemeral and do not make a package modified versus clean. See Package Metadata Fields for how modification_status and metadata release relate to rebuilds.

During updates, Release lines are normalized to avoid conflicts, and the automation ignores Release-only changes, so automatic Fedora updates continue normally after a rebuild.

If you want to explicitly prevent automatic updates (e.g., you’re investigating an issue), you can manually mark the package as modified:

./ci/dist_git.py mark-modified <package> --modified --reason "Investigating build issue"

Note: This will block automatic Fedora updates until you mark it clean again.

Rebuilding Reverse Dependencies

When a compiler, runtime, or toolchain package changes, you may need to rebuild all packages that depend on it. The rebuild-rev-deps command automates finding and rebuilding reverse dependencies.

When to use

Common scenarios for rebuilding reverse dependencies:

• Golang/Python runtime updates: When updating golang1.26, python3.14, etc., rebuild all Go/Python packages
• Toolchain changes: When updating gcc, rebuild packages that BuildRequire it
• Library ABI changes: When a library’s ABI changes, rebuild packages that BuildRequire it

Usage

./ci/dist_git.py rebuild-rev-deps <package> --reason "<reason>"

The command:

1. Finds all packages that have BuildRequires: <package> in their spec files
2. Rebuilds each package (bumps Release field and commits)
3. Reports a summary of successful/failed rebuilds

Examples:

# Rebuild all Go packages when golang updates
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "golang 1.26.2 update"

# Rebuild all Python packages when python updates
./ci/dist_git.py rebuild-rev-deps python3.14 --reason "python 3.14.1 update"

# Rebuild packages that depend on a specific library
./ci/dist_git.py rebuild-rev-deps openssl --reason "openssl 3.4.0 update"

Virtual BuildRequires (golang/python)

For virtual BuildRequires like golang1.25, golang1.26, python3.13, python3.14, the command automatically handles translation to the actual BuildRequires target:

# These all work the same way:
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "..."
./ci/dist_git.py rebuild-rev-deps go-rpm-macros --reason "..."

Important: Only the latest version triggers rebuilds. This is because:

1. Multiple golang versions exist: golang1.25, golang1.26
2. They all provide the same virtual package: Provides: golang = <version>
3. DNF always picks the highest version to satisfy Requires: golang

Therefore:

• ✅ rebuild-rev-deps golang1.26 → Rebuilds all Go packages (latest version)
• ❌ rebuild-rev-deps golang1.25 → Error: Not the latest version

This ensures rebuilds only happen when the active runtime actually changes.

Creating MRs for reverse dependency rebuilds

After creating rebuild commits, use rebuild_multi_mr.sh to push each commit as its own MR:

./ci/rebuild_multi_mr.sh --base main

See the “Creating MRs for rebuild commits” section above for full details.

Dry-run mode

Preview which packages would be rebuilt without making changes:

./ci/dist_git.py --dry-run rebuild-rev-deps golang1.26 --reason "test"

This shows:

• Which packages have the BuildRequires dependency
• What the new Release values would be
• Does not commit any changes

Backporting a Patch

Use this workflow when you need to fast-track an upstream fix or feature that hasn’t yet been released in Fedora.

1. Obtain the patch

Fetch the patch from the upstream repository. For GitHub PRs, append .patch to the PR URL:

curl -L https://github.com/<org>/<repo>/pull/<number>.patch \
  > rpms/<package>/<NNNN>-<short-description>.patch

Name the patch file with a numeric prefix matching the next available PatchN: slot in the spec file (e.g., 0004-fix-foo.patch if Patch1-3 already exist).

2. Add the patch to the spec file

Add a PatchN: declaration after the existing patches:

Patch3:         0003-existing-patch.patch
Patch4:         0004-fix-foo.patch

The patch will be applied automatically if the spec uses %autosetup -p1. If the spec uses explicit %patchN macros, add the corresponding apply line in the %prep section.

3. Check for a gorget source-pipeline

If metadata/<package>.source-pipeline.yaml exists, check whether its transform: step applies patches or otherwise runs against patched source (e.g. resolving a lockfile with yarn install, pnpm fetch, npm ci):

# run from the root of the rpms repo checkout
grep -n -E "patch -p[0-9]|git apply" metadata/<package>.source-pipeline.yaml

This catches the common forms, but isn’t exhaustive — if the transform applies patches some other way (a different tool, or a script that wraps patch/git apply), read the transform: step directly rather than trusting a grep miss.

If it does, and your new patch touches a file that step depends on (a lockfile or manifest such as yarn.lock, package.json, pnpm-lock.yaml, go.sum, Cargo.lock), add the same patch -p1 < "${PACKAGE_DIR}/<your-patch>.patch" line there too, in the same relative order as its PatchN: slot in the spec. The patch file itself needs no extra fetch step — it’s already in ${PACKAGE_DIR}/ because you committed it there in step 2.

The pipeline’s transform: step and the spec’s %prep/%goprep are two independent lists of patches to apply — nothing keeps them in sync automatically. A patch declared only in the spec is invisible to whatever artifact the transform step generates (e.g. an offline yarn/pnpm cache built from the lockfile), and the generated artifact silently drifts from what %build actually applies. See “Known sharp edge: patch-list duplication” in the source-pipeline design doc for the grafana12.4/grafana13.1 incident this rule comes from. test/test_source_pipeline_patches.py checks for this drift, but treat it as a safety net, not a substitute for updating the pipeline yourself here.

4. For Go packages: check go-vendor-tools.toml’s pre_commands

This applies whether or not the package uses gorget — rpms/<package>/go-vendor-tools.toml’s [archive] pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately:

grep -n -A2 "pre_commands" rpms/<package>/go-vendor-tools.toml

If your new patch targets a vendored Go module (or you’re editing pre_commands directly instead of adding a patch — don’t), and any pre_commands entry mutates go.mod/go.sum (directly, via go get, or via go mod tidy/edit), your patch must apply the same go.mod/go.sum change to the actual source tree, not just to go-vendor-tools.toml. pre_commands cannot substitute for this: they require live network access to run go get, which %prep doesn’t have (Konflux builds are hermetic). Compute the resulting go.mod/go.sum diff offline (e.g. clone upstream, apply the same edits, run go mod tidy if pre_commands does) and commit that as the patch.

Skipping this leaves go.mod in the build tree and vendor/modules.txt in the generated vendor archive silently requiring different versions of the same package, which go build -mod=vendor rejects as inconsistent vendoring. See the source-pipeline design doc for the trivy incident this rule comes from. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

5. Bump the Release

Follow the same .N suffix pattern as no-change rebuilds:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

6. Commit the patch

Use this commit message format:

<package>: backport <short description>

Backport: <link to PR or commit>
<ticket link if applicable>

Example:

dnf5: backport reproducible build sorting fix

Backport: https://github.com/rpm-software-management/dnf5/pull/2522

7. Mark package as modified

Mark the package as modified to prevent automatic Fedora updates from overwriting your backport:

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Backport fix for <issue description>"

Example:

./ci/dist_git.py mark-modified dnf5 --modified \
  --reason "Backport reproducible build sorting fix from upstream PR#2522"

This ensures the package won’t be automatically updated from Fedora until the backported patch lands upstream and you explicitly mark it clean again.

8. Test the build locally (optional)

Build the package locally to verify the patch applies cleanly:

./ci/build_rpms.sh <package>

Built RPMs will be in builds/<package>/RPMS/.

Related Operations

• Excluding Packages from Images - temporarily block faulty packages in container builds

16 - Setting Up Pulp Repositories

Overview

• This document and script should be used to initialize and setup a pulp domain and a set of repositories
• We choose the domain prefix of public- to ensure that the repositories are publicly available

Setup

• make sure you have followed the local development setup instructions in the infrastructure repository
• install pulp cli via dnf install pulp-cli
• setup the local pulp environment via:

cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

• Run the creation script

Create

./create-pulp-resources.sh public-hummingbird "source,x86_64,s390x,ppc64le,aarch64"

17 - Package Metadata Fields

Overview

Each package has a metadata file at metadata/<package>.json. Two fields control how Hummingbird treats local changes and rebuilds relative to Fedora:

This page is the canonical reference for configuring those fields. For day-to-day workflows (marking packages, rebuilding, importing), see the related docs at the end.

modification_status

modification_status records whether a package still matches its Fedora upstream import, has local source-level changes, or is Hummingbird-independent (not from Fedora).

When to use each value

• clean: Default for packages imported from Fedora with no local source changes. Automatic Fedora updates are allowed. Conflicts with modification_reason (must be absent).
• modified: Use after backports, custom patches, or other spec/source edits that must not be overwritten by an automatic update. Requires modification_reason.
• independent: Required for packages that are not imported from Fedora. These packages have no source / branch / sha fields. Conflicts with modification_reason (must be absent).

Imports set the status automatically (clean for Fedora imports, independent for Hummingbird-independent imports). After local source changes, set status with dist_git.py mark-modified — see Package Modification Tracking.

Rebuilds that only bump the spec Release: line do not change modification_status. Release-only changes are ephemeral and are ignored when deciding whether a package is modified versus clean.

modification_reason

When modification_status is modified, modification_reason is required. It should be a short explanation of why the package cannot be auto-updated (for example, a CVE backport or a custom spec change). Clear the reason when marking the package clean again.

modification_reason applies only to modified packages. Do not add it for clean or independent packages — see When not to update these fields.

release

The release field in metadata is not the same as the Release: line in the package spec. It is overloaded: the same field stores one of two kinds of base release (always without a dist tag such as .fc42 or .hum1):

1. Fedora/rawhide base — the upstream Fedora release this package was imported or last synced from (dist suffix stripped). Used while Hummingbird still tracks a Fedora build of the current version.
2. Local base — a Hummingbird-chosen base for packages with no Fedora upstream (independent), or for Fedora-imported packages that Hummingbird has version-bumped ahead of Fedora. In the ahead-of-Fedora case this is typically 0.1 — a locally invented placeholder so a later Fedora import with Release >= 1 sorts higher; it is not a value confirmed from an actual Fedora build.

dist_git.py rebuild uses this base (when a Fedora source is present) to compute the next .N micro-bump on the spec Release: line.

Who writes metadata release

Fedora-imported packages

For packages with a Fedora source:

• release is set on import and refreshed on dist_git.py update / sync from the upstream Fedora package (dist suffixes like .fc42 are stripped). That is the Fedora/rawhide base.
• When check_upstream_versions.py check --update bumps the package ahead of Fedora, it resets the spec to Release: 0.1%{?dist} (unless %autorelease) and writes that local base (0.1) into metadata release. Until Fedora ships the same version and update / sync runs again, metadata no longer holds a Fedora-confirmed baseline.
• Do not change metadata release during local rebuilds or backports. Rebuilds bump only the spec Release: line; metadata continues to record the current base (Fedora or local) so rebuild can compute the next .N suffix correctly.
• When Fedora’s baseline is Release: %autorelease, metadata release stores the resolved numeric release from MDAPI (not the literal %autorelease). Import/update replace %autorelease in the local spec with that value plus %{?dist}. For rebuilds that still see %autorelease, follow Rebuilding Packages.

Independent packages

For Hummingbird-independent packages (modification_status: "independent", no source field):

• There is no Fedora upstream release. Set release to a local base such as 1 or 0.1 when adding the package (no dist tag — same storage convention as Fedora-imported packages).
• The .hum1 suffix comes from the spec Release: line via %{?dist} (for example Release: 1%{?dist}), not from metadata release.
• dist_git.py rebuild does not treat metadata release as an upstream Fedora baseline for independent packages (it only uses that baseline when a source field is present).

Spec Release: vs metadata release

No-change rebuilds and reverse-dependency rebuilds change the spec Release: only. See Rebuilding Packages. When a Fedora update for the same version lands later, dist_git.py update / sync replaces the spec Release: from upstream and restores metadata release to the new Fedora/rawhide baseline.

When not to update these fields

Most day-to-day package work changes the spec, sources, or patches — not metadata modification_status, modification_reason, or release. Leave those fields alone unless the package’s relationship to Fedora actually changed.

Examples

Independent package + CVE fix: Edit the spec and add the patch. Keep:

{
  "modification_status": "independent",
  "release": "1",
  "version": "1.3.0"
}

Do not add modification_reason, and do not change status to modified.

Fedora package + CVE backport: After the patch, mark modified so auto-updates stay blocked:

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Backport CVE-2024-12345"

Do not edit metadata release as part of that change.

Rebuild only: Use dist_git.py rebuild (or bump the spec Release:). Leave metadata modification_status and release unchanged.

Related documentation

• Package Modification Tracking — mark packages modified/clean, check status, validation, and update hooks
• Rebuilding Packages — bump spec Release: for no-change rebuilds and backports
• Adding Independent Packages — create metadata for packages not from Fedora
• Updating Dist-git Packages — import and update from Fedora

18 - Debugging Build Failures

Overview

When ./ci/build_rpms.sh fails to build a package, you can use the --shell-after flag to drop into an interactive debugging environment. This preserves the build state and allows you to investigate failures, modify files, and re-run the build without starting over.

Quick Start

# Run build and drop into shell after completion (even on failure)
./ci/build_rpms.sh --shell-after <package>

After the build completes (success or failure), you’ll be dropped into a mock shell inside the build environment.

Understanding the Build Environment

The build happens in two layers:

1. Outer container: The rpm-build-pipeline podman container (Red Hat build environment)
2. Mock shell: An isolated RPM build environment inside the container (similar to a chroot)

When you use --shell-after, you’re placed directly in the mock shell.

Key directories in the outer container

• /results/build.log - Full build output with all commands that were run
• /sources/ - Package source files (spec, patches, etc.)
• /config/ - Mock configuration
• /repo/ - Your local rpms repository (read-only mount)

Key directories in the mock shell

• /builddir/build/BUILD/<package>-*/ - Unpacked source tree (modify and re-run build/test commands here)
• /builddir/build/originals/<package>.spec - The spec file (modify to test spec changes)
• /builddir/build/SOURCES/ - Source tarballs and patches
• /builddir/build/BUILDROOT/ - Install root (where %install places files)

Debugging Workflow

1. Connect to the outer container

To get a second shell without killing your existing mock shell session, connect to the outer container:

# In another terminal, find the running container
podman ps | grep rpm-build-pipeline

# Get a shell in the outer container
podman exec -it <container-id> bash

2. Examine the build failure

From the outer container, check the build log to understand what failed:

# View the end of the build log
tail -100 /results/build.log

# Search for specific errors
grep -i error /results/build.log

3. Enter the mock shell

To investigate or modify the build environment, you can get an additional shell in the mock chroot, particularly for agents which cannot use the initial interactive --shell-after from above.

# From the outer container, chroot into the mock build environment
chroot /var/lib/mock/local-x86_64/root /bin/bash

4. Iterate on the fix

The recommended approach for debugging is to extract and re-run specific commands directly:

Fast iteration: Extract and re-run specific commands (recommended)

For quick iteration while developing a fix, extract the exact build or test command from the build log and run it directly:

# From the outer container, find what command the failing phase runs
grep -A 30 'Executing(%check)' /results/build.log

# Look for lines starting with "+ " that show the actual commands
# Example output:
#   + cd /builddir/build/BUILD/dnf5-5.4.0.0-build/dnf5-5.4.0.0
#   + /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure ...

# Run that command in the mock chroot
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && <actual-command>"

For build failures during %build: Look for Executing(%build) in the log, find commands like make or ninja, then re-run them:

# Example: extract the build command
grep -A 30 'Executing(%build)' /results/build.log | grep -E '^\+ (make|ninja|cmake)'

# Re-run in the build directory
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/<build-subdir> && make -j14"

For test failures during %check: Look for Executing(%check) and extract the test command:

# Example: extract the test command
grep -A 30 'Executing(%check)' /results/build.log | grep -E '^\+ (ctest|make check|pytest)'

# Re-run tests
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && ctest --output-on-failure"

This approach is fast because it skips rpmbuild overhead and goes straight to the failing command. It works with the preserved BUILD directory from --shell-after and is ideal when you’re modifying source files and want quick feedback.

Modifying source files: Edit files directly in /builddir/build/BUILD/<package>-*/ and re-run the command to test your changes quickly.

Modifying the spec file: You can also test spec file changes by editing /builddir/build/originals/<package>.spec in the mock chroot:

# From the host, edit the spec file in the mock chroot
podman exec -it -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  vim /builddir/build/originals/<package>.spec

# Run rpmbuild to see the change
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  env HOME=/builddir LANG=C.UTF-8 su mockbuild -c \
  'rpmbuild -bb --target x86_64 --nodeps /builddir/build/originals/<package>.spec'

Full validation: Apply fixes and rebuild with build_rpms.sh

Manual rpmbuild in the preserved environment may not work reliably for all packages due to complex build dependencies and environment requirements.

Once you’ve identified a fix using the fast iteration approach:

1. Exit the debug environment and apply your fix to the actual source files or spec file in rpms/<package>/

2. Test the complete build using the build script:

   ./ci/build_rpms.sh <package>
   

This ensures your fix works through the entire build process in a clean environment.

Related Operations

• Rebuilding Packages - Bump release or backport patches
• Package Modification Tracking - Mark packages as modified

19 - Package Modification Tracking

Overview

The RPMs repository tracks whether packages have been locally modified from their Fedora upstream source. This tracking prevents automatic updates from overwriting local changes like backported patches or custom modifications.

Modification Status

Each package metadata file (metadata/<package>.json) has a modification_status of clean, modified, or independent. That field (and related modification_reason / release configuration) is documented in Package Metadata Fields. This page covers the workflows for checking status, marking packages, viewing diffs, and configuring update hooks.

An optional track_upstream string field controls whether a package is checked by check_upstream_versions.py for new upstream releases (via release-monitoring.org). Set it to "latest" to track the latest version, or to a version prefix like "1.26" to constrain updates to that series. Its presence enables tracking; omit the field to disable it. The check subcommand only checks packages with track_upstream set when no explicit package arguments are given. The list subcommand shows all packages regardless of this field.

Checking Package Status

View a package’s modification status:

jq .modification_status metadata/<package>.json

View reason for modification (if modified):

jq .modification_reason metadata/<package>.json

List all modified packages:

for f in metadata/*.json; do
  status=$(jq -r .modification_status "$f" 2>/dev/null)
  if [ "$status" = "modified" ]; then
    pkg=$(basename "$f" .json)
    reason=$(jq -r .modification_reason "$f" 2>/dev/null)
    echo "$pkg: $reason"
  fi
done

Viewing Package Differences

To see what changes exist in a modified package compared to upstream Fedora:

# Show full diff for a package
./ci/dist_git.py diff bash

# Show summary statistics
./ci/dist_git.py diff bash --stat

# Show only which files changed
./ci/dist_git.py diff bash --name-only

# Show raw diff (includes Release: bumps and whitespace)
./ci/dist_git.py diff bash --raw

# Diff all modified packages
./ci/dist_git.py diff --all

What’s shown:

• By default, the diff ignores Release: number changes (no-change rebuilds)
• Trailing whitespace and blank line changes are ignored
• Use --raw to see absolutely everything, including Release: bumps

Package types:

• Modified packages: Shows the differences
• Clean packages: Shows nothing (useful for verification)
• Independent packages: Skips with message “no upstream to diff against”

Marking Packages

Mark as Modified

Use this when you make local changes to a package (backports, custom patches, etc.):

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Brief explanation of why"

Examples:

# After backporting a patch
./ci/dist_git.py mark-modified gcc --modified \
  --reason "Backport CVE-2024-12345 fix from upstream"

# After custom spec change
./ci/dist_git.py mark-modified systemd --modified \
  --reason "Add custom service unit for Hummingbird"

The reason field is required and should be concise but descriptive. It helps future maintainers understand why the package can’t be auto-updated.

Mark as Clean

Use this to re-enable automatic updates after confirming your changes are no longer needed (e.g., the fix landed in Fedora):

./ci/dist_git.py mark-modified <package> --clean

This removes the modified status and allows the package to receive automatic updates from Fedora again.

Configure Upstream Tracking

Use set-upstream to configure upstream tracking settings for a package. Each flag independently sets or clears one metadata field. At least one flag is required; omitted flags leave their fields untouched.

./ci/dist_git.py set-upstream <package> [flags]

Each set/clear pair is mutually exclusive (can’t pass --track-version and --no-track-version together).

Examples:

# Enable upstream version tracking (any version)
./ci/dist_git.py set-upstream bash --track-version latest

# Disable upstream version tracking
./ci/dist_git.py set-upstream bash --no-track-version

# Set upstream name with version constraint for versioned packages
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Set release-monitoring.org project ID (integer)
./ci/dist_git.py set-upstream python3.11 --track-version 3.11 --project-id 13254

# Remove project ID (reverts to RPM name lookup)
./ci/dist_git.py set-upstream python3.11 --no-project-id

# Combine multiple flags in one call
./ci/dist_git.py set-upstream golang1.26 \
  --track-version 1.26 --project-id 13254

Metadata fields:

These fields affect two systems:

• dist_git.py update: When track_upstream is a version prefix, skips upstream versions that don’t match. For example, track_upstream: "1.26" allows 1.26, 1.26.0, 1.26.3 but rejects 1.27.0.
• check_upstream_versions.py: When release_monitoring_project_id is an integer, queries the v2 API directly by Anitya project ID. When it is a string, queries release-monitoring.org using that name instead of the RPM package name (e.g., looks up golang instead of golang1.26). When absent, uses the RPM package name. When track_upstream is a version prefix, filters the reported upstream versions to only those matching the prefix. Only packages with track_upstream set are checked by check_upstream_versions.py check when no explicit package arguments are given.

Find release-monitoring.org project IDs by searching on https://release-monitoring.org.

The project ID can be combined with a version prefix to filter versions returned by the project ID lookup:

{
  "release_monitoring_project_id": 13254,
  "track_upstream": "3.11"
}

Some upstream projects tag releases with a suffix that is not part of the RPM version (e.g., swift-6.3.3-RELEASE). After Anitya strips the version prefix, the reported version still contains the suffix (6.3.3-RELEASE), which is incompatible with RPM’s Version: field (hyphens are not allowed). Use version_suffix_strip to remove it before comparison and update:

{
  "release_monitoring_project_id": 21267,
  "version_suffix_strip": "-RELEASE"
}

Some upstream projects publish source tarballs independently of tag creation, and Anitya may report a version before the tarball is available. Use source_availability_check to name a checker function (registered in SOURCE_AVAILABILITY_CHECKERS in check_upstream_versions.py) that HEAD-probes the source URL before selecting a version. Versions whose source returns 404 are skipped in favour of the next available version. Available checkers:

• openjdk_osci — probes https://openjdk-sources.osci.io/openjdk{feature}/openjdk-{version}.tar.xz

{
  "release_monitoring_project_id": 369281,
  "track_upstream": "21",
  "upstream_version_transform": "openjdk_to_rpm",
  "source_availability_check": "openjdk_osci"
}

version_from_ref (fixed-ref, no-tagged-release packages)

Most gorget source-pipeline.yaml fetch steps are version-templated (ref: "v${VERSION}"), so dist_git.py update can invoke gorget with metadata/<package>.json’s own version directly. A few packages instead pin a fixed git-snapshot commit with no tagged upstream release at all (gcc, glibc, libyuv, php-patchwork-jsqueeze, vim) – their version field doesn’t change when the pinned commit does, so it can’t drive gorget’s --version argument.

version_from_ref tells dist_git.py update how to derive gorget’s --version string from a newly-pinned commit, instead of requiring someone to compute it by hand every time (the previous process, documented as a manual step in each such package’s source-pipeline.yaml comments).

Currently one type is supported:

• commit-date (used by gcc): "<metadata.json's version>-<commit's own YYYYMMDD date>", where the date comes from git log -1 --format=%cd --date=format:%Y%m%d <ref> run against the newly-pinned commit.

{
  "version_from_ref": {"type": "commit-date"}
}

Currently gcc-only. dist_git.py update only fully automates a fixed-ref pin refresh for gcc’s simple case (a full 40-char commit SHA lives directly in a spec %global). The other four packages pin via a git describe-style string (e.g. glibc’s %{glibcsrcdir} macro, glibc-2.43-47-gbc95068f5f) whose trailing hash is only a 10-character abbreviation – resolving that back to a full SHA needs a local clone containing the commit object, real extra network work with its own failure modes (ambiguous hashes, upstream unavailability), not yet implemented (see HUM-4621’s “Remaining work”). For those four, dist_git.py update detects when the pipeline’s pinned ref: no longer matches the merged spec and refuses to auto-commit – it’s routed to a draft, no-test-labeled MR for manual resolution instead, the same way a real git merge conflict is, rather than silently committing a sources file gorget was never actually run against.

Per-Package Update Hooks

When check_upstream_versions.py check --update updates a package, by default it sets Version: to the new upstream version and Release: to 0.1%{?dist} (unless %autorelease is used), adds a changelog entry, and downloads new sources from the URLs declared in the spec. Some packages need custom logic (e.g. generating stripped tarballs or patching macro-based version lines). A per-package hooks file lets you override or extend these default phases without changing check_upstream_versions.py itself.

Hooks file location

metadata/<package>.update-hooks.yaml

For example, metadata/nodejs25.update-hooks.yaml.

Hook phases

The YAML file supports three optional keys. Each value is a shell command string executed with bash -eo pipefail -c in the package directory as the working directory.

Omitting a phase means the default logic runs for that phase. Packages without a hooks file behave identically to before.

Unknown phase keys in the YAML cause a ValueError (fail-fast).

Environment variables

Every hook receives these environment variables:

Example

See metadata/nodejs25.update-hooks.yaml for a working example that uses all three hook phases.

How Auto-Updates Work

The ./ci/dist_git.py update command (used by automation) checks modification status before updating packages:

• clean packages: Updated automatically when new Fedora versions are available
• modified packages: Automatically merged with upstream changes (conflicts create draft MRs)
• independent packages: Update blocked (not sourced from Fedora)
• version-constrained packages: Skipped if upstream version doesn’t match track_upstream prefix

To force-update a modified package (discarding local changes):

./ci/dist_git.py sync <package>

The sync command bypasses the modification check and force-updates to the latest upstream version. After syncing, the package is automatically marked clean.

Resolving Merge Conflicts

When modified packages are updated from Fedora, dist_git.py update attempts to automatically merge local changes with the new upstream version using git’s three-way merge. When conflicts occur, the update still succeeds but creates a commit with conflict markers, and the automation files a draft merge request labeled with CONFLICT: for manual resolution.

Understanding Conflict Markers

Git uses this conflict marker structure:

<<<<<<< HEAD
Fedora's version (new upstream)
=======
Hummingbird's local modifications
>>>>>>> hummingbird-local

ALL THREE markers must be removed for a clean resolution.

Update branch/MR structure

MRs are created on branches following the pattern:

chore/dist-git-update-PACKAGENAME

These branches are automatically created by the dist_git_update GitLab schedule. If they have conflicts, they result in draft MRs with:

• Title prefix: CONFLICT: chore(rpms): Update ...
• Description listing the conflicting files
• no-test label to skip CI tests (saves resources since conflicts need manual resolution)

Resolution Process

1. Check out the conflict branch:

   git fetch origin
   git checkout origin/chore/dist-git-update-PACKAGENAME
   

2. Examine the conflict:

   # Find all files with conflict markers
   git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/
   
   # View the specific conflict
   git show HEAD:rpms/PACKAGENAME/PACKAGENAME.spec | grep -B5 -A10 "^<<<<<<< HEAD"
   

3. Understand the local changes:

   # Review commit history to understand why changes were made
   git log --oneline -- rpms/PACKAGENAME/
   git log -p -- rpms/PACKAGENAME/  # With diffs
   
   # Check the modification reason
   jq -r .modification_reason metadata/PACKAGENAME.json
   

   Understand the context for correct resolution:

   • What was the original purpose of the local change? Is it transient or permanent?
   • Is it a workaround for a bug, a security patch, or a configuration difference?
   • Does it affect other packages (e.g., nss builds nspr as a subpackage)?
   • Check spec file comments (e.g., NOTE: comments) for packaging details

   Decide which version to accept:

   • Accept HEAD (Fedora) for: release number lags, fixed workarounds that Fedora improved or addressed differently
   • Keep hummingbird-local for: security patches not in Fedora, FIPS requirements, critical fixes, and other permanent modifications
   • Merge both for: test skip lists, independent changes that don’t conflict logically
   • When in doubt: Accept Fedora’s version for packaging metadata (Release:, subpackage versions), keep Hummingbird’s version for functional changes (patches, dependencies, build options)

4. Resolve the conflict: Edit the file to choose the appropriate version (HEAD, hummingbird-local, or merge both). Verify no markers remain:

   git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/
   

5. Validate the resolution: Check that local modifications are preserved:

   # Check the diff against upstream (works on working tree, staging not required)
   ./ci/dist_git.py diff PACKAGENAME
   
   # Compare with previous modification commits to verify
   git log -p -- rpms/PACKAGENAME/
   

   The diff should show only the intended local modifications (ignoring Release: bumps). This confirms the merge preserved your changes correctly. Note: dist_git.py diff compares the filesystem working tree against upstream, so it works before or after staging.

6. Amend the commit: Record the original conflicted commit SHA, then amend:

   # Record the original conflicted commit SHA
   ORIGINAL_SHA=$(git rev-parse HEAD)
   
   # Stage the resolved files and amend the commit
   git add rpms/PACKAGENAME/
   git commit --amend -m "$(git log -1 --format=%B | head -n -1)
   
   Conflicted-Update: $ORIGINAL_SHA"
   

   This preserves the original commit message while adding a Conflicted-Update: trailer that records which commit contained the conflict markers. This helps track the resolution history and can be useful for auditing or debugging later.

7. Push the resolution:

   git push origin HEAD:chore/dist-git-update-PACKAGENAME --force-with-lease --push-option merge_request.unlabel=no-test
   

   This removes the no-test label from the MR, which triggers CI tests to run and verifies the resolution works correctly. Some developers might have origin as read-only remote, and a different writable remote (e.g. originw).

Common Conflicts

nss: Subpackage Release Numbers

The nss package builds nspr as a subpackage with its own release number offset.

BACKGROUND:

• nss builds both nss and nspr RPMs from the same source
• nspr_release uses an offset (%[%baserelease+n]) to avoid NVR clashes
• The spec file NOTE explains: reset to 1 when nspr_version changes, increment when only nss changes
• Fedora manages these offsets in their ecosystem to prevent conflicts

CONFLICT EXAMPLE:

<<<<<<< HEAD
%global nspr_release %[%baserelease+3]
=======
%global nspr_release %[%baserelease+1]
>>>>>>> hummingbird-local

REASONING: When updating to a new upstream nss version from Fedora:

• Accept Fedora’s nspr_release offset (HEAD) - they manage NVR clashes
• Our local offset was specific to Hummingbird rebuilds
• New upstream version should reset to Fedora’s packaging values
• Don’t try to “calculate” what it should be - trust Fedora’s packaging

RESOLUTION: Accept HEAD (Fedora’s value)

Special Case: Rebuild-Only Changes

Release-only changes (no-change rebuilds) are automatically ignored by the modification detection logic. This means:

• Bumping Release: 3%{?dist} → Release: 3.1%{?dist} does not mark the package as modified
• The package can still receive automatic Fedora updates
• The Release bump will be preserved if the update doesn’t change the upstream Release field

You do not need to mark packages as modified for rebuild-only changes, unless you want to explicitly prevent automatic updates for other reasons.

CI Validation

The CI pipeline validates modification status consistency using make check, which runs:

./ci/validate_package_modifications.py --all

This validation ensures:

1. All packages have a modification_status field
2. The value is one of: clean, modified, independent
3. If track_upstream is present, it must be a string ("latest" or a version prefix)
4. Modified packages have a modification_reason
5. Independent packages do not have source/branch/sha fields (Hummingbird-independent only)
6. Git commit history matches the declared modification status

The validation runs on every merge request and push to main, failing the build if metadata is inconsistent.

For local development, run the full validation:

./ci/validate_package_modifications.py --all

Or validate specific packages:

./ci/validate_package_modifications.py bash glibc gcc

Validation Modes

The validation script has two modes:

Fast mode (default): Checks git commit history patterns

./ci/validate_package_modifications.py --all

This validates that all commits since the last Sync follow standard patterns (have Upstream: trailers). Runs in less than a minute for all packages.

Thorough mode: Clones upstream repos and compares filesystems

./ci/validate_package_modifications.py --all --thorough

This performs full filesystem comparisons with upstream Fedora repositories. Slow and unreliable (hundreds of upstream dist-git clones) but authoritative - validates actual state regardless of git commit history.

For CI and daily development, fast mode is sufficient. Use thorough mode when:

• Debugging discrepancies between metadata and actual state
• Auditing the entire repository for hidden modifications
• Investigating why a package can’t be updated

Workflow Examples

Backporting a Patch

1. Add patch file and modify spec (see Rebuilding Packages)

2. Commit the changes

3. Mark as modified:

   ./ci/dist_git.py mark-modified dnf5 --modified \
     --reason "Backport reproducible build fix (upstream PR#2522)"
   

4. Package is now protected from automatic Fedora updates

Re-enabling Auto-Updates

When your backported fix lands in Fedora:

1. Verify the fix is in the latest Fedora version:

   ./ci/dist_git.py update dnf5  # This will fail with "modified" error
   

2. Mark the package clean:

   ./ci/dist_git.py mark-modified dnf5 --clean
   

3. Update from Fedora:

   ./ci/dist_git.py update dnf5  # Now succeeds
   

Importing New Packages

When importing packages, modification status is set automatically:

# Fedora package → marked as "clean"
./ci/dist_git.py import fedora/neofetch

# Hummingbird-independent package → marked as "independent"
./ci/dist_git.py import hummingbird/custom-tool

No manual marking needed for imports.

Troubleshooting

CI Fails: “Missing modification_status field”

This means a metadata file is missing the required field. This means that the package was not imported properly.

CI Fails: “Marked as clean but package has modifications”

The package has local changes but metadata says it’s clean. To fix:

1. Check what changed:

   git log -p -- rpms/<package>/
   

2. Mark as modified with the appropriate reason:

   ./ci/dist_git.py mark-modified <package> --modified --reason "..."
   

CI Fails: “Marked as modified but package is actually clean”

The package has no local changes but is marked modified. To fix:

1. Verify it’s actually clean:

   ./ci/dist_git.py update <package>  # Check if upstream matches
   

2. If confirmed clean, remove the modified status:

   ./ci/dist_git.py mark-modified <package> --clean
   

Update Blocked: “Cannot auto-update <package>”

This is expected for modified packages. Options:

1. Wait for fix to land in Fedora, then mark clean and update

2. Force-sync to discard local changes:

   ./ci/dist_git.py sync <package>
   

3. Keep blocked if the local changes are still needed

Related Documentation

• Package Metadata Fields - modification_status and release configuration
• Rebuilding Packages - How to rebuild and backport patches
• Updating Dist-git Packages - How automatic updates work

20 - Private RPM Repositories

AI Agent Note: When asked to route a package to a private repository, add private_product: <product> to the package’s entry in ci/package-overrides.yaml, then run make generate to regenerate resources. The product name must match an existing RPA entry in ci/konflux_rpa_config.yml.

Overview

Hummingbird LTS products (e.g., LTS OpenJDK, LTS DotNet) require subscription-gated RPMs that must not be publicly accessible. Each LTS product gets its own private Pulp repository with independent credentials, enabling per-product subscription enforcement.

The private RPM pipeline uses the same build infrastructure as public RPMs but routes packages to separate Pulp domains via dedicated Konflux Applications and ReleasePlanAdmissions (RPAs).

How it works

1. Each private product has a dedicated Konflux Application (e.g., private-<product>-rpms-main) separate from the public rpms-main application.
2. A ReleasePlan connects the private Application to its RPA, enabling auto-releases when builds complete.
3. Packages are assigned to a private product via private_product in ci/package-overrides.yaml.
4. A dedicated RPA per product routes those packages to private Pulp repositories.
5. Packages with private_product are automatically excluded from the public RPA.

Architecture

package-overrides.yaml          konflux_rpa_config.yml
  <package>:                      rpas:
    private_product: <product>      - name: ...public...
                                    - name: ...private-<product>...
                                      private_product: <product>
        |                                    |
        v                                    v
  Component resource              ReleasePlanAdmission
  application: private-           targets: private-<product>-rpms-main
    <product>-rpms-main           pulp_signed_domain: private-hummingbird-<product>
                                           ^
                                           |
                                  ReleasePlan
                                  application: private-<product>-rpms-main
                                  releasePlanAdmission: hummingbird-rpms-private-<product>

Assigning a Package to a Private Product

Add private_product to the package’s entry in ci/package-overrides.yaml:

<package>:
  private_product: <product>
  timeout_hours: 8

Then regenerate all resources:

make generate

This will:

• Set the package’s Konflux Component to use application: private-<product>-rpms-main
• Include the package in the private product’s RPA component list
• Automatically exclude the package from the public RPA

Verifying the assignment

After running make generate, verify the changes:

# Check the component's application assignment
grep -A5 '<package>-main' konflux-templates/rendered.yml | grep application

# Check the package appears in the private RPA
grep '<package>-main' releng/hummingbird-rpms-private-<product>.yaml

# Check the package is excluded from the public RPA
grep '<package>-main' releng/hummingbird-rpms-tech-preview-staging.yaml
# (should return no results)

Onboarding a New Private Product

Automated onboarding

Use the add-private-product subcommand to automate Pulp setup, RPA configuration, package assignment, and resource regeneration in a single step:

./ci/dist_git.py add-private-product <product> \
  --packages pkg1,pkg2 \
  --infra-repo <path-to-infrastructure-repo> \
  --pulp-config <path-to-cli.toml>

This handles Pulp infrastructure (step 2), RPA config (step 5), package assignment (step 6), infrastructure repo templates (step 4 file creation), and resource regeneration. You still need to manually complete: credentials secret deployment (step 3), merging the infrastructure repo MR created by this tool (step 4), and content guard setup (step 7).

Use --dry-run to preview changes without modifying anything.

Manual onboarding

Adding a new LTS product (beyond existing ones) requires these steps: service account, Pulp infrastructure, credentials secret, Konflux Application, RPA configuration, resource regeneration, and content guard setup.

1. Create a Pulp service account (optional)

If you want isolated credentials for private repo operations (recommended), create a dedicated service account before setting up the Pulp infrastructure. See Pulp Access for instructions on creating a service account, configuring the CLI, and storing credentials in the vault. Otherwise, you can reuse the existing public Pulp credentials.

2. Create Pulp infrastructure

Use the existing Pulp setup script to create the private domains, RPM repositories, and file repositories (for SBOMs and attestations). See ci/pulp-setup/README.md for the required pulp-cli-console plugin install before running these commands:

# Unsigned (staging) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned

# Unsigned (staging) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned \
  --type file

# Signed (production) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>

# Signed (production) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product> \
  --type file

If using a dedicated service account, add --config <path-to-cli.toml> to each command.

This creates per-architecture RPM repositories (source, x86_64, s390x, ppc64le, aarch64) with distributions, and file repositories (metadata, rpm-catalog) for SBOM and attestation storage.

3. Deploy the Pulp credentials secret

If using a dedicated service account (step 1), add a Kubernetes Secret template to the infrastructure repo so the credentials are deployed to the Konflux cluster:

1. Create kubernetes/setup-konflux/47-pulp-private-hummingbird-config-file-secret.yml.j2:

   ---
   kind: Secret
   apiVersion: v1
   metadata:
     name: pulp-private-hummingbird-config-file-secret
   stringData:
     cli.toml: «{ lookup_secret("HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE[deployed]:cli.toml") | forceescape }»
   type: Opaque
   

2. Add an entry to secrets.yml for the vault key:

   HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE:
     backend: hv
     meta:
       active: true
       created_at: '<timestamp>'
       deployed: true
   

   The secret name (pulp-private-hummingbird-config-file-secret) must match the pulp_secret_name in the private RPA configuration (step 5).

3. Add an ExternalSecret to the konflux-release-data repo (rhtap-release-data) so the credentials are available to the release pipeline. Create the ExternalSecret in the cluster-specific directory (e.g., tenants-config/cluster/kflux-prd-rh03/managed/rhtap-releng-tenant/):

   ---
   apiVersion: external-secrets.io/v1
   kind: ExternalSecret
   metadata:
     name: hummingbird-private-pulp-bot-production-secret
     annotations:
       argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
       argocd.argoproj.io/sync-wave: "0"
   spec:
     dataFrom:
       - extract:
           key: releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production
     refreshInterval: 1h
     secretStoreRef:
       kind: SecretStore
       name: releng-vault
     target:
       creationPolicy: Owner
       deletionPolicy: Delete
       name: hummingbird-pulp-credentials-private-production-secret
   

   Add it to the cluster’s kustomization.yaml and regenerate auto-generated files with tenants-config/build-manifests.sh.

4. Add the secret to the releng vault at https://vault.devshift.net under the path releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production with the following keys:

   Key	Value
   cli.toml	Full contents of the Pulp CLI config file
   expires-on	na
   owner	Your Kerberos username

4. Create the Konflux Application

The Application resource is managed in the infrastructure repo, not this repo. Create it there following the same pattern as rpms-main:

1. Create a new directory kubernetes/private-<product>-rpms-main/ in the infrastructure repo.

2. Add 00-application.yml.j2 with the standard Application template:

   ---
   apiVersion: appstudio.redhat.com/v1alpha1
   kind: Application
   metadata:
     name: {{ env["PROJECT_NAME"] }}
   spec:
     appModelRepository: {url: ""}
     displayName: {{ env["PROJECT_NAME"] }}
     gitOpsRepository: {url: ""}
   

3. Add the new project to the CI matrix in infrastructure/.gitlab-ci.yml. Find the PROJECT_NAME list under the konflux-rh03/hummingbird-tenant context and add the new application name:

   - PROJECT_NAME:
       - rpms-main
       - private-<product>-rpms-main   # add this line
     PROJECT_CONTEXT:
       - konflux-rh03/hummingbird-tenant
   

4. Add 01-release-plans.yml.j2 with a ReleasePlan that references the private RPA. Copy from kubernetes/rpms-main/01-release-plans.yml.j2 and update the releasePlanAdmission label to match the private RPA name:

   metadata:
     name: hummingbird-rpm-release-private-<product>
     labels:
       release.appstudio.openshift.io/auto-release: "true"
       release.appstudio.openshift.io/standing-attribution: "true"
       release.appstudio.openshift.io/releasePlanAdmission: hummingbird-rpms-private-<product>
   spec:
     application: {{ env["PROJECT_NAME"] }}
     target: rhtap-releng-tenant
     # ... copy remaining spec from rpms-main/01-release-plans.yml.j2
   

   The ReleasePlan connects the private Application to its ReleasePlanAdmission. Without it, builds in the private Application will not trigger releases.

5. Copy 10-integration-test-scenarios-testing-farm.yml.j2 from kubernetes/rpms-main/. The template is fully parameterized with {{ env["PROJECT_NAME"] }} so no edits are needed. This may be needed to prevent the Konflux PR group integration test from blocking MRs that touch private product packages.

Merge this MR in the infrastructure repo first — the CI pipeline will deploy the Application, ReleasePlan, and IntegrationTestScenarios to the cluster. Components and RPAs in this repo reference it by name, so the Application must exist before they are applied.

5. Add RPA configuration

Add a new entry to the rpas: list in ci/konflux_rpa_config.yml:

rpas:
  # Existing public RPA (private packages are automatically excluded)
  - name: hummingbird-rpms-tech-preview-staging
    # ... existing config ...

  # New private product RPA
  - name: hummingbird-rpms-private-<product>
    application_prefix: private-<product>-rpms
    release_org: registry.stage.redhat.io/hummingbird-tech-preview
    single_component_mode: true
    service_account_name: hummingbird-rpm-release-staging
    pulp_unsigned_domain: private-hummingbird-<product>-unsigned
    pulp_signed_domain: private-hummingbird-<product>
    pulp_secret_name: hummingbird-pulp-credentials-private-production-secret
    pipeline_revision: <release-pipeline-branch>
    pipeline_url: https://github.com/scoheb/release-service-catalog.git
    private_product: <product>
    component_filter:
      path_prefix: rpms/

Key fields:

6. Assign packages and regenerate

Add private_product: <product> to each package in ci/package-overrides.yaml, then regenerate:

make generate

This produces:

• Updated konflux-templates/rendered.yml with per-component application assignments
• A new RPA file at releng/hummingbird-rpms-private-<product>.yaml
• Updated public RPA excluding private packages

7. Set up Pulp content guard

To restrict access to the private Pulp repositories, a content guard must be configured so only customers with the correct subscription can access the content:

1. Obtain a SKU for the private product’s subscription offering.
2. Create a feature for that SKU in the Feature service.
3. Contact the Pulp team to:
   • Map the organization ID to the feature
   • Set up the content guard on the private Pulp domain(s)

This step is required before customers can access the private repositories. Without it, the repositories are created but have no access control.

Pulp credentials

If using a dedicated service account (step 1), see Pulp Access for vault storage instructions.

If using the shared service account, private RPAs reuse the same Pulp publishing credentials as the public RPA (hummingbird-pulp-credentials-production-secret). Per-product access control for customers is handled downstream by Red Hat’s subscription entitlement system, not at the Pulp publishing layer.

Public/Private Routing

Packages with private_product set in ci/package-overrides.yaml are automatically excluded from the public RPA and included only in their product’s private RPA. Removing the private_product field returns the package to public-only publishing.

Related Files

21 - Reporting CVE Data Issues

Overview

Sometimes CVE data published on cve.org or NIST NVD is incorrect – wrong affected package, wrong severity score, wrong affected version range, or missing fix information. When this happens, the incorrect data propagates into vulnerability scanners and can cause false positives (or false negatives) for Hummingbird container images.

This document describes how to report these issues to get the data corrected at the source.

When to Report

Report a CVE data issue when you find any of the following:

• Wrong affected package – the CVE Record lists a package that is not actually affected
• Wrong version range – the affected or fixed version boundaries are incorrect
• Wrong severity – the CVSS score or severity rating does not match the actual impact
• Missing fix information – a fix exists upstream but the CVE Record does not reflect it
• Wrong CPE match – the NVD CPE configuration matches products that are not affected

How to Report

Reporting to the CVE Program (cve.org)

The CVE Program accepts corrections through the CVE Numbering Authority (CNA) that owns the record.

1. Find the CVE Record at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX.
2. Identify the CNA listed in the record (shown in the “Assigning CNA” field).
3. Contact the CNA directly to request a correction. Most CNAs accept reports via:
   • Their security reporting email (often listed in their CNA page)
   • GitHub issues if the CNA is an open source project
4. If the CNA is unresponsive, use the CVE Program Request form to dispute the record.

Reporting to NIST NVD

NIST NVD enriches CVE Records with CVSS scores and CPE match data. To request corrections:

1. Navigate to the NVD entry at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX.
2. Click the “Contact” link or email nvd@nist.gov with:
   • The CVE ID
   • The specific field that is incorrect
   • Evidence of the correct data (links to upstream commits, release notes, etc.)

Reporting to Red Hat Product Security

If the incorrect CVE data is causing issues in Red Hat’s vulnerability tracking:

1. File a Jira ticket in the HUM project with the CVE ID and a description of the data issue.
2. Red Hat Product Security can update the VEX feed to reflect the correct disposition for Hummingbird, even before the upstream CVE data is corrected.

Tracking the Correction

After reporting, track the status of the correction:

• CVE Record updates are published at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX
• NVD updates appear at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX (NVD may take days to weeks to process corrections)
• Update the HUM Jira ticket with the correction status so the team is aware

Requesting information on the fix status of an existing CVE

File a support request at https://access.redhat.com/support.

22 - Lookaside Cache Access

Overview

Source tarballs for RPM packages are stored in an S3-based lookaside cache rather than in git. To upload files to the cache (e.g., when updating a package to a new upstream version), you need AWS credentials with the appropriate permissions.

The ci/upload-to-lookaside-cache.sh script handles uploads directly, and ci/check_upstream_versions.py --update calls it automatically when downloading new source archives.

Prerequisites

You must be a poweruser in the arr-cloud-aws-core group (it-cloud-aws-727920394381-poweruser). Request access to this group if you do not already have it.

Obtaining AWS Credentials

There are two ways to authenticate.

Option A: Browser-based login

Run aws login, which opens a browser for authentication:

aws login

Option B: Kerberos-based login via container

Use the CKI tools container to obtain credentials via Kerberos:

$ podman run --rm -it \
    -e KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u) \
    -v $HOME/.aws:/cki/.aws:U,Z \
    quay.io/cki/cki-tools:latest

bash-5.2# kinit <userid>@IPA.REDHAT.COM

bash-5.2# AWS_IDP_URL=https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws \
    cki_aws_login --duration 43200 \
    --account 727920394381 --role poweruser

Replace <userid> with your Kerberos user ID. This writes credentials to $HOME/.aws on the host (bind-mounted into the container).

Note: This overwrites the AWS default profile credentials. If you use the default profile for other purposes, back up $HOME/.aws/credentials before running this command.

Uploading Files

Once credentials are configured in $HOME/.aws, you can upload files to the lookaside cache.

Manual upload

./ci/upload-to-lookaside-cache.sh -f <file> -p <package>

Example:

./ci/upload-to-lookaside-cache.sh -f rpms/tar/tar-1.35.tar.xz -p tar

Automated upload via version checker

check_upstream_versions.py --update downloads new source archives and uploads them to the lookaside cache automatically:

./ci/check_upstream_versions.py check --update <package>

The check subcommand only processes packages with "track_upstream": true in their metadata when no explicit package arguments are given. To see all packages and their upstream status, use check_upstream_versions.py list. See Package Modification Tracking for how to enable tracking.

See Also

• Adding Independent Packages - Adding new packages with source tarballs
• Rebuilding Packages - Rebuilding existing packages

23 - Upstream Diff Analysis

Overview

The upstream diff analysis tool classifies locally modified RPM packages by how their changes relate to upstream Fedora. This helps the team systematically decide which modifications should be proposed upstream, which are Hummingbird-specific, and which need more complex handling.

The primary interface is the /upstream-diff Claude Code skill, which orchestrates the full workflow. The skill delegates all deterministic operations (diffing, metadata lookup, caching) to ci/upstream_diff.py and applies its own judgment only for the classification step.

Classification Categories

Each modified package is classified into one of five categories:

Workflow

Invoke the skill in Claude Code with /upstream-diff. When called with no arguments, it offers three modes:

Analyze

Prepares the next batch of unanalyzed or stale packages, reviews each diff and any upstream PR activity, then classifies each package into a category with reasoning and a recommended action. Results are cached locally so subsequent runs skip already-analyzed packages.

You can also target specific packages by name (e.g., /upstream-diff analyze bash glibc).

View Results

Displays the cached analysis as a summary table. Supports filtering by category and showing details for a single package. Use this to get a quick overview of where things stand across all modified packages.

JIRA

Creates or updates a JIRA issue for a specific package to track the upstream proposal. The skill generates a pre-filled description and analysis comment from the cached data, previews it for confirmation, and then files or updates the issue. The JIRA issue key is recorded in the cache so it appears in the view output.

Script Reference

The skill uses ci/upstream_diff.py under the hood. The script can also be called directly for automation, scripting, or when working outside Claude Code.

Note that prepare and save are two halves of a classification pipeline: prepare gathers raw data (diffs, metadata, upstream PRs, category definitions), but its output requires human or LLM judgment to select a category and write the reasoning fields before calling save. The skill’s Analyze mode bridges this gap automatically.

Prepare

Gather metadata, diffs, upstream PR status, and the classification schema — this is the data the skill’s Analyze mode feeds into its classification step:

# Specific packages
./ci/upstream_diff.py prepare bash glibc

# Next batch of unanalyzed packages
./ci/upstream_diff.py prepare --batch 10

Save

Store a classification result. The skill’s Analyze mode calls this after classifying each package:

./ci/upstream_diff.py save <package> \
  --category <category> \
  --changes-summary "One-line summary of changes" \
  --reasoning "Why this category was chosen" \
  --recommendation "Recommended next action" \
  --upstream-prs <pr_id1> <pr_id2> ...

# No related upstream PRs
./ci/upstream_diff.py save <package> \
  --category hummingbird-specific \
  --changes-summary "Custom Hummingbird macros for FIPS" \
  --reasoning "FIPS build flags are specific to Hummingbird" \
  --upstream-prs ""

The --hummingbird-macros flag can be added when changes use Hummingbird-specific RPM macros.

Results are cached in .cache/upstream-diff-analysis.json.

View

Display cached results — this is what the skill’s View Results mode calls:

# Summary table
./ci/upstream_diff.py view

# Single package detail
./ci/upstream_diff.py view <package>

# Filter by category
./ci/upstream_diff.py view --category upstreamable

# Include unanalyzed modified packages
./ci/upstream_diff.py view --all

# Markdown-formatted links (for JIRA or rendered contexts)
./ci/upstream_diff.py view --markdown

# Raw JSON
./ci/upstream_diff.py view --json

Check Upstream PRs

Check whether local changes have already been submitted or merged in the upstream Fedora dist-git repository:

./ci/upstream_diff.py check-prs <package1> <package2> ...

Shows open and recently merged (last 90 days) pull requests from the upstream Fedora dist-git repo.

Update Related PRs

After initial analysis, update which upstream PRs are related to a package’s local changes:

# Set related PRs
./ci/upstream_diff.py save --set-upstream-prs <package> <pr_id1> <pr_id2> ...

# Clear related PRs
./ci/upstream_diff.py save --set-upstream-prs <package>

JIRA Templates

Generate pre-filled JIRA content from cached analysis — this is what the skill’s JIRA mode uses:

./ci/upstream_diff.py jira-template <package> [--epic HUM-1613]

Record a JIRA issue key after creating an issue:

./ci/upstream_diff.py save --set-jira <package> HUM-XXXX

Related Documentation

• Package Modification Tracking — how packages are marked modified and how diffs are computed
• Rebuilding Packages — how to rebuild and backport patches

24 - GPG Source Verification

Overview

Some packages verify the authenticity of their upstream source tarball with a detached OpenPGP signature where upstream signs each release with a private key and publishes a .asc/.sig alongside the tarball. During source fetching, the gorget source-pipeline tool checks that signature against the project’s public key before the bytes are ever used in a build.

The trusted public keys live centrally in metadata/gpg-keys/<project>.gpg, one keyring file per upstream project. Centralized storage means:

• the full set of signers we trust is auditable in one directory, and
• rotating or revoking a key is a single, reviewable commit.

The directory is handed to gorget as --gpg-keys-dir metadata/gpg-keys (see ci/check_upstream_versions.py), so a pipeline’s keyring: field is just a filename within it.

How the pieces fit together

Three things wire up verification for a package:

1. The keyring — metadata/gpg-keys/<project>.gpg, the trusted public key(s).

2. The pipeline step — a verify: entry in metadata/<package>.source-pipeline.yaml:

   fetch:
     - type: url
       url: "https://curl.se/download/curl-${VERSION}.tar.xz"
     - type: url
       url: "https://curl.se/download/curl-${VERSION}.tar.xz.asc"
   
   verify:
     - type: gpg-signature
       target: "curl-${VERSION}.tar.xz"       # the artifact to verify
       signature: "curl-${VERSION}.tar.xz.asc" # its detached signature
       keyring: "curl.gpg"                      # filename in metadata/gpg-keys/
   

   gorget imports keyring into a fresh, throwaway GPG homedir per check, then runs the equivalent of gpg --verify <signature> <target>. A bad or missing signature fails the fetch.

3. CI validation — test/test_gpg_keys.py (run by make check) confirms every file in metadata/gpg-keys/ is a parseable public key, and that every keyring: referenced by a pipeline actually exists.

Adding a key for a new package

Prerequisite: gpg (from the gnupg2 package). It ships in the CI image, run the commands below inside a container, e.g. podman run --rm -it -v "$PWD:$PWD:z" -w "$PWD" quay.io/hummingbird-ci/gitlab-ci:latest bash.

1. Obtain the upstream public key. Prefer a key you can already trust: many packages already ship the maintainer’s key next to their spec (e.g. rpms/curl/mykey.asc, rpms/bash/chet-gpgkey.asc). Otherwise download it from the project’s official key page.

2. Store it as a keyring in metadata/gpg-keys/. The keyring may be ASCII-armored or binary; this repo standardizes on binary .gpg. Convert an armored key with --dearmor:

   gpg --dearmor < rpms/<package>/<upstream-key>.asc > metadata/gpg-keys/<project>.gpg
   

   Name the file after the upstream project, not the RPM (so multiple versioned packages, e.g. python3.11/python3.12, can share one keyring).

3. Verify the key is what you expect. Print its fingerprints and confirm they match the fingerprints published on the upstream’s official channel:

   gpg --show-keys --with-fingerprint metadata/gpg-keys/<project>.gpg
   

4. Wire up the pipeline. Add (or extend) metadata/<package>.source-pipeline.yaml with the fetch steps for the tarball + signature and the verify: [{type: gpg-signature, ...}] step shown above.

5. Prove the whole chain end to end before committing import the keyring into a throwaway homedir and verify a real release signature against it:

   V=<version>
   curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz"
   curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz.asc"
   export GNUPGHOME=$(mktemp -d)
   gpg --import metadata/gpg-keys/<project>.gpg
   gpg --verify "<tarball>-$V.tar.xz.asc" "<tarball>-$V.tar.xz"   # expect "Good signature"
   

6. Run the checks: make check (validates the keyring and the pipeline reference).

Rotating or replacing a key

Upstream may roll to a new signing key (expiry, policy, new maintainer). Because the trusted set is just files in one directory, rotation is a single commit:

1. Obtain the new public key from the upstream’s official channel and confirm its fingerprint out of band.

2. Replace the contents of metadata/gpg-keys/<project>.gpg (re-run the --dearmor step). If upstream signs a transition period with both keys, you may keep both by importing them into the same keyring:

   gpg --dearmor < old-key.asc  > metadata/gpg-keys/<project>.gpg
   gpg --dearmor < new-key.asc >> metadata/gpg-keys/<project>.gpg
   

3. Re-run the end-to-end verification (step 5 above) against the latest release, then make check.

4. Commit with a message recording why the key changed and how you confirmed the new fingerprint, this file is the audit trail for what we trust.

Revoking / removing a key

• If a project drops GPG verification, delete both metadata/gpg-keys/<project>.gpg and the verify: step from its pipeline in the same commit. (CI fails a pipeline that references a missing keyring, and if the key is orphaned the reverse is easy to spot.)
• If a key is compromised, remove it immediately and replace it with the upstream’s revocation / replacement key.

Troubleshooting

• gpg: no valid OpenPGP data found — the file isn’t a real keyring (empty, truncated, or you saved an HTML error page). Re-download and re---dearmor. test_gpg_keys.py catches this in CI.
• gpg: Can't check signature: No public key — the signature was made with a key that isn’t in the keyring. Upstream likely rotated keys, follow Rotating or replacing a key above.
• BAD signature — the tarball does not match the signature. Do not paper over this. It means a corrupted download or, in the worst case, tampering. Re-fetch from the canonical source and if it persists, escalate rather than accepting the artifact.

25 - Rebasing the Buildroot to a New Fedora Release

AI Agent Note: This document is the procedural runbook for a Fedora buildroot rebase (e.g. F43 -> F44). It was written after executing the F44 rebase (see HUM-2018 and its child stories HUM-3369, HUM-3370, HUM-4536, HUM-4537 for the detailed history) and should be followed, and updated, the next time this operation is performed (e.g. F44 -> F45). Read this whole document before starting; the ordering between steps matters and several steps have hard dependencies on the one before it.

Overview

Periodically, Fedora ships a new stable release and Hummingbird needs to move its buildroot (mock/mock.cfg) and the packages that track a specific Fedora branch (rather than rawhide) onto it. This is a multi-day, multi-phase effort touching the core toolchain, CI/testing configuration, and potentially every package in the repository (since changing the compiler/linker toolchain can affect any build). The F44 rebase took about 2 calendar weeks.

Pre-flight checks

Before starting, confirm:

• The target Fedora release is actually stable (not just branched) and its package repos are reachable from the build environment.

• Which packages currently track the outgoing release branch explicitly (i.e. branch is a versioned Fedora branch like f43, not rawhide) via:

  # List every package whose branch is pinned to a specific Fedora release (not rawhide)
  ./ci/dist_git.py list | xargs -I{} sh -c 'echo -n "{}: "; python3 -c "import json; print(json.load(open(\"metadata/{}.json\"))[\"branch\"])"' 2>/dev/null | grep -E ': f[0-9]+$'
  

  For each, decide up front whether it should move to the new branch, move to rawhide, or stay pinned (see “Packages that intentionally stay behind” below) — don’t leave this undecided until a package fails to build.

• Whether any of those packages carry FIPS-validation-critical modifications. Packages whose modification_reason references FIPS certification should generally not be rebased to new branch content, even though they still need to be rebuilt against the new toolchain. Moving them would risk invalidating the FIPS-validated source. (See HUM-4989 for a related, more general proposal to record why a package is pinned in its metadata rather than relying on the modification reason text.)

• Whether any package whose version is the “which version is default” signal for an adjacent language/tooling ecosystem (e.g. python-rpm-macros) needs the same branch-pinning treatment as the interpreter itself — especially if that ecosystem is mid-transition to a new major version upstream at the same time as this rebase. See “Known gotchas: python3dist / rawhide-tracked toolchain-adjacent packages” below for why this matters and what already went wrong once.

• Whether to pause the routine automated dist-git sync bot for the duration of the Step 4 mass rebuild. See “Known gotchas: automation racing a manual mass rebuild” below for the tradeoff.

Step 1 — Buildroot config + core toolchain (separate MRs)

Do not bundle the buildroot config change with the toolchain rebuilds in one MR — split them so the buildroot config lands and validates quickly, independent of the (much slower) toolchain rebuilds.

1. mock/mock.cfg: update dist, releasever, bootstrap_image, description, and the [fedora]/[fedora-updates] repo baseurls and names to the new release. Merge this on its own or bundled only with fedora-repos (see next point) and binutils if it’s a clean (unmodified) package — do not bundle glibc/gcc/llvm here.
2. Testing/CI fallback config (rpms/ci/repos/fedora-43.repo → fedora-44.repo, and ci/default-tests/tests-rpm.yml’s repo copy + GPG key import): these reference the buildroot’s Fedora version directly, and there’s exactly one of each — bump them in the same MR as mock.cfg (or immediately alongside it), not later. During the F44 rebase this was deferred and wasn’t caught until php failed its install test days later in Step 4, against a fallback repo that still pointed at the outgoing release. (This is distinct from the per-package test fixture gotcha below, which recurs throughout every step — see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references.”)
3. fedora-repos: rebase to the new release. If Fedora’s spec uses a conditional ELN-style macro for Release:, evaluate whether to keep it or hardcode a plain Release: N%{?dist} — this is a judgment call each time depending on what’s simplest to maintain; document whichever choice is made in the package’s modification_reason.
4. binutils (if it carries local CVE backport patches): before assuming the patches still apply, check whether the CVEs they address have already been fixed upstream in the new Fedora version. If so:
   • Pull Fedora’s current spec as the new baseline (don’t try to merge patch-by-patch on top of the old spec).
   • Re-add only the local patches that are not already fixed upstream.
   • For a quick validation build, it’s sufficient to confirm the patch set applies and compilation begins — you don’t need to wait for a full build to complete just to validate the patch set is sane. Kill the build once compilation starts.
5. glibc: this is the highest-risk package in this phase because of a circular bootstrap dependency (see “Known gotcha: glibc/gcc bootstrap circularity” below). Expect this to require a temporary workaround, tracked as a modified package, to be reverted in Step 2.
6. gcc: reapply any local modification (e.g. disabling non-essential language frontends) against the new spec. Expect this to be one of the longest-running builds in the whole rebase (multiple hours per architecture) — plan CI capacity accordingly.
7. llvm: rebase and rebuild; this can run in parallel with gcc and typically finishes faster.

Step 2 — Bootstrap cleanup

Once the new compiler (from Step 1) is published to Pulp:

1. Rebuild annobin against the new compiler. This must happen before the next step, or reverting the glibc bootstrap workaround will immediately re-trigger the same circular dependency.
2. Revert the glibc bootstrap workaround from Step 1, now that the new compiler and annobin are both available. Clear the modified status/reason that was recording the workaround.
3. Rebuild libtool against the new compiler. This is easy to miss (it was missed during the F44 rebase and had to be done out-of-band mid-mass-rebuild) but several packages (apr, audit, authselect, automake, avahi, bind, catatonit, cryptsetup, find, and likely others) will fail to build against the new toolchain without it. Do this here, not in Step 4.

Step 3 — Toolchain-adjacent, release-pinned packages

Rebase the remaining packages that were explicitly tracking the outgoing Fedora branch and don’t carry a permanent reason to stay behind (see “Packages that intentionally stay behind”). For each:

• Decide whether it should move to the new versioned branch (e.g. f44) or to rawhide — moving to rawhide can simplify build-dependency resolution for clean (unmodified) packages, but ties future updates to rawhide’s churn.
• For packages with local modifications, check whether the modification is still needed (e.g. a security backport that may have landed upstream by now) before reapplying it blindly.
• Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before opening this package’s MR (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — fix it in this same MR if present.

Step 4 — Mass rebuild of everything else

Rebuild every remaining package against the new toolchain so that linking, annobin annotations, and compiler hardening are consistent across the whole package set.

1. Build the exclusion list first: everything already rebuilt in Steps 1–3, plus anything merged in the hours immediately before starting this step (check recent merge history). Use dist_git.py rebuild --all’s --exclude option (HUM-5183):

   ./ci/dist_git.py rebuild --all --exclude pkg1,pkg2,... --reason "<toolchain> toolchain rebuild"
   

   During the F44 rebase this flag didn’t exist yet, so the exclusion list had to be hand-rolled with ./ci/dist_git.py list | grep -v -E '^(pkg1|pkg2|...)$' piped into rebuild — no longer necessary now that --exclude has landed.

2. Watch for packages whose Release: field doesn’t fit the simple .N bump pattern (e.g. packages using build-time macros for Release:) — these will fail to commit cleanly and need to be handled in a separate pass.

3. Submit in batches, not one giant MR wave — use ./ci/rebuild_multi_mr.sh --dry-run to preview, then ./ci/rebuild_multi_mr.sh --max-updates=N to submit and auto-merge-on-green in controlled increments. Plan for several days; a run of ~460 packages took about a week of elapsed time across 5 batches during the F44 rebase, gated by CI/Testing Farm throughput more than build time itself.

4. Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before each package’s rebuild MR merges (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — this is what let php’s failure slip through undetected for days during the F44 rebase, so don’t rely on remembering it only once at the end.

5. Expect a small number of packages to need bespoke fixes that a routine rebuild can’t resolve automatically — license metadata regressions surfaced by newer scanners, compiler/linker regressions in vendored dependencies (e.g. a bindgen version incompatible with a newer clang), confirmed upstream compiler-triggered bugs in JIT/JVM-style software, or stale version-gates in local patches that assumed an older toolchain. Budget time for a handful of these near the end of the mass rebuild.

Known gotchas

glibc/gcc bootstrap circularity

A new Fedora release’s glibc spec may use a feature of the new gcc (e.g. a new compiler flag), while the new gcc may in turn require symbols only present in the new glibc. If Hummingbird’s own Pulp repos still serve the old toolchain at a higher priority than the new Fedora repos during the transition window, this becomes a real chicken-and-egg problem, not just a theoretical one.

The pragmatic fix used for F44 was to drop the new-toolchain-specific flag from glibc temporarily (if it’s a defensive/hardening-only flag whose absence doesn’t break correctness), rebuild glibc and gcc against each other with it disabled, then re-enable it once both are published and annobin/libtool have also been rebuilt against the new compiler (Step 2 above).

Package-specific test fixtures with hardcoded Fedora-version references

Beyond the single global fallback config bumped in Step 1, some packages carry their own test/rpms/<pkg>.yml test suite that independently hardcodes a Fedora repo/GPG-key reference for setting up a test chroot (during F44 this was glib2, hummingbird-release, ca-certificates, crypto-policies, and openssl). Don’t wait for a one-time catch-all audit to find these — fix each one in the same MR as that package’s own rebuild/update, whenever it happens to land during the rebase (Step 1, 3, or 4, whichever touches that specific package — see the reminders in each of those steps above). Relying on an exhaustive upfront sweep of every test/rpms/*.yml is fragile and is exactly what let php’s failure slip through undetected for days during the F44 rebase — its own MR was fine, but a different package’s stale GPG key/repo reference broke shared test infrastructure php depended on. When you open the rebuild/update MR for any package, grep test/rpms/<pkg>.yml for the outgoing release string first.

See HUM-5184 for tracking a structural fix (single source of truth or a CI lint check) so this class of drift is caught automatically instead of relying on either of the above being remembered.

Automation racing a manual mass rebuild

The routine automated dist-git sync bot keeps running on its normal schedule regardless of an in-progress manual mass rebuild. If both touch the same packages in the same window, you’ll get a wave of merge-conflicted bot MRs that need manual cleanup — during F44 this was roughly 70 MRs at once. Decide before starting Step 4 (see “Pre-flight checks” above) whether to pause routine sync automation for the duration of the mass rebuild, or accept the cleanup cost.

External CI capacity constraints

Large rebuild waves can coincide with unrelated capacity crunches on shared infrastructure (e.g. Testing Farm). These are outside Hummingbird’s control but can stall a batch for a day; check external status pages when a batch is unexpectedly slow (for reference, the F44 rebase hit https://status.testing-farm.io/issues/2026-07-14-big-queue-in-red-hat-ranch/).

python3dist / rawhide-tracked toolchain-adjacent packages

If any packages central to a specific language ecosystem’s “what version is default” (e.g. python-rpm-macros) are tracking rawhide rather than a pinned branch, a routine automated sync during this window can silently pull in an experimental version bump (e.g. Fedora bootstrapping the next language version) that corrupts auto-generated package metadata across dozens of downstream packages without any build failing immediately. See HUM-4988 and HUM-4989 for the full incident report and proposed guardrails from the F44 rebase. Decide during “Pre-flight checks” above whether any “default version” declaration packages need the same branch-pinning treatment as the interpreter itself before you start.

Packages that intentionally stay behind

Not every package needs to move to the new branch. Document (in modification_reason and/or the epic tracking the rebase) any package that’s staying pinned to the old branch, and why — otherwise this looks like unfinished work to the next person. Reasons seen so far:

• FIPS validation: the package carries a modification tied to FIPS certification, and moving to newer upstream content risks invalidating that certification. The package is still rebuilt against the new toolchain — it just doesn’t pick up newer Fedora source content.
• Package no longer exists upstream: Fedora sometimes drops a package entirely in a new release (e.g. an older LTS language runtime superseded by a newer one). If Hummingbird still needs to carry it, it has no newer branch to rebase to and simply stays on its last available branch, rebuilt against the new toolchain.

Reference: commands used during the F44 rebase

These are shown verbatim from the F44 rebase for concreteness. When reusing them for the next rebase (e.g. F45), remember to substitute the actual target release everywhere F44 appears below — including the exclusion list of already-rebuilt packages, which will differ each time.

# Preview a rebuild plan across all packages
./ci/rebuild_multi_mr.sh --dry-run

# Submit rebuild MRs in controlled batches, auto-merge on green
./ci/rebuild_multi_mr.sh --max-updates=50

# Rebuild everything except a known set of already-handled packages, using the
# --exclude flag (HUM-5183, landed after the F44 rebase — see Step 4 above)
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm,annobin,binutils,fedora-repos,python3.13,python-gitlab --reason "F44 toolchain rebuild"

During the F44 rebase itself, --exclude didn’t exist yet, so the equivalent exclusion list had to be hand-rolled:

./ci/dist_git.py rebuild $(./ci/dist_git.py list | grep -v -E '^(glibc|gcc|llvm|annobin|binutils|fedora-repos|python3.13|python-gitlab)$') --reason "F44 toolchain rebuild"

Related tickets

• HUM-2018 — F44 buildroot rebase epic (full history)
• HUM-3369 / HUM-3370 / HUM-4536 / HUM-4537 — Steps 1-4, each with a detailed what/why/fix writeup
• HUM-4988 / HUM-4989 — python-rpm-macros/python3dist regression incident and guardrails
• HUM-5182 — bump_release() double-suffix bug
• HUM-5183 — dist_git.py rebuild --exclude option (implemented)
• HUM-5184 — generalize the testing-infra Fedora-version drift fix