Rebuilding Packages

AI Agent Note: When asked to rebuild packages, use the rebuild command: ./ci/dist_git.py rebuild <package> --reason "<reason>". Always ask the user for a ticket link or explanation first to use as the reason. If the command fails with “uses macros in Release field”, see the “Packages requiring manual rebuild” section below for instructions.

Overview

This document covers two scenarios for triggering a new package build:

1. No-change rebuild: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes).

2. Backporting a patch: Add an upstream patch that hasn’t yet landed in Fedora to fast-track a fix or feature.

Both scenarios use the .N release suffix pattern to ensure our builds sort higher than the upstream Fedora release while remaining lower than the next upstream version.

No-Change Rebuild

Using the rebuild command (recommended)

The rebuild command automates the Release field bump:

# Rebuild a single package
./ci/dist_git.py rebuild <package> --reason "<reason>"

# Rebuild multiple packages (one commit per package)
./ci/dist_git.py rebuild <package1> <package2> ... --reason "<reason>"

# Rebuild all packages (one commit per package)
./ci/dist_git.py rebuild --all --reason "<reason>"

Examples:

# Single package
./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes"

# Multiple packages
./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds"

# All packages (useful after toolchain updates)
./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15"

The command:

• Automatically bumps the Release field using the .N suffix pattern
• Handles %autorelease by resolving and replacing with explicit values
• Preserves macros in the Release field (e.g., %{revision})
• Creates a properly formatted commit message (one per package with --all)
• Does not mark the package as modified (release-only changes are ephemeral)

Supports --dry-run to preview changes without committing:

./ci/dist_git.py --dry-run rebuild <package> --reason "test"

Creating MRs for rebuild commits

After creating rebuild commits locally, use rebuild_multi_mr.sh to push each commit as its own merge request (one MR per package, auto-merge enabled):

./ci/rebuild_multi_mr.sh

By default the script compares against origin/main. For local development, use --base to point at a different ref:

# Use local main branch as base (useful when origin/main is not up to date)
./ci/rebuild_multi_mr.sh --base main

# Use a specific commit SHA as base
./ci/rebuild_multi_mr.sh --base abc1234

# Preview what would be created without pushing
./ci/rebuild_multi_mr.sh --dry-run

# Limit to at most N MRs
./ci/rebuild_multi_mr.sh --max-updates=5

The script:

• Creates a chore/rebuild-{package} branch per commit and pushes it
• Titles each MR chore(rpms): Rebuild {package}: {reason}
• Enables auto-merge on all rebuild MRs
• Leaves the current branch untouched
• Skips branches that already exist on the remote (idempotent)
• Only processes commits whose message matches Rebuild {package}: {reason}; other commits in the range are silently skipped

Full workflow example:

# 1. Create the rebuild commits
./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update"

# 2. Preview the MRs that would be created
./ci/rebuild_multi_mr.sh --base main --dry-run

# 3. Create the MRs
./ci/rebuild_multi_mr.sh --base main

Packages requiring manual rebuild

Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file.

Macro indirection patterns

These packages define the Release field using a macro, where the macro itself contains %{?dist}. The rebuild command cannot detect or manipulate these without expanding all macros, which would break the macro system.

nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):

%{load:%{_sourcedir}/nodejs.srpm.macros}
%nodejs_define_version node 1:25.8.2-%{autorelease} -p
...
Release: %{node_release}

The %{node_release} macro is defined by an external macro system loaded from nodejs.srpm.macros. The release component is embedded in the version definition.

How to rebuild: Edit the %nodejs_define_version node line to bump the release component (e.g., change -%{autorelease} to -1.1 or increment existing .N).

kernel-headers:

%define specrelease 59%{?buildid}%{?dist}
...
Release: %{specrelease}

How to rebuild: Edit the %define specrelease line to add/increment the .N suffix before %{?buildid}:

%define specrelease 59.1%{?buildid}%{?dist}

krb5:

%global krb5_release 4%{?dist}
...
Release: %{krb5_release}

How to rebuild: Edit the %global krb5_release line to add/increment the .N suffix:

%global krb5_release 4.1%{?dist}

Why these can’t be automated

The rebuild command can handle:

• ✅ Simple numeric: Release: 5%{?dist}
• ✅ Macros ending with dist: Release: %{baserelease}%{?dist} (e.g., rpm, gcc)
• ✅ Complex macros with dist: Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist}
• ✅ Content after dist: Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}} (e.g., unbound)

The rebuild command cannot handle:

• ❌ Macros without %{?dist}: Release: %{node_release}
• ❌ Macros where dist is inside the macro definition: %{krb5_release} contains %{?dist}

This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM’s full macro parser.

Manual rebuild process

If you need to rebuild manually or the automated command doesn’t work for your use case, follow these steps:

1. Identify the package to rebuild

Identify the source package name and locate its spec file in rpms/<package>/<package>.spec.

If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name:

podman run --rm quay.io/hummingbird-ci/builder:latest-hatchling \
  dnf5 repoquery --queryformat '%{SOURCERPM}' <binary-package> 2>/dev/null

Example: ncurses-libs-6.5-8.20250614.hum1 -> SRPM ncurses-6.5-8.20250614.hum1.src.rpm -> spec file at rpms/ncurses/ncurses.spec

2. Determine the Release bump pattern

The .N bump suffix must always appear immediately before %{?dist}. The %{?dist} suffix should always be the final component since it identifies the build environment.

For %autorelease, first resolve its value using rpmspec, then replace with the resolved value plus .1. In the Hummingbird monorepo, %autorelease always evaluates to 1.

Note: If the Release field is missing %{?dist} entirely or looks unusual (e.g., 1build1 instead of 1.1%{?dist}), flag this to the user for resolution. Check the git history to understand the original value:

git log -p -S "Release:" -- rpms/<package>/<package>.spec

This helps determine the correct fix when a previous bump was malformed.

3. Modify the spec file

Use sed to edit only the Release: line, avoiding any unintended whitespace changes that text editors may introduce:

sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms/<package>/<package>.spec

Verify the change with git diff before committing:

git diff rpms/<package>/<package>.spec

The diff should show only the Release line change:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

Important: Only modify the Release line. Do not introduce any other changes such as whitespace fixes or trailing newline modifications. If the diff shows additional changes, reset and retry with sed.

4. Verify the bump is correct

Use rpm --eval to confirm the new release sorts higher than the original:

# Returns -1 if first < second (correct), 1 if first > second (wrong)
rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}'
# Expected output: -1

5. Commit the change

Use this commit message format:

Rebuild <package>: <reason>

<ticket link or explanation>

Example:

Rebuild ncurses: published multiple times with different hashes

HUM-1234

6. Verify the commit

After committing, verify only the Release line was changed:

git show --stat HEAD

Expected output should show exactly 1 insertion and 1 deletion:

 rpms/<package>/<package>.spec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

If the commit shows more changes, amend or reset and redo the change using sed.

Important notes about rebuilds

Modification status

Rebuilds do not change a package’s modification_status. Release-only changes are ephemeral and don’t affect whether a package is considered modified vs clean:

• Release fields are temporary: When updating from Fedora later, the Release gets replaced anyway
• Merges normalize Release: During updates, Release lines are normalized to avoid conflicts
• No source changes: Rebuilds don’t modify sources, patches, or spec logic

The automation already ignores Release-only changes, so automatic Fedora updates will continue normally after a rebuild.

If you want to explicitly prevent automatic updates (e.g., you’re investigating an issue), you can manually mark the package as modified:

./ci/dist_git.py mark-modified <package> --modified --reason "Investigating build issue"

Note: This will block automatic Fedora updates until you mark it clean again.

Backporting a Patch

Use this workflow when you need to fast-track an upstream fix or feature that hasn’t yet been released in Fedora.

1. Obtain the patch

Fetch the patch from the upstream repository. For GitHub PRs, append .patch to the PR URL:

curl -L https://github.com/<org>/<repo>/pull/<number>.patch \
  > rpms/<package>/<NNNN>-<short-description>.patch

Name the patch file with a numeric prefix matching the next available PatchN: slot in the spec file (e.g., 0004-fix-foo.patch if Patch1-3 already exist).

2. Add the patch to the spec file

Add a PatchN: declaration after the existing patches:

Patch3:         0003-existing-patch.patch
Patch4:         0004-fix-foo.patch

The patch will be applied automatically if the spec uses %autosetup -p1. If the spec uses explicit %patchN macros, add the corresponding apply line in the %prep section.

3. Bump the Release

Follow the same .N suffix pattern as no-change rebuilds:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

4. Add a changelog entry

Add a new changelog entry at the top of the %changelog section:

%changelog
* Wed Jan 08 2026 Your Name <email@example.com> - 1.2.3-3.1
- Backport upstream PR#1234: short description of the fix

* Mon Jan 06 2026 Previous Maintainer <prev@example.com> - 1.2.3-3
- Previous changelog entry

5. Commit the change

Use this commit message format:

<package>: backport <short description>

Upstream: <link to PR or commit>
<ticket link if applicable>

Example:

dnf5: backport reproducible build sorting fix

Upstream: https://github.com/rpm-software-management/dnf5/pull/2522

6. Mark package as modified

Mark the package as modified to prevent automatic Fedora updates from overwriting your backport:

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Backport fix for <issue description>"

Example:

./ci/dist_git.py mark-modified dnf5 --modified \
  --reason "Backport reproducible build sorting fix from upstream PR#2522"

This ensures the package won’t be automatically updated from Fedora until the backported patch lands upstream and you explicitly mark it clean again.

7. Test the build locally (optional)

Build the package locally to verify the patch applies cleanly:

./ci/build_rpms.sh <package>

Built RPMs will be in builds/<package>/RPMS/.

Related Operations

• Excluding Packages from Images - temporarily block faulty packages in container builds