This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Using a Project Hummingbird container image

1.1 - Using a Project Hummingbird container image

Project Hummingbird builds a collection of minimal, hardened, and secure container images with a significantly reduced attack surface. This strong focus on security combined with a highly automated update workflow aims to minimize CVE counts, targeting near-zero vulnerabilities. All images support amd64 and arm64 architectures.

Quick Start

All images are available from the Red Hat Hardened Images registry and work directly with Podman, Docker, or Kubernetes:

# Run a command directly with the curl image
podman run registry.access.redhat.com/hi/curl:latest -v https://example.com

# Start a PostgreSQL database
podman run -e POSTGRES_PASSWORD=mysecret -p 5432:5432 registry.access.redhat.com/hi/postgresql:latest

Example container build using the Hummingbird Python image:

FROM registry.access.redhat.com/hi/python:latest
COPY myapp.py /app/
WORKDIR /app
CMD ["python3", "myapp.py"]

Documentation for each image is available on the Red Hat Hardened Images Catalog.

Contents

Available Images

Available images span language runtimes, databases, web servers, CLI tools, and base runtime images.

Browse the complete catalog at the Red Hat Hardened Images Catalog.

Hardened for Security

Hummingbird applies several measures to harden container images:

  • Minimal Software Footprint: Images include only essential packages required for the workload, significantly reducing the attack surface and CVE count.
  • Rapid Update Deployment: Package updates ship as quickly as possible, ensuring fixes are consumed early.
  • Non-Root User Default: Containers default to a non-root user (UID 65532) where technically possible, reducing privileges within the container.
  • Hermetic Build Environment: All containers are built in a hermetic environment without network access, preventing unintended package drift and maintaining full control over software versions.
  • Distroless Security: Shipping only what is strictly necessary for the workload reduces the attack surface and makes certain types of attacks impossible.
  • FIPS-Validated Cryptography: FIPS variants provide NIST-validated cryptographic modules for compliance-sensitive workloads.

Distroless Containers

Hummingbird builds distroless containers — images that do not ship with a package manager and most do not even provide a shell.

The distroless design makes the bundled application the container’s entrypoint, offering a streamlined experience. For example, with the curl image, arguments can be passed directly: podman run registry.access.redhat.com/hi/curl:latest -v https://www.redhat.com/en.

Purpose-built containers reduce the burden on users. Instead of building custom container images and managing their vulnerabilities, a Hummingbird image with the needed application avoids CVE management overhead entirely.

Understanding Image Variants

Hummingbird provides different variants to support various use cases while maintaining security by default.

Default Variant (:latest)

  • Distroless: no package manager, no shell
  • Minimal attack surface
  • Recommended for production
  • Example: registry.access.redhat.com/hi/python:latest

Builder Variant (:latest-builder)

  • Includes dnf package manager and bash
  • For installing additional dependencies
  • Intended for multi-stage builds and development
  • Example: registry.access.redhat.com/hi/python:latest-builder

Some images provide additional variants such as PHP’s FPM variant.

FIPS Variants (:latest-fips, :latest-fips-builder)

FIPS variants ship FIPS 140-3 validated cryptographic modules for workloads that require certified cryptography. Validation applies only when running on RHEL systems installed in FIPS mode.

Two cryptographic stacks are available depending on the image:

Stack Images FIPS packages
OpenSSL Go, Nginx, Node.js, Python, Ruby, curl openssl-config-fips, crypto-policies-config-fips
NSS OpenJDK nss-softokn-fips, nss-softokn-freebl-fips

FIPS is available in both distroless (:latest-fips) and builder (:latest-fips-builder) variants. FIPS variants are available for the Hummingbird distro only (not Rawhide). For implementation details, see the FIPS Variant Guide.

Support Levels

Images are published under two support levels:

  • Red Hat supported — the majority of images. Built, tested, and maintained by the Hummingbird team with full CI/CD coverage, security updates, and vulnerability tracking. Available at registry.access.redhat.com/hi/ with official Red Hat signing keys — this is the recommended registry for all users.
  • Community — images where the upstream project does not follow standard open-source release practices (e.g., MinIO) or where the image is experimental (e.g., bootc-os). Built and tested in the same infrastructure but without the same support commitments. Community also serves as a staging ground for images being evaluated for promotion to Red Hat supported status. Published to quay.io/hummingbird-community/.

Registry Organizations

Registry Description
registry.access.redhat.com/hi/ Red Hat Hardened Container Images (recommended)
quay.io/hummingbird/ Mirror of Red Hat supported images without official signing
quay.io/hummingbird-community/ Community-supported images (e.g., MinIO, bootc-os)
quay.io/hummingbird-rawhide/ Upstream Fedora Rawhide packages directly
quay.io/hummingbird-ci/ Build infrastructure images

Tool images used by pipelines (e.g., quay.io/hummingbird-ci/gitlab-ci) are published from the tools repository.

Tagging Strategy

Images follow a version-based tagging scheme:

  • :latest — most recent version (may change)
  • :<version> — specific version (e.g., :3.11, :16)
  • :<version>-builder — builder variant of a specific version
  • :<version>-fips — FIPS variant of a specific version
  • :<version>-fips-builder — FIPS builder variant of a specific version

Use versioned tags in production for reproducible builds. The :latest tag is convenient for development but may introduce unexpected changes.

For best practices on tag selection and digest pinning, see How to name, version, and reference container images.

Multi-version images:

For languages with multiple supported version families (Python, Node.js, .NET, etc.), separate images exist for each major version:

  • registry.access.redhat.com/hi/python:latest — latest Python (currently 3.14)
  • registry.access.redhat.com/hi/python:3.11 — Python 3.11.x (stays on 3.11)
  • registry.access.redhat.com/hi/python:3.13 — Python 3.13.x (stays on 3.13)

Version constraints prevent these images from accidentally upgrading to incompatible versions. See version constraints for details.

Multi-Stage Build Pattern

The recommended pattern for compiled languages:

# Build stage: use builder variant to install dependencies and compile
FROM registry.access.redhat.com/hi/go:latest-builder AS builder
RUN dnf install -y <build dependencies>
COPY . /src
WORKDIR /src
RUN go build -o /app .

# Runtime stage: use minimal base image for the compiled binary
FROM registry.access.redhat.com/hi/core-runtime:latest
COPY --from=builder /app /app
ENTRYPOINT ["/app"]

This approach provides build-time flexibility while maintaining a minimal production image.

Image Verification

Hummingbird images are signed and can be verified using cosign:

cosign verify \
  --key "https://security.access.redhat.com/data/63405576.txt" \
  --insecure-ignore-tlog \
  registry.access.redhat.com/hi/<image>:<tag>

Vulnerability Scanning

Use Syft and Grype to locally inspect and scan images. For details on how production SBOMs are generated, see Software Bill of Materials.

To scan an image for vulnerabilities:

grype registry.access.redhat.com/hi/<image>:<tag>

For current vulnerability information across all variants and versions, see the Red Hat Hardened Images Catalog.

Sharing Host Data

By default, containers do not have access to host filesystem content. Volume mounts must be added explicitly:

podman run -v /path/on/host:/path/in/container registry.access.redhat.com/hi/curl:latest ...

SELinux Relabeling

On systems with SELinux enabled (such as Fedora and RHEL), mounted volumes must be relabeled to allow container access. Use the :z or :Z option:

podman run -v /path/on/host:/path/in/container:z registry.access.redhat.com/hi/curl:latest ...

The :z option relabels content to be accessible by any container sharing the mount. The :Z option relabels content to be uniquely accessible to one container.

File Permissions

Most Hummingbird images default to a non-root user. When mounting host directories that the container needs to write to, either make the directory world-writable (within a private directory):

mkdir -m 700 /path/on/host
mkdir -m 777 /path/on/host/mnt
podman run -v /path/on/host/mnt:/path/in/container:z ...

Or run as the root user (recommended only with rootless Podman, where the root user in the container maps to the calling user on the host):

podman run --user root -v /path/on/host:/path/in/container:z ...

For read-only access, no permission changes are needed as long as files are world-readable.

Custom CA Certificates

Hummingbird images can be configured to trust custom Certificate Authority (CA) certificates for TLS connections. The approach depends on the image’s TLS stack.

OpenSSL-Based Images (curl, Nginx, Python, etc.)

Mount a CA bundle to replace or extend the system trust store:

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
  registry.access.redhat.com/hi/curl:latest https://internal-server/

For details on building derived images with merged trust stores, see the Custom CA (OpenSSL) guide.

Java-Based Images (OpenJDK, Tomcat)

Java uses its own PKCS12 truststore format. A custom truststore can be created using keytool and mounted at runtime:

podman run --rm \
  -v /path/to/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z \
  registry.access.redhat.com/hi/openjdk:latest java -jar app.jar

For step-by-step truststore creation, see the Custom CA (Java) guide.

Compatibility

Hummingbird images are designed for compatibility with popular images from Docker Hub, Red Hat UBI, and other registries, enabling straightforward migration of existing workloads.

Key difference: Hummingbird images default to a non-root user (UID 65532) where technically possible, while most traditional images run as root. This may require adjusting file permissions on mounted volumes:

# Ensure correct ownership for mounted data
chown -R 65532:65532 /path/to/data

For detailed comparisons (environment variables, ports, sizes, default users), see the compatibility report, also available in machine-readable form.

Reproducible Builds

Reproducible builds ensure that the same inputs and build environment always produce bit-for-bit identical output. This allows independent verification that a published image corresponds to its claimed source materials, making it straightforward to detect if malware was injected during the build process.

Hummingbird images are fully reproducible. Given the signed SLSA provenance attestation that accompanies each image, anyone can rebuild the image from its inputs (this git repo and the RPMs it describes) to verify it matches the published version exactly.

For further background, see Reproducible builds: Project Hummingbird.

Verifying Reproducibility

The cosign and podman tools are required. First, download and verify the SLSA provenance attestation using cosign (see Image Verification for key details):

IMAGE=registry.access.redhat.com/hi/curl:latest

cosign verify-attestation --key "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/ci/key.pub?ref_type=heads" --insecure-ignore-tlog \
  --type slsaprovenance $IMAGE > attestation.json

Then, feed the attestation into the rebuild tool (capture the image ID for comparison):

iid=$(podman run -i --rm --privileged -v /mnt \
  quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json)

To verify reproducibility, pull the published image and compare image IDs:

iid2=$(podman pull $IMAGE)
[ $iid = $iid2 ] && echo "Identical"

To keep the rebuilt image for further inspection, use the DUMP_OCIARCHIVE environment variable:

podman run -i --rm --privileged -e DUMP_OCIARCHIVE=1 -v /mnt \
  quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json | podman load

Content-Based Layers

Most container images have layers that mirror the Containerfile structure: each RUN or COPY instruction creates a layer. A single package update can invalidate a layer much larger than the package itself, requiring clients to re-pull content that has not changed.

Hummingbird images use chunkah to split images into content-based layers. Instead of layers reflecting the build process, files are grouped by the packages they belong to. This benefits both the network level (only new layers need downloading) and the storage level (common layers are stored once on disk).

Inspect which packages live in which layers using podman history (or docker history):

podman pull registry.access.redhat.com/hi/jq
podman history registry.access.redhat.com/hi/jq

Source Containers

The source code for all Hummingbird containers is available as source containers. A source container includes RPM and non-RPM content shipped in an image, pushed alongside each available tag with the -source suffix.

Inspect source container contents with skopeo:

IMAGE=registry.access.redhat.com/hi/curl:8.19.0-source

cd $(mktemp -d)
mkdir source
skopeo copy --override-os=linux docker://$IMAGE dir:source

cd source
mv version manifest.json ..
for f in $(ls); do tar xvf $f; done

Roadmap

  • Image catalog expansion: The catalog of available images will continue to expand significantly.
  • Image size optimization: Container-specific package configurations and dependency management to further reduce image sizes.
  • Enhanced CVE and SBOM tooling: Improved vulnerability metadata and scanning integration, including methods like cargo-auditable for Rust supply chain transparency.
  • Bootable containers: Expanding support for image-based OS deployments and updates.

Relationship to Fedora

Hummingbird builds and maintains its own RPM packages independently. Spec files are derived from Fedora but built in Hummingbird’s own infrastructure, allowing project-specific modifications and optimizations. Key differences from Fedora:

  • Use of monorepos — one monorepo for packages and one for containers, simplifying management and automation.
  • Heavy reliance on CI/CD — dependency management, testing, and releasing are automated.
  • Secure supply chain requirements — container images and packages are built using Konflux, providing SLSA level 3 compliance.
  • Greater emphasis on upstream tracking — software closely tracks upstream projects’ lifecycles, including parallel stable versions where offered.

There is a natural affinity between the two projects, and opportunities exist to contribute work from Hummingbird back into Fedora.

Contributing

Interested in contributing? The following resources cover the development workflow:

License

This project is licensed under the MIT License - see the LICENSE.txt file for details.

1.2 - Custom CA Certificates (OpenSSL)

Overview

You can configure OpenSSL-based container images (curl, nginx, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections.

There are two approaches for custom CAs depending on your needs:

  1. Custom bundle file - Replace system CAs entirely with your own bundle
  2. Derived image - Build a new image with merged trust store

Approach 1: Custom Bundle File

Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.

Mount your CA bundle to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:

Custom bundle with Podman

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
  quay.io/hummingbird/curl https://your-server/

Custom bundle with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-bundle
data:
  tls-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    ... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: quay.io/hummingbird/curl
    volumeMounts:
    - name: custom-ca
      mountPath: /etc/pki/ca-trust/extracted/pem
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca-bundle

Custom bundle on OpenShift with CA injection

On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.

Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:

# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  labels:
    config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: quay.io/hummingbird/curl
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      subPath: ca-bundle.crt
      readOnly: true
  volumes:
  - name: trusted-ca
    configMap:
      name: trusted-ca

The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (tls-ca-bundle.pem) that OpenSSL expects.

See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.

Approach 2: Derived Image

Use this approach when you need to trust both your custom CA and the image’s builtin default CAs. This creates a new image with a merged trust store with the trust anchor command.

Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the curl image:

FROM quay.io/hummingbird/curl:latest-builder AS builder

USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt

FROM quay.io/hummingbird/curl:latest
COPY --from=builder /etc/pki /etc/pki

Build and use your custom image:

podman build -t my-curl-with-ca .
podman run --rm my-curl-with-ca https://your-server/

1.3 - Custom CA Certificates (Java)

Overview

You can configure Java-based container images (openjdk, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections. Java uses its own truststore format (PKCS12) rather than PEM files.

Java keystores require a password for integrity checking. The system truststore uses the standard password changeit, which is a well-known default used for tamper detection rather than confidentiality.

Volume Mount Approach

Create a custom truststore using the keytool command. You can either trust only your specific CAs (for internal-only applications) or merge with the default public CAs (typical for applications connecting to both internal and external services).

CERTDIR=$(mktemp -d)  # or use a persistent directory for repeated runs
chmod 777 "$CERTDIR"  # to work with non-privileged containers

# OPTIONAL: Copy the default Java truststore to merge with public CAs
# Skip this step to trust ONLY your custom CAs
podman run --rm -v "${CERTDIR}:/work:Z" \
  quay.io/hummingbird/openjdk:latest \
  cp /etc/pki/ca-trust/extracted/java/cacerts /work/cacerts

# Import your custom CA
podman run --rm -v "${CERTDIR}:/work:Z" \
  quay.io/hummingbird/openjdk:latest \
  keytool -import -noprompt -trustcacerts -alias my-custom-ca \
    -file /work/my-ca.crt \
    -keystore /work/cacerts \
    -storepass changeit -storetype PKCS12 \
    -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE

The -storetype PKCS12 and -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE flags are required for compatibility with FIPS images. The system truststore is PKCS12 format with unencrypted certificate entries (generated by p11-kit). Without these flags, keytool defaults to PBE-encrypted certificate bags that the FIPS security provider cannot decrypt.

Podman Example

Mount the custom truststore to the system location:

podman run --rm \
  -v "${CERTDIR}/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z" \
  quay.io/hummingbird/openjdk:latest \
  java -jar myapp.jar

Kubernetes Example

In Kubernetes, use an init container to create the truststore, then share it via an emptyDir volume:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
  initContainers:
  - name: prepare-truststore
    image: quay.io/hummingbird/openjdk:latest
    command: ["sh", "-c"]
    args:
    - |
      # OPTIONAL: Copy the default truststore to merge with public CAs
      # Skip this line to trust ONLY your custom CAs
      cp /etc/pki/ca-trust/extracted/java/cacerts /truststore/cacerts
      chmod 666 /truststore/cacerts
      keytool -import -noprompt -trustcacerts -alias custom-ca \
        -file /ca/ca.crt -keystore /truststore/cacerts \
        -storepass changeit -storetype PKCS12 \
        -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE
    volumeMounts:
    - name: custom-ca
      mountPath: /ca
    - name: truststore
      mountPath: /truststore
  containers:
  - name: app
    image: quay.io/hummingbird/openjdk:latest
    volumeMounts:
    - name: truststore
      mountPath: /etc/pki/ca-trust/extracted/java
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca
  - name: truststore
    emptyDir: {}

For the trust-only-specific-CAs approach, omit the cp command and create the keystore directly with keytool -import.

Derived Image Approach

Use this approach when it’s more practical to bake the certificates into the image instead of generating the truststore on each container startup. This creates a new image with a merged trust store with the trust anchor command.

Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the openjdk image:

FROM quay.io/hummingbird/openjdk:latest-builder AS builder

USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt

FROM quay.io/hummingbird/openjdk:latest
COPY --from=builder /etc/pki /etc/pki

Build and use your custom image:

podman build -t my-openjdk-with-ca .
podman run --rm my-openjdk-with-ca java -jar myapp.jar

1.4 - Custom CA Certificates (Python)

Overview

You can configure Python container images to trust custom Certificate Authority (CA) certificates for TLS connections. Python libraries have different trust store behaviors compared to other OpenSSL-based images:

  • urllib (built-in) reads CAs from both of these locations:
    1. /etc/pki/tls/cert.pem - the main CA bundle file
    2. /etc/pki/tls/certs/ - directory with hashed certificate symlinks
  • requests (third-party) ignores /etc/pki entirely and requires the REQUESTS_CA_BUNDLE environment variable to use system CAs

Note: The requests library is not included in the standard Python image. To use it, you need to build a derived image:

FROM quay.io/hummingbird/python:latest
RUN ["pip3", "install", "requests"]

podman build -t localhost/my-python-with-requests .

There are two approaches for custom CAs depending on your needs:

  1. Custom bundle file - Replace system CAs entirely with your own bundle
  2. Add custom CAs - Merge your custom CAs with the system defaults

Approach 1: Custom Bundle File

Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.

Mount your CA bundle to /etc/pki/tls/cert.pem, set $REQUESTS_CA_BUNDLE to that, and block the hashed directory with an empty directory:

Custom bundle with Podman

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/tls/cert.pem:ro,Z \
  --tmpfs /etc/pki/tls/certs:notmpcopyup \
  -e REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem \
  localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
import requests
print(requests.get('https://your-server/').status_code)
"

Note: Podman requires the :notmpcopyup option as mounting tmpfs behaves like an overlay. Docker’s tmpfs creates an empty directory by default, and does not know that option, so drop that option with Docker.

Custom bundle with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-bundle
data:
  cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    ... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use quay.io/hummingbird/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/cert.pem
    volumeMounts:
    - name: custom-ca
      mountPath: /etc/pki/tls/cert.pem
      subPath: cert.pem
      readOnly: true
    - name: empty-certs
      mountPath: /etc/pki/tls/certs
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca-bundle
  - name: empty-certs
    emptyDir: {}

Custom bundle on OpenShift with CA injection

On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.

Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:

# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  labels:
    config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use quay.io/hummingbird/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/cert.pem
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/pki/tls/cert.pem
      subPath: ca-bundle.crt
      readOnly: true
    - name: empty-certs
      mountPath: /etc/pki/tls/certs
      readOnly: true
  volumes:
  - name: trusted-ca
    configMap:
      name: trusted-ca
  - name: empty-certs
    emptyDir: {}

The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (/etc/pki/tls/cert.pem) that Python expects.

See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.

Approach 2: Add Custom CAs

Use this when you want to merge your custom CAs with the system defaults.

Mount your CA certificate as an individual file in /etc/pki/tls/certs/ with a properly hashed filename. The filename must be the output of openssl x509 -noout -subject_hash -in ca.crt followed by .0. Also set $REQUESTS_CA_BUNDLE to that directory:

Custom CA file with Podman

# Generate the hashed filename
CA_HASH=$(openssl x509 -noout -subject_hash -in ca.crt)
echo "Hashed filename: ${CA_HASH}.0"

# Mount the CA certificate as an individual file in the certs directory
podman run --rm \
  -v ./ca.crt:/etc/pki/tls/certs/${CA_HASH}.0:ro,Z \
  -e REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ \
  localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
print(urllib.request.urlopen('https://google.com/').status)  # Default CAs still work
import requests
print(requests.get('https://your-server/').status_code)
print(requests.get('https://google.com/').status_code)  # Default CAs still work
"

Custom CA file with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use quay.io/hummingbird/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/certs/
    volumeMounts:
    - name: custom-ca
      # REQUIRED: replace example hash with: `openssl x509 -noout -subject_hash -in ca.crt` and append .0
      mountPath: /etc/pki/tls/certs/12abc456.0
      subPath: ca.crt
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca

2 - Contributing to a Project Hummingbird container image

Welcome to the Project Hummingbird contributor documentation. This section provides guides and references for contributing to container images.

Getting Started

New to contributing? Start here:

Guides

Step-by-step guides for common tasks:

Reference

Detailed reference documentation:

2.1 - Quickstart Guide

Get your first contribution done in 5 minutes

Welcome! This guide will help you make your first contribution to Project Hummingbird container images.

For more detailed information, see the full contributor documentation.

Prerequisites

  • Container tools: Podman or Docker
  • Build tools: buildah, make, git
  • Python: Python 3.11+ with dependencies from requirements.txt

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3 python3-pip
pip install -r requirements.txt

Quick Start Workflow

1. Fork and Clone

  1. Fork the repository on GitLab
  2. Clone your fork:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

2. Make Your Changes

Edit files in the images/<image-name>/ directory:

  • properties.yml - Image configuration
  • Containerfile.j2 - Container build template
  • tests-container.yml - Integration tests

3. Generate and Build

# Update dependent files
make

# Build the image
ci/build_images.sh <image-name>

4. Test Your Changes

# Run integration tests
ci/run_tests_container.sh <image-name>

# Run linters and checks
make check

5. Submit a Merge Request

  1. Create a branch: git checkout -b fix-nginx-config
  2. Commit your changes: git commit -am "Fix nginx configuration"
  3. Push to your fork: git push origin fix-nginx-config
  4. Open a merge request from your fork to the upstream repository on GitLab

Note for external contributors: If you’re not a project member, the CI pipeline won’t run automatically. Please ping one of the project maintainers in your merge request to trigger the build and test pipeline.

License

By contributing to this project, you agree that your contributions will be licensed under the MIT License. See LICENSE.txt for the full license text.

Common Tasks

Adding a New Image

# Create image directory
mkdir images/myapp

# Copy templates
cp images/Containerfile.j2 images/myapp/Containerfile.j2
cp images/properties.yml images/myapp/properties.yml

# Edit the template, configuration, and tests
vim images/myapp/properties.yml        # Configure packages and variants
vim images/myapp/Containerfile.j2      # Customize the build template
vim images/myapp/tests-container.yml   # Add integration tests

# Update dependent files
make

# Build and test
ci/build_images.sh myapp
ci/run_tests_container.sh myapp

Testing with Docker

# Automatic Docker-in-Docker setup
ci/build_images.sh --engine docker --setup <image-name>
ci/run_tests_container.sh --engine docker --setup <image-name>

Building Specific Variants

# Build only the rawhide builder variant
ci/build_images.sh nginx/rawhide/builder

# Test only the rawhide default variant
ci/run_tests_container.sh nginx/rawhide/default

# Build/test all variants for a specific distro
ci/build_images.sh nginx/rawhide
ci/run_tests_container.sh nginx/rawhide

Next Steps

Getting Help

2.2 - Development Workflow

Detailed development environment setup and contribution workflow

Prerequisites

Ensure the required tools are installed:

  • Container tools: Podman or Docker
  • Build tools: buildah, make, git
  • Python: Python 3 with PyYAML package

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3-pyyaml

macOS Setup

macOS requires bash 5+ and GNU command-line tools. Install via Homebrew:

brew install podman bash grep coreutils
podman machine init && podman machine start

# Add to ~/.zshrc for persistence
export PATH="/opt/homebrew/bin:/opt/homebrew/opt/grep/libexec/gnubin:/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"

Note: On Intel Macs, use /usr/local/ instead of /opt/homebrew/.

Repository Setup

Fork and Clone

  1. Fork the containers repository on GitLab
  2. Clone your fork with submodules:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
  1. Add the upstream remote:
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

Development Workflow

1. Create a Branch

Create a descriptive branch starting from upstream main:

git fetch upstream
git checkout -b feature-add-redis-image upstream/main

2. Make Changes

Edit the relevant files in images/<image-name>/:

  • properties.yml - Image configuration (packages, variants, tags)
  • Containerfile.j2 - Container build template
  • tests-container.yml - Container tests
  • tests-k8s.yml - K8s tests
  • README.md.j2 - Image documentation template

3. Generate Files

Regenerate derived files:

make

4. Build Locally

Build the image to verify changes:

ci/build_images.sh <image-name>

# Build specific variant
ci/build_images.sh <image-name>/rawhide/builder

# Build with verbose output
ci/build_images.sh --verbose <image-name>

5. Test Locally

Container tests:

ci/run_tests_container.sh <image-name>

# Test specific variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_container.sh --verbose <image-name>

By default, tests use Podman. To test with Docker:

# Automatic Docker-in-Docker setup
ci/run_tests_container.sh --engine docker --setup <image-name>

K8s tests:

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

See the Testing Guide for K8s test prerequisites and local development workflow.

Testing base images:

When modifying base images (like core-runtime), test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
ci/run_tests_k8s.sh --include-reverse-deps --context <context> core-runtime

Troubleshooting test failures:

Use --pause flag to inspect resources before cleanup:

ci/run_tests_container.sh --pause <image-name>
ci/run_tests_k8s.sh --pause --context <context> <image-name>

6. Run Linters

Ensure code quality:

make check

7. Commit and Push

Commit changes with a descriptive message and push to the fork:

git add .
git commit -m "feat: add Redis container image

- Add Redis 7.x image with default and builder variants
- Include basic health check tests
- Add compatibility notes for official Redis image"

git push origin feature-add-redis-image

8. Open a Merge Request

Open a merge request on GitLab:

  1. Go to the containers repository
  2. Select your fork and branch as the source, main as the target
  3. Fill in the description with what changed, why, and what tests were added

9. CI Pipeline

The CI pipeline automatically:

  • Builds images for multiple architectures
  • Runs integration tests
  • Checks for linting issues

Note for external contributors: If not a project member, the CI pipeline won’t run automatically. Ping one of the project maintainers in the merge request to trigger the pipeline.

Next Steps

2.3 - Resolving Merge Conflicts in Generated Files

How to resolve merge conflicts in generated files during rebases

Overview

During rebases of branches in this repository, you can commonly trigger merge conflicts in files that are generated by the various toolings and Makefile targets.

Rules

  • This documentation is only for generated files.
  • If there is not a section below for the conflicted file, you may need to check make help or search the Makefile for how to regenerate it (e.g., grep -r "target_file" Makefile).

Procedures for generated files

The following sections describe how to rebase given merge conflicts in specific paths.

Regenerating rpms.lock.yaml

When rebasing a branch, you may encounter conflicts in rpms.lock.yaml files. You should first delete the conflicting file, then run make container. Once in the container, rerun make and pass the relative path of the target. The file will be regenerated.

Below shows an interactive example. If attempting to do this non-interactively, remove the file as described and then run something like:

podman run --pull=newer --rm -v "$PWD:$PWD:z" -w "$PWD" \
    quay.io/hummingbird-ci/gitlab-ci:latest \
    make <target_file>

# 1. Delete the conflicted lockfile (replace with your actual conflicting path)
rm <path/to/conflicting/rpms.lock.yaml>

# 2. Enter the CI container
make container

# 3. Inside container, regenerate the lockfile
make <path/to/conflicting/rpms.lock.yaml>

# 4. Exit container
exit

# 5. Stage the regenerated file and continue the rebase
git add <path/to/conflicting/rpms.lock.yaml>
git rebase --continue

# 6. After rebase completes, validate the result
make -j$(nproc) check

Regenerating Containerfile

If the merge conflict is in one or more of the generated Containerfiles, then always accept the upstream (rebasing-onto) branch as the source of truth (i.e., git checkout --theirs -- <Containerfile> and git add <Containerfile>). When you have committed the upstream versions and are done with the rebase, you must then regenerate the Containerfiles so they pick up your changes. Containerfiles can be generated in two ways:

To rebuild Containerfiles for all images that are changed by your PR:

make -j$(nproc)

To rebuild a specific Containerfile, you can also use make and pass the path of the generated Containerfile as an argument.

make images/nginx/rawhide/default/Containerfile

After regenerating Containerfiles:

# Stage regenerated Containerfiles
git add <path/to/regenerated/Containerfile>

# If still in rebase, continue
git rebase --continue

# After rebase completes, validate the result
make -j$(nproc) check

2.4 - Adding New Images

Step-by-step guide for adding new container images to the project

This guide covers two scenarios:

  • Adding a new image - Creating an entirely new image (e.g., adding nginx for the first time)
  • Adding a versioned image - Adding a new version of an existing image (e.g., go-1-25 when go already exists)

Choose the appropriate section below based on your use case.

Adding Completely New Images

Follow these steps to add an entirely new container image to the project.

1. Create Image Directory

Create a directory for the new image:

mkdir images/your-service

2. Copy Base Templates

Copy the template files:

cp images/properties.yml images/your-service/properties.yml
cp images/Containerfile.j2 images/your-service/Containerfile.j2

3. Configure Image Properties

Edit images/your-service/properties.yml to configure:

  • Variants: Define variants (default: [default, builder])
  • Packages: List RPM packages needed in rpm_packages.all
  • Main package: Set main_package for version labeling
  • Repository and stream: Set repository and stream (both required). Add a YAML comment above stream documenting the reasoning (see Choosing a Stream Value)
  • Tags: Configure image tags
  • Release metadata: Set application_category, summary, description, and url
  • Support level: New images start with support_level: community

Example minimal configuration:

---
rpm_packages:
  all:
    - coreutils-single
    - your-main-package
  default:
    - ca-certificates-bundle  # smaller than ca-certificates; sufficient for non-builder variants
  builder:
    - ca-certificates         # full package needed for builder toolchains

main_package: your-main-package
repository: your-service
# <reasoning for stream choice> -- see "Choosing a Stream Value" section
stream: "latest"
# deprecated: true  # Optional: mark the image stream as deprecated
support_level: community
application_category: "Other"
summary: "One-line description of the image"
description: >-
  Description of the software, its purpose, and its main capabilities.
url: "https://upstream-project.example.com/"

tags:
  - value: latest

Prefer ca-certificates-bundle over ca-certificates in non-builder variants — it is smaller and sufficient for TLS trust roots. Builder variants need the full ca-certificates package for toolchain compatibility.

See the Image Configuration Reference for complete properties.yml options.

4. Customize Containerfile Template

Edit images/your-service/Containerfile.j2:

  • Add service-specific configuration (entrypoint, exposed ports, volumes, environment variables)
  • The template automatically includes packages from properties.yml via the {{ main_packages_arg() }} macro

The base template macros handle the standard container setup automatically:

Macro What it provides
{{ final_stage() }} ENV HOME=/tmp, WORKDIR /tmp, labels, OCI archive extraction
{{ set_user() }} USER 65532 (when user: default in properties.yml)

The user field in properties.yml defaults to default (UID 65532) if not specified. No passwd entry is needed — Linux allows running as any UID. See the Container Configuration reference for per-variant and custom user options.

When to override the defaults

The macros provide sensible defaults, but the image should match what the upstream image does. If the upstream image uses a custom user, WORKDIR, or HOME, the Hummingbird image should match for compatibility. Check the upstream image’s Dockerfile or inspect it with podman inspect before deciding.

When the defaults are sufficient, avoid duplicating what the macros already provide:

{# The macros already handle HOME=/tmp, WORKDIR=/tmp, USER 65532.     #}
{# Only add useradd/groupadd, WORKDIR, HOME, or USER when the         #}
{# upstream image does something different that users depend on.       #}

When the upstream image requires a specific user or working directory, set user: in properties.yml and add WORKDIR in Containerfile.j2 as needed. Use {{ default_user }} instead of hardcoding 65532.

Docker/registry auth in README templates

For images that interact with container registries, the auth config must be mounted to $HOME/.docker/config.json inside the container. Check what HOME is set to — for default-user images it is /tmp:

-v ${XDG_RUNTIME_DIR}/containers/auth.json:/tmp/.docker/config.json:ro,Z

Multi-Architecture Support

When Containerfiles or build scripts need to reference the target architecture (e.g., downloading architecture-specific binaries), use the correct naming convention for each context.

Docker Linux
Intel amd64 x86_64
ARM arm64 aarch64
Containerfile $TARGETARCH ARG variable $(arch) output
  • In Containerfiles or Go builds: Use the $TARGETARCH ARG variable provided by buildah (values: amd64, arm64)
  • In shell scripts or RPM builds: Use the arch or uname -m commands (values: x86_64, aarch64)

5. Add Comparison Report

Create the required images/your-service/report.yml. Its first comparison supplies the upstream image and documentation URL used by compatibility reports and generated README sections:

---
compare:
  - pull: docker.io/upstream/your-service:latest
    url: https://upstream-project.example.com/docs/

Add ignore entries only for intentional differences from the upstream image. See existing images with similar entrypoints and runtime behavior for examples.

6. Add Integration Tests

Create test files in images/your-service/:

  • tests-container.yml - Required container tests (Podman/Docker)
  • tests-k8s.yml - Optional Kubernetes tests for workload or cluster integration behavior

Container test example:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

K8s test example:

---
run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

See the Testing Guide for how to write and run tests.

7. Add Image Documentation

Create images/your-service/README.md.j2 with a high-level description, usage instructions, and compatibility notes. The template generates multiple README files for different target audiences (defined in images/variables.yml under readme_targets).

Important notes:

  • The readme_heading() macro automatically generates the heading with the correct product name
  • The readme_description() macro renders the image description from description in properties.yml; use it after the heading instead of hardcoding introductory text
  • Use {{ readme_targets[target].registry }} for registry references (not hardcoded quay.io/hummingbird)
  • The readme_available_tags() macro renders the variants and tags table
  • The readme_trailing_sections() macro appends compatibility, verification, vulnerability, and about sections

Example template:

{{ readme_heading() }}

{{ readme_description() }}

{{ readme_available_tags(variants, tags) }}

## Usage

podman run -d -p 8080:8080 {{ readme_targets[target].registry }}/your-service:latest

{{ readme_trailing_sections() }}

For more details, see README Generation.

8. Generate, Validate, and Submit

Follow the standard contribution workflow from the Development Workflow guide:

  • Generate files (make); commit all generated changes
  • Build locally (ci/build_images.sh your-service)
  • Run container tests (ci/run_tests_container.sh your-service)
  • Run Kubernetes tests when tests-k8s.yml exists (ci/run_tests_k8s.sh your-service)
  • Run linters (make check)
  • Commit and push changes
  • Open a merge request
  • Wait for CI pipeline

9. Add Konflux Components and Trigger Testing

When generated konflux-templates/rendered.yml changes, the merge request pipeline runs add_components_to_konflux automatically. This adds only the new components. After the job completes, add a /retest comment to the merge request so Konflux starts the new image pipelines. The manual deploy_components_to_konflux job redeploys all components and is not needed for normal image onboarding.

Adding Versioned Images

When adding a new version of an existing image (e.g., go-1-25, helm-4, or rabbitmq-4-3), follow this streamlined workflow.

1. Check Package Availability

Versioned packages may be available in different repositories:

  • Hummingbird repository - Custom packages built by the Hummingbird project (e.g., golang1.25, nodejs24, dotnet-sdk-10.0)
  • Fedora repositories - Standard Fedora packages (e.g., golang, python3)

The project uses two distros. Their current repository files are defined in images/variables.yml:

  • rawhide - Uses only Fedora Rawhide packages (fedora-44.repo)
  • hummingbird - Uses Fedora 44 + Hummingbird packages (fedora-44.repo + hummingbird.repo)

To check if a package is available in Hummingbird:

# Check the Hummingbird RPMs repository
ls ../rpms/rpms/ | grep <package-name>

Important: If the package is only available in the Hummingbird repository, add distros: [hummingbird] to properties.yml (see step 3).

2. Create Image Directory and Copy Files

Follow the naming pattern <language>-<version> with dashes (e.g., go-1-25, nodejs-24, dotnet-sdk-10-0):

# Create directory
mkdir images/go-1-25

# Copy templates from the closest existing version, not the generic templates
cp images/go-1-26/Containerfile.j2 images/go-1-25/
cp -P images/go-1-26/tests-container.yml images/go-1-25/
cp images/go-1-26/report.yml images/go-1-25/

Tests shared by a version family may instead live under ci/shared-tests/, with each image’s tests-container.yml and tests-k8s.yml symlinked to the shared files. Preserve the sibling image’s arrangement.

Only the version that publishes latest owns README.md.j2 and the generated README files for the shared repository. When a new version becomes the latest, move or copy the template to that version and remove it from the previous version. Older versions without latest do not need a README template. README files are generated from a template in the same image directory; they are not shared automatically.

3. Configure properties.yml

Create images/go-1-25/properties.yml with the versioned package:

---
main_package: golang1.25
repository: go
# endoflife.date: parallel at major.minor (Docker Hub + Wolfi)
stream: "1.25"
application_category: "Programming Languages & Runtimes"
summary: "Statically-compiled language with concurrency and a large standard library"
description: >-
  Statically typed, compiled programming language with garbage collection,
  memory safety, and a large standard library. It includes the complete Go
  toolchain.
url: "https://go.dev"
distros:
  - hummingbird  # Only if package is unavailable in Fedora
user: root
version_constraints: '1.25.*'
tags:
  # NOTE: Do NOT include 'latest' or major-version tags for older versions
  # Only the newest version (e.g., go-1-26) should have those tags
  - value: '{{ package_major_minor_version("golang1.25") }}'
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'
    label: org.opencontainers.image.version
rpm_packages:
  all:
    - coreutils-single
    - git-core
    - glibc-devel
    - golang1.25
    - make

Key differences from a new image:

  • Set main_package to the versioned package name
  • Set repository to match the base image family (for registry organization)
  • Set stream to the version this image tracks (e.g., "1.25" for go-1-25), with a YAML comment above it documenting the reasoning (copy from an existing sibling image)
  • Set version_constraints so lockfile updates cannot cross the image’s version boundary
  • Copy application_category, summary, description, and url from the sibling image, then update any version-specific text
  • Add distros: [hummingbird] if the package is only in the Hummingbird repository
  • Preserve sibling settings such as variants, FIPS packages, user, and support level unless the new upstream version requires a deliberate change
  • Tag strategy for multiple versions:
    • Only the newest version includes latest
    • Include a major-version tag on each stream when it is unique (e.g., Helm 3 and Helm 4)
    • If several streams share a major version (e.g., Go 1.25 and 1.26), only the newest includes that shared major-version tag
    • Every version includes its non-conflicting stream tag and full version tag

4. Update Containerfile Template

Edit images/go-1-25/Containerfile.j2 to reference the versioned package in any environment variables or version-specific commands:

 ENV GOPATH=/go \
     GOTOOLCHAIN=local \
-    GOLANG_VERSION={{ package_version('golang') }} \
+    GOLANG_VERSION={{ package_version('golang1.25') }} \
     PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

5. Update Comparison Images

Edit images/go-1-25/report.yml to compare against version-specific upstream images:

compare:
  - pull: docker.io/golang:1.25
    url: https://hub.docker.com/_/golang

  - pull: cgr.dev/chainguard/go:latest-dev
    url: https://images.chainguard.dev/directory/image/go/overview

6. Tag Strategy for Multiple Versions

When multiple versions share a repository, coordinate tags to avoid conflicts. The following Go versions both have major version 1, so only the newest can publish the 1 tag.

Newest version (go-1-26):

tags:
  - value: latest                                        # Points to newest
  - value: '{{ package_major_version("golang1.26") }}'   # e.g., "1"
    label: io.hummingbird-project.major-version
  - value: '{{ package_major_minor_version("golang1.26") }}'  # e.g., "1.26"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.26") }}'         # Full version
    label: org.opencontainers.image.version

Older versions (go-1-25):

tags:
  # NO 'latest' tag
  # NO major-version tag (would conflict with go-1-26's "1" tag)
  - value: '{{ package_major_minor_version("golang1.25") }}'  # e.g., "1.25"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'         # Full version
    label: org.opencontainers.image.version

Why: This ensures users can reference:

  • quay.io/hummingbird/go:latest → go-1-26
  • quay.io/hummingbird/go:1 → go-1-26 (latest 1.x)
  • quay.io/hummingbird/go:1.25 → go-1-25 (specific minor version)
  • quay.io/hummingbird/go:1.26 → go-1-26 (specific minor version)

When to update: If you add go-1-27, update go-1-26 to remove its latest and major-version tags, and add them to go-1-27.

7. Generate Files and Test

# Generate distro directories, lockfiles, VERSION, TAGS, etc.
make

# Build the image
ci/build_images.sh go-1-25/hummingbird/default

# Run tests
ci/run_tests_container.sh go-1-25/hummingbird/default

# Run linters
make check

Common Issues

Error: No match for argument: golang1.25

The package is not available in the configured distro repositories. If it is available only in the Hummingbird repository, add distros: [hummingbird] to properties.yml, then regenerate files:

make

Generation removes stale distro directories. Do not edit or remove generated files individually.

Choosing a Stream Value

Every image must have a stream field in properties.yml. The stream is a version series identity (e.g., "24", "3.11", "10.1", "latest") that is emitted as the io.hummingbird-project.stream label on the container image. It enables the image catalog to provide consistent structured data about multi-stream repositories.

Why Stream Matters

The image catalog must provide consistent structured data across time, including for images published before a second stream exists. Labels on published images cannot be changed retroactively. If a repository gains a new stream, all previously published images must already carry the correct version-based stream value — otherwise the catalog has inconsistent data that requires fragile repository-specific fixups.

Decision Process

flowchart TD
    Start["New image: choose stream value"] --> Q1{"Does endoflife.date show\nmultiple concurrently\nsupported branches?"}
    Q1 -->|Yes| Assess["Assess ecosystem granularity:\nupstream project, Docker Hub,\nDebian/Ubuntu, Wolfi/Chainguard,\nFedora, endoflife.date"]
    Assess --> Version["Use version-based stream\n(match ecosystem granularity)"]
    Q1 -->|No| Q2{"Does the software have a\nmeaningful version series\nidentity?"}
    Q2 -->|Yes| Current["Use version-based stream\n(current major version)"]
    Q2 -->|No| Latest["stream: latest"]
    Version --> Doc["Document reasoning\nas YAML comment"]
    Current --> Doc
    Latest --> Doc

Document the reasoning for the stream choice as a YAML comment above the stream field in properties.yml. This ensures future maintainers understand why a particular granularity was chosen:

# endoflife.date: parallel at major.minor (2.8, 3.0, 3.2)
stream: "3.0"

# rolling release with date-based versions, no version branches
stream: "latest"

Step 1: Does endoflife.date show multiple concurrently supported branches?

Check endoflife.date for the upstream project. It provides structured data on upstream release branches, security support timelines, and versioning schemes for most projects. If it shows multiple concurrently supported branches, use a version-based stream at the granularity of those branches. If the product is not on endoflife.date, that is itself evidence that parallel version lifecycle management is not a concern for that software — proceed to step 2.

The stream granularity must match the level at which upstream maintains parallel security-supported branches. Check:

  • Upstream project: At what version granularity are parallel branches maintained?
  • Docker Hub official images: Are there parallel version tags? At what granularity?
  • Debian/Ubuntu: Are there parallel versioned packages?
  • Wolfi/Chainguard: Are there versioned package variants?
  • Fedora: Are there parallel versioned packages? Single-version availability is not evidence against versioning, but multi-version availability is strong evidence for it.

Important: the image’s directory name is not evidence for granularity. The directory may use a coarser version or no version at all. The stream must reflect upstream’s actual branch structure.

Similarly, the purpose of upstream’s branches (LTS vs stable, stable vs mainline, STS vs LTS) is irrelevant to granularity. If they are parallel branches receiving security updates, the stream granularity must match.

Match the upstream project’s version naming convention. If the project and ecosystem consistently identify branches using a format that includes a minor component (e.g., .NET uses 8.0, 9.0 — never just 8 or 9), use that format for the stream value. This applies even when endoflife.date abbreviates to just the major number.

Step 2: Does the software have a meaningful version series identity?

If the software uses numbered versioning where the major (or major.minor) number identifies a version series, use a version-based stream at the current major version. The risk is asymmetric: using a version-based stream when a new major version never ships is harmless (the label is still correct), while using "latest" when a new version does ship breaks catalog consistency.

Note: if endoflife.date lists minor versions (e.g., memcached 1.4, 1.5, 1.6) but each immediately supersedes the previous with no overlap in support periods, the minor number is a sequential release counter, not a branch identity. Use the major version as the stream.

Reserve "latest" only for images where no meaningful version series identity exists. This is a narrow criterion with two cases:

  • Rolling/date-based releases with no version series (e.g., minio uses RELEASE.2024-01-18...)
  • Base images that track the distro rather than independent software (e.g., core-runtime)

Step 3: For unversioned images, track the current version

For images without a version in their directory name or version constraints, the stream value reflects the version currently shipped and must be updated when the upstream version changes. This is analogous to version_constraints — an explicit declaration that is updated periodically.

Field Reference

See the Image Configuration Reference for the stream field definition and examples.

Next Steps

2.5 - Adding FIPS Variants

How FIPS variants work and step-by-step guide for adding them

Overview

FIPS variants provide container images that use only FIPS 140-3 validated cryptographic modules. FIPS is implemented as a cross-cutting modifier variant (like builder), not a separate image — the FIPS variant uses the same base image structure but layers FIPS configuration on top via additional RPM packages.

FIPS variants are restricted to the Hummingbird distro only, because FIPS-validated packages are not available in Fedora Rawhide.

FIPS images must not contain any non-validated cryptographic libraries. The presence of an unvalidated crypto library (such as libgcrypt or gnutls) would undermine the FIPS compliance guarantee, even if the application itself does not use that library. Global tests enforce this constraint.

FIPS Validation Scope

The cryptographic modules shipped in FIPS images are FIPS 140-3 validated through the NIST Cryptographic Module Validation Program (CMVP). This validation applies only when running on RHEL systems installed in FIPS mode. Running FIPS images outside this environment (e.g., on Fedora, on RHEL not in FIPS mode, or on other Linux distributions) is outside the validated configuration.

As a best-effort goal, FIPS images aim to behave similarly on both FIPS and non-FIPS hosts — restricting cryptographic operations to FIPS-approved algorithms regardless of host configuration. This is not a guarantee; some images (particularly NSS-based ones like OpenJDK) may behave differently depending on host FIPS mode.

Validated Cryptographic Modules

All validated modules in Hummingbird are pre-built binaries from RHEL 9.2, submitted to NIST for FIPS 140-3 validation:

Module Package Verified by global test
OpenSSL FIPS provider openssl-config-fips fips-provider-matches-ubi9
NSS softokn nss-softokn-fips fips-nss-modules-match-rhel92
NSS freebl nss-softokn-freebl-fips fips-nss-modules-match-rhel92

Global tests validate each module’s checksum against the known-good RHEL 9.2 validated binaries. Checksums are architecture-specific because the binaries differ per architecture.

All FIPS variants also install crypto-policies-config-fips, which sets the system-wide crypto policy to FIPS.

Two FIPS Stacks

Two FIPS stacks exist depending on which crypto library the image’s software uses:

OpenSSL stack — crypto-policies-config-fips + openssl-config-fips. The OpenSSL FIPS provider is enabled via a drop-in configuration file at /etc/pki/tls/openssl.d/fips-provider-enable.cnf. Enforcement is container-side: non-approved algorithms are rejected even on non-FIPS hosts. Used by: Nginx, Node.js, Python, Ruby. Go images also include these packages for system tools (e.g., git-core), but Go itself uses its own native FIPS 140-3 module.

NSS stack — crypto-policies-config-fips + nss-softokn-fips + nss-softokn-freebl-fips. NSS checks the host kernel’s FIPS mode (/proc/sys/crypto/fips_enabled), so enforcement behavior differs between FIPS and non-FIPS hosts. The primary Java security provider switches to SunPKCS11-NSS-FIPS. Used by: OpenJDK.

Both stacks share crypto-policies-config-fips for the system crypto policy, but differ in which cryptographic library provides the validated implementation.

Adding a FIPS Variant

1. Determine the FIPS Stack

Identify which crypto library the image’s software uses:

  • OpenSSL — most languages and servers (Python, Node.js, Ruby, Nginx, Go)
  • NSS — Java/OpenJDK (uses NSS via PKCS#11)

This determines which FIPS packages to install.

2. Update properties.yml

Add fips to additional_variants with a Hummingbird distro restriction, and add the FIPS packages to rpm_packages.fips.

OpenSSL-based image:

additional_variants:
  - name: fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - openssl-config-fips

NSS-based image (OpenJDK):

OpenJDK has existing runtime and runtime-builder base variants. FIPS is added as both fips (full JDK) and runtime-fips (headless JRE):

additional_variants:
  - name: fips
    distros: [hummingbird]
  - name: runtime-fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - java-21-openjdk-devel              # variant-specific packages
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips
  runtime-fips:
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips

See the Image Configuration Reference for complete properties.yml options, including additional_variants and rpm_packages.

3. Generate Files

Run make to generate the hummingbird/fips/ directory structure (Containerfile, RPM lockfiles, VERSION, TAGS):

make

4. Write FIPS Tests

Add FIPS-specific tests to tests-container.yml using filters.variants to target FIPS variants. A standard set of FIPS tests validates:

  1. Non-approved algorithm rejected — e.g., MD5 (for security use), Blowfish, RC2
  2. Approved algorithm works — e.g., SHA-256
  3. Non-approved cipher rejected — e.g., 3DES, CHACHA20-POLY1305
  4. Approved cipher works — e.g., AES-256-GCM (encrypt/decrypt roundtrip)

Example (Python):

fips-rejects-md5:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    try:
        hashlib.new('md5')
        print('MD5_ALLOWED')
    except ValueError:
        print('MD5_REJECTED')
    " | grep -q MD5_REJECTED \
        || test_fail "FIPS mode should reject MD5"

fips-allows-sha256:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    print(hashlib.sha256(b'test').hexdigest())
    " | grep -q 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 \
        || test_fail "FIPS mode should allow SHA-256"

Host FIPS mode considerations:

Where possible, write tests without filters.fips so they validate behavior on both FIPS and non-FIPS hosts. This supports the best-effort goal of consistent behavior across host types. Use filters.fips: true only for tests that inherently require a FIPS-enabled host kernel — primarily NSS-based images where enforcement depends on the host:

# NSS-based: rejection only works on FIPS hosts
fips-rejects-blowfish:
  filters:
    variants: ["*fips*"]
    fips: true                    # requires FIPS-enabled host kernel
  command: |
    ...

See the Test Configuration Reference for variant filters and FIPS mode selection.

5. Build and Test

# Build the FIPS variant
ci/build_images.sh <image>/hummingbird/fips

# Run tests
ci/run_tests_container.sh <image>/hummingbird/fips

# Run linters
make check

Go-Specific FIPS Requirements

Go 1.24+ includes a native FIPS 140-3 cryptographic module that has been CMVP validated (CMVP Certificate #5247). Unlike the previous golang-fips approach (which replaced the compiler with a fork using OpenSSL as the crypto backend), native FIPS uses the same golang package with build-time and runtime environment variables:

  • GOFIPS140=v1.0.0 (build-time): Tells go build to include the CMVP-validated FIPS module (v1.0.0) in the resulting binary. Set in the image environment so all builds in the container use it.
  • GODEBUG=fips140=on (runtime): Enables FIPS 140-3 mode — uses NIST DRBG for randomness, negotiates only FIPS-approved TLS, and runs mandatory self-tests. Set in the image environment as the default for binaries run in the container.
  • Same package: The FIPS variant installs the standard golang1.25 package (no version_package override needed).
  • No cgo dependency: Native FIPS works with both CGO_ENABLED=0 and CGO_ENABLED=1 binaries.
main_package: golang1.25
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - golang1.25
    - openssl-config-fips

The openssl-config-fips package is retained for system tools in the image (e.g., git-core) that use OpenSSL for TLS connections.

Go-specific FIPS tests verify:

  • crypto/fips140.Enabled() returns true (native FIPS module is active)
  • go version -m output includes build GOFIPS140=v1.0.0-c2097c7c (CMVP-validated module, per the security policy section 11.1)
  • FIPS works with CGO_ENABLED=0 static binaries (a key advantage over the old golang-fips approach)

OpenJDK-Specific FIPS Requirements

OpenJDK uses the NSS stack instead of OpenSSL, which has several implications:

  • No openssl-config-fips — uses nss-softokn-fips + nss-softokn-freebl-fips instead
  • Host FIPS mode dependency — NSS checks /proc/sys/crypto/fips_enabled, so FIPS enforcement behavior differs between FIPS and non-FIPS hosts. Crypto rejection tests must use filters.fips: true.
  • Compound variants — OpenJDK introduces runtime-fips alongside fips, combining the runtime base variant with the FIPS modifier. Both share the same NSS FIPS packages; fips additionally includes java-*-openjdk-devel.
  • Security provider — in FIPS mode, the primary Java security provider becomes SunPKCS11-NSS-FIPS instead of the default SUN/SunJCE providers.

Global FIPS Tests

The following tests run automatically for all FIPS images via ci/global-tests/tests-container.yml. Each test skips gracefully for images using the other FIPS stack.

Test Validates Applies to
fips-provider-matches-ubi9 OpenSSL FIPS module checksum matches RHEL 9.2 binaries OpenSSL stack
fips-nss-modules-match-rhel92 NSS module checksums match RHEL 9.2 validated binaries NSS stack
fips-no-libgcrypt libgcrypt is not present (not a validated module) All FIPS
openssl-fips-config-installed OpenSSL FIPS drop-in config exists OpenSSL stack
crypto-policy-is-fips System crypto policy is set to FIPS All FIPS

Image-specific FIPS tests (algorithm rejection/acceptance) are defined in each image’s tests-container.yml.

Testing on FIPS Hosts

The test runner detects host FIPS mode from /proc/sys/crypto/fips_enabled and exports the TEST_FIPS environment variable (true or false). Tests can filter on this using filters.fips:

# Runs only on FIPS-enabled hosts
fips-host-required:
  filters:
    fips: true
  command: |
    ...

# Runs only on non-FIPS hosts
non-fips-only:
  filters:
    fips: false
  command: |
    ...

# Runs regardless of host FIPS mode (default when filters.fips is omitted)
always-runs:
  command: |
    ...

OpenSSL-based images: Most FIPS tests run regardless of host FIPS mode, because the OpenSSL FIPS provider enforces algorithm restrictions at the container level.

NSS-based images: Crypto rejection tests require filters.fips: true, because NSS enforcement depends on the host kernel’s FIPS mode.

Next Steps

2.6 - Testing Guide

How to run and write integration tests locally and work with CI

Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments.

Running Container Tests Locally

Prerequisites

  • Container Engine: Podman or Docker
  • Python: PyYAML package (pip install PyYAML or dnf install python3-pyyaml)

Tests work directly with Podman:

ci/run_tests_container.sh <image-name>

# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide

# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>

With Docker

Use --setup to automatically configure Docker-in-Docker:

ci/run_tests_container.sh --engine docker --setup <image-name>

The --setup flag:

  • Starts hummingbird-docker-dind container with mirror.gcr.io/docker:dind
  • Configures environment variables (DOCKER_HOST, DOCKER_CERT_PATH, etc.)
  • Waits for Docker daemon to be ready
  • Reuses existing container on subsequent runs

Prerequisites:

  • Docker CLI (dnf install docker-cli)
  • Podman (to run the dind container)

Building Images

Build images before testing:

# With Podman
ci/build_images.sh <image-name>

# With Docker
ci/build_images.sh --engine docker --setup <image-name>

Testing Base Images

When modifying base images, test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime

Reproducing CI Failures

Reproduce Testing Farm failures locally:

# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default

# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/rawhide/group
ci/run_tests_container.sh --engine docker --setup <image-name>/rawhide/group

Troubleshooting Container Tests

Inspecting Failed Tests

Use --pause to inspect containers before cleanup:

ci/run_tests_container.sh --pause <image-name>

Viewing Test Output

By default, only failing tests show output. Use --verbose to see passing test output:

ci/run_tests_container.sh --verbose <image-name>

This also enables bash trace mode (set -x), showing each command as it executes.

Running K8s Tests Locally

K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).

Basic Usage

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Local Development Workflow

Build images locally and push to the internal registry:

# Build the image
podman build -t my-nginx:dev images/nginx/hummingbird/default/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

This requires oc login to the target cluster and oc project <namespace> to set the target namespace.

Testing Published Images

Test published images without building locally:

IMAGE_URL=quay.io/hummingbird/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Troubleshooting K8s Tests

Use --pause to inspect resources before cleanup, or --verbose to see passing test output:

ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Tests that encounter recognized transient infrastructure failures run up to three times. The runner cleans up test resources and waits 60 seconds before each retry. Application assertion failures are reported immediately. Recognized failures include transient registry and container daemon errors, package-download timeouts from the Hummingbird repository, and transient Go crypto/rand entropy-read failures on FIPS-enabled hosts.

Writing Tests

Test File Locations

  • Container tests: images/<name>/tests-container.yml
  • K8s tests: images/<name>/tests-k8s.yml

Both use the same YAML format with different available environment variables.

Basic Tests

Create a test file with named tests:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

basic-functionality:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --help

Container tests must use test_engine_run to start containers. When a detached container has both a name and a network, the helper waits for Podman network DNS registration automatically before returning. Fresh lookup processes avoid negative DNS caching in a single client process:

test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}"

For K8s tests:

---
cluster-access:
  command: |
    kubectl auth can-i get pods || test_fail "No pod access"

run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

External Test Scripts

For complex tests, use a separate shell script:

---
complex-test:
  command: ./test-complex-scenario.sh

Create images/<name>/test-complex-scenario.sh:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test

Variant-Specific Tests

Limit tests to specific variants:

build-tools-test:
  variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

Cross-Variant Tests

Test interactions between variants:

cross-variant-test:
  variants: [group]
  command: |
    builder="${TEST_IMAGES[nginx/builder]:?}"
    default="${TEST_IMAGES[nginx/default]:?}"

    test_engine_run --rm "${builder}" nginx -version
    test_engine_run --rm "${default}" nginx -version

Known Issues (Container Tests Only)

Mark container tests with known failures:

test-with-known-issue:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"

See the Test Configuration Reference for complete configuration options.

Working with CI Tests

Container Tests (Testing Farm)

Container tests run on Testing Farm with RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Viewing CI Test Results

  1. Open the merge request in GitLab
  2. Navigate to the Jobs page for the pipeline
  3. Identify the Testing Farm job (e.g., containers-rawhide-testing-farm-x86-64)
  4. Click on the job name to go to the Konflux PipelineRun page
  5. If wait-for-results is already finished, select it in the pipeline details, and then switch to the Testing Farm job via the ARTIFACTS_URL link in the right side pane; otherwise, select the scheduler job, switch to the Testing Farm API details via the tf-request link in the right side pane, and then follow the run.artifacts link in the JSON data

K8s Tests (Konflux Ephemeral Namespace)

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

Viewing K8s Test Results

  1. Open the merge request in GitLab
  2. Navigate to the Jobs page for the pipeline
  3. Identify the K8s test job (e.g., containers-rawhide-k8s-test)
  4. Click on the job name to go to the Konflux PipelineRun page
  5. View the run-tests task logs for test output

Rerunning CI Tests

Retrigger a specific pipeline using slash commands in merge request comments:

/retest nginx--hummingbird--default-on-pull-request

For automated retriggers of only failed checks, see Retrying Konflux Checks.

Next Steps

2.7 - Adding Kubernetes Tests

How to add Kubernetes and OpenShift integration tests to container images

This guide describes how to add Kubernetes tests to container images.

Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

  • Directory permissions (important for OpenShift arbitrary UIDs)
  • Version checks
  • Application functionality (web servers, databases, etc.)
  • Multi-container scenarios (sidecar patterns)

Confirm Test Design

Before implementing:

  1. Read images/{image_name}/tests-container.yml to understand existing tests
  2. Identify which tests need K8s validation:
    • Directory permissions (important for OpenShift arbitrary UIDs)
    • Version check
    • Application functionality (web server, database, etc.)
    • Multi-container scenarios (sidecar patterns)
  3. Confirm test approach before creating files

Rules

  • Script naming: test-k8s-*.sh prefix (e.g., test-k8s-version.sh)
  • Script location: images/{image_name}/test-scripts/
  • YAML location: images/{image_name}/tests-k8s.yml
  • Keep YAML lean: complex logic goes in scripts, not inline YAML
  • No external downloads: avoid network flakes (exception: container images)
  • OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

Validation

  • If images/{image_name}/ doesn’t exist → error “Image not found”
  • If no tests-container.yml → error “Add container tests first”

Create Test Scripts

If directory doesn’t exist → create images/{image_name}/test-scripts/

Test Script Template

#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0

Directory Permissions Test

If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:

  • Check writable directories exist and are writable
  • Verify running as non-root (UID != 0)
  • Test actual write with touch/rm

Version Check Test

If testing version → create test-k8s-version.sh:

  • Run version command
  • Check major version only (not major.minor.patch)
  • Avoids breaking on patch updates

Application-Specific Tests

  • If web server → deployment test with sidecar (curl container)
  • If database → connection/persistence test
  • If tool → primary functionality test

Create tests-k8s.yml

Pattern for each test:

test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"

Multi-Container Tests

If test needs multiple containers (app + client):

  • Use emptyDir volume mounted in both containers
  • emptyDir has 777 permissions, works with arbitrary UIDs

Readiness Probes

If testing app functionality:

  • Add readinessProbe to app container
  • Set timeoutSeconds: 5 for OpenShift network latency

OpenShift Compatibility

Read images/{image_name}/Containerfile.j2:

  • If writable dirs use user:user ownership → change to user:0
  • If writable dirs lack g+w → add chmod -R g+w
  • Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
  • IMPORTANT: Use {{ default_user }} template variable, never hardcode UIDs:
    • ✅ Correct: chown -R {{ default_user }}:0 ${NEWROOT}/dir
    • ❌ Wrong: chown -R 65532:0 ${NEWROOT}/dir
    • Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

touch images/{image_name}/Containerfile.j2
make images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default

Lint and Test

  1. Run make -j$(nproc) check → if fails, fix errors before proceeding

  2. Test in minikube:

    minikube start
ci/run_tests_k8s.sh --context minikube {image_name}

  3. Monitor pods immediately: kubectl get pods -w

  4. If Containerfile changed → test in OpenShift:

    ci/build_images.sh {image_name}/{distro}/default
# Tag with simple localhost name for --push-image
podman tag quay.io/hummingbird/{image_name}:latest localhost/{image_name}:test
ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default

    Note: Push images to localhost/{image_name}:test, not to quay.io. Tests should never push to the official registry.

  5. Monitor pods: oc get pods -w

Common Issues

  • If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
  • If wrong image used in OpenShift → verify ci/run_tests_k8s.sh doesn’t override TEST_IMAGE when --push-image set
  • If multi-container test can’t see files → add emptyDir volume
  • If readiness probe times out in OpenShift → add timeoutSeconds: 5

Automated Skill

For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.

2.8 - Image Configuration Reference

Complete reference for properties.yml configuration and image settings

Complete reference for configuring container images via properties.yml.

properties.yml Overview

Each image can have custom settings via a properties.yml file. This reference documents all available configuration options.

Minimal example:

---
rpm_packages:
  all:
    - nginx
main_package: nginx
tags:
  - value: latest

Complete structure:

---
# Distro configuration (see Image Variants section)
# distros: [hummingbird]  # Override default distros (use when package unavailable in some distros)

# Image variants (see Image Variants section)
# Use additional_variants to extend defaults, or variants to override completely
additional_variants: [fpm, fpm-builder]  # Adds to default variants
# variants: [default, builder]  # Alternative: completely replaces defaults

# Container runtime configuration (see Container Configuration section)
user: default  # Optional: 'default' for default_user (65532), or literal user ID/name

# Package management (see Package Management section)
rpm_packages:
  build-deps: [...]
  all:
    - package-name                   # All architectures
    - name: arch-specific-package    # Arch-specific (see Package Management)
      arches:
        only: x86_64
  <distro>: [...]
  <variant>: [...]
additional_repos: [...]
allow_fedora_repos: true  # Optional: allow Fedora repos for hummingbird images (temporary workaround)

# Build configuration (see Build Configuration section)
hermetic: true
# Exceptional legacy source builds only; do not use for new images:
# build_from_source: true
# prefetch_gomod_path: upstream-submodule

# Versioning (see Versioning section)
main_package: package-name
# version_package:            # Optional: override package for version lookup per variant/distro
#   fips: package-fips-name   # Example: variant-specific package name

# Testing configuration (see Testing Configuration section)
reverse_dependency_tests: false  # Optional: disable for images like curl

# Release configuration (see Release Configuration section)
repository: image-name           # Required: Quay.io repository name
# <reasoning for stream choice>
stream: "latest"                 # Required: version series identity for the image catalog
application_category: "Storage"  # Required: Pyxis application category
# support_level: community        # Optional: community or experimental (default: Red Hat supported)
# deprecated: true                # Optional: mark the image stream as deprecated

# Image metadata (see Image Metadata section)
summary: "Short one-liner description"   # Required when adding labels
description: >-                          # Required when adding labels
  Short paragraph describing the image.
  1-2 sentences about key features.
url: "https://upstream-project.org"      # Required when adding labels

tags:
  - value: latest
  - value: '{{ package_major_version("package-name") }}'
    label: io.hummingbird-project.major-version

# Compliance scanning (enabled by default, see Compliance Scanning section)
# oscap:
#   enabled: false         # Optional: set to false to opt out
#   profiles:              # Optional: override which profiles run per variant
#     cis: true
#     stig:
#       variants: ["*fips*"]
#   exclude_rules:         # Optional: exclude specific rules (concatenated with globals)
#     - id: xccdf_org.ssgproject.content_rule_example
#       reason: "Reason for exclusion"
#       variants: ["*fips*"]   # Optional: glob patterns
#       profiles: [cis]        # Optional: limit to specific profiles

Image Metadata

Images can declare metadata fields for container image labels:

summary

  • Type: String
  • Required: When adding image labels (currently caddy only, expanding to all)
  • Length: ~40-80 characters (one-liner)

Short phrase describing what the image is. Used in table/list views and catalog cards.

Do not start with the image name – the name is always visible from context (UI headings, name label, inspect output).

Example: "Web server with automatic HTTPS"

description

  • Type: String (use >- YAML scalar for multi-line readability)
  • Required: When adding image labels
  • Length: ~100-250 characters (1-2 sentences)

Short paragraph for card views, podman inspect, and Kubernetes UIs (io.k8s.description). Describes what the software is and its key distinguishing features.

Use >- (folded scalar) in YAML to keep the value readable across multiple lines – it folds to a single-line string. Do not use | (literal block) as embedded newlines break label syntax.

Avoid embedded double quotes and backslashes – values are emitted as-is in Containerfile LABEL instructions with no escaping.

The same description value is used when generating the image README: the readme_description() macro in README.md.j2 templates renders it as the introductory paragraph. Do not duplicate this text in the template; use {{ readme_description() }}.

Example:

description: >-
  Extensible server platform with automatic HTTPS by default.
  Provides HTTP/3, reverse proxying, load balancing, and static
  file serving with minimal configuration.

url

  • Type: String (URL)
  • Required: When adding image labels

URL for the upstream project homepage. Used in org.opencontainers.image.url and the Conforma url label.

Example: "https://caddyserver.com"

deprecated

  • Type: Boolean
  • Default: Omitted, which means non-deprecated
  • Description: Marks the image stream as deprecated. The generated image carries the io.hummingbird-project.deprecated: true label so the image catalog can mark the stream as deprecated after the final release.
  • Example: deprecated: true

The field applies to the stream represented by the image directory. When all streams sharing a repository are deprecated, the catalog can mark the repository as deprecated. When only some streams are deprecated, the repository remains visible as a mixed-status repository.

Image Variants

Images can have multiple variants (e.g., default and builder):

  • default: Minimal runtime environment
  • builder: Includes development tools and build dependencies
  • custom: Define custom variants in properties.yml

Each variant gets its own:

  • Containerfile in images/<name>/<variant>/Containerfile
  • RPM lockfiles in images/<name>/<variant>/rpms
  • Build pipeline and release tags (non-default variants get -<variant> suffix)

variants

  • Type: Array of strings
  • Default: [default, builder]
  • Description: List of image variants to generate. Completely overrides the default variants. Each variant gets its own Containerfile, lockfiles, build pipeline, and release tags. Non-default variants get a -<variant> suffix in their image tags.
  • Example: [default] (CI images that only need the default variant)
  • Note: Prefer additional_variants when extending defaults; use variants only when you need to exclude default variants.

additional_variants

  • Type: Array of strings or objects
  • Default: None
  • Description: Additional variants to add to the base variants. The base is variants if specified, otherwise the default variants. Use this when you want to extend without repeating.

Simple format (string array):

additional_variants: [fpm, fpm-builder]  # Adds FPM variants to base variants for all distros

Object format with distro restrictions:

Each variant can be an object with name and optional distros fields to restrict which distros the variant is built for:

additional_variants:
  - name: fips
    distros: [hummingbird]  # Only build fips variant for hummingbird distro
  - name: fpm               # Simple string still works in array
  - name: fpm-builder
    distros: [hummingbird, rawhide]  # Build for specific distros

The distros field supports glob patterns:

additional_variants:
  - name: fips
    distros: ["*bird"]      # Matches hummingbird
  - name: special
    distros: ["raw*"]       # Matches rawhide

When to use distro restrictions:

Use distro restrictions when a variant requires packages or features only available in specific distros. For example, FIPS variants may only be available for Hummingbird where FIPS-validated packages are present

variant_descriptions

  • Type: Object (string-to-string mapping)
  • Required: When image has description and defines additional_variants with non-default base specializations
  • Description: Maps base variant names to human-readable descriptions. These descriptions are emitted as the io.hummingbird-project.variant.description label on each container image. The global description for default is defined in images/variables.yml; image-specific entries add descriptions for bases like fpm, runtime, or openssl. Modifier display (builder, fips) is handled by catalog consumers using the boolean variant labels, not by these descriptions.

Writing guidelines:

  • Describe the variant holistically — what the image variant is, not just what packages it adds on top of default. Someone reading the description with no surrounding context (e.g., via podman inspect) should understand the variant’s identity.
  • Use a noun phrase (2-6 words), not a full sentence.
  • Do not start with the image name (the image context is already provided by the image name label).
  • Avoid the word “runtime” in the default description (the OpenJDK runtime variant is a separate base, and OpenJDK’s default includes the full JDK).
  • Avoid expanding abbreviations redundantly (e.g., “JRE runtime” is redundant because JRE already means “Java Runtime Environment”).
  • Prefer functional descriptions (what capability the variant provides) over repeating the variant name as a product name.

Example (PHP image with FPM base):

variant_descriptions:
  fpm: "PHP FastCGI process manager"

Example (OpenJDK image with runtime base):

variant_descriptions:
  runtime: "Headless Java runtime"

distros

  • Type: Array of strings
  • Default: [rawhide, hummingbird] (from images/variables.yml)
  • Description: List of distros to build for this image. Overrides the global default_distros. Use this to disable a distro for images where a required package is not available in that distro.
  • Example: [hummingbird] disables Rawhide builds for this image
  • Note: Each enabled distro creates a full set of variants, so the final combinations are distros × variants.

Container Configuration

user

  • Type: String, integer, or object (per-variant dict)
  • Required: No (defaults to default)
  • Description: Specifies which user the container runs as. The USER directive is rendered via the {{ set_user() }} macro at the end of Containerfile templates. The special (and default) value default maps to the default_user variable (UID 65532). For user: default, the {{ final_stage() }} macro sets HOME=/tmp and WORKDIR /tmp early in the image to ensure unprivileged users have a writable default environment. Images can override WORKDIR later in their Containerfile.j2. For builder variants with user: default, it additionally sets ENV CONTAINER_DEFAULT_USER=65532.

Same user for all variants:

user: default

user: root  # Root user

user: postgres  # Literal username

Per-variant configuration:

Use when different variants need different users:

user:
  default: root
  builder: root
  fpm: default
  fpm-builder: default

Package Management

Packages are defined in properties.yml under rpm_packages:

rpm_packages:
  build-deps:             # Build-time dependencies (installed in builder layer, not in final
                          # image). Included in lockfiles for hermetic builds.
    - golang              # Example: Go compiler for building from source
    - rpm-build           # Example: Tools for building RPMs during image construction

  all:                    # Included in all variants (Containerfiles and lockfiles)
    - coreutils-single
    - your-main-package
    - name: grub2-efi-x64           # Arch-specific: only on x86_64
      arches:
        only: x86_64
    - name: grub2-efi-aa64          # Arch-specific: only on aarch64
      arches:
        only: aarch64

  hummingbird:            # Included in all Hummingbird variants (distro-level key)
    - versioned-package-1.2

  rawhide:                # Included in all Rawhide variants (distro-level key)
    - unversioned-package

  builder:                # Included only in "builder" variant
    - compiler-packages
    - debug-tools

  custom-variant:         # Included only in "custom-variant" variant
    - variant-specific-packages

Package entry format

Each entry in an rpm_packages list can be either a string (installed on all architectures) or an object with architecture constraints:

String entry (all architectures):

- coreutils-single

Object entry (arch-specific):

- name: grub2-efi-x64
  arches:
    only: x86_64            # Install only on x86_64

- name: some-package
  arches:
    not: aarch64             # Install on all architectures except aarch64

The arches.only and arches.not fields accept either a single string or a list of strings. Supported architectures: aarch64, x86_64.

How arch-specific packages work:

  1. Lockfiles: Arch-specific entries are passed to rpm-lockfile-prototype using its native per-arch package format. The lockfile resolver only includes matching packages for each architecture.
  2. Containerfiles: Arch-specific packages are installed via a case statement on TARGETARCH, so the correct packages are installed during multi-arch builds.

rpm_packages.build-deps

  • Type: Array of strings or arch-specific objects
  • Description: Build-time dependencies needed during image construction. Automatically installed in the builder layer (not in the final image) and included in lockfiles for hermetic builds. Use for compilers, build tools, and other packages needed only during the build process.

rpm_packages.all

  • Type: Array of strings or arch-specific objects
  • Description: Packages included in all variants, in both Containerfiles and lockfiles

rpm_packages.<distro>

  • Type: Array of strings or arch-specific objects
  • Description: Packages included in all variants for the specified distro. Useful when different distros ship different package names (e.g. versioned vs unversioned) and the packages should apply to every variant within that distro.

rpm_packages.<variant>

  • Type: Array of strings or arch-specific objects
  • Description: Packages included only in the specified variant, regardless of distro

additional_repos

  • Type: Array of strings (repository filenames)
  • Default: None
  • Description: Additional yum repository files from yum-repos/ to include in the RPM lockfile generation. These repos are added to the variant-specific repos (see [Global Variables Reference
  • Example: [konflux-ci-rpm-lockfile-prototype-main-fedora-rawhide.repo]
  • Usage: Only needed for packages not available in standard Fedora repos or variant-specific repos

allow_fedora_repos

  • Type: Boolean
  • Default: false
  • Description: Allow Fedora repositories during lockfile generation for Hummingbird production images. By default, production Hummingbird images exclude Fedora repos to ensure packages come only from the Hummingbird repository. Set to true to temporarily allow Fedora repos when required packages are not yet available in the Hummingbird repository.
  • Example: true
  • Usage: Typically used temporarily for packages not yet built for Hummingbird (e.g., .NET runtime on x86_64, OpenJDK packages). Should be removed once packages are available in the Hummingbird repository.
  • Note: Toolchain and similar images under images/ (such as hummingbird-builder) use Fedora repositories for signature validation regardless of this setting.

Hermetic Builds

For hermetic builds (enabled by default), all packages from:

  • rpm_packages.build-deps - included in lockfiles and Containerfiles (builder layer only)
  • rpm_packages.all - included in lockfiles and Containerfiles (final image)
  • rpm_packages.<distro> - included in lockfiles and Containerfiles for all variants of a distro
  • rpm_packages.<variant> - included in lockfiles and Containerfiles for matching variants
  • default_rpm_packages.builder - automatically added for builder variants (see Global Variables Reference - default_rpm_packages)

are added to rpms.lock.yaml and prefetched using cachi2 for offline builds.

Build-time vs Runtime Packages:

  • rpm_packages.build-deps: Installed in the builder layer only (not in final image)
  • rpm_packages.all: Installed in ${NEWROOT} (the final image)
  • rpm_packages.<distro>: Installed in ${NEWROOT} for all variants of the specified distro
  • rpm_packages.<variant>: Installed in ${NEWROOT} for the specified variant

Build Configuration

hermetic

  • Type: Boolean
  • Default: true
  • Description: Controls whether the Konflux build pipeline passes HERMETIC=true to the buildah task, enabling hermetic (offline, network-isolated) builds with prefetched dependencies.
  • Important: Images with hermetic: false are blocked from release by the Conforma hermetic_task policy. hermetic: false should not be used for any image in the containers repository.

build_from_source

  • Type: Boolean
  • Default: false
  • Description: Controls version extraction in the package_version() macro. When true, extracts version from .gitmodules for images built from git submodules. When false, extracts version from RPM package metadata. Does not control whether source code is compiled in the Containerfile.
  • Warning: Do not use for new images. Software built directly from source is not represented by RPM package metadata and cannot be scanned for CVEs by the project’s package-based scanner. This setting exists only for exceptional legacy images.

prefetch_gomod_path

  • Type: String (path relative to image directory)
  • Default: None
  • Description: Path to a Golang project directory (typically a git submodule) for prefetching Go modules via cachi2. Enables hermetic builds for Go projects without a vendor tree. The path is relative to the image directory.
  • Usage: Existing exceptional source-built images only. Do not use for new images because source-built software cannot be scanned for CVEs by the project’s package-based scanner.

Versioning

main_package

  • Type: String
  • Description: Specifies the main package for version labeling

For RPM-based images:

Set to the package name. Version is extracted from RPM package metadata.

main_package: nginx

For source-built images:

Existing exceptional source-built images set this to the submodule path. Version is extracted from the branch field in .gitmodules. Do not use this pattern for new images: source-built software cannot be scanned for CVEs by the project’s package-based scanner.

main_package: images/minio/minio-upstream
build_from_source: true

version_constraints

  • Type: String (glob pattern)
  • Default: None (no constraints)
  • Description: Defines a version constraint pattern to prevent unwanted major or minor version updates. The constraint applies to all distros for this image. Constraints are validated during make check, and builds fail if VERSION files violate the constraint. Primarily used for multi-version images (e.g., python-3-11, nodejs-20) to ensure they stay on their intended version family.

Constraint patterns:

Patterns use glob-style matching:

  • 3.11.* - Match any 3.11.X version (recommended)
  • 20.* - Match any 20.X version
  • 8.* - Match any 8.X version
  • 1.25.* - Match any 1.25.X version (for 3-part versioning like Go)

The pattern matches against the content of VERSION files generated during the build process.

Example:

main_package: python3.11
version_constraints: '3.11.*'  # Ensure python-3-11 stays on 3.11.X across all distros

main_package: dotnet-sdk-8.0
version_constraints: '8.*'  # Applies to rawhide, hummingbird, and any other distros

When to use:

  • Multi-version images where each image targets a specific version family
  • Preventing automatic major/minor version bumps during Renovate updates
  • Ensuring version tags remain accurate (e.g., python:3.11 doesn’t become python:3.12)

Violation handling:

When a constraint is violated, make check fails with a detailed error message showing:

  • The VERSION file path
  • The actual version found
  • The constraint that was violated
  • The properties.yml file where the constraint is defined

Options for resolution:

  1. Create a new image for the new version (recommended for major/minor bumps)
  2. Exclude the violating version via yum-repos/*.repo excludepkgs
  3. Update the constraint to allow the new version

See Version Constraints for detailed documentation.

version_package

  • Type: Object (string-to-string mapping)
  • Default: None
  • Description: Override the package name used for version extraction on a per-distro, per-variant, or per-distro/variant basis. When a variant or distro installs a different package that provides the same software (e.g., a FIPS-enabled build), this field tells the version/tag macros which package name to look up in the lockfile. The lookup order is: distro/variant, then variant, then distro, then main_package.

Per-distro override:

Use when different distros ship the same software under different package names:

main_package: ruby
version_package:
  hummingbird: ruby4.0  # hummingbird ships ruby4.0, rawhide ships ruby

Per-variant override:

Use when a variant installs a differently-named package:

main_package: ruby
version_package:
  minimal: ruby-minimal  # minimal variant resolves version from ruby-minimal

Per-distro/variant override:

Use when both distro and variant affect the package name:

main_package: ruby
version_package:
  hummingbird/fips: ruby4.0-fips

Usage in templates:

The resolved package name is available as package_name_for_version in Jinja2 templates:

tags:
  - value: '{{ package_version(package_name_for_version) }}'
    label: org.opencontainers.image.version

ENV GOLANG_VERSION={{ package_version(package_name_for_version) }}

Testing Configuration

reverse_dependency_tests

  • Type: Boolean
  • Default: true
  • Description: Controls whether this image participates in reverse dependency workflows (both building and testing). When true, changes to this image trigger rebuilds and tests of all images that depend on it. Dependencies are detected by searching for TEST_IMAGES[...] references in test scripts. Set to false for images that should not trigger reverse dependency workflows when changed.
  • Example: Set to false for curl to avoid triggering unnecessary reverse dependency tests

How it works in CI:

When an image is tested, the CI pipeline performs two phases:

  1. Build Phase: Builds all dependencies needed for testing

    • Uses ci/build_images.sh --build-deps to build:
      • Forward dependencies (images the main image’s tests depend on)
      • Reverse dependencies (images that depend on the main image, filtered by reverse_dependency_tests: true)
      • Forward dependencies of those reverse dependencies
    • Automatically deduplicates to ensure each image is built exactly once

  2. Test Phase: Runs tests on the main image and its reverse dependencies

    • Uses ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps to test:
      • The main image
      • All reverse dependencies (filtered by reverse_dependency_tests: true)

Why set to false?

Set reverse_dependency_tests: false for images like curl that are:

  • Used pervasively across many images (too many reverse dependencies)
  • Have a small, stable API that can be fully tested in their own tests
  • Would cause excessive CI load if every dependent image were tested on each change

Release Configuration

The repository and stream fields together define the release identity for an image. Both are required and validated during CI. The repository specifies the Quay.io repository name, and the stream specifies the version series within that repository.

repository

  • Type: String
  • Required: Yes
  • Description: The Quay.io repository name for this image. For images where the directory name matches the desired repository name (e.g., images/nginx/ → repository: nginx), this is redundant but still required for explicitness. For versioned image directories, this groups multiple versions under one repository (e.g., nodejs-20 and nodejs-24 both use repository: nodejs).
  • Example: nodejs (used by nodejs-20 and nodejs-24 directories)

stream

  • Type: String
  • Required: Yes
  • Description: The version series identity for this image within its repository. Emitted as the io.hummingbird-project.stream label on the container image, enabling the image catalog to provide consistent structured data about multi-stream repositories.

The stream value must reflect upstream’s actual branch structure at the granularity where parallel security-maintained branches exist. Use endoflife.date to determine the correct granularity.

Key rules:

  • The stream is never derived from the image directory name. The directory name is not evidence for stream granularity (e.g., a directory named foo/ for an upstream with branches 2.8, 3.0, 3.2 still requires a major.minor stream like "3.0").
  • The purpose of upstream branches is irrelevant to granularity. Whether upstream calls them “LTS”, “stable”, “mainline”, or “STS” does not matter. If they are parallel branches receiving security updates, the stream granularity must match.
  • Match the upstream project’s version naming convention. If the project identifies branches as 8.0, 9.0 (not 8, 9), use that format even if endoflife.date abbreviates it.
  • Sequential minor versions are not parallel branches. If endoflife.date lists minor versions that each immediately supersede the previous with no support overlap, the minor is a release counter. Use the major version as the stream.
  • Use "latest" only for images where no meaningful version series identity exists: rolling-release tools with date-based versioning (e.g., minio) or base images that track the distro rather than independent software (e.g., core-runtime).
  • For unversioned images (no version in directory name, no version constraints), the stream value reflects the version currently shipped and must be updated when the upstream version changes.

See the stream assignment guidelines for the full decision process and rationale.

Document the reasoning for the stream choice as a YAML comment above the stream field. This ensures future maintainers understand why a particular granularity was chosen.

Examples:

# endoflife.date: parallel at major (Docker Hub + Debian + Wolfi)
stream: "24"
# endoflife.date: parallel at major.minor (Docker Hub + Debian + Wolfi)
stream: "3.11"
# endoflife.date: branches are 10.1, 11.0, not 10, 11
stream: "10.1"
# rolling release with date-based versions, no version branches
stream: "latest"

application_category

  • Type: String
  • Required: Yes
  • Description: The Pyxis application category for the image’s delivery repository. Emitted as the io.hummingbird-project.application-category label on the container image and used in the generated Pyxis repo config. All images sharing the same repository must use the same value. Validated during make check against the list in ci/application-categories.txt.

Common values for Hummingbird images:

  • "Programming Languages & Runtimes" – language runtimes and SDKs (Go, Python, Node.js, .NET, Ruby, etc.)
  • "Web Services" – web servers, reverse proxies, and load balancers (Nginx, httpd, Caddy, HAProxy, Tomcat)
  • "Database & Data Management" – databases and caches (PostgreSQL, MariaDB, Valkey, Memcached)
  • "Developer Tools" – CLI utilities and build tools (git, curl, jq)
  • "Storage" – object storage (MinIO, MinIO Client)
  • "Operating System" – base runtime images (core-runtime)

Example:

application_category: "Web Services"

tags

  • Type: Array of objects
  • Description: Define image tags for Quay.io releases. Each variant gets these tags, with non-default variants receiving a -<variant> suffix.

Tag Object Fields:

  • value (required): The tag value. Can be a literal string (e.g., latest) or a Jinja2 template using helper functions.
  • label (optional): If specified, the tag value is written as a LABEL in the Containerfile, and the actual rendered value from the label is used as the image tag. This ensures version tags match the actual package versions in the image.

Tag Helper Functions:

  • package_version("<package-name>") - Full version (e.g., 1.2.3-4.fc42)
  • package_major_version("<package-name>") - Major version only (e.g., 1)
  • package_major_minor_version("<package-name>") - Major.minor version (e.g., 1.2)

Example:

tags:
  - value: latest                                        # Literal tag, no label
  - value: '{{ package_major_version("nginx") }}'        # Template tag with label
    label: io.hummingbird-project.major-version
  - value: '{{ package_major_minor_version("nginx") }}'  # Tag becomes actual version from label
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("nginx") }}'
    label: org.opencontainers.image.version

Note: For images with multiple versions, only set latest on the latest version.

tag_suffix_aliases

  • Type: Mapping of variant names to arrays of suffixes
  • Description: Add aliases for every configured tag without creating another image variant. The suffix is appended directly to each base tag. Existing canonical tags remain unchanged.
  • Example:
tag_suffix_aliases:
  default:
    - jdk25
  builder:
    - jdk25-builder

support_level

  • Type: String

  • Default: Omitted (image is Red Hat supported)

  • Values: community, experimental

  • Description: Controls the support level, Quay organization, and Konflux application assignment for hummingbird-distro images. The mapping from support level to Quay org is defined in images/variables.yml under support_levels; all other names (Konflux app, service account, RPA suffix) are derived from the Quay org name. When omitted, the image is Red Hat supported and assigned to containers-hummingbird. Rawhide images are always assigned to containers-rawhide regardless of this property.

  • Example: support_level: community

  • Mapping:

    Value Quay org Konflux app CPE Pyxis/RPA
    (omitted) hummingbird containers-hummingbird Yes Included
    community hummingbird-community containers-community-hummingbird No Excluded
    experimental hummingbird-ci containers-ci-hummingbird No Excluded

  • Usage: Use community for images maintained by the community (e.g., minio, bootc-os). Use experimental for internal build infrastructure (e.g., hummingbird-builder). These images are built and tested normally but published to their designated Quay organization, not the official Red Hat registry.

How Labels Work:

  1. Template is rendered and written to Containerfile: LABEL org.opencontainers.image.version=1.27.3-1.fc42
  2. Build system extracts the actual value from the image label
  3. Image is tagged with the extracted value: quay.io/hummingbird/nginx:1.27.3-1.fc42

This ensures tags always match the actual package versions built into the image.

Compliance Scanning

OpenSCAP compliance scanning verifies CIS and STIG compliance during the build. Scanning runs against the container rootfs using oscap-chroot and fails the build if any enabled rules are violated.

Scanning is enabled by default for all images. It only runs for the Hummingbird distro and is skipped for the builder base image.

oscap.enabled

  • Type: Boolean
  • Default: true (from images/variables.yml)
  • Description: Controls whether OpenSCAP compliance scanning runs for this image. When true, a verify-compliance step is added to the generated Containerfile for all Hummingbird variants. Set to false to opt out.
oscap:
  enabled: false

oscap.profiles

  • Type: Object mapping profile names to enablement rules
  • Default: {cis: true, stig: {variants: ["*fips*"]}} (from images/variables.yml)
  • Description: Controls which compliance profiles run for each variant. Each profile can be set to true (all variants), false (disabled), or an object with a variants list of glob patterns.

Default behavior (CIS for all variants, STIG only for FIPS variants):

oscap:
  profiles:
    cis: true
    stig:
      variants:
        - "*fips*"

Enable STIG for all variants:

oscap:
  profiles:
    stig: true

Disable CIS for a specific image:

oscap:
  profiles:
    cis: false

The variants field supports glob patterns using fnmatch syntax (e.g., *fips* matches fips, fips-builder, openssl-fips).

oscap.exclude_rules

  • Type: Array of rule exclusion objects
  • Default: Global rule exclusions from images/variables.yml
  • Description: Rules to exclude from compliance scanning. Image-level exclusions are concatenated with global exclusions (not replaced). Each exclusion object has the following fields:
Field Type Required Description
id String Yes XCCDF rule ID (e.g., xccdf_org.ssgproject.content_rule_configure_crypto_policy)
reason String Yes Explanation for excluding this rule
variants Array of strings No Glob patterns for variants this exclusion applies to. Omit for all variants.
profiles Array of strings No Profile names (cis, stig) this exclusion applies to. Omit for all profiles.

Exclude a rule for all variants and profiles:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "Caddy uses Go's built-in TLS stack, not system crypto-policies"

Exclude a rule only for specific variants:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot"
      variants:
        - "*fips*"

Exclude a rule only from a specific profile:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_some_rule
      reason: "Not applicable to CIS for this image"
      profiles:
        - stig

Combine variant and profile filters:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot for FIPS variants"
      variants: ["*fips*"]
      profiles: [cis]

How compliance scanning works

  1. Profile resolution: generate_jinja2.py determines which profiles (CIS, STIG) are active for the current variant based on oscap.profiles.
  2. Rule filtering: Exclusion rules are filtered by variant (glob matching) and profile, producing per-profile exclusion lists.
  3. Tailoring file: If any active profile has exclusions, an XCCDF tailoring file (oscap-tailoring.xml) is generated in the variant directory with custom profiles extending the upstream CIS/STIG profiles.
  4. Containerfile: The verify-compliance command is added with explicit --cis and/or --stig flags based on active profiles, plus a --tailoring-file argument if exclusions exist.
  5. Build-time scan: oscap-chroot evaluates the rootfs against each active profile. The build fails if any non-excluded rule fails.

Next Steps

2.9 - Test Configuration Reference

Complete reference for test definition format and configuration options

Complete reference for configuring image tests via tests-container.yml and tests-k8s.yml.

Test Definition Format

Each image can have tests defined in:

  • images/<name>/tests-container.yml - Container tests (run with Podman/Docker)
  • images/<name>/tests-k8s.yml - K8s tests (run in Kubernetes cluster)

Both files use the same YAML format:

---
script:
  command: ./test.sh

inline:
  command: |
    image=${TEST_IMAGE:?}
    test_engine_run --rm "${image}" --version

Environment Variables

The test runners provide environment variables to test commands. Some are common to both test types, others are specific to container or K8s tests.

Common Variables

Variable Description
TEST_IMAGE The container image being tested
TEST_IMAGES Associative array with all group/variant image URLs
TEST_IMAGES_PATH Path to file containing serialized TEST_IMAGES array
TEST_VERBOSE Show test command output (true or false)
TEST_GROUP The image group being tested (nginx, python, etc.)
TEST_DISTRO The distro being tested (hummingbird, rawhide)
TEST_VARIANT The variant being tested (default, builder, etc.)

Container Test Variables

Variable Description
TEST_ENGINE Container engine to use (podman or docker)
TEST_USER_ID Current user ID (for permission handling)
TEST_FIPS Whether host is in FIPS mode (true or false)

K8s Test Variables

Variable Description
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (hum-k8s-test=<id>)

Helper Functions

Common to All Tests

Both test types provide the test_fail function:

Function Description
test_fail Fail the test with a custom error message

In K8s tests, kubectl is pre-configured with the context/kubeconfig from CLI args.

Container Tests Only

Container tests provide helper functions that wrap the container engine with appropriate flags. Always use these functions instead of calling ${TEST_ENGINE} directly for create and run commands. This ensures tests respect globally set options such as the hermetic mode setting (--pull=never).

Function Description
test_engine_create Run ${TEST_ENGINE} create with appropriate pull flags
test_engine_run Run ${TEST_ENGINE} run with appropriate pull flags

Note: For other container engine commands (inspect, network, volume, etc.), use ${TEST_ENGINE:?} directly.

Variant-Aware Testing

The testing system automatically includes variant-specific default tests:

  • Global tests from ci/{variant}-tests/tests-container.yml
  • Image-specific tests from images/<name>/tests-container.yml

Variant Selection

Tests can specify which variants they apply to using the filters.variants field. The field supports both exact matches and glob patterns.

# Test that only runs for builder variants
build-tools-test:
  filters:
    variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

# Test that runs for multiple specific variants
multi-variant-test:
  filters:
    variants: [default, builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Hello"

# Test that runs for all variants (no filters.variants field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

Glob Patterns in Variant Filters

Use glob patterns to match multiple variants dynamically:

# Test that runs for any FIPS variant (fips, fips-builder, etc.)
fips-test:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-fips-mode

# Test that runs for all builder variants (builder, fpm-builder, etc.)
builder-test:
  filters:
    variants: ["*-builder", "builder"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" dnf --version

# Mix exact matches and patterns
mixed-filter:
  filters:
    variants: [default, "*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on default and all fips variants"

Supported glob syntax:

  • * - matches any sequence of characters
  • ? - matches any single character

Distro Selection

Tests can specify which distros they apply to using the filters.distros field. Like variant filters, this field supports both exact matches and glob patterns.

# Test that only runs for hummingbird distro
hummingbird-only-test:
  filters:
    distros: [hummingbird]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-hummingbird-specific-check

# Test that runs for multiple distros
multi-distro-test:
  filters:
    distros: [hummingbird, rawhide]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on both"

# Test that runs for all distros (no filters.distros field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

This is useful for tests that check distro-specific features. For example, CPE labels are only present on hummingbird images (Fedora/rawhide has no official CPE).

Glob Patterns in Distro Filters

Use glob patterns to match multiple distros dynamically:

# Test that runs for any distro ending in "bird"
bird-distro-test:
  filters:
    distros: ["*bird"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-something

# Test that runs for any distro starting with "raw"
raw-distro-test:
  filters:
    distros: ["raw*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-rawhide-feature

Supported glob syntax:

  • * - matches any sequence of characters
  • ? - matches any single character

FIPS Mode Selection

Tests can specify whether they should run based on the host’s FIPS mode using filters.fips:

# Test that only runs on FIPS-enabled hosts
fips-required-test:
  filters:
    fips: true
  command: |
    test_engine_run --rm "${TEST_IMAGE}" openssl list -providers

# Test that only runs on non-FIPS hosts
non-fips-test:
  filters:
    fips: false
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-non-fips-check

# Test that runs regardless of FIPS mode (no filters.fips field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

The TEST_FIPS environment variable is set to true when running on a FIPS-enabled host (detected from /proc/sys/crypto/fips_enabled), and false otherwise.

Group Test Mode

When running tests for all variants (without specifying a specific variant), the system supports a special “group” test mode for tests that need to work across multiple variants simultaneously:

# Test that runs only in group mode
cross-variant-compatibility-test:
  filters:
    variants: [group]
  command: |
    # Access specific group/variant combinations using TEST_GROUP
    builder_image=${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    default_image=${TEST_IMAGES[${TEST_GROUP}/default]:?}

    echo "Builder variant: ${builder_image}"
    echo "Default variant: ${default_image}"

    # Test that both variants have compatible APIs
    test_engine_run --rm "${builder_image}" nginx -version
    test_engine_run --rm "${default_image}" nginx -version

Using TEST_IMAGES in External Scripts

External shell scripts that need to reference other image variants must source the TEST_IMAGES array file at the beginning:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Now TEST_IMAGES is available - always use :? for proper error checking
test_engine_run --rm "${TEST_IMAGES[curl/default]:?}" ...
test_engine_run --rm "${TEST_IMAGES[httpd/default]:?}" ...

Note: Inline test commands in YAML files automatically have access to TEST_IMAGES and do not need this sourcing pattern.

Using TEST_GROUP for Dynamic References

The TEST_GROUP variable contains the current image group name (e.g., nginx, python, aspnet-runtime-8-0). Use it with TEST_IMAGES to dynamically reference the current image’s variants without hardcoding the group name:

multi-stage-build:
  filters:
    variants: [builder]
  command: |
    # Build in builder variant, run in default variant
    "${TEST_ENGINE}" build -t localhost/myapp -f - . <<EOF
    FROM ${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    # ... build steps ...

    FROM ${TEST_IMAGES[${TEST_GROUP}/default]:?}
    COPY --from=0 /app /app
    EOF

    test_engine_run --rm localhost/myapp /app/myapp

Test Helper Functions

The test runner provides helper functions available in test commands:

test_fail(message)

Immediately fails the test with a custom error message sent to stderr:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Expected 'expected', got '$result'"

Known Issues (Container Tests Only)

Container tests can specify known log patterns that should not cause the test to report an error (for which support is still unimplemented) or failure. This helps distinguish between expected failures (tracked issues) and unexpected failures (new regressions), and detects when expected failures stop occurring.

Add known_issues to any test to specify patterns matching expected failures:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"  # optional, defaults to "always"

Fields

  • description: Human-readable explanation of the issue (required)
  • issue: Full URL to the issue tracking this problem (required)
  • pattern: Regular expression(s) to match against test output (required)
  • fails: Frequency of failure (optional, defaults to “always”)

Pattern Support

The pattern field supports both single patterns and arrays of patterns:

known_issues:
  # Single pattern
  - description: "Simple failure case"
    issue: "https://issues.redhat.com/browse/PROJ-1"
    pattern: "Connection failed"

  # Multiple patterns (any match triggers)
  - description: "Network connectivity issues"
    issue: "https://issues.redhat.com/browse/PROJ-2"
    pattern:
      - "Connection timeout"
      - "Network unreachable"
      - "curl: \\(28\\)"
    fails: "sometimes"

Failure Frequency

  • always: Known failures that consistently fail every time (default)
  • sometimes: Intermittent failures that may pass on retry (flaky tests)

Unexpected Pass Detection

The system automatically detects when tests with known issues labeled as fails: "always" suddenly start passing. This will not prevent missing known issues being masked by other (still occurring) known issues.

Automatic Retries

The container and K8s test runners automatically retry tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary registry, network, package-download, image-pull, container-startup, or K8s scheduling problems.

Retry Behavior:

  • Failed tests are checked against a list of retriable error patterns
  • If a match is found, the test is automatically retried
  • If the test still fails after all attempts, it’s reported as a normal failure

K8s tests run up to three times, with cleanup and a 60-second delay between attempts. Retriable conditions include image-pull and container-creation errors, as well as kubectl wait timeouts. Application assertion failures are not retried.

Next Steps

3 - Operating Hummingbird infrastructure

Operations guides for managing Hummingbird infrastructure, including Konflux cluster setup and CI/CD operations.

3.1 - Excluding Packages from Images

AI Agent Note: When asked to exclude packages, always ask the user for a ticket link or explanation first. This is required for the comment documenting the exclusion.

Overview

When a faulty RPM build is published to Hummingbird repositories, it can be excluded from container images by adding an excludepkgs directive to the repository configuration.

Workflow

1. Identify the source package NVR

Identify the Name-Version-Release of the faulty source package. Example: ncurses-6.5-8.20250614.hum1

2. Generate excludepkgs value

Query the Hummingbird repositories and generate the excludepkgs value:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest sh -c '
dnf5 repoquery --queryformat="%{sourcerpm} %{name}-%{evr}.*
" 2>/dev/null | sed -n "s/^ncurses-6.5-8.20250614.hum1.src.rpm //p" | tr "\n" " "
'

Replace ncurses-6.5-8.20250614.hum1 with the source NVR. This outputs the ready-to-use excludepkgs value:

ncurses-6.5-8.20250614.hum1.* ncurses-base-6.5-8.20250614.hum1.* ...

3. Add excludepkgs to repo file

Edit yum-repos/hummingbird.repo and add the binary package NEVRs to excludepkgs in the [hummingbird] section (the binary repo, not [hummingbird-source]).

Document the exclusion with a one-line comment containing the ticket link:

[hummingbird]
...
# HUM-1234: ncurses-6.5-8.20250614.hum1 causes segfault in terminfo parsing
excludepkgs=ncurses-base-6.5-8.20250614.hum1.* ncurses-libs-6.5-8.20250614.hum1.*

Multiple packages are space-separated. Use .* suffix to match all architectures.

4. Merge the change

Merge the repo file change to main. Renovate will automatically apply the exclusions when it next rebases existing lockfile update branches or creates new ones.

Verification

Check that the excluded package no longer appears in any rpms/rpms.lock.yaml files in subsequent Renovate-generated merge requests.

Removing Exclusions

When a newer version is available, exclusions can be removed.

1. Verify a newer version exists

Check the latest available version (should differ from excluded 6.5-8.20250614.hum1):

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --latest-limit=1 --queryformat="%{evr}" "ncurses-libs" 2>/dev/null

2. Remove the exclusion

Remove the excludepkgs line and its comment from yum-repos/hummingbird.repo.

3. Merge and verify

Merge the change. Renovate will pick up the newer version in subsequent lockfile updates.

3.2 - Managing Konflux Comments

Clean up and manage excessive Konflux comments on merge requests using the comment deletion tool.

When Konflux comments become excessive on merge requests, you can clean them up using the provided comment deletion tool.

Prerequisites

  • COM_GITLAB_TOKEN: GitLab API token with api scope and Maintainer access

Usage

If Konflux comments become excessive on merge requests, you can clean them up using:

# Show help for the comment deletion tool
make delete-konflux-comments ARGS="--help"

# Example: Delete comments from a specific merge request
make delete-konflux-comments ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete only failed build/test comments (dry run)
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete all Konflux comments from a merge request
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

Comment Categories

The tool can target specific types of Konflux comments:

  • failed: Only comments related to failed builds/tests
  • all: All Konflux-related comments (use with --all flag)

Safety Features

  • Dry Run Mode: Use --dry-run to preview what would be deleted without actually removing comments
  • Selective Deletion: Target specific comment categories to avoid removing important information
  • Confirmation: The tool will show what it plans to delete before taking action

Examples

# Set your GitLab token
export COM_GITLAB_TOKEN=your_token_here

# Preview what failed comments would be deleted
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete only failed build comments
make delete-konflux-comments ARGS="--comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete all Konflux comments (use with caution)
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

When to Use

Consider cleaning up Konflux comments when:

  • Merge requests have accumulated many outdated failure comments
  • Comments are making it difficult to find relevant discussion
  • You want to start fresh after resolving systematic issues
  • The comment thread has become cluttered with automated messages

Note: Use this tool judiciously, as it permanently removes comments that might contain useful debugging information.

3.3 - Running Conforma Checks Locally

Overview

Conforma (Enterprise Contract) policy checks run automatically in the Konflux pipeline before release (see Image Pipeline — Stage 5). These checks can also be run locally against any Konflux-built image to validate compliance, test policy changes, or investigate failures.

Prerequisites

Install the Conforma CLI:

curl -sLO https://github.com/conforma/cli/releases/download/snapshot/ec_linux_amd64
chmod 755 ec_linux_amd64
mkdir -p ~/.local/bin
mv ec_linux_amd64 ~/.local/bin/ec

Running All Checks

The full set of Conforma checks (signatures, attestations, SLSA provenance, labels, etc.) can be run against any Konflux-built image using ec validate image with the Konflux signing key.

Public Key

The Konflux signing public key is committed at ci/key.pub. This is the same key stored in the Konflux member cluster at k8s://openshift-pipelines/public-key.

Obtaining the Image Reference

Combine the IMAGE_URL and IMAGE_DIGEST results from a successful build PipelineRun:

<IMAGE_URL>@<IMAGE_DIGEST>

Merge request build images follow the pattern:

quay.io/redhat-user-workloads/hummingbird-tenant/<component>:on-mr-<MR_ID>-<COMMIT_SHA>

Running the Validation

Create a policy file that matches the pipeline policy but with any modifications needed for testing. For example, to run the pipeline policy with label checks enabled (removing the labels.required_labels and labels.optional_labels exclusions):

cat > /tmp/policy.yaml << 'YAML'
sources:
  - name: Release Policies
    config:
      exclude:
        - test.required_tests_passed:sast-snyk-check
        - test.no_skipped_tests:sast-snyk-check
        - test.required_tests_passed:sast-snyk-check-oci-ta
        - test.no_skipped_tests:sast-snyk-check-oci-ta
        - test.no_failed_tests:ecosystem-cert-preflight-checks
        - test.no_erred_tests:ecosystem-cert-preflight-checks
        - test.no_failed_informative_tests
        - test.no_test_warnings:deprecated-image-check
        - trusted_task.current
        - rpm_repos.ids_known
        - buildah_build_task.privileged_nested_param
        - schedule.weekday_restriction
      include:
        - '@redhat'
    data:
      - github.com/release-engineering/rhtap-ec-policy//data
      - oci::quay.io/konflux-ci/tekton-catalog/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/konflux-vanguard/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/integration-service-catalog/data-acceptable-bundles:latest
    policy:
      - oci::quay.io/enterprise-contract/ec-release-policy:konflux
    ruleData:
      allowed_registry_prefixes:
        - "quay.io/hummingbird-rawhide/"
        - "quay.io/hummingbird-ci/"
        - "quay.io/hummingbird-community/"
        - "quay.io/hummingbird/"
        - "quay.io/redhat-user-workloads/"
YAML

Then validate:

ec validate image \
    --image "quay.io/redhat-user-workloads/hummingbird-tenant/<component>@sha256:<digest>" \
    --policy /tmp/policy.yaml \
    --public-key ci/key.pub \
    --ignore-rekor \
    --strict=false \
    --show-successes \
    --output text

Options

Flag Purpose
--ignore-rekor Required — Konflux does not use a Rekor transparency log
--strict=false Return exit code 0 even on failures (useful for investigation)
--show-successes Include passing checks in output (off by default)
--output json Machine-readable output (also: text, yaml, summary)

Running Label Checks Only

The label checks (labels.required_labels, labels.optional_labels) are the only Conforma rules that can be evaluated without signatures or attestations, because they only read OCI image metadata. This lighter-weight approach does not require the signing key and works against any registry image, including published (non-Konflux-built) images.

Setup

Fetch the policy rego files and upstream rule data (one-time):

ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

git clone --depth=1 \
    https://github.com/release-engineering/rhtap-ec-policy.git \
    /tmp/rhtap-ec-policy

Running Checks

ec opa eval \
    --data /tmp/ec-policy/policy/*/policy \
    --data /tmp/rhtap-ec-policy/data/rule_data.yml \
    --input <(echo '{"image":{"ref":"quay.io/hummingbird-rawhide/caddy:latest"}}') \
    '{"deny": data.labels.deny, "warn": data.labels.warn}' \
    --format pretty

This works against any image accessible from a container registry — both Konflux-built and published images. It uses the ec CLI’s embedded OPA engine with ec.oci.* built-in functions to read image config directly from the registry.

Note: The --data flag (not --bundle) must be used for the rego files. Loading the policy as a bundle creates an OPA root conflict that prevents the rule data from being resolved. The ec.oci.* built-in functions are only available in ec opa, not in standalone opa or conftest.

Interpreting Results

For ec validate image:

  • Successes indicate passing policy rules (visible with --show-successes).
  • Violations are failures that block release. Each includes a code (rule name) and msg.
  • Warnings are non-blocking advisories.

For ec opa eval (label checks only):

  • deny entries are failures. Each includes code, msg, and optionally effective_on (a date when the rule becomes enforced — future dates mean the rule is not yet active).
  • warn entries are non-blocking warnings for optional labels.
  • Empty arrays mean the image passes all label checks.

Updating Cached Data

The fetched policy and rule data are point-in-time snapshots. To pick up upstream changes:

# Re-fetch the policy rego files (for ec opa eval)
rm -rf /tmp/ec-policy
ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

# Update the rule data (for ec opa eval)
git -C /tmp/rhtap-ec-policy pull

The ec validate image command fetches policy and data bundles on each run based on the policy.yaml references, so no manual update is needed.

3.4 - Upgrading Docsy

How to upgrade the Docsy theme and forward-port our layout overrides.

Overview

Both Hummingbird documentation sites use Docsy as their Hugo theme, installed as a Hugo module. Docsy releases roughly quarterly. This runbook covers how to upgrade to a new Docsy version.

The public docs (documentation) own all layout overrides. The internal docs (infrastructure/internal-docs/) only have build scaffolding and pick up layouts from the public docs via Hugo module import.

Layout override inventory

We override four Docsy templates: three to inject the companion page banner and sidebar lock icons, and one to add llms-full.txt cross-linking. Each override is a copy of the upstream template with a small patch applied.

layouts/docs/_td-content.html

Overrides Docsy’s shared layouts/_td-content.html for docs pages only.

Patch: One line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/docs/list.html

Overrides Docsy’s layouts/docs/list.html for docs section pages.

Patch: Same one line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/partials/sidebar-tree.html

Overrides Docsy’s layouts/_partials/sidebar-tree.html for the navigation sidebar.

Patch: One snippet inserted in two places (foldable and non-foldable branches), after the {{ $s.LinkTitle }} </span> and before </a>:

{{- if isset $s.Params "companion" }}{{ if not $s.Params.companion }} <i class="fa-solid fa-lock companion-icon internal-only" title="Internal only"></i>{{ end }}{{ end -}}

layouts/all.md

Overrides Docsy’s layouts/all.md (the catch-all markdown output template).

Patch: One line changed in the LLMS index section to add a link to llms-full.txt alongside the existing llms.txt link.

Other overrides (no forward-porting needed)

These files are fully custom and don’t track an upstream template:

File Purpose
layouts/shortcodes/include.html Custom file-include shortcode
layouts/_default/_markup/render-link.html Link rewriting for hummingbird URLs and /l/ aliases
layouts/_default/_markup/render-heading.html Delegates to Docsy’s td/render-heading.html
layouts/partials/companion.html Environment-aware companion page banner
layouts/partials/breadcrumb.html Custom breadcrumb with stable-URL display
layouts/partials/footer.html Fully custom 3-column footer
layouts/home.html Custom home page with cover image and feature cards
layouts/index.llms.txt Custom recursive llms.txt with full doc tree
layouts/index.llms-full.txt Full content dump for LLM consumption
layouts/partials/llms-doc-tree.txt Recursive tree-walking partial for llms.txt
layouts/partials/llms-full-doc-tree.txt Recursive content partial for llms-full.txt

Upgrade procedure

1. Read the release notes

Check the Docsy changelog and the release upgrade guide for the target version. Note any breaking changes that affect templates we override.

2. Update the Hugo module

From the documentation repo worktree:

# Update to new version (replace vX.Y.Z with the target version)
podman run --pull=newer -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy"

3. Update npm dependencies

Docsy v0.16.0+ sources Bootstrap and Font Awesome from npm:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod npm pack && npm install --no-package-lock"

4. Forward-port layout overrides

For each of the four tracked overrides:

  1. Find the new upstream template in the Hugo module cache:

    find .hugo_cache -path "*docsy/theme@vX.Y.Z*" \
  -name "_td-content.html" -o \
  -name "sidebar-tree.html" -o \
  -name "all.md" -o \
  \( -name "list.html" -path "*docs*" \)

  2. Compare the new upstream against our current override to see what changed upstream.

  3. Start from the new upstream template and re-apply the patch documented in the inventory above.

  4. For docs/list.html, keep all upstream hooks (especially {{ .Render "_td-content-after-header" -}}).

5. Verify dark mode and code highlighting

The site uses Docsy’s light/dark mode toggle and dark-aware code syntax highlighting. After upgrading, confirm these config and SCSS settings are still in place:

  • config.yaml: params.ui.showLightDarkModeMenu: true
  • config.yaml: markup.highlight.noClasses: false (required for CSS-based Chroma themes to work)
  • assets/scss/_styles_project.scss: @import 'td/code-dark'; (loads light/dark Chroma themes: friendly for light, native for dark)

If a Docsy upgrade changes the dark mode mechanism, check the Look and Feel docs for updated instructions.

6. Build and validate

make build  # or: make build-host (if Hugo/Go are available locally)

The build should complete with no errors and no deprecation warnings.

Validation checklist

After upgrading, verify:

  • Build passes without errors or deprecation warnings
  • Companion banners render on docs pages with companion: true front matter
  • Lock icons appear in the internal site sidebar for internal-only pages
  • Home page cover image and search render correctly
  • Light/dark mode toggle works and all page elements adapt (navbar, sidebar, footer, code blocks)
  • Code blocks have dark background with readable syntax colors in dark mode
  • Agent-support outputs generate, if enabled (.md URL variants, /llms.txt, /llms-full.txt)
  • Stable URL aliases (/l/...) still resolve

For the internal site, also build infrastructure/internal-docs/ after updating its config/internal/config.yaml and go.mod to the same Docsy version.

Updating the internal site

The internal site only needs config and module changes (no layout work):

  1. Update internal-docs/config/internal/config.yaml: change the theme: list to match the public site.

  2. Update the Hugo module and rebuild:

    cd internal-docs
make container
# Inside container:
hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy
exit
make setup  # npm dependencies
make build

Rollback

To revert to a previous Docsy version:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vPREVIOUS && hugo mod tidy"

Then restore the layout overrides from the previous commit (git checkout HEAD~1 -- layouts/).

3.5 - Version Constraints for Multi-Version Images

Overview

Version constraints prevent unwanted major or minor version updates for multi-version container images (e.g., python-3-11, nodejs-20). Constraints are defined in properties.yml and validated during make check by comparing VERSION files against defined patterns.

Adding Version Constraints

Edit the image’s properties.yml to add a version constraint:

---
main_package: python3.11
version_constraints: '3.11.*'  # Allow 3.11.X, block 3.12+ and 4.X

The constraint applies to all distros for the image:

---
main_package: dotnet-sdk-8.0
version_constraints: '8.*'  # Applies to rawhide, hummingbird, and any other distros

Constraint Patterns

Constraints use glob-style patterns:

  • 3.11.* - Match any 3.11.X version (recommended for version families)
  • 20.* - Match any 20.X version
  • 8.* - Match any 8.X version
  • 1.25.* - Match any 1.25.X version (for Go-style 3-part versions)

The pattern matches against the version from the VERSION file. For example, if the VERSION file contains 3.11.14, the constraint 3.11.* will match.

How Validation Works

  1. Lockfile Generation: When make all runs, lockfiles are generated with resolved package versions
  2. VERSION File Creation: generate_jinja2.py creates VERSION files from resolved package versions
  3. Constraint Checking: When make check runs, check_version_constraints.py validates VERSION files against constraints
  4. Failure: If a violation is detected, the build fails with a detailed error message
  5. Renovate Integration: Renovate MRs that violate constraints fail CI and are blocked from automerge

Handling Constraint Violations

When a build fails due to a version constraint violation, you have several options:

Option 1: Accept the new version (create new image)

If the new major/minor version is acceptable:

  1. Create a new image directory (e.g., images/python-3-14/)

  2. Copy properties and configuration from the old version

  3. Update the constraint in the new directory:

    version_constraints: '3.14.*'

  4. Update repository field to group versions together

  5. Consider adding latest tag to the new version if appropriate

Option 2: Block the version update (exclude package)

If the new version should be blocked entirely:

  1. Add the package to yum-repos/*.repo excludepkgs
  2. Follow the procedure in Excluding Packages from Images
  3. Renovate will skip the excluded version in future updates
  4. When the issue is resolved, remove the exclusion

Option 3: Update the constraint (allow version bump)

If the version bump is acceptable for the existing image:

  1. Update the constraint in properties.yml:

    version_constraints: '3.12.*'  # Allow 3.12.X now

  2. Consider if this changes the image’s purpose (major version change)

  3. Update image documentation and tags accordingly

  4. Consider renaming the image directory to reflect the new version

Examples

Python Multi-Version Images

# images/python-3-11/properties.yml
distros:
  - hummingbird
main_package: python3.11
repository: python
version_constraints: '3.11.*'

# images/python-3-13/properties.yml
distros:
  - hummingbird
main_package: python3.13
repository: python
version_constraints: '3.13.*'

Node.js Multi-Version Images

# images/nodejs-20/properties.yml
main_package: nodejs20
repository: nodejs
version_constraints: '20.*'

# images/nodejs-24/properties.yml
main_package: nodejs24
repository: nodejs
version_constraints: '24.*'

.NET SDK Multi-Version Images

# images/dotnet-sdk-8-0/properties.yml
main_package: dotnet-sdk-8.0
repository: dotnet-sdk
version_constraints: '8.*'

.NET Runtime Multi-Version Images

# images/dotnet-runtime-8-0/properties.yml
main_package: dotnet-runtime-8.0
repository: dotnet-runtime
version_constraints: '8.*'

ASP.NET Runtime Multi-Version Images

# images/aspnet-runtime-8-0/properties.yml
main_package: aspnetcore-runtime-8.0
repository: aspnet-runtime
version_constraints: '8.*'

OpenJDK Multi-Version Images

# images/openjdk-21/properties.yml
main_package: java-21-openjdk-headless
repository: openjdk
version_constraints: '21.*'

Go Multi-Version Images (3-part versions)

# images/go-1-25/properties.yml
main_package: golang1.25
repository: go
distros:
  - hummingbird
version_constraints: '1.25.*'  # Matches 1.25.0, 1.25.3, etc.

Troubleshooting

Constraint not working

  • Verify package name matches exactly (case-sensitive)
  • Ensure pattern syntax is correct (use * wildcard)
  • Run make all to regenerate VERSION files before make check

False positive violations

  • Check for typos in the constraint pattern
  • Verify the pattern matches the version format (e.g., 3.11.* not 3.11*)
  • Ensure you’re using glob patterns, not regex

Package renamed in repositories

If a package is renamed (e.g., python3.11 → python311):

  1. Update main_package field
  2. Update constraint with new package name (not needed - constraints check VERSION files)
  3. Regenerate lockfiles and VERSION files with make all

Implementation Details

The version constraint system consists of:

  1. properties.yml: Optional version_constraints field with a glob pattern string
  2. ci/check_version_constraints.py: Python script that validates VERSION files
  3. Makefile integration: Script runs as part of make check
  4. CI integration: Failures block merge, including Renovate automerge

The validation process:

  1. Reads properties.yml for each image
  2. If version_constraints defined, finds all VERSION files across all distros
  3. Applies the same constraint pattern to all distros for the image
  4. Compares VERSION file content against glob pattern using Python’s fnmatch
  5. Reports violations with file path, version, constraint, and properties file location
  6. Exits with error code 1 if any violations found

See Also

3.6 - Retrying Konflux Checks

Automatically retrigger failed Konflux CI checks on merge requests using the retrigger script.

When Konflux CI checks fail on merge requests, you can automatically retrigger them using the provided retrigger script.

Prerequisites

  • GITLAB_TOKEN: GitLab API access token with api scope and Maintainer access

Usage

For automated retriggers of failed Konflux pipeline runs:

# Retrigger only failed pipeline runs and wait for them to start
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Preview what would be retriggered without posting comments
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

How It Works

The retrigger script will:

  1. Extract failed pipeline runs from the commit statuses on the merge request
  2. Generate /retest pipeline-name commands for each failed run
  3. Post the retest commands as merge request comments (unless --dry-run)
  4. Wait for the pipeline runs to start (unless --dry-run)

Manual Retrigger Commands

You can also manually retrigger specific checks using slash commands in merge request comments:

/retest gitlab-ci--default--main-on-pull-request

To retrigger a specific pipeline run as listed in .tekton/images-on-pull-request.yaml.

See the Tekton Pipelines as Code documentation for more details on available commands.

Examples

# Set your GitLab token
export GITLAB_TOKEN=your_token_here

# Retrigger failed checks for a specific merge request
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

# Dry run to see what would be retriggered
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

The script will output which pipeline runs are being retriggered and wait for them to start before completing.

3.7 - Switching Fedora Streams

AI Agent Note: When asked to switch Fedora streams, follow the workflow below. The repo file naming convention determines the GPG key automatically.

Overview

The rawhide distro can be pinned to a specific Fedora release branch instead of following the rolling Rawhide stream. This provides stability during the period when Rawhide moves to N+1 but the current stable release is still N-1.

For example, when Fedora 44 branches from Rawhide, the rawhide distro can be pinned to Fedora 44 packages while Rawhide itself starts tracking Fedora 45.

Repo ID naming convention

All Fedora repo files use a fedora-<stream> naming convention for their repo IDs, where <stream> matches the filename stem:

  • fedora-43.repo uses [fedora-43], [fedora-43-source], [fedora-43-updates], etc.
  • fedora-44.repo uses [fedora-44], [fedora-44-source], etc.
  • fedora-rawhide.repo uses [fedora-rawhide], [fedora-rawhide-source], etc.

This convention ensures:

  • No DNF repo ID conflicts when multiple repo files coexist in yum-repos/
  • Predictable, greppable repo IDs in lockfiles
  • The GPG key name can be derived automatically from the filename

Workflow

1. Create or update the repo file

Create a new file yum-repos/fedora-<version>.repo (e.g., fedora-45.repo).

The repo file must follow these conventions:

  • Repo IDs match the filename stem: Use [fedora-45], [fedora-45-debuginfo], and [fedora-45-source] as section names
  • Version-specific URLs: Point to the correct Fedora version’s repository (e.g., /development/45/ or /releases/45/)
  • Version-specific GPG key: Reference the correct signing key (e.g., RPM-GPG-KEY-fedora-45-$basearch)

Use an existing repo file (e.g., yum-repos/fedora-44.repo) as a template.

Example for a development branch:

[fedora-45]
name=Fedora 45 - Developmental packages for the next Fedora release
baseurl=https://koji-s3-cache.hummingbird-project.io/download-ib01.fedoraproject.org/pub/fedora/linux/development/45/Everything/$basearch/os/
enabled=1
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-45-$basearch
skip_if_unavailable=False

2. Update variables.yml

Edit images/variables.yml and change the default_variant_repos.rawhide entry to reference the new repo file:

default_variant_repos:
  rawhide:
    - fedora-45.repo

3. Regenerate all derived files

Run regeneration to update rpms.in.yaml files, Containerfiles, and lockfiles:

make -j 16 FORCE=true

4. Verify GPG key derivation

The Jinja templates in macros/setup_newroot.yml.j2 automatically derive the GPG key name from the repo filename:

  • fedora-45.repo becomes RPM-GPG-KEY-fedora-45-primary
  • fedora-44.repo becomes RPM-GPG-KEY-fedora-44-primary
  • fedora-rawhide.repo becomes RPM-GPG-KEY-fedora-rawhide-primary

No template changes are needed when switching versions. Verify by checking the generated Containerfile for the builder image contains the correct key:

grep rpmkeys images/hummingbird-builder/rawhide/default/Containerfile

Switching back to rolling Rawhide

To switch back to the rolling Rawhide stream, update variables.yml to reference fedora-rawhide.repo:

default_variant_repos:
  rawhide:
    - fedora-rawhide.repo

Then regenerate all derived files with make -j 16 FORCE=true.

3.8 - Disabling Rawhide for Images

AI Agent Note: When asked to disable Rawhide for images, follow the workflow below. Run make all after editing properties to clean up stale directories automatically.

Overview

When Rawhide is unstable (mass rebuild, broken critical packages, Fedora branching), Rawhide variants can be temporarily disabled for specific images or globally. This stops building and testing Rawhide variants while keeping Hummingbird variants unaffected.

Disabling Rawhide for a single image

1. Edit properties.yml

Add an explicit distros field to images/<name>/properties.yml that excludes rawhide:

distros:
  - hummingbird

This overrides the default distros (which include rawhide) for this image only.

2. Regenerate

make all

The stale Rawhide directories are removed automatically. Generated resources (.gitattributes, Konflux templates, Tekton pipelines) are updated to exclude the Rawhide variants.

3. Commit and merge

Commit the properties change together with the removed generated files.

Disabling Rawhide for all images

1. Edit variables.yml

Remove rawhide from default_distros in images/variables.yml:

default_distros:
  - hummingbird

2. Edit per-image overrides

Images that explicitly list distros including rawhide in their own properties.yml are not affected by the variables.yml change. Edit those files to remove rawhide as well.

3. Regenerate

make all

4. Commit and merge

Commit all properties changes together with the removed generated files.

Re-enabling Rawhide

Reverse the properties change (add rawhide back to default_distros or remove the per-image distros override), then regenerate:

make all

The Rawhide directories no longer exist, so Make creates all targets from scratch.

Verification

After running make all:

  • git status shows Rawhide directories as deleted (when disabling) or new files (when re-enabling)
  • .gitattributes no longer lists Rawhide paths for affected images
  • konflux-templates/rendered.yml and .tekton/ pipelines no longer reference Rawhide variants for affected images

See Also

3.9 - Setting Up a Konflux Cluster

How to set up a new Konflux cluster

Setting Up the Cluster

  1. Set up the new cluster in the infrastructure repository, mirroring the existing cluster configuration
  2. In the Konflux UI, create a new Component for the Application with the same name as the git repository
  3. Verify the Pipeline as Code Repository resource is created
  4. Run two successive full pipelines in the infrastructure repository to deploy resources from konflux/rendered.yml and fix up owner references
  5. Delete the initial Component created from the UI
  6. Disable Konflux build status comments (requires elevated permissions):
kubectl patch \
  --namespace hummingbird-tenant \
  repository REPOSITORY_NAME \
  --type merge \
  --patch '{"spec":{"settings":{"gitlab":{"comment_strategy":"disable_all"}}}}'

Next Steps

3.10 - Deleting RPMs from Hummingbird Repos

How to delete RPM packages from Hummingbird repositories

Overview

  • This document should be used to remove RPMs from the Hummingbird repositories

Setup

  • make sure you have followed the local development setup instructions in the infrastructure repository
  • install pulp cli via dnf install pulp-cli
  • setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

Steps

  • Note the names of the rpms to delete. In this example, we’ll use systemd-stub
  • Obtain the relevant metadata required to delete.
    • You need to execute this command once for each architecture.
pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.pulp_href)"'

or for prettier content,

pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.name) \(.version)-\(.release)  \(.arch)  \(.pulp_created)  \(.pulp_href)"'

Example output from basic command:

/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/
  • Using the hrefs, construct your delete commands:
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/

3.11 - Updating Dist-git Packages

How packages are automatically updated from Fedora dist-git

Packages are automatically updated from Fedora dist-git. Each update creates a separate MR that is automatically approved and merged when CI passes.

Testing Locally

# Dry-run (check all packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone

# Dry-run (check first 5 packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone --max-packages=5

# Check only a specific package (for testing/debugging)
./ci/dist_git_update_multi_mr.sh --clone --only-package=libgcrypt

# Check only clean packages (skip modified/independent)
./ci/dist_git_update_multi_mr.sh --clone --clean-only

# Check only modified packages (skip clean/independent)
./ci/dist_git_update_multi_mr.sh --clone --modified-only

# Create up to 3 test MRs (checks all packages, stops after finding 3 updates)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-updates=3 --create-mrs

# Check first 10 packages, create up to 3 MRs
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-packages=10 --max-updates=3 --create-mrs

# Create MRs only for clean packages (skip modified/independent)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --clean-only --create-mrs

Environment Variables

  • CHORE_MR_GITLAB_TOKEN - GitLab token with write_repository scope (required for --create-mrs)
  • CHORE_MR_APPROVAL_GITLAB_TOKEN - GitLab token with api scope for auto-approving MRs (used by CI)
  • GITLAB_REMOTE_URL - Target repo (default: https://gitlab.com/redhat/hummingbird/rpms.git)

Flags

  • --clone - Clone from GitLab to /tmp (safe for local testing, uses latest main branch)
  • --max-packages=N - Check only the first N packages (limits input set)
  • --max-updates=N - Stop after finding N updates (limits output MRs created)
  • --create-mrs - Actually create MRs (requires token)
  • --only-package=NAME - Check only the specified package (for testing/debugging specific packages)
  • --clean-only - Skip packages with modification_status of ‘modified’ or ‘independent’, only process clean packages
  • --modified-only - Skip packages with modification_status of ‘clean’ or ‘independent’, only process modified packages (mutually exclusive with --clean-only)

Using –clean-only

The --clean-only flag filters out packages marked as ‘modified’ or ‘independent’ before attempting updates. This is useful for:

  1. Better failure detection - Exit code 1 indicates real update failures, not expected errors from modified/independent packages
  2. Cleaner output - No error messages for packages that can’t be auto-updated by design
  3. Efficient CI - Focus on packages that should update automatically
  4. Performance - Avoids invoking dist_git.py for packages that will fail

Without --clean-only, the script attempts to update all packages. Modified/independent packages fail with:

ERROR: Cannot auto-update <package>
       Status: modified/independent
       Reason: <reason>
       Use 'sync' to force update or 'mark-modified --clean' to allow updates

These expected failures can mask genuine update issues. Using --clean-only prevents these false failures.

Using –modified-only

The --modified-only flag filters to only process packages marked as ‘modified’, skipping clean and independent packages. This is useful for checking the merge logic, as well as getting an overview of current merge conflicts.

Auto-Merge and Auto-Approval

MRs created by this script are configured to:

  • Auto-merge when pipeline succeeds (set via merge_request.merge_when_pipeline_succeeds)
  • Auto-approve after 10 minutes via the chore_mr_approval CI job (gives Konflux time to post commit statuses)

Version-Constrained Updates

Packages with track_upstream set to a version prefix in their metadata are version-constrained. This is used for versioned packages like golang1.26 that track a specific upstream version line.

How It Works

The track_upstream and release_monitoring_project_id metadata fields work together across two systems:

dist_git.py update (dist-git sync from Fedora):

When track_upstream is a version prefix (e.g., "1.26"), the update command uses prefix matching:

  • 1.26 matches 1.26, 1.26.0, 1.26.3 (allowed)
  • 1.26 does not match 1.27.0, 2.0 (skipped)

Skipped packages log a warning:

WARNING: Skipping golang1.26: upstream version 1.27.0 doesn't match tracked version 1.26

check_upstream_versions.py (release-monitoring.org checks):

When release_monitoring_project_id is a string, Anitya is queried using that name instead of the RPM package name. For example, golang1.26 with release_monitoring_project_id: "golang" queries Anitya for golang. When it is an integer, the v2 API is queried directly by project ID. When track_upstream is a version prefix, the list of upstream versions returned by Anitya is filtered to only those matching the prefix. This means check_upstream_versions.py check will report 1.26.5 as the latest version for golang1.26 even if Anitya reports 1.27.0 as the latest golang release.

When --update is used, the script updates the spec file, downloads new sources, and commits the result. Packages that need custom update logic can provide a hooks file at metadata/<package>.update-hooks.yaml to override the spec update, source download, or add a post-update step. See Package Modification Tracking for details.

Setting Up Version Constraints

# Constrain golang1.26 to only receive 1.26.x updates
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Remove the version constraint
./ci/dist_git.py set-upstream golang1.26 --no-track-version

Behavior

  • dist_git.py: Version constraint is checked before pre-release and Koji build checks
  • dist_git.py: The sync command bypasses the version constraint (explicit force operation)
  • check_upstream_versions.py: release_monitoring_project_id determines the Anitya lookup (int for project ID, string for name, absent for RPM name); track_upstream filters versions when set to a prefix
  • Batch operations continue processing other packages after skipping constrained ones

Pre-Release Version Filtering

By default, the update mechanism skips pre-release versions to prevent unstable packages from entering the repository automatically. Pre-release patterns include:

  • Tilde notation: 5.3.0~rc1, 2.0~beta1, 1.0~alpha (RPM standard)
  • Suffix notation: 5.3.0-rc1, 2.0.beta1, 3.0-alpha, 1.0.dev
  • Development markers: 1.5git20240101, 2024.01.snapshot, 1.0dev

Manual Override

To explicitly update to a pre-release version:

# Update single package to pre-release version
./ci/dist_git.py update --allow-prerelease package-name --skip-build-check

# Batch update allowing pre-releases
./ci/dist_git.py update --allow-prerelease --skip-build-check

Pre-release filtering behavior

  • Pre-release detection occurs before Koji build checks (saves API calls)
  • Skipped packages log a warning with the detected pattern
  • Batch updates continue processing other packages
  • The sync command bypasses this check (explicit force operation)

3.12 - Writing Documentation

How to contribute to the Project Hummingbird documentation.

Overview

New documentation is always welcome! This guide explains how to contribute to the Project Hummingbird documentation.

Documentation Sites

The Project Hummingbird documentation is open by default and available at two sites:

Documentation Repositories

Documentation content comes from six repositories:

  1. Containers repository - Documentation focused on container images (cross-cutting layout):

    • documentation/contributing/ - How to add and modify container images
    • documentation/background/ - Container image architecture and reference docs
    • documentation/ci-scripts/ - CI script documentation
    • documentation/operating/ - Operator runbooks for image maintenance
    • documentation/using/ - End-user guides (custom CA certificates, etc.)
    • README.md - User guide
    • CONTRIBUTING.md - Quickstart guide

  2. RPMs repository - Documentation focused on RPM packages (cross-cutting layout):

    • documentation/background/ - RPM pipeline architecture and Konflux deployment
    • documentation/operating/ - Operator runbooks for package management
    • CONTRIBUTING.md - Contributing guide for RPM packages

  3. Tools repository - Documentation for infrastructure tools and services (per-component layout):

    • documentation/ - One page per tool/service (flat structure)

  4. K8s test pipeline repository - Documentation for Kubernetes integration testing (per-component layout):

    • documentation/ - Pipeline design, test format, and EaaS debugging

  5. Public documentation repository - Site infrastructure and cross-cutting guides:

    • content/docs/using/ - End-user guides for using container images
    • content/docs/operating/ - Infrastructure operation guides (Konflux, GitLab, etc.)
    • content/docs/background/ - General architecture and concepts

  6. Infrastructure repository - Internal-only documentation (cross-cutting layout):

    • documentation/operating/ - Internal operational docs (AWS, DNS, GitLab, etc.)
    • documentation/background/ - Internal background docs (error budgets, etc.)
    • documentation/presentations/ - Team presentations and talks

How They Work Together

The repositories are layered using Hugo Modules:

  1. Public documentation imports specific directories from the containers, rpms, tools, and k8s-test-pipeline repositories
  2. Internal documentation is built from the infrastructure repository’s internal-docs/ subtree. It imports the public documentation (which transitively includes the imported repository documentation) and mounts the infrastructure repo’s documentation/ directory as the Internal section

Hugo Module Mounts:

# In documentation/config.yaml
module:
  imports:
    - path: gitlab.com/redhat/hummingbird/containers
      mounts:
        - {source: 'README.md', target: content/snippets/readme.md}
        - {source: 'CONTRIBUTING.md', target: content/snippets/contributing.md}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/background', target: content/docs/background/containers, files: "*.md"}
        - {source: 'documentation/ci-scripts', target: content/docs/background/containers/ci-scripts, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/rpms
      mounts:
        - {source: 'CONTRIBUTING.md', target: content/snippets/rpms-contributing.md}
        - {source: 'documentation/background', target: content/docs/background/rpms, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/tools
      mounts:
        - {source: 'documentation', target: content/docs/background/tools, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
      mounts:
        - {source: 'documentation', target: content/docs/background/k8s-test-pipeline, files: "*.md"}

For README.md and CONTRIBUTING.md, front matter is provided via wrapper files.

This creates a unified documentation site from multiple repositories without duplication.

Documentation Layout Patterns

Source repositories use one of two layout patterns depending on their content:

Cross-cutting layout (containers, rpms): The repository documents cross-cutting concerns like contributing, operating, and background information. Documentation is organized into subdirectories matching the site sections:

  • documentation/background/ - Explanation and reference material
  • documentation/contributing/ - How to contribute
  • documentation/operating/ - Operator runbooks
  • documentation/using/ - End-user guides

Each subdirectory is mounted into the matching site section, so operating docs from different repositories appear together under Operating.

Per-component layout (tools, k8s-test-pipeline): The repository is a monorepo where each documentation page covers one tool or component. Documentation is a flat directory with one file per component:

  • documentation/<component-name>.md

The entire directory is mounted under background/<repo-name>/ on the site.

Getting Started

Quick Start

Clone, install dependencies, and start the documentation server:

git clone https://gitlab.com/redhat/hummingbird/documentation
cd documentation
make setup  # Installs npm dependencies; only needed once per checkout
make serve  # Available at http://localhost:1313/

For the internal documentation:

git clone https://gitlab.com/redhat/hummingbird/infrastructure
cd infrastructure/internal-docs
make serve  # Available at http://localhost:1314/

For the best development experience, set up all repositories with direnv:

mkdir -p ~/git/hummingbird
cd ~/git/hummingbird

# Clone all repositories
git clone https://gitlab.com/redhat/hummingbird/containers
git clone https://gitlab.com/redhat/hummingbird/rpms
git clone https://gitlab.com/redhat/hummingbird/tools
git clone https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
git clone https://gitlab.com/redhat/hummingbird/documentation
git clone https://gitlab.com/redhat/hummingbird/infrastructure

# Set up environment variables
cat > .envrc << 'EOF'
export CONTAINERS_REPO_PATH=$(pwd)/containers
export RPMS_REPO_PATH=$(pwd)/rpms
export TOOLS_REPO_PATH=$(pwd)/tools
export K8S_TEST_PIPELINE_REPO_PATH=$(pwd)/k8s-test-pipeline
export DOCUMENTATION_REPO_PATH=$(pwd)/documentation
export INFRASTRUCTURE_REPO_PATH=$(pwd)/infrastructure
EOF
direnv allow

This automatically uses your local repositories for imported content, making changes immediately visible in the documentation preview.

Adding Documentation

Add new documentation as markdown files inside the appropriate directory under content/. Every file should end in .md.

Preview your changes:

make serve  # Starts local server with live reload

Check for problems:

make check  # Runs linters and validation

Documentation uses two types of internal links depending on the context:

Use relative .md links for pages in the same directory, and for cross-section links in the documentation repository:

[Testing Guide](testing-images.md)
[Adding Images](adding-images.md)

Use full URLs with /l/ aliases for links across repositories or across sections in all repositories:

[Image Pipeline][image-pipeline]
[Global Variables][global-vars]
[Retrying Checks][retrying-checks]

[image-pipeline]: https://hummingbird-project.io/l/image-pipeline
[global-vars]: https://hummingbird-project.io/l/global-variables-reference
[retrying-checks]: https://hummingbird-project.io/l/retrying-konflux-checks

Why full URLs?

  • Work in standalone contexts (GitHub, text editors, when files are copied)
  • Hugo’s link render hook automatically converts them to relative links in the rendered site
  • Stable even when files are moved or reorganized
  • Clear and explicit about which page is being referenced

The documentation site includes a custom link render hook (layouts/_default/_markup/render-link.html) that:

  1. Detects https://hummingbird-project.io/l/... URLs
  2. Looks up the page with the matching alias
  3. Converts to a relative link in the rendered HTML
  4. Keeps external links as-is with target="_blank"

This provides the best of both worlds: full URLs in markdown source files that work everywhere, and optimized relative links in the rendered site.

All cross-referenced pages should have stable /l/ aliases. These aliases serve two purposes:

  1. Incoming links from external sources (documentation, scripts, merge request comments)
  2. Cross-references within documentation (across sections or repositories)

Adding an Alias

Add an alias in the page front matter:

---
title: "Konflux Integration"
aliases: [/l/konflux-integration]
---

Now you can link to https://hummingbird-project.io/l/konflux-integration and it will continue working even if the page moves.

When to Add Aliases

Add /l/ aliases to pages that:

  • Are referenced from other repositories (e.g., containers repo → docs repo)
  • Are linked to from CI scripts or infrastructure
  • Are shared in merge request comments or external documentation
  • Are referenced across documentation sections

Naming Convention

  • Use /l/ prefix for all stable links
  • Keep names short, memorable, and descriptive
  • Use kebab-case (lowercase with hyphens)
  • No nested paths (flat structure only)

Submitting Changes

  1. Fork the repository on GitLab
  2. Clone your fork locally
  3. Create a new branch for your changes
  4. Make your changes and test them with make serve
  5. Run make check to ensure there are no errors
  6. Commit your changes with a descriptive commit message
  7. Push to your fork
  8. Create a merge request against the main repository

Documentation Structure

The public documentation is organized into several main sections:

  • Using - Guides for users of Project Hummingbird container images
  • Contributing - Guides for contributors to the project
  • Operating - Guides for infrastructure operators
  • Background - Technical background and architecture information

The internal documentation adds:

  • Internal - Internal-only infrastructure documentation, organized into:
    • Operating - AWS, DNS, GitLab, OpenShift, and other service runbooks
    • Background - Error budgets, SLIs/SLOs
    • Presentations - Team talks and lightning talks

When adding new documentation, place it in the most appropriate section.

3.13 - Adding Independent Packages

How to add packages that originate in the Hummingbird repository

Independent packages are packages that originate in the Hummingbird repository rather than being imported from Fedora dist-git.

Steps to Add an Independent Package

1. Create Package Directory

mkdir rpms/<package-name>
cd rpms/<package-name>

2. Add Package Files

Create the following files in the package directory:

  • <package-name>.spec - RPM spec file
  • sources - SHA512 checksums of source tarballs
  • Source tarballs (.tar.gz, .tar.bz2, .tar.xz)
  • Any patches or additional configuration files

3. Generate the sources File

The sources file must contain SHA512 checksums in BSD-style format:

sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

This produces the correct format:

SHA512 (oras-1.3.0.tar.gz) = fb871c0577f621f7e1f56a54f249f96c659b29c771dad529f9a9e838379b2d56bca9cef03a90071915a568e28e266326989e67359e589a87908c4fc077b14689
SHA512 (oras-1.3.0-vendor.tar.bz2) = fa23324bf3910c3dc050eb3c0ebfef3224168489679cf1195d668c656c6c53beaac94b4786e45c2cf3782f9437b0d166da9c18e608a3b474b04957ff1dec5226

Important: Use sha512sum --tag to generate the BSD-style format. Do not use the default GNU format.

4. Create Package Metadata

Create metadata/<package-name>.json:

{
  "modification_status": "independent",
  "upstream_repo": "https://github.com/example/project",
  "version": "1.3.0"
}

Fields:

  • modification_status: Must be "independent" for packages not imported from Fedora (see Package Metadata Fields)
  • upstream_repo: Canonical upstream git repository URL (required — CI enforces this). If no upstream repo exists, use https://src.fedoraproject.org/rpms/<name> as a fallback.
  • version: Package version (must match spec file)

5. Generate Konflux Resources

Generate the Konflux Component and ImageRepository resources:

make generate-host

Or directly:

python3 ci/generate_resources.py all

This updates:

  • konflux-templates/rendered.yml
  • .tekton/ pipeline files

6. Commit Changes

git add rpms/<package-name>/ metadata/<package-name>.json konflux-templates/ .tekton/
git commit -m "Add <package-name>-<version>-<release>

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"

Note: Source tarballs (.tar.gz, .tar.bz2, .tar.xz) are automatically ignored by .gitignore and should be uploaded to the lookaside cache instead.

Example: Adding oras

# 1. Create directory
mkdir rpms/oras
cd rpms/oras

# 2. Add spec file and sources
# ... create oras.spec ...

# 3. Generate sources file
sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

# 4. Create metadata
cat > ../../metadata/oras.json << 'EOF'
{
  "modification_status": "independent",
  "upstream_repo": "https://github.com/oras-project/oras",
  "version": "1.3.0"
}
EOF

# 5. Generate Konflux resources
make generate-host

# 6. Commit
git add rpms/oras/ metadata/oras.json konflux-templates/ .tekton/
git commit -m "Add oras-1.3.0-1.hum1"

See Also

3.14 - Pulp Access

Overview

Pulp is used to host and distribute Hummingbird RPM repositories. Access to the Pulp API requires a service account with credentials stored in the vault.

Existing credentials

The public Pulp service account credentials are stored in the vault at rhel-primitives/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE. Retrieve the cli.toml field and save it locally (see ci/pulp-setup/README.md for usage).

Creating a new Pulp service account

When you need a dedicated service account (e.g., for private product repositories):

  1. Go to https://access.redhat.com/terms-based-registry/
  2. Log in with your @redhat.com account
  3. Click New Service Account
  4. Enter a username like hummingbird-<qualifier>-pulp-bot
  5. Enter a description like bot account for hummingbird <qualifier> pulp operations
  6. Click Create
  7. Capture the full username (in the format <id>|<username>) and the token

Configure the CLI

Create a cli.toml with the new credentials:

[cli]
base_url = "https://packages.redhat.com"
api_root = "/api/pulp/"
username = "<id>|hummingbird-<qualifier>-pulp-bot"
password = "<token>"
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
domain = "<target-domain>"

You can either save this to ~/.config/pulp/cli.toml or pass it explicitly with --config ./cli.toml when running scripts.

Store credentials in the vault

Store the credentials in the vault at apps/hummingbird with the key HUMMINGBIRD_<QUALIFIER>_PULP_BOT_CONFIG_FILE (e.g., HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE). Create two subkeys:

  • cli.toml — the full config file contents
  • value — the token value

3.15 - Measuring Documentation Quality

The documentation quality evaluation measures whether public documentation is complete enough for an LLM with web search to answer questions correctly. Low scores indicate documentation gaps; score improvements after content changes validate the fix.

For config schema, CLI options, and scoring details, see the Documentation Quality Evaluation reference in the tools repo. For the general evaluation framework, see Hummingbird Agent Evals.

How it works

The evaluation runs in two phases:

  1. Ask – multiple LLM configurations answer each golden question using web search against public documentation. Models never see internal sources; the test is whether public docs contain the information.
  2. Judge – a separate LLM scores each response against expected facts and coherence criteria defined in the golden dataset.

Documentation quality is the variable being measured, not model quality. Models are the instrument – if scores are low across all models, the documentation is incomplete.

The evaluation is resume-friendly: existing response and scored files are skipped on re-run, so interrupted runs can be continued without repeating expensive model calls.

Golden dataset

The config and golden dataset live in evals/docs_eval.yml in this repo. The file combines evaluation settings (models, judge, repetitions) with scoring criteria (questions, expected facts, coherence checks, thresholds).

Running

Prerequisites

  • GCP Application Default Credentials:

    gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=<gcp-project>

  • The hummingbird-agent package installed (pip install -e . from the hummingbird-agent/ directory in the tools repo), or the container image.

Via Makefile

make eval-docs

Manual

python -m hummingbird_agent.evals.docs_eval.run_eval \
  -c evals/docs_eval.yml

Writing questions

Each question tests a documentation topic, not a model capability. When adding or editing questions:

  • Facts should be verifiable from public documentation. Avoid opinions or information that requires internal knowledge.
  • Assign higher weights to more important facts. A weight-2 fact contributes twice as much to the score as a weight-1 fact.
  • The coherence criterion checks narrative quality, not just fact presence. A response that lists correct facts without connecting them should score lower on coherence.
  • Calibrate the threshold by running a baseline: set the threshold where current documentation should fail but improved documentation should pass.

3.16 - Rebuilding Packages

AI Agent Note: When asked to rebuild packages, use the rebuild command: ./ci/dist_git.py rebuild <package> --reason "<reason>". For rebuilding reverse dependencies (e.g., “rebuild all Go packages”), use rebuild-rev-deps <package> --reason "<reason>". Always ask the user for a ticket link or explanation first to use as the reason. If the command fails with “uses macros in Release field”, see the “Packages requiring manual rebuild” section below for instructions.

Overview

This document covers three scenarios for triggering package builds:

  1. No-change rebuild: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes).

  2. Rebuilding reverse dependencies: Rebuild all packages that depend on a changed package (e.g., rebuild all Go packages when golang updates).

  3. Backporting a patch: Add an upstream patch that hasn’t yet landed in Fedora to fast-track a fix or feature.

All scenarios use the .N release suffix pattern to ensure our builds sort higher than the upstream Fedora release while remaining lower than the next upstream version.

No-Change Rebuild

The rebuild command automates the Release field bump:

# Rebuild a single package
./ci/dist_git.py rebuild <package> --reason "<reason>"

# Rebuild multiple packages (one commit per package)
./ci/dist_git.py rebuild <package1> <package2> ... --reason "<reason>"

# Rebuild all packages (one commit per package)
./ci/dist_git.py rebuild --all --reason "<reason>"

# Rebuild all packages except specific ones (requires --all)
./ci/dist_git.py rebuild --all --exclude <pkg1>,<pkg2> --reason "<reason>"

Examples:

# Single package
./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes"

# Multiple packages
./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds"

# All packages (useful after toolchain updates)
./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15"

# All packages except a few already rebuilt earlier in the same rollout
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm --reason "toolchain update: GCC 15"

--exclude takes a comma-separated list of package names and only works together with --all — it is rejected when combined with an explicit package list, since you can simply omit the packages you don’t want to rebuild in that case.

The command:

  • Automatically bumps the Release field using the .N suffix pattern
  • Handles %autorelease by resolving and replacing with explicit values
  • Preserves macros in the Release field (e.g., %{revision})
  • Creates a properly formatted commit message (one per package with --all)
  • Does not mark the package as modified (release-only changes are ephemeral)

Supports --dry-run to preview changes without committing:

./ci/dist_git.py --dry-run rebuild <package> --reason "test"

Creating MRs for rebuild commits

After creating rebuild commits locally, use rebuild_multi_mr.sh to push each commit as its own merge request (one MR per package, auto-merge enabled):

./ci/rebuild_multi_mr.sh

By default the script compares against origin/main. For local development, use --base to point at a different ref:

# Use local main branch as base (useful when origin/main is not up to date)
./ci/rebuild_multi_mr.sh --base main

# Use a specific commit SHA as base
./ci/rebuild_multi_mr.sh --base abc1234

# Preview what would be created without pushing
./ci/rebuild_multi_mr.sh --dry-run

# Limit to at most N MRs
./ci/rebuild_multi_mr.sh --max-updates=5

The script:

  • Creates a chore/rebuild-{package} branch per commit and pushes it
  • Titles each MR chore(rpms): Rebuild {package}: {reason}
  • Enables auto-merge on all rebuild MRs
  • Leaves the current branch untouched
  • Skips branches that already exist on the remote (idempotent)
  • Only processes commits whose message matches Rebuild {package}: {reason}; other commits in the range are silently skipped

Full workflow example:

# 1. Create the rebuild commits
./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update"

# 2. Preview the MRs that would be created
./ci/rebuild_multi_mr.sh --base main --dry-run

# 3. Create the MRs
./ci/rebuild_multi_mr.sh --base main

Packages requiring manual rebuild

Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file.

Macro indirection patterns

These packages define the Release field using a macro, where the macro itself contains %{?dist}. The rebuild command cannot detect or manipulate these without expanding all macros, which would break the macro system.

nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):

%{load:%{_sourcedir}/nodejs.srpm.macros}
%nodejs_define_version node 1:25.8.2-%{autorelease} -p
...
Release: %{node_release}

The %{node_release} macro is defined by an external macro system loaded from nodejs.srpm.macros. The release component is embedded in the version definition.

How to rebuild: Edit the %nodejs_define_version node line to bump the release component (e.g., change -%{autorelease} to -1.1 or increment existing .N).

kernel-headers:

%define specrelease 59%{?buildid}%{?dist}
...
Release: %{specrelease}

How to rebuild: Edit the %define specrelease line to add/increment the .N suffix before %{?buildid}:

%define specrelease 59.1%{?buildid}%{?dist}

krb5:

%global krb5_release 4%{?dist}
...
Release: %{krb5_release}

How to rebuild: Edit the %global krb5_release line to add/increment the .N suffix:

%global krb5_release 4.1%{?dist}

Why these can’t be automated

The rebuild command can handle:

  • ✅ Simple numeric: Release: 5%{?dist}
  • ✅ Macros ending with dist: Release: %{baserelease}%{?dist} (e.g., rpm, gcc)
  • ✅ Complex macros with dist: Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist}
  • ✅ Content after dist: Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}} (e.g., unbound)

The rebuild command cannot handle:

  • ❌ Macros without %{?dist}: Release: %{node_release}
  • ❌ Macros where dist is inside the macro definition: %{krb5_release} contains %{?dist}

This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM’s full macro parser.

Manual rebuild process

If you need to rebuild manually or the automated command doesn’t work for your use case, follow these steps:

1. Identify the package to rebuild

Identify the source package name and locate its spec file in rpms/<package>/<package>.spec.

If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --queryformat '%{SOURCERPM}' <binary-package> 2>/dev/null

Example: ncurses-libs-6.5-8.20250614.hum1 -> SRPM ncurses-6.5-8.20250614.hum1.src.rpm -> spec file at rpms/ncurses/ncurses.spec

2. Determine the Release bump pattern

The .N bump suffix must always appear immediately before %{?dist}. The %{?dist} suffix should always be the final component since it identifies the build environment.

Current Pattern Example Before Example After
Simple numeric Release: 3%{?dist} Release: 3.1%{?dist}
Already bumped Release: 3.1%{?dist} Release: 3.2%{?dist}
With macro Release: 8.%{revision}%{?dist} Release: 8.%{revision}.1%{?dist}
autorelease Release: %autorelease Release: 1.1%{?dist}

For %autorelease, first resolve its value using rpmspec, then replace with the resolved value plus .1. In the Hummingbird monorepo, %autorelease always evaluates to 1.

Note: If the Release field is missing %{?dist} entirely or looks unusual (e.g., 1build1 instead of 1.1%{?dist}), flag this to the user for resolution. Check the git history to understand the original value:

git log -p -S "Release:" -- rpms/<package>/<package>.spec

This helps determine the correct fix when a previous bump was malformed.

3. Modify the spec file

Use sed to edit only the Release: line, avoiding any unintended whitespace changes that text editors may introduce:

sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms/<package>/<package>.spec

Verify the change with git diff before committing:

git diff rpms/<package>/<package>.spec

The diff should show only the Release line change:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

Important: Only modify the Release line. Do not introduce any other changes such as whitespace fixes or trailing newline modifications. If the diff shows additional changes, reset and retry with sed.

Important: Do not change the release field in metadata/<package>.json during local rebuilds or backports. That field records the Fedora release for the version currently shipped and is present only while Hummingbird ships Fedora’s release; see Package Metadata Fields.

4. Verify the bump is correct

Use rpm --eval to confirm the new release sorts higher than the original:

# Returns -1 if first < second (correct), 1 if first > second (wrong)
rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}'
# Expected output: -1

5. Commit the change

Use this commit message format:

Rebuild <package>: <reason>

<ticket link or explanation>

Example:

Rebuild ncurses: published multiple times with different hashes

HUM-1234

6. Verify the commit

After committing, verify only the Release line was changed:

git show --stat HEAD

Expected output should show exactly 1 insertion and 1 deletion:

 rpms/<package>/<package>.spec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

If the commit shows more changes, amend or reset and redo the change using sed.

Important notes about rebuilds

Modification status

Rebuilds do not change a package’s modification_status. Spec Release: bumps are ephemeral and do not make a package modified versus clean. See Package Metadata Fields for how modification_status and metadata release relate to rebuilds.

During updates, Release lines are normalized to avoid conflicts, and the automation ignores Release-only changes, so automatic Fedora updates continue normally after a rebuild.

If you want to explicitly prevent automatic updates (e.g., you’re investigating an issue), you can manually mark the package as modified:

./ci/dist_git.py mark-modified <package> --modified --reason "Investigating build issue"

Note: This will block automatic Fedora updates until you mark it clean again.

Rebuilding Reverse Dependencies

When a compiler, runtime, or toolchain package changes, you may need to rebuild all packages that depend on it. The rebuild-rev-deps command automates finding and rebuilding reverse dependencies.

When to use

Common scenarios for rebuilding reverse dependencies:

  • Golang/Python runtime updates: When updating golang1.26, python3.14, etc., rebuild all Go/Python packages
  • Toolchain changes: When updating gcc, rebuild packages that BuildRequire it
  • Library ABI changes: When a library’s ABI changes, rebuild packages that BuildRequire it

Usage

./ci/dist_git.py rebuild-rev-deps <package> --reason "<reason>"

The command:

  1. Finds all packages that have BuildRequires: <package> in their spec files
  2. Rebuilds each package (bumps Release field and commits)
  3. Reports a summary of successful/failed rebuilds

Examples:

# Rebuild all Go packages when golang updates
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "golang 1.26.2 update"

# Rebuild all Python packages when python updates
./ci/dist_git.py rebuild-rev-deps python3.14 --reason "python 3.14.1 update"

# Rebuild packages that depend on a specific library
./ci/dist_git.py rebuild-rev-deps openssl --reason "openssl 3.4.0 update"

Virtual BuildRequires (golang/python)

For virtual BuildRequires like golang1.25, golang1.26, python3.13, python3.14, the command automatically handles translation to the actual BuildRequires target:

# These all work the same way:
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "..."
./ci/dist_git.py rebuild-rev-deps go-rpm-macros --reason "..."

Important: Only the latest version triggers rebuilds. This is because:

  1. Multiple golang versions exist: golang1.25, golang1.26
  2. They all provide the same virtual package: Provides: golang = <version>
  3. DNF always picks the highest version to satisfy Requires: golang

Therefore:

  • ✅ rebuild-rev-deps golang1.26 → Rebuilds all Go packages (latest version)
  • ❌ rebuild-rev-deps golang1.25 → Error: Not the latest version

This ensures rebuilds only happen when the active runtime actually changes.

Creating MRs for reverse dependency rebuilds

After creating rebuild commits, use rebuild_multi_mr.sh to push each commit as its own MR:

./ci/rebuild_multi_mr.sh --base main

See the “Creating MRs for rebuild commits” section above for full details.

Dry-run mode

Preview which packages would be rebuilt without making changes:

./ci/dist_git.py --dry-run rebuild-rev-deps golang1.26 --reason "test"

This shows:

  • Which packages have the BuildRequires dependency
  • What the new Release values would be
  • Does not commit any changes

Backporting a Patch

Use this workflow when you need to fast-track an upstream fix or feature that hasn’t yet been released in Fedora.

1. Obtain the patch

Fetch the patch from the upstream repository. For GitHub PRs, append .patch to the PR URL:

curl -L https://github.com/<org>/<repo>/pull/<number>.patch \
  > rpms/<package>/<NNNN>-<short-description>.patch

Name the patch file with a numeric prefix matching the next available PatchN: slot in the spec file (e.g., 0004-fix-foo.patch if Patch1-3 already exist).

2. Add the patch to the spec file

Add a PatchN: declaration after the existing patches:

Patch3:         0003-existing-patch.patch
Patch4:         0004-fix-foo.patch

The patch will be applied automatically if the spec uses %autosetup -p1. If the spec uses explicit %patchN macros, add the corresponding apply line in the %prep section.

3. Check for a gorget source-pipeline

If metadata/<package>.source-pipeline.yaml exists, check whether its transform: step applies patches or otherwise runs against patched source (e.g. resolving a lockfile with yarn install, pnpm fetch, npm ci):

# run from the root of the rpms repo checkout
grep -n -E "patch -p[0-9]|git apply" metadata/<package>.source-pipeline.yaml

This catches the common forms, but isn’t exhaustive — if the transform applies patches some other way (a different tool, or a script that wraps patch/git apply), read the transform: step directly rather than trusting a grep miss.

If it does, and your new patch touches a file that step depends on (a lockfile or manifest such as yarn.lock, package.json, pnpm-lock.yaml, go.sum, Cargo.lock), add the same patch -p1 < "${PACKAGE_DIR}/<your-patch>.patch" line there too, in the same relative order as its PatchN: slot in the spec. The patch file itself needs no extra fetch step — it’s already in ${PACKAGE_DIR}/ because you committed it there in step 2.

The pipeline’s transform: step and the spec’s %prep/%goprep are two independent lists of patches to apply — nothing keeps them in sync automatically. A patch declared only in the spec is invisible to whatever artifact the transform step generates (e.g. an offline yarn/pnpm cache built from the lockfile), and the generated artifact silently drifts from what %build actually applies. See “Known sharp edge: patch-list duplication” in the source-pipeline design doc for the grafana12.4/grafana13.1 incident this rule comes from. test/test_source_pipeline_patches.py checks for this drift, but treat it as a safety net, not a substitute for updating the pipeline yourself here.

4. For Go packages: check go-vendor-tools.toml’s pre_commands

This applies whether or not the package uses gorget — rpms/<package>/go-vendor-tools.toml’s [archive] pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately:

grep -n -A2 "pre_commands" rpms/<package>/go-vendor-tools.toml

If your new patch targets a vendored Go module (or you’re editing pre_commands directly instead of adding a patch — don’t), and any pre_commands entry mutates go.mod/go.sum (directly, via go get, or via go mod tidy/edit), your patch must apply the same go.mod/go.sum change to the actual source tree, not just to go-vendor-tools.toml. pre_commands cannot substitute for this: they require live network access to run go get, which %prep doesn’t have (Konflux builds are hermetic). Compute the resulting go.mod/go.sum diff offline (e.g. clone upstream, apply the same edits, run go mod tidy if pre_commands does) and commit that as the patch.

Skipping this leaves go.mod in the build tree and vendor/modules.txt in the generated vendor archive silently requiring different versions of the same package, which go build -mod=vendor rejects as inconsistent vendoring. See the source-pipeline design doc for the trivy incident this rule comes from. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

5. Bump the Release

Follow the same .N suffix pattern as no-change rebuilds:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

6. Commit the patch

Use this commit message format:

<package>: backport <short description>

Backport: <link to PR or commit>
<ticket link if applicable>

Example:

dnf5: backport reproducible build sorting fix

Backport: https://github.com/rpm-software-management/dnf5/pull/2522

7. Mark package as modified

Mark the package as modified to prevent automatic Fedora updates from overwriting your backport:

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Backport fix for <issue description>"

Example:

./ci/dist_git.py mark-modified dnf5 --modified \
  --reason "Backport reproducible build sorting fix from upstream PR#2522"

This ensures the package won’t be automatically updated from Fedora until the backported patch lands upstream and you explicitly mark it clean again.

8. Test the build locally (optional)

Build the package locally to verify the patch applies cleanly:

./ci/build_rpms.sh <package>

Built RPMs will be in builds/<package>/RPMS/.

3.17 - Setting Up Pulp Repositories

How to create and set up Pulp domain and repositories

Overview

  • This document and script should be used to initialize and setup a pulp domain and a set of repositories
  • We choose the domain prefix of public- to ensure that the repositories are publicly available

Setup

  • make sure you have followed the local development setup instructions in the infrastructure repository
  • install pulp cli via dnf install pulp-cli
  • setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

Create

./create-pulp-resources.sh public-hummingbird "source,x86_64,s390x,ppc64le,aarch64"

3.18 - Package Metadata Fields

Overview

Each package has a metadata file at metadata/<package>.json. Two fields control how Hummingbird treats local changes and rebuilds relative to Fedora:

Field Purpose
modification_status Whether the package may be auto-updated from Fedora
release Fedora release from which this package was imported

This page is the canonical reference for configuring those fields. For day-to-day workflows (marking packages, rebuilding, importing), see the related docs at the end.

modification_status

modification_status records whether a package still matches its Fedora upstream import, has local source-level changes, or is Hummingbird-independent (not from Fedora).

Status Meaning Auto-updates from Fedora
clean Unmodified Fedora import Allowed
modified Local changes (patches, spec modifications) Blocked
independent Hummingbird-independent package (not from Fedora) Blocked

When to use each value

  • clean: Default for packages imported from Fedora with no local source changes. Automatic Fedora updates are allowed. Conflicts with modification_reason (must be absent).
  • modified: Use after backports, custom patches, or other spec/source edits that must not be overwritten by an automatic update. Requires modification_reason.
  • independent: Required for packages that are not imported from Fedora. These packages have no source / branch / sha fields. Conflicts with modification_reason (must be absent).

Imports set the status automatically (clean for Fedora imports, independent for Hummingbird-independent imports). After local source changes, set status with dist_git.py mark-modified — see Package Modification Tracking.

Rebuilds that only bump the spec Release: line do not change modification_status. Release-only changes are ephemeral and are ignored when deciding whether a package is modified versus clean.

modification_reason

When modification_status is modified, modification_reason is required. It should be a short explanation of why the package cannot be auto-updated (for example, a CVE backport or a custom spec change). Clear the reason when marking the package clean again.

modification_reason applies only to modified packages. Do not add it for clean or independent packages — see When not to update these fields.

release

The release field in metadata is not the same as the Release: line in the package spec. It records the Fedora release from which the package was imported, without a dist tag such as .fc42 or .hum1. It is present only when Hummingbird ships Fedora’s release.

Location What it represents
metadata/<package>.json → release Fedora release from which the package was imported, without its dist tag
Spec Release: The release used for the Hummingbird build (includes %{?dist} / .hum1, and may include local .N rebuild suffixes)

Who writes metadata release

Writer When Value stored
dist_git.py import / update / sync Fedora import or refresh Resolved Fedora release with dist suffix stripped (e.g. 5 from 5.fc42)
check_upstream_versions.py Local upstream version bump (check --update) Removes release, because the new spec release is not from Fedora

Fedora-imported packages

For packages with a Fedora source:

  • release is set on import and refreshed on dist_git.py update / sync from the upstream Fedora package (dist suffixes like .fc42 are stripped). That is the Fedora/rawhide base.
  • When check_upstream_versions.py check --update bumps the package ahead of Fedora, it resets the spec to Release: 0.1%{?dist} (unless %autorelease) and removes metadata release. The local 0.1 is not a Fedora-confirmed release.
  • Do not change metadata release during local rebuilds or backports. Rebuilds bump only the spec Release: line; when no Fedora baseline exists, rebuild advances 0.1 to 0.2.
  • When Fedora’s baseline is Release: %autorelease, metadata release stores the resolved numeric release from MDAPI (not the literal %autorelease). Import/update replace %autorelease in the local spec with that value plus %{?dist}. For rebuilds that still see %autorelease, follow Rebuilding Packages.

Independent packages

For Hummingbird-independent packages (modification_status: "independent", no source field):

  • There is no Fedora upstream release, so metadata does not contain release.
  • The .hum1 suffix comes from the spec Release: line via %{?dist} (for example Release: 1%{?dist}), not from metadata release.
  • dist_git.py rebuild does not use metadata release for independent packages.

Spec Release: vs metadata release

No-change rebuilds and reverse-dependency rebuilds change the spec Release: only. See Rebuilding Packages. When dist_git.py update or sync imports a Fedora version, it records that Fedora release in metadata. Local patches merged onto that version retain the Fedora release baseline. When check_upstream_versions.py moves a package to an upstream version ahead of Fedora, it removes metadata release.

When not to update these fields

Most day-to-day package work changes the spec, sources, or patches — not metadata modification_status, modification_reason, or release. Leave those fields alone unless the package’s relationship to Fedora actually changed.

Situation Do not change Why
Independent package gets a CVE patch, backport, or other source edit modification_status, modification_reason Status stays independent. There is no Fedora auto-update to block, so do not switch to modified or add a modification_reason.
No-change rebuild (spec Release: bump only) modification_status, modification_reason, metadata release Rebuilds are ephemeral. Status stays unchanged; metadata release continues to record a Fedora baseline when one exists.
Fedora-imported package gets a local patch or backport metadata release Mark the package modified with a reason, but keep metadata release as the last Fedora baseline. Only bump the spec Release: if needed.
Package is already modified and you add another local change modification_status Leave status as modified. Update modification_reason only if the existing reason no longer describes why auto-updates must stay blocked.
Upstream Fedora update lands and you want auto-updates again (do not leave stale fields) Mark clean (clears modification_reason). Do not hand-edit release; let dist_git.py update refresh version/release from Fedora.

Examples

Independent package + CVE fix: Edit the spec and add the patch. Keep:

{
  "modification_status": "independent",
  "version": "1.3.0"
}

Do not add modification_reason, and do not change status to modified.

Fedora package + CVE backport: After the patch, mark modified so auto-updates stay blocked:

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Backport CVE-2024-12345"

Do not edit metadata release as part of that change.

Rebuild only: Use dist_git.py rebuild (or bump the spec Release:). Leave metadata modification_status and release unchanged.

3.19 - Debugging Build Failures

Overview

When ./ci/build_rpms.sh fails to build a package, you can use the --shell-after flag to drop into an interactive debugging environment. This preserves the build state and allows you to investigate failures, modify files, and re-run the build without starting over.

Quick Start

# Run build and drop into shell after completion (even on failure)
./ci/build_rpms.sh --shell-after <package>

After the build completes (success or failure), you’ll be dropped into a mock shell inside the build environment.

Understanding the Build Environment

The build happens in two layers:

  1. Outer container: The rpm-build-pipeline podman container (Red Hat build environment)
  2. Mock shell: An isolated RPM build environment inside the container (similar to a chroot)

When you use --shell-after, you’re placed directly in the mock shell.

Key directories in the outer container

  • /results/build.log - Full build output with all commands that were run
  • /sources/ - Package source files (spec, patches, etc.)
  • /config/ - Mock configuration
  • /repo/ - Your local rpms repository (read-only mount)

Key directories in the mock shell

  • /builddir/build/BUILD/<package>-*/ - Unpacked source tree (modify and re-run build/test commands here)
  • /builddir/build/originals/<package>.spec - The spec file (modify to test spec changes)
  • /builddir/build/SOURCES/ - Source tarballs and patches
  • /builddir/build/BUILDROOT/ - Install root (where %install places files)

Debugging Workflow

1. Connect to the outer container

To get a second shell without killing your existing mock shell session, connect to the outer container:

# In another terminal, find the running container
podman ps | grep rpm-build-pipeline

# Get a shell in the outer container
podman exec -it <container-id> bash

2. Examine the build failure

From the outer container, check the build log to understand what failed:

# View the end of the build log
tail -100 /results/build.log

# Search for specific errors
grep -i error /results/build.log

3. Enter the mock shell

To investigate or modify the build environment, you can get an additional shell in the mock chroot, particularly for agents which cannot use the initial interactive --shell-after from above.

# From the outer container, chroot into the mock build environment
chroot /var/lib/mock/local-x86_64/root /bin/bash

4. Iterate on the fix

The recommended approach for debugging is to extract and re-run specific commands directly:

For quick iteration while developing a fix, extract the exact build or test command from the build log and run it directly:

# From the outer container, find what command the failing phase runs
grep -A 30 'Executing(%check)' /results/build.log

# Look for lines starting with "+ " that show the actual commands
# Example output:
#   + cd /builddir/build/BUILD/dnf5-5.4.0.0-build/dnf5-5.4.0.0
#   + /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure ...

# Run that command in the mock chroot
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && <actual-command>"

For build failures during %build: Look for Executing(%build) in the log, find commands like make or ninja, then re-run them:

# Example: extract the build command
grep -A 30 'Executing(%build)' /results/build.log | grep -E '^\+ (make|ninja|cmake)'

# Re-run in the build directory
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/<build-subdir> && make -j14"

For test failures during %check: Look for Executing(%check) and extract the test command:

# Example: extract the test command
grep -A 30 'Executing(%check)' /results/build.log | grep -E '^\+ (ctest|make check|pytest)'

# Re-run tests
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && ctest --output-on-failure"

This approach is fast because it skips rpmbuild overhead and goes straight to the failing command. It works with the preserved BUILD directory from --shell-after and is ideal when you’re modifying source files and want quick feedback.

Modifying source files: Edit files directly in /builddir/build/BUILD/<package>-*/ and re-run the command to test your changes quickly.

Modifying the spec file: You can also test spec file changes by editing /builddir/build/originals/<package>.spec in the mock chroot:

# From the host, edit the spec file in the mock chroot
podman exec -it -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  vim /builddir/build/originals/<package>.spec

# Run rpmbuild to see the change
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  env HOME=/builddir LANG=C.UTF-8 su mockbuild -c \
  'rpmbuild -bb --target x86_64 --nodeps /builddir/build/originals/<package>.spec'

Full validation: Apply fixes and rebuild with build_rpms.sh

Manual rpmbuild in the preserved environment may not work reliably for all packages due to complex build dependencies and environment requirements.

Once you’ve identified a fix using the fast iteration approach:

  1. Exit the debug environment and apply your fix to the actual source files or spec file in rpms/<package>/

  2. Test the complete build using the build script:

    ./ci/build_rpms.sh <package>

This ensures your fix works through the entire build process in a clean environment.

3.20 - Package Modification Tracking

Overview

The RPMs repository tracks whether packages have been locally modified from their Fedora upstream source. This tracking prevents automatic updates from overwriting local changes like backported patches or custom modifications.

Modification Status

Each package metadata file (metadata/<package>.json) has a modification_status of clean, modified, or independent. That field (and related modification_reason / release configuration) is documented in Package Metadata Fields. This page covers the workflows for checking status, marking packages, viewing diffs, and configuring update hooks.

An optional track_upstream string field controls whether a package is checked by check_upstream_versions.py for new upstream releases (via release-monitoring.org). Set it to "latest" to track the latest version, or to a version prefix like "1.26" to constrain updates to that series. Its presence enables tracking; omit the field to disable it. The check subcommand only checks packages with track_upstream set when no explicit package arguments are given. The list subcommand shows all packages regardless of this field.

Checking Package Status

View a package’s modification status:

jq .modification_status metadata/<package>.json

View reason for modification (if modified):

jq .modification_reason metadata/<package>.json

List all modified packages:

for f in metadata/*.json; do
  status=$(jq -r .modification_status "$f" 2>/dev/null)
  if [ "$status" = "modified" ]; then
    pkg=$(basename "$f" .json)
    reason=$(jq -r .modification_reason "$f" 2>/dev/null)
    echo "$pkg: $reason"
  fi
done

Viewing Package Differences

To see what changes exist in a modified package compared to upstream Fedora:

# Show full diff for a package
./ci/dist_git.py diff bash

# Show summary statistics
./ci/dist_git.py diff bash --stat

# Show only which files changed
./ci/dist_git.py diff bash --name-only

# Show raw diff (includes Release: bumps and whitespace)
./ci/dist_git.py diff bash --raw

# Diff all modified packages
./ci/dist_git.py diff --all

What’s shown:

  • By default, the diff ignores Release: number changes (no-change rebuilds)
  • Trailing whitespace and blank line changes are ignored
  • Use --raw to see absolutely everything, including Release: bumps

Package types:

  • Modified packages: Shows the differences
  • Clean packages: Shows nothing (useful for verification)
  • Independent packages: Skips with message “no upstream to diff against”

Marking Packages

Mark as Modified

Use this when you make local changes to a package (backports, custom patches, etc.):

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Brief explanation of why"

Examples:

# After backporting a patch
./ci/dist_git.py mark-modified gcc --modified \
  --reason "Backport CVE-2024-12345 fix from upstream"

# After custom spec change
./ci/dist_git.py mark-modified systemd --modified \
  --reason "Add custom service unit for Hummingbird"

The reason field is required and should be concise but descriptive. It helps future maintainers understand why the package can’t be auto-updated.

Mark as Clean

Use this to re-enable automatic updates after confirming your changes are no longer needed (e.g., the fix landed in Fedora):

./ci/dist_git.py mark-modified <package> --clean

This removes the modified status and allows the package to receive automatic updates from Fedora again.

Configure Upstream Tracking

Use set-upstream to configure upstream tracking settings for a package. Each flag independently sets or clears one metadata field. At least one flag is required; omitted flags leave their fields untouched.

./ci/dist_git.py set-upstream <package> [flags]
Flag Sets field Clears with
--track-version latest track_upstream: "latest" --no-track-version
--track-version VER track_upstream: "VER" --no-track-version
--project-id ID release_monitoring_project_id (int) --no-project-id
--project-id NAME release_monitoring_project_id (str) --no-project-id

Each set/clear pair is mutually exclusive (can’t pass --track-version and --no-track-version together).

Examples:

# Enable upstream version tracking (any version)
./ci/dist_git.py set-upstream bash --track-version latest

# Disable upstream version tracking
./ci/dist_git.py set-upstream bash --no-track-version

# Set upstream name with version constraint for versioned packages
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Set release-monitoring.org project ID (integer)
./ci/dist_git.py set-upstream python3.11 --track-version 3.11 --project-id 13254

# Remove project ID (reverts to RPM name lookup)
./ci/dist_git.py set-upstream python3.11 --no-project-id

# Combine multiple flags in one call
./ci/dist_git.py set-upstream golang1.26 \
  --track-version 1.26 --project-id 13254

Metadata fields:

Field Description Example
track_upstream "latest" or version prefix to constrain updates "latest", "1.26"
release_monitoring_project_id Anitya project ID (int) or upstream name (str) 13254, "golang"
version_suffix_strip Suffix to strip from Anitya-reported versions "-RELEASE"
upstream_version_transform Named transform from Anitya version to RPM scheme "openjdk_to_rpm"
source_availability_check Source probe to run before selecting a version {"type": "http", "urls": [...]}

These fields affect two systems:

  • dist_git.py update: When track_upstream is a version prefix, skips upstream versions that don’t match. For example, track_upstream: "1.26" allows 1.26, 1.26.0, 1.26.3 but rejects 1.27.0.
  • check_upstream_versions.py: When release_monitoring_project_id is an integer, queries the v2 API directly by Anitya project ID. When it is a string, queries release-monitoring.org using that name instead of the RPM package name (e.g., looks up golang instead of golang1.26). When absent, uses the RPM package name. When track_upstream is a version prefix, filters the reported upstream versions to only those matching the prefix. Only packages with track_upstream set are checked by check_upstream_versions.py check when no explicit package arguments are given.

Find release-monitoring.org project IDs by searching on https://release-monitoring.org.

The project ID can be combined with a version prefix to filter versions returned by the project ID lookup:

{
  "release_monitoring_project_id": 13254,
  "track_upstream": "3.11"
}

Some upstream projects tag releases with a suffix that is not part of the RPM version (e.g., swift-6.3.3-RELEASE). After Anitya strips the version prefix, the reported version still contains the suffix (6.3.3-RELEASE), which is incompatible with RPM’s Version: field (hyphens are not allowed). Use version_suffix_strip to remove it before comparison and update:

{
  "release_monitoring_project_id": 21267,
  "version_suffix_strip": "-RELEASE"
}

Some upstream projects publish source tarballs independently of tag creation, and Anitya may report a version before the tarball is available. Use source_availability_check to HEAD-probe one or more required source URLs before selecting a version. Versions whose source returns 404 are skipped in favour of the next available version. The skip is reported as a warning in the scheduled-job summary and Slack notification, but does not fail the update job.

The general HTTP probe accepts a non-empty urls list. Every URL must be available. URL templates may use ${VERSION}, ${UPSTREAM_REPO}, and ${TRACK_UPSTREAM}:

{
  "source_availability_check": {
    "type": "http",
    "urls": [
      "${UPSTREAM_REPO}/releases/download/v${VERSION}/libfabric-${VERSION}.tar.bz2"
    ]
  }
}

Legacy named checkers remain available for source layouts that need package-specific logic:

  • openjdk_osci — probes https://openjdk-sources.osci.io/openjdk{feature}/openjdk-{version}.tar.xz
  • tomcat_apache — probes the Tomcat source archive on downloads.apache.org, and its detached signature for Tomcat 10 before allowing an update
{
  "release_monitoring_project_id": 369281,
  "track_upstream": "21",
  "upstream_version_transform": "openjdk_to_rpm",
  "source_availability_check": "openjdk_osci"
}

version_from_ref (fixed-ref, no-tagged-release packages)

Most gorget source-pipeline.yaml fetch steps are version-templated (ref: "v${VERSION}"), so dist_git.py update can invoke gorget with metadata/<package>.json’s own version directly. A few packages instead pin a fixed git-snapshot commit with no tagged upstream release at all (gcc, glibc, libyuv, php-patchwork-jsqueeze, vim) – their version field doesn’t change when the pinned commit does, so it can’t drive gorget’s --version argument.

version_from_ref tells dist_git.py update how to derive gorget’s --version string from a newly-pinned commit, instead of requiring someone to compute it by hand every time (the previous process, documented as a manual step in each such package’s source-pipeline.yaml comments).

Currently one type is supported:

  • commit-date (used by gcc): "<metadata.json's version>-<commit's own YYYYMMDD date>", where the date comes from git log -1 --format=%cd --date=format:%Y%m%d <ref> run against the newly-pinned commit.
{
  "version_from_ref": {"type": "commit-date"}
}

Currently gcc-only. dist_git.py update only fully automates a fixed-ref pin refresh for gcc’s simple case (a full 40-char commit SHA lives directly in a spec %global). The other four packages pin via a git describe-style string (e.g. glibc’s %{glibcsrcdir} macro, glibc-2.43-47-gbc95068f5f) whose trailing hash is only a 10-character abbreviation – resolving that back to a full SHA needs a local clone containing the commit object, real extra network work with its own failure modes (ambiguous hashes, upstream unavailability), not yet implemented (see HUM-4621’s “Remaining work”). For those four, dist_git.py update detects when the pipeline’s pinned ref: no longer matches the merged spec and refuses to auto-commit – it’s routed to a draft, no-test-labeled MR for manual resolution instead, the same way a real git merge conflict is, rather than silently committing a sources file gorget was never actually run against.

Per-Package Update Hooks

When check_upstream_versions.py check --update updates a package, by default it sets Version: to the new upstream version and Release: to 0.1%{?dist} (unless %autorelease is used), adds a changelog entry, and downloads new sources from the URLs declared in the spec. Some packages need custom logic (e.g. generating stripped tarballs or patching macro-based version lines). A per-package hooks file lets you override or extend these default phases without changing check_upstream_versions.py itself.

Hooks file location

metadata/<package>.update-hooks.yaml

For example, metadata/nodejs25.update-hooks.yaml.

Hook phases

The YAML file supports three optional keys. Each value is a shell command string executed with bash -eo pipefail -c in the package directory as the working directory.

Phase Behaviour
update_spec Replaces the default update that sets Version: to the new upstream version and Release: to 0.1%{?dist}. A changelog entry is still added automatically.
download_sources Replaces the default URL-based source download. Must print one filename per line to stdout for files to upload to the lookaside cache. Redirect any other output to stderr (>&2).
post_update Additive — runs after spec + sources are ready. No default equivalent.

Omitting a phase means the default logic runs for that phase. Packages without a hooks file behave identically to before.

Unknown phase keys in the YAML cause a ValueError (fail-fast).

Environment variables

Every hook receives these environment variables:

Variable Example
UPDATE_PACKAGE nodejs25
UPDATE_OLD_VERSION 25.6.1
UPDATE_NEW_VERSION 25.8.2
UPDATE_SPEC_FILE /home/rpms/rpms/nodejs25/nodejs25.spec
UPDATE_PACKAGE_DIR /home/rpms/rpms/nodejs25
UPDATE_SOURCES_FILE /home/rpms/rpms/nodejs25/sources
UPDATE_ROOT_DIR /home/rpms

Example

See metadata/nodejs25.update-hooks.yaml for a working example that uses all three hook phases.

How Auto-Updates Work

The ./ci/dist_git.py update command (used by automation) checks modification status before updating packages:

  • clean packages: Updated automatically when new Fedora versions are available
  • modified packages: Automatically merged with upstream changes (conflicts create draft MRs)
  • independent packages: Update blocked (not sourced from Fedora)
  • version-constrained packages: Skipped if upstream version doesn’t match track_upstream prefix

To force-update a modified package (discarding local changes):

./ci/dist_git.py sync <package>

The sync command bypasses the modification check and force-updates to the latest upstream version. After syncing, the package is automatically marked clean.

Resolving Merge Conflicts

When modified packages are updated from Fedora, dist_git.py update attempts to automatically merge local changes with the new upstream version using git’s three-way merge. When conflicts occur, the update still succeeds but creates a commit with conflict markers, and the automation files a draft merge request labeled with CONFLICT: for manual resolution.

Understanding Conflict Markers

Git uses this conflict marker structure:

<<<<<<< HEAD
Fedora's version (new upstream)
=======
Hummingbird's local modifications
>>>>>>> hummingbird-local

ALL THREE markers must be removed for a clean resolution.

Update branch/MR structure

MRs are created on branches following the pattern:

chore/dist-git-update-PACKAGENAME

These branches are automatically created by the dist_git_update GitLab schedule. If they have conflicts, they result in draft MRs with:

  • Title prefix: CONFLICT: chore(rpms): Update ...
  • Description listing the conflicting files
  • no-test label to skip CI tests (saves resources since conflicts need manual resolution)

Resolution Process

  1. Check out the conflict branch:

    git fetch origin
git checkout origin/chore/dist-git-update-PACKAGENAME

  2. Examine the conflict:

    # Find all files with conflict markers
git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/

# View the specific conflict
git show HEAD:rpms/PACKAGENAME/PACKAGENAME.spec | grep -B5 -A10 "^<<<<<<< HEAD"

  3. Understand the local changes:

    # Review commit history to understand why changes were made
git log --oneline -- rpms/PACKAGENAME/
git log -p -- rpms/PACKAGENAME/  # With diffs

# Check the modification reason
jq -r .modification_reason metadata/PACKAGENAME.json

    Understand the context for correct resolution:

    • What was the original purpose of the local change? Is it transient or permanent?
    • Is it a workaround for a bug, a security patch, or a configuration difference?
    • Does it affect other packages (e.g., nss builds nspr as a subpackage)?
    • Check spec file comments (e.g., NOTE: comments) for packaging details

    Decide which version to accept:

    • Accept HEAD (Fedora) for: release number lags, fixed workarounds that Fedora improved or addressed differently
    • Keep hummingbird-local for: security patches not in Fedora, FIPS requirements, critical fixes, and other permanent modifications
    • Merge both for: test skip lists, independent changes that don’t conflict logically
    • When in doubt: Accept Fedora’s version for packaging metadata (Release:, subpackage versions), keep Hummingbird’s version for functional changes (patches, dependencies, build options)

  4. Resolve the conflict: Edit the file to choose the appropriate version (HEAD, hummingbird-local, or merge both). Verify no markers remain:

    git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/

  5. Validate the resolution: Check that local modifications are preserved:

    # Check the diff against upstream (works on working tree, staging not required)
./ci/dist_git.py diff PACKAGENAME

# Compare with previous modification commits to verify
git log -p -- rpms/PACKAGENAME/

    The diff should show only the intended local modifications (ignoring Release: bumps). This confirms the merge preserved your changes correctly. Note: dist_git.py diff compares the filesystem working tree against upstream, so it works before or after staging.

  6. Amend the commit: Record the original conflicted commit SHA, then amend:

    # Record the original conflicted commit SHA
ORIGINAL_SHA=$(git rev-parse HEAD)

# Stage the resolved files and amend the commit
git add rpms/PACKAGENAME/
git commit --amend -m "$(git log -1 --format=%B | head -n -1)

Conflicted-Update: $ORIGINAL_SHA"

    This preserves the original commit message while adding a Conflicted-Update: trailer that records which commit contained the conflict markers. This helps track the resolution history and can be useful for auditing or debugging later.

  7. Push the resolution:

    git push origin HEAD:chore/dist-git-update-PACKAGENAME --force-with-lease --push-option merge_request.unlabel=no-test

    This removes the no-test label from the MR, which triggers CI tests to run and verifies the resolution works correctly. Some developers might have origin as read-only remote, and a different writable remote (e.g. originw).

Common Conflicts

nss: Subpackage Release Numbers

The nss package builds nspr as a subpackage with its own release number offset.

BACKGROUND:

  • nss builds both nss and nspr RPMs from the same source
  • nspr_release uses an offset (%[%baserelease+n]) to avoid NVR clashes
  • The spec file NOTE explains: reset to 1 when nspr_version changes, increment when only nss changes
  • Fedora manages these offsets in their ecosystem to prevent conflicts

CONFLICT EXAMPLE:

<<<<<<< HEAD
%global nspr_release %[%baserelease+3]
=======
%global nspr_release %[%baserelease+1]
>>>>>>> hummingbird-local

REASONING: When updating to a new upstream nss version from Fedora:

  • Accept Fedora’s nspr_release offset (HEAD) - they manage NVR clashes
  • Our local offset was specific to Hummingbird rebuilds
  • New upstream version should reset to Fedora’s packaging values
  • Don’t try to “calculate” what it should be - trust Fedora’s packaging

RESOLUTION: Accept HEAD (Fedora’s value)

Special Case: Rebuild-Only Changes

Release-only changes (no-change rebuilds) are automatically ignored by the modification detection logic. This means:

  • Bumping Release: 3%{?dist} → Release: 3.1%{?dist} does not mark the package as modified
  • The package can still receive automatic Fedora updates
  • The Release bump will be preserved if the update doesn’t change the upstream Release field

You do not need to mark packages as modified for rebuild-only changes, unless you want to explicitly prevent automatic updates for other reasons.

CI Validation

The CI pipeline validates modification status consistency using make check, which runs:

./ci/validate_package_modifications.py --all

This validation ensures:

  1. All packages have a modification_status field
  2. The value is one of: clean, modified, independent
  3. If track_upstream is present, it must be a string ("latest" or a version prefix)
  4. Modified packages have a modification_reason
  5. Independent packages do not have source/branch/sha fields (Hummingbird-independent only)
  6. Git commit history matches the declared modification status

The validation runs on every merge request and push to main, failing the build if metadata is inconsistent.

For local development, run the full validation:

./ci/validate_package_modifications.py --all

Or validate specific packages:

./ci/validate_package_modifications.py bash glibc gcc

Validation Modes

The validation script has two modes:

Fast mode (default): Checks git commit history patterns

./ci/validate_package_modifications.py --all

This validates that all commits since the last Sync follow standard patterns (have Upstream: trailers). Runs in less than a minute for all packages.

Thorough mode: Clones upstream repos and compares filesystems

./ci/validate_package_modifications.py --all --thorough

This performs full filesystem comparisons with upstream Fedora repositories. Slow and unreliable (hundreds of upstream dist-git clones) but authoritative - validates actual state regardless of git commit history.

For CI and daily development, fast mode is sufficient. Use thorough mode when:

  • Debugging discrepancies between metadata and actual state
  • Auditing the entire repository for hidden modifications
  • Investigating why a package can’t be updated

Workflow Examples

Backporting a Patch

  1. Add patch file and modify spec (see Rebuilding Packages)

  2. Commit the changes

  3. Mark as modified:

    ./ci/dist_git.py mark-modified dnf5 --modified \
  --reason "Backport reproducible build fix (upstream PR#2522)"

  4. Package is now protected from automatic Fedora updates

Re-enabling Auto-Updates

When your backported fix lands in Fedora:

  1. Verify the fix is in the latest Fedora version:

    ./ci/dist_git.py update dnf5  # This will fail with "modified" error

  2. Mark the package clean:

    ./ci/dist_git.py mark-modified dnf5 --clean

  3. Update from Fedora:

    ./ci/dist_git.py update dnf5  # Now succeeds

Importing New Packages

When importing packages, modification status is set automatically:

# Fedora package → marked as "clean"
./ci/dist_git.py import fedora/neofetch

# Hummingbird-independent package → marked as "independent"
./ci/dist_git.py import hummingbird/custom-tool

No manual marking needed for imports.

Troubleshooting

CI Fails: “Missing modification_status field”

This means a metadata file is missing the required field. This means that the package was not imported properly.

CI Fails: “Marked as clean but package has modifications”

The package has local changes but metadata says it’s clean. To fix:

  1. Check what changed:

    git log -p -- rpms/<package>/

  2. Mark as modified with the appropriate reason:

    ./ci/dist_git.py mark-modified <package> --modified --reason "..."

CI Fails: “Marked as modified but package is actually clean”

The package has no local changes but is marked modified. To fix:

  1. Verify it’s actually clean:

    ./ci/dist_git.py update <package>  # Check if upstream matches

  2. If confirmed clean, remove the modified status:

    ./ci/dist_git.py mark-modified <package> --clean

Update Blocked: “Cannot auto-update <package>”

This is expected for modified packages. Options:

  1. Wait for fix to land in Fedora, then mark clean and update

  2. Force-sync to discard local changes:

    ./ci/dist_git.py sync <package>

  3. Keep blocked if the local changes are still needed

3.21 - Private RPM Repositories

AI Agent Note: When asked to route a package to a private repository, add private_product: <product> to the package’s entry in ci/package-overrides.yaml, then run make generate to regenerate resources. The product name must match an existing RPA entry in ci/konflux_rpa_config.yml.

Overview

Hummingbird LTS products (e.g., LTS OpenJDK, LTS DotNet) require subscription-gated RPMs that must not be publicly accessible. Each LTS product gets its own private Pulp repository with independent credentials, enabling per-product subscription enforcement.

The private RPM pipeline uses the same build infrastructure as public RPMs but routes packages to separate Pulp domains via dedicated Konflux Applications and ReleasePlanAdmissions (RPAs).

How it works

  1. Each private product has a dedicated Konflux Application (e.g., private-<product>-rpms-main) separate from the public rpms-main application.
  2. A ReleasePlan connects the private Application to its RPA, enabling auto-releases when builds complete.
  3. Packages are assigned to a private product via private_product in ci/package-overrides.yaml.
  4. A dedicated RPA per product routes those packages to private Pulp repositories.
  5. Packages with private_product are automatically excluded from the public RPA.

Architecture

package-overrides.yaml          konflux_rpa_config.yml
  <package>:                      rpas:
    private_product: <product>      - name: ...public...
                                    - name: ...private-<product>...
                                      private_product: <product>
        |                                    |
        v                                    v
  Component resource              ReleasePlanAdmission
  application: private-           targets: private-<product>-rpms-main
    <product>-rpms-main           pulp_signed_domain: private-hummingbird-<product>
                                           ^
                                           |
                                  ReleasePlan
                                  application: private-<product>-rpms-main
                                  releasePlanAdmission: hummingbird-rpms-private-<product>

Assigning a Package to a Private Product

Add private_product to the package’s entry in ci/package-overrides.yaml:

<package>:
  private_product: <product>
  timeout_hours: 8

Then regenerate all resources:

make generate

This will:

  • Set the package’s Konflux Component to use application: private-<product>-rpms-main
  • Include the package in the private product’s RPA component list
  • Automatically exclude the package from the public RPA

Verifying the assignment

After running make generate, verify the changes:

# Check the component's application assignment
grep -A5 '<package>-main' konflux-templates/rendered.yml | grep application

# Check the package appears in the private RPA
grep '<package>-main' releng/hummingbird-rpms-private-<product>.yaml

# Check the package is excluded from the public RPA
grep '<package>-main' releng/hummingbird-rpms-tech-preview-staging.yaml
# (should return no results)

Onboarding a New Private Product

Automated onboarding

Use the add-private-product subcommand to automate Pulp setup, RPA configuration, package assignment, and resource regeneration in a single step:

./ci/dist_git.py add-private-product <product> \
  --packages pkg1,pkg2 \
  --infra-repo <path-to-infrastructure-repo> \
  --pulp-config <path-to-cli.toml>

This handles Pulp infrastructure (step 2), RPA config (step 5), package assignment (step 6), infrastructure repo templates (step 4 file creation), and resource regeneration. You still need to manually complete: credentials secret deployment (step 3), merging the infrastructure repo MR created by this tool (step 4), and content guard setup (step 7).

Use --dry-run to preview changes without modifying anything.

Manual onboarding

Adding a new LTS product (beyond existing ones) requires these steps: service account, Pulp infrastructure, credentials secret, Konflux Application, RPA configuration, resource regeneration, and content guard setup.

1. Create a Pulp service account (optional)

If you want isolated credentials for private repo operations (recommended), create a dedicated service account before setting up the Pulp infrastructure. See Pulp Access for instructions on creating a service account, configuring the CLI, and storing credentials in the vault. Otherwise, you can reuse the existing public Pulp credentials.

2. Create Pulp infrastructure

Use the existing Pulp setup script to create the private domains, RPM repositories, and file repositories (for SBOMs and attestations). See ci/pulp-setup/README.md for the required pulp-cli-console plugin install before running these commands:

# Unsigned (staging) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned

# Unsigned (staging) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned \
  --type file

# Signed (production) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>

# Signed (production) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product> \
  --type file

If using a dedicated service account, add --config <path-to-cli.toml> to each command.

This creates per-architecture RPM repositories (source, x86_64, s390x, ppc64le, aarch64) with distributions, and file repositories (metadata, rpm-catalog) for SBOM and attestation storage.

3. Deploy the Pulp credentials secret

If using a dedicated service account (step 1), add a Kubernetes Secret template to the infrastructure repo so the credentials are deployed to the Konflux cluster:

  1. Create kubernetes/setup-konflux/47-pulp-private-hummingbird-config-file-secret.yml.j2:

    ---
kind: Secret
apiVersion: v1
metadata:
  name: pulp-private-hummingbird-config-file-secret
stringData:
  cli.toml: «{ lookup_secret("HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE[deployed]:cli.toml") | forceescape }»
type: Opaque

  2. Add an entry to secrets.yml for the vault key:

    HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE:
  backend: hv
  meta:
    active: true
    created_at: '<timestamp>'
    deployed: true

    The secret name (pulp-private-hummingbird-config-file-secret) must match the pulp_secret_name in the private RPA configuration (step 5).

  3. Add an ExternalSecret to the konflux-release-data repo (rhtap-release-data) so the credentials are available to the release pipeline. Create the ExternalSecret in the cluster-specific directory (e.g., tenants-config/cluster/kflux-prd-rh03/managed/rhtap-releng-tenant/):

    ---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: hummingbird-private-pulp-bot-production-secret
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: "0"
spec:
  dataFrom:
    - extract:
        key: releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: releng-vault
  target:
    creationPolicy: Owner
    deletionPolicy: Delete
    name: hummingbird-pulp-credentials-private-production-secret

    Add it to the cluster’s kustomization.yaml and regenerate auto-generated files with tenants-config/build-manifests.sh.

  4. Add the secret to the releng vault at https://vault.devshift.net under the path releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production with the following keys:

    Key Value
    cli.toml Full contents of the Pulp CLI config file
    expires-on na
    owner Your Kerberos username

4. Create the Konflux Application

The Application resource is managed in the infrastructure repo, not this repo. Create it there following the same pattern as rpms-main:

  1. Create a new directory kubernetes/private-<product>-rpms-main/ in the infrastructure repo.

  2. Add 00-application.yml.j2 with the standard Application template:

    ---
apiVersion: appstudio.redhat.com/v1alpha1
kind: Application
metadata:
  name: {{ env["PROJECT_NAME"] }}
spec:
  appModelRepository: {url: ""}
  displayName: {{ env["PROJECT_NAME"] }}
  gitOpsRepository: {url: ""}

  3. Add the new project to the CI matrix in infrastructure/.gitlab-ci.yml. Find the PROJECT_NAME list under the konflux-rh03/hummingbird-tenant context and add the new application name:

    - PROJECT_NAME:
    - rpms-main
    - private-<product>-rpms-main   # add this line
  PROJECT_CONTEXT:
    - konflux-rh03/hummingbird-tenant

  4. Add 01-release-plans.yml.j2 with a ReleasePlan that references the private RPA. Copy from kubernetes/rpms-main/01-release-plans.yml.j2 and update the releasePlanAdmission label to match the private RPA name:

    metadata:
  name: hummingbird-rpm-release-private-<product>
  labels:
    release.appstudio.openshift.io/auto-release: "true"
    release.appstudio.openshift.io/standing-attribution: "true"
    release.appstudio.openshift.io/releasePlanAdmission: hummingbird-rpms-private-<product>
spec:
  application: {{ env["PROJECT_NAME"] }}
  target: rhtap-releng-tenant
  # ... copy remaining spec from rpms-main/01-release-plans.yml.j2

    The ReleasePlan connects the private Application to its ReleasePlanAdmission. Without it, builds in the private Application will not trigger releases.

  5. Copy 10-integration-test-scenarios-testing-farm.yml.j2 from kubernetes/rpms-main/. The template is fully parameterized with {{ env["PROJECT_NAME"] }} so no edits are needed. This may be needed to prevent the Konflux PR group integration test from blocking MRs that touch private product packages.

Merge this MR in the infrastructure repo first — the CI pipeline will deploy the Application, ReleasePlan, and IntegrationTestScenarios to the cluster. Components and RPAs in this repo reference it by name, so the Application must exist before they are applied.

5. Add RPA configuration

Add a new entry to the rpas: list in ci/konflux_rpa_config.yml:

rpas:
  # Existing public RPA (private packages are automatically excluded)
  - name: hummingbird-rpms-tech-preview-staging
    # ... existing config ...

  # New private product RPA
  - name: hummingbird-rpms-private-<product>
    application_prefix: private-<product>-rpms
    release_org: registry.stage.redhat.io/hummingbird-tech-preview
    single_component_mode: true
    service_account_name: hummingbird-rpm-release-staging
    pulp_unsigned_domain: private-hummingbird-<product>-unsigned
    pulp_signed_domain: private-hummingbird-<product>
    pulp_secret_name: hummingbird-pulp-credentials-private-production-secret
    pipeline_revision: <release-pipeline-branch>
    pipeline_url: https://github.com/scoheb/release-service-catalog.git
    private_product: <product>
    component_filter:
      path_prefix: rpms/

Key fields:

Field Purpose
name Kubernetes resource name for the RPA
application_prefix Combined with branch to form the Konflux Application name
private_product Matches the private_product value in package-overrides.yaml
pulp_unsigned_domain Pulp domain for unsigned RPMs (staging)
pulp_signed_domain Pulp domain for signed RPMs (production)
pulp_secret_name Kubernetes secret containing Pulp publishing credentials

6. Assign packages and regenerate

Add private_product: <product> to each package in ci/package-overrides.yaml, then regenerate:

make generate

This produces:

  • Updated konflux-templates/rendered.yml with per-component application assignments
  • A new RPA file at releng/hummingbird-rpms-private-<product>.yaml
  • Updated public RPA excluding private packages

7. Set up Pulp content guard

To restrict access to the private Pulp repositories, a content guard must be configured so only customers with the correct subscription can access the content:

  1. Obtain a SKU for the private product’s subscription offering.
  2. Create a feature for that SKU in the Feature service.
  3. Contact the Pulp team to:
    • Map the organization ID to the feature
    • Set up the content guard on the private Pulp domain(s)

This step is required before customers can access the private repositories. Without it, the repositories are created but have no access control.

Pulp credentials

If using a dedicated service account (step 1), see Pulp Access for vault storage instructions.

If using the shared service account, private RPAs reuse the same Pulp publishing credentials as the public RPA (hummingbird-pulp-credentials-production-secret). Per-product access control for customers is handled downstream by Red Hat’s subscription entitlement system, not at the Pulp publishing layer.

Public/Private Routing

Packages with private_product set in ci/package-overrides.yaml are automatically excluded from the public RPA and included only in their product’s private RPA. Removing the private_product field returns the package to public-only publishing.

File Purpose
ci/package-overrides.yaml Per-package private_product assignment
ci/konflux_rpa_config.yml RPA definitions (public + private)
ci/generate_resources.py Generates Components, RPAs, and PipelineRuns
ci/pulp-setup/create-pulp-resources.sh Creates Pulp domains and repositories
konflux-templates/macros/releng/release-plan-admission.yml.j2 RPA template
konflux-templates/macros/component.yml.j2 Component template (per-component application)
konflux-templates/macros/image-repository.yml.j2 ImageRepository template (per-component application)

3.22 - Reporting CVE Data Issues

Overview

Sometimes CVE data published on cve.org or NIST NVD is incorrect – wrong affected package, wrong severity score, wrong affected version range, or missing fix information. When this happens, the incorrect data propagates into vulnerability scanners and can cause false positives (or false negatives) for Hummingbird container images.

This document describes how to report these issues to get the data corrected at the source.

When to Report

Report a CVE data issue when you find any of the following:

  • Wrong affected package – the CVE Record lists a package that is not actually affected
  • Wrong version range – the affected or fixed version boundaries are incorrect
  • Wrong severity – the CVSS score or severity rating does not match the actual impact
  • Missing fix information – a fix exists upstream but the CVE Record does not reflect it
  • Wrong CPE match – the NVD CPE configuration matches products that are not affected

How to Report

Reporting to the CVE Program (cve.org)

The CVE Program accepts corrections through the CVE Numbering Authority (CNA) that owns the record.

  1. Find the CVE Record at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX.
  2. Identify the CNA listed in the record (shown in the “Assigning CNA” field).
  3. Contact the CNA directly to request a correction. Most CNAs accept reports via:
    • Their security reporting email (often listed in their CNA page)
    • GitHub issues if the CNA is an open source project
  4. If the CNA is unresponsive, use the CVE Program Request form to dispute the record.

Reporting to NIST NVD

NIST NVD enriches CVE Records with CVSS scores and CPE match data. To request corrections:

  1. Navigate to the NVD entry at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX.
  2. Click the “Contact” link or email nvd@nist.gov with:
    • The CVE ID
    • The specific field that is incorrect
    • Evidence of the correct data (links to upstream commits, release notes, etc.)

Reporting to Red Hat Product Security

If the incorrect CVE data is causing issues in Red Hat’s vulnerability tracking:

  1. File a Jira ticket in the HUM project with the CVE ID and a description of the data issue.
  2. Red Hat Product Security can update the VEX feed to reflect the correct disposition for Hummingbird, even before the upstream CVE data is corrected.

Tracking the Correction

After reporting, track the status of the correction:

  • CVE Record updates are published at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX
  • NVD updates appear at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX (NVD may take days to weeks to process corrections)
  • Update the HUM Jira ticket with the correction status so the team is aware

Requesting information on the fix status of an existing CVE

File a support request at https://access.redhat.com/support.

3.23 - Lookaside Cache Access

How to access the S3-based lookaside cache for RPM source tarballs

Overview

Source tarballs for RPM packages are stored in an S3-based lookaside cache rather than in git. To upload files to the cache (e.g., when updating a package to a new upstream version), you need AWS credentials with the appropriate permissions.

The ci/upload-to-lookaside-cache.sh script handles uploads directly, and ci/check_upstream_versions.py --update calls it automatically when downloading new source archives.

Prerequisites

You must be a poweruser in the arr-cloud-aws-core group (it-cloud-aws-727920394381-poweruser). Request access to this group if you do not already have it.

Obtaining AWS Credentials

There are two ways to authenticate.

Option A: Browser-based login

Run aws login, which opens a browser for authentication:

aws login

Option B: Kerberos-based login via container

Use the CKI tools container to obtain credentials via Kerberos:

$ podman run --rm -it \
    -e KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u) \
    -v $HOME/.aws:/cki/.aws:U,Z \
    quay.io/cki/cki-tools:latest

bash-5.2# kinit <userid>@IPA.REDHAT.COM

bash-5.2# AWS_IDP_URL=https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws \
    cki_aws_login --duration 43200 \
    --account 727920394381 --role poweruser

Replace <userid> with your Kerberos user ID. This writes credentials to $HOME/.aws on the host (bind-mounted into the container).

Note: This overwrites the AWS default profile credentials. If you use the default profile for other purposes, back up $HOME/.aws/credentials before running this command.

Uploading Files

Once credentials are configured in $HOME/.aws, you can upload files to the lookaside cache.

Manual upload

./ci/upload-to-lookaside-cache.sh -f <file> -p <package>

Example:

./ci/upload-to-lookaside-cache.sh -f rpms/tar/tar-1.35.tar.xz -p tar

Automated upload via version checker

check_upstream_versions.py --update downloads new source archives and uploads them to the lookaside cache automatically:

./ci/check_upstream_versions.py check --update <package>

The check subcommand only processes packages with "track_upstream": true in their metadata when no explicit package arguments are given. To see all packages and their upstream status, use check_upstream_versions.py list. See Package Modification Tracking for how to enable tracking.

See Also

3.24 - Upstream Diff Analysis

Overview

The upstream diff analysis tool classifies locally modified RPM packages by how their changes relate to upstream Fedora. This helps the team systematically decide which modifications should be proposed upstream, which are Hummingbird-specific, and which need more complex handling.

The primary interface is the /upstream-diff Claude Code skill, which orchestrates the full workflow. The skill delegates all deterministic operations (diffing, metadata lookup, caching) to ci/upstream_diff.py and applies its own judgment only for the classification step.

Classification Categories

Each modified package is classified into one of five categories:

Category Meaning Action
upstreamable Changes can be submitted to Fedora as-is File upstream PR
hummingbird-specific Changes are intentional and specific to Hummingbird Keep locally, no upstream action needed
complex Changes need investigation or partial upstreaming Requires human decision
mixed Some changes are upstreamable, others are Hummingbird-specific Split and handle separately
no-diff Package is marked modified but has no actual diff Consider unmarking as modified

Workflow

Invoke the skill in Claude Code with /upstream-diff. When called with no arguments, it offers three modes:

Analyze

Prepares the next batch of unanalyzed or stale packages, reviews each diff and any upstream PR activity, then classifies each package into a category with reasoning and a recommended action. Results are cached locally so subsequent runs skip already-analyzed packages.

You can also target specific packages by name (e.g., /upstream-diff analyze bash glibc).

View Results

Displays the cached analysis as a summary table. Supports filtering by category and showing details for a single package. Use this to get a quick overview of where things stand across all modified packages.

JIRA

Creates or updates a JIRA issue for a specific package to track the upstream proposal. The skill generates a pre-filled description and analysis comment from the cached data, previews it for confirmation, and then files or updates the issue. The JIRA issue key is recorded in the cache so it appears in the view output.

Script Reference

The skill uses ci/upstream_diff.py under the hood. The script can also be called directly for automation, scripting, or when working outside Claude Code.

Note that prepare and save are two halves of a classification pipeline: prepare gathers raw data (diffs, metadata, upstream PRs, category definitions), but its output requires human or LLM judgment to select a category and write the reasoning fields before calling save. The skill’s Analyze mode bridges this gap automatically.

Prepare

Gather metadata, diffs, upstream PR status, and the classification schema — this is the data the skill’s Analyze mode feeds into its classification step:

# Specific packages
./ci/upstream_diff.py prepare bash glibc

# Next batch of unanalyzed packages
./ci/upstream_diff.py prepare --batch 10

Save

Store a classification result. The skill’s Analyze mode calls this after classifying each package:

./ci/upstream_diff.py save <package> \
  --category <category> \
  --changes-summary "One-line summary of changes" \
  --reasoning "Why this category was chosen" \
  --recommendation "Recommended next action" \
  --upstream-prs <pr_id1> <pr_id2> ...

# No related upstream PRs
./ci/upstream_diff.py save <package> \
  --category hummingbird-specific \
  --changes-summary "Custom Hummingbird macros for FIPS" \
  --reasoning "FIPS build flags are specific to Hummingbird" \
  --upstream-prs ""

The --hummingbird-macros flag can be added when changes use Hummingbird-specific RPM macros.

Results are cached in .cache/upstream-diff-analysis.json.

View

Display cached results — this is what the skill’s View Results mode calls:

# Summary table
./ci/upstream_diff.py view

# Single package detail
./ci/upstream_diff.py view <package>

# Filter by category
./ci/upstream_diff.py view --category upstreamable

# Include unanalyzed modified packages
./ci/upstream_diff.py view --all

# Markdown-formatted links (for JIRA or rendered contexts)
./ci/upstream_diff.py view --markdown

# Raw JSON
./ci/upstream_diff.py view --json

Check Upstream PRs

Check whether local changes have already been submitted or merged in the upstream Fedora dist-git repository:

./ci/upstream_diff.py check-prs <package1> <package2> ...

Shows open and recently merged (last 90 days) pull requests from the upstream Fedora dist-git repo.

After initial analysis, update which upstream PRs are related to a package’s local changes:

# Set related PRs
./ci/upstream_diff.py save --set-upstream-prs <package> <pr_id1> <pr_id2> ...

# Clear related PRs
./ci/upstream_diff.py save --set-upstream-prs <package>

JIRA Templates

Generate pre-filled JIRA content from cached analysis — this is what the skill’s JIRA mode uses:

./ci/upstream_diff.py jira-template <package> [--epic HUM-1613]

Record a JIRA issue key after creating an issue:

./ci/upstream_diff.py save --set-jira <package> HUM-XXXX

3.25 - GPG Source Verification

How upstream signing keys are stored, and how to add, rotate, or revoke them

Overview

Some packages verify the authenticity of their upstream source tarball with a detached OpenPGP signature where upstream signs each release with a private key and publishes a .asc/.sig alongside the tarball. During source fetching, the gorget source-pipeline tool checks that signature against the project’s public key before the bytes are ever used in a build.

The trusted public keys live centrally in metadata/gpg-keys/<project>.gpg, one keyring file per upstream project. Centralized storage means:

  • the full set of signers we trust is auditable in one directory, and
  • rotating or revoking a key is a single, reviewable commit.

The directory is handed to gorget as --gpg-keys-dir metadata/gpg-keys (see ci/check_upstream_versions.py), so a pipeline’s keyring: field is just a filename within it.

How the pieces fit together

Three things wire up verification for a package:

  1. The keyring — metadata/gpg-keys/<project>.gpg, the trusted public key(s).

  2. The pipeline step — a verify: entry in metadata/<package>.source-pipeline.yaml:

    fetch:
  - type: url
    url: "https://curl.se/download/curl-${VERSION}.tar.xz"
  - type: url
    url: "https://curl.se/download/curl-${VERSION}.tar.xz.asc"

verify:
  - type: gpg-signature
    target: "curl-${VERSION}.tar.xz"       # the artifact to verify
    signature: "curl-${VERSION}.tar.xz.asc" # its detached signature
    keyring: "curl.gpg"                      # filename in metadata/gpg-keys/

    gorget imports keyring into a fresh, throwaway GPG homedir per check, then runs the equivalent of gpg --verify <signature> <target>. A bad or missing signature fails the fetch.

  3. CI validation — test/test_gpg_keys.py (run by make check) confirms every file in metadata/gpg-keys/ is a parseable public key, and that every keyring: referenced by a pipeline actually exists.

Adding a key for a new package

Prerequisite: gpg (from the gnupg2 package). It ships in the CI image, run the commands below inside a container, e.g. podman run --rm -it -v "$PWD:$PWD:z" -w "$PWD" quay.io/hummingbird-ci/gitlab-ci:latest bash.

  1. Obtain the upstream public key. Prefer a key you can already trust: many packages already ship the maintainer’s key next to their spec (e.g. rpms/curl/mykey.asc, rpms/bash/chet-gpgkey.asc). Otherwise download it from the project’s official key page.

  2. Store it as a keyring in metadata/gpg-keys/. The keyring may be ASCII-armored or binary; this repo standardizes on binary .gpg. Convert an armored key with --dearmor:

    gpg --dearmor < rpms/<package>/<upstream-key>.asc > metadata/gpg-keys/<project>.gpg

    Name the file after the upstream project, not the RPM (so multiple versioned packages, e.g. python3.11/python3.12, can share one keyring).

  3. Verify the key is what you expect. Print its fingerprints and confirm they match the fingerprints published on the upstream’s official channel:

    gpg --show-keys --with-fingerprint metadata/gpg-keys/<project>.gpg

  4. Wire up the pipeline. Add (or extend) metadata/<package>.source-pipeline.yaml with the fetch steps for the tarball + signature and the verify: [{type: gpg-signature, ...}] step shown above.

  5. Prove the whole chain end to end before committing import the keyring into a throwaway homedir and verify a real release signature against it:

    V=<version>
curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz"
curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz.asc"
export GNUPGHOME=$(mktemp -d)
gpg --import metadata/gpg-keys/<project>.gpg
gpg --verify "<tarball>-$V.tar.xz.asc" "<tarball>-$V.tar.xz"   # expect "Good signature"

  6. Run the checks: make check (validates the keyring and the pipeline reference).

Rotating or replacing a key

Upstream may roll to a new signing key (expiry, policy, new maintainer). Because the trusted set is just files in one directory, rotation is a single commit:

  1. Obtain the new public key from the upstream’s official channel and confirm its fingerprint out of band.

  2. Replace the contents of metadata/gpg-keys/<project>.gpg (re-run the --dearmor step). If upstream signs a transition period with both keys, you may keep both by importing them into the same keyring:

    gpg --dearmor < old-key.asc  > metadata/gpg-keys/<project>.gpg
gpg --dearmor < new-key.asc >> metadata/gpg-keys/<project>.gpg

  3. Re-run the end-to-end verification (step 5 above) against the latest release, then make check.

  4. Commit with a message recording why the key changed and how you confirmed the new fingerprint, this file is the audit trail for what we trust.

Revoking / removing a key

  • If a project drops GPG verification, delete both metadata/gpg-keys/<project>.gpg and the verify: step from its pipeline in the same commit. (CI fails a pipeline that references a missing keyring, and if the key is orphaned the reverse is easy to spot.)
  • If a key is compromised, remove it immediately and replace it with the upstream’s revocation / replacement key.

Troubleshooting

  • gpg: no valid OpenPGP data found — the file isn’t a real keyring (empty, truncated, or you saved an HTML error page). Re-download and re---dearmor. test_gpg_keys.py catches this in CI.
  • gpg: Can't check signature: No public key — the signature was made with a key that isn’t in the keyring. Upstream likely rotated keys, follow Rotating or replacing a key above.
  • BAD signature — the tarball does not match the signature. Do not paper over this. It means a corrupted download or, in the worst case, tampering. Re-fetch from the canonical source and if it persists, escalate rather than accepting the artifact.

3.26 - Rebasing the Buildroot to a New Fedora Release

Step-by-step runbook for moving mock.cfg, the core toolchain, and the rest of the package set to a new Fedora release

AI Agent Note: This document is the procedural runbook for a Fedora buildroot rebase (e.g. F43 -> F44). It was written after executing the F44 rebase (see HUM-2018 and its child stories HUM-3369, HUM-3370, HUM-4536, HUM-4537 for the detailed history) and should be followed, and updated, the next time this operation is performed (e.g. F44 -> F45). Read this whole document before starting; the ordering between steps matters and several steps have hard dependencies on the one before it.

Overview

Periodically, Fedora ships a new stable release and Hummingbird needs to move its buildroot (mock/mock.cfg) and the packages that track a specific Fedora branch (rather than rawhide) onto it. This is a multi-day, multi-phase effort touching the core toolchain, CI/testing configuration, and potentially every package in the repository (since changing the compiler/linker toolchain can affect any build). The F44 rebase took about 2 calendar weeks.

Pre-flight checks

Before starting, confirm:

  • The target Fedora release is actually stable (not just branched) and its package repos are reachable from the build environment.

  • Which packages currently track the outgoing release branch explicitly (i.e. branch is a versioned Fedora branch like f43, not rawhide) via:

    # List every package whose branch is pinned to a specific Fedora release (not rawhide)
./ci/dist_git.py list | xargs -I{} sh -c 'echo -n "{}: "; python3 -c "import json; print(json.load(open(\"metadata/{}.json\"))[\"branch\"])"' 2>/dev/null | grep -E ': f[0-9]+$'

    For each, decide up front whether it should move to the new branch, move to rawhide, or stay pinned (see “Packages that intentionally stay behind” below) — don’t leave this undecided until a package fails to build.

  • Whether any of those packages carry FIPS-validation-critical modifications. Packages whose modification_reason references FIPS certification should generally not be rebased to new branch content, even though they still need to be rebuilt against the new toolchain. Moving them would risk invalidating the FIPS-validated source. (See HUM-4989 for a related, more general proposal to record why a package is pinned in its metadata rather than relying on the modification reason text.)

  • Whether any package whose version is the “which version is default” signal for an adjacent language/tooling ecosystem (e.g. python-rpm-macros) needs the same branch-pinning treatment as the interpreter itself — especially if that ecosystem is mid-transition to a new major version upstream at the same time as this rebase. See “Known gotchas: python3dist / rawhide-tracked toolchain-adjacent packages” below for why this matters and what already went wrong once.

  • Whether to pause the routine automated dist-git sync bot for the duration of the Step 4 mass rebuild. See “Known gotchas: automation racing a manual mass rebuild” below for the tradeoff.

Step 1 — Buildroot config + core toolchain (separate MRs)

Do not bundle the buildroot config change with the toolchain rebuilds in one MR — split them so the buildroot config lands and validates quickly, independent of the (much slower) toolchain rebuilds.

  1. mock/mock.cfg: update dist, releasever, bootstrap_image, description, and the [fedora]/[fedora-updates] repo baseurls and names to the new release. Merge this on its own or bundled only with fedora-repos (see next point) and binutils if it’s a clean (unmodified) package — do not bundle glibc/gcc/llvm here.
  2. Testing/CI fallback config (rpms/ci/repos/fedora-43.repo → fedora-44.repo, and ci/default-tests/tests-rpm.yml’s repo copy + GPG key import): these reference the buildroot’s Fedora version directly, and there’s exactly one of each — bump them in the same MR as mock.cfg (or immediately alongside it), not later. During the F44 rebase this was deferred and wasn’t caught until php failed its install test days later in Step 4, against a fallback repo that still pointed at the outgoing release. (This is distinct from the per-package test fixture gotcha below, which recurs throughout every step — see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references.”)
  3. fedora-repos: rebase to the new release. If Fedora’s spec uses a conditional ELN-style macro for Release:, evaluate whether to keep it or hardcode a plain Release: N%{?dist} — this is a judgment call each time depending on what’s simplest to maintain; document whichever choice is made in the package’s modification_reason.
  4. binutils (if it carries local CVE backport patches): before assuming the patches still apply, check whether the CVEs they address have already been fixed upstream in the new Fedora version. If so:
    • Pull Fedora’s current spec as the new baseline (don’t try to merge patch-by-patch on top of the old spec).
    • Re-add only the local patches that are not already fixed upstream.
    • For a quick validation build, it’s sufficient to confirm the patch set applies and compilation begins — you don’t need to wait for a full build to complete just to validate the patch set is sane. Kill the build once compilation starts.
  5. glibc: this is the highest-risk package in this phase because of a circular bootstrap dependency (see “Known gotcha: glibc/gcc bootstrap circularity” below). Expect this to require a temporary workaround, tracked as a modified package, to be reverted in Step 2.
  6. gcc: reapply any local modification (e.g. disabling non-essential language frontends) against the new spec. Expect this to be one of the longest-running builds in the whole rebase (multiple hours per architecture) — plan CI capacity accordingly.
  7. llvm: rebase and rebuild; this can run in parallel with gcc and typically finishes faster.

Step 2 — Bootstrap cleanup

Once the new compiler (from Step 1) is published to Pulp:

  1. Rebuild annobin against the new compiler. This must happen before the next step, or reverting the glibc bootstrap workaround will immediately re-trigger the same circular dependency.
  2. Revert the glibc bootstrap workaround from Step 1, now that the new compiler and annobin are both available. Clear the modified status/reason that was recording the workaround.
  3. Rebuild libtool against the new compiler. This is easy to miss (it was missed during the F44 rebase and had to be done out-of-band mid-mass-rebuild) but several packages (apr, audit, authselect, automake, avahi, bind, catatonit, cryptsetup, find, and likely others) will fail to build against the new toolchain without it. Do this here, not in Step 4.

Step 3 — Toolchain-adjacent, release-pinned packages

Rebase the remaining packages that were explicitly tracking the outgoing Fedora branch and don’t carry a permanent reason to stay behind (see “Packages that intentionally stay behind”). For each:

  • Decide whether it should move to the new versioned branch (e.g. f44) or to rawhide — moving to rawhide can simplify build-dependency resolution for clean (unmodified) packages, but ties future updates to rawhide’s churn.
  • For packages with local modifications, check whether the modification is still needed (e.g. a security backport that may have landed upstream by now) before reapplying it blindly.
  • Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before opening this package’s MR (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — fix it in this same MR if present.

Step 4 — Mass rebuild of everything else

Rebuild every remaining package against the new toolchain so that linking, annobin annotations, and compiler hardening are consistent across the whole package set.

  1. Build the exclusion list first: everything already rebuilt in Steps 1–3, plus anything merged in the hours immediately before starting this step (check recent merge history). Use dist_git.py rebuild --all’s --exclude option (HUM-5183):

    ./ci/dist_git.py rebuild --all --exclude pkg1,pkg2,... --reason "<toolchain> toolchain rebuild"

    During the F44 rebase this flag didn’t exist yet, so the exclusion list had to be hand-rolled with ./ci/dist_git.py list | grep -v -E '^(pkg1|pkg2|...)$' piped into rebuild — no longer necessary now that --exclude has landed.

  2. Watch for packages whose Release: field doesn’t fit the simple .N bump pattern (e.g. packages using build-time macros for Release:) — these will fail to commit cleanly and need to be handled in a separate pass.

  3. Submit in batches, not one giant MR wave — use ./ci/rebuild_multi_mr.sh --dry-run to preview, then ./ci/rebuild_multi_mr.sh --max-updates=N to submit and auto-merge-on-green in controlled increments. Plan for several days; a run of ~460 packages took about a week of elapsed time across 5 batches during the F44 rebase, gated by CI/Testing Farm throughput more than build time itself.

  4. Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before each package’s rebuild MR merges (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — this is what let php’s failure slip through undetected for days during the F44 rebase, so don’t rely on remembering it only once at the end.

  5. Expect a small number of packages to need bespoke fixes that a routine rebuild can’t resolve automatically — license metadata regressions surfaced by newer scanners, compiler/linker regressions in vendored dependencies (e.g. a bindgen version incompatible with a newer clang), confirmed upstream compiler-triggered bugs in JIT/JVM-style software, or stale version-gates in local patches that assumed an older toolchain. Budget time for a handful of these near the end of the mass rebuild.

Known gotchas

glibc/gcc bootstrap circularity

A new Fedora release’s glibc spec may use a feature of the new gcc (e.g. a new compiler flag), while the new gcc may in turn require symbols only present in the new glibc. If Hummingbird’s own Pulp repos still serve the old toolchain at a higher priority than the new Fedora repos during the transition window, this becomes a real chicken-and-egg problem, not just a theoretical one.

The pragmatic fix used for F44 was to drop the new-toolchain-specific flag from glibc temporarily (if it’s a defensive/hardening-only flag whose absence doesn’t break correctness), rebuild glibc and gcc against each other with it disabled, then re-enable it once both are published and annobin/libtool have also been rebuilt against the new compiler (Step 2 above).

Package-specific test fixtures with hardcoded Fedora-version references

Beyond the single global fallback config bumped in Step 1, some packages carry their own test/rpms/<pkg>.yml test suite that independently hardcodes a Fedora repo/GPG-key reference for setting up a test chroot (during F44 this was glib2, hummingbird-release, ca-certificates, crypto-policies, and openssl). Don’t wait for a one-time catch-all audit to find these — fix each one in the same MR as that package’s own rebuild/update, whenever it happens to land during the rebase (Step 1, 3, or 4, whichever touches that specific package — see the reminders in each of those steps above). Relying on an exhaustive upfront sweep of every test/rpms/*.yml is fragile and is exactly what let php’s failure slip through undetected for days during the F44 rebase — its own MR was fine, but a different package’s stale GPG key/repo reference broke shared test infrastructure php depended on. When you open the rebuild/update MR for any package, grep test/rpms/<pkg>.yml for the outgoing release string first.

See HUM-5184 for tracking a structural fix (single source of truth or a CI lint check) so this class of drift is caught automatically instead of relying on either of the above being remembered.

Automation racing a manual mass rebuild

The routine automated dist-git sync bot keeps running on its normal schedule regardless of an in-progress manual mass rebuild. If both touch the same packages in the same window, you’ll get a wave of merge-conflicted bot MRs that need manual cleanup — during F44 this was roughly 70 MRs at once. Decide before starting Step 4 (see “Pre-flight checks” above) whether to pause routine sync automation for the duration of the mass rebuild, or accept the cleanup cost.

External CI capacity constraints

Large rebuild waves can coincide with unrelated capacity crunches on shared infrastructure (e.g. Testing Farm). These are outside Hummingbird’s control but can stall a batch for a day; check external status pages when a batch is unexpectedly slow (for reference, the F44 rebase hit https://status.testing-farm.io/issues/2026-07-14-big-queue-in-red-hat-ranch/).

python3dist / rawhide-tracked toolchain-adjacent packages

If any packages central to a specific language ecosystem’s “what version is default” (e.g. python-rpm-macros) are tracking rawhide rather than a pinned branch, a routine automated sync during this window can silently pull in an experimental version bump (e.g. Fedora bootstrapping the next language version) that corrupts auto-generated package metadata across dozens of downstream packages without any build failing immediately. See HUM-4988 and HUM-4989 for the full incident report and proposed guardrails from the F44 rebase. Decide during “Pre-flight checks” above whether any “default version” declaration packages need the same branch-pinning treatment as the interpreter itself before you start.

Packages that intentionally stay behind

Not every package needs to move to the new branch. Document (in modification_reason and/or the epic tracking the rebase) any package that’s staying pinned to the old branch, and why — otherwise this looks like unfinished work to the next person. Reasons seen so far:

  • FIPS validation: the package carries a modification tied to FIPS certification, and moving to newer upstream content risks invalidating that certification. The package is still rebuilt against the new toolchain — it just doesn’t pick up newer Fedora source content.
  • Package no longer exists upstream: Fedora sometimes drops a package entirely in a new release (e.g. an older LTS language runtime superseded by a newer one). If Hummingbird still needs to carry it, it has no newer branch to rebase to and simply stays on its last available branch, rebuilt against the new toolchain.

Reference: commands used during the F44 rebase

These are shown verbatim from the F44 rebase for concreteness. When reusing them for the next rebase (e.g. F45), remember to substitute the actual target release everywhere F44 appears below — including the exclusion list of already-rebuilt packages, which will differ each time.

# Preview a rebuild plan across all packages
./ci/rebuild_multi_mr.sh --dry-run

# Submit rebuild MRs in controlled batches, auto-merge on green
./ci/rebuild_multi_mr.sh --max-updates=50

# Rebuild everything except a known set of already-handled packages, using the
# --exclude flag (HUM-5183, landed after the F44 rebase — see Step 4 above)
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm,annobin,binutils,fedora-repos,python3.13,python-gitlab --reason "F44 toolchain rebuild"

During the F44 rebase itself, --exclude didn’t exist yet, so the equivalent exclusion list had to be hand-rolled:

./ci/dist_git.py rebuild $(./ci/dist_git.py list | grep -v -E '^(glibc|gcc|llvm|annobin|binutils|fedora-repos|python3.13|python-gitlab)$') --reason "F44 toolchain rebuild"
  • HUM-2018 — F44 buildroot rebase epic (full history)
  • HUM-3369 / HUM-3370 / HUM-4536 / HUM-4537 — Steps 1-4, each with a detailed what/why/fix writeup
  • HUM-4988 / HUM-4989 — python-rpm-macros/python3dist regression incident and guardrails
  • HUM-5182 — bump_release() double-suffix bug
  • HUM-5183 — dist_git.py rebuild --exclude option (implemented)
  • HUM-5184 — generalize the testing-infra Fedora-version drift fix

4 - Lot's of background information

Background and explanation documentation for understanding Project Hummingbird systems and concepts.

4.1 - RPMs repository

Background documentation and contributing guide for Project Hummingbird RPM packages.

Thank you for your interest in contributing! This repository contains RPM packaging, minimal tests, and CI plumbing used by Hummingbird to build, lint, and validate RPMs in containers and Testing Farm.

This guide is adapted from the Hummingbird containers contribution guide and aligned with our workflows and tooling. See the original for broader context: Hummingbird Containers CONTRIBUTING.

Code of Conduct

Be respectful and constructive. By participating, you agree to uphold a professional and inclusive environment.

Repository layout

  • rpms/<package>/ – RPM dist-gits (spec, sources), mostly auto-imported
  • ci/ – helper scripts and default tests
    • build_rpms.sh – build RPMs using Mock in the Konflux-compatible container
    • run_tests_rpm.sh – run rpmlint and install/rebuild tests in a container
    • default-tests/tests-rpm.yml – default tests included for all packages
    • run_tests_rpm.fmf – tmt/Testing Farm entry point
  • konflux-templates/ – Konflux/PAC resources
  • mock/ – mock configuration
  • test/rpms/<package>.yml – package-specific tests

Prerequisites

  • Fedora or RHEL-like environment
  • Podman
  • rpmlint
  • tmt (optional, for local TF-style runs)
  • jq, python3, python3-yaml (used by run_tests_rpm.sh)

Build locally

Build a package’s SRPM and RPMs using the Konflux-aligned environment:

./ci/build_rpms.sh <package_name>
# Results are written to: /tmp/konflux-build-<package_name>-*/results/

Building in Lima VM

When building inside a Lima VM (macOS users), use the --build-dir flag to specify a directory on the VM’s native filesystem. This avoids permission issues with mock’s bootstrap process on macOS mounts:

limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build <package_name>'

The build directory must be on the VM’s native filesystem (not a macOS mount) to ensure Linux file ownership, permissions, and symlinks work correctly with /var/lib/mock.

Interactive repository debugging

To investigate package/dependency/installability issues, you can run an interactive shell in the same environment used by the package builds:

./ci/build_rpms.sh --shell-before setup

This will show you the mock command that would be used to build the setup package, and drop you into a shell in the same environment before executing the build, with dnf available. You can then run the command manually and inspect the environment, repositories, and package metadata.

If you want it to actually build the local package first, so that you can include it in your investigation, use --shell-after instead:

./ci/build_rpms.sh --shell-after setup

Test locally (containerized)

Run the default and package-specific tests against one or more built RPMs:

# Basic usage (binary rpm)
./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm <package_name>

# Include source RPM tests
./ci/run_tests_rpm.sh \
  --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \
  --src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \
  <package_name>

# Test all RPMs in the build output directory after build_rpms.sh
./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds/<package_name>/RPMS/*.rpm) \
  --repo-dir builds/<package_name>/RPMS/ \
  --src-rpm builds/<package_name>/SRPMS/*.src.rpm \
  <package_name>

Notes:

  • Tests run inside a Podman container.
  • The test image defaults to quay.io/hummingbird/core-runtime:latest-builder. You can override via TEST_IMAGE:
TEST_IMAGE=quay.io/hummingbird/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm <package_name>
  • Container execution is performed as root with HOME=/root to avoid XDG state permission issues in dnf5.

Test in tmt/Testing Farm locally

You can drive the same FMF test locally with tmt. The FMF test expects an OCI artifact that contains RPMs, referenced via IMAGE_URL (Testing Farm sets this automatically). For local trials you can point to any compatible OCI artifact or skip the ORAS pull logic and directly call the script.

# If using an OCI artifact with RPMs
IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a

# Or run the script directly with built RPMs (bypassing the FMF wrapper)
./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm <package_name>

If you need longer time in Testing Farm, the FMF test includes duration, which you can adjust in ci/run_tests_rpm.fmf.

Dist-git Imports

The ci/dist_git.py tool imports Fedora/CentOS dist-git packages into the rpms/ directory. We expect most packages to not have (permanent) Hummingbird specific changes, so most of them will keep syncing with upstream dist-gits. In most cases that will be Fedora rawhide, but for some packages we may pick a different upstream, e.g. stable Fedora or even CentOS Stream.

The status of all imports is tracked in imports.json. Active Fedora releases are tracked in upstream-releases.json, which is updated from the Bodhi API. Note that rawhide is automatically resolved to the highest numbered Fedora release at runtime and is not stored in the JSON file.

See ./ci/dist_git.py --help for all available options. Some examples:

  • Update upstream-releases.json from Bodhi API (should be done periodically, e.g., when new Fedora versions are released):
./ci/dist_git.py update-releases

This fetches the latest Fedora releases from Bodhi. The rawhide branch automatically resolves to the highest numbered Fedora version (e.g., if f43 and f44 are available, rawhide uses f44).

  • Import a new package. This requires specifying the dist-git URL (with fedora/ being a shortcut for the Fedora dist-git URL) and optionally a branch (default: rawhide):
./ci/dist_git.py import fedora/bash

./ci/dist_git.py import --branch f42 fedora/glibc
./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git
  • Update all or a single package:
./ci/dist_git.py update

./ci/dist_git.py update bash

This only imports changes if these were actually built in Koji, to ensure we only import changes which are meant to be released. You can disable this check with --skip-build-check.

  • Re-sync a package to upstream, discarding any local modifications. We use this after Fedora adopted our change, or it is no longer relevant:
./ci/dist_git.py sync bash

All of these commands automatically commit changes with descriptive commit messages including the upstream SHA. To avoid that, you can use the --dry-run option.

  • Enable upstream version tracking for a package (used by check_upstream_versions.py check). These commands modify the metadata file but do not create a git commit:
./ci/dist_git.py set-upstream bash --track
./ci/dist_git.py set-upstream bash --no-track
  • Check upstream version status. The check subcommand checks tracked packages; list shows all packages in a table:
./ci/check_upstream_versions.py check
./ci/check_upstream_versions.py list
./ci/check_upstream_versions.py list --json

Package-specific overrides

Per-package build configuration can be customized in ci/package-overrides.yaml. If a package is not listed, it uses default settings.

Available options:

Option Description Default
timeout_hours Build timeout in hours 4
build_platforms List of MPLs (instance sizes) for multi-platform builds Pipeline defaults

Example configuration:

# Long-running package with custom timeout
setup:
  timeout_hours: 12

# Package requiring larger build instances
llvm:
  timeout_hours: 12
  build_platforms:
    - "linux-d160-c8xlarge/arm64"
    - "linux-d160-c8xlarge/amd64"

All available MPLs can be found in the Konflux multi-platform builds documentation.

After modifying overrides, regenerate the pipeline files:

make generate

This updates .tekton/rpms-on-push.yaml and .tekton/rpms-on-pull-request.yaml with the new configuration.

Branching and pull requests

  • Create feature branches from the default branch.
  • Keep edits focused and small; separate unrelated changes into separate PRs.
  • Include meaningful commit messages (why + what). Reference related issues if applicable.
  • PRs must pass CI (build + tests). Fix lint/test failures or mark known failures properly.

Commit message conventions

  • First line: short imperative summary (≤ 72 chars)
  • Body (optional): context, rationale, and user/ops impact
  • Reference issues using standard notation (e.g. “Fixes: #123”)

Spec guidelines

  • Don’t use %changelog for Hummingbird specific changes. It’s just a point of conflict with Fedora imports, commit messages are good enough.
  • Hummingbird specific changes increase Release: number in steps of 0.1 (e.g. 2 → 2.1 → 2.2). This avoids colliding with Fedora’s Release number namespace and allows us to sync again to a future -3.

Packaging and testing guidelines

  • Spec files should be reproducible and minimal.
  • Prefer pinned container digests for CI images where feasible. If a tag and digest are both present (name:tag@sha256:<digest>), the digest is authoritative for content selection.
  • Default tests in ci/default-tests/tests-rpm.yml must be fast, deterministic, and safe for all packages.
  • Package-specific tests (rpms/<package>/tests-rpm.yml) can override or extend defaults. Keep them bounded in runtime and dependencies.
  • When possible, capture flaky or environmental issues under known_issues in test YAML with clear matching patterns and descriptions.

Style

  • Shell: bash with set -euo pipefail, readable variable names, and early returns where reasonable.
  • YAML: consistent indentation and quoting; prefer explicitness over magic.
  • Comments: add when necessary to explain non-obvious rationale; avoid restating the code.

Security and supply chain

  • Avoid embedding credentials or secrets; use environment variables or CI secret stores.
  • Prefer pulling images by digest for stability and repeatability in CI.
  • Validate inputs and sanitize any paths used by scripts.

Resolving CVEs

When manually fixing a CVE (e.g., adding a patch or performing a manual update), a Fixes: CVE-YYYY-XXXX must be added to the MR description and/or a commit. This will indicate to the build system that a CVE fix has been submitted for review.

Reporting issues

Open an issue describing the problem, reproduction steps, and environment. Attach logs (build/test) when possible. If the failure is intermittent, call that out explicitly.

Licensing

Ensure files include appropriate licenses and that any third-party content is compatible with the project’s license.

Thank you

Your contributions make the Hummingbird RPMs better for everyone. We appreciate your time and feedback!

4.1.1 - RPM Pipeline

How an RPM spec change flows through the build pipeline to the package repository

Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.

Overview

The RPM pipeline consists of five main stages:

  1. Spec Change - A package spec file is modified in rpms/<package>/
  2. Merge Request & Validation - CI validates the change
  3. Build - Konflux builds RPMs per architecture via mock in hermetic mode
  4. Signing - Built RPMs are cryptographically signed
  5. Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository

After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.

flowchart TD
    A["Spec Change<br/>rpms/&lt;package&gt;/"] --> B["Merge Request"]
    B --> C["CI Validation<br/>(check, tree_status)"]
    C --> D["Merge to main"]
    D --> E["Konflux RPM Build<br/>Tekton PipelineRun per package<br/>mock hermetic build per arch"]
    E --> F["RPM Signing<br/>Kerberos-based"]
    F --> G["Publish to Pulp<br/>packages.redhat.com<br/>/public-hummingbird/&lt;arch&gt;/"]
    G --> H["Container lockfile updates<br/>(see Image Pipeline)"]

Stage 1: Spec Change

An RPM change begins as a commit in rpms/<package>/. The change typically includes a modified .spec file (updated version, new patch, or Release bump) and any associated source files or patches.

How changes originate

Method Description Modification Status
Automated Fedora sync ci/dist_git.py update merges upstream Fedora changes No independent packages
Upstream version update ci/check_upstream_versions.py --update bumps to new upstream release No independent packages
Manual patch backport Developer adds a CVE patch and references it in the .spec Marked modified afterward
No-change rebuild ci/dist_git.py rebuild <package> bumps the Release field Status unchanged
Reverse dependency rebuild ci/dist_git.py rebuild-rev-deps <package> rebuilds all dependents Status unchanged

See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.

Package metadata

Each package has a metadata file at metadata/<package>.json that tracks its relationship to upstream:

{
  "source": "https://src.fedoraproject.org/rpms/curl.git",
  "branch": "rawhide",
  "sha": "abc123...",
  "version": "8.12.1",
  "release": "1",
  "modification_status": "clean",
  "upstream_repo": "https://github.com/curl/curl"
}

Key fields:

Field Purpose
modification_status clean (auto-updates enabled), modified (auto-updates blocked), or independent (no upstream)
modification_reason Explanation of local modifications (when modified)
release Fedora release at last import/update (not the spec Release:), without dist tag; present only while shipping Fedora’s release
upstream_repo Canonical upstream git repository URL
track_upstream Version prefix constraint (e.g., "1.26" for golang1.26)
version_transform Version mapping rule for CVE analysis (e.g., dotnet_sdk_to_runtime)
cve_product CVE vendor/product override (string, or a list for multiple products; e.g. "Oracle Corporation / Oracle Java SE")

See Package Metadata Fields for modification_status and release configuration, and Package Modification Tracking for managing modification status day to day.

Stage 2: Merge Request & Validation

Changes reach main via merge requests. Automated MRs are created using:

  • ci/create_mr.sh - Single MR with configurable options
  • ci/rebuild_multi_mr.sh - One MR per package with auto-merge enabled
  • ci/dist_git_update_multi_mr.sh - Automated Fedora update MRs (scheduled CI job)
  • ci/upstream_update_multi_mr.sh - Automated upstream version update MRs (scheduled CI job)

The scheduled upstream-version update job creates normal MRs for successful package updates. If a package update fails after a new version has been identified, it creates (or reuses) a draft diagnostic MR instead. The diagnostic MR includes the original error but no partial package changes, and must be resolved manually before a package update can be proposed.

CI validation

GitLab CI runs validation jobs on every merge request:

  • check - Linting, type checking, and spec validation
  • check_nevr_conflicts - Predicts the NEVR(A) each changed package’s spec would build (using rpmspec with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale main, e.g. two rebuild MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch rpms/**/*, and within that, only checks packages whose rpms/<package>/ directory itself changed – matching Konflux’s own build trigger – not packages whose only change is under metadata/, since a metadata-only change (e.g. ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally with make check-nevr-conflicts ARGS='<package>', or against real built RPMs with make check-nevr-conflicts ARGS='--rpms-dir builds/<package>' after ci/build_rpms.sh. See ci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages routed to a private_product are noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode some OK results can be for a package name a real build never actually produces (e.g. a base package with no top-level %files section, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount a CONFLICT found elsewhere in the same run.
  • tree_status - Repository consistency checks (metadata integrity, spec file presence)
  • Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)

Auto-approval for chore MRs

Automated update MRs (chore/* branches) follow an accelerated path:

  1. MR is created with auto-merge enabled
  2. Konflux builds the package and posts commit statuses
  3. After a delay, the chore_mr_approval CI job auto-approves the MR
  4. GitLab merges the MR once the pipeline succeeds

Stage 3: Build

After merging to main, Konflux builds RPMs automatically.

Build triggers

Each package has a Tekton PipelineRun definition generated from its directory in rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered .tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers a build when changes to that package’s directory are pushed to main. When a single push touches multiple packages, one PipelineRun is triggered per affected package directory. Merge request pushes also trigger builds for validation.

The setup package acts as a canary: changes to ci/ or mock/ directories also trigger a setup build on pull requests, validating build infrastructure changes before they reach main.

Build process

The build pipeline (build-rpm-package) executes for each target architecture:

  1. Checks out the repository at the merge commit
  2. Calculates build dependencies
  3. Runs mock in hermetic mode (network-isolated with pre-fetched dependencies)
  4. Produces RPM artifacts and build logs
  5. Generates an SBOM (Software Bill of Materials)
  6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry

Build architectures

Packages are built for the architectures specified in their PipelineRun definition. Most packages build for x86_64 and aarch64. Some packages also build for s390x and ppc64le.

Build output

Build artifacts are stored in the Konflux OCI registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main

Konflux resources

Build infrastructure is defined across two repositories:

  • rpms repo (konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy
  • infrastructure repo (kubernetes/rpms-main/) - Application, ReleasePlan, IntegrationTestScenario, ServiceAccount, and Secret resources

See Konflux Resource Deployment for details on how these resources are managed and deployed.

Testing

RPM packages are validated through integration tests that run on Testing Farm infrastructure.

Tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test triggering

Tests are only triggered on merge requests, not on main branch builds. For a package at rpms/<package>/, tests are triggered for all changes below that directory.

Integration Test Scenario

Tests are triggered via an IntegrationTestScenario resource defined in the infrastructure repository:

Configuration: infrastructure/kubernetes/rpms-main/10-integration-test-scenarios-testing-farm.yml.j2

The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:

Scenario parameters:

  • COMPOSE: RHEL-9-Nightly - Test environment OS/compose
  • PIPELINE_MODE: rpm - Configures RPM testing mode (vs. container)
  • ARCH: x86_64,aarch64 - Architectures to test (creates separate test runs per arch)
  • PASS_SNAPSHOT_TO_TF: false - Don’t pass full snapshot JSON (package info comes via IMAGE_NAME/IMAGE_URL instead)
  • IMAGE_TAG: - Version of the tmt-via-testing-farm pipeline bundle

Key characteristics:

  • Single scenario for all packages - Not per-package, runs for every package build
  • Context-based triggering - Runs on pull_request snapshots only
  • Package identification - Konflux automatically provides IMAGE_NAME (e.g., openssl-main) and IMAGE_URL (OCI image with built RPMs) as environment variables

FMF test plan

The root folder of the rpms repository is marked with a .fmf directory to identify it as an FMF metadata tree for tmt. The test plan in ci/run_tests_rpm.fmf sets up the test instance and runs default and package-specific tests via ci/run_tests_rpm.sh.

Test environment

These environment variables are available to the tmt test execution:

Variable Description
IMAGE_NAME Component name (e.g., openssl-main)
IMAGE_URL OCI image URL with built RPMs (e.g., quay.io/.../openssl-main@sha...)
SNAPSHOT Snapshot metadata (if PASS_SNAPSHOT_TO_TF is true)
COMPOSE OS/compose being tested (RHEL-9-Nightly)
ARCH Architectures being tested (x86_64,aarch64)
TMT_VERSION tmt version running the tests

Stage 4: Signing

The Konflux Release Service signs built RPMs before publishing:

  • Method: Kerberos-based signing via konflux-release-signing-prod@IPA.REDHAT.COM
  • Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
  • Signing image: quay.io/konflux-ci/signing:latest (verify pinned digest in ReleasePlanAdmission)

Stage 5: Publishing to Pulp

After signing, RPMs are uploaded to the Hummingbird Pulp repository.

Release pipeline

The push-rpms-to-pulp release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (releng/hummingbird-rpms-tech-preview-staging.yaml):

mapping:
  rpm-repositories:
    - name: x86_64
      repository_id: public-hummingbird-x86_64-rpms
    - name: aarch64
      repository_id: public-hummingbird-aarch64-rpms
    - name: src
      repository_id: public-hummingbird-source-rpms
    # See ReleasePlanAdmission for the full list of published architectures

Pulp repository

Published RPMs are available in the Pulp content index:

Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.

What Happens Next

Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.

4.1.2 - Konflux Resource Deployment

How Konflux resources are defined and deployed across repositories

Konflux resources for RPM packages are split between two repositories:

  • rpms: Per-package resources generated from package directories
  • infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Resource Type Source RPMs MR RPMs Push Infra MR Infra Push
Component rpms manual manual - -
ImageRepository rpms manual manual - -
ReleasePlanAdmission rpms - automatic manual automatic
EnterpriseContractPolicy rpms - automatic manual automatic
Application infrastructure - - manual automatic
ReleasePlan infrastructure - - manual automatic
IntegrationTestScenario infrastructure - - manual automatic
ServiceAccount, Secret infrastructure - - manual automatic

Legend:

  • RPMs MR: Downstream pipeline triggered from rpms MR (deploys from MR commit)
  • RPMs Push: Downstream pipeline triggered from rpms push to main
  • Infra MR: Infrastructure MR pipeline (manual trigger)
  • Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

RPMs Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

  1. Ownership: Components are managed by the rpms repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle.
  2. Dynamic generation: Generated from rpms/*/ directories via ci/generate_konflux_resources.sh, depending on package presence and configuration.
  3. Timing: Must be deployed early during MR review so Konflux can build and test new packages.

ReleasePlanAdmission:

  1. Static configuration: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern.
  2. Main branch only: Deployed only on push to main to maintain deployment consistency.

EnterpriseContractPolicy:

  1. Consistency: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern.

Infrastructure Repo

Application, ReleasePlan, and IntegrationTestScenario are defined in kubernetes/rpms-main/:

  1. Independent iteration: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo.
  2. Testable before merge: If defined in the rpms repo, these would only deploy after merging to main, making iteration difficult.
  3. Static configuration: These resources don’t depend on per-package data.

ServiceAccount and Secret for releases are defined in kubernetes/setup-konflux/:

  1. Security: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo.
  2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo.

The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp.

4.1.3 - Source Pipeline Tool

Source Pipeline Tool

Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.

Problem Statement

RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:

  1. Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.

  2. Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.

  3. Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.

  4. Unverified patches. Patches (.patch files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.

Tool Overview

A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.

                          ┌──────────────────────────────┐
                          │     source-pipeline tool     │
                          │      (container image)       │
                          │                              │
  ┌───────────┐           │  ┌───────┐    ┌───────────┐  │    ┌───────────────┐
  │  package  │           │  │ fetch │───▸│ transform │  │    │   tarballs    │
  │   dir     │──────────▸│  └───────┘    └─────┬─────┘  │───▸│   sources     │
  │ (spec,    │           │                     │        │    │   report      │
  │  patches, │           │  ┌───────────┐      │        │    └───────┬───────┘
  │  config)  │           │  │  verify   │◂─────┘        │            │
  │           │           │  └─────┬─────┘               │            ▼
  └───────────┘           │        │                     │    ┌───────────────┐
                          │  ┌─────▼─────┐               │    │   lookaside   │
  ┌───────────┐           │  │  enforce  │               │    │    cache      │
  │ pipeline  │           │  │  policy   │               │    └───────────────┘
  │   yaml    │──────────▸│  └─────┬─────┘               │
  │           │           │        │                     │
  └───────────┘           │  ┌─────▼─────┐               │
                          │  │   emit    │               │
                          │  └───────────┘               │
                          └──────────────────────────────┘

Design principles

  • Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
  • Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
  • Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
  • Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.

Container interface

podman run --rm \
  -v ./<package-dir>:/package:ro \
  -v ./pipeline.yaml:/pipeline.yaml:ro \
  -v ./gpg-keys:/gpg-keys:ro \
  -v ./output:/output \
  source-pipeline:latest \
  --version <new-version> \
  [--old-version <old-version>] \
  [--verify-patches /patches-from-upstream] \
  [--dry-run]

Inputs (mounted read-only):

  • /package — the package directory (spec file, patches, existing sources file)
  • /pipeline.yaml — the declarative pipeline definition
  • /gpg-keys — centralized GPG keyring directory

Outputs (written to /output):

  • Source tarballs (ready for lookaside upload)
  • sources — updated sources manifest with checksums
  • report.json — verification and policy results (pass/fail per check, patch classifications)

Exit codes:

  • 0 — success, all checks passed
  • 1 — error (download failure, tool error)
  • 2 — policy violation (verification or constraint failure)

--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to /output. Exit codes and report.json behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing.

On any non-zero exit, report.json is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no package update commit is created, the package stays at its current version, and the automation continues to the next package. In the scheduled upstream-version-update job, the wrapper also creates (or reuses) a draft diagnostic MR for a per-package update failure. That MR contains the full error and no partial package changes, so a maintainer can reproduce and resolve the problem safely. Transient failures (exit 1) may self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention.

Pipeline Stages

1. Fetch

Downloads source artifacts directly from upstream.

  • Parses Source: URLs from the spec file (resolving RPM macros like %{version}, %{name}, %{url})
  • Downloads each source from the upstream URL
  • Supports fetching from git repositories at a tag, branch, or commit
  • For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)

2. Transform

Applies per-package source modifications.

  • Vendor archive generation — runs ecosystem-specific tooling (e.g., go_vendor_archive for Go, npm pack for Node.js, cargo vendor for Rust)
  • Vendor dependency pinning — modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix.
  • Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
  • UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
  • Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
  • Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a toolchain: section. The container ships defaults; per-package overrides ensure that build-ui, vendor, and run: steps use the correct toolchain without requiring separate container images per package

Known sharp edge: patch-list duplication

Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill point back here instead of re-telling the story.

Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ... followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain their own copy of “which patches touch this source tree” rather than reading it from the spec’s PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see Rebuilding Packages) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform’s install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation step, and the mismatch didn’t surface until the next automated version bump re-ran the transform from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact.

go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up requiring different versions of the same package — go build -mod=vendor rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s %check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

3. Verify

Validates integrity and authenticity of fetched sources.

  • GPG signature verification — downloads signature files (.asc, .sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation.
  • Checksum verification — compares against published checksums where available
  • Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
  • Re-publication detection — if a previously committed checksum exists in the sources file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an accepted-checksums entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.

4. Enforce policy

Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect) and catches violations in packages that don’t use vendor-pin.

  • Vendor dependency constraints — ensures vendored dependencies meet version requirements (e.g., sanitize-html >= 2.17.5 for a CVE fix)
  • Ecosystem-specific checks — Go module verification, npm audit, cargo audit
  • License compliance — flags vendored dependencies with incompatible licenses
  • Fail-closed: any policy violation exits with code 2

5. Emit

Produces final artifacts.

  • Writes tarballs to the output directory
  • Generates sources file in dist-git format (SHA512 (filename) = hash)
  • Writes report.json with verification results, policy check results, and patch classification

Pipeline Schema

Per-package pipeline definitions are YAML files that declare the full source generation workflow.

Packages without a pipeline definition use a built-in default that fetches sources from the spec’s Source: URLs — no transforms, no verification, no policy.

Schema definition

# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
  # Macro substitutions applied before Source URL resolution
  macros:
    - name: go_patch                     # %global go_patch <new-value>
      value: "${VERSION_PATCH}"          # extracted from VERSION (e.g., 1.25.3 → 3)

    - name: k8s_ver                      # %global k8s_ver <new-value>
      value: "${VERSION}"

  # Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
  reset-release: true

# Source fetching
fetch:
  sources:
    # Fetch Source0 from the URL declared in the spec
    - spec-source: 0

    # Fetch Source1 (e.g., a GPG signature file)
    - spec-source: 1

    # Or fetch from an explicit URL (for sources not in the spec)
    - url: "https://example.com/extra-source-${VERSION}.tar.gz"

    # Or fetch from a git repository at a tag/commit
    - git:
        repo: "https://github.com/example/project"
        ref: "v${VERSION}"              # tag, branch, or commit hash
        include-history: false           # include .git dir in tarball (default: false)

  # Vendor archive generation (optional)
  vendor:
    ecosystem: go                        # go | npm | cargo | composer
    config: go-vendor-tools.toml         # ecosystem-specific config file in the package dir
    source-dir: .                        # directory to vendor from (default: extracted source root)
    submodules:                          # (optional) vendor multiple Go submodules independently
      - server
      - etcdctl
      - etcdutl

# Source transforms (optional, ordered)
transform:
  # Built-in transform types
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
        - "deps/ngtcp2/ngtcp2/crypto/"
      output: "node-v${VERSION}-stripped.tar.gz"

  # Pin vendored dependency versions before vendor archive generation.
  # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
  # bump specific dependencies, then re-resolves the dependency graph.
  # Runs before the vendor stage so the pinned versions are included in
  # the vendor archive. Re-applied on every source generation, so upstream
  # updates cannot silently revert a security fix.
  #
  # Version constraints are always minimum versions — the version field
  # means "at least this version." The tool translates to ecosystem-native
  # operations:
  #   Go:    go get <package>@v<version> (minimum version selection)
  #   npm:   npm install <package>@">= <version>"
  #   Cargo: set dependency requirement to ">= <version>" in Cargo.toml
  - vendor-pin:
      - package: golang.org/x/crypto
        ecosystem: go
        version: "0.31.0"
        reason: "CVE-2024-45337"

      - package: sanitize-html
        ecosystem: npm
        version: "2.17.5"
        reason: "CVE-2024-XXXXX"

      - package: tokio
        ecosystem: cargo
        version: "1.38.1"
        reason: "CVE-2024-YYYYY"

  # Custom command (escape hatch)
  - run: "./packaging/make-tarball.sh ${VERSION}"
    outputs:
      - "package-${VERSION}-stripped.tar.gz"

  # UI asset build
  - build-ui:
      ecosystem: npm                     # npm | yarn
      source-dir: "web/ui"
      output: "${PACKAGE}-${VERSION}-ui.tar.gz"

# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
  node: "20"                             # Node.js version for build-ui, npm vendor, and run: steps
  go: "1.23"                             # Go version for go vendor and run: steps
  rust: "1.80"                           # Rust version for cargo vendor and run: steps
  python: "3.12"                         # Python version for run: steps

# Integrity verification (optional)
verify:
  gpg:
    signature-source: 1                  # spec Source index containing the .asc/.sig
    keyring: "curl.gpg"                  # key name in the GPG keys directory
  checksums:
    url: "https://example.com/SHA256SUMS"
    algorithm: sha256

  # Override for upstream re-publications. Required when upstream re-publishes
  # a release artifact with different content under the same version. The
  # pipeline refuses to accept changed content automatically — a human must
  # add the new checksum here after investigation.
  accepted-checksums:
    - file: "example-1.2.3.tar.gz"
      sha512: "abc123..."
      reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"

# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
  vendor-constraints:
    - package: sanitize-html
      ecosystem: npm
      version: "2.17.5"
      reason: "CVE-2024-XXXXX"

    - package: golang.org/x/crypto
      ecosystem: go
      version: "0.31.0"
      reason: "CVE-2024-45337"

# Patch verification (optional)
patches:
  verify: true                           # diff-detect new/changed patches
  classify: true                         # attempt to classify by origin
  upstream-repo: "https://github.com/curl/curl"
  fail-on-unverified: false              # warn-only by default

  # Patch lifecycle rules (optional)
  # Declares version-scoped applicability for patches. The pipeline tool reads
  # these rules from patch headers (preferred) or from this YAML, and enforces
  # them during updates: patches outside their valid range are flagged or dropped.
  lifecycle:
    - file: "fix-memory-leak.patch"
      applies-to: "< 1.5.0"             # drop this patch at version 1.5.0+
      reason: "Fixed upstream in 1.5.0 (commit abc123)"
      action: drop                       # drop | warn (default: warn)

    - file: "cve-2024-45337.patch"
      applies-to: "< 0.31.0"
      reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
      action: drop

    - file: "distro-branding.patch"
      applies-to: "*"                    # carry forward unconditionally
      reason: "Distribution-specific branding, always required"

# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
  # Extract bundled dependency versions and splice into spec between markers
  - bundled-provides:
      modules-txt: "vendor/modules.txt"  # Go modules.txt path inside extracted source
      start-marker: "# --- bundled-deps.sh ---"
      end-marker: "# --- end bundled-deps.sh ---"

  # Run a custom command (escape hatch for complex metadata extraction)
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"

Variable substitution

The following variables are available in all string values:

Variable Value
${VERSION} New upstream version (e.g., 1.25.3)
${VERSION_MAJOR} Major version component (e.g., 1)
${VERSION_MINOR} Minor version component (e.g., 25)
${VERSION_PATCH} Patch version component (e.g., 3)
${OLD_VERSION} Previous version
${PACKAGE} Package name (directory name)
${SPEC_FILE} Path to the spec file

Version constraint semantics

All version: fields in vendor-pin and policy.vendor-constraints use minimum-version semantics: the value means “at least this version.” The tool translates this into ecosystem-native operations:

  • Go: go get <package>@v<version> (minimum version selection via MVS)
  • npm: npm install <package>@">= <version>"
  • Cargo: sets the dependency requirement to >= <version> in Cargo.toml

Default behavior (no pipeline YAML)

When no pipeline YAML is provided, the tool applies a built-in default:

  1. Parse all Source: URLs from the spec
  2. Download each from the upstream URL
  3. Generate sources file with SHA512 checksums
  4. No transforms, no verification, no policy enforcement

This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.

Patch Verification

When packages are updated from an upstream distribution, patch files (.patch) may arrive without inspection. The pipeline tool can optionally verify and manage patches.

Verification approach

The pipeline tool can optionally verify patches when invoked with --verify-patches:

  1. Diff detection. Compare the set of .patch files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches.

  2. Header parsing. Extract metadata from patch headers:

    • From: — author identity
    • Subject: — description
    • Commit hash references (e.g., From <hash>, cherry picked from commit <hash>)
    • Bug: / CVE: references

  3. Classification. Attempt to classify each patch:

    • Upstream backport — references a commit hash that exists in the upstream repo
    • CVE fix — references a CVE identifier
    • Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
    • Distribution-specific — modifies paths, branding, or distribution-specific integration
    • Unclassified — cannot be automatically categorized

  4. Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via git ls-remote or the upstream API).

  5. Reporting. Output classification and verification results in report.json. Optionally fail on unverified patches (controlled by patches.fail-on-unverified in the pipeline YAML).

Patch lifecycle enforcement

Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.

The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML’s patches.lifecycle section.

Header-based declaration (preferred). Patch authors add structured keywords to the patch header:

From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---

YAML-based declaration (fallback). For patches from upstream distributions that cannot have headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch.

Enforcement behavior:

  1. During an update to version ${VERSION}, the tool evaluates each patch’s applies-to range against the new version.
  2. If a patch is outside its valid range:
    • action: drop — the patch file is deleted from the package directory and the corresponding Patch: declaration and %patch / %autopatch application directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in report.json.
    • action: warn (default) — the patch is flagged in report.json but not removed. The update proceeds.
  3. If a patch has applies-to: *, it is always carried forward.
  4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to applies-to: *, action: warn).

Carry-forward enforcement. The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked applies-to: * is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error.

Example report output

{
  "patches": {
    "new": [
      {
        "file": "fix-memory-leak.patch",
        "classification": "upstream-backport",
        "upstream_commit": "abc123def456",
        "verified": true
      }
    ],
    "removed": ["old-workaround.patch"],
    "unchanged": ["distro-paths.patch"],
    "lifecycle": [
      {
        "file": "cve-2024-45337.patch",
        "applies_to": "< 0.31.0",
        "current_version": "0.31.0",
        "action": "drop",
        "result": "removed — version 0.31.0 is outside applies-to range"
      },
      {
        "file": "cve-2024-99999.patch",
        "applies_to": "< 2.0.0",
        "current_version": "2.0.0",
        "action": "drop",
        "result": "flagged — patch is applied inside a conditional %if block; manual removal required"
      },
      {
        "file": "distro-branding.patch",
        "applies_to": "*",
        "action": "carry-forward",
        "result": "present"
      }
    ]
  }
}

Examples

Trivial package (curl)

No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you want GPG verification:

fetch:
  sources:
    - spec-source: 0
    - spec-source: 1

verify:
  gpg:
    signature-source: 1
    keyring: "curl.gpg"

Transformed package (Node.js)

Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:

spec-update:
  macros:
    - name: nodejs_define_version node
      value: "${VERSION}"
  reset-release: true

toolchain:
  node: "22"

fetch:
  sources:
    - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"

transform:
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
      output: "node-v${VERSION}-stripped.tar.gz"

verify:
  checksums:
    url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
    algorithm: sha256

post:
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"

Simple Go vendor (caddy)

A pattern shared by nats-server, oauth2-proxy, and similar Go projects:

fetch:
  sources:
    - git:
        repo: "https://github.com/caddyserver/caddy"
        ref: "v${VERSION}"

  vendor:
    ecosystem: go

Multi-submodule Go vendor (etcd)

fetch:
  sources:
    - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    submodules:
      - server
      - etcdctl
      - etcdutl

Code generation + Go vendor (opentelemetry-collector-contrib)

Built-in primitives handle fetch and vendor; the code generation step requires run::

fetch:
  sources:
    - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    source-dir: _build

transform:
  - run: |
      curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
      curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
      sha256sum -c --ignore-missing ocb_checksums.txt
      chmod +x ocb
      tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
      cd "opentelemetry-collector-releases-${VERSION}"
      if [ -f scripts/prepare-obi.sh ]; then
        bash scripts/prepare-obi.sh otelcol-contrib
      fi
      ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
    outputs:
      - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"

Git snapshot (libXtst)

A pattern shared by libX11, libXext, libXi, libXrender:

fetch:
  sources:
    - git:
        repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
        ref: "libXtst-${VERSION}"
        include-history: true

Built-in Primitives Coverage

An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:

Fully declarative (no run: needed) — ~30 script files (15 unique patterns):

Built-in primitive Packages covered
vendor: {ecosystem: go} caddy, nats-server×2, oauth2-proxy
vendor: {ecosystem: go, submodules: [...]} etcd (3 submodules)
strip-tarball: {remove: [...]} nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs×2
fetch: {git: {repo, ref}} oniguruma, php-jsqueeze, libX11/Xext/Xi/Xrender/Xtst
fetch + verify: {checksums} nodejs download+verify
vendor: {ecosystem: composer} composer

Partially declarative (generic primitives + 1–2 run: steps):

Package Generic part Custom run: step
otel-collector, otel-collector-contrib vendor: {ecosystem: go} OCB binary code generation
java-openjdk×2 strip-tarball, fetch: {git} ./configure + make store-source-revision (needs boot JDK)
selinux-policy fetch: {git} (×3 repos) Multi-repo selective archiving

Genuinely custom (run: required) — 7 packages:

Package Why it can’t be declarative
nss-fips Container-based RPM download with subscription-manager credentials + QEMU cross-arch
openssl-fips-provider Container-based SRPM download + nested RPM extraction
ca-certificates Interactive multi-source crypto trust data merging
gcc GCC-specific changelog/PR extraction from git history
gdb Interactive patch management tooling (stgit)
python3.14 Koji task-specific JIT stencil artifact extraction
erlang27 Complex patch reformatting + spec rewriting

The run: escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract.

Open Questions

  1. Implementation approach. The pipeline YAML schema (ordered transform stages, run: escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:
    • Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
    • Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
    • Shared shell function library — the majority of scripts decompose into 3–4 operations (vendor_go, strip_tarball, fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands.
    • Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no run: blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.

Hummingbird Integration

This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.

Current Architecture

Two update systems

System Packages Source of tarballs Hook system
dist_git.py update 450 (clean + modified) Fedora lookaside cache (via sources file copied from Fedora dist-git) None
check_upstream_versions.py 22 (independent) Upstream URLs (via download_sources hooks or default spec URL download) Yes: update_spec, download_sources, post_update

dist_git.py update flow:

  1. Clones Fedora dist-git for the package
  2. Checks version, Koji build status, pre-release filtering
  3. Copies entire Fedora dist-git checkout into rpms/<package>/ via shutil.copytree — this includes the spec, patches, .gitignore, and sources file
  4. For modified packages, performs a 3-way git merge to preserve local changes
  5. Commits everything (spec, patches, sources file, metadata)

The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash). Actual tarballs are excluded by .gitignore and stored in a lookaside cache.

check_upstream_versions.py flow:

  1. Queries release-monitoring.org for new upstream versions
  2. Runs three hook phases per package:
    • update_spec — updates the spec’s Version/Release (default: specfile.update_version())
    • download_sources — downloads source tarballs (default: fetches from spec Source URLs)
    • post_update — additional steps (default: no-op)
  3. Uploads downloaded tarballs to Hummingbird’s lookaside cache
  4. Updates the sources file with new checksums
  5. Commits

Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.

Three lookaside cache backends

Configured in mock/dist-git-client.ini:

Backend URL pattern Used by
Fedora src.fedoraproject.org/repo/pkgs/rpms/{name}/{filename}/{hashtype}/{hash}/{filename} ~410 packages (default)
CentOS Stream sources.stream.centos.org/sources/rpms/{name}/... 1 package (rust-rpm-sequoia)
Hummingbird d1766whheab9hg.cloudfront.net/rpms/{name}/... (S3: arr-hummingbird-prod-dist-git-cache) ~40 packages (independent + forked)

The forked_from field in ci/package-overrides.yaml determines which cache a package uses at build time. Packages without forked_from default to Fedora’s cache.

How Konflux builds consume sources

Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):

  • monorepo-subdir: rpms/<package> — locates the spec, patches, and sources file
  • package-name: <name> — upstream name for lookaside URL construction
  • forked-from: <url> — (optional) selects the lookaside backend
  • dist-git-client-configdir: mock/ — (optional) points to the custom dist-git-client.ini

The pipeline uses dist-git-client to read the sources file and download tarballs from the appropriate cache, then runs mock to build RPMs.

Existing source generation patterns

Some independent packages already have tarball generation scripts:

  • rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh — downloads upstream release, runs OCB to generate source code, vendors Go dependencies
  • rpms/caddy/create-vendor-tarball.sh — downloads upstream tarball, generates Go vendor archive
  • rpms/nodejs25/packaging/make-nodejs-tarball.sh — downloads upstream, strips bundled content

These scripts are wired into the update path via download_sources hooks in metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache.

Integration with dist_git.py update

The current flow copies everything from Fedora dist-git (including the sources file) via shutil.copytree. The pipeline tool inserts after this step:

Current:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Commit

Proposed:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Run source-pipeline tool             ← fetches from upstream, replaces sources file
  4. Upload tarballs to Hummingbird lookaside
  5. Commit                               ← sources file now points to Hummingbird lookaside

Implementation:

  • After the shutil.copytree (or merge for modified packages), check if metadata/<package>.source-pipeline.yaml exists
  • If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the sources file
  • If it does not exist: run the built-in default (fetch from spec URLs, upload, replace sources)
  • A --skip-pipeline flag allows falling back to the current behavior during migration
  • Update ci/package-overrides.yaml to set forked_from to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)

Integration with check_upstream_versions.py

The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating per-package update behavior into a single *.source-pipeline.yaml file:

Current:
  1. update_spec hook/default             ← *.update-hooks.yaml
  2. download_sources hook/default        ← *.update-hooks.yaml
  3. post_update hook/default             ← *.update-hooks.yaml

Proposed:
  1. spec-update (pipeline YAML)          ← replaces update_spec hooks
  2. fetch + transform (pipeline YAML)    ← replaces download_sources hooks
  3. post (pipeline YAML)                 ← replaces post_update hooks

This reduces per-package metadata from three files (metadata/<package>.json, *.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for identity/tracking, *.source-pipeline.yaml for all update behavior).

Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed.

Integration with Konflux

Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via dist-git-client as it does today.

Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.

Hummingbird-specific configuration

Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.

GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.

The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.

Local development

Developers can run the tool directly:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  -v /tmp/output:/output \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0

# Outputs in /tmp/output/:
#   curl-8.21.0.tar.xz
#   curl-8.21.0.tar.xz.asc
#   sources
#   report.json

To preview without producing artifacts:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0 \
  --dry-run

# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.

Migration Path

Phase 1: Independent Go packages

Migrate the 22 independent packages that already fetch from upstream. Convert their existing create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions. Validates the tool against known-good packages.

Scope: 7 packages with existing scripts + 15 without.

Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code generation, but their vendor step is generic.

Phase 2: Trivial clean packages

Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.

Prerequisites:

  • Audit to classify packages as trivial vs. transformed (HUM-4620)
  • dist_git.py update integration complete (HUM-4621)

Per-package steps:

  1. Run the pipeline tool, compare output against existing Fedora-sourced tarball
  2. If identical: add forked_from: hummingbird to package-overrides.yaml
  3. If different: flag for Phase 3

Phase 3: Transformed packages

Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:

  • Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): strip-tarball
  • Git snapshots (libX11 family×5, oniguruma, php-jsqueeze): fetch: {git: ...}
  • Composer vendor (composer): vendor: {ecosystem: composer}

Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require run: blocks for genuinely custom logic.

Scope: determined by Phase 2 audit.

Phase 4: Policy enforcement

Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates.

Phase 5: Patch verification

Enable patches.verify and patches.classify for packages. Start with warn-only (fail-on-unverified: false), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages.

Operational Health

Escape hatch ratio

The primary health metric for the source pipeline is the escape hatch ratio: the percentage of pipeline YAML definitions that contain run: blocks (in transform or post sections) relative to the total number of pipeline YAML definitions.

A run: block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the same way: a package that cannot be expressed declaratively gets a run: block. This makes the ratio a single number that aggregates all pressures on the declarative model.

Interpretation:

Ratio Signal
< 15% Healthy. The primitive set covers real-world needs.
15-25% Watch. Look for repeated patterns in run: blocks that should become primitives.
> 25% Action needed. The YAML is becoming a workflow engine. Either expand the primitive set or reconsider the abstraction.

When the ratio climbs, examine the run: blocks for repeated patterns. If three or more packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad – it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set.

CI enforcement

The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files, calculates the escape hatch ratio, and reports the result. It runs as part of make check.

  • During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need run: blocks before new primitives are added.
  • After Phase 3: enable --fail mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a run: block is added.

4.2 - Agent-Friendly Documentation

How AI agents and LLMs can discover and consume the Project Hummingbird documentation site.

The documentation site provides several features that make it easier for AI agents and LLMs to discover, index, and consume its content.

Discovery and access methods

llms.txt

The site publishes an llms.txt file at the root, following the llmstxt.org specification. This file is a Markdown-formatted index of all documentation pages, grouped by section, with links to each page’s markdown URL. It serves as the primary entry point for agents discovering the site.

llms-full.txt

The site also publishes llms-full.txt – a full content dump of all documentation pages in a single file. This is intended for LLMs with large context windows that want to ingest the entire site in a single request without crawling individual pages.

Pages whose rendered content exceeds a size threshold (currently 50,000 characters) are replaced with a placeholder that includes the page title, description, and a link to the individual page’s markdown URL for on-demand fetching. This keeps the file at a manageable size while ensuring no content is completely inaccessible.

Per-page markdown output

Every page on the site is available in clean markdown format by appending /index.md to its URL. For example:

  • HTML: https://hummingbird-project.io/docs/contributing/quickstart/
  • Markdown: https://hummingbird-project.io/docs/contributing/quickstart/index.md

The markdown output resolves all shortcodes (such as file includes from other repositories) while preserving the content as markdown rather than converting to HTML. Each markdown page includes links to both llms.txt and llms-full.txt so agents can discover the site-wide indexes from any page.

Specifications and tools

The site aims to follow the emerging standards for agent-friendly documentation:

  • llmstxt.org – the llms.txt specification
  • Agent-Friendly Documentation Spec – broader specification for documentation sites that serve AI agents
  • AFDocs – scoring tool that evaluates a site’s compliance with the Agent-Friendly Documentation Spec

Planned features

Content negotiation (HUM-6570)

A CloudFront content-negotiation layer will allow agents to request text/markdown via the HTTP Accept header and receive the markdown variant without knowing the /index.md URL convention. This is a requirement of the Agent-Friendly Documentation Spec.

HTML llms-directive (HUM-6571)

A Docsy upgrade to v0.17.0 will embed a pointer to llms.txt inside the HTML <body> of every page, so agents landing on any HTML page can discover the LLMS index without prior knowledge of the site structure.

Documentation quality evaluation (HUM-6575)

A promptfoo-based evaluation harness will measure how accurately LLMs can answer questions about Hummingbird using the published documentation. This provides a feedback loop for identifying documentation pages that need improvement for agent consumption.

4.3 - Containers repository

Background documentation sourced from the containers repository.

4.3.1 - Image Pipeline

How the container image pipeline works from source to release

Explanation of the complete container image pipeline, from RPM dependency updates through source templates, generation, build, testing, release, and advisory publication.

Overview

The container image pipeline consists of six main stages:

  1. Source Templates - Jinja2 templates in images/*/ define container images
  2. Generation - Templates are rendered into Containerfiles and Konflux resources
  3. Build - Konflux builds multi-architecture container images
  4. Testing - Testing Farm runs integration tests via Konflux
  5. Enterprise Contract Validation - Conforma validates policy compliance before release
  6. Release - Images are published to registries and advisories are created

RPM packages are built and published separately via the RPM Pipeline. This document covers what happens after RPMs are available in the Hummingbird package repository.

RPM Dependency Updates

Container images pin exact RPM versions via lockfiles. When new RPMs are published to the Hummingbird Pulp repository, lockfiles must be regenerated for images to pick up the updates.

Lockfile structure

Each image variant has two RPM-related files:

File Purpose
images/<image>/<distro>/<variant>/rpms/rpms.in.yaml Declares required packages
images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml Pins exact versions, URLs, and checksums

The lockfile contains every RPM (direct and transitive dependencies) with its exact version, architecture, URL pointing to the Pulp repository, and checksum. During build, the locked RPMs are downloaded in hermetic mode — no network access to external repositories.

Automatic lockfile updates

MintMaker / Renovate — A custom Renovate instance (forked rpm-lockfile manager from redhat-exd-rebuilds/renovate) runs as a Kubernetes CronJob. It scans each rpms.in.yaml file individually, resolves dependencies against Pulp repositories, and opens auto-merge MRs — one per image group and distro (e.g., “Refresh RPM lockfiles for go-1-25/hummingbird”).

The underlying tool is rpm-lockfile-prototype, which wraps the DNF dependency solver. For a given rpms.in.yaml, it resolves all direct and transitive dependencies against the yum repositories defined in yum-repos/*.repo and outputs a complete rpms.lock.yaml with pinned versions, download URLs, and checksums. Locally, generate_rpms_lock.py adds hash-based optimization: it hashes the input file and skips regeneration when the hash matches the existing lockfile, avoiding expensive solver runs when nothing has changed.

Manual lockfile updates

To refresh the lockfile for a single image variant:

make images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml FORCE_REFRESH=true

For example:

make images/caddy/hummingbird/default/rpms/rpms.lock.yaml FORCE_REFRESH=true

To regenerate all lockfiles and open MRs for the changes:

# Regenerate all rpms.lock.yaml files
make all-host FORCE_REFRESH=true
# Open MRs for any changed lockfiles
ci/create_lockfile_update_mrs.sh

From lockfile MR to container build

When a lockfile MR merges to main, PipelinesAsCode triggers a container build (Stage 3). Each image variant has CEL expressions in its PipelineRun definition that match on file paths, so only images whose lockfiles changed get rebuilt.

Stage 1: Source Templates

Each container image is defined by templates in images/<image-name>/:

  • properties.yml - Image configuration (packages, variants, tags, etc.)
  • Containerfile.j2 - Jinja2 template for the container build
  • README.md.j2 - Documentation template
  • tests-container.yml - Integration test definitions

Templates use reusable macros from macros/*.yml.j2:

  • setup_newroot() - Configures DNF and filesystem
  • install_newroot() - Installs packages
  • cleanup_newroot() - Cleans up files
  • final_stage() - Creates scratch-based final image

Shared configuration is defined in images/variables.yml. See Global Variables Reference for details.

Stage 2: Generation

Templates are rendered into concrete artifacts that drive the pipeline:

  • Containerfiles - Build instructions for each image variant
  • README documentation - Image documentation for Quay.io
  • Konflux resources - CI/CD pipeline definitions

Containerfile Generation

Containerfiles are generated from templates for each image variant using Make’s incremental build system:

make

This combines:

  • Reusable macros from macros/
  • Service-specific templates from images/*/Containerfile.j2
  • Configuration from properties.yml (see Image Configuration Reference)
  • Variables from images/variables.yml (see Global Variables Reference)
  • RPM versions from rpms.lock.yaml files
  • Git submodule information from .gitmodules

Output: images/<image-name>/<variant>/Containerfile, along with VERSION and TAGS files

The build system uses timestamp-based dependency tracking, so only changed files are regenerated.

README Generation

After Containerfiles are generated, README documentation is generated from README.md.j2 templates:

  1. Tag values are extracted from the generated Containerfile labels
  2. README is rendered using macros from macros/readme.yml.j2
  3. Generated README includes actual version tags from the Containerfile

Output: images/<image-name>/README.md

Konflux Resource Generation

Konflux CI/CD resources are generated from templates in konflux-templates/:

make

This generates:

  • Components - Define what to build (one per image variant)
  • ImageRepositories - Define where to push images
  • ReleasePlanAdmissions - Define how to release images

Output: konflux-templates/rendered.yml

These resources must be deployed to Konflux before builds can run. See the Konflux Resource Deployment guide for how and when resources are deployed.

Stage 3: Build

Konflux builds container images automatically when changes are pushed to GitLab.

Build Triggers

  • Merge Requests: Builds all changed images and triggers tests
  • Main Branch: Builds all changed images (tests do not run on main)

Build Process

For each image variant:

  1. Konflux Component watches the GitLab repository
  2. On changes, Konflux triggers a build PipelineRun
  3. The build uses the generated Containerfile from images/<name>/<variant>/Containerfile
  4. Images are built for multiple architectures (x86_64 and aarch64)
  5. Built images are pushed to the development registry

Build Output

Development images are pushed to the Red Hat User Workloads registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main

Merge request builds are tagged with:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main:on-mr-<MR_ID>-<COMMIT_SHA>

Examples:

  • quay.io/redhat-user-workloads/hummingbird-tenant/curl--default--main
  • quay.io/redhat-user-workloads/hummingbird-tenant/nginx--builder--main:on-mr-123-abc1234

SBOM Generation

Each per-architecture build also produces an SPDX 2.3 Software Bill of Materials (SBOM), attached as an OCI artifact to the image. The SBOM is assembled from two independent scans, merged into a single document:

flowchart LR
    syft["Syft\n(buildah-remote-oci-ta)"] --> mobster["Mobster\n(buildah-remote-oci-ta)"]
    hermeto["Hermeto\n(prefetch-dependencies-oci-ta)"] --> mobster
    mobster --> sbom["Per-arch SPDX SBOM"]
    sbom --> index["Mobster\n(build-image-index)"]
    index --> oci["Index SBOM\n(.sbom OCI artifact)"]
  • Syft runs as the sbom-syft-generate step inside the buildah-remote-oci-ta Tekton task. It scans the RPM database of the built per-arch image and finds installed binary RPMs plus non-RPM packages (Go modules, pip packages, etc.). One SBOM is produced per architecture.
  • Hermeto runs inside the prefetch-dependencies-oci-ta Tekton task. It records all build-time dependencies from lockfiles. A single Hermeto SBOM covers all architectures and source RPMs.
  • Mobster runs as the prepare-sboms step inside buildah-remote-oci-ta, merging the Syft and Hermeto SBOMs into one SPDX document per architecture. A second Mobster invocation in the build-image-index task generates an index-level SBOM and attaches it as an OCI artifact.

See Security Labels and Metadata for the SBOM entry structure and how to access SBOMs from the registry.

Stage 4: Testing

Images are validated through two types of integration tests:

  • Container tests (tests-container.yml) - Run with Podman and Docker via Testing Farm on RHEL-9-Nightly systems
  • K8s tests (tests-k8s.yml) - Run in Konflux ephemeral Kubernetes namespaces

Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test Triggering

Tests are only triggered on merge requests, not on main branch builds.

For an image group at images/<group-name>/, tests are triggered for all changes below that directory, excluding documentation-only changes.

Any changes below .tekton/ or ci/ will trigger tests for the caddy image to ensure infrastructure changes work before merging.

Test Execution

For each image group, pipelines are created per variant. If there is more than one variant in an image group, additional group pipelines are created. See Konflux group snapshot documentation for details.

Tests are selected based on the variants field in test files. Additionally, global tests from ci/{variant}-tests/ are included. Group pipelines run tests that specify variants: [group], allowing validation across multiple variants.

Reverse Dependency Testing

When a base image like core-runtime changes, dependent images (rust, xcaddy, etc.) need to be tested to ensure compatibility. Reverse dependency testing is enabled by default. Images can opt-out by setting reverse_dependency_tests: false in their properties.yml. This is recommended for images like “curl” which are widely used and only have a small API which their own tests cover well enough.

For container tests, dependent images are rebuilt locally in the testing environment to prevent version skew and ensure dependent images are in sync with the repository status in the merge request under test.

Integration Test Scenarios

Integration tests are triggered via IntegrationTestScenario resources defined in the infrastructure repository:

Application Purpose
containers-hummingbird Red Hat supported Hummingbird images
containers-community-hummingbird Community images (support_level: community)
containers-ci-hummingbird Experimental/infrastructure images (support_level: experimental)
containers-rawhide All Rawhide-based images

Container Testing

Container tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Container Testing Flow

  1. Developer opens MR modifying images/nginx/
  2. GitLab CI pipeline and Konflux pipelines start in parallel
  3. Konflux builds nginx image variants
  4. IntegrationTestScenario triggers Testing Farm job
  5. tmt discovers fmf plan (ci/run_tests_container.fmf)
  6. Testing Farm provisions machines (x86_64 and aarch64 with RHEL-9-Nightly)
  7. tmt sets up the testing environment
  8. tmt runs tests via ci/run_tests_container.sh

Container Test ITS Configuration

The container tests use the upstream Testing Farm pipeline for Konflux CI:

kind: IntegrationTestScenario
spec:
  contexts:
    - {name: pull_request}
  params:
    - {name: COMPOSE, value: RHEL-9-Nightly}
    - {name: ARCH, value: x86_64|aarch64}  # one for each
    - {name: IMAGE_TAG, value: v3.2}
  resolverRef:
    resolver: bundles
    params:
      - {name: bundle, value: quay.io/testing-farm/tmt-via-testing-farm:$(params.IMAGE_TAG)}
      - {name: name, value: tmt-via-testing-farm}
      - {name: kind, value: pipeline}

FMF Test Plan

The root folder of the containers repository is marked with a .fmf directory to enable Testing Farm support. The actual test plan in ci/run_tests_container.fmf runs the following steps:

  1. Install podman for container testing
  2. Install Docker and start the Docker daemon on the host for Docker integration tests
  3. Fix git submodules until TFT-3991 is resolved
  4. For regular pipelines:
    • Verify the image built by Konflux is reproducible using ci/test_rebuild.sh
    • If needed, build reverse dependency images locally via Buildah
    • Run Podman and Docker tests via ci/run_tests_container.sh --component-name
  5. For group pipelines:
    • Run Podman and Docker group tests via ci/run_tests_container.sh --group-component-name

Container Test Environment

Testing Farm provides these environment variables to the test plan:

Variable Description
IMAGE_NAME Single component (e.g., curl--default--main)
IMAGE_NAMES Multiple components for group pipelines
IMAGE_URL Image URL from Konflux
IMAGE_URL_... Image URLs from Konflux for group pipelines
SNAPSHOT_b64 Snapshot metadata (base64 encoded)

K8s Testing

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

K8s Testing Flow

  1. Developer opens MR modifying an image in the containers repository (e.g., nginx)
  2. GitLab CI pipeline and Konflux pipelines start in parallel
  3. Konflux builds image variants
  4. IntegrationTestScenario triggers K8s test pipeline
  5. Pipeline checks for tests-k8s.yml; skips with SUCCESS if not found
  6. Pipeline provisions ephemeral namespace via Konflux EaaS (tied to PipelineRun lifecycle)
  7. Pipeline fetches source via Trusted Artifacts
  8. Tests run via ci/run_tests_k8s.sh with kubeconfig for ephemeral namespace
  9. Pipeline fails if any test reports non-SUCCESS, ensuring GitLab sees correct status

K8s Test ITS Configuration

The K8s tests use the k8s-test-pipeline:

kind: IntegrationTestScenario
spec:
  contexts:
    - {name: pull_request}
  resolverRef:
    resolver: bundles
    params:
      - {name: bundle, value: quay.io/hummingbird-ci/k8s-test-pipeline:latest}
      - {name: name, value: k8s-test}
      - {name: kind, value: pipeline}

Stage 5: Enterprise Contract Validation

Before images can be released, they must pass Enterprise Contract (also known as Conforma) policy validation. This ensures images meet security, compliance, and build quality standards. These checks can also be run locally against Konflux-built images.

Policy Validation

Enterprise Contract validates that:

  • Images are built using trusted, verified Tekton tasks
  • Builds are hermetic (network-isolated with pre-fetched dependencies)
  • Required security tests have passed
  • Images have proper metadata and labels
  • Build artifacts meet supply chain security requirements

Policy Configuration

Policies are defined as EnterpriseContractPolicy resources in konflux-templates/macros/policy.yml.j2:

  • containers-hummingbird / containers-rawhide - Strict policies for production images
  • containers-community-hummingbird - Policy for community-supported images in the containers repository (same base exclusions as production policies)

These policies use the @redhat rule collection from the ec-release-policy.

Policy Exclusions

The following checks are excluded from the default @redhat policy set. When modifying exclusions, update the policy macro and this documentation.

Test Package

The test package verifies that each build was subjected to a set of tests and that those tests all passed.

Snyk SAST checks (test.required_tests_passed:sast-snyk-check, test.no_skipped_tests:sast-snyk-check, test.required_tests_passed:sast-snyk-check-oci-ta, test.no_skipped_tests:sast-snyk-check-oci-ta) are excluded because Hummingbird images are currently not supported by Snyk.

Red Hat certification preflight checks (test.no_failed_tests:ecosystem-cert-preflight-checks, test.no_erred_tests:ecosystem-cert-preflight-checks) are excluded because Hummingbird images are not yet published to the Red Hat certified container registry.

Informative test failures (test.no_failed_informative_tests) are excluded because these produce warnings for advisory purposes only and are explicitly non-blocking.

Deprecated image warnings (test.no_test_warnings:deprecated-image-check) are excluded because final images are built FROM scratch, and the builder image is updated via Renovate like all other images.

Trusted Task Package

The trusted_task package verifies that all Tekton Tasks involved in building the image are trusted by comparing Task references with a pre-defined list of trusted Tasks.

The trusted_task.current check warns when newer versions of tasks are available. This is excluded because we use stable pinned task versions and control upgrade timing via Renovate rather than requiring the latest version at all times.

RPM Repos Package

The rpm_repos package confirms that all RPM packages listed in SBOMs specify a known and permitted repository ID.

The rpm_repos.ids_known check is excluded because images use the internal hummingbird repository and Fedora repositories, which are not in the upstream known_rpm_repositories.yml list (that file only contains Red Hat official repositories).

Labels Package

The labels package checks if the image has the expected labels set, including required and optional labels for Red Hat container certification.

Both labels.required_labels and labels.optional_labels are excluded because images currently only include basic labels (maintainer, license_terms, name, cpe) and version labels, not the full set of Red Hat certification labels (vendor, version, release, summary, description, url, etc.) required for Red Hat Ecosystem Catalog publishing.

Buildah Build Task Package

The buildah_build_task package verifies buildah build task parameters.

The buildah_build_task.privileged_nested_param check verifies that PRIVILEGED_NESTED is not set to true. This is excluded because images use the dnf-installroot helper from the builder image to build all containers, including the builder image itself. This script requires privileged operations (unshare, mount -t tmpfs, mount --bind for /proc and /dev/*) to set up the install root environment (see commit 3668af16).

Schedule Package

The schedule package verifies that releases conform to a given schedule, including weekday restrictions.

The schedule.weekday_restriction check is excluded to allow releases any day including weekends. The @redhat policy restricts weekend releases, but this project needs the ability to ship urgent CVE fixes immediately regardless of the day of week.

CVE Package

The cve package checks for blocking and non-blocking CVEs in container images.

The cve.cve_blockers check is excluded because blocking on known CVEs would prevent releasing images that fix other CVEs. A VEX feed could suppress false positives for RPM-level CVEs, but CVEs can also originate from other artifact types where VEX does not apply (see MR !2227).

Hermetic Task Package (CI-Only)

The hermetic_task package verifies that tasks were invoked with the proper parameters to perform a hermetic (network-isolated) execution.

All Konflux applications in the containers repository enforce the hermetic task check. All applications allow quay.io/hummingbird-ci/ as a base image source because images build FROM quay.io/hummingbird-ci/hummingbird-builder via setup_newroot.yml.j2.

Stage 6: Release

After images pass testing in merge requests and are merged to main, they are released to public registries.

Registry Organization

Images are published to different registries based on distro and purpose:

Registry Purpose
registry.access.redhat.com/hi/ Red Hat supported Hummingbird images (production)
quay.io/hummingbird/ Red Hat supported Hummingbird images (mirror)
quay.io/hummingbird-community/ Community-supported Hummingbird images
quay.io/hummingbird-rawhide/ All Rawhide-based images
quay.io/hummingbird-ci/ CI and tools images

Production Registry: Red Hat supported Hummingbird images are published to registry.access.redhat.com/hi/ via the rh-advisories release pipeline and mirrored to quay.io/hummingbird/.

Community Registry (hummingbird-community): Publishes community-supported images (those with support_level: community in properties.yml). Examples include minio, minio-client, and bootc-os.

CI/tools Registry (hummingbird-ci): Publishes tooling and infrastructure images. This includes tools-repo images (built outside the containers repository) and experimental images from the containers repository (support_level: experimental), such as hummingbird-builder.

Release Process

  1. Merge request passes all tests
  2. Merge request is merged to main branch
  3. Konflux builds images from main
  4. ReleasePlanAdmission resources trigger the release pipeline
  5. Images are signed via Cosign for supply chain attestation
  6. Images are copied from the Konflux registry to target registries
  7. Tags are applied based on properties.yml configuration
  8. For production releases: images are registered in the Red Hat container catalog via Pyxis

Release Mechanism

Releases are configured via ReleasePlanAdmission (RPA) resources in the containers repo and ReleasePlan resources in the infrastructure repo:

  • ReleasePlanAdmission - Defines per-registry release configuration (target registry, tags, visibility settings)
  • ReleasePlan - Triggers the release pipeline for a specific application and registry

Each distro/registry combination has its own RPA.

Production release pipeline

The production release uses the rh-advisories pipeline from the release-service-catalog. The push-snapshot task pushes images from the development registry to:

  • Production: registry.access.redhat.com/hi/<image>:<tags>

Configuration is defined in releng/hummingbird-containers-prod.yaml.

Image signing

Images are signed using Cosign as part of the release pipeline. Signing provides supply chain attestation, allowing consumers to verify image provenance.

Pyxis catalog registration

The release pipeline registers images in the Red Hat container catalog via the Pyxis API:

  • Product: Red Hat Hardened Images (Product ID 1071)
  • Metadata: version, categories, layer information, SBOM references
  • Configuration: releng/pyxis-hummingbird.yaml

Pyxis registration makes images discoverable in the Red Hat Ecosystem Catalog and links them to security advisories.

Release Output

Released images are published to registries based on distro and support level:

registry.access.redhat.com/hi/<image-repository>:<image-tag>
quay.io/hummingbird/<image-repository>:<image-tag>
quay.io/hummingbird-community/<image-repository>:<image-tag>
quay.io/hummingbird-rawhide/<image-repository>:<image-tag>
quay.io/hummingbird-ci/<image-repository>:<image-tag>

Examples:

  • registry.access.redhat.com/hi/curl:8 - Production Red Hat supported image
  • quay.io/hummingbird/nodejs:20 - Red Hat supported Hummingbird image
  • quay.io/hummingbird-community/minio:latest - Community-supported image
  • quay.io/hummingbird-ci/hummingbird-builder:latest - Hummingbird Builder image
  • quay.io/hummingbird-rawhide/curl:latest - Rawhide image

Release Tags

Tags are extracted from Containerfile labels as defined in properties.yml:

  • latest - Latest version of the image
  • <major> - Major version (e.g., 20 for Node.js 20.x)
  • <major>.<minor> - Major.minor version (e.g., 20.11)
  • <full-version> - Complete version with release (e.g., 20.11.1-1.fc42)
  • <timestamp> - Build timestamp (production releases only)

Non-default variants receive a -<variant> suffix (e.g., latest-builder).

Advisory Creation

When the production release pipeline runs, advisories are created:

  1. The Release Service generates an Advisory YAML from ReleasePlan and ReleasePlanAdmission metadata
  2. The advisory is pushed to the advisories repo on CEE GitLab
  3. GitLab CI validates the advisory against the schema and enforces field-level permissions
  4. On merge to main, the advisory is published via UMB and Kafka

Advisory types:

Type Meaning
RHSA Security Advisory (contains CVE fixes)
RHBA Bug Fix Advisory
RHEA Enhancement Advisory

After publication, the Pyxis API links each image record to its advisory via image_advisory_id.

VEX Feed Update

After advisory publication, Red Hat SDEngine generates public vulnerability data:

  • CSAF VEX documents (per-CVE) published at https://security.access.redhat.com/data/csaf/v2/vex-feed/
  • CSAF Advisory documents (per-advisory) published at https://security.access.redhat.com/data/csaf/v2/advisories/

The hummingbird-vex-feed repository maintains CPE mappings that associate Hummingbird package repositories with cpe:/a:redhat:hummingbird:1.

Image Documentation

When a commit to main changes a README.md file, the content is automatically pushed to quay.io as the image description via the update_quay_description job.

References

4.3.2 - Global Variables Reference

Complete reference for global configuration in images/variables.yml

Complete reference for global configuration settings that apply to all container images.

Overview

The images/variables.yml file contains global configuration that applies to all container images in the repository. These settings define project-wide defaults, security parameters, and variant-specific behavior.

Location: images/variables.yml (in the containers repository root)

Scope: All images inherit these settings unless overridden in image-specific properties.yml

Configuration Reference

cpe

  • Type: String (CPE 2.2 formatted string)
  • Default: "cpe:/a:redhat:hummingbird:1"
  • Description: Common Platform Enumeration (CPE) identifier for all Hummingbird containers.
  • Usage: Automatically added as a label to all generated container images and passed to inject-source-info during the build process.
  • Format: CPE 2.2 URI format: cpe:/part:vendor:product:version

default_user

  • Type: String (numeric UID)
  • Default: "65532"
  • Description: The default unprivileged user ID that all containers run as.
  • Usage: Referenced by the {{ set_user() }} macro when user: default is specified (or when user: is omitted, as default is the default value).
  • Value: 65532 is chosen as a high, non-conflicting UID that:
    • Avoids conflicts with system users (typically < 1000)
    • Avoids conflicts with regular users (typically 1000-60000)

default_distros

  • Type: Array of strings
  • Default: ["rawhide", "hummingbird"]
  • Description: Defines the distribution variants (distros) for which images are built. Each distro represents a different base package source.
  • Directory Structure: Images are organized as images/<name>/<distro>/<variant>/
  • Common Values:
    • rawhide - Uses Fedora Rawhide repositories only
    • hummingbird - Uses both Fedora Rawhide and Hummingbird repositories
  • Usage: Scripts iterate over distros and variants to find all image variants to build. Each distro/variant combination produces a separate container image.
  • See Also: default_variant_repos for repository configuration per distro

default_variants

  • Type: Array of strings
  • Default: ["default", "builder"]
  • Description: Defines the default set of variants to generate for each image within each distro when not explicitly specified in the image’s properties.yml.
  • Directory Structure: Variants are subdirectories within each distro: images/<name>/<distro>/<variant>/
  • Common Values:
    • default - The minimal runtime variant
    • builder - Extended variant with build tools and package managers
  • Extending: Images can use additional_variants in properties.yml to add extra variants while keeping the defaults (preferred approach)
  • Override: Images can specify variants in properties.yml to replace these defaults entirely (use when you need to exclude default variants)
  • See Also: Image Configuration Reference - variants

default_rpm_packages

  • Type: Object with variant names as keys, arrays of package names as values

  • Default:

    default_rpm_packages:
  builder:
    - bash
    - dnf5
    - shadow-utils

  • Description: Defines packages that are automatically included in specific variants across all images. These are added in addition to packages defined in image-specific properties.yml.

  • Variants: Currently only builder is defined, but other variants can be added

  • Usage: Use this to provide variant-specific packages that should be available in all images using that variant. For example, builder variants include package management tools for development workflows.

default_variant_repos

  • Type: Object with distro names as keys, arrays of repository filenames as values

  • Default:

    default_variant_repos:
  rawhide:
    - fedora-44.repo
  hummingbird:
    - fedora-44.repo
    - hummingbird.repo

  • Description: Defines which yum repositories each distro uses by default for package installation and lockfile generation. The Rawhide distro uses the latest branched Fedora development repo, while Hummingbird uses a stable Fedora release plus the Hummingbird package repository.

  • Repository Files: References files in the yum-repos/ directory

  • Usage: This allows different distros to use different package sources. The hummingbird distro includes additional repositories that provide Hummingbird-specific packages.

  • Override: Image-specific additional_repos in properties.yml are appended to the distro-specific repos

  • See Also: Image Configuration Reference - additional_repos

oscap

  • Type: Object

  • Default:

    oscap:
  enabled: true
  profiles:
    cis: true
    stig:
      variants:
        - "*fips*"
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_root_path_no_dot
      reason: "False positive: environmentvariable58 probe requires /proc
        which is unavailable in offline chroot scanning"

  • Description: Global defaults for OpenSCAP compliance scanning. Scanning is enabled by default (enabled: true). Images can opt out by setting enabled: false in their properties.yml.

  • Fields:

    • enabled - Whether compliance scanning is active (overridden per image)
    • profiles - Which compliance profiles run per variant. Each profile can be true (all variants), false (disabled), or {variants: [...]} with glob patterns. Default: CIS for all variants, STIG for FIPS variants.
    • exclude_rules - Global rule exclusions applied to all oscap-enabled images. Image-specific exclusions in properties.yml are concatenated with these (not replaced).

  • Merge Behavior: When an image overrides oscap fields:

    • enabled (scalar) is replaced by the image value
    • profiles (dict) is recursively merged, so an image can override individual profiles without affecting others
    • exclude_rules (list) is concatenated, so image rules are added after global rules

  • See Also: Image Configuration Reference - Compliance Scanning

readme_targets

  • Type: Object with target names as keys, configuration objects as values

  • Default: See the actual file

  • Description: Defines target-specific configuration for generating multiple README files from a single template. Each target represents a different build/distribution channel (e.g., Hummingbird project vs Red Hat product).

  • Target Configuration Fields:

    • filename - Output filename for this target’s README
    • registry - Container registry URL for image references
    • product_name - Full product name for documentation
    • product_name_short - Short product name for headings and titles
    • doc_base_url - Base URL for documentation links

  • Usage: Templates access these values via {{ readme_targets[target].registry }} and similar expressions. The build system generates one README file per target from the same README.md.j2 template.

  • See Also: Image Pipeline - README Generation

Next Steps

4.3.3 - Konflux Resource Deployment

How Konflux resources are defined and deployed across repositories

Konflux resources for container images are split between two repositories:

  • containers: Per-image resources generated from properties.yml
  • infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Resource Type Source Containers MR Containers Push Infra MR Infra Push
Component containers manual manual - -
ImageRepository containers manual manual - -
ReleasePlanAdmission containers - automatic manual automatic
EnterpriseContractPolicy containers - automatic manual automatic
Application infrastructure - - manual automatic
ReleasePlan infrastructure - - manual automatic
IntegrationTestScenario infrastructure - - manual automatic
ServiceAccount, Secret infrastructure - - manual automatic

Legend:

  • Containers MR: Downstream pipeline triggered from containers MR (deploys from MR commit)
  • Containers Push: Downstream pipeline triggered from containers push to main
  • Infra MR: Infrastructure MR pipeline (manual trigger)
  • Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

Containers Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

  1. Ownership: Components are managed by the containers repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from containers deploy changes, giving the containers repo full control over the component lifecycle.
  2. Dynamic generation: Generated from images/*/properties.yml via ci/internal/generate_konflux_resources.sh, depending on per-image configuration (variants, tags, repository names).
  3. Timing: Must be deployed early during MR review so Konflux can build and test new images.

ReleasePlanAdmission:

  1. Dynamic generation: Includes per-image tag mappings extracted from properties.yml.
  2. Main branch only: Prevents race conditions—deploying from a feature branch risks undoing other images’ onboarding. If branch A adds image-foo and branch B (created earlier) adds image-bar, deploying B’s RPA would remove image-foo from the release config.

EnterpriseContractPolicy:

  1. Documentation co-location: Policy exclusions are documented in Image Pipeline. Keeping policy and documentation together eases maintenance.

Infrastructure Repo

Application, ReleasePlan, and IntegrationTestScenario are defined in kubernetes/containers-*/ directories (one per Konflux application):

  1. Independent iteration: Changes are decoupled from containers repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the containers repo.
  2. Testable before merge: If defined in the containers repo, these would only deploy after merging to main (like RPAs), making iteration difficult.
  3. Static configuration: These resources don’t depend on per-image data from properties.yml.

ServiceAccount and Secret for releases are defined in kubernetes/release-quay-hummingbird*/ directories (one per Quay organization):

  1. Security: Secret specifications (names, structure, credential references) should not be exposed in the containers repo.
  2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the containers repo.

The service accounts are referenced by ReleasePlanAdmissions to authorize pushing images to each quay.io/hummingbird* organization.

See Also

4.3.4 - Security Labels and Metadata

Container labels, embedded metadata, and SBOMs for vulnerability scanning

Hummingbird container images include labels, embedded metadata, and SBOMs that enable security scanners to perform container-first vulnerability reporting.

Overview

Security scanners need a way to determine which vulnerabilities are applicable to a container image. Three mechanisms provide this information:

  1. Container Labels - OCI image labels (name, cpe) set in the Containerfile
  2. Embedded Metadata - A labels.json file written to the container filesystem during build
  3. Software Bill of Materials (SBOM) - An SPDX document listing all packages, attached as an OCI artifact alongside the image

Labels and embedded metadata identify the product; the SBOM identifies the packages within it. Labels and embedded metadata provide the same core information but serve different access paths: labels are accessible via container inspection tools, while embedded metadata is accessible to scanners with only filesystem access.

Applicability

The cpe label is only added to released Hummingbird distro images (no CPE identifier for non-product images). All other labels are added to all images.

The labels.json embedded metadata file contains all Containerfile LABEL values plus auto-computed fields.

SBOMs are attached to all released images (both Rawhide and Hummingbird).

Container Labels

name

  • Type: String
  • Format: <org>/<image> or <org>/<image>-<variant>
  • Examples: hummingbird/nodejs-24, hummingbird/nodejs-24-builder, hummingbird/caddy
  • Description: Canonical name for the container image. This is the name that appears in VEX (Vulnerability Exploitability eXchange) statements and is used by scanners to map vulnerabilities to specific images.

The canonical name is derived from the image directory name (which includes the version, e.g. nodejs-24) and the variant. For default variants, the name is hummingbird/<image> (e.g. hummingbird/nodejs-24). For non-default variants, the variant is appended with a hyphen (e.g. hummingbird/nodejs-24-builder). Note that the canonical name may differ from the registry repository path. For example, nodejs-24 and nodejs-20 are both pushed to the hummingbird/nodejs registry repository with different tags, but their canonical names are hummingbird/nodejs-24 and hummingbird/nodejs-20 respectively.

See container-image-labels.md for the full name label format across all image types.

cpe

  • Type: String (CPE 2.2 formatted)
  • Format: cpe:/a:redhat:hummingbird:1
  • Description: Common Platform Enumeration (CPE) identifier for the container. CPE is a standardized naming scheme for software products that enables correlation with vulnerability databases.

The CPE value is defined globally in images/variables.yml and applied to all Hummingbird images. Containers with the same CPE are considered the same software product for vulnerability reporting purposes.

org.opencontainers.image.created

  • Type: String (RFC3339 timestamp)
  • Format: 2025-01-15T12:00:00Z
  • Description: Creation timestamp of the container image. Used by scanners to determine if an image predates or postdates a vulnerability fix.

If SOURCE_DATE_EPOCH is set during the build, that timestamp is used instead of the actual build time, supporting reproducible builds.

Embedded Metadata (labels.json)

The labels.json file is written to /usr/share/buildinfo/labels.json inside the container filesystem by the inject-source-info script during build. It contains all Containerfile LABEL values plus auto-computed fields like architecture and org.opencontainers.image.created.

Schema

The file follows the embedded_metadata.v1 schema published by Red Hat Product Security.

Fields

The embedded_metadata.v1 schema defines the minimum fields required by security scanners:

Field Type Description
name string Canonical container name (e.g., hummingbird/nodejs-24)
cpe string CPE identifier (e.g., cpe:/a:redhat:hummingbird:1)
architecture string Target architecture (e.g., amd64, arm64)
org.opencontainers.image.created string RFC3339 creation timestamp

The file also includes all other Containerfile LABEL values (e.g., description, summary, vendor, version). See container-image-labels.md for the complete label reference.

Example

{
  "name": "hummingbird/caddy",
  "cpe": "cpe:/a:redhat:hummingbird:1",
  "architecture": "amd64",
  "org.opencontainers.image.created": "2025-01-15T12:00:00Z"
}

Software Bill of Materials (SBOM)

Each released image has a per-architecture SPDX 2.3 SBOM attached as an OCI artifact. See SBOM Generation for how the production SBOMs are built from Syft, Hermeto, and Mobster.

Accessing SBOMs

SBOMs are stored as OCI artifacts alongside each per-architecture image. To download one, resolve the per-arch digest and use cosign:

IMAGE="quay.io/hummingbird/caddy"
TAG="latest"
ARCH="amd64"

# Get per-architecture digests from the image index
skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq '.manifests[] | {digest, platform}'

# Download the SBOM for the chosen architecture
ARCH_DIGEST="$(skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq -r --arg arch "${ARCH}" \
    '.manifests[] | select(.platform.architecture == $arch) | .digest')"
cosign download sbom "${IMAGE}@${ARCH_DIGEST}" > sbom.json

Entry Sources

The merged SBOM contains entries from two tools:

Source Content Scope
Syft Installed binary RPMs + non-RPM packages Image architecture only
Hermeto Build-time dependencies from lockfiles All architectures + source RPMs

A package installed in the final image may have entries from both Syft and Hermeto (with different metadata), since it was both a build dependency and is present at runtime.

Distinguishing Entry Sources

After the Mobster merge, Syft and Hermeto entries carry mutually exclusive markers:

Marker Syft Hermeto
sourceInfo field present ✓
licenseDeclared set ✓
CPE references ✓
upstream= in PURL ✓
Annotation containing “hermeto” ✓
repository_id= in PURL ✓

As a general rule, any entry with an annotation containing the word “hermeto” (case-insensitive) originates from Hermeto (build-time provenance). All other entries originate from Syft (runtime image scan).

PURL Format

Syft and Hermeto use different PURL qualifier sets for the same package. Examples from caddy:latest (Hummingbird, amd64):

Syft (runtime):

pkg:rpm/hummingbird/caddy@2.10.2-1.hum1?arch=x86_64&distro=hummingbird-20251124&upstream=caddy-2.10.2-1.hum1.src.rpm

Hermeto (build, binary):

pkg:rpm/caddy@2.10.2-1.hum1?arch=x86_64&checksum=sha256:ca02a0...&repository_id=public-hummingbird-x86_64-rpms

Hermeto (build, source):

pkg:rpm/caddy@2.10.2-1.hum1?arch=src&checksum=sha256:42912d...&repository_id=public-hummingbird-source-rpms

Key differences:

  • Distro namespace: Syft includes the distro in the PURL path (pkg:rpm/hummingbird/...). Hermeto omits it (pkg:rpm/...).
  • EVR: Syft populates versionInfo with the full epoch:version-release. Hermeto sets versionInfo to the bare upstream version; the full EVR is only in the PURL @version.
  • Architecture: Syft entries match the image architecture. Hermeto entries carry an arch= PURL qualifier that may be the image arch, a different arch (cross-arch), noarch, or src (source RPMs).
  • Source RPM: Syft entries carry upstream=<srpm> in the PURL. Hermeto entries have separate arch=src entries instead.

Hermeto Annotation Format

Hermeto entries carry annotations with JSON-encoded metadata:

{
  "annotationDate": "2026-03-02T11:58:12Z",
  "annotationType": "OTHER",
  "annotator": "Tool: hermeto:jsonencoded",
  "comment": "{\"name\": \"hermeto:found_by\", \"value\": \"hermeto\"}"
}

Entry Breakdown Example

Typical counts for caddy:latest (Hummingbird, amd64); exact counts vary as image dependencies change:

Source Typical count Content
Syft ~53 Binary RPMs (x86_64 + noarch)
Syft 1 Go module (stdlib)
Syft 2 OCI image metadata
Hermeto ~41 Binary RPMs (x86_64)
Hermeto ~41 Binary RPMs (aarch64, cross-arch)
Hermeto ~11 Binary RPMs (noarch)
Hermeto ~36 Source RPMs (arch=src)

How Scanners Use This Metadata

  1. CPE Matching: Scanners use the cpe value to look up applicable VEX statements for the product
  2. Name Matching: The name identifies which specific container the VEX statements apply to
  3. Version Comparison: The creation timestamp enables comparison between the scanned image and fixed versions reported in VEX statements
  4. Package Enumeration: Scanners use the SBOM to enumerate all packages in the image and correlate them with vulnerability databases via PURLs and CPEs
File Purpose
documentation/background/container-image-labels.md Complete reference for all image labels
documentation/background/image-pipeline.md SBOM generation pipeline (Stage 3)
images/variables.yml Defines the global cpe value
images/hummingbird-builder/inject-source-info.sh Script that creates labels.json
macros/inject_source_info_labels.yml.j2 Macro that adds LABEL to Containerfile
macros/install_newroot.yml.j2 Macro that invokes inject-source-info

See Also

References

4.3.5 - Container Image Labels

Complete reference for all container image labels

Hummingbird container images carry labels from multiple standards and namespaces. This page is the single reference for every label.

Standards: C = Conforma (rule dataset), O = OCI Image Spec, H = Hummingbird project, S = Security schema

Labels

Label Aliases Value C O H S
architecture Host architecture (e.g., x86_64) ✓
com.redhat.component hummingbird ✓
com.redhat.license_terms UBI EULA ¹
cpe CPE identifier (Hummingbird only) ² ✓ ✓
distribution-scope public ✓
io.hummingbird-project.containerfile Containerfile path relative to repo root ⁹ ✓
io.hummingbird-project.deprecated true for deprecated image streams ✓
io.hummingbird-project.major-minor-version Major.minor from tags (e.g., 2.10) ³ ✓
io.hummingbird-project.major-version Major from tags (e.g., 2) ³ ✓
io.hummingbird-project.repository Publishing name (e.g., caddy) ⁴ ✓
io.hummingbird-project.stream Version stream (e.g., 2) ⁴ ✓
io.hummingbird-project.variant Variant name (e.g., fpm-builder) ⁸ ✓
io.hummingbird-project.variant.base Base specialization (e.g., fpm) ⁸ ✓
io.hummingbird-project.variant.builder true when builder (absent otherwise) ⁸ ✓
io.hummingbird-project.variant.description Base variant description (no modifiers) ⁸ ✓
io.hummingbird-project.variant.fips true when FIPS (absent otherwise) ⁸ ✓
io.k8s.description Long description ⁵ ✓
maintainer Project Hummingbird / Red Hat ✓
name Canonical publishing repository ² ✓ ✓
org.opencontainers.image.created build-date RFC3339 build timestamp ✓ ✓ ✓
org.opencontainers.image.description description Long description ⁵ ✓ ✓
org.opencontainers.image.revision vcs-ref Full git commit SHA ✓ ✓
org.opencontainers.image.source GitLab repository URL ✓
org.opencontainers.image.title Image name (e.g., caddy) ✓
org.opencontainers.image.url url Upstream project URL ⁵ ✓ ✓
org.opencontainers.image.vendor vendor Red Hat, Inc. ⁶ ✓ ✓
org.opencontainers.image.version version Full version from tags (e.g., 2.10.2) ³ ✓ ✓
release SOURCE_DATE_EPOCH (commit timestamp) ⁷ ✓
summary Short one-liner ⁵ ✓
vcs-type git ✓

Notes

¹ Licenses

The com.redhat.license_terms label is not required by Conforma, OCI, or the security schema. It is a Red Hat convention present on all Red Hat container images (UBI, language runtimes, etc.), pointing to the UBI EULA.

The org.opencontainers.image.licenses label is intentionally not set. Per-package license data is derived from SBOMs stored as OCI artifacts alongside each image. A manually-set SPDX expression would be incomplete compared to the SBOM-derived data available in the image catalog.

² Name and identity

The name label identifies the canonical publishing repository. It uses the repository value from properties.yml, prefixed by the registry organization:

  • Red Hat supported Hummingbird images: hi/<repository>
  • Community images: hummingbird-community/<repository>
  • Experimental images: hummingbird-ci/<repository>
  • Rawhide images: hummingbird-rawhide/<repository>

All variants published to the same repository have the same name label. For example, the default and builder variants of Caddy both use hi/caddy.

The cpe label is only set for Hummingbird distro images (not Rawhide). See Security Labels and Metadata for details on the scanning workflow and labels.json.

³ Version labels

Three version granularities are derived from the image’s tags: version (full, e.g., 2.10.2), major-minor-version (e.g., 2.10), and major-version (e.g., 2). The full version is the OCI/Conforma org.opencontainers.image.version; the two coarser granularities are Hummingbird project labels used for tag aliasing. There is no io.hummingbird-project.version label — it would duplicate the OCI label.

⁴ Repository and stream

The io.hummingbird-project.repository and io.hummingbird-project.stream fields together form the release identity pair. Both are defined in properties.yml. See image-configuration-reference.md for field definitions.

The io.hummingbird-project.deprecated label is present with the value true on the final release of a deprecated stream. It is omitted for non-deprecated streams. The catalog uses the label to preserve the current deprecation state without changing older image digests.

⁵ Description, summary, and URL

The description, summary, and url fields are defined in properties.yml. Description and summary target different display contexts:

  • summary: One-liner (~40-80 chars) for table/list views
  • description: Short paragraph (~100-250 chars, 1-2 sentences) for card views and podman inspect

Style rules:

  • Do not start summary or description with the image name
  • Use >- YAML scalar for multi-line readability in properties.yml
  • Avoid embedded double quotes and backslashes (no escaping in templates)

⁶ Vendor

All images use Red Hat, Inc. as vendor (the distributing entity). The OCI org.opencontainers.image.vendor uses Red Hat (without “, Inc.”) per OCI convention. Both values are set on all distros.

⁷ Release

The release label uses the commit timestamp as a Unix epoch (matching Red Hat convention). In Konflux this is set via the generate-labels pipeline task.

⁸ Variant labels

Each variant name is decomposed into a base specialization and cross-cutting modifiers (builder, fips). The naming convention is <base>[-fips][-builder], where modifier order does not matter. A bare modifier like builder has base default.

Examples: fpm-builder → base=fpm, builder=yes; fips-builder → base=default, builder=yes, fips=yes.

The .description label stores only the base description (e.g., “PHP FastCGI process manager”). Modifier display is handled by consumers using the .builder and .fips boolean labels. Base descriptions come from variant_descriptions in images/variables.yml (for default) and image-specific properties.yml (for bases like fpm, runtime).

⁹ Containerfile path

The io.hummingbird-project.containerfile label contains the path to the Containerfile source relative to the repository root (e.g., images/caddy/hummingbird/default/Containerfile). A full URL to the source can be constructed from org.opencontainers.image.source + org.opencontainers.image.revision + this path.

Embedded Metadata (labels.json)

All labels are written to /usr/share/buildinfo/labels.json inside the container filesystem, providing filesystem-level access for security scanners. See Security Labels and Metadata for schema details.

File Purpose
images/<image>/properties.yml Per-image metadata fields
ci/internal/generate_jinja2.py Builds inject_labels dict from properties
macros/image_metadata_labels.yml.j2 Emits LABEL instructions from inject_labels
macros/inject_source_info_labels.yml.j2 Emits name and cpe LABEL instructions
macros/install_newroot.yml.j2 Invokes inject-source-info.sh with labels
images/hummingbird-builder/inject-source-info.sh Writes labels.json to container filesystem
ci/build_images.sh Adds build-time labels
ci/check_release_fields.py Validates label field values in properties.yml

4.3.6 - CI Scripts

This section documents the CI scripts used in the Project Hummingbird container build and test pipeline.

These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline.

4.3.6.1 - build_images.sh

Build container images using buildah with support for multiple architectures and container engines

Purpose

Build container images using buildah with support for multiple architectures and container engines.

Usage

Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...]

OPTIONS:
    --verbose, -v        Enable verbose output during build
    --arch ARCHITECTURE  Specify target architecture (e.g., amd64, arm64, arm/v7)
    --engine ENGINE      Specify runtime engine for testing/export (podman or docker)
    --setup              Set up Docker-in-Docker environment before exporting
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --local-rpms-dir DIR Directory containing custom RPMs to use during build.
                         These have a higher priority and thus override the
                         standard repositories.
    --build-deps         Build all dependencies of specified groups, but NOT the
                         specified groups themselves. Collects forward, reverse,
                         and transitive dependencies automatically.
    --pull               Pull images from registry instead of building locally.
                         Fast alternative for local development. Uses published
                         images from quay.io.
    --dryrun             Show what would be built (or pulled with --pull) without
                         actually building/pulling.
    --help, -h           Show this help message

Examples

# Build single image group (all distro/variants)
ci/build_images.sh nginx
ci/build_images.sh nodejs-20

# Build specific distro/variant only
ci/build_images.sh nginx/rawhide/builder
ci/build_images.sh nodejs-20/hummingbird/default

# Build multiple image groups at once
ci/build_images.sh nginx curl git
ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder

# Build with options
ci/build_images.sh --verbose curl
ci/build_images.sh --arch arm64 nginx
ci/build_images.sh --engine docker nginx
ci/build_images.sh --engine podman --verbose nginx

# Build multiple image groups with options
ci/build_images.sh --verbose nginx curl git
ci/build_images.sh --arch arm64 nginx postgresql

# Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies from registry instead of building (faster for local dev)
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Pull a specific image from registry
ci/build_images.sh --pull nginx/rawhide/default

# Build from CI component name format (used in CI environments)
ci/build_images.sh --component-name curl--rawhide--default

# Build with custom RPMs (for testing modified packages)
ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder

Note: Some images require git submodules initialized (use git init --recurse-submodules when cloning or git submodule update --init if already cloned). When building for foreign architectures, make sure qemu-user-static is available.

Dependency Building

The --build-deps flag builds all dependencies needed for testing an image, but not the image itself. This is used in CI to ensure all required images are available before running tests.

Pulling vs Building Dependencies

For local test development, use --pull with --build-deps to pull pre-built images from the registry instead of building them locally. This is much faster and uses the same dependency collection logic:

# Slow: Build all dependencies locally
ci/build_images.sh --build-deps nginx

# Fast: Pull all dependencies from registry
ci/build_images.sh --build-deps --pull nginx

The --pull flag works without --build-deps too:

# Pull a specific image instead of building it
ci/build_images.sh --pull caddy/rawhide/default

How Dependency Collection Works

When you run ci/build_images.sh --build-deps <image>, it performs a 2-level expansion:

  1. Level 1: Collects forward and reverse dependencies of the specified image

    • Forward dependencies: Images that the specified image’s tests depend on (detected by TEST_IMAGES[...] references in test files)
    • Reverse dependencies: Images that depend on the specified image (filtered by reverse_dependency_tests: true in properties.yml)

  2. Level 2: Collects forward dependencies of the reverse dependencies

  3. Stops: No further expansion (avoids infinite graph traversal)

All dependencies are automatically deduplicated to ensure each image is built exactly once.

Relationship to Testing

The reverse_dependency_tests property in properties.yml affects both building and testing:

  • Build phase (--build-deps): Filters which reverse dependencies to build
  • Test phase (ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps): Filters which reverse dependencies to test

Set reverse_dependency_tests: false for images like curl that are used pervasively but don’t need reverse dependency workflows.

Dependency Building Examples

# Build dependencies for dotnet-runtime-10-0
# Builds: dotnet-sdk-10-0 (forward dependency)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies for dotnet-runtime-10-0 (faster alternative)
# Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Build dependencies for core-runtime
# Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true)
# Plus their forward dependencies
ci/build_images.sh --build-deps core-runtime

# Use --dryrun to see what would be built/pulled without actually doing it
ci/build_images.sh --dryrun --build-deps core-runtime
ci/build_images.sh --dryrun --build-deps --pull core-runtime

Building with custom RPMs

The --local-rpms-dir option enables testing container images with custom-built RPM packages, for iterating on package changes before committing to the RPM repository.

Workflow

  1. Build custom RPMs in the rpms repository (see its documentation for the complete workflow):

    cd ../rpms
# modify a package
ci/build_rpms.sh packagename
# Built RPMs will be in builds/packagename/RPMS/

  2. Build container image using the custom RPMs:

    cd ../containers
ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder

  3. Verify the custom package was installed:

    podman run --rm --entrypoint '' quay.io/hummingbird/imagename:latest-builder rpm -qa

4.3.6.2 - run_tests_container.sh

Run tests for image groups using containerized test environment with support for both Docker and Podman engines

Purpose

Run tests for image groups using containerized test environment with support for both Docker and Podman engines.

Usage

Run ci/run_tests_container.sh --help for full usage information.

Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --hermetic           Use hermetic builds (--pull=never, only use prefetched/built images)
                         Default for CI. Without this, missing images are pulled on demand.
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Engine Selection: Use --engine to specify the container engine (podman or docker). When using Docker, add --setup for automatic Docker-in-Docker environment setup.

Hermetic vs Non-Hermetic Testing:

  • Local development (default): Without --hermetic, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first.
  • CI environment: Use --hermetic to enforce that tests only use prefetched or locally-built images (via --pull=never). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built.

Building and Testing

For local development, you’ll typically want to build the images first, then test them:

# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups

The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.

When working on base images (like core-runtime) that other images depend on, build the base image and its dependencies before running reverse-dependency tests:

# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime

The -p/--pause option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging.

Testing with Specific Image Builds

To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT> environment variables (uppercase, hyphens/dots → underscores, __ separates parts):

# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
  ci/run_tests_container.sh tomcat-10/hummingbird/builder

Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.

Retry Behavior:

  • Failed tests are checked against a list of retriable error patterns
  • If a match is found, the test is automatically retried
  • If the test still fails after all attempts, it’s reported as a normal failure

Examples

# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder

# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default

Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:

  • Find images that depend on it by scanning for TEST_IMAGES[group/distro/variant] references
  • Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting reverse_dependency_tests: false in properties.yml (e.g., curl, which is widely used but typically doesn’t need reverse dependency testing).

4.3.6.3 - run_tests_k8s.sh

Run K8s tests for image groups using kubectl with support for local development and CI environments

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
podman build -t my-nginx:dev images/nginx/hummingbird/default/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

The --push-image flag:

  • Compares local image digest with remote
  • Skips push if image already exists with same digest
  • Uses port-forward to registry-proxy for pushing
  • Sets TEST_IMAGE to the internal registry reference

Prerequisites:

  • oc login to the target cluster
  • oc project <namespace> to set the target namespace
  • Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=quay.io/hummingbird/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Environment Variables

Tests have access to these environment variables:

Variable Description
TEST_IMAGE Container image under test (with digest)
TEST_IMAGES Associative array with group/variant image URLs (uses TEST_DISTRO context)
TEST_IMAGES_PATH Path to file containing serialized TEST_IMAGES array
TEST_GROUP The image group being tested
TEST_DISTRO The distro being tested (e.g., rawhide)
TEST_VARIANT The variant being tested (e.g., default)
TEST_VERBOSE Show test command output (true or false)
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (hum-k8s-test=<id>)

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Function Description
test_fail Fail the test with a custom error message

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

# Build locally and push to internal registry
podman build -t my-nginx:dev images/nginx/hummingbird/default/
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

  • --kubeconfig pointing to the ephemeral namespace kubeconfig
  • --component-name or --group-component-name for component identification
  • --output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.

4.3.6.4 - retrigger_failed_checks.py

Retrigger failed Konflux CI checks for a given GitLab merge request

Purpose

Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running.

Usage

Run ci/retrigger_failed_checks.py --help for full usage information.

Usage: ci/retrigger_failed_checks.py <MR_URL> [OPTIONS]

OPTIONS:
    -h, --help              show this help message and exit
    --dry-run               just print what comments would be posted
    --token-path, -t PATH   path to file containing GitLab API token

Authentication (in order of precedence):
  1. --token-path: Path to file containing GitLab API token (highest priority)
  2. GITLAB_TOKEN_PATH: Environment variable with path to token file
  3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback)

Behavior

The script will:

  1. Extract failed pipeline runs from the commit statuses on the MR
  2. Generate /retest {pipeline-name} commands for each failed run
  3. Post the retest commands as MR comments (unless –dry-run)
  4. Wait for the pipeline runs to start (unless –dry-run)

Examples

# Using --token-path option (highest priority)
ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN_PATH environment variable
GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN environment variable (legacy)
GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Dry-run mode (show what would be done, don't post or wait)
ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234

4.3.6.5 - gitlab_sync.py

Sync a generated file to an external GitLab repository via merge request, with automatic merge wait, pipeline retry, and preemption support

Purpose

Sync a file from a public source URL to a target GitLab repository via merge request. The script always waits for the MR to be merged and for the post-merge pipeline to complete before returning. It is invoked by infrastructure CI jobs to keep external repos (pyxis-repo-configs, konflux-release-data) in sync with generated files from the source repos.

Usage

Run gitlab_sync.py --help for full usage information.

usage: gitlab_sync.py [-h] --source-url SOURCE_URL
                      --target-project TARGET_PROJECT
                      --target-file TARGET_FILE --sync-branch SYNC_BRANCH
                      --mr-title MR_TITLE --gitlab-url GITLAB_URL
                      [--merge-timeout MERGE_TIMEOUT]
                      [--post-merge-timeout POST_MERGE_TIMEOUT]
                      [--mr-description MR_DESCRIPTION] [--squash-on-merge]
                      [--dry-run]

Authentication:
  GITLAB_TOKEN    Environment variable with GitLab API token (required)

Monitoring:
  SENTRY_DSN      Sentry DSN for alerting on failures (optional)

How It Works

  1. Fetch source from the public --source-url (with HTTP retries)
  2. Determine comparison ref: use the sync branch if it exists, otherwise the project’s default branch
  3. Compare source content against the comparison ref (stripped whitespace)
  4. Early exit if content matches and the comparison ref is the default branch (nothing to deploy)
  5. Commit the source to the sync branch using the GitLab Commits API with force: true (creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits.
  6. Create or update MR targeting the default branch
  7. Self-approve the MR (best-effort; continues if already approved)
  8. Post takeover comment with CI_JOB_ID for preemption tracking
  9. Wait for MR merge (polling every 30s, up to --merge-timeout)
  10. Wait for post-merge pipeline to succeed (up to --post-merge-timeout)

Exit Codes

Code Meaning
0 Success (MR merged) or no changes needed
1 Failure (timeout, pipeline failure, unrecoverable error)
42 Preempted by a newer CI job managing the same MR

Preemption

When multiple CI jobs target the same sync branch, the script uses MR comments to coordinate. Each job posts a comment containing its CI_JOB_ID (monotonically increasing within a GitLab instance). Before each merge attempt, the script checks for comments with a higher job ID. If found, it yields by exiting with code 42.

The caller should handle exit 42 to stop processing (a newer job will handle all remaining syncs). When running outside CI (CI_JOB_ID not set), preemption is disabled.

Error Recovery

  • Pipeline failure: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted.
  • Merge conflict: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert.
  • Post-merge pipeline failure: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA).
  • Draft MR: Keeps polling without attempting to merge (allows manual intervention).

Examples

# Dry run: show diff without modifying anything
GITLAB_TOKEN=$TOKEN gitlab_sync.py \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --dry-run

# Sync Pyxis config (requires squash merge)
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --squash-on-merge

# Sync RPM RPA to konflux-release-data
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \
  --target-project "releng/konflux-release-data" \
  --target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \
  --sync-branch "hummingbird/sync-rpms-rpa" \
  --mr-title "Update hummingbird RPM ReleasePlanAdmission"

Preemption wrapper for sequential syncs in a CI job:

gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge
rc=$?
if [ "$rc" -eq 42 ]; then
  echo "Preempted by newer job, stopping"
  exit 0
elif [ "$rc" -ne 0 ]; then
  exit "$rc"
fi
# Proceed to next sync only on success
gitlab_sync.py --gitlab-url ... --source-url ...

Environment Variables

Variable Required Description
GITLAB_TOKEN Yes API token with Developer+ access on the target project
CI_JOB_ID No Set automatically in CI; enables preemption detection
SENTRY_DSN No Sentry DSN for failure alerting

Development

The script and gitlab-ci tool image are maintained in the Hummingbird tools repository, not this repository. Make implementation and test changes there.

4.4 - K8s Test Pipeline

Background documentation sourced from the K8s test pipeline repository, covering pipeline design, test format, and EaaS debugging for Kubernetes integration tests.

4.4.1 - Pipeline Design

Design guidelines and architecture of the K8s test pipeline.

Task/Step Overview

Task Type Steps Results Retries Timeout
check-for-tests Inline taskSpec write-snapshot, resolve-component, fetch-source, check-for-tests, write-results HAS_TESTS, TEST_OUTPUT 2 10m
provision-namespace External bundle (task-eaas-provision-space) (managed by task) secretRef 2 10m
run-tests Inline taskSpec write-snapshot, resolve-component, fetch-source, run-tests, write-results TEST_OUTPUT 2 24h
fail-pipeline-on-test-failure Inline taskSpec (finally) check-results — 0 10m

Both check-for-tests and run-tests independently fetch source — each task gets its own emptyDir volume, so no state carries across tasks.

Design Guidelines

1. Filesystem over results

Pass data between steps via files on a shared emptyDir, not Tekton results. Tekton results have a 4KB termination message limit (tektoncd/pipeline#4060) and large $(params.*) substitutions hit ARG_MAX.

Each task mounts an emptyDir at /workdir. Steps communicate via files: snapshot.json, component, image_names, src/, kubeconfig, result, error. Zero $(params.*) references remain inside script: blocks — all param values are passed via env vars or written to /workdir.

2. Distinguish retryable from non-retryable errors

Use exit 1 for transient failures (network, registry) that benefit from retry. Use exit 0 + a sentinel file for permanent failures (bad data) to prevent wasted retries.

Non-retryable errors write Konflux-format JSON to /workdir/error and exit 0. Subsequent steps check for this sentinel and skip. The write-results step copies it to TEST_OUTPUT. Tasks default to retries: 2, but fail-pipeline-on-test-failure has retries: 0 — test failures should not be retried.

3. Fail fast

Every shell script enables set -euo pipefail so unexpected command failures surface immediately rather than cascading silently.

4. Separate data from status

Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration.

Konflux reads TEST_OUTPUT results as data but does not fail the pipeline based on them. The fail-pipeline-on-test-failure finally-task reads both TEST_OUTPUT values and exit 1 on non-SUCCESS. This is needed because GitLab PAC integration requires the pipeline itself to fail for MR status reporting.

5. Gate expensive work behind cheap checks

Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do.

provision-namespace and run-tests are gated on check-for-tests.results.HAS_TESTS == "true" via Tekton when expressions. If no tests-k8s.yml exists, the pipeline completes with just check-for-tests, skipping EaaS provisioning. This is the common case for components without K8s tests.

6. Make failures reproducible

Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally.

Stderr from cosign/jq/oras is not swallowed. The full run_tests_k8s.sh command line is logged.

7. Set task timeouts deliberately

Tekton applies a global default TaskRun timeout (typically 1h from default-timeout-minutes in config-defaults) to every task that has no explicit timeout, regardless of pipeline-level timeouts. This means omitting a task timeout does NOT let the pipeline timeout control the bound — the task gets killed at 1h.

Every task in this pipeline has an explicit timeout. Bounded operations (check-for-tests, provision-namespace, fail-pipeline-on-test-failure) use 10m. The variable-duration run-tests task uses 24h so the pipeline-level timeout (controlled per-application via ITS annotations like test.appstudio.openshift.io/pipeline_timeout) is the effective bound, following the Testing Farm pipeline pattern.

8. Define shared logic once

When multiple tasks need identical steps, use YAML anchors to define them once and reuse via aliases. Tekton pipelines have no function/import mechanism, so anchors are the only DRY tool available within a single pipeline file. Without this, step logic drifts between tasks when one copy is updated but not the other.

write-snapshot, resolve-component, and fetch-source are defined as anchors (&write-snapshot-step, etc.) in check-for-tests and reused via *write-snapshot-step in run-tests. Both tasks independently execute the same steps to fetch and process source, staying in sync.

Trusted Artifacts Source Fetch

Source code is fetched via the Konflux Trusted Artifacts chain:

  1. cosign download attestation — retrieves build attestations (handles multiple JSONL attestations)
  2. Extract SOURCE_ARTIFACT from .predicate.buildConfig.tasks[].results[]
  3. oras blob fetch — downloads the source tarball
  4. tar -xzf — extracts to /workdir/src

The oci: prefix is stripped from the artifact URI. --no-same-owner avoids permission issues during extraction.

4.4.2 - Test Format

Pipeline-level test discovery, environment variables, parameters, and group snapshot handling.

Test Discovery

The pipeline locates tests-k8s.yml using source.git.context from the Konflux Snapshot. For example, if source.git.context is images/curl, the pipeline looks for images/curl/tests-k8s.yml in the source tree.

When source.git.context is not populated (currently the case for the triggering component in PR-triggered pipelines, tracked as KONFLUX-12674), a heuristic fallback derives the context from the component name: it tries images/<base> and then <base>.

The actual tests-k8s.yml format is opaque to the pipeline — it is passed to the repository’s test runner script, which defines the structure and semantics. See the respective repository documentation for the YAML format.

Environment Variables

The test runner receives these variables from the pipeline:

Variable Description
TEST_IMAGE Container image under test (with digest)
TEST_GROUP Image group name (e.g., git, postgresql)
TEST_VARIANT Variant being tested (e.g., default, builder)
KUBECONFIG Path to kubeconfig for the ephemeral namespace
TEST_IMAGES Bash associative array of all images in the snapshot
TEST_IMAGES_PATH Path to source TEST_IMAGES in external scripts
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (e.g., hum-k8s-test=<id>)
test_fail Shell function — call test_fail "message" to fail a test

Pipeline Parameters

Parameter Description Default
SNAPSHOT Konflux snapshot JSON or snapshot name (with snapshot-param-as-name annotation) —
EXTRA_SINGLE_COMPONENT_ARGS Additional arguments appended to the test runner for single-component (PR) snapshots --include-reverse-deps

Group Snapshot Handling

For group snapshots (post-merge), the pipeline exports:

  • IMAGE_NAMES — space-separated list of all component names
  • IMAGE_URL_<COMPONENT> — image URL for each component (name uppercased, dashes replaced with underscores)

The test runner script is called once with multiple --group-component-name arguments. Test discovery checks all components for tests-k8s.yml. Source is fetched from the first component’s build attestation (shared source tree).

4.4.3 - EaaS and Debugging

How the pipeline uses Konflux Environment as a Service, and how to debug test failures.

Environment as a Service (EaaS)

EaaS is a Konflux-managed service that provisions ephemeral OpenShift namespaces on member clusters for integration testing.

Provisioning Flow

  1. The provision-namespace task creates an eaas.konflux-ci.dev/v1alpha1 Namespace claim, owned by the PipelineRun.
  2. The EaaS controller allocates a namespace on a member cluster (currently kflux-prd-es01) and writes a scoped kubeconfig into a Kubernetes Secret.
  3. The run-tests step reads the kubeconfig from the Secret to get cluster access.
  4. When the PipelineRun is deleted, the owner reference triggers garbage collection of the claim, namespace, and Secret.

The member cluster may change — it is dynamically assigned by the EaaS controller.

Debugging Test Failures

Using Kubearchive for Historical PLRs

Kubearchive stores historical PipelineRun, TaskRun, and Pod data. The containers repo includes a helper script at ci/internal/k8s_helper.py.

Example: find PLRs for a specific PR:

cd /path/to/rprm-containers
python3 -c "
from ci.internal import k8s_helper
h = k8s_helper.K8sHelper('https://konflux-ui.apps.kflux-prd-rh03.nnv1.p1.openshiftapps.com/ns/hummingbird-tenant/')
endpoint = h._tekton_endpoint('pipelineruns')
items = h._fetch_from_both_sources(endpoint, 'pac.test.appstudio.openshift.io/pull-request=421,test.appstudio.openshift.io/scenario=tools-k8s-test')
for plr in items:
    print(plr['metadata']['name'], plr['status']['conditions'][0]['reason'])
"

Useful label selectors:

  • test.appstudio.openshift.io/scenario=<scenario-name> — filter by ITS name
  • pac.test.appstudio.openshift.io/pull-request=<number> — filter by MR
  • appstudio.openshift.io/component=<component> — filter by component

Accessing the EaaS Namespace During a Live PLR

While a PipelineRun is running, you can access the ephemeral namespace:

  1. Find the provision secret name from the provision-namespace task results (visible in the PLR status or Konflux UI).

  2. Extract the kubeconfig:

    kubectl get secret <secret-name> -n hummingbird-tenant \
  -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/eaas-kubeconfig

  3. Use it to inspect the namespace:

    export KUBECONFIG=/tmp/eaas-kubeconfig
kubectl get pods
kubectl get events --sort-by='.lastTimestamp'
kubectl logs <pod-name>

Finding Which Cluster EaaS Uses

To determine the current EaaS member cluster, extract the server URL from a kubeconfig provisioned by EaaS:

kubectl get secret <secret-name> -n hummingbird-tenant \
  -o jsonpath='{.data.kubeconfig}' | base64 -d | grep server

As of this writing, the EaaS member cluster is kflux-prd-es01.1ion.p1.openshiftapps.com. This may change as the EaaS controller dynamically assigns clusters.

4.5 - Tools repository

Background documentation sourced from the tools repository, covering infrastructure components like event forwarders, the message bus, and monitoring tools.

4.5.1 - Message Bus Architecture

The Hummingbird message bus is an event-driven architecture built on AWS SNS. Events from multiple sources flow through a central topic, enabling subscribers to filter and process only the events they need.

Architecture

flowchart TD
    subgraph Publishers
        GL[GitLab Webhooks]
        K8S[Kubernetes Clusters]
    end

    subgraph MessageBus [Message Bus]
        SNS[(SNS Topic)]
    end

    subgraph Archiver
        ARCH[SNS S3 Archiver]
        S3[(S3 Bucket)]
    end

    subgraph StatusDB [Status Database]
        SQS1[SQS Queue]
        WORKER[hummingbird-status]
        PG[(PostgreSQL)]
    end

    subgraph ConsumerQueue [Consumer Queue]
        SQS2[SQS Queue]
    end

    subgraph Catalog [Container Catalog]
        SYNC[SyncFunction]
        DDB[(DynamoDB)]
    end

    subgraph Consumers
        CONSUMER[Event Consumer]
    end

    GL -->|gitlab-event-forwarder| SNS
    K8S -->|kubernetes-event-forwarder| SNS
    SNS --> ARCH
    ARCH --> S3
    SNS --> SQS1
    SQS1 --> WORKER
    WORKER --> PG
    SNS -->|"FilterPolicy\nkind=Release"| SYNC
    SYNC -->|fetch manifests| Registry[(OCI Registry)]
    SYNC -->|read/write| DDB
    SNS --> SQS2
    SQS2 -->|new events| CONSUMER
    S3 -.->|historic events| CONSUMER
    PG -.->|aggregated status| CONSUMER

Components

Component Role Description Staging
hummingbird-events-topic Infrastructure Central SNS topic for all events Yes
gitlab-event-forwarder Publisher Receives GitLab webhooks, publishes to SNS Yes
kubernetes-event-forwarder Publisher Watches K8s resources, publishes changes to SNS Yes
sns-s3-archiver Subscriber Archives all events to S3 for querying/replay Yes
hummingbird-status Subscriber Ingests events to PostgreSQL for structured queries Yes
hummingbird-agent Subscriber Processes events to drive automated workflows Yes
workqueue-service Subscriber Queues and dispatches work items from events Yes
container-catalog Subscriber Incrementally syncs image metadata to DynamoDB on Release events No

Message Format

All messages include standard attributes for filtering:

Attribute Description Examples
source Event origin gitlab, kubernetes
event_type Type of event push, merge_request, ADDED, MODIFIED

Additional attributes vary by source - see individual publisher docs for details.

GitLab Events

Published by gitlab-event-forwarder:

Attribute Description Example
project_path Full project path redhat/hummingbird/containers
group_path Full group path redhat/hummingbird

Kubernetes Events

Published by kubernetes-event-forwarder:

Attribute Description Example
cluster API server URL https://api.cluster:6443
namespace Object namespace production
kind Resource kind Pod, Deployment
api_version API version v1, apps/v1
object_name Resource name nginx-7d8c4c9d6f

Subscription Filtering

SNS filter policies enable subscribers to receive only relevant events:

{
  "source": ["gitlab"],
  "event_type": ["push", "merge_request"]
}

{
  "source": ["kubernetes"],
  "kind": ["Deployment"],
  "event_type": ["MODIFIED"]
}

Kubernetes Release events (used by container-catalog sync Lambda):

{
  "kind": ["Release"]
}

See hummingbird-events-topic for subscription setup instructions.

Event Flow Example

  1. Developer pushes to GitLab repository
  2. GitLab sends webhook to gitlab-event-forwarder Lambda
  3. Lambda validates token, extracts metadata, publishes to SNS
  4. SNS delivers to all matching subscribers:
    • sns-s3-archiver stores event in S3
    • Event consumers process new events in real-time

Consuming Events

Consumers can receive events from multiple sources:

  • New events: Subscribe to SNS topic for real-time processing
  • Historic events: Download from S3 archive for replay or catch-up
  • Structured queries: Query PostgreSQL via hummingbird-status for pipeline status, component state, and cross-referenced data

The S3 archive enables consumers to bootstrap state, then switch to live SNS events. The PostgreSQL database provides a queryable view of pipeline status with relationships between pushes, builds, snapshots, and releases.

Replaying Events

The sns-s3-archiver stores complete SNS records with decoded payloads, enabling easy replay:

# Download archived events
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/

# Browse events
zcat ./local-events/2025/12/13/12/34/*.json.gz | jq

import gzip
import json
from pathlib import Path

# Replay archived events to handler
for path in sorted(Path("./local-events").rglob("*.json.gz")):
    with gzip.open(path, "rt") as f:
        sns_record = json.load(f)
    event = {"Records": [{"Sns": sns_record}]}
    my_handler.lambda_handler(event, None)

See sns-s3-archiver documentation for details on storage format and the _decode_message pattern for handlers.

Staging Environment

Production and staging use independent SNS topics. Each environment has its own publishers, archiver, and consumer queues so that changes can be tested in staging before production deployment.

flowchart TD
    GL["GitLab Projects\n(dual webhook delivery)"]
    K8S["Konflux Cluster\n(K8s resources)"]

    subgraph production ["Production"]
        GEF_P["gitlab-event-forwarder\n(Lambda)"]
        KEF_P["kubernetes-event-forwarder\n(K8s pod)"]
        SNS_P["SNS: arr-hummingbird-prod-events"]
        prodConsumers["consumers\n(status, agent, workqueue,\narchiver, etc.)"]

        GEF_P --> SNS_P
        KEF_P --> SNS_P
        SNS_P --> prodConsumers
    end

    subgraph staging ["Staging"]
        GEF_S["gitlab-event-forwarder\n(Lambda)"]
        KEF_S["kubernetes-event-forwarder\n(K8s pod)"]
        SNS_S["SNS: arr-hummingbird-staging-events"]
        stagConsumers["consumers\n(status, agent, workqueue,\narchiver)"]

        GEF_S --> SNS_S
        KEF_S --> SNS_S
        SNS_S --> stagConsumers
    end

    GL --> GEF_P
    GL --> GEF_S
    K8S --> KEF_P
    K8S --> KEF_S

How it works

  • Duplicate webhook delivery: GitLab sends every event to both gitlab-event-forwarder.hummingbird-project.io (prod) and gitlab-event-forwarder.staging.hummingbird-project.io (staging). Each forwarder publishes to its own SNS topic.
  • Dual Kubernetes watches: Both prod and staging kubernetes-event-forwarder pods watch the same Konflux cluster resources and publish to their respective topics.
  • Consumer isolation: Staging consumers (hummingbird-status, hummingbird-agent, workqueue-service) have separate SQS queues subscribed to the staging topic, with dedicated Postgres databases.
  • Staging archiver: A separate sns-s3-archiver subscribes to the staging topic and archives to its own S3 bucket with a shorter retention period than production.

4.5.2 - Alloy CloudWatch

An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana Alloy’s prometheus.exporter.cloudwatch component. Grants access to CloudWatch Metrics APIs so selected AWS service metrics (Lambda, SQS, SNS, DynamoDB) can be ingested into Mimir for alerting.

Features

  • Read-Only IAM User: Dedicated user for Alloy CloudWatch metrics exporter
  • CloudWatch Metrics: GetMetricData, GetMetricStatistics, ListMetrics
  • Resource Discovery: tag:GetResources for filtering by app-code=RPRM-001
  • Account Discovery: sts:GetCallerIdentity and ec2:DescribeRegions for YACE auto-discovery

Architecture

Single SAM template that creates:

  1. IAM User - Read-only service account (${ResourcePrefix}-user)
  2. IAM Policy - CloudWatch Metrics read, tag and region discovery, STS caller identity

The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Alloy Hub pod on MPP via a Kubernetes Secret.

flowchart LR
  subgraph aws [AWS]
    iam["IAM User\n(read-only)"]
    cwMetrics["CloudWatch Metrics"]
  end
  subgraph mpp [MPP Hub]
    alloy["Alloy Hub"]
    mimir["Mimir"]
  end
  alloy -->|"GetMetricData"| cwMetrics
  alloy -->|"remote_write"| mimir
  alloy -.->|"credentials from Vault"| iam

IAM Permissions

All permissions are read-only with Resource: "*" (CloudWatch does not support resource-level restrictions for most read operations).

Category Actions
Metrics GetMetricData, GetMetricStatistics, ListMetrics
Discovery ec2:DescribeRegions, tag:GetResources, sts:GetCallerIdentity

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd alloy-cloudwatch
sam deploy --guided   # First deployment
sam deploy            # Subsequent deployments

After deployment, create an access key for the IAM user and store it in Vault using cki_tools.credentials.manager.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.3 - Deployment Bot

An AWS CloudFormation/SAM stack that provisions IAM resources for automated AWS deployments from GitLab CI. Creates a service account with permissions to deploy SAM applications including Lambda functions, API Gateway, SNS topics, and related resources.

Features

  • IAM Service Account: Dedicated user for CI/CD pipelines
  • Scoped Permissions: Least-privilege access for SAM deployments
  • Resource Naming: All managed resources use consistent prefix

Architecture

Single SAM template that creates:

  1. IAM User - Service account for deployments (${ResourcePrefix}-deployment-bot)
  2. IAM Policy - Permissions for SAM deployment operations

Prerequisites

  • AWS CLI configured with admin credentials
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Deploy using containerized AWS SAM CLI:

cd deployment-bot
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely in HashiCorp Vault.

SAM Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
AdditionalResourcePrefix Additional prefix this bot can manage (e.g. staging) ""

Resource naming: IAM resources follow {ResourcePrefix}-deployment-bot pattern (e.g., myapp-prod-deployment-bot). Use AdditionalResourcePrefix to allow the same bot to manage resources in another environment.

Permissions

The deployment bot has permissions to manage:

Service Scope
CloudFormation Stacks matching ${ResourcePrefix}-*, SAM bootstrap stack
S3 SAM CLI managed buckets
Lambda Functions matching ${ResourcePrefix}-*
API Gateway All REST APIs and domains
SNS Topics matching ${ResourcePrefix}-*
IAM Users/policies/roles matching ${ResourcePrefix}-*
CloudWatch Log groups for managed Lambda functions
Route53 Record changes (for custom domains)
ACM Certificate management

Usage

Bootstrap (One-Time)

The deployment bot must be deployed first using admin credentials:

# With admin AWS credentials
cd deployment-bot
sam build
sam deploy --guided

Create access keys for the new IAM user and store in HashiCorp Vault.

Automated Deployments

Once bootstrapped, the deployment bot credentials are used by aws_deploy.sh in the infrastructure repository for all subsequent SAM deployments:

export PROJECT_NAME=gitlab-event-forwarder
./aws_deploy.sh

GitLab CI runs this automatically via the aws matrix job.

Development

This project contains only a SAM template with no application code.

Security

  • Access keys should be stored in HashiCorp Vault
  • Permissions are scoped to resources with the configured prefix
  • Route53 and ACM have broader permissions due to API limitations

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.4 - Grafana CloudWatch

An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana’s CloudWatch datasource plugin. Grants access to CloudWatch Logs and Metrics APIs so Lambda execution logs and AWS service metrics are browsable directly in Grafana.

Features

  • Read-Only IAM User: Dedicated user for Grafana CloudWatch datasource
  • CloudWatch Logs: Browse Lambda and AWS service log groups via Logs Insights
  • CloudWatch Metrics: Browse metrics across all AWS namespaces (Lambda, SQS, SNS, DynamoDB, etc.)
  • Resource Discovery: tag:GetResources and ec2:DescribeRegions for Grafana plugin autodiscovery

Architecture

Single SAM template that creates:

  1. IAM User - Read-only service account (${ResourcePrefix}-user)
  2. IAM Policy - CloudWatch Logs read, Metrics read, tag and region discovery

The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Grafana pod on MPP via a Kubernetes Secret.

flowchart LR
  subgraph aws [AWS]
    iam["IAM User\n(read-only)"]
    cwLogs["CloudWatch Logs"]
    cwMetrics["CloudWatch Metrics"]
  end
  subgraph mpp [MPP Hub]
    grafana["Grafana\n(CloudWatch datasource)"]
  end
  grafana -->|"Logs Insights API"| cwLogs
  grafana -->|"Metrics API"| cwMetrics
  grafana -.->|"credentials from Vault"| iam

IAM Permissions

All permissions are read-only with Resource: "*" (CloudWatch does not support resource-level restrictions for most read operations).

Category Actions
Logs DescribeLogGroups, DescribeLogStreams, GetLogEvents, FilterLogEvents, StartQuery, StopQuery, GetQueryResults, DescribeQueries
Metrics GetMetricData, GetMetricStatistics, ListMetrics, DescribeAlarms, DescribeAlarmsForMetric
Discovery ec2:DescribeRegions, tag:GetResources

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd grafana-cloudwatch
sam deploy --guided   # First deployment
sam deploy            # Subsequent deployments

After deployment, create an access key for the IAM user and store it in Vault using cki_tools.credentials.manager.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.5 - Hummingbird Events Topic

An SNS topic for all Hummingbird project events. This topic serves as a central hub for distributing events from multiple sources (GitLab webhooks, Kubernetes, etc.) to multiple subscribers, enabling event-driven architectures and integrations.

Features

  • Central Event Hub: Dedicated SNS topic per environment (production and staging) for all project events from multiple sources
  • Multiple Subscribers: Supports Lambda, SQS, HTTP endpoints, email, SMS, and more
  • Event Filtering: Subscribers can filter events using SNS subscription filter policies

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for SNS, CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-events-topic
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs:

  • TopicArn - SNS topic ARN (for event publishers)
  • TopicName - SNS topic name

Parameters

Parameter Description Default
TopicName SNS topic name myapp-prod-events

Resource naming: The SNS topic uses the provided TopicName parameter.

Usage

Publishing Events

Event publishers need the topic ARN to publish messages:

# Get topic ARN from CloudFormation stack
aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --query 'Stacks[0].Outputs[?OutputKey==`TopicArn`].OutputValue' \
  --output text

Event publishers:

Subscribing to Events

Subscribe services to receive events:

Via AWS Console:

  1. Open SNS console → Topics
  2. Select the topic
  3. Create subscription (choose protocol: Lambda, SQS, HTTP, Email, etc.)
  4. Add subscription filter policy (optional)

Via AWS CLI:

aws sns subscribe \
  --topic-arn <topic-arn> \
  --protocol lambda \
  --notification-endpoint <lambda-arn>

Add filter policy:

aws sns set-subscription-attributes \
  --subscription-arn <subscription-arn> \
  --attribute-name FilterPolicy \
  --attribute-value '{"source": ["gitlab"], "event_type": ["push"]}'

Subscription Filter Examples

GitLab push events from specific project:

{
  "source": ["gitlab"],
  "event_type": ["push"],
  "project_path": ["redhat/hummingbird/containers"]
}

All merge request events:

{
  "source": ["gitlab"],
  "event_type": ["merge_request"]
}

All events from GitLab:

{
  "source": ["gitlab"]
}

Event metadata: See publisher documentation for available metadata:

Development

This is a pure infrastructure project (no application code). See the main README for SAM build/deploy commands.

Security & Limitations

Security:

  • SNS topic follows least privilege principle
  • Access controlled via IAM policies
  • Supports server-side encryption (optional)
  • Publishers require sns:Publish permission
  • Subscribers require appropriate protocol permissions

Limitations:

  • Message size: Up to 256 KB
  • Maximum subscriptions: 12,500,000 per topic
  • Message retention: Not supported (use SQS for durable queuing)
  • Delivery retries: Protocol-dependent

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.6 - Vertex AI Cost Metrics

This document defines the shared cost-observability contract for all Hummingbird services that call Vertex AI. It covers two obligations:

  1. Prometheus metrics — estimated cost counters scraped into Mimir.
  2. Vertex request labels — billing metadata attached to every API call.

All services that make Vertex AI calls MUST implement both.

1. Prometheus metrics (estimated cost)

Every Vertex AI caller MUST emit the following counters on its scraped /metrics endpoint. Using identical names and label sets across services enables a single Grafana dashboard without per-app query logic.

Counters

Name Unit Labels Description
hummingbird_vertex_cost_dollars_total USD model, workflow Estimated cost based on MODEL_PRICING x tokens
hummingbird_vertex_tokens_total tokens model, workflow, direction Token counts per API call
hummingbird_vertex_requests_total requests model, workflow, status API call attempts

Labels

Applications MUST set these labels. Additional labels are permitted if cardinality remains bounded.

Label Values
model Model name as sent to Vertex (e.g. gemini-2.5-flash, claude-sonnet-4-6)
workflow Workflow or operation name (e.g. code-review, renovate-babysit)
direction input, output, cache_read
status success, error, retry

Applications MUST NOT set cluster, namespace, app, service, or pod — Alloy adds these automatically at scrape time via relabel rules.

Cardinality

  • model: bounded by deployed model set (currently 2-4)
  • workflow: bounded by workflow config entries (~10 for agent, 1 for dashboard)
  • direction: fixed set of 3
  • status: fixed set of 3

Total series per app: model x workflow x max(direction, status) = 4 x 10 x 3 = 120. Well within Mimir per-metric limits.

2. Vertex request labels (billed cost)

Every Vertex API call MUST include billing labels so that GCP Cloud Billing rows can be attributed to a specific application and workflow once BigQuery export is enabled.

Required labels

Key Value Example
app Application name (matches Kubernetes app) agent, dashboard
workflow Workflow or operation name within the app code-review, analyze-failures

Callers MUST set both keys on every generateContent, streamGenerateContent, or rawPredict request.

How to attach

Endpoint Mechanism
generateContent (Gemini) Top-level "labels" key in JSON body
rawPredict / streamRawPredict X-Vertex-AI-Labels HTTP header

For generateContent, add the labels object to the request body:

{
  "contents": [...],
  "labels": {"app": "agent", "workflow": "code-review"}
}

For rawPredict, base64-encode the labels JSON and pass as a header (shown decoded for clarity):

import base64
import json

labels = {"app": "dashboard", "workflow": "analyze-failures"}
headers["X-Vertex-AI-Labels"] = base64.b64encode(
    json.dumps(labels).encode(),
).decode()

Constraints (GCP requirements)

  • Keys and values: lowercase letters, numbers, underscores, dashes only
  • Max 63 characters each
  • Keys MUST start with a letter
  • Labels are only forwarded for PayGo consumption (Provisioned Throughput silently ignores them)

Reference

Example PromQL

Total estimated spend yesterday (all apps):

sum(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-app daily cost:

sum by (app)(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-workflow breakdown for the agent:

sum by (workflow, model)(
  increase(hummingbird_vertex_cost_dollars_total{app="hummingbird-agent"}[1d])
)

Token consumption by direction:

sum by (app, direction)(increase(hummingbird_vertex_tokens_total[1d]))

Implementing apps

Application Port Workflow values
hummingbird-agent 9090 from wf_cfg.name per workflow
hummingbird-dashboard 8080 analyze-failures

4.5.7 - SNS S3 Archiver

A Lambda function that archives all messages from the Hummingbird Events SNS topic to S3. Events are stored as clean JSON with decoded payloads, enabling easy browsing with zcat | jq and replay to handlers.

Features

  • S3 Storage: Events stored with time-hierarchical key structure for efficient time-range queries
  • Clean JSON: Decodes gzip+base64 Message payload for human-readable storage
  • Replay Support: Preserves all MessageAttributes with updated content_encoding for replay fidelity
  • Easy Access: Download via aws s3 sync, browse with zcat | jq

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, S3, SNS, CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd sns-s3-archiver
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs:

  • BucketName - S3 bucket name for archived events
  • BucketArn - S3 bucket ARN
  • FunctionArn - Lambda function ARN

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
SnsTopicArn ARN of the SNS topic to subscribe to (required)
SentryDsn Optional Sentry DSN for error tracking ""
RetentionDays Days to retain events (0=indefinite) 0

Resource naming: Bucket and Lambda names are derived from ResourcePrefix:

  • S3 bucket: {ResourcePrefix}-s3-events
  • Lambda function: {ResourcePrefix}-lambda-s3-archiver

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

S3 Key Structure

Events are stored with a time-hierarchical key structure optimized for time-range queries:

sns/YYYY/MM/DD/HH/MM/TIMESTAMP#MSGID::source::kind::name.json.gz

Examples:

sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#abc12345::kubernetes::Snapshot::my-snapshot.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#def67890::gitlab::push::abc123def456.json.gz

The sns/ prefix separates the new format from legacy data, enabling incremental migration. The :: delimiter separates metadata fields (avoiding conflicts with K8s names that contain --). The resource name in the key enables efficient “latest state per resource” queries without downloading all files.

The minute-level subdirectory (/MM/) provides write distribution for high-throughput scenarios.

Storage Format

Each S3 object contains a gzip-compressed JSON record:

{
  "MessageId": "abc12345-1234-5678-9abc-def012345678",
  "Timestamp": "2025-12-13T12:34:56.789Z",
  "Message": {
    "apiVersion": "appstudio.redhat.com/v1alpha1",
    "kind": "Snapshot",
    "metadata": { ... },
    "spec": { ... }
  },
  "MessageAttributes": {
    "source": { "Type": "String", "Value": "kubernetes" },
    "kind": { "Type": "String", "Value": "Snapshot" },
    "content_encoding": { "Type": "String", "Value": "json" }
  }
}

Key points:

  • Message is a decoded JSON object (not the original base64+gzip string from SNS)
  • content_encoding is set to "json" to indicate the decoded format

Usage

Download Events Locally

# Sync entire bucket
aws s3 sync s3://bucket-name/ ./local-events/

# Sync specific time range
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/2025/12/13/

# Sync with filtering
aws s3 sync s3://bucket-name/ ./local-events/ --exclude "*" --include "*kubernetes*Snapshot*"

Browse Events

# List events for a specific hour
aws s3 ls s3://bucket-name/2025/12/13/12/ --recursive

# View a single event
aws s3 cp s3://bucket-name/2025/12/13/12/34/event.json.gz - | zcat | jq

# Local browsing
zcat ./local-events/2025/12/13/12/34/event.json.gz | jq

Replay Events

Consumer handlers should support both live SNS format and archived format:

def _decode_message(sns_record: dict) -> dict:
    """Decode Message - handles live SNS and archived formats."""
    message = sns_record.get("Message", "")
    encoding = sns_record.get("MessageAttributes", {}).get("content_encoding", {}).get("Value")

    # Archived format: Message is already decoded
    if encoding == "json" or isinstance(message, dict):
        return message if isinstance(message, dict) else json.loads(message)

    # Live SNS format: Message is gzip+base64
    if encoding == "gzip+base64":
        return json.loads(gzip.decompress(base64.b64decode(message)))

    # Fallback: plain JSON string
    return json.loads(message)

Compression Flow

Forwarder → SNS → Archiver → S3

1. Forwarder: gzip(payload) → base64 → SNS Message
2. SNS: Passes through with content_encoding="gzip+base64"
3. Archiver: Decodes Message, sets content_encoding="json", gzip(record) → S3
4. S3: Stores clean JSON record, gzip-compressed at object level

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

S3 Lifecycle Policies

The bucket has two lifecycle rules:

  • AbortIncompleteUploads: Automatically cleans up incomplete multipart uploads after 1 day (prevents storage cost from failed uploads)
  • ExpireOldEvents: When RetentionDays > 0, automatically deletes objects older than the specified retention period. Disabled when RetentionDays = 0 (indefinite retention, the default).

Security & Limitations

Security:

  • S3 bucket blocks all public access by default
  • S3 objects encrypted at rest (AES256)
  • Lambda follows least privilege principle (PutObject only)
  • CloudWatch logs capture all archive operations (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • S3 object key length: max 1024 bytes
  • Single-region deployment (follows SNS topic region)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.8 - Hummingbird Tools

Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup.

Features

  • Streaming Backup: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases
  • Streaming Restore: Downloads backup to temp file, streams through psql - no memory constraints
  • Automatic Rotation: Keeps configurable number of backups per group (daily/weekly/monthly)
  • Separate IAM Users: Write user for production backups, read-only user for staging restore

Prerequisites

  • AWS CLI configured with appropriate credentials
  • PostgreSQL client tools (pg_dump, psql)
  • Python 3.11 or later
  • Access to the hummingbird-status PostgreSQL database

Deployment

The SAM template creates:

  • S3 bucket for database backups
  • IAM user with write permissions (for production backup CronJobs)
  • IAM user with read-only permissions (for staging restore CronJob)

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-tools
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Note: Only deployed in production - staging restore reads from the production backup bucket.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod

Resource naming:

  • S3 bucket: {ResourcePrefix}
  • Write IAM user: {ResourcePrefix}-write-user
  • Read IAM user: {ResourcePrefix}-read-user

Usage

Backup

Run a backup with rotation:

python3 -m hummingbird_tools.backup <group> <rotation>

Arguments:

  • group: Backup group name (e.g., daily, weekly, monthly)
  • rotation: Number of backups to keep for this group

Environment variables:

Variable Description
S3_BUCKET S3 bucket name
S3_PREFIX Optional prefix for S3 keys
POSTGRESQL_HOST Database hostname
POSTGRESQL_USER Database username
POSTGRESQL_PASSWORD Database password
POSTGRESQL_DATABASE Database name

Example:

export S3_BUCKET=myapp-prod-db-backups
export S3_PREFIX=mydb/
export POSTGRESQL_HOST=myapp-postgres
export POSTGRESQL_USER=postgres
export POSTGRESQL_PASSWORD=secret
export POSTGRESQL_DATABASE=mydb

python3 -m hummingbird_tools.backup daily 7

This creates a backup like mydb/2026-01-16-02-30.daily.sql.gz and removes any daily backups beyond 7.

Restore

Restore from the latest backup:

python3 -m hummingbird_tools.restore

Environment variables are the same as for backup.

The restore process:

  1. Lists all backups matching *.{daily,weekly,monthly}.sql.gz
  2. Selects the latest by filename sort (most recent timestamp)
  3. Downloads to temp file
  4. Drops all existing tables in the public schema
  5. Streams the backup through psql

CronJob Schedule

Deployed via Kubernetes CronJobs in kubernetes/hummingbird-status/:

CronJob Schedule Environment Command
daily 30 2 * * * production python3 -m hummingbird_tools.backup daily 7
weekly 30 6 * * 0 production python3 -m hummingbird_tools.backup weekly 4
monthly 30 10 1 * * production python3 -m hummingbird_tools.backup monthly 12
restore (suspended) staging python3 -m hummingbird_tools.restore

To manually trigger a staging restore:

kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s)

S3 Key Structure

Backups are stored with a flat key structure per database:

{database}/{timestamp}.{group}.sql.gz

Examples:

mydb/2026-01-16-02-30.daily.sql.gz
mydb/2026-01-12-06-30.weekly.sql.gz
mydb/2026-01-01-10-30.monthly.sql.gz

Multiple databases can share the same bucket using different prefixes.

Development

See the main README for development workflows.

make hummingbird-tools/setup  # Install dependencies
make check                     # Lint code (ruff)
make fmt                       # Format code
make test                      # Run unit tests
make coverage                  # Run tests with coverage

Security

  • S3 bucket blocks all public access
  • S3 objects encrypted at rest (AES256)
  • Separate IAM users with least privilege:
    • Write user: s3:PutObject, s3:DeleteObject, s3:ListBucket
    • Read user: s3:GetObject, s3:ListBucket
  • Database passwords passed via environment variables (Kubernetes secrets)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.9 - ProdSec RPM Catalog

Builds an RPM catalog from the Hummingbird Pulp repos and publishes it to the metadata distribution for ProdSec. Also includes a monitoring companion that prints the last publication timestamp.

Features

  • DNF-Based Repo Query: Reads package metadata from all 5 Hummingbird repos (source, x86_64, aarch64, ppc64le, s390x) using the DNF API
  • Pulp REST API Upload: Creates artifacts, file content, publications, and updates the distribution in a single automated workflow
  • Artifact Deduplication: Skips upload when content SHA-256 matches an existing artifact
  • Async Task Polling: Handles Pulp’s asynchronous tasks with configurable timeout (10 minutes)
  • Sentry Error Reporting: Errors are reported to Sentry when SENTRY_DSN is set
  • Monitoring Check: Companion module prints the publication timestamp for automated freshness monitoring

Prerequisites

  • Python 3.11 or later
  • dnf system library (available on Fedora/RHEL)
  • Network access to packages.redhat.com (repo metadata)
  • Pulp API access with client certificate or username/password auth
  • A file repository and distribution named metadata in the public-hummingbird Pulp domain

Usage

Build and Publish Catalog

python3 -m hummingbird_tools.prodsec_catalog

No arguments. All configuration is via environment variables.

Check Last Publication Time

python3 -m hummingbird_tools.prodsec_catalog_check

Prints the timestamp of the most recent catalog publication in local timezone. Uses the same authentication as the catalog builder.

Configuration

Authentication

Client certificate auth (preferred for CronJob):

Variable Description
HUMMINGBIRD_PULP_BOT_CERTIFICATE Path to client certificate PEM file
HUMMINGBIRD_PULP_BOT_KEY Path to private key PEM file (optional)
HUMMINGBIRD_PULP_BOT_PASSWORD Optional passphrase for the key

Basic auth (alternative for local use):

Variable Description
PULP_USERNAME Pulp API username
PULP_PASSWORD Pulp API password

Credentials can also be configured in ~/.config/pulp/cli.toml (same format as the pulp CLI). Set PULP_CONFIG to override the config file path (useful in containers where the home directory may vary).

Pulp API

Variable Description
PULP_BASE_URL Pulp API base URL (or read from cli.toml)
PULP_CONFIG Path to pulp CLI config file (default ~/.config/pulp/cli.toml)

Error Reporting

Variable Description
SENTRY_DSN Optional Sentry DSN for error tracking

CronJob Schedule

Deployed via Kubernetes CronJob in kubernetes/hummingbird-status/:

CronJob Schedule Environment Command
prodsec-catalog 0 4 * * * production python3 -m hummingbird_tools.prodsec_catalog

Catalog Format

The catalog is a tab-separated file with one RPM per line:

{name}-{version}-{release}.{arch}.rpm\t{repo}\t{build_timestamp}

Example:

nginx-1.28.0-1.hum1.x86_64.rpm  x86_64  2026-01-15 14:30:00
kernel-6.12.5-1.hum1.src.rpm    source  2026-01-10 08:00:00

Published at: https://packages.redhat.com/api/pulp-content/public-hummingbird/metadata/hummingbird-rpm-catalog.txt

Development

See the main README for development workflows.

make hummingbird-tools/setup  # Install dependencies
make check                     # Lint code (ruff)
make test                      # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.10 - CVE Analysis

Automated analysis of HUM Jira Security (CVE) tickets. For each ticket, the tool fetches vulnerability data from MITRE and NVD, compares it against the versions shipped in the Hummingbird package repository, searches for upstream and Fedora fixes, and computes whether the ticket should be closed, moved to In Progress, or flagged for manual investigation.

CVE Lifecycle & State Vocabulary

A CVE tracker moves through a shared lifecycle that both this tool and the CVE agent reason about. The deterministic analysis decides a target resolution for each ticket; the residual it cannot resolve (cve-needs-attention) is handed to the agent; closed tickets are later reconciled against the VEX feed.

stateDiagram-v2
    direction TB

    [*] --> New

    New --> InProgress: affected / needs investigation
    New --> DoneErrata: fixed by analysis
    New --> NotABug: component not present
    New --> WontDo: package EOL

    InProgress --> Agent: upstream-fix-available / cve-needs-attention

    Agent: CVE Agent
    Agent --> Review: fix MR opened
    Agent --> DoneErrata: already_fixed
    Agent --> NotABug: not_affected
    Agent --> InProgress: no_upstream_fix / needs_human / error

    Review --> DoneErrata: MR merged
    Review --> Review: MR conflicts / needs human

    state "Closed (JiraStatus)" as Closed {
        DoneErrata: Done-Errata
        NotABug: Not a Bug
        WontDo: Won't Do
    }

    DoneErrata --> AwaitingVEX: enqueue awaiting-vex
    NotABug --> AwaitingVEX: enqueue awaiting-vex
    WontDo --> [*]: no VEX reconciliation
    AwaitingVEX --> [*]: VEX matches resolution
    AwaitingVEX --> AwaitingVEX: pending / mismatch

    classDef jiraStatus fill:#e5e7e9,stroke:#27ae60,color:#1f2937
    classDef closedStatus fill:#e5e7e9,stroke:#1f5f8b,color:#fff
    classDef resolution fill:#d5f5e3,stroke:#27ae60,color:#1f2937
    classDef automation fill:#e5e7e9,stroke:#7f8c8d,color:#1f2937
    class New,InProgress,Review jiraStatus
    class Closed closedStatus
    class DoneErrata,NotABug,WontDo resolution
    class Agent,AwaitingVEX automation

Notes on the diagram:

  • Agent is the CVE agent. See that tool’s doc for the full internals (currently not deployed, work is done by humans).
  • Blue states are Jira statuses, green states are resolutions, and gray states represent supporting automation.
  • Won’t Do (package EOL) closes without VEX reconciliation; Done-Errata and Not a Bug enqueue awaiting-vex and leave AwaitingVEX only once the VEX feed matches the resolution.

These state names live in exactly one place, hummingbird_cve_analysis/lib/states.py, imported by both tools:

  • JiraStatus — New, In Progress, Review, Closed.
  • Resolution — Done-Errata, Not a Bug, Won't Do (set when closing).
  • Disposition — the second field of a computed_resolution line: the three resolutions above plus the open-ticket pseudo-dispositions affected and needs investigation.
  • Verdict — the CVE agent’s per-ticket outcome (needs_change, not_affected, already_fixed, no_upstream_fix, needs_human, error), with VERDICT_OUTCOME mapping each verdict to the Jira status/resolution it implies.

How It Works

1. Ticket Selection

The tool queries Jira for HUM project tickets with component Security. Tickets can be filtered by creation period (--show-since "2 weeks"), limited to specific keys (HUM-796 HUM-518), or scanned in bulk. Closed tickets are skipped by default unless --include-closed is specified or specific ticket keys are given on the command line.

Embargoed tickets (security level “Embargoed Security Issue” or embargo status field set to true) are always skipped.

Tickets with no valid CVE-YYYY-NNNN in the CVE ID field or Summary (for example GHSA/OSV-only trackers) are postponed: analysis is skipped, an INFO message is logged, and with --resolve a one-time tracker comment is posted and the cve-needs-attention label is applied (open tickets only). Use --skip-no-cve to omit those tickets without posting the postpone notice or adding the label.

2. CVE Data Collection

For each ticket the tool:

  1. Extracts the CVE ID from the Jira CVE ID field (customfield_10667) when populated, otherwise from the Summary via a regex match for CVE-YYYY-NNNN patterns. If neither source has a CVE ID, analysis is postponed (see above).

  2. Loads the CVE record from a local clone of the cvelistV5 repository and parses the CVE 5.0 affected block, following the CVE 5.0 Product and Version Encodings specification. This includes:

    • Parsing lessThan and lessThanOrEqual version ranges
    • Handling lessThan: "*" (no upper bound) and lessThan: "4.*" (end-of-series, all versions in the 4.x series) wildcards per the spec
    • Processing changes lists that subdivide ranges into affected and unaffected segments
    • Respecting defaultStatus at the product level
    • Filtering out versionType: "git" entries that use commit hashes instead of numeric versions (these are preserved as informational data but not used for version comparison)
    • Detecting CNA data errors where git commit hashes are used in version fields without versionType: "git", including hashes wrapped in operator syntax (e.g., < 6374ae0bcdfe...)
    • Parsing inline comparison operators (< 12.3.0, >= 4.0, <= 2.5) that are not part of the CVE 5.0 schema but are widely used by CNAs in practice
    • Parsing compound range bands (>= 2.0, < 2.2.26 and > 1.32.3, < 1.34.6) used by some CNAs to express a closed-open range in a single version string

  3. Looks up version ranges from the NVD data feeds and merges them with the MITRE data. NVD data is obtained from the JSON 2.0 data feeds at nvd.nist.gov/feeds/json/cve/2.0/ (per-year .json.gz tarballs updated daily, plus a CVE-Modified overlay updated every 2 hours). Feeds are cached locally (--nvd-cache-dir) and only re-downloaded when the NVD .meta file SHA256 indicates newer data is available. When MITRE has no vendor-specific data (all vendors are “n/a”), NVD ranges replace the MITRE data entirely. NVD references tagged “Patch” or matching commit URL patterns are also extracted and used for upstream fix detection (see below).

  4. Looks up the Hummingbird package version by scraping the Pulp repository index at packages.redhat.com for the latest RPM matching the package name extracted from the ticket summary.

3. Resolution Computation

The tool compares the shipped Hummingbird version against the affected version ranges to compute a recommended resolution:

  • Closed / Done-Errata: The repo version is not in any affected range, or the repo version is >= the fix version, or the repo version appears in the “not affected” list.
  • In Progress / affected: The repo version falls within an affected range.
  • In Progress / affected (no version data): Neither MITRE nor NVD has affected version information. The ticket is moved to In Progress for manual review but does not receive the cve-needs-attention label.
  • In Progress / needs investigation: Version data was available but could not be compared (e.g., CVE only provides git commit hashes), or multiple distinct products with different versioning schemes are listed and no cve_product override is configured. These cases receive the cve-needs-attention label.

4. Product Mismatch Detection

The tool detects when a CVE was filed against the wrong Hummingbird package. It compares the CVE vendor/product names against the Hummingbird package name and upstream repo URL from the package map. For example, a CVE for isaacs/node-tar (an npm package) filed against the tar RPM (GNU tar) is flagged as a mismatch. This detection works even without a repo URL by comparing normalized product names.

Mismatch triage uses a two-phase evidence gate:

  1. Identity mismatch — CVE product/vendor does not match the package (cve_product override or heuristic name/repo matching)
  2. SBOM (+ binary) evidence — only then decide misfiled vs vendored vs needs investigation

Phase 2 fetches the latest package SBOM from the Hummingbird Pulp repository and records the artifact used (NVR/URL) plus match evidence. Go module matching accepts the module root and major-version paths (github.com/vendor/product or .../v5). Deeper import paths such as .../api or .../daemon only count as a hit when a CVE path hint (from malformed/version-path fields) aligns with that subpackage, so a client/API module does not satisfy a daemon-only CVE. If the CVE product is found as a vendored dependency, the tool runs binary confirmation in this order:

  1. SPDX SBOM evidence — sourceInfo pointing at shipped paths (e.g. Go buildinfo under /usr/bin/...) or CONTAINS from binary RPM roots (*.x86_64, *.aarch64, …) marks present_in_binary without Syft. Evidence only under .src / go.mod is treated as source-only for the next step.
  2. RPM Provides: bundled(...) — when SBOM evidence is source-only or ambiguous, downloaded non-debug binary RPMs are queried with rpm -qp --provides. A matching Fedora bundled Provide (npm bundled(npm(name)), Go bundled(golang(IMPORT_PATH)), Node core bundled(nodejs-undici), etc.) marks present_in_binary with evidence_method=rpm_bundled_provides. Ecosystem wrappers such as npm(...) are unwrapped so a CVE/SBOM search for nanoid matches bundled(npm(nanoid)). This catches dependencies embedded into binaries (including minified frontend JS) that Syft cannot inventory as on-disk packages.
  3. Syft on binary RPMs — when neither SPDX nor bundled Provides confirm presence, Syft scans those same RPMs. When OSIDB reports subpackage names that match RPMs for the NVR, only those RPMs are scanned; if OSIDB data is missing or the names do not match this package’s RPMs (for example a Redis CVE filed against boost), the full NVR set is used. Pulp download URLs use the arch from the RPM filename (aarch64 files under aarch64/, x86_64 under x86_64/; noarch stays on the x86_64 listing).

Syft absence only becomes absent_in_binary_confirmed when SBOM evidence is source-only (or no SBOM document was available) and Syft actually ran. A miss with ambiguous SBOM evidence stays unknown so embedded Go/Rust modules are not closed as source-only when SPDX already ties them to a binary. A Provides-only scan (Syft unavailable) never confirms absence. Per-RPM download or scan failures are skipped so sibling RPMs can still confirm presence; if any RPM failed and no hit was found, the result stays unknown (absence is never confirmed on a partial scan). A Provides query failure soft-misses and falls through to Syft for that same RPM.

  • present_in_binary (mismatch_vendored): legitimate vendored ticket; compare the confirmed binary/Provide version (falling back to the SBOM vendored dep version) against the CVE ranges (parent latest_repo_version / Hummingbird repo version stay the package), and apply cve-needs-attention so operators can review the binary hit. When several bundled Provides match the same dependency (for example npm(nanoid) at 3.3.16 and 5.1.16), every copy is collected. Resolution is worst-of / all-must-be-fixed: the ticket stays affected if any copy is still affected; Done-Errata only when every copy is not affected. The tool does not invent multi-major fix mappings from a single CVE fixed line — Fixed in Build remains the override when CVE version data only describes one release line. Placeholder dep versions such as cargo std@0.0.0 are not comparable: fall back to the parent package version when available; otherwise set non_comparable_vendored_version, recommend New / needs investigation, and apply cve-needs-attention — never Done-Errata from “0.0.0 is not in any affected range”
  • absent_in_binary_confirmed (mismatch_source_only): source-only; eligible to close as Not a Bug / Component not Present. Must never set Fixed in Build or take the advisory / Done-Errata path.
  • unknown (mismatch_binary_unknown): leave for investigation; do not auto-close or call the ticket misfiled

Stable reason codes are emitted in analysis output (Reason codes:) and on each CVE entry (reason_code / mismatch_gate) for automation:

Reason code Meaning
mismatch_sbom_miss Identity mismatch + SBOM miss → misfiled candidate
mismatch_sbom_unavailable Identity mismatch but SBOM could not be checked
mismatch_binary_unknown SBOM hit but binary presence unknown
mismatch_source_only SBOM hit confirmed absent from binaries
mismatch_vendored SBOM hit confirmed present in binaries → label for review
non_comparable_vendored_version Vendored version is a placeholder (e.g. 0.0.0) with no parent fallback → needs attention

A misfiled recommendation requires identity mismatch + SBOM miss (no vendored hit). Ambiguous checks and confirmed binary RPM matches keep cve-needs-attention with the reason code above. When present in binaries, normal resolution flow still proceeds (In Progress / Close) in addition to the attention label. The upstream fix search targets the parent package’s repo. Each analysis cycle re-fetches the SBOM and re-runs binary confirmation.

5. Upstream Fix Detection

Fix search uses a precedence model so tickets are not dual-labeled from both Fedora and upstream when a definitive path exists:

  1. CVE/NVD fix links (preferred): NVD references tagged “Patch” or matching commit URL patterns, plus CVE/NVD PR/MR references that match the package’s upstream repo. When these yield a fix label, Bodhi and forge search are skipped.
  2. Otherwise, repo-aware search from metadata/*.json upstream_repo:
    • Fedora-based (src.fedoraproject.org, pagure.io, or missing repo URL): Bodhi + DistGit only (see §6)
    • Non-Fedora forge (GitHub/GitLab/cgit): forge search only

Forge Search (non-Fedora fallback):

  • GitHub: Searches PRs via the GitHub API (/search/issues) for PRs mentioning the CVE ID. Fetches commit counts from the pulls API.
  • GitLab: Searches merge requests via the GitLab API on supported instances (gitlab.com, gitlab.gnome.org, gitlab.freedesktop.org, etc.). The --gitlab-token is only sent to gitlab.com to avoid 401 errors on other instances.
  • cgit: Scrapes commit log pages on cgit hosts (Savannah, Sourceware, kernel.org, busybox.net, etc.) for commits mentioning the CVE ID. Handles URL rewrites for Savannah hosts (e.g., git.savannah.gnu.org/git/ to cgit.git.savannah.gnu.org/cgit/).

Fix status is classified as:

  • upstream-fix-available: At least one merged/closed PR, committed fix, or NVD patch reference
  • upstream-fix-in-progress: At least one open PR with no merged fixes

6. Fedora Fix Detection

Used only for Fedora-based packages (or when no upstream repo is configured) and only when CVE/NVD fix links did not already produce a label. Hits are classified with the same upstream-fix-* labels as forge/NVD evidence (Bodhi update links are still shown in analysis output):

  • Bodhi: Queries the Fedora Bodhi API for updates matching the package name that reference the CVE ID (in the cves list or notes field). Supports multi-page results. Stable → upstream-fix-available; testing/pending → upstream-fix-in-progress.
  • DistGit Spec Scan: When Bodhi has no matches, fetches the Fedora DistGit spec file and scans for CVE references in patch filenames, changelog entries, and comments to detect backported fixes (upstream-fix-available).

7. Fixed Build Detection

The tool automatically detects whether the current Hummingbird build fixed a CVE by checking if all CVE IDs are mentioned anywhere in the package directory (patch names, changelogs, comments in the .spec file and so on). If found, it returns the current source RPM name. Assuming that we run the analysis frequently enough, this is precise enough, otherwise it errs on the side of caution (i.e. it possibly marks a higher version as fixed even if an earlier one already carried the fix).

The detected build is shown in the output as Detected Fix: package-version-release.hum1.src.rpm.

When --resolve is specified and a fixed build is detected, the tool automatically populates the “Fixed in Build” Jira field with the detected SRPM name, but only if:

  • The field is not already set (manual values take precedence)
  • The issue is not in Closed status (respects human closure decisions)
  • The assessment is not mismatch_source_only / source_only_confirmed (Component not Present closes as Not a Bug and clears any Fixed in Build)
  • The computed resolution is already Closed / Done-Errata or every ticket CVE ID appears on an uncommented PatchN: line (HUM-6182). A spec comment or leftover unapplied file that merely mentions the CVE is not enough. Product-mismatch and EOL tickets never take this PatchN override.

Version-range analysis can still say “affected” after a backport that does not bump Version (for example an unbounded range such as 1.1.1+ on popt 1.19). In that case an applied CVE-YYYY-NNNN.patch listed as PatchN: is allowed to set Fixed in Build. The SRPM must already be in Pulp (the existing fix_committed_not_built gate is unchanged). The ticket is not closed here; advisory review remains the backstop.

The RPMs repository can be provided via --rpms-repo, pointing to a local clone. This is preferable with multiple runs to avoid the git clone operation. When not given, the repository will be (shallow) cloned into a temporary directory.

Similarly, the cvelistV5 repository (used for CVE record lookups) can be provided via --cve-repo. When not given, a shallow clone is performed into a temporary directory. For production use, pre-clone and update the repo externally (e.g. via a CronJob) to avoid the clone overhead on each run. The tool does not attempt to update the provided repo itself.

8. Advisory Integration (with --resolve)

When a CVE is resolved as “Closed / Done-Errata”, the tool integrates with the CEE GitLab advisories repo to document the fix:

  1. Clone: The advisories repo (default: releng/advisories) is shallow-cloned via a bot fork using netrc-based authentication (no tokens in process arguments or logs).
  2. Modify: For each resolved ticket, the advisory YAML is updated: cves.fixed entries are added, the type is changed from RHBA to RHSA, and CVE references are appended.
  3. Batch MR: All advisory changes are accumulated as individual commits on a single branch (cve-analysis/batch). One merge request is created at the end of the run covering all tickets.
  4. Review state: Tickets are transitioned to “Review” (not directly closed) and the batch MR URL is posted to each affected ticket.
  5. Auto-close: On subsequent runs, tickets in “Review” are checked: if the advisory MR has been merged, the ticket is closed as “Done-Errata”. If the MR has unresolvable rebase conflicts, a comment with manual resolution steps is posted and the advisory-mr-failed label is added.
  6. Slack notifications: When --slack-webhook-url (or the SLACK_WEBHOOK_URL env var) is set, the tool sends a Slack message when the batch MR fails to merge or the post-merge pipeline fails. Merge-failure messages @-mention prarit and jstibran.

The advisory flow is skipped when --advisories-project points to a non-production URL, allowing safe testing without modifying Jira.

8.1 Recovery for failed advisory MRs

In some failure cases (for example, advisory MR conflicts followed by manual ticket closure), Jira status can be corrected to Done-Errata while dashboard lifecycle metadata remains incomplete. This leaves R-Time rows PENDING (elapsed-to-now) until image, VEX, and close all exist.

Use the recovery script to backfill delivery events:

  • rpm_fix_published_to_pulp (hb_rpm_fix) from packages.redhat.com source RPM timestamps
  • image_rebuilt_on_quay (hb_image_fix) from catalog/quay image history
  • hum_ticket_closed from Jira resolutiondate

Only tickets that are Closed / Done-Errata are used for event derivation and dashboard import. Tickets that remain in Review (or any non-closed state) are reported and skipped for event backfill.

For tickets still in Review, the script also attempts advisory reconciliation before deriving events:

  • Read advisory MR URL from Jira comments (Advisory MR: https://...)
  • If no MR URL is present in comments, search merged advisories MRs for the HUM ticket key in MR description/title
  • When a merged MR is found and --apply is used, transition the ticket to Closed / Done-Errata and then derive hum_ticket_closed

Script location:

  • hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py

Run in dry-run mode first (default):

cd /path/to/tools
export JIRA_TOKEN=your_jira_token
export CEE_GITLAB_TOKEN=your_cee_gitlab_token
export CVE_REPORT_TOKEN=your_dashboard_token

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json

Apply the backfill to dashboard event storage:

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json \
  --apply

Target specific tickets (instead of default JQL):

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --apply \
  HUM-4884 HUM-4871 HUM-4844

Verification example:

# Replace with the actual dashboard URL (same as --dashboard-url default)
DASHBOARD_URL=https://hummingbird-dashboard-hummingbird--internal.apps.int.spoke.prod.us-east-1.aws.paas.redhat.com
curl -s "${DASHBOARD_URL}/api/cve/r-time?days=30" | jq -r '
  .entries[]
  | select(.key=="HUM-4884")
  | [.key, (.fix_delivered_at // "-"), (.stages.hb_rpm_fix // "-"), (.stages.hb_image_fix // "-")]
  | @tsv
'

If hb_rpm_fix and hb_image_fix remain empty in the derived JSON output, the issue is source-data availability (no resolvable RPM/image publish signal), not dashboard ingestion.

If a ticket is in Review and no advisory MR can be found in Jira comments or merged advisories MRs, it is skipped and reported for manual follow-up. By default, the script targets advisory-mr-failed tickets (for both Closed / Done-Errata and Review states); override with --jql when needed.

8.2 Dashboard collection ticket selection

Dashboard lifecycle collection (HUM-5603) uses a separate ticket-selection path from cve_analysis mutations. Helpers live in hummingbird_cve_analysis/dashboard_selection.py.

Watermark (start of previous run): read the newest entry from the dashboard Run Log (GET /api/cve/run-log?limit=1). Approximate previous-run start as run_at - duration_seconds (Run Log run_at is ingest time), then subtract a small buffer (default 2 minutes) for clock skew. Using start — not end — avoids missing tickets updated while the previous collector run was in progress. Overlap is intentional; /api/cve-report event upserts are idempotent.

JQL selection:

  • All still-open HUM Security tickets (rescan for updates)
  • Closed tickets with Jira updated >= watermark
  • Bootstrap (empty Run Log): open tickets only
  • Explicit ticket keys: key in (...) (any status)

Use --start-time (ISO-8601, e.g. 2026-08-01T00:00:00Z) to override the Run Log watermark and recollect Closed tickets with updated >= that start, so dashboard data can be overwritten/backfilled.

Inspect watermark and JQL without modifying Jira or the dashboard. From the tools repo root, either install the package (make hummingbird-cve-analysis/setup) or set PYTHONPATH:

cd /path/to/tools
export PYTHONPATH=hummingbird-cve-analysis
export CVE_REPORT_TOKEN=your_dashboard_token

python3 -m hummingbird_cve_analysis.dashboard_selection --prod

# Optional: also search Jira and list matching keys
export JIRA_TOKEN=your_jira_token
python3 -m hummingbird_cve_analysis.dashboard_selection \
  --prod \
  --jira-user user@example.com \
  --fetch-issues -o json-pretty
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for /api/cve/run-log
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--watermark-buffer-minutes Minutes subtracted from previous-run start estimate (default: 2)
--fetch-issues Also search Jira with the constructed JQL
--jira-token / --jira-user JIRA_TOKEN Jira credentials (only with --fetch-issues)
--jira-url JIRA_URL Jira base URL (only with --fetch-issues)
--max-results Max issues to fetch with --fetch-issues (default: 2000)
--output, -o human, json, or json-pretty

8.3 Analysis → collector handoff file

cve_analysis writes a small JSON handoff file for collect_cve_dashboard (HUM-5799 / HUM-5800). This carries Run Log counters, captured log_output, and per-ticket fields that are expensive to recompute (upstream_fix, fedora_fix) plus human_text for Run Log details. Delivery timestamps (hb_rpm_fix / hb_image_fix) and hum_ticket_closed are gathered by the collector, not this file. Analysis stdout is human text only; it does not emit /api/cve-report JSON.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.cve_analysis \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --handoff-file /tmp/cve_analysis_handoff.json \
  ...

Handoff shape (schema_version: 1):

  • Top level: run_started_at, duration_seconds, mutation counters (labels_changed, advisories_created / advisories_failed, tickets_closed, comments_posted, attachments_uploaded, fatal_errors), log_output
  • tickets[]: key, upstream_fix, fedora_fix, human_text
  • vex_updates[] (HUM-5843): key, vex_status, vex_match_state, vex_resolved, labels from the awaiting-vex reconcile pass

8.4 collect_cve_dashboard

collect_cve_dashboard (HUM-5798 / HUM-5800) owns dashboard JSON emission. cve_analysis no longer prints /api/cve-report JSON on stdout; it prints human analysis text and optionally writes --handoff-file for the collector.

  1. Resolve Run Log watermark + open-ticket rescan (section 8.2). Selection always includes Closed tickets labeled awaiting-vex (HUM-5843), then any vex_updates[] keys missing from that JQL
  2. For each ticket, gather slim lifecycle fields from Jira / cvelistV5 / Fixed-in-Build → Pulp (cve_published, hum_ticket_created, hum_ticket_closed, hb_rpm_fix) and catalog image history (hb_image_fix), plus OSIDB osidb_flaw_created / osidb_affect_created from the public OSIDB API (omitted when OSIDB has no data), computed_resolution from Jira status/resolution (R-Time gates on Done-Errata in that string) and catalog_image_source from the catalog source map (R-Time delivery is image publish when true, RPM publish when false). Timestamp helpers live in lib/lifecycle.py and are shared with analyze_issue
  3. Merge --handoff-file counters, log_output, per-ticket upstream_fix / fedora_fix / human_text, and vex_updates
  4. Dry-run prints /api/cve-report JSON; --apply POSTs it

The collector builds the catalog source map once per run for hb_image_fix and catalog_image_source. When a delivery timestamp is omitted, the dashboard keeps any existing rpm_fix_published_to_pulp / image_rebuilt_on_quay value rather than clearing it. An empty or failed catalog map omits catalog_image_source so the dashboard keeps requiring an image.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json

# POST to preprod dashboard
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json \
  --apply
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for Run Log + /api/cve-report
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Analysis handoff JSON from cve_analysis --handoff-file
--apply POST report to /api/cve-report (default: dry-run JSON on stdout)
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--cve-repo Local cvelistV5 repo for cve_published
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--output, -o human, json, or json-pretty

8.5 verify_image_publish

verify_image_publish (HUM-721) catches CVE tickets that claim a fix via image rebuild (hb_image_fix in the whiteboard cve_cycle) but whose image was never actually published to the registry — hb_image_fix is set from catalog release-history metadata (oldest_created), which can lag or be wrong; this re-verifies with a live check.

  1. Search Jira for Closed HUM Security tickets updated within the last --window-minutes (default 60)
  2. For each, read hb_rpm_fix / hb_image_fix from the whiteboard; skip tickets where hb_image_fix falls outside the window
  3. Re-derive the source package name from the summary/labels (no SBOM / network calls) and re-run the same catalog-history lookup (find_image_publish_release) that originally produced hb_image_fix, recovering the specific image/stream/variant/digest
  4. HEAD the image manifest by digest directly against the registry (image_manifest_exists) — this is the part hb_image_fix itself never did
  5. Post a Slack alert for each unverified claim; log a summary either way

Read-only: never modifies Jira, the dashboard, or the CVE analysis pipeline.

PYTHONPATH=hummingbird-cve-analysis python3 -m hummingbird_cve_analysis.verify_image_publish \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --window-minutes 60
Option Environment Variable Description
--window-minutes Look back this many minutes for hb_image_fix (default: 60)
--slack-webhook-url SLACK_WEBHOOK_URL Webhook for unverified-claim alerts
--dry-run Check and log, but do not post to Slack
--catalog-api-base Catalog API base URL
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--jira-url JIRA_URL Jira base URL

9. Jira Actions (with --resolve)

When --resolve is specified, the tool modifies Jira tickets:

  • Fixed in Build: When a fixed build is detected and the field is not already set, the tool populates it with the detected SRPM name (only for non-closed issues). Not-affected Closed / Done-Errata closes also set Fixed in Build from detected_fixed_build or latest_srpm when empty, so R-Time image timestamps can be resolved.
  • Closed / Done-Errata: Tickets where the shipped version is not affected are transitioned to Review with an advisory MR (see above). After the MR merges, they are closed as Done-Errata on the next run.
  • Move to In Progress: Tickets where the shipped version is affected are transitioned to In Progress with a comment including affected version ranges, upstream fix status, and Fedora update links.
  • Needs Investigation: Tickets with incomplete version data get a comment explaining why automatic resolution was not possible.
  • Product Mismatch: A comment is posted explaining the mismatch (with SBOM artifact evidence) and the cve-needs-attention label is applied. No transition is performed. Auto-detected Fixed-build / delivery timestamps (hb_rpm_fix, hb_image_fix) are cleared so R-Time does not treat misfiled tickets as done. Human Fixed in Build overrides are kept. When --resolve runs against the production advisories project, the analyzed package SBOM JSON is also attached to the ticket as {nvr}.sbom.json (skipped if that filename is already present). The same attachment is applied for other SBOM-backed review paths (vendored binary hits, source-only closes, binary-unknown).
  • Package Not Present: If the package is missing from the rpms repo, the tool still runs an SBOM-first check against the CVE product(s). An SBOM hit triggers Syft binary confirmation: absent_in_binary_confirmed closes as Not a Bug / Component not Present; presence or unknown leaves the ticket for investigation. An SBOM miss/unavailable closes as Not a Bug with VEX Component not Present. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Package EOL (fix_status: 0): Tickets for packages marked End-Of-Life in rpms metadata are closed as Won’t Do with no VEX Justification. This takes precedence over Done-Errata and Fixed-in-Build / advisory paths. Automation labels (upstream-fix-*, legacy fedora-fix-*, cve-needs-attention, cve-next-release, advisory-mr-failed) are removed; fedora-bz-filed is kept as an audit record. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Label Management: Labels are applied and updated:
    • upstream-fix-available / upstream-fix-in-progress (including Bodhi/DistGit evidence for Fedora-based packages)
    • cve-needs-attention (applied when human review is needed; removed when resolved)
    • Legacy fedora-fix-* labels are stripped on subsequent runs
    • Labels are upgraded (in-progress to available) and stale labels are cleaned up on ticket closure.

Every Jira action (label add/remove, status transition, product mismatch) includes a single comment with a bold action summary heading followed by the full analysis output in a preformatted code block, giving the reader the same detail they would see on the CLI.

10. Assignee-Based Automation Control

When --resolve is active, the tool checks each ticket’s assignee. If the assignee is not the bot account (--jira-user), the tool skips all automated actions (comments, labels, transitions) for that ticket. A one-time comment is posted explaining that automation is skipped. This allows humans to take ownership of a ticket by assigning it to themselves, preventing the bot from interfering with manual work. Reassigning back to the bot account re-enables automation.

For testing purposes, use --skip-assignee-check to process all tickets regardless of assignee.

11. Human Closure Protection

When the tool encounters a Closed ticket whose analysis recommends a non-Closed resolution (e.g., In Progress), it checks the Jira changelog to determine who performed the last status transition. If a human (not the bot) closed the ticket, the tool skips the transition to respect the human’s decision. This prevents the tool from reopening tickets where a fix was backported without a version bump or where a human determined the CVE does not apply.

12. Stale Closure Detection (with --include-closed)

When --include-closed is used, the tool also analyzes Closed tickets. If the current analysis recommends a non-Closed resolution (e.g., the package version changed and is now affected), a warning is emitted flagging the ticket for review.

13. Sprint and Epic Assignment

When --resolve is active, the tool automatically adds CVE tickets to the current Hummingbird sprint and links them to a per-sprint CVE tracking epic when human interaction is detected. This ensures sprint metrics capture human work on CVE tickets.

The tool queries the Jira Agile API for the active sprint on the Hummingbird board (--board-id, default 1489) and filters by name prefix (--sprint-prefix, default “Hum S”) to identify the correct sprint among multiple teams sharing the HUM project.

Sprint and epic assignment is triggered in two cases:

  • Self-assigned tickets: When a ticket is assigned to someone other than the bot, the ticket is added to the current sprint and linked to the sprint’s CVE epic.
  • Human-set Fixed in Build: When the Fixed in Build field was set by a human (not the bot), as determined by the issue changelog, the ticket is added to the current sprint.

If no matching epic exists for the current sprint, one is created automatically with the summary “CVE tickets for sprint <name>”. The epic key is cached for the duration of the run. A Jira comment is posted on the ticket recording the sprint and epic assignment.

Tickets that already have a sprint assigned are skipped. Tickets closed manually without going through either automated path should be added to the sprint by hand (see the manual process guide).

14. Continuous Operation

When --time-between-runs N is set (N > 0), the tool runs in a loop, re-executing the full analysis every N minutes. The default is 0 (single run). Shutdown is graceful: SIGINT/SIGTERM finishes the current cycle before exiting.

15. Feature Flag

When --feature-flag-name is set (default: cve_analysis_enabled), the tool checks a GitLab feature flag on the project specified by --feature-flag-project-id (default: 73447720, i.e. redhat/hummingbird/rpms) at the start of each cycle. If the flag is inactive, the cycle is skipped and the tool sleeps for 60 seconds before checking again.

The check uses the --gitlab-token / GITLAB_TOKEN credential, which must have Developer role or higher on the target project. The check is fail-open: if GitLab is unreachable or the token lacks permissions, the tool assumes the flag is enabled and proceeds.

Set --feature-flag-name "" to disable the check entirely.

Package Map

The package map is loaded from the rpms repo’s per-package metadata files (metadata/<package>.json). Each file contains an upstream_repo field pointing to the canonical upstream git repository, plus optional fields:

  • upstream_branch – upstream branch (for versioned packages sharing a repo)
  • cve_product – CVE vendor/product override
  • version_transform – version transform rule
  • fix_status – package fix policy (0 = End-Of-Life / will not fix CVEs; close as Won’t Do)

If fix_status is 0, analysis always resolves to Closed / Won’t Do (including when the shipped version is outside the CVE affected range or a product mismatch would otherwise need investigation). No VEX Justification is set. If fix_status is missing or set to 1, behavior is unchanged from the default analysis flow.

The cve_product field is an optional human-curated value that specifies which CVE vendor/product entry maps to this package. It supports these formats:

  • Vendor / Product (exact match): matches a specific vendor and product pair. Example: "F5 / NGINX Open Source" for the nginx package, which excludes NGINX Plus entries that use incompatible R-versioning.
  • Vendor (vendor prefix match): matches any product whose vendor starts with the given string. Example: "Go " for golang packages, where the vendor varies (Go standard library, Go toolchain, etc.) and the product varies by module (net/url, os, crypto/x509).
  • Multiple selectors: provide more than one acceptable value either as a JSON list in metadata (preferred) or as a semicolon-delimited string. Example: ["vda-linux / busybox_mirror", "BusyBox / BusyBox"] or "vda-linux / busybox_mirror; BusyBox / BusyBox". The tool treats this as “match any selector”.

When cve_product is set, the tool uses it for exact product matching in both resolution computation (filtering to the correct product in multi-product CVEs) and mismatch detection. When empty, the tool falls back to heuristic name matching.

The rpms repo is cloned automatically at startup (or provided via --rpms-repo). A CSV override can be passed via --package-map for backward compatibility.

Vendored Dependency Detection

Vendored dependency detection uses live SBOM lookups from the Hummingbird Pulp repository. When a CVE product mismatch is detected, the tool fetches the SPDX SBOM for the ticket’s package from packages.redhat.com/.../metadata/sboms/{package}-main/ (same public index used by the rpms CVE skill), selects the newest dated sha256-….sbom entry from that listing, and searches that document for the CVE product(s) only (via purl references), stopping on the first hit. The analysis output always records the SBOM artifact used and whether the lookup hit, missed, or was unavailable.

The same SBOM-first check also gates Component not Present closures for packages missing from the rpms repo. An SBOM hit alone no longer blocks that closure: binary confirmation must reach absent_in_binary_confirmed (no bundled Provide match and Syft miss on source-only SPDX evidence) before Not a Bug / Component not Present is recommended. A binary hit or unknown result keeps the ticket open for investigation.

Go subpackages are matched by progressively stripping path components (e.g., github.com/jackc/pgx/v5/pgproto3 matches github.com/jackc/pgx/v5). When a match is found and confirmed in binaries, the confirmed binary/Provide version is preferred for resolution over the SBOM lockfile version. The SBOM and binary confirmation are re-run each analysis cycle.

Requires the rpm and/or syft CLI on PATH (both are available in the analysis container image via rpm-build and the Syft install). Without either tool, binary confirmation returns unknown. With only rpm, a bundled Provide can still confirm present_in_binary, but absence is never auto-confirmed without Syft.

The generate_vendored_map_sbom.py script in package_maps/ can still be used for auditing vendored dependencies across all packages, but is no longer required at runtime.

Prerequisites

  • Python 3.11 or later
  • Jira API token (Bearer or Basic auth)
  • Network access to redhat.atlassian.net, github.com, nvd.nist.gov, bodhi.fedoraproject.org, src.fedoraproject.org, and packages.redhat.com (note: github.com is only needed when --cve-repo is not provided and the tool must clone cvelistV5 itself)
  • rpm and syft CLIs on PATH (for binary RPM confirmation after SBOM hits: bundled Provides via rpm, component inventory via Syft)
  • GitHub API token (optional, for upstream PR search)
  • GitLab API token (optional, for gitlab.com MR search)
  • CEE GitLab API token (required with --resolve, for advisory repo)

Usage

# Basic usage with Jira token from environment
export JIRA_TOKEN=your_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Analyze specific tickets
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com HUM-796 HUM-518

# Show only tickets from the last 2 weeks
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --show-since "2 weeks"

# Analyze with upstream fix detection (report findings without modifying Jira)
export GITHUB_TOKEN=your_github_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Resolve tickets and apply labels
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --resolve

# Include closed tickets and check for stale closures
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --include-closed

# Include closed tickets but exclude specific ones
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --include-closed --exclude "HUM-555,HUM-552"

# Skip issues without CVE links; write collector handoff
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --skip-no-cve --handoff-file /tmp/cve_analysis_handoff.json

# Continuous mode: resolve tickets every 30 minutes
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --resolve --time-between-runs 30

Dashboard /api/cve-report JSON is produced by collect_cve_dashboard (section 8.4), not by cve_analysis stdout (HUM-5800).

Inspecting lifecycle timing with dump_lifecycle

scripts/dump_lifecycle reads lifecycle milestones and package data from the dashboard HTTP API and prints them for one or more HUM tickets. CVE_REPORT_TOKEN is needed.

# Text output for one ticket against prod
scripts/dump_lifecycle --prod HUM-1234

# JSON output for multiple tickets against preprod
scripts/dump_lifecycle --preprod --json HUM-1234 HUM-5678

# Custom dashboard URL
scripts/dump_lifecycle --dashboard-url https://... HUM-1234

Output includes all milestone timestamps (cve_published, osidb_flaw_created, osidb_affect_created, hum_ticket_created, …), computed duration legs, and rpm_first_published (package) sourced from package_lifecycle. The --json flag emits the output as a JSON array instead of human-readable text.

Backfilling OSIDB timestamps with backfill_osidb_timestamps.py

scripts/backfill_osidb_timestamps.py is a one-time helper that fills osidb_flaw_created and osidb_affect_created for tickets already stored in cve_ticket_events. It reads existing rows from /api/cve-export, looks up timestamps from the public OSIDB API, and POSTs new rows to /api/cve-import with reset=false (write-once; existing timestamps are kept). Tickets that already have both OSIDB events are skipped unless --force is set. CVE_REPORT_TOKEN is needed.

# Backfill prod
scripts/backfill_osidb_timestamps.py --prod

# Dry run against preprod
scripts/backfill_osidb_timestamps.py --preprod --dry-run

# Limit to specific tickets
scripts/backfill_osidb_timestamps.py --prod HUM-1234 HUM-5678

Ongoing collection is handled by collect_cve_dashboard. After a prod backfill, copy_prod_to_preprod.sh copies the new event types with the rest of cve_ticket_events (they are on the dashboard import allowlist).

Collecting rpm_first_published with collect_rpm_first_published

scripts/collect_rpm_first_published sweeps the Hummingbird Pulp repo for all packages, finds the earliest SRPM upload timestamp for each, and writes the results to the package_lifecycle table via POST /api/cve-export - no direct DB access.

# Sweep all packages from the rpms repo and write to prod
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

# Only process specific packages
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --package curl wget

# Dry run — log what would be written without posting
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --dry-run

Re-runs are safe: the dashboard upsert only updates occurred_at if the incoming timestamp is earlier than the stored one.

Analyzing CVE Response Delays with analyze_rpm_first_published

scripts/analyze_rpm_first_published inspects the relation between CVE publication, package onboarding (rpm_first_published), and HUM ticket creation:

# Analyze all tickets on prod
scripts/analyze_rpm_first_published --prod --token $CVE_REPORT_TOKEN

# Filter to new packages onboarded after CVE publication
scripts/analyze_rpm_first_published --prod --filter new-pkg

# Filter to pre-existing packages with at least 30 days actionable delay
scripts/analyze_rpm_first_published --prod --filter pre-existing --min-days 30 --sort actionable

CLI options:

  • --filter {all,new-pkg,pre-existing}: filter by onboarding timing relative to CVE.
  • --min-days N: minimum delay threshold in days (default: 0).
  • --sort {cve_to_hum,actionable,cve_to_rpm}: sort column (default: actionable).
  • --limit N: max rows to display (default: 50).

Configuration

Option Environment Variable Description
--jira-token JIRA_TOKEN Jira API or Bearer token
--jira-url JIRA_URL Jira base URL (default: https://redhat.atlassian.net)
--jira-user Username for Basic auth
--output, -o Stdout format: human only (json / json-pretty removed in HUM-5800)
--show-since Filter by creation period (e.g. 2 weeks, 3 hours)
--resolve Transition Jira tickets and apply labels
--skip-assignee-check Skip assignee validation (for testing)
--include-closed Include Closed tickets in analysis; warn on stale closures
--exclude Comma-separated ticket keys to skip (e.g. HUM-555,HUM-552)
--skip-no-cve Omit issues with no CVE ID (CVE ID field or Summary); no postpone comment/label
--max-results Max number of issues to fetch (default: 2000)
--github-token GITHUB_TOKEN GitHub API token for upstream PR search
--gitlab-token GITLAB_TOKEN GitLab API token for gitlab.com MR search
--nvd-cache-dir NVD_CACHE_DIR Directory for caching NVD data feed files; avoids re-downloading unchanged feeds
--cee-gitlab-token CEE_GITLAB_TOKEN CEE GitLab token for advisory repo operations
--advisories-project ADVISORIES_REPO Advisories repo URL (default: releng/advisories)
--advisories-fork ADVISORIES_FORK Bot’s fork URL for advisory MR creation
--slack-webhook-url SLACK_WEBHOOK_URL Slack webhook URL for failure notifications; empty or unset disables Slack
--no-merge-request Skip advisory MR creation during --resolve
--keep-advisory-repo Do not delete the cloned advisory repo after the run (useful for debugging)
--test-advisory Create advisory MR then immediately close it (for testing)
--package-map Path to CSV override; by default uses rpms repo metadata
--rpms-repo Path to RPMs git repo for fixed build detection
--cve-repo Path to local cvelistV5 git repo for CVE record lookups
--time-between-runs Re-run every N minutes; 0 = single run (default)
--feature-flag-project-id GitLab project ID for feature flag lookup (default: 73447720, rpms project)
--feature-flag-name Feature flag name to check each cycle (default: cve_analysis_enabled; empty = disabled)
--board-id Jira Agile board ID for sprint lookup (default: 1489)
--sprint-prefix Sprint name prefix to identify Hummingbird sprints (default: Hum S)
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Write analysis→collector handoff JSON (counters, log, fix times, human_text)
SENTRY_DSN Optional Sentry DSN for error tracking

Managed Labels

The tool manages the following labels on Jira tickets. These are automatically applied, upgraded, and cleaned up:

Label Meaning
upstream-fix-available A fix exists (forge PR/commit, NVD/CVE link, or Fedora Bodhi/DistGit)
upstream-fix-in-progress A fix is in progress (open forge PR, or testing/pending Bodhi update)
fedora-bz-filed A Fedora Bugzilla has been filed for this CVE
cve-needs-attention Analysis needs human attention (see below)
advisory-mr-failed Advisory MR has unresolvable rebase conflicts
cve-next-release Fix will arrive via the next upstream release (set manually; see below)
awaiting-vex Closed ticket awaiting VEX agreement with Jira resolution (HUM-5843 work queue)

Legacy fedora-fix-available / fedora-fix-in-progress labels are no longer applied; the bot removes them on subsequent runs (Bodhi/DistGit evidence now uses upstream-fix-*).

Labels are upgraded automatically (e.g., upstream-fix-in-progress is replaced by upstream-fix-available when a fix is merged). Stale labels are removed when tickets are closed, except fedora-bz-filed which is preserved as an audit record and awaiting-vex which is managed by the VEX reconcile pass. The cve-needs-attention label is removed automatically when the warning condition no longer applies.

awaiting-vex (HUM-5843)

When a ticket is closed as Done-Errata or Not a Bug in production, analysis adds awaiting-vex. Each --resolve cycle then:

  1. Queries Closed tickets labeled awaiting-vex, plus Closed Done-Errata / Not a Bug tickets resolved in the last 7 days (so human/agent closes that skipped the close-time enqueue still enter the queue)
  2. Fetches Hummingbird status from the Red Hat CSAF VEX feed for every CVE ID in the ticket summary, scoped to that ticket’s package (not a CVE-wide worst-case across other Hummingbird products)
  3. Compares to the Jira resolution for the awaiting-vex work queue (Done-Errata → fixed, Not a Bug → known_not_affected or package_not_listed when CSAF omits the package); match requires all CVE IDs to agree
  4. On match: records vex_status + vex_resolved (scan time) in the collector handoff and removes the label if present
  5. On pending/mismatch: adds awaiting-vex if missing, otherwise keeps it, and still emits current vex_status. Catch-up closes with no CVE ID in the summary are not labeled: MATCH is impossible, and adding the label would re-select the ticket forever.

The Closed tab computes MATCH from stored vex_status plus Jira resolution; it does not use stored vex_match_state as source of truth. Dashboard reads merged event metadata (later rows overlay earlier ones) so a ticket closed after cve_published does not keep leftover New / In Progress analysis text as Resolution (HUM-6091).

The collector also fetches any vex_updates[] keys that the watermark JQL missed, so a same-cycle catch-up match still lands on the Closed tab.

Won’t Do closures do not enqueue awaiting-vex. Open-ticket VEX mismatches remain a vex-checker reconcile concern.

cve-next-release

The cve-next-release label is used for CVEs where no immediate action is possible and the fix will arrive via the next upstream release. It is set manually. Common scenarios include:

  • Vendored dependencies: A fix exists in a vendored package (e.g. ws inside dotnet) but cannot be consumed until the parent package updates its vendored copy.
  • No backport path: The fix cannot be backported to the current release and must wait for a future upstream version.
  • Upstream fix pending: A fix is expected upstream but has not landed yet, and no interim mitigation is available.

Package metadata fix_status: 0 (EOL) does not use this label; those tickets are closed as Won’t Do instead.

When this label is present on a ticket:

  • The automation skips applying upstream-fix-available and upstream-fix-in-progress labels, since they would be misleading, and removes them if already present
  • The SBOM version check continues each analysis cycle, so when a new upstream release containing the fix is consumed, the ticket is closed normally
  • Fedora labels and cve-needs-attention are still managed normally
  • The label is not removed automatically; it must be removed manually when no longer applicable

cve-next-release tickets are not closed just because Hummingbird shipped a new build. They close when analysis concludes the CVE is no longer affected; for vendored dependencies, that means the SBOM shows the fixed upstream version, not merely a new parent NVR.

cve-needs-attention conditions

The cve-needs-attention label is applied when any of these conditions are detected:

  • Product mismatch: CVE vendor/product does not match the Hummingbird package (e.g. node-tar CVE filed against GNU tar) after an SBOM-first check finds no vendored dependency hit
  • Package missing from rpms but present in SBOM/binaries: package directory is absent from the rpms repo, yet the CVE product appears in the package SBOM and binary confirmation is present or unknown
  • Multiple products: CVE lists multiple distinct products with different versioning schemes and no cve_product override is configured (e.g. NGINX Open Source + NGINX Plus)
  • Git-only version data: CVE version data uses commit hashes instead of numeric versions (e.g. libsodium with lessThan: ad3004ec...)
  • CNA data error: A git commit hash is used where a version number is expected without setting versionType: "git", including hashes embedded in operator syntax (e.g., "version": "< 6374ae0bcdfe..."). The warning includes a link to the CVE 5.0 source control versions spec
  • Malformed version field: The version field contains syntax that could not be parsed (e.g., compound operator+hash ranges like >= hash1, < hash2)
  • CVE record not found: the CVE record file was not found in the cvelistV5 repo
  • No repo version: the Hummingbird package was not found in the Pulp repository (package name mismatch or missing SRPM)
  • Stale closure: ticket is Closed in Jira but current analysis recommends a different resolution

Human override workflow

When a human wants to take over a ticket from the bot:

  1. Assign the ticket to yourself – the bot skips all automation on tickets not assigned to the bot account. Reassign to the bot to re-enable automation.

  2. Set “Fixed in Build” – if you know the fix is in a specific SRPM, set the “Fixed in Build” field to the SRPM name (e.g. libarchive-3.8.7-1.hum1.src.rpm). The bot will use this to create the advisory MR and close the ticket, bypassing its own analysis.

Migration note: The cve-analysis-okay label is deprecated and no longer suppresses automation. Existing tickets with this label will be re-processed by the bot on its next run. To keep the bot from touching a specific ticket, reassign it to yourself before the next run. The cve-analysis-okay label can then be removed manually.

Output

Human-Readable

Project: HUM  Component: Security

  HUM-796  CVE-2026-2673 openssl: buffer overflow [hummingbird-1]
    Open since:       7 days (Mar 24 2026)
    Labels resolution: upstream-fix-available
    OpenSSL / OpenSSL:
      Affected versions: 3.5.0 < 3.5.6
      Fixed in:          3.5.6
      Hummingbird repo (latest):  3.5.5 / openssl-3.5.5-1.hum1.src.rpm
    Repo:             https://github.com/openssl/openssl
    CVE-2026-2673: upstream-fix-available (3 PRs)
      [CLOSED, 3 commits] Fix group tuple handling in DEFAULT expansion (3.5)
              https://github.com/openssl/openssl/pull/30110
    Fedora update: FEDORA-2026-abc123 openssl-3.5.6-1.fc44 (stable, security)
              https://bodhi.fedoraproject.org/updates/FEDORA-2026-abc123
    Jira current:     In Progress / (none)
    Jira resolution:  In Progress / affected (repo 3.5.5 is in affected range 3.5.0 < 3.5.6)

JSON

Each issue includes: key, summary, labels, open_since, cve_ids, cves (with version and resolution data), jira_current (status/resolution), computed_resolution, and upstream (with PR/MR search results, Bodhi updates, and Fedora version data).

Viewing Logs

The CVE analysis tool runs as a pod in the hummingbird--internal namespace on mpp-prod. See the OpenShift MP+ internal docs page for general log access instructions.

For CVE analysis specifically:

  • Live: Open the mpp-prod pods list in the OpenShift console, filter for cve-analysis, and open the Logs tab.
  • Grafana/Loki: Open the Hummingbird Grafana logs dashboard, select cluster mpp-prod and namespace hummingbird--internal, then filter for hummingbird-cve-analysis pods.

Library layout

Reusable library code lives under hummingbird_cve_analysis/lib/. The CLI entry point remains cve_analysis.py at the package root.

Module Responsibility
lib/github_client.py GitHub PR and release API reads
lib/gitlab_client.py gitlab.com merge request, release, and feature-flag reads
lib/fedora.py Fedora Bodhi, DistGit spec parsing, and Bugzilla helpers
lib/upstream.py Upstream forge search, cgit commits, and fix-status analysis
lib/net.py Shared network error tuple used by library HTTP callers
lib/nvd.py NVD JSON 2.0 feed cache, download, and reference parsing
lib/cvelist.py Local cvelistV5 repository access and CVE 5.0 affected parsing
lib/jira_client.py Jira REST read/write primitives and ADF comment helpers
lib/versions.py Pure version comparison and range helpers
lib/analysis.py CVE decision pipeline, product matching, and lifecycle analysis
lib/formatting.py Human and Jira comment/output formatting
lib/resolve.py Jira/advisory mutation helpers (labels, close, attach SBOM)
lib/ticket.py Shared one-ticket analyze/resolve API
lib/advisory_handler.py CEE advisories repo clone/edit/MR helpers
lib/osidb_client.py OSIDB subpackage PURL lookups and flaw/affect created timestamps
lib/pulp.py Hummingbird Pulp repo RPM/SRPM listings and repodata lookups
lib/catalog.py Container catalog API for image publish times
lib/rpms_repo.py Local rpms git repo, package map loading, fixed-build detection
lib/sbom.py SBOM fetch, vendored dependency lookup, binary RPM confirmation
lib/slack.py Slack webhook helper
lib/version_transforms.py Named version-transform helpers

cve_analysis.py is the CLI entry point only (argument parsing, JQL building, signal handling, and the main orchestration loop). Library code lives under lib/; import and patch those modules directly. New cron jobs, webhooks, and other integrations should prefer ticket.process_ticket for analyzing (and optionally resolving) a single Jira issue:

from hummingbird_cve_analysis.lib import ticket

result = ticket.process_ticket(
    issue,
    pkg_map,
    catalog_source_map,
    github_token,
    gitlab_token,
    base_url=base_url,
    token=token,
    resolve=False,
)

Development

See the main README for development workflows.

make hummingbird-cve-analysis/setup  # Install dependencies
make check                            # Lint code (ruff)
make test                             # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.11 - Dashboard MR Linker

An AWS Lambda function that posts internal notes on newly opened GitLab merge requests linking to the Hummingbird dashboard status page. This provides an easy way for developers to check Konflux build status directly from their MRs.

Features

  • Internal Notes: Posts an internal note with dashboard link when MRs are opened (visible to project members only)
  • Configurable Projects: Only monitors specified projects via space-delimited list
  • SNS Integration: Subscribes to GitLab events via SNS with filter policy

Architecture

The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for merge_request events:

GitLab Webhook → gitlab-event-forwarder → SNS Topic → dashboard-mr-linker → GitLab API

When a new MR is opened on a configured repository, the Lambda posts an internal note:

:robot: **Hummingbird Status**

View Konflux build status for this MR: [org/group/repo!123](https://dashboard.example.com/mr/org/group/repo/123)

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)
  • GitLab API token with api scope (Reporter role or higher on monitored projects)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd dashboard-mr-linker
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabToken GitLab API token (api scope, Reporter+) (required)
GitLabUrl GitLab instance URL https://gitlab.com
DashboardBaseUrl Base URL of the dashboard (required)
GitLabProjects Space-delimited list of GitLab project paths (required)
SentryDsn Optional Sentry DSN ``

Resource naming: Lambda follows {ResourcePrefix}-lambda-linker pattern.

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Usage

Configure the GITLAB_PROJECTS parameter with space-delimited project paths:

org/group/containers org/group/rpms org/other/project

Only MRs opened on these projects will receive dashboard link notes.

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

Configuration

Lambda function receives configuration via environment variables (automatically set by CloudFormation):

Variable Description
GITLAB_TOKEN GitLab API token (api scope, Reporter+)
GITLAB_URL GitLab instance URL
DASHBOARD_BASE_URL Base URL of the dashboard
GITLAB_PROJECTS Space-delimited list of GitLab project paths
SENTRY_DSN Optional Sentry DSN
AWS_REGION AWS region (auto-set)

Security & Limitations

Security:

  • GitLab API token should have minimal required scopes (api)
  • Token stored as CloudFormation parameter with NoEcho: true
  • SNS subscription uses filter policy to only receive relevant events
  • Notes are posted as internal (visible to project members only)
  • CloudWatch logs capture all note posts (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • Only processes action: open events (not updates or other actions)
  • Skips MRs with the managed-by::workqueue label (created by workqueue-service). The workqueue-service manages its own dashboard integration for these MRs, so the Lambda backs off to avoid duplicate notes

Integration

The workqueue-service creates the dashboard note itself for MRs it manages. The managed-by::workqueue scoped label, applied by the create_mr executor, causes this Lambda to skip those MRs. The workqueue-service’s ensure_dashboard_note executor then posts the note and stores the dashboard_link dimension with the note ID.

For non-managed MRs (those not created by the workqueue-service), the Lambda remains the sole creator of the dashboard note.

Both systems use the same marker in the note body: :robot: **Hummingbird Status**. The Lambda’s COMMENT_MARKER and the workqueue-service’s DASHBOARD_LINK_MARKER constant must stay in sync — changing one without the other will break identification and may cause issues with future reconciliation or status rendering.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.12 - GitLab Event Forwarder

An AWS Lambda function that receives GitLab webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates webhook signatures and extracts minimal metadata (source, event type, project/group paths) as SNS message attributes, enabling downstream subscribers to filter events precisely.

The original GitLab JSON payload is forwarded unchanged as the SNS message body.

Features

  • Webhook Signature Validation: Verifies authenticity using HMAC-SHA256 signing tokens (Standard Webhooks), with legacy X-Gitlab-Token fallback
  • Structured Metadata: Extracts event type, project/group paths as SNS message attributes
  • SNS Subscription Filtering: Enables precise event routing to subscribers
  • Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Single Lambda function handles the webhook processing:

  1. Handler - API Gateway endpoint (/webhook) that validates GitLab webhook signatures, extracts metadata, and publishes to SNS

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd gitlab-event-forwarder
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs: ApiEndpoint - API Gateway endpoint URL (<api-endpoint>)

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabWebhookTokens GitLab webhook tokens (JSON) (required)
GitLabSigningTokens Signing tokens (JSON array) []
SentryDsn Optional Sentry DSN ``
CustomDomainName Custom domain name ``
HostedZoneId Route53 hosted zone ``

Resource naming: Lambda and API resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-lambda-handler, myapp-prod-api).

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Usage

Configure GitLab projects or groups to send webhooks to <api-endpoint>:

webhooks:
  <api-endpoint>:
    token: <secret-token>
    signing_token: <whsec-signing-token>
    push_events: true
    merge_requests_events: true
    pipeline_events: true

signing_token (recommended) uses HMAC-SHA256 verification with replay protection (±5 min tolerance) per the Standard Webhooks spec. token uses a static X-Gitlab-Token header. When signing tokens are configured on the forwarder, they take priority and webhook-signature is required on all requests.

SNS Subscription Filter Examples:

Push events from a specific project:

{
  "source": ["gitlab"],
  "event_type": ["push"],
  "project_path": ["redhat/hummingbird/containers"]
}

All merge request events:

{
  "source": ["gitlab"],
  "event_type": ["merge_request"]
}

Member events from a group:

{
  "source": ["gitlab"],
  "event_type": ["member"],
  "group_path": ["redhat/hummingbird"]
}

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

Configuration

Lambda function receives configuration via environment variables (automatically set by CloudFormation):

Variable Description
SNS_TOPIC_ARN SNS topic ARN
GITLAB_WEBHOOK_TOKENS GitLab webhook tokens (JSON array)
GITLAB_SIGNING_TOKENS Signing tokens (JSON array)
SENTRY_DSN Optional Sentry DSN
AWS_REGION AWS region (auto-set)

Event Metadata

The Lambda function extracts minimal metadata from GitLab webhook events and adds them as SNS message attributes:

Attribute Description Example
source Always "gitlab" gitlab
event_type From object_kind push, merge_request
project_path Full project path redhat/hummingbird/containers
group_path Full group path redhat/hummingbird

Security & Limitations

Security:

  • HMAC-SHA256 signing token verification via webhook-signature header (Standard Webhooks spec) with ±5 min replay protection
  • When signing tokens are configured, static X-Gitlab-Token is not accepted
  • Legacy X-Gitlab-Token validation when no signing tokens are configured
  • Supports multiple active tokens for zero-downtime rotation
  • Invalid/missing tokens return HTTP 401
  • SNS topic follows least privilege principle
  • CloudWatch logs capture all webhook deliveries (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • Webhook payload size: Up to 6 MB (API Gateway limit)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.13 - Hummingbird MR Human Tracker

A CLI tool that detects human intervention on automation-opened GitLab merge requests (Renovate, Dependabot, etc.) and links them to a per-sprint Jira epic for tracking.

Features

  • Bot MR Detection: Identifies MRs authored by bot accounts across multiple GitLab projects using configurable username patterns
  • Human Activity Detection: Finds MRs where humans have commented or pushed commits (via GitLab system notes)
  • Per-Sprint Jira Epic: Automatically finds or creates an epic per sprint to track automation fix work (e.g. “Automation MR fixes for sprint Hum S16”)
  • Remote Link Tracking: Adds affected MR URLs as remote links on the Jira epic, with deduplication to avoid duplicates on repeated runs
  • Merged/Closed MR Scanning: --since flag catches MRs that were merged or closed before the next scheduled run
  • Dry Run: Preview what would be linked without making any changes

Prerequisites

  • Python 3.11 or later
  • GitLab API token with read access to target projects (including MR notes)
  • Jira API token with permission to create epics and remote links in the HUM project

Installation

pip install -e hummingbird-mr-human-tracker

Usage

mr-human-tracker --jira-user user@redhat.com --dry-run -v

Common invocations

# Preview what would be linked (no changes made)
mr-human-tracker --jira-user user@redhat.com --dry-run -v

# Run for real, including MRs merged in the last 2 days
mr-human-tracker --jira-user user@redhat.com --since 2d

# Scan only specific projects
mr-human-tracker --jira-user user@redhat.com --projects redhat/hummingbird/tools

# Verbose output for debugging
mr-human-tracker --jira-user user@redhat.com --since 1w -v

Output

On completion, the tool prints a summary of linked MRs:

INFO hummingbird_mr_human_tracker.tracker: Active sprint: Hum S16 [June 11 - June 25] (id=68851)
INFO hummingbird_mr_human_tracker.tracker: Using epic HUM-2602
INFO hummingbird_mr_human_tracker.tracker: Linked 2 MR(s) to HUM-2602:
INFO hummingbird_mr_human_tracker.tracker:   https://gitlab.com/redhat/hummingbird/rpms/-/merge_requests/2395
INFO hummingbird_mr_human_tracker.tracker:   https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/8476

Configuration

All configuration is via CLI arguments and environment variables.

CLI arguments

Argument Description Default
--gitlab-url GitLab instance URL GITLAB_URL env
--gitlab-token GitLab private token GITLAB_TOKEN env
--jira-url Jira base URL JIRA_URL env
--jira-token Jira API token JIRA_TOKEN env
--jira-user Jira email for Basic auth (omit for Bearer) None
--projects GitLab project paths to scan tools, rpms, containers
--board-id Jira board ID for sprint lookup 1489
--sprint-prefix Sprint name prefix to match Hum S
--since Also scan merged/closed MRs updated within window None (open MRs only)
--dry-run Report without making changes False
-v, --verbose Enable debug logging False

Environment variables

Variable Description
GITLAB_URL GitLab instance URL
GITLAB_TOKEN GitLab private token
JIRA_URL Jira base URL
JIRA_TOKEN Jira API token

Duration format for --since

The --since flag accepts a number followed by a unit:

Unit Meaning
h Hours
d Days
w Weeks

Examples: 12h, 2d, 1w

How it works

  1. Fetch active sprint from the Jira Agile API (board 1489, prefix “Hum S”)
  2. Find or create a Jira epic named “Automation MR fixes for sprint {sprint_name}” in the HUM project, assigned to the active sprint
  3. Scan GitLab projects for open MRs authored by bot accounts; if --since is set, also scan recently merged/closed MRs
  4. Detect human activity on each bot MR by checking:
    • Non-bot, non-system comments
    • System notes indicating a human pushed commits (“added N commit”)
    • Reopen events by non-bot users
  5. Add remote links on the Jira epic for each human-touched MR (skipping URLs already linked)

Bot detection

A username is considered a bot if it matches any of:

  • Contains bot_ or bot- (e.g. renovate_bot)
  • Ends with _bot (e.g. some_bot)
  • Ends with [bot] (e.g. renovate[bot], dependabot[bot])
  • Starts with project_<digits>_bot or group_<digits>_bot (GitLab service accounts)

Development

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.14 - MR Auto-Approver

An AWS Lambda function that auto-approves GitLab merge requests based on configurable per-project rules. Replaces CI/CD-based approval jobs and moves approval tokens out of GitLab CI/CD variables.

Features

  • Rules-Based Config: YAML config with per-project ordered rules matching on user IDs, username patterns, branch patterns, and Konflux pipeline status
  • First-Match-Wins: Rules evaluated in order; first match determines action (approve or deny)
  • Konflux Integration: Optional check that Konflux pipelines have posted commit statuses before approving
  • Fork Rejection: Unconditionally rejects MRs from forked projects
  • User Verification: Matches against the authenticated webhook event user, not forgeable Git commit metadata
  • Stateless & Idempotent: No queues or stored state; re-approving is a no-op

Architecture

The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for merge_request and pipeline events:

GitLab Webhook → gitlab-event-forwarder → SNS Topic → mr-auto-approver → GitLab API

Merge request events (open, update): evaluate rules using the authenticated pusher from the webhook payload. If matched and no Konflux check needed, approve immediately. If Konflux check needed, check statuses now and approve only if present and not failed.

Pipeline events (success, failed): look up the MR from the pipeline event via the GitLab API, evaluate rules, and approve if Konflux statuses are ready. This handles the case where Konflux posts commit statuses after the initial MR event.

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)
  • GitLab API token(s) with api scope and permission to approve MRs on target projects
  • Target project webhooks must send merge_requests_events and pipeline_events to the gitlab-event-forwarder endpoint

Deployment

Build and deploy using containerized AWS SAM CLI:

cd mr-auto-approver
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabUrl GitLab instance URL https://gitlab.com
ConfigPath Path to YAML config (bundled with Lambda) config.yaml
ApprovalTokens JSON map of env var names to tokens {}
SentryDsn Optional Sentry DSN ``

Resource naming: Lambda follows {ResourcePrefix}-lambda-approver pattern.

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Configuration

The Lambda uses a YAML config file bundled at deploy time. The config defines per-project rules:

settings:
  gitlab_url: https://gitlab.com

projects:
  org/group/repo:
    token_env: APPROVAL_GITLAB_TOKEN_REPO
    rules:
      - branch_regexes: ["renovate/skip/.*"]
        action: deny
      - user_ids: [12345678]
        branch_regexes: ["renovate/.*"]
      - user_ids: [87654321]
        check_konflux: true

Rule fields

Field Type Default Description
user_ids list[int] [] GitLab user IDs (immutable, preferred)
user_regexes list[str] [] Fullmatch regexes for username
branch_regexes list[str] [] Fullmatch regexes for source branch
check_konflux bool false Require Konflux statuses before approval
action str approve approve or deny

Match logic: AND across field types, OR within a field. Empty fields match anything. First matching rule wins. user_ids and user_regexes are both user identity constraints – if either is set, the user must match at least one entry from across both lists. Prefer user_ids over user_regexes to prevent username confusion attacks.

Adding a new project

  1. Create a project-scoped GitLab token (Developer, api scope) in Vault
  2. Add the token to vars.sh and the jq command that builds APPROVAL_TOKENS
  3. Add a project entry to config.yaml in the infrastructure repo with token_env and rules
  4. Deploy the Lambda
  5. Ensure the project’s GitLab webhook sends events to the gitlab-event-forwarder endpoint

Environment variables

Variable Description
CONFIG_PATH Path to YAML config file
GITLAB_URL GitLab instance URL
APPROVAL_TOKENS JSON object mapping env var names to GitLab tokens
SENTRY_DSN Optional Sentry DSN

Security

  • Fork MRs are unconditionally rejected before any rule evaluation (source_project_id != target_project_id)
  • User matching supports immutable user.id (preferred) in addition to user.username from the webhook payload, preventing username confusion attacks. Never uses Git commit author/committer metadata
  • For pipeline events, the Lambda looks up head_pipeline.user (both id and username) via the GitLab API to verify the last pusher
  • Approval tokens stored as CloudFormation parameters with NoEcho: true
  • SNS subscription filter policy limits events to merge_request and pipeline
  • CloudWatch logs capture all approval decisions (7-day retention)

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.15 - Container Catalog

Per-distro serverless catalog API for Hummingbird container images.

Features

  • Image Directory - Browse all container images with metadata
  • Tag Browser - View tags, digests, architectures per image
  • Specifications - Per-architecture OCI config details (env, cmd, user, labels)
  • SBOM - Per-architecture package lists from SPDX attestations
  • Vulnerabilities - CVE scanning results via Grype
  • CVE Metrics - Vulnerability aggregate and structured exposure logs
  • Provenance - Source traceability from SLSA attestations
  • Release History - Timeline of past builds with drill-down
  • Deprecation status - Stream and release status from OCI labels
  • OpenAPI Spec - Machine-readable API documentation at /v1/openapi.json
  • Swagger UI - Interactive API explorer at /v1/docs/

Architecture

Rust serverless stack deployed as two isolated per-distro stacks:

  • API Lambda - DynamoDB pass-through (~10ms response)
  • Sync Lambda - Incremental DynamoDB sync from SNS Release events (~22 registry calls per release)
  • Index Lambda (index-lambda) - SNS-triggered Syft JSON generation, uploads native Syft JSON to S3 for scanner consumption
  • Scan Lambda (scan-lambda) - SQS-triggered CVE scanning via Grype from native Syft JSON in S3, per-digest processing with first_seen tracking and partial batch failure reporting
  • Enqueue Lambda (enqueue-lambda) - Hourly EventBridge-triggered fan-out that checks Grype DB for updates and enqueues non-superseded digests to SQS
  • Metrics Lambda (metrics-lambda) - DynamoDB Stream-triggered vulnerability aggregate and structured CVE exposure logs
  • DynamoDB - Pre-computed JSON items (single-table PK/SK design, Streams: KEYS_ONLY)
  • S3 ScanDataBucket - Native Syft JSON storage for scanner data (gzipped, keyed by grype/{image}/{digest_hex}.json.gz)
  • SQS ScanQueue - Central queue for scan tasks (fed by S3 events and Enqueue Lambda)
  • CloudFront - CDN with per-endpoint cache TTLs
  • CloudWatch - Structured CVE exposure logs
  • catalog sync - Full DynamoDB population from GitLab + Quay.io OCI v2 registry
  • catalog scan - CLI CVE scanning via Grype; reads native Syft JSON from S3 (with --bucket) or builds synthetic JSON from DynamoDB SBOMs (fallback)
  • catalog index - CLI Syft JSON generation for backfilling S3
  • Catalog SPA - Lit 3 web app served from S3 via CloudFront

Scanner Architecture

CVE vulnerability data must match direct grype <image> scans exactly for all images. Synthetic Syft JSON (built from stored SBOM packages) cannot reliably reproduce the native output because Grype relies on metadata fields, artifact relationships, and deduplication logic that are lost in the SPDX-to-API roundtrip. Additionally, raw SPDX from the build system contains package bloat (e.g. Go sub-modules, empty-version entries) that native Syft filters out when scanning a compiled binary directly.

To guarantee exact-match results, the scanner chain stores native Syft JSON in S3 rather than DynamoDB: a single image’s Syft JSON is typically 1-10 MB (gzipped to 100 KB - 1 MB), which exceeds DynamoDB’s 400 KB item limit. S3 has no per-object size constraint, avoids provisioned throughput costs for large blobs, and supports event notifications for triggering downstream scanners. An independent Index Lambda runs syft <image> on each new release and uploads the full output to s3://{bucket}/grype/{image}/{hex}.json.gz. The S3 upload fires an event notification to SQS, which triggers the Scan Lambda to read the native JSON, run Grype, and write per-canonical vulnerability data to DynamoDB.

The scanner chain is intentionally independent of the catalog data chain: the same SNS Release event triggers both the Sync Lambda (catalog data to DynamoDB) and the Index Lambda (Syft JSON to S3), with no ordering dependency between them.

CVE Data Flow

graph TD
    Konflux([Konflux Release]) -->|SNS| Sync[SyncFunction]
    Konflux -->|SNS| Index[IndexFunction]
    Sync -->|update| Tags[(Tags)]
    Tags -->|"stream via ESM"| Metrics
    Tags -->|read| Scan[ScanFunction]
    Index -->|upload| S3Syft[(S3 Syft JSON)]
    S3Syft -->|"S3 event via SQS"| Scan
    Hourly([Schedule]) -->|hourly| Enqueue[EnqueueFunction]
    Enqueue --> Gate[check Grype DB]
    Gate -->|"fan out via SQS"| Scan
    Scan -->|update| Vulns[(Image Vulnerabilities)]
    Vulns -->|"stream via ESM"| Metrics[MetricsFunction]
    Metrics -->|"structured logs"| CloudWatch[CloudWatch Logs]
    Metrics -->|"write aggregate"| CatalogVulns[(Catalog Vulnerabilities)]

    classDef lambda fill:#d4e6f1,stroke:#2980b9
    classDef dynamo fill:#fdebd0,stroke:#e67e22
    classDef trigger fill:#d5f5e3,stroke:#27ae60
    classDef aws fill:#e8daf1,stroke:#8e44ad
    classDef gate fill:#e5e7e9,stroke:#7f8c8d
    class Sync,Scan,Enqueue,Metrics,Index lambda
    class Vulns,Tags,CatalogVulns,S3Syft dynamo
    class Konflux,Hourly trigger
    class CloudWatch aws
    class Gate gate

Prerequisites

  • Rust 1.75+ (for building backend)
  • Node.js 22+ (for building frontend)
  • AWS credentials (for DynamoDB access and deployment)
  • SAM CLI (for deployment)

API Endpoints

Endpoint Description
GET /v1/images Image directory
GET /v1/images/{name} Image overview (README)
GET /v1/images/{name}/tags Tags for an image
GET /v1/images/{name}/details/{canonical} Per-canonical details
GET /v1/images/{name}/sbom/{canonical} Package list
GET /v1/images/{name}/vulnerabilities/{canonical} Vulnerability scan
GET /v1/images/{name}/history/{stream}/{variant} Release timeline
GET /v1/images/{name}/releases/details/{digest} Release details (immutable)
GET /v1/images/{name}/releases/sbom/{digest} Release SBOM (immutable)
GET /v1/images/{name}/releases/vulnerabilities/{digest} Release vulnerabilities
GET /v1/vulnerabilities Catalog-wide CVE aggregate
GET /v1/openapi.json OpenAPI 3.1 specification
GET /v1/docs/ Interactive Swagger UI

Timestamp Fields

The oldest_created field on ImageSummary, Tag, and HistorySummary is the earliest OCI created timestamp across all architectures in the release. All architectures were built at or after this date, making it useful for conservative staleness detection.

The specifications endpoint returns per-architecture data keyed by architecture name. Each architecture’s created field is the direct OCI config root timestamp for that specific architecture.

Deprecation status

The containers repository can mark an image stream with the io.hummingbird-project.deprecated=true OCI label. The catalog exposes this status in two places:

  • Tag.deprecated identifies a release carrying the label.
  • ImageSummary.deprecated_streams lists streams whose current release is deprecated.

Both fields are additive API fields. Older DynamoDB items without these fields are read as active releases with no deprecated streams.

Usage

All tools are built as a single catalog binary with subcommands (api, sync, sync-lambda, scan, scan-lambda, index, index-lambda, enqueue-lambda, metrics, metrics-lambda). The binary is built in the Rust container and CLI subcommands are run in the gitlab-ci container (which provides grype, syft, and other tools). Only make and podman are required.

Sync

# Dry run (print items to stdout)
make container-catalog/sync ARGS="--distro rawhide --dry-run"

# Populate DynamoDB
make container-catalog/sync ARGS="--distro rawhide --table-name <table>"

Sync Lambda

The sync-lambda subcommand runs as an AWS Lambda function triggered by SNS Release events from kubernetes-event-forwarder. It incrementally syncs a single image release to DynamoDB (~22 registry API calls per release vs ~104 for a full sync).

The Lambda:

  1. Decodes gzip+base64 SNS messages
  2. Filters for Succeeded releases targeting the configured Quay.io namespace
  3. Fetches OCI manifest data for the new digest
  4. Writes per-digest items (DETAILS, SBOM, RELEASE_DETAILS, RELEASE_SBOM)
  5. Merges into aggregate items (TAGS, HISTORY, OVERVIEW, DIRECTORY)
  6. Fetches README from GitLab for OVERVIEW content (uses README.redhat.md for hummingbird, README.md for rawhide)

Registry fetch errors (manifest, SBOM, attestation) propagate as hard failures so the Lambda retries automatically (up to 2 retries with backoff) before sending to the DLQ. GitLab README failures are non-fatal – the existing README is preserved if the fetch fails.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Index

The index subcommand generates native Syft JSON for all non-superseded image digests and uploads them to S3 for scanner consumption. It reads the image directory and TAGS from DynamoDB, checks S3 for existing objects (dedup via HeadObject), runs syft <image> --platform linux/amd64, gzips the output, and uploads to s3://{bucket}/grype/{image}/{hex}.json.gz.

# Dry run (show what would be uploaded)
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --dry-run"

# Backfill S3 for all images
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket>"

# Backfill a single image
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --image caddy"

Index Lambda

The index-lambda subcommand runs as an AWS Lambda function triggered by the same SNS Release events as the Sync Lambda. It operates independently of the catalog chain (no DynamoDB access) and generates native Syft JSON for scanner consumption.

The Lambda:

  1. Decodes gzip+base64 SNS messages (shared with Sync Lambda via release_event module)
  2. Filters for Succeeded releases targeting the configured Quay.io namespace
  3. Checks S3 for existing objects (HeadObject dedup by digest)
  4. Runs syft <image>@<digest> --platform linux/amd64 --output syft-json
  5. Gzips and uploads to s3://{bucket}/grype/{image}/{hex}.json.gz

Failures propagate for Lambda retry (up to 2 retries) before sending to the IndexDLQ. The catalog index CLI backfills any gaps.

Environment Variable Description
SCAN_DATA_BUCKET S3 bucket for scanner data
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Scan

The scan subcommand reads image listings, tags, and SBOMs from DynamoDB (no registry access needed) and runs Grype against each image’s stored SBOM packages. Results include a first_seen timestamp per CVE, tracked at the group+variant+stream level and carried across releases for SLI computation.

# Dry run (print items to stdout)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --dry-run"

# Scan and write to DynamoDB (purge stale vuln data first, implies --scope=all)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --purge"

# Scan only non-superseded (current) tags
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --scope non-superseded"

# Scan all releases including historic (tagless) releases
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --scope all"

# Scan a single image
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --image caddy --dry-run"

Scan Lambda

The scan-lambda subcommand runs as an AWS Lambda function triggered by SQS messages. Two paths feed the ScanQueue:

  1. Real-time (S3 event notification): When the Index Lambda uploads a new Syft JSON to S3, an s3:ObjectCreated event (filtered on grype/ prefix) is sent directly to SQS
  2. Hourly (Enqueue Lambda): Fans out all non-superseded digests when the Grype vulnerability database has been updated, sending {"bucket": "...", "key": "grype/{image}/{hex}.json.gz"} messages

The Lambda accepts both S3 event notification JSON and direct {"bucket", "key"} messages. For each message it:

  1. Downloads and decompresses the native Syft JSON from S3
  2. Runs Grype on the raw bytes
  3. Looks up all canonical tags matching the digest from the TAGS item
  4. Writes VULNERABILITIES#{canonical} for each matching tag (with first_seen tracking) and RELEASE_VULNERABILITIES#{hex} once per digest

Processing is per-digest: a single Syft JSON serves all canonicals sharing that digest, avoiding redundant Grype invocations.

The Lambda loads the Grype DB on cold start (cached in /tmp for warm invocations), processes up to 10 messages per batch, and reports partial batch failures so only failed records return to the queue.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Enqueue Lambda

The enqueue-lambda subcommand runs hourly via EventBridge Schedule. On each invocation it performs a lightweight HTTP GET of the public Grype DB listing (latest.json, ~200 bytes) and compares the built timestamp against the stored CATALOG/LAST_FULL_SCAN_DB item in DynamoDB. If the DB hasn’t changed, it returns early (~23 of 24 hourly invocations short-circuit). When an update is detected, it reads the image directory and all TAGS items, deduplicates by digest, and sends one SQS message per unique digest using SendMessageBatch. Messages use the format {"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SCAN_QUEUE_URL SQS queue URL for scan messages
SCAN_DATA_BUCKET S3 bucket for scanner data
GRYPE_DB_LATEST_URL Grype DB listing URL (has sensible default)
SENTRY_DSN Optional Sentry DSN for error tracking

Metrics

The metrics subcommand performs a one-shot read of all non-superseded vulnerability data from DynamoDB, outputs structured CVE exposure logs, and writes the catalog-wide vulnerability aggregate to DynamoDB (PK=CATALOG, SK=VULNERABILITIES). With --dry-run, it skips the DynamoDB aggregate write (useful for local inspection).

# Dry run (print structured logs to stdout)
make container-catalog/metrics ARGS="--distro hummingbird --table-name <table> --dry-run"

# Write aggregate to DynamoDB and print structured logs
make container-catalog/metrics ARGS="--distro hummingbird --table-name <table>"

Metrics Lambda

The metrics-lambda subcommand runs as a DynamoDB Stream-triggered Lambda that writes a catalog-wide vulnerability aggregate to DynamoDB (served by GET /v1/vulnerabilities) and emits structured CVE exposure logs to CloudWatch Logs.

How It Works

The Lambda is triggered by DynamoDB Stream events filtered on VULNERABILITIES# and TAGS changes, with a 60-second batching window and reserved concurrency of 1 (single instance). It maintains an in-memory active CVE table across warm invocations:

  • Cold start: Reads CATALOG/DIRECTORY, all TAGS, and all non-superseded VULNERABILITIES# items from DynamoDB to build the full table (~2-3s at 10000 tags)
  • Warm invocations: Incrementally updates the table from stream event keys (~50-100 GetItem calls per batch)
  • After each invocation: Recomputes aggregate and emits structured logs

Structured Logs

Each invocation emits one JSON log line per active CVE to stdout (captured by CloudWatch Logs). Example query for all active CVEs:

filter message = "active_cve"
| fields cve, severity, exposure_hours, repository, stream, variant, component
| sort exposure_hours desc
Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN

Deployment

make container-catalog/build
make container-catalog/deploy

Configuration

catalog sync

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name
--purge Delete all items before writing
--cache-dir Cache directory (auto-detected)
--image Sync only a specific repo
--legacy-discovery Use GitLab-based repo discovery

catalog index

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name (for image/tag discovery)
--bucket S3 bucket for scanner data
--dry-run Print what would be uploaded without uploading
--image Index only a specific image
--parallel Number of concurrent operations (default: 2)

catalog scan

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name (required)
--bucket S3 bucket with native Syft JSON (uses S3 scan path)
--scope non-superseded, tags (default), or all
--dry-run Print items without writing
--purge Purge vuln data before writing (implies --scope all)
--cache-dir Cache directory (auto-detected)
--parallel Number of concurrent scans (default: 4)
--image Scan only a specific image
--tag Scan only a specific tag (requires --image)

SAM Parameters

Parameter Description
Distro rawhide or hummingbird
CacheEnabled Enable CloudFront caching
CatalogDomainName Catalog web UI domain
ApiDomainName API domain
HostedZoneId Route53 hosted zone
CorsOrigins Comma-separated CORS origins (default *)
SnsTopicArn SNS topic ARN for Release events (enables sync Lambda)

Frontend

The catalog web UI is a Lit 3 SPA (Web Components) with Tailwind CSS, built per-distro with Vite. Source is in container-catalog/frontend/.

Only make and podman are required (no local Node.js needed). Defaults from .envrc.defaults are applied automatically.

# Install dependencies
make container-catalog/frontend/setup

# Development server at http://localhost:5173
make container-catalog/frontend/dev

# Production build
make container-catalog/frontend/build

Host variants (*-host) run without podman (for CI or local Node.js).

Frontend Build Variables

Variable Description
VITE_API_URL API base URL for the distro
VITE_DISTRO rawhide or hummingbird
VITE_DISTRO_LABEL Display label for current distro
VITE_OTHER_CATALOG_URL URL of the other distro’s catalog (optional, hides link if unset)
VITE_OTHER_DISTRO_LABEL Display label for other distro (optional)
VITE_VULNERABILITIES_ENABLED Show vulnerabilities tab

Development

# Backend
cargo test                    # Run tests
cargo clippy --all-targets   # Lint
cargo fmt                     # Format

# Frontend (host variants, requires local Node.js)
cd container-catalog/frontend
npm run typecheck             # Type check
npm run build                 # Production build

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.16 - Hummingbird Agent

An event-driven LLM agent that investigates CI/CD failures and posts findings as GitLab merge request notes. The agent executes markdown-defined workflows using tool calling, with all data processing running inside an isolated sandbox container.

For the architectural design rationale, module boundaries, security invariants, and design decision registry, see Agent Design. For the model loop wire format, see Agent Model Loop.

flowchart TD
    Pipeline["Pipeline Fails"]
    MREvent["MR Created / Updated"]
    Slash["/hummingbird command"]

    Pipeline -->|"event"| Agent
    MREvent -->|"event"| Agent
    Slash -->|"command"| Agent

    subgraph Agent ["Hummingbird Agent"]
        APIs["GitLab + Konflux +<br/>Testing Farm"]
        Model["LLM<br/>(Gemini / Claude)"]
        subgraph Sandbox ["Isolated Sandbox (no network)"]
            Tools["jq / python3 / yq<br/>data processing"]
        end
        APIs <-->|"data"| Model
        Model <-->|"commands"| Tools
    end

    Note["MR Note<br/>(analysis or review)"]

    Agent -->|"posts"| Note
    Note -->|"reply to continue"| Agent

Features

  • Workflow-driven analysis - Investigation logic lives in .md files, not in code; easy to iterate without redeployment
  • Centralized YAML config - A single config file defines operational settings, workflows, enabled data sources with token env var names, project allowlists, and per-project limits
  • Sandboxed execution - All untrusted commands (jq, python3, shell) run in an isolated container, never on the host
  • Five sandbox backends - Podman for local development (network-isolated), direct K8s pod creation, Deployment-backed pod pool for low-latency production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or VMIRS-backed VM pool for heavier workloads requiring full OS isolation
  • Data source abstraction - GitLab, Konflux, and Testing Farm are registered as tool-calling functions the model invokes directly
  • Auto-spill for large outputs - Stdout/stderr and data source responses exceeding 4 KB are automatically saved to sandbox files with a compact preview returned to the model, keeping context usage bounded
  • Prompt caching - Gemini uses implicit server-side caching automatically; Claude uses explicit sliding-window cache breakpoints that reduce input token costs by ~80% on multi-turn agent runs
  • Token budget management - Per-call context ceiling and iteration-based soft/hard limits prevent runaway sessions
  • Session persistence - Conversation history, transcript, and sandbox files saved to S3 (production) or local directory (development) for debugging and future session resumption. Sessions are stored in a provider-neutral format, enabling model switching between conversations.

Architecture

flowchart LR
    subgraph input [Input]
        SQS["SQS Queue"]
        CLI["CLI --event"]
    end

    subgraph agentLoop [Agent Loop]
        WF["Workflow .md<br/>(system prompt)"]
        LLM["LLM&lt;br/&gt;(Gemini / Claude)"]
        TR["Tool Registry"]
    end

    subgraph tools [Tools]
        SE["sandbox_exec"]
        FTS["fetch_to_sandbox"]
        DS["Data Sources"]
    end

    subgraph sandbox [Sandbox Container]
        JQ["jq / python3 / yq"]
        Files["Spilled files"]
    end

    subgraph external [External APIs]
        GL["GitLab API"]
        KX["Konflux K8s"]
        TF["Testing Farm"]
    end

    subgraph output [Output]
        Note["GitLab MR Note"]
        Session["Session State"]
    end

    SQS --> WF
    CLI --> WF
    WF --> LLM
    LLM -->|"tool calls"| TR
    TR --> SE --> sandbox
    TR --> FTS --> sandbox
    TR --> DS
    DS --> GL
    DS --> KX
    DS --> TF
    DS -->|"auto-spill"| sandbox
    LLM -->|"final text"| Note
    LLM --> Session

An event (CLI --event JSON or SQS message) identifies a GitLab project and MR IID. The config file maps project paths to workflows and provides action, max_iterations, enabled data sources, and token env var names. Both run and serve use this config. The markdown body of the prompt file becomes the LLM system prompt. The agent loop iterates until the model produces a final text response or hits the iteration/context limit. Tool calls are dispatched through the ToolRegistry: sandbox_exec runs shell commands, fetch_to_sandbox pipes data source output into the sandbox, and direct data source calls return results inline (or auto-spill large responses to files).

For a detailed walkthrough of the model loop – what gets sent to the model each iteration, how tool calls flow, the exact wire format, and how user replies integrate for session resumption – see Agent Model Loop.

Event-Driven Triggers

In production, the agent consumes events from an SQS queue subscribed to the central SNS topic. The SNS filter policy delivers three event types:

  • gitlab::pipeline – Fires when a GitLab CI pipeline completes. The agent triggers on status=failed pipelines from merge_request_event sources where the triggering user has at least Developer access on the project. Only workflows with trigger: pipeline are executed. Since the pipeline stays open until all Konflux external stages resolve, this naturally waits for all builds and tests to finish before triggering.

  • gitlab::merge_request – Fires on MR open and update events. Only workflows with trigger: merge_request are executed. The agent triggers in three cases: a new MR is opened (action=open), code is pushed to an existing MR (action=update with oldrev), or a draft MR is marked as ready (action=update with a changes.draft transition from true to false). All other update events (labels, assignees, description changes) are skipped – they carry no new code. Draft MRs are skipped; SHA-based deduplication prevents reviewing the same code revision twice. Access checks and trigger_rules filtering evaluate the push author: for open and push events the webhook user is the pusher; for undraft transitions the agent resolves the actual pusher from MR system notes so that the person undrafting cannot bypass the access check on code pushed by an untrusted user.

  • gitlab::note – MR comment events. Two sub-flows:

    • Slash command (/hummingbird <workflow-name>): triggers a specific workflow. Prefix matching is supported (e.g. /hummingbird analyze matches analyze-failures). /hummingbird or /hummingbird help lists available workflows. The note author must have Developer+ access on the project. Optional runtime overrides can be appended: /hummingbird code-review model=claude-sonnet-4-6 max_iterations=25.
    • Reply to agent note: when a user replies to an existing agent note (which contains a session marker), the agent loads the previous session from S3 (conversation history + sandbox files) and continues the conversation with the user’s reply as input. The system prompt includes a CONTINUATION_PROMPT that prevents the model from re-running the full workflow. If the session is not found (expired/deleted), the agent falls back to a cold start. Replies may include overrides on a separate line (e.g. /hummingbird model=claude-opus-4-6); override lines are stripped from the user message. Overrides persist in the session until explicitly changed.

    Notes generated by the agent itself (containing session markers) are skipped to prevent infinite loops. Reply threading respects the internal_notes config: if the project requires internal notes but the original thread was public, the reply is posted as a new top-level internal note instead.

Both pipeline and merge_request triggers apply per-workflow trigger rules filtering. Each workflow may define an ordered trigger_rules array; the first matching rule decides whether the event is allowed or denied (implicit deny on fallthrough). Rules can match on pipeline status, user_regex/user_regexes, branch_regex/branch_regexes, and title_regex/title_regexes. All conditions within a rule are ANDed; multiple patterns within a regex field are ORed. The ! prefix negates a pattern. If no trigger_rules are specified, merge_request triggers allow all events and pipeline triggers default to allowing only failed status (preserving backward compatibility).

The legacy ignore_users and ignore_branches fields are still accepted and automatically desugared into equivalent trigger_rules (deny + catch-all allow). They cannot be mixed with explicit trigger_rules.

Rate limiting is per-workflow: each workflow’s thread count is tracked independently via JSON session markers that embed the workflow name and commit SHA. The max_runs_per_mr limit applies separately to each workflow on a given MR.

Events flow through a two-stage SQS pipeline. A slim ingress router forwards webhooks from the standard queue to an SQS FIFO queue, grouped by discussion_id for note events (ensuring same-discussion ordering) and by SQS MessageId for pipeline/MR events (no serialization needed). Phase 1 handlers validate the event and resolve the session, then post a continuation message back to the FIFO grouped by session_id. Phase 2 picks up the continuation and runs the workflow. This ensures all work for a given session is serialized even across multiple discussion threads.

The SQS infrastructure is defined in template.yaml (SAM/CloudFormation): a standard ingress queue (60s visibility timeout for the router hop) and a FIFO work queue (30-minute visibility timeout for workflow execution).

Model Configuration

The agent supports multiple LLM providers via a unified adapter interface.

Gemini

  • API key (local dev) – Set GOOGLE_API_KEY. Calls generativelanguage.googleapis.com directly. No region configuration needed.
  • Vertex AI (production) – Set GOOGLE_CLOUD_PROJECT and configure model_regions in the YAML settings. Uses Application Default Credentials (ADC) via google-auth. For OpenShift, mount a service account key JSON file and set GOOGLE_APPLICATION_CREDENTIALS, or use workload identity.

Gemini uses implicit server-side caching – repeated prefixes are automatically cached by the Vertex AI backend with no opt-in required. Cached input tokens are billed at 25% of the base input rate. The agent tracks cached token counts from API responses for cost estimation.

Anthropic Claude

Claude models are accessed via Vertex AI using the same GCP project (GOOGLE_CLOUD_PROJECT) with the region resolved from model_regions. Enable the desired model in the GCP Model Garden. Use model: claude-sonnet-4-20250514 in workflow config.

Prompt caching is enabled automatically for Claude models. The agent places explicit cache breakpoints on messages so that the full conversation prefix is served from cache on every turn after the first. Cache writes cost 1.25x the base input rate; cache reads cost 0.10x (90% discount). For a typical 20-iteration agent run, this reduces input token costs by approximately 80%. Ephemeral messages (iteration warnings, nudges) are excluded from cache writes to avoid polluting the cache with transient content.

Region configuration

GCP regions are configured via settings.model_regions in the config file – a map of model name prefixes to GCP regions. The agent resolves the region by longest-prefix match on the effective model name (same algorithm as cost estimation). Example:

settings:
  model_regions:
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global

At least one of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT must be set. If both are set, the API key takes precedence for Gemini models. Claude models always require Vertex AI mode (GOOGLE_CLOUD_PROJECT + model_regions); GOOGLE_API_KEY direct mode is for Gemini only.

Prerequisites

  • Python 3.11+
  • Podman (for local sandbox) or kubectl (for K8s sandbox)
  • Authentication: GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT (see above)
  • For Claude models: enable the desired model in GCP Model Garden
  • GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens)
  • Kubeconfig with access to the Konflux cluster (if using Konflux data sources)

Installation

cd hummingbird-agent
pip install -e .

Usage

Local development (run)

The run command uses the config file for workflow lookup, data source registration, and token resolution – the same code path as serve. By default, results are printed to stdout (dry-run). Use --execute to post the result as a GitLab MR note.

There are two ways to select what to run:

Direct workflow selection (--workflow + --project + --event):

# Run a specific workflow on an MR, Podman sandbox, print to stdout
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}'

# Same but with K8s sandbox
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --context my-cluster/my-namespace

# Post the result as a GitLab MR note (also saves session to S3 if configured)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --execute

# Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --save-session /tmp/my-session

# Resume from a saved session with a follow-up question
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --resume-session /tmp/my-session \
    --message "Can you look at the clair-scan timeout more closely?"

# Chain: resume and save the new session for another round
hummingbird-agent run \
    --event-file event.json \
    --resume-session /tmp/my-session \
    --message "What layer hash failed?" \
    --save-session /tmp/my-session-2

Event-file replay (--event-file): routes the event through the same project-index lookup as serve, but skips rate limiting and status filtering:

# Replay a real webhook event, dry-run with Podman
hummingbird-agent run \
    --event-file event.json

# Replay with K8s sandbox and post notes
hummingbird-agent run \
    --event-file event.json \
    --context my-cluster/my-namespace \
    --execute

Production (serve)

# Poll SQS queue for events (pool sandbox by default, posts results as MR notes)
CONFIG_PATH=config.yml hummingbird-agent serve

Requires CONFIG_PATH pointing to a config file with settings.sqs_queue_url and settings.sqs_fifo_queue_url set. The config file defines which workflows run on which projects, with per-project limits and data source token mappings. Handles SIGTERM/SIGINT for graceful shutdown. A background router thread forwards events from the standard queue to the FIFO; the main thread consumes the FIFO with a semaphore-gated thread pool (controlled by settings.max_concurrent_agents) so excess messages stay in SQS for other instances.

Config hot-reload: In serve mode a background thread polls the config file for changes (every 5 seconds by default). When the file changes, the new config is validated and atomically swapped in – subsequent event dispatches use the updated config. If the new config is invalid, the previous config is kept and a warning is logged. No restart required for config changes.

The serve command also accepts --sandbox, --context, and --namespace for local development with a different sandbox backend (e.g. Podman).

CLI Options

run subcommand:

Option Description Default
--event Inline event JSON string (mutually exclusive with --event-file) -
--event-file Read event from a JSON file (mutually exclusive with --event) -
--workflow Workflow name from config file (requires --project) -
--project GitLab project path (requires --workflow) -
--execute Post result as GitLab MR note and save session to S3 -
--save-session Save session artifacts to this directory -
--resume-session Resume from a saved session directory (requires --message) -
--message Follow-up message for session resumption (requires --resume-session) -
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) podman
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

serve subcommand:

Option Description Default
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) k8spool
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

Configuration

Config file

Both run and serve use a single YAML config file. Set the path via CONFIG_PATH (default: config.example.yaml). The config has two sections: settings for operational parameters, and workflows for workflow definitions. The settings section provides defaults that can be omitted for local development (sensible defaults are used).

settings:
  gitlab_url: https://gitlab.com                # GitLab instance URL
  sandbox:                                      # sandbox pod configuration
    image: quay.io/.../gitlab-ci:latest         #   container image (k8s mode only)
    namespace: default                          #   K8s namespace (required for K8s/pool/kubevirt)
    active_deadline_seconds: 1800               #   pod hard timeout / reap interval
    linger_seconds: 300                         #   keep pod alive after success (pool mode, 0=disable)
    max_lingering_pods: 2                       #   max idle lingering pods before eviction (pool mode)
    vm_image: quay.io/.../vm-disk:latest        #   containerDisk image (kubevirt mode only)
    vm_memory: 1Gi                              #   VM guest memory (kubevirt mode only)
    vm_ssh_private_key: /path/to/key            #   SSH private key for pool VMs (kubevirtpool mode)
    metadata:                                   #   pod/VMI metadata (k8s/kubevirt mode)
      labels:
        app.kubernetes.io/name: hummingbird-agent-sandbox
    resources:                                  #   K8s resource requests/limits (k8s mode)
      requests:
        cpu: "100m"
        memory: "256Mi"
        ephemeral-storage: "256Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
        ephemeral-storage: "2Gi"
  max_concurrent_agents: 4                      # max concurrent workflows (serve)
  sqs_queue_url: ""                             # standard SQS ingress queue URL (serve)
  sqs_fifo_queue_url: ""                        # FIFO work queue URL (serve)
  s3_session_bucket: ""                         # S3 bucket for session persistence
  model: gemini-3.1-pro-preview  # or claude-sonnet-4-20250514
  model_regions:                                 # model prefix -> GCP region
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global
  max_iterations: 30                             # default iteration limit
  max_runs_per_mr: 5                             # default per-MR rate limit
  internal_notes: true                           # default note visibility
  docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md
  source_url: https://gitlab.com/org/group/project
  slack_url: https://slack.example.com/archives/C0123456789
  slack_label: "#my-channel"

workflows:
  code-review:
    trigger: merge_request                     # auto-trigger on MR events
    description: Performs AI-powered code review
    sandbox: k8spool                           # per-workflow sandbox override (optional)
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md
    trigger_rules:                             # ordered rule chain, first match wins
      - user_regex: "renovate\\[bot\\]"        # deny bot MRs (regex fullmatch)
        action: deny
      - branch_regex: "chore/.*"               # deny maintenance branches
        action: deny
      - action: allow                          # allow everything else
    prompt: workflows/code-review.md
    action: post_gitlab_note
    model: gemini-3.1-pro-preview  # or claude-sonnet-4-20250514
    max_iterations: 15
    max_inline_size: 200000                    # keep full diffs in context
    context_limit: 500000                      # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows
    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO
    projects:
      org/group/project: {}

  analyze-failures:
    trigger: pipeline                          # auto-trigger on failed pipelines
    description: Investigates CI/CD pipeline failures
    sandbox: kubevirtpool                      # use VM sandbox for heavier workloads
    auto_resolve_on_push: true                 # resolve threads when a new SHA is pushed
    auto_resolve_on_success: true              # resolve threads when pipeline succeeds
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md
    prompt: workflows/analyze-failures.md       # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview  # or claude-sonnet-4-20250514
    max_iterations: 50                          # per-workflow iteration override

    data_sources:                               # model tool tokens (read-only)
      gitlab:
        token_env: GITLAB_TOKEN_RO              # env var name, not the token
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens:                                 # per-project model token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO
        action_tokens:                           # per-workflow write tokens
          gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS

With --workflow/--project, the workflow and project are looked up directly in the config. With --event-file, the project is extracted from the event body and matched against the project index to find applicable workflows.

Discussion threads

All workflow results are posted as discussion threads: a placeholder note starts the discussion and the full result is posted as a reply. The placeholder is never edited, so email notifications include the actual result text. For slash commands, the result is posted as a reply in the triggering discussion.

Auto-resolve

Workflows can opt in to automatic resolution of their discussion threads:

  • auto_resolve_on_push (default false): When a new commit is pushed to the MR (i.e. a merge_request event with action: update), all agent discussion threads for the workflow whose SHA differs from the new HEAD are resolved. This clears stale failure analyses when the developer pushes a fix.
  • auto_resolve_on_success (default false): When the head pipeline succeeds, all agent discussion threads for the workflow on that MR are resolved (regardless of SHA). This handles pipeline reruns on the same SHA where a transient failure is now green.

Both flags are independent and can be combined. Resolution runs before rate limit checks, so threads are resolved even if the workflow’s per-MR run limit has been reached.

Workflows can enable Anthropic’s built-in web search server tool by listing web_search as a data source:

workflows:
  renovate-babysit:
    model: claude-sonnet-4-20250514
    data_sources:
      gitlab: {}
      web_search: {}
    # ...

When web_search is present in data_sources, the web_search_20250305 server tool is included in API requests to Claude. The model decides autonomously when to search. Search execution happens server-side (no client-side tool dispatch), and results appear as server_tool_use / web_search_tool_result content blocks in the response. These blocks are preserved through session save/resume.

Unlike other data sources, web_search has no configuration options and does not register any orchestrator-side tools – it is handled entirely by the model provider.

If the API returns a pause_turn stop reason (server-side search loop hit its iteration limit), the adapter automatically re-sends the conversation to continue, up to 5 continuations per generate() call.

Web search is only supported with Claude models. The setting is ignored for Gemini.

The first agent-authored note (placeholder) includes a footer with links to documentation, source code, the Slack channel, the workflow prompt, and a continuation prompt. These links are configured via global settings:

Setting Description
settings.docs_url Link to agent documentation
settings.source_url Link to agent source repository
settings.slack_url Link to support Slack channel
settings.slack_label Display text for Slack link (default: “Slack”)

Per-workflow, set workflow_url to link to the workflow’s prompt file. The footer stays on the placeholder and the result is a separate reply.

Token separation

Tokens are split into three tiers that never mix:

  • Model tool tokens (in YAML data_sources / tokens): read-only tokens passed to the LLM’s tool calls. Declared in the config file as env var names. These are user-defined and resolved at runtime from the referenced env vars. Create as project access tokens with Reporter role and read_api scope. Reporter is the minimum role required to see internal (confidential) notes in the discussions tool.
  • Workflow action tokens (in YAML action_tokens, per-project): write tokens used by the orchestrator for per-workflow GitLab writes (notes, thread resolution). Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the project config as env var names. Create as project access tokens with Developer role and api scope. These tokens are never exposed to the LLM and never enter the ToolRegistry. Naming convention: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
  • Orchestrator tokens (ORCHESTRATOR_* env vars, NOT in YAML): tokens used by the runner for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Create as project access tokens with Developer role and read_api scope (or api for infrastructure notes). Developer role is required because the discussions tool trust-filters notes by author access level (>= Developer); if the orchestrator bot has only Reporter access, its own notes are redacted. Resolved by convention: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (fallback). The ORCHESTRATOR_ prefix makes these impossible to confuse with model tokens.

Environment Variables

The agent reads only secrets and authentication from environment variables. All operational settings come from the config file’s settings section.

Variable Required Default Description
CONFIG_PATH no config.example.yaml Path to config YAML
GOOGLE_API_KEY yes* - Gemini API key; Gemini direct mode only (not for Claude)
GOOGLE_CLOUD_PROJECT yes* - GCP project ID (Vertex AI mode); required for Claude models
ORCHESTRATOR_GITLAB_TOKEN serve - Orchestrator GitLab token (global fallback)
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> no - Per-project orchestrator token
SENTRY_DSN no - Sentry DSN for error tracking

*One of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT is required. Claude models require Vertex AI (GOOGLE_CLOUD_PROJECT); GOOGLE_API_KEY is for Gemini direct mode only.

Model tool tokens (e.g. GITLAB_TOKEN_RO, GITLAB_TOKEN_CONTAINERS_RO) and data source credentials (e.g. KONFLUX_CLUSTER_URL, KUBECONFIG) are referenced by name in the config file’s data_sources and tokens sections. They are not listed in the table above because their names are user-defined.

Security and Design Constraints

The agent is designed to run in a shared OpenShift cluster without cluster-admin access, processing potentially untrusted merge requests. These constraints shaped the architecture:

Sandbox isolation. All arbitrary commands executed by the LLM run inside an ephemeral container, never on the host:

  • Podman (local): --network=none, --user 65532, no host mounts. Complete network isolation.
  • Kubernetes (production): Pods comply with OpenShift’s restricted-v2 Security Context Constraint: runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], automountServiceAccountToken: false (no K8s API access from sandbox), activeDeadlineSeconds (configurable, default 1800). Security context fields are hardcoded in the pod manifest for portability to vanilla Kubernetes with Pod Security Admission (restricted level). Resource requests/limits, metadata, and activeDeadlineSeconds are configurable via settings.sandbox in the config file. Network access is denied by a NetworkPolicy on the sandbox namespace that blocks all egress from all pods (podSelector: {}).

No cluster-admin required. The agent operates with namespace-scoped permissions only. The orchestrator’s ServiceAccount needs only:

  • pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
  • pods/exec: create – command execution via kubectl exec
  • virtualmachineinstances.kubevirt.io: create, get, list, delete, patch – KubeVirt VMI lifecycle (kubevirt/kubevirtpool modes only)
  • secrets: create, get, delete – SSH key Secrets for VMI provisioning (kubevirt/kubevirtpool modes only)

These permissions are granted via a Role in the sandbox namespace, not the orchestrator’s own namespace. No CRDs, no custom runtimes, no cluster-scoped resources. Konflux data is fetched via bearer token from kubeconfig, not from inside the cluster.

Namespace separation. Sandbox pods are created in a dedicated namespace, separate from the orchestrator. This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens. The sandbox namespace is locked down with standard K8s resources:

  • RBAC: Role + RoleBinding scoped to the namespace, granting only the permissions above to the orchestrator’s ServiceAccount
  • NetworkPolicy: uses podSelector: {} to select all pods in the dedicated sandbox namespace, denying all egress (egress: []). The sandbox cannot reach the internet, the K8s API, or other pods.
  • activeDeadlineSeconds: sandbox pods self-terminate after the configured timeout (default 1800s / 30 minutes) even if the orchestrator crashes or is killed, preventing orphaned pods

Credential separation. Data source credentials (GitLab tokens, kubeconfig) live in the orchestrator process only, injected via K8s Secrets. The sandbox container has no credentials, no SA token (automountServiceAccountToken: false), and no network access. Data flows into the sandbox via stdin piping through write_file.

Command execution via kubectl exec. The K8s sandbox uses a hybrid approach: the Kubernetes Python client manages pod lifecycle (create, wait, delete), while kubectl exec handles command execution. This avoids the complexity and reliability issues of the websocket-based exec API.

KubeVirt VM isolation. KubeVirt sandbox VMs run as root inside the guest OS, but the VM itself is contained by the KubeVirt hypervisor (QEMU/KVM). The VM has no access to the Kubernetes API, no ServiceAccount token, and no credentials. SSH keys are ephemeral (generated per sandbox start for direct mode, or shared per pool for pool mode) and cleaned up with the VMI.

Sandbox Backends

Podman (local) K8s (direct) K8sPool (production) KubeVirt (direct) KubeVirtPool
Start podman run -d --network=none create_namespaced_pod Claim standby pod from Deployment Create VMI + SSH Secret Claim standby VMI from VMIRS
Exec podman exec kubectl exec kubectl exec ssh ssh
Auth Local Podman socket In-cluster SA or kubeconfig In-cluster SA or kubeconfig In-cluster SA or kubeconfig In-cluster SA or kubeconfig
Network None (--network=none) None (deny-all NetworkPolicy) None (deny-all NetworkPolicy) Cluster pod network (SSH) Cluster pod network (SSH)
User 65532 (fixed) Namespace UID range (SCC) Namespace UID range (SCC) root (inside VM) root (inside VM)
Cleanup podman rm -f delete_namespaced_pod delete_namespaced_pod Delete VMI + Secret + temp keys Delete VMI

All five implement the Sandbox protocol: start(), exec(), write_file(), read_file(), cleanup(), linger().

The pod pool backend (k8spool) eliminates pod startup latency by claiming pre-warmed pods from a Kubernetes Deployment. Claimed pods are detached from the ReplicaSet and the Deployment automatically creates replacements. After a successful workflow, pool pods linger for linger_seconds (default 300, configurable, 0 to disable) so that user replies can reuse the same pod without re-creating it or restoring from S3. The reaper runs once per workflow execution and deletes expired lingering pods. It also evicts excess lingering pods beyond max_lingering_pods (default 2), starting with those closest to their deadline. See the design doc (section 8.8) for details.

The KubeVirt backends (kubevirt, kubevirtpool) provide full VM isolation using KubeVirt VirtualMachineInstances. The VM boots from a containerDisk image and SSH keys are injected via a Kubernetes Secret volume (the VM image’s inject-ssh-keys.service reads from /dev/disk/by-id/virtio-ssh-pubkeys). Command execution uses SSH instead of kubectl exec. The kubevirtpool backend claims pre-warmed VMIs from a VirtualMachineInstanceReplicaSet, with the same claim/reap/linger semantics as the pod pool. The sandbox backend can be set per-workflow via the sandbox: field in the workflow config, with resolution order: workflow config > CLI --sandbox > default.

Data Sources

Data sources are registered as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (or auto-spills large responses to the sandbox).

GitLab

Tool Description
gitlab_get_mr_details MR metadata (title, author, state, SHA, labels)
gitlab_get_mr_unified_diff Complete unified diff in patch format
gitlab_get_mr_diff Per-file structured change data
gitlab_get_mr_commits List of commits in a merge request
gitlab_get_mr_discussions Discussion threads with redacted agent transcripts and trust-filtered comments
gitlab_get_commit_statuses CI/CD pipeline statuses for a commit
gitlab_get_file_at_ref Raw file content at a git ref
gitlab_get_repo_archive Repository tar.gz (binary, auto-spilled)
gitlab_get_job_log CI job trace output (ANSI codes stripped)

Konflux

Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and Kubearchive (for completed resources), with deduplication by UID.

Tool Description
konflux_list_pipelineruns All PipelineRuns for a commit SHA
konflux_list_taskruns All TaskRuns for a commit SHA
konflux_list_pods All pods for a commit SHA
konflux_get_pod Full pod resource (spec, status, conditions, container statuses)
konflux_get_pod_log Pod logs; optional container param, fetches all containers when omitted

Response metadata includes konflux_ui base URL for building reviewer-facing links.

Testing Farm

Tool Description
tf_get_results JUnit XML results for a request ID
tf_get_test_log Individual test log by URL (restricted to Testing Farm artifact URLs)
tf_get_request_status Request state, queue/run times

Response metadata includes artifacts_base URL for building artifact links.

Workflow System

Workflows are .md files whose content becomes the LLM system prompt verbatim. Workflow metadata (action, model, max_iterations, enabled data sources, project allowlists) is defined in the config file. The .md file is pure system prompt text.

Available workflows:

  • analyze-failures.md - Investigates CI/CD pipeline failures by fetching MR details, identifying failed pipelines via commit statuses, retrieving PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm, and producing a grouped root-cause report with reviewer-facing URLs.
  • code-review.md - Performs AI-powered code review by fetching MR details, the unified diff, and prior discussion threads in parallel, then attempting to load per-project rules from workflows/repo-rules/ (a mandatory step – rules take precedence over generic standards when present) and producing structured feedback with severity ratings, code examples, and actionable suggestions. On follow-up reviews (after SHA updates), the agent sees its own previous findings, developer responses, and resolved threads – avoiding duplicate findings and respecting developer explanations. Uses elevated max_inline_size (200 KB) and context_limit (500K tokens) to keep the full diff in context.
  • renovate-babysit.md - Triages a single Renovate-authored MR triggered by a successful pipeline. Classifies the MR as safe or risky based on diff scope, upstream dependency changes (from MR description, web search), and local codebase impact (via gitlab_get_repo_archive + grep). Posts a structured note with verdict, upstream change summary, and suggested actions for risky MRs. Requires a Claude model with web_search data source.

Token Budget Management

Both Gemini and Claude benefit from prompt caching that reduces the effective cost of full history replay. Cached token counts from both providers feed into the estimate_cost() calculation. See Agent Model Loop – Prompt caching for details on how caching works per provider.

The agent uses a dual-limit approach instead of a cumulative token budget:

  • Iteration limit (settings.max_iterations, default 30, overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up prompt is injected at 80% (SOFT_ITERATION_RATIO).
  • Context limit (CONTEXT_LIMIT, default 60,000 tokens, overridable per-workflow via context_limit) - Per-call input token ceiling. When exceeded, a wrap-up prompt forces the model to finalize.

Large outputs are automatically redirected to sandbox files to keep the LLM context small. The spill threshold defaults to 4 KB (MAX_INLINE_SIZE) but can be overridden per-workflow via max_inline_size in the config:

  • sandbox_exec - stdout/stderr exceeding the threshold saved to /tmp/_out/{N}.txt; model receives a preview (head + tail) with file path
  • Data sources - text exceeding the threshold saved to /tmp/_out/{name}_{N}.txt with preview; binary data saved to .bin
  • fetch_to_sandbox - always writes to the caller-specified path; returns metadata only

All tuning constants are centralized in config.py:

Constant Default Purpose
DEFAULT_MAX_ITERATIONS 30 Hard iteration cap
SOFT_ITERATION_RATIO 0.8 Inject wrap-up at this fraction
CONTEXT_LIMIT 60,000 Per-call input token ceiling
OUTPUT_PREVIEW_BYTES 4,096 Preview size for spilled outputs
OUTPUT_TAIL_BYTES 512 Extra tail appended to previews
MAX_INLINE_SIZE 4,096 Max inline size for data source responses

Evaluation Framework

The agent includes an evaluation framework for scientific measurement of prompt and model quality. See Hummingbird Agent Evals for the full guide covering the shared hummingbird_agent.eval library, existing evaluations, and how to write new ones.

Development

See the main README for development workflows.

make hummingbird-agent/setup  # Install dependencies
make check                    # Lint code (ruff)
make fmt                      # Format code
make test                     # Run unit tests
make coverage                 # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.17 - Hummingbird Agent Model Loop

How the Hummingbird agent communicates with the LLM (Gemini or Claude), calls tools, and how user replies integrate into the conversation.

For the architectural design rationale behind the loop (why full history replay, why iteration-based budgets, why nudge via system prompt), see Agent Design – Section 6. For operational usage documentation, see Hummingbird Agent.

What gets sent to the model

Every call to the model API sends three things:

1. systemInstruction

A single text string, rebuilt every iteration. Gemini sends it as systemInstruction.parts[].text; Anthropic sends it as a top-level system string. Composed of layers:

BASE_SYSTEM_PROMPT            # agent.py: sandbox rules, tool usage tips
+ tool_notes                  # per-data-source notes from ToolRegistry
+ workflow_prompt             # full content of e.g. workflows/analyze-failures.md

Near the iteration or context limit, a warning suffix is appended to the system prompt for that call. The suffix escalates through three levels:

  • ITERATION_WARNING / CONTEXT_WARNING – soft “start wrapping up” at 80% of the iteration or context limit. Tools remain available.
  • FINAL_TURN_WARNING – hard stop on the absolute last iteration or after the context limit is exceeded. Combined with tool-calling disabled for that request (toolConfig.functionCallingConfig.mode: NONE on Gemini, tool_choice.type: none on Anthropic) to mechanically prevent further tool calls.
  • EMPTY_RESPONSE_NUDGE – appended when the model returns an empty response (no text, no tool calls). Tools remain available.

2. contents

A list of message dicts – the full conversation history. Grows every iteration. Each entry has a role (user, model, or tool) and parts. On Anthropic, tool results use role: "user" (not a separate role); the adapter merges consecutive user messages so the wire shape matches what each API expects.

{
  "contents": [
    {"role": "user",  "parts": [{"text": "{\"project\":\"org/repo\",\"iid\":42,...}"}]},
    {"role": "model", "parts": [{"functionCall": {"name": "gitlab_get_mr_details", "args": {...}}}]},
    {"role": "tool",  "parts": [{"functionResponse": {"name": "gitlab_get_mr_details", "response": {...}}}]},
    ...
  ]
}

3. tools

Tool definitions, static across all iterations. Gemini uses functionDeclarations (JSON Schema in parametersJsonSchema); Anthropic uses input_schema per tool.

{
  "tools": [{"functionDeclarations": [
    {"name": "sandbox_exec", "description": "...", "parametersJsonSchema": {...}},
    {"name": "fetch_to_sandbox", ...},
    {"name": "fetch_batch_to_sandbox", ...},
    {"name": "gitlab_get_mr_details", ...},
    {"name": "konflux_list_pipelineruns", ...}
  ]}]
}

4. toolConfig (conditional)

On the final turn (last iteration or after context limit exceeded), the request disables tools to mechanically prevent function calls. Gemini:

{
  "toolConfig": {"functionCallingConfig": {"mode": "NONE"}}
}

Anthropic equivalent: tool_choice: {"type": "none"}.

This is only sent when FINAL_TURN_WARNING is active. All other iterations omit this constraint, allowing the model to choose freely between text and tools.

The agent loop, turn by turn

The loop in run_agent_loop runs up to max_iterations times. Each iteration is one round-trip to the model.

Initialization

system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt
tool_defs     = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, <data sources...>]
contents      = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}]

Iteration 1

-> Send: system_instruction + contents (1 item) + tool_defs
<- Response: functionCall(gitlab_get_mr_details, {project: "org/repo", iid: 42})

contents.append(model: response.raw_content)
  execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"}
contents.append(tool: model.make_tool_responses([(name, result)]))

Iteration 2

-> Send: system_instruction + contents (3 items) + tool_defs
<- Response: functionCall(fetch_batch_to_sandbox, {requests: [...]})

contents.append(model: ...)
  execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]}
contents.append(tool: ...)

Iteration 3

-> Send: system_instruction + contents (5 items) + tool_defs
<- Response: functionCall(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"})

contents.append(model: ...)
  execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"}
contents.append(tool: ...)

Iteration N (final)

-> Send: system_instruction + contents (2N-1 items) + tool_defs
<- Response: text("## Konflux Failure Analysis\n\n...")    // text + no tool_calls = DONE

contents.append(model: {text: "## Konflux Failure Analysis..."})
BREAK -- return (text, usage, transcript, contents)

Response handling and termination

Each iteration, the model’s response can contain text, tool calls, both, or neither. The agent handles each case:

Response Action
Text only Final response. Save text, break. (Happy path.)
Text + tool calls Save text as last_text fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting.
Tool calls only Execute the tool calls, continue loop.
Neither Empty response – retry up to MAX_EMPTY_RETRIES (2) times. Each retry nudges the model via EMPTY_RESPONSE_NUDGE appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries.

The text + tool calls case is worth noting: the model’s text is not returned to the user immediately. It is stored as last_text (a fallback in case the loop terminates later without a clean final text, e.g. by hitting max iterations). The raw_content dict – which includes both the text and functionCall parts – is appended to contents as a single model turn:

{"role": "model", "parts": [
    {"text": "Let me check the clair-scan logs..."},
    {"functionCall": {"name": "sandbox_exec", "args": {"command": "grep timeout ..."}}}
]}

The loop terminates on:

  1. Text only – the model produced a final response.
  2. Max iterations reached – the final iteration uses FINAL_TURN_WARNING + toolConfig NONE to force text output. Falls back to whatever last_text was seen, or "[Agent did not produce a final response]".
  3. Context limit exceeded – input_tokens >= CONTEXT_LIMIT sets context_exceeded, making the next iteration final (same as #2). A soft warning (CONTEXT_WARNING) fires earlier at 80% of the limit.
  4. Fatal model error – HTTP error, timeout, or connection error after retries exhausted. Transport errors (timeouts, connection failures) are wrapped as retryable ModelError and retried with exponential backoff alongside HTTP 5xx/429.
  5. Empty responses exhausted – MAX_EMPTY_RETRIES (2) nudge retries failed.
  6. Unexpected error – any exception not handled by the retry mechanism (e.g. malformed API response) breaks the loop and returns partial results. Accumulated contents, transcript, and sandbox are preserved and the session is saved normally.

Budget escalation

Both iterations and context size use the same two-tier pattern:

Soft warning (80%) Hard stop (100%)
Iterations ITERATION_WARNING at 80% of max_iterations FINAL_TURN_WARNING on last iteration
Context CONTEXT_WARNING at 80% of CONTEXT_LIMIT FINAL_TURN_WARNING on next iteration after exceeding CONTEXT_LIMIT

Soft warnings are advisory (“start wrapping up”) and keep tools available. The hard stop uses toolConfig.mode: NONE to mechanically prevent further tool calls, ensuring the model produces text.

The contents list in detail

Each entry in contents follows the provider’s native wire format during execution.

Canonical format (persistence)

During the agent loop, contents use the provider’s native format. On save, to_canonical() converts them to OpenAI Chat Completions-style messages. On load, from_canonical() converts back to the current provider’s native format. That enables cross-provider session resumption.

Anthropic-specific blocks are stored as opaque pass-through fields on the canonical assistant message: thinking_blocks for thinking/redacted_thinking, server_tool_blocks for server_tool_use/web_search_tool_result. These are restored in correct position order (thinking first, then text/tool_use, then server tool blocks) by from_canonical().

User turn

{"role": "user", "parts": [{"text": "..."}]}

Created by model.make_user_content(text). In a cold start there is exactly one user turn at the start: the JSON event. No additional user turns appear during a normal run – empty-response nudges are delivered via the system prompt, not as user messages.

Model turn

{"role": "model", "parts": [
    {"functionCall": {"name": "sandbox_exec", "args": {"command": "jq ..."}}}
]}

Or for the final response:

{"role": "model", "parts": [{"text": "## Konflux Failure Analysis..."}]}

This is response.raw_content – the exact dict from the model response candidate, appended verbatim. Can contain text, tool calls, or both. Parallel tool calls appear as multiple functionCall parts in one model turn.

Tool turn

{"role": "tool", "parts": [
    {"functionResponse": {"name": "sandbox_exec", "response": {"exit_code": 0, "stdout": "..."}}}
]}

Created by model.make_tool_responses(results). One functionResponse part per tool call in the preceding model turn. Tool results are always JSON dicts; large outputs are spilled to sandbox files and only a preview is included.

Typical 8-iteration conversation shape

contents[0]  = user:  {"project":"org/repo","iid":42,...}          # initial event
contents[1]  = model: functionCall(gitlab_get_mr_details)          # iter 1
contents[2]  = tool:  functionResponse(gitlab_get_mr_details)      # iter 1
contents[3]  = model: functionCall(fetch_batch_to_sandbox)         # iter 2
contents[4]  = tool:  functionResponse(fetch_batch_to_sandbox)     # iter 2
contents[5]  = model: functionCall(sandbox_exec)                   # iter 3
contents[6]  = tool:  functionResponse(sandbox_exec)               # iter 3
...
contents[13] = model: functionCall(sandbox_exec)                   # iter 7
contents[14] = tool:  functionResponse(sandbox_exec)               # iter 7
contents[15] = model: text("## Konflux Failure Analysis...")       # iter 8 (final)

This entire list is persisted as context.json in the session (see Canonical format (persistence) above). It is the full state needed to resume a conversation.

How tools are called

When the model’s response contains functionCall parts, _execute_tool_calls iterates them sequentially:

tool_registry.execute(ToolCall(name, args))
  match name:
     "sandbox_exec"           -> runs sh -c in container -> {exit_code, stdout, stderr}
     "fetch_to_sandbox"       -> calls data source func -> writes to sandbox file -> {saved_to, bytes}
     "fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call
     any data source name     -> calls func directly -> returns inline or auto-spills large output

Large output handling

When stdout or a data source response exceeds 4KB, it is automatically saved to a sandbox file (/tmp/_out/N.txt) and only a preview (head + tail) is returned to the model. The model gets the file path and can use sandbox_exec with jq/grep/head to process it. This keeps the context window manageable.

How a user reply integrates (session resumption)

When a user replies to an agent note on GitLab, the orchestrator resumes the conversation by restoring the previous state and appending the reply.

What gets restored from S3

  • context.json – the persisted conversation in canonical (OpenAI Chat Completions-style) form; from_canonical() maps it to the active provider’s native contents before the loop runs
  • sandbox.tar.gz – /tmp/_out/ files (PipelineRun JSONs, test logs, jq output, etc.) restored into the new sandbox

Because load uses from_canonical(), you can switch models across providers (e.g. Gemini to Claude or the reverse) when resuming, as long as the session was saved with canonical contents.

The resumed contents list

// Restored from context.json (previous run)
contents[0]  = user:  {"project":"org/repo","iid":42,...}           # original event
contents[1]  = model: functionCall(gitlab_get_mr_details)           # iter 1
contents[2]  = tool:  functionResponse(gitlab_get_mr_details)       # iter 1
...                                                                  # all prior turns
contents[15] = model: text("## Konflux Failure Analysis...")        # previous final text

// NEW: user reply appended
contents[16] = user:  "Can you look at the clair-scan timeout more closely?"

The agent loop continues

ITERATION 1 (resumed):
  -> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs
  <- Response: functionCall(sandbox_exec, {command: "grep -i timeout /tmp/data/..."})
                                                      ^ using restored sandbox files
  contents[17] = model: functionCall(sandbox_exec)
  contents[18] = tool:  functionResponse(sandbox_exec)

ITERATION 2 (resumed):
  <- Response: text("The clair-scan timeout is caused by...")
  contents[19] = model: text("The clair-scan timeout is caused by...")
  BREAK

Cold start vs resumed – key differences

Aspect Cold start Resumed
System prompt BASE + tool_notes + workflow BASE + tool_notes + workflow + CONTINUATION_PROMPT
First user message {"project":..., "iid":...} {"project":..., "iid":...} (restored)
Conversation history Empty (just the event) Full prior conversation
Latest user message – "Can you look at the clair-scan timeout?"
Sandbox files Empty Restored from archive
Tool definitions Same Same
Session format Native (per provider) in memory Canonical on disk; native after from_canonical()

The CONTINUATION_PROMPT

Without this, the workflow prompt (e.g. analyze-failures.md) tells the model to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1… The model might try to re-run the entire analysis. The continuation prompt overrides this:

## Continuation

This is a follow-up to a previous conversation. The conversation history
contains your prior analysis and tool calls. The user is replying with a
question or request about your previous work.

**Do NOT re-run the full workflow from scratch.** Instead:
- Respond directly to the user's question
- Use your tools to investigate further if needed (logs, data are still
  available in the sandbox)
- Reference your previous findings where relevant
- Keep your response focused on what the user asked

The model now understands: “I already did the analysis (it’s all in the conversation history). The user has a specific question. Let me answer it.”

Iteration and token budget for resumed sessions

The resumed session gets a full fresh iteration budget. run_workflow sets max_iterations from the workflow config (e.g. 50 for analyze-failures), and the loop counter starts at 0 regardless of how many iterations the previous session used. The model can make as many tool calls as it needs to answer the follow-up.

The practical constraint is the context window, not iterations. The restored contents can be large – a typical 8-iteration cold start uses ~50-100K input tokens. Since the model re-counts the full history every iteration, the resumed session starts at roughly the token count where the previous session ended, plus the new user message. Each new tool call adds more. Token counts logged by the agent are billed tokens (cumulative across API calls; each call re-sends the full history, so these overlap).

The existing CONTEXT_LIMIT check still applies: at 80% of the limit, a soft CONTEXT_WARNING tells the model to start wrapping up. If the limit is exceeded, the next iteration becomes final (FINAL_TURN_WARNING + toolConfig NONE).

For the typical case (user asks one focused follow-up, agent does 1-3 tool calls), this is fine. Multi-turn deep dives will eventually hit the context limit, at which point the agent wraps up – and the user can start a fresh conversation if needed.

Prompt caching

Since the agent replays the full conversation history on every API call, prompt caching significantly reduces the cost of repeated prefixes. Both providers support caching, but with different mechanisms.

Gemini (implicit)

Gemini’s Vertex AI backend automatically caches repeated request prefixes server-side. No request-side annotations are needed. The agent reads cachedContentTokenCount from usageMetadata in each response and reports it as cache_read_tokens. There is no cache_creation_tokens for Gemini – implicit caching has no write surcharge, only a read discount (25% of the base input rate).

Claude (explicit breakpoints)

Vertex AI does not support Anthropic’s automatic caching. The agent places explicit cache_control: {"type": "ephemeral"} annotations on message content blocks. The API hashes the cumulative prefix – tools, system prompt, and all messages from the start of the request up to the annotated block – and caches the result. A breakpoint on a message therefore covers everything before it; separate breakpoints on the system prompt or tools would be redundant.

On the wire, an annotated message looks like this:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "Describe the sandbox.", "cache_control": {"type": "ephemeral"}}
  ]
}

For messages with multiple content blocks (e.g. tool_result arrays), the cache_control is placed on the last block in the array.

Sliding-window breakpoints

The agent uses two of the four available breakpoint slots per request. A sliding pair of breakpoints (B1, B2) ensures the full conversation prefix is cached and only the newest turn is processed at full price:

  • B2 is placed on the last message (writes the current prefix to cache)
  • B1 is placed where B2 was on the previous call (reads the prior prefix from cache)
Call 1:  [msg0 B2]
         B2: WRITE entire prefix to cache

Call 2:  [msg0 B1] [msg1] [msg2 B2]
         B1: READ  (matches call 1's B2 -- same position, same prefix)
         B2: WRITE (extends cache to include new messages)

Call 3:  [msg0] [msg1] [msg2 B1] [msg3] [msg4 B2]
         B1: READ  (matches call 2's B2)
         B2: WRITE

Call N:  ... [msg(N-2) B1] [msg(N-1)] [msg(N) B2]
         B1: READ  (matches call N-1's B2)
         B2: WRITE

The cache_control metadata is not part of the prefix hash. Moving B1 to a position that previously had B2 (and no longer has cache_control) does not invalidate the cache – the hash matches because the content is identical.

The cache has a 5-minute TTL, refreshed on each hit. Cache writes cost 1.25x the base input rate (25% surcharge); cache reads cost 0.10x (90% discount). The minimum cacheable prefix length is 1024 tokens for Sonnet/Opus and 4096 tokens for Haiku – the system prompt alone exceeds these thresholds.

Ephemeral turns

When the agent loop injects a transient message (iteration warning, context warning, empty-response nudge), the generate() call receives ephemeral=True. The transient message is appended as a user message, merged with the preceding tool-result user message by _merge_consecutive_user, and popped from contents after the call.

  • B1 is still placed at the previous B2 position, so the cache read for the stable prefix still works
  • B2 is placed on the second-to-last merged message (the last stable message before the merged ephemeral tail), so the cached prefix advances without including transient content
  • The internal B2 position advances to the second-to-last index, so successive ephemeral turns keep sliding B1 forward

This ensures the B1=previous-B2 invariant holds through ephemeral turns:

T1:     U1·B2                                          prev=0
T2:     U1·B1  M1  U2·B2                               prev=2
T3(E):  U1  M1  U2·B1  M2·B2  U3+E                    prev=3
T4(E):  U1  M1  U2  M2·B1  U3  M3·B2  U4+E            prev=5
T5:     U1  M1  U2  M2  U3  M3·B1  U4  M4  U5·B2      prev=8

(U = user message, M = model message, E = ephemeral, +E = merged with preceding user message.)

When the merged message list has fewer than two entries during an ephemeral call (e.g. a single merged user+ephemeral on the first turn), no breakpoints are placed and the B2 position is not updated.

Thinking blocks

Extended thinking blocks in assistant responses are part of the cached prefix. They do not break cache hits when the following user message contains only tool_result blocks (the normal case in the agent loop).

When web_search is listed in a workflow’s data sources, the Anthropic adapter adds the web_search_20250305 server tool to the request. The API executes searches server-side and returns server_tool_use and web_search_tool_result content blocks in the assistant response alongside regular text/tool_use blocks.

These blocks are:

  • Transparent to the agent loop – they don’t generate ToolCall entries, so the agent loop doesn’t attempt to dispatch them.
  • Preserved in raw_content – the full content block array (including server tool blocks) is stored in raw_content and persisted via to_canonical().
  • Restored on session resume – from_canonical() reconstructs them as opaque pass-through blocks, similar to thinking blocks.

If the API returns pause_turn (server-side search loop hit its iteration limit), the adapter automatically re-sends the partial response to continue, merging content blocks and accumulating usage tokens across continuations.

Caching and session resumption

After session resumption, the model instance is fresh and has no record of previous breakpoint positions. The first call has no B1 (no cache read), only B2 (cache write). From call 2 onward, caching works normally. This is the same behavior as a cold start.

4.5.18 - Red Hat Catalog

Web UI for browsing Hummingbird container images, security advisories, and API documentation.

Features

  • Image Catalog - Browse container images with tags, architectures, and specifications
  • Security Feed - Security advisory feed with CVE details
  • API Documentation - Interactive API documentation browser
  • Image Details - SBOM, provenance, and release history per image

Architecture

React 18 + PatternFly v6 SPA. Data fetching via TanStack Query against the container-catalog API (proxied through webpack-dev-server in development, direct in production). Webpack 5 build. Hosted on CloudFront + S3 (production) with MR previews on GitLab Pages.

Prerequisites

  • Node.js 22+ (for host targets)
  • make + podman (for container targets)

Usage

Only make and podman are required for container targets. Host variants (*-host) run without podman.

Target Description
redhat-catalog/setup Install dependencies (container)
redhat-catalog/check Type-check, lint, and test (container)
redhat-catalog/build Production build (container)
redhat-catalog/dev Dev server on port 9000 (container)
redhat-catalog/setup-host Install dependencies (host)
redhat-catalog/check-host Type-check, lint, and test (host)
redhat-catalog/build-host Production build (host)
redhat-catalog/dev-host Dev server (host)

Set ASSET_PATH to control the base path for non-root deployments (used by webpack output.publicPath and React Router basename).

Configuration

Build Variables

Variable Description
ASSET_PATH Webpack public path / React Router basename

API base URLs are configured in app-config.ts.

SAM Parameters

Parameter Description
ResourcePrefix Prefix for all resource names
CatalogDomainName Custom domain (optional, leave empty for CloudFront default)

Deployment

Production deployment uses a CloudFront + S3 stack defined in template.yaml. The infrastructure post-deploy script builds the SPA in a Node.js container, syncs it to S3, and creates a CloudFront cache invalidation. Custom domain support is optional via the CatalogDomainName SAM parameter. When set, an ACM certificate is created in-template with DNS validation – the validation CNAME must already exist in the external DNS zone before deployment.

MR previews are deployed to GitLab Pages with path_prefix per MR, auto-cleaned on merge/close with a 1-week expiry.

Environments and CI promotion

Experimental, staging, and production URLs, MR vs main pipelines, deploy triggers, and UAT sign-off are documented in Red Hat Catalog Environment Promotion (monorepo documentation/) and Red Hat Catalog UAT Program.

Development

Detailed contributor documentation lives in-tree:

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.19 - Hummingbird Agent Design

Architectural design document for the hummingbird-agent. Covers the reasoning behind every major design choice so that future changes can be made safely, without accidentally violating invariants that hold the system together.

For operational usage (CLI, config reference, deployment), see Hummingbird Agent. For the model loop wire format, see Agent Model Loop.

1. Design Philosophy

Five principles shaped the agent’s architecture. Every component traces back to at least one of these.

Security by default. The agent processes untrusted merge requests. All LLM-driven commands run inside isolated sandbox containers with no network and no credentials. Orchestrator write tokens and model read tokens live in separate code paths that never cross. No cluster-admin is required.

Config over code. Investigation logic lives in markdown workflow files that become the LLM system prompt verbatim – changing the analysis strategy requires no code change and no redeployment. Operational settings (data sources, project allowlists, iteration limits) live in a YAML config file that is auditable and committable. Secrets are the only thing in environment variables.

Bounded cost. Every output path has a size cap. Large tool outputs are auto-spilled to sandbox files with only a preview returned to the model. Each session has both an iteration limit and a per-call context token limit with two-tier escalation (soft warning, then hard stop with tool disabling). The relationship between these constants is documented and centralized.

Partial over nothing. When some data is unavailable (expired pod logs, unreachable Konflux cluster, Testing Farm 404), the agent continues with whatever data it has and notes the gap in its output. A partial report is more valuable than a crash.

Model-agnostic agent loop. The agent loop (agent.py) does not inspect the internal structure of the contents list. It appends raw_content from model responses and make_tool_responses() output without looking inside. Each model backend owns its wire format. This makes it possible to add new model backends (Claude, GPT) without touching the agent loop.

2. System Architecture

2.1. Component overview

flowchart TB
    subgraph input [Event Sources]
        SQS["SQS Queue<br/>(gitlab::pipeline, gitlab::note)"]
        CLI["CLI<br/>(--event / --event-file)"]
    end

    subgraph orchestrator [Orchestrator Process]
        Events["events.py<br/>SQS consumer"]
        Runner["runner.py<br/>Event routing"]
        Agent["agent.py<br/>Model loop"]
        Tools["tools.py<br/>Tool registry"]
        Actions["actions.py<br/>GitLab notes"]
        Sessions["sessions.py<br/>Persistence"]
        WfConfig["workflow_config.py<br/>YAML config"]
    end

    subgraph models [Model Backends]
        Gemini["models/gemini.py<br/>Gemini API / Vertex AI"]
        Anthropic["models/anthropic.py<br/>Anthropic via Vertex AI"]
    end

    subgraph sandbox [Sandbox Container]
        SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox<br/>jq, python3, yq"]
        SpillFiles["/tmp/data/_out/ spill files"]
        DataFiles["/tmp/data/ working files"]
    end

    subgraph dataSources [Data Source Modules]
        GL["gitlab.py"]
        KX["konflux.py"]
        TF["testing_farm.py"]
    end

    subgraph external [External APIs]
        GitLabAPI["GitLab API"]
        KonfluxAPI["K8s + Kubearchive"]
        TFAPI["Testing Farm API"]
    end

    subgraph storage [Storage]
        S3["S3 Sessions"]
        MRNote["GitLab MR Notes"]
    end

    SQS --> Events --> Runner
    CLI --> Runner
    Runner --> Agent
    Agent --> Tools
    Tools --> SB
    Tools --> dataSources
    dataSources --> external
    Agent --> Gemini
    Agent --> Anthropic
    Runner --> Actions --> MRNote
    Runner --> Sessions --> S3
    Runner --> WfConfig

2.2. Request lifecycle

A complete run proceeds through these stages:

  1. Event arrival. An SQS message (production) or CLI invocation (dev) provides a GitLab project path and MR IID.

  2. Config lookup. workflow_config.py maps the project to one or more workflows via the project_index. Each match yields a WorkflowConfig and ProjectEntry with data source declarations and per-project settings.

  3. Rate-limit check and SHA dedup. actions.scan_agent_threads() scans MR discussions in a single pass, parsing JSON session markers to determine the per-workflow thread count and whether the current SHA has already been reviewed. If the SHA was already reviewed, the workflow is skipped. If the thread count meets or exceeds max_runs_per_mr, the workflow is skipped. Slash commands and replies bypass this check.

  4. Placeholder note. actions.create_placeholder_note() posts a placeholder so the reviewer knows analysis is in progress. The note contains a JSON session marker (<!-- hummingbird-session: {"id":"UUID","wf":"name","sha":"abc"} -->) for rate limiting, SHA dedup, and session resumption.

  5. Sandbox start. sandbox.create_sandbox() starts a Podman container or K8s pod. /tmp/data/ is pre-created for the model’s use.

  6. Data source registration. data_sources.register_selected() resolves tokens and URLs from the config and registers tool functions on the ToolRegistry. Only data sources declared in the workflow config are registered.

  7. Agent loop. agent.run_agent_loop() runs the model loop: the workflow markdown becomes the system prompt, tool definitions are provided, and the model iterates calling tools and producing text until it emits a final response or hits a budget limit.

  8. Note update. The final text, wrapped with a session marker and reply prompt, replaces the placeholder note.

  9. Session save. Conversation history (contents), transcript, and sandbox archive are saved to S3 (production) or a local directory (dev).

  10. Sandbox cleanup. The container/pod is deleted. K8s pods also have a configurable activeDeadlineSeconds backstop (default 1800s) in case the orchestrator dies.

2.3. Architectural boundaries

The codebase is organized around four strict boundaries:

Runner (runner.py) is the orchestration layer. It owns event routing, the placeholder/update note lifecycle, session save/load, and sandbox lifecycle. It calls agent.run_agent_loop() but never reaches into the agent’s internals.

Agent (agent.py) is the model loop. It knows about the model interface, tool definitions, and the contents list, but nothing about GitLab, SQS, sessions, or actions. It returns (text, usage, transcript, contents) and is completely unaware of what happens with those values.

Tools (tools.py) bridge the agent and the sandbox/data-sources. The agent calls tool_registry.execute(tool_call) and gets a dict back. It never calls sandbox methods directly. This indirection is what enables auto-spill: the tool registry can transparently save large outputs to sandbox files and return previews.

Models (models/) own wire format conversion. Each model adapter’s generate() accepts internal types (contents, ToolDef) and returns a ModelResponse. make_user_content() and make_tool_responses() produce the model-specific dicts that go into contents (Gemini and Anthropic backends each implement the ModelAdapter protocol). The agent treats these as opaque values – it appends them but never inspects their internal structure.

3. Module Architecture

3.1. Dependency graph

flowchart TD
    main["__main__.py"]
    config_mod["config.py"]
    wf_config["workflow_config.py"]
    events_mod["events.py"]
    runner_mod["runner.py"]
    agent_mod["agent.py"]
    workflow_mod["workflow.py"]
    tools_mod["tools.py"]
    sandbox_mod["sandbox.py"]
    actions_mod["actions.py"]
    sessions_mod["sessions.py"]
    transcript_mod["transcript.py"]
    ds_init["data_sources/__init__.py"]
    ds_gitlab["data_sources/gitlab.py"]
    ds_konflux["data_sources/konflux.py"]
    ds_tf["data_sources/testing_farm.py"]
    http_mod["_http.py"]
    config_watch_mod["config_watch.py"]
    models_init["models/__init__.py"]
    models_types["models/_types.py"]
    models_gemini["models/gemini.py"]
    models_anthropic["models/anthropic.py"]

    main --> config_mod
    main --> config_watch_mod
    main --> wf_config
    main --> events_mod
    main --> runner_mod
    main --> sessions_mod
    main --> transcript_mod
    main --> actions_mod

    config_watch_mod --> config_mod
    config_watch_mod --> wf_config

    runner_mod --> actions_mod
    runner_mod --> agent_mod
    runner_mod --> config_mod
    runner_mod --> ds_init
    runner_mod --> sandbox_mod
    runner_mod --> sessions_mod
    runner_mod --> tools_mod
    runner_mod --> workflow_mod
    runner_mod --> wf_config
    runner_mod --> transcript_mod

    agent_mod --> config_mod
    agent_mod --> models_init
    agent_mod --> tools_mod

    tools_mod --> config_mod
    tools_mod --> models_init
    tools_mod --> sandbox_mod

    workflow_mod --> config_mod
    workflow_mod --> models_init

    ds_init --> tools_mod
    ds_init --> wf_config
    ds_init --> ds_gitlab
    ds_init --> ds_konflux
    ds_init --> ds_tf

    ds_konflux --> http_mod

    sessions_mod --> sandbox_mod

    models_init --> models_types
    models_init --> models_gemini
    models_init --> models_anthropic
    models_gemini --> http_mod
    models_gemini --> models_types
    models_anthropic --> http_mod
    models_anthropic --> models_types

3.2. Module responsibilities

Each module has a single, well-defined responsibility. The boundary rules below are invariants – violating them breaks the security model or the separation of concerns.

config.py – Runtime configuration and constants

  • Owns: Config dataclass, estimate_cost(), all tuning constants (DEFAULT_MAX_ITERATIONS, CONTEXT_LIMIT, OUTPUT_PREVIEW_BYTES, etc.)
  • Boundary: Pure data. No I/O except reading env vars in Config.load(). No imports from other hummingbird modules.
  • Invariant: All interdependent budget/spill constants must be defined here with their relationship documented in the comment block.

workflow_config.py – YAML config loader and token resolution

  • Owns: AgentConfig, WorkflowConfig (with trigger, description, trigger_rules, ignore_users, ignore_branches fields), TriggerRule, ProjectEntry, DataSourceConfig dataclasses; load(), validate_env(), evaluate_trigger_rules(), resolve_tool_token(), resolve_cluster_url(), get_prompt().
  • Boundary: Reads YAML from disk and env vars. Never instantiates network clients or model objects.
  • Invariant: resolve_tool_token() resolves model tool tokens only. It must never return orchestrator tokens. The resolution chain is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> "" (empty).

events.py – SQS consumer

  • Owns: decode_sns_message() (SNS envelope decoding), poll_loop() (blocking SQS consumer with concurrency control).
  • Boundary: Knows about SQS/SNS wire formats. Calls a generic EventHandler callback. Does not know about GitLab, workflows, or the agent.
  • Invariant: Failed messages are not deleted from SQS (they go to the DLQ after visibility timeout expires). Successful messages are deleted after handler returns.

config_watch.py – Background config hot-reload

  • Owns: ConfigHolder (thread-safe config pair with atomic swap), start_watcher() (daemon thread that polls config file mtime), _watch_loop().
  • Boundary: Knows about config.Config.load() and workflow_config.load(). Does not know about events, the agent, or any runtime state.
  • Invariant: On reload failure (parse error, missing env var), the previous config is kept and a warning is logged. The watcher never crashes the serve loop.

runner.py – Event routing and workflow execution

  • Owns: handle_event(), handle_pipeline(), handle_merge_request(), handle_note(), _execute_workflow(), _handle_reply(), run_workflow(), _acquire_sandbox(), _resolve_slash_workflow(), _format_help(), _is_ignored_user(), WorkflowRequest, WorkflowResult, SandboxOpts.
  • Boundary: Orchestrates everything: config lookup, rate limiting, placeholder notes, sandbox lifecycle, agent invocation, session save. This is the only module that touches both actions and agent.
  • Invariant: run_workflow() either lingers the sandbox (success) or cleans it up (exception).

agent.py – Model-agnostic tool-calling loop

  • Owns: run_agent_loop(), system prompt constants (BASE_SYSTEM_PROMPT, CONTINUATION_PROMPT, warning/nudge strings), budget logic.
  • Boundary: Knows about models (the ModelAdapter protocol: generate() and content construction) and tools (for execute()). Does not know about GitLab, SQS, sessions, or actions. Does not know which model backend is in use.
  • Invariant: The contents list is treated as opaque. Items are appended via response.raw_content and model.make_tool_responses(). The agent never inspects, modifies, or deletes items in the list (except for empty response retries, which pop() the last appended item before the model has seen any tool results for it).

tools.py – Tool registry

  • Owns: ToolRegistry class with sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox built-in tools; data source registration and dispatch; auto-spill logic.
  • Boundary: Owns the sandbox reference and all tool execution. The agent never touches the sandbox directly.
  • Invariant: _spill_counter is shared across all spill paths (sandbox exec stdout, sandbox exec stderr, data source auto-spill) to prevent filename collisions in /tmp/data/_out/.

sandbox.py – Sandbox backends

  • Owns: Sandbox protocol, PodmanSandbox, K8sSandbox, K8sPoolSandbox, SandboxPool, KubeVirtSandbox, VmiPool, KubeVirtPoolSandbox, create_sandbox(), ExecResult.
  • Boundary: Knows about container/pod/VM lifecycle and command execution. Does not know about tools, models, or the agent.
  • Invariant: All backends must implement the same 8-method protocol (start, exec, write_file, stream_to_file, stream_exec, read_file_iter, cleanup, linger). exec() accepts optional stdin_data for piping raw bytes. All must pre-create /tmp/data/ in start(). All must use stdin piping for write_file() (never host volume mounts).

actions.py – GitLab note lifecycle

  • Owns: Orchestrator token resolution (resolve_orchestrator_token()), workflow action token client creation (_make_client()), bot user ID collection (collect_workflow_bot_ids()), note CRUD, JSON marker parsing (marker_tag, parse_marker), scan_agent_threads (per-workflow rate limiting + SHA dedup), post_simple_reply, member access checks, session-for-reply lookup, AgentThreadInfo, SessionRef.
  • Boundary: Functions are split by token tier:
    • Orchestrator token (operational reads + infrastructure notes, use _get_client): check_member_access, get_latest_push_author, get_head_pipeline_id, is_first_note_in_discussion, reply_access_denied, notify_rate_limit, post_simple_reply.
    • Workflow action token (per-workflow-attributable writes, take explicit token: str): create_placeholder_discussion, post_discussion_note, resolve_agent_threads, scan_agent_threads. Never touches model tool tokens, the tool registry, or the agent.
  • Invariant: Orchestrator tokens are resolved from ORCHESTRATOR_* env vars only. Workflow action tokens are resolved from action_tokens in YAML and passed as explicit parameters. Neither tier is exposed to the model.

sessions.py – Session persistence

  • Owns: archive_sandbox(), save_local(), save_s3(), load_s3(), load_local(), restore_sandbox(), SessionData (including format_version for canonical vs legacy session JSON).
  • Boundary: Knows about sandbox (for archiving) and S3/filesystem (for storage). Does not know about the agent, tools, or models. Load/save detects canonical envelope (format_version: 1) vs legacy raw contents lists.
  • Invariant: The sandbox archive is never extracted on the orchestrator. It is created inside the sandbox (tar czf), streamed out via read_file_iter(), and restored inside a new sandbox via stream_exec("tar xzf -").

transcript.py – Transcript rendering

  • Owns: render_markdown(), per-tool rendering functions, truncation.
  • Boundary: Pure transformation from TranscriptEntry list to markdown. No I/O, no side effects.

data_sources/__init__.py – Data source registration

  • Owns: register_selected() – resolves tokens/URLs from config and calls each data source’s register() function.
  • Boundary: Bridges workflow_config (for token/URL resolution) and tools (for registration). Only registers data sources declared in the workflow config.

data_sources/gitlab.py, konflux.py, testing_farm.py

  • Own: register() function that creates tool definitions and closures over credentials, and registers them on the ToolRegistry.
  • Boundary: Each module talks to one external API. Credentials are captured at registration time via closure, never stored globally.

models/_types.py – Shared data classes

  • Owns: ModelError, ToolDef, ToolCall, Usage, ModelResponse, TranscriptEntry; ModelAdapter protocol (implemented by model backends).
  • Boundary: Pure data. No imports from other hummingbird modules.

_http.py – Shared HTTP infrastructure

  • Owns: new_session() (requests session with retry adapter and configurable 429 handling), VertexAuth (Google ADC credentials).
  • Boundary: Used by model backends (models/gemini.py) and data sources (data_sources/konflux.py). No model-specific or data-source- specific logic.

models/gemini.py – Gemini model adapter

  • Owns: GeminiModel with generate(), make_user_content(), make_tool_responses(), to_canonical(), from_canonical().
  • Boundary: Translates between internal types and the Gemini REST API. Supports API key (direct) and Vertex AI (ADC) authentication modes.
  • Note: Parses cachedContentTokenCount from Gemini’s implicit server-side caching into Usage.cache_read_tokens.

models/anthropic.py – Anthropic model adapter (Vertex AI)

  • Owns: AnthropicVertexModel with generate(), make_user_content(), make_tool_responses(), to_canonical(), from_canonical().
  • Boundary: Translates between internal types and the Anthropic Messages API via Vertex AI rawPredict.
  • Note: make_tool_responses() uses _last_tool_ids instance state to pair tool results with tool-use IDs.
  • Note: generate() merges consecutive user messages before sending (Anthropic requires strict user/assistant alternation).
  • Note: generate() annotates messages with sliding-window cache breakpoints (cache_control) on shallow copies to avoid mutating contents. The ephemeral parameter skips cache writes for transient warning messages.

4. Security Model

The agent runs in a shared OpenShift cluster without cluster-admin access, processing merge requests from repositories where external contributors can submit code. The security model addresses two threat vectors: (1) the LLM executing arbitrary commands chosen by an attacker-controlled MR, and (2) credential leakage between the model, the sandbox, and orchestrator actions.

4.1. Sandbox isolation

All commands generated by the LLM run inside an ephemeral container, never on the orchestrator host. The sandbox has no credentials, no network, and no visibility into the orchestrator.

flowchart LR
    subgraph orchestrator [Orchestrator]
        Agent["Agent Loop"]
        Creds["Secrets<br/>(tokens, kubeconfig)"]
    end
    subgraph sandbox [Sandbox Container]
        Shell["sh -c commands"]
        Files["/tmp/data/<br/>(incl. _out/ spill dir)"]
    end
    Agent -->|"stdin pipe<br/>(write_file)"| sandbox
    Agent -->|"exec command"| sandbox
    sandbox -->|"stdout/stderr"| Agent
    sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
    sandbox -.-x|"NO access"| Creds

Podman (local development):

  • --network=none – complete network isolation; curl, wget, pip install all fail
  • --user 65532 – fixed non-root UID; no privilege escalation
  • No host volume mounts – data enters only via stdin piping through write_file()

Kubernetes (production):

Every field in the pod manifest is set explicitly for portability to vanilla Kubernetes with Pod Security Admission (restricted level), not just reliance on OpenShift’s restricted-v2 SCC admission:

  • automountServiceAccountToken: false – no K8s API access from sandbox
  • runAsNonRoot: true – enforced at pod level
  • seccompProfile: RuntimeDefault – required by restricted level
  • allowPrivilegeEscalation: false – container level
  • capabilities.drop: ["ALL"] – container level
  • activeDeadlineSeconds: 1800 – pod self-terminates after 30 minutes even if the orchestrator crashes; prevents orphaned pods
  • restartPolicy: Never – pod does not restart on failure
  • NetworkPolicy on the sandbox namespace blocks all egress from all pods in the namespace (podSelector: {}, egress: [])

The pod does not set runAsUser explicitly. On OpenShift, the SCC assigns a UID from the namespace range. On vanilla K8s, the image’s USER directive is used.

4.2. Credential separation (four-tier token model)

Tokens are split into four tiers with strict code-path separation:

flowchart TB
    subgraph tier1 [Tier 1: Orchestrator Tokens]
        OT["ORCHESTRATOR_GITLAB_TOKEN_*<br/>Operational reads<br/>Env vars only, never in YAML"]
    end
    subgraph tier2 [Tier 2: Workflow Action Tokens]
        AT["action_tokens in YAML<br/>Per-workflow bot identity<br/>Write scope (api)"]
    end
    subgraph tier3 [Tier 3: Model Tool Tokens]
        MT["GITLAB_TOKEN_RO, etc.<br/>Read scope (read_api)<br/>Env var NAMES in YAML"]
    end
    subgraph tier4 [Tier 4: Sandbox]
        SB["Zero credentials<br/>Zero network<br/>Zero SA token"]
    end

    OT -->|"used by"| OpsReads["actions.py<br/>(member access, push author,<br/>rate-limit notices)"]
    AT -->|"used by"| WfWrites["actions.py<br/>(notes, thread resolution)"]
    MT -->|"used by"| DataSources["data_sources/<br/>(GitLab, Konflux, TF)"]
    SB -->|"used by"| SandboxExec["sandbox_exec<br/>(sh -c commands)"]

    OpsReads -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| SandboxExec
    DataSources -.-x|"NEVER"| OpsReads
    DataSources -.-x|"NEVER"| WfWrites

Tier 1 – Orchestrator tokens. Tokens used by actions.py for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Resolved by convention from env vars: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (global fallback). The ORCHESTRATOR_ prefix is a structural safeguard – these env var names can never appear in the YAML config’s data_sources, tokens, or action_tokens sections. These tokens can be scoped to read_api since all per-workflow writes moved to Tier 2.

Tier 2 – Workflow action tokens. Per-workflow bot identity tokens (api scope) used by actions.py for all workflow-attributable GitLab writes: placeholder notes, discussion notes, and thread resolution. Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the YAML config as env var names at the project level:

projects:
  redhat/hummingbird/containers:
    action_tokens:
      gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS

Resolved by resolve_action_token(project_entry, ds_name) which reads the env var name from action_tokens and resolves it from os.environ. These tokens are passed as explicit token: str parameters to write functions in actions.py. They NEVER enter the ToolRegistry or model tool calls.

At startup, collect_workflow_bot_ids() authenticates each unique workflow token to build a set of all workflow bot user IDs. This set is used by handle_note for self-filtering (skipping notes from any workflow bot) and by find_session_for_reply for session marker scanning.

The naming convention groups tokens alphabetically: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.

Tier 3 – Model tool tokens. Read-only tokens (read_api scope for GitLab) used by data source modules during tool execution. Declared in the YAML config as env var names (not values):

data_sources:
  gitlab:
    token_env: GITLAB_TOKEN_RO          # name of the env var

Per-project overrides are possible:

projects:
  redhat/hummingbird/containers:
    tokens:
      gitlab: GITLAB_TOKEN_CONTAINERS_RO  # overrides token_env for this project

The resolution chain in resolve_tool_token() is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""

Storing env var names (not values) in YAML means the config file is safe to commit and audit. Actual secret values live in environment variables, injected via K8s Secrets at deployment time.

Tier 4 – Sandbox. The sandbox container has zero credentials, zero network access, and automountServiceAccountToken: false (no K8s API access). Data enters the sandbox only via write_file() (stdin piping). The model cannot instruct the sandbox to reach external APIs – it must use the orchestrator’s data source tools.

4.3. Namespace separation

Sandbox pods run in a dedicated namespace (hummingbird--agent-sandbox), separate from the orchestrator namespace (hummingbird--internal). This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens.

RBAC setup:

The orchestrator’s ServiceAccount gets a Role in the sandbox namespace (not its own namespace) granting only:

  • pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
  • pods/exec: create – command execution via kubectl exec

No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator needs only namespace-scoped permissions, so it works with standard OpenShift RBAC without requesting cluster-admin.

Konflux data is fetched via bearer token from kubeconfig credentials, not from inside the cluster. The orchestrator’s ServiceAccount does not need access to Konflux namespaces.

NetworkPolicy:

A deny-all-egress NetworkPolicy in the sandbox namespace uses podSelector: {} to match all pods and sets egress: []. The sandbox cannot reach the internet, the K8s API, or other pods in the cluster.

4.4. Security invariants

These must hold for the security model to be effective. Any change that violates one of these is a security regression:

  1. Orchestrator tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are resolved in actions.py only.

  2. Workflow action tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are passed as explicit token: str parameters within actions.py and runner.py only.

  3. Model tool tokens must NEVER flow into actions.py. They are resolved in workflow_config.py and consumed in data_sources/.

  4. The sandbox must NEVER have network access or credentials. No host mounts, no SA token, no env vars with secrets.

  5. The sandbox archive is NEVER extracted on the orchestrator. It is created inside one sandbox and restored inside another. The orchestrator only transports the bytes.

  6. No cluster-admin required. Only namespace-scoped resources (Role, RoleBinding, Pod, NetworkPolicy) are used.

  7. Agent-generated notes are skipped on re-processing. Notes containing SESSION_MARKER_PREFIX are filtered out in handle_note() to prevent infinite loops. Self-filtering checks against all workflow bot user IDs (collected at startup) plus the orchestrator bot ID.

  8. All auto-triggers require Developer access. handle_pipeline(), handle_merge_request(), and handle_note() each check check_member_access() on the event’s user before executing a workflow. Events from non-developers are silently skipped (or replied to with an access-denied message for slash commands). This ensures that the target MR’s work products – its diff, description, CI logs, and commit messages – originate from a trusted author. Fork MRs from external contributors are blocked because the MR author lacks Developer+ on the target project.

    Scope limitation: this gate only covers the target MR. Once the agent is running, the model controls tool arguments and can direct tools at content beyond the target MR – other MRs, other refs, other job IDs, even other projects reachable by the read-only token. The current mitigations are: (a) workflow prompts instruct the model to use the event’s {project, iid, sha}, (b) the model would need to be manipulated via prompt injection from already-trusted content to deviate, (c) the sandbox prevents the model from acting on manipulated reasoning beyond producing text output, and (d) the token is read-only with minimal scope. However, if a data source tool is added that returns user-authored prose from arbitrary resources (e.g. issue bodies, wiki pages), it should apply per-author trust filtering (invariant #10).

  9. Data source tools that accept URLs must validate them against an allowlist. tf_get_test_log restricts URLs to the Testing Farm artifacts prefix to prevent the LLM from directing the orchestrator to fetch arbitrary URLs (SSRF).

  10. Third-party commentary entering the model prompt must be trust-filtered. Data sources that feed text from users other than the event trigger into contents (discussion comments, issue bodies) must gate on project membership at Developer+ level per author. Content from untrusted authors must be redacted to a fixed placeholder, never sanitized or escaped – there is no reliable way to escape adversarial text for an LLM prompt. Discussions where all notes are untrusted must be dropped entirely. See §9.1 for the reference implementation in gitlab_get_mr_discussions.

This invariant covers commentary (what other people said about the MR), not work products (the diff, CI logs, commit messages). Work products are inherently the input to the agent’s analysis and cannot be content-filtered without defeating the agent’s purpose. For the target MR, their trust comes from invariant #8 (the event trigger is Developer+). For content the model fetches beyond the target MR, trust depends on the tool: discussion tools must filter per-author (#10), while code/log/metadata tools rely on the mitigations described in #7.

5. Configuration System

5.1. Two sources, strict separation

Configuration comes from exactly two sources with no overlap:

  • YAML config file (CONFIG_PATH): operational settings, workflow definitions, project allowlists, data source declarations, and token env var names. This file is safe to commit, review, and audit.
  • Environment variables: secrets only (API keys, GitLab tokens, kubeconfig paths). These are injected at deployment time via K8s Secrets.

This split exists because YAML provides structure, validation, and audit trails, while secrets must stay out of version control.

5.2. Config loading

Two config objects are built at startup:

Config (from config.py): loaded by Config.load(settings), where settings is the settings: section from the YAML file. Auth/bootstrap fields come from env vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, etc.). Operational fields come from the settings dict with sensible defaults.

AgentConfig (from workflow_config.py): loaded by workflow_config.load(path). Contains all workflow definitions, project entries, and a pre-built project_index.

The two are kept separate because Config is needed everywhere (model construction, sandbox creation, session storage), while AgentConfig is only needed for event routing and data source registration.

5.3. YAML config structure

settings:
  gitlab_url: https://gitlab.com         # operational, not a secret
  sandbox:
    image: quay.io/.../image:tag
    namespace: my-namespace              # K8s only
  model: gemini-3.1-pro-preview
  # model: claude-sonnet-4-5@20250929   # Anthropic via Vertex AI (alternative)
  max_iterations: 30
  max_runs_per_mr: 5
  internal_notes: true
  max_concurrent_agents: 4
  sqs_queue_url: ""                      # empty = no SQS (local dev)
  s3_session_bucket: ""                  # empty = no S3 (local dev)
  trigger_prefix: /hummingbird           # slash command prefix
  session_marker_prefix: hummingbird-session  # HTML comment marker ID
  auto_trigger: true                     # auto-run on failed pipelines

workflows:
  analyze-failures:
    prompt: workflows/analyze-failures.md  # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview                  # per-workflow override
    max_iterations: 50                     # per-workflow override

    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO         # env var name, not the value
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens:                            # per-project token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO

Design choices in this structure:

Workflow-first organization. Each workflow owns its project list, not the other way around. This scopes data source permissions per workflow-project pair. A future code-review workflow can have different GitLab tokens (with different scopes) than the analyze-failures workflow, with no ambiguity.

Token env var names in YAML (not values). The YAML file is committed to the repo. It contains token_env: GITLAB_TOKEN_RO (a name), not the actual token. The actual secret value is resolved at runtime via os.environ.get(env_var). This allows the config to be reviewed and audited without exposing secrets.

Inline cluster_url only. The Konflux cluster URL is operational configuration (it identifies which cluster to talk to), not a secret. Putting it inline in YAML makes it visible and auditable. The URL must include the namespace path (e.g. /ns/my-tenant).

5.4. Project index

At load time, _build_project_index() constructs a reverse lookup:

project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}

This provides O(1) lookup when a pipeline event arrives with a project path. A single project can appear in multiple workflows (e.g. both analyze-failures and a future code-review), and each matching workflow will be triggered independently.

5.5. Token resolution chains

Model tool tokens (for data source API calls):

resolve_tool_token(project_entry, ds_name, workflow_cfg):
  1. project_entry.tokens[ds_name]           -> per-project override
  2. workflow_cfg.data_sources[ds_name].token_env  -> workflow default
  3. "" (empty string)                        -> no token
  Each step resolves the env var NAME, then reads os.environ[name].

Workflow action tokens (for per-workflow GitLab writes):

resolve_action_token(project_entry, ds_name):
  1. project_entry.action_tokens[ds_name]  -> env var NAME
  2. os.environ[name]                      -> resolved token value
  3. "" (empty)                            -> not configured
  No fallback chain. Missing tokens cause validate_env to fail at startup.

Orchestrator tokens (for operational reads and infrastructure notes):

resolve_orchestrator_token(project_path):
  1. ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT>  -> per-project
  2. ORCHESTRATOR_GITLAB_TOKEN                     -> global fallback
  Mangling: "/" -> "_", "-" -> "_", uppercase.
  e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS

Cluster URL (for Konflux):

resolve_cluster_url(ds_cfg):
  -> ds_cfg.cluster_url (inline value in YAML)

5.6. Environment validation

validate_env(agent_cfg) checks at startup that all referenced env vars exist. It walks every workflow’s data sources, project token overrides, and project action token entries, collecting missing vars into a single error message. This catches configuration errors early instead of failing mid-run when a specific data source or workflow action is first used. Missing action_tokens env vars are treated the same as missing model tool tokens: startup fails unconditionally.

6. Agent Loop Design

For the wire-format walkthrough (what bytes go to the model API, what comes back), see Agent Model Loop. This section covers the design rationale behind the loop.

6.1. Full history replay

The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list from the beginning of the conversation. This means every large tool output sitting in history inflates every subsequent API call.

This property is fundamental to why auto-spill exists (section 7). Without auto-spill, a single cat of a 32KB file early in the conversation adds ~8K tokens to every remaining API call. Over a 15-iteration run, that is ~120K wasted tokens.

The alternative – conversation compaction (replacing old tool results with summaries) – was considered and deferred. Each provider has strict requirements about content structure (e.g. Gemini model turns must match the preceding tool turns; Anthropic enforces user/assistant alternation), and modifying history risks confusing the model or violating API constraints. Auto-spill solves 90% of the problem with none of the risk.

6.2. Budget model: iterations + context limit

The agent uses a dual-limit approach rather than a cumulative token budget:

Iteration limit (max_iterations, default 30 per workflow config). Hard cap on the number of model round-trips. This is the primary cost control lever. With auto-spill keeping per-call context bounded, iteration count is roughly proportional to cost.

Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens). Checked after each API call using response.usage.input_tokens. This is a safety net for cases where auto-spill is not sufficient (e.g., many small tool results that individually stay under the spill threshold but cumulatively fill the context).

Why not a cumulative token budget? Because with full history replay, each API call re-sends everything. “Cumulative billed tokens” double-counts: call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed total is 15K but actual new content is only 10K. Iteration count is a simpler and more predictable proxy for cost.

6.3. Two-tier budget escalation

Both limits use the same escalation pattern:

flowchart LR
    Normal["Normal<br/>tools available"]
    Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
    Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]

    Normal -->|"80% reached"| Soft
    Soft -->|"100% reached"| Hard

Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is injected into contents for that turn only, then popped before the response is persisted. Tools remain available so the model can finish in-progress work.

Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral user message AND tool_defs is set to [] (empty list) to physically prevent further tool calls. The model must produce text. This is more reliable than disabling tools at the API layer (toolConfig.functionCallingConfig.mode: NONE on Gemini, tool_choice: {"type": "none"} on Anthropic), which models sometimes ignore (Gemini may return UNEXPECTED_TOOL_CALL).

6.3.1 Ephemeral messages and Anthropic alternation

All per-turn warnings use ephemeral user messages: they are appended to contents before the API call and popped immediately after. This keeps the system prompt stable across all iterations and prevents warnings from polluting the conversation history saved in sessions.

Anthropic requires strict user/assistant alternation. The Anthropic adapter’s generate() merges consecutive user messages on a copy of contents before sending the request, so ephemeral warnings (and other adjacent user turns) do not break the API contract.

6.4. Empty response handling

Models occasionally return empty responses (no text, no tool calls). The agent retries up to MAX_EMPTY_RETRIES (2) times:

  1. Pop the empty response from contents. It adds nothing and may confuse the model on the next call.
  2. Inject EMPTY_RESPONSE_NUDGE as an ephemeral user message on the next turn: “Your previous response was empty. Please continue…”
  3. Continue the loop (consuming an iteration).

The nudge is delivered as an ephemeral user message (injected before the API call and popped after). This avoids mutating the system prompt and keeps the conversation history clean for session persistence.

The empty_retries counter resets to 0 after any successful iteration (one where tool calls were executed). This means the model gets fresh retries if it produces empty responses at different points in the conversation.

MALFORMED_FUNCTION_CALL handling. Models sometimes return a finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls. This is treated as a special case of empty response: the retry mechanism kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which tells the model to retry with simpler arguments and avoid large text payloads in tool call arguments.

6.5. Error recovery

Model API errors. _generate_with_retry() catches ModelError and retries up to MODEL_RETRY_COUNT (4) times if retryable is True (HTTP 5xx and 429). Non-retryable errors (4xx, auth failures) fail immediately. Retries use exponential backoff: MODEL_RETRY_BASE_DELAY * 2^attempt, capped at MODEL_RETRY_MAX_DELAY (60s), giving delays of 5s, 10s, 20s, 40s. Transport-level 429 retry is disabled on the model’s HTTP session (retry_429=False) so that rate-limit responses are handled at the model retry layer with proper backoff instead of being silently retried by urllib3.

Transport errors. Each model adapter’s generate() catches requests.Timeout and requests.ConnectionError from the HTTP call and wraps them as retryable ModelError. This makes timeouts (read and connect) and connection failures subject to the same retry logic as HTTP 5xx/429. Timeout is caught before ConnectionError because ConnectTimeout inherits from both. Note that the urllib3 retry adapter does not retry POST requests (not idempotent), so transport errors from model calls always propagate to our code.

Unexpected loop errors. run_agent_loop wraps the _generate_with_retry call in a try/except Exception that logs and breaks instead of propagating. This ensures the function always returns partial results (accumulated contents, transcript, sandbox state) even when an unexpected exception occurs (e.g. JSONDecodeError from a truncated API response). The caller saves the session normally – conversation history and sandbox archive are preserved for resumption. This follows the “partial over nothing” principle: 32 iterations of work are more valuable than a crash.

No failure-path session save. _execute_workflow does not save a session when run_workflow raises. With the agent loop catching unexpected errors, the failure path only fires for infrastructure errors (sandbox start, config) where there is no useful state. Not saving avoids overwriting a previous good session when a reply attempt fails.

Tool execution errors. _execute_tool_calls() wraps each tool_registry.execute() in a try/except. Unhandled exceptions are caught and returned to the model as {"error": "Tool X failed: ..."}. This prevents a single broken tool from crashing the entire session – the model sees the error and can adapt.

Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the TimeoutExpired exception is caught in the tool registry and returned as a structured error dict. The model can retry with a different command or proceed without the result.

6.6. Session resumption

When resuming from a previous session:

  1. initial_contents (the full contents list from the previous run) is prepended, followed by the user’s reply as a new user turn.
  2. CONTINUATION_PROMPT is appended to the system prompt.
  3. The sandbox is restored from the archived tarball.
  4. A fresh iteration budget starts from 0.

Saved sessions use a versioned JSON envelope (format_version: 1) whose contents are in a canonical (OpenAI-style) message format, independent of whether the run used Gemini or Anthropic. On resume, run_workflow converts initial_contents into the active model’s native wire format: for format_version >= 1, via model.from_canonical(); for legacy sessions without a version, GeminiModel.to_canonical() migrates Gemini-native history to canonical form first, then model.from_canonical() loads it into the current backend. After a successful run, model.to_canonical() converts the native contents back to canonical form before persistence.

CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g., analyze-failures.md) tells the model to follow a rigid multi-phase workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the entire analysis. The continuation prompt overrides this: “Do NOT re-run the full workflow. Respond directly to the user’s question.”

The practical constraint on resumed sessions is the context window, not iterations. The restored history from an 8-iteration cold start uses ~50-100K input tokens. Each new tool call adds more. The existing CONTEXT_LIMIT check still applies and will force wrap-up if the context grows too large.

Canonical session format (rationale)

Storing sessions in canonical form decouples persisted history from any one provider’s JSON shape. The same saved thread can be resumed on a different model adapter (including cross-provider migration) because conversion happens at load and save boundaries only; the agent loop continues to treat native contents as opaque between those steps.

6.7. Prompt caching

Since the agent replays the full conversation history on every API call (see 6.1), prompt caching reduces the cost of re-processing unchanged prefixes. The two model backends handle caching differently.

Gemini. Vertex AI caches prefixes implicitly on the server side. The adapter parses cachedContentTokenCount from usageMetadata and reports it as cache_read_tokens. No request-side changes are needed.

Claude. Vertex AI does not support Anthropic’s automatic caching (which requires opt-in at the API level and is not yet available on Vertex). The adapter uses explicit cache_control: {"type": "ephemeral"} annotations on message content blocks. Up to 4 breakpoint slots are available per request; the agent uses 2.

Why 2 breakpoints on messages, not 4 on system/tools/messages. The Anthropic prefix hash is cumulative: it covers everything from the start of the request (tools, system prompt, messages) up to the annotated block. A single breakpoint on a message already caches the entire prefix including tools and system. Separate breakpoints on earlier components would be redundant and waste slots. Using only 2 of the 4 slots leaves room for future use.

Why a sliding window. The conversation grows by 2 messages per turn (assistant response + user tool result). Two breakpoints slide forward in lockstep:

  • B2 on the last message writes the full prefix to cache.
  • B1 on the previous B2 position reads the prior prefix from cache.

Each call after the first gets a cache hit for the entire prefix minus the newest turn. _prev_cache_index on the model instance tracks where B2 was placed so that B1 can be positioned on the next call.

Why shallow copies. The contents list is owned by the agent loop and persisted in sessions. _annotate_cache_breakpoints() creates a shallow copy of the messages list and replaces only the B1/B2 entries with copies that have cache_control injected. The originals are never mutated, so no cleanup is needed after the API call and cache_control never leaks into saved sessions.

Ephemeral message interaction. When the agent loop injects a transient message (see 6.3.1), generate() receives ephemeral=True. B2 is not placed (no cache write) so the transient content never enters the cache. _prev_cache_index is not updated, so the next non-ephemeral call’s B1 still points to the last valid write and produces a cache hit. B1 is still placed to provide a cache read for the stable prefix on the ephemeral call itself.

Cost estimation. MODEL_PRICING in config.py stores a 4-tuple per model prefix: (input, output, cache_read, cache_write) per million tokens. estimate_cost() computes: uncached * input + cache_read_tokens * cache_read_rate + cache_creation_tokens * cache_write_rate + output * output_rate. Gemini has cache_write = 0 (implicit caching has no write surcharge); Claude has non-zero cache_write (1.25x the base input rate for explicit breakpoints).

7. Tool System and Auto-Spill

7.1. Tool registry design

ToolRegistry is the single point of dispatch for all tool calls. The agent calls registry.execute(tool_call) and gets a dict back. It never calls sandbox methods or data source functions directly.

Three built-in tools are always available:

  • sandbox_exec – runs sh -c <command> in the sandbox container. Returns {exit_code, stdout, stderr}, with auto-spill for large outputs.
  • fetch_to_sandbox – calls a data source function and writes the result to a specified path in the sandbox. Returns metadata only (path, byte count, line count). Used when the model wants explicit path control.
  • fetch_batch_to_sandbox – calls multiple fetch_to_sandbox in one tool call. Saves iterations vs. sequential calls (e.g., fetching both pipelineruns and taskruns in one round-trip).

Data source tools are registered dynamically per workflow config. Each data source module provides a register() function that creates ToolDef objects and closures over credentials, then calls registry.register_data_source(tool_def, func, response_metadata).

7.2. Auto-spill architecture

Auto-spill is the key mechanism for keeping the LLM context bounded. Without it, large outputs accumulate in the contents list and inflate every subsequent API call (because both APIs replay the full history).

flowchart TD
    Output["Tool produces output"]
    SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
    Inline["Return full output inline"]
    Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
    Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]

    Output --> SizeCheck
    SizeCheck -->|"<= 4 KB"| Inline
    SizeCheck -->|"> 4 KB"| Spill --> Preview

Three spill paths share the same _spill_counter to prevent filename collisions:

sandbox_exec spill. When stdout or stderr exceeds OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to /tmp/data/_out/{counter}.txt via _spill_field(). The model receives:

{
  "exit_code": 0,
  "stdout": "<first 4KB>",
  "stdout_truncated": true,
  "stdout_file": "/tmp/data/_out/0.txt",
  "stdout_bytes": 32768,
  "stdout_lines": 1024,
  "stdout_tail": "<last 512 bytes>"
}

The preview (head + tail) gives the model enough context to decide whether to process the full file with jq/grep/head.

Data source inline spill. When a direct data source call returns text larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes it to /tmp/data/_out/{name}_{counter}.txt and returns:

{
  "saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
  "bytes": 85432,
  "lines": 2100,
  "preview": "<first 4KB>"
}

Streaming responses (e.g., gitlab_get_repo_archive which returns a StreamingResponse with suffix .tar.gz) are written via _spill_streaming(). Text streams (binary=False, the default) capture a UTF-8 preview from the head while piping chunks to the sandbox. Binary streams (binary=True) skip preview capture entirely and return only {saved_to, bytes}, avoiding meaningless decoded output for formats like tar.gz. The StreamingResponse.suffix field controls the file extension (.jsonl, .tar.gz, .log).

Non-streaming binary responses are written to .bin files with no preview via _spill_binary().

Sandbox-optional mode. ToolRegistry accepts sb=None for use cases that do not require a sandbox (e.g. evaluation harnesses that only use native model tools like Google Search grounding). When no sandbox is available:

  • Sandbox tools (sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox) are omitted from get_tool_defs() so the model never sees them.
  • Text data sources (inline and streaming) degrade to truncated previews (OUTPUT_PREVIEW_BYTES). The model receives {preview, truncated: true} instead of {saved_to, preview}. Streaming text is consumed into a memory buffer up to the preview limit and the rest is discarded.
  • Binary data (streaming and non-streaming) returns an error with the byte count ({error, bytes}). Binary content cannot be meaningfully previewed as text.

fetch_to_sandbox spill. Always writes to the caller-specified path (not /tmp/data/_out/). Returns metadata only (no preview). The model uses this when it wants a specific filename for later processing.

7.3. Why auto-spill instead of rejecting large outputs

The earlier design rejected large data source responses with “use fetch_to_sandbox instead.” This caused two wasted iterations per rejection: the model makes the call, gets rejected, then has to repeat with fetch_to_sandbox. With auto-spill, the data flows to a sandbox file transparently. Validation showed ~10% fewer iterations with auto-spill.

7.4. Why fetch_to_sandbox still exists alongside auto-spill

Auto-spill handles the common case, but fetch_to_sandbox provides:

  • Explicit path control. The model can choose meaningful filenames (/tmp/data/pipelineruns.json) instead of getting auto-generated names (/tmp/data/_out/konflux_list_pipelineruns_1.txt).
  • Batch fetching. fetch_batch_to_sandbox combines multiple fetches in one tool call, saving iterations.
  • No preview overhead. fetch_to_sandbox returns metadata only, which is useful when the model knows it will process the file with sandbox_exec anyway.

7.5. ToolDef notes

ToolDef has an optional notes field for domain knowledge that belongs in the system prompt but not in the tool’s JSON schema. Examples:

  • Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and the two label selectors (BUILD vs. TEST).
  • Testing Farm notes explain XML result structure and usage patterns.

ToolRegistry.get_tool_notes() collects all non-None notes into a ## Data Source Notes section that is prepended to the workflow body in the system prompt:

BASE_SYSTEM_PROMPT + tool_notes + workflow_body

This keeps domain knowledge close to the tool definitions (in the data source module) rather than duplicated in every workflow .md file.

7.6. Response metadata

register_data_source() accepts optional response_metadata – a dict that is merged into every response from that tool (inline, auto-spill, and fetch_to_sandbox). Used for:

  • Konflux: {"konflux_ui": "https://..."} so the model can build reviewer-facing PipelineRun links.
  • Testing Farm: {"artifacts_base": "https://..."} so the model can build artifact links.

This avoids having the model ask “what is the Konflux UI URL?” – the information arrives with every tool response.

8. Sandbox Architecture

8.1. The Sandbox protocol

All backends implement an 8-method typing.Protocol:

class Sandbox(Protocol):
    def start(self) -> None                                                  # create container/pod, pre-create /tmp/data/
    def exec(command, *, stdin_data: bytes | None) -> ExecResult             # sh -c in sandbox
    def write_file(path, data: bytes) -> None                               # stdin pipe: cat > path
    def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int]    # Popen stdin pipe, returns (bytes, lines)
    def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult         # Popen with streaming stdin
    def read_file_iter(path) -> Iterator[bytes]                             # Popen stdout pipe in chunks
    def cleanup(self) -> None                                               # rm -f container / delete pod
    def linger(session_id) -> None                                          # keep alive for reuse, or fall back to cleanup

ExecResult is (exit_code: int, stdout: str, stderr: str).

stream_to_file() and stream_exec() use subprocess.Popen with a stdin pipe, writing chunks incrementally. This avoids buffering large payloads in orchestrator memory (e.g., streaming a tar.gz archive into the sandbox). read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.

write_file() uses stdin piping (cat > path), never host volume mounts. This is critical: it means data flows through the orchestrator process, not through a shared filesystem. The sandbox has no host mounts.

8.2. Backend selection

Five sandbox backends are available:

Backend --sandbox When to use
PodmanSandbox podman Local development (default for run)
K8sSandbox k8s Direct pod creation on a K8s cluster
K8sPoolSandbox k8spool Pre-warmed pool via Deployment (default for serve)
KubeVirtSandbox kubevirt Direct VMI creation with SSH exec
KubeVirtPoolSandbox kubevirtpool Pre-warmed VMI pool via VMIRS

create_sandbox() factory handles podman, k8s, and kubevirt. Pool backends (k8spool, kubevirtpool) are handled in __main__.py, which creates a SandboxPool or VmiPool and passes it to run_workflow().

The sandbox backend can be set per-workflow via the sandbox: field in the workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox

default. This allows different workflows to use different backends (e.g., lightweight code review uses pod pool, heavier analysis uses VM pool).

K8s is never auto-detected from the environment. It requires explicit CLI flags. This prevents accidental use of a K8s sandbox when developing locally.

For K8s namespace resolution (k8s and k8spool):

  1. If --namespace is provided, use it.
  2. If --context is provided, extract the namespace from the kubeconfig context. If the context has no default namespace, raise an error.
  3. If neither is provided (in-cluster), use sandbox.namespace from the config file.

8.3. PodmanSandbox

podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
    --workdir /tmp {image} sleep infinity
  • Unique container name with UUID suffix prevents collisions
  • sleep infinity keeps the container alive for repeated exec calls
  • --network=none provides complete network isolation
  • --user 65532 is a fixed non-root UID (matches nonroot in distroless)
  • Cleanup: podman rm -f (force, in case exec is still running)

8.4. K8sSandbox: the hybrid approach

The K8s sandbox uses a hybrid of two tools:

  • kubernetes Python library for pod lifecycle: create_namespaced_pod, read_namespaced_pod (poll for Running), delete_namespaced_pod.
  • kubectl exec subprocess for command execution.

Why not use the kubernetes Python library for exec too? Three problems discovered during development:

  1. No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions 1-4 have no mechanism to signal “stdin is done.” Commands like cat > /file hang forever waiting for more input. Python client v5 support does not exist.

  2. BrokenPipeError on large stdin. Sending more than ~1MB through the WebSocket stream() API causes pipe errors, breaking write_file() for large data source responses.

  3. Unbounded memory from stream(). The stream() function accumulates all stdout/stderr data in memory with no streaming control. A command producing megabytes of output would consume unbounded memory.

kubectl exec as a subprocess avoids all three problems and provides the same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr captured, exit code from return code. The implementation in exec() is nearly identical between PodmanSandbox and K8sSandbox.

8.5. Pod manifest design

The K8s pod manifest is built by _build_pod_manifest():

metadata:
  labels:
    app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
  automountServiceAccountToken: false  # no K8s API from sandbox
  activeDeadlineSeconds: 1800          # 30min hard timeout, backstop
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    seccompProfile: RuntimeDefault
  containers:
  - name: sandbox
    command: ["sleep", "infinity"]
    workingDir: /tmp
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
    resources:
      requests: {cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi}
      limits: {cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi}

runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC assigns a UID from the namespace UID range. On vanilla K8s, the image’s USER directive is used. Setting an explicit UID would conflict with OpenShift’s SCC admission.

8.6. Auth modes

  • Local development: --context flag passes the kubeconfig context to config.new_client_from_config(context=...), creating a per-instance ApiClient.
  • Production (in-cluster): context=None triggers config.load_incluster_config(), using the pod’s ServiceAccount token.

The kubectl exec commands include --context when running locally but omit it when in-cluster (kubectl uses the default in-cluster config).

8.7. Pre-created directories

Both backends run mkdir -p /tmp/data in start() after the container/pod is up. This provides a working directory for the model without consuming an iteration. /tmp/data/_out (the spill directory) is created on demand by mkdir -p $(dirname ...) in write_file().

8.8. K8sPoolSandbox: Deployment-backed pool

On-demand pod creation adds 5-30 seconds of latency per workflow (image pull, scheduling, container start). For interactive use (slash commands, reply-based resumption), this delay is user-facing. The pool eliminates it.

Mechanism. A Kubernetes Deployment maintains a set of pods labelled hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim() finds a Running standby pod, patches it to role: active, clears its ownerReferences (detaching it from the ReplicaSet), and annotates it with hummingbird/reap-by (an absolute UTC deadline). The Deployment controller sees the ReplicaSet is under the desired replica count and creates a replacement.

Pod lifecycle.

  1. Deployment creates pod -> role: standby (managed by ReplicaSet)
  2. claim() patches -> role: active, ownerReferences: [], reap-by: now + max_active_seconds
  3. Agent uses the pod via inherited K8sSandbox.exec()
  4. On success: linger() patches -> reap-by: now + linger_seconds, session-id: <id> (pod stays alive)
  5. On reply within linger window: try_reclaim() patches -> reap-by: now + max_active_seconds, clears session-id
  6. On failure or linger expiry: pod is deleted (by cleanup() or the reaper)

Pod lingering. After a successful workflow, pool sandboxes are kept alive for linger_seconds (default 300) instead of being deleted immediately. The pod is annotated with hummingbird/session-id to link it back to the session. If a user reply arrives within the linger window, try_reclaim() finds the pod by session-id, clears the annotation (marking it as in-use), and resets reap-by. This skips pod creation and S3 archive restoration. If no reply arrives, the reaper deletes the pod when its reap-by deadline passes.

Concurrency safety. session-id is only present while lingering. Its absence during active execution prevents concurrent replies from adopting the same pod. try_reclaim() is serialized by a threading lock; the first caller wins, others fall back to S3 restore with a fresh pod.

Reaping. reap_expired() is called once per workflow execution (after sandbox acquisition, before the model loop). It performs a single sorted pass over all active pods:

  1. Non-lingering pods (no session-id) past their reap-by deadline are deleted immediately.
  2. Lingering pods (with session-id) are sorted by reap-by ascending. The loop deletes pods that are either expired (now > reap-by) or exceed max_lingering_pods (default 2, configurable), starting with those closest to their deadline. The loop stops when both conditions are satisfied (now <= reap-by and remaining count <= limit).

This replaces the previous design where _reap_expired() ran inside claim() on every poll iteration. Moving it to a once-per-workflow call reduces API load and centralizes cleanup. The max_lingering_pods cap prevents unbounded pod accumulation from many short-lived workflows.

Indefinite wait. claim() polls until a pod is available or the shutdown event fires. It logs at DEBUG level on each poll, escalating to WARNING after ~2 minutes. This is normal behavior when the pool (Deployment replicas) is smaller than max_concurrent_agents – the SQS semaphore caps concurrency, so demand will not permanently exceed supply.

Config simplification. In pool mode, the pod image, resources, metadata, and security context are defined solely in the Deployment template. The agent config only needs sandbox.namespace and sandbox.active_deadline_seconds. This eliminates duplication between the agent configmap and the Deployment.

Class design. K8sPoolSandbox inherits from K8sSandbox to reuse exec(), write_file(), read_file_iter(), and cleanup(). It overrides __init__() (does not call super().__init__() since that resolves kubeconfig), start() (claims from pool instead of creating a pod), and linger() (patches annotations instead of deleting). It adds start_from_existing() for the reclaim path. All sandbox backends implement linger(session_id): non-pool backends fall back to cleanup().

8.9. KubeVirtSandbox: SSH-based VM execution

KubeVirtSandbox provides full VM isolation using KubeVirt VirtualMachineInstances. It parallels K8sSandbox but uses CustomObjectsApi for VMI lifecycle and ssh for command execution instead of kubectl exec.

Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides virtctl console (serial) and VNC, but these are not suitable for programmatic command execution. SSH provides a reliable, well-understood exec channel with stdin/stdout/stderr piping.

SSH key injection. The VM sandbox image includes a custom inject-ssh-keys.service that reads SSH public keys from a KubeVirt Secret volume attached as a virtio disk with serial ssh-pubkeys (visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to /root/.ssh/authorized_keys before sshd starts. This avoids the need for cloudInitNoCloud or qemu-guest-agent.

Single-VM lifecycle:

  1. Generate an ephemeral ed25519 key pair via ssh-keygen
  2. Create a K8s Secret with the public key
  3. Create the VMI referencing the containerDisk image and the SSH Secret
  4. Poll VMI status until Running with a pod network IP
  5. Poll SSH readiness (ssh sandbox@ip true)
  6. Execute workflow commands via SSH
  7. Cleanup: delete VMI, delete Secret, remove temp key files

Polling strategy. Two-phase wait: first poll status.phase via get_namespaced_custom_object until Running (same interval as pod polling), then poll SSH readiness with ConnectTimeout=5. Total timeout matches POD_START_TIMEOUT (300s) for VMI startup plus _SSH_READY_TIMEOUT (120s) for SSH.

8.10. KubeVirtPoolSandbox: VMIRS-backed pool

KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox + SandboxPool, operating on VMI custom resources instead of Pods.

VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native controller maintains a desired number of identical VMIs. When a VMI is claimed (label flipped to active, ownerReferences cleared), the VMIRS automatically creates a replacement – the same pattern as a Deployment replenishing claimed pods.

Shared SSH key. Pool mode uses a single key pair for all standby VMIs. The public key is stored in a K8s Secret referenced by all VMIs in the VMIRS template. The private key path is provided via sandbox.vm_ssh_private_key in the config. On pool restart, a new key pair can be generated; existing VMIs drain naturally with the old key.

Claim/reap/linger semantics. Identical to the pod pool: claim() polls for Running VMIs with hummingbird/role=standby, patches to active, clears ownerReferences. reap_expired() deletes VMIs past their hummingbird/reap-by deadline. try_reclaim() finds lingering VMIs by hummingbird/session-id. The only difference is the API: all operations go through CustomObjectsApi with kubevirt.io/v1 virtualmachineinstances.

Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox to reuse exec(), write_file(), read_file_iter() (all SSH-based). It overrides __init__() (takes pool, skips key generation), start() (claims from pool), cleanup() (only deletes VMI, not Secret/keys which are pool-owned), and linger() (patches annotations).

9. Data Sources

Data sources are external APIs wrapped as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (inline or auto-spilled). Each data source module follows the same pattern:

  1. Define TOOL_DEFS – a list of ToolDef objects with names, descriptions, parameter schemas, and optional notes.
  2. Define implementation functions that take a pre-configured client as the first argument.
  3. Provide a register() function that creates the client, wraps implementations with functools.partial, and calls registry.register_data_source() for each tool.

Credentials are captured at registration time via functools.partial closures. They are never stored as global state and never leak into tool definitions or model contents.

9.1. GitLab

Uses python-gitlab library with retry_transient_errors=True for automatic retry on transient HTTP errors.

Tools:

  • gitlab_get_mr_details – MR metadata (title, author, SHA, labels, URLs)
  • gitlab_get_commit_statuses – all CI/CD statuses for a commit SHA (paginated automatically). Covers both Konflux external statuses and native GitLab CI job statuses. Each status includes target_url (job page link containing the job ID) and allow_failure.
  • gitlab_get_mr_diff – changed files in the MR
  • gitlab_get_file_at_ref – raw file content at a git ref
  • gitlab_get_repo_archive – repository tar.gz via repository_archive(iterator=True). Returns a StreamingResponse with chunked binary data (suffix .tar.gz), so the archive streams to the sandbox without buffering in orchestrator memory.
  • gitlab_get_job_log – job trace (log output) for a GitLab CI job. Uses lazy=True on the job object and trace(iterator=True) for streaming. Each chunk is decoded with errors='replace' and ANSI escape codes are stripped per-chunk before re-encoding. Returns a StreamingResponse (suffix .log). Per-chunk ANSI stripping is safe because escape sequences are <20 bytes and chunks are 1024+ bytes.

No response_metadata is set because GitLab commit statuses already contain target_url fields that the model uses for linking.

gitlab_get_mr_discussions – trust filtering and redaction.

MR discussions are the first data source that feeds user-generated free text into the model prompt. Unlike diffs and CI logs (which are code or machine output), discussion comments can contain arbitrary prose written by anyone who can post on the MR – including external contributors on public projects. This creates a prompt injection vector: an attacker posts a comment containing instructions that manipulate the model’s review output.

The discussions tool addresses this with a three-layer defence:

  1. Author trust gate. Each note’s author is checked for Developer+ access (level >= 30) on the project via members_all.get(author_id). Results are cached per author_id within a single call to avoid repeated API lookups. System notes (merge events, label changes) bypass the author check. Agent-authored notes are trusted via their author’s Developer+ access (the bot account holds Developer access on configured projects). Note: agent note detection for redaction purposes (stripping transcripts and footers) uses session marker presence, but trust is always based on the author’s access level, not the marker. This prevents marker injection from granting trust to non-member notes.

  2. Redaction, not sanitization. Untrusted notes in a discussion that also contains trusted notes are replaced with a fixed placeholder ([redacted: non-member comment]). The placeholder preserves the conversation structure (the model sees that someone replied, but not what they said). Discussions where all notes are untrusted are dropped entirely. No attempt is made to sanitize or escape untrusted content – there is no reliable escaping mechanism for LLM prompts, so the only safe option is to withhold the content entirely.

  3. Agent note stripping. Agent-authored notes contain large <details><summary>Agent transcript</summary>...</details> blocks (15K-90K chars of raw tool calls), metadata footers, and session markers. These are stripped before the note enters the model prompt, leaving only the review text. This serves dual purposes: token efficiency and avoiding feeding the model its own raw tool call history (which could cause degenerate self-referential loops).

Residual risks and scope limitations:

  • A compromised Developer+ account can inject adversarial content. This is accepted as equivalent to the existing risk of a compromised developer pushing malicious code (which the agent would also process).
  • The trust threshold is project-level (Developer role on the project), not MR-level. A developer on the project can influence any MR’s review.
  • Comments posted after the discussions are fetched but before the review is posted are not seen. This is a TOCTOU gap but has no security impact (the model simply misses late comments).

9.2. Konflux

Uses raw requests against K8s and Kubearchive APIs (not the kubernetes Python SDK). This avoids the heavy kubernetes client dependency for what is essentially bearer-token HTTP with label selectors.

Dual-fetch architecture:

flowchart LR
    subgraph fetch [Fetch Phase]
        KA["Kubearchive<br/>(historical)"]
        K8s["K8s API<br/>(live)"]
    end
    Combine["Combine items"]
    Dedup["Deduplicate<br/>by metadata.uid"]

    KA --> Combine
    K8s --> Combine
    Combine --> Dedup

For each resource type (PipelineRuns, TaskRuns), the client fetches from both Kubearchive (completed/historical resources) and the live K8s API, combines the results, and deduplicates by metadata.uid. This ensures no resources are missed regardless of whether they have been archived yet.

Two label selectors. Konflux uses different labels for BUILD and TEST PipelineRuns:

  • BUILD: pipelinesascode.tekton.dev/sha=<commit_sha>
  • TEST: pac.test.appstudio.openshift.io/sha=<commit_sha>

Each get_pipelineruns()/get_taskruns() call queries both selectors and combines the results.

Pod log fetching. get_pod_log() accepts an optional container parameter. When specified, it fetches logs for that single container. When omitted, it discovers all containers from the pod spec and concatenates their logs with === container_name === headers. Each individual log fetch tries Kubearchive first, then falls back to the live K8s API. Logs may be unavailable from both sources if the pod has expired.

Streaming pagination. K8s list endpoints can return pages of 80+ MB when a commit touches many components (e.g. 500 PipelineRuns per page). iter_paginated() uses session.get(stream=True) and ijson.parse() to stream-parse each page: items are yielded one at a time via ObjectBuilder, and the metadata.continue pagination token is captured from the same parse pass. resp.raw.decode_content = True is set so urllib3 transparently decompresses gzip/deflate Content-Encoding inline (Kubearchive returns gzip-compressed responses). Only one item is in memory at a time – O(single_item) regardless of page size. Truncated or malformed streams raise IncompleteJSONError, which is caught and treated like a network error (log a warning, stop iterating that source).

Credential resolution. KonfluxClient.__init__() parses the kubeconfig file to extract the API server URL and bearer token for the cluster. The cluster domain is extracted from the cluster_url config value. This approach avoids depending on kubectl or the kubernetes Python library for API authentication.

Response metadata: {"konflux_ui": "https://konflux-ui.apps.<domain>/ns/<namespace>"} is merged into every response so the model can build reviewer-facing links like [{name}]({konflux_ui}/pipelinerun/{name}).

9.3. Testing Farm

Uses requests.Session with a module-level retry adapter (429, 5xx) for resilience against transient errors.

Tools:

  • tf_get_results – JUnit XML results for a request ID
  • tf_get_test_log – individual test log by URL (from results.xml); restricted to the ARTIFACTS_BASE prefix to prevent SSRF
  • tf_get_request_status – request state, queue/run times

ToolDef notes on tf_get_results document the XML structure: //testsuites/@overall-result, //testcase/@result, //testcase/logs/log with @name and @href. This domain knowledge goes into the system prompt so the model knows how to parse the XML with sandbox_exec using python3 xml.etree.ElementTree.

Response metadata: {"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"} is merged into every response so the model can build artifact links like [{request_id}]({artifacts_base}/{request_id}/).

9.4. Registration flow

data_sources.register_selected() is the entry point called by runner.py. It iterates the workflow config’s data_sources dict and registers only the declared sources:

for ds_name, ds_cfg in wf_cfg.data_sources.items():
    if ds_name == "gitlab":
        token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
        gitlab.register(registry, gitlab_url, token)
    elif ds_name == "konflux":
        cluster_url = resolve_cluster_url(ds_cfg)
        kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
        konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
    elif ds_name == "testing_farm":
        testing_farm.register(registry)

This selective registration means a workflow with data_sources: {gitlab: {...}} only exposes GitLab tools to the model. Konflux and Testing Farm tools do not appear in the tool definitions, preventing the model from attempting to use unconfigured data sources.

9.5. Data flow: streaming vs buffered

Data moves from external APIs through the orchestrator to the model (inline or auto-spilled to sandbox). The memory profile of each tool depends on whether the HTTP response is consumed incrementally or buffered entirely.

Streaming (preferred for large/unbounded responses). The HTTP response is consumed incrementally – via ijson stream parsing, iterator=True in python-gitlab, or lazy pagination. Only one chunk or item is in memory at a time. Used by:

  • iter_paginated() (Konflux) – session.get(stream=True) + ijson
  • get_commit_statuses() (GitLab) – statuses.list(iterator=True) via _iter_commit_statuses(), yielding JSONL bytes one status at a time
  • get_repo_archive() (GitLab) – repository_archive(iterator=True), returns chunked tar.gz binary via StreamingResponse(suffix=".tar.gz", binary=True) – no text preview is generated
  • get_job_log() (GitLab) – trace(iterator=True) with per-chunk ANSI stripping, returns StreamingResponse(suffix=".log")

All produce StreamingResponse objects that flow through stream_to_file() into the sandbox without accumulating in orchestrator memory. StreamingResponse.suffix controls the auto-spill filename extension (.jsonl, .tar.gz, .log). StreamingResponse.binary controls whether a head preview is captured (False for text, True to skip for opaque binary formats).

Buffered (acceptable for small/bounded responses). The full response is loaded into memory as a string or dict. This is fine when the response size is bounded and small (typically < 1 MB). Used by:

  • get_mr_details() (GitLab) – single MR object, < 10 KB
  • get_file_at_ref() (GitLab) – single file, bounded by repo constraints
  • tf_get_results(), tf_get_test_log(), tf_get_request_status() (Testing Farm) – XML/text/JSON, typically < 1 MB

Buffered with risk (candidates for future streaming). Same buffered pattern but the response size is not bounded by design. Auto-spill mitigates the context growth problem (large outputs are written to sandbox files), but the orchestrator still spikes RSS during the fetch:

  • get_pod_log() (Konflux) – resp.text per container, concatenated. Multi-container pods accumulate all logs.
  • get_mr_diff() (GitLab) – full diff JSON, scales with MR size. GitLab truncates server-side but the result can still be large.

Invariant: paginated K8s list endpoints must always use streaming. Page sizes scale with the number of components in the commit – a single page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON). Buffering these responses risks OOM under normal production workloads, especially with concurrent workflows.

10. Event Pipeline and Production Operations

10.1. Two-stage SQS pipeline (ingress router + FIFO worker)

Events flow through a two-stage pipeline to serialize per-session work:

flowchart LR
  STD["Standard SQS<br/>(ingress)"]
  R["Router"]
  FIFO["SQS FIFO<br/>(work queue)"]
  W["Worker<br/>(poll_loop)"]

  STD -->|ReceiveMessage| R
  R -->|"SendMessage<br/>MessageGroupId"| FIFO
  W -->|"continuation<br/>group=session_id"| FIFO
  FIFO -->|ReceiveMessage| W

Ingress router. events.router_loop() is a single-threaded loop that long-polls the standard (ingress) queue, peeks into each SNS envelope to assign a MessageGroupId, forwards the raw message body to the FIFO queue, then deletes from the standard queue.

  • Note events: MessageGroupId = discussion_id from the webhook body. This serializes replies to the same agent session – FIFO delivers at most one in-flight message per group.
  • Pipeline / MR events: MessageGroupId = SQS MessageId (unique per message). No serialization – each event is its own group.
  • MessageDeduplicationId: Always the SQS MessageId from the standard queue. Absorbs duplicate deliveries from standard SQS’s at-least-once semantics (5-minute dedup window).

If SendMessage to FIFO fails, the message is not deleted from the standard queue and retries via visibility timeout.

FIFO worker. events.poll_loop() long-polls the FIFO queue, decodes SNS envelopes, and dispatches to handle_event. The max_concurrent_agents semaphore caps parallel groups being processed. FIFO guarantees that within a group, only one message is in-flight.

Two-phase processing. When the FIFO worker picks up a webhook (Phase 1), the handler validates the event, resolves the session_id, then posts a continuation message back to the same FIFO with MessageGroupId = session_id. The Phase 1 message is deleted quickly (~1-5s). Phase 2 processes the continuation: loading session state, building WorkflowRequest, and calling _execute_workflow. Because Phase 2 continuations share a MessageGroupId per session, all work for a given session is serialized – even if it spans multiple GitLab discussion threads.

Uniform message format. Both webhook messages (from SNS) and continuation messages use the same SNS-style envelope with gzip+base64 compression. encode_envelope() is the inverse of decode_sns_message(). The consumer doesn’t need to distinguish between webhook and continuation messages at the transport layer.

Graceful shutdown. SIGTERM/SIGINT set a shutdown_event. Both loops exit. Messages in SQS become visible again after the visibility timeout for other consumers.

Why two queues. The standard queue receives events from SNS (via subscription). The FIFO queue serializes per-session work. The gitlab-event-forwarder and SNS subscription are unchanged – the routing logic lives entirely in the agent codebase.

10.2. SNS envelope decoding

Messages arrive as SNS notifications with:

  • MessageAttributes.source – event source (e.g., "gitlab")
  • MessageAttributes.event_type – event type (e.g., "pipeline", "merge_request", "note")
  • MessageAttributes.content_encoding – "gzip+base64" for gitlab-event-forwarder, absent for kubernetes-event-forwarder
  • Message – the actual event body (JSON string, or gzip+base64 encoded)

decode_sns_message() handles both encoding formats transparently.

10.3. Event routing

runner.handle_event() dispatches on (source, event_type):

flowchart TD
    Event["Event arrives"]
    Check{"source/type?"}
    Cont["handle_continuation()"]
    Pipeline["handle_pipeline()"]
    MRHandler["handle_merge_request()"]
    Note["handle_note()"]
    Ignore["Ignore"]

    Event --> Check
    Check -->|"agent/continuation"| Cont
    Check -->|"gitlab/pipeline"| Pipeline
    Check -->|"gitlab/merge_request"| MRHandler
    Check -->|"gitlab/note"| Note
    Check -->|"other"| Ignore

    Pipeline --> FilterSource{"source = merge_request_event?"}
    FilterSource -->|"yes"| AutoResolve["auto-resolve on success<br/>(unconditional)"]
    FilterSource -->|"no"| Skip1["Skip"]
    AutoResolve --> PipeRules["evaluate trigger_rules<br/>(status + user + branch + title)"]
    PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
    PipeRules -->|"deny"| Skip1a["Skip"]

    MRHandler --> FilterMRAction{"action in open/update?<br/>draft = false?"}
    FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules<br/>(user + branch + title)"]
    FilterMRAction -->|"no"| Skip1b["Skip"]
    MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
    MRRules -->|"deny"| Skip1c["Skip"]

    Note --> FilterMRNote{"MR note?<br/>action = create?"}
    FilterMRNote -->|"yes"| NoteType
    FilterMRNote -->|"no"| Skip3["Skip"]

    NoteType{"Note type?"}
    NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
    NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
    NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()<br/>→ _enqueue_continuation()"]
    NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
    NoteType -->|"other"| Skip5["Skip"]

    FindSession --> Found{"session found?"}
    Found -->|"yes"| Enqueue3["_enqueue_continuation()<br/>(resume_session)"]
    Found -->|"no"| Skip6["Skip"]

    Cont --> ContType{"work_type?"}
    ContType -->|"new_session"| ExecNew["_execute_workflow()"]
    ContType -->|"resume_session"| LoadS3["load session from S3<br/>→ _execute_workflow()"]

Phase 1 handlers (handle_pipeline, handle_merge_request, handle_note) perform validation, auth checks, and session resolution, then post a WorkOrder continuation message via _enqueue_continuation(). They do not call _execute_workflow directly.

Phase 2 (handle_continuation) receives the WorkOrder, resolves the workflow config, builds a WorkflowRequest, and calls _execute_workflow. For resume_session orders, it loads the previous session from S3. _execute_workflow handles SHA dedup and per-workflow rate limiting via scan_agent_threads.

Slash commands dispatch to a single named workflow via _resolve_slash_workflow (exact match, then prefix match). Help and error replies are posted via post_simple_reply (no session marker, no rate limit impact).

10.4. Pipeline trigger design

Why gitlab::pipeline instead of kubernetes::PipelineRun: A GitLab pipeline event fires once when the pipeline completes. Since Konflux external stages are attached to the pipeline, the event naturally waits for all builds and tests to finish before triggering. This means the agent sees the full picture in one event, without needing to deduplicate or wait for stragglers.

Trigger filters:

  • status in {failed} – only failed pipelines trigger analysis
  • source == "merge_request_event" – only MR pipelines, not branch/tag

10.4a. Merge request trigger design

The handle_merge_request handler fires on MR open, reopen, and update events (action in {"open", "reopen", "update"}). Draft MRs are skipped (object_attributes.draft == True); marking a draft as ready triggers a review since the event arrives with draft: false.

Event classification is deliberately simple: the handler does not inspect oldrev or changes.draft fields. Instead, SHA-based deduplication in _execute_workflow (via JSON session markers) ensures each code revision is reviewed at most once. Metadata-only updates (title, label changes) on an already-reviewed SHA are silently skipped.

10.5. Note trigger design

Two sub-flows:

Slash command (/hummingbird <workflow-name>): Triggers a specific named workflow, bypassing rate limiting. _resolve_slash_workflow performs exact-match lookup first, then falls back to unique prefix matching. If the subcommand is missing or is “help”, _format_help returns a list of available workflows with descriptions. Ambiguous or unknown subcommands produce an error message via post_simple_reply. The note author must have Developer+ access (level >= 30) on the project, checked via check_member_access(). If denied, the agent replies in the same discussion thread with a short access-denied message.

Reply to agent note: When a user replies to an existing agent note (which contains a JSON session marker):

  1. find_session_for_reply() walks the discussion thread, filtering by bot author ID (get_bot_user_id()), and returns the first matching session marker. All bot markers in a thread share the same session ID.
  2. handle_note posts a resume_session continuation to the FIFO (grouped by session_id).
  3. handle_continuation loads the session from S3 (conversation history and sandbox archive), builds a WorkflowRequest, and calls _execute_workflow() to resume with the user’s reply.
  4. If the session is not found (expired/deleted), the agent replies with a message explaining the session has expired and suggesting to start a new run.

Reply authors are also subject to the Developer+ access check. If denied, the agent replies in the discussion thread with the same access-denied message.

Non-agent-directed notes: Regular comments that are neither slash commands nor replies to agent threads are silently ignored (debug-level log). No access check is performed for these.

Self-note filtering: Notes where author_id matches the bot’s own user ID (resolved via get_bot_user_id()) are skipped immediately. This prevents infinite loops where the agent’s own output triggers another agent run. The author-ID check replaces the previous substring check (marker_prefix_str in note_body), which was vulnerable to denial-of-service: a user including the marker prefix in their reply would cause the bot to silently ignore it.

Threading logic for replies: If the project requires internal notes (internal_notes: true) but the original discussion was public, the reply is posted as a new top-level internal note instead of replying in the public thread. This prevents leaking internal analysis into public threads.

10.6. Note lifecycle

1. create_placeholder_note()   -> "Running the <workflow> workflow..."
                                   + JSON session marker (id, wf, sha)
2. run_workflow()              -> agent loop
3. update_note()               -> replace placeholder with result
                                   + reply prompt + JSON session marker

Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.

On failure, the placeholder is updated to “Hummingbird analysis failed.” The JSON session marker is always present so the note can be identified as agent-generated (for rate limiting, SHA dedup, and self-note filtering). The marker embeds the workflow name and commit SHA, enabling per-workflow rate limiting and SHA-based deduplication.

10.7. Rate limiting and SHA deduplication

scan_agent_threads() performs a single pass over MR discussions, parsing JSON session markers (<!-- hummingbird-session: {"id":...,"wf":...,"sha":...} -->) to determine:

  1. Per-workflow thread count – how many threads the given workflow has created on this MR. If the count meets or exceeds max_runs_per_mr, the workflow is skipped.
  2. SHA dedup – whether the current commit SHA has already been reviewed by this workflow. If so, the workflow is skipped (prevents re-reviewing the same code on metadata-only MR updates).

Slash commands and replies bypass both checks entirely.

The rate limit is per-workflow per-MR: different workflows maintain independent thread counts on the same MR. JSON markers are backward compatible – old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} with no workflow or SHA information, so they are not counted toward any specific workflow’s limit.

11. Session System

Sessions enable conversation continuity: a user can reply to an agent note and the agent picks up where it left off, with full context and sandbox files restored.

11.1. What gets saved

Three artifacts are saved after each run:

  • context.json – the full contents list from the agent loop. This is the complete conversation history in Gemini wire format (user turns, model turns with tool calls, tool response turns). It is the minimum state needed to resume the conversation.
  • transcript.md – a human-readable markdown rendering of the run for debugging and auditing. Not used for resumption.
  • sandbox.tar.gz – an archive of /tmp/data/ (the sandbox working directory, which includes the _out/ spill subdirectory). Contains PipelineRun JSONs, test logs, jq output, and any other files the model created during the run. Restored into the new sandbox on resumption so the model can reference its previous work.

11.2. Storage backends

S3 (production):

s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz

Local directory (development, --save-session):

{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz

Both backends have the same interface. S3 save is best-effort: wrapped in try/except so a transient S3 error does not prevent the MR note from being posted. The note always gets delivered first.

11.3. Sandbox archive transport

The sandbox archive requires special handling because the orchestrator cannot directly access the sandbox filesystem (no volume mounts):

Archive (inside sandbox):
  sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")

Stream out (sandbox -> host temp file):
  for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
      tmp.write(chunk)                     # 64 KB chunks, no base64

Restore (host temp file -> new sandbox):
  sb.stream_exec("tar xzf - -C /tmp", file_chunks())

Both directions use streaming binary I/O via Popen stdin/stdout pipes. No base64 encoding is needed: read_file_iter() yields raw bytes from the sandbox via a Popen stdout pipe, and stream_exec() feeds raw bytes into a Popen stdin pipe. The archive is never extracted on the orchestrator – it exists only as opaque bytes being transported between sandboxes. This is a security invariant: the orchestrator never parses or inspects the archive contents.

11.4. Resumption flow

flowchart TD
    Reply["User replies to agent note"]
    FindSession["find_session_for_reply()<br/>walks discussion thread"]
    LoadS3["load_s3(session_id)"]
    NotFound{"session found?"}
    Expired["Post session-expired reply"]
    NewSandbox["Start new sandbox"]
    Restore["restore_sandbox(archive)"]
    BuildContents["contents = old_contents + user_reply"]
    RunLoop["run_agent_loop(<br/>initial_contents, CONTINUATION_PROMPT)"]

    Reply --> FindSession --> LoadS3
    LoadS3 --> NotFound
    NotFound -->|"no"| Expired
    NotFound -->|"yes"| NewSandbox
    NewSandbox --> Restore --> BuildContents --> RunLoop

Key aspects:

  • Fresh iteration budget. The resumed session starts iteration 0 with the full max_iterations budget, regardless of how many iterations the previous session used.
  • Context window is the real limit. A typical 8-iteration cold start uses ~50-100K input tokens. The restored history is sent in full on every API call. The CONTEXT_LIMIT check applies and will force wrap-up if needed.
  • Session ID reuse. The resumed session keeps the original session ID. S3 state is overwritten in place with the updated conversation history. Since GitLab discussions are linear (not branching), there is no need for a tree of sessions – the conversation is a single sequential thread.
  • Graceful degradation. If the S3 session is not found (expired, deleted), the agent posts a reply explaining the session has expired and suggesting to start a new run. It does not fall back to a cold start.

11.5. Session markers

Every agent note contains a hidden HTML comment with a JSON payload:

<!-- hummingbird-session: {"id":"UUID","wf":"code-review","sha":"abc123"} -->

The JSON payload contains:

  • id – session UUID (always present)
  • wf – workflow name (present for new-format markers)
  • sha – commit SHA at the time of review (present for auto-triggered runs)

This marker serves four purposes:

  1. Resumption: find_session_for_reply() searches discussion threads for this marker to find the session ID. Only bot-authored notes are considered; the first matching marker wins.
  2. Per-workflow rate limiting: scan_agent_threads() counts discussion threads for a specific workflow using the wf field. Only bot-authored notes are scanned.
  3. SHA deduplication: scan_agent_threads() checks whether the current SHA has already been reviewed by the workflow using the sha field.
  4. Self-filtering: handle_note() compares the webhook event’s author_id against the bot’s own user ID (via get_bot_user_id()) to prevent infinite loops.

All marker-scanning functions resolve the bot user ID via gl.auth() (GET /user) on the orchestrator token. This ensures markers in non-bot notes (user replies, external contributors) are never parsed, preventing session hijacking via injected markers.

Old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} for backward compatibility. They are counted for resumption but not for per-workflow rate limiting or SHA dedup (no wf/sha information).

12. Workflow System

12.1. Separation of prompt and metadata

A workflow has two parts that live in different places:

  • Prompt (.md file): the LLM system prompt, verbatim. This is the investigation strategy, tool usage guidance, data patterns, and output format. Pure text, no code.
  • Metadata (YAML config): operational settings – action, model, max_iterations, data_sources, projects. This controls what the orchestrator does with the workflow, not what the model does.

This separation means changing the analysis strategy (e.g., adding a new investigation step) requires editing a markdown file. Changing operational parameters (e.g., which projects use this workflow, which model to use) requires editing the YAML config. Neither requires a code change.

12.2. System prompt layering

The system prompt sent to the model is built from three layers:

BASE_SYSTEM_PROMPT        # agent.py: sandbox rules, tool usage tips
+ tool_notes              # from ToolRegistry: per-data-source domain knowledge
+ workflow_body           # full content of e.g. workflows/analyze-failures.md

On session resumption, a fourth layer is appended:

+ CONTINUATION_PROMPT     # agent.py: "do not re-run the full workflow"

The system prompt is rebuilt every iteration (to allow warning suffixes to be appended), but the base content is stable. Warning suffixes are appended at the end so they override earlier instructions.

12.3. Design choice: strategy, not procedure

The workflow .md file describes strategy and guidance, not a rigid script. The model decides when and how to use each tool based on what it sees.

This matters because merge request failures are diverse. A fixed procedure would either miss edge cases (e.g., build failures before tests ran) or waste iterations on steps that don’t apply. By giving the model a strategy (“identify failing tests, fetch details, analyze root causes, group by similarity”), it can adapt to whatever it encounters.

The workflow does structure the investigation into phases (Data Collection, Analysis, Output) for clarity, but these are guidelines, not enforced checkpoints.

12.4. Workflow file anatomy (analyze-failures.md)

The primary workflow is structured as:

  1. Scope and approach – what this workflow analyzes and what it ignores
  2. Input – event JSON format (project, iid)
  3. Data Collection Phase:
    • Data.1: Fetch MR details (direct call, small response)
    • Data.2: Fetch commit statuses, identify failures
    • Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
    • Data.4: Process with jq, extract Testing Farm data in bulk
  4. Analysis Phase:
    • Analysis.1a: Investigate each failed PipelineRun individually
    • Analysis.1b: Summarize and group by root cause
  5. Output – markdown template with root causes, collapsible details, clickable links (PipelineRun, Testing Farm, test logs)
  6. Error Handling – partial report philosophy

Design considerations embedded in the workflow:

  • Efficient bulk fetching: PipelineRuns and TaskRuns are fetched in one fetch_batch_to_sandbox call, not individually.
  • Testing Farm data extracted in Data.4, analyzed in Analysis.1: All TF results.xml are fetched in bulk before analysis starts, enabling cross-failure pattern detection.
  • Log fetching is selective: Only 1-2 representative logs per failure pattern, not all logs. This keeps iteration count bounded.
  • Reviewer-facing URLs: The output template instructs the model to include clickable links using konflux_ui and artifacts_base from response metadata.

12.5. Prompt file resolution

get_prompt(workflow_cfg, config_dir) resolves the prompt: field relative to the config file’s directory:

prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md

In the container image, workflows are baked in at /app/workflows/. A ConfigMap can override them at deploy time by mounting at the same path.

13. Deployment and Container

13.1. Container build strategy

The Containerfile uses an all-RPM builder+installroot+scratch pattern (no pip, no venv):

  1. Builder stage: Uses a Fedora-based builder image with dnf. Installs all dependencies as RPMs into a clean --installroot.
  2. Application code: Copies hummingbird_agent/ and workflows/ into the installroot at /app/.
  3. Final stage: FROM scratch, copies the entire installroot. No package manager, no shell beyond what RPMs provide.

RPM dependencies: python3-boto3, python3-google-auth+requests, python3-gitlab, python3-kubernetes, python3-pyyaml, python3-requests, python3-sentry-sdk, kubernetes1.35-client (for kubectl).

This approach was chosen over pip because:

  • All deps come from Fedora’s package repository – no PyPI supply chain risk
  • Smaller image (~22 MB saved vs. google-genai SDK alone)
  • Reproducible builds from known RPM versions
  • No compilation step (no gcc/python3-devel in the image)

13.2. Container runtime properties

CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532

The default CMD runs serve mode for production. Local development uses run mode via explicit command override. USER 65532 matches the standard nonroot UID used by distroless images and the Podman sandbox.

13.3. K8s deployment manifests

Located in hummingbird-agent/kubernetes/:

deployment.yaml:

  • Single replica with rolling update (maxSurge: 1, maxUnavailable: 0)
  • terminationGracePeriodSeconds: 900 (15 minutes) to allow in-flight agent runs to complete on shutdown
  • ServiceAccount: hummingbird-agent
  • Secrets mounted from K8s Secret hummingbird-agent
  • Commented-out mounts for workflow ConfigMap and custom CA trust

rbac.yaml (applied in the sandbox namespace):

  • ServiceAccount hummingbird-agent
  • Role with pods: create, get, list, delete, patch and pods/exec: create
  • RoleBinding linking the SA to the Role
  • patch is required for pool mode (relabeling pods during claim)

sandbox-pool.yaml (applied in the sandbox namespace):

  • Deployment with replicas: 3 (tuned to balance latency vs cost)
  • Pods labelled hummingbird/role: standby for pool discovery
  • Same security context and resource limits as direct-creation pods
  • revisionHistoryLimit: 2, rolling update with maxSurge: 1, maxUnavailable: 0

networkpolicy.yaml:

  • podSelector: {} – applies to all pods in the namespace
  • egress: [] – deny all outbound traffic

secret.yaml:

  • Template with all required env vars (API keys, tokens, URLs)
  • Values must be populated per deployment

13.4. Workflow mounting

Workflows are baked into the image at /app/workflows/. To update workflows without rebuilding the image:

  1. Create a ConfigMap from the workflows directory
  2. Uncomment the volume mount in the deployment YAML
  3. The ConfigMap mount replaces the baked-in directory (all-or-nothing)

This is useful for rapid iteration in staging without waiting for a new image build.

13.5. Custom CA trust

For clusters with internal CA certificates (common in enterprise environments), the deployment supports OpenShift’s CA injection:

  1. Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle label
  2. Mount it at /etc/pki/custom
  3. Set REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt

OpenShift automatically injects the cluster’s CA bundle into the ConfigMap.

14. Design Decision Registry

Each entry records a decision, the alternatives considered, why the chosen approach won, and what would break if the decision were reversed. This is the most important section for avoiding regressions.

kubectl exec over kubernetes Python exec API

  • Chosen: kubectl exec as subprocess for sandbox command execution.
  • Alternative: kubernetes Python client stream() API.
  • Why: Three showstopper bugs in the Python client: (1) WebSocket v1-v4 has no stdin EOF signal, so cat > /file hangs forever; (2) BrokenPipeError on stdin larger than ~1MB; (3) stream() accumulates all stdout in memory with no control. kubectl avoids all three and provides the same subprocess interface as Podman.
  • If reversed: write_file() would hang or fail on large data. Pod log retrieval with large outputs would OOM. Sandbox reliability would drop significantly.

YAML config over env vars for operational settings

  • Chosen: Single YAML file for workflows, projects, limits, data source declarations.
  • Alternative: Everything in env vars (original design).
  • Why: Env vars cannot express structured data (workflow-project mappings, per-project token overrides, data source config with multiple fields). YAML provides structure, validation, and audit trails.
  • If reversed: Token scoping would be lost (no per-workflow-project token overrides). Project allowlists would be impossible. The config would be unauditable.

Token env var names in YAML, not values

  • Chosen: YAML contains token_env: GITLAB_TOKEN_RO (the env var name), not the actual token value.
  • Alternative: Inline secrets in YAML, or env vars for everything.
  • Why: The YAML file can be committed, reviewed, and audited. Actual secrets stay in env vars (injected via K8s Secrets). Inline secrets would make the config file a secret itself.
  • If reversed: The config file would become a secret, breaking audit trails and code review workflows.

Orchestrator token prefix convention (ORCHESTRATOR_*)

  • Chosen: Orchestrator tokens use ORCHESTRATOR_GITLAB_TOKEN_* env var names by convention.
  • Alternative: Same env var namespace as model tokens, distinguished by context.
  • Why: Structural separation makes it impossible to accidentally pass an orchestrator token to a model tool (or vice versa). The ORCHESTRATOR_ prefix is never used in YAML token_env fields.
  • If reversed: A misconfiguration could leak write-capable tokens to the model, which could then expose them via tool calls.

Auto-spill over conversation compaction

  • Chosen: Large outputs are saved to sandbox files with previews returned to the model.
  • Alternative: Replace old tool results in contents with compact summaries (conversation compaction).
  • Why: Compaction requires modifying the contents list, which risks violating Gemini API constraints (model turns must match preceding tool turns). Auto-spill achieves ~90% of the token reduction with zero risk of breaking the conversation structure.
  • If reversed: Token usage would increase ~15-30%. Per-call context would grow unbounded. Sessions would hit the context limit much sooner. Conversation compaction could be added on top of auto-spill in the future, but is not needed with current workloads.

Iteration count over cumulative token budget

  • Chosen: max_iterations as the primary cost control lever.
  • Alternative: Cumulative token budget (stop when total billed tokens exceed a threshold).
  • Why: With full history replay, each API call re-sends everything. Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K (including 5K again), total = 15K billed but only 10K new content. With auto-spill keeping per-call size bounded, iteration count is a much simpler and more predictable proxy for actual cost.
  • If reversed: The budget model would be confusing and inaccurate. Cost estimates would be wrong. The cumulative metric is still logged for observability, but it does not drive termination.

ToolDef notes in system prompt, not tool schema

  • Chosen: Domain knowledge (Konflux dual-fetch, TF XML structure) goes in ToolDef.notes, injected into the system prompt.
  • Alternative: Put everything in the tool schema description.
  • Why: Tool schemas have character limits and are sent in the tools field of every API call. Long descriptions waste tokens on tool defs. System prompt notes are sent once and can be arbitrarily detailed.
  • If reversed: Tool descriptions would be bloated. Domain knowledge would need to be duplicated in every workflow .md file.

Full history replay (no compaction)

  • Chosen: The contents list grows monotonically. Items are never removed or modified (except empty response retries).
  • Alternative: Compact old turns to reduce context size.
  • Why: The Gemini API requires strict turn-by-turn structure. Modifying or removing items risks invalid conversation structure. Auto-spill handles the growth problem at the source (preventing large items from entering history).
  • If reversed: Risk of Gemini API errors from malformed conversation structure. Risk of confusing the model (it references previous results that have been summarized away).

fetch_to_sandbox kept alongside auto-spill

  • Chosen: Both fetch_to_sandbox and auto-spill coexist.
  • Alternative: Remove fetch_to_sandbox since auto-spill handles large outputs automatically.
  • Why: fetch_to_sandbox provides explicit path control (model can choose meaningful filenames), batch fetching (one tool call for multiple fetches), and no-preview responses (useful when the model will process with jq anyway).
  • If reversed: The model would lose path control and batch fetching would require multiple auto-spilled calls. The workflow would need more iterations to achieve the same result.

K8s library for lifecycle + kubectl for exec (hybrid)

  • Chosen: Use the kubernetes Python library for pod create/read/delete, kubectl subprocess for exec.
  • Alternative: All-kubectl (subprocess for everything) or all-library.
  • Why: The library provides typed pod status polling and clean error handling for lifecycle. kubectl provides reliable exec (see “kubectl exec over kubernetes Python exec API” above). Using the library for exec would require solving the three WebSocket bugs.
  • If reversed: Either lifecycle management would be fragile (parsing kubectl JSON output for pod status) or exec would be unreliable.

Single-file YAML config over multiple config files

  • Chosen: Everything in one config.yaml with settings and workflows sections.
  • Alternative: Separate files per workflow, or settings.yaml + workflows.yaml.
  • Why: Single source of truth. All project-workflow mappings visible in one place. Settings-level defaults flow down to all projects. No file discovery logic needed.
  • If reversed: Token resolution would need cross-file lookups. Project index would need multi-file aggregation. Config validation would be more complex.

Workflow .md as pure prompt, metadata in YAML

  • Chosen: The workflow .md file is pure system prompt text. Metadata (action, model, max_iterations, data_sources, projects) lives in the YAML config.
  • Alternative: YAML frontmatter in the .md file (original design).
  • Why: Separation of concerns. The .md file is the model’s instructions – it should be readable and editable by anyone writing prompts. The YAML config is the orchestrator’s instructions – it controls routing, limits, and credentials. Mixing them in one file conflates two audiences.
  • If reversed: Prompt authors would need to understand YAML config structure. Token/project config would be scattered across .md files instead of centralized.

Raw HTTP for Gemini instead of google-genai SDK

  • Chosen: requests + google-auth for Gemini API calls.
  • Alternative: google-genai Python SDK.
  • Why: The SDK brings ~22 MB of transitive deps (pydantic, httpx, websockets). The Gemini REST API is simple camelCase JSON over HTTPS – one endpoint, one request format. Raw HTTP enables: smaller container, simpler tests (mock requests.post), plain dict conversation history (easy session serialization), no SDK breakage risk, and a models/ package structure that supports multiple backends.
  • If reversed: Container would be ~22 MB larger. contents would use SDK objects instead of plain dicts, complicating session serialization. Adding Claude support would require a separate approach.

Nudge via system prompt suffix, not user message

  • Chosen: Empty response nudge and budget warnings are appended to the system prompt as suffixes.
  • Alternative: Inject synthetic user messages into contents.
  • Why: User messages in contents must come from the actual user (the initial event, or a reply). Injecting synthetic messages pollutes the conversation history that is saved in sessions. System prompt suffixes are transient – they affect one API call without permanently modifying the conversation state.
  • If reversed: Saved sessions would contain synthetic user messages. Resumed conversations would be confusing. The model might respond to the synthetic messages instead of the user’s actual input.

Session ID reuse over new-ID-per-resume

  • Chosen: Resumed sessions keep the original session ID. S3 state is overwritten in place.
  • Alternative: Each resume generates a new UUID, saving alongside the original (append-only tree of sessions).
  • Why: GitLab discussions are linear, not branching. There is no scenario where two different sessions from the same thread are both valid. A new UUID per resume creates orphaned S3 snapshots that are never referenced again, since find_session_for_reply() always picks the latest marker. Reusing the ID is simpler, uses less storage, and matches the linear conversation model.
  • If reversed: S3 would accumulate orphaned session snapshots. Each resume would need a new note marker, but the old markers would still be in the thread, creating confusion about which session is current.

Reply to unauthorized agent-directed notes instead of silent skip

  • Chosen: When an unauthorized user sends a slash command or replies to an agent thread, the agent replies in the same discussion with a short access-denied message. Non-agent-directed notes are silently ignored (debug log, no access check).
  • Alternative: Log a warning and skip silently for all unauthorized notes (the original behavior).
  • Why: Silent skip gives no feedback to someone who intentionally tried to engage the agent, which is confusing. On the other hand, checking access and logging a warning for every random comment on a public project is noisy and pointless. Moving the access check after intent detection cleanly separates the two cases. The reply uses the discussions API so it automatically inherits confidentiality from the parent note/thread.
  • If reversed: Users without sufficient access who try /hummingbird would get no feedback. The agent log would be noisy with warnings for every comment on public MRs.

Deployment-backed pod pool over Python-thread pool

  • Chosen: A Kubernetes Deployment maintains pre-warmed standby pods. The agent claims a pod by relabeling it; the Deployment replaces it.
  • Alternative: A Python-side thread pool that pre-creates pods and queues them for use.
  • Why: The Deployment provides self-healing (restart crashed pods), native scaling (kubectl scale), rolling updates (image changes), and monitoring via standard K8s tooling. A Python pool would need to reimplement all of these.
  • If reversed: Pod replenishment, crash recovery, and image updates would all need custom code. Scaling would require agent redeployment.

Pod isolation: never reuse sandbox pods

  • Chosen: Each workflow run gets its own pod. After cleanup, the pod is deleted. Claimed pods are detached from the ReplicaSet.
  • Alternative: Return used pods to the pool and reset them.
  • Why: Residual state from a previous run (files, environment, running processes) could leak between MR investigations, creating security and correctness risks. Deletion is simple and foolproof.
  • If reversed: Would need a reliable pod-reset mechanism and auditing that no state survives between runs.

Absolute reap-by deadline over relative claimed-at age

  • Chosen: Active pods are annotated with hummingbird/reap-by (an absolute UTC timestamp). The reaper deletes pods where now > reap-by.
  • Alternative: Annotate with claimed-at and compute age relative to max_active_seconds; or use activeDeadlineSeconds on the pod spec.
  • Why: An absolute deadline simplifies the reaper to a single comparison. It also supports varying deadlines: claim sets reap-by = now + max_active_seconds, while linger sets reap-by = now + linger_seconds. With a relative timestamp, the reaper would need to know which mode the pod is in. activeDeadlineSeconds applies from pod creation, not from claim – standby pods would expire before being used.
  • If reversed: The reaper would need mode-aware age calculations. Lingering pods would require a separate annotation or reaper path.

Indefinite claim wait over timeout

  • Chosen: SandboxPool.claim() polls indefinitely (governed by shutdown event), logging at DEBUG then WARNING.
  • Alternative: Timeout after N seconds and raise an error.
  • Why: The pool is typically smaller than max_concurrent_agents for cost reasons. Waiting for a replacement pod is normal operational behavior, not an error. A timeout would cause spurious failures during burst traffic. The SQS semaphore already bounds concurrency.
  • If reversed: Burst traffic would cause avoidable failures instead of brief delays.

Pool config in Deployment only, not in agent configmap

  • Chosen: In pool mode (k8spool), the pod image, resources, labels, and security context are defined solely in the Deployment template. The agent config only needs namespace and active_deadline_seconds.
  • Alternative: Keep image/resources/metadata in the agent configmap too (as in k8s mode).
  • Why: Eliminates config duplication. The Deployment template is the single source of truth. Changes to pod resources or image only require updating one place and re-rolling the Deployment.
  • If reversed: Config drift between the Deployment and the agent configmap would be a constant risk.

Pod lingering over immediate cleanup for reply latency

  • Chosen: After a successful workflow, pool sandbox pods linger for linger_seconds (default 300) with a session-id annotation. Replies within the window reclaim the pod via try_reclaim(), skipping pod creation and S3 archive restoration.
  • Alternative: Always delete the pod immediately and restore from S3 on every reply.
  • Why: Reply latency drops from seconds (pod claim + S3 restore) to near-zero. The S3 archive is still saved as a fallback if the pod is gone. The reap-by annotation ensures lingering pods are cleaned up if no reply arrives. The session-id annotation doubles as a concurrency guard: it is only present while lingering, preventing active pods from being reclaimed.
  • If reversed: Every reply would pay full pod + restore latency, even for immediate follow-ups. User experience for conversational interactions would degrade noticeably.

linger() on Sandbox protocol over isinstance checks in runner

  • Chosen: All sandbox backends implement linger(session_id). Non-pool backends fall back to cleanup(). The runner calls sb.linger() without type checks.
  • Alternative: isinstance(sb, K8sPoolSandbox) in run_workflow().
  • Why: Keeps the runner backend-agnostic. Adding a new backend requires only implementing the protocol, not touching the runner. The fallback behavior is co-located with each backend.
  • If reversed: The runner would need to know about every backend type and their linger capabilities.

GCP SA key over Workload Identity Federation

  • Chosen: GCP service account key stored in Vault, rotated via cki-tools credential manager (prepare/switch/clean cycle).
  • Alternative: Workload Identity Federation (WIF) with projected SA token. Two sub-options: (a) automatic OIDC discovery if the cluster issuer is public, (b) manual JWKS upload for internal issuers.
  • Why: mpp-prod’s OIDC issuer is https://kubernetes.default.svc (internal, not publicly reachable), so GCP STS cannot discover it automatically. Manual JWKS upload works but requires re-upload after SRE-triggered signing key rotations. SA key integrates with the existing credential manager rotation infrastructure (same pattern as AWS keys and GitLab tokens), needs no OIDC reachability, and enables automated validate/update via google-auth in CI. The application code (VertexAuth / google.auth.default()) works identically with both approaches – switching to WIF later requires only infrastructure changes.
  • If reversed: Replace SA key with WIF projected token + credential config ConfigMap. The gcp_service_account_key token type in cki-tools would no longer be needed for this use case.

Sliding-window breakpoints over per-component breakpoints (Claude)

  • Chosen: Two breakpoints on messages (B1 at the previous write position, B2 at the latest message) that slide forward each turn.
  • Alternative: Separate breakpoints on system prompt, tools, and messages (using 3-4 of the 4 available slots).
  • Why: The Anthropic prefix hash is cumulative – it covers everything from the start of the request (tools, system, messages) up to the breakpoint. A single breakpoint on a message already caches the entire prefix. Separate breakpoints on earlier components would be redundant and waste slots. Two sliding breakpoints cover the full conversation history with cache reads on every turn after the first.
  • If reversed: Three breakpoint slots wasted on content already covered by the message breakpoint. Only one slot left for the sliding window, making it impossible to have both a read (B1) and write (B2) breakpoint on messages.

Developer+ trust filtering over unfiltered or sanitized discussion comments

  • Chosen: gitlab_get_mr_discussions checks each note author’s project access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed placeholder in mixed-trust discussions, or the entire discussion is dropped if all notes are untrusted. Agent notes are always trusted (detected by session marker) but have transcripts and metadata stripped.
  • Alternatives considered:
    • (a) Include all comments unfiltered. Simplest, but any external contributor can craft comments that manipulate the model’s review output (prompt injection).
    • (b) Sanitize/escape untrusted content (strip markdown, quote as code blocks, prefix with “[external]”). There is no reliable escaping mechanism for LLM prompts – the model interprets natural language regardless of formatting. Escaping gives a false sense of security.
    • (c) Only include agent-authored notes (skip all human comments). Safe, but defeats the purpose: the model would never see developer responses to its own findings.
    • (d) Include only discussions where the agent participated. Better, but still misses developer-initiated review threads that provide relevant context.
  • Why: Developer+ is the same threshold used for pipeline trigger authorization (invariant #8) and slash command access. It matches the trust boundary already established: people who can push code and approve MRs are trusted to provide review context. The placeholder approach preserves discussion structure (the model sees that someone replied) without exposing the content. Full-drop for all-untrusted discussions avoids noise from discussions that contain zero useful context.
  • If reversed: (a) opens a prompt injection vector on any project that accepts external MRs. (b) provides no actual protection. (c) makes follow-up reviews unable to see developer explanations, causing repeated false positives. (d) misses developer-initiated context.

Bot-author filtering for session marker parsing over unfiltered note scanning

  • Chosen: find_session_for_reply(), scan_agent_threads(), and the handle_note() self-filter all resolve the bot user ID via get_bot_user_id() (which calls gl.auth() on the orchestrator token) and only consider notes where author.id matches. In find_session_for_reply(), the first matching marker wins.
  • Alternative: Parse markers from any note and keep the last match; substring self-filter in handle_note (previous behavior).
  • Why: The previous unfiltered approach allowed session hijacking: a user reply or agent review prose containing an example marker (<!-- hummingbird-session: ... -->) was parsed as a real session, causing “session expired” errors in production. For handle_note, the substring check caused the reverse problem: a user embedding the marker prefix in a reply silently suppressed the event (denial of service). The gl.auth() call is one GET /user per invocation – negligible cost. _get_client_with_bot_id() returns both the authenticated client and the bot user ID, avoiding redundant client construction.
  • If reversed: Any MR participant can inject a marker to hijack the session ID or inflate rate-limit counts. The agent’s own review prose containing example markers causes spurious “session expired” messages.

ijson streaming over buffered JSON for K8s list responses

  • Chosen: session.get(stream=True) + ijson.parse(resp.raw) for iter_paginated(). Items are yielded one at a time via ObjectBuilder; the metadata.continue pagination token is captured in the same parse pass. resp.raw.decode_content = True is required because Kubearchive returns gzip Content-Encoding; without it ijson sees compressed bytes.
  • Alternative: session.get().json() loads the full page into memory (the original implementation).
  • Why: A single K8s list page can contain 500 PipelineRuns (80+ MB JSON). Parsing this with .json() creates a +247 MB RSS spike (raw bytes + decoded string + parsed dict coexist). With 4 concurrent workflows, this exceeds any reasonable pod memory limit. ijson stream-parsing yields items one at a time, reducing the peak to +12 MB for the same data – a 95% reduction. The continue token appears in the metadata object (before or after items depending on the server); the event-driven parser captures it regardless of order.
  • If reversed: Large MRs (100+ components) would OOM the agent pod. Concurrent workflows would multiply the problem. The pod memory limit would need to scale with the largest possible page size, which is unbounded.

Action token tier for per-workflow identity

  • Chosen: A dedicated Tier 2 of workflow action tokens, separate from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets a dedicated GitLab bot user per project for all write operations (notes, thread resolution).
  • Alternative: Extend orchestrator tokens with a per-workflow fallback chain (ORCHESTRATOR_<WORKFLOW>_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN).
  • Why: Orchestrator tokens serve a fundamentally different purpose (operational reads + infrastructure notes) than workflow output (analysis results, review comments). Mixing them in one fallback chain risks a single bot identity posting notes on behalf of multiple workflows, eliminating the audit trail. A separate tier with explicit token: str parameters makes the code path unambiguous: callers must choose which tier they use. The naming convention (HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>) groups tokens alphabetically by workflow, making env var auditing straightforward.
  • If reversed: All workflow notes would appear from the same bot user, making it impossible to distinguish which agent (code-review vs analyze-failures vs renovate-babysit) produced a given note in the MR timeline. GitLab’s audit log would attribute all actions to one identity.

Per-project-only action token config (no workflow-level default)

  • Chosen: action_tokens is defined at the project level only in YAML config. No workflow-level action_tokens default is supported.
  • Alternative: Allow workflow-level action_tokens that projects inherit by default, with per-project overrides.
  • Why: GitLab project access tokens are project-scoped. A workflow-level default would be misleading – a single token cannot write to multiple projects. Forcing per-project declaration makes the mapping explicit and prevents configuration errors where a cross-project default silently fails.
  • If reversed: Operators could set a workflow-level token that only works for one project, causing silent 403 errors for other projects in the same workflow.

Migration: workflow action tokens (first deployment)

On the first deployment with workflow action tokens, scan_agent_threads and resolve_agent_threads authenticate as the workflow bot and attribute notes exclusively by that bot’s user ID. Threads previously authored by the orchestrator bot will not be visible to these functions. Expect:

  • One extra workflow run per open MR on the first trigger after deployment (sha_reviewed returns False for already-reviewed SHAs).
  • Pre-migration orchestrator-authored threads will not be auto-resolved on push or pipeline success.

Pre-migration threads can be manually closed, or they will age out naturally as new workflow-bot-authored threads are created for subsequent MR activity.

SSH over kubectl exec for KubeVirt VMs

  • Chosen: SSH (ssh -T sandbox@ip command) for VM command execution.
  • Alternative: virtctl console (serial console), KubeVirt websocket API, or qemu-guest-agent exec.
  • Why: VMs have no kubectl exec equivalent. virtctl console provides a serial terminal, not a programmable exec channel. The websocket API requires a VNC or serial connection. qemu-guest-agent is not installed in the VM image. SSH provides clean stdin/stdout/stderr separation, timeout control, and is already well-tested in subprocess pipelines.
  • If reversed: Would need qemu-guest-agent in the image and the exec endpoint, which has poorer error reporting and no streaming stdin.

Secret volume for SSH key injection

  • Chosen: Kubernetes Secret mounted as a virtio disk (serial ssh-pubkeys) read by the VM’s inject-ssh-keys.service at boot.
  • Alternative: cloudInitNoCloud or accessCredentials with qemu-guest-agent.
  • Why: The VM image already has inject-ssh-keys.service which reads from the virtio disk. cloudInitNoCloud requires cloud-init in the image. accessCredentials requires qemu-guest-agent. The Secret volume approach requires no additional software in the VM image.
  • If reversed: Would need cloud-init or guest agent in the bootc image, adding complexity and attack surface.

VMIRS for VM pool replenishment

  • Chosen: VirtualMachineInstanceReplicaSet (VMIRS) as the replenishment controller for the VM pool.
  • Alternative: Custom controller, or VirtualMachine with RunStrategy.
  • Why: VMIRS is KubeVirt’s native equivalent of a Deployment’s ReplicaSet – it maintains a desired count of identical VMIs. When a VMI is claimed (ownerReferences cleared), the VMIRS creates a replacement automatically. Same pattern as the pod pool with Deployment.
  • If reversed: Would need a custom controller to maintain the VMI pool, adding operational complexity.

Per-workflow sandbox backend

  • Chosen: sandbox: field in workflow config, resolved per-workflow with fallback to CLI --sandbox flag.
  • Alternative: Global-only sandbox selection via CLI.
  • Why: Different workflows have different isolation requirements. Code review needs only a lightweight container, while failure analysis may need a full VM with root access. Per-workflow selection avoids running all workflows in VMs (wasteful) or all in containers (insufficient isolation).
  • If reversed: All workflows would share the same sandbox backend, requiring either over-provisioning (VMs for everything) or accepting weaker isolation for some workflows.

4.5.20 - Hummingbird Agent Evals

Evaluation framework for scientific measurement of LLM output quality. For the agent itself, see Hummingbird Agent. For architecture details, see Hummingbird Agent Design.

What evals are

Evals measure the quality of LLM outputs under controlled conditions. They answer questions like “does prompt v5 produce better code reviews than v4?” or “can an LLM answer this question correctly using only public documentation?”

Evals are not unit tests. Unit tests verify deterministic behavior; evals quantify stochastic output quality through structured scoring.

General pattern

Evals generally follow a pattern of:

  1. Inputs – a set of test cases: golden dataset with scoring criteria, real MRs to review, or other structured inputs.
  2. Model execution – run one or more LLM configurations against the inputs, optionally with tools (web search, sandbox).
  3. Scoring – evaluate outputs via LLM-as-judge, comparison against reference answers, or human review.
  4. Iteration – adjust prompts or config, re-run, check for improvement.

Not all evals use every step. Baseline measurements may stop at step 3; comparison evals may skip the golden dataset entirely.

Shared infrastructure

hummingbird_agent.eval

The hummingbird_agent.eval module provides generic evaluation primitives used across all evals. Key components:

  • EvalJudge – LLM-as-judge wrapper over Vertex AI with explicit temperature and token limit control. Supports both strict judging (low temperature) and open-ended analysis (higher temperature).
  • Snapshot management – saves timestamped evaluation results with metadata, maintains a latest symlink.
  • Variant tracking – persists the best-performing prompt or config variant across iterations.
  • Convergence detection – checks whether scores have plateaued across consecutive iterations.
  • Holdout validation – compares holdout scores against dev scores using standard error of the mean.

See the module docstrings for the full API.

hummingbird_agent.evals.mr_helpers

MR-experiment-specific helpers built on top of hummingbird_agent.eval. Provides GitLab auth, MR data I/O, review generation orchestration, and CLI scaffolding. Used by the code-review and renovate-triage evals.

Running an eval

Prerequisites

  1. GCP Application Default Credentials:

    gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=<your-gcp-project>

  2. The agent package installed in editable mode:

    cd hummingbird-agent
pip install -e .

Execution

Each eval is a Python module under hummingbird_agent/evals/. Run from hummingbird-agent/:

python -m hummingbird_agent.evals.<name>.<script> [options]

Refer to each eval’s own README for specific options and configuration.

Existing evals

Writing a new eval

Create a new directory under hummingbird_agent/evals/ with:

  • README.md – what the eval measures and how to run it, linking back to this document for the general framework.
  • Golden dataset – inputs and scoring criteria. Can live outside the eval directory (e.g. in the documentation repo alongside the content being measured) and be passed via CLI.
  • run_*.py – orchestration script using hummingbird_agent.eval primitives (or hummingbird_agent.evals.mr_helpers for MR-based evals).

The eval’s run script should be importable as a module (python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed. Do not hardcode GCP project IDs; VertexAuth resolves them at runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.

Authentication

All evals use VertexAuth from hummingbird_agent._http, which resolves credentials and project in this order:

  1. Explicit project= parameter (if passed).
  2. GOOGLE_CLOUD_PROJECT environment variable.
  3. ADC default project (from gcloud auth application-default login).

Service account keys work automatically when GOOGLE_APPLICATION_CREDENTIALS is set.

4.5.21 - Documentation Quality Evaluation

Reference for the documentation quality evaluation, which measures whether public documentation is complete enough for LLMs with web search to answer questions correctly. For the general evaluation framework, see Hummingbird Agent Evals.

Config file

A single YAML file combines evaluation settings and the golden dataset. It is passed to the run script via -c:

python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml

The config file lives in the documentation repo alongside the content being measured.

Evaluation settings

Key Required Default Description
models yes – List of model configurations
models[].id yes – Vertex AI model identifier
models[].region yes – Vertex AI region
models[].google_search no false Enable Google Search grounding (Gemini)
models[].web_search no false Enable web search tool (Claude)
judge yes – Judge model configuration
judge.model yes – Judge model identifier
judge.region no global Judge model region
repetitions no 3 Runs per (question, model) pair
max_iterations no 5 Agent loop iteration cap per question

Golden dataset

The questions list defines what the evaluation measures:

Field Required Default Description
id yes – Unique identifier (used in output paths)
question yes – Question sent to each model
facts yes – List of fact checks
facts[].question yes – Yes/no question about the response
facts[].weight no 1 Relative importance
coherence no – Criterion for narrative quality scoring
coherence_weight no 1.5 Weight of coherence in the final score
threshold no 0.4 Minimum weighted score to pass

CLI options

Option Default Description
-c / --config required Path to config YAML
--phase all ask, judge, or all
--results-dir results/ next to config Output directory
--run-id UTC timestamp Subdirectory under results

Scoring

Each response is scored independently:

  • Fact scoring – the judge answers each fact question with yes or no. Passing facts contribute their weight to the numerator.
  • Coherence scoring – the judge rates narrative quality from 0.0 to 1.0. The coherence score is multiplied by its weight and added to the total.
  • Weighted score – sum(passed_fact_weights + coherence_contribution) / total_weight. A response passes if weighted_score >= threshold.

The summary aggregates per-model averages across all repetitions.

4.5.22 - Hummingbird Data Flow

Overview of all data sources, what is stored where, and how data moves between systems.

Data sources

Source What it provides
Jira REST API CVE tracker ticket fields (status, timestamps, labels, custom fields)
Jira whiteboard (customfield_10841) Per-ticket timestamp cache written back by the analysis tool
Red Hat Pulp (packages.redhat.com) SRPM/RPM publish timestamps for Hummingbird repos
Hummingbird container catalog API Image rebuild history (when a package first appeared in a rebuilt image)
Red Hat CSAF VEX feed (security.access.redhat.com) Advisory fix/not-affected status per CVE
OSIDB (osidb.prodsec.redhat.com) Subpackage-level affectedness; flaw created_dt and hummingbird-1 affect created_dt
NVD / CVE list CVE publish dates, product/version data
GitHub / GitLab Upstream fix commit / PR / release dates
Fedora updates Fedora update availability timestamps
Konflux SNS/SQS events Pipeline run, snapshot, release, MR, push events

PostgreSQL databases

There are two Postgres databases.

1. hummingbird-status — Konflux pipeline events

Fed by SNS/SQS events via the hummingbird-status ingestor.

Table Primary key What is stored
gitlab_pushes (sha, repo, ref) Git push events — sha, branch, commit list
components name Konflux component definitions and git context
pipelineruns name Build/test PLR outcomes, status, start/completion times
snapshots name Multi-component snapshots, source PLR, sha
releases name Release outcomes, images, LLM analysis text
gitlab_merge_requests (project, iid) Current MR state, merge commit sha
gitlab_mr_versions (project, iid, sha) Per-commit history of each MR head

2. hummingbird-dashboard — CVE lifecycle + dashboard overlays

Table Primary key / unique What is stored
cve_ticket_events (ticket_key, event_type) One row per lifecycle milestone per ticket. occurred_at = timestamp; metadata = JSONB (see below). The source of all R-Time computations.
cve_analysis_log id Each analysis tool run: timestamp, ticket count, log output
cve_ticket_claims ticket_key Claim/lock for deduplicating concurrent analysis runs
cve_ticket_blame (ticket_key, category) unique Human-assigned R-Time blame categories (hummingbird, prodsec, konflux, testing, pulp, other); a ticket may have multiple
dashboard_settings key Runtime flags: auto-rerun enabled, analysis enabled, etc.
auto_rerun_log id History of automatic retrigger attempts
blocked_error_patterns id Regex patterns that suppress auto-rerun
blocked_snapshots snapshot_name Manually blocked snapshots
analysis_log / analysis_costs id LLM failure analysis runs and token costs
blocked_push_builds / push_build_rerun_log id Push build blocking and retry history
package_lifecycle (package, event_type) Package-scoped lifecycle milestones (e.g. rpm_first_published). Distinct from cve_ticket_events — one row per package/milestone, not per ticket.

cve_ticket_events lifecycle milestone event_type values

These are the canonical names after the HUM-5918 migration:

event_type Meaning
cve_published CVE published date (NVD / CVE list)
osidb_flaw_created OSIDB flaw created_dt (when ProdSec ingested the CVE)
osidb_affect_created hummingbird-1 affect created_dt for this ticket’s pscomponent (when a HUM tracker could be filed)
hum_ticket_created HUM Jira ticket creation date
hum_ticket_closed HUM Jira ticket resolution date
upstream_fix_merged Upstream fix commit / release merged
fedora_update_available Fedora update containing the fix became available
rpm_fix_published_to_pulp Fix RPM published to Hummingbird Pulp repo
image_rebuilt_on_quay Hummingbird container image rebuilt with the fix
vex_resolved Red Hat CSAF VEX advisory confirmed the resolution (HUM-5843)

rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable — they are updated when a newer delivery event supersedes the previous one. All other event types are write-once: once set, occurred_at is never overwritten.

cve_ticket_events.metadata JSONB fields

Each row carries a metadata blob that reflects the ticket’s state at the time of the most recent analysis run:

Field Source Description
computed_resolution analysis tool Hummingbird’s computed fix status
jira_status Jira status field Current Jira issue status
labels Jira labels field All labels on the ticket
fixed_in_build Jira customfield_10578 SRPM set manually in the “Fixed in Build” field
detected_fixed_build Pulp / SRPM detection SRPM filename auto-detected from Pulp repodata
vex_status Red Hat VEX feed fixed / known_not_affected / …
vex_match_state reconciliation logic matched / pending / mismatch
vex_resolved VEX reconciliation Timestamp when VEX first agreed with Jira resolution
catalog_image_source collector catalog map true when the package is a catalog image SBOM source (R-Time delivery is image publish); false means RPM publish. Missing on old rows: dashboard still requires image

Jira whiteboard (customfield_10841)

The CVE analysis tool uses the Jira whiteboard field as a per-ticket timestamp cache. This is being phased out in favour of Postgres as the durable store (HUM-5917), but is still the source for a subset of fields.

The whiteboard holds compact JSON. The relevant sub-key is cve_cycle:

cve_cycle key Meaning Mutable?
cve_published CVE publish date No — write-once
jira_created HUM ticket creation date No — write-once
jira_closed HUM ticket close date No — disabled (255-char limit)
upstream_fix Upstream fix timestamp No — write-once
fedora_fix Fedora update timestamp No — write-once
hb_rpm_fix RPM published to Pulp Yes — re-evaluated each run
hb_image_fix Image rebuilt on Quay Yes — re-evaluated each run

The whiteboard is read and written only by the CVE analysis tool. The dashboard never reads it directly; its authoritative source is always cve_ticket_events.

Jira fields read by the CVE analysis tool

On each run the analysis tool fetches every open HUM CVE tracker ticket and reads the following fields from the Jira REST API:

Jira field / custom field Purpose
summary Ticket title / package detection
status, resolution, resolutiondate Ticket lifecycle state
created Jira ticket creation date → hum_ticket_created
description, comment CVE ID extraction, fix evidence
labels cve-next-release, fix-in-progress, etc.
security Embargo level
assignee Ticket owner
customfield_10578 (Fixed in Build) Manually set SRPM identifying the fix build
customfield_10841 (Whiteboard) Cached cve_cycle timestamps (read + write)
customfield_10667 (CVE ID) Structured CVE identifiers
customfield_10860 (Embargo Status) Embargo flag
customfield_10020 (Sprint) Sprint membership
customfield_10014 (Epic Link) Parent epic

Data flow

┌─────────────────────────────────────────────────────────────────┐
│  External data sources                                          │
│                                                                 │
│  Jira REST API ─────────────────────────────────────┐           │
│  NVD / CVE list ────────────────────────────────────┤           │
│  GitHub / GitLab (upstream fix commits) ────────────┤           │
│  Fedora updates ────────────────────────────────────┤           │
│  Red Hat Pulp (RPM publish times) ─────────────────►│           │
│  Hummingbird catalog API (image rebuild times) ─────┤           │
│  Red Hat CSAF VEX feed ─────────────────────────────┤           │
│  OSIDB (subpackage affectedness) ───────────────────┘           │
└──────────────────────────────┬──────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│  CVE analysis tool (cron, hummingbird-cve-analysis)              │
│                                                                  │
│  • Reads Jira ticket fields + whiteboard cache                   │
│  • Resolves fix SRPM from Pulp repodata                          │
│  • Resolves image rebuild timestamp from catalog API             │
│  • Checks upstream PR/release dates (GitHub/GitLab)              │
│  • Fetches CVE publish date (NVD/CVE list)                       │
│  • Reconciles VEX status (CSAF feed)                             │
│  • Checks subpackage affectedness (OSIDB)                        │
│                                                                  │
│  Writes back to:                                                 │
│  ├── Jira whiteboard (cve_cycle timestamp cache)                 │
│  ├── Jira labels / Fixed-in-Build field                          │
│  ├── Jira comments                                               │
│  └── dashboard DB → cve_ticket_events (one row per milestone)    │
│                      cve_analysis_log (run record)               │
└──────────────────────────────┬───────────────────────────────────┘
                               │
           ┌───────────────────┴───────────────────┐
           │                                       │
           ▼                                       ▼
┌──────────────────────────┐      ┌───────────────────────────────┐
│  Jira whiteboard         │      │  dashboard DB                 │
│  (per-ticket JSON cache) │      │  cve_ticket_events table      │
│  cve_cycle timestamps    │      │  one row per (ticket,         │
│  ← being phased out      │      │    event_type) milestone      │
│    (HUM-5917)            │      └───────────────┬───────────────┘
└──────────────────────────┘                      │
                                                  ▼
                               ┌──────────────────────────────────┐
                               │  hummingbird-dashboard API       │
                               │                                  │
                               │  Pivots cve_ticket_events rows   │
                               │  into per-ticket dicts, computes:│
                               │  • entry-level R-Time fields     │
                               │    (HUM-5920)                    │
                               │  • duration legs inside stages   │
                               │    (HUM-5921)                    │
                               │                                  │
                               │  Serves JSON + Jinja templates   │
                               └──────────────────────────────────┘


┌─────────────────────────────────────────────────────────┐
│  Konflux SNS/SQS event stream                           │
│                                                         │
│  GitLab pushes, PipelineRuns, Snapshots, Releases, MRs  │
│  → hummingbird-status ingestor                          │
│  → hummingbird-status DB (pipeline/release tables)      │
│  → hummingbird-dashboard API (build/release views)      │
└─────────────────────────────────────────────────────────┘

API field names

The dashboard API exposes these computed fields per R-Time entry:

Entry-level fields

API field Meaning
cve_published_at Start of R-Time: earlier of CVE published and HUM created (Include CVE-HUM), or HUM created (Exclude)
fix_delivered_at Done timestamp: latest of delivery and Jira close (Exclude VEX), plus VEX when Include VEX is on; unset until the required gates exist. Delivery is image publish for catalog image sources, RPM publish otherwise
cve_to_delivery_hours Completed R-Time only (cve_published_at → fix_delivered_at); None while still accumulating
cve_to_hum_created_hours Filing lag after NVD: max(HUM created − NVD, 0); 0.0 when HUM is first
advisory_to_vex_hours Completed ADV-VEX only (hum_ticket_closed → vex_resolved); None until VEX exists
display_advisory_to_vex_hours ADV-VEX hours, or elapsed-to-now when the VEX feed has not updated yet
fix_before_cve_published True when the delivery timestamp predates notification (informational)
hum_ticket_open True until the selected done gates exist
deferred_to_next_release True for cve-next-release tickets
display_duration_hours R-Time hours, or elapsed-to-now when the ticket is not yet done

Duration legs (stages dict)

API field Interval
cve_published_to_osidb_flaw CVE publish → OSIDB flaw created
osidb_flaw_to_osidb_affect OSIDB flaw created → hummingbird-1 affect created
osidb_affect_to_hum_created hummingbird-1 affect created → HUM ticket opened
cve_published_to_hum_created CVE publish → HUM ticket opened
cve_published_to_upstream_fix CVE publish → upstream fix merged
hum_created_to_upstream_fix HUM ticket → upstream fix merged
upstream_fix_to_fedora_update Upstream fix → Fedora update available
fedora_update_to_rpm_fix_published Fedora update → fix RPM in Pulp
upstream_fix_to_rpm_fix_published Upstream fix → fix RPM in Pulp
hum_created_to_rpm_fix_published HUM ticket → fix RPM in Pulp
rpm_fix_published_to_image_rebuilt Fix RPM in Pulp → image rebuilt on Quay
image_rebuilt_to_hum_closed Image rebuild → HUM ticket closed
hum_closed_to_vex_resolved HUM ticket closed (advisory MR merge) → VEX feed update

Aggregate stats

API field Meaning
avg_hum_created_to_rpm_fix_hours Mean HUM ticket → RPM fix (skips missing legs)
avg_rpm_fix_to_image_rebuilt_hours Mean RPM publish → image rebuild
avg_advisory_to_vex_hours Mean Done-Errata close → VEX feed (completed ADV-VEX only)

Package lifecycle data

rpm_first_published is the earliest timestamp at which any SRPM for a given package appeared in the Hummingbird Pulp repo. It is package-scoped — one row per package — as opposed to cve_ticket_events which is ticket-scoped.

Collection

scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

Flow:

  1. Loads the package list from rpms_repo.load_package_map_from_metadata().
  2. For each package: calls pulp.fetch_earliest_srpm_time(), which browses packages.redhat.com/.../source/Packages/{letter}/, parses the HTML listing already used by the analysis tool, and returns the earliest upload timestamp across all SRPMs for that package.
  3. POSTs all results in one request to /api/cve-import (same endpoint used for cve_ticket_events sync). The dashboard upserts with COALESCE(LEAST(existing, incoming), existing, incoming) — re-runs only update occurred_at if the incoming timestamp is earlier, and NULLs are never stored over a real value.

Surfaced in dump_lifecycle

scripts/dump_lifecycle --prod HUM-1234

After fetching ticket milestones from cve_ticket_events, the script resolves the package name from those rows and makes a second call to /api/cve-export?section=package_lifecycle to fetch rpm_first_published for that package. It is shown at the bottom of the text output and under rpm_first_published in the JSON output.

4.5.23 - Kubernetes Event Forwarder

A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch.

The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded.

Features

  • Multi-Cluster Support: Watch resources across multiple Kubernetes clusters using kubeconfig contexts
  • Dynamic Resource Watching: Configure any namespaced resource type using standard Kubernetes apiVersion and kind
  • SNS Integration: Publish events with structured message attributes for precise filtering
  • Compression: Events are compressed (gzip+base64) to reduce SNS message size
  • Automatic Reconnection: Handles watch connection failures and reconnects automatically

Architecture

Threading Model

The forwarder uses a multi-threaded architecture:

  • Watcher threads (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn’t affect others.

  • Publisher thread (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness.

Memory optimization: When SPOOL_DIR is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills).

LIST + Watch Pattern

The forwarder uses explicit LIST followed by Watch rather than resource_version="0":

  1. LIST retrieves all current objects and a snapshot resourceVersion
  2. Watch starts from the LIST’s resourceVersion (not object RVs)
  3. On graceful watch timeout (300s), restart watch from last known RV (no re-list)
  4. On errors, fresh LIST to ensure current state

This is necessary because synthetic ADDED events from resource_version="0" have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade.

Prerequisites

  • Target cluster: ServiceAccount with watch permissions on the resources you want to monitor
  • Deployment cluster: Kubernetes cluster to run the forwarder
  • AWS credentials: SNS publish permissions (optional - logs events if not configured)

Deployment

The container image is built via Konflux CI/CD and published to quay.io/hummingbird-ci/kubernetes-event-forwarder:latest.

Example Kubernetes manifests are provided in the kubernetes/ directory:

  • rbac.yaml - ServiceAccount, Role, RoleBinding for the target cluster
  • secret.yaml - Kubeconfig and AWS credentials
  • configmap.yaml - Resource watch configuration
  • deployment.yaml - Forwarder deployment

Quick Start

  1. On the target cluster (the one you want to watch), apply RBAC and create a token:

    kubectl apply -f kubernetes/rbac.yaml
kubectl create token kubernetes-event-forwarder --duration=8760h

  2. Update the example manifests with your values:

    • secret.yaml: cluster URL, token, AWS credentials
    • configmap.yaml: resources to watch
    • deployment.yaml: SNS topic ARN, AWS region

  3. Apply the manifests to your deployment cluster:

    kubectl apply -f kubernetes/secret.yaml
kubectl apply -f kubernetes/configmap.yaml
kubectl apply -f kubernetes/deployment.yaml

Prerequisites: Deploy hummingbird-events-topic first to create the SNS topic, then deploy the AWS resources (see below).

AWS Resources

The SAM template (template.yaml) provisions IAM resources for SNS publishing:

  • IAM User (${ResourcePrefix}-user) - Service account for the forwarder
  • IAM Policy (${ResourcePrefix}-policy) - Grants sns:Publish to the SNS topic

Deploy using containerized AWS SAM CLI:

cd kubernetes-event-forwarder
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely.

SAM Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)

Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern (e.g., myapp-prod-user, myapp-prod-policy).

Usage

Configure resources to watch in config.yaml:

resources:
  - apiVersion: v1
    kind: Pod
  - apiVersion: apps/v1
    kind: Deployment
  - apiVersion: v1
    kind: ConfigMap

SNS Subscription Filter Examples:

Pod events in a specific namespace:

{
  "source": ["kubernetes"],
  "kind": ["Pod"],
  "namespace": ["production"]
}

All deployment changes:

{
  "source": ["kubernetes"],
  "kind": ["Deployment"]
}

Deleted resources across all clusters:

{
  "source": ["kubernetes"],
  "event_type": ["DELETED"]
}

Development

See the main README for development workflows.

make kubernetes-event-forwarder/setup  # Install dependencies
make check                             # Lint code (ruff)
make fmt                               # Format code
make test                              # Run unit tests
make coverage                          # Run tests with coverage

Configuration

The deployment is configured via environment variables:

Variable Description
CONFIG_PATH Path to config YAML file
CONFIG Inline config YAML (alternative)
SNS_TOPIC_ARN SNS topic ARN (optional - logs if unset)
SPOOL_DIR Optional spool directory for file-based message queue
AWS_ACCESS_KEY_ID AWS access key ID
AWS_SECRET_ACCESS_KEY AWS secret access key
AWS_DEFAULT_REGION AWS region
METRICS_PORT Prometheus metrics port (default: 9090)
SENTRY_DSN Optional Sentry DSN

Event Metadata

The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes:

Attribute Description Example
source Always "kubernetes" kubernetes
cluster API server host https://api.cluster:6443
namespace Object namespace production
api_version Resource API version v1, apps/v1
kind Resource kind Pod, Deployment
event_type Event type ADDED, MODIFIED, DELETED
object_name Name of the object nginx-7d8c4c9d6f
content_encoding Message encoding gzip+base64

Security & Limitations

Security:

  • AWS credentials stored in Kubernetes Secret
  • Kubeconfig credentials stored in Kubernetes Secret
  • SNS topic follows least privilege principle (publish-only)
  • Sentry integration for error tracking

Limitations:

  • Only namespaced resources supported
  • One thread per context + resource type combination
  • SNS message size limit: 256 KB (after compression)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.24 - Jira Image Requests

AWS Lambda proxy that serves Hummingbird Image Request issues from Jira (with DynamoDB caching) and creates Stories/Epics from the catalog request form.

Features

  • Public list: GET /image-requests returns Ready Image Request issues
  • Form submit: POST /image-requests creates a version Story. An Epic is created only when a second request uses the same image name; later requests reuse that Epic. A first-time name is Story-only.
  • Customer scoring: Writes the Customer criterion on the Epic from paid email domains (placeholder until subscriber login)
  • Total scoring: Maintained by Jira Automation from manual triage fields

Prerequisites

  • AWS CLI with permissions for Lambda, API Gateway, DynamoDB, SSM, CloudFormation
  • Podman or Docker for SAM build
  • Jira bot token in SSM (SecureString)
  • HUM project custom fields for scoring (see below)

Deployment

make jira-image-requests/build
make jira-image-requests/deploy

Configuration

Parameter / env Description
ResourcePrefix Prefix for AWS resource names
JiraTokenParameter SSM parameter name for the Jira API token
CacheTtlSeconds Cache TTL for the public list (default 300)
CorsAllowOrigin Allowed CORS origin
ScoringCustomerField Jira custom field id for Customer (customfield_…); empty skips write
PaidCustomerDomains Comma-separated email domains scored as paid (e.g. acme.com,contoso.com)
SentryDsn Optional Sentry DSN

Image request scoring

Triage scores live on the Image Request Epic, not on customer-facing Stories.

Custom fields (HUM Epics)

Create (or map) these fields, then record each customfield_XXXXX id:

Field Type / values Who sets it
IR Legal Select: Y / N Manual
IR Competitive Number (typically 0–1) Manual
IR Marketable Number Manual
IR Supportability Number Manual
IR Upstream Health Number Manual
IR Level of Effort Number: 3 easy, 2 medium, 1 hard Manual
IR Technical Feasibility Number: 0 blocking, 1 feasible Manual
IR Portfolio Conflicts Number: 0 conflict, 1 good Manual
IR Customer Number: 0 or 1 Lambda (ScoringCustomerField)
IR Total Number Jira Automation only

Gate: Legal must be Y to continue scoring. Competitive, Marketable, and Supportability / Maturity are also expected to be 1 before a meaningful Total (team convention from the triage spreadsheet).

Lambda: Customer criterion

When a POST creates or reuses an Epic (second or later request for a name), the Lambda:

  1. Loads Stories under the Epic
  2. Sets Customer to 1 if any submitter email domain is in PaidCustomerDomains, otherwise 0
  3. Writes the value to ScoringCustomerField when configured

A first request for a name does not create an Epic, so Customer is not written until a second request for that name arrives.

When subscriber login lands, replace the domain list with real subscription status.

Configure two rules in Jira (Project settings → Automation). Use the real field names/ids from your instance.

Rule 1 — Sum Total when Legal is Y

  • Trigger: Field value changed for any of: Competitive, Marketable, Supportability, Upstream Health, Level of Effort, Technical Feasibility, Portfolio Conflicts, Customer (and optionally Legal)
  • Condition: Issue type = Epic AND component = Image Request AND IR Legal = Y
  • Action: Edit issue → set IR Total to the sum of the numeric criteria (exclude Legal). Example smart value shape:
{{#=}}
{{issue.IR Competitive}} + {{issue.IR Marketable}} + {{issue.IR Supportability}} +
{{issue.IR Upstream Health}} + {{issue.IR Level of Effort}} +
{{issue.IR Technical Feasibility}} + {{issue.IR Portfolio Conflicts}} +
{{issue.IR Customer}}
{{/}}

Use your instance’s smart-value field keys (often customfield_XXXXX).

  • Optional action: Remove label ir-legal-blocked when Legal is Y

Rule 2 — Flag when Legal is not Y

  • Trigger: IR Legal changed
  • Condition: Issue type = Epic AND component = Image Request AND IR Legal is empty OR IR Legal = N
  • Actions:
    1. Clear IR Total (or set to empty)
    2. Add label ir-legal-blocked

Jira Automation recalculates Total whenever triage fields change — no webhook or Lambda rescoring path is required.

Development

make jira-image-requests/setup
make test

License

GPL-3.0-or-later

4.5.25 - PAC Trigger

A CLI tool to manually trigger Konflux PAC (Pipelines-as-Code) pipelines for a specific component and branch. Useful for debugging pipeline issues or re-running builds without pushing a new commit.

Usage

# Trigger pipeline for a component (uses branch HEAD)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --cluster-url https://konflux-ui.apps.cluster.example.com

# Trigger with specific commit
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --commit abc123def456 \
            --cluster-url https://konflux-ui.apps.cluster.example.com

# Dry run (preview without creating)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --cluster-url https://konflux-ui.apps.cluster.example.com \
            --dry-run

Options

Option Description Default
--gitlab-project-url GitLab project URL (required) -
--component Component name (required) -
--branch Branch name main
--commit Specific commit SHA HEAD
--cluster-url Konflux cluster URL (required) -
--dry-run Preview PipelineRun without creating false
-v, --verbose Enable debug logging false

Installation

cd pac-trigger
pip install -e .

Prerequisites

  • Kubeconfig: Context with access to the target Konflux cluster/namespace
  • Repository resource: PAC Repository must exist for the GitLab project
  • Push template: Component must have a push template in .tekton/*.yaml

How It Works

  1. Fetch commit: Gets HEAD commit SHA from GitLab (or uses provided commit)
  2. Fetch template: Downloads .tekton/*.yaml files via GitLab API, finds push template matching the component
  3. Get namespace: Extracts namespace from template metadata
  4. Find credentials: Looks up PAC Repository resource to find git secret
  5. Create git-auth secret: Creates ephemeral pac-trigger-gitauth-* secret with git credentials
  6. Create PipelineRun: Substitutes template variables and creates the PipelineRun
  7. Link secret: Sets ownerReference on secret for garbage collection
  8. Print URL: Outputs Konflux UI URL for the PipelineRun

Template Variables

The following PAC template variables are supported:

Variable Substituted with
{{revision}} Commit SHA
{{target_branch}} Branch name
{{repo_url}} GitLab project URL
{{git_auth_secret}} Created secret name

Unsupported template variables will cause an error.

Labels Added

Label Value Purpose
pac-trigger/manual true Identifies manual triggers

Features

  • Template-based: Fetches PipelineRun templates from .tekton/*.yaml files via anonymous GitLab API
  • Push-only: Only triggers push templates (filters by CEL expression)
  • Automatic credentials: Creates ephemeral git-auth secrets from existing PAC Repository secrets
  • Garbage collection: Secrets are linked to PipelineRun via ownerReference
  • Kubeconfig-based: Uses local kubeconfig for cluster authentication

Development

See the main README for development workflows.

make pac-trigger/setup  # Install dependencies
make check              # Lint code (ruff)
make fmt                # Format code
make test               # Run unit tests
make coverage           # Run tests with coverage

Comparison with PAC-triggered Runs

Manually triggered PipelineRuns differ from PAC-triggered ones:

Present in both:

  • appstudio.openshift.io/application
  • appstudio.openshift.io/component
  • pipelines.appstudio.openshift.io/type: build
  • build.appstudio.redhat.com/commit_sha
  • build.appstudio.redhat.com/target_branch

Unique to pac-trigger:

  • pac-trigger/manual: true label

Missing (PAC internal metadata):

  • pipelinesascode.tekton.dev/event-type
  • pipelinesascode.tekton.dev/sha
  • GitLab status reporting annotations

These differences don’t affect pipeline execution—they’re used for PAC’s internal tracking and GitLab commit status updates.

Limitations

  • Only push templates supported (not pull-request)
  • Only GitLab repositories supported
  • Requires existing PAC Repository resource
  • No GitLab commit status reporting

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.26 - Hummingbird Status

SQS worker and database management for ingesting Konflux pipeline events.

Features

  • SQS Worker - Real-time database updates from SNS pipeline events
  • GitLab Event Ingestion - Push events and merge request events with version tracking
  • Database Init - Populate from pg_dump, S3 archive, or local filesystem mirror
  • Prometheus Metrics - Built-in metrics endpoint for monitoring
  • MR Reconciliation - Fix stale MR state by checking GitLab API
  • Schema Management - Idempotent database schema creation

Prerequisites

  • Python 3.11+
  • PostgreSQL 16
  • AWS credentials (for SQS worker and S3 initialization)

Installation

cd hummingbird-status
pip install -e .

Usage

Local Development

cd hummingbird-status

# Database management
./dev.sh db-start              # Start PostgreSQL container
./dev.sh db-init               # Initialize from available source
./dev.sh db-shell              # PostgreSQL interactive shell
./dev.sh db-dump               # Create pg_dump file
./dev.sh db-stop               # Stop PostgreSQL
./dev.sh db-reset              # Stop and delete volume

# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init   # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init        # From S3

# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker

Container Deployment

The container runs the SQS worker by default:

podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status

For database initialization:

podman run -e DATABASE_URL=... -e S3_BUCKET=... \
    hummingbird-status python3 -m hummingbird_status.worker.init_db

MR Reconciliation

MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear as “opened” in the dashboard when it is actually merged or closed. The reconcile command checks all open MRs against the GitLab API and corrects stale entries:

# Using environment variables
GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile

# Using CLI arguments
python -m hummingbird_status.reconcile \
    --database-url postgresql://... \
    --gitlab-token glpat-...

In a container:

podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \
    hummingbird-status python3 -m hummingbird_status.reconcile

Configuration

Environment Variables

Variable Default Description
DATABASE_URL - PostgreSQL connection URL
SQS_QUEUE_URL - SQS queue URL (worker)
S3_BUCKET - S3 bucket for init
S3_PREFIX sns/ S3 key prefix
LOCAL_MIRROR_PATH /data/sns-mirror Local S3 mirror path
INIT_DUMP_PATH /data/init/dump.sql pg_dump file path
GITLAB_URL https://gitlab.com GitLab instance URL (reconcile)
GITLAB_TOKEN - GitLab API token (reconcile)
METRICS_PORT 9090 Prometheus metrics port

Database Initialization Priority

  1. pg_dump file - $INIT_DUMP_PATH if exists
  2. Local S3 mirror - $LOCAL_MIRROR_PATH/sns/*.json.gz if exists
  3. S3 bucket - $S3_BUCKET/$S3_PREFIX with AWS credentials

Database Schema

The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).

flowchart TD
    push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
    mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
    mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
    comp[["<b>components</b><br/>git_context → component mapping"]]
    build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
    snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
    test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
    rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
    relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]

    push -- "+ affected" --> build
    mr -- "sha" --> mrv
    mrv -- "sha joins" --> build
    comp -- "components" --> build
    build -- "success<br/>creates" --> snap
    snap -- "triggers" --> test
    test -- "success<br/>creates" --> rel
    rel -- "managed<br/>by" --> relplr

Tables

Table Primary Key Description
gitlab_pushes sha,repo,ref GitLab push events to main branch
gitlab_merge_requests project,iid Current state of merge requests
gitlab_mr_versions project,iid,sha Head commit history for each MR
components name Konflux Component resources
pipelineruns name Build, test, and release pipelines
snapshots name Image snapshots after successful builds
releases name Published releases to target registry

Merge Request Tracking

The MR tables enable tracking build status across MR versions:

  • gitlab_merge_requests - Stores current MR metadata (title, state, branches, author, latest head SHA). Updated via ON CONFLICT ... WHERE updated_at < to keep the most recent state. author_name captures the webhook user.name alongside author (the username) so the dashboard can show a bot/access-token account’s real GitLab name (e.g. “chore-mr”) instead of its generated username – like author, it’s set once and preserved via COALESCE on conflict, so it isn’t backfilled for MRs that predate this column.

  • gitlab_mr_versions - Records each unique head commit SHA for an MR. When force-pushing the same SHA, created_at is updated to the latest event timestamp via GREATEST(), ensuring correct ordering even after force pushes.

Development

Running Tests

cd hummingbird-status
pip install -e ".[dev]"
pytest

Project Structure

hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml         # SAM template for AWS resources
├── hummingbird_status/
│   ├── db.py             # Database schema and utilities
│   ├── ingest.py         # SNS event parsing and ingestion
│   ├── reconcile.py      # MR state reconciliation with GitLab
│   └── worker/
│       ├── sqs.py        # SQS consumer
│       └── init_db.py    # Database initializer
└── tests/

AWS Resources

Deploy the SQS queue using SAM:

cd hummingbird-status
make build     # Build SAM application
make deploy    # First deployment (guided)
make redeploy  # Subsequent deployments

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
SnsTopicArn ARN of the SNS topic to subscribe (required)

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first.

See the main README for development workflows.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.27 - Lambda S3 Cache

An AWS Lambda setup that caches files from public URLs in S3. When a URL is requested, the service returns a 302 redirect to either the cached S3 copy (via presigned URL) or the original source. Cache misses trigger asynchronous downloads to S3, ensuring future requests are served from the cache.

Caching uses the original URL’s host and path as the S3 key - each unique URL maps to a single cache entry that persists until expiration. Frequently accessed content automatically extends its cache lifetime on each access.

Note: Cached content behavior depends on file type:

  • Immutable content (matching immutable extension filter, e.g., .rpm): Changes at origin won’t be reflected until cache expires
  • Mutable content (non-matching extension, e.g., repomd.xml): Cache is revalidated on each request via HEAD check; stale behavior is configurable

Cache Revalidation: Mutable content uses HTTP ETags for efficient validation. The uploader stores the origin’s ETag in S3 metadata when caching content. On subsequent requests, the handler sends a HEAD request with If-None-Match header containing the stored ETag. If the origin responds with 304 Not Modified, the cache is fresh and served directly. If the ETag differs, the cache is stale and an async refresh is triggered; response behavior depends on StaleCacheBehavior.

Note: If the origin doesn’t provide ETags, the cache cannot validate freshness. In this case, mutable content always redirects to origin without triggering cache updates (caching would be ineffective since every request would redirect anyway).

Features

  • Streaming Upload: Handles files efficiently by streaming directly from source to S3 without loading into memory
  • Presigned S3 URLs: Returns short-lived signed URLs for cached content
  • Automatic Expiration: Cached content expires after a certain time, with lifetime extended on each access
  • URL Prefix Allowlist: Only caches content matching explicitly allowed host+path prefixes
  • Immutable Extension Filter: Identifies immutable content that doesn’t require revalidation
  • Cache Revalidation: Mutable content (non-matching extensions) is validated on each request; stale cache behavior is configurable (origin or cache)
  • Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Three Lambda functions handle the caching workflow:

  1. Handler - API Gateway endpoint that checks cache, returns 302 redirects, and triggers async operations. Implements touch cooldown to prevent S3 throttling by only touching objects after a configurable time period has elapsed since last modification.
  2. Uploader - Downloads from origin and streams to S3 on cache misses (invoked asynchronously)
  3. Touch - Updates S3 object timestamps to extend cache lifetime on cache hits (invoked asynchronously only when cooldown period has elapsed)

Request Flow

flowchart TD
    Start([Request]) --> CheckURL{URL matches<br/>AllowedPrefixes?}

    CheckURL -->|No| RedirectOrigin
    CheckURL -->|Yes| HeadS3[HEAD S3]:::network
    HeadS3 --> CheckCache{Cache exists?}

    CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async
    CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?}

    CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin<br/>If-None-Match: stored ETag]:::network
    HeadOrigin --> CheckChanged{Origin changed?}

    CheckChanged -->|No ETag from origin| RedirectOrigin
    CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async
    CheckChanged -->|ETag matches| CheckCooldown
    CheckChanged -->|304 Not Modified| CheckCooldown
    CheckChanged -->|"Error/timeout"| CheckCooldown

    CheckImmutable -->|Yes| CheckCooldown

    CheckCooldown{Touch cooldown<br/>elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async

    InvokeUploaderMiss --> RedirectOrigin
    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin

    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache
    CheckCooldown -->|No| RedirectCache
    InvokeTouch --> RedirectCache

    subgraph redirectGroup [ ]
        RedirectOrigin[302 to Origin URL]:::redirect
        RedirectCache[302 to S3 Presigned URL]:::redirect
    end
    style redirectGroup fill:none,stroke:none

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff

Legend: Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

make lambda-s3-cache/build     # Build Lambda package
make lambda-s3-cache/deploy    # First deployment (interactive/guided)
make lambda-s3-cache/redeploy  # Subsequent deployments (non-interactive)

Deployment output: ApiEndpoint - the API Gateway URL to use for requests

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
PresignedUrlExpiration Presigned URL expiration (seconds) 3600
AllowedPrefixes Whitespace-separated list of allowed URL prefixes example.com/path/
ImmutableExtensions File extensions for immutable content (no revalidation) .rpm
CacheExpirationDays Days to keep cached content (minimum: 1) 14
TouchCooldownMinutes Minimum minutes between touch operations 60
StaleCacheBehavior When cache is stale: origin or cache origin
CustomDomainName Optional custom domain name ``
HostedZoneId Route53 hosted zone ID (required if custom domain) ``
SentryDsn Optional Sentry DSN for error tracking ``

Resource naming: All AWS resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-bucket, myapp-prod-lambda-handler).

Usage

Make GET requests to the API endpoint (or custom domain if configured). The URL to cache is encoded in the path (without the https:// prefix):

curl -L "https://<api-endpoint>/example.com/path/to/file.rpm"

Behavior:

  • First request (cache miss): Redirects to original URL while triggering async S3 upload
  • Subsequent requests (cache hit):
    • Immutable content (matching immutable extension filter): Redirects to presigned S3 URL and resets cache expiration
    • Mutable content (non-matching extension): HEAD request validates cache freshness (10s timeout). If stale, behavior depends on StaleCacheBehavior; if fresh, serves from cache
  • Disallowed prefixes: URLs not matching allowed prefixes are transparently redirected to the original URL (302 pass-through)

Immutable Extension Filter Behavior

The ImmutableExtensions parameter determines caching behavior:

  • Matching extension (e.g., .rpm): Treated as immutable content. Cached without revalidation - changes at origin won’t be reflected until cache expires. Served directly from cache on all requests.

  • Non-matching extension (e.g., .xml, .gz): Treated as mutable content. Cached with revalidation - HEAD request on each cache hit validates freshness using ETags. If stale, behavior depends on StaleCacheBehavior:

    • origin (default): Redirects to origin for fresh content, async refresh
    • cache: Serves stale cache immediately (faster), async refresh in background

Important: All files matching AllowedPrefixes are cached, regardless of extension. The extension filter only determines whether revalidation is performed.

Usage as Repository Proxy

The cache can be used as a DNF/yum baseurl for RPM repositories. Configure ImmutableExtensions to include .rpm:

  • .rpm files (matching extension): Cached as immutable - fast, no revalidation
  • Metadata files (non-matching extension, e.g., repomd.xml, primary.xml.gz): Cached with revalidation - ensures fresh metadata while benefiting from cache
[myrepo]
name=My Repository
baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/
enabled=1

This provides caching benefits for RPM downloads while ensuring repository metadata stays fresh.

Development

See the main README for development workflows.

make lambda-s3-cache/setup  # Install dependencies
make check                  # Lint code
make fmt                    # Format code
make test                   # Run unit tests
make coverage               # Run tests with coverage

Configuration

Lambda functions receive configuration via environment variables (automatically set by CloudFormation):

Variable Handler Uploader Touch Description
S3_BUCKET_NAME ✅ ✅ ✅ S3 bucket name
PRESIGNED_URL_EXPIRATION ✅ Presigned URL expiration (sec)
UPLOADER_LAMBDA_ARN ✅ Uploader Lambda ARN
TOUCH_LAMBDA_ARN ✅ Touch Lambda ARN
TOUCH_COOLDOWN_MINUTES ✅ Min minutes between touch ops
ALLOWED_PREFIXES ✅ Allowed URL prefixes
IMMUTABLE_EXTENSIONS ✅ Extensions for immutable content
STALE_CACHE_BEHAVIOR ✅ Stale behavior: origin or cache
SENTRY_DSN ✅ ✅ ✅ Optional Sentry DSN

Security & Limitations

Security:

  • S3 bucket has public access blocked; all objects encrypted at rest (AES256)
  • Presigned URLs expire after a certain time
  • IAM policies follow least privilege principle
  • URL prefix allowlist prevents caching arbitrary URLs
  • Immutable extension filter distinguishes immutable vs mutable content for revalidation

Limitations:

  • Lambda timeout: 15 min (uploader), 30 sec (handler, touch)
  • Lambda memory: 1024 MB (uploader), 256 MB (handler, touch)
  • S3 object size: Up to 5 TB (AWS limit)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.28 - RPM CVE Count

A CLI tool to count the number of known CVEs for a given list of RPM packages by querying the Red Hat OSIDB database.

Features

  • Batch processing: Query multiple packages from a file
  • Impact filtering: Filter by CVE severity (CRITICAL, IMPORTANT, MODERATE, LOW)
  • Date filtering: Count only CVEs created after a specific date
  • CSV output: Easy to import into spreadsheets or process with other tools

Prerequisites

  • Red Hat VPN: Must be connected to access the OSIDB database
  • Go 1.21+: For building from source

Installation

Install the latest version:

go install gitlab.com/redhat/hummingbird/tools/rpm-cve-count@latest

Or build from source:

git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/rpm-cve-count
go build

Usage

rpm-cve-count -file <package-file> [-after <date>] [-impact <level>]

Options

Option Description Required
-file Read packages from file (one per line) Yes
-after Count CVEs created after date (YYYY-MM-DD) No
-impact Filter by impact: CRITICAL, IMPORTANT, MODERATE, LOW No

Examples

Create a file with package names (one per line):

# packages.txt
kernel
systemd
openssl
glibc

Count all CVEs for the packages:

$ rpm-cve-count -file packages.txt
kernel,342
systemd,87
openssl,156
glibc,234

Count only CRITICAL CVEs:

$ rpm-cve-count -file packages.txt -impact CRITICAL
kernel,23
systemd,5
openssl,18
glibc,12

Count CVEs created after a specific date:

$ rpm-cve-count -file packages.txt -after 2024-01-01
kernel,45
systemd,12
openssl,28
glibc,31

Combine filters to count CRITICAL CVEs from the last year:

$ rpm-cve-count -file packages.txt -impact CRITICAL -after 2024-01-01
kernel,8
systemd,2
openssl,5
glibc,3

Save results to CSV:

rpm-cve-count -file packages.txt > results.csv

Output Format

CSV format with two columns:

  • Package name
  • CVE count

Development

# Build
go build

# Run tests
go test ./...

# Install locally
go install

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.29 - Hummingbird Dashboard

Web dashboard and CLI for monitoring Konflux build pipeline status.

Features

  • Web Dashboard - Real-time view of build, test, and release status
  • Commits View - Detailed pipeline status per commit with expandable releases
  • Components View - Latest status per component grouped by state
  • Merge Requests View - Track build status across MR versions
  • CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
  • Failed Releases - View and manage failed releases with LLM-powered analysis
  • Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
  • Auto-Rerun - Automatically retry transient release and push build failures
  • Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
  • Smart Triggering - Application-aware rules for determining affected components
  • Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
  • R-Time Blame Tagging - Assign blame categories to late tickets from the stage dropdown
  • ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
  • Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks

Prerequisites

Installation

cd hummingbird-dashboard
pip install -e .

Usage

Local Development

cd hummingbird-dashboard

# Option 1: Port-forward to production database
./dev.sh port-forward  # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start  # In terminal 2

# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start

The dashboard runs at http://localhost:8080 with live reload.

Web UI

Endpoint Description
/ Dashboard with all applications
/apps/{app}/commits Commits view for an application
/apps/{app}/components Components view for an application
/mrs Merge requests overview (filterable)
/mr/{project}/{iid} MR detail with build status per version
/releases/failed Failed releases with analysis
/releases/blocked Blocked snapshots management
/releases/analysis-log LLM analysis run history
/cve/status CVE ticket status dashboard
/cve/run-log CVE analysis run history
/cve/open Open CVE trackers with SLO status
/cve/closed Closed tickets with VEX reconciliation
/cve/r-time CVE R-Time (notified to done) metrics
/availability Estimated image pull availability (SLI-UE1-B)
/health Health check endpoint
/metrics Prometheus metrics

Each CVE tracker page (/cve/status, /cve/open, /cve/closed, /cve/r-time) has a “Download CSV” button at the bottom of the page that downloads the current view (same filters as the page) as a CSV file, via a matching /cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID” toggle in the header reveals the CVE ID next to each HUM ticket key (HUM-1234/CVE-2026-56789); the preference persists via localStorage.

CVE Search API

GET /api/cve-runs/search — search CVE analysis run history.

Parameter Type Default Description
package string — Filter by package name (substring match)
ticket string — Filter by ticket key (substring match)
q string — Free-text search across ticket details
log string — Search CronJob log output
days int 30 Time window in days (1–365)
limit int 500 Max results (1–2000)

At least one of package, ticket, q, or log is required.

The response includes two result sets:

  • results — per-ticket matches from the JSONB details column
  • log_matches — per-run matches from CronJob log output, each with a log_excerpt showing the matching lines in context

Example:

# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {run_at, log_excerpt}'

# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
  | jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'

# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'

CLI

# View latest commit status
hummingbird-dashboard --application myapp --format table commit

# View specific commit
hummingbird-dashboard --application myapp commit abc1234

# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10

# Component status overview
hummingbird-dashboard --application myapp component

# JSON output
hummingbird-dashboard --format json commit | jq .

# Auto-rerun failed releases
hummingbird-dashboard auto-rerun

# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures

# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force

REST API Reference

All endpoints return JSON. Read-only GET endpoints are unauthenticated. Interactive OpenAPI/Swagger documentation is available at /docs.

Tier 1 — Core Pipeline Status

GET /api/apps/{app_name}/commits

Pipeline status for commits in an application.

Parameter Type Default Description
sha string — Filter by commit SHA (prefix)
component string — Filter by component name
limit int 10 Max commits to return (max 100) 
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .

GET /api/apps/{app_name}/components

Latest build status per component, grouped by state.

Parameter Type Default Description
search string — Filter components by name (substring) 
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .

Tier 2 — Operational Visibility

GET /api/dashboard

Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.

curl -s 'http://localhost:8080/api/dashboard' | jq .

GET /api/releases/failed

List of currently failed releases with analysis results.

Parameter Type Default Description
application string — Filter by application name 
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .

GET /api/releases/{name}/analysis

LLM failure analysis for a specific release (root cause, classification, recommendation).

curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .

GET /api/releases/auto-rerun-log

History of automatic rerun attempts.

Parameter Type Default Description
limit int 50 Max entries 
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .

GET /api/releases/analysis-log

History of LLM analysis runs.

Parameter Type Default Description
limit int 50 Max entries 
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .

GET /api/releases/analysis-costs

Cumulative token usage and estimated cost for LLM analysis runs.

curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .

GET /api/mrs

Merge requests across monitored repositories.

Parameter Type Default Description
state string opened MR state: opened, merged, closed, all
project string — Filter by project (repository) name 
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .

GET /api/mrs/projects

List of all projects (repositories) with tracked merge requests.

curl -s 'http://localhost:8080/api/mrs/projects' | jq .

GET /api/mr/{project}/{iid}

Detail for a single merge request, including all versions and per-version build status.

curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .

GET /api/cve/status

Current CVE ticket status across all tracked packages.

curl -s 'http://localhost:8080/api/cve/status' | jq .

GET /api/cve/open

Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).

Parameter Type Default Description
slo float 24 SLO threshold hours (24, 72, or 168)
include_next_release bool true Include cve-next-release tickets 
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .

GET /api/cve/closed

Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation (HUM-5843). Analysis stores package-scoped vex_status and Jira resolution; the dashboard computes MATCH from those facts (Done-Errata → fixed, Not a Bug → known_not_affected or package_not_listed). Shows Hummingbird vex_status, match state (matched / pending / mismatch), and vex_resolved (first scan time where VEX agreed with the Jira resolution). Event metadata is merged across a ticket’s rows in occurred_at order (HUM-6091): later non-empty fields overlay earlier ones, so Close/VEX facts are not stuck on the first cve_published row. The Resolution column uses Jira Closed / {resolution} when stored analysis text is not already Closed. The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat CSAF VEX document when the package is named in CSAF. package_not_listed (Not a Bug, package absent from the document) shows N/A with no link: this ticket did not produce a VEX change.

Parameter Type Default Description
days int 30 Time window (0 = all time) 
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .

GET /api/cve/r-time

CVE R-Time (notified to done) for Done-Errata tickets. Include CVE-HUM starts at the earlier of CVE publication (NVD datePublished / cve_published) and HUM ticket creation. Exclude CVE-HUM starts at HUM created. Delivery is the catalog image publish when the package is a catalog image source, otherwise the RPM publish. Include VEX requires delivery, VEX updated, and Closed / Done-Errata; Exclude VEX requires delivery and close only. The done timestamp is the latest of the required gates, and only when all of them exist. Incomplete tickets stay on the table with elapsed time until now; the days window filters by Done-Errata close (fallback: notified), same as completed tickets use done. CVE-HUM is filing lag after NVD (max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is Done-Errata close to VEX feed update (max(VEX − close, 0)); close is recorded when the advisory MR merges. Upstream/Fedora are not start or end. Pre-built (delivery before notification) is informational only. Rows without catalog_image_source in metadata still require an image until the collector restamps them. Each entry includes package onboarding timestamps and delay analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours). Default excludes cve-next-release, CVE-HUM time, and VEX time.

Parameter Type Default Description
days int 30 Time window (0 = all time)
slo float 168 SLO threshold hours (24, 72, or 168)
include_next_release bool false Include cve-next-release tickets
include_hum_cve_time bool false Include NVD-to-HUM filing lag in R-Time
include_vex_time bool false Require VEX feed update before R-Time ends 
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .

GET /api/cve/run-log

History of CVE analysis CronJob runs.

Parameter Type Default Description
limit int 100 Max entries 
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .

GET /api/cve-export

Export CVE sync data as JSON for prod-to-preprod replication. Valid section values are cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame.

Optional pagination parameters:

Parameter Type Default Description
section string — Section to export
limit int 2000 Rows per page
offset int 0 Pagination offset 
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'

curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
  | jq '.count,.has_more'

# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
  | jq '.rows[] | select(.event_type == "rpm_first_published")'

Event type validation. The import path (POST /api/cve-import) validates every event_type value against a canonical allowlist defined in hummingbird_dashboard/sources.py:

  • CANONICAL_TICKET_EVENT_TYPES — values accepted for cve_ticket_events: cve_published, osidb_flaw_created, osidb_affect_created, hum_ticket_created, hum_ticket_closed, upstream_fix_merged, fedora_update_available, rpm_fix_published_to_pulp, image_rebuilt_on_quay, vex_resolved.
  • CANONICAL_PACKAGE_EVENT_TYPES — values accepted for package_lifecycle: rpm_first_published.

Legacy names (e.g. jira_created, delivered_in_rpm, …) are mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP. Unknown names are rejected with HTTP 400.

POST /api/cve-import

Replace local CVE sync data from a payload previously returned by /api/cve-export.

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data @cve-export.json \
  'http://localhost:8080/api/cve-import' | jq .

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
  'http://localhost:8080/api/cve-import' | jq .

To copy prod CVE sync rows onto preprod (both sections; first page of each section uses reset=true):

CVE_REPORT_TOKEN=... ./hummingbird-cve-analysis/scripts/copy_prod_to_preprod.sh
CVE_REPORT_TOKEN=... ./hummingbird-cve-analysis/scripts/copy_prod_to_preprod.sh --dry-run

The script prefers paginated /api/cve-export. If that endpoint returns HTTP 500, it retries unpaged /api/cve-export. Analysis logs may fall back to /api/cve/run-log (latest 100 runs). Ticket events are not reconstructed from Open/Closed/R-Time: those views omit ticket-event metadata. Before import, ticket events collapse pre-HUM-5918 names (jira_closed, delivered_in_image, …) onto canonical types so a null legacy row cannot wipe a timestamp or VEX label. Prod ticket-event rows often omit catalog_image_source; the copy fills that flag from the catalog source map (same rule as the collector) and keeps prod’s True/False when present. The copy fails if the catalog map cannot be built. Ticket-event types such as osidb_flaw_created and osidb_affect_created copy with the rest of cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.

occurred_at conflict resolution on import (both the paginated and full-replace code paths) matches the live collector path (HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable delivery timestamps where a non-null incoming value wins, so a sync can update them, without letting a null incoming value erase an already-recorded delivery timestamp; every other event_type is write-once, keeping the existing timestamp once set so a sync or backfill re-run cannot overwrite an established immutable milestone.

cve_analysis_log is paged 5 rows at a time by default. Each row includes full log_output (up to 512 KiB). Ticket events default to 1000 rows per page.

Tier 3 — Analytics & SRE Metrics

GET /api/stats/build-throughput

Build counts over time (successful, failed, total).

Parameter Type Default Description
days int 30 Lookback window in days 
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .

GET /api/stats/release-success-rate

Release success/failure ratios over time.

Parameter Type Default Description
days int 30 Lookback window in days
application string — Filter by application 
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .

GET /api/stats/auto-rerun-effectiveness

Success rate and time-to-resolution for automatic reruns.

Parameter Type Default Description
days int 30 Lookback window in days 
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .

GET /api/components/staleness

Components ranked by time since last successful build.

curl -s 'http://localhost:8080/api/components/staleness' | jq .

GET /api/releases/similar-failures

Find releases with error messages similar to a given string.

Parameter Type Default Description
error string — Error text to match against 
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .

GET /api/apps/{app_name}/last-successful

Timestamp and SHA of the last fully successful pipeline per component.

curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .

Tier 4 — System

GET /api/service-status/json

Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).

curl -s 'http://localhost:8080/api/service-status/json' | jq .

GET /api/settings

Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).

curl -s 'http://localhost:8080/api/settings' | jq .

GET /health

Enriched health check returning service version, uptime, database connectivity, and dependency status.

curl -s 'http://localhost:8080/health' | jq .

Configuration

Environment Variables

Variable Default Description
DATABASE_URL postgresql://...localhost:5432/ PostgreSQL connection URL
PORT 8080 Web server port

Authentication Variables

These control the retrigger functionality (requires OAuth proxy in production):

Variable Default Description
TRIGGER_AUTH_MODE oauth oauth (production) or local
TRIGGER_AUTH_GROUP konflux-hummingbird-admin-access OpenShift group required to retrigger
TRIGGER_LOCAL_USER local-dev Username when TRIGGER_AUTH_MODE=local
CVE_REPORT_TOKEN Bearer token for CVE API auth

For local development with retrigger enabled:

TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start

Auto-Rerun Variables

These control the auto-rerun cronjob and failure analysis:

Variable Default Description
AUTO_RERUN_MIN_AGE_MINUTES 30 Minimum failure age before retrying
AUTO_RERUN_MAX_RETRIES 3 Max rerun attempts per snapshot+plan
KONFLUX_KUBECONFIG_PATH Path to Konflux kubeconfig file
RELEASE_NAMESPACE Namespace where releases are created
MANAGED_NAMESPACE Namespace for fetching release PLRs
KUBEARCHIVE_URL KubeArchive API URL for archived resources
GOOGLE_APPLICATION_CREDENTIALS Path to GCP SA key for Vertex AI
GOOGLE_CLOUD_PROJECT GCP project ID for Vertex AI
ANALYSIS_MODEL_API_KEY Gemini API key (fallback if no GCP creds)
ANALYSIS_MODEL gemini-2.5-flash LLM model for failure analysis
ANALYSIS_MODEL_REGION global Vertex AI region
MAX_ANALYSES_PER_CYCLE 5 Max releases to analyze per cycle
SLACK_WEBHOOK_URL Slack webhook for rerun/analysis notifications
DASHBOARD_URL Dashboard base URL for Slack links
GITLAB_SLACK_MAP_PATH /etc/hummingbird/gitlab-slack-map.yaml GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention)
PUSH_RERUN_MIN_AGE_MINUTES 10 Min push build failure age before retrying
PUSH_RERUN_MAX_RETRIES 3 Max retry attempts per component+sha

Error Budget Alert Variables

The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:

Remaining budget Slack alert
≤20% Warning
≤5% Critical
0% Exhausted (sprint-stop)

Incoming webhooks always post to the channel they were created for; the payload cannot select #team-hummingbird. Create the webhook for that channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared SLACK_WEBHOOK_URL at that webhook).

State is stored in dashboard_settings (error_budget_rtime_alert_state) so the same threshold is not re-alerted. The stored state is claimed with a compare-and-set before Slack is called, and rolled back if the post fails, so a transient Slack error is retried on the next interval and concurrent evaluators cannot double-post the same transition. Recovery updates the stored state silently so a later re-worsening can alert again. No message is sent when the webhook URL is unset, or when alerts are disabled via the R-Time page toggle (error_budget_alerts_enabled in dashboard_settings; default enabled).

Variable Default Description
ERROR_BUDGET_SLACK_WEBHOOK_URL SLACK_WEBHOOK_URL Incoming webhook created for the alert channel (typically #team-hummingbird)
SLACK_WEBHOOK_URL Fallback webhook if the error-budget URL is unset
DASHBOARD_URL Base URL for the R-Time deep link in alerts
ERROR_BUDGET_ALERT_INTERVAL_SECONDS 900 Seconds between background checks

Failures

The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.

Failed Releases

The /failures/releases page shows all failed releases with:

  • LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
  • Failure Classification — Transient, Configuration, Code, External Service, or Unknown
  • Rerun History — Past rerun attempts and outcomes per snapshot
  • Error Pattern Matching — Auto-block snapshots matching known error patterns
  • Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
  • Analysis Toggle — Enable/disable LLM analysis from the dashboard

CLI Subcommands

The auto-rerun subcommand retries eligible failed releases:

hummingbird-dashboard auto-rerun --application myapp

The auto-rerun-push subcommand retries eligible failed push builds:

hummingbird-dashboard auto-rerun-push --application myapp

The analyze-failures subcommand runs LLM analysis on unanalyzed failures:

hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant

All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR.

Image Pull Availability

The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually pull the images we publish. Continuously pulling every catalog image on a tight schedule is not feasible (terabytes of registry traffic), so the check is split into three tiers, run by check-image-pulls:

  • Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes transferred, against every latest* image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours.
  • Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
  • Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.

Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.

  • SLO: 95% target, 28-day rolling window, based on the manifest tier
  • Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
Option Description
--dry-run Probe without writing results to the database
--force Force live manifest HEAD checks (ignore digest cache)
--catalog-api-base Catalog API base URL (default: CATALOG_API_BASE or built-in)

Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be disabled from the dashboard via the image_pull_checks_enabled setting.

Merge Requests

The /mrs page shows merge requests across all monitored repositories with:

  • State filter - Open, merged, closed, or all MRs
  • Project filter - Filter by specific repository
  • Build status - Aggregate status across all components

The /mr/{project}/{iid} detail page shows:

  • All MR versions - Each head commit SHA that was pushed to the MR
  • Build status per version - Full pipeline status (build, snapshot, test, release)
  • Links to Konflux UI - Direct links to PipelineRuns and Snapshots

Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.

Component Status

The dashboard tracks component build status with these states:

Status Icon Description
Success ✅ Build passed for expected commit
Superseded 🔄 Expected commit not built, but newer succeeded
Failed ❌ Build failed for expected commit
Stale ⚠️ Build not triggered for expected commit
Running ⏳ Build in progress
Missing ❓ No build found

Components are grouped by status: Failed/Stale → Running → OK → Missing.

Trigger Rules

The dashboard uses application-specific rules to determine which components are affected by a push:

  • containers: Changes in images/{component} trigger builds
  • rpms: Changes in rpms/{component} or mock/mock.cfg trigger builds
  • tools: Changes in {component} directory trigger builds

Certain files are excluded from triggering (README, templates, etc.).

Development

See the main README for development workflows.

Running Tests

cd hummingbird-dashboard
pip install -e ".[dev]"
pytest

Project Structure

hummingbird-dashboard/
├── Containerfile
├── dev.sh
├── hummingbird_dashboard/
│   ├── analysis.py     # LLM failure analysis (Gemini)
│   ├── cli.py          # CLI entry point
│   ├── db/             # SQLAlchemy engine/session + ORM models
│   │   ├── engine.py
│   │   └── models.py
│   ├── konflux.py      # Konflux API client
│   ├── models.py       # Data models (Component, PushEvent)
│   ├── sources.py      # PostgreSQL queries
│   ├── table.py        # CLI table formatting
│   ├── triggers.py     # Application-specific trigger rules
│   ├── views.py        # Status computation and aggregation
│   └── web/
│       ├── app.py      # FastAPI application
│       └── templates/  # Jinja2 templates
└── tests/

Building Container Image

cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.30 - Work Queue Service

Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation; designed to support CVE remediation, advisory tracking, and other work item types.

For design decisions and architectural rationale, see Work Queue Service Design.

Architecture

The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).

flowchart TD
    clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
    gl_events["GitLab\n(events)"] -- webhook --> fwd["gitlab-event-forwarder"]
    fwd --> sns["SNS"]
    sns -->|"filter: source=gitlab,\nevent_type=merge_request"| sqs["SQS Queue"]
    sqs --> svc
    svc -- "Commits API,\ncreate MR" --> gl["GitLab\n(API)"]
    svc -- "shared DB" --> pg[("PostgreSQL")]

The SQS consumer thread receives GitLab merge_request events from the SNS topic via a filtered SQS subscription. Events are stored in the pending_events table and processed by existing work-item threads, which map MR state changes to phase transitions.

Design

Generic + detail tables

The schema separates queue machinery from type-specific data:

  • work_items — generic: id, type, phase, pending_action, attempts, leased_until, error, timestamps. Shared by all work item types.
  • mr_details — MR-specific: source, package, project, file_actions, commit_message, mr_iid, etc. Joined to work_items via foreign key.
  • work_item_statuses — independent readiness dimensions with composite key (work_item_id, dimension, variant).

New work item types add their own detail table (e.g. cve_details) — no changes to the generic layer. SQLAlchemy joined table inheritance loads the correct subclass automatically via polymorphic_load="selectin".

Phases

Four generic phases track coarse lifecycle status:

Phase Description
active Being processed — the only mutable phase
completed All lifecycle steps done (terminal)
failed Error occurred (terminal, retryable)
cancelled Cancelled (terminal, retryable)

Items start as active. Only active items accept event-driven phase transitions and executor actions. The three terminal phases (completed, failed, cancelled) are immutable to events — recovery requires explicit action via the retry API. GitLab merge_request events drive transitions from active: merge -> completed, close -> cancelled.

Dimensions

Independent readiness signals stored in work_item_statuses. Each dimension has a status and an optional variant for multi-instance signals (e.g. per-component Konflux builds). Dimensions are set by executors or event handlers via EventResult.dimensions; the generic handle_event layer applies event-driven dimensions and emits audit entries.

Decider

A pure function registered per type that derives the next action from the work item’s current state. The decider reads the selectin-loaded item.statuses list (in-memory, no DB call) to check dimensions:

def _mr_decide(item: MRWorkItem) -> str | None:
    if item.phase != "active":
        return None
    if item.mr_iid is None:
        return "create_mr"
    if not any(s.dimension == "dashboard_link" for s in item.statuses):
        return "ensure_dashboard_note"
    return None

The decider only determines the next pending_action — it does not set phases or modify state.

MR action chain

The MR work item lifecycle chains two executor actions:

  1. create_mr — creates the branch, commit, and MR via the GitLab Commits API. Applies the managed-by::workqueue scoped label.
  2. ensure_dashboard_note — posts an internal note with the Hummingbird dashboard link on the MR and stores the dashboard_link dimension with the note_id.

After both actions complete, the decider returns None and the item sits in active phase waiting for event-driven lifecycle tracking (merge → completed, close → cancelled).

The ensure_dashboard_note executor stores a dashboard_link dimension with detail {"note_id": int, "url": str}. The note_id is the GitLab note ID used for future status updates.

stateDiagram-v2
    direction LR
    absent --> posted : ensure_dashboard_note executor

Label coordination with dashboard-mr-linker. The managed-by::workqueue label causes the dashboard-mr-linker Lambda to skip the MR, so the workqueue-service creates the dashboard note for its MRs. When dashboard_base_url is not configured, the ensure_dashboard_note executor is not registered and the decider’s action triggers a failure (visible in audit and metrics).

Processing loop

The ProcessingManager runs per-item threads:

  • Claims items where pending_action IS NOT NULL using SELECT ... FOR UPDATE SKIP LOCKED
  • Serial per item — one action at a time per work item
  • Parallel across items — independent items processed concurrently
  • Heartbeat context manager extends leased_until during execution
  • Executors apply state updates directly (phase, MR fields) and must not call session.commit() or session.rollback()
  • After execution, the decider determines the next pending_action
  • Periodic sweep reclaims items with expired leases

MR labeling

The executor always adds the managed-by::workqueue scoped label to every MR it creates, merged with any caller-provided labels. This label is a coordination signal: other services (e.g. dashboard-mr-linker) check for this label and skip managed MRs, since the workqueue-service handles dashboard note creation itself via the ensure_dashboard_note action.

Event processing

The SQS consumer thread receives merge_request events from the SNS topic and stores them in the pending_events table. Work-item threads drain and process these events as part of their main loop.

Consumer thread. A single daemon thread (poll_loop) long-polls SQS with MaxNumberOfMessages=10 and WaitTimeSeconds=20. For each message it decodes the SNS envelope (handles gzip+base64), resolves matching work items via the resolver registry, INSERTs into pending_events, commits, deletes the SQS message, and calls manager.notify() for each affected work item. The consumer is started only when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is set.

State mapping. MR state changes are mapped to phases:

GitLab state Phase Notes
merged completed Terminal
closed cancelled Terminal (recover via retry API)
opened — Ignored (only active items accept events)
locked — Ignored (transient during merge)

Only items in active phase accept event-driven transitions. Events for terminal items (completed, failed, cancelled) are silently discarded.

Out-of-order guard. The last_event_at column on work_items rejects events with updated_at older than the last processed event. Same-timestamp events pass the <= guard (processing is idempotent).

Advisory lock. A per-work-item pg_advisory_xact_lock coordinates the consumer’s INSERT with the work-item thread’s exit decision, preventing events from being orphaned between the final drain check and thread exit.

Sweep extension. The periodic sweep (every 5 minutes) also checks for work items with unprocessed pending_events, providing startup recovery and a safety net behind the advisory lock.

Adding a new work item type

  1. Create a detail table and SQLAlchemy model (e.g. CveWorkItem)
  2. Register a decider function: decider.register("cve", _cve_decide)
  3. Register executor functions for each action
  4. Add a submission endpoint: POST /api/work-items/cve

No changes needed to the processing loop, claiming, or generic API.

API

Endpoints

Method Path Description
POST /api/work-items/mr Submit an MR work item
GET /api/work-items/mr List MR items (source, project, etc.)
GET /api/work-items List all items (type, phase filters)
GET /api/work-items/{id} Get item with nested details
POST /api/work-items/{id}/retry Retry a failed item
GET /api/whoami Introspect caller identity and permissions
GET /oauth/start Initiate OAuth login (handled by oauth-proxy)
GET /oauth/callback OAuth provider callback (handled by oauth-proxy)
POST /oauth/token Exchange OAuth session cookie for Bearer token
GET /healthz Health check
GET /metrics Prometheus metrics

Responses include a nested details dict with type-specific fields:

id: "..."
type: mr
phase: active
details:
  source: rpms-ci
  mr_iid: 42
  mr_url: https://gitlab.com/org/group/project/-/merge_requests/42

Authentication

All callers authenticate via Authorization: Bearer tokens. The service validates each token in one of two ways:

Caller Token type Validation
K8s SA JWT (projected) JWKS signature verification
GitLab CI JWT (OIDC) JWKS signature verification
Human (web/CLI) sha256~... (OCP) OCP users/~ API
Local dev none WORKQUEUE_SERVICE_LOCAL_AUTH_USER env var

X-Forwarded-* headers are never trusted on API paths. The proxy protects only the OAuth login flow (/oauth/start, /oauth/callback) and the POST /oauth/token endpoint, which bridges cookie sessions to Bearer tokens for browser and CLI users. The /oauth/token endpoint uses POST (not GET) to prevent CSRF — browser-initiated GETs via <img> or <script> tags cannot trigger it.

OAuth proxy deployment requirements

The oauth-proxy sidecar handles only the OAuth login flow. All /api/ paths bypass the proxy — the app handles auth itself. The proxy must be configured with these flags:

Flag Purpose
--bypass-auth-except-for=^/oauth/ Proxy only protects the OAuth flow (login + token exchange)
--pass-access-token Sets X-Forwarded-Access-Token on cookie-authenticated requests (required for /oauth/token). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size)
--pass-user-headers Sets X-Forwarded-User on protected paths
--cookie-refresh=1h Refreshes the OCP token inside the cookie before it expires (see below)
--cookie-expire=24h Hard session limit, forces re-login
--cookie-samesite=lax CSRF defense-in-depth for /oauth/token (POST already blocks the main vectors)

Why --cookie-refresh is critical

The proxy manages two independent lifetimes that are easy to confuse:

  • Session cookie — controlled by the proxy (--cookie-expire, defaults to browser session). Contains the encrypted OCP access token.
  • OCP access token — controlled by OCP (default 24h). The actual credential the app uses to call users/~.

The proxy does not know when the OCP token inside its cookie expires. It only checks whether the cookie is valid (signature, expiry). Without --cookie-refresh, the cookie outlives the token:

Time What happens
T+0h User logs in. Proxy creates cookie with fresh OCP token (expires T+24h)
T+24h OCP token expires. Cookie still valid.
T+25h User makes request. Proxy sets X-Forwarded-User (valid cookie) and X-Forwarded-Access-Token (expired token). App calls users/~ with expired token → 401 → anonymous → “not authorized”.

--cookie-refresh=1h tells the proxy to silently refresh the OCP token inside the cookie every hour, keeping it well within the 24h expiry. The workqueue-service also uses a TTL-bounded cache (5 minutes) for users/~ results, so expired-token failures are never permanent.

Authorization

Access rules are defined in the config file. Each rule maps a remote identity to permissions (read, retry, submit), optionally scoped to work item types. Rules use either groups (OCP group membership) or issuer+claims (JWT bearer). Claim matching follows Vault’s bound_claims pattern: all conditions AND, list values OR.

All permissions (including read) must be granted by a matching rule. Authenticated callers with no matching rules get 403. Multiple matching rules are merged (union of permissions).

See Configuration for the config file format.

Whoami introspection

GET /api/whoami returns the caller’s identity and effective permissions. Example response:

authentication:
  tier: ocp-user
  identity: admin-user
  groups:
    - konflux-hummingbird-admin-access
authorization:
  effective_permissions:
    mr:
      - read
      - retry
      - submit
  matching_rules:
    - groups:
        - konflux-hummingbird-admin-access
      permissions:
        - read
        - submit
        - retry
      item_types: []

API Documentation

Interactive API documentation available without authentication: /docs (Swagger UI), /redoc (ReDoc), /openapi.json.

Audit Log

State-changing operations (processing, API calls, auth decisions) emit audit entries. Every entry is logged at INFO level. When WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set, entries are additionally archived to S3 as gzipped JSON objects for long-term querying via Athena.

Audited events

Event type Source Description
action_started Worker Action execution began
action_completed Worker Action finished (success or failure)
action_decided Decider Decider assigned next action
action_no_op Decider Decider returned no next action
phase_change Worker / API / Events Phase transition (max attempts, submit, event)
dimension_change Events / Executor Dimension upsert (dashboard_link posted, etc.)
retry_requested API Retry endpoint reset a failed item
auth_failed Auth Anonymous request rejected (401)
auth_denied Auth Authenticated caller lacks permission (403)

S3 key format

When S3 archival is enabled, entries are stored as gzipped JSON with an Athena-partitionable key layout:

{prefix}/YYYY/MM/DD/HH/MM/{timestamp}#{work_item_id}::{event_type}.json.gz

The ContentEncoding: gzip header is set so S3 API consumers get transparent decompression.

Prometheus Metrics

The service exposes Prometheus metrics at /metrics on the same port as the API (no sidecar). Metrics follow the repo naming convention: generic operational metrics use the shared hummingbird_ prefix, service-specific metrics use hummingbird_workqueue_.

Generic operational metrics

Metric Type Labels Description
hummingbird_http_requests_total Counter method, path, status Total HTTP requests
hummingbird_http_request_duration_seconds Histogram method, path HTTP request latency
hummingbird_sqs_messages_received_total Counter — SQS messages pulled
hummingbird_sqs_messages_processed_total Counter status SQS messages processed

Service-specific metrics

Metric Type Labels Description
hummingbird_workqueue_work_items_created_total Counter item_type Work items submitted via API
hummingbird_workqueue_work_items_completed_total Counter item_type, result Action executions completed (success/failed)
hummingbird_workqueue_work_items_retried_total Counter item_type Work items retried via API
hummingbird_workqueue_action_duration_seconds Histogram action Executor action duration
hummingbird_workqueue_active_processing_threads Gauge — Running work item threads
hummingbird_workqueue_pending_events_inserted_total Counter — Pending events inserted into DB

UUID path segments in HTTP metrics are normalized to /{id} to limit label cardinality. The /healthz and /metrics paths are excluded from HTTP metrics.

CLI

export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)

# Update an existing file (local_path:repo_path)
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --file /path/to/sources.spec:sources.spec \
  --commit-message "Sync sources from upstream" \
  --source-branch mr-service/sync-1 --title "Sync sources" \
  --idempotency-key kernel-sync-1

# Create a new file
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --create /path/to/new-file.txt:new-file.txt \
  --commit-message "Add new file" \
  --source-branch mr-service/add-1 --title "Add file" \
  --idempotency-key kernel-add-1

# Check status
workqueue-service status --id <uuid>

# List active MR items
workqueue-service status --phase active

# Show identity and permissions
workqueue-service whoami

# Raw JSON output
workqueue-service whoami --json

--url overrides WORKQUEUE_SERVICE_URL. --no-auth skips all credential loading and sends unauthenticated requests (like aws --no-sign-request). Useful with whoami to verify anonymous access.

CLI authentication

The CLI resolves credentials in this order:

  1. WORKQUEUE_SERVICE_TOKEN env var — used directly as a Bearer token. Recommended for CI (OIDC/projected SA tokens).
  2. Cached token — if a valid token exists in the per-server cache, it is reused automatically.
  3. OAuth/Kerberos login — when neither env var nor cache is available, the CLI performs a synchronous headless SPNEGO login against the oauth-proxy, exchanges the session cookie for an OCP Bearer token via POST /oauth/token, and caches it for subsequent calls. Requires requests-gssapi and a valid Kerberos ticket (kinit).

Tokens are cached per-server under $XDG_CACHE_HOME/workqueue-service/tokens/ (defaults to ~/.cache/workqueue-service/tokens/) with 0600 permissions and a 20-hour TTL. If a cached token is rejected (HTTP 401), the CLI invalidates the cache and retries once with a fresh OAuth login.

Configuration

Config file

The service reads its configuration from a YAML file at startup. The path defaults to /etc/workqueue/config.yml and can be overridden with WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing or invalid (fail-closed).

authorization:
  rules:
    # Any in-cluster SA with the right audience can read
    - issuer: k8s
      claims:
        aud: workqueue-service
      permissions: [read]

    # Human users -- OCP group membership
    - groups:
        - konflux-hummingbird-admin-access
      permissions: [read, submit, retry]

    # Specific K8s SA for CI automation
    - issuer: k8s
      claims:
        aud: workqueue-service
        sub: system:serviceaccount:hummingbird--internal:rpms-ci
      permissions: [submit, retry]
      item_types: [mr]

    # GitLab CI OIDC
    - issuer: https://gitlab.com
      claims:
        aud: workqueue-service
        project_path: redhat/hummingbird/rpms
      permissions: [read, submit, retry]
      item_types: [mr]

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

Authorization rules

Each rule has either groups (OCP group membership from users/~ response) or issuer+claims (JWT bearer), not both.

  • issuer determines which JWKS keys validate the token signature. Use k8s as shorthand for the auto-detected cluster OIDC issuer.
  • claims matches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numeric project_id, boolean ref_protected).
  • permissions grants read, retry, and/or submit.
  • item_types optionally restricts the rule to specific work item types. Omit to apply to all types.
  • Every issuer rule must include aud in claims to prevent cross-service token reuse.

Handled types

handled_types declares which work item types this instance serves. Only listed types get their routers and executors registered. The processing sweep only picks up items matching these types. Empty or omitted = handle nothing.

Each type can have per-type operational config:

Type Field Description
mr gitlab_url GitLab instance URL (default: https://gitlab.com)
mr dashboard_base_url Dashboard URL for note creation (when unset, notes are not created)
mr projects Per-project/group token and URL configuration

Per-project token configuration

The projects map under handled_types.mr controls which GitLab API token the executor uses for each target project. Keys are GitLab project paths or group prefixes. Every project submitted to the service must match an entry – there is no default fallback token.

Matching order:

  1. Exact match on the full project path
  2. Longest group-prefix match on a / boundary (e.g. redhat/rhel/src matches redhat/rhel/src/kernel)

Each entry requires token_env (the name of the environment variable holding the GitLab API token) and optionally overrides gitlab_url:

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
        gitlab_url: https://gitlab.example.com
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

At startup, the service validates that all referenced token_env variables exist in the process environment and refuses to start if any are missing.

Adding a new issuer

Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:

- issuer: https://token.actions.githubusercontent.com
  claims:
    aud: workqueue-service
    repository: org/repo
  permissions: [submit]
  types: [mr]

Environment variables

Variable Default Description
DATABASE_URL — PostgreSQL connection URL (required)
SENTRY_DSN — Sentry DSN for error reporting
WORKQUEUE_SERVICE_CONFIG /etc/workqueue/config.yml Config file path
WORKQUEUE_SERVICE_LOCAL_AUTH_USER — Dev-only auth bypass (warns at startup)
WORKQUEUE_SERVICE_URL — CLI: service URL
WORKQUEUE_SERVICE_TOKEN — CLI: Bearer token
(per-project token_env) — GitLab API token(s) for MR executor (set in config projects)
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL — SQS queue URL for events (when unset, event consumer is disabled)
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET — S3 bucket for audit log (production)
WORKQUEUE_SERVICE_AUDIT_S3_PREFIX workqueue-service-audit Key prefix within the audit S3 bucket
AWS_ACCESS_KEY_ID — AWS credentials for S3 audit archival
AWS_SECRET_ACCESS_KEY — AWS credentials for S3 audit archival
AWS_DEFAULT_REGION — AWS region for S3 audit archival

Development

Local Development

cd workqueue-service
./dev.sh db-start    # Start local PostgreSQL
./dev.sh db-migrate  # Run Alembic migrations
./dev.sh start       # Start the service
./dev.sh db-shell    # PostgreSQL shell
./dev.sh db-stop     # Stop PostgreSQL
./dev.sh db-reset    # Stop and delete volume

Running Tests

cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests

Tests use testcontainers[postgres] locally (auto-detects Podman) or a CI-provided PostgreSQL sidecar via DATABASE_URL.

License

This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.

4.5.31 - Work Queue Service Design

Architectural design document for workqueue-service. Covers the reasoning behind every major design choice so that future changes can be made safely. For operational usage, see Work Queue Service.

1. Design Philosophy

Four principles shaped the service’s architecture:

Security boundary. The service sits between untrusted callers (AI agents, CI jobs, automation scripts) and trusted write operations (GitLab pushes, MR creation). All changes are inspectable structured data, not opaque blobs. Path blocklists and action validation run before any GitLab write.

Type-agnostic queue, type-specific behaviour. The processing loop, claiming, heartbeat, and lease management are generic. Type-specific logic (MR creation, CVE remediation) lives in pluggable subpackages that register deciders and executors at import time. Adding a new work item type requires no changes to the generic layer.

Single pending action. Only one action per work item at a time. After each action, the decider re-evaluates with current state. This avoids queuing actions that assume a future state which may not hold.

Phase + dimensions, not a linear state machine. The coarse lifecycle (active/completed/failed/cancelled) is separate from independent readiness signals (CI, approval, Konflux build). Dimensions can regress (force-push resets CI), which a linear state machine cannot express.

2. Architecture

Callers submit work items via REST API. The ProcessingManager drains pending actions in per-item threads, dispatching to type-specific executors. Currently the only executor creates GitLab MRs via the Commits API.

flowchart LR
    callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
    API --> DB[("PostgreSQL")]
    Processing["ProcessingManager"] --> DB
    Processing --> GL["GitLab\nCommits API"]
    API -.-> Processing

3. Design Decisions Record

DDR-1: Phase + dimensions vs linear state machine

Decision: Replace the 12-state linear state machine with a phase + dimensions model.

Context: The initial design used a single state column: pending → in_progress → created → mr_ci → approved → merged → post_merge_ci → building → releasing → completed. This broke because post-creation concerns (CI, approval, Konflux build) are independent — CI can pass before approval, approval can be revoked, force-push resets CI but not approval.

Alternatives: (a) Composite states (ci_passed_approval_pending) — exponential explosion. (b) Multiple boolean columns — rigid, schema migration for each new signal.

Rationale: Phase tracks coarse lifecycle; independent dimensions in a separate table track readiness signals with variant support for multi-instance cases (per-component Konflux builds). Dimensions can regress naturally. New dimension types need no schema migration.

DDR-2: Four generic phases (active/completed/failed/cancelled)

Decision: Narrow from 6 MR-specific phases to 4 generic phases.

Context: The intermediate design had pending, active, merged, completed, failed, closed — mixing generic concepts with MR-specific milestones.

Alternatives: Keep MR-specific phases and add more for each type.

Rationale: Phase answers “is this item done, and how?” Type-specific milestones (MR merged, CI passed) are tracked via dimensions and detail-table columns. Items start as active — there is no pending phase because the decider immediately assigns an action at creation time.

DDR-3: Dimensions in a separate table with variant support

Decision: Store dimensions in work_item_statuses table, not as columns on work_items.

Context: Initially considered a column per dimension (ci_status, approval_status, etc.).

Alternatives: (a) Column per dimension — schema migration for each new one. (b) JSONB column — flexible but loses CHECK constraints and complicates queries. (c) Separate table with composite PK (work_item_id, dimension, variant).

Rationale: Option (c). New dimensions need no migration. Variants handle multi-instance signals (multiple Konflux components). Each work item only has rows for dimensions that apply to it. The “are all dimensions satisfied?” query is a natural GROUP BY/HAVING.

DDR-4: Decider as pure function, not state machine transitions

Decision: The decider is a pure function decide(item) → action | None registered per type. It only determines the next pending_action.

Context: The original design had a TRANSITIONS dict mapping state → frozenset[state] with validate_transition().

Alternatives: (a) Transition table — rigid, can’t express “depends on which dimensions passed”. (b) Decider function — flexible, testable.

Rationale: The decider is trivially testable (given state, assert action). State transitions come from executors (they apply phase changes directly) and events (they update dimensions). The decider just picks the next action based on current state.

DDR-5: Single pending action, not a queue

Decision: One pending_action per work item at a time.

Context: Considered queuing multiple actions (“merge, then trigger Konflux build”).

Alternatives: Action queue table per work item.

Rationale: Queued actions assume future state that may not hold. If “merge” fails, the queued “trigger Konflux build” is nonsense. After each action, the decider re-evaluates with current state. The attempts counter tracks consecutive failures, not total actions.

DDR-6: Claiming via leased_until (reusable)

Decision: Time-based lease (leased_until column) for work item claiming, reusable across all phases.

Context: The original design used state = 'in_progress' as a one-time claiming mechanism.

Alternatives: (a) claimed_by + claimed_at columns — identifies the worker but adds complexity. (b) state = 'in_progress' — ties claiming to phase, not reusable. (c) leased_until timestamp — simple, reusable, self-recovering.

Rationale: Option (c). Items can be reclaimed repeatedly throughout their lifecycle. Expired leases are automatically reclaimable by other workers. The heartbeat context manager extends the lease during processing.

DDR-7: Generic + detail tables (joined table inheritance)

Decision: Split work_items into generic queue table + per-type detail tables (mr_details, future cve_details).

Context: The initial design had all MR-specific columns on the work_items table.

Alternatives: (a) Single table with nullable type-specific columns — sparse, doesn’t enforce type constraints. (b) JSONB payload column — flexible but loses type safety. (c) Joined table inheritance.

Rationale: Option (c). SQLAlchemy’s polymorphic_load="selectin" automatically loads the correct subclass. New types add their own detail table — no changes to the generic layer. Type-specific columns have proper types and constraints.

DDR-8: Type-dispatched decider/executor registries

Decision: Registry pattern for deciders and executors. Each type registers at import time via decider.register("mr", _mr_decide).

Context: Needed a way to dispatch to type-specific logic without the generic layer knowing about types.

Alternatives: (a) if/elif chains in the processing loop. (b) Class-based dispatch (strategy pattern). (c) Registry functions.

Rationale: Option (c). Simple, explicit, no class hierarchy needed. The mr/__init__.py import triggers registration. New types just add their own register() calls.

DDR-9: Structured file actions vs git patch_data

Decision: Replace opaque patch_data (BYTEA, git format-patch output) with structured file_actions (JSONB) and commit_message.

Context: The MR service sits on the security boundary — it writes to GitLab repos on behalf of untrusted callers. HUM-851 and the AI Investigation design rules require pre-commit inspection (path blocklists, content validation).

Alternatives: (a) Keep patch_data, clone repo, apply patch, then inspect diff — inspection happens after creating a working tree with credentials. (b) Parse patches with unidiff library — gives diffs not full content, still needs original files. (c) Structured file actions.

Rationale: Option (c). Inspection is a pure function over data the service already has — no clone, no git binary, no temp dirs. Path blocklists and action validation run before any API call. The file_actions JSONB column is also a queryable audit record.

DDR-10: GitLab Commits API vs git subprocess

Decision: Use the GitLab Commits API for branch creation and file commits. Delete git_ops.py.

Context: The original executor shelled out to git (clone, am, push), requiring: git binary in the container, subprocess timeout handling, GIT_ASKPASS credential management, temp directory lifecycle.

Alternatives: (a) Keep git subprocess with GIT_ASKPASS and timeout. (b) GitLab Commits API.

Rationale: Option (b). gitlab_sync.py in the same repo already proves the pattern. The Commits API’s start_branch parameter creates the branch automatically. No subprocess, no credentials in URLs, no temp dirs. Callers have full file contents (not patches), so git am semantics aren’t needed.

DDR-11: OIDC issuer from discovery, not hardcoded

Decision: Read the OIDC issuer from the /.well-known/openid-configuration discovery response. Rewrite the JWKS URI to use kubernetes.default.svc.

Context: The initial implementation hardcoded issuer = "https://kubernetes.default.svc". This failed on EaaS clusters where --service-account-issuer is https://oidc.op1.openshiftapps.com/.... The JWKS URI from discovery pointed to an IP:6443 that was unreachable from pods.

Alternatives: (a) Environment variable override (KUBE_OIDC_ISSUER). (b) Read from discovery.

Rationale: Option (b). No configuration needed — works on any cluster automatically. The JWKS fetch uses the path from discovery but rewrites the host to kubernetes.default.svc (always reachable on port 443). SA bearer token authenticates both requests (OpenShift blocks anonymous JWKS access).

DDR-12: item_type not type

Decision: Name the discriminator column item_type everywhere (DB, ORM, API, schemas).

Context: type is a Python builtin. Using it as a function parameter triggers ruff A002.

Alternatives: (a) Use type with alias in different contexts. (b) Rename to item_type everywhere.

Rationale: Option (b). Clean rename, no aliases, no special casing. Consistent at every layer.

DDR-13: source field removed

Decision: Remove the source column from mr_details. The caller_identity field (set automatically from authentication) serves the same purpose.

Context: source was a user-declared label (“rpms-ci”, “dependency-updater”) while caller_identity is the authenticated identity. Having both was redundant.

Alternatives: Keep both for cases where a shared SA submits on behalf of different logical sources.

Rationale: Removed until needed. caller_identity captures who submitted. If per-source filtering is needed later, it can be re-introduced as a field on MRWorkItemCreate.

DDR-14: attempts counts consecutive failures, not total actions

Decision: attempts increments only on executor failure and resets to 0 on success.

Context: The initial implementation incremented attempts before every action execution. A work item chaining 4 actions (create_mr → merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the third action.

Alternatives: (a) Separate total_actions counter. (b) Only count failures.

Rationale: Option (b). attempts answers “how many consecutive times has this action failed?” — the signal for escalation. Total actions executed is an audit concern, not a processing concern.

DDR-15: Executor applies state directly; decider only picks next action

Decision: Executors directly modify item.phase, item.mr_iid, etc. The decider only returns the next pending_action.

Context: Initially the processing loop set the phase based on ExecutorResult.success. This mixed two concerns.

Alternatives: (a) Processing loop sets phase. (b) Decider sets phase. (c) Executor sets phase.

Rationale: Option (c). The executor knows what happened (“I created an MR, so phase stays active”). Events that update dimensions also directly set state. The pattern is the same: something happens → state is updated → decider evaluates for the next action. Symmetry with event-driven dimension updates.

DDR-16: Items stay active after MR creation

Decision: After the create_mr executor succeeds, the item remains in active phase with pending_action = None.

Context: After MR creation, the decider returns None (no action). The item sits in active indefinitely.

Alternatives: (a) Set phase to completed after MR creation. (b) Keep active for future event-driven lifecycle tracking.

Rationale: Option (b). The active phase covers the entire lifecycle — from submission through CI, approval, merge, and post-merge tracking. The decider returns no actions until event-driven dimension updates (future SQS consumer) trigger the next step.

DDR-17: Audit log to S3, not database

Decision: Service-level audit (state transitions, decider decisions, actions taken) goes to S3 as immutable objects.

Context: Raw webhook events are already archived by sns-s3-archiver. The service needs its own audit trail for state transitions and decider evaluations.

Alternatives: (a) Database table (work_item_events). (b) S3 immutable objects. (c) Both.

Rationale: Option (b). S3 is immutable and cheap. The operational DB stays lean. Raw events are already in S3 via sns-s3-archiver; service-level audit captures the service’s reaction to those events.

DDR-18: Agent direction — dumber, not smarter

Decision: The hummingbird-agent should become a pure compute engine. The work queue service is the MR lifecycle orchestrator. The agent never calls the work queue service; the work queue service calls the agent.

Context: HUM-857 proposed adding create_merge_request as a new action in the agent’s actions.py. This was the wrong direction.

Alternatives: (a) Agent gains more actions (push, merge, MR management). (b) Agent becomes stateless compute; work queue service orchestrates.

Rationale: Option (b). The agent keeps LLM loop, tool registry, and sandbox — nothing else. Event routing, rate limiting, placeholder notes, result posting, session management, and all GitLab writes move to the work queue service. HUM-857 closed as Obsolete.

DDR-19: Shared database

Decision: The work queue service uses the same events PostgreSQL database as hummingbird-status and hummingbird-dashboard.

Context: Cross-service joins (e.g. work_items JOIN gitlab_merge_requests JOIN pipelineruns) are useful for the dashboard.

Alternatives: Separate database per service.

Rationale: Shared DB with service-specific Alembic migration chains (alembic_version_mr_service). Each service manages its own tables independently. The migration chain is isolated so one service can evolve its schema without affecting others.

DDR-20: Pydantic config over cki-lib/jsonschema

Decision: Use Pydantic for config validation. Load config from a YAML file at startup.

Context: Needed structured config for authorization rules and per-type operational settings. Alternatives: cki-lib config helpers, jsonschema, or plain dicts.

Alternatives: (a) cki-lib — adds a dependency, the service stays self-contained without it. (b) jsonschema — separate schema file, no type-safe access. (c) Pydantic — already a dependency (API schemas).

Rationale: Option (c). Pydantic provides type-safe attribute access, defaults, cross-field validators (e.g. mutual exclusivity of groups vs issuer+claims), and optional JSON Schema export. No new dependencies.

DDR-21: Claim-based rules (Vault bound_claims pattern)

Decision: Authorization rules match on arbitrary JWT claims, not just sub. All claim conditions AND; list values OR.

Context: Studied JWT trust/authz config from AWS API Gateway, HashiCorp Vault, Kubernetes Structured Auth, Istio, Traefik Hub, and oauth2-proxy. All support matching on arbitrary claims.

Alternatives: (a) Subject allowlist — rigid, requires exact sub values. (b) Claim-based matching — flexible, supports project_path, ref, groups, etc.

Rationale: Option (b). Vault’s bound_claims pattern is proven and simple. Values are coerced to lowercase strings before comparison to handle int/string mismatches (GitLab’s numeric project_id) and bool/string mismatches (ref_protected: true).

DDR-22: Issuer per rule, audience as claim

Decision: issuer is the only special field on a token rule — it determines which JWKS keys to use. All other JWT conditions (including aud) are standard claim matches in the claims dict.

Context: Considered a separate jwt.issuers section and/or treating aud as a first-class field alongside issuer.

Alternatives: (a) Global jwt section with trusted issuers + separate rules section. (b) aud as a dedicated field on each rule. (c) Issuer per rule, audience as a claim.

Rationale: Option (c). Keeps each rule self-contained and self-documenting. The Pydantic validator enforces that every issuer rule includes aud in claims to prevent cross-service token reuse.

DDR-23: Unified rule model for groups and tokens

Decision: A single rules list where every rule maps a remote identity (groups or issuer+claims) to local permissions.

Context: The original design had admin_groups/retry_groups (local permission → remote groups) alongside rules (remote identity → local permissions) — opposite mapping directions.

Alternatives: Keep separate group and token rule formats.

Rationale: Unified model. Every rule follows the same structure: remote identity → local permissions. Pydantic cross-field validators enforce mutual exclusivity.

DDR-24: Bearer-only auth with users/~ self-authentication

Decision: All API auth uses Authorization: Bearer tokens. Opaque OCP tokens (sha256~...) are validated by calling the OCP users/~ API with the user’s own token (self-authenticating).

Context: The oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. The TokenReview API would be cleaner but requires system:auth-delegator ClusterRoleBinding — a cluster-scoped permission not available on managed OCP clusters.

Alternatives: (a) Trust X-Forwarded-User from the proxy on all paths — injection risk on bypassed paths. (b) TokenReview API — requires cluster RBAC. (c) users/~ self-authentication.

Rationale: Option (c). No special RBAC needed. The users/~ endpoint returns username and groups. Results are cached with a thread-safe TTL cache (5 minutes). The /me endpoint bridges OAuth cookie sessions to Bearer tokens.

DDR-25: handled_types as explicit opt-in

Decision: handled_types declares which work item types this instance serves. Empty means handle nothing.

Context: Different deployments may serve different types from the same container image. Types carry per-type config (e.g. gitlab_url for MR).

Alternatives: (a) Handle all types by default. (b) Explicit opt-in.

Rationale: Option (b). Prevents accidental catch-all deployments. The sweep returns early when handled_types is empty — no items stolen from other instances in a shared-database deployment.

DDR-26: k8s shorthand skipped when not in-cluster

Decision: Rules with issuer: k8s are silently skipped when the in-cluster CA bundle is absent.

Context: The K8s OIDC issuer URL varies by cluster and is auto-detected at runtime. Local development doesn’t have a cluster.

Alternatives: (a) Error on startup if k8s issuer can’t be resolved. (b) Skip silently.

Rationale: Option (b). Allows using a production config file for local development and testing without startup errors.

Decision: The oauth-proxy sidecar handles only the OAuth login flow. API endpoints bypass the proxy. A proxy-protected /me endpoint bridges cookie sessions to Bearer tokens. The proxy must be configured with --cookie-refresh=1h to prevent stale-token failures.

Context: The OCP oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. Protected paths validate Bearer tokens via TokenReview (requires cluster RBAC we cannot obtain) or reject non-OCP tokens. Bypassed paths pass all Bearer tokens through but ignore cookies. Additionally, the OCP access token embedded in the proxy’s session cookie has its own expiry (default 24h), independent of the cookie lifetime. Without --cookie-refresh, the cookie outlives the token, causing the proxy to forward expired credentials while still reporting the user as authenticated — the “logged in but not authorized” failure observed in the hummingbird-dashboard deployment.

Alternatives: (a) All paths protected — blocks non-OCP Bearer tokens. (b) --openshift-delegate-urls — requires system:auth-delegator ClusterRoleBinding. (c) App reads proxy cookies directly — tight coupling to proxy internals. (d) Login-only proxy with /me bridge.

Rationale: Option (d). The /me endpoint is the only proxy-protected API path. It reads X-Forwarded-Access-Token (set by the proxy from the cookie session) and returns the OCP access token to the caller. Subsequent API calls use Authorization: Bearer sha256~... on bypassed paths, where the app validates via the OCP users/~ API. --cookie-refresh=1h keeps the embedded token fresh. TTL-bounded caching (5 minutes) on users/~ results ensures expired-token failures are never permanent (unlike @functools.cache which caches failures indefinitely).

DDR-28: Audit log with logging + optional S3 archival

Decision: Always log audit entries via Python logging at INFO level, and optionally archive to S3 as gzipped JSON objects when WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set.

Context: Needed an immutable record of state transitions, action outcomes, and auth decisions for compliance and debugging.

Alternatives: (a) Database table — bloats the operational DB with a different concern. (b) S3-only — no visibility without querying Athena; loses events if S3 is unreachable.

Why logging + S3: Logging to stdout is zero-config and integrates with existing container log collection for real-time visibility. S3 provides the durable, queryable archive (Athena) for compliance.

Key design rules:

  • Audit failures never propagate — write_audit_entry() catches all exceptions and logs them, but never raises.
  • S3 keys use YYYY/MM/DD/HH/MM/ prefix for Athena partitioning.
  • ContentEncoding: gzip set so S3 API consumers get transparent decompression.

Field conventions (follow these when adding new audit calls):

  • actor: caller identity for API/auth events, thread name for worker events, "system" for decider/system-triggered transitions.
  • old_state: state before the change. Include phase and pending_action when both are relevant. Omit on creation (no prior state) and on auth events (no work item state).
  • new_state: state after the change. Same keys as old_state. Omit fields that are None (they are filtered by to_dict()).
  • trigger: what caused this event. Always include source (one of "api", "worker", "decider"). Add action for deliberate operations, reason for error/system transitions, error for failure details.
  • metadata: context orthogonal to the state transition — request info on auth events, type-specific data on submit. Not for duplicating state already in old_state/new_state.

DDR-29: Events processed in work-item threads, not consumer threads

Decision: The SQS consumer thread stores events in the pending_events table. Existing per-work-item threads drain and process them.

Context: The ProcessingManager already runs one thread per work item with lease protection. Processing events in the same thread means events and actions are naturally serialized per work item.

Alternatives: (a) Consumer threads handle events directly — concurrency conflicts with action threads. (b) FIFO queue + in-memory dispatch — receipt handle management, visibility timeout extensions, message loss on crash.

Rationale: Option (c). Events join the existing thread structure. The decider runs after each event, determining follow-up actions. The single-thread-per-item invariant is preserved.

DDR-30: pending_events table as durable event store

Decision: Use a PostgreSQL table (pending_events) instead of FIFO queues or in-memory queues for event delivery to work-item threads.

Context: Since work-item threads do the processing, the SQS consumer just needs to deliver events to them durably.

Alternatives: (a) FIFO queue + in-memory dispatch — complex EventBroker, receipt handle management, message loss on crash. (b) FIFO queue + direct handling — concurrency conflicts.

Rationale: The DB provides durability (crash recovery), serialization (single thread per work item), and ordering (last_event_at guard). No FIFO queue, router thread, MessageGroupId computation, in-memory queues, or visibility timeout management needed.

DDR-31: SQS delete after INSERT, not after full processing

Decision: Delete the SQS message after INSERT+COMMIT into pending_events, not after full event processing.

Context: The hummingbird-agent deletes after handling (no DB intermediary). The workqueue-service can delete after INSERT+COMMIT because the event is crash-recoverable in PostgreSQL.

Alternatives: Delete after full processing — requires visibility timeout extensions, receipt handle passing, heartbeat management.

Rationale: After INSERT+COMMIT the event is durable in PostgreSQL. Duplicate SQS delivery may INSERT the same event twice; the last_event_at guard makes processing idempotent.

DDR-32: Resolver filters by phase, consumer stays read-only

Decision: The resolver filters by phase = 'active' on the MRWorkItem query. SQLAlchemy’s joined table inheritance handles the work_items table access implicitly. The consumer does SELECT + INSERT only, no UPDATE, no row lock contention.

Context: Only active items accept event-driven transitions (see DDR-37). Filtering at resolve time avoids unnecessary INSERTs into pending_events, thread spawns, and no-op processing for terminal items.

Alternatives: (a) Resolve without phase filter, guard only at processing time — wastes work for terminal items. (b) Consumer checks phase and applies transitions directly — contention with work-item threads.

Rationale: The phase filter adds negligible cost (PK lookup on work_items, typically one row). The consumer remains read-only. Business logic (phase transitions, decider, audit) is applied by the generic handle_event layer in the work-item thread.

DDR-33: Raw events stored, interpretation separated from application

Decision: The pending_events table stores the raw event JSON body and event type. Type-specific handlers interpret events (read-only on the work item), the generic layer applies the result.

Context: Event processing has three concerns: interpretation (what does this event mean?), application (phase transition, decider, audit), and routing (which work items?). Mixing them makes handlers hard to test and inconsistent across types.

Alternatives: (a) Handlers apply all changes directly — mixes interpretation with mutation, each handler reimplements decider/audit boilerplate. (b) Pre-process events in the consumer — consumer needs type-specific knowledge.

Rationale: Handlers return an EventResult dataclass (phase, event_ts, trigger metadata). The generic handle_event applies the phase transition via update_work_item_from_event, runs the decider, and emits audit entries. Handlers have read-only access to the work item and session (e.g. to check existing dimensions or detail fields) but EventResult is their only output — they never mutate the work item directly. This ensures decider/audit logic is consistent across all item types. See DDR-38 for the dimension delivery extension.

DDR-34: Per-work-item advisory lock for insert/drain coordination

Decision: Use pg_advisory_xact_lock(key) (transaction-scoped) keyed by work_item_id.int & 0x7FFFFFFFFFFFFFFF to coordinate the consumer’s INSERT with the work-item thread’s exit decision.

Context: Without coordination, the consumer can INSERT an event between the thread’s final drain check and its exit, orphaning the event until the next sweep (~5 minutes).

Alternatives: (a) Accept the race — events wait up to 5 minutes. (b) Global lock — unnecessary contention.

Rationale: Per-work-item advisory lock closes the gap. Either the INSERT commits before the thread’s re-drain (thread sees it and continues), or the INSERT commits after the thread exits (notify() spawns a new thread). The hash space is 2^63, making collisions negligible. The sweep is the safety net for the astronomically unlikely collision case.

DDR-35: Sweep for startup recovery

Decision: Extend the periodic sweep to also check for work items with unprocessed pending_events rows.

Context: Events that arrive while the service is stopped, or that survive a mid-processing crash, must be processed on restart.

Alternatives: Separate startup-only scan.

Rationale: Unifying startup recovery with the periodic sweep keeps a single code path. The sweep runs immediately on manager.start(), catching events from downtime. It also serves as a safety net behind the advisory lock.

DDR-36: Consumer lifecycle and shutdown ordering

Decision: The SQS consumer is a single daemon thread started from the FastAPI lifespan. Shutdown stops the consumer before work-item threads.

Context: If work-item threads stop while the consumer is still INSERTing, events could be orphaned. The consumer must stop first.

Alternatives: Non-daemon thread with explicit join.

Rationale: Single thread is sufficient for low event volume (MR state changes). Daemon thread ensures process exit even if the thread hangs. Shutdown ordering: (1) set shutdown_event, (2) join consumer thread (timeout=30s), (3) call manager.stop(). No consumer when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is unset.

DDR-37: Terminal phase semantics — only active items accept events

Decision: Only items in phase = 'active' accept event-driven phase transitions. completed, failed, and cancelled are terminal phases — recovery requires explicit action via the retry API.

Context: Initially only failed was guarded (phase != 'failed'). completed and cancelled items could be modified by events: a closed event could transition a completed item to cancelled, and an opened event could reactivate a cancelled item.

Alternatives: (a) Guard only failed — inconsistent, completed items are accidentally mutable. (b) Growing exclusion list (NOT IN ('failed', 'completed')) — fragile if new phases are added. (c) Positive match phase = 'active'.

Rationale: Option (c). A single positive match is clearer and future-proof. If a work item was cancelled (MR closed) or completed (MR merged), only deliberate human action (retry API) should reactivate it — not an automated event. The retry API accepts both failed and cancelled items. The resolver also filters by phase = 'active' at query time (DDR-32) to avoid unnecessary INSERT/processing for terminal items.

DDR-38: Dimension updates via EventResult

Decision: Event handlers return dimension upserts as part of EventResult. The generic handle_event layer applies them.

Context: DDR-33 established that handlers return structured results and the generic layer applies them. Dimensions (DDR-1, DDR-3) need the same pattern — handlers identify which dimensions changed, the generic layer upserts them and emits audit entries.

Alternatives: (a) Handlers call crud.upsert_dimension() directly — breaks the read-only handler contract from DDR-33. (b) Separate dimension-specific consumer — duplicates the resolver/handler infrastructure.

Rationale: EventResult.dimensions is a list of DimensionUpdate dataclasses. After phase transition logic, handle_event iterates over dimensions and calls crud.upsert_dimension() for each, emitting an AuditEntry with event_type="dimension_change" using the existing dimensions field. Dimension upserts run even when the phase transition is stale (the early return is refactored to skip only the phase/decider block). This ensures dimension-only events (e.g. note events with no phase change) are still applied.

4. Future Work

Event-driven dimension updates

The dimension mechanism is implemented with dashboard_link as the first dimension (DDR-38). MR state events drive phase transitions (merge -> completed, close -> cancelled). Remaining dimensions: CI pipeline events updating ci, MR approval events updating approval, and Konflux PipelineRun events updating konflux_build/konflux_release. Each follows the same EventResult.dimensions pattern.

Non-managed MR tracking

Design how the service handles MRs it did not create (MRs without the managed-by::workqueue label). Requires subscribing to note events so the service can discover dashboard notes posted by the Lambda and adopt them as dashboard_link dimensions. Prerequisite for retiring the Lambda.

Status rendering

An executor that edits the dashboard note (using the note_id from the dashboard_link dimension) with aggregated dimension status (CI, approval, Konflux).

CVE work item type

Add cve/ subpackage with CveWorkItem, CVE-specific decider (analyze → label → create advisory MR → check VEX → close), and CVE-specific executors. No changes to the generic layer.

Content validation and size limits

Add content-level inspection to the pre-commit checks: maximum file size, binary detection, and content pattern matching. Currently only path blocklists and action validation are enforced.

Reconciliation sweep

Periodic job queries GitLab API for items in active phase, compares actual state with dimension rows, corrects drift from missed events.

Service rename

The service was renamed from hummingbird-mr-service to workqueue-service to reflect its generalized purpose. A further rename to hummingbird-work-queue or similar is deferred until a second work item type is implemented.

4.5.32 - Service metrics

Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes Metrics API CPU/memory gauges, OpenShift cluster resource quota usage, PVC filesystem usage, and GitLab CI/CD schedule status.

Features

  • Explicit collectors - nothing runs unless listed in METRICS_CONFIG
  • Kubernetes Metrics API - per-container CPU and memory gauges
  • Cluster resource quotas - AppliedClusterResourceQuota usage and limits
  • AWS Cost Explorer - yesterday’s NetAmortizedCost by team and service
  • Volume filesystem usage - PVC disk usage via shutil.disk_usage()
  • GitLab CI/CD schedules - pipeline schedule health via GraphQL
  • Prometheus - HTTP metrics on port 9090

Configuration

Collectors are off unless listed in METRICS_CONFIG:

enabled: [kubernetes]

Unknown names fail startup.

Variable Purpose
METRICS_CONFIG YAML with enabled collector list (required)
KUBERNETES_CONFIG YAML with namespaces for the kubernetes collector
VOLUMES_CONFIG YAML with volumes list for the volume collector
GITLAB_URL GitLab instance URL (default https://gitlab.com)
GITLAB_TOKEN GitLab API token with read_api scope (required for gitlab collector)
GITLAB_CONFIG YAML with namespaces list for the gitlab collector
METRICS_PORT HTTP port (default 9090)
SENTRY_DSN Optional Sentry DSN
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY Cost Explorer IAM user (hub)
AWS_DEFAULT_REGION Must be us-east-1 for Cost Explorer

Kubernetes Metrics API

hummingbird_k8s_resource_usage{namespace,pod,container,resource,app} with resource=cpu|memory and app sourced from the pod’s app.kubernetes.io/name label (empty string if absent). Instant gauges from /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods. This is not cAdvisor: managed OpenShift does not grant nodes/metrics or cluster-monitoring-view, so kubelet /metrics/cadvisor and platform Prometheus federation are not available.

If KUBERNETES_CONFIG.namespaces is unset, the collector uses the pod’s ServiceAccount namespace.

Cluster resource quotas

The kubernetes collector also reads namespace-scoped AppliedClusterResourceQuota objects from the quota.openshift.io/v1 API and exposes two gauges that mirror the openshift-state-metrics schema with a hummingbird_ prefix:

  • hummingbird_clusterresourcequota_usage{name,resource,type} – cluster-wide totals from status.total (type=hard|used)
  • hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type} – per-namespace breakdown from status.namespaces (type=hard|used)

resource values match Kubernetes quantity names: cpu, memory, requests.cpu, limits.memory, pods, etc. Quantities are converted to base units (cores, bytes, count).

The same ACRQ may appear in multiple namespaces; the global metric is idempotent across duplicates. Stale series are removed when ACRQs disappear.

Requires list on appliedclusterresourcequotas in quota.openshift.io (namespace-scoped Role – no ClusterRole needed).

Volume filesystem usage

The volume collector reports disk usage for PVC-backed volumes. It is designed to run as a sidecar on pods that mount PVCs, since kubelet_volume_stats_* metrics are inaccessible on managed clusters (no ClusterRole for nodes/metrics) and RWO volumes cannot be mounted cross-namespace.

Configure via VOLUMES_CONFIG:

VOLUMES_CONFIG: |
  volumes:
    - {name: my-postgres, path: /data}

Metrics (collected every 5 minutes):

  • hummingbird_volume_usage_bytes{name} – used bytes
  • hummingbird_volume_total_bytes{name} – total capacity
  • hummingbird_volume_available_bytes{name} – available bytes

Deploy with enabled: [volume] and a non-default METRICS_PORT (e.g. 9091) to avoid port conflicts when running as a sidecar alongside other containers. See the hummingbird-status StatefulSet in the infrastructure repo for an example.

GitLab CI/CD schedules

The gitlab collector monitors the health of pipeline schedules across all projects in configured GitLab namespaces. It queries the GitLab GraphQL API once per hour, paginating through projects to fetch each schedule’s last pipeline status.

Configure via GITLAB_CONFIG:

GITLAB_CONFIG: |
  namespaces:
    - redhat/hummingbird

The GITLAB_TOKEN environment variable must contain a token with read_api scope that has access to the monitored namespaces. Keep it in an OCP Secret (not the ConfigMap).

Metric (collected every hour):

  • hummingbird_gitlab_schedule_status{instance,project,active,description,cron} – Enum reporting the last pipeline status of each CI/CD schedule

States: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled, unknown. Schedules with no last pipeline report unknown.

Stale series are removed automatically when schedules disappear.

AWS costs

hummingbird_aws_cost{group,type,key} is yesterday’s NetAmortizedCost. Queries run at process start and every 24 hours.

group=all is the shared account by app-code and SERVICE. group=hummingbird filters app-code=RPRM-001 and groups by SERVICE, OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION.

Untagged Cost Explorer keys such as app-code$ become key=untagged. Well-known SERVICE names are shortened (S3, EC2). Dual grouping uses S3|PutObject.

The SAM template creates an IAM user with ce:GetCostAndUsage only. Access keys are created after deploy and stored in Vault. Deployment lives in the infrastructure repo.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.33 - VEX Checker

A CLI tool that works with the Red Hat CSAF VEX feed and the Hummingbird catalog API to check CVE statuses and track coverage across Jira, VEX advisories, and container image scan results.

Subcommands

Subcommand Purpose
check Look up a single CVE and show Hummingbird product statuses
reconcile Cross-reference Jira CVE tickets with the VEX feed
catalog Cross-reference catalog API CVEs with Jira tickets

Prerequisites

  • Python 3.11+
  • Internet access to security.access.redhat.com (no VPN required) for check/reconcile; to api-hummingbird.hummingbird-project.io (no VPN required) for catalog
  • JIRA_EMAIL Atlassian account email — required for reconcile and catalog
  • JIRA_TOKEN Jira API token — required for reconcile and catalog

check — Single CVE lookup

Fetches the CSAF VEX document for a given CVE and prints the status of all Hummingbird products, filtering out all other RHEL/OpenShift/etc. entries.

check: usage

./check_vex.py check CVE-YYYY-NNNNN [--json]

check: options

Option Description
CVE-ID CVE to look up
--json, -j Output results as JSON

check: examples

$ ./check_vex.py check CVE-2014-8090

CVE-2014-8090 — CVE-2014-8090 ruby: REXML billion laughs attack (Moderate)
  hummingbird-1:ruby.src      known_affected  none_available
  hummingbird-1:ruby3.3.src   known_affected  none_available
  hummingbird-1:ruby3.4.src   known_affected  none_available
  hummingbird-1:ruby4.0.src   known_affected  none_available

$ ./check_vex.py check CVE-2014-8090 --json
{
  "cve": "CVE-2014-8090",
  "title": "...",
  "severity": "Moderate",
  "hummingbird": [
    { "product_id": "hummingbird-1:ruby.src", "status": "known_affected", "remediation": "none_available" },
    ...
  ]
}

check: exit codes

Code Meaning
0 No VEX document found, no Hummingbird products listed, or no known_affected status
1 One or more Hummingbird products are known_affected
2 Invalid CVE ID format

reconcile — Jira/VEX sync check

Queries the Jira HUM project (Security component) for all CVE tickets, then checks each CVE in the VEX feed. Reports two types of mismatches:

  • Open in Jira but not known_affected in VEX — VEX says the issue is resolved but the Jira ticket is still open.
  • Closed in Jira but known_affected in VEX — the Jira ticket was closed but Red Hat’s advisory still marks Hummingbird as affected.

Continuous closed-ticket VEX reconciliation (Done-Errata → fixed, Not a Bug → known_not_affected or package_not_listed, scoped to the ticket’s package, with awaiting-vex work queue and dashboard persistence) lives in hummingbird-cve-analysis / the dashboard Closed tab (HUM-5843). reconcile remains the one-shot CLI audit.

Jira tickets are considered closed when their status is one of: Done, Closed, Won't Fix, Not a Bug.

VEX documents are fetched in parallel (--workers, default 20) so the command is fast even with hundreds of CVE tickets.

reconcile: usage

export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py reconcile [--workers N] [--json]

Credentials are read exclusively from environment variables to avoid exposing them in process listings.

reconcile: options

Option Description
--jira-url Jira base URL (default: https://redhat.atlassian.net)
--workers Parallel VEX fetch workers (default: 20)
--json, -j Output results as JSON

reconcile: example

$ ./check_vex.py reconcile

Fetching Jira tickets... 142 ticket(s)
Fetching VEX statuses for 89 unique CVE(s)...
  89/89
2 mismatch(es) found:

  HUM-1234  CVE-2025-1234  closed → known_affected  (VEX still open)
    Jira: https://redhat.atlassian.net/browse/HUM-1234
    VEX:  https://security.access.redhat.com/data/csaf/v2/vex-feed/2025/cve-2025-1234.json

  HUM-5678  CVE-2024-5678  open → fixed  (VEX not affected)
    Jira: https://redhat.atlassian.net/browse/HUM-5678
    VEX:  https://security.access.redhat.com/data/csaf/v2/vex-feed/2024/cve-2024-5678.json

reconcile: exit codes

Code Meaning
0 No mismatches found
1 One or more mismatches detected
2 Missing JIRA_EMAIL or JIRA_TOKEN

catalog — Catalog API / Jira coverage check

Fetches all CVEs currently detected in Hummingbird container images (via Grype scan results stored in the catalog API) and cross-references them with Jira HUM Security tickets. Reports CVEs that have no corresponding Jira ticket.

CVEs with a closed Jira ticket are reported separately as expected propagation delay — the typical flow is CVE fix → RPM → image rebuild → Grype DB update, and Jira tickets are closed early while the catalog API reflects the latest Grype scan (updated ~once daily).

Non-CVE vulnerability identifiers (e.g. GHSA-*) from Grype are filtered out since they are not tracked in Jira.

catalog: usage

export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py catalog [--api-url URL] [--workers N] [--json]

Each CVE line shows severity, age since first detection, affected images, and for tracked CVEs the Jira ticket key. Components with known fix versions are shown in brackets when available.

Closed Jira tickets are cross-referenced with the VEX feed to distinguish between different root causes:

  • Done-Errata, VEX still affected – fix shipped in RPM but not yet propagated to container images; needs rebuild or lockfile refresh.
  • Done-Errata, VEX resolved – fix fully propagated, Grype DB just needs its daily update to stop flagging.
  • Won’t Do / Not a Bug, VEX resolved – VEX already marks CVE as not affected; Grype DB delay, will self-resolve.
  • Won’t Do / Not a Bug, VEX still affected – genuine known issue that will persist in scans (accepted risk, upstream won’t fix, etc.).

catalog: options

Option Description
--api-url Catalog API base URL (default: https://api-hummingbird.hummingbird-project.io/v1)
--jira-url Jira base URL (default: https://redhat.atlassian.net)
--workers Parallel VEX fetch workers (default: 20)
--json, -j Output results as JSON

catalog: example

$ ./check_vex.py catalog

Catalog: 72 CVEs across 29 images (scanned 2026-04-28T10:45:38Z)
Jira: 748 tickets (437 unique CVEs)

11 untracked CVE(s) (no Jira ticket):

  CVE-2008-2662  High     8d  ruby (12 tags)  [ruby3.3@3.3.10, +36 more]
  CVE-2026-2950  Medium   8d  aspnet-runtime, ... (30 tags)  [lodash@4.17.21]
  ...

45 tracked CVE(s) open in Jira:

  CVE-2025-68114  High  8d  php (3 tags)  [capstone@5.0.6]  HUM-925
  ...

8 closed in Jira (Done-Errata) but VEX still affected -- verify fix propagated:

  CVE-2026-27143  Critical  8d  caddy, go-fdo-client, ... (10 tags)  HUM-969

4 closed in Jira (Done-Errata), VEX resolved -- Grype DB delay:

  CVE-2025-61732  High  8d  go, xcaddy (6 tags)  HUM-1119

3 closed in Jira (Won't Do / Not a Bug), VEX resolved -- Grype DB delay:

  CVE-2026-27140  High  8d  caddy, go, ... (10 tags)  HUM-967

1 closed in Jira (Won't Do / Not a Bug), VEX still affected -- genuine:

  CVE-2025-12781  Medium  8d  postgresql, python, ... (17 tags)  HUM-1387

catalog: exit codes

Code Meaning
0 All catalog CVEs are tracked (have a Jira ticket)
1 One or more untracked CVEs found
2 Missing JIRA_EMAIL or JIRA_TOKEN

Development

# Run tests
cd vex-checker && python3 -m unittest discover tests -v

# Run linter (from repo root)
ruff check vex-checker

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.34 - Hummingbird CVE agent

CLI that investigates cve-needs-attention Jira tickets, using AI for the tickets that cve_analysis.py could not resolve. Later integrated with a cve work-item type in workqueue-service.

Diagram

flowchart TB

  TICKET["HUM ticket key"] --> CLI

  subgraph AGENT["CVE Agent"]
      direction TB

      CLI["CLI: run HUM-XXXX"]
      COL["Collect + validate<br/>Jira · Pulp · SBOM"]
      CAND{"Backport candidate?"}
      RESULT["InvestigationResult"]

      subgraph SANDBOX["Ephemeral rpms sandbox (one clone per ticket)"]
          AIINV["AI investigation<br/>read-only SBOM · package file tools"]
          BACKPORT["Secret-free sparse clone<br/>fetch patches · generate file_actions"]
      end

      AIADAPT["AI patch adaptation<br/>fresh sandbox"]

      CLI --> COL --> CAND
      COMMENT["Jira comment<br/>no label · state · VEX writes"]

      CAND -->|"analysis says affected + names a fix"| BACKPORT
      CAND -->|"no · or the fix already shipped"| AIINV
      BACKPORT -->|"needs_change + file_actions"| RESULT
      BACKPORT -->|"unsupported · needs_human, or the AI verdict unchanged"| RESULT
      AIINV -->|"needs_change + a fix named by analysis or cited by AI"| BACKPORT
      AIINV -->|Verdict| RESULT
      RESULT -->|"deterministic ADF comment"| COMMENT
  end

  subgraph QUEUE["workqueue-service"]
      direction TB

      CVEITEM["cve work item"]
      MR["mr work item<br/>Draft RPM MR"]
      KONFLUX["Konflux build"]

      CVEITEM --> MR --> KONFLUX
  end

  RESULT --> CVEITEM
  KONFLUX -.->|failed pipeline webhook| AIADAPT
  AIADAPT -->|adapted file_actions| MR

  classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
  classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
  classDef external fill:#f3f4f6,stroke:#6b7280,color:#111827
  class CLI,COL,CAND,RESULT,COMMENT,AIINV,AIADAPT agent
  class BACKPORT sandbox
  class TICKET,CVEITEM,MR,KONFLUX external

The diagram shows the agent as a whole. Today it runs as a standalone CLI that prints or persists the result locally and makes no remote writes; running it in production as the cve work-item type behind workqueue-service, together with the Jira/MR hand-off, is planned. The two writes have different owners: the agent posts the Jira comment itself through cve_analysis’s jira_client, which already handles ADF formatting and duplicate suppression, while the fix goes out as an mr work item that workqueue-service owns. Label, Fixed-in-Build, VEX, and close transitions stay with cve_analysis and are deliberately not written by the agent. Infrastructure errors propagate; only unsupported or ambiguous deterministic inputs become needs_human. The ticket-level state machine lives in the CVE Lifecycle & State Vocabulary.

Features

  • Scoped to needs-attention - starts from cve_analysis output and adds investigation only where deterministic analysis already gave up.

  • AI behind one seam - all LLM calls go through a small, swappable ai interface. The only backend today reuses the hummingbird-agent model adapter, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own. The initial prompt includes SBOM identity but not the complete document; the model can use bounded read-only tools to search its SBOM and inspect package files.

  • Structured results - every investigation produces a validated, versioned result contract. The result status is a Verdict from the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent and cve_analysis speak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine.

  • Deterministic backports - for an unambiguous single-CVE ticket whose analysis names upstream commits, the agent fetches those commits as patches and mechanically emits MR-compatible patch and spec file_actions. No AI call is needed when the analysis itself asserted affectedness; when it punted but still named a fix, AI judges affectedness and the same preparation attaches the file_actions. AI can also cite an exact upstream commit as structured evidence; deterministic validation still fetches the patch and generates all file actions, so the model never authors a patch. PR-only analysis is resolved to its commits first, in apply order (GitHub returns PR commits oldest-first, GitLab newest-first). Unsupported or ambiguous RPM layouts are left for human review. Two rpms-repo conventions are applied:

    • Patch files are named <NNNNN>-<cve-id>-<slug>.patch, where the slug is up to five words taken from the Jira summary’s description (for example 00003-cve-2026-82474-policy-bypass-allows-unauthorized-program.patch). When one CVE needs several commits, the patch number keeps the names unique. Tickets with no usable summary fall back to the short commit ID.
    • Release gets a trailing micro bump rather than an increment of the base release, which stays owned by rebases: 4 becomes 4.1, 4.1 becomes 4.2, and 16.p2%{?dist} becomes 16.p2.1%{?dist}. Date-shaped releases are refused and left for human review.

  • Ephemeral rpms sandbox - fixes are prepared in a throwaway, uncredentialed clone of the rpms repo: read the package’s code, fetch an upstream commit as a patch, and stage it into the package for an MR. Cloning is the slowest step in a run, so one clone is opened per ticket and shared by the AI’s read-only file tools and the deterministic backport; a ticket that inspects no package files never clones at all. The clone is removed on exit and never holds a push token. It runs no untrusted code; git calls use a minimal, secret-free environment with credential helpers, external config, and submodule recursion disabled. Patch application and package build verification are left to Konflux. HUM-7513 will consume failed-pipeline webhooks and invoke AI to adapt patches that do not apply or build cleanly.

Security model

The agent processes untrusted input — Jira ticket content, upstream commits, and package spec/patch files — so the sandbox is built so that no untrusted code is ever executed and untrusted input cannot reach secrets or a shared host.

  • No untrusted code execution. The sandbox only clones the rpms repo, reads files, resolves upstream PRs, downloads patches, and stages them into the package. It never runs rpmbuild, a spec’s %prep scriptlet, or any code supplied by a spec or patch. Verifying the fix by building the package — the step that would execute spec-controlled code — is deliberately delegated to Konflux, which builds the resulting MR in its own hardened pipeline. This keeps a whole class of remote-code-execution surface (and slow, resource-heavy builds of large packages) out of the agent entirely.
  • Deployment isolation. The agent ships as a container image and is meant to run as a Kubernetes Job or inside workqueue-service; it is not deployed yet. The rpms fork lives in a private temp dir inside that boundary and is removed on exit, so git operations run inside the deployment sandbox rather than on a shared host — no separate nested container is needed, because nothing here executes untrusted code.
  • Secret-free subprocesses. git runs with a minimal, allowlisted environment. The service process env (Jira and model tokens) is never inherited by child processes, so a subprocess cannot read or exfiltrate them.
  • Uncredentialed, hardened git. The fork is a public, read-only clone with no push token. Credential helpers and interactive prompts are disabled, system/global git config is neutralized (an attacker-controlled GIT_CONFIG_GLOBAL cannot change behaviour), and submodule recursion is off.
  • Input validation. Package names are restricted to a safe charset (no path traversal or absolute paths), file reads are confined to the package directory, saved patch names must be *.patch and cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.

Prerequisites

  • Python 3.11+
  • Jira API credentials (JIRA_TOKEN, and JIRA_USER where basic auth applies)
  • Vertex AI access for the AI path: GOOGLE_CLOUD_PROJECT, or GOOGLE_API_KEY for the direct Gemini API. The deterministic fast path needs neither.

Installation

cd hummingbird-cve-agent
pip install -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"

Usage

hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json

This gathers facts for the ticket, runs the investigation, and prints the result as JSON or persists it to --output. Eligible backports include base64-encoded file_actions and a commit_message ready for an mr work item; everything else is judged by the AI client. A backport that cannot be prepared is never retried by AI — adapting a patch that does not apply is HUM-7513’s reaction to a failed Konflux pipeline. No remote writes.

Container

Build from the tools repository root so the image can include the shared hummingbird-agent and hummingbird-cve-analysis packages:

podman build -f hummingbird-cve-agent/Containerfile \
  -t hummingbird-cve-agent .
podman run --rm \
  -e JIRA_TOKEN \
  -e JIRA_USER \
  hummingbird-cve-agent run HUM-1234

Configuration

Operational settings come from the YAML config file; secrets and model credentials come from the environment.

Variable Purpose
HUMMINGBIRD_CVE_AGENT_CONFIG Path to a YAML config file (optional)
JIRA_TOKEN Jira API token
JIRA_USER Jira basic-auth user (optional)
GITHUB_TOKEN GitHub token for PR commits (optional)
GITLAB_TOKEN GitLab.com token for MR commits (optional)
GOOGLE_API_KEY Gemini API key (direct API mode)
GOOGLE_CLOUD_PROJECT GCP project for Vertex AI mode

YAML settings:

Key Default Purpose
jira_url Red Hat Jira Jira base URL
ai_backend agent AI backend selector (swappable seam)
model claude-sonnet-4-6 Model name passed to build_model
model_regions Vertex regions for the Gemini prefixes Model-name prefix to Vertex region

model_regions maps a model-name prefix to the Vertex region to call, longest prefix first. It ships with the regions the Gemini models need and is meant to be overridden, not to be exhaustive.

Development

See the main README for development workflows.

make check                         # lint
make test                          # run tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.35 - Red Hat Catalog Environment Promotion

Environments

Environment URL Source Deploy trigger
MR Preview GitLab Pages (/mr-{IID}) MR branch Auto on MR pipeline
Experimental images.experimental.hummingbird-project.io/<branch-name> experiment/* branch Auto on push to experiment branch
Staging images.staging.hummingbird-project.io main Auto on merge to main
Production images.redhat.com main Manual after staging deploy

Disambiguation: The infra experimental host (images.experimental.hummingbird-project.io, experiment/* branches) is an off-main deploy sandbox. The former in-app /experimental product route was removed in HUM-2171 — use /api instead. See Archived surfaces.

Promotion Paths

Routine: staging to production

  1. MR merged to main
  2. redhat-catalog-visual-staging-reference captures live visuals on current staging
  3. deploy_redhat_catalog_staging runs automatically
  4. redhat-catalog-e2e-smoke (smoke) and visual_redhat_catalog_staging (presentation diff) validate staging
  5. UAT sign-off on staging — product, design, and engineering release approvers complete the UAT checklist and file a record (UAT program runbook)
  6. Approval owner clicks deploy_redhat_catalog (manual gate) after automated gates pass and UAT is approved
  7. Production deploys from same commit

Exploration: experimental to staging to production

  1. Create branch experiment/<name> from main (must use exactly experiment/, not experimental/ or other prefixes — without it, no deploy triggers)
  2. Push triggers deploy_redhat_catalog_experimental automatically
  3. Branch deploys to images.experimental.hummingbird-project.io/<name>/
  4. Iterate on experimental host for days or weeks
  5. Light UAT (optional spot-check on experimental URL — see UAT program — experimental path)
  6. When ready: open MR from experiment/<name> to main
  7. Normal MR review, merge, then routine promotion path (full UAT on staging)

Retiring an experiment

  1. Delete the experiment/<name> branch

  2. Remove the branch prefix from the experimental S3 bucket:

    aws s3 rm s3://redhat-catalog-experimental-spa/<name>/ --recursive

  3. Invalidate the CloudFront cache:

    aws cloudfront create-invalidation \
  --distribution-id <EXPERIMENTAL_DISTRIBUTION_ID> \
  --paths "/<name>/*"

Build Configuration Per Environment

The infrastructure pipeline sets these environment variables at build time:

Variable Experimental Staging Production
ASSET_PATH /<branch-name>/ / /
DPAL_USE_STAGING true true (unset, uses prod)
TRUSTARC_DATA_DOMAIN images.experimental.hummingbird-project.io images.staging.hummingbird-project.io images.redhat.com
CATALOG_API_BASE_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
CATALOG_BASE_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
HUMMINGBIRD_API_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
JIRA_API_URL Jira API URL Jira API URL Jira API URL

Infrastructure

SAM Stacks

Stack Template ResourcePrefix CatalogDomainName
redhat-catalog-prod template.yaml redhat-catalog images.redhat.com
redhat-catalog-staging template.yaml redhat-catalog-staging images.staging.hummingbird-project.io
redhat-catalog-experimental template-experimental.yaml redhat-catalog-experimental images.experimental.hummingbird-project.io

Staging uses the same template.yaml as production (single-site CloudFront distribution). Experimental uses template-experimental.yaml which adds a CloudFront Function for path-prefix SPA routing across multiple branches.

DNS

ALIAS records (images.staging/experimental.hummingbird-project.io → CloudFront) are created automatically by post_deploy.sh after each deploy.

First-time bootstrap: The SAM template creates an ACM certificate with DNS validation. The ACM validation CNAME must exist in the hummingbird-project.io Route 53 zone before the first sam deploy, or CloudFormation will wait 30 minutes and roll back. To bootstrap:

  1. Deploy with CatalogDomainName="" (uses CloudFront default domain, no cert)
  2. Create the ACM validation CNAME in Route 53 (get the value from the ACM console or aws acm describe-certificate)
  3. Redeploy with CatalogDomainName set to the real domain

After the first successful deploy, no manual DNS steps are needed.

CI/CD Variables

Set in GitLab project settings:

Variable Value
REDHAT_CATALOG_STAGING_URL https://images.staging.hummingbird-project.io

Pipeline Flow

experiment/* push:
  redhat-catalog-experimental-check
    -> deploy_redhat_catalog_experimental

MR (target main):
  redhat-catalog (build + test)
    -> redhat-catalog-browser-tests
    -> redhat-catalog-visual-mock (Playwright mock API; MR vs merge-base diff)
    -> pages (preview)

main push:
  redhat-catalog-visual-staging-reference (pre-deploy live capture)
    -> deploy_redhat_catalog_staging (auto)
      -> redhat-catalog-e2e-smoke (smoke)
      -> visual_redhat_catalog_staging (post-deploy vs pre-deploy diff)
    -> deploy_redhat_catalog (manual; needs staging deploy + smoke + visual + UAT sign-off)

Visual regression details: redhat-catalog testing guide — Visual regression and e2e/visual/README.md.

4.5.36 - S3 Lookaside Cache

Infrastructure for caching dist-git lookaside content, providing access to source artifacts for the Hummingbird build pipeline.

Overview

This stack provides S3-based infrastructure for dist-git artifacts with CloudFront CDN. It caches content from upstream sources like Git tarballs.

A companion stack (s3-lookaside-cache-upload-role) creates a GitLab OIDC identity provider and an IAM role that GitLab CI jobs can assume using short-lived JWT tokens, eliminating the need for long-lived access keys.

Architecture

Cache Infrastructure

flowchart LR
    clients["Build Clients"]
    cf["CloudFront\nDistribution"]
    s3["S3 Bucket\n(dist-git-cache)"]
    logs["S3 Logs\nBucket"]
    backup["AWS Backup\nVault"]
    headers["Security\nHeaders"]

    clients --> cf --> s3
    cf --> headers
    s3 --> logs --> backup

GitLab CI Upload Role

flowchart LR
    gitlab["GitLab CI\n(.gitlab-ci.yml)"]
    sts["AWS STS\n(validates JWT\nvia OIDC/JWKS)"]
    role["IAM Role\n(scoped to\nS3 PutObject)"]
    s3["S3 Bucket\n(dist-git cache)"]

    gitlab -- "id_token" --> sts
    sts -- "AssumeRoleWithWebIdentity" --> role
    role -- "temporary credentials" --> sts
    sts -- "credentials" --> gitlab
    gitlab -- "s3:PutObject" --> s3

Components

Cache Stack (s3-lookaside-cache)

Resource Type Description
DistGitCacheBucket S3 Bucket Main cache storage with versioning and object lock
DistGitLogBucket S3 Bucket Access logs with tiered storage lifecycle
DistGitCacheDistribution CloudFront CDN with HTTP/2+3, IPv6, TLS 1.2+
DistGitCacheOAC Origin Access Ctrl Secure S3 access from CloudFront
DistGitCacheCachePolicy Cache Policy 1-day default TTL, 1-year max, Gzip/Brotli
DistGitCacheResponseHeadersPolicy Response Headers HSTS, X-Frame-Options, XSS protection
DistGitBackupVault Backup Vault AWS Backup vault for data protection
DistGitBackupPlan Backup Plan Daily (35-day) and weekly (365-day) backups
DistGitUploadPolicy IAM Managed Policy Grants s3:PutObject to the cache bucket

Upload Role Stack (s3-lookaside-cache-upload-role)

Resource Type Description
GitLabOIDCProvider IAM OIDC Provider Registers gitlab.com as a trusted identity provider
DistGitUploadRole IAM Role Web identity role assumable by the configured GitLab project

Parameters

Cache Stack Parameters

Parameter Description
ResourcePrefix Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-cache)
BucketName Globally unique name for the cache bucket (logs bucket appends -logs)

Upload Role Parameters

Parameter Description
ResourcePrefix Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-upload)
CacheStackName Name of the deployed s3-lookaside-cache CloudFormation stack
AdditionalCacheStackName Optional second cache stack whose upload policy is attached to the role
GitLabProjectPath GitLab project path allowed to assume the role (e.g., redhat/hummingbird/rpms)

S3 Key Structure

Files are stored using the dist-git lookaside path convention:

{namespace}/{package}/{filename}/{hashType}/{hash}/{filename}

Example:

rpms/tar/tar-1.35.tar.xz/sha512/abc123.../tar-1.35.tar.xz

Trust Policy

The upload role’s trust policy restricts access using three conditions (all StringEquals):

  • Audience (gitlab.com:aud): Must be https://gitlab.com
  • Subject (gitlab.com:sub): Must match project_path:<GitLabProjectPath>:ref_type:branch:ref:main
  • Protected ref (gitlab.com:ref_protected): Must be "true"

Only the main branch of the configured project can assume the role.

GitLab CI Usage

Configure your .gitlab-ci.yml to assume the role using id_tokens. The AWS CLI automatically calls AssumeRoleWithWebIdentity when AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN are set:

upload to cache:
  image:
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  script:
    - set +x
    - printenv GITLAB_OIDC_TOKEN > /tmp/oidc-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/oidc-token
    - export AWS_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME}"
    - export AWS_ROLE_SESSION_NAME="gitlab-ci-${CI_JOB_ID}"
    - aws s3 cp "$FILE" "s3://${BUCKET}/${S3_KEY}"
    - rm -f /tmp/oidc-token

The aud value in id_tokens must match the audience configured in the OIDC identity provider (https://gitlab.com). The token is written to a file because the AWS SDK reads it from AWS_WEB_IDENTITY_TOKEN_FILE rather than accepting it inline.

Security Features

  • Encryption: AES256 server-side encryption with S3 bucket keys
  • Public Access: All public access blocked on both buckets
  • Transport Security: HTTPS enforced via bucket policy
  • Object Lock: GOVERNANCE mode with 1-day default retention (will be increased in the future)
  • CloudFront OAC: Modern Origin Access Control (not legacy OAI)
  • Security Headers: HSTS, X-Frame-Options (DENY), X-Content-Type-Options, X-XSS-Protection, strict referrer policy
  • TLS 1.2 Minimum: For all CloudFront connections
  • OIDC Federation: Short-lived tokens instead of long-lived access keys

Backup Strategy

Schedule Retention Cold Storage
Daily (5:00 AM UTC) 35 days -
Weekly (Sunday 5:00 AM UTC) 365 days After 30 days

Continuous backup is enabled for point-in-time recovery.

Log Lifecycle

Access logs transition through storage tiers:

Age Storage Class
0-30 days Standard
30-60 days Standard-IA
60-90 days Glacier IR
90+ days Expired

Outputs

Cache Stack Outputs

Output Description
BucketName Name of the cache S3 bucket
BucketArn ARN of the cache S3 bucket
LogBucketName Name of the logs S3 bucket
LogBucketArn ARN of the logs S3 bucket
DistributionDomainName CloudFront domain name
DistributionId CloudFront distribution ID
DistributionArn CloudFront distribution ARN
BackupVaultName AWS Backup vault name
BackupVaultArn AWS Backup vault ARN
BackupPlanId AWS Backup plan ID
UploadPolicyArn ARN of the managed policy granting upload access

Upload Role Outputs

Output Description
RoleArn ARN of the upload role (pass to GitLab CI as ROLE_ARN)
RoleName Name of the upload role
OIDCProviderArn ARN of the GitLab OIDC identity provider

References

4.5.37 - Red Hat Catalog UAT Program

User Acceptance Testing (UAT) for the Red Hat Catalog SPA — human sign-off on staging before production promotion.

Epic: HUM-2179 · Story: HUM-2183 · Depends on: HUM-2060 (staging host + promotion gates)

Purpose

UAT complements automated validation (smoke, E2E, visual regression, security scans). It catches product, content, and UX issues that automation does not cover. UAT does not replace automated gates.

Layer Owner Blocks prod?
Unit / lint / MR security CI MR merge
Staging smoke + visual CI Manual prod job availability
UAT sign-off Product + design + eng Manual prod deploy
Post-prod smoke CI / on-call Rollback decision

Environments (disambiguation)

Term Meaning URL pattern
Staging Pre-prod validation target for UAT https://images.staging.hummingbird-project.io
Experimental host Infra sandbox for experiment/* branches https://images.experimental.hummingbird-project.io/<branch>/
Former /experimental route Removed in-app product route (HUM-2171) Returns 404 — use /api instead. See archived-surfaces.md

See Red Hat Catalog Environment Promotion for full promotion paths.

Promotion gates

Staging → production (full UAT)

All of the following are required before clicking deploy_redhat_catalog:

  1. deploy_redhat_catalog_staging succeeded on main
  2. redhat-catalog-e2e-smoke passed against staging
  3. visual_redhat_catalog_staging passed (pre/post deploy diff)
  4. MR-blocking security jobs clean (or approved exceptions) — see security-scanning.md
  5. UAT sign-off recorded (checklist + Jira comment; git record in redhat-catalog/docs/uat-records/)

The prod deploy job is manual in GitLab; the release owner must verify UAT before triggering it.

Light UAT on experimental host

Experimental hosts are for iteration, not customer release. When opening an MR from experiment/<name> to main, use a light UAT pass on the experimental URL — not the full checklist:

Check Required?
Feature under test works on experimental host Yes
No console errors on primary journey Yes
Masthead/footer not broken Yes
Full staging UAT checklist No — runs on staging after merge
Named approver sign-off Eng owner only (no prod gate yet)

After merge, staging receives the full UAT before prod.

Named approvers (staging → prod)

Confirm names with the product owner during checklist review (HUM-2060 open question on approval owners). Until confirmed, use these roles:

Role Primary Backup Responsibility
Product TBD — Catalog journeys, copy, feature completeness
Design TBD — Visual polish, layout, theme, responsive UX
Engineering release TBD TBD Pipeline green, staging commit matches intent, prod deploy authorization

Any Block from product or design stops prod promotion until defects are fixed on staging and UAT re-run.

Runbook — request and run UAT

1. Request UAT (release owner)

When main pipeline completes staging deploy + automated gates:

  1. Create or reuse a Release ticket in Jira (label redhat-catalog, link HUM-2179 epic).
  2. Comment with pipeline URL, commit SHA, and summary of user-facing changes.
  3. Label ticket uat-requested.
  4. Notify approvers (#hummingbird or team channel — use team default).

2. Execute UAT (approvers / delegates)

  1. Open uat-checklist.md against staging.
  2. File results in redhat-catalog/docs/uat-records/YYYY-MM-DD-<topic>.md.
  3. Log defects as Jira bugs linked to the release ticket.

Expected turnaround: 2 business days from uat-requested for routine releases; same-day for hotfixes when approvers are available.

3. Sign off or block

  • Pass: All checklist items pass or have accepted exceptions; all three approver rows signed; Jira comment posted (template below); label uat-pass.
  • Fail: Label uat-fail; do not trigger deploy_redhat_catalog until staging is fixed and UAT re-run.

4. Promote to production (engineering release)

  1. Verify Jira release ticket has uat-pass and linked sign-off record.
  2. In GitLab main pipeline, run manual job deploy_redhat_catalog.
  3. Comment on release ticket with prod pipeline URL.

What blocks release

Blocker Who resolves
Staging smoke or visual CI failed Engineering
Open Critical/High security finding without exception Engineering + security
UAT checklist failure (product/design) Engineering fixes → re-UAT
Missing approver sign-off Approver
Staging commit ≠ intended release SHA Release owner

Jira sign-off template

Post on the release ticket when UAT completes:

h3. UAT sign-off — staging → prod

*Staging URL:* https://images.staging.hummingbird-project.io
*Commit:* {full SHA}
*Pipeline:* {GitLab pipeline URL}
*Checklist record:* redhat-catalog/docs/uat-records/{filename}.md

||Role||Name||Decision||
|Product|{name}|Approve / Block|
|Design|{name}|Approve / Block|
|Engineering release|{name}|Approve / Block|

*Result:* Approved for prod / Blocked
*Defects:* {HUM-xxx links or "none"}
*Tester:* {name}, {date}

Suggested Jira labels

Label Meaning
uat-requested Staging ready; waiting for human validation
uat-pass Full sign-off complete
uat-fail Blocked; defects logged
uat-light Experimental → staging spot-check only

Optional custom field (if added in Jira project settings): UAT status (Not started / In progress / Passed / Failed).

Release ticket template

Create a Jira Task or Release issue per prod promotion:

## Release summary
- **Target:** staging → production
- **Commit:** 
- **User-facing changes:** 

## Automated gates
- [ ] Staging deploy green
- [ ] Smoke passed
- [ ] Visual regression passed
- [ ] Security scans clean (or exceptions documented)

## UAT
- [ ] Checklist completed ([uat-checklist.md](link))
- [ ] Sign-off record filed ([uat-records/](link))
- [ ] Product approve
- [ ] Design approve
- [ ] Engineering release approve

## Prod promote
- [ ] deploy_redhat_catalog triggered
- [ ] Post-deploy verification

Out of scope

  • Automating UAT (Playwright covers regression; UAT stays human)
  • Legal/compliance sign-off (separate security program)

4.5.38 - Hummingbird MR Collaboration

How to open merge requests and help on someone else’s MR in the redhat/hummingbird/tools monorepo. Applies to all components (including redhat-catalog/).

Features

  • Same-repo branches — contributors work in the shared GitLab project, not personal forks for day-to-day changes
  • One MR per change — seniors push fixes to the author’s source branch instead of opening a nested MR
  • Automatic CI — each push to the MR branch re-runs the pipeline, GitLab Pages preview, and dashboard status links

Prerequisites

  • GitLab account with Developer (or higher) access on redhat/hummingbird/tools
  • Git clone of the monorepo
  • Optional: glab CLI authenticated to GitLab

Git remotes

Many developers use two remotes:

Remote Typical URL Use for
upstream git@gitlab.com:redhat/hummingbird/tools.git Fetch MRs, push to shared branches
origin Personal fork (if configured) Optional; not used for team MRs

Add upstream if missing:

git remote add upstream git@gitlab.com:redhat/hummingbird/tools.git
git fetch upstream

Do not open fork MRs for routine team work. mr-auto-approver unconditionally rejects MRs where source_project_id != target_project_id.

Open an MR (author)

git clone git@gitlab.com:redhat/hummingbird/tools.git
cd tools
git checkout -b hum-XXXX-short-description
# edit, commit
git push -u upstream hum-XXXX-short-description

Open an MR targeting main in the GitLab UI.

Branch naming: prefer hum-XXXX-* or a descriptive feat/*, fix/*, or experiment/* prefix (see Environment Promotion for experiment/* deploy behavior).

Help on someone else’s MR (reviewer)

Use the MR source branch name from the GitLab MR page (not necessarily your local checkout name).

1. Check out the MR locally

Option A — fetch by MR number (works without glab):

git fetch upstream merge-requests/<IID>/head:mr-<IID>
git checkout mr-<IID>

Option B — checkout the source branch:

git fetch upstream <source-branch>
git checkout -B <source-branch> upstream/<source-branch>

Option C — glab:

glab mr checkout <IID> --repo redhat/hummingbird/tools

2. Edit and verify locally

Component-specific checks (example for Red Hat Catalog):

cd redhat-catalog
npm run ci-checks
npm run start:dev   # optional UI smoke test

3. Push to the author’s branch

If your local branch name matches the MR source branch:

git push upstream <source-branch>

If you checked out via mr-<IID> (local name differs from remote), push explicitly:

git push upstream HEAD:<source-branch>

Example: local branch mr-748, remote source branch feat/image-request-form:

git push upstream HEAD:feat/image-request-form

Optional — rename locally so future pushes are simpler:

git branch -m mr-<IID> <source-branch>
git push -u upstream <source-branch>

The existing MR updates; CI re-runs; no second MR is needed.

What runs on each MR push

For redhat-catalog/** changes, see Red Hat Catalog Environment Promotion — Pipeline Flow. Summary:

Stage Result
Test Build, lint, type-check, unit tests, audit, SAST
Visual Mock visual diff (MR head vs merge-base)
Preview GitLab Pages at /mr-{IID}
Status Internal MR note with Hummingbird dashboard link
Agent Code review on open/update; /hummingbird analyze-failures on CI fail

After merge to main, staging deploys automatically; production requires manual gate and UAT sign-off.

Troubleshooting

error: src refspec <branch> does not match any

Git has no local branch with that name. You are probably on mr-<IID> while pushing git push upstream feat/....

Fix: push the current branch to the remote source branch:

git push upstream HEAD:<source-branch>

Push rejected / permission denied

  • Confirm Developer+ access on redhat/hummingbird/tools
  • Confirm you are pushing to upstream, not a personal fork
  • Check whether the MR author enabled “Prevent pushing to source branch” (uncommon)

fatal: couldn't find remote ref merge-requests/.../head

Fetch from upstream, not origin:

git fetch upstream merge-requests/<IID>/head:mr-<IID>

Team tips

Tip Why
Assign yourself when taking over pushes Clear ownership on the MR
Use MR Pages preview Review UI without local dev
Use dashboard link in MR internal note Per-commit Konflux/build status
Use GitLab suggestions Small fixes without a local checkout
Use experiment/<name> for long exploration Live host before opening an MR

4.5.39 - K8s Integration Tests

Smoke tests that verify tools container images work in a real Kubernetes cluster. Each test creates short-lived pods, checks expected behavior, and cleans up. Tests run automatically on every MR via the Konflux K8s test pipeline.

Test Runner

Tests are executed by ci/run_tests_k8s.sh. For each component it:

  1. Reads {component}/tests-k8s.yml
  2. Parses each test entry (YAML -> JSON via Python)
  3. Runs the command block with kubectl wired to the target cluster
  4. Cleans up labeled resources (hum-k8s-test=<run-id>) after each test

Running Locally

Test against a local cluster (kind, minikube, or remote):

# Single component with a published image
IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \
IMAGE_NAME=gitlab-ci--tools \
  ci/run_tests_k8s.sh --context kind-kind gitlab-ci--tools

# Single component with a locally-built image
podman build -t localhost/hummingbird-dashboard:dev hummingbird-dashboard/
kind load docker-image localhost/hummingbird-dashboard:dev
IMAGE_URL=localhost/hummingbird-dashboard:dev \
IMAGE_NAME=hummingbird-dashboard--tools \
  ci/run_tests_k8s.sh --context kind-kind hummingbird-dashboard--tools

CI Integration

In Konflux, the tools-k8s-test IntegrationTestScenario triggers on every MR. The pipeline:

  1. Checks whether tests-k8s.yml exists for the changed component
  2. Provisions an ephemeral EaaS namespace
  3. Runs ci/run_tests_k8s.sh with the built image

Components without tests-k8s.yml skip the test (the pipeline exits early after the check step).

When an MR touches multiple components in one PR-group snapshot, Konflux runs a single group PipelineRun instead of one per component, invoking the script once with --group-component-name (repeated) and per-component IMAGE_URL_<COMPONENT> env vars instead of the single-component IMAGE_URL/IMAGE_NAME pair. --group-component-name is handled identically to --component-name.

Adding Tests for a New Component

  1. Create {component}/tests-k8s.yml with one or more named tests:

    ---
smoke-test:
  command: |
    name="${TEST_GROUP}-smoke-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE:?}" --restart=Never \
      --labels="${TEST_RUN_LABEL}" \
      --command -- echo "hello"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \
      "pod/${name}" --timeout=120s || test_fail "Pod did not succeed"
    kubectl logs "${name}"

  2. Use ${TEST_IMAGE} for the image under test, ${TEST_RUN_ID} and ${TEST_RUN_LABEL} for unique naming and cleanup.

  3. Call test_fail "message" to fail a test with a clear message.

  4. Keep tests fast (under 2 minutes) — these are smoke tests, not integration suites.

Common pitfalls

  • All lines in command: | block scalars must be indented relative to the command key. Unindented lines break YAML parsing.
  • Use command -v instead of which to check for commands — which is not available in all container images.
  • For multi-statement Python one-liners, use semicolons on a single line: python3 -c 'import foo; import bar; print("ok")'

Debugging Failures

See the EaaS and Debugging guide for accessing ephemeral namespaces and using Kubearchive for historical PipelineRun data.

5 - Agentic SDLC

How Project Hummingbird uses agentic AI to drive velocity across the software development lifecycle.

Project Hummingbird’s core mission is velocity: shipping RPM updates, CVE fixes, and new container images as fast as possible. The pipeline already delivers at a scale unprecedented within Red Hat — consistently over 1,000 automated commits per week across the containers and rpms repositories, a throughput that would be impossible to sustain manually — work that in most Linux distributions is still performed manually by dedicated teams. Managing 400+ packages and 1,750+ image builds with a 24-hour security SLO, that pipeline is not a nice-to-have — it is what makes the project viable with a small team.

The pipeline solves throughput. What it does not solve is the full software development lifecycle: assessing what to build, implementing it, documenting it, testing it, reviewing UX, coordinating releases. For a small team maintaining hundreds of containers and building an entire OS, staffing all of those functions manually does not scale — and agentic AI is what changes that equation. Where rule-based automation handles the deterministic, agents take on everything that has historically required a human to pick up, think through, and act on. Hummingbird is building toward becoming an open source reference implementation for responsible agentic AI in software engineering: demonstrating through production use, not promises, that this kind of automation can be adopted incrementally and grounded in engineering discipline.

The Agentic SDLC Vision

The target state is a development pipeline driven by agents at every stage — with humans engaged where their judgment is needed, not where their time is consumed by routine work.

Every change flows through the same lifecycle:

1. Request. A customer, product team, or automated trigger submits a change: a new container image, an RPM update, a feature request, a bug fix.

2. Analysis. A product management agent assesses the request — evaluating feasibility, priority, and fit against existing components and conventions. For straightforward requests it proceeds automatically; for complex or ambiguous ones it surfaces the relevant context and flags the case for human input before continuing. This is the work that normally sits in a product manager’s queue.

3. Implementation. The request splits based on its nature:

  • Deterministic changes — dependency updates, lockfile syncs, rebuilds triggered by upstream changes — are implemented by rule-based automation and merged automatically once CI passes. No ticket, no approval, no delay.
  • Everything else is handled by an engineering agent: a developer co-programming with an agent, or an agent working autonomously to write the code or configuration, run validation, and open a merge request. Agentic AI is the default answer for everything that has historically required a human to pick it up, think through it, and act on it.
  • Specialist agents contribute alongside: a tech writer agent drafts or updates the documentation and long-form Red Hat product content; a testing agent generates or extends tests; and a user design agent applies UX patterns and practices to any interface-related changes.

4. Review. Automated agents check for policy compliance and code quality, including the production code review agent. The same agent capabilities that contribute during implementation also apply here: a tech writer agent reviews documentation changes for clarity and correctness; a testing agent reviews test coverage and flags gaps; and a user design agent reviews UX-related changes against established patterns. A human engineer makes the final call on all of it. No agent-produced work is merged without human approval — the human is not eliminated, but elevated to the role where judgment actually matters.

5. Merge. The change ships.

The goal is maximum automation, pursued pragmatically rather than dogmatically — the right tool for each kind of work. Where agentic AI excels is in the space just before human judgment is required: gathering context, surfacing options, resolving the routine, so that when a human does engage, the decision in front of them is sharp, informed, and fast.

Responsible AI

Agent-produced code and decisions require human approval before merge. This is not a temporary constraint — it is a design principle. Humans may stay in the loop to steer or closely supervise complex work when that helps, but the standing requirement is approval — not constant supervision of every agent run. The feedback between human judgment and agent behavior is what keeps the system trustworthy over time.

The high-velocity automated pipeline — over 1,000 commits per week — operates through rule-based automation and does not involve agents; no human review is required there. The failure analysis agent does run at high volume — triggering on every pipeline, build, or test failure — but it produces analysis and comments, not code changes requiring approval. That alone saves hundreds of minutes per week of manual log triage. Agent work that requires human approval is scoped to SDLC tasks: feature requests, documentation, code review — a much lower volume where the requirement is entirely manageable.

Security

Agents run in containers — locally via Podman, in production on OpenShift. This is not a novel security model: it is the same approach the industry has long used for untrusted code, applied consistently. The Hummingbird agent platform is architecturally designed to ensure agents have no access to secrets or tokens. That single constraint effectively neutralizes prompt injection as a threat: an agent that cannot access credentials or act on external systems is, in practice, just another contributor — and the review process treats it as such. The same gatekeeping that applies to any third-party contribution applies to agent-produced work. The worst-case outcome is a low-quality merge request — and that is precisely what the review process, including the review agents themselves, is designed to catch.

Learning from Feedback

Agents learn from feedback — and this is already working in production. When a reviewer comments on an agent-produced merge request, the agent reads those conversations, incorporates the feedback, and updates its work. It avoids re-raising issues already addressed and respects when a developer has explained an intentional design choice. That feedback is also preserved — stored so that future agent runs can draw on it, improving with each iteration rather than repeating the same mistakes — an approach grounded in the frameworks established in Pascal Bornet’s Agentic Artificial Intelligence.

From Pair Programmer to Autonomous Agent

Deploying production agents requires a foundation. The approach that works is not to automate first and document later — it is the reverse.

Phase 1 — Agent as pair programmer. Agents assist individual developers: drafting code, explaining unfamiliar systems, proposing changes for review. This phase builds understanding of what agents need: context, conventions, constraints.

Phase 2 — Documentation as fuel. Agent output quality is directly proportional to documentation quality. Every repository carries an AGENTS.md — a file with instructions targeting agents specifically, but one that mostly points to other documentation rather than duplicating it. That underlying documentation is written for humans and agents alike: clear, specific, and rich with rationale. The AGENTS.md is the entry point; the broader docs are the substance. An agent with good documentation reasons from the same foundation as the team; an agent without it guesses.

Phase 3 — Agents in production. With the foundation in place, agents move into the critical path: first as reviewers proposing concrete changes, then as autonomous contributors on well-defined tasks. The next section describes what that looks like in practice for Hummingbird today.

An agent running autonomously in production is not fundamentally different from a developer running an agent during pair programming. The same agent, the same context, the same documentation — the difference is who initiates the run and how much human supervision follows. If an agent consistently does good work when a developer is in the loop, that is a strong signal the project is ready to let it operate more independently. The quality of pair programming performance is the leading indicator for autonomous deployment readiness. Trust is built incrementally, one well-handled task at a time.

See From Pair Programmer to Autonomous Agent by Valentin Rothberg for the full treatment.

Documentation as Infrastructure

Writing Documentation with Agents

In an agentic system, documentation is infrastructure — the primary interface between the team and its agents, and the quality of what an agent produces is bounded by the quality of what it can read. That makes writing documentation with agent assistance not a separate task but a natural extension of the work: the same agent that helps an engineer write and review code can help write the documentation for it — capturing decisions, explaining rationale, drafting contribution guides. This is not a shortcut; it is the same pair-programming model applied to a different kind of output. This is how Hummingbird built its documentation from the start: engineers worked alongside agents to draft and improve it in the same way they wrote code — iteratively, collaboratively, with humans in the lead. The result is a virtuous cycle: better docs make agents more capable, and more capable agents help produce better docs.

Tech writers are not a prerequisite — engineers writing clear, intentional documentation with agent assistance is sufficient to start. But their skills become more valuable in this model, not less: precision over generality, rationale alongside rules, explicit anti-patterns. A tech writer contributing to an agentic codebase is directly shaping the behavior of the agents that work within it.

In Hummingbird, this is not theoretical. The containers repository is developed in active collaboration with Red Hat tech writers, whose customer-facing product documentation lives in the same repository as the code — keeping internal conventions, agent context, and customer-facing content in one place, shared across engineers, tech writers, and agents alike.

Knowledge Transfer Across Teams

The same principle — knowledge encoded in markdown, readable by humans and agents alike — also answers a broader question: how does one team’s expertise become available to agents maintained by another? The answer is the same as everything else in this system — markdown. A UX team encodes their patterns and design practices in a skills file; the user design agent picks it up. The documentation team maintains their style guide, standards, and best practices — including Red Hat’s documentation guidelines — encoded once and available to every agent that produces or reviews written content. No code changes, no integrations, no handoff meetings. Any team can contribute their domain knowledge to the agentic system simply by writing it down in a form that both humans and agents can read. The markdown file is the interface between teams.

Agents in Production Today

Two agents are running in production today, built on the Hummingbird Agent platform. They operate on every merge request across the containers and rpms repositories, which together see over 1,000 automated commits per week — a throughput that would be impossible to review and maintain manually without agent assistance.

More agents are under active development in parallel, with production deployment planned after Red Hat Summit 2026 — likely by end of Q2. One example is the merge conflict agent — which will automatically resolve merge conflicts that arise during automated RPM updates. The full roadmap is tracked in the HUM-687 epic.

Failure Analysis Agent

When a merge request pipeline fails, this agent automatically investigates and posts its findings directly on the merge request. It pulls data from multiple sources — GitLab CI job logs, Konflux build and test pipeline runs, Testing Farm test results — and correlates them to identify root causes. Where the same underlying issue causes multiple failures across different images or architectures, it groups them so engineers see patterns rather than an overwhelming list of individual failures. Every finding includes links to the specific logs and artifacts so engineers can dig deeper without having to hunt.

What previously required clicking through multiple levels of web UI, downloading logs, and working through thousands of lines of output to find a single root cause now happens automatically. It is not rocket science — but it is some of the most time-consuming and draining work an engineer can be asked to do repeatedly.

Workflow definition: analyze-failures.md

Code Review Agent

When a merge request is opened, this agent performs a technical code review with the priorities of a senior engineer: security first, then correctness, then performance, then maintainability. It provides specific line references and concrete code examples for every issue it raises — not vague suggestions, but actual fixes. It reads prior review discussions before posting, avoids re-raising issues already addressed, and respects when a developer has explained an intentional design choice.

The result is consistent, high-quality review coverage on every merge request, with human reviewers free to focus on the issues that genuinely need their judgment.

The team is already pleased with the quality of the reviews — and this is only the beginning. A false positive costs nothing: a human reviewer reads it, disagrees, and moves on. The design ensures that bad agent output has no blast radius. And as human reviewers comment on agent-produced work, the agent learns and incorporates that feedback — improving with every iteration.

Workflow definition: code-review.md

The near-term impact is measured in cycle time. A feature request that previously took weeks to move from intake through product assessment, implementation, documentation, and review can be compressed to days, sometimes hours. That changes what a small team can realistically take on. The longer-term picture is equally straightforward: the system already delivers value today. If agents continue to improve, Hummingbird benefits more. If they plateau, the team continues to operate at a level that would otherwise require far more people. The infrastructure built now is not a bet on a technology that might not materialize — it is an investment that is already paying off, with optionality for more.

6 - Architecture Decision Records

Architecture Decision Records (ADRs) capture significant technical decisions made across the Hummingbird project, along with their context and consequences.

Records

ID Title Status Date
ADR-0001 Unified Observability Stack Implemented 2026-07-19
ADR-0002 AI-Assisted SDLC Workqueue Proposed 2026-08-24
ADR-0003 Metadata and Versioning Specification Proposed 2026-08-31

6.1 - ADR-0001: Unified Observability Stack

  • Status: Implemented
  • Date: 2026-07-19 (updated 2026-08-19)
  • Author: Robert Sturla
  • Jira: HUM-4790 (Epic), HUM-4791 (Design task)

Context

Hummingbird had no single place to view the health of builds, infrastructure, and services. Multiple disconnected monitoring and alerting implementations existed — CloudWatch dashboards, a custom cloudwatch-log-forwarder Lambda, an SNS AlertsTopic for consumer CloudWatch alarms, and per-service custom dashboards. Workloads span multiple clusters with no cross-cluster visibility. This created blind spots, duplicated effort, and inconsistent alerting behaviour.

One standardised observability stack is needed. No duplication. One toolchain, used everywhere, federated across clusters from day one.

Operational runbooks live in documentation/monitoring.md in the infrastructure repository. This ADR records the architecture decision.

Cluster Constraints

Hummingbird has no cluster-admin access on any cluster. All resources must be deployed within tenant-provisioned namespaces. Operator CRDs and RBAC grants are controlled by the cluster platform team — what is available differs between clusters.

This constraint drives a hub-spoke split architecture: the hub (MPP) runs the backends and Grafana, while spoke clusters run lightweight collectors that forward all telemetry to the hub.

Decision

Stack

Hub (MPP, hummingbird--monitoring-hub):

Signal Backend Storage
Metrics Mimir (monolithic) S3 via ObjectBucketClaim
Logs Loki (monolithic) S3 via ObjectBucketClaim
Collection Alloy Hub (YACE + textfile) N/A
Visualisation Grafana N/A
Alerting Mimir ruler + Alertmanager N/A

Spoke (every cluster with hummingbird--monitoring):

Signal Backend Storage
Metrics Alloy prometheus.scrape → hub Mimir remote-write N/A
Logs Alloy loki.source.kubernetes → hub Loki N/A
Collection Alloy + kube-state-metrics N/A
Visualisation None (use hub Grafana) N/A

Spoke clusters have no local backends. All storage, alerting, and visualisation is centralised on the hub. Each spoke forwards to both the staging and production hubs.

Distributed tracing (Tempo) is not deployed. See Future Work.

Alerting Architecture

Mimir evaluates PromQL alerting rules (ruler) and delivers notifications through its built-in Alertmanager. Grafana is not in the evaluation path.

Mimir ruler (PromQL) → Mimir Alertmanager → Slack / email
  • Rules are Git-provisioned in the infrastructure repository and mounted into Mimir as a ConfigMap.
  • All alerting runs on the hub. Spoke clusters do not run Alertmanager. Spoke alerting depends on network connectivity to the hub — if a spoke loses connectivity, alerts for that cluster are delayed until the connection is restored. This is an accepted tradeoff given the operator constraints on spoke clusters (see Cluster Constraints).
  • Staging Alertmanager receivers are empty (a blackhole) so staging does not page Slack or email.

Alert Routing

Severity Channel
critical Slack + email
info Slack
everything else Slack + email

Production only; staging is a blackhole.

Deployment Model

The hub runs the backends, Grafana, and Alloy Hub. Spoke clusters run Alloy and kube-state-metrics.

Hub (MPP):

  • Namespace: dedicated hummingbird--monitoring-hub namespace (tenant-provisioned)
  • Tenant-scoped: all resources deployed in namespaces owned by Hummingbird, no cluster-admin dependency
  • Mimir (monolithic): accepts Prometheus remote-write from spoke Alloy and Alloy Hub, stores TSDB blocks in S3, serves PromQL, and runs the ruler and Alertmanager. Grafana queries Mimir in-cluster (no nginx basic auth on the Service port).
  • Loki (monolithic): receives logs from spoke Alloy via authenticated Routes. Grafana queries Loki in-cluster.
  • Alloy Hub: a dedicated Alloy instance in the hub namespace. CloudWatch metrics are ingested with prometheus.exporter.cloudwatch (YACE), filtered to Hummingbird resources (app-code=RPRM-001). Credential-expiry metrics are ingested with the textfile collector from a CI-generated ConfigMap. Alloy Hub remote-writes to in-cluster Mimir (Service DNS; no Route or CA bundle).
  • Grafana: exposed via an internal Route. Authentication is Generic OAuth against gitlab.com (redhat/hummingbird group). GitLab Owner and Maintainer map to Grafana Admin, Developer to Editor, everyone else in the allowed group to Viewer. Datasources and dashboards live in grafana_data/ and are synced with grafana_data/deploy.sh. Grafana also has a CloudWatch datasource for ad-hoc browsing of AWS logs and metrics; that path is query-only and is not used for alerting or pod-log forwarding.
  • External access: Loki and Mimir Routes sit behind an nginx sidecar with HTTP basic auth. In-cluster Service access (Grafana, Alloy Hub) does not require auth.

Spoke (every cluster):

  • Namespace: hummingbird--monitoring
  • Alloy: collects container logs via loki.source.kubernetes and scrapes prometheus.io/scrape Services, including kube-state-metrics. Forwards logs to hub Loki and metrics to hub Mimir. Dual-writes to staging and production hubs.
  • kube-state-metrics: Kubernetes object metrics (deployments, pods, jobs, PVCs, resource quotas) with a curated allowlist.
  • No local Mimir, Loki, Grafana, or Alertmanager.

AWS Lambda Services

Some Hummingbird services (e.g. container-catalog) run as AWS Lambda functions. Observability for those services uses the existing AWS signals rather than an OpenTelemetry Lambda extension:

  • Metrics: Alloy Hub YACE scrapes curated AWS/Lambda metrics (Invocations, Errors, Duration, Throttles) into Mimir. Mimir alerting rules cover error rate and throttling.
  • Logs: Grafana queries CloudWatch Logs directly via the CloudWatch datasource. Pod logs are not involved; Lambda logs stay in CloudWatch.
  • Low-traffic utility Lambdas are covered by the same YACE allowlist when tagged app-code=RPRM-001.

Log Collection

Container stdout/stderr is tailed by spoke Alloy using loki.source.kubernetes (Kubernetes API; no DaemonSet, no sidecar, no ClusterLogForwarder). Logs are labeled with cluster, namespace, pod, container, and app, then dual-written to hub Loki.

This requires no application code changes. ClusterLogForwarder is available on some clusters but is not used.

High Availability

Hub:

Component Replicas Rationale
Mimir 1 Monolithic; brief downtime acceptable
Loki 1 Monolithic; brief downtime acceptable
Grafana 1 View-only; brief downtime acceptable
Alloy Hub 1 Stateless scrape; gap until next scrape is OK

Spoke:

Component Replicas Rationale
Alloy 1 Stateless forwarder; brief gap OK
kube-state-metrics 1 Stateless; brief gap OK

Spoke clusters have no stateful observability components. HA concerns are concentrated on the hub. Single-replica monolithic Mimir and Loki are an accepted tradeoff against the proposed Prometheus + Thanos HA pair: there is no existing Prometheus fleet to retrofit, and Alloy already remote-writes.

Retention

Environment Mimir Loki
Production 90d 90d
Staging 14d 14d

S3 is provisioned with ObjectBucketClaim. There is no separate warm/cold tier or multi-year archive.

Operator Availability

CRD availability was audited on 2026-08-09. Hummingbird has no cluster-admin access — only operator CRDs explicitly granted to tenant namespaces are usable. That audit is why this stack does not use Cluster Observability Operator (MonitoringStack / Thanos), LokiStack, TempoStack, ClusterLogForwarder, or an OpenTelemetry Collector as the live path:

  • LokiStack and TempoStack are not available to tenant namespaces on any cluster. Loki is a standalone Deployment. Tempo is deferred.
  • COO MonitoringStack is available on MPP but not on spoke, and would have required Prometheus + Thanos for a fleet we do not have. Mimir accepts remote-write directly from Alloy.
  • ClusterLogForwarder is available, but Alloy loki.source.kubernetes collects the same pod logs without a second operator.
  • OpenTelemetry Collector was deployed earlier and removed once Alloy wrote directly to Loki and Mimir.

Relationship to Existing Dashboard

The existing hummingbird-dashboard app serves two distinct roles:

  1. Data visualisation — CVE status charts, failed release counts, build metrics
  2. Operational workflows — click to rerun a failed release, interactive triage, actionable controls

This observability stack takes over data collection, storage, and visualisation (role 1). Grafana is the primary place for charts, time-series dashboards, and alert-driven visualisations. The dashboard app retains operational workflow features (role 2) — interactive actions that Grafana cannot provide.

The two systems complement each other:

  • The dashboard app can embed Grafana panels for richer visualisation without reimplementing charting
  • The dashboard app can query the same backends (Mimir, Loki) directly for data it needs to drive workflows
  • Over time, the dashboard app consumes data from the unified backends instead of maintaining parallel collection

Alerting (e.g. failed release Slack notifications) moves to Alertmanager. The dashboard app no longer needs its own alerting path.

What This Replaces

Previous Replaced by
cloudwatch-log-forwarder Lambda Spoke Alloy → hub Loki
SNS AlertsTopic + consumer CloudWatch alarms Mimir ruler + Alertmanager
Per-service custom dashboards Grafana dashboards
Dashboard app data collection Unified backends (Mimir, Loki)
Dashboard app alerting (Slack) Alertmanager
Dashboard app visualisations Grafana (embeddable in the dashboard app)

The dashboard app itself is not decommissioned — its operational workflow features (release reruns, interactive triage) remain. Only its data collection and visualisation responsibilities shift to the unified stack.

hummingbird-events-topic is the live event bus. It was not an alerting channel and is not replaced by this stack.

Consumer CloudWatch alarms and the log-forwarder were removed after the Mimir/Loki path was in place. Remaining CloudWatch use is intentional:

  • Grafana CloudWatch datasource for ad-hoc AWS log and metric browsing
  • CVE error-budget SLO dashboards and hummingbird-slo-alerts (no Grafana port yet)

Operations

Day-to-day endpoints, datasources, query examples, and onboarding a new cluster are documented in documentation/monitoring.md in the infrastructure repository.

Future Work

  • Distributed tracing: Tempo (or equivalent) for request and pipeline traces. Not deployed; traces are not a current signal.
  • CVE lifecycle tracing: instrument CVE analysis, RPM build, and container image pipelines with spans. Trace a CVE from detection → RPM fix → image rebuild → publish. Requires a trace backend and a schema (separate story).
  • Spoke local alerting: if operator access on spoke clusters changes, a local evaluation path could remove the hub dependency for spoke alerts. Not required today.

Alternatives Considered

Alerting: Grafana-only vs Alertmanager-only vs hybrid

  • Grafana-only: less mature routing, dedup, and silencing.
  • Hybrid Grafana + Alertmanager (original proposal): Grafana would evaluate rules across metrics, logs, and traces, then forward to Alertmanager. That is not what shipped: there is no Tempo, and log alerting is not required yet.
  • Mimir ruler + Alertmanager (chosen): PromQL rules and delivery in one component. Matches the metrics-first rollout. Applications can still POST to the Alertmanager API later if needed.

Metrics backend: Prometheus + Thanos vs Mimir

Prometheus + Thanos (MonitoringStack) was the original hub design. It fits retrofitting an existing Prometheus fleet. Hummingbird had no such fleet — Alloy already remote-writes. Mimir is one component for receive, storage, PromQL, ruler, and Alertmanager, instead of Prometheus + Thanos Sidecar + Store Gateway + Querier + a separate Alertmanager.

Logs: Sumo Logic vs Loki

Red Hat has a company-wide Sumo Logic license. However, the Sumo Logic Grafana plugin requires Grafana Enterprise or Grafana Cloud — it is not available in open-source Grafana. Loki integrates natively with open-source Grafana and keeps logs on-cluster.

Log collection: ClusterLogForwarder vs Alloy

ClusterLogForwarder is available on hub and spoke. Alloy loki.source.kubernetes collects the same pod logs and already runs on every spoke for metrics, so a second operator was not justified.

Spoke architecture: full stack vs collector-only

  • Full stack per cluster: each cluster runs its own backends and Grafana. Local alerting survives network partitions. Requires operator CRDs Hummingbird does not have on spoke.
  • Collector-only on spoke (chosen): Alloy + kube-state-metrics forwarding to the hub. Simpler spoke footprint. Tradeoff: spoke alerting depends on hub connectivity.

Lambda: OTel extension vs CloudWatch ingest

An OpenTelemetry Collector Lambda extension would export OTLP to the hub. That needs a network path from Lambda to MPP and a collector that no longer exists. YACE metrics into Mimir plus Grafana’s CloudWatch datasource for logs reuse AWS signals already produced by the functions.

HA: all components vs selective vs single replica

Full HA doubles cost with little benefit for view-only tools. The original proposal kept HA only on Prometheus and Alertmanager. The as-built stack uses single-replica monolithic Mimir and Loki; Alertmanager is in-process with Mimir. Brief downtime is accepted.

Consequences

  • Team must learn Grafana dashboarding and PromQL/LogQL basics
  • New hub services should expose Prometheus metrics (scrape annotation) and may add a Grafana dashboard under grafana_data/
  • New spoke services must be scraped by Alloy (scrape annotation) and emit logs to stdout/stderr
  • Spoke alerting depends on hub connectivity — no local alerting on spoke
  • Deploying a new hub cluster requires Mimir, Loki, Grafana, and Alloy Hub, plus ObjectBucketClaims
  • Deploying a new spoke cluster requires a hummingbird--monitoring namespace, Alloy, and kube-state-metrics pointing at both hubs
  • CloudWatch remains for Lambda logs (Grafana datasource) and CVE SLO error-budget dashboards; it is not used for pod-log forwarding or consumer alarms

6.2 - ADR-0002: AI-Assisted SDLC Workqueue

  • Status: Proposed
  • Date: 2026-08-24
  • Author: Michael Hofmann
  • Jira: HUM-6342 (spike)

Context

The containers and rpms repositories produce over 1,000 automated commits per week. The automation that drives this throughput is spread across multiple independent mechanisms: GitLab CI scheduled jobs running shell scripts, Renovate CronJobs in the infrastructure repo, AWS Lambda functions for MR approval, and manual developer workflows run from dev machines. Each mechanism creates and manages MRs with its own lifecycle, credentials, and error handling.

There is no unified system that tracks an MR from creation through approval to merge. The Hummingbird Agent already runs AI-powered failure analysis and code review on every MR, but cannot create or modify MRs. The security foundation for agent-authored MRs is designed in HUM-851 but not yet implemented.

This ADR documents the current state of MR automation across all repositories and entrypoints as discovery for a spike to design a unified workqueue-based system.

Scope

This document inventories what exists today. It does not prescribe the target architecture — that is the output of the spike. Design questions and patterns are included to inform the spike investigation.

Current State

Repositories

rpms (redhat/hummingbird/rpms): RPM package definitions. Each package lives in its own directory with a .spec file and metadata/<package>.json for upstream repo URL, CVE product mapping, and modification status. Fedora dist-git imports and upstream version bumps are the primary automated flows.

containers (redhat/hummingbird/containers): Container image definitions. Each image lives in images/<name>/ with Containerfiles and RPM lockfiles (rpms.lock.yaml). Lockfile refresh and Renovate dependency updates are the primary automated flows.

tools (redhat/hummingbird/tools): Lambdas, CLI tools, and the Hummingbird Agent framework. Contains mr-auto-approver, gitlab-event-forwarder, hummingbird-agent, and the CVE analysis pipeline.

infrastructure (redhat/hummingbird/infrastructure): Kubernetes manifests, AWS SAM templates, and deployment configuration. Hosts Renovate CronJob definitions and agent deployment manifests.

MR Creation Entrypoints

Entrypoint Repo Trigger What it creates
dist_git_update_multi_mr.sh rpms GitLab schedule (every 4h) One MR per package needing Fedora dist-git sync
upstream_update_multi_mr.sh rpms GitLab schedule One MR per package with a newer upstream release
rebuild_multi_mr.sh rpms Manual (operator-triggered) One MR per package needing rebuild
create_lockfile_update_mrs.sh containers GitLab schedule One MR per image with changed RPM lockfiles
ci/create_mr.sh (metrics) containers GitLab schedule (daily) Single chore MR for metrics report
Renovate (rpms) rpms K8s CronJob (hourly) Tekton pipeline digest and Konflux release-catalog updates only
Renovate (containers) containers K8s CronJob (hourly) Dependency updates, lockfile maintenance, pipeline migration
Renovate (tools/infra/k8s-test) tools/infra/k8s-test K8s CronJob (hourly) Dependency updates
CVE analysis (advisory_handler) CEE advisories K8s CronJob (30 min) Advisory MRs; handles rebase on conflicts
/cve skill rpms Manual (Cursor/Claude) Fix MRs from staged CVE analysis data in Jira
gitlab_sync tools Various File sync MRs with poll, rebase, and best-effort approve
Manual dev workflows rpms/containers Human on dev machine Package updates, CVE fixes, spec changes

MR Creation Mechanism: Git Push Options

All shell-script-based MR creation uses git push options, not the GitLab Merge Requests REST API. The push options control MR creation, title, description, labels, draft status, and merge-when-pipeline-succeeds (MWPS):

git push \
  --push-option merge_request.create \
  --push-option "merge_request.title=chore(rpms): Update foo to 1.2.3" \
  --push-option merge_request.remove_source_branch \
  --push-option merge_request.merge_when_pipeline_succeeds \
  origin HEAD:refs/heads/chore/upstream-update-foo

Approval is always a separate step — never part of the creation scripts.

MR Lifecycle and Approval Flows

GitLab Project Settings (containers and rpms)

Setting Value
Approvals required 1 (any_approver); author cannot self-approve
Pipeline must succeed Yes
Discussions resolved Required
Jira required Yes (prevent_merge_without_jira_issue)
Merge trains / merge pipelines Off (Konflux external statuses must stay blocking)
Merge method Merge commit
Auto-merge MWPS via merge_request.merge_when_pipeline_succeeds push option

Bot Identities

Role Identity Token type
Create chore MRs chore-mr Project access token
Approve chore MRs (CI) chore-mr-approval Separate project access token
Create lockfile MRs lockfile-update Project access token
Approve lockfile MRs (CI) lockfile-update-approval Separate project access token
Renovate + Konflux PAC pipelines-as-code MintMaker token
Lambda approvals Per-project tokens Separate project access tokens

Approval Paths

Three parallel approval mechanisms exist, with overlapping scope:

1. CI approval jobs — delayed jobs in each repo’s .gitlab-ci.yml:

  • rpms chore_mr_approval: gates on check_konflux_statuses.py (polls every 60s up to 2 hours for Konflux statuses to appear and succeed), then approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. Triggered for author chore-mr or pipelines-as-code on chore/* branches.

  • containers chore_mr_approval: same author/branch gate, approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. No Konflux polling. allow_failure: true.

  • containers lockfile_update_approval: triggered for pipelines-as-code on lock-file-maintenance(-vulnerability)? branches. Real when: delayed / start_in: 10 minutes. No Konflux polling.

2. mr-auto-approver Lambda — event-driven via GitLab webhooks:

GitLab webhook → gitlab-event-forwarder → SNS → mr-auto-approver Lambda

Approves only (does not set MWPS). Rejects forks. Checks Konflux statuses where configured.

Project Allowed authors Branch patterns Konflux check
containers PAC + chore-mr renovate/.*, chore/.* Yes
rpms PAC + chore-mr chore/.* only Yes
tools PAC renovate/.* (deny renovate/redhat-catalog/.*) No
infrastructure renovate token renovate/.* No

The Lambda is intended to replace CI approval jobs; both currently run in parallel for chore/lockfile paths.

3. Renovate automerge — Renovate sets automerge: true in its MR configuration, but GitLab still requires a non-author approval before merge. For rpms Renovate MRs, neither CI approval jobs nor the Lambda cover renovate/.* branches — these MRs are stuck until a human approves.

Konflux Status Gate

External commit statuses prefixed Konflux kflux-prd-rh03 / are set by Konflux PipelineRuns triggered via Pipelines-as-Code. The check_konflux_statuses.py script (rpms) polls these statuses:

  • Missing → wait (poll continues)
  • Failed → do not approve
  • Invoked/running/pending/success → approve (MWPS then waits for final green, including Testing Farm via IntegrationTestScenarios)

Lifecycle by MR Type

RPMs dist-git clean (chore/dist-git-update-<pkg>): schedule creates → MWPS at creation → GitLab CI + Konflux builds + Testing Farm → CI approval job (poll Konflux, up to 2h) + Lambda → merge. Typical: tens of minutes to hours.

RPMs dist-git conflict: same branch, but MR is created as draft with no-test label and no MWPS. Requires human conflict resolution (or claude_resolve_conflict.sh), then remove draft/no-test and re-trigger.

RPMs upstream (chore/upstream-update-<pkg>): same as dist-git clean. Uses OIDC AWS lookaside upload for tarballs.

RPMs rebuild (chore/rebuild-<pkg>): same approval path. Typically operator-triggered, not scheduled.

RPMs Renovate (renovate/…): Renovate sets automerge, but neither CI approval job nor Lambda covers renovate/.* branches on rpms → stuck until human approves. The renovate-babysit agent workflow posts a triage note but does not approve or merge.

Containers Renovate lockfile (lock-file-maintenance): lockfile_update_approval (10 min delay) + Lambda (renovate/.* with Konflux check) → Renovate automerge / MWPS. Typical: 30–90+ minutes.

Containers other Renovate: Lambda covers renovate/.*. No lockfile_update_approval unless branch matches lockfile pattern.

Containers metrics report (chore/metrics-report): daily schedule → MWPS → chore_mr_approval + Lambda.

Containers CI lockfile (rpm-lockfile-updates/<image>): schedule is currently active: false. Approval path is incomplete — CI rule matches lock-file-maintenance, Lambda does not match rpm-lockfile-updates/.*.

Shell Script Automation Detail

rpms create_mr.sh (shared helper)

Used by all rpms multi-MR scripts. Expects commits already on HEAD.

Inputs: --branch NAME (required), --title TEXT (required), optional --description, --auto-merge, --draft, --label NAME (repeatable).

Algorithm:

  1. Count commits ahead of target branch; if 0 → exit 0.
  2. Dedup: git ls-remote --heads for branch name. If present → exit 2 (“MR already exists”).
  3. Create branch, push with MR push options.
  4. Parse push stdout for MR URL; if missing → exit 1.

Exit code contract: 0 = MR created, 2 = remote branch exists (skip), 1 = failure.

Limitation: stale remote branch from a closed MR still triggers exit 2. Does not update existing MR/branch content. No --force-with-lease.

rpms dist_git_update_multi_mr.sh

Creates one MR per package needing Fedora dist-git sync.

Modes: production (cwd = repo, always create MRs) vs clone dry-run vs clone + --create-mrs.

Algorithm:

  1. Select metadata/*.json files; apply --clean-only / --modified-only filter; skip native packages.
  2. Record START_COMMIT on target branch.
  3. For each package: run ./ci/dist_git.py update PACKAGE. Exit 0 + new commit = clean update. Exit 2 + new commit = conflict update.
  4. Collect all new commits, hard-reset target to START_COMMIT.
  5. Per commit: parse subject (^(Update|Sync) ([^ ]+) from ... to ...), create branch chore/dist-git-update-<pkg>, cherry-pick. Conflict → draft + no-test, no auto-merge. Clean → --auto-merge.
  6. Call create_mr.sh.

Branch convention: chore/dist-git-update-<pkg>. Title: chore(rpms): <commit subject> or CONFLICT: chore(rpms): <subject>.

rpms upstream_update_multi_mr.sh

Same multi-MR / cherry-pick pattern as dist-git, for upstream version bumps.

Algorithm: runs ./ci/check_upstream_versions.py check --update --sign-off, collects commits, splits into per-package MRs. Always --auto-merge. Branch: chore/upstream-update-<pkg>.

rpms rebuild_multi_mr.sh

Does not generate rebuilds. Expects local commits from dist_git.py rebuild or rebuild-rev-deps, then splits them into per-package MRs.

Unique feature: --force flag deletes existing remote branch before create, working around stale branches from closed MRs. Branch: chore/rebuild-<pkg>.

containers create_lockfile_update_mrs.sh

Creates or updates one MR per image group when RPM lockfiles changed. Prerequisite: lockfiles already refreshed (make all-host FORCE_REFRESH=true).

Algorithm (process_single_image):

  1. Branch: rpm-lockfile-updates/<image>.
  2. For each distro/variant lockfile: compare to HEAD after stripping .arches[].packages[].url and .arches[].source[].url via yq (URL churn ignored).
  3. If no meaningful changes → skip.
  4. If push-mode ≠ force and origin/<branch> exists: compare to remote; if identical → skip.
  5. git switch --force-create branch, commit lockfiles.
  6. Push --force-with-lease + MR push options.

Key difference from rpms: uses --force-with-lease to update existing branches in place. Dedup compares lockfile content, not just branch existence.

containers create_mr.sh (simpler helper)

Used by metrics report. Creates from working tree (not existing commits). git add --all, commit, --force-with-lease push. No dedup — overwrites existing branch.

Deduplication Summary

Script Existing branch Content refresh
rpms create_mr.sh Skip (exit 2) Never
rpms multi-MR scripts Via create_mr.sh Never (unless rebuild --force)
containers lockfile Skip if remote content matches Yes (--force-with-lease)
containers create_mr.sh Overwrites Yes

Hummingbird Agent Framework

The Hummingbird Agent is an event-driven LLM agent deployed on OpenShift that processes GitLab events via SQS.

Architecture

  • Event-driven: consumes SQS events (pipeline failures, MR events, slash commands). No REST API — triggered only by SQS or CLI.
  • Workflow-driven: investigation logic defined in markdown files as LLM system prompts. New behaviors do not require code changes.
  • Sandboxed execution: all LLM commands run in isolated containers (Podman/K8s/KubeVirt) with no network access and no credentials.
  • Multi-model: supports Gemini and Claude via Vertex AI.
  • Token separation: model tokens (read-only, Reporter-level) are distinct from orchestrator tokens (write-capable, used only by deterministic code outside the sandbox).
  • Single output action: post_gitlab_note is the only way the agent affects the outside world. It cannot create, modify, or merge MRs.

Current Workflows

analyze-failures: triggered on pipeline failure. Bulk-fetches GitLab CI job logs, Konflux PipelineRun/TaskRun metadata and logs, and Testing Farm results. Groups failures by root cause. Posts a structured MR note with collapsible forensic evidence and links.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_diff, gitlab_get_commit_statuses, konflux_list_pipelineruns, konflux_get_pod_log, tf_get_results, tf_get_test_log, sandbox_exec
  • Pattern: bulk metadata fetch → selective deep log pulls → grouped analysis → structured note
  • Flexibility: medium-low (tightly coupled to Konflux/Tekton/TF pipelines)

code-review: triggered on MR open/update. Reads the unified diff, prior review discussions, and per-repo rules. Posts review findings with severity, category, and concrete code fix examples.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, gitlab_get_mr_discussions, gitlab_get_file_at_ref, sandbox_exec
  • Pattern: diff-first analysis, discussion-aware follow-up (avoids re-raising resolved issues), per-project rules overlay
  • Per-repo rules: workflows/repo-rules/<group>-<repo>.md files provide project-specific review guidance (e.g., hummingbird-rpms.md documents package metadata conventions)
  • Flexibility: high (generic for any GitLab MR; domain knowledge plugs in via repo-rules/)

renovate-babysit: triggered on successful Renovate MR pipeline. Classifies the update as safe or risky based on diff scope, upstream changelogs (via web search), and local API usage. Posts a triage verdict.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, web_search, gitlab_get_repo_archive, sandbox_exec
  • Pattern: diff scope classification, embedded changelog first then web search, archive + grep for call-site impact
  • Flexibility: medium-high (ecosystem-agnostic risk matrix; does not approve/merge — triage note only)

What the Agent Has That Maps to a Workqueue

  • SQS FIFO queue with message grouping (serializes per session)
  • Session persistence (S3 context.json)
  • Config hot-reload in serve mode
  • Per-project token separation (model / action / orchestrator)
  • Rate limiting (per-workflow per-MR)
  • Data source abstraction (GitLab, Konflux, Testing Farm)
  • Auto-resolve on push/success

What the Agent Lacks for MR Lifecycle Management

  • No MR creation capability (no git write operations, no push access)
  • Read-only GitLab tokens for the model
  • No workqueue for MR lifecycle state (events are fire-and-forget)
  • Single output action (post_gitlab_note only)
  • No safe-outputs pattern (no validation of proposed writes)
  • No deterministic fallback paths for well-understood operations
  • No request/response API (SQS consumption and CLI only; external services cannot invoke a workflow and retrieve structured results)

Event Bus

The existing event pipeline could serve as the backbone for a workqueue:

GitLab MR / note / pipeline webhooks
  → gitlab-event-forwarder (API Gateway → Lambda)
  → SNS hummingbird-events-topic
  → mr-auto-approver Lambda (approve safe bot MRs)
  → hummingbird-agent SQS (analyze / review / babysit)
  → hummingbird-status SQS (pipeline tracking)

mr-auto-approver is a pure rule-based approver — no LLM involvement. hummingbird-mr-human-tracker detects human fixes on Renovate/bot MRs and tracks them in Jira epics.

Security Foundation: HUM-851

HUM-851 (“Enable Secure Agent MR Creation”) is an existing epic (status: New) that designs the security posture for agents creating MRs.

Core Principle

Instead of protecting CI secrets from agent-authored MR pipelines, eliminate all secrets from containers/rpms/tools CI entirely. Move all secret-dependent jobs to the infrastructure repo’s CI. With zero secrets, a compromised agent MR pipeline can only access CI_JOB_TOKEN (single-project-scoped for bot users) and the runner’s network.

What Moves to Infrastructure Repo CI

Job Current project Current trigger
deploy_hugo_tag containers, rpms, tools On merge to main
lockfile_update containers Scheduled
Lockfile/chore approval containers, rpms After MR CI passes
Chore MR creation containers, rpms Scheduled
dist_git_update rpms Scheduled
upstream_version_update rpms Scheduled
metrics_report containers Scheduled
update_quay_description containers Scheduled/on merge

This migration directly enables centralized MR creation.

Target Security Posture

  • Zero CI secrets in target projects
  • Restricted runners with egress filtered to package registries only
  • Agent never holds write tokens — writes happen in deterministic orchestrator code
  • Diff safety validation — blocklist prevents changes to CI config, .tekton/, dependency files
  • Human review required for every agent MR, auto-labeled agent-generated
  • Rate limiting per-project and global; auto-close stale MRs
  • Monitoring: every agent MR logged and Slack-notified

Task Breakdown

Task Status What it does
HUM-852 New Threat model and security design
HUM-853 New Move ops jobs to infra CI
HUM-854 Closed K8s CronJob/Job manifests (done)
HUM-855 New Validate and remove CI variables
HUM-856 New Restrict runner egress
HUM-857 New MR creation orchestrator action
HUM-858 New Rate limiting and abuse prevention
HUM-859 New Monitoring and incident response

Renovate Deployment

Deployed as Kubernetes CronJobs in the infrastructure repo (kubernetes/renovate/20-cronjobs.yml.j2):

  • 5 repos: containers, infrastructure, k8s-test-pipeline, rpms, tools
  • Each gets its own CronJob running hourly with concurrencyPolicy: Replace
  • Uses MintMaker image (MINTMAKER_IMAGE)
  • containers/rpms use bash wrappers and custom config files
  • Custom env variables for RPM lockfile cache, Rust/Cargo paths

CVE Analysis Pipeline

The CVE analysis pipeline is a complex, mature system. The relevant interface points for a workqueue are:

  1. hummingbird-cve-analysis CronJob (K8s, every 30 min prod) — runs cve_analysis --resolve which triages Jira Security tickets, transitions statuses, creates advisory MRs, manages labels.
  2. collect_cve_dashboard — runs after analysis, posts lifecycle data to the dashboard API.
  3. /cve skill — reads staged CVE analysis data from Jira tickets and creates fix MRs in the rpms repo.

The workqueue integration point: when /cve or CVE analysis determines an MR should be created, that request becomes a work item.

RPM-to-Container Pipeline Flow

A single RPM spec change triggers a cascade that spans both repos and multiple build systems. The full flow is documented in the RPM Pipeline and Image Pipeline docs.

RPM spec change (rpms repo)
  → MR validation (GitLab CI + Konflux build + Testing Farm)
  → Merge to main
  → Konflux RPM build (Tekton PipelineRun per package, mock hermetic)
  → RPM signing (Kerberos-based)
  → Publish to Pulp (packages.redhat.com)
  → Lockfile update detects new RPM versions (containers repo)
  → Lockfile MR created (Renovate or create_lockfile_update_mrs.sh)
  → Container build (Konflux, multi-arch)
  → Container testing (Testing Farm + K8s tests)
  → Enterprise Contract validation (Conforma)
  → Release to Quay.io

Key observations for the workqueue:

  • The cascade is implicit — there is no explicit trigger from RPM publish to container lockfile update. Lockfile updates discover new RPMs by polling Pulp (via make all-host FORCE_REFRESH=true or Renovate’s RPM lockfile maintenance).
  • A single RPM change can fan out to many container images (any image that includes that package in its lockfile).
  • The RPM pipeline has 5 stages (spec change, MR validation, build, signing, publishing). The container pipeline has 6 stages (source templates, generation, build, testing, Enterprise Contract, release).
  • Testing Farm runs integration tests on merge requests only, not on main branch builds. Container tests include reverse dependency testing (rebuilding dependent images locally in the test environment).
  • Enterprise Contract (Conforma) validates supply chain security, hermetic builds, and policy compliance before release.

Decision

To be determined by the spike investigation. This section will be updated with the chosen architecture.

The spike should investigate a unified workqueue-based system where:

  • Queue feeders replace or wrap the current scheduled scripts and Renovate outputs
  • Queue processors handle MR creation, approval, rebasing, and merge through deterministic code
  • Agentic capabilities (review, patch generation, triage) are invoked as composable services with structured output
  • The security model follows HUM-851’s design: agents reason read-only, orchestrators execute writes

Design Patterns and Prior Art

GitHub Agentic Workflows (gh-aw)

gh-aw provides a reference architecture for AI-assisted repository automation with security guardrails:

  • Read-only agent execution: agents run with no write permissions, no secrets
  • Safe Outputs: agents buffer intended write operations as structured JSON artifacts; separate permission-controlled jobs validate and execute them
  • Threat detection: a separate AI agent scans agent outputs before write execution
  • Plan-level trust: trust is embedded in the workflow definition (what the agent is allowed to do), not in the agent’s runtime decisions

Key pattern: separate reasoning (agentic, sandboxed, read-only) from execution (deterministic, permission-controlled, auditable).

Existing Hummingbird Patterns

  • SQS FIFO queue with message grouping: already serializes work per session; can be extended to serialize per-MR
  • Workflow-as-prompt: markdown files define behavior; new behaviors do not need code changes
  • Data source abstraction: GitLab API access is modular
  • Token tier separation: model read tokens, action write tokens, orchestrator tokens — maps directly to the safe-outputs pattern
  • Sandbox isolation: reusable for agentic review/analysis tasks

Agent as Composable AI Service

One approach to bridge the agent framework and a workqueue is to evolve the agent from a self-contained event processor into a building block that non-AI services can call:

  • Request/response API: HTTP endpoint accepts workflow requests, returns a UUID; caller polls for structured JSON results. The caller (e.g., a workqueue herder) applies deterministic logic to decide what to do with the output.
  • Structured output: agent produces typed JSON (not just a GitLab note), enabling callers to process results programmatically.
  • Safe-output actions: new action types beyond post_gitlab_note (e.g., create_mr, update_mr) with per-workflow path allowlists, diff size limits, and branch naming constraints. The orchestrator validates each action against the workflow’s configured constraints before execution.

Open Questions

  1. Queue technology: extend SQS FIFO (already in use by the agent) or use a different queue (K8s-native, Redis, PostgreSQL)?
  2. Processor deployment: K8s Deployments polling SQS, GitLab CI jobs triggered by webhooks, Lambda functions, or a combination?
  3. Renovate strategy: keep Renovate as a separate MR creator and add a herder, or replace Renovate’s MR creation with the workqueue?
  4. Write credentials: per-repo project access tokens (current model) or a single bot account? How does HUM-851’s “eliminate CI secrets” model interact with workqueue processors that need write access?
  5. Safe outputs validation: simple schema validation, deterministic allowlist checks, or full threat detection (as in gh-aw)?
  6. Review scope: which MRs need AI review before merging vs which can be auto-merged with deterministic checks only?
  7. State tracking: queue, GitLab labels, a database, or the existing hummingbird-status PostgreSQL?
  8. Agent evolution: request/response API, safe-output actions, or hybrid? How does this relate to HUM-857?
  9. HUM-851 relationship: is the workqueue spike a superset of HUM-851, or a separate epic that depends on HUM-851 Phase 1?

Consequences

To be updated after the spike investigation determines the architecture.

Known consequences of any workqueue approach:

  • Centralizing MR creation provides a single point for audit, rate limiting, and monitoring
  • Moving secret-dependent jobs to the infrastructure repo (per HUM-851) is a prerequisite regardless of workqueue design
  • The existing CI approval jobs, Lambda approver, and Renovate automerge have overlapping scope that must be reconciled
  • RPMs Renovate MRs currently have no automated approval path — any design must address this gap
  • The agent’s current workflows (failure analysis, code review, Renovate babysit) complement the workqueue and should not be disrupted

File and System Inventory

Infrastructure repo

Path Contents
kubernetes/renovate/ Renovate CronJob definitions (5 repos)
kubernetes/hummingbird-agent/ Agent deployment manifests (8 files)
kubernetes/hummingbird-cve-analysis/ CVE analysis CronJob
aws/hummingbird-agent/ Agent SQS/SNS SAM templates

Tools repo

Path Contents
hummingbird-agent/ Agent framework (loop, sandbox, models, data sources, workflows)
hummingbird-agent/workflows/ Workflow definitions (analyze-failures, code-review, renovate-babysit)
hummingbird-agent/workflows/repo-rules/ Per-project review rules (hummingbird-rpms.md)
hummingbird-cve-analysis/ CVE analysis pipeline (preserve)
hummingbird-dashboard/ Dashboard web app (preserve)
mr-auto-approver/ Lambda: rule-based MR approval
hummingbird-mr-human-tracker/ Lambda: tracks human fixes on bot MRs
gitlab-ci/gitlab_sync/ File sync MR creation with rebase
gitlab-event-forwarder/ Webhook → SNS bridge
hummingbird-events-topic/ SNS topic infrastructure

RPMs repo

Path Contents
ci/create_mr.sh Shared MR creation helper (push options, dedup by branch)
ci/dist_git_update_multi_mr.sh Dist-git sync: commit-all → reset → cherry-pick per package
ci/upstream_update_multi_mr.sh Upstream version bumps: same cherry-pick pattern
ci/rebuild_multi_mr.sh Rebuild MR splitter (with --force for stale branches)
ci/dist_git.py CLI for import/update/sync/rebuild operations
ci/check_konflux_statuses.py Polls Konflux external statuses before approval
ci/claude_resolve_conflict.sh AI-assisted conflict resolution for dist-git MRs
metadata/<package>.json Per-package metadata (upstream URL, CVE mapping, status)

Containers repo

Path Contents
ci/create_lockfile_update_mrs.sh Per-image lockfile MR creation (force-with-lease, content dedup)
ci/create_mr.sh Generic chore MR helper (force-with-lease, no dedup)
ci/internal/shared_lib.sh Shared utilities (get_distro_variants, build helpers)
images/<name>/ Per-image Containerfiles and RPM lockfiles

6.3 - ADR-0003: Metadata and Versioning Specification

  • Status: Proposed
  • Date: 2026-08-31
  • Author: Brent Baude
  • Jira: HUM-6212

Context

The metadata file at metadata/<package>.json and the package directory at rpms/<package>/ form a pair, one per package. The package directory holds the build input: the spec file, patches, and the sources checksum manifest that mock consumes to produce RPMs. This specification does not change anything about that directory.

The metadata file is the machine-readable record of a package’s relationship to Fedora and upstream. Automation reads it to decide what it may do: whether dist_git.py update may overwrite the package directory, which upstream project check_upstream_versions.py tracks, and how CVE analysis maps the package to its upstream identity.

The metadata file does not duplicate build inputs, and it does not record the release currently shipped: the spec’s Release: line is the sole authoritative value for what ships, and is what NVR computation uses (see Computing the built NVR). For packages on Fedora’s version, fedora.release also appears in metadata — but only as the base value the spec’s Release: is built from, never as a substitute for it.

The two locations are updated independently; neither is derived from the other. They must stay consistent by convention, not by construction — modification_status is correct only if it accurately reflects whether the package directory actually diverges from Fedora’s import.

This document normatively defines the target structure of metadata/<package>.json and how release and versioning behave as a function of modification_status. It describes the target metadata shape, not what is present on disk in metadata/*.json today. Bringing existing metadata in line with this specification is separate, tracked work.

Decision

Package states

Every package occupies exactly one of three states, describing its relationship to Fedora at a given moment. A state is not itself a metadata field — it is what modification_status means in combination with which version the package currently ships. The rest of this specification refers to these states by name, so they are defined here first.

A package’s state follows from its modification_status, which is itself defined by the package’s relationship to a Fedora origin. modification_status is one of three values:

  • clean — the package has a Fedora origin, and currently matches that Fedora import.
  • modified — the package has a Fedora origin, but currently diverges from it — either through local changes while still on Fedora’s version, or by having moved ahead to a newer upstream version.
  • independent — the package has no Fedora origin at all.

For a modified package, the state further depends on whether it currently ships Fedora’s exact version or has moved ahead to a newer upstream version. The fedora object’s shape — complete or partial — records which of those is the case; it does not itself define the state.

The table below names all three states by the version they currently ship, not just by modification_status, since modified alone is ambiguous between the two Fedora-origin cases:

Version currently shipped modification_status
Fedora’s version clean or modified
An upstream version ahead of Fedora modified only
No Fedora origin (independent) independent

Field reference

This is the complete field-by-field reference for metadata/<package>.json — every field consumed by tooling in this repository, not only the fields that determine modification_status or the Fedora relationship. Fields are listed in four groups, in order: package identity and state (version through modification_reason), the Fedora-origin group (fedora and its sub-fields), fields that configure upstream-version tracking, and fields unrelated to either Fedora or upstream tracking (cve_product, version_transform, fix_status). The “Presence” column states the condition under which a field appears in the JSON; fields are omitted entirely rather than set to null or an empty value when their condition does not hold.

Field Type Presence Description
version string Always The version currently shipped.
modification_status enum: clean, modified, independent Always Whether the package matches its Fedora import unmodified (clean), carries local changes (modified), or has no Fedora origin at all (independent).
modification_reason string Only in the modified state Why the package carries local changes. Absent for clean and independent packages.
fedora object In the clean state or either sub-case of the modified state Everything about the Fedora build this package tracks or last tracked. Absent entirely in the independent state.
fedora.git_url string Whenever fedora is present The Fedora dist-git repository URL.
fedora.branch string Whenever fedora is present The Fedora dist-git branch (e.g. rawhide).
fedora.sha string Whenever fedora is present The exact Fedora commit imported or last synced.
fedora.release string Only in the on-Fedora-version state (clean, or modified while still on Fedora’s version) The release number of the Fedora build tracked, dist tag stripped. Absent once the package moves to an upstream version ahead of Fedora.
upstream_repo string (git URL) Always The canonical upstream project repository. Falls back to the Fedora dist-git URL when no independent upstream repository exists.
upstream_branch string Optional — independent of state Pins one package’s upstream line when several packages share a single upstream_repo (e.g. an nodejs2x-style family).
version_from_ref object: {"type": "commit-date"} Optional — independent of state; requires upstream_repo Configures dist_git.py update’s fixed-ref pin-refresh logic to compute a gorget source-pipeline’s --version string automatically from a newly-pinned commit, instead of requiring manual resolution. commit-date is currently the only supported type.
track_upstream string: "latest" or a version prefix (e.g. "1.26") Optional — independent of state Enables check_upstream_versions.py tracking of this package; a version prefix constrains which upstream releases are accepted.
release_monitoring_project_id integer or string Optional — independent of state; only meaningful paired with track_upstream The release-monitoring.org (Anitya) project ID or name check_upstream_versions.py queries.
version_source string: currently only "gitlab_tags" Optional — independent of state Selects an alternate upstream-version source for check_upstream_versions.py instead of Anitya.
tag_strip_prefix string, default "v" Optional — independent of state Used with version_source: "gitlab_tags" to strip a tag-name prefix before comparing versions.
version_suffix_strip string Optional — independent of state A suffix check_upstream_versions.py strips from an Anitya-reported version before comparison and update (e.g. "-RELEASE").
upstream_version_transform string Optional — independent of state A named transform check_upstream_versions.py applies to normalize an Anitya-reported version into RPM version syntax (e.g. openjdk_to_rpm). A distinct field from version_transform below, with a different consumer — do not conflate the two.
source_availability_check string Optional — independent of state A named checker (registered in SOURCE_AVAILABILITY_CHECKERS) that HEAD-probes a candidate source URL before check_upstream_versions.py accepts that version.
cve_product string, or list of strings Optional — independent of state A CVE vendor/product override, independent of Fedora or upstream tracking.
version_transform string Optional — independent of state A version-mapping rule consumed by CVE analysis tooling in the tools repository (e.g. dotnet_sdk_to_runtime). Not consumed by anything in this repository’s own ci/ scripts.
fix_status integer Optional — independent of state Not consumed by any tooling in this repository today. Carried over from history and likely stale — not a field to use for new work.

Computing the built NVR

The built RPM is identified by its NVR: Name, Version, and Release. Name is the spec/package name and is unaffected by any of this — only Version: and Release: vary with state.

Metadata version and the spec Version: line always match — metadata stores it separately not because the two can diverge, but because reading Version: reliably from every spec (which may build it from macros or indirection) isn’t practical, so metadata is the field automation trusts instead. Release: works the other way: the spec Release: line is the sole authoritative value for what ships, in every state. fedora.release, when present, is only the base value Release: is built from — never a substitute for it.

The table below shows, for each state, where the spec’s Version: and Release: lines come from. The prose beneath it gives the precise rebuild-increment rules that don’t compress into a table cell.

Version currently shipped Spec Version: Spec Release:
Fedora’s version (clean or modified) Matches metadata version fedora.release plus %{?dist}, with an optional trailing .N for local rebuilds
An upstream version ahead of Fedora (modified only) Matches metadata version Starts at 0.1%{?dist} the first time the package moves ahead of Fedora; local rebuilds increment the trailing .N directly from the spec’s own current value
No Fedora origin (independent) Matches metadata version A one-time human-chosen base (typically 1 or 0.1) plus %{?dist}; local rebuilds increment the trailing .N directly

On Fedora version (clean or modified): fedora.release holds Fedora’s confirmed release number, dist tag stripped. The spec Release: is that value plus %{?dist}, with an optional trailing .N if the package has been locally rebuilt without a source change since the last import or sync. dist_git.py rebuild computes the next .N by comparing the spec’s current Release: against fedora.release: if they match exactly, this is the first local rebuild since import or sync, and .1 is appended; if the spec already carries a trailing .N ahead of fedora.release, that trailing number is incremented.

On upstream version (always modified): No fedora.release, and no other metadata field records a base value. The spec uses Release: 0.1%{?dist} the first time a package moves ahead of Fedora. Every subsequent local rebuild increments the trailing dot-number directly from the spec’s own current Release: value (0.1 → 0.2 → 0.3, and so on) — never by re-deriving it from a stored metadata value, because none exists. This is what makes the bug class behind HUM-5182 structurally impossible: that bug occurred because a stored placeholder value could disagree with the spec’s actual current Release:; with nothing stored to disagree with, the mis-increment cannot happen.

Independent: The same rule as the on-upstream-version case applies — no metadata field records a base value, and the spec Release: is authoritative. Rebuilds increment its trailing .N directly. The only difference is the very first value chosen when the package is added (typically 1 or 0.1) is a one-time human decision, not derived from anything.

Validation rules

Every rule below is a single, unconditional statement: given the field values present, exactly one outcome is correct. None of them carry a qualifier like “should,” “generally,” or “in most cases,” and none fold an exception into the wording — a rule that needs an exception is really two rules, stated separately. The rationale for why each rule holds lives elsewhere in this document (Package states, Field reference, Computing the built NVR); this section exists so a validator can implement each line as a direct check with nothing left to interpret.

  1. version is present.
  2. modification_status is present.
  3. upstream_repo is present.
  4. modification_status is clean, modified, or independent.
  5. version_source, when present, is gitlab_tags.
  6. source_availability_check, when present, is a name registered in SOURCE_AVAILABILITY_CHECKERS.
  7. The top-level field release does not appear.
  8. If modification_status is clean, fedora is present.
  9. If modification_status is modified, fedora is present.
  10. If modification_status is independent, fedora is absent.
  11. If fedora is present, it includes git_url, branch, and sha.
  12. If modification_status is clean, fedora.release is present.
  13. If modification_status is modified, fedora.release is present if and only if version equals the version Fedora ships at fedora.sha.
  14. fedora.release, when present, does not include a dist-tag suffix (e.g. .fc42, .el9).
  15. If modification_status is modified, modification_reason is present.
  16. If modification_status is clean or independent, modification_reason is absent.
  17. If track_upstream is present and version_source is absent, release_monitoring_project_id is present.
  18. If release_monitoring_project_id is present, track_upstream is present.
  19. If version_source is gitlab_tags, release_monitoring_project_id is absent.
  20. If tag_strip_prefix is present, version_source is gitlab_tags.
  21. The spec’s Version: line, resolved through macro expansion, equals metadata version.
  22. If version_from_ref is present, upstream_repo is present.
  23. version_from_ref.type, when present, is commit-date.

Examples

Metadata for a package on Fedora version, clean:

{
  "version": "5.4.3.0",
  "modification_status": "clean",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/dnf5.git",
    "branch": "rawhide",
    "sha": "f174d0fcedc78a34d74b800697862b93683ad5e0"
  },
  "upstream_repo": "https://github.com/rpm-software-management/dnf5"
}

Metadata for a package on Fedora version, modified:

{
  "version": "5.3.15",
  "modification_status": "modified",
  "modification_reason": "Added gorget source-pipeline.yaml (HUM-5841; pattern from HUM-4622): a fresh fetch of Source0's bash-5.3.tar.gz no longer matches the recorded checksum -- upstream re-published the base tarball (translation file + build stamp only, GPG signature re-verified against the already-trusted key) -- so Fedora's cache no longer serves bytes matching the corrected checksum",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/bash.git",
    "branch": "rawhide",
    "sha": "2768211b5135c7169f513965d79e9f89e0ca6124"
  },
  "upstream_repo": "https://git.savannah.gnu.org/git/bash.git"
}

Metadata for a package on an upstream version ahead of Fedora, modified:

{
  "version": "0.9~rc4",
  "modification_status": "modified",
  "modification_reason": "update to 0.9rc4",
  "fedora": {
    "git_url": "https://src.fedoraproject.org/rpms/avahi.git",
    "branch": "rawhide",
    "sha": "195919c9f9a8dff0e996921afd8736df8f12715b"
  },
  "upstream_repo": "https://github.com/avahi/avahi"
}

Note the absence of fedora.release and of any top-level release field — this package is ahead of whatever Fedora currently ships, so there is no confirmed Fedora release to record.

Metadata for an independent package:

{
  "version": "1.3.3",
  "modification_status": "independent",
  "upstream_repo": "https://github.com/oras-project/oras",
  "track_upstream": "latest",
  "release_monitoring_project_id": 205787
}

No fedora object and no release-related field — the release currently shipped lives only in that package’s spec Release: line.

Consequences

  • metadata/*.json across the rpms repository must be brought in line with this specification; that migration is separate, tracked work and is not complete as of this ADR.
  • dist_git.py, check_upstream_versions.py, and CVE analysis tooling can implement the Validation rules directly, since each rule is a single unconditional check with no interpretation left to the implementer.
  • The spec’s Release: line remains the sole authoritative value for what ships; no metadata field may be treated as a substitute for it, including fedora.release.
  • Removing a stored base value for the on-upstream-version and independent states (see Computing the built NVR) makes the bug class behind HUM-5182 structurally impossible, since there is no longer a stored value that can disagree with the spec’s actual current Release:.