This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Using a Project Hummingbird container image

1.1 - Using a Project Hummingbird container image

Project Hummingbird builds a collection of minimal, hardened, and secure container images with a significantly reduced attack surface. This strong focus on security combined with a highly automated update workflow aims to minimize CVE counts, targeting near-zero vulnerabilities. All images support amd64 and arm64 architectures.

Quick Start

All images are available from the Red Hat Hardened Images registry and work directly with Podman, Docker, or Kubernetes:

# Run a command directly with the curl image
podman run registry.access.redhat.com/hi/curl:latest -v https://example.com

# Start a PostgreSQL database
podman run -e POSTGRES_PASSWORD=mysecret -p 5432:5432 registry.access.redhat.com/hi/postgresql:latest

Example container build using the Hummingbird Python image:

FROM registry.access.redhat.com/hi/python:latest
COPY myapp.py /app/
WORKDIR /app
CMD ["python3", "myapp.py"]

Documentation for each image is available on the Red Hat Hardened Images Catalog.

Contents

Available Images

Available images span language runtimes, databases, web servers, CLI tools, and base runtime images.

Browse the complete catalog at the Red Hat Hardened Images Catalog.

Hardened for Security

Hummingbird applies several measures to harden container images:

  • Minimal Software Footprint: Images include only essential packages required for the workload, significantly reducing the attack surface and CVE count.
  • Rapid Update Deployment: Package updates ship as quickly as possible, ensuring fixes are consumed early.
  • Non-Root User Default: Containers default to a non-root user (UID 65532) where technically possible, reducing privileges within the container.
  • Hermetic Build Environment: All containers are built in a hermetic environment without network access, preventing unintended package drift and maintaining full control over software versions.
  • Distroless Security: Shipping only what is strictly necessary for the workload reduces the attack surface and makes certain types of attacks impossible.
  • FIPS-Validated Cryptography: FIPS variants provide NIST-validated cryptographic modules for compliance-sensitive workloads.

Distroless Containers

Hummingbird builds distroless containers — images that do not ship with a package manager and most do not even provide a shell.

The distroless design makes the bundled application the container’s entrypoint, offering a streamlined experience. For example, with the curl image, arguments can be passed directly: podman run registry.access.redhat.com/hi/curl:latest -v https://www.redhat.com/en.

Purpose-built containers reduce the burden on users. Instead of building custom container images and managing their vulnerabilities, a Hummingbird image with the needed application avoids CVE management overhead entirely.

Understanding Image Variants

Hummingbird provides different variants to support various use cases while maintaining security by default.

Default Variant (:latest)

  • Distroless: no package manager, no shell
  • Minimal attack surface
  • Recommended for production
  • Example: registry.access.redhat.com/hi/python:latest

Builder Variant (:latest-builder)

  • Includes dnf package manager and bash
  • For installing additional dependencies
  • Intended for multi-stage builds and development
  • Example: registry.access.redhat.com/hi/python:latest-builder

Some images provide additional variants such as PHP’s FPM variant.

FIPS Variants (:latest-fips, :latest-fips-builder)

FIPS variants ship FIPS 140-3 validated cryptographic modules for workloads that require certified cryptography. Validation applies only when running on RHEL systems installed in FIPS mode.

Two cryptographic stacks are available depending on the image:

Stack Images FIPS packages
OpenSSL Go, Nginx, Node.js, Python, Ruby, curl openssl-config-fips, crypto-policies-config-fips
NSS OpenJDK nss-softokn-fips, nss-softokn-freebl-fips

FIPS is available in both distroless (:latest-fips) and builder (:latest-fips-builder) variants. FIPS variants are available for the Hummingbird distro only (not Rawhide). For implementation details, see the FIPS Variant Guide.

Support Levels

Images are published under two support levels:

  • Red Hat supported — the majority of images. Built, tested, and maintained by the Hummingbird team with full CI/CD coverage, security updates, and vulnerability tracking. Available at registry.access.redhat.com/hi/ with official Red Hat signing keys — this is the recommended registry for all users.
  • Community images include projects with nonstandard open-source release practices, such as MinIO, and experimental images such as os-minimal. Build and test infrastructure is shared with Red Hat supported images, but support commitments differ. Community images can be evaluated for promotion to Red Hat supported status. Published to quay.io/hummingbird-community/.

Registry Organizations

Registry Description
registry.access.redhat.com/hi/ Red Hat Hardened Container Images (recommended)
quay.io/hummingbird/ Mirror of Red Hat supported images without official signing
quay.io/hummingbird-community/ Community-supported images (e.g., MinIO, os-minimal)
quay.io/hummingbird-rawhide/ Upstream Fedora Rawhide packages directly
quay.io/hummingbird-ci/ Build infrastructure images

Tool images used by pipelines (e.g., quay.io/hummingbird-ci/gitlab-ci) are published from the tools repository.

Tagging Strategy

Images follow a version-based tagging scheme:

  • :latest — most recent version (may change)
  • :<version> — specific version (e.g., :3.11, :16)
  • :<version>-builder — builder variant of a specific version
  • :<version>-fips — FIPS variant of a specific version
  • :<version>-fips-builder — FIPS builder variant of a specific version

Use versioned tags in production to stay within a supported version stream. The :latest tag is convenient for development but may introduce unexpected changes.

For general background on tag selection and digest pinning, see How to name, version, and reference container images.

Multi-version images:

For languages with multiple supported version families (Python, Node.js, .NET, etc.), separate images exist for each major version:

  • registry.access.redhat.com/hi/python:latest — latest Python (currently 3.14)
  • registry.access.redhat.com/hi/python:3.11 — Python 3.11.x (stays on 3.11)
  • registry.access.redhat.com/hi/python:3.13 — Python 3.13.x (stays on 3.13)

Version constraints prevent these images from accidentally upgrading to incompatible versions. See version constraints for details.

Multi-Stage Build Pattern

The recommended pattern for compiled languages:

# Build stage: use builder variant to install dependencies and compile
FROM registry.access.redhat.com/hi/go:latest-builder AS builder
RUN dnf install -y <build dependencies>
COPY . /src
WORKDIR /src
RUN go build -o /app .

# Runtime stage: use minimal base image for the compiled binary
FROM registry.access.redhat.com/hi/core-runtime:latest
COPY --from=builder /app /app
ENTRYPOINT ["/app"]

This approach provides build-time flexibility while maintaining a minimal production image.

Referencing Images in Production

Version tags like :3.11 track the main package version but not its dependencies. When packages such as glibc or openssl receive security patches, the image is rebuilt and pushed under the same tag with a new digest. Using the tag alone means silently receiving different content on each pull. To pin the exact image while still communicating the intended version stream, combine the tag with a digest:

FROM registry.access.redhat.com/hi/python:3.11@sha256:<digest>

When both a tag and a digest are present, the container engine uses the digest. The tag serves as documentation — it makes clear which version stream the image belongs to. Retrieve the current digest with:

skopeo inspect --format '{{.Digest}}' \
  docker://registry.access.redhat.com/hi/python:3.11

Because tags float on every rebuild, the pinned digest will fall behind as new security updates ship. A dependency-management bot such as Renovate can watch the tag and open merge requests whenever the digest changes, keeping images current without manual tracking.

Image Verification

Hummingbird images are signed and can be verified using the Hummingbird cosign image:

podman run --rm registry.access.redhat.com/hi/cosign:latest verify \
  --key "https://security.access.redhat.com/data/63405576.txt" \
  --insecure-ignore-tlog \
  registry.access.redhat.com/hi/<image>:<tag>

Vulnerability Scanning

Use Syft and Grype to locally inspect and scan images. For details on how production SBOMs are generated, see Software Bill of Materials.

To scan an image for vulnerabilities:

grype registry.access.redhat.com/hi/<image>:<tag>

For current vulnerability information across all variants and versions, see the Red Hat Hardened Images Catalog.

Sharing Host Data

By default, containers do not have access to host filesystem content. Volume mounts must be added explicitly:

podman run -v /path/on/host:/path/in/container registry.access.redhat.com/hi/curl:latest ...

SELinux Relabeling

On systems with SELinux enabled (such as Fedora and RHEL), mounted volumes must be relabeled to allow container access. Use the :z or :Z option:

podman run -v /path/on/host:/path/in/container:z registry.access.redhat.com/hi/curl:latest ...

The :z option relabels content to be accessible by any container sharing the mount. The :Z option relabels content to be uniquely accessible to one container.

File Permissions

Most Hummingbird images default to a non-root user. When mounting host directories that the container needs to write to, either make the directory world-writable (within a private directory):

mkdir -m 700 /path/on/host
mkdir -m 777 /path/on/host/mnt
podman run -v /path/on/host/mnt:/path/in/container:z ...

Or run as the root user (recommended only with rootless Podman, where the root user in the container maps to the calling user on the host):

podman run --user root -v /path/on/host:/path/in/container:z ...

For read-only access, no permission changes are needed as long as files are world-readable.

Custom CA Certificates

Hummingbird images can be configured to trust custom Certificate Authority (CA) certificates for TLS connections. The approach depends on the image’s TLS stack.

OpenSSL-Based Images (curl, Nginx, Python, etc.)

Mount a CA bundle to replace or extend the system trust store:

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
  registry.access.redhat.com/hi/curl:latest https://internal-server/

For details on building derived images with merged trust stores, see the Custom CA (OpenSSL) guide.

Java-Based Images (OpenJDK, Tomcat)

Java uses its own PKCS12 truststore format. A custom truststore can be created using keytool and mounted at runtime:

podman run --rm \
  -v /path/to/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z \
  registry.access.redhat.com/hi/openjdk:latest java -jar app.jar

For step-by-step truststore creation, see the Custom CA (Java) guide.

Compatibility

Hummingbird images are designed for compatibility with popular images from Docker Hub, Red Hat UBI, and other registries, enabling straightforward migration of existing workloads.

Key difference: Hummingbird images default to a non-root user (UID 65532) where technically possible, while most traditional images run as root. This may require adjusting file permissions on mounted volumes:

# Ensure correct ownership for mounted data
chown -R 65532:65532 /path/to/data

For detailed comparisons (environment variables, ports, sizes, default users), see the compatibility report, also available in machine-readable form.

Reproducible Builds

Reproducible builds ensure that the same inputs and build environment always produce bit-for-bit identical output. This allows independent verification that a published image corresponds to its claimed source materials, making it straightforward to detect if malware was injected during the build process.

Hummingbird images are fully reproducible. Given the signed SLSA provenance attestation that accompanies each image, anyone can rebuild the image from its inputs (this git repo and the RPMs it describes) to verify it matches the published version exactly.

For further background, see Reproducible builds: Project Hummingbird.

Verifying Reproducibility

The podman tool is required. First, download and verify the SLSA provenance attestation using cosign (see Image Verification for key details):

IMAGE=registry.access.redhat.com/hi/curl:latest

podman run --rm registry.access.redhat.com/hi/cosign:latest verify-attestation \
  --key "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/ci/key.pub?ref_type=heads" \
  --insecure-ignore-tlog \
  --type slsaprovenance $IMAGE > attestation.json

Then, feed the attestation into the rebuild tool (capture the image ID for comparison):

iid=$(podman run -i --rm --privileged -v /mnt \
  quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json)

To verify reproducibility, pull the published image and compare image IDs:

iid2=$(podman pull $IMAGE)
[ $iid = $iid2 ] && echo "Identical"

To keep the rebuilt image for further inspection, use the DUMP_OCIARCHIVE environment variable:

podman run -i --rm --privileged -e DUMP_OCIARCHIVE=1 -v /mnt \
  quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json | podman load

Content-Based Layers

Most container images have layers that mirror the Containerfile structure: each RUN or COPY instruction creates a layer. A single package update can invalidate a layer much larger than the package itself, requiring clients to re-pull content that has not changed.

Hummingbird images use chunkah to split images into content-based layers. Instead of layers reflecting the build process, files are grouped by the packages they belong to. This benefits both the network level (only new layers need downloading) and the storage level (common layers are stored once on disk).

Inspect which packages live in which layers using podman history (or docker history):

podman pull registry.access.redhat.com/hi/jq
podman history registry.access.redhat.com/hi/jq

Source Containers

The source code for all Hummingbird containers is available as source containers. A source container includes RPM and non-RPM content shipped in an image, pushed alongside each available tag with the -source suffix.

Inspect source container contents with skopeo:

IMAGE=registry.access.redhat.com/hi/curl:8.19.0-source

cd $(mktemp -d)
mkdir source
skopeo copy --override-os=linux docker://$IMAGE dir:source

cd source
mv version manifest.json ..
for f in $(ls); do tar xvf $f; done

Roadmap

  • Image catalog expansion: The catalog of available images will continue to expand significantly.
  • Image size optimization: Container-specific package configurations and dependency management to further reduce image sizes.
  • Enhanced CVE and SBOM tooling: Improved vulnerability metadata and scanning integration, including methods like cargo-auditable for Rust supply chain transparency.
  • Bootable containers: Expanding support for image-based OS deployments and updates.

Relationship to Fedora

Hummingbird builds and maintains its own RPM packages independently. Spec files are derived from Fedora but built in Hummingbird’s own infrastructure, allowing project-specific modifications and optimizations. Key differences from Fedora:

  • Use of monorepos — one monorepo for packages and one for containers, simplifying management and automation.
  • Heavy reliance on CI/CD — dependency management, testing, and releasing are automated.
  • Secure supply chain requirements — container images and packages are built using Konflux, providing SLSA level 3 compliance.
  • Greater emphasis on upstream tracking — software closely tracks upstream projects’ lifecycles, including parallel stable versions where offered.

There is a natural affinity between the two projects, and opportunities exist to contribute work from Hummingbird back into Fedora.

Contributing

Interested in contributing? The following resources cover the development workflow:

License

This project is licensed under the MIT License - see the LICENSE.txt file for details.

1.2 - Custom CA Certificates (OpenSSL)

Overview

You can configure OpenSSL-based container images (curl, nginx, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections.

There are two approaches for custom CAs depending on your needs:

  1. Custom bundle file - Replace system CAs entirely with your own bundle
  2. Derived image - Build a new image with merged trust store

Approach 1: Custom Bundle File

Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.

Mount your CA bundle to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:

Custom bundle with Podman

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
  registry.access.redhat.com/hi/curl https://your-server/

Custom bundle with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-bundle
data:
  tls-ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    ... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: registry.access.redhat.com/hi/curl
    volumeMounts:
    - name: custom-ca
      mountPath: /etc/pki/ca-trust/extracted/pem
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca-bundle

Custom bundle on OpenShift with CA injection

On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.

Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:

# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  labels:
    config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: registry.access.redhat.com/hi/curl
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      subPath: ca-bundle.crt
      readOnly: true
  volumes:
  - name: trusted-ca
    configMap:
      name: trusted-ca

The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (tls-ca-bundle.pem) that OpenSSL expects.

See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.

Approach 2: Derived Image

Use this approach when you need to trust both your custom CA and the image’s builtin default CAs. This creates a new image with a merged trust store with the trust anchor command.

Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the curl image:

FROM registry.access.redhat.com/hi/curl:latest-builder AS builder

USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt

FROM registry.access.redhat.com/hi/curl:latest
COPY --from=builder /etc/pki /etc/pki

Build and use your custom image:

podman build -t my-curl-with-ca .
podman run --rm my-curl-with-ca https://your-server/

1.3 - Custom CA Certificates (Java)

Overview

You can configure Java-based container images (openjdk, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections. Java uses its own truststore format (PKCS12) rather than PEM files.

Java keystores require a password for integrity checking. The system truststore uses the standard password changeit, which is a well-known default used for tamper detection rather than confidentiality.

Volume Mount Approach

Create a custom truststore using the keytool command. You can either trust only your specific CAs (for internal-only applications) or merge with the default public CAs (typical for applications connecting to both internal and external services).

CERTDIR=$(mktemp -d)  # or use a persistent directory for repeated runs
chmod 777 "$CERTDIR"  # to work with non-privileged containers

# OPTIONAL: Copy the default Java truststore to merge with public CAs
# Skip this step to trust ONLY your custom CAs
podman run --rm -v "${CERTDIR}:/work:Z" \
  registry.access.redhat.com/hi/openjdk:latest \
  cp /etc/pki/ca-trust/extracted/java/cacerts /work/cacerts

# Import your custom CA
podman run --rm -v "${CERTDIR}:/work:Z" \
  registry.access.redhat.com/hi/openjdk:latest \
  keytool -import -noprompt -trustcacerts -alias my-custom-ca \
    -file /work/my-ca.crt \
    -keystore /work/cacerts \
    -storepass changeit -storetype PKCS12 \
    -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE

The -storetype PKCS12 and -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE flags are required for compatibility with FIPS images. The system truststore is PKCS12 format with unencrypted certificate entries (generated by p11-kit). Without these flags, keytool defaults to PBE-encrypted certificate bags that the FIPS security provider cannot decrypt.

Podman Example

Mount the custom truststore to the system location:

podman run --rm \
  -v "${CERTDIR}/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z" \
  registry.access.redhat.com/hi/openjdk:latest \
  java -jar myapp.jar

Kubernetes Example

In Kubernetes, use an init container to create the truststore, then share it via an emptyDir volume:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
  initContainers:
  - name: prepare-truststore
    image: registry.access.redhat.com/hi/openjdk:latest
    command: ["sh", "-c"]
    args:
    - |
      # OPTIONAL: Copy the default truststore to merge with public CAs
      # Skip this line to trust ONLY your custom CAs
      cp /etc/pki/ca-trust/extracted/java/cacerts /truststore/cacerts
      chmod 666 /truststore/cacerts
      keytool -import -noprompt -trustcacerts -alias custom-ca \
        -file /ca/ca.crt -keystore /truststore/cacerts \
        -storepass changeit -storetype PKCS12 \
        -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE
    volumeMounts:
    - name: custom-ca
      mountPath: /ca
    - name: truststore
      mountPath: /truststore
  containers:
  - name: app
    image: registry.access.redhat.com/hi/openjdk:latest
    volumeMounts:
    - name: truststore
      mountPath: /etc/pki/ca-trust/extracted/java
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca
  - name: truststore
    emptyDir: {}

For the trust-only-specific-CAs approach, omit the cp command and create the keystore directly with keytool -import.

Derived Image Approach

Use this approach when it’s more practical to bake the certificates into the image instead of generating the truststore on each container startup. This creates a new image with a merged trust store with the trust anchor command.

Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the openjdk image:

FROM registry.access.redhat.com/hi/openjdk:latest-builder AS builder

USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt

FROM registry.access.redhat.com/hi/openjdk:latest
COPY --from=builder /etc/pki /etc/pki

Build and use your custom image:

podman build -t my-openjdk-with-ca .
podman run --rm my-openjdk-with-ca java -jar myapp.jar

1.4 - Custom CA Certificates (Python)

Overview

You can configure Python container images to trust custom Certificate Authority (CA) certificates for TLS connections. Python libraries have different trust store behaviors compared to other OpenSSL-based images:

  • urllib (built-in) reads CAs from both of these locations:
    1. /etc/pki/tls/cert.pem - the main CA bundle file
    2. /etc/pki/tls/certs/ - directory with hashed certificate symlinks
  • requests (third-party) ignores /etc/pki entirely and requires the REQUESTS_CA_BUNDLE environment variable to use system CAs

Note: The requests library is not included in the standard Python image. To use it, you need to build a derived image:

FROM registry.access.redhat.com/hi/python:latest
RUN ["pip3", "install", "requests"]
podman build -t localhost/my-python-with-requests .

There are two approaches for custom CAs depending on your needs:

  1. Custom bundle file - Replace system CAs entirely with your own bundle
  2. Add custom CAs - Merge your custom CAs with the system defaults

Approach 1: Custom Bundle File

Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.

Mount your CA bundle to /etc/pki/tls/cert.pem, set $REQUESTS_CA_BUNDLE to that, and block the hashed directory with an empty directory:

Custom bundle with Podman

podman run --rm \
  -v /path/to/ca.crt:/etc/pki/tls/cert.pem:ro,Z \
  --tmpfs /etc/pki/tls/certs:notmpcopyup \
  -e REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem \
  localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
import requests
print(requests.get('https://your-server/').status_code)
"

Note: Podman requires the :notmpcopyup option as mounting tmpfs behaves like an overlay. Docker’s tmpfs creates an empty directory by default, and does not know that option, so drop that option with Docker.

Custom bundle with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-bundle
data:
  cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    ... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use registry.access.redhat.com/hi/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/cert.pem
    volumeMounts:
    - name: custom-ca
      mountPath: /etc/pki/tls/cert.pem
      subPath: cert.pem
      readOnly: true
    - name: empty-certs
      mountPath: /etc/pki/tls/certs
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca-bundle
  - name: empty-certs
    emptyDir: {}

Custom bundle on OpenShift with CA injection

On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.

Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:

# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  labels:
    config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use registry.access.redhat.com/hi/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/cert.pem
    volumeMounts:
    - name: trusted-ca
      mountPath: /etc/pki/tls/cert.pem
      subPath: ca-bundle.crt
      readOnly: true
    - name: empty-certs
      mountPath: /etc/pki/tls/certs
      readOnly: true
  volumes:
  - name: trusted-ca
    configMap:
      name: trusted-ca
  - name: empty-certs
    emptyDir: {}

The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (/etc/pki/tls/cert.pem) that Python expects.

See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.

Approach 2: Add Custom CAs

Use this when you want to merge your custom CAs with the system defaults.

Mount your CA certificate as an individual file in /etc/pki/tls/certs/ with a properly hashed filename. The filename must be the output of openssl x509 -noout -subject_hash -in ca.crt followed by .0. Also set $REQUESTS_CA_BUNDLE to that directory:

Custom CA file with Podman

# Generate the hashed filename
CA_HASH=$(openssl x509 -noout -subject_hash -in ca.crt)
echo "Hashed filename: ${CA_HASH}.0"

# Mount the CA certificate as an individual file in the certs directory
podman run --rm \
  -v ./ca.crt:/etc/pki/tls/certs/${CA_HASH}.0:ro,Z \
  -e REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ \
  localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
print(urllib.request.urlopen('https://google.com/').status)  # Default CAs still work
import requests
print(requests.get('https://your-server/').status_code)
print(requests.get('https://google.com/').status_code)  # Default CAs still work
"

Custom CA file with Kubernetes

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    # Use registry.access.redhat.com/hi/python for urllib only
    # Use your derived image with requests installed for both libraries
    image: registry.example.com/my-python-with-requests:latest
    env:
    - name: REQUESTS_CA_BUNDLE
      value: /etc/pki/tls/certs/
    volumeMounts:
    - name: custom-ca
      # REQUIRED: replace example hash with: `openssl x509 -noout -subject_hash -in ca.crt` and append .0
      mountPath: /etc/pki/tls/certs/12abc456.0
      subPath: ca.crt
      readOnly: true
  volumes:
  - name: custom-ca
    configMap:
      name: custom-ca

2 - Contributing to a Project Hummingbird container image

Welcome to the Project Hummingbird contributor documentation. This section provides guides and references for contributing to container images.

Getting Started

New to contributing? Start here:

Guides

Step-by-step guides for common tasks:

Reference

Detailed reference documentation:

2.1 - Quickstart Guide

Get your first contribution done in 5 minutes

Welcome! This guide will help you make your first contribution to Project Hummingbird container images.

For more detailed information, see the full contributor documentation.

Prerequisites

  • Container tools: Podman or Docker
  • Build tools: buildah, make, git
  • Python: Python 3.11+ with dependencies from requirements.txt

For macOS-specific setup, see macOS setup in the Development Workflow.

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3 python3-pip
pip install -r requirements.txt

Quick Start Workflow

1. Fork and Clone

  1. Fork the repository on GitLab
  2. Clone your fork:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

2. Make Your Changes

Edit files in the images/<image-name>/ directory:

  • properties.yml - Image configuration
  • Containerfile.j2 - Container build template
  • tests-container.yml - Integration tests

3. Generate and Build

# Update dependent files
make

# Build the image
ci/build_images.sh <image-name>

4. Test Your Changes

# Run integration tests
ci/run_tests_container.sh <image-name>

# Run linters and checks
make check

5. Submit a Merge Request

  1. Create a branch: git checkout -b fix-nginx-config
  2. Commit your changes: git commit -am "Fix nginx configuration"
  3. Push to your fork: git push origin fix-nginx-config
  4. Open a merge request from your fork to the upstream repository on GitLab

Note for external contributors: If you’re not a project member, the CI pipeline won’t run automatically. Please ping one of the project maintainers in your merge request to trigger the build and test pipeline.

License

By contributing to this project, you agree that your contributions will be licensed under the MIT License. See LICENSE.txt for the full license text.

Common Tasks

Adding a New Image

# Create image directory
mkdir images/myapp

# Copy templates
cp images/Containerfile.j2 images/myapp/Containerfile.j2
cp images/properties.yml images/myapp/properties.yml

# Edit the template, configuration, and tests
vim images/myapp/properties.yml        # Configure packages and variants
vim images/myapp/Containerfile.j2      # Customize the build template
vim images/myapp/tests-container.yml   # Add integration tests

# Update dependent files
make

# Build and test
ci/build_images.sh myapp
ci/run_tests_container.sh myapp

Testing with Docker

# Automatic Docker-in-Docker setup
ci/build_images.sh --engine docker --setup <image-name>
ci/run_tests_container.sh --engine docker --setup <image-name>

Building Specific Variants

# Build only the rawhide builder variant
ci/build_images.sh nginx/rawhide/builder

# Test only the rawhide default variant
ci/run_tests_container.sh nginx/rawhide/default

# Build/test all variants for a specific distro
ci/build_images.sh nginx/rawhide
ci/run_tests_container.sh nginx/rawhide

Next Steps

Getting Help

2.2 - Development Workflow

Detailed development environment setup and contribution workflow

Prerequisites

Ensure the required tools are installed:

  • Container tools: Podman or Docker
  • Build tools: buildah, make, git
  • Python: Python 3.11+ with dependencies from requirements.txt

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3 python3-pip

macOS Setup

macOS requires Python 3.11+, Bash 5+, and GNU command-line tools. Install via Homebrew:

brew install podman bash grep coreutils python
podman machine init && podman machine start

# Add to ~/.zshrc for persistence, then source it or open a new shell
export PATH="/opt/homebrew/bin:/opt/homebrew/opt/grep/libexec/gnubin:/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"

Build scripts use Bash 5+, GNU grep and date, and the Python selected by PATH.

Note: On Intel Macs, use /usr/local/ instead of /opt/homebrew/.

Repository Setup

Fork and Clone

  1. Fork the containers repository on GitLab
  2. Clone your fork with submodules:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
  1. Add the upstream remote:
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

Install Python dependencies

Create and activate a virtual environment, then install the project requirements:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

Keep the virtual environment active when running make or CI scripts so their Python commands use these dependencies.

Development Workflow

1. Create a Branch

Create a descriptive branch starting from upstream main:

git fetch upstream
git checkout -b feature-add-redis-image upstream/main

2. Make Changes

Edit the relevant files in images/<image-name>/:

  • properties.yml - Image configuration (packages, variants, tags)
  • Containerfile.j2 - Container build template
  • tests-container.yml - Container tests
  • tests-k8s.yml - K8s tests
  • README.md.j2 - Image documentation template

3. Generate Files

Regenerate derived files:

make

4. Build Locally

Build the image to verify changes:

ci/build_images.sh <image-name>

# Build specific variant
ci/build_images.sh <image-name>/rawhide/builder

# Build with verbose output
ci/build_images.sh --verbose <image-name>

5. Test Locally

Container tests:

ci/run_tests_container.sh <image-name>

# Test specific variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_container.sh --verbose <image-name>

By default, tests use Podman. To test with Docker:

# Automatic Docker-in-Docker setup
ci/run_tests_container.sh --engine docker --setup <image-name>

K8s tests:

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

See the Testing Guide for K8s test prerequisites and local development workflow.

Testing base images:

When modifying base images (like core-runtime), test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
ci/run_tests_k8s.sh --include-reverse-deps --context <context> core-runtime

Troubleshooting test failures:

Use --pause flag to inspect resources before cleanup:

ci/run_tests_container.sh --pause <image-name>
ci/run_tests_k8s.sh --pause --context <context> <image-name>

6. Run Linters

Ensure code quality:

make check

7. Commit and Push

Commit changes with a descriptive message and push to the fork:

git add .
git commit -m "feat: add Redis container image

- Add Redis 7.x image with default and builder variants
- Include basic health check tests
- Add compatibility notes for official Redis image"

git push origin feature-add-redis-image

8. Open a Merge Request

Open a merge request on GitLab:

  1. Go to the containers repository
  2. Select your fork and branch as the source, main as the target
  3. Fill in the description with what changed, why, and what tests were added

9. CI Pipeline

The CI pipeline automatically:

  • Builds images for multiple architectures
  • Runs integration tests
  • Checks for linting issues

Note for external contributors: If not a project member, the CI pipeline won’t run automatically. Ping one of the project maintainers in the merge request to trigger the pipeline.

Next Steps

2.3 - Resolving Merge Conflicts in Generated Files

How to resolve merge conflicts in generated files during rebases

Overview

During rebases of branches in this repository, you can commonly trigger merge conflicts in files that are generated by the various toolings and Makefile targets.

Rules

  • This documentation is only for generated files.
  • If there is not a section below for the conflicted file, you may need to check make help or search the Makefile for how to regenerate it (e.g., grep -r "target_file" Makefile).

Procedures for generated files

The following sections describe how to rebase given merge conflicts in specific paths.

Regenerating rpms.lock.yaml

When rebasing a branch, you may encounter conflicts in rpms.lock.yaml files. You should first delete the conflicting file, then run make container. Once in the container, rerun make and pass the relative path of the target. The file will be regenerated.

Below shows an interactive example. If attempting to do this non-interactively, remove the file as described and then run something like:

podman run --pull=newer --rm -v "$PWD:$PWD:z" -w "$PWD" \
    quay.io/hummingbird-ci/gitlab-ci:latest \
    make <target_file>
# 1. Delete the conflicted lockfile (replace with your actual conflicting path)
rm <path/to/conflicting/rpms.lock.yaml>

# 2. Enter the CI container
make container

# 3. Inside container, regenerate the lockfile
make <path/to/conflicting/rpms.lock.yaml>

# 4. Exit container
exit

# 5. Stage the regenerated file and continue the rebase
git add <path/to/conflicting/rpms.lock.yaml>
git rebase --continue

# 6. After rebase completes, validate the result
make -j$(nproc) check

Regenerating Containerfile

If the merge conflict is in one or more of the generated Containerfiles, then always accept the upstream (rebasing-onto) branch as the source of truth (i.e., git checkout --theirs -- <Containerfile> and git add <Containerfile>). When you have committed the upstream versions and are done with the rebase, you must then regenerate the Containerfiles so they pick up your changes. Containerfiles can be generated in two ways:

To rebuild Containerfiles for all images that are changed by your PR:

make -j$(nproc)

To rebuild a specific Containerfile, you can also use make and pass the path of the generated Containerfile as an argument.

make generated/images/nginx/rawhide/default/Containerfile

After regenerating Containerfiles:

# Stage regenerated Containerfiles
git add <path/to/regenerated/Containerfile>

# If still in rebase, continue
git rebase --continue

# After rebase completes, validate the result
make -j$(nproc) check

2.4 - Adding New Images

Step-by-step guide for adding new container images to the project

This guide covers two scenarios:

  • Adding a new image - Creating an entirely new image (e.g., adding nginx for the first time)
  • Adding a versioned image - Adding a new version of an existing image (e.g., go-1-25 when go already exists)

Choose the appropriate section below based on your use case.

Adding Completely New Images

Follow these steps to add an entirely new container image to the project.

1. Create Image Directory

Create a directory for the new image:

mkdir images/your-service

2. Copy Base Templates

Copy the template files:

cp images/properties.yml images/your-service/properties.yml
cp images/Containerfile.j2 images/your-service/Containerfile.j2

3. Configure Image Properties

Edit images/your-service/properties.yml to configure:

  • Variants: Define variants (default: [default, builder])
  • Packages: List RPM packages needed in rpm_packages.all
  • Main package: Set main_package for version labeling
  • Repository and stream: Set repository and stream (both required). Add a YAML comment above stream documenting the reasoning (see Choosing a Stream Value)
  • Tags: Configure image tags
  • Release metadata: Set application_category, summary, description, and url
  • Support level: New images start with support_level: community

Example minimal configuration:

---
rpm_packages:
  all:
    - coreutils-single
    - your-main-package
  default:
    - ca-certificates-bundle  # smaller than ca-certificates; sufficient for non-builder variants
  builder:
    - ca-certificates         # full package needed for builder toolchains

main_package: your-main-package
repository: your-service
# <reasoning for stream choice> -- see "Choosing a Stream Value" section
stream: "latest"
# deprecated: true  # Optional: mark the image stream as deprecated
support_level: community
application_category: "Other"
summary: "One-line description of the image"
description: >-
  Description of the software, its purpose, and its main capabilities.
url: "https://upstream-project.example.com/"

tags:
  - value: latest

Prefer ca-certificates-bundle over ca-certificates in non-builder variants — it is smaller and sufficient for TLS trust roots. Builder variants need the full ca-certificates package for toolchain compatibility.

See the Image Configuration Reference for complete properties.yml options.

4. Customize Containerfile Template

Edit images/your-service/Containerfile.j2:

  • Add service-specific configuration (entrypoint, exposed ports, volumes, environment variables)
  • The template automatically includes packages from properties.yml via the {{ main_packages_arg() }} macro

The base template macros handle the standard container setup automatically:

Macro What it provides
{{ final_stage() }} ENV HOME=/tmp, WORKDIR /tmp, labels, OCI archive extraction
{{ set_user() }} USER 65532 (when user: default in properties.yml)

The user field in properties.yml defaults to default (UID 65532) if not specified. No passwd entry is needed — Linux allows running as any UID. See the Container Configuration reference for per-variant and custom user options.

When to override the defaults

The macros provide sensible defaults, but the image should match what the upstream image does. If the upstream image uses a custom user, WORKDIR, or HOME, the Hummingbird image should match for compatibility. Check the upstream image’s Dockerfile or inspect it with podman inspect before deciding.

When the defaults are sufficient, avoid duplicating what the macros already provide:

{# The macros already handle HOME=/tmp, WORKDIR=/tmp, USER 65532.     #}
{# Only add useradd/groupadd, WORKDIR, HOME, or USER when the         #}
{# upstream image does something different that users depend on.       #}

When the upstream image requires a specific user or working directory, set user: in properties.yml and add WORKDIR in Containerfile.j2 as needed. Use {{ default_user }} instead of hardcoding 65532.

Docker/registry auth in README templates

For images that interact with container registries, the auth config must be mounted to $HOME/.docker/config.json inside the container. Check what HOME is set to — for default-user images it is /tmp:

-v ${XDG_RUNTIME_DIR}/containers/auth.json:/tmp/.docker/config.json:ro,Z

Multi-Architecture Support

When Containerfiles or build scripts need to reference the target architecture (e.g., downloading architecture-specific binaries), use the correct naming convention for each context.

Context OCI architecture RPM / Linux name
Intel amd64 x86_64
ARM arm64 aarch64
IBM Z s390x s390x
POWER ppc64le ppc64le
Containerfile $TARGETARCH ARG variable $(arch) output
  • In Containerfiles or Go builds: Use the $TARGETARCH ARG variable provided by buildah (e.g., amd64, arm64, s390x)
  • In shell scripts or RPM builds: Use the arch or uname -m commands (values: x86_64, aarch64, s390x)

Set an image’s architectures property to override the distro defaults for that image or a specific variant. See Architecture configuration.

5. Add Comparison Report

Create the required images/your-service/report.yml. Its first comparison supplies the upstream image and documentation URL used by compatibility reports and generated README sections:

---
compare:
  - pull: docker.io/upstream/your-service:latest
    url: https://upstream-project.example.com/docs/

Add ignore entries only for intentional differences from the upstream image. See existing images with similar entrypoints and runtime behavior for examples.

6. Add Integration Tests

Create test files in images/your-service/:

  • tests-container.yml - Required container tests (Podman/Docker)
  • tests-k8s.yml - Optional Kubernetes tests for workload or cluster integration behavior

Container test example:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

K8s test example:

---
run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

See the Testing Guide for how to write and run tests.

7. Add Image Documentation

Create images/your-service/README.md.j2 with a high-level description, usage instructions, and compatibility notes. The template generates multiple README files for different target audiences (defined in images/variables.yml under readme_targets).

Important notes:

  • The readme_heading() macro automatically generates the heading with the correct product name
  • The readme_description() macro renders the image description from description in properties.yml; use it after the heading instead of hardcoding introductory text
  • Use {{ readme_targets[target].registry }} for registry references (not hardcoded quay.io/hummingbird)
  • The readme_trailing_sections() macro appends compatibility, verification, vulnerability, and about sections

Example template:

{{ readme_heading() }}

{{ readme_description() }}

## Usage

podman run -d -p 8080:8080 {{ readme_targets[target].registry }}/your-service:latest

{{ readme_trailing_sections() }}

For more details, see README Generation.

8. Generate, Validate, and Submit

Follow the standard contribution workflow from the Development Workflow guide:

  • Generate files (make); commit all generated changes
  • Build locally (ci/build_images.sh your-service)
  • Run container tests (ci/run_tests_container.sh your-service)
  • Run Kubernetes tests when tests-k8s.yml exists (ci/run_tests_k8s.sh your-service)
  • Run linters (make check)
  • Commit and push changes
  • Open a merge request
  • Wait for CI pipeline

9. Add Konflux Components and Trigger Testing

When generated konflux-templates/rendered.yml changes, the merge request pipeline runs add_components_to_konflux automatically. This adds only the new components. After the job completes, add a /retest comment to the merge request so Konflux starts the new image pipelines. The manual deploy_components_to_konflux job redeploys all components and is not needed for normal image onboarding.

Adding Versioned Images

When adding a new version of an existing image (e.g., go-1-25, helm-4, or rabbitmq-4-3), follow this streamlined workflow.

1. Check Package Availability

Versioned packages may be available in different repositories:

  • Hummingbird repository - Custom packages built by the Hummingbird project (e.g., golang1.25, nodejs24, dotnet-sdk-10.0)
  • Fedora repositories - Standard Fedora packages (e.g., golang, python3)

The project uses two distros. Their current repository files are defined in images/variables.yml:

  • rawhide - Uses only Fedora Rawhide packages (fedora-44.repo)
  • hummingbird - Uses Fedora 44 + Hummingbird packages (fedora-44.repo + hummingbird.repo)

To check if a package is available in Hummingbird:

# Check the Hummingbird RPMs repository
ls ../rpms/rpms/ | grep <package-name>

Important: If the package is only available in the Hummingbird repository, add distros: [hummingbird] to properties.yml (see step 3).

2. Create Image Directory and Copy Files

Follow the naming pattern <language>-<version> with dashes (e.g., go-1-25, nodejs-24, dotnet-sdk-10-0):

# Create directory
mkdir images/go-1-25

# Copy templates from the closest existing version, not the generic templates
cp images/go-1-26/Containerfile.j2 images/go-1-25/
cp -P images/go-1-26/tests-container.yml images/go-1-25/
cp images/go-1-26/report.yml images/go-1-25/

Tests shared by a version family may instead live under ci/shared-tests/, with each image’s tests-container.yml and tests-k8s.yml symlinked to the shared files. Preserve the sibling image’s arrangement.

Only the version that publishes latest owns README.md.j2 and the generated README files for the shared repository. When a new version becomes the latest, move or copy the template to that version and remove it from the previous version. Older versions without latest do not need a README template. README files are generated from a template in the same image directory; they are not shared automatically.

3. Configure properties.yml

Create images/go-1-25/properties.yml with the versioned package:

---
main_package: golang1.25
repository: go
# endoflife.date: parallel at major.minor (Docker Hub + Wolfi)
stream: "1.25"
application_category: "Programming Languages & Runtimes"
summary: "Statically-compiled language with concurrency and a large standard library"
description: >-
  Statically typed, compiled programming language with garbage collection,
  memory safety, and a large standard library. It includes the complete Go
  toolchain.
url: "https://go.dev"
distros:
  - hummingbird  # Only if package is unavailable in Fedora
user: root
version_constraints: '1.25.*'
tags:
  # NOTE: Do NOT include 'latest' or major-version tags for older versions
  # Only the newest version (e.g., go-1-26) should have those tags
  - value: '{{ package_major_minor_version("golang1.25") }}'
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'
    label: org.opencontainers.image.version
rpm_packages:
  all:
    - coreutils-single
    - git-core
    - glibc-devel
    - golang1.25
    - make

Key differences from a new image:

  • Set main_package to the versioned package name
  • Set repository to match the base image family (for registry organization)
  • Set stream to the version this image tracks (e.g., "1.25" for go-1-25), with a YAML comment above it documenting the reasoning (copy from an existing sibling image)
  • Set version_constraints so lockfile updates cannot cross the image’s version boundary
  • Copy application_category, summary, description, and url from the sibling image, then update any version-specific text
  • Add distros: [hummingbird] if the package is only in the Hummingbird repository
  • Preserve sibling settings such as variants, FIPS packages, user, and support level unless the new upstream version requires a deliberate change
  • Tag strategy for multiple versions:
    • Only the newest version includes latest
    • Include a major-version tag on each stream when it is unique (e.g., Helm 3 and Helm 4)
    • If several streams share a major version (e.g., Go 1.25 and 1.26), only the newest includes that shared major-version tag
    • Every version includes its non-conflicting stream tag and full version tag

4. Update Containerfile Template

Edit images/go-1-25/Containerfile.j2 to reference the versioned package in any environment variables or version-specific commands:

 ENV GOPATH=/go \
     GOTOOLCHAIN=local \
-    GOLANG_VERSION={{ package_version('golang') }} \
+    GOLANG_VERSION={{ package_version('golang1.25') }} \
     PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

5. Update Comparison Images

Edit images/go-1-25/report.yml to compare against version-specific upstream images:

compare:
  - pull: docker.io/golang:1.25
    url: https://hub.docker.com/_/golang

  - pull: cgr.dev/chainguard/go:latest-dev
    url: https://images.chainguard.dev/directory/image/go/overview

6. Tag Strategy for Multiple Versions

When multiple versions share a repository, coordinate tags to avoid conflicts. The following Go versions both have major version 1, so only the newest can publish the 1 tag.

Newest version (go-1-26):

tags:
  - value: latest                                        # Points to newest
  - value: '{{ package_major_version("golang1.26") }}'   # e.g., "1"
    label: io.hummingbird-project.major-version
  - value: '{{ package_major_minor_version("golang1.26") }}'  # e.g., "1.26"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.26") }}'         # Full version
    label: org.opencontainers.image.version

Older versions (go-1-25):

tags:
  # NO 'latest' tag
  # NO major-version tag (would conflict with go-1-26's "1" tag)
  - value: '{{ package_major_minor_version("golang1.25") }}'  # e.g., "1.25"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'         # Full version
    label: org.opencontainers.image.version

Why: This ensures users can reference:

  • registry.access.redhat.com/hi/go:latest → go-1-26
  • registry.access.redhat.com/hi/go:1 → go-1-26 (latest 1.x)
  • registry.access.redhat.com/hi/go:1.25 → go-1-25 (specific minor version)
  • registry.access.redhat.com/hi/go:1.26 → go-1-26 (specific minor version)

When to update: If you add go-1-27, update go-1-26 to remove its latest and major-version tags, and add them to go-1-27.

7. Generate Files and Test

# Generate distro directories, lockfiles, VERSION, TAGS, etc.
make

# Build the image
ci/build_images.sh go-1-25/hummingbird/default

# Run tests
ci/run_tests_container.sh go-1-25/hummingbird/default

# Run linters
make check

Common Issues

Error: No match for argument: golang1.25

The package is not available in the configured distro repositories. If it is available only in the Hummingbird repository, add distros: [hummingbird] to properties.yml, then regenerate files:

make

Generation removes stale distro directories. Do not edit or remove generated files individually.

Choosing a Stream Value

Every image must have a stream field in properties.yml. The stream is a version series identity (e.g., "24", "3.11", "10.1", "latest") that is emitted as the io.hummingbird-project.stream label on the container image. It enables the image catalog to provide consistent structured data about multi-stream repositories.

Why Stream Matters

The image catalog must provide consistent structured data across time, including for images published before a second stream exists. Labels on published images cannot be changed retroactively. If a repository gains a new stream, all previously published images must already carry the correct version-based stream value — otherwise the catalog has inconsistent data that requires fragile repository-specific fixups.

Decision Process

flowchart TD
    Start["New image: choose stream value"] --> Q1{"Does endoflife.date show\nmultiple concurrently\nsupported branches?"}
    Q1 -->|Yes| Assess["Assess ecosystem granularity:\nupstream project, Docker Hub,\nDebian/Ubuntu, Wolfi/Chainguard,\nFedora, endoflife.date"]
    Assess --> Version["Use version-based stream\n(match ecosystem granularity)"]
    Q1 -->|No| Q2{"Does the software have a\nmeaningful version series\nidentity?"}
    Q2 -->|Yes| Current["Use version-based stream\n(current major version)"]
    Q2 -->|No| Latest["stream: latest"]
    Version --> Doc["Document reasoning\nas YAML comment"]
    Current --> Doc
    Latest --> Doc

Document the reasoning for the stream choice as a YAML comment above the stream field in properties.yml. This ensures future maintainers understand why a particular granularity was chosen:

# endoflife.date: parallel at major.minor (2.8, 3.0, 3.2)
stream: "3.0"
# rolling release with date-based versions, no version branches
stream: "latest"

Step 1: Does endoflife.date show multiple concurrently supported branches?

Check endoflife.date for the upstream project. It provides structured data on upstream release branches, security support timelines, and versioning schemes for most projects. If it shows multiple concurrently supported branches, use a version-based stream at the granularity of those branches. If the product is not on endoflife.date, that is itself evidence that parallel version lifecycle management is not a concern for that software — proceed to step 2.

The stream granularity must match the level at which upstream maintains parallel security-supported branches. Check:

  • Upstream project: At what version granularity are parallel branches maintained?
  • Docker Hub official images: Are there parallel version tags? At what granularity?
  • Debian/Ubuntu: Are there parallel versioned packages?
  • Wolfi/Chainguard: Are there versioned package variants?
  • Fedora: Are there parallel versioned packages? Single-version availability is not evidence against versioning, but multi-version availability is strong evidence for it.

Important: the image’s directory name is not evidence for granularity. The directory may use a coarser version or no version at all. The stream must reflect upstream’s actual branch structure.

Similarly, the purpose of upstream’s branches (LTS vs stable, stable vs mainline, STS vs LTS) is irrelevant to granularity. If they are parallel branches receiving security updates, the stream granularity must match.

Match the upstream project’s version naming convention. If the project and ecosystem consistently identify branches using a format that includes a minor component (e.g., .NET uses 8.0, 9.0 — never just 8 or 9), use that format for the stream value. This applies even when endoflife.date abbreviates to just the major number.

Step 2: Does the software have a meaningful version series identity?

If the software uses numbered versioning where the major (or major.minor) number identifies a version series, use a version-based stream at the current major version. The risk is asymmetric: using a version-based stream when a new major version never ships is harmless (the label is still correct), while using "latest" when a new version does ship breaks catalog consistency.

Note: if endoflife.date lists minor versions (e.g., memcached 1.4, 1.5, 1.6) but each immediately supersedes the previous with no overlap in support periods, the minor number is a sequential release counter, not a branch identity. Use the major version as the stream.

Reserve "latest" only for images where no meaningful version series identity exists. This is a narrow criterion with two cases:

  • Rolling/date-based releases with no version series (e.g., minio uses RELEASE.2024-01-18...)
  • Base images that track the distro rather than independent software (e.g., core-runtime)

Step 3: For unversioned images, track the current version

For images without a version in their directory name or version constraints, the stream value reflects the version currently shipped and must be updated when the upstream version changes. This is analogous to version_constraints — an explicit declaration that is updated periodically.

Field Reference

See the Image Configuration Reference for the stream field definition and examples.

Next Steps

2.5 - Adding FIPS Variants

How FIPS variants work and step-by-step guide for adding them

Overview

FIPS variants provide container images that use only FIPS 140-3 validated cryptographic modules. FIPS is implemented as a cross-cutting modifier variant (like builder), not a separate image — the FIPS variant uses the same base image structure but layers FIPS configuration on top via additional RPM packages.

FIPS variants are restricted to the Hummingbird distro only, because FIPS-validated packages are not available in Fedora Rawhide.

FIPS images must not contain any non-validated cryptographic libraries. The presence of an unvalidated crypto library (such as libgcrypt or gnutls) would undermine the FIPS compliance guarantee, even if the application itself does not use that library. Global tests enforce this constraint.

FIPS Validation Scope

The cryptographic modules shipped in FIPS images are FIPS 140-3 validated through the NIST Cryptographic Module Validation Program (CMVP). This validation applies only when running on RHEL systems installed in FIPS mode. Running FIPS images outside this environment (e.g., on Fedora, on RHEL not in FIPS mode, or on other Linux distributions) is outside the validated configuration.

As a best-effort goal, FIPS images aim to behave similarly on both FIPS and non-FIPS hosts — restricting cryptographic operations to FIPS-approved algorithms regardless of host configuration. This is not a guarantee; some images (particularly NSS-based ones like OpenJDK) may behave differently depending on host FIPS mode.

Validated Cryptographic Modules

All validated modules in Hummingbird are pre-built binaries from RHEL 9.2, submitted to NIST for FIPS 140-3 validation:

Module Package Verified by global test
OpenSSL FIPS provider openssl-config-fips fips-provider-matches-ubi9
NSS softokn nss-softokn-fips fips-nss-modules-match-rhel92
NSS freebl nss-softokn-freebl-fips fips-nss-modules-match-rhel92

Global tests validate each module’s checksum against the known-good RHEL 9.2 validated binaries. Checksums are architecture-specific because the binaries differ per architecture.

All FIPS variants also install crypto-policies-config-fips, which sets the system-wide crypto policy to FIPS.

Two FIPS Stacks

Two FIPS stacks exist depending on which crypto library the image’s software uses:

OpenSSL stack — crypto-policies-config-fips + openssl-config-fips. The OpenSSL FIPS provider is enabled via a drop-in configuration file at /etc/pki/tls/openssl.d/fips-provider-enable.cnf. Enforcement is container-side: non-approved algorithms are rejected even on non-FIPS hosts. Used by: Nginx, Node.js, Python, Ruby. Go images also include these packages for system tools (e.g., git-core), but Go itself uses its own native FIPS 140-3 module.

NSS stack — crypto-policies-config-fips + nss-softokn-fips + nss-softokn-freebl-fips. NSS checks the host kernel’s FIPS mode (/proc/sys/crypto/fips_enabled), so enforcement behavior differs between FIPS and non-FIPS hosts. The primary Java security provider switches to SunPKCS11-NSS-FIPS. Used by: OpenJDK.

Both stacks share crypto-policies-config-fips for the system crypto policy, but differ in which cryptographic library provides the validated implementation.

Adding a FIPS Variant

1. Determine the FIPS Stack

Identify which crypto library the image’s software uses:

  • OpenSSL — most languages and servers (Python, Node.js, Ruby, Nginx, Go)
  • NSS — Java/OpenJDK (uses NSS via PKCS#11)

This determines which FIPS packages to install.

2. Update properties.yml

Add fips to additional_variants with a Hummingbird distro restriction, and add the FIPS packages to rpm_packages.fips.

OpenSSL-based image:

additional_variants:
  - name: fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - openssl-config-fips

NSS-based image (OpenJDK):

OpenJDK has existing runtime and runtime-builder base variants. FIPS is added as both fips (full JDK) and runtime-fips (headless JRE):

additional_variants:
  - name: fips
    distros: [hummingbird]
  - name: runtime-fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - java-21-openjdk-devel              # variant-specific packages
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips
  runtime-fips:
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips

See the Image Configuration Reference for complete properties.yml options, including additional_variants and rpm_packages.

3. Generate Files

Run make to generate the hummingbird/fips/ directory structure (Containerfile, RPM lockfiles, VERSION, TAGS):

make

4. Write FIPS Tests

Add FIPS-specific tests to tests-container.yml using filters.variants to target FIPS variants. A standard set of FIPS tests validates:

  1. Non-approved algorithm rejected — e.g., MD5 (for security use), Blowfish, RC2
  2. Approved algorithm works — e.g., SHA-256
  3. Non-approved cipher rejected — e.g., 3DES, CHACHA20-POLY1305
  4. Approved cipher works — e.g., AES-256-GCM (encrypt/decrypt roundtrip)

Example (Python):

fips-rejects-md5:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    try:
        hashlib.new('md5')
        print('MD5_ALLOWED')
    except ValueError:
        print('MD5_REJECTED')
    " | grep -q MD5_REJECTED \
        || test_fail "FIPS mode should reject MD5"

fips-allows-sha256:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    print(hashlib.sha256(b'test').hexdigest())
    " | grep -q 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 \
        || test_fail "FIPS mode should allow SHA-256"

Host FIPS mode considerations:

Where possible, write tests without filters.fips so they validate behavior on both FIPS and non-FIPS hosts. This supports the best-effort goal of consistent behavior across host types. Use filters.fips: true only for tests that inherently require a FIPS-enabled host kernel — primarily NSS-based images where enforcement depends on the host:

# NSS-based: rejection only works on FIPS hosts
fips-rejects-blowfish:
  filters:
    variants: ["*fips*"]
    fips: true                    # requires FIPS-enabled host kernel
  command: |
    ...

See the Test Configuration Reference for variant filters and FIPS mode selection.

5. Build and Test

# Build the FIPS variant
ci/build_images.sh <image>/hummingbird/fips

# Run tests
ci/run_tests_container.sh <image>/hummingbird/fips

# Run linters
make check

Go-Specific FIPS Requirements

Go 1.24+ includes a native FIPS 140-3 cryptographic module that has been CMVP validated (CMVP Certificate #5247). Unlike the previous golang-fips approach (which replaced the compiler with a fork using OpenSSL as the crypto backend), native FIPS uses the same golang package with build-time and runtime environment variables:

  • GOFIPS140=v1.0.0 (build-time): Tells go build to include the CMVP-validated FIPS module (v1.0.0) in the resulting binary. Set in the image environment so all builds in the container use it.
  • GODEBUG=fips140=on (runtime): Enables FIPS 140-3 mode — uses NIST DRBG for randomness, negotiates only FIPS-approved TLS, and runs mandatory self-tests. Set in the image environment as the default for binaries run in the container.
  • Same package: The FIPS variant installs the standard golang1.25 package (no version_package override needed).
  • No cgo dependency: Native FIPS works with both CGO_ENABLED=0 and CGO_ENABLED=1 binaries.
main_package: golang1.25
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - golang1.25
    - openssl-config-fips

The openssl-config-fips package is retained for system tools in the image (e.g., git-core) that use OpenSSL for TLS connections.

Go-specific FIPS tests verify:

  • crypto/fips140.Enabled() returns true (native FIPS module is active)
  • go version -m output includes build GOFIPS140=v1.0.0-c2097c7c (CMVP-validated module, per the security policy section 11.1)
  • FIPS works with CGO_ENABLED=0 static binaries (a key advantage over the old golang-fips approach)

OpenJDK-Specific FIPS Requirements

OpenJDK uses the NSS stack instead of OpenSSL, which has several implications:

  • No openssl-config-fips — uses nss-softokn-fips + nss-softokn-freebl-fips instead
  • Host FIPS mode dependency — NSS checks /proc/sys/crypto/fips_enabled, so FIPS enforcement behavior differs between FIPS and non-FIPS hosts. Crypto rejection tests must use filters.fips: true.
  • Compound variants — OpenJDK introduces runtime-fips alongside fips, combining the runtime base variant with the FIPS modifier. Both share the same NSS FIPS packages; fips additionally includes java-*-openjdk-devel.
  • Security provider — in FIPS mode, the primary Java security provider becomes SunPKCS11-NSS-FIPS instead of the default SUN/SunJCE providers.

Global FIPS Tests

The following tests run automatically for all FIPS images via ci/global-tests/tests-container.yml. Each test skips gracefully for images using the other FIPS stack.

Test Validates Applies to
fips-provider-matches-ubi9 OpenSSL FIPS module checksum matches RHEL 9.2 binaries OpenSSL stack
fips-nss-modules-match-rhel92 NSS module checksums match RHEL 9.2 validated binaries NSS stack
fips-no-libgcrypt libgcrypt is not present (not a validated module) All FIPS
openssl-fips-config-installed OpenSSL FIPS drop-in config exists OpenSSL stack
crypto-policy-is-fips System crypto policy is set to FIPS All FIPS

Image-specific FIPS tests (algorithm rejection/acceptance) are defined in each image’s tests-container.yml.

Testing on FIPS Hosts

The test runner detects host FIPS mode from /proc/sys/crypto/fips_enabled and exports the TEST_FIPS environment variable (true or false). Tests can filter on this using filters.fips:

# Runs only on FIPS-enabled hosts
fips-host-required:
  filters:
    fips: true
  command: |
    ...

# Runs only on non-FIPS hosts
non-fips-only:
  filters:
    fips: false
  command: |
    ...

# Runs regardless of host FIPS mode (default when filters.fips is omitted)
always-runs:
  command: |
    ...

OpenSSL-based images: Most FIPS tests run regardless of host FIPS mode, because the OpenSSL FIPS provider enforces algorithm restrictions at the container level.

NSS-based images: Crypto rejection tests require filters.fips: true, because NSS enforcement depends on the host kernel’s FIPS mode.

Next Steps

2.6 - Testing Guide

How to run and write integration tests locally and work with CI

Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments.

Running Container Tests Locally

Prerequisites

  • Container Engine: Podman or Docker
  • Python: PyYAML package (pip install PyYAML or dnf install python3-pyyaml)

Tests work directly with Podman:

ci/run_tests_container.sh <image-name>

# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide

# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>

With Docker

Use --setup to automatically configure Docker-in-Docker:

ci/run_tests_container.sh --engine docker --setup <image-name>

The --setup flag:

  • Starts hummingbird-docker-dind container with mirror.gcr.io/docker:dind
  • Configures environment variables (DOCKER_HOST, DOCKER_CERT_PATH, etc.)
  • Waits for Docker daemon to be ready
  • Reuses existing container on subsequent runs

Prerequisites:

  • Docker CLI (dnf install docker-cli)
  • Podman (to run the dind container)

Building Images

Build images before testing:

# With Podman
ci/build_images.sh <image-name>

# With Docker
ci/build_images.sh --engine docker --setup <image-name>

Testing Base Images

When modifying base images, test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime

Reproducing CI Failures

Reproduce Testing Farm failures locally:

# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default

# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/rawhide/group
ci/run_tests_container.sh --engine docker --setup <image-name>/rawhide/group

Troubleshooting Container Tests

Inspecting Failed Tests

Use --pause to inspect containers before cleanup:

ci/run_tests_container.sh --pause <image-name>

Viewing Test Output

By default, only failing tests show output. Use --verbose to see passing test output:

ci/run_tests_container.sh --verbose <image-name>

This also enables bash trace mode (set -x), showing each command as it executes.

Running K8s Tests Locally

K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).

Basic Usage

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Local Development Workflow

Build images locally and push to the internal registry:

# Build the image
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

This requires oc login to the target cluster and oc project <namespace> to set the target namespace.

Testing Published Images

Test published images without building locally:

IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Troubleshooting K8s Tests

Use --pause to inspect resources before cleanup, or --verbose to see passing test output:

ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Tests that encounter recognized transient infrastructure failures run up to three times. The runner cleans up test resources and waits 60 seconds before each retry. Application assertion failures are reported immediately. Recognized failures include transient registry and container daemon errors, package-download timeouts from the Hummingbird repository, and transient Go crypto/rand entropy-read failures on FIPS-enabled hosts.

Writing Tests

Test File Locations

  • Container tests: images/<name>/tests-container.yml
  • K8s tests: images/<name>/tests-k8s.yml

Both use the same YAML format with different available environment variables.

Basic Tests

Create a test file with named tests:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

basic-functionality:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --help

Container tests must use test_engine_run to start containers. When a detached container has both a name and a network, the helper waits for Podman network DNS registration automatically before returning. Fresh lookup processes avoid negative DNS caching in a single client process:

test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}"

For K8s tests:

---
cluster-access:
  command: |
    kubectl auth can-i get pods || test_fail "No pod access"

run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

External Test Scripts

For complex tests, use a separate shell script:

---
complex-test:
  command: ./test-complex-scenario.sh

Create images/<name>/test-complex-scenario.sh:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test

Variant-Specific Tests

Limit tests to specific variants:

build-tools-test:
  variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

Cross-Variant Tests

Test interactions between variants:

cross-variant-test:
  variants: [group]
  command: |
    builder="${TEST_IMAGES[nginx/builder]:?}"
    default="${TEST_IMAGES[nginx/default]:?}"

    test_engine_run --rm "${builder}" nginx -version
    test_engine_run --rm "${default}" nginx -version

Known Issues (Container Tests Only)

Mark container tests with known failures:

test-with-known-issue:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"

See the Test Configuration Reference for complete configuration options.

Working with CI Tests

Container Tests (Testing Farm)

Container tests run on Testing Farm with RHEL-9-Nightly systems for x86_64 and aarch64. The runner skips variants unsupported by the host architecture. Affected-test planning rejects required helper images that lack that architecture. Native s390x coverage needs a Testing Farm scenario and s390x test helper images.

Viewing CI Test Results

  1. Open the merge request in GitLab
  2. Navigate to the Jobs page for the pipeline
  3. Identify the Testing Farm job (e.g., containers-rawhide-testing-farm-x86-64)
  4. Click on the job name to go to the Konflux PipelineRun page
  5. If wait-for-results is already finished, select it in the pipeline details, and then switch to the Testing Farm job via the ARTIFACTS_URL link in the right side pane; otherwise, select the scheduler job, switch to the Testing Farm API details via the tf-request link in the right side pane, and then follow the run.artifacts link in the JSON data

Each Testing Farm request runs one or more FMF plans in parallel. For Hummingbird images, results appear under two plans: /ci/run_tests_container/standard (non-FIPS) and /ci/run_tests_container/fips (FIPS-enabled). Other applications only run /standard.

K8s Tests (Konflux Ephemeral Namespace)

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

Viewing K8s Test Results

  1. Open the merge request in GitLab
  2. Navigate to the Jobs page for the pipeline
  3. Identify the K8s test job (e.g., containers-rawhide-k8s-test)
  4. Click on the job name to go to the Konflux PipelineRun page
  5. View the run-tests task logs for test output

Rerunning CI Tests

Retrigger a specific pipeline using slash commands in merge request comments:

/retest nginx--hummingbird--default-on-pull-request

For automated retriggers of only failed checks, see Retrying Konflux Checks.

Next Steps

2.7 - Adding Kubernetes Tests

How to add Kubernetes and OpenShift integration tests to container images

This guide describes how to add Kubernetes tests to container images.

Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

  • Directory permissions (important for OpenShift arbitrary UIDs)
  • Version checks
  • Application functionality (web servers, databases, etc.)
  • Multi-container scenarios (sidecar patterns)

Confirm Test Design

Before implementing:

  1. Read images/{image_name}/tests-container.yml to understand existing tests
  2. Identify which tests need K8s validation:
    • Directory permissions (important for OpenShift arbitrary UIDs)
    • Version check
    • Application functionality (web server, database, etc.)
    • Multi-container scenarios (sidecar patterns)
  3. Confirm test approach before creating files

Rules

  • Script naming: test-k8s-*.sh prefix (e.g., test-k8s-version.sh)
  • Script location: images/{image_name}/test-scripts/
  • YAML location: images/{image_name}/tests-k8s.yml
  • Keep YAML lean: complex logic goes in scripts, not inline YAML
  • No external downloads: avoid network flakes (exception: container images)
  • OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

Validation

  • If images/{image_name}/ doesn’t exist → error “Image not found”
  • If no tests-container.yml → error “Add container tests first”

Create Test Scripts

If directory doesn’t exist → create images/{image_name}/test-scripts/

Test Script Template

#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0

Directory Permissions Test

If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:

  • Check writable directories exist and are writable
  • Verify running as non-root (UID != 0)
  • Test actual write with touch/rm

Version Check Test

If testing version → create test-k8s-version.sh:

  • Run version command
  • Check major version only (not major.minor.patch)
  • Avoids breaking on patch updates

Application-Specific Tests

  • If web server → deployment test with sidecar (curl container)
  • If database → connection/persistence test
  • If tool → primary functionality test

Create tests-k8s.yml

Pattern for each test:

test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"

Multi-Container Tests

If test needs multiple containers (app + client):

  • Use emptyDir volume mounted in both containers
  • emptyDir has 777 permissions, works with arbitrary UIDs

Readiness Probes

If testing app functionality:

  • Add readinessProbe to app container
  • Set timeoutSeconds: 5 for OpenShift network latency

OpenShift Compatibility

Read images/{image_name}/Containerfile.j2:

  • If writable dirs use user:user ownership → change to user:0
  • If writable dirs lack g+w → add chmod -R g+w
  • Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
  • IMPORTANT: Use {{ default_user }} template variable, never hardcode UIDs:
    • ✅ Correct: chown -R {{ default_user }}:0 ${NEWROOT}/dir
    • ❌ Wrong: chown -R 65532:0 ${NEWROOT}/dir
    • Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

touch images/{image_name}/Containerfile.j2
make generated/images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default

Lint and Test

  1. Run make -j$(nproc) check → if fails, fix errors before proceeding

  2. Test in minikube:

    minikube start
    ci/run_tests_k8s.sh --context minikube {image_name}
    
  3. Monitor pods immediately: kubectl get pods -w

  4. If Containerfile changed → test in OpenShift:

    ci/build_images.sh {image_name}/{distro}/default
    # Tag with simple localhost name for --push-image
    podman tag registry.access.redhat.com/hi/{image_name}:latest localhost/{image_name}:test
    ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default
    

    Note: Push images to localhost/{image_name}:test, not to quay.io. Tests should never push to the official registry.

  5. Monitor pods: oc get pods -w

Common Issues

  • If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
  • If wrong image used in OpenShift → verify ci/run_tests_k8s.sh doesn’t override TEST_IMAGE when --push-image set
  • If multi-container test can’t see files → add emptyDir volume
  • If readiness probe times out in OpenShift → add timeoutSeconds: 5

Automated Skill

For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.

2.8 - Image Configuration Reference

Complete reference for properties.yml configuration and image settings

Complete reference for configuring container images via properties.yml.

properties.yml Overview

Each image can have custom settings via a properties.yml file. This reference documents all available configuration options.

Minimal example:

---
rpm_packages:
  all:
    - nginx
main_package: nginx
tags:
  - value: latest

Complete structure:

---
# Distro configuration (see Image Variants section)
# distros: [hummingbird]  # Override default distros (use when package unavailable in some distros)

# Image variants (see Image Variants section)
# Use additional_variants to extend defaults, or variants to override completely
additional_variants: [fpm, fpm-builder] # Adds to default variants
# variants: [default, builder]  # Alternative: completely replaces defaults

# Container runtime configuration (see Container Configuration section)
user: default # Optional: 'default' for default_user (65532), or literal user ID/name

# Package management (see Package Management section)
rpm_packages:
  build-deps: [...]
  all:
    - package-name # All architectures
    - name: arch-specific-package # Arch-specific (see Package Management)
      arches:
        only: x86_64
  <distro>: [...]
  <variant>: [...]
additional_repos: [...]
allow_fedora_repos: true # Optional: allow Fedora repos for hummingbird images (temporary workaround)

# Build configuration (see Build Configuration section)
architectures:
  hummingbird: [aarch64, x86_64, s390x] # Optional: replace distro architectures
hermetic: true
# Exceptional legacy source builds only; do not use for new images:
# build_from_source: true
# prefetch_gomod_path: upstream-submodule

# Versioning (see Versioning section)
main_package: package-name
# version_package:            # Optional: override package for version lookup per variant/distro
#   fips: package-fips-name   # Example: variant-specific package name

# Testing configuration (see Testing Configuration section)
reverse_dependency_tests: false # Optional: disable for images like curl

# Release configuration (see Release Configuration section)
repository: image-name # Required: Quay.io repository name
# <reasoning for stream choice>
stream: "latest" # Required: version series identity for the image catalog
application_category: "Storage" # Required: Pyxis application category
# support_level: community        # Optional: community or experimental (default: Red Hat supported)
# deprecated: true                # Optional: mark the image stream as deprecated

# Image metadata (see Image Metadata section)
summary: "Short one-liner description" # Required when adding labels
description: >- # Required when adding labels
  Short paragraph describing the image.
  1-2 sentences about key features.
url: "https://upstream-project.org" # Required when adding labels

tags:
  - value: latest
  - value: '{{ package_major_version("package-name") }}'
    label: io.hummingbird-project.major-version

# Compliance scanning (enabled by default, see Compliance Scanning section)
# oscap:
#   enabled: false         # Optional: set to false to opt out
#   profiles:              # Optional: override which profiles run per variant
#     cis: true
#     stig:
#       variants: ["*fips*"]
#   exclude_rules:         # Optional: exclude specific rules (concatenated with globals)
#     - id: xccdf_org.ssgproject.content_rule_example
#       reason: "Reason for exclusion"
#       variants: ["*fips*"]   # Optional: glob patterns
#       profiles: [cis]        # Optional: limit to specific profiles

Image Metadata

Images can declare metadata fields for container image labels:

summary

  • Type: String
  • Required: When adding image labels (currently caddy only, expanding to all)
  • Length: ~40-80 characters (one-liner)

Short phrase describing what the image is. Used in table/list views and catalog cards.

Do not start with the image name – the name is always visible from context (UI headings, name label, inspect output).

Example: "Web server with automatic HTTPS"

description

  • Type: String (use >- YAML scalar for multi-line readability)
  • Required: When adding image labels
  • Length: ~100-250 characters (1-2 sentences)

Short paragraph for card views, podman inspect, and Kubernetes UIs (io.k8s.description). Describes what the software is and its key distinguishing features.

Use >- (folded scalar) in YAML to keep the value readable across multiple lines – it folds to a single-line string. Do not use | (literal block) as embedded newlines break label syntax.

Avoid embedded double quotes and backslashes – values are emitted as-is in Containerfile LABEL instructions with no escaping.

The same description value is used when generating the image README: the readme_description() macro in README.md.j2 templates renders it as the introductory paragraph. Do not duplicate this text in the template; use {{ readme_description() }}.

Example:

description: >-
  Extensible server platform with automatic HTTPS by default.
  Provides HTTP/3, reverse proxying, load balancing, and static
  file serving with minimal configuration.

url

  • Type: String (URL)
  • Required: When adding image labels

URL for the upstream project homepage. Used in org.opencontainers.image.url and the Conforma url label.

Example: "https://caddyserver.com"

deprecated

  • Type: Boolean
  • Default: Omitted, which means non-deprecated
  • Description: Marks the image stream as deprecated. The generated image carries the io.hummingbird-project.deprecated: true label so the image catalog can display deprecation status for streams it includes.
  • Example: deprecated: true

The field applies to the stream represented by the image directory, including all variants. The catalog tracks deprecated streams within a repository, so active and deprecated streams can remain visible together.

Package updates, builds, tests, and releases continue, and published images remain available. Add deprecation notices and replacement guidance to the image README template separately.

The catalog currently covers Red Hat supported Hummingbird and Rawhide images. Community images carry the label but are not listed in those catalogs.

Image Variants

Images can have multiple variants (e.g., default and builder):

  • default: Minimal runtime environment
  • builder: Includes development tools and build dependencies
  • custom: Define custom variants in properties.yml

Each variant gets its own:

  • Containerfile in generated/images/<name>/<distro>/<variant>/Containerfile
  • RPM lockfiles in generated/images/<name>/<distro>/<variant>/rpms
  • Build pipeline and release tags (non-default variants get -<variant> suffix)

variants

  • Type: Array of strings
  • Default: [default, builder]
  • Description: List of image variants to generate. Completely overrides the default variants. Each variant gets its own Containerfile, lockfiles, build pipeline, and release tags. Non-default variants get a -<variant> suffix in their image tags.
  • Example: [default] (CI images that only need the default variant)
  • Note: Prefer additional_variants when extending defaults; use variants only when you need to exclude default variants.

additional_variants

  • Type: Array of strings or objects
  • Default: None
  • Description: Additional variants to add to the base variants. The base is variants if specified, otherwise the default variants. Use this when you want to extend without repeating.

Simple format (string array):

additional_variants: [fpm, fpm-builder] # Adds FPM variants to base variants for all distros

Object format with distro restrictions:

Each variant can be an object with name and optional distros fields to restrict which distros the variant is built for:

additional_variants:
  - name: fips
    distros: [hummingbird] # Only build fips variant for hummingbird distro
  - name: fpm # Simple string still works in array
  - name: fpm-builder
    distros: [hummingbird, rawhide] # Build for specific distros

The distros field supports glob patterns:

additional_variants:
  - name: fips
    distros: ["*bird"] # Matches hummingbird
  - name: special
    distros: ["raw*"] # Matches rawhide

When to use distro restrictions:

Use distro restrictions when a variant requires packages or features only available in specific distros. For example, FIPS variants may only be available for Hummingbird where FIPS-validated packages are present

variant_descriptions

  • Type: Object (string-to-string mapping)
  • Required: When image has description and defines additional_variants with non-default base specializations
  • Description: Maps base variant names to human-readable descriptions. These descriptions are emitted as the io.hummingbird-project.variant.description label on each container image. The global description for default is defined in images/variables.yml; image-specific entries add descriptions for bases like fpm, runtime, or openssl. Modifier display (builder, fips) is handled by catalog consumers using the boolean variant labels, not by these descriptions.

Writing guidelines:

  • Describe the variant holistically — what the image variant is, not just what packages it adds on top of default. Someone reading the description with no surrounding context (e.g., via podman inspect) should understand the variant’s identity.
  • Use a noun phrase (2-6 words), not a full sentence.
  • Do not start with the image name (the image context is already provided by the image name label).
  • Avoid the word “runtime” in the default description (the OpenJDK runtime variant is a separate base, and OpenJDK’s default includes the full JDK).
  • Avoid expanding abbreviations redundantly (e.g., “JRE runtime” is redundant because JRE already means “Java Runtime Environment”).
  • Prefer functional descriptions (what capability the variant provides) over repeating the variant name as a product name.

Example (PHP image with FPM base):

variant_descriptions:
  fpm: "PHP FastCGI process manager"

Example (OpenJDK image with runtime base):

variant_descriptions:
  runtime: "Headless Java runtime"

distros

  • Type: Array of strings
  • Default: [rawhide, hummingbird] (from images/variables.yml)
  • Description: List of distros to build for this image. Overrides the global default_distros. Use this to disable a distro for images where a required package is not available in that distro.
  • Example: [hummingbird] disables Rawhide builds for this image
  • Note: Each enabled distro creates a full set of variants, so the final combinations are distros × variants.

Container Configuration

user

  • Type: String, integer, or object (per-variant dict)
  • Required: No (defaults to default)
  • Description: Specifies which user the container runs as. The USER directive is rendered via the {{ set_user() }} macro at the end of Containerfile templates. The special (and default) value default maps to the default_user variable (UID 65532). For user: default, the {{ final_stage() }} macro sets HOME=/tmp and WORKDIR /tmp early in the image to ensure unprivileged users have a writable default environment. Images can override WORKDIR later in their Containerfile.j2. For builder variants with user: default, it additionally sets ENV CONTAINER_DEFAULT_USER=65532.

Same user for all variants:

user: default
user: root # Root user
user: postgres # Literal username

Per-variant configuration:

Use when different variants need different users:

user:
  default: root
  builder: root
  fpm: default
  fpm-builder: default

Package Management

Packages are defined in properties.yml under rpm_packages:

rpm_packages:
  build-deps: # Build-time dependencies (installed in builder layer, not in final
    # image). Included in lockfiles for hermetic builds.
    - golang # Example: Go compiler for building from source
    - rpm-build # Example: Tools for building RPMs during image construction

  all: # Included in all variants (Containerfiles and lockfiles)
    - coreutils-single
    - your-main-package
    - name: grub2-efi-x64 # Arch-specific: only on x86_64
      arches:
        only: x86_64
    - name: grub2-efi-aa64 # Arch-specific: only on aarch64
      arches:
        only: aarch64

  hummingbird: # Included in all Hummingbird variants (distro-level key)
    - versioned-package-1.2

  rawhide: # Included in all Rawhide variants (distro-level key)
    - unversioned-package

  builder: # Included only in "builder" variant
    - compiler-packages
    - debug-tools

  custom-variant: # Included only in "custom-variant" variant
    - variant-specific-packages

Package entry format

Each entry in an rpm_packages list can be either a string (installed on all architectures) or an object with architecture constraints:

String entry (all architectures):

- coreutils-single

Object entry (arch-specific):

- name: grub2-efi-x64
  arches:
    only: x86_64 # Install only on x86_64
- name: some-package
  arches:
    not: aarch64 # Install on all architectures except aarch64

The arches.only and arches.not fields accept either a single string or a list of strings. Package constraints use RPM architecture names: aarch64, x86_64, s390x, and ppc64le.

How arch-specific packages work:

  1. Lockfiles: Arch-specific entries are passed to rpm-lockfile-prototype using its native per-arch package format. The lockfile resolver only includes matching packages for each architecture.
  2. Containerfiles: Arch-specific packages are installed via a case statement on TARGETARCH, so the correct packages are installed during multi-arch builds.

rpm_packages.build-deps

  • Type: Array of strings or arch-specific objects
  • Description: Build-time dependencies needed during image construction. Automatically installed in the builder layer (not in the final image) and included in lockfiles for hermetic builds. Use for compilers, build tools, and other packages needed only during the build process.

rpm_packages.all

  • Type: Array of strings or arch-specific objects
  • Description: Packages included in all variants, in both Containerfiles and lockfiles

rpm_packages.<distro>

  • Type: Array of strings or arch-specific objects
  • Description: Packages included in all variants for the specified distro. Useful when different distros ship different package names (e.g. versioned vs unversioned) and the packages should apply to every variant within that distro.

rpm_packages.<variant>

  • Type: Array of strings or arch-specific objects
  • Description: Packages included only in the specified variant, regardless of distro

additional_repos

  • Type: Array of strings (repository filenames)
  • Default: None
  • Description: Additional yum repository files from yum-repos/ to include in the RPM lockfile generation. These repos are added to the variant-specific repos (see [Global Variables Reference
  • Example: [konflux-ci-rpm-lockfile-prototype-main-fedora-rawhide.repo]
  • Usage: Only needed for packages not available in standard Fedora repos or variant-specific repos

allow_fedora_repos

  • Type: Boolean
  • Default: false
  • Description: Allow Fedora repositories during lockfile generation for Hummingbird production images. By default, production Hummingbird images exclude Fedora repos to ensure packages come only from the Hummingbird repository. Set to true to temporarily allow Fedora repos when required packages are not yet available in the Hummingbird repository.
  • Example: true
  • Usage: Typically used temporarily for packages not yet built for Hummingbird (e.g., .NET runtime on x86_64, OpenJDK packages). Should be removed once packages are available in the Hummingbird repository.
  • Note: Toolchain and similar images under images/ (such as hummingbird-builder) use Fedora repositories for signature validation regardless of this setting.

Hermetic Builds

For hermetic builds (enabled by default), all packages from:

  • rpm_packages.build-deps - included in lockfiles and Containerfiles (builder layer only)
  • rpm_packages.all - included in lockfiles and Containerfiles (final image)
  • rpm_packages.<distro> - included in lockfiles and Containerfiles for all variants of a distro
  • rpm_packages.<variant> - included in lockfiles and Containerfiles for matching variants
  • default_rpm_packages.builder - automatically added for builder variants (see Global Variables Reference - default_rpm_packages)

are added to rpms.lock.yaml and prefetched using cachi2 for offline builds.

Build-time vs Runtime Packages:

  • rpm_packages.build-deps: Installed in the builder layer only (not in final image)
  • rpm_packages.all: Installed in ${NEWROOT} (the final image)
  • rpm_packages.<distro>: Installed in ${NEWROOT} for all variants of the specified distro
  • rpm_packages.<variant>: Installed in ${NEWROOT} for the specified variant

Build Configuration

architectures

  • Type: Mapping of distro or distro/variant to a non-empty list of RPM architecture names
  • Default: default_architectures[distro] in images/variables.yml

Lists replace defaults. A distro/variant entry takes precedence over a distro entry. Unknown names, duplicate entries, empty lists, and unknown selection keys are rejected. Package arches.only and arches.not constraints use the same architecture names. Exclusions apply to the effective architecture list for the selected variant.

architectures:
  hummingbird: [aarch64, x86_64, s390x]
  hummingbird/fips: [aarch64, x86_64]

Rawhide defaults to aarch64 and x86_64. Builds convert x86_64 to amd64 and aarch64 to arm64; s390x and ppc64le keep their names. The resolved lists drive RPM inputs and Konflux build platforms.

Before enabling another architecture, verify package repositories, builder manifests, native build workers, test workers, and test helper images. Generated files must be regenerated from their source configuration.

hermetic

  • Type: Boolean
  • Default: true
  • Description: Controls whether the Konflux build pipeline passes HERMETIC=true to the buildah task, enabling hermetic (offline, network-isolated) builds with prefetched dependencies.
  • Important: Images with hermetic: false are blocked from release by the Conforma hermetic_task policy. hermetic: false should not be used for any image in the containers repository.

build_from_source

  • Type: Boolean
  • Default: false
  • Description: Controls version extraction in the package_version() macro. When true, extracts version from .gitmodules for images built from git submodules. When false, extracts version from RPM package metadata. Does not control whether source code is compiled in the Containerfile.
  • Warning: Do not use for new images. Software built directly from source is not represented by RPM package metadata and cannot be scanned for CVEs by the project’s package-based scanner. This setting exists only for exceptional legacy images.

prefetch_gomod_path

  • Type: String (path relative to image directory)
  • Default: None
  • Description: Path to a Golang project directory (typically a git submodule) for prefetching Go modules via cachi2. Enables hermetic builds for Go projects without a vendor tree. The path is relative to the image directory.
  • Usage: Existing exceptional source-built images only. Do not use for new images because source-built software cannot be scanned for CVEs by the project’s package-based scanner.

Versioning

main_package

  • Type: String
  • Description: Specifies the main package for version labeling

For RPM-based images:

Set to the package name. Version is extracted from RPM package metadata.

main_package: nginx

For source-built images:

Existing exceptional source-built images set this to the submodule path. Version is extracted from the branch field in .gitmodules. Do not use this pattern for new images: source-built software cannot be scanned for CVEs by the project’s package-based scanner.

main_package: images/minio/minio-upstream
build_from_source: true

version_constraints

  • Type: String (glob pattern)
  • Default: None (no constraints)
  • Description: Defines a version constraint pattern to prevent unwanted major or minor version updates. The constraint applies to all distros for this image. Constraints are validated during make check, and builds fail if VERSION files violate the constraint. Primarily used for multi-version images (e.g., python-3-11, nodejs-20) to ensure they stay on their intended version family.

Constraint patterns:

Patterns use glob-style matching:

  • 3.11.* - Match any 3.11.X version (recommended)
  • 20.* - Match any 20.X version
  • 8.* - Match any 8.X version
  • 1.25.* - Match any 1.25.X version (for 3-part versioning like Go)

The pattern matches against the content of VERSION files generated during the build process.

Example:

main_package: python3.11
version_constraints: "3.11.*" # Ensure python-3-11 stays on 3.11.X across all distros
main_package: dotnet-sdk-8.0
version_constraints: "8.*" # Applies to rawhide, hummingbird, and any other distros

When to use:

  • Multi-version images where each image targets a specific version family
  • Preventing automatic major/minor version bumps during Renovate updates
  • Ensuring version tags remain accurate (e.g., python:3.11 doesn’t become python:3.12)

Violation handling:

When a constraint is violated, make check fails with a detailed error message showing:

  • The VERSION file path
  • The actual version found
  • The constraint that was violated
  • The properties.yml file where the constraint is defined

Options for resolution:

  1. Create a new image for the new version (recommended for major/minor bumps)
  2. Exclude the violating version via yum-repos/*.repo excludepkgs
  3. Update the constraint to allow the new version

See Version Constraints for detailed documentation.

version_package

  • Type: Object (string-to-string mapping)
  • Default: None
  • Description: Override the package name used for version extraction on a per-distro, per-variant, or per-distro/variant basis. When a variant or distro installs a different package that provides the same software (e.g., a FIPS-enabled build), this field tells the version/tag macros which package name to look up in the lockfile. The lookup order is: distro/variant, then variant, then distro, then main_package.

Per-distro override:

Use when different distros ship the same software under different package names:

main_package: ruby
version_package:
  hummingbird: ruby4.0 # hummingbird ships ruby4.0, rawhide ships ruby

Per-variant override:

Use when a variant installs a differently-named package:

main_package: ruby
version_package:
  minimal: ruby-minimal # minimal variant resolves version from ruby-minimal

Per-distro/variant override:

Use when both distro and variant affect the package name:

main_package: ruby
version_package:
  hummingbird/fips: ruby4.0-fips

Usage in templates:

The resolved package name is available as package_name_for_version in Jinja2 templates:

tags:
  - value: "{{ package_version(package_name_for_version) }}"
    label: org.opencontainers.image.version
ENV GOLANG_VERSION={{ package_version(package_name_for_version) }}

Testing Configuration

reverse_dependency_tests

  • Type: Boolean
  • Default: true
  • Description: Controls whether this image participates in reverse dependency workflows (both building and testing). When true, changes to this image trigger rebuilds and tests of all images that depend on it. Dependencies are detected by searching for TEST_IMAGES[...] references in test scripts. Set to false for images that should not trigger reverse dependency workflows when changed.
  • Example: Set to false for curl to avoid triggering unnecessary reverse dependency tests

How it works in CI:

When an image is tested, the CI pipeline performs two phases:

  1. Build Phase: Builds all dependencies needed for testing

    • Uses ci/build_images.sh --build-deps to build:
      • Forward dependencies (images the main image’s tests depend on)
      • Reverse dependencies (images that depend on the main image, filtered by reverse_dependency_tests: true)
      • Forward dependencies of those reverse dependencies
    • Automatically deduplicates to ensure each image is built exactly once
  2. Test Phase: Runs tests on the main image and its reverse dependencies

    • Uses ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps to test:
      • The main image
      • All reverse dependencies (filtered by reverse_dependency_tests: true)
    • Kubernetes runs exclude reverse dependencies from the same image group, since a single-component run cannot provide their sibling images; those run in the group pipelines. See run_tests_k8s.sh.

Why set to false?

Set reverse_dependency_tests: false for images like curl that are:

  • Used pervasively across many images (too many reverse dependencies)
  • Have a small, stable API that can be fully tested in their own tests
  • Would cause excessive CI load if every dependent image were tested on each change

Release Configuration

The repository and stream fields together define the release identity for an image. Both are required and validated during CI. The repository specifies the Quay.io repository name, and the stream specifies the version series within that repository.

repository

  • Type: String
  • Required: Yes
  • Description: The Quay.io repository name for this image. For images where the directory name matches the desired repository name (e.g., images/nginx/ → repository: nginx), this is redundant but still required for explicitness. For versioned image directories, this groups multiple versions under one repository (e.g., nodejs-20 and nodejs-24 both use repository: nodejs).
  • Example: nodejs (used by nodejs-20 and nodejs-24 directories)

stream

  • Type: String
  • Required: Yes
  • Description: The version series identity for this image within its repository. Emitted as the io.hummingbird-project.stream label on the container image, enabling the image catalog to provide consistent structured data about multi-stream repositories.

The stream value must reflect upstream’s actual branch structure at the granularity where parallel security-maintained branches exist. Use endoflife.date to determine the correct granularity.

Key rules:

  • The stream is never derived from the image directory name. The directory name is not evidence for stream granularity (e.g., a directory named foo/ for an upstream with branches 2.8, 3.0, 3.2 still requires a major.minor stream like "3.0").
  • The purpose of upstream branches is irrelevant to granularity. Whether upstream calls them “LTS”, “stable”, “mainline”, or “STS” does not matter. If they are parallel branches receiving security updates, the stream granularity must match.
  • Match the upstream project’s version naming convention. If the project identifies branches as 8.0, 9.0 (not 8, 9), use that format even if endoflife.date abbreviates it.
  • Sequential minor versions are not parallel branches. If endoflife.date lists minor versions that each immediately supersede the previous with no support overlap, the minor is a release counter. Use the major version as the stream.
  • Use "latest" only for images where no meaningful version series identity exists: rolling-release tools with date-based versioning (e.g., minio) or base images that track the distro rather than independent software (e.g., core-runtime).
  • For unversioned images (no version in directory name, no version constraints), the stream value reflects the version currently shipped and must be updated when the upstream version changes.

See the stream assignment guidelines for the full decision process and rationale.

Document the reasoning for the stream choice as a YAML comment above the stream field. This ensures future maintainers understand why a particular granularity was chosen.

Examples:

# endoflife.date: parallel at major (Docker Hub + Debian + Wolfi)
stream: "24"
# endoflife.date: parallel at major.minor (Docker Hub + Debian + Wolfi)
stream: "3.11"
# endoflife.date: branches are 10.1, 11.0, not 10, 11
stream: "10.1"
# rolling release with date-based versions, no version branches
stream: "latest"

application_category

  • Type: String
  • Required: Yes
  • Description: The Pyxis application category for the image’s delivery repository. Emitted as the io.hummingbird-project.application-category label on the container image and used in the generated Pyxis repo config. All images sharing the same repository must use the same value. Validated during make check against the list in ci/application-categories.txt.

Common values for Hummingbird images:

  • "Programming Languages & Runtimes" – language runtimes and SDKs (Go, Python, Node.js, .NET, Ruby, etc.)
  • "Web Services" – web servers, reverse proxies, and load balancers (Nginx, httpd, Caddy, HAProxy, Tomcat)
  • "Database & Data Management" – databases and caches (PostgreSQL, MariaDB, Valkey, Memcached)
  • "Developer Tools" – CLI utilities and build tools (git, curl, jq)
  • "Storage" – object storage (MinIO, MinIO Client)
  • "Operating System" – base runtime images (core-runtime)

Example:

application_category: "Web Services"

tags

  • Type: Array of objects
  • Description: Define image tags for Quay.io releases. Each variant gets these tags, with non-default variants receiving a -<variant> suffix.

Tag Object Fields:

  • value (required): The tag value. Can be a literal string (e.g., latest) or a Jinja2 template using helper functions.
  • label (optional): If specified, the tag value is written as a LABEL in the Containerfile, and the actual rendered value from the label is used as the image tag. This ensures version tags match the actual package versions in the image.

Tag Helper Functions:

  • package_version("<package-name>") - Full version (e.g., 1.2.3-4.fc42)
  • package_major_version("<package-name>") - Major version only (e.g., 1)
  • package_major_minor_version("<package-name>") - Major.minor version (e.g., 1.2)
  • rpm_lockfile.content_date - YYYYMMDD UTC date when the lockfile content last changed. Available for all images with an RPM lockfile. Only advances when the resolved RPM set changes, not on every lockfile regeneration.

Example:

tags:
  - value: latest # Literal tag, no label
  - value: '{{ package_major_version("nginx") }}' # Template tag with label
    label: io.hummingbird-project.major-version
  - value: '{{ package_major_minor_version("nginx") }}' # Tag becomes actual version from label
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("nginx") }}'
    label: org.opencontainers.image.version

Note: For images with multiple versions, only set latest on the latest version.

tag_suffix_aliases

  • Type: Mapping of variant names to arrays of suffixes
  • Description: Add aliases for every configured tag without creating another image variant. The suffix is appended directly to each base tag. Existing canonical tags remain unchanged.
  • Example:
tag_suffix_aliases:
  default:
    - jdk25
  builder:
    - jdk25-builder

support_level

  • Type: String

  • Default: Omitted (image is Red Hat supported)

  • Values: community, experimental

  • Description: Controls the support level, Quay organization, and Konflux application assignment for hummingbird-distro images. The mapping from support level to Quay org is defined in images/variables.yml under support_levels; all other names (Konflux app, service account, RPA suffix) are derived from the Quay org name. When omitted, the image is Red Hat supported and assigned to containers-hummingbird. Rawhide images are always assigned to containers-rawhide regardless of this property.

  • Example: support_level: community

  • Mapping:

    Value Quay org Konflux app CPE Pyxis/RPA
    (omitted) hummingbird containers-hummingbird Yes Included
    community hummingbird-community containers-community-hummingbird No Excluded
    experimental hummingbird-ci containers-ci-hummingbird No Excluded
  • Usage: Use community for images maintained by the community (e.g., minio, os-minimal). Use experimental for internal build infrastructure (e.g., hummingbird-builder). These images are built and tested normally but published to their designated Quay organization, not the official Red Hat registry.

How Labels Work:

  1. Template is rendered and written to Containerfile: LABEL org.opencontainers.image.version=1.27.3-1.fc42
  2. Build system extracts the actual value from the image label
  3. Image is tagged with the extracted value: registry.access.redhat.com/hi/nginx:1.27.3-1.fc42

This ensures tags always match the actual package versions built into the image.

Upstream Comparison Configuration

The compare section configures which upstream and third-party images to track for automated vulnerability comparison via the container catalog’s external image scanning pipeline. Place it at the end of properties.yml, after all other sections.

Repository and stream values are whatever the image itself uses. Variant names use Hummingbird names where an equivalent exists (e.g. builder for upstream dev images); upstream-only variants like slim or alpine keep their upstream names.

compare

  • Type: Object
  • Required: No (omit for images with no upstream or third-party equivalent)
  • Description: Configures which upstream and third-party images to track. When present, the aggregation script (ci/aggregate_compare.py) includes this image’s sources in the combined config uploaded to DynamoDB, where the external scanning pipeline picks it up.

compare.sources

  • Type: Array of source objects
  • Required: Yes (when compare is present)
  • Description: List of external image sources to compare against. Each source object documents one upstream registry.

source registry

  • Type: String
  • Required: Yes
  • Description: Full registry pull reference for the upstream image. Must include the full path — use docker.io/library/nginx, not docker.io/nginx.

Example: docker.io/library/nginx, cgr.dev/chainguard/python

source tags

  • Type: Array of tag objects
  • Required: Yes (non-empty)
  • Description: Tags to track on the upstream registry. Each tag object has two fields:
Field Required Description
tag Yes The registry tag to track. Must be a tag that exists on the upstream registry.
variant Yes The variant this tag represents (e.g. default, alpine, slim). Use Hummingbird variant names where equivalents exist (e.g. builder for upstream dev images).

source variants

  • Type: Array of variant description objects
  • Required: Yes
  • Description: Describes the variants referenced in tags. Each variant object has:
Field Required Description
name Yes Variant identifier matching the variant values used in tags.
description Yes Human-readable description of this variant.
base Yes Base variant name (use the variant name, e.g. default, alpine, slim).
builder Yes Whether this is a builder variant. Use false for upstream images.
fips Yes Whether this is a FIPS variant. Use false for upstream images.

Tag selection guidance

Use floating tags, not version-pinned tags. Floating tags (e.g. mainline, 3.14) track the latest patch release automatically. Version-pinned tags (e.g. 1.31.7, 3.14.10) go stale and require manual updates.

Match the upstream project’s branch or release structure. Check the upstream registry for available tags. Common patterns:

  • Branch-based tags (nginx): mainline, mainline-alpine — these float to the latest release in the mainline branch. Only track the branch that matches your image’s stream.
  • Minor-version tags (python, node): 3.14, 3.14-slim, 3.14-alpine — these float to the latest patch within the minor version.
  • Latest-only (Chainguard): latest — the only publicly available tag on the free tier.

Place source entries in the directory of the stream they track. The aggregated config derives every tag’s stream from its directory’s own top-level stream: value, so a source belongs in the directory whose stream matches what the upstream tag delivers:

  • Upstream has named branches (e.g. nginx mainline) → the source lives in the directory whose stream: is mainline
  • Upstream has 3.14.x and 3.15.x release series → the sources live in the stream: "3.14" and stream: "3.15" directories respectively
  • Rolling tags like latest map to the numbered stream they currently deliver — e.g. a Chainguard latest tag that is currently Python 3.15 belongs in the python-3-15 directory
  • Directories whose stream: is latest (rolling releases, date-based versions) emit stream: latest automatically

Chainguard images: Chainguard’s free tier only offers latest and latest-dev tags. Place the Chainguard source in the image directory whose stream matches the Chainguard image’s actual version. Check with podman run --rm cgr.dev/chainguard/<image>:latest --version (or equivalent). Chainguard python:latest is currently Python 3.15, so it belongs in python-3-15/properties.yml. When Chainguard moves to a newer version, move the source to the new stream’s directory. Use variant: "builder" for latest-dev (the dev image with a shell and package manager).

Variant naming: use Hummingbird variant names where upstream equivalents exist — default for plain runtime images, builder for upstream dev images that add a shell and package manager (e.g. Chainguard’s latest-dev). Upstream-only variants with no Hummingbird equivalent (e.g. Docker Hub slim, alpine) keep their upstream names. CVE data for all tracked variants is served the same way, so frontends can present them however fits.

Repository association: the aggregation script automatically attaches the image’s top-level repository: value to each aggregated tag. The catalog uses it to find the external sources belonging to an image detail page — no need to repeat it in the compare: section.

Multi-directory images: When multiple image directories compare against the same upstream registry (e.g. python-3-11/ through python-3-15/ all compare against docker.io/library/python), each directory’s compare.sources lists only its own stream’s tags. The aggregation script merges entries with the same registry — duplicate tags across directories are an error. Because the repository is attached per tag, different Hummingbird repositories may share one upstream registry (e.g. caddy and xcaddy comparing different tags of docker.io/library/caddy).

Examples

nginx — branch-based tags, two upstream branches, plus Chainguard:

compare:
  sources:
    - registry: docker.io/library/nginx
      tags:
        - { tag: "mainline", variant: "default" }
        - { tag: "mainline-alpine", variant: "alpine" }
      variants:
        - {
            name: "default",
            description: "Debian-based image",
            base: "default",
            builder: false,
            fips: false,
          }
        - {
            name: "alpine",
            description: "Alpine-based image",
            base: "alpine",
            builder: false,
            fips: false,
          }
    - registry: cgr.dev/chainguard/nginx
      tags:
        - { tag: "latest", variant: "default" }
      variants:
        - {
            name: "default",
            description: "Chainguard image",
            base: "default",
            builder: false,
            fips: false,
          }

python-3-14 — minor-version tags, one stream per directory (the aggregation script merges all python-3-XX directories into a single docker.io/library/python entry):

compare:
  sources:
    - registry: docker.io/library/python
      tags:
        - { tag: "3.14", variant: "default" }
        - { tag: "3.14-slim", variant: "slim" }
        - { tag: "3.14-alpine", variant: "alpine" }
      variants:
        - {
            name: "default",
            description: "Debian-based image",
            base: "default",
            builder: false,
            fips: false,
          }
        - {
            name: "slim",
            description: "Minimal Debian-based image",
            base: "slim",
            builder: false,
            fips: false,
          }
        - {
            name: "alpine",
            description: "Alpine-based image",
            base: "alpine",
            builder: false,
            fips: false,
          }

python-3-15 — adds a Chainguard source: rolling latest tags mapped to the current stream, with latest-dev using the builder variant:

compare:
  sources:
    - registry: docker.io/library/python
      tags:
        - { tag: "3.15", variant: "default" }
        - { tag: "3.15-slim", variant: "slim" }
        - { tag: "3.15-alpine", variant: "alpine" }
      variants:
        - {
            name: "default",
            description: "Debian-based image",
            base: "default",
            builder: false,
            fips: false,
          }
        - {
            name: "slim",
            description: "Minimal Debian-based image",
            base: "slim",
            builder: false,
            fips: false,
          }
        - {
            name: "alpine",
            description: "Alpine-based image",
            base: "alpine",
            builder: false,
            fips: false,
          }
    - registry: cgr.dev/chainguard/python
      tags:
        - { tag: "latest", variant: "default" }
        - { tag: "latest-dev", variant: "builder" }
      variants:
        - {
            name: "default",
            description: "Chainguard image",
            base: "default",
            builder: false,
            fips: false,
          }
        - {
            name: "builder",
            description: "Chainguard image with shell and package manager",
            base: "default",
            builder: true,
            fips: false,
          }

minio (hypothetical) — rolling release with only a latest tag and no version series on either side:

compare:
  sources:
    - registry: docker.io/minio/minio
      tags:
        - { tag: "latest", variant: "default" }
      variants:
        - {
            name: "default",
            description: "Official MinIO image",
            base: "default",
            builder: false,
            fips: false,
          }

Compliance Scanning

OpenSCAP compliance scanning verifies CIS and STIG compliance during the build. Scanning runs against the container rootfs using oscap-chroot and fails the build if any enabled rules are violated.

Scanning is enabled by default for all images. It only runs for the Hummingbird distro and is skipped for the builder base image.

oscap.enabled

  • Type: Boolean
  • Default: true (from images/variables.yml)
  • Description: Controls whether OpenSCAP compliance scanning runs for this image. When true, a verify-compliance step is added to the generated Containerfile for all Hummingbird variants. Set to false to opt out.
oscap:
  enabled: false

oscap.profiles

  • Type: Object mapping profile names to enablement rules
  • Default: {cis: true, stig: {variants: ["*fips*"]}} (from images/variables.yml)
  • Description: Controls which compliance profiles run for each variant. Each profile can be set to true (all variants), false (disabled), or an object with a variants list of glob patterns.

Default behavior (CIS for all variants, STIG only for FIPS variants):

oscap:
  profiles:
    cis: true
    stig:
      variants:
        - "*fips*"

Enable STIG for all variants:

oscap:
  profiles:
    stig: true

Disable CIS for a specific image:

oscap:
  profiles:
    cis: false

The variants field supports glob patterns using fnmatch syntax (e.g., *fips* matches fips, fips-builder, openssl-fips).

oscap.exclude_rules

  • Type: Array of rule exclusion objects
  • Default: Global rule exclusions from images/variables.yml
  • Description: Rules to exclude from compliance scanning. Image-level exclusions are concatenated with global exclusions (not replaced). Each exclusion object has the following fields:
Field Type Required Description
id String Yes XCCDF rule ID (e.g., xccdf_org.ssgproject.content_rule_configure_crypto_policy)
reason String Yes Explanation for excluding this rule
variants Array of strings No Glob patterns for variants this exclusion applies to. Omit for all variants.
profiles Array of strings No Profile names (cis, stig) this exclusion applies to. Omit for all profiles.

Exclude a rule for all variants and profiles:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "Caddy uses Go's built-in TLS stack, not system crypto-policies"

Exclude a rule only for specific variants:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot"
      variants:
        - "*fips*"

Exclude a rule only from a specific profile:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_some_rule
      reason: "Not applicable to CIS for this image"
      profiles:
        - stig

Combine variant and profile filters:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot for FIPS variants"
      variants: ["*fips*"]
      profiles: [cis]

How compliance scanning works

  1. Profile resolution: generate_jinja2.py determines which profiles (CIS, STIG) are active for the current variant based on oscap.profiles.
  2. Rule filtering: Exclusion rules are filtered by variant (glob matching) and profile, producing per-profile exclusion lists.
  3. Tailoring file: If any active profile has exclusions, an XCCDF tailoring file (oscap-tailoring.xml) is generated in the variant directory with custom profiles extending the upstream CIS/STIG profiles.
  4. Containerfile: The verify-compliance command is added with explicit --cis and/or --stig flags based on active profiles, plus a --tailoring-file argument if exclusions exist.
  5. Build-time scan: oscap-chroot evaluates the rootfs against each active profile. The build fails if any non-excluded rule fails.

Next Steps

2.9 - Test Configuration Reference

Complete reference for test definition format and configuration options

Complete reference for configuring image tests via tests-container.yml and tests-k8s.yml.

Test Definition Format

Each image can have tests defined in:

  • images/<name>/tests-container.yml - Container tests (run with Podman/Docker)
  • images/<name>/tests-k8s.yml - K8s tests (run in Kubernetes cluster)

Both files use the same YAML format:

---
script:
  command: ./test.sh

inline:
  command: |
    image=${TEST_IMAGE:?}
    test_engine_run --rm "${image}" --version

Environment Variables

The test runners provide environment variables to test commands. Some are common to both test types, others are specific to container or K8s tests.

Common Variables

Variable Description
TEST_IMAGE The container image being tested
TEST_IMAGES Associative array with all group/variant image URLs
TEST_IMAGES_PATH Path to file containing serialized TEST_IMAGES array
TEST_VERBOSE Show test command output (true or false)
TEST_GROUP The image group being tested (nginx, python, etc.)
TEST_DISTRO The distro being tested (hummingbird, rawhide)
TEST_VARIANT The variant being tested (default, builder, etc.)
TEST_GENERATED_VARIANT_DIR Absolute path to the generated variant directory

Container Test Variables

Variable Description
TEST_ENGINE Container engine to use (podman or docker)
TEST_USER_ID Current user ID (for permission handling)
TEST_FIPS Whether host is in FIPS mode (true or false)

K8s Test Variables

Variable Description
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (hum-k8s-test=<id>)

Helper Functions

Common to All Tests

Both test types provide the test_fail function:

Function Description
test_fail Fail the test with a custom error message

In K8s tests, kubectl is pre-configured with the context/kubeconfig from CLI args.

Container Tests Only

Container tests provide helper functions that wrap the container engine with appropriate flags. Always use these functions instead of calling ${TEST_ENGINE} directly for create and run commands. This ensures tests respect globally set options such as the hermetic mode setting (--pull=never).

Function Description
test_engine_create Run ${TEST_ENGINE} create with appropriate pull flags
test_engine_run Run ${TEST_ENGINE} run with appropriate pull flags

Note: For other container engine commands (inspect, network, volume, etc.), use ${TEST_ENGINE:?} directly.

Variant-Aware Testing

The testing system automatically includes variant-specific default tests:

  • Global tests from ci/{variant}-tests/tests-container.yml
  • Image-specific tests from images/<name>/tests-container.yml

Variant Selection

Tests can specify which variants they apply to using the filters.variants field. The field supports both exact matches and glob patterns.

# Test that only runs for builder variants
build-tools-test:
  filters:
    variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

# Test that runs for multiple specific variants
multi-variant-test:
  filters:
    variants: [default, builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Hello"

# Test that runs for all variants (no filters.variants field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

Glob Patterns in Variant Filters

Use glob patterns to match multiple variants dynamically:

# Test that runs for any FIPS variant (fips, fips-builder, etc.)
fips-test:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-fips-mode

# Test that runs for all builder variants (builder, fpm-builder, etc.)
builder-test:
  filters:
    variants: ["*-builder", "builder"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" dnf --version

# Mix exact matches and patterns
mixed-filter:
  filters:
    variants: [default, "*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on default and all fips variants"

Supported glob syntax:

  • * - matches any sequence of characters
  • ? - matches any single character

Distro Selection

Tests can specify which distros they apply to using the filters.distros field. Like variant filters, this field supports both exact matches and glob patterns.

# Test that only runs for hummingbird distro
hummingbird-only-test:
  filters:
    distros: [hummingbird]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-hummingbird-specific-check

# Test that runs for multiple distros
multi-distro-test:
  filters:
    distros: [hummingbird, rawhide]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on both"

# Test that runs for all distros (no filters.distros field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

This is useful for tests that check distro-specific features. For example, CPE labels are only present on hummingbird images (Fedora/rawhide has no official CPE).

Glob Patterns in Distro Filters

Use glob patterns to match multiple distros dynamically:

# Test that runs for any distro ending in "bird"
bird-distro-test:
  filters:
    distros: ["*bird"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-something

# Test that runs for any distro starting with "raw"
raw-distro-test:
  filters:
    distros: ["raw*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-rawhide-feature

Supported glob syntax:

  • * - matches any sequence of characters
  • ? - matches any single character

FIPS Mode Selection

Tests can specify whether they should run based on the host’s FIPS mode using filters.fips:

# Test that only runs on FIPS-enabled hosts
fips-required-test:
  filters:
    fips: true
  command: |
    test_engine_run --rm "${TEST_IMAGE}" openssl list -providers

# Test that only runs on non-FIPS hosts
non-fips-test:
  filters:
    fips: false
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-non-fips-check

# Test that runs regardless of FIPS mode (no filters.fips field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

The TEST_FIPS environment variable is set to true when running on a FIPS-enabled host (detected from /proc/sys/crypto/fips_enabled), and false otherwise.

Group Test Mode

When running tests for all variants (without specifying a specific variant), the system supports a special “group” test mode for tests that need to work across multiple variants simultaneously:

# Test that runs only in group mode
cross-variant-compatibility-test:
  filters:
    variants: [group]
  command: |
    # Access specific group/variant combinations using TEST_GROUP
    builder_image=${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    default_image=${TEST_IMAGES[${TEST_GROUP}/default]:?}

    echo "Builder variant: ${builder_image}"
    echo "Default variant: ${default_image}"

    # Test that both variants have compatible APIs
    test_engine_run --rm "${builder_image}" nginx -version
    test_engine_run --rm "${default_image}" nginx -version

Using TEST_IMAGES in External Scripts

External shell scripts that need to reference other image variants must source the TEST_IMAGES array file at the beginning:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Now TEST_IMAGES is available - always use :? for proper error checking
test_engine_run --rm "${TEST_IMAGES[curl/default]:?}" ...
test_engine_run --rm "${TEST_IMAGES[httpd/default]:?}" ...

Note: Inline test commands in YAML files automatically have access to TEST_IMAGES and do not need this sourcing pattern.

Using TEST_GROUP for Dynamic References

The TEST_GROUP variable contains the current image group name (e.g., nginx, python, aspnet-runtime-8-0). Use it with TEST_IMAGES to dynamically reference the current image’s variants without hardcoding the group name:

multi-stage-build:
  filters:
    variants: [builder]
  command: |
    # Build in builder variant, run in default variant
    "${TEST_ENGINE}" build -t localhost/myapp -f - . <<EOF
    FROM ${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    # ... build steps ...

    FROM ${TEST_IMAGES[${TEST_GROUP}/default]:?}
    COPY --from=0 /app /app
    EOF

    test_engine_run --rm localhost/myapp /app/myapp

Test Helper Functions

The test runner provides helper functions available in test commands:

test_fail(message)

Immediately fails the test with a custom error message sent to stderr:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Expected 'expected', got '$result'"

Known Issues (Container Tests Only)

Container tests can specify known log patterns that should not cause the test to report an error (for which support is still unimplemented) or failure. This helps distinguish between expected failures (tracked issues) and unexpected failures (new regressions), and detects when expected failures stop occurring.

Add known_issues to any test to specify patterns matching expected failures:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"  # optional, defaults to "always"

Fields

  • description: Human-readable explanation of the issue (required)
  • issue: Full URL to the issue tracking this problem (required)
  • pattern: Regular expression(s) to match against test output (required)
  • fails: Frequency of failure (optional, defaults to “always”)

Pattern Support

The pattern field supports both single patterns and arrays of patterns:

known_issues:
  # Single pattern
  - description: "Simple failure case"
    issue: "https://issues.redhat.com/browse/PROJ-1"
    pattern: "Connection failed"

  # Multiple patterns (any match triggers)
  - description: "Network connectivity issues"
    issue: "https://issues.redhat.com/browse/PROJ-2"
    pattern:
      - "Connection timeout"
      - "Network unreachable"
      - "curl: \\(28\\)"
    fails: "sometimes"

Failure Frequency

  • always: Known failures that consistently fail every time (default)
  • sometimes: Intermittent failures that may pass on retry (flaky tests)

Unexpected Pass Detection

The system automatically detects when tests with known issues labeled as fails: "always" suddenly start passing. This will not prevent missing known issues being masked by other (still occurring) known issues.

Automatic Retries

The container and K8s test runners automatically retry tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary registry, network, package-download, image-pull, container-startup, or K8s scheduling problems.

Retry Behavior:

  • Failed tests are checked against a list of retriable error patterns
  • If a match is found, the test is automatically retried
  • If the test still fails after all attempts, it’s reported as a normal failure

K8s tests run up to three times, with cleanup and a 60-second delay between attempts. Retriable conditions include image-pull and container-creation errors, as well as kubectl wait timeouts. Application assertion failures are not retried.

Next Steps

3 - Operating Hummingbird infrastructure

Operations guides for managing Hummingbird infrastructure, including Konflux cluster setup and CI/CD operations.

3.1 - Excluding Packages from Images

AI Agent Note: When asked to exclude packages, always ask the user for a ticket link or explanation first. This is required for the comment documenting the exclusion.

Overview

When a faulty RPM build is published to Hummingbird repositories, it can be excluded from container images by adding an excludepkgs directive to the repository configuration.

Workflow

1. Identify the source package NVR

Identify the Name-Version-Release of the faulty source package. Example: ncurses-6.5-8.20250614.hum1

2. Generate excludepkgs value

Query the Hummingbird repositories and generate the excludepkgs value:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest sh -c '
dnf5 repoquery --queryformat="%{sourcerpm} %{name}-%{evr}.*
" 2>/dev/null | sed -n "s/^ncurses-6.5-8.20250614.hum1.src.rpm //p" | tr "\n" " "
'

Replace ncurses-6.5-8.20250614.hum1 with the source NVR. This outputs the ready-to-use excludepkgs value:

ncurses-6.5-8.20250614.hum1.* ncurses-base-6.5-8.20250614.hum1.* ...

3. Add excludepkgs to repo file

Edit yum-repos/hummingbird.repo and add the binary package NEVRs to excludepkgs in the [hummingbird] section (the binary repo, not [hummingbird-source]).

Document the exclusion with a one-line comment containing the ticket link:

[hummingbird]
...
# HUM-1234: ncurses-6.5-8.20250614.hum1 causes segfault in terminfo parsing
excludepkgs=ncurses-base-6.5-8.20250614.hum1.* ncurses-libs-6.5-8.20250614.hum1.*

Multiple packages are space-separated. Use .* suffix to match all architectures.

4. Merge the change

Merge the repo file change to main. Renovate will automatically apply the exclusions when it next rebases existing lockfile update branches or creates new ones.

Verification

Check that the excluded package no longer appears in any rpms/rpms.lock.yaml files in subsequent Renovate-generated merge requests.

Removing Exclusions

When a newer version is available, exclusions can be removed.

1. Verify a newer version exists

Check the latest available version (should differ from excluded 6.5-8.20250614.hum1):

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --latest-limit=1 --queryformat="%{evr}" "ncurses-libs" 2>/dev/null

2. Remove the exclusion

Remove the excludepkgs line and its comment from yum-repos/hummingbird.repo.

3. Merge and verify

Merge the change. Renovate will pick up the newer version in subsequent lockfile updates.

3.2 - Managing Konflux Comments

Clean up and manage excessive Konflux comments on merge requests using the comment deletion tool.

When Konflux comments become excessive on merge requests, you can clean them up using the provided comment deletion tool.

Prerequisites

  • COM_GITLAB_TOKEN: GitLab API token with api scope and Maintainer access

Usage

If Konflux comments become excessive on merge requests, you can clean them up using:

# Show help for the comment deletion tool
make delete-konflux-comments ARGS="--help"

# Example: Delete comments from a specific merge request
make delete-konflux-comments ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete only failed build/test comments (dry run)
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Example: Delete all Konflux comments from a merge request
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

Comment Categories

The tool can target specific types of Konflux comments:

  • failed: Only comments related to failed builds/tests
  • all: All Konflux-related comments (use with --all flag)

Safety Features

  • Dry Run Mode: Use --dry-run to preview what would be deleted without actually removing comments
  • Selective Deletion: Target specific comment categories to avoid removing important information
  • Confirmation: The tool will show what it plans to delete before taking action

Examples

# Set your GitLab token
export COM_GITLAB_TOKEN=your_token_here

# Preview what failed comments would be deleted
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete only failed build comments
make delete-konflux-comments ARGS="--comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

# Delete all Konflux comments (use with caution)
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"

When to Use

Consider cleaning up Konflux comments when:

  • Merge requests have accumulated many outdated failure comments
  • Comments are making it difficult to find relevant discussion
  • You want to start fresh after resolving systematic issues
  • The comment thread has become cluttered with automated messages

Note: Use this tool judiciously, as it permanently removes comments that might contain useful debugging information.

3.3 - Running Conforma Checks Locally

Overview

Conforma (Enterprise Contract) policy checks run automatically in the Konflux pipeline before release (see Image Pipeline — Stage 5). These checks can also be run locally against any Konflux-built image to validate compliance, test policy changes, or investigate failures.

Prerequisites

Install the Conforma CLI:

curl -sLO https://github.com/conforma/cli/releases/download/snapshot/ec_linux_amd64
chmod 755 ec_linux_amd64
mkdir -p ~/.local/bin
mv ec_linux_amd64 ~/.local/bin/ec

Running All Checks

The full set of Conforma checks (signatures, attestations, SLSA provenance, labels, etc.) can be run against any Konflux-built image using ec validate image with the Konflux signing key.

Public Key

Testing Farm and local verification use the Konflux build signing public key committed at ci/key.pub. The authoritative source is the Konflux member cluster Secret at k8s://openshift-pipelines/public-key. Update this file whenever Konflux rotates its build signing key so both checks can verify newly signed attestations.

Obtaining the Image Reference

Combine the IMAGE_URL and IMAGE_DIGEST results from a successful build PipelineRun:

<IMAGE_URL>@<IMAGE_DIGEST>

Merge request build images follow the pattern:

quay.io/redhat-user-workloads/hummingbird-tenant/<component>:on-mr-<MR_ID>-<COMMIT_SHA>

Running the Validation

Create a policy file that matches the pipeline policy but with any modifications needed for testing. For example, to run the pipeline policy with label checks enabled (removing the labels.required_labels and labels.optional_labels exclusions):

cat > /tmp/policy.yaml << 'YAML'
sources:
  - name: Release Policies
    config:
      exclude:
        - test.required_tests_passed:sast-snyk-check
        - test.no_skipped_tests:sast-snyk-check
        - test.required_tests_passed:sast-snyk-check-oci-ta
        - test.no_skipped_tests:sast-snyk-check-oci-ta
        - test.no_failed_tests:ecosystem-cert-preflight-checks
        - test.no_erred_tests:ecosystem-cert-preflight-checks
        - test.no_failed_informative_tests
        - test.no_test_warnings:deprecated-image-check
        - trusted_task.current
        - rpm_repos.ids_known
        - buildah_build_task.privileged_nested_param
        - schedule.weekday_restriction
        - cve.cve_blockers
        - schedule.date_restriction
      include:
        - '@redhat'
    data:
      - github.com/release-engineering/rhtap-ec-policy//data
      - oci::quay.io/konflux-ci/tekton-catalog/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/konflux-vanguard/data-acceptable-bundles:latest
      - oci::quay.io/konflux-ci/integration-service-catalog/data-acceptable-bundles:latest
    policy:
      - oci::quay.io/enterprise-contract/ec-release-policy:konflux
    ruleData:
      allowed_registry_prefixes:
        - "quay.io/hummingbird-ci/"
YAML

Then validate:

ec validate image \
    --image "quay.io/redhat-user-workloads/hummingbird-tenant/<component>@sha256:<digest>" \
    --policy /tmp/policy.yaml \
    --public-key ci/key.pub \
    --ignore-rekor \
    --strict=false \
    --show-successes \
    --output text

Options

Flag Purpose
--ignore-rekor Required — Konflux does not use a Rekor transparency log
--strict=false Return exit code 0 even on failures (useful for investigation)
--show-successes Include passing checks in output (off by default)
--output json Machine-readable output (also: text, yaml, summary)

Running Label Checks Only

The label checks (labels.required_labels, labels.optional_labels) are the only Conforma rules that can be evaluated without signatures or attestations, because they only read OCI image metadata. This lighter-weight approach does not require the signing key and works against any registry image, including published (non-Konflux-built) images.

Setup

Fetch the policy rego files and upstream rule data (one-time):

ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

git clone --depth=1 \
    https://github.com/release-engineering/rhtap-ec-policy.git \
    /tmp/rhtap-ec-policy

Running Checks

ec opa eval \
    --data /tmp/ec-policy/policy/*/policy \
    --data /tmp/rhtap-ec-policy/data/rule_data.yml \
    --input <(echo '{"image":{"ref":"quay.io/hummingbird-rawhide/caddy:latest"}}') \
    '{"deny": data.labels.deny, "warn": data.labels.warn}' \
    --format pretty

This works against any image accessible from a container registry — both Konflux-built and published images. It uses the ec CLI’s embedded OPA engine with ec.oci.* built-in functions to read image config directly from the registry.

Note: The --data flag (not --bundle) must be used for the rego files. Loading the policy as a bundle creates an OPA root conflict that prevents the rule data from being resolved. The ec.oci.* built-in functions are only available in ec opa, not in standalone opa or conftest.

Interpreting Results

For ec validate image:

  • Successes indicate passing policy rules (visible with --show-successes).
  • Violations are failures that block release. Each includes a code (rule name) and msg.
  • Warnings are non-blocking advisories.

For ec opa eval (label checks only):

  • deny entries are failures. Each includes code, msg, and optionally effective_on (a date when the rule becomes enforced — future dates mean the rule is not yet active).
  • warn entries are non-blocking warnings for optional labels.
  • Empty arrays mean the image passes all label checks.

Updating Cached Data

The fetched policy and rule data are point-in-time snapshots. To pick up upstream changes:

# Re-fetch the policy rego files (for ec opa eval)
rm -rf /tmp/ec-policy
ec fetch policy \
    --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
    --dest /tmp/ec-policy

# Update the rule data (for ec opa eval)
git -C /tmp/rhtap-ec-policy pull

The ec validate image command fetches policy and data bundles on each run based on the policy.yaml references, so no manual update is needed.

3.4 - Upgrading Docsy

How to upgrade the Docsy theme and forward-port our layout overrides.

Overview

Both Hummingbird documentation sites use Docsy as their Hugo theme, installed as a Hugo module. Docsy releases roughly quarterly. This runbook covers how to upgrade to a new Docsy version.

The public docs (documentation) own all layout overrides. The internal docs (infrastructure/internal-docs/) only have build scaffolding and pick up layouts from the public docs via Hugo module import.

Layout override inventory

We override four Docsy templates: three to inject the companion page banner and sidebar lock icons, and one to add llms-full.txt cross-linking. Each override is a copy of the upstream template with a small patch applied.

layouts/docs/_td-content.html

Overrides Docsy’s shared layouts/_td-content.html for docs pages only.

Patch: One line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/docs/list.html

Overrides Docsy’s layouts/docs/list.html for docs section pages.

Patch: Same one line added after the description lead, before <header>:

{{ partial "companion.html" . }}

layouts/partials/sidebar-tree.html

Overrides Docsy’s layouts/_partials/sidebar-tree.html for the navigation sidebar.

Patch: One snippet inserted in two places (foldable and non-foldable branches), after the {{ $s.LinkTitle }} </span> and before </a>:

{{- if isset $s.Params "companion" }}{{ if not $s.Params.companion }} <i class="fa-solid fa-lock companion-icon internal-only" title="Internal only"></i>{{ end }}{{ end -}}

layouts/all.md

Overrides Docsy’s layouts/all.md (the catch-all markdown output template).

Patch: One line changed in the LLMS index section to add a link to llms-full.txt alongside the existing llms.txt link.

Other overrides (no forward-porting needed)

These files are fully custom and don’t track an upstream template:

File Purpose
layouts/shortcodes/include.html Custom file-include shortcode
layouts/_default/_markup/render-link.html Link rewriting for hummingbird URLs and /l/ aliases
layouts/_default/_markup/render-heading.html Delegates to Docsy’s td/render-heading.html
layouts/partials/companion.html Environment-aware companion page banner
layouts/partials/breadcrumb.html Custom breadcrumb with stable-URL display
layouts/partials/footer.html Fully custom 3-column footer
layouts/home.html Custom home page with cover image and feature cards
layouts/index.llms.txt Custom recursive llms.txt with full doc tree
layouts/index.llms-full.txt Full content dump for LLM consumption
layouts/partials/llms-doc-tree.txt Recursive tree-walking partial for llms.txt
layouts/partials/llms-full-doc-tree.txt Recursive content partial for llms-full.txt

Upgrade procedure

1. Read the release notes

Check the Docsy changelog and the release upgrade guide for the target version. Note any breaking changes that affect templates we override.

2. Update the Hugo module

From the documentation repo worktree:

# Update to new version (replace vX.Y.Z with the target version)
podman run --pull=newer -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy"

3. Update npm dependencies

Docsy v0.16.0+ sources Bootstrap and Font Awesome from npm:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod npm pack && npm install --no-package-lock"

4. Forward-port layout overrides

For each of the four tracked overrides:

  1. Find the new upstream template in the Hugo module cache:

    find .hugo_cache -path "*docsy/theme@vX.Y.Z*" \
      -name "_td-content.html" -o \
      -name "sidebar-tree.html" -o \
      -name "all.md" -o \
      \( -name "list.html" -path "*docs*" \)
    
  2. Compare the new upstream against our current override to see what changed upstream.

  3. Start from the new upstream template and re-apply the patch documented in the inventory above.

  4. For docs/list.html, keep all upstream hooks (especially {{ .Render "_td-content-after-header" -}}).

5. Verify dark mode and code highlighting

The site uses Docsy’s light/dark mode toggle and dark-aware code syntax highlighting. After upgrading, confirm these config and SCSS settings are still in place:

  • config.yaml: params.ui.showLightDarkModeMenu: true
  • config.yaml: markup.highlight.noClasses: false (required for CSS-based Chroma themes to work)
  • assets/scss/_styles_project.scss: @import 'td/code-dark'; (loads light/dark Chroma themes: friendly for light, native for dark)

If a Docsy upgrade changes the dark mode mechanism, check the Look and Feel docs for updated instructions.

6. Build and validate

make build  # or: make build-host (if Hugo/Go are available locally)

The build should complete with no errors and no deprecation warnings.

Validation checklist

After upgrading, verify:

  • Build passes without errors or deprecation warnings
  • Companion banners render on docs pages with companion: true front matter
  • Lock icons appear in the internal site sidebar for internal-only pages
  • Home page cover image and search render correctly
  • Light/dark mode toggle works and all page elements adapt (navbar, sidebar, footer, code blocks)
  • Code blocks have dark background with readable syntax colors in dark mode
  • Agent-support outputs generate, if enabled (.md URL variants, /llms.txt, /llms-full.txt)
  • Stable URL aliases (/l/...) still resolve

For the internal site, also build infrastructure/internal-docs/ after updating its config/internal/config.yaml and go.mod to the same Docsy version.

Updating the internal site

The internal site only needs config and module changes (no layout work):

  1. Update internal-docs/config/internal/config.yaml: change the theme: list to match the public site.

  2. Update the Hugo module and rebuild:

    cd internal-docs
    make container
    # Inside container:
    hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy
    exit
    make setup  # npm dependencies
    make build
    

Rollback

To revert to a previous Docsy version:

podman run --pull=never -it --rm -u 0 \
  -v $(pwd):/src:z -w /src \
  -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
  quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
  "hugo mod get github.com/google/docsy/theme@vPREVIOUS && hugo mod tidy"

Then restore the layout overrides from the previous commit (git checkout HEAD~1 -- layouts/).

3.5 - Version Constraints for Multi-Version Images

Overview

Version constraints prevent unwanted major or minor version updates for multi-version container images (e.g., python-3-11, nodejs-20). Constraints are defined in properties.yml and validated during make check by comparing VERSION files against defined patterns.

Adding Version Constraints

Edit the image’s properties.yml to add a version constraint:

---
main_package: python3.11
version_constraints: '3.11.*'  # Allow 3.11.X, block 3.12+ and 4.X

The constraint applies to all distros for the image:

---
main_package: dotnet-sdk-8.0
version_constraints: '8.*'  # Applies to rawhide, hummingbird, and any other distros

Constraint Patterns

Constraints use glob-style patterns:

  • 3.11.* - Match any 3.11.X version (recommended for version families)
  • 20.* - Match any 20.X version
  • 8.* - Match any 8.X version
  • 1.25.* - Match any 1.25.X version (for Go-style 3-part versions)

The pattern matches against the version from the VERSION file. For example, if the VERSION file contains 3.11.14, the constraint 3.11.* will match.

How Validation Works

  1. Lockfile Generation: When make all runs, lockfiles are generated with resolved package versions
  2. VERSION File Creation: generate_jinja2.py creates VERSION files from resolved package versions
  3. Constraint Checking: When make check runs, check_version_constraints.py validates VERSION files against constraints
  4. Failure: If a violation is detected, the build fails with a detailed error message
  5. Renovate Integration: Renovate MRs that violate constraints fail CI and are blocked from automerge

Handling Constraint Violations

When a build fails due to a version constraint violation, you have several options:

Option 1: Accept the new version (create new image)

If the new major/minor version is acceptable:

  1. Create a new image directory (e.g., images/python-3-14/)

  2. Copy properties and configuration from the old version

  3. Update the constraint in the new directory:

    version_constraints: '3.14.*'
    
  4. Update repository field to group versions together

  5. Consider adding latest tag to the new version if appropriate

Option 2: Block the version update (exclude package)

If the new version should be blocked entirely:

  1. Add the package to yum-repos/*.repo excludepkgs
  2. Follow the procedure in Excluding Packages from Images
  3. Renovate will skip the excluded version in future updates
  4. When the issue is resolved, remove the exclusion

Option 3: Update the constraint (allow version bump)

If the version bump is acceptable for the existing image:

  1. Update the constraint in properties.yml:

    version_constraints: '3.12.*'  # Allow 3.12.X now
    
  2. Consider if this changes the image’s purpose (major version change)

  3. Update image documentation and tags accordingly

  4. Consider renaming the image directory to reflect the new version

Examples

Python Multi-Version Images

# images/python-3-11/properties.yml
distros:
  - hummingbird
main_package: python3.11
repository: python
version_constraints: '3.11.*'
# images/python-3-13/properties.yml
distros:
  - hummingbird
main_package: python3.13
repository: python
version_constraints: '3.13.*'

Node.js Multi-Version Images

# images/nodejs-20/properties.yml
main_package: nodejs20
repository: nodejs
version_constraints: '20.*'
# images/nodejs-24/properties.yml
main_package: nodejs24
repository: nodejs
version_constraints: '24.*'

.NET SDK Multi-Version Images

# images/dotnet-sdk-8-0/properties.yml
main_package: dotnet-sdk-8.0
repository: dotnet-sdk
version_constraints: '8.*'

.NET Runtime Multi-Version Images

# images/dotnet-runtime-8-0/properties.yml
main_package: dotnet-runtime-8.0
repository: dotnet-runtime
version_constraints: '8.*'

ASP.NET Runtime Multi-Version Images

# images/aspnet-runtime-8-0/properties.yml
main_package: aspnetcore-runtime-8.0
repository: aspnet-runtime
version_constraints: '8.*'

OpenJDK Multi-Version Images

# images/openjdk-21/properties.yml
main_package: java-21-openjdk-headless
repository: openjdk
version_constraints: '21.*'

Go Multi-Version Images (3-part versions)

# images/go-1-25/properties.yml
main_package: golang1.25
repository: go
distros:
  - hummingbird
version_constraints: '1.25.*'  # Matches 1.25.0, 1.25.3, etc.

Troubleshooting

Constraint not working

  • Verify package name matches exactly (case-sensitive)
  • Ensure pattern syntax is correct (use * wildcard)
  • Run make all to regenerate VERSION files before make check

False positive violations

  • Check for typos in the constraint pattern
  • Verify the pattern matches the version format (e.g., 3.11.* not 3.11*)
  • Ensure you’re using glob patterns, not regex

Package renamed in repositories

If a package is renamed (e.g., python3.11 → python311):

  1. Update main_package field
  2. Update constraint with new package name (not needed - constraints check VERSION files)
  3. Regenerate lockfiles and VERSION files with make all

Implementation Details

The version constraint system consists of:

  1. properties.yml: Optional version_constraints field with a glob pattern string
  2. ci/check_version_constraints.py: Python script that validates VERSION files
  3. Makefile integration: Script runs as part of make check
  4. CI integration: Failures block merge, including Renovate automerge

The validation process:

  1. Reads properties.yml for each image
  2. If version_constraints defined, finds all VERSION files across all distros
  3. Applies the same constraint pattern to all distros for the image
  4. Compares VERSION file content against glob pattern using Python’s fnmatch
  5. Reports violations with file path, version, constraint, and properties file location
  6. Exits with error code 1 if any violations found

See Also

3.6 - Retrying Konflux Checks

Automatically retrigger failed Konflux CI checks on merge requests using the retrigger script.

When Konflux CI checks fail on merge requests, you can automatically retrigger them using the provided retrigger script.

Prerequisites

  • GITLAB_TOKEN: GitLab API access token with api scope and Maintainer access

Usage

For automated retriggers of failed Konflux pipeline runs:

# Retrigger only failed pipeline runs and wait for them to start
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

# Preview what would be retriggered without posting comments
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"

How It Works

The retrigger script will:

  1. Extract failed pipeline runs from the commit statuses on the merge request
  2. Generate /retest pipeline-name commands for each failed run
  3. Post the retest commands as merge request comments (unless --dry-run)
  4. Wait for the pipeline runs to start (unless --dry-run)

Manual Retrigger Commands

You can also manually retrigger specific checks using slash commands in merge request comments:

/retest gitlab-ci--default--main-on-pull-request

To retrigger a specific pipeline run as listed in .tekton/images-on-pull-request.yaml.

See the Tekton Pipelines as Code documentation for more details on available commands.

Examples

# Set your GitLab token
export GITLAB_TOKEN=your_token_here

# Retrigger failed checks for a specific merge request
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

# Dry run to see what would be retriggered
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"

The script will output which pipeline runs are being retriggered and wait for them to start before completing.

3.7 - Switching Fedora Streams

AI Agent Note: When asked to switch Fedora streams, follow the workflow below. The repo file naming convention determines the GPG key automatically.

Overview

The rawhide distro can be pinned to a specific Fedora release branch instead of following the rolling Rawhide stream. This provides stability during the period when Rawhide moves to N+1 but the current stable release is still N-1.

For example, when Fedora 44 branches from Rawhide, the rawhide distro can be pinned to Fedora 44 packages while Rawhide itself starts tracking Fedora 45.

Repo ID naming convention

All Fedora repo files use a fedora-<stream> naming convention for their repo IDs, where <stream> matches the filename stem:

  • fedora-43.repo uses [fedora-43], [fedora-43-source], [fedora-43-updates], etc.
  • fedora-44.repo uses [fedora-44], [fedora-44-source], etc.
  • fedora-rawhide.repo uses [fedora-rawhide], [fedora-rawhide-source], etc.

This convention ensures:

  • No DNF repo ID conflicts when multiple repo files coexist in yum-repos/
  • Predictable, greppable repo IDs in lockfiles
  • The GPG key name can be derived automatically from the filename

Workflow

1. Create or update the repo file

Create a new file yum-repos/fedora-<version>.repo (e.g., fedora-45.repo).

The repo file must follow these conventions:

  • Repo IDs match the filename stem: Use [fedora-45], [fedora-45-debuginfo], and [fedora-45-source] as section names
  • Version-specific URLs: Point to the correct Fedora version’s repository (e.g., /development/45/ or /releases/45/)
  • Version-specific GPG key: Reference the correct signing key (e.g., RPM-GPG-KEY-fedora-45-$basearch)

Use an existing repo file (e.g., yum-repos/fedora-44.repo) as a template.

Example for a development branch:

[fedora-45]
name=Fedora 45 - Developmental packages for the next Fedora release
baseurl=https://koji-s3-cache.hummingbird-project.io/download-ib01.fedoraproject.org/pub/fedora/linux/development/45/Everything/$basearch/os/
enabled=1
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-45-$basearch
skip_if_unavailable=False

2. Update variables.yml

Edit images/variables.yml and change the default_variant_repos.rawhide entry to reference the new repo file:

default_variant_repos:
  rawhide:
    - fedora-45.repo

3. Regenerate all derived files

Run regeneration to update rpms.in.yaml files, Containerfiles, and lockfiles:

make -j 16 FORCE=true

4. Verify GPG key derivation

The Jinja templates in macros/setup_newroot.yml.j2 automatically derive the GPG key name from the repo filename:

  • fedora-45.repo becomes RPM-GPG-KEY-fedora-45-primary
  • fedora-44.repo becomes RPM-GPG-KEY-fedora-44-primary
  • fedora-rawhide.repo becomes RPM-GPG-KEY-fedora-rawhide-primary

No template changes are needed when switching versions. Verify by checking the generated Containerfile for the builder image contains the correct key:

grep rpmkeys images/hummingbird-builder/rawhide/default/Containerfile

Switching back to rolling Rawhide

To switch back to the rolling Rawhide stream, update variables.yml to reference fedora-rawhide.repo:

default_variant_repos:
  rawhide:
    - fedora-rawhide.repo

Then regenerate all derived files with make -j 16 FORCE=true.

3.8 - Disabling Rawhide for Images

AI Agent Note: When asked to disable Rawhide for images, follow the workflow below. Run make all after editing properties to clean up stale directories automatically.

Overview

When Rawhide is unstable (mass rebuild, broken critical packages, Fedora branching), Rawhide variants can be temporarily disabled for specific images or globally. This stops building and testing Rawhide variants while keeping Hummingbird variants unaffected.

Disabling Rawhide for a single image

1. Edit properties.yml

Add an explicit distros field to images/<name>/properties.yml that excludes rawhide:

distros:
  - hummingbird

This overrides the default distros (which include rawhide) for this image only.

2. Regenerate

make all

The stale Rawhide directories are removed automatically. Generated resources (.gitattributes, Konflux templates, Tekton pipelines) are updated to exclude the Rawhide variants.

3. Commit and merge

Commit the properties change together with the removed generated files.

Disabling Rawhide for all images

1. Edit variables.yml

Remove rawhide from default_distros in images/variables.yml:

default_distros:
  - hummingbird

2. Edit per-image overrides

Images that explicitly list distros including rawhide in their own properties.yml are not affected by the variables.yml change. Edit those files to remove rawhide as well.

3. Regenerate

make all

4. Commit and merge

Commit all properties changes together with the removed generated files.

Re-enabling Rawhide

Reverse the properties change (add rawhide back to default_distros or remove the per-image distros override), then regenerate:

make all

The Rawhide directories no longer exist, so Make creates all targets from scratch.

Verification

After running make all:

  • git status shows Rawhide directories as deleted (when disabling) or new files (when re-enabling)
  • .gitattributes no longer lists Rawhide paths for affected images
  • konflux-templates/rendered.yml and .tekton/ pipelines no longer reference Rawhide variants for affected images

See Also

3.9 - MR Artifact Promotion

Overview

An RPM package releases the image built and tested in its merge request (MR) by default. Merging the MR starts a promotion PipelineRun, not another RPM build. The promoted image keeps the MR source revision and digest; the merge commit records why it was released.

The promotion flow, from the merge-commit parents through MR Snapshot lookup and the push-style Component Snapshot to the automatic Release, is described in MR artifact promotion. On the RPM side, the MR build produces a digest-pinned OCI image containing unsigned RPMs, required Testing Farm tests gate the merge, and Release Service adds duplicate-RPM filtering and Pulp publication to the managed Conforma, signing, and publication checks.

Prerequisites

  • The package uses two-parent merge commits on main, and its MR build and required tests passed.
  • The Component’s build ServiceAccount can push source Trusted Artifacts to its Quay image repository and use the pipeline SCC. The hummingbird-promotion ServiceAccount can read build evidence and create Snapshots; it must be able to read the source Trusted Artifact.
  • The Hummingbird tools image contains both promotion commands and is pinned by digest. The Hummingbird Tools Task reference describes their inputs and validation.
  • The MR Snapshot, linked PipelineRun, and image remain available through merge and retry. KubeArchive can supply removed Konflux objects but cannot restore an expired image; check the actual Quay expiry for the selected digest and its tags and do not assume a fixed retention period.

Choose the push pipeline

Imported packages use promotion by default. Set push_pipeline: build in ci/package-overrides.yaml only for a documented exception, then run make generate. The generator places each package’s push-triggered PipelineRun in .tekton/rpms-on-promotion.yaml or .tekton/rpms-on-push.yaml accordingly. Every package remains in trigger-manifest-push.json (see trigger manifests documentation) so its pushes reach Pipelines as Code. Remove the build override to restore promotion.

Review the generated YAML and confirm the Hummingbird tools image digest is current. Keep the MR build in .tekton/rpms-on-pull-request.yaml. A package merge should start one promotion run and no push-build run.

Verify a promotion

Find these resources in the tenant’s Konflux UI or with oc. Follow their names and links rather than treating the promotion run as a build:

oc -n hummingbird-tenant get pipelineruns -l appstudio.openshift.io/component=syft-main
oc -n hummingbird-tenant get snapshots -l appstudio.openshift.io/component=syft-main
oc -n hummingbird-tenant get releases -o custom-columns=NAME:.metadata.name,SNAPSHOT:.spec.snapshot

Select the promotion PipelineRun for merge commit M; its COMPONENT_SNAPSHOT result names the created Snapshot. Its annotations name the tested MR Snapshot and build PipelineRun. Find the Release whose spec.snapshot matches, then follow its managed PipelineRun.

Resource Check
MR build PipelineRun type=build, MR event, successful build result, source revision B, and digest D.
Tested MR Snapshot The package’s Component entry has source B and image D; required MR tests passed.
Promotion PipelineRun type=promotion, merge revision M, three Tasks (clone-repository, find-tested-mr-artifact, create-promotion-component), and a COMPONENT_SNAPSHOT result.
Promotion Component Snapshot One Component entry with the same source B and image D; type=component, push event and Component labels; annotations link M and the source evidence. Its start-time annotation comes from the promotion PipelineRun.
Automatic Release spec.snapshot names the promotion Snapshot. Check its conditions and managed PipelineRun; the promotion Task did not create it.
Managed release Confirm Conforma and Pulp Tasks succeeded. Compare signed RPM SHA256 values in the Release’s status.artifacts.rpmfiles with matching Pulp package records. Signing changes RPM bytes, so these hashes need not equal the unsigned RPM hashes inside image D.

The Component Snapshot normally reports that no required IntegrationTestScenarios were found and testing was skipped. The promotion Task fails before Snapshot creation if a configured scenario could retest it. Verify that the MR image digest and the Component Snapshot digest are identical; this proves promotion reused the tested image. Look up the merge SHA in Konflux PipelineRuns to confirm that no push build ran for this Component.

The clone Task runs under the Component build ServiceAccount and stores the source Trusted Artifact in the Component’s Quay repository. Lookup and Snapshot creation run under hummingbird-promotion.

Recover a failed run

Failure Action
Clone cannot push or restore the source Trusted Artifact Check the Component build ServiceAccount, its Quay access, and the hummingbird-promotion read path. The clone Task needs the pipeline SCC.
Lookup cannot find passed MR evidence Check the MR Snapshot, its build PipelineRun, required test result, and KubeArchive access. If the evidence cannot be recovered, build and test again before release.
Package source changed on main after M Stop the stale promotion. Use a fresh MR build and tests for the newer package inputs.
Tested image expired or cannot be pulled Do not create a Snapshot with a missing image. Rebuild and run the required tests before release.
Snapshot creation hits quota or finds a conflict Check the tenant quota or the existing deterministic Snapshot. A retry reuses only a matching live Snapshot; it does not reuse an archived or conflicting one.
Promotion Task or PipelineRun needs a fix Start a new promotion PipelineRun for the original merge SHA using the corrected manifest and Task pins. An old PipelineRun keeps its old Task revisions, and a garbage-collected run cannot be rerun. Do not bump the RPM Release solely to retry promotion.
Release fails after the Component Snapshot exists Recover the failed Release through the Release Service workflow using that Snapshot. Do not create another promotion Snapshot or rebuild RPMs to diagnose a release-service failure.

Record the MR Snapshot, tested digest, promotion PipelineRun and Component Snapshot, automatic Release, managed PipelineRun, Conforma result, and Pulp RPM checksums in the work item. Keep a compact evidence attachment when Konflux resources may be garbage-collected. An automatic Release can fail and later succeed through a retry using the same Snapshot; link both outcomes.

Development

Run python3 ci/generate_resources.py all and pytest test/test_generate_resources.py. Review the generated push, promotion, and MR PipelineRuns for the package.

3.10 - Red Hat Catalog Environment Promotion

Environments

Environment URL Source Deploy trigger
MR Preview GitLab Pages (/mr-{IID}) MR branch Auto on MR pipeline
Experimental images.experimental.hummingbird-project.io/<branch-name> experiment/* branch Auto on push to experiment branch
Staging images.staging.hummingbird-project.io main Auto on merge to main
Production images.redhat.com main Auto on merge to main

Disambiguation: The infra experimental host (images.experimental.hummingbird-project.io, experiment/* branches) is an off-main deploy sandbox. The former in-app /experimental product route was removed in HUM-2171 — use /api instead. See Archived surfaces.

Promotion Paths

Routine: staging to production

  1. MR merged to main
  2. deploy_redhat_catalog_staging and deploy_redhat_catalog run automatically in parallel (both trigger on push to main when catalog files change)
  3. UAT sign-off on staging — product, design, and engineering release approvers complete the UAT checklist and file a record (UAT program runbook)

Exploration: experimental to staging to production

  1. Create branch experiment/<name> from main (must use exactly experiment/, not experimental/ or other prefixes — without it, no deploy triggers)
  2. Push triggers deploy_redhat_catalog_experimental automatically
  3. Branch deploys to images.experimental.hummingbird-project.io/<name>/
  4. Iterate on experimental host for days or weeks
  5. Light UAT (optional spot-check on experimental URL — see UAT program — experimental path)
  6. When ready: open MR from experiment/<name> to main
  7. Normal MR review, merge, then routine promotion path (full UAT on staging)

Retiring an experiment

  1. Delete the experiment/<name> branch

  2. Remove the branch prefix from the experimental S3 bucket:

    aws s3 rm s3://redhat-catalog-experimental-spa/<name>/ --recursive
    
  3. Invalidate the CloudFront cache:

    aws cloudfront create-invalidation \
      --distribution-id <EXPERIMENTAL_DISTRIBUTION_ID> \
      --paths "/<name>/*"
    

Build Configuration Per Environment

The infrastructure pipeline sets these environment variables at build time:

Variable Experimental Staging Production
ASSET_PATH /<branch-name>/ / /
DPAL_USE_STAGING true true (unset, uses prod)
TRUSTARC_DATA_DOMAIN images.experimental.hummingbird-project.io images.staging.hummingbird-project.io images.redhat.com
CATALOG_API_BASE_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
CATALOG_BASE_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
HUMMINGBIRD_API_URL (unset, uses app default) (unset, uses app default) (unset, uses prod)
JIRA_API_URL Jira API URL Jira API URL Jira API URL

Infrastructure

SAM Stacks

Stack Template ResourcePrefix CatalogDomainName
redhat-catalog-prod template.yaml redhat-catalog images.redhat.com
redhat-catalog-staging template.yaml redhat-catalog-staging images.staging.hummingbird-project.io
redhat-catalog-experimental template-experimental.yaml redhat-catalog-experimental images.experimental.hummingbird-project.io

Staging uses the same template.yaml as production (single-site CloudFront distribution). Experimental uses template-experimental.yaml which adds a CloudFront Function for path-prefix SPA routing across multiple branches.

DNS

ALIAS records (images.staging/experimental.hummingbird-project.io → CloudFront) are created automatically by post_deploy.sh after each deploy.

First-time bootstrap: The SAM template creates an ACM certificate with DNS validation. The ACM validation CNAME must exist in the hummingbird-project.io Route 53 zone before the first sam deploy, or CloudFormation will wait 30 minutes and roll back. To bootstrap:

  1. Deploy with CatalogDomainName="" (uses CloudFront default domain, no cert)
  2. Create the ACM validation CNAME in Route 53 (get the value from the ACM console or aws acm describe-certificate)
  3. Redeploy with CatalogDomainName set to the real domain

After the first successful deploy, no manual DNS steps are needed.

CI/CD Variables

Set in GitLab project settings:

Variable Value
REDHAT_CATALOG_STAGING_URL https://images.staging.hummingbird-project.io

Pipeline Flow

experiment/* push:
  redhat-catalog-experimental-check
    -> deploy_redhat_catalog_experimental

MR (target main):
  redhat-catalog (build + test)
    -> redhat-catalog-browser-tests
    -> redhat-catalog-visual-mock (Playwright mock API; MR vs merge-base diff)
    -> pages (preview)

main push:
  deploy_redhat_catalog_staging (auto)
  deploy_redhat_catalog (auto)

Visual regression details: redhat-catalog testing guide — Visual regression and e2e/visual/README.md.

3.11 - Setting Up a Konflux Cluster

How to set up a new Konflux cluster

Setting Up the Cluster

  1. Set up the new cluster in the infrastructure repository, mirroring the existing cluster configuration
  2. In the Konflux UI, create a new Component for the Application with the same name as the git repository
  3. Verify the Pipeline as Code Repository resource is created
  4. Run two successive full pipelines in the infrastructure repository to deploy resources from konflux/rendered.yml and fix up owner references
  5. Delete the initial Component created from the UI
  6. Disable Konflux build status comments (requires elevated permissions):
kubectl patch \
  --namespace hummingbird-tenant \
  repository REPOSITORY_NAME \
  --type merge \
  --patch '{"spec":{"settings":{"gitlab":{"comment_strategy":"disable_all"}}}}'

Next Steps

3.12 - Deleting RPMs from Hummingbird Repos

How to delete RPM packages from Hummingbird repositories

Overview

  • This document should be used to remove RPMs from the Hummingbird repositories

Setup

  • make sure you have followed the local development setup instructions in the infrastructure repository
  • install pulp cli via dnf install pulp-cli
  • setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

Steps

  • Note the names of the rpms to delete. In this example, we’ll use systemd-stub
  • Obtain the relevant metadata required to delete.
    • You need to execute this command once for each architecture.
pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.pulp_href)"'

or for prettier content,

pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.name) \(.version)-\(.release)  \(.arch)  \(.pulp_created)  \(.pulp_href)"'

Example output from basic command:

/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/
  • Using the hrefs, construct your delete commands:
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/

3.13 - Red Hat Catalog UAT Program

User Acceptance Testing (UAT) for the Red Hat Catalog SPA — human sign-off on staging before production promotion.

Epic: HUM-2179 · Story: HUM-2183 · Depends on: HUM-2060 (staging host + promotion gates)

Purpose

UAT complements automated validation (smoke, E2E, visual regression, security scans). It catches product, content, and UX issues that automation does not cover. UAT does not replace automated gates.

Layer Owner Blocks prod?
Unit / lint / MR security CI MR merge
UAT sign-off Product + design + eng Human review before prod is live
Post-prod smoke CI / on-call Rollback decision

Environments (disambiguation)

Term Meaning URL pattern
Staging Pre-prod validation target for UAT https://images.staging.hummingbird-project.io
Experimental host Infra sandbox for experiment/* branches https://images.experimental.hummingbird-project.io/<branch>/
Former /experimental route Removed in-app product route (HUM-2171) Returns 404 — use /api instead. See archived-surfaces.md

See Red Hat Catalog Environment Promotion for full promotion paths.

Promotion gates

Staging → production (full UAT)

Both deploy_redhat_catalog_staging and deploy_redhat_catalog trigger automatically on merge to main (when catalog files change). UAT is a human review layer that runs concurrently with the automated deploys — if UAT reveals issues, the release owner must roll back production.

Required before a release is considered complete:

  1. deploy_redhat_catalog_staging succeeded on main
  2. MR-blocking security jobs clean (or approved exceptions) — see security-scanning.md
  3. UAT sign-off recorded (checklist + Jira comment; git record in redhat-catalog/docs/uat-records/)

Light UAT on experimental host

Experimental hosts are for iteration, not customer release. When opening an MR from experiment/<name> to main, use a light UAT pass on the experimental URL — not the full checklist:

Check Required?
Feature under test works on experimental host Yes
No console errors on primary journey Yes
Masthead/footer not broken Yes
Full staging UAT checklist No — runs on staging after merge
Named approver sign-off Eng owner only (no prod gate yet)

After merge, staging receives the full UAT before prod.

Named approvers (staging → prod)

Confirm names with the product owner during checklist review (HUM-2060 open question on approval owners). Until confirmed, use these roles:

Role Primary Backup Responsibility
Product TBD — Catalog journeys, copy, feature completeness
Design TBD — Visual polish, layout, theme, responsive UX
Engineering release TBD TBD Pipeline green, staging commit matches intent, prod deploy authorization

Any Block from product or design stops prod promotion until defects are fixed on staging and UAT re-run.

Runbook — request and run UAT

1. Request UAT (release owner)

When main pipeline completes staging deploy + automated gates:

  1. Create or reuse a Release ticket in Jira (label redhat-catalog, link HUM-2179 epic).
  2. Comment with pipeline URL, commit SHA, and summary of user-facing changes.
  3. Label ticket uat-requested.
  4. Notify approvers (#hummingbird or team channel — use team default).

2. Execute UAT (approvers / delegates)

  1. Open uat-checklist.md against staging.
  2. File results in redhat-catalog/docs/uat-records/YYYY-MM-DD-<topic>.md.
  3. Log defects as Jira bugs linked to the release ticket.

Expected turnaround: 2 business days from uat-requested for routine releases; same-day for hotfixes when approvers are available.

3. Sign off or block

  • Pass: All checklist items pass or have accepted exceptions; all three approver rows signed; Jira comment posted (template below); label uat-pass.
  • Fail: Label uat-fail; roll back production if already deployed, fix defects on staging, and re-run UAT.

4. Record production promotion (engineering release)

  1. Verify Jira release ticket has uat-pass and linked sign-off record.
  2. Production deploys automatically on merge to main — confirm deploy_redhat_catalog succeeded in the GitLab main pipeline.
  3. Comment on release ticket with prod pipeline URL.

What blocks release

Blocker Who resolves
Staging smoke or visual CI failed Engineering
Open Critical/High security finding without exception Engineering + security
UAT checklist failure (product/design) Engineering fixes → re-UAT
Missing approver sign-off Approver
Staging commit ≠ intended release SHA Release owner

Jira sign-off template

Post on the release ticket when UAT completes:

h3. UAT sign-off — staging → prod

*Staging URL:* https://images.staging.hummingbird-project.io
*Commit:* {full SHA}
*Pipeline:* {GitLab pipeline URL}
*Checklist record:* redhat-catalog/docs/uat-records/{filename}.md

||Role||Name||Decision||
|Product|{name}|Approve / Block|
|Design|{name}|Approve / Block|
|Engineering release|{name}|Approve / Block|

*Result:* Approved for prod / Blocked
*Defects:* {HUM-xxx links or "none"}
*Tester:* {name}, {date}

Suggested Jira labels

Label Meaning
uat-requested Staging ready; waiting for human validation
uat-pass Full sign-off complete
uat-fail Blocked; defects logged
uat-light Experimental → staging spot-check only

Optional custom field (if added in Jira project settings): UAT status (Not started / In progress / Passed / Failed).

Release ticket template

Create a Jira Task or Release issue per prod promotion:

## Release summary

- **Target:** staging → production
- **Commit:**
- **User-facing changes:**

## Automated gates

- [ ] Staging deploy green
- [ ] Production deploy green (auto-triggered on merge)
- [ ] Security scans clean (or exceptions documented)

## UAT

- [ ] Checklist completed ([uat-checklist.md](link))
- [ ] Sign-off record filed ([uat-records/](link))
- [ ] Product approve
- [ ] Design approve
- [ ] Engineering release approve

## Prod verify

- [ ] deploy_redhat_catalog succeeded in main pipeline
- [ ] Post-deploy verification

Out of scope

  • Automating UAT (Playwright covers regression; UAT stays human)
  • Legal/compliance sign-off (separate security program)

3.14 - Updating Dist-git Packages

How packages are automatically updated from Fedora dist-git

Packages are automatically updated from Fedora dist-git. Each update creates a separate MR that is automatically approved and merged when CI passes.

Packages configured for MR artifact promotion release the image built and tested in that MR after it merges. The operating guide shows how to follow the promotion PipelineRun, automatic Release, and Pulp publication, including retries. Other packages build again on main.

Portable package filenames

Import, update, and sync escape Windows-invalid filename characters with ~ followed by their hexadecimal character code. For example, fix:one.patch becomes fix~3aone.patch. Reserved device names and trailing dots or spaces are escaped too. Safe filenames stay unchanged. If an escaped name would collide with another filename, including on a case-insensitive filesystem, sanitization fails before changing files.

Packaging specs, scripts, and configuration files use local names. Patch contents and source archives stay unchanged. Download URLs keep their original locations. A tracked Source or Patch declared by URL gets an RPM #/ suffix for its local filename. The sources file keeps Fedora lookaside names.

Merges normalize both the old and new Fedora snapshots before applying local changes. The transformation is deterministic and idempotent, so no filename map is stored in metadata. Modification checks and ordinary diff compare sanitized upstream trees. Use diff --raw to see the filename substitutions themselves. Sanitization does not change Release, version, or modification status.

To migrate existing packages:

make dist-git ARGS='--dry-run sanitize-paths --all'
make dist-git ARGS='sanitize-paths patch'
# Or migrate every package:
make dist-git ARGS='sanitize-paths --all'

This command leaves changes in the working tree without committing. It includes tracked files and new files that Git does not ignore, and skips downloaded lookaside archives. Review and commit the renamed files and updated references together. For a clean Fedora package, include its existing Upstream: SHA trailer in the package commit to preserve history validation.

make check runs dist_git.py check-paths on tracked paths throughout the repository. Stage the migration before checking, so Git’s index contains the new names. The check also rejects Windows reserved names and paths that differ only by case.

New clones can check out the corrected tree on Windows. Older commits still contain the original filenames, so checking out those commits requires a filesystem that supports them.

dist_git.py --dry-run policy

--dry-run preserves each command’s established behavior. It is not a universal side-effect-free preview: several commands still modify package files or metadata. Place the flag before the command, for example make dist-git ARGS='--dry-run update bash'. Each command’s --help also describes its policy.

Command Behavior with --dry-run
import Imports files, writes metadata, and regenerates resources, without staging or committing.
update Updates package files, but does not save metadata, run source pipelines, commit, or write recovery state. Existing safety checks still apply.
update --continue Checks for unresolved conflict markers and reports the intended commit, without source verification, spec finalization, or committing.
sync Replaces package files, but does not save metadata, run source pipelines, commit, or write recovery state. With --mark, files are left unchanged only when the requested SHA already matches the recorded SHA and the branch is unchanged. If upstream has advanced or the branch changes, normal sync still replaces files and can discard local edits, even with --mark.
rebuild Modifies spec releases without committing.
rebuild-rev-deps Modifies reverse-dependency spec releases without committing.
sanitize-paths Previews portable path changes without changing files.
add-private-product Previews configuration and infrastructure changes without contacting Pulp, modifying files, or committing.
mark-modified, set-upstream Still write metadata. These commands never commit, with or without --dry-run.
rename, update-releases, list, diff, check-paths, ls-sources Reject the flag before executing the command.

Compatibility decision: mark-modified and set-upstream continue accepting --dry-run because existing callers use it for their metadata-only workflows. The flag does not turn either command into a preview or suppress metadata writes. Other unsupported combinations, including read-only commands where the flag would be ignored, are rejected explicitly. A new side-effect-free preview mode is outside this policy.

Version-validation failures exit with code 1, both with and without --dry-run. A normal update saves recovery state for correction and update --continue. A failed dry-run does not create recovery state, so it cannot be resumed with --continue. Its package-file edits remain available for inspection; correct the problem before starting another update.

Testing Locally

# Dry-run (check all packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone

# Dry-run (check first 5 packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone --max-packages=5

# Check only a specific package (for testing/debugging)
./ci/dist_git_update_multi_mr.sh --clone --only-package=libgcrypt

# Check only clean packages (skip modified/independent)
./ci/dist_git_update_multi_mr.sh --clone --clean-only

# Check only modified packages (skip clean/independent)
./ci/dist_git_update_multi_mr.sh --clone --modified-only

# Create up to 3 test MRs (checks all packages, stops after finding 3 updates)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-updates=3 --create-mrs

# Check first 10 packages, create up to 3 MRs
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-packages=10 --max-updates=3 --create-mrs

# Create MRs only for clean packages (skip modified/independent)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --clean-only --create-mrs

Environment Variables

  • CHORE_MR_GITLAB_TOKEN - GitLab token with api and write_repository scopes (required for --create-mrs and resolved diagnostic-MR cleanup)
  • CHORE_MR_APPROVAL_GITLAB_TOKEN - GitLab token with api scope for auto-approving MRs (used by CI)
  • GITLAB_REMOTE_URL - Target repo (default: https://gitlab.com/redhat/hummingbird/rpms.git)

Flags

  • --clone - Clone from GitLab to /tmp (safe for local testing, uses latest main branch)
  • --max-packages=N - Check only the first N packages (limits input set)
  • --max-updates=N - Stop after finding N updates (limits output MRs created)
  • --create-mrs - Actually create MRs (requires token)
  • --only-package=NAME - Check only the specified package (for testing/debugging specific packages)
  • --clean-only - Skip packages with modification_status of ‘modified’ or ‘independent’, only process clean packages
  • --modified-only - Skip packages with modification_status of ‘clean’ or ‘independent’, only process modified packages (mutually exclusive with --clean-only)

Using –clean-only

The --clean-only flag filters out packages marked as ‘modified’ or ‘independent’ before attempting updates. This is useful for:

  1. Better failure detection - Exit code 1 indicates real update failures, not expected errors from modified/independent packages
  2. Cleaner output - No error messages for packages that can’t be auto-updated by design
  3. Efficient CI - Focus on packages that should update automatically
  4. Performance - Avoids invoking dist_git.py for packages that will fail

Without --clean-only, the script attempts to update all packages. Modified/independent packages fail with:

ERROR: Cannot auto-update <package>
       Status: modified/independent
       Reason: <reason>
       Use 'sync' to force update or 'mark-modified --clean' to allow updates

These expected failures can mask genuine update issues. Using --clean-only prevents these false failures.

Using –modified-only

The --modified-only flag filters to only process packages marked as ‘modified’, skipping clean and independent packages. This is useful for checking the merge logic, as well as getting an overview of current merge conflicts.

Auto-Merge and Auto-Approval

MRs created by this script are configured to:

  • Auto-merge when pipeline succeeds (set via merge_request.merge_when_pipeline_succeeds)
  • Auto-approve after 10 minutes via the chore_mr_approval CI job (gives Konflux time to post commit statuses)

Version-Constrained Updates

Packages with track_upstream set to a version prefix in their metadata are version-constrained. This is used for versioned packages like golang1.26 that track a specific upstream version line.

How It Works

The track_upstream and release_monitoring_project_id metadata fields work together across two systems:

dist_git.py update (dist-git sync from Fedora):

When track_upstream is a version prefix (e.g., "1.26"), the update command uses prefix matching:

  • 1.26 matches 1.26, 1.26.0, 1.26.3 (allowed)
  • 1.26 does not match 1.27.0, 2.0 (skipped)

Skipped packages log a warning:

WARNING: Skipping golang1.26: upstream version 1.27.0 doesn't match tracked version 1.26

check_upstream_versions.py (release-monitoring.org checks):

When release_monitoring_project_id is a string, Anitya is queried using that name instead of the RPM package name. For example, golang1.26 with release_monitoring_project_id: "golang" queries Anitya for golang. When it is an integer, the v2 API is queried directly by project ID. When track_upstream is a version prefix, the list of upstream versions returned by Anitya is filtered to only those matching the prefix. This means check_upstream_versions.py check will report 1.26.5 as the latest version for golang1.26 even if Anitya reports 1.27.0 as the latest golang release.

When --update is used, the script updates the spec file, downloads new sources, and commits the result. Packages that need custom update logic can provide a hooks file at metadata/<package>.update-hooks.yaml to override the spec update, source download, or add a post-update step. See Package Modification Tracking for details.

Setting Up Version Constraints

# Constrain golang1.26 to only receive 1.26.x updates
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Remove the version constraint
./ci/dist_git.py set-upstream golang1.26 --no-track-version

Behavior

  • dist_git.py: Version constraint is checked before pre-release and Koji build checks
  • dist_git.py: The sync command bypasses the version constraint (explicit force operation)
  • check_upstream_versions.py: release_monitoring_project_id determines the Anitya lookup (int for project ID, string for name, absent for RPM name); track_upstream filters versions when set to a prefix
  • Batch operations continue processing other packages after skipping constrained ones

Pre-Release Version Filtering

By default, the update mechanism skips pre-release versions to prevent unstable packages from entering the repository automatically. Pre-release patterns include:

  • Tilde notation: 5.3.0~rc1, 2.0~beta1, 1.0~alpha (RPM standard)
  • Suffix notation: 5.3.0-rc1, 2.0.beta1, 3.0-alpha, 1.0.dev
  • Development markers: 1.5git20240101, 2024.01.snapshot, 1.0dev

Manual Override

To explicitly update to a pre-release version:

# Update single package to pre-release version
./ci/dist_git.py update --allow-prerelease package-name --skip-build-check

# Batch update allowing pre-releases
./ci/dist_git.py update --allow-prerelease --skip-build-check

Pre-release filtering behavior

  • Pre-release detection occurs before Koji build checks (saves API calls)
  • Skipped packages log a warning with the detected pattern
  • Batch updates continue processing other packages
  • The sync command bypasses this check (explicit force operation)

3.15 - Writing Documentation

How to contribute to the Project Hummingbird documentation.

Overview

New documentation is always welcome! This guide explains how to contribute to the Project Hummingbird documentation.

Documentation Sites

The Project Hummingbird documentation is open by default and available at two sites:

Documentation Repositories

Documentation content comes from six repositories:

  1. Containers repository - Documentation focused on container images (cross-cutting layout):

    • documentation/contributing/ - How to add and modify container images
    • documentation/background/ - Container image architecture and reference docs
    • documentation/ci-scripts/ - CI script documentation
    • documentation/operating/ - Operator runbooks for image maintenance
    • documentation/using/ - End-user guides (custom CA certificates, etc.)
    • README.md - User guide
    • CONTRIBUTING.md - Quickstart guide
  2. RPMs repository - Documentation focused on RPM packages (cross-cutting layout):

    • documentation/background/ - RPM pipeline architecture and Konflux deployment
    • documentation/operating/ - Operator runbooks for package management
    • CONTRIBUTING.md - Contributing guide for RPM packages
  3. Tools repository - Documentation for infrastructure tools and services (cross-cutting layout):

    • documentation/background/ - Cross-cutting pages: architecture (message bus, data flow, artifact promotion, shared contracts) and team workflow guides
    • documentation/components/ - One page per tool or service; components with design docs or DDRs are page bundles: a directory with _index.md as the main page plus child pages like design.md/design-decisions.md
    • documentation/operating/ - Operator runbooks for tools and services

    The contributing/ and using/ mounts are pre-provisioned for future use; the shared Contributing section is containers-scoped, so the team workflow guides live in background/.

  4. K8s test pipeline repository - Documentation for Kubernetes integration testing (per-component layout):

    • documentation/ - Pipeline design, test format, and EaaS debugging

    Section mounts are pre-provisioned for a future split into the standard subdirectories.

  5. Public documentation repository - Site infrastructure and cross-cutting guides:

    • content/docs/using/ - End-user guides for using container images
    • content/docs/operating/ - Infrastructure operation guides (Konflux, GitLab, etc.)
    • content/docs/background/ - General architecture and concepts
  6. Infrastructure repository - Internal-only documentation (cross-cutting layout):

    • documentation/operating/ - Internal operational docs (AWS, DNS, GitLab, etc.)
    • documentation/background/ - Internal background docs (error budgets, etc.)
    • documentation/presentations/ - Team presentations and talks

How They Work Together

The repositories are layered using Hugo Modules:

  1. Public documentation imports specific directories from the containers, rpms, tools, and k8s-test-pipeline repositories
  2. Internal documentation is built from the infrastructure repository’s internal-docs/ subtree. It imports the public documentation (which transitively includes the imported repository documentation) and mounts the infrastructure repo’s documentation/ directory as the Internal section

Hugo Module Mounts:

# In documentation/config.yaml
module:
  imports:
    - path: gitlab.com/redhat/hummingbird/containers
      mounts:
        - {source: 'README.md', target: content/snippets/readme.md}
        - {source: 'CONTRIBUTING.md', target: content/snippets/contributing.md}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/background', target: content/docs/background/containers, files: "*.md"}
        - {source: 'documentation/ci-scripts', target: content/docs/background/containers/ci-scripts, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/rpms
      mounts:
        - {source: 'CONTRIBUTING.md', target: content/snippets/rpms-contributing.md}
        - {source: 'documentation/background', target: content/docs/background/rpms, files: "*.md"}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/tools
      mounts:
        - {source: 'documentation/background', target: content/docs/background/tools, files: "*.md"}
        # Per-component page bundles: multi-file components are branch
        # bundles (_index.md + child pages); the recursive glob plus the
        # shared target keeps each component's main-page URL flat.
        - {source: 'documentation/components', target: content/docs/background/tools, files: ["*.md", "**/*.md"]}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}
    - path: gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
      mounts:
        # Serves the current flat layout; the section mounts below are
        # ready for whenever the repository adopts the standard subdirs.
        - {source: 'documentation', target: content/docs/background/k8s-test-pipeline, files: "*.md"}
        - {source: 'documentation/background', target: content/docs/background/k8s-test-pipeline, files: "*.md"}
        - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
        - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
        - {source: 'documentation/using', target: content/docs/using, files: "*.md"}

For README.md and CONTRIBUTING.md, front matter is provided via wrapper files.

This creates a unified documentation site from multiple repositories without duplication.

Documentation Layout Patterns

Source repositories use one of two layout patterns depending on their content:

Cross-cutting layout (containers, rpms, tools): The repository documents cross-cutting concerns like contributing, operating, and background information. Documentation is organized into subdirectories matching the site sections:

  • documentation/background/ - Explanation and reference material
  • documentation/contributing/ - How to contribute
  • documentation/operating/ - Operator runbooks
  • documentation/using/ - End-user guides

Each subdirectory is mounted into the matching site section, so operating docs from different repositories appear together under Operating. Not every repository uses every subdirectory: the tools repository keeps its team workflow guides in background/ (the shared Contributing section is currently containers-scoped) and leaves contributing/ and using/ to the pre-provisioned mounts, unused for now.

The tools repository combines the cross-cutting sections with per-component page bundles inside components/: one page per component, and components with design docs or DDRs become bundles — a directory with _index.md as the main page plus child pages like design.md or design-decisions.md. The recursive components mount targets the same content/docs/background/tools path as the other background mounts, so a bundle’s main page keeps a flat URL (/docs/background/tools/<component>/) while its child pages nest one level below it (/docs/background/tools/<component>/design/).

Per-component layout (k8s-test-pipeline): The repository is a monorepo where each documentation page covers one tool or component. Documentation is a flat directory with one file per component:

  • documentation/<component-name>.md

The entire directory is mounted under background/<repo-name>/ on the site. Section mounts for the standard subdirectories are pre-provisioned in the site configuration, so the repository can adopt the cross-cutting layout without further site changes.

Getting Started

Quick Start

Clone, install dependencies, and start the documentation server:

git clone https://gitlab.com/redhat/hummingbird/documentation
cd documentation
make setup  # Installs npm dependencies; only needed once per checkout
make serve  # Available at http://localhost:1313/

For the internal documentation:

git clone https://gitlab.com/redhat/hummingbird/infrastructure
cd infrastructure/internal-docs
make serve  # Available at http://localhost:1314/

For the best development experience, set up all repositories with direnv:

mkdir -p ~/git/hummingbird
cd ~/git/hummingbird

# Clone all repositories
git clone https://gitlab.com/redhat/hummingbird/containers
git clone https://gitlab.com/redhat/hummingbird/rpms
git clone https://gitlab.com/redhat/hummingbird/tools
git clone https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
git clone https://gitlab.com/redhat/hummingbird/documentation
git clone https://gitlab.com/redhat/hummingbird/infrastructure

# Set up environment variables
cat > .envrc << 'EOF'
export CONTAINERS_REPO_PATH=$(pwd)/containers
export RPMS_REPO_PATH=$(pwd)/rpms
export TOOLS_REPO_PATH=$(pwd)/tools
export K8S_TEST_PIPELINE_REPO_PATH=$(pwd)/k8s-test-pipeline
export DOCUMENTATION_REPO_PATH=$(pwd)/documentation
export INFRASTRUCTURE_REPO_PATH=$(pwd)/infrastructure
EOF
direnv allow

This automatically uses your local repositories for imported content, making changes immediately visible in the documentation preview.

Adding Documentation

Add new documentation as markdown files inside the appropriate directory under content/. Every file should end in .md.

Preview your changes:

make serve  # Starts local server with live reload

Check for problems:

make check  # Runs linters and validation

After make setup, run the Docsy print-layout regression tests:

make test

These tests build a small site with Mermaid diagrams in section and child pages. They check renderer inclusion in normal and combined print views, including nested sections and no_print exclusions. After changing print rendering, also check a diagram page and its “Print entire section” view in a browser, then export a PDF and confirm that the diagrams remain visible.

Documentation uses two types of internal links depending on the context:

Use relative .md links for pages in the same directory, and for cross-section links in the documentation repository:

[Testing Guide](testing-images.md)
[Adding Images](adding-images.md)

Use full URLs with /l/ aliases for links across repositories or across sections in all repositories:

[Image Pipeline][image-pipeline]
[Global Variables][global-vars]
[Retrying Checks][retrying-checks]

[image-pipeline]: https://hummingbird-project.io/l/image-pipeline
[global-vars]: https://hummingbird-project.io/l/global-variables-reference
[retrying-checks]: https://hummingbird-project.io/l/retrying-konflux-checks

Why full URLs?

  • Work in standalone contexts (GitHub, text editors, when files are copied)
  • Hugo’s link render hook automatically converts them to relative links in the rendered site
  • Stable even when files are moved or reorganized
  • Clear and explicit about which page is being referenced

URL Stability

Documentation URLs (for example, /docs/background/tools/page/) are not stable. They change whenever content is reorganized, and a moved page’s previous URL returns a 404 by design.

Only /l/ aliases are stable and guaranteed to keep resolving. Use them for any reference that must survive reorganization:

  • Cross-repository links
  • Cross-section links
  • Merge request comments
  • Scripts
  • Bookmarks

The documentation site includes a custom link render hook (layouts/_default/_markup/render-link.html) that:

  1. Detects https://hummingbird-project.io/l/... URLs
  2. Looks up the page with the matching alias
  3. Converts to a relative link in the rendered HTML
  4. Keeps external links as-is with target="_blank"

This provides the best of both worlds: full URLs in markdown source files that work everywhere, and optimized relative links in the rendered site.

All cross-referenced pages should have stable /l/ aliases. These aliases serve two purposes:

  1. Incoming links from external sources (documentation, scripts, merge request comments)
  2. Cross-references within documentation (across sections or repositories)

Stability Policy

Only /l/ aliases are stable and guaranteed to keep resolving. Documentation URLs change whenever content is reorganized, and a moved page’s previous URL returns a 404 by design.

Never add an alias for a page’s previous URL after a move — aliases are curated stable links, not a redirect backlog.

Adding an Alias

Add an alias in the page front matter:

---
title: "Konflux Integration"
aliases: [/l/konflux-integration]
---

Now you can link to https://hummingbird-project.io/l/konflux-integration and it will continue working even if the page moves.

When to Add Aliases

Add /l/ aliases to pages that:

  • Are referenced from other repositories (e.g., containers repo → docs repo)
  • Are linked to from CI scripts or infrastructure
  • Are shared in merge request comments or external documentation
  • Are referenced across documentation sections

Naming Convention

  • Use /l/ prefix for all stable links
  • Keep names short, memorable, and descriptive
  • Use kebab-case (lowercase with hyphens)
  • No nested paths (flat structure only)

Submitting Changes

  1. Fork the repository on GitLab
  2. Clone your fork locally
  3. Create a new branch for your changes
  4. Make your changes and test them with make serve
  5. Run make check to ensure there are no errors
  6. Commit your changes with a descriptive commit message
  7. Push to your fork
  8. Create a merge request against the main repository

Source Repository Checks

Merge requests that change mounted documentation are checked in the source repository’s own pipeline. The containers, rpms, tools, and pipelines/k8s-test-pipeline repositories each have a check-afdocs job that builds this site with the merge request’s working tree mounted in place of the published module, then runs the same agent-friendly docs (afdocs) suite the documentation repository runs on its own merge requests. The infrastructure repository has the equivalent job for the internal documentation site (check-afdocs-internal-docs), which also covers the internal documentation mounted from that repository.

  • The job runs automatically when a merge request changes a path the site mounts, and is available as an optional manual job otherwise.
  • A failure means the rendered site would break. Fix it in the source repository — a broken link or oversized page is a content problem there, not a build problem here.
  • Like a production build, the check resolves the other repositories' documentation at their default branches, so a failure can also come from another repository’s main. Re-run the job once that repository is fixed.
# Run the same check locally from a documentation checkout:
make check-afdocs TOOLS_REPO_PATH=/path/to/tools

Documentation Structure

The public documentation is organized into several main sections:

  • Using - Guides for users of Project Hummingbird container images
  • Contributing - Guides for contributors to the project
  • Operating - Guides for infrastructure operators
  • Background - Technical background and architecture information

The internal documentation adds:

  • Internal - Internal-only infrastructure documentation, organized into:
    • Operating - AWS, DNS, GitLab, OpenShift, and other service runbooks
    • Background - Error budgets, SLIs/SLOs
    • Presentations - Team talks and lightning talks

When adding new documentation, place it in the most appropriate section.

3.16 - Adding Independent Packages

How to add packages that originate in the Hummingbird repository

Independent packages are packages that originate in the Hummingbird repository rather than being imported from Fedora dist-git.

Steps to Add an Independent Package

1. Create Package Directory

mkdir rpms/<package-name>
cd rpms/<package-name>

2. Add Package Files

Create the following files in the package directory:

  • <package-name>.spec - RPM spec file
  • sources - SHA512 checksums of source tarballs
  • Source tarballs (.tar.gz, .tar.bz2, .tar.xz)
  • Any patches or additional configuration files

Rust specs can opt in to isolated tests by requiring cargo-nextest when checks are enabled and replacing %cargo_test with %cargo_nextest. Keep existing feature flags and exclusions; the macro preserves RPM compiler flags and the rpm profile, then runs doctests separately. Use NEXTEST_PROFILE=ci to select an upstream CI profile. For serial suites, retain RUST_TEST_THREADS=1 and also set NEXTEST_TEST_THREADS=1. Publish the runner before its consumers.

3. Generate the sources File

The sources file must contain SHA512 checksums in BSD-style format:

sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

This produces the correct format:

SHA512 (oras-1.3.0.tar.gz) = fb871c0577f621f7e1f56a54f249f96c659b29c771dad529f9a9e838379b2d56bca9cef03a90071915a568e28e266326989e67359e589a87908c4fc077b14689
SHA512 (oras-1.3.0-vendor.tar.bz2) = fa23324bf3910c3dc050eb3c0ebfef3224168489679cf1195d668c656c6c53beaac94b4786e45c2cf3782f9437b0d166da9c18e608a3b474b04957ff1dec5226

Important: Use sha512sum --tag to generate the BSD-style format. Do not use the default GNU format.

4. Create Package Metadata

Create metadata/<package-name>.json:

{
  "modification_status": "independent",
  "upstream_repo": "https://github.com/example/project",
  "version": "1.3.0"
}

Fields:

  • modification_status: Must be "independent" for packages not imported from Fedora (see Package Metadata Fields)
  • upstream_repo: Canonical upstream git repository URL (required — CI enforces this). If no upstream repo exists, use https://src.fedoraproject.org/rpms/<name> as a fallback.
  • version: Package version (must match spec file)

5. Generate Konflux Resources

Generate the Konflux Component and ImageRepository resources:

make generate-host

Or directly:

python3 ci/generate_resources.py all

This updates:

  • konflux-templates/rendered.yml
  • .tekton/ pipeline files

6. Commit Changes

git add rpms/<package-name>/ metadata/<package-name>.json konflux-templates/ .tekton/
git commit -m "Add <package-name>-<version>-<release>

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"

Note: Source tarballs (.tar.gz, .tar.bz2, .tar.xz) are automatically ignored by .gitignore and should be uploaded to the lookaside cache instead.

Example: Adding oras

# 1. Create directory
mkdir rpms/oras
cd rpms/oras

# 2. Add spec file and sources
# ... create oras.spec ...

# 3. Generate sources file
sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources

# 4. Create metadata
cat > ../../metadata/oras.json << 'EOF'
{
  "modification_status": "independent",
  "upstream_repo": "https://github.com/oras-project/oras",
  "version": "1.3.0"
}
EOF

# 5. Generate Konflux resources
make generate-host

# 6. Commit
git add rpms/oras/ metadata/oras.json konflux-templates/ .tekton/
git commit -m "Add oras-1.3.0-1.hum1"

See Also

3.17 - Pulp Access

Overview

Pulp is used to host and distribute Hummingbird RPM repositories. Access to the Pulp API requires a service account with credentials stored in the vault.

Existing credentials

The public Pulp service account credentials are stored in the vault at rhel-primitives/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE. Retrieve the cli.toml field and save it locally (see ci/pulp-setup/README.md for usage).

Creating a new Pulp service account

When you need a dedicated service account (e.g., for private product repositories):

  1. Go to https://access.redhat.com/terms-based-registry/
  2. Log in with your @redhat.com account
  3. Click New Service Account
  4. Enter a username like hummingbird-<qualifier>-pulp-bot
  5. Enter a description like bot account for hummingbird <qualifier> pulp operations
  6. Click Create
  7. Capture the full username (in the format <id>|<username>) and the token

Configure the CLI

Create a cli.toml with the new credentials:

[cli]
base_url = "https://packages.redhat.com"
api_root = "/api/pulp/"
username = "<id>|hummingbird-<qualifier>-pulp-bot"
password = "<token>"
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
domain = "<target-domain>"

You can either save this to ~/.config/pulp/cli.toml or pass it explicitly with --config ./cli.toml when running scripts.

Store credentials in the vault

Store the credentials in the vault at apps/hummingbird with the key HUMMINGBIRD_<QUALIFIER>_PULP_BOT_CONFIG_FILE (e.g., HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE). Create two subkeys:

  • cli.toml — the full config file contents
  • value — the token value

3.18 - Measuring Documentation Quality

The documentation quality evaluation measures whether public documentation is complete enough for an LLM with web search to answer questions correctly. Low scores indicate documentation gaps; score improvements after content changes validate the fix.

For config schema, CLI options, and scoring details, see the Documentation Quality Evaluation reference in the tools repo. For the general evaluation framework, see Hummingbird Agent Evals.

How it works

The evaluation runs in two phases:

  1. Ask – multiple LLM configurations answer each golden question using web search against public documentation. Models never see internal sources; the test is whether public docs contain the information.
  2. Judge – a separate LLM scores each response against expected facts and coherence criteria defined in the golden dataset.

Documentation quality is the variable being measured, not model quality. Models are the instrument – if scores are low across all models, the documentation is incomplete.

The evaluation is resume-friendly: existing response and scored files are skipped on re-run, so interrupted runs can be continued without repeating expensive model calls.

Golden dataset

The config and golden dataset live in evals/docs_eval.yml in this repo. The file combines evaluation settings (models, judge, repetitions) with scoring criteria (questions, expected facts, coherence checks, thresholds).

Running

Prerequisites

  • GCP Application Default Credentials:

    gcloud auth application-default login
    export GOOGLE_CLOUD_PROJECT=<gcp-project>
    
  • The hummingbird-agent package installed (pip install -e . from the hummingbird-agent/ directory in the tools repo), or the container image.

Via Makefile

make eval-docs

Manual

python -m hummingbird_agent.evals.docs_eval.run_eval \
  -c evals/docs_eval.yml

Writing questions

Each question tests a documentation topic, not a model capability. When adding or editing questions:

  • Facts should be verifiable from public documentation. Avoid opinions or information that requires internal knowledge.
  • Assign higher weights to more important facts. A weight-2 fact contributes twice as much to the score as a weight-1 fact.
  • The coherence criterion checks narrative quality, not just fact presence. A response that lists correct facts without connecting them should score lower on coherence.
  • Calibrate the threshold by running a baseline: set the threshold where current documentation should fail but improved documentation should pass.

3.19 - Rebuilding Packages

AI Agent Note: When asked to rebuild packages, use the rebuild command: ./ci/dist_git.py rebuild <package> --reason "<reason>". For rebuilding reverse dependencies (e.g., “rebuild all Go packages”), use rebuild-rev-deps <package> --reason "<reason>". Always ask the user for a ticket link or explanation first to use as the reason. If the command fails with “uses macros in Release field”, see the “Packages requiring manual rebuild” section below for instructions.

Overview

This document covers three scenarios for triggering package builds:

  1. No-change rebuild: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes).

  2. Rebuilding reverse dependencies: Rebuild all packages that depend on a changed package (e.g., rebuild all Go packages when golang updates).

  3. Backporting a patch: Add an upstream patch that hasn’t yet landed in Fedora to fast-track a fix or feature.

All scenarios use the .N release suffix pattern to ensure our builds sort higher than the upstream Fedora release while remaining lower than the next upstream version.

No-Change Rebuild

The rebuild command automates the Release field bump:

# Rebuild a single package
./ci/dist_git.py rebuild <package> --reason "<reason>"

# Rebuild multiple packages (one commit per package)
./ci/dist_git.py rebuild <package1> <package2> ... --reason "<reason>"

# Rebuild all packages (one commit per package)
./ci/dist_git.py rebuild --all --reason "<reason>"

# Rebuild all packages except specific ones (requires --all)
./ci/dist_git.py rebuild --all --exclude <pkg1>,<pkg2> --reason "<reason>"

Examples:

# Single package
./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes"

# Multiple packages
./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds"

# All packages (useful after toolchain updates)
./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15"

# All packages except a few already rebuilt earlier in the same rollout
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm --reason "toolchain update: GCC 15"

--exclude takes a comma-separated list of package names and only works together with --all — it is rejected when combined with an explicit package list, since you can simply omit the packages you don’t want to rebuild in that case.

The command:

  • Automatically bumps the Release field using the .N suffix pattern
  • Handles %autorelease by resolving and replacing with explicit values
  • Preserves macros in the Release field (e.g., %{revision})
  • Creates a properly formatted commit message (one per package with --all)
  • Does not mark the package as modified (release-only changes are ephemeral)

Supports --dry-run to preview changes without committing:

./ci/dist_git.py --dry-run rebuild <package> --reason "test"

Creating MRs for rebuild commits

For packages configured for MR artifact promotion, the MR build and required tests produce the image that is released after merge. Follow that guide to verify the promotion PipelineRun and automatic Release or recover a failure. Other packages build again on main.

After creating rebuild commits locally, use rebuild_multi_mr.sh to push each commit as its own merge request (one MR per package, auto-merge enabled):

./ci/rebuild_multi_mr.sh

By default the script compares against origin/main. For local development, use --base to point at a different ref:

# Use local main branch as base (useful when origin/main is not up to date)
./ci/rebuild_multi_mr.sh --base main

# Use a specific commit SHA as base
./ci/rebuild_multi_mr.sh --base abc1234

# Preview what would be created without pushing
./ci/rebuild_multi_mr.sh --dry-run

# Limit to at most N MRs
./ci/rebuild_multi_mr.sh --max-updates=5

The script:

  • Creates a chore/rebuild-{package} branch per commit and pushes it
  • Titles each MR chore(rpms): Rebuild {package}: {reason}
  • Enables auto-merge on all rebuild MRs
  • Leaves the current branch untouched
  • Skips branches that already exist on the remote (idempotent)
  • Only processes commits whose message matches Rebuild {package}: {reason}; other commits in the range are silently skipped

Full workflow example:

# 1. Create the rebuild commits
./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update"

# 2. Preview the MRs that would be created
./ci/rebuild_multi_mr.sh --base main --dry-run

# 3. Create the MRs
./ci/rebuild_multi_mr.sh --base main

Packages requiring manual rebuild

Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file.

Macro indirection patterns

These packages define the Release field using a macro, where the macro itself contains %{?dist}. The rebuild command cannot detect or manipulate these without expanding all macros, which would break the macro system.

nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):

%{load:%{_sourcedir}/nodejs.srpm.macros}
%nodejs_define_version node 1:25.8.2-%{autorelease} -p
...
Release: %{node_release}

The %{node_release} macro is defined by an external macro system loaded from nodejs.srpm.macros. The release component is embedded in the version definition.

How to rebuild: Edit the %nodejs_define_version node line to bump the release component (e.g., change -%{autorelease} to -1.1 or increment existing .N).

kernel-headers:

%define specrelease 59%{?buildid}%{?dist}
...
Release: %{specrelease}

How to rebuild: Edit the %define specrelease line to add/increment the .N suffix before %{?buildid}:

%define specrelease 59.1%{?buildid}%{?dist}

krb5:

%global krb5_release 4%{?dist}
...
Release: %{krb5_release}

How to rebuild: Edit the %global krb5_release line to add/increment the .N suffix:

%global krb5_release 4.1%{?dist}

Why these can’t be automated

The rebuild command can handle:

  • ✅ Simple numeric: Release: 5%{?dist}
  • ✅ Macros ending with dist: Release: %{baserelease}%{?dist} (e.g., rpm, gcc)
  • ✅ Complex macros with dist: Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist}
  • ✅ Content after dist: Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}} (e.g., unbound)

The rebuild command cannot handle:

  • ❌ Macros without %{?dist}: Release: %{node_release}
  • ❌ Macros where dist is inside the macro definition: %{krb5_release} contains %{?dist}

This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM’s full macro parser.

Manual rebuild process

If you need to rebuild manually or the automated command doesn’t work for your use case, follow these steps:

1. Identify the package to rebuild

Identify the source package name and locate its spec file in rpms/<package>/<package>.spec.

If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name:

podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
  dnf5 repoquery --queryformat '%{SOURCERPM}' <binary-package> 2>/dev/null

Example: ncurses-libs-6.5-8.20250614.hum1 -> SRPM ncurses-6.5-8.20250614.hum1.src.rpm -> spec file at rpms/ncurses/ncurses.spec

2. Determine the Release bump pattern

The .N bump suffix must always appear immediately before %{?dist}. The %{?dist} suffix should always be the final component since it identifies the build environment.

Current Pattern Example Before Example After
Simple numeric Release: 3%{?dist} Release: 3.1%{?dist}
Already bumped Release: 3.1%{?dist} Release: 3.2%{?dist}
With macro Release: 8.%{revision}%{?dist} Release: 8.%{revision}.1%{?dist}
autorelease Release: %autorelease Release: 1.1%{?dist}

For %autorelease, first resolve its value using rpmspec, then replace with the resolved value plus .1. In the Hummingbird monorepo, %autorelease always evaluates to 1.

Note: If the Release field is missing %{?dist} entirely or looks unusual (e.g., 1build1 instead of 1.1%{?dist}), flag this to the user for resolution. Check the git history to understand the original value:

git log -p -S "Release:" -- rpms/<package>/<package>.spec

This helps determine the correct fix when a previous bump was malformed.

3. Modify the spec file

Use sed to edit only the Release: line, avoiding any unintended whitespace changes that text editors may introduce:

sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms/<package>/<package>.spec

Verify the change with git diff before committing:

git diff rpms/<package>/<package>.spec

The diff should show only the Release line change:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

Important: Only modify the Release line. Do not introduce any other changes such as whitespace fixes or trailing newline modifications. If the diff shows additional changes, reset and retry with sed.

Important: Do not change the release field in metadata/<package>.json during local rebuilds or backports. That field records the Fedora release for the version currently shipped and is present only while Hummingbird ships Fedora’s release; see Package Metadata Fields.

4. Verify the bump is correct

Use rpm --eval to confirm the new release sorts higher than the original:

# Returns -1 if first < second (correct), 1 if first > second (wrong)
rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}'
# Expected output: -1

5. Commit the change

Use this commit message format:

Rebuild <package>: <reason>

<ticket link or explanation>

Example:

Rebuild ncurses: published multiple times with different hashes

HUM-1234

6. Verify the commit

After committing, verify only the Release line was changed:

git show --stat HEAD

Expected output should show exactly 1 insertion and 1 deletion:

 rpms/<package>/<package>.spec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

If the commit shows more changes, amend or reset and redo the change using sed.

Important notes about rebuilds

Modification status

Rebuilds do not change a package’s modification_status. Spec Release: bumps are ephemeral and do not make a package modified versus clean. See Package Metadata Fields for how modification_status and metadata release relate to rebuilds.

During updates, Release lines are normalized to avoid conflicts, and the automation ignores Release-only changes, so automatic Fedora updates continue normally after a rebuild.

If you want to explicitly prevent automatic updates (e.g., you’re investigating an issue), you can manually mark the package as modified:

./ci/dist_git.py mark-modified <package> --modified --reason "Investigating build issue"

Note: This will block automatic Fedora updates until you mark it clean again.

Rebuilding Reverse Dependencies

When a compiler, runtime, or toolchain package changes, you may need to rebuild all packages that depend on it. The rebuild-rev-deps command automates finding and rebuilding reverse dependencies.

When to use

Common scenarios for rebuilding reverse dependencies:

  • Golang/Python runtime updates: When updating golang1.26, python3.14, etc., rebuild all Go/Python packages
  • Toolchain changes: When updating gcc, rebuild packages that BuildRequire it
  • Library ABI changes: When a library’s ABI changes, rebuild packages that BuildRequire it

Usage

./ci/dist_git.py rebuild-rev-deps <package> --reason "<reason>"

The command:

  1. Finds all packages that have BuildRequires: <package> in their spec files
  2. Rebuilds each package (bumps Release field and commits)
  3. Reports a summary of successful/failed rebuilds

Examples:

# Rebuild all Go packages when golang updates
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "golang 1.26.2 update"

# Rebuild all Python packages when python updates
./ci/dist_git.py rebuild-rev-deps python3.14 --reason "python 3.14.1 update"

# Rebuild packages that depend on a specific library
./ci/dist_git.py rebuild-rev-deps openssl --reason "openssl 3.4.0 update"

Virtual BuildRequires (golang/python)

For virtual BuildRequires like golang1.25, golang1.26, python3.13, python3.14, the command automatically handles translation to the actual BuildRequires target:

# These all work the same way:
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "..."
./ci/dist_git.py rebuild-rev-deps go-rpm-macros --reason "..."

Important: Only the latest version triggers rebuilds. This is because:

  1. Multiple golang versions exist: golang1.25, golang1.26
  2. They all provide the same virtual package: Provides: golang = <version>
  3. DNF always picks the highest version to satisfy Requires: golang

Therefore:

  • ✅ rebuild-rev-deps golang1.26 → Rebuilds all Go packages (latest version)
  • ❌ rebuild-rev-deps golang1.25 → Error: Not the latest version

This ensures rebuilds only happen when the active runtime actually changes.

Creating MRs for reverse dependency rebuilds

After creating rebuild commits, use rebuild_multi_mr.sh to push each commit as its own MR:

./ci/rebuild_multi_mr.sh --base main

See the “Creating MRs for rebuild commits” section above for full details.

Dry-run mode

Preview which packages would be rebuilt without making changes:

./ci/dist_git.py --dry-run rebuild-rev-deps golang1.26 --reason "test"

This shows:

  • Which packages have the BuildRequires dependency
  • What the new Release values would be
  • Does not commit any changes

Backporting a Patch

Use this workflow when you need to fast-track an upstream fix or feature that hasn’t yet been released in Fedora.

1. Obtain the patch

Fetch the patch from the upstream repository. For GitHub PRs, append .patch to the PR URL:

curl -L https://github.com/<org>/<repo>/pull/<number>.patch \
  > rpms/<package>/<NNNN>-<short-description>.patch

Name the patch file with a numeric prefix matching the next available PatchN: slot in the spec file (e.g., 0004-fix-foo.patch if Patch1-3 already exist).

Keep the upstream provenance in the patch header:

Upstream: https://github.com/<org>/<repo>/commit/<direct-commit>
Origin: https://github.com/<org>/<repo>/commit/<original-commit>

Upstream is the commit that the patch applies. Add Origin only when Upstream adapts or backports an earlier commit.

2. Add the patch to the spec file

Add a PatchN: declaration after the existing patches:

Patch3:         0003-existing-patch.patch
Patch4:         0004-fix-foo.patch

The patch will be applied automatically if the spec uses %autosetup -p1. If the spec uses explicit %patchN macros, add the corresponding apply line in the %prep section.

3. Check for a gorget source-pipeline

If metadata/<package>.source-pipeline.yaml exists, check whether its transform: step applies patches or otherwise runs against patched source (e.g. resolving a lockfile with yarn install, pnpm fetch, npm ci):

# run from the root of the rpms repo checkout
grep -n -E "patch -p[0-9]|git apply" metadata/<package>.source-pipeline.yaml

This catches the common forms, but isn’t exhaustive — if the transform applies patches some other way (a different tool, or a script that wraps patch/git apply), read the transform: step directly rather than trusting a grep miss.

If it does, and your new patch touches a file that step depends on (a lockfile or manifest such as yarn.lock, package.json, pnpm-lock.yaml, go.sum, Cargo.lock), add the same patch -p1 < "${PACKAGE_DIR}/<your-patch>.patch" line there too, in the same relative order as its PatchN: slot in the spec. The patch file itself needs no extra fetch step — it’s already in ${PACKAGE_DIR}/ because you committed it there in step 2.

The pipeline’s transform: step and the spec’s %prep/%goprep are two independent lists of patches to apply — nothing keeps them in sync automatically. A patch declared only in the spec is invisible to whatever artifact the transform step generates (e.g. an offline yarn/pnpm cache built from the lockfile), and the generated artifact silently drifts from what %build actually applies. See “Known sharp edge: patch-list duplication” in the source-pipeline design doc for the grafana12.4/grafana13.1 incident this rule comes from. test/test_source_pipeline_patches.py checks for this drift, but treat it as a safety net, not a substitute for updating the pipeline yourself here.

4. For Go packages: check go-vendor-tools.toml’s pre_commands

This applies whether or not the package uses gorget — rpms/<package>/go-vendor-tools.toml’s [archive] pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately:

grep -n -A2 "pre_commands" rpms/<package>/go-vendor-tools.toml

If your new patch targets a vendored Go module (or you’re editing pre_commands directly instead of adding a patch — don’t), and any pre_commands entry mutates go.mod/go.sum (directly, via go get, or via go mod tidy/edit), your patch must apply the same go.mod/go.sum change to the actual source tree, not just to go-vendor-tools.toml. pre_commands cannot substitute for this: they require live network access to run go get, which %prep doesn’t have (Konflux builds are hermetic). Compute the resulting go.mod/go.sum diff offline (e.g. clone upstream, apply the same edits, run go mod tidy if pre_commands does) and commit that as the patch.

Skipping this leaves go.mod in the build tree and vendor/modules.txt in the generated vendor archive silently requiring different versions of the same package, which go build -mod=vendor rejects as inconsistent vendoring. See the source-pipeline design doc for the trivy incident this rule comes from. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

5. Bump the Release

Follow the same .N suffix pattern as no-change rebuilds:

- Release: 3%{?dist}
+ Release: 3.1%{?dist}

6. Mark the package as modified

Mark a Fedora-imported package as modified before you commit. This prevents an automatic Fedora update from overwriting the patch:

make dist-git ARGS='mark-modified <package> --modified --reason "Backport <issue or CVE>"'

Preserve existing modification reasons. Do not change an independent package to modified. mark-modified updates the metadata file but does not commit it.

7. Commit the patch

Use this commit message format:

Backport <short description> to <package>

Backport: <link to PR or commit>
<ticket link if applicable>

Example:

Backport reproducible build sorting fix to dnf5

Backport: https://github.com/rpm-software-management/dnf5/pull/2522

Include the patch, spec file, and metadata file in the same package commit. Do not use dist_git.py rebuild for this step. That command describes and commits a no-change rebuild.

8. Verify patch application

Apply the patch to the exact packaged source release.

3.20 - Setting Up Pulp Repositories

How to create and set up Pulp domain and repositories

Overview

  • This document and script should be used to initialize and setup a pulp domain and a set of repositories
  • We choose the domain prefix of public- to ensure that the repositories are publicly available

Setup

  • make sure you have followed the local development setup instructions in the infrastructure repository
  • install pulp cli via dnf install pulp-cli
  • setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE  > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml

Create

./create-pulp-resources.sh public-hummingbird "source,x86_64,s390x,ppc64le,aarch64"

3.21 - Package Metadata Fields

Overview

Each package has a metadata file at metadata/<package>.json. This page explains metadata configuration and tooling behavior for day-to-day package operations. The RPM metadata and versioning specification defines the authoritative metadata contract, including field definitions, package states, and validation rules.

Two fields control how Hummingbird treats local changes and rebuilds relative to Fedora:

Field Purpose
modification_status Whether the package may be auto-updated from Fedora
fedora.release Fedora release baseline when the package still ships the imported Fedora version

Fedora provenance is stored in the fedora object, with git_url, branch, and sha identifying the import or last sync. fedora.release is present only while the package ships that Fedora version. A package ahead of Fedora retains provenance but has no fedora.release; an independent package has no fedora object. There is no top-level release field.

modification_status

modification_status records whether a package still matches its Fedora import, diverges from that import, or has no Fedora origin.

Status Meaning Auto-updates from Fedora
clean Unmodified Fedora import Allowed
modified Local changes or an upstream version ahead of Fedora Blocked
independent Hummingbird-independent package (not from Fedora) Blocked

When to use each value

  • clean: Default for packages imported from Fedora with no local source changes. Automatic Fedora updates are allowed. Conflicts with modification_reason (must be absent).
  • modified: Use after backports, custom patches, or updates ahead of Fedora that must not be overwritten by an automatic update. Requires modification_reason.
  • independent: Required for packages that are not imported from Fedora. These packages have no fedora object. Conflicts with modification_reason (must be absent).

Imports set the status automatically, using clean for Fedora imports and independent for Hummingbird-independent imports. After local source changes, set status with dist_git.py mark-modified. See Package Modification Tracking.

Rebuilds that only bump the spec Release: line do not change modification_status. Release-only changes are ephemeral and are ignored when deciding whether a package is modified versus clean.

modification_reason

When modification_status is modified, modification_reason is required. It should be a short explanation of why the package cannot be auto-updated (for example, a CVE backport or a custom spec change). Clear the reason when marking the package clean again.

modification_reason applies only to modified packages. Do not add it for clean or independent packages. See When not to update these fields.

fedora.release

The fedora.release field in metadata is not the same as the Release: line in the package spec. It records the imported Fedora release without a dist tag such as .fc42. It remains present after local patches or release-only rebuilds while the package stays on the imported Fedora version. The spec’s Release: line is authoritative for the Hummingbird build.

Location What it represents
metadata/<package>.json → fedora.release Fedora release baseline for the imported version, without its dist tag
Spec Release: The release used for the Hummingbird build (includes %{?dist} / .hum1, and may include local .N rebuild suffixes)

Who writes metadata fedora.release

Writer When Value stored
dist_git.py import / update / sync Fedora import or refresh Resolved Fedora release with dist suffix stripped (e.g. 5 from 5.fc42)
check_upstream_versions.py Local upstream version bump (check --update) Removes fedora.release for Fedora-origin packages moving ahead of Fedora, retaining the other provenance fields

Fedora-imported packages

For packages with a fedora object:

  • fedora.release is set on import and refreshed when dist_git.py update / sync brings in a Fedora version. Dist suffixes such as .fc42 are stripped.
  • When check_upstream_versions.py check --update moves the package ahead of Fedora, it removes fedora.release and marks the package modified. The default updater resets a literal spec release to 0.1%{?dist}. It preserves %autorelease and other package-specific release macros; an update_spec hook can replace the default spec-update behavior. See Package Modification Tracking for update hooks.
  • Do not change fedora.release during local rebuilds or backports. Rebuilds bump only the spec’s Release: line. With a Fedora baseline, the first rebuild appends .1 and later rebuilds increment that suffix. Without a Fedora baseline, rebuilding 0.1 advances it to 0.2.
  • When Fedora uses %autorelease, fedora.release stores the resolved numeric release, not the literal macro. Imports and updates resolve it through Fedora build services and replace the local spec’s %autorelease with that value plus %{?dist}. For rebuilds that still see %autorelease, follow Rebuilding Packages.

Independent packages

For Hummingbird-independent packages with modification_status: "independent":

  • There is no Fedora origin, so metadata does not contain a fedora object or fedora.release.
  • The .hum1 suffix comes from the spec Release: line via %{?dist} (for example Release: 1%{?dist}), not from metadata.
  • dist_git.py rebuild increments the spec’s current release directly for independent packages.

Spec Release: vs metadata fedora.release

No-change rebuilds and reverse-dependency rebuilds change the spec Release: only. See Rebuilding Packages. When dist_git.py update or sync imports a Fedora version, it records that Fedora release in fedora.release. Local patches merged onto that version retain the Fedora release baseline. When check_upstream_versions.py moves a package ahead of Fedora, it removes fedora.release. See the specification’s built NVR rules for the complete versioning contract.

private_product

private_product records an internal product assignment for private RPM distribution. It is independent of the package’s Fedora relationship and versioning state. See the metadata specification for the field’s format.

Read and write assignments through hum.meta. The library checks that a product matches an RPA entry in ci/konflux_rpa_config.yml; resource generation uses that assignment to select the private Konflux application and exclude the package from public RPAs. Packages without an assignment remain public, regardless of their names.

See Private RPM Repositories for assignment and onboarding commands.

When not to update these fields

Most day-to-day package work changes the spec, sources, or patches, not metadata modification_status, modification_reason, or fedora.release. Leave those fields alone unless the package’s relationship to Fedora actually changed.

Situation Do not change Why
Independent package gets a CVE patch, backport, or other source edit modification_status, modification_reason Status stays independent. There is no Fedora auto-update to block, so do not switch to modified or add a modification_reason.
No-change rebuild (spec Release: bump only) modification_status, modification_reason, fedora.release Status stays unchanged; fedora.release continues to record a Fedora baseline when one exists.
Fedora-imported package gets a local patch or backport fedora.release Mark the package modified with a reason, but keep fedora.release if present. Only bump the spec’s Release: if needed.
Package is already modified and you add another local change modification_status Leave status as modified. Update modification_reason only if the existing reason no longer describes why auto-updates must stay blocked.
Upstream Fedora update lands and automatic Fedora updates should resume Do not leave stale fields Use dist_git.py update to refresh version and Fedora provenance. Mark clean once the package matches Fedora; this clears modification_reason.

Examples

Independent package + CVE fix: Edit the spec and add the patch. Keep the metadata status and version unchanged, as shown in this excerpt, not a complete metadata file:

{
  "modification_status": "independent",
  "version": "1.3.0"
}

Do not add modification_reason, and do not change status to modified.

Fedora package + CVE backport: After the patch, mark modified so auto-updates stay blocked:

make dist-git ARGS='mark-modified <package> --modified --reason "Backport CVE-2024-12345"'

Do not edit fedora.release as part of that change.

Rebuild only: Use dist_git.py rebuild. Leave metadata modification_status and fedora.release unchanged.

3.22 - Debugging Build Failures

Overview

When ./ci/build_rpms.sh fails to build a package, you can use the --shell-after flag to drop into an interactive debugging environment. This preserves the build state and allows you to investigate failures, modify files, and re-run the build without starting over.

Quick Start

# Run build and drop into shell after completion (even on failure)
./ci/build_rpms.sh --shell-after <package>

After the build completes (success or failure), you’ll be dropped into a mock shell inside the build environment.

Checking Sources and Patches Only

Many build failures come from simple problems: a missing or incorrect source tarball, or a patch that does not apply. To catch these without running a full build, use --prep-only:

./ci/build_rpms.sh --prep-only <package>

This builds the SRPM, which fails if any Source/Patch file is missing. It then installs the BuildRequires and runs only %prep (rpmbuild -rp, equivalent to rpmbuild -bp), which unpacks the tarballs and applies the patches. The script exits non-zero if any step fails. The %prep output is saved to builds/<package>/results/prep.log. --prep-only cannot be combined with --hermetic, --shell-before, or --shell-after.

Understanding the Build Environment

The build happens in two layers:

  1. Outer container: The rpm-build-pipeline podman container (Red Hat build environment)
  2. Mock shell: An isolated RPM build environment inside the container (similar to a chroot)

When you use --shell-after, you’re placed directly in the mock shell.

Key directories in the outer container

  • /results/build.log - Full build output with all commands that were run
  • /sources/ - Package source files (spec, patches, etc.)
  • /config/ - Mock configuration
  • /repo/ - Your local rpms repository (read-only mount)

Key directories in the mock shell

  • /builddir/build/BUILD/<package>-*/ - Unpacked source tree (modify and re-run build/test commands here)
  • /builddir/build/originals/<package>.spec - The spec file (modify to test spec changes)
  • /builddir/build/SOURCES/ - Source tarballs and patches
  • /builddir/build/BUILDROOT/ - Install root (where %install places files)

Debugging Workflow

1. Connect to the outer container

To get a second shell without killing your existing mock shell session, connect to the outer container:

# In another terminal, find the running container
podman ps | grep rpm-build-pipeline

# Get a shell in the outer container
podman exec -it <container-id> bash

2. Examine the build failure

From the outer container, check the build log to understand what failed:

# View the end of the build log
tail -100 /results/build.log

# Search for specific errors
grep -i error /results/build.log

3. Enter the mock shell

To investigate or modify the build environment, you can get an additional shell in the mock chroot, particularly for agents which cannot use the initial interactive --shell-after from above.

# From the outer container, chroot into the mock build environment
chroot /var/lib/mock/local-x86_64/root /bin/bash

4. Iterate on the fix

The recommended approach for debugging is to extract and re-run specific commands directly:

For quick iteration while developing a fix, extract the exact build or test command from the build log and run it directly:

# From the outer container, find what command the failing phase runs
grep -A 30 'Executing(%check)' /results/build.log

# Look for lines starting with "+ " that show the actual commands
# Example output:
#   + cd /builddir/build/BUILD/dnf5-5.4.0.0-build/dnf5-5.4.0.0
#   + /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure ...

# Run that command in the mock chroot
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && <actual-command>"

For build failures during %build: Look for Executing(%build) in the log, find commands like make or ninja, then re-run them:

# Example: extract the build command
grep -A 30 'Executing(%build)' /results/build.log | grep -E '^\+ (make|ninja|cmake)'

# Re-run in the build directory
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/<build-subdir> && make -j14"

For test failures during %check: Look for Executing(%check) and extract the test command:

# Example: extract the test command
grep -A 30 'Executing(%check)' /results/build.log | grep -E '^\+ (ctest|make check|pytest)'

# Re-run tests
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && ctest --output-on-failure"

This approach is fast because it skips rpmbuild overhead and goes straight to the failing command. It works with the preserved BUILD directory from --shell-after and is ideal when you’re modifying source files and want quick feedback.

Modifying source files: Edit files directly in /builddir/build/BUILD/<package>-*/ and re-run the command to test your changes quickly.

Modifying the spec file: You can also test spec file changes by editing /builddir/build/originals/<package>.spec in the mock chroot:

# From the host, edit the spec file in the mock chroot
podman exec -it -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  vim /builddir/build/originals/<package>.spec

# Run rpmbuild to see the change
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
  env HOME=/builddir LANG=C.UTF-8 su mockbuild -c \
  'rpmbuild -bb --target x86_64 --nodeps /builddir/build/originals/<package>.spec'

Full validation: Apply fixes and rebuild with build_rpms.sh

Manual rpmbuild in the preserved environment may not work reliably for all packages due to complex build dependencies and environment requirements.

Once you’ve identified a fix using the fast iteration approach:

  1. Exit the debug environment and apply your fix to the actual source files or spec file in rpms/<package>/

  2. Test the complete build using the build script:

    ./ci/build_rpms.sh <package>
    

This ensures your fix works through the entire build process in a clean environment.

3.23 - Package Modification Tracking

Overview

The RPMs repository tracks whether packages have been locally modified from their Fedora upstream source. This tracking prevents automatic updates from overwriting local changes like backported patches or custom modifications.

Modification Status

Each package metadata file (metadata/<package>.json) has a modification_status of clean, modified, or independent. That field (and related modification_reason / release configuration) is documented in Package Metadata Fields. This page covers the workflows for checking status, marking packages, viewing diffs, and configuring update hooks.

An optional track_upstream string field controls whether a package is checked by check_upstream_versions.py for new upstream releases (via release-monitoring.org). Set it to "latest" to track the latest version, or to a version prefix like "1.26" to constrain updates to that series. Its presence enables tracking; omit the field to disable it. The check subcommand only checks packages with track_upstream set when no explicit package arguments are given. The list subcommand shows all packages regardless of this field.

Checking Package Status

View a package’s modification status:

jq .modification_status metadata/<package>.json

View reason for modification (if modified):

jq .modification_reason metadata/<package>.json

List all modified packages:

for f in metadata/*.json; do
  status=$(jq -r .modification_status "$f" 2>/dev/null)
  if [ "$status" = "modified" ]; then
    pkg=$(basename "$f" .json)
    reason=$(jq -r .modification_reason "$f" 2>/dev/null)
    echo "$pkg: $reason"
  fi
done

Viewing Package Differences

To see what changes exist in a modified package compared to upstream Fedora:

# Show full diff for a package
./ci/dist_git.py diff bash

# Show summary statistics
./ci/dist_git.py diff bash --stat

# Show only which files changed
./ci/dist_git.py diff bash --name-only

# Show raw diff (includes Release: bumps and whitespace)
./ci/dist_git.py diff bash --raw

# Diff all modified packages
./ci/dist_git.py diff --all

What’s shown:

  • By default, the diff ignores Release: number changes (no-change rebuilds)
  • Trailing whitespace and blank line changes are ignored
  • Use --raw to see absolutely everything, including Release: bumps

Package types:

  • Modified packages: Shows the differences
  • Clean packages: Shows nothing (useful for verification)
  • Independent packages: Skips with message “no upstream to diff against”

Marking Packages

Mark as Modified

Use this when you make local changes to a package (backports, custom patches, etc.):

./ci/dist_git.py mark-modified <package> --modified \
  --reason "Brief explanation of why"

Examples:

# After backporting a patch
./ci/dist_git.py mark-modified gcc --modified \
  --reason "Backport CVE-2024-12345 fix from upstream"

# After custom spec change
./ci/dist_git.py mark-modified systemd --modified \
  --reason "Add custom service unit for Hummingbird"

The reason field is required and should be concise but descriptive. It helps future maintainers understand why the package can’t be auto-updated.

Mark as Clean

Use this to re-enable automatic updates after confirming your changes are no longer needed (e.g., the fix landed in Fedora):

./ci/dist_git.py mark-modified <package> --clean

This removes the modified status and allows the package to receive automatic updates from Fedora again.

Configure Upstream Tracking

Use set-upstream to configure upstream tracking settings for a package. Each flag independently sets or clears one metadata field. At least one flag is required; omitted flags leave their fields untouched.

./ci/dist_git.py set-upstream <package> [flags]
Flag Sets field Clears with
--track-version latest track_upstream: "latest" --no-track-version
--track-version VER track_upstream: "VER" --no-track-version
--project-id ID release_monitoring_project_id (int) --no-project-id
--project-id NAME release_monitoring_project_id (str) --no-project-id

Each set/clear pair is mutually exclusive (can’t pass --track-version and --no-track-version together).

Examples:

# Enable upstream version tracking (any version)
./ci/dist_git.py set-upstream bash --track-version latest

# Disable upstream version tracking
./ci/dist_git.py set-upstream bash --no-track-version

# Set upstream name with version constraint for versioned packages
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26

# Set release-monitoring.org project ID (integer)
./ci/dist_git.py set-upstream python3.11 --track-version 3.11 --project-id 13254

# Remove project ID (reverts to RPM name lookup)
./ci/dist_git.py set-upstream python3.11 --no-project-id

# Combine multiple flags in one call
./ci/dist_git.py set-upstream golang1.26 \
  --track-version 1.26 --project-id 13254

Metadata fields:

Field Description Example
track_upstream "latest" or version prefix to constrain updates "latest", "1.26"
release_monitoring_project_id Anitya project ID (int) or upstream name (str) 13254, "golang"
version_suffix_strip Suffix to strip from Anitya-reported versions "-RELEASE"
upstream_version_transform Named transform from Anitya version to RPM scheme "openjdk_to_rpm"
source_availability_check Source probe to run before selecting a version {"type": "http", "urls": [...]}

These fields affect two systems:

  • dist_git.py update: When track_upstream is a version prefix, skips upstream versions that don’t match. For example, track_upstream: "1.26" allows 1.26, 1.26.0, 1.26.3 but rejects 1.27.0.
  • check_upstream_versions.py: When release_monitoring_project_id is an integer, queries the v2 API directly by Anitya project ID. When it is a string, queries release-monitoring.org using that name instead of the RPM package name (e.g., looks up golang instead of golang1.26). When absent, uses the RPM package name. When track_upstream is a version prefix, filters the reported upstream versions to only those matching the prefix. Only packages with track_upstream set are checked by check_upstream_versions.py check when no explicit package arguments are given.

When a successful lookup reports releases but none match the configured version prefix, the checker skips that package with a warning instead of failing the job. For example, Python 3.15 tracking can wait while Anitya only reports Python 3.14 releases. The package is not counted as up to date, no spec or source update is attempted, and a warning-only run exits successfully. Empty upstream results and HTTP, connection, or JSON parsing failures during lookup remain errors. Existing warnings for unpublished source archives remain nonfatal as well.

Find release-monitoring.org project IDs by searching on https://release-monitoring.org.

The project ID can be combined with a version prefix to filter versions returned by the project ID lookup:

{
  "release_monitoring_project_id": 13254,
  "track_upstream": "3.11"
}

Some upstream projects tag releases with a suffix that is not part of the RPM version (e.g., swift-6.3.3-RELEASE). After Anitya strips the version prefix, the reported version still contains the suffix (6.3.3-RELEASE), which is incompatible with RPM’s Version: field (hyphens are not allowed). Use version_suffix_strip to remove it before comparison and update:

{
  "release_monitoring_project_id": 21267,
  "version_suffix_strip": "-RELEASE"
}

Some upstream projects publish source tarballs independently of tag creation, and Anitya may report a version before the tarball is available. Use source_availability_check to HEAD-probe one or more required source URLs before selecting a version. Versions whose source returns 404 are skipped in favour of the next available version. The skip is reported as a warning in the scheduled-job log and summary, but does not fail the update job. Routine warnings are omitted from Slack notifications.

The general HTTP probe accepts a non-empty urls list. Every URL must be available. URL templates may use ${VERSION}, ${UPSTREAM_REPO}, and ${TRACK_UPSTREAM}:

{
  "source_availability_check": {
    "type": "http",
    "urls": [
      "${UPSTREAM_REPO}/releases/download/v${VERSION}/libfabric-${VERSION}.tar.bz2"
    ]
  }
}

Legacy named checkers remain available for source layouts that need package-specific logic:

  • openjdk_osci — probes https://openjdk-sources.osci.io/openjdk{feature}/openjdk-{version}.tar.xz
  • openjdk_ga checks for a jdk-<version>-ga tag in upstream_repo and excludes build zero. It selects base releases with at most four version components. Backport fixes from extra-component emergency releases separately, as Fedora does. Use it when fetching OpenJDK archives directly from GitHub after Fedora maintenance ends.
  • tomcat_apache — probes the Tomcat source archive on downloads.apache.org, and its detached signature for Tomcat 10 before allowing an update
{
  "release_monitoring_project_id": 369281,
  "track_upstream": "21",
  "upstream_version_transform": "openjdk_to_rpm",
  "source_availability_check": "openjdk_osci"
}

version_from_ref (fixed-ref, no-tagged-release packages)

Most gorget source-pipeline.yaml fetch steps are version-templated (ref: "v${VERSION}"), so dist_git.py update can invoke gorget with metadata/<package>.json’s own version directly. A few packages instead pin a fixed git-snapshot commit with no tagged upstream release at all (gcc, glibc, libyuv, php-patchwork-jsqueeze, vim) – their version field doesn’t change when the pinned commit does, so it can’t drive gorget’s --version argument.

version_from_ref tells dist_git.py update how to derive gorget’s --version string from a newly-pinned commit, instead of requiring someone to compute it by hand every time (the previous process, documented as a manual step in each such package’s source-pipeline.yaml comments).

Currently one type is supported:

  • commit-date (used by gcc): "<metadata.json's version>-<commit's own YYYYMMDD date>", where the date comes from git log -1 --format=%cd --date=format:%Y%m%d <ref> run against the newly-pinned commit.
{
  "version_from_ref": {"type": "commit-date"}
}

Currently gcc-only. dist_git.py update only fully automates a fixed-ref pin refresh for gcc’s simple case (a full 40-char commit SHA lives directly in a spec %global). The other four packages pin via a git describe-style string (e.g. glibc’s %{glibcsrcdir} macro, glibc-2.43-47-gbc95068f5f) whose trailing hash is only a 10-character abbreviation – resolving that back to a full SHA needs a local clone containing the commit object, real extra network work with its own failure modes (ambiguous hashes, upstream unavailability), not yet implemented (see HUM-4621’s “Remaining work”). For those four, dist_git.py update detects when the pipeline’s pinned ref: no longer matches the merged spec and refuses to auto-commit – it’s routed to a draft, no-test-labeled MR for manual resolution instead, the same way a real git merge conflict is, rather than silently committing a sources file gorget was never actually run against.

Sudo Patchlevel Comparisons

For sudo, upstream version comparisons include the spec’s optional extraver macro. Fedora’s Version: 1.9.17 with extraver p2 therefore compares as 1.9.17p2, preventing an unnecessary update and a doubled p2 in the source URL. This comparison does not change the spec, the RPM version/release, or the version stored in package metadata. Other packages' extraver macros are not treated as upstream version suffixes.

Per-Package Update Hooks

When check_upstream_versions.py check --update updates a package, by default it sets Version: to the new upstream version and Release: to 0.1%{?dist} (unless %autorelease is used), adds a changelog entry, and downloads new sources from the URLs declared in the spec. Some packages need custom logic (e.g. generating stripped tarballs or patching macro-based version lines). A per-package hooks file lets you override or extend these default phases without changing check_upstream_versions.py itself.

Hooks file location

metadata/<package>.update-hooks.yaml

For example, metadata/nodejs25.update-hooks.yaml.

Hook phases

The YAML file supports three optional keys. Each value is a shell command string executed with bash -eo pipefail -c in the package directory as the working directory.

Phase Behaviour
update_spec Replaces the default update that sets Version: to the new upstream version and Release: to 0.1%{?dist}. A changelog entry is still added automatically.
download_sources Replaces the default URL-based source download. Must print one filename per line to stdout for files to upload to the lookaside cache. Redirect any other output to stderr (>&2).
post_update Additive — runs after spec + sources are ready. No default equivalent.

Omitting a phase means the default logic runs for that phase. Packages without a hooks file behave identically to before.

Unknown phase keys in the YAML cause a ValueError (fail-fast).

Environment variables

Every hook receives these environment variables:

Variable Example
UPDATE_PACKAGE nodejs25
UPDATE_OLD_VERSION 25.6.1
UPDATE_NEW_VERSION 25.8.2
UPDATE_SPEC_FILE /home/rpms/rpms/nodejs25/nodejs25.spec
UPDATE_PACKAGE_DIR /home/rpms/rpms/nodejs25
UPDATE_SOURCES_FILE /home/rpms/rpms/nodejs25/sources
UPDATE_ROOT_DIR /home/rpms

Example

See metadata/nodejs25.update-hooks.yaml for a working example that uses all three hook phases.

How Auto-Updates Work

The ./ci/dist_git.py update command (used by automation) checks modification status before updating packages:

  • clean packages: Updated automatically when new Fedora versions are available
  • modified packages: Automatically merged with upstream changes (conflicts create draft MRs)
  • independent packages: Update blocked (not sourced from Fedora)
  • version-constrained packages: Skipped if upstream version doesn’t match track_upstream prefix

To force-update a modified package (discarding local changes):

./ci/dist_git.py sync <package>

The sync command bypasses the modification check and force-updates to the latest upstream version. After syncing, the package is automatically marked clean.

Resolving Merge Conflicts

When modified packages are updated from Fedora, dist_git.py update attempts to automatically merge local changes with the new upstream version using git’s three-way merge. When conflicts occur, the update still succeeds but creates a commit with conflict markers, and the automation files a draft merge request labeled with CONFLICT: for manual resolution.

Understanding Conflict Markers

Git uses this conflict marker structure:

<<<<<<< HEAD
Fedora's version (new upstream)
=======
Hummingbird's local modifications
>>>>>>> hummingbird-local

ALL THREE markers must be removed for a clean resolution.

Update branch/MR structure

MRs are created on branches following the pattern:

chore/dist-git-update-PACKAGENAME

These branches are automatically created by the dist_git_update GitLab schedule. If they have conflicts, they result in draft MRs with:

  • Title prefix: CONFLICT: chore(rpms): Update ...
  • Description listing the conflicting files
  • no-test label to skip CI tests (saves resources since conflicts need manual resolution)

Resolution Process

  1. Check out the conflict branch:

    git fetch origin
    git checkout origin/chore/dist-git-update-PACKAGENAME
    
  2. Examine the conflict:

    # Find all files with conflict markers
    git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/
    
    # View the specific conflict
    git show HEAD:rpms/PACKAGENAME/PACKAGENAME.spec | grep -B5 -A10 "^<<<<<<< HEAD"
    
  3. Understand the local changes:

    # Review commit history to understand why changes were made
    git log --oneline -- rpms/PACKAGENAME/
    git log -p -- rpms/PACKAGENAME/  # With diffs
    
    # Check the modification reason
    jq -r .modification_reason metadata/PACKAGENAME.json
    

    Understand the context for correct resolution:

    • What was the original purpose of the local change? Is it transient or permanent?
    • Is it a workaround for a bug, a security patch, or a configuration difference?
    • Does it affect other packages (e.g., nss builds nspr as a subpackage)?
    • Check spec file comments (e.g., NOTE: comments) for packaging details

    Decide which version to accept:

    • Accept HEAD (Fedora) for: release number lags, fixed workarounds that Fedora improved or addressed differently
    • Keep hummingbird-local for: security patches not in Fedora, FIPS requirements, critical fixes, and other permanent modifications
    • Merge both for: test skip lists, independent changes that don’t conflict logically
    • When in doubt: Accept Fedora’s version for packaging metadata (Release:, subpackage versions), keep Hummingbird’s version for functional changes (patches, dependencies, build options)
  4. Resolve the conflict: Edit the file to choose the appropriate version (HEAD, hummingbird-local, or merge both). Verify no markers remain:

    git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/
    
  5. Validate the resolution: Check that local modifications are preserved:

    # Check the diff against upstream (works on working tree, staging not required)
    ./ci/dist_git.py diff PACKAGENAME
    
    # Compare with previous modification commits to verify
    git log -p -- rpms/PACKAGENAME/
    

    The diff should show only the intended local modifications (ignoring Release: bumps). This confirms the merge preserved your changes correctly. Note: dist_git.py diff compares the filesystem working tree against upstream, so it works before or after staging.

  6. Amend the commit: Record the original conflicted commit SHA, then amend:

    # Record the original conflicted commit SHA
    ORIGINAL_SHA=$(git rev-parse HEAD)
    
    # Stage the resolved files and amend the commit
    git add rpms/PACKAGENAME/
    git commit --amend -m "$(git log -1 --format=%B | head -n -1)
    
    Conflicted-Update: $ORIGINAL_SHA"
    

    This preserves the original commit message while adding a Conflicted-Update: trailer that records which commit contained the conflict markers. This helps track the resolution history and can be useful for auditing or debugging later.

  7. Push the resolution:

    git push origin HEAD:chore/dist-git-update-PACKAGENAME --force-with-lease --push-option merge_request.unlabel=no-test
    

    This removes the no-test label from the MR, which triggers CI tests to run and verifies the resolution works correctly. Some developers might have origin as read-only remote, and a different writable remote (e.g. originw).

Common Conflicts

nss: Subpackage Release Numbers

The nss package builds nspr as a subpackage with its own release number offset.

BACKGROUND:

  • nss builds both nss and nspr RPMs from the same source
  • nspr_release uses an offset (%[%baserelease+n]) to avoid NVR clashes
  • The spec file NOTE explains: reset to 1 when nspr_version changes, increment when only nss changes
  • Fedora manages these offsets in their ecosystem to prevent conflicts

CONFLICT EXAMPLE:

<<<<<<< HEAD
%global nspr_release %[%baserelease+3]
=======
%global nspr_release %[%baserelease+1]
>>>>>>> hummingbird-local

REASONING: When updating to a new upstream nss version from Fedora:

  • Accept Fedora’s nspr_release offset (HEAD) - they manage NVR clashes
  • Our local offset was specific to Hummingbird rebuilds
  • New upstream version should reset to Fedora’s packaging values
  • Don’t try to “calculate” what it should be - trust Fedora’s packaging

RESOLUTION: Accept HEAD (Fedora’s value)

Special Case: Rebuild-Only Changes

Release-only changes (no-change rebuilds) are automatically ignored by the modification detection logic. This means:

  • Bumping Release: 3%{?dist} → Release: 3.1%{?dist} does not mark the package as modified
  • The package can still receive automatic Fedora updates
  • The Release bump will be preserved if the update doesn’t change the upstream Release field

You do not need to mark packages as modified for rebuild-only changes, unless you want to explicitly prevent automatic updates for other reasons.

CI Validation

The CI pipeline validates modification status consistency using make check, which runs:

./ci/validate_package_modifications.py --all

This validation ensures:

  1. All packages have a modification_status field
  2. The value is one of: clean, modified, independent
  3. If track_upstream is present, it must be a string ("latest" or a version prefix)
  4. Modified packages have a modification_reason
  5. Independent packages do not have source/branch/sha fields (Hummingbird-independent only)
  6. Git commit history matches the declared modification status

The validation runs on every merge request and push to main, failing the build if metadata is inconsistent.

For local development, run the full validation:

./ci/validate_package_modifications.py --all

Or validate specific packages:

./ci/validate_package_modifications.py bash glibc gcc

Validation Modes

The validation script has two modes:

Fast mode (default): Checks git commit history patterns

./ci/validate_package_modifications.py --all

This validates that all commits since the last Sync follow standard patterns (have Upstream: trailers). Runs in less than a minute for all packages.

Thorough mode: Clones upstream repos and compares filesystems

./ci/validate_package_modifications.py --all --thorough

This performs full filesystem comparisons with upstream Fedora repositories. Slow and unreliable (hundreds of upstream dist-git clones) but authoritative - validates actual state regardless of git commit history.

For CI and daily development, fast mode is sufficient. Use thorough mode when:

  • Debugging discrepancies between metadata and actual state
  • Auditing the entire repository for hidden modifications
  • Investigating why a package can’t be updated

Workflow Examples

Backporting a Patch

  1. Add patch file and modify spec (see Rebuilding Packages)

  2. Commit the changes

  3. Mark as modified:

    ./ci/dist_git.py mark-modified dnf5 --modified \
      --reason "Backport reproducible build fix (upstream PR#2522)"
    
  4. Package is now protected from automatic Fedora updates

Re-enabling Auto-Updates

When your backported fix lands in Fedora:

  1. Verify the fix is in the latest Fedora version:

    ./ci/dist_git.py update dnf5  # This will fail with "modified" error
    
  2. Mark the package clean:

    ./ci/dist_git.py mark-modified dnf5 --clean
    
  3. Update from Fedora:

    ./ci/dist_git.py update dnf5  # Now succeeds
    

Importing New Packages

When importing packages, modification status is set automatically:

# Fedora package → marked as "clean"
./ci/dist_git.py import fedora/neofetch

# Hummingbird-independent package → marked as "independent"
./ci/dist_git.py import hummingbird/custom-tool

No manual marking needed for imports.

Troubleshooting

CI Fails: “Missing modification_status field”

This means a metadata file is missing the required field. This means that the package was not imported properly.

CI Fails: “Marked as clean but package has modifications”

The package has local changes but metadata says it’s clean. To fix:

  1. Check what changed:

    git log -p -- rpms/<package>/
    
  2. Mark as modified with the appropriate reason:

    ./ci/dist_git.py mark-modified <package> --modified --reason "..."
    

CI Fails: “Marked as modified but package is actually clean”

The package has no local changes but is marked modified. To fix:

  1. Verify it’s actually clean:

    ./ci/dist_git.py update <package>  # Check if upstream matches
    
  2. If confirmed clean, remove the modified status:

    ./ci/dist_git.py mark-modified <package> --clean
    

Update Blocked: “Cannot auto-update <package>”

This is expected for modified packages. Options:

  1. Wait for fix to land in Fedora, then mark clean and update

  2. Force-sync to discard local changes:

    ./ci/dist_git.py sync <package>
    
  3. Keep blocked if the local changes are still needed

3.24 - Private RPM Repositories

AI Agent Note: Assign a private product through hum.meta, which saves private_product in metadata/<package>.json, then run make generate. The product must match an existing RPA entry in ci/konflux_rpa_config.yml.

Overview

Hummingbird LTS products (e.g., LTS OpenJDK, LTS DotNet) require subscription-gated RPMs that must not be publicly accessible. Each LTS product gets its own private Pulp repository with independent credentials, enabling per-product subscription enforcement.

The private RPM pipeline uses the same build infrastructure as public RPMs but routes packages to separate Pulp domains via dedicated Konflux Applications and ReleasePlanAdmissions (RPAs).

How it works

  1. Each private product has a dedicated Konflux Application (e.g., private-<product>-rpms-main) separate from the public rpms-main application.
  2. A ReleasePlan connects the private Application to its RPA, enabling auto-releases when builds complete.
  3. Packages are assigned to a private product via private_product in metadata/<package>.json, read and written through hum.meta.
  4. A dedicated RPA per product routes those packages to private Pulp repositories.
  5. Packages with private_product are automatically excluded from the public RPA.

Architecture

metadata/<package>.json         konflux_rpa_config.yml
  "private_product": "<product>"   rpas:
                                    - name: ...public...
                                    - name: ...private-<product>...
                                      private_product: <product>
        |                                    |
        v                                    v
  Component resource              ReleasePlanAdmission
  application: private-           targets: private-<product>-rpms-main
    <product>-rpms-main           pulp_signed_domain: private-hummingbird-<product>
                                           ^
                                           |
                                  ReleasePlan
                                  application: private-<product>-rpms-main
                                  releasePlanAdmission: hummingbird-rpms-private-<product>

Assigning a Package to a Private Product

Set private_product through hum.meta for an existing product:

make container <<'SH'
python3 - <<'PY'
from pathlib import Path
from hum.meta import MetadataRepository

repo = MetadataRepository(Path.cwd())
metadata = repo.load("<package>")
metadata.private_product = "<product>"
repo.save(metadata)
PY
SH

This updates metadata/<package>.json without changing the package’s state. The library checks that the product has a matching RPA entry. Do not infer private or LTS status from a package-name suffix, and do not add assignments to ci/package-overrides.yaml.

Then regenerate all resources:

make generate

This will:

  • Set the package’s Konflux Component to use application: private-<product>-rpms-main
  • Include the package in the private product’s RPA component list
  • Automatically exclude the package from the public RPA

Verifying the assignment

After running make generate, verify the changes:

# Check the component's application assignment
grep -A5 '<package>-main' konflux-templates/rendered.yml | grep application

# Check the package appears in the private RPA
grep '<package>-main' releng/hummingbird-rpms-private-<product>.yaml

# Check the package is excluded from the public RPA
grep '<package>-main' releng/hummingbird-rpms-tech-preview-staging.yaml
# (should return no results)

Onboarding a New Private Product

Automated onboarding

Use the add-private-product subcommand to automate Pulp setup, RPA configuration, package assignment, and resource regeneration in a single step:

./ci/dist_git.py add-private-product <product> \
  --packages pkg1,pkg2 \
  --infra-repo <path-to-infrastructure-repo> \
  --pulp-config <path-to-cli.toml>

This handles Pulp infrastructure (step 2), RPA config (step 5), package assignment (step 6), infrastructure repo templates (step 4 file creation), and resource regeneration. You still need to manually complete: credentials secret deployment (step 3), merging the infrastructure repo MR created by this tool (step 4), and content guard setup (step 7).

Use --dry-run to preview changes without modifying anything.

Manual onboarding

Adding a new LTS product (beyond existing ones) requires these steps: service account, Pulp infrastructure, credentials secret, Konflux Application, RPA configuration, resource regeneration, and content guard setup.

1. Create a Pulp service account (optional)

If you want isolated credentials for private repo operations (recommended), create a dedicated service account before setting up the Pulp infrastructure. See Pulp Access for instructions on creating a service account, configuring the CLI, and storing credentials in the vault. Otherwise, you can reuse the existing public Pulp credentials.

2. Create Pulp infrastructure

Use the existing Pulp setup script to create the private domains, RPM repositories, and file repositories (for SBOMs and attestations). See ci/pulp-setup/README.md for the required pulp-cli-console plugin install before running these commands:

# Unsigned (staging) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned

# Unsigned (staging) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>-unsigned \
  --type file

# Signed (production) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product>

# Signed (production) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
  --domain private-hummingbird-<product> \
  --type file

If using a dedicated service account, add --config <path-to-cli.toml> to each command.

This creates per-architecture RPM repositories (source, x86_64, s390x, ppc64le, aarch64) with distributions, and file repositories (metadata, rpm-catalog) for SBOM and attestation storage.

3. Deploy the Pulp credentials secret

If using a dedicated service account (step 1), add a Kubernetes Secret template to the infrastructure repo so the credentials are deployed to the Konflux cluster:

  1. Create kubernetes/setup-konflux/47-pulp-private-hummingbird-config-file-secret.yml.j2:

    ---
    kind: Secret
    apiVersion: v1
    metadata:
      name: pulp-private-hummingbird-config-file-secret
    stringData:
      cli.toml: «{ lookup_secret("HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE[deployed]:cli.toml") | forceescape }»
    type: Opaque
    
  2. Add an entry to secrets.yml for the vault key:

    HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE:
      backend: hv
      meta:
        active: true
        created_at: "<timestamp>"
        deployed: true
    

    The secret name (pulp-private-hummingbird-config-file-secret) must match the pulp_secret_name in the private RPA configuration (step 5).

  3. Add an ExternalSecret to the konflux-release-data repo (rhtap-release-data) so the credentials are available to the release pipeline. Create the ExternalSecret in the cluster-specific directory (e.g., tenants-config/cluster/kflux-prd-rh03/managed/rhtap-releng-tenant/):

    ---
    apiVersion: external-secrets.io/v1
    kind: ExternalSecret
    metadata:
      name: hummingbird-private-pulp-bot-production-secret
      annotations:
        argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
        argocd.argoproj.io/sync-wave: "0"
    spec:
      dataFrom:
        - extract:
            key: releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production
      refreshInterval: 1h
      secretStoreRef:
        kind: SecretStore
        name: releng-vault
      target:
        creationPolicy: Owner
        deletionPolicy: Delete
        name: hummingbird-pulp-credentials-private-production-secret
    

    Add it to the cluster’s kustomization.yaml and regenerate auto-generated files with tenants-config/build-manifests.sh.

  4. Add the secret to the releng vault at https://vault.devshift.net under the path releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production with the following keys:

    Key Value
    cli.toml Full contents of the Pulp CLI config file
    expires-on na
    owner Your Kerberos username

4. Create the Konflux Application

The Application resource is managed in the infrastructure repo, not this repo. Create it there following the same pattern as rpms-main:

  1. Create a new directory kubernetes/private-<product>-rpms-main/ in the infrastructure repo.

  2. Add 00-application.yml.j2 with the standard Application template:

    ---
    apiVersion: appstudio.redhat.com/v1alpha1
    kind: Application
    metadata:
      name: {{ env["PROJECT_NAME"] }}
    spec:
      appModelRepository: {url: ""}
      displayName: {{ env["PROJECT_NAME"] }}
      gitOpsRepository: {url: ""}
    
  3. Add the new project to the CI matrix in infrastructure/.gitlab-ci.yml. Find the PROJECT_NAME list under the konflux-rh03/hummingbird-tenant context and add the new application name:

    - PROJECT_NAME:
        - rpms-main
        - private-<product>-rpms-main # add this line
      PROJECT_CONTEXT:
        - konflux-rh03/hummingbird-tenant
    
  4. Add 01-release-plans.yml.j2 with a ReleasePlan that references the private RPA. Copy from kubernetes/rpms-main/01-release-plans.yml.j2 and update the releasePlanAdmission label to match the private RPA name:

    metadata:
      name: hummingbird-rpm-release-private-<product>
      labels:
        release.appstudio.openshift.io/auto-release: "true"
        release.appstudio.openshift.io/standing-attribution: "true"
        release.appstudio.openshift.io/releasePlanAdmission: hummingbird-rpms-private-<product>
    spec:
      application: {{ env["PROJECT_NAME"] }}
      target: rhtap-releng-tenant
      # ... copy remaining spec from rpms-main/01-release-plans.yml.j2
    

    The ReleasePlan connects the private Application to its ReleasePlanAdmission. Without it, builds in the private Application will not trigger releases.

    The IntegrationTestScenario for the new private application is generated automatically by this repo’s ci/generate_resources.py — it derives the application list from ci/konflux_rpa_config.yml. Once you add the RPA entry in step 5 and run make generate, the new application’s testing-farm ITS will appear in konflux-templates/rendered.yml and the trigger manifest. The ITS is deployed to the cluster via the *-releases infrastructure sync after this MR merges to main.

Merge this MR in the infrastructure repo first — the CI pipeline will deploy the Application and ReleasePlan to the cluster. Components and RPAs in this repo reference it by name, so the Application must exist before they are applied.

5. Add RPA configuration

Add a new entry to the rpas: list in ci/konflux_rpa_config.yml:

rpas:
  # Existing public RPA (private packages are automatically excluded)
  - name: hummingbird-rpms-tech-preview-staging
    # ... existing config ...

  # New private product RPA
  - name: hummingbird-rpms-private-<product>
    application_prefix: private-<product>-rpms
    release_org: registry.stage.redhat.io/hummingbird-tech-preview
    single_component_mode: true
    service_account_name: hummingbird-rpm-release-staging
    pulp_unsigned_domain: private-hummingbird-<product>-unsigned
    pulp_signed_domain: private-hummingbird-<product>
    pulp_secret_name: hummingbird-pulp-credentials-private-production-secret
    pipeline_revision: <release-pipeline-branch>
    pipeline_url: https://github.com/scoheb/release-service-catalog.git
    private_product: <product>
    component_filter:
      path_prefix: rpms/

Key fields:

Field Purpose
name Kubernetes resource name for the RPA
application_prefix Combined with branch to form the Konflux Application name
private_product Matches the product assignment in metadata/<package>.json
pulp_unsigned_domain Pulp domain for unsigned RPMs (staging)
pulp_signed_domain Pulp domain for signed RPMs (production)
pulp_secret_name Kubernetes secret containing Pulp publishing credentials

6. Assign packages and regenerate

Assign each package through hum.meta as described in Assigning a package, then regenerate:

make generate

This produces:

  • Updated konflux-templates/rendered.yml with per-component application assignments
  • A new RPA file at releng/hummingbird-rpms-private-<product>.yaml
  • Updated public RPA excluding private packages

7. Set up Pulp content guard

To restrict access to the private Pulp repositories, a content guard must be configured so only customers with the correct subscription can access the content:

  1. Obtain a SKU for the private product’s subscription offering.
  2. Create a feature for that SKU in the Feature service.
  3. Contact the Pulp team to:
    • Map the organization ID to the feature
    • Set up the content guard on the private Pulp domain(s)

This step is required before customers can access the private repositories. Without it, the repositories are created but have no access control.

Pulp credentials

If using a dedicated service account (step 1), see Pulp Access for vault storage instructions.

If using the shared service account, private RPAs reuse the same Pulp publishing credentials as the public RPA (hummingbird-pulp-credentials-production-secret). Per-product access control for customers is handled downstream by Red Hat’s subscription entitlement system, not at the Pulp publishing layer.

Public/Private Routing

Packages with private_product set in metadata/<package>.json are automatically excluded from the public RPA and included only in their product’s private RPA. Set metadata.private_product to None and save through hum.meta to return the package to public-only publishing.

File Purpose
metadata/<package>.json Per-package private_product assignment
ci/konflux_rpa_config.yml RPA definitions (public + private)
ci/generate_resources.py Generates Components, RPAs, and PipelineRuns
ci/pulp-setup/create-pulp-resources.sh Creates Pulp domains and repositories
konflux-templates/macros/releng/release-plan-admission.yml.j2 RPA template
konflux-templates/macros/component.yml.j2 Component template (per-component application)
konflux-templates/macros/image-repository.yml.j2 ImageRepository template (per-component application)

3.25 - Reporting CVE Data Issues

Overview

Sometimes CVE data published on cve.org or NIST NVD is incorrect – wrong affected package, wrong severity score, wrong affected version range, or missing fix information. When this happens, the incorrect data propagates into vulnerability scanners and can cause false positives (or false negatives) for Hummingbird container images.

This document describes how to report these issues to get the data corrected at the source.

When to Report

Report a CVE data issue when you find any of the following:

  • Wrong affected package – the CVE Record lists a package that is not actually affected
  • Wrong version range – the affected or fixed version boundaries are incorrect
  • Wrong severity – the CVSS score or severity rating does not match the actual impact
  • Missing fix information – a fix exists upstream but the CVE Record does not reflect it
  • Wrong CPE match – the NVD CPE configuration matches products that are not affected

How to Report

Reporting to the CVE Program (cve.org)

The CVE Program accepts corrections through the CVE Numbering Authority (CNA) that owns the record.

  1. Find the CVE Record at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX.
  2. Identify the CNA listed in the record (shown in the “Assigning CNA” field).
  3. Contact the CNA directly to request a correction. Most CNAs accept reports via:
    • Their security reporting email (often listed in their CNA page)
    • GitHub issues if the CNA is an open source project
  4. If the CNA is unresponsive, use the CVE Program Request form to dispute the record.

Reporting to NIST NVD

NIST NVD enriches CVE Records with CVSS scores and CPE match data. To request corrections:

  1. Navigate to the NVD entry at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX.
  2. Click the “Contact” link or email nvd@nist.gov with:
    • The CVE ID
    • The specific field that is incorrect
    • Evidence of the correct data (links to upstream commits, release notes, etc.)

Reporting to Red Hat Product Security

If the incorrect CVE data is causing issues in Red Hat’s vulnerability tracking:

  1. File a Jira ticket in the HUM project with the CVE ID and a description of the data issue.
  2. Red Hat Product Security can update the VEX feed to reflect the correct disposition for Hummingbird, even before the upstream CVE data is corrected.

Tracking the Correction

After reporting, track the status of the correction:

  • CVE Record updates are published at https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX
  • NVD updates appear at https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX (NVD may take days to weeks to process corrections)
  • Update the HUM Jira ticket with the correction status so the team is aware

Requesting information on the fix status of an existing CVE

File a support request at https://access.redhat.com/support.

3.26 - Lookaside Cache Access

How to access the S3-based lookaside cache for RPM source tarballs

Overview

Source tarballs for RPM packages are stored in an S3-based lookaside cache rather than in git. To upload files to the cache (e.g., when updating a package to a new upstream version), you need AWS credentials with the appropriate permissions.

The ci/upload-to-lookaside-cache.sh script handles uploads directly, and ci/check_upstream_versions.py --update calls it automatically when downloading new source archives.

Prerequisites

You must be a poweruser in the arr-cloud-aws-core group (it-cloud-aws-727920394381-poweruser). Request access to this group if you do not already have it.

Obtaining AWS Credentials

There are two ways to authenticate.

Option A: Browser-based login

Run aws login, which opens a browser for authentication:

aws login

Option B: Kerberos-based login via container

Use the CKI tools container to obtain credentials via Kerberos:

$ podman run --rm -it \
    -e KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u) \
    -v $HOME/.aws:/cki/.aws:U,Z \
    quay.io/cki/cki-tools:latest

bash-5.2# kinit <userid>@IPA.REDHAT.COM

bash-5.2# AWS_IDP_URL=https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws \
    cki_aws_login --duration 43200 \
    --account 727920394381 --role poweruser

Replace <userid> with your Kerberos user ID. This writes credentials to $HOME/.aws on the host (bind-mounted into the container).

Note: This overwrites the AWS default profile credentials. If you use the default profile for other purposes, back up $HOME/.aws/credentials before running this command.

Uploading Files

Once credentials are configured in $HOME/.aws, you can upload files to the lookaside cache.

Manual upload

./ci/upload-to-lookaside-cache.sh -f <file> -p <package>

Example:

./ci/upload-to-lookaside-cache.sh -f rpms/tar/tar-1.35.tar.xz -p tar

Automated upload via version checker

check_upstream_versions.py --update downloads new source archives and uploads them to the lookaside cache automatically:

./ci/check_upstream_versions.py check --update <package>

The check subcommand only processes packages with "track_upstream": true in their metadata when no explicit package arguments are given. To see all packages and their upstream status, use check_upstream_versions.py list. See Package Modification Tracking for how to enable tracking.

See Also

3.27 - Upstream Diff Analysis

Overview

The upstream diff analysis tool classifies locally modified RPM packages by how their changes relate to upstream Fedora. This helps the team systematically decide which modifications should be proposed upstream, which are Hummingbird-specific, and which need more complex handling.

The primary interface is the /upstream-diff Claude Code skill, which orchestrates the full workflow. The skill delegates all deterministic operations (diffing, metadata lookup, caching) to ci/upstream_diff.py and applies its own judgment only for the classification step.

Classification Categories

Each modified package is classified into one of five categories:

Category Meaning Action
upstreamable Changes can be submitted to Fedora as-is File upstream PR
hummingbird-specific Changes are intentional and specific to Hummingbird Keep locally, no upstream action needed
complex Changes need investigation or partial upstreaming Requires human decision
mixed Some changes are upstreamable, others are Hummingbird-specific Split and handle separately
no-diff Package is marked modified but has no actual diff Consider unmarking as modified

Workflow

Invoke the skill in Claude Code with /upstream-diff. When called with no arguments, it offers three modes:

Analyze

Prepares the next batch of unanalyzed or stale packages, reviews each diff and any upstream PR activity, then classifies each package into a category with reasoning and a recommended action. Results are cached locally so subsequent runs skip already-analyzed packages.

You can also target specific packages by name (e.g., /upstream-diff analyze bash glibc).

View Results

Displays the cached analysis as a summary table. Supports filtering by category and showing details for a single package. Use this to get a quick overview of where things stand across all modified packages.

JIRA

Creates or updates a JIRA issue for a specific package to track the upstream proposal. The skill generates a pre-filled description and analysis comment from the cached data, previews it for confirmation, and then files or updates the issue. The JIRA issue key is recorded in the cache so it appears in the view output.

Script Reference

The skill uses ci/upstream_diff.py under the hood. The script can also be called directly for automation, scripting, or when working outside Claude Code.

Note that prepare and save are two halves of a classification pipeline: prepare gathers raw data (diffs, metadata, upstream PRs, category definitions), but its output requires human or LLM judgment to select a category and write the reasoning fields before calling save. The skill’s Analyze mode bridges this gap automatically.

Prepare

Gather metadata, diffs, upstream PR status, and the classification schema — this is the data the skill’s Analyze mode feeds into its classification step:

# Specific packages
./ci/upstream_diff.py prepare bash glibc

# Next batch of unanalyzed packages
./ci/upstream_diff.py prepare --batch 10

Save

Store a classification result. The skill’s Analyze mode calls this after classifying each package:

./ci/upstream_diff.py save <package> \
  --category <category> \
  --changes-summary "One-line summary of changes" \
  --reasoning "Why this category was chosen" \
  --recommendation "Recommended next action" \
  --upstream-prs <pr_id1> <pr_id2> ...

# No related upstream PRs
./ci/upstream_diff.py save <package> \
  --category hummingbird-specific \
  --changes-summary "Custom Hummingbird macros for FIPS" \
  --reasoning "FIPS build flags are specific to Hummingbird" \
  --upstream-prs ""

The --hummingbird-macros flag can be added when changes use Hummingbird-specific RPM macros.

Results are cached in .cache/upstream-diff-analysis.json.

View

Display cached results — this is what the skill’s View Results mode calls:

# Summary table
./ci/upstream_diff.py view

# Single package detail
./ci/upstream_diff.py view <package>

# Filter by category
./ci/upstream_diff.py view --category upstreamable

# Include unanalyzed modified packages
./ci/upstream_diff.py view --all

# Markdown-formatted links (for JIRA or rendered contexts)
./ci/upstream_diff.py view --markdown

# Raw JSON
./ci/upstream_diff.py view --json

Check Upstream PRs

Check whether local changes have already been submitted or merged in the upstream Fedora dist-git repository:

./ci/upstream_diff.py check-prs <package1> <package2> ...

Shows open and recently merged (last 90 days) pull requests from the upstream Fedora dist-git repo.

After initial analysis, update which upstream PRs are related to a package’s local changes:

# Set related PRs
./ci/upstream_diff.py save --set-upstream-prs <package> <pr_id1> <pr_id2> ...

# Clear related PRs
./ci/upstream_diff.py save --set-upstream-prs <package>

JIRA Templates

Generate pre-filled JIRA content from cached analysis — this is what the skill’s JIRA mode uses:

./ci/upstream_diff.py jira-template <package> [--epic HUM-1613]

Record a JIRA issue key after creating an issue:

./ci/upstream_diff.py save --set-jira <package> HUM-XXXX

3.28 - GPG Source Verification

How upstream signing keys are stored, and how to add, rotate, or revoke them

Overview

Some packages verify the authenticity of their upstream source tarball with a detached OpenPGP signature where upstream signs each release with a private key and publishes a .asc/.sig alongside the tarball. During source fetching, the gorget source-pipeline tool checks that signature against the project’s public key before the bytes are ever used in a build.

The trusted public keys live centrally in metadata/gpg-keys/<project>.gpg, one keyring file per upstream project. Centralized storage means:

  • the full set of signers we trust is auditable in one directory, and
  • rotating or revoking a key is a single, reviewable commit.

The directory is handed to gorget as --gpg-keys-dir metadata/gpg-keys (see ci/check_upstream_versions.py), so a pipeline’s keyring: field is just a filename within it.

How the pieces fit together

Three things wire up verification for a package:

  1. The keyring — metadata/gpg-keys/<project>.gpg, the trusted public key(s).

  2. The pipeline step — a verify: entry in metadata/<package>.source-pipeline.yaml:

    fetch:
      - type: url
        url: "https://curl.se/download/curl-${VERSION}.tar.xz"
      - type: url
        url: "https://curl.se/download/curl-${VERSION}.tar.xz.asc"
    
    verify:
      - type: gpg-signature
        target: "curl-${VERSION}.tar.xz"       # the artifact to verify
        signature: "curl-${VERSION}.tar.xz.asc" # its detached signature
        keyring: "curl.gpg"                      # filename in metadata/gpg-keys/
    

    gorget imports keyring into a fresh, throwaway GPG homedir per check, then runs the equivalent of gpg --verify <signature> <target>. A bad or missing signature fails the fetch.

  3. CI validation — test/test_gpg_keys.py (run by make check) confirms every file in metadata/gpg-keys/ is a parseable public key, and that every keyring: referenced by a pipeline actually exists.

Adding a key for a new package

Prerequisite: gpg (from the gnupg2 package). It ships in the CI image, run the commands below inside a container, e.g. podman run --rm -it -v "$PWD:$PWD:z" -w "$PWD" quay.io/hummingbird-ci/gitlab-ci:latest bash.

  1. Obtain the upstream public key. Prefer a key you can already trust: many packages already ship the maintainer’s key next to their spec (e.g. rpms/curl/mykey.asc, rpms/bash/chet-gpgkey.asc). Otherwise download it from the project’s official key page.

  2. Store it as a keyring in metadata/gpg-keys/. The keyring may be ASCII-armored or binary; this repo standardizes on binary .gpg. Convert an armored key with --dearmor:

    gpg --dearmor < rpms/<package>/<upstream-key>.asc > metadata/gpg-keys/<project>.gpg
    

    Name the file after the upstream project, not the RPM (so multiple versioned packages, e.g. python3.11/python3.12, can share one keyring).

  3. Verify the key is what you expect. Print its fingerprints and confirm they match the fingerprints published on the upstream’s official channel:

    gpg --show-keys --with-fingerprint metadata/gpg-keys/<project>.gpg
    
  4. Wire up the pipeline. Add (or extend) metadata/<package>.source-pipeline.yaml with the fetch steps for the tarball + signature and the verify: [{type: gpg-signature, ...}] step shown above.

  5. Prove the whole chain end to end before committing import the keyring into a throwaway homedir and verify a real release signature against it:

    V=<version>
    curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz"
    curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz.asc"
    export GNUPGHOME=$(mktemp -d)
    gpg --import metadata/gpg-keys/<project>.gpg
    gpg --verify "<tarball>-$V.tar.xz.asc" "<tarball>-$V.tar.xz"   # expect "Good signature"
    
  6. Run the checks: make check (validates the keyring and the pipeline reference).

Rotating or replacing a key

Upstream may roll to a new signing key (expiry, policy, new maintainer). Because the trusted set is just files in one directory, rotation is a single commit:

  1. Obtain the new public key from the upstream’s official channel and confirm its fingerprint out of band.

  2. Replace the contents of metadata/gpg-keys/<project>.gpg (re-run the --dearmor step). If upstream signs a transition period with both keys, you may keep both by importing them into the same keyring:

    gpg --dearmor < old-key.asc  > metadata/gpg-keys/<project>.gpg
    gpg --dearmor < new-key.asc >> metadata/gpg-keys/<project>.gpg
    
  3. Re-run the end-to-end verification (step 5 above) against the latest release, then make check.

  4. Commit with a message recording why the key changed and how you confirmed the new fingerprint, this file is the audit trail for what we trust.

Revoking / removing a key

  • If a project drops GPG verification, delete both metadata/gpg-keys/<project>.gpg and the verify: step from its pipeline in the same commit. (CI fails a pipeline that references a missing keyring, and if the key is orphaned the reverse is easy to spot.)
  • If a key is compromised, remove it immediately and replace it with the upstream’s revocation / replacement key.

Troubleshooting

  • gpg: no valid OpenPGP data found — the file isn’t a real keyring (empty, truncated, or you saved an HTML error page). Re-download and re---dearmor. test_gpg_keys.py catches this in CI.
  • gpg: Can't check signature: No public key — the signature was made with a key that isn’t in the keyring. Upstream likely rotated keys, follow Rotating or replacing a key above.
  • BAD signature — the tarball does not match the signature. Do not paper over this. It means a corrupted download or, in the worst case, tampering. Re-fetch from the canonical source and if it persists, escalate rather than accepting the artifact.

3.29 - Rebasing the Buildroot to a New Fedora Release

Step-by-step runbook for moving mock.cfg, the core toolchain, and the rest of the package set to a new Fedora release

AI Agent Note: This document is the procedural runbook for a Fedora buildroot rebase (e.g. F43 -> F44). It was written after executing the F44 rebase (see HUM-2018 and its child stories HUM-3369, HUM-3370, HUM-4536, HUM-4537 for the detailed history) and should be followed, and updated, the next time this operation is performed (e.g. F44 -> F45). Read this whole document before starting; the ordering between steps matters and several steps have hard dependencies on the one before it.

Overview

Periodically, Fedora ships a new stable release and Hummingbird needs to move its buildroot (mock/mock.cfg) and the packages that track a specific Fedora branch (rather than rawhide) onto it. This is a multi-day, multi-phase effort touching the core toolchain, CI/testing configuration, and potentially every package in the repository (since changing the compiler/linker toolchain can affect any build). The F44 rebase took about 2 calendar weeks.

Pre-flight checks

Before starting, confirm:

  • The target Fedora release is actually stable (not just branched) and its package repos are reachable from the build environment.

  • Which packages currently track the outgoing release branch explicitly (i.e. branch is a versioned Fedora branch like f43, not rawhide) via:

    # List every package whose branch is pinned to a specific Fedora release (not rawhide)
    ./ci/dist_git.py list | xargs -I{} sh -c 'echo -n "{}: "; python3 -c "import json; print(json.load(open(\"metadata/{}.json\"))[\"branch\"])"' 2>/dev/null | grep -E ': f[0-9]+$'
    

    For each, decide up front whether it should move to the new branch, move to rawhide, or stay pinned (see “Packages that intentionally stay behind” below) — don’t leave this undecided until a package fails to build.

  • Whether any of those packages carry FIPS-validation-critical modifications. Packages whose modification_reason references FIPS certification should generally not be rebased to new branch content, even though they still need to be rebuilt against the new toolchain. Moving them would risk invalidating the FIPS-validated source. (See HUM-4989 for a related, more general proposal to record why a package is pinned in its metadata rather than relying on the modification reason text.)

  • Whether any package whose version is the “which version is default” signal for an adjacent language/tooling ecosystem (e.g. python-rpm-macros) needs the same branch-pinning treatment as the interpreter itself — especially if that ecosystem is mid-transition to a new major version upstream at the same time as this rebase. See “Known gotchas: python3dist / rawhide-tracked toolchain-adjacent packages” below for why this matters and what already went wrong once.

  • Whether to pause the routine automated dist-git sync bot for the duration of the Step 4 mass rebuild. See “Known gotchas: automation racing a manual mass rebuild” below for the tradeoff.

Step 1 — Buildroot config + core toolchain (separate MRs)

Do not bundle the buildroot config change with the toolchain rebuilds in one MR — split them so the buildroot config lands and validates quickly, independent of the (much slower) toolchain rebuilds.

  1. mock/mock.cfg: update dist, releasever, bootstrap_image, description, and the [fedora]/[fedora-updates] repo baseurls and names to the new release. Merge this on its own or bundled only with fedora-repos (see next point) and binutils if it’s a clean (unmodified) package — do not bundle glibc/gcc/llvm here.
  2. Testing/CI fallback config (rpms/ci/repos/fedora-43.repo → fedora-44.repo, and ci/default-tests/tests-rpm.yml’s repo copy + GPG key import): these reference the buildroot’s Fedora version directly, and there’s exactly one of each — bump them in the same MR as mock.cfg (or immediately alongside it), not later. During the F44 rebase this was deferred and wasn’t caught until php failed its install test days later in Step 4, against a fallback repo that still pointed at the outgoing release. (This is distinct from the per-package test fixture gotcha below, which recurs throughout every step — see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references.”)
  3. fedora-repos: rebase to the new release. If Fedora’s spec uses a conditional ELN-style macro for Release:, evaluate whether to keep it or hardcode a plain Release: N%{?dist} — this is a judgment call each time depending on what’s simplest to maintain; document whichever choice is made in the package’s modification_reason.
  4. binutils (if it carries local CVE backport patches): before assuming the patches still apply, check whether the CVEs they address have already been fixed upstream in the new Fedora version. If so:
    • Pull Fedora’s current spec as the new baseline (don’t try to merge patch-by-patch on top of the old spec).
    • Re-add only the local patches that are not already fixed upstream.
    • For a quick validation build, it’s sufficient to confirm the patch set applies and compilation begins — you don’t need to wait for a full build to complete just to validate the patch set is sane. Kill the build once compilation starts.
  5. glibc: this is the highest-risk package in this phase because of a circular bootstrap dependency (see “Known gotcha: glibc/gcc bootstrap circularity” below). Expect this to require a temporary workaround, tracked as a modified package, to be reverted in Step 2.
  6. gcc: reapply any local modification (e.g. disabling non-essential language frontends) against the new spec. Expect this to be one of the longest-running builds in the whole rebase (multiple hours per architecture) — plan CI capacity accordingly.
  7. llvm: rebase and rebuild; this can run in parallel with gcc and typically finishes faster.

Step 2 — Bootstrap cleanup

Once the new compiler (from Step 1) is published to Pulp:

  1. Rebuild annobin against the new compiler. This must happen before the next step, or reverting the glibc bootstrap workaround will immediately re-trigger the same circular dependency.
  2. Revert the glibc bootstrap workaround from Step 1, now that the new compiler and annobin are both available. Clear the modified status/reason that was recording the workaround.
  3. Rebuild libtool against the new compiler. This is easy to miss (it was missed during the F44 rebase and had to be done out-of-band mid-mass-rebuild) but several packages (apr, audit, authselect, automake, avahi, bind, catatonit, cryptsetup, find, and likely others) will fail to build against the new toolchain without it. Do this here, not in Step 4.

Step 3 — Toolchain-adjacent, release-pinned packages

Rebase the remaining packages that were explicitly tracking the outgoing Fedora branch and don’t carry a permanent reason to stay behind (see “Packages that intentionally stay behind”). For each:

  • Decide whether it should move to the new versioned branch (e.g. f44) or to rawhide — moving to rawhide can simplify build-dependency resolution for clean (unmodified) packages, but ties future updates to rawhide’s churn.
  • For packages with local modifications, check whether the modification is still needed (e.g. a security backport that may have landed upstream by now) before reapplying it blindly.
  • Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before opening this package’s MR (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — fix it in this same MR if present.

Step 4 — Mass rebuild of everything else

Rebuild every remaining package against the new toolchain so that linking, annobin annotations, and compiler hardening are consistent across the whole package set.

  1. Build the exclusion list first: everything already rebuilt in Steps 1–3, plus anything merged in the hours immediately before starting this step (check recent merge history). Use dist_git.py rebuild --all’s --exclude option (HUM-5183):

    ./ci/dist_git.py rebuild --all --exclude pkg1,pkg2,... --reason "<toolchain> toolchain rebuild"
    

    During the F44 rebase this flag didn’t exist yet, so the exclusion list had to be hand-rolled with ./ci/dist_git.py list | grep -v -E '^(pkg1|pkg2|...)$' piped into rebuild — no longer necessary now that --exclude has landed.

  2. Watch for packages whose Release: field doesn’t fit the simple .N bump pattern (e.g. packages using build-time macros for Release:) — these will fail to commit cleanly and need to be handled in a separate pass.

  3. Submit in batches, not one giant MR wave — use ./ci/rebuild_multi_mr.sh --dry-run to preview, then ./ci/rebuild_multi_mr.sh --max-updates=N to submit and auto-merge-on-green in controlled increments. Plan for several days; a run of ~460 packages took about a week of elapsed time across 5 batches during the F44 rebase, gated by CI/Testing Farm throughput more than build time itself.

  4. Check test/rpms/<pkg>.yml for a hardcoded Fedora-version reference before each package’s rebuild MR merges (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — this is what let php’s failure slip through undetected for days during the F44 rebase, so don’t rely on remembering it only once at the end.

  5. Expect a small number of packages to need bespoke fixes that a routine rebuild can’t resolve automatically — license metadata regressions surfaced by newer scanners, compiler/linker regressions in vendored dependencies (e.g. a bindgen version incompatible with a newer clang), confirmed upstream compiler-triggered bugs in JIT/JVM-style software, or stale version-gates in local patches that assumed an older toolchain. Budget time for a handful of these near the end of the mass rebuild.

Known gotchas

glibc/gcc bootstrap circularity

A new Fedora release’s glibc spec may use a feature of the new gcc (e.g. a new compiler flag), while the new gcc may in turn require symbols only present in the new glibc. If Hummingbird’s own Pulp repos still serve the old toolchain at a higher priority than the new Fedora repos during the transition window, this becomes a real chicken-and-egg problem, not just a theoretical one.

The pragmatic fix used for F44 was to drop the new-toolchain-specific flag from glibc temporarily (if it’s a defensive/hardening-only flag whose absence doesn’t break correctness), rebuild glibc and gcc against each other with it disabled, then re-enable it once both are published and annobin/libtool have also been rebuilt against the new compiler (Step 2 above).

Package-specific test fixtures with hardcoded Fedora-version references

Beyond the single global fallback config bumped in Step 1, some packages carry their own test/rpms/<pkg>.yml test suite that independently hardcodes a Fedora repo/GPG-key reference for setting up a test chroot (during F44 this was glib2, hummingbird-release, ca-certificates, crypto-policies, and openssl). Don’t wait for a one-time catch-all audit to find these — fix each one in the same MR as that package’s own rebuild/update, whenever it happens to land during the rebase (Step 1, 3, or 4, whichever touches that specific package — see the reminders in each of those steps above). Relying on an exhaustive upfront sweep of every test/rpms/*.yml is fragile and is exactly what let php’s failure slip through undetected for days during the F44 rebase — its own MR was fine, but a different package’s stale GPG key/repo reference broke shared test infrastructure php depended on. When you open the rebuild/update MR for any package, grep test/rpms/<pkg>.yml for the outgoing release string first.

See HUM-5184 for tracking a structural fix (single source of truth or a CI lint check) so this class of drift is caught automatically instead of relying on either of the above being remembered.

Automation racing a manual mass rebuild

The routine automated dist-git sync bot keeps running on its normal schedule regardless of an in-progress manual mass rebuild. If both touch the same packages in the same window, you’ll get a wave of merge-conflicted bot MRs that need manual cleanup — during F44 this was roughly 70 MRs at once. Decide before starting Step 4 (see “Pre-flight checks” above) whether to pause routine sync automation for the duration of the mass rebuild, or accept the cleanup cost.

External CI capacity constraints

Large rebuild waves can coincide with unrelated capacity crunches on shared infrastructure (e.g. Testing Farm). These are outside Hummingbird’s control but can stall a batch for a day; check external status pages when a batch is unexpectedly slow (for reference, the F44 rebase hit https://status.testing-farm.io/issues/2026-07-14-big-queue-in-red-hat-ranch/).

python3dist / rawhide-tracked toolchain-adjacent packages

If any packages central to a specific language ecosystem’s “what version is default” (e.g. python-rpm-macros) are tracking rawhide rather than a pinned branch, a routine automated sync during this window can silently pull in an experimental version bump (e.g. Fedora bootstrapping the next language version) that corrupts auto-generated package metadata across dozens of downstream packages without any build failing immediately. See HUM-4988 and HUM-4989 for the full incident report and proposed guardrails from the F44 rebase. Decide during “Pre-flight checks” above whether any “default version” declaration packages need the same branch-pinning treatment as the interpreter itself before you start.

Packages that intentionally stay behind

Not every package needs to move to the new branch. Document (in modification_reason and/or the epic tracking the rebase) any package that’s staying pinned to the old branch, and why — otherwise this looks like unfinished work to the next person. Reasons seen so far:

  • FIPS validation: the package carries a modification tied to FIPS certification, and moving to newer upstream content risks invalidating that certification. The package is still rebuilt against the new toolchain — it just doesn’t pick up newer Fedora source content.
  • Package no longer exists upstream: Fedora sometimes drops a package entirely in a new release (e.g. an older LTS language runtime superseded by a newer one). If Hummingbird still needs to carry it, it has no newer branch to rebase to and simply stays on its last available branch, rebuilt against the new toolchain.

Reference: commands used during the F44 rebase

These are shown verbatim from the F44 rebase for concreteness. When reusing them for the next rebase (e.g. F45), remember to substitute the actual target release everywhere F44 appears below — including the exclusion list of already-rebuilt packages, which will differ each time.

# Preview a rebuild plan across all packages
./ci/rebuild_multi_mr.sh --dry-run

# Submit rebuild MRs in controlled batches, auto-merge on green
./ci/rebuild_multi_mr.sh --max-updates=50

# Rebuild everything except a known set of already-handled packages, using the
# --exclude flag (HUM-5183, landed after the F44 rebase — see Step 4 above)
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm,annobin,binutils,fedora-repos,python3.13,python-gitlab --reason "F44 toolchain rebuild"

During the F44 rebase itself, --exclude didn’t exist yet, so the equivalent exclusion list had to be hand-rolled:

./ci/dist_git.py rebuild $(./ci/dist_git.py list | grep -v -E '^(glibc|gcc|llvm|annobin|binutils|fedora-repos|python3.13|python-gitlab)$') --reason "F44 toolchain rebuild"
  • HUM-2018 — F44 buildroot rebase epic (full history)
  • HUM-3369 / HUM-3370 / HUM-4536 / HUM-4537 — Steps 1-4, each with a detailed what/why/fix writeup
  • HUM-4988 / HUM-4989 — python-rpm-macros/python3dist regression incident and guardrails
  • HUM-5182 — bump_release() double-suffix bug
  • HUM-5183 — dist_git.py rebuild --exclude option (implemented)
  • HUM-5184 — generalize the testing-infra Fedora-version drift fix

4 - Lot's of background information

Background and explanation documentation for understanding Project Hummingbird systems and concepts.

4.1 - RPMs repository

Background documentation and contributing guide for Project Hummingbird RPM packages.

Thank you for your interest in contributing! This repository contains RPM packaging, minimal tests, and CI plumbing used by Hummingbird to build, lint, and validate RPMs in containers and Testing Farm.

This guide is adapted from the Hummingbird containers contribution guide and aligned with our workflows and tooling. See the original for broader context: Hummingbird Containers CONTRIBUTING.

Code of Conduct

Be respectful and constructive. By participating, you agree to uphold a professional and inclusive environment.

Repository layout

  • rpms/<package>/ – RPM dist-gits (spec, sources), mostly auto-imported
  • ci/ – helper scripts and default tests
    • build_rpms.sh – build RPMs using Mock in the Konflux-compatible container
    • run_tests_rpm.sh – run rpmlint and install/rebuild tests in a container
    • default-tests/tests-rpm.yml – default tests included for all packages
    • run_tests_rpm.fmf – tmt/Testing Farm entry point
  • konflux-templates/ – Konflux/PAC resources
  • mock/ – mock configuration
  • test/rpms/<package>.yml – package-specific tests

Prerequisites

  • Fedora or RHEL-like environment
  • Podman
  • rpmlint
  • tmt (optional, for local TF-style runs)
  • jq, python3, python3-yaml (used by run_tests_rpm.sh)

Build locally

Build a package’s SRPM and RPMs using the Konflux-aligned environment:

./ci/build_rpms.sh <package_name>
# Results are written to: /tmp/konflux-build-<package_name>-*/results/

Building in Lima VM

When building inside a Lima VM (macOS users), use the --build-dir flag to specify a directory on the VM’s native filesystem. This avoids permission issues with mock’s bootstrap process on macOS mounts:

limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build <package_name>'

The build directory must be on the VM’s native filesystem (not a macOS mount) to ensure Linux file ownership, permissions, and symlinks work correctly with /var/lib/mock.

Interactive repository debugging

To investigate package/dependency/installability issues, you can run an interactive shell in the same environment used by the package builds:

./ci/build_rpms.sh --shell-before setup

This will show you the mock command that would be used to build the setup package, and drop you into a shell in the same environment before executing the build, with dnf available. You can then run the command manually and inspect the environment, repositories, and package metadata.

If you want it to actually build the local package first, so that you can include it in your investigation, use --shell-after instead:

./ci/build_rpms.sh --shell-after setup

Test locally (containerized)

Run the default and package-specific tests against one or more built RPMs:

# Basic usage (binary rpm)
./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm <package_name>

# Include source RPM tests
./ci/run_tests_rpm.sh \
  --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \
  --src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \
  <package_name>

# Test all RPMs in the build output directory after build_rpms.sh
./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds/<package_name>/RPMS/*.rpm) \
  --repo-dir builds/<package_name>/RPMS/ \
  --src-rpm builds/<package_name>/SRPMS/*.src.rpm \
  <package_name>

Notes:

  • Tests run inside a Podman container.
  • The test image defaults to registry.access.redhat.com/hi/core-runtime:latest-builder. You can override via TEST_IMAGE:
TEST_IMAGE=registry.access.redhat.com/hi/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm <package_name>
  • Container execution is performed as root with HOME=/root to avoid XDG state permission issues in dnf5.

Test in tmt/Testing Farm locally

You can drive the same FMF test locally with tmt. The FMF test expects an OCI artifact that contains RPMs, referenced via IMAGE_URL (Testing Farm sets this automatically). For local trials you can point to any compatible OCI artifact or skip the ORAS pull logic and directly call the script.

# If using an OCI artifact with RPMs
IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a

# Or run the script directly with built RPMs (bypassing the FMF wrapper)
./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm <package_name>

If you need longer time in Testing Farm, the FMF test includes duration, which you can adjust in ci/run_tests_rpm.fmf.

Dist-git Imports

The ci/dist_git.py tool imports Fedora/CentOS dist-git packages into the rpms/ directory. We expect most packages to not have (permanent) Hummingbird specific changes, so most of them will keep syncing with upstream dist-gits. In most cases that will be Fedora rawhide, but for some packages we may pick a different upstream, e.g. stable Fedora or even CentOS Stream.

The status of all imports is tracked in imports.json. Active Fedora releases are tracked in upstream-releases.json, which is updated from the Bodhi API.

Note that rawhide is automatically resolved to the highest numbered Fedora release at runtime and is not stored in the JSON file.

See ./ci/dist_git.py --help for all available options. Some examples:

  • Update upstream-releases.json from Bodhi API (should be done periodically, e.g., when new Fedora versions are released):
./ci/dist_git.py update-releases

This fetches the latest Fedora releases from Bodhi. The rawhide branch automatically resolves to the highest numbered Fedora version (e.g., if f43 and f44 are available, rawhide uses f44).

  • Import a new package. This requires specifying the dist-git URL (with fedora/ being a shortcut for the Fedora dist-git URL) and optionally a branch (default: rawhide):
./ci/dist_git.py import fedora/bash

./ci/dist_git.py import --branch f42 fedora/glibc
./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git
  • Update all or a single package:
./ci/dist_git.py update

./ci/dist_git.py update bash

This only imports changes if these were actually built in Koji, to ensure we only import changes which are meant to be released. You can disable this check with --skip-build-check.

  • Re-sync a package to upstream, discarding any local modifications. We use this after Fedora adopted our change, or it is no longer relevant:
./ci/dist_git.py sync bash

All of these commands automatically commit changes with descriptive commit messages including the upstream SHA. To avoid that, you can use the --dry-run option.

  • Enable upstream version tracking for a package (used by check_upstream_versions.py check). These commands modify the metadata file but do not create a git commit:
./ci/dist_git.py set-upstream bash --track
./ci/dist_git.py set-upstream bash --no-track
  • Check upstream version status. The check subcommand checks tracked packages; list shows all packages in a table:
./ci/check_upstream_versions.py check
./ci/check_upstream_versions.py list
./ci/check_upstream_versions.py list --json

Package-specific overrides

Per-package build configuration can be customized in ci/package-overrides.yaml. If a package is not listed, it uses default settings.

Available options:

Option Description Default
timeout_hours Build timeout in hours 4
build_platforms List of MPLs (instance sizes) for multi-platform builds Pipeline defaults

Example configuration:

# Long-running package with custom timeout
setup:
  timeout_hours: 12

# Package requiring larger build instances
llvm:
  timeout_hours: 12
  build_platforms:
    - "linux-d160-c8xlarge/arm64"
    - "linux-d160-c8xlarge/amd64"

All available MPLs can be found in the Konflux multi-platform builds documentation.

After modifying overrides, regenerate the pipeline files:

make generate

This updates .tekton/rpms-on-push.yaml and .tekton/rpms-on-pull-request.yaml with the new configuration.

Branching and pull requests

  • Create feature branches from the default branch.
  • Keep edits focused and small; separate unrelated changes into separate PRs.
  • Include meaningful commit messages (why + what). Reference related issues if applicable.
  • PRs must pass CI (build + tests). Fix lint/test failures or mark known failures properly.

Commit message conventions

  • First line: short imperative summary (≤ 72 chars)
  • Body (optional): context, rationale, and user/ops impact
  • Reference issues using standard notation (e.g. “Fixes: #123”)

Spec guidelines

  • Don’t use %changelog for Hummingbird specific changes. It’s just a point of conflict with Fedora imports, commit messages are good enough.
  • Hummingbird specific changes increase Release: number in steps of 0.1 (e.g. 2 → 2.1 → 2.2). This avoids colliding with Fedora’s Release number namespace and allows us to sync again to a future -3.

Packaging and testing guidelines

  • Spec files should be reproducible and minimal.
  • Prefer pinned container digests for CI images where feasible. If a tag and digest are both present (name:tag@sha256:<digest>), the digest is authoritative for content selection.
  • Default tests in ci/default-tests/tests-rpm.yml must be fast, deterministic, and safe for all packages.
  • Package-specific tests (rpms/<package>/tests-rpm.yml) can override or extend defaults. Keep them bounded in runtime and dependencies.
  • When possible, capture flaky or environmental issues under known_issues in test YAML with clear matching patterns and descriptions.

Style

  • Shell: bash with set -euo pipefail, readable variable names, and early returns where reasonable.
  • YAML: consistent indentation and quoting; prefer explicitness over magic.
  • Comments: add when necessary to explain non-obvious rationale; avoid restating the code.

Security and supply chain

  • Avoid embedding credentials or secrets; use environment variables or CI secret stores.
  • Prefer pulling images by digest for stability and repeatability in CI.
  • Validate inputs and sanitize any paths used by scripts.

Resolving CVEs

When manually fixing a CVE (e.g., adding a patch or performing a manual update), a Fixes: CVE-YYYY-XXXX must be added to the MR description and/or a commit. This will indicate to the build system that a CVE fix has been submitted for review.

Reporting issues

Open an issue describing the problem, reproduction steps, and environment. Attach logs (build/test) when possible. If the failure is intermittent, call that out explicitly.

Licensing

Ensure files include appropriate licenses and that any third-party content is compatible with the project’s license.

Thank you

Your contributions make the Hummingbird RPMs better for everyone. We appreciate your time and feedback!

4.1.1 - RPM Pipeline

How an RPM spec change flows through the build pipeline to the package repository

Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.

Overview

The RPM pipeline consists of five main stages:

  1. Spec Change - A package spec file is modified in rpms/<package>/
  2. MR Build & Validation - Konflux builds RPMs and required tests validate them before merge
  3. Promotion or Push Build - After merge, packages reuse the tested MR image by default; explicit build exceptions build again on main
  4. Release & Signing - Integration Service creates the Release; the managed pipeline checks Conforma and signs the RPMs
  5. Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository

After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.

flowchart TD
    A["Spec Change<br/>rpms/&lt;package&gt;/"] --> B["Merge Request"]
    B --> C["MR CI + Konflux build<br/>required Testing Farm tests"]
    C --> D["Merge to main"]
    D --> E{"Promotion enabled?"}
    E -->|Yes| F["Promotion PipelineRun<br/>reuse tested image digest"]
    E -->|No| G["Konflux push build<br/>mock hermetic build per arch"]
    F --> H["Automatic Release<br/>Conforma, signing"]
    G --> H
    H --> I["Publish signed RPMs to Pulp"]
    I --> J["Container lockfile updates<br/>(see Image Pipeline)"]

Stage 1: Spec Change

An RPM change begins as a commit in rpms/<package>/. The change typically includes a modified .spec file (updated version, new patch, or Release bump) and any associated source files or patches.

How changes originate

Method Description Modification Status
Automated Fedora sync ci/dist_git.py update merges upstream Fedora changes No independent packages
Upstream version update ci/check_upstream_versions.py --update bumps to new upstream release No independent packages
Manual patch backport Developer adds a CVE patch and references it in the .spec Marked modified afterward
No-change rebuild ci/dist_git.py rebuild <package> bumps the Release field Status unchanged
Reverse dependency rebuild ci/dist_git.py rebuild-rev-deps <package> rebuilds all dependents Status unchanged

See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.

Package metadata

Each package has a metadata file at metadata/<package>.json that tracks its relationship to upstream:

{
  "source": "https://src.fedoraproject.org/rpms/curl.git",
  "branch": "rawhide",
  "sha": "abc123...",
  "version": "8.12.1",
  "release": "1",
  "modification_status": "clean",
  "upstream_repo": "https://github.com/curl/curl"
}

Key fields:

Field Purpose
modification_status clean (auto-updates enabled), modified (auto-updates blocked), or independent (no upstream)
modification_reason Explanation of local modifications (when modified)
release Fedora release at last import/update (not the spec Release:), without dist tag; present only while shipping Fedora’s release
upstream_repo Canonical upstream git repository URL
track_upstream Version prefix constraint (e.g., "1.26" for golang1.26)
version_transform Version mapping rule for CVE analysis (e.g., dotnet_sdk_to_runtime)
cve_product CVE vendor/product override (string, or a list for multiple products; e.g. "Oracle Corporation / Oracle Java SE")

See Package Metadata Fields for modification_status and release configuration, and Package Modification Tracking for managing modification status day to day.

Stage 2: Merge Request & Validation

Changes reach main via merge requests. Automated MRs are created using:

  • ci/create_mr.sh - Single MR with configurable options
  • ci/rebuild_multi_mr.sh - One MR per package with auto-merge enabled
  • ci/dist_git_update_multi_mr.sh - Automated Fedora update MRs (scheduled CI job)
  • ci/upstream_update_multi_mr.sh - Automated upstream version update MRs (scheduled CI job)

The scheduled upstream-version update job creates normal MRs for successful package updates. If a package update fails after a new version has been identified, it creates (or reuses) a draft diagnostic MR instead. The diagnostic MR includes the original error but no partial package changes, and must be resolved manually before a package update can be proposed.

Both scheduled update jobs send a Slack completion message when SLACK_WEBHOOK_URL is configured: success sends a plain success message, including when upstream warnings were logged, while failure includes actionable package failures and links to MRs needing attention. Routine upstream warnings remain in the job log and summary, not in Slack.

These completion messages also act as a heartbeat. The notification runs in the job’s after_script, so a job that never starts, or a runner failure before after_script can run, may send no message. A missing expected message is a signal to check the schedule and runner availability; it is not an automatic missed-run alert.

CI validation

GitLab CI runs validation jobs on every merge request:

  • check - Linting, type checking, and spec validation
  • check_nevr_conflicts - Predicts the NEVR(A) each changed package’s spec would build (using rpmspec with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale main, e.g. two rebuild MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch rpms/**/*, and within that, only checks packages whose rpms/<package>/ directory itself changed – matching Konflux’s own build trigger – not packages whose only change is under metadata/, since a metadata-only change (e.g. ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally with make check-nevr-conflicts ARGS='<package>', or against real built RPMs with make check-nevr-conflicts ARGS='--rpms-dir builds/<package>' after ci/build_rpms.sh. See ci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages assigned a private_product in metadata/<package>.json through hum.meta are noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode some OK results can be for a package name a real build never actually produces (e.g. a base package with no top-level %files section, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount a CONFLICT found elsewhere in the same run.
  • tree_status - Repository consistency checks (metadata integrity, spec file presence)
  • Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)

Auto-approval for chore MRs

Automated update MRs (chore/* branches) follow an accelerated path:

  1. MR is created with auto-merge enabled
  2. Konflux builds the package and posts commit statuses
  3. After a delay, the chore_mr_approval CI job auto-approves the MR
  4. GitLab merges the MR once the pipeline succeeds

Stage 3: Promotion or Push Build

Konflux builds RPMs for merge requests. After merge, packages use MR Artifact Promotion to release that tested image by default. Packages explicitly configured with push_pipeline: build build again on main.

Build triggers

Each package has a Tekton MR build PipelineRun definition generated from its directory in rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered .tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers it for MR changes. A push to main triggers one PipelineRun per affected package: a promotion run from .tekton/rpms-on-promotion.yaml by default, or a build from .tekton/rpms-on-push.yaml for an explicit push_pipeline: build exception.

The setup package acts as a canary: changes to ci/ or mock/ directories also trigger a setup build on pull requests, validating build infrastructure changes before they reach main.

Build process

The build pipeline (build-rpm-package) executes for each target architecture:

  1. Checks out the repository at the MR head or push commit
  2. Calculates build dependencies
  3. Runs mock in hermetic mode (network-isolated with pre-fetched dependencies)
  4. Produces RPM artifacts and build logs
  5. Generates an SBOM (Software Bill of Materials)
  6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry

Build architectures

Packages are built for the architectures specified in their PipelineRun definition. Packages build for x86_64, aarch64, and s390x by default. Per-package overrides can select other architectures or build platforms.

Build output

Build artifacts are stored in the Konflux OCI registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main

Konflux resources

Build infrastructure is defined across two repositories:

  • rpms repo (konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy, IntegrationTestScenario
  • infrastructure repo (kubernetes/rpms-main/) - Application, ReleasePlan, ServiceAccount, and Secret resources

See Konflux Resource Deployment for details on how these resources are managed and deployed.

Testing

RPM packages are validated through integration tests that run on Testing Farm infrastructure.

Tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test triggering

Tests are only triggered on merge requests, not on main branch builds. For a package at rpms/<package>/, tests are triggered for all changes below that directory.

Integration Test Scenario

Tests are triggered via an IntegrationTestScenario resource generated by this (rpms) repository:

Configuration: konflux-templates/scenarios.yml (application × template matrix) and konflux-templates/scenarios/<type>.yml.j2 (full CR content per scenario type); both consumed by the generator for CR rendering and manifest emission
Rendered output: konflux-templates/rendered.yml
Deploy path: via the *-releases infrastructure sync on merge to main (not the components sync — ITS is static repo-level config not intended to change in MRs; concurrent MR pipelines would otherwise reset the live CR). Renovate IMAGE_TAG bumps auto-deploy on merge via the same path.

The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:

The generator emits two scenario types per application (instantiated from templates in konflux-templates/scenarios/):

  • testing-farm (non-optional, pull_request context) — runs on every pre-merge component snapshot. Parameters:

    • COMPOSE: RHEL-9-Nightly - Test environment OS/compose
    • PIPELINE_MODE: rpm - Configures RPM testing mode (vs. container)
    • ARCH: x86_64,aarch64 - Architectures to test (creates separate test runs per arch)
    • PASS_SNAPSHOT_TO_TF: false - Don’t pass full snapshot JSON (package info comes via IMAGE_NAME/IMAGE_URL instead)
    • IMAGE_TAG: - Version of the tmt-via-testing-farm pipeline bundle
    • GIT_URL: - Allowlisted upstream repo URL (public rpms-main application only; omitted for private applications)
  • testing-farm-upstream (optional, component_attr-main + component_setup-main canary contexts) — runs only on canary component snapshots. Additional parameter:

    • TMT_PATH_FROM_CONTEXT: "true" - Derives the tmt test path from the component context

Key characteristics:

  • Single scenario per application - Not per-package; runs for every package build in that application
  • Context-based triggering - testing-farm runs on pull_request snapshots only; testing-farm-upstream runs on canary component snapshots and is optional (not listed in the trigger manifest)
  • Package identification - Konflux automatically provides IMAGE_NAME (e.g., openssl-main) and IMAGE_URL (OCI image with built RPMs) as environment variables

FMF test plan

The root folder of the rpms repository is marked with a .fmf directory to identify it as an FMF metadata tree for tmt. The test plan in ci/run_tests_rpm.fmf sets up the test instance and runs default and package-specific tests via ci/run_tests_rpm.sh.

Test environment

These environment variables are available to the tmt test execution:

Variable Description
IMAGE_NAME Component name (e.g., openssl-main)
IMAGE_URL OCI image URL with built RPMs (e.g., quay.io/.../openssl-main@sha...)
SNAPSHOT Snapshot metadata (if PASS_SNAPSHOT_TO_TF is true)
COMPOSE OS/compose being tested (RHEL-9-Nightly)
ARCH Architectures being tested (x86_64,aarch64)
TMT_VERSION tmt version running the tests

Stage 4: Release and Signing

Integration Service creates an automatic Release for a passing push Component Snapshot. For an MR artifact promotion, the Snapshot reuses the tested digest and does not run the MR tests again. Release Service validates Conforma and signs the RPMs before publishing:

  • Method: Kerberos-based signing via konflux-release-signing-prod@IPA.REDHAT.COM
  • Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
  • Signing image: quay.io/konflux-ci/signing:latest (verify pinned digest in ReleasePlanAdmission)

Stage 5: Publishing to Pulp

After signing, RPMs are uploaded to the Hummingbird Pulp repository.

Release pipeline

The push-rpms-to-pulp release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (releng/hummingbird-rpms-tech-preview-staging.yaml):

mapping:
  rpm-repositories:
    - name: x86_64
      repository_id: public-hummingbird-x86_64-rpms
    - name: aarch64
      repository_id: public-hummingbird-aarch64-rpms
    - name: src
      repository_id: public-hummingbird-source-rpms
    # See ReleasePlanAdmission for the full list of published architectures

Pulp repository

Published RPMs are available in the Pulp content index:

Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.

What Happens Next

Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.

4.1.2 - Konflux Resource Deployment

How Konflux resources are defined and deployed across repositories

Konflux resources for RPM packages are split between two repositories:

  • rpms: Per-package resources generated from package directories
  • infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Resource Type Source RPMs MR RPMs Push Infra MR Infra Push
Component rpms manual manual - -
ImageRepository rpms manual manual - -
ReleasePlanAdmission rpms - automatic manual automatic
EnterpriseContractPolicy rpms - automatic manual automatic
IntegrationTestScenario rpms - automatic manual automatic
Application infrastructure - - manual automatic
ReleasePlan infrastructure - - manual automatic
ServiceAccount, Secret infrastructure - - manual automatic

Legend:

  • RPMs MR: Downstream pipeline triggered from rpms MR (deploys from MR commit)
  • RPMs Push: Downstream pipeline triggered from rpms push to main
  • Infra MR: Infrastructure MR pipeline (manual trigger)
  • Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

RPMs Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

  1. Ownership: Components are managed by the rpms repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle.
  2. Dynamic generation: Generated from rpms/*/ directories via ci/generate_konflux_resources.sh, depending on package presence and configuration.
  3. Timing: Must be deployed early during MR review so Konflux can build and test new packages.

ReleasePlanAdmission:

  1. Static configuration: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern.
  2. Main branch only: Deployed only on push to main to maintain deployment consistency.

EnterpriseContractPolicy:

  1. Consistency: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern.

RPMs Repo — IntegrationTestScenario

IntegrationTestScenario resources are defined via two config files:

  • konflux-templates/scenarios.yml — the application × template matrix (which app gets which scenario type, with which triggers and optional per-instance vars)
  • konflux-templates/scenarios/<type>.yml.j2 — full CR content per scenario type (params, resolverRef, Renovate markers, timeout annotations)

The generator renders these into konflux-templates/rendered.yml alongside Components and ImageRepositories. The same matrix drives the trigger-manifest integration_test_scenarios section, so the manifest cannot drift from deployed ITS.

Deploy path: ITS is deployed by the *-releases downstream pipeline (not *-components). The releases sync fetches rendered.yml from the main branch and runs on every merge to main, so ITS changes deploy automatically after merge — including Renovate bundle-digest and IMAGE_TAG bumps. The components sync is deliberately excluded: it deploys from PARENT_COMMIT_SHA (MR heads), and ITS is static repo-level config that is not supposed to change in an MR — concurrent or stale MR pipelines would otherwise reset the live CR to older versions. Pre-merge ITS deployment is intentionally unavailable by design.

No-prune behaviour: The deploy uses server-side apply without pruning — resources removed or renamed in rendered.yml are not deleted from the cluster. A renamed or removed ITS remains live and continues producing test PipelineRuns until deleted manually:

kubectl delete integrationtestscenario <name> -n hummingbird-tenant

See the infrastructure repo’s kubernetes-rendering.md “No pruning” section for details.

The integration_test_scenarios section of .tekton/trigger-manifest-pull-request.json is derived from the same scenarios.yml matrix, ensuring the manifest never drifts from deployed scenarios. See the trigger manifests documentation for the full manifest schema.

Infrastructure Repo

Application and ReleasePlan are defined in kubernetes/rpms-main/:

  1. Independent iteration: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo.
  2. Static configuration: These resources don’t depend on per-package data.

ServiceAccount and Secret for releases are defined in kubernetes/setup-konflux/:

  1. Security: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo.
  2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo.

The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp.

4.1.3 - Source Pipeline Tool

Source Pipeline Tool

Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.

Problem Statement

RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:

  1. Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.

  2. Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.

  3. Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.

  4. Unverified patches. Patches (.patch files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.

Tool Overview

A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.

                          ┌──────────────────────────────┐
                          │     source-pipeline tool     │
                          │      (container image)       │
                          │                              │
  ┌───────────┐           │  ┌───────┐    ┌───────────┐  │    ┌───────────────┐
  │  package  │           │  │ fetch │───▸│ transform │  │    │   tarballs    │
  │   dir     │──────────▸│  └───────┘    └─────┬─────┘  │───▸│   sources     │
  │ (spec,    │           │                     │        │    │   report      │
  │  patches, │           │  ┌───────────┐      │        │    └───────┬───────┘
  │  config)  │           │  │  verify   │◂─────┘        │            │
  │           │           │  └─────┬─────┘               │            ▼
  └───────────┘           │        │                     │    ┌───────────────┐
                          │  ┌─────▼─────┐               │    │   lookaside   │
  ┌───────────┐           │  │  enforce  │               │    │    cache      │
  │ pipeline  │           │  │  policy   │               │    └───────────────┘
  │   yaml    │──────────▸│  └─────┬─────┘               │
  │           │           │        │                     │
  └───────────┘           │  ┌─────▼─────┐               │
                          │  │   emit    │               │
                          │  └───────────┘               │
                          └──────────────────────────────┘

Design principles

  • Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
  • Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
  • Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
  • Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.

Container interface

podman run --rm \
  -v ./<package-dir>:/package:ro \
  -v ./pipeline.yaml:/pipeline.yaml:ro \
  -v ./gpg-keys:/gpg-keys:ro \
  -v ./output:/output \
  source-pipeline:latest \
  --version <new-version> \
  [--old-version <old-version>] \
  [--verify-patches /patches-from-upstream] \
  [--dry-run]

Inputs (mounted read-only):

  • /package — the package directory (spec file, patches, existing sources file)
  • /pipeline.yaml — the declarative pipeline definition
  • /gpg-keys — centralized GPG keyring directory

Outputs (written to /output):

  • Source tarballs (ready for lookaside upload)
  • sources — updated sources manifest with checksums
  • report.json — verification and policy results (pass/fail per check, patch classifications)

Exit codes:

  • 0 — success, all checks passed
  • 1 — error (download failure, tool error)
  • 2 — policy violation (verification or constraint failure)

--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to /output. Exit codes and report.json behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing.

On any non-zero exit, report.json is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no package update commit is created, the package stays at its current version, and the automation continues to the next package. In the scheduled upstream-version-update job, the wrapper also creates (or reuses) a draft diagnostic MR for a per-package update failure. That MR contains the full error and no partial package changes, so a maintainer can reproduce and resolve the problem safely. Transient failures (exit 1) may self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention. When a later scheduled run successfully processes the same package, it closes and removes the still-empty diagnostic MR branch. A diagnostic MR with maintainer changes is left open for manual handling.

Pipeline Stages

1. Fetch

Downloads source artifacts directly from upstream.

  • Parses Source: URLs from the spec file (resolving RPM macros like %{version}, %{name}, %{url})
  • Downloads each source from the upstream URL
  • Supports fetching from git repositories at a tag, branch, or commit
  • For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)

2. Transform

Applies per-package source modifications.

  • Vendor archive generation — runs ecosystem-specific tooling (e.g., go_vendor_archive for Go, npm pack for Node.js, cargo vendor for Rust)
  • Vendor dependency pinning — modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix.
  • Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
  • UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
  • Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
  • Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a toolchain: section. The container ships defaults; per-package overrides ensure that build-ui, vendor, and run: steps use the correct toolchain without requiring separate container images per package

Known sharp edge: patch-list duplication

Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill point back here instead of re-telling the story.

Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ... followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain their own copy of “which patches touch this source tree” rather than reading it from the spec’s PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see Rebuilding Packages) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform’s install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation step, and the mismatch didn’t surface until the next automated version bump re-ran the transform from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact.

go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up requiring different versions of the same package — go build -mod=vendor rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s %check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this drift, same caveat as above.

3. Verify

Validates integrity and authenticity of fetched sources.

  • GPG signature verification — downloads signature files (.asc, .sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation.
  • Checksum verification — compares against published checksums where available
  • Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
  • Re-publication detection — if a previously committed checksum exists in the sources file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an accepted-checksums entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.

4. Enforce policy

Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect) and catches violations in packages that don’t use vendor-pin.

  • Vendor dependency constraints — ensures vendored dependencies meet version requirements (e.g., sanitize-html >= 2.17.5 for a CVE fix)
  • Ecosystem-specific checks — Go module verification, npm audit, cargo audit
  • License compliance — flags vendored dependencies with incompatible licenses
  • Fail-closed: any policy violation exits with code 2

5. Emit

Produces final artifacts.

  • Writes tarballs to the output directory
  • Generates sources file in dist-git format (SHA512 (filename) = hash)
  • Writes report.json with verification results, policy check results, and patch classification

Pipeline Schema

Per-package pipeline definitions are YAML files that declare the full source generation workflow.

Packages without a pipeline definition use a built-in default that fetches sources from the spec’s Source: URLs — no transforms, no verification, no policy.

Schema definition

# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
  # Macro substitutions applied before Source URL resolution
  macros:
    - name: go_patch                     # %global go_patch <new-value>
      value: "${VERSION_PATCH}"          # extracted from VERSION (e.g., 1.25.3 → 3)

    - name: k8s_ver                      # %global k8s_ver <new-value>
      value: "${VERSION}"

  # Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
  reset-release: true

# Source fetching
fetch:
  sources:
    # Fetch Source0 from the URL declared in the spec
    - spec-source: 0

    # Fetch Source1 (e.g., a GPG signature file)
    - spec-source: 1

    # Or fetch from an explicit URL (for sources not in the spec)
    - url: "https://example.com/extra-source-${VERSION}.tar.gz"

    # Or fetch from a git repository at a tag/commit
    - git:
        repo: "https://github.com/example/project"
        ref: "v${VERSION}"              # tag, branch, or commit hash
        include-history: false           # include .git dir in tarball (default: false)

  # Vendor archive generation (optional)
  vendor:
    ecosystem: go                        # go | npm | cargo | composer
    config: go-vendor-tools.toml         # ecosystem-specific config file in the package dir
    source-dir: .                        # directory to vendor from (default: extracted source root)
    submodules:                          # (optional) vendor multiple Go submodules independently
      - server
      - etcdctl
      - etcdutl

# Source transforms (optional, ordered)
transform:
  # Built-in transform types
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
        - "deps/ngtcp2/ngtcp2/crypto/"
      output: "node-v${VERSION}-stripped.tar.gz"

  # Pin vendored dependency versions before vendor archive generation.
  # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
  # bump specific dependencies, then re-resolves the dependency graph.
  # Runs before the vendor stage so the pinned versions are included in
  # the vendor archive. Re-applied on every source generation, so upstream
  # updates cannot silently revert a security fix.
  #
  # Version constraints are always minimum versions — the version field
  # means "at least this version." The tool translates to ecosystem-native
  # operations:
  #   Go:    go get <package>@v<version> (minimum version selection)
  #   npm:   npm install <package>@">= <version>"
  #   Cargo: set dependency requirement to ">= <version>" in Cargo.toml
  - vendor-pin:
      - package: golang.org/x/crypto
        ecosystem: go
        version: "0.31.0"
        reason: "CVE-2024-45337"

      - package: sanitize-html
        ecosystem: npm
        version: "2.17.5"
        reason: "CVE-2024-XXXXX"

      - package: tokio
        ecosystem: cargo
        version: "1.38.1"
        reason: "CVE-2024-YYYYY"

  # Custom command (escape hatch)
  - run: "./packaging/make-tarball.sh ${VERSION}"
    outputs:
      - "package-${VERSION}-stripped.tar.gz"

  # UI asset build
  - build-ui:
      ecosystem: npm                     # npm | yarn
      source-dir: "web/ui"
      output: "${PACKAGE}-${VERSION}-ui.tar.gz"

# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
  node: "20"                             # Node.js version for build-ui, npm vendor, and run: steps
  go: "1.23"                             # Go version for go vendor and run: steps
  rust: "1.80"                           # Rust version for cargo vendor and run: steps
  python: "3.12"                         # Python version for run: steps

# Integrity verification (optional)
verify:
  gpg:
    signature-source: 1                  # spec Source index containing the .asc/.sig
    keyring: "curl.gpg"                  # key name in the GPG keys directory
  checksums:
    url: "https://example.com/SHA256SUMS"
    algorithm: sha256

  # Override for upstream re-publications. Required when upstream re-publishes
  # a release artifact with different content under the same version. The
  # pipeline refuses to accept changed content automatically — a human must
  # add the new checksum here after investigation.
  accepted-checksums:
    - file: "example-1.2.3.tar.gz"
      sha512: "abc123..."
      reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"

# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
  vendor-constraints:
    - package: sanitize-html
      ecosystem: npm
      version: "2.17.5"
      reason: "CVE-2024-XXXXX"

    - package: golang.org/x/crypto
      ecosystem: go
      version: "0.31.0"
      reason: "CVE-2024-45337"

# Patch verification (optional)
patches:
  verify: true                           # diff-detect new/changed patches
  classify: true                         # attempt to classify by origin
  upstream-repo: "https://github.com/curl/curl"
  fail-on-unverified: false              # warn-only by default

  # Patch lifecycle rules (optional)
  # Declares version-scoped applicability for patches. The pipeline tool reads
  # these rules from patch headers (preferred) or from this YAML, and enforces
  # them during updates: patches outside their valid range are flagged or dropped.
  lifecycle:
    - file: "fix-memory-leak.patch"
      applies-to: "< 1.5.0"             # drop this patch at version 1.5.0+
      reason: "Fixed upstream in 1.5.0 (commit abc123)"
      action: drop                       # drop | warn (default: warn)

    - file: "cve-2024-45337.patch"
      applies-to: "< 0.31.0"
      reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
      action: drop

    - file: "distro-branding.patch"
      applies-to: "*"                    # carry forward unconditionally
      reason: "Distribution-specific branding, always required"

# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
  # Extract bundled dependency versions and splice into spec between markers
  - bundled-provides:
      modules-txt: "vendor/modules.txt"  # Go modules.txt path inside extracted source
      start-marker: "# --- bundled-deps.sh ---"
      end-marker: "# --- end bundled-deps.sh ---"

  # Run a custom command (escape hatch for complex metadata extraction)
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"

Variable substitution

The following variables are available in all string values:

Variable Value
${VERSION} New upstream version (e.g., 1.25.3)
${VERSION_MAJOR} Major version component (e.g., 1)
${VERSION_MINOR} Minor version component (e.g., 25)
${VERSION_PATCH} Patch version component (e.g., 3)
${OLD_VERSION} Previous version
${PACKAGE} Package name (directory name)
${SPEC_FILE} Path to the spec file

Version constraint semantics

All version: fields in vendor-pin and policy.vendor-constraints use minimum-version semantics: the value means “at least this version.” The tool translates this into ecosystem-native operations:

  • Go: go get <package>@v<version> (minimum version selection via MVS)
  • npm: npm install <package>@">= <version>"
  • Cargo: sets the dependency requirement to >= <version> in Cargo.toml

Default behavior (no pipeline YAML)

When no pipeline YAML is provided, the tool applies a built-in default:

  1. Parse all Source: URLs from the spec
  2. Download each from the upstream URL
  3. Generate sources file with SHA512 checksums
  4. No transforms, no verification, no policy enforcement

This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.

Patch Verification

When packages are updated from an upstream distribution, patch files (.patch) may arrive without inspection. The pipeline tool can optionally verify and manage patches.

Verification approach

The pipeline tool can optionally verify patches when invoked with --verify-patches:

  1. Diff detection. Compare the set of .patch files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches.

  2. Header parsing. Extract metadata from patch headers:

    • From: — author identity
    • Subject: — description
    • Commit hash references (e.g., From <hash>, cherry picked from commit <hash>)
    • Bug: / CVE: references
  3. Classification. Attempt to classify each patch:

    • Upstream backport — references a commit hash that exists in the upstream repo
    • CVE fix — references a CVE identifier
    • Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
    • Distribution-specific — modifies paths, branding, or distribution-specific integration
    • Unclassified — cannot be automatically categorized
  4. Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via git ls-remote or the upstream API).

  5. Reporting. Output classification and verification results in report.json. Optionally fail on unverified patches (controlled by patches.fail-on-unverified in the pipeline YAML).

Patch lifecycle enforcement

Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.

The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML’s patches.lifecycle section.

Header-based declaration (preferred). Patch authors add structured keywords to the patch header:

From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---

YAML-based declaration (fallback). For patches from upstream distributions that cannot have headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch.

Enforcement behavior:

  1. During an update to version ${VERSION}, the tool evaluates each patch’s applies-to range against the new version.
  2. If a patch is outside its valid range:
    • action: drop — the patch file is deleted from the package directory and the corresponding Patch: declaration and %patch / %autopatch application directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in report.json.
    • action: warn (default) — the patch is flagged in report.json but not removed. The update proceeds.
  3. If a patch has applies-to: *, it is always carried forward.
  4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to applies-to: *, action: warn).

Carry-forward enforcement. The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked applies-to: * is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error.

Example report output

{
  "patches": {
    "new": [
      {
        "file": "fix-memory-leak.patch",
        "classification": "upstream-backport",
        "upstream_commit": "abc123def456",
        "verified": true
      }
    ],
    "removed": ["old-workaround.patch"],
    "unchanged": ["distro-paths.patch"],
    "lifecycle": [
      {
        "file": "cve-2024-45337.patch",
        "applies_to": "< 0.31.0",
        "current_version": "0.31.0",
        "action": "drop",
        "result": "removed — version 0.31.0 is outside applies-to range"
      },
      {
        "file": "cve-2024-99999.patch",
        "applies_to": "< 2.0.0",
        "current_version": "2.0.0",
        "action": "drop",
        "result": "flagged — patch is applied inside a conditional %if block; manual removal required"
      },
      {
        "file": "distro-branding.patch",
        "applies_to": "*",
        "action": "carry-forward",
        "result": "present"
      }
    ]
  }
}

Examples

Trivial package (curl)

No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you want GPG verification:

fetch:
  sources:
    - spec-source: 0
    - spec-source: 1

verify:
  gpg:
    signature-source: 1
    keyring: "curl.gpg"

Transformed package (Node.js)

Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:

spec-update:
  macros:
    - name: nodejs_define_version node
      value: "${VERSION}"
  reset-release: true

toolchain:
  node: "22"

fetch:
  sources:
    - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"

transform:
  - strip-tarball:
      source: "node-v${VERSION}.tar.gz"
      remove:
        - "deps/openssl/"
      output: "node-v${VERSION}-stripped.tar.gz"

verify:
  checksums:
    url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
    algorithm: sha256

post:
  - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"

Simple Go vendor (caddy)

A pattern shared by nats-server, oauth2-proxy, and similar Go projects:

fetch:
  sources:
    - git:
        repo: "https://github.com/caddyserver/caddy"
        ref: "v${VERSION}"

  vendor:
    ecosystem: go

Multi-submodule Go vendor (etcd)

fetch:
  sources:
    - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    submodules:
      - server
      - etcdctl
      - etcdutl

Code generation + Go vendor (opentelemetry-collector-contrib)

Built-in primitives handle fetch and vendor; the code generation step requires run::

fetch:
  sources:
    - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"

  vendor:
    ecosystem: go
    config: go-vendor-tools.toml
    source-dir: _build

transform:
  - run: |
      curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
      curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
      sha256sum -c --ignore-missing ocb_checksums.txt
      chmod +x ocb
      tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
      cd "opentelemetry-collector-releases-${VERSION}"
      if [ -f scripts/prepare-obi.sh ]; then
        bash scripts/prepare-obi.sh otelcol-contrib
      fi
      ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
    outputs:
      - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"

Git snapshot (libXtst)

A pattern shared by libX11, libXext, libXi, libXrender:

fetch:
  sources:
    - git:
        repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
        ref: "libXtst-${VERSION}"
        include-history: true

Built-in Primitives Coverage

An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:

Fully declarative (no run: needed) — ~30 script files (15 unique patterns):

Built-in primitive Packages covered
vendor: {ecosystem: go} caddy, nats-server×2, oauth2-proxy
vendor: {ecosystem: go, submodules: [...]} etcd (3 submodules)
strip-tarball: {remove: [...]} nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs×2
fetch: {git: {repo, ref}} oniguruma, php-jsqueeze, libX11/Xext/Xi/Xrender/Xtst
fetch + verify: {checksums} nodejs download+verify
vendor: {ecosystem: composer} composer

Partially declarative (generic primitives + 1–2 run: steps):

Package Generic part Custom run: step
otel-collector, otel-collector-contrib vendor: {ecosystem: go} OCB binary code generation
java-openjdk×2 strip-tarball, fetch: {git} ./configure + make store-source-revision (needs boot JDK)
selinux-policy fetch: {git} (×3 repos) Multi-repo selective archiving

Genuinely custom (run: required) — 7 packages:

Package Why it can’t be declarative
nss-fips Container-based RPM download with subscription-manager credentials + QEMU cross-arch
openssl-fips-provider Container-based SRPM download + nested RPM extraction
ca-certificates Interactive multi-source crypto trust data merging
gcc GCC-specific changelog/PR extraction from git history
gdb Interactive patch management tooling (stgit)
python3.14 Koji task-specific JIT stencil artifact extraction
erlang27 Complex patch reformatting + spec rewriting

The run: escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract.

Open Questions

  1. Implementation approach. The pipeline YAML schema (ordered transform stages, run: escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:
    • Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
    • Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
    • Shared shell function library — the majority of scripts decompose into 3–4 operations (vendor_go, strip_tarball, fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands.
    • Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no run: blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.

Hummingbird Integration

This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.

Current Architecture

Two update systems

System Packages Source of tarballs Hook system
dist_git.py update 450 (clean + modified) Fedora lookaside cache (via sources file copied from Fedora dist-git) None
check_upstream_versions.py 22 (independent) Upstream URLs (via download_sources hooks or default spec URL download) Yes: update_spec, download_sources, post_update

dist_git.py update flow:

  1. Clones Fedora dist-git for the package
  2. Checks version, Koji build status, pre-release filtering
  3. Copies entire Fedora dist-git checkout into rpms/<package>/ via shutil.copytree — this includes the spec, patches, .gitignore, and sources file
  4. For modified packages, performs a 3-way git merge to preserve local changes
  5. Commits everything (spec, patches, sources file, metadata)

The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash). Actual tarballs are excluded by .gitignore and stored in a lookaside cache.

check_upstream_versions.py flow:

  1. Queries release-monitoring.org for new upstream versions
  2. Runs three hook phases per package:
    • update_spec — updates the spec’s Version/Release (default: specfile.update_version())
    • download_sources — downloads source tarballs (default: fetches from spec Source URLs)
    • post_update — additional steps (default: no-op)
  3. Uploads downloaded tarballs to Hummingbird’s lookaside cache
  4. Updates the sources file with new checksums
  5. Commits

Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.

Three lookaside cache backends

Configured in mock/dist-git-client.ini:

Backend URL pattern Used by
Fedora src.fedoraproject.org/repo/pkgs/rpms/{name}/{filename}/{hashtype}/{hash}/{filename} ~410 packages (default)
CentOS Stream sources.stream.centos.org/sources/rpms/{name}/... 1 package (rust-rpm-sequoia)
Hummingbird d1766whheab9hg.cloudfront.net/rpms/{name}/... (S3: arr-hummingbird-prod-dist-git-cache) ~40 packages (independent + forked)

The forked_from field in ci/package-overrides.yaml determines which cache a package uses at build time. Packages without forked_from default to Fedora’s cache.

How Konflux builds consume sources

Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):

  • monorepo-subdir: rpms/<package> — locates the spec, patches, and sources file
  • package-name: <name> — upstream name for lookaside URL construction
  • forked-from: <url> — (optional) selects the lookaside backend
  • dist-git-client-configdir: mock/ — (optional) points to the custom dist-git-client.ini

The pipeline uses dist-git-client to read the sources file and download tarballs from the appropriate cache, then runs mock to build RPMs.

Existing source generation patterns

Some independent packages already have tarball generation scripts:

  • rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh — downloads upstream release, runs OCB to generate source code, vendors Go dependencies
  • rpms/caddy/create-vendor-tarball.sh — downloads upstream tarball, generates Go vendor archive
  • rpms/nodejs25/packaging/make-nodejs-tarball.sh — downloads upstream, strips bundled content

These scripts are wired into the update path via download_sources hooks in metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache.

Integration with dist_git.py update

The current flow copies everything from Fedora dist-git (including the sources file) via shutil.copytree. The pipeline tool inserts after this step:

Current:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Commit

Proposed:
  1. Clone Fedora dist-git
  2. copytree into rpms/<package>/        ← sources file points to Fedora lookaside
  3. Run source-pipeline tool             ← fetches from upstream, replaces sources file
  4. Upload tarballs to Hummingbird lookaside
  5. Commit                               ← sources file now points to Hummingbird lookaside

Implementation:

  • After the shutil.copytree (or merge for modified packages), check if metadata/<package>.source-pipeline.yaml exists
  • If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the sources file
  • If it does not exist: run the built-in default (fetch from spec URLs, upload, replace sources)
  • A --skip-pipeline flag allows falling back to the current behavior during migration
  • Update ci/package-overrides.yaml to set forked_from to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)

Integration with check_upstream_versions.py

The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating per-package update behavior into a single *.source-pipeline.yaml file:

Current:
  1. update_spec hook/default             ← *.update-hooks.yaml
  2. download_sources hook/default        ← *.update-hooks.yaml
  3. post_update hook/default             ← *.update-hooks.yaml

Proposed:
  1. spec-update (pipeline YAML)          ← replaces update_spec hooks
  2. fetch + transform (pipeline YAML)    ← replaces download_sources hooks
  3. post (pipeline YAML)                 ← replaces post_update hooks

This reduces per-package metadata from three files (metadata/<package>.json, *.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for identity/tracking, *.source-pipeline.yaml for all update behavior).

Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed.

Integration with Konflux

Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via dist-git-client as it does today.

Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.

Hummingbird-specific configuration

Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.

GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.

The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.

Local development

Developers can run the tool directly:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  -v /tmp/output:/output \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0

# Outputs in /tmp/output/:
#   curl-8.21.0.tar.xz
#   curl-8.21.0.tar.xz.asc
#   sources
#   report.json

To preview without producing artifacts:

podman run --rm \
  -v ./rpms/curl:/package:ro \
  -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
  -v ./metadata/gpg-keys:/gpg-keys:ro \
  quay.io/hummingbird-ci/source-pipeline:latest \
  --version 8.21.0 \
  --dry-run

# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.

Migration Path

Phase 1: Independent Go packages

Migrate the 22 independent packages that already fetch from upstream. Convert their existing create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions. Validates the tool against known-good packages.

Scope: 7 packages with existing scripts + 15 without.

Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code generation, but their vendor step is generic.

Phase 2: Trivial clean packages

Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.

Prerequisites:

  • Audit to classify packages as trivial vs. transformed (HUM-4620)
  • dist_git.py update integration complete (HUM-4621)

Per-package steps:

  1. Run the pipeline tool, compare output against existing Fedora-sourced tarball
  2. If identical: add forked_from: hummingbird to package-overrides.yaml
  3. If different: flag for Phase 3

Phase 3: Transformed packages

Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:

  • Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): strip-tarball
  • Git snapshots (libX11 family×5, oniguruma, php-jsqueeze): fetch: {git: ...}
  • Composer vendor (composer): vendor: {ecosystem: composer}

Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require run: blocks for genuinely custom logic.

Scope: determined by Phase 2 audit.

Phase 4: Policy enforcement

Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates.

Phase 5: Patch verification

Enable patches.verify and patches.classify for packages. Start with warn-only (fail-on-unverified: false), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages.

Operational Health

Escape hatch ratio

The primary health metric for the source pipeline is the escape hatch ratio: the percentage of pipeline YAML definitions that contain run: blocks (in transform or post sections) relative to the total number of pipeline YAML definitions.

A run: block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the same way: a package that cannot be expressed declaratively gets a run: block. This makes the ratio a single number that aggregates all pressures on the declarative model.

Interpretation:

Ratio Signal
< 15% Healthy. The primitive set covers real-world needs.
15-25% Watch. Look for repeated patterns in run: blocks that should become primitives.
> 25% Action needed. The YAML is becoming a workflow engine. Either expand the primitive set or reconsider the abstraction.

When the ratio climbs, examine the run: blocks for repeated patterns. If three or more packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad – it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set.

CI enforcement

The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files, calculates the escape hatch ratio, and reports the result. It runs as part of make check.

  • During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need run: blocks before new primitives are added.
  • After Phase 3: enable --fail mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a run: block is added.

4.2 - Agent-Friendly Documentation

How AI agents and LLMs can discover and consume the Project Hummingbird documentation site.

The documentation site provides several features that make it easier for AI agents and LLMs to discover, index, and consume its content.

Discovery and access methods

llms.txt

The site publishes an llms.txt file at the root, following the llmstxt.org specification. This file is a Markdown-formatted index of all documentation pages, grouped by section, with links to each page’s markdown URL. It serves as the primary entry point for agents discovering the site.

llms-full.txt

The site also publishes llms-full.txt – a full content dump of all documentation pages in a single file. This is intended for LLMs with large context windows that want to ingest the entire site in a single request without crawling individual pages.

Pages whose rendered content exceeds a size threshold (currently 50,000 characters) are replaced with a placeholder that includes the page title, description, and a link to the individual page’s markdown URL for on-demand fetching. This keeps the file at a manageable size while ensuring no content is completely inaccessible.

Per-page markdown output

Every page on the site is available in clean markdown format by appending /index.md to its URL. For example:

  • HTML: https://hummingbird-project.io/docs/contributing/quickstart/
  • Markdown: https://hummingbird-project.io/docs/contributing/quickstart/index.md

The markdown output resolves all shortcodes (such as file includes from other repositories) while preserving the content as markdown rather than converting to HTML. Each markdown page includes links to both llms.txt and llms-full.txt so agents can discover the site-wide indexes from any page.

Content negotiation

A CloudFront distribution sits between DNS and the site’s page host. A viewer-request CloudFront Function inspects the Accept header on every request: when the header includes text/markdown, the function rewrites the URI to serve the pre-built .md equivalent. Only HTML-like patterns (trailing slash, .html, bare paths) are rewritten; all other URIs pass through unchanged.

This means agents can use standard HTTP content negotiation on any page URL without needing to know the /index.md convention:

GET /docs/contributing/quickstart/ HTTP/1.1
Accept: text/markdown

Pages referenced from external sources have stable short URLs under the /l/ prefix (e.g., /l/adding-images). These return HTTP 301 to the canonical page URL, so pages can move without breaking incoming links. The Writing Documentation guide describes how to add aliases.

The CloudFront viewer-request function recognizes /l/ paths and normalizes them before they reach the origin. Content negotiation (Accept header) is skipped for /l/ paths — instead, the URL form determines the response: /l/foo and /l/foo/ redirect to the HTML page, while /l/foo.md and /l/foo/index.md redirect to the markdown file. If no matching alias exists, the origin returns a 404.

Specifications and tools

The site aims to follow the emerging standards for agent-friendly documentation:

  • llmstxt.org – the llms.txt specification
  • Agent-Friendly Documentation Spec – broader specification for documentation sites that serve AI agents
  • AFDocs – scoring tool that evaluates a site’s compliance with the Agent-Friendly Documentation Spec

Planned features

HTML llms-directive (HUM-6571)

A Docsy upgrade to v0.17.0 will embed a pointer to llms.txt inside the HTML <body> of every page, so agents landing on any HTML page can discover the LLMS index without prior knowledge of the site structure.

Documentation quality evaluation (HUM-6575)

A promptfoo-based evaluation harness will measure how accurately LLMs can answer questions about Hummingbird using the published documentation. This provides a feedback loop for identifying documentation pages that need improvement for agent consumption.

4.3 - Containers repository

Background documentation sourced from the containers repository.

4.3.1 - Image Pipeline

How the container image pipeline works from source to release

Explanation of the complete container image pipeline, from RPM dependency updates through source templates, generation, build, testing, release, and advisory publication.

Overview

The container image pipeline consists of six main stages:

  1. Source Templates - Jinja2 templates in images/*/ define container images
  2. Generation - Templates are rendered into Containerfiles and Konflux resources
  3. Build - Konflux builds multi-architecture container images
  4. Testing - Testing Farm runs integration tests via Konflux
  5. Enterprise Contract Validation - Conforma validates policy compliance before release
  6. Release - Images are published to registries and advisories are created

RPM packages are built and published separately via the RPM Pipeline. This document covers what happens after RPMs are available in the Hummingbird package repository.

RPM Dependency Updates

Container images pin exact RPM versions via lockfiles. When new RPMs are published to the Hummingbird Pulp repository, lockfiles must be regenerated for images to pick up the updates.

Lockfile structure

Each image variant has two RPM-related files:

File Purpose
generated/images/<image>/<distro>/<variant>/rpms/rpms.in.yaml Declares required packages
generated/images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml Pins exact versions, URLs, and checksums

The lockfile contains every RPM (direct and transitive dependencies) with its exact version, architecture, URL pointing to the Pulp repository, and checksum. During build, the locked RPMs are downloaded in hermetic mode — no network access to external repositories.

Automatic lockfile updates

MintMaker / Renovate — A custom Renovate instance (forked rpm-lockfile manager from redhat-exd-rebuilds/renovate) runs as a Kubernetes CronJob. It scans each rpms.in.yaml file individually, resolves dependencies against Pulp repositories, and opens auto-merge MRs — one per image group and distro (e.g., “Refresh RPM lockfiles for go-1-25/hummingbird”).

The underlying tool is rpm-lockfile-prototype, which wraps the DNF dependency solver. For a given rpms.in.yaml, it resolves all direct and transitive dependencies against the yum repositories defined in yum-repos/*.repo and outputs a complete rpms.lock.yaml with pinned versions, download URLs, and checksums. Locally, generate_rpms_lock.py adds hash-based optimization: it hashes the input file and skips regeneration when the hash matches the existing lockfile, avoiding expensive solver runs when nothing has changed.

Manual lockfile updates

To refresh the lockfile for a single image variant:

make generated/images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml FORCE_REFRESH=true

For example:

make generated/images/caddy/hummingbird/default/rpms/rpms.lock.yaml FORCE_REFRESH=true

To regenerate all lockfiles and open MRs for the changes:

# Regenerate all rpms.lock.yaml files
make all-host FORCE_REFRESH=true
# Open MRs for any changed lockfiles
ci/create_lockfile_update_mrs.sh

From lockfile MR to container build

When a lockfile MR merges to main, PipelinesAsCode triggers a container build (Stage 3). Each image variant has CEL expressions in its PipelineRun definition that match on file paths, so only images whose lockfiles changed get rebuilt.

Stage 1: Source Templates

Each container image is defined by templates in images/<image-name>/:

  • properties.yml - Image configuration (packages, variants, tags, etc.)
  • Containerfile.j2 - Jinja2 template for the container build
  • README.md.j2 - Documentation template
  • tests-container.yml - Integration test definitions

Templates use reusable macros from macros/*.yml.j2:

  • setup_newroot() - Configures DNF and filesystem
  • install_newroot() - Installs packages
  • cleanup_newroot() - Cleans up files
  • final_stage() - Creates scratch-based final image

Shared configuration is defined in images/variables.yml. See Global Variables Reference for details.

Stage 2: Generation

Templates are rendered into concrete artifacts that drive the pipeline:

  • Containerfiles - Build instructions for each image variant
  • README documentation - Image documentation for Quay.io
  • Konflux resources - CI/CD pipeline definitions

Containerfile Generation

Containerfiles are generated from templates for each image variant using Make’s incremental build system:

make

This combines:

  • Reusable macros from macros/
  • Service-specific templates from images/*/Containerfile.j2
  • Configuration from properties.yml (see Image Configuration Reference)
  • Variables from images/variables.yml (see Global Variables Reference)
  • RPM versions from rpms.lock.yaml files
  • Git submodule information from .gitmodules

Output: generated/images/<image-name>/<distro>/<variant>/Containerfile, along with VERSION and TAGS files

The build system uses timestamp-based dependency tracking, so only changed files are regenerated.

README Generation

After Containerfiles are generated, README documentation is generated from README.md.j2 templates using macros from macros/readme.yml.j2. Each image template produces one README per target defined in images/variables.yml (Hummingbird, Rawhide, Red Hat).

Output: generated/images/<image-name>/hummingbird/README.md, generated/images/<image-name>/rawhide/README.md, generated/images/<image-name>/README.md

Konflux Resource Generation

Konflux CI/CD resources are generated from templates in konflux-templates/:

make

This generates:

  • Components - Define what to build (one per image variant)
  • ImageRepositories - Define where to push images
  • ReleasePlanAdmissions - Define how to release images

Output: konflux-templates/rendered.yml

These resources must be deployed to Konflux before builds can run. See the Konflux Resource Deployment guide for how and when resources are deployed.

Stage 3: Build

Konflux builds container images automatically when changes are pushed to GitLab.

Build Triggers

  • Merge Requests: Builds all changed images and triggers tests
  • Main Branch: Builds all changed images (tests do not run on main)

Build Process

For each image variant:

  1. Konflux Component watches the GitLab repository
  2. On changes, Konflux triggers a build PipelineRun
  3. The build uses the generated Containerfile from generated/images/<name>/<distro>/<variant>/Containerfile
  4. Images are built for multiple architectures (x86_64 and aarch64)
  5. Built images are pushed to the development registry

Build Output

Development images are pushed to the Red Hat User Workloads registry:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main

Merge request builds are tagged with:

quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main:on-mr-<MR_ID>-<COMMIT_SHA>

Examples:

  • quay.io/redhat-user-workloads/hummingbird-tenant/curl--default--main
  • quay.io/redhat-user-workloads/hummingbird-tenant/nginx--builder--main:on-mr-123-abc1234

SBOM Generation

Each per-architecture build also produces an SPDX 2.3 Software Bill of Materials (SBOM), attached as an OCI artifact to the image. The SBOM is assembled from two independent scans, merged into a single document:

flowchart LR
    syft["Syft\n(buildah-remote-oci-ta)"] --> mobster["Mobster\n(buildah-remote-oci-ta)"]
    hermeto["Hermeto\n(prefetch-dependencies-oci-ta)"] --> mobster
    mobster --> sbom["Per-arch SPDX SBOM"]
    sbom --> index["Mobster\n(build-image-index)"]
    index --> oci["Index SBOM\n(.sbom OCI artifact)"]
  • Syft runs as the sbom-syft-generate step inside the buildah-remote-oci-ta Tekton task. It scans the RPM database of the built per-arch image and finds installed binary RPMs plus non-RPM packages (Go modules, pip packages, etc.). One SBOM is produced per architecture.
  • Hermeto runs inside the prefetch-dependencies-oci-ta Tekton task. It records all build-time dependencies from lockfiles. A single Hermeto SBOM covers all architectures and source RPMs.
  • Mobster runs as the prepare-sboms step inside buildah-remote-oci-ta, merging the Syft and Hermeto SBOMs into one SPDX document per architecture. A second Mobster invocation in the build-image-index task generates an index-level SBOM and attaches it as an OCI artifact.

See Security Labels and Metadata for the SBOM entry structure and how to access SBOMs from the registry.

Stage 4: Testing

Images are validated through two types of integration tests:

  • Container tests (tests-container.yml) - Run with Podman and Docker via Testing Farm on RHEL-9-Nightly systems
  • K8s tests (tests-k8s.yml) - Run in Konflux ephemeral Kubernetes namespaces

Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.

Test Triggering

Tests are only triggered on merge requests, not on main branch builds.

For an image group at images/<group-name>/, tests are triggered for all changes below that directory, excluding documentation-only changes.

Any changes below .tekton/ or ci/ will trigger tests for the caddy image to ensure infrastructure changes work before merging.

Test Execution

For each image group, pipelines are created per variant. If there is more than one variant in an image group, additional group pipelines are created. See Konflux group snapshot documentation for details.

Tests are selected based on the variants field in test files. Additionally, global tests from ci/{variant}-tests/ are included. Group pipelines run tests that specify variants: [group], allowing validation across multiple variants.

Per-component Kubernetes runs execute two categories of tests:

  • Own-variant tests: tests defined for the component’s pair (the snapshot image is available via IMAGE_URL).
  • Cross-group reverse dependencies: tests in other image groups that reference the changed component via TEST_IMAGES[group/variant]. The changed image comes from the snapshot; their own images are unaffected by the change.

Reverse dependency tests that reference a same-group sibling image — a variant of the same group other than the component under test — are excluded from per-component Kubernetes runs and run only in the group pipelines. Own-variant test selection is unchanged. A per-component run provides only the image under test; a test needing a sibling would fall back to a released image — silently testing the changed component against an older sibling — or fail to pull when no release exists. Group snapshot pipelines (the pr group GitLab status) cover all same-group combination tests at the exact commit, with digest-pinned siblings provided by Konflux.

Reverse Dependency Testing

When a base image like core-runtime changes, dependent images (rust, xcaddy, etc.) need to be tested to ensure compatibility. Reverse dependency testing is enabled by default. Images can opt-out by setting reverse_dependency_tests: false in their properties.yml. This is recommended for images like “curl” which are widely used and only have a small API which their own tests cover well enough.

For container tests, dependent images are rebuilt locally in the testing environment to prevent version skew and ensure dependent images are in sync with the repository status in the merge request under test.

For Kubernetes tests, reverse dependency testing selects cross-group consumers of the changed component. Same-group combination tests (those referencing sibling variants of the same group) are excluded from per-component runs; they execute in the group pipelines where all sibling images are available at the exact commit.

Integration Test Scenarios

Integration tests are triggered via IntegrationTestScenario resources defined in this repository (konflux-templates/scenarios/<type>.yml.j2 templates, instantiated by konflux-templates/scenarios.yml) and deployed through the *-releases sync on merge to main. The trigger manifests (integration_test_scenarios section) are generated from the same scenarios.yml matrix so they cannot drift from the deployed CRs:

Application Purpose
containers-hummingbird Red Hat supported Hummingbird images
containers-community-hummingbird Community images (support_level: community)
containers-ci-hummingbird Experimental/infrastructure images (support_level: experimental)
containers-rawhide All Rawhide-based images

Container Testing

Container tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Container Testing Flow

  1. Developer opens MR modifying images/nginx/
  2. GitLab CI pipeline and Konflux pipelines start in parallel
  3. Konflux builds nginx image variants
  4. IntegrationTestScenario triggers Testing Farm job
  5. tmt discovers enabled child plans in ci/run_tests_container.fmf
  6. Testing Farm provisions one RHEL-9-Nightly machine per enabled plan (per architecture)
  7. tmt sets up the testing environment on each machine
  8. tmt runs tests via ci/run_tests_container.sh

Container Test ITS Configuration

The container tests use the upstream Testing Farm pipeline for Konflux CI:

kind: IntegrationTestScenario
spec:
  contexts:
    - { name: pull_request }
  params:
    - { name: COMPOSE, value: RHEL-9-Nightly }
    - { name: ARCH, value: x86_64|aarch64 } # one for each
    - { name: CONTEXT, value: fips=enabled } # containers-hummingbird only
    - { name: IMAGE_TAG, value: v3.22 }
  resolverRef:
    resolver: bundles
    params:
      - {
          name: bundle,
          value: quay.io/testing-farm/tmt-via-testing-farm:$(params.IMAGE_TAG),
        }
      - { name: name, value: tmt-via-testing-farm }
      - { name: kind, value: pipeline }

FMF Test Plan

The root folder of the containers repository is marked with a .fmf directory to enable Testing Farm support. The test plan in ci/run_tests_container.fmf uses FMF virtual child plans to define two execution variants:

  • /standard — inherits all parent configuration, always enabled
  • /fips — inherits all parent configuration plus a FIPS prepare step; disabled by default, enabled via adjust when the fips=enabled context is set

The parent node defines shared configuration (prepare, discover, execute) inherited by both children. Because it has children, FMF treats it as a branch node and does not select it as a runnable plan. The containers-hummingbird ITS passes CONTEXT=fips=enabled, so both /standard and /fips run in parallel. Other applications (rawhide, ci-hummingbird, community-hummingbird) pass no CONTEXT, so only /standard runs.

Both child plans run the following steps:

  1. Install podman for container testing
  2. Install Docker and start the Docker daemon on the host for Docker integration tests
  3. Fix git submodules until TFT-3991 is resolved
  4. For regular pipelines:
    • Verify the image built by Konflux is reproducible using ci/test_rebuild.sh
    • If needed, build reverse dependency images locally via Buildah
    • Run Podman and Docker tests via ci/run_tests_container.sh --component-name
  5. For group pipelines:
    • Run Podman and Docker group tests via ci/run_tests_container.sh --group-component-name

The /fips child additionally enables FIPS mode on the provisioned RHEL machine before all other prepare steps (via order: 0).

Container Test Environment

Testing Farm provides these environment variables to the test plan:

Variable Description
IMAGE_NAME Single component (e.g., curl--default--main)
IMAGE_NAMES Multiple components for group pipelines
IMAGE_URL Image URL from Konflux
IMAGE_URL_... Image URLs from Konflux for group pipelines
SNAPSHOT_b64 Snapshot metadata (base64 encoded)

K8s Testing

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

K8s Testing Flow

  1. Developer opens MR modifying an image in the containers repository (e.g., nginx)
  2. GitLab CI pipeline and Konflux pipelines start in parallel
  3. Konflux builds image variants
  4. IntegrationTestScenario triggers K8s test pipeline
  5. Pipeline checks for tests-k8s.yml; skips with SUCCESS if not found
  6. Pipeline provisions ephemeral namespace via Konflux EaaS (tied to PipelineRun lifecycle)
  7. Pipeline fetches source via Trusted Artifacts
  8. Tests run via ci/run_tests_k8s.sh with kubeconfig for ephemeral namespace
  9. Pipeline fails if any test reports non-SUCCESS, ensuring GitLab sees correct status

K8s Test ITS Configuration

The K8s tests use the k8s-test-pipeline:

kind: IntegrationTestScenario
spec:
  contexts:
    - { name: pull_request }
  resolverRef:
    resolver: bundles
    params:
      - { name: bundle, value: quay.io/hummingbird-ci/k8s-test-pipeline:latest }
      - { name: name, value: k8s-test }
      - { name: kind, value: pipeline }

Stage 5: Enterprise Contract Validation

Before images can be released, they must pass Enterprise Contract (also known as Conforma) policy validation. This ensures images meet security, compliance, and build quality standards. These checks can also be run locally against Konflux-built images.

Policy Validation

Enterprise Contract validates that:

  • Images are built using trusted, verified Tekton tasks
  • Builds are hermetic (network-isolated with pre-fetched dependencies)
  • Required security tests have passed
  • Images have proper metadata and labels
  • Build artifacts meet supply chain security requirements

Policy Configuration

Policies are defined as EnterpriseContractPolicy resources in konflux-templates/macros/policy.yml.j2:

  • containers-hummingbird / containers-rawhide - Strict policies for production images
  • containers-community-hummingbird - Policy for community-supported images in the containers repository (same base exclusions as production policies)

These policies use the @redhat rule collection from the ec-release-policy.

Policy Exclusions

The following checks are excluded from the default @redhat policy set. When modifying exclusions, update the policy macro and this documentation.

Test Package

The test package verifies that each build was subjected to a set of tests and that those tests all passed.

Snyk SAST checks (test.required_tests_passed:sast-snyk-check, test.no_skipped_tests:sast-snyk-check, test.required_tests_passed:sast-snyk-check-oci-ta, test.no_skipped_tests:sast-snyk-check-oci-ta) are excluded because Hummingbird images are currently not supported by Snyk.

Red Hat certification preflight checks (test.no_failed_tests:ecosystem-cert-preflight-checks, test.no_erred_tests:ecosystem-cert-preflight-checks) are excluded because Hummingbird images are not yet published to the Red Hat certified container registry.

Informative test failures (test.no_failed_informative_tests) are excluded because these produce warnings for advisory purposes only and are explicitly non-blocking.

Deprecated image warnings (test.no_test_warnings:deprecated-image-check) are excluded because final images are built FROM scratch, and the builder image is updated via Renovate like all other images.

Trusted Task Package

The trusted_task package verifies that all Tekton Tasks involved in building the image are trusted by comparing Task references with a pre-defined list of trusted Tasks.

The trusted_task.current check warns when newer versions of tasks are available. This is excluded because we use stable pinned task versions and control upgrade timing via Renovate rather than requiring the latest version at all times.

RPM Repos Package

The rpm_repos package confirms that all RPM packages listed in SBOMs specify a known and permitted repository ID.

The rpm_repos.ids_known check is excluded because images use the internal hummingbird repository and Fedora repositories, which are not in the upstream known_rpm_repositories.yml list (that file only contains Red Hat official repositories).

RPM Signature Package

The rpm_signature package verifies that all RPMs in the image are signed by an allowed signing key.

The upstream default only allows the Red Hat release key 2 (199e2f91fd431d51). Rawhide, community, and experimental images contain Fedora-signed RPMs, so the containers-rawhide, containers-community-hummingbird, and containers-ci-hummingbird policies add Fedora signing keys via the additional_rpm_signature_keys macro parameter. The containers-hummingbird policy uses the default (Red Hat key only) because Hummingbird images install RPMs exclusively from the Hummingbird repository.

The Fedora key list in konflux-resources.yml.j2 should be kept in sync with the keys published at https://fedoraproject.org/security.

Labels Package

The labels package checks if the image has the expected labels set, including required and optional labels for Red Hat container certification.

Both labels.required_labels and labels.optional_labels are excluded because images currently only include basic labels (maintainer, license_terms, name, cpe) and version labels, not the full set of Red Hat certification labels (vendor, version, release, summary, description, url, etc.) required for Red Hat Ecosystem Catalog publishing.

Buildah Build Task Package

The buildah_build_task package verifies buildah build task parameters.

The buildah_build_task.privileged_nested_param check verifies that PRIVILEGED_NESTED is not set to true. This is excluded because images use the dnf-installroot helper from the builder image to build all containers, including the builder image itself. This script requires privileged operations (unshare, mount -t tmpfs, mount --bind for /proc and /dev/*) to set up the install root environment (see commit 3668af16).

Schedule Package

The schedule package verifies that releases conform to a given schedule, including weekday restrictions.

The schedule.weekday_restriction check is excluded to allow releases any day including weekends. The @redhat policy restricts weekend releases, but this project needs the ability to ship urgent CVE fixes immediately regardless of the day of week.

CVE Package

The cve package checks for blocking and non-blocking CVEs in container images.

The cve.cve_blockers check is excluded because blocking on known CVEs would prevent releasing images that fix other CVEs. A VEX feed could suppress false positives for RPM-level CVEs, but CVEs can also originate from other artifact types where VEX does not apply (see MR !2227).

Hermetic Task Package (CI-Only)

The hermetic_task package verifies that tasks were invoked with the proper parameters to perform a hermetic (network-isolated) execution.

All Konflux applications in the containers repository enforce the hermetic task check. All applications allow quay.io/hummingbird-ci/ as a base image source because images build FROM quay.io/hummingbird-ci/hummingbird-builder via setup_newroot.yml.j2.

Stage 6: Release

After images pass testing in merge requests and are merged to main, they are released to public registries.

Registry Organization

Images are published to different registries based on distro and purpose:

Registry Purpose
registry.access.redhat.com/hi/ Red Hat supported Hummingbird images (production)
quay.io/hummingbird/ Red Hat supported Hummingbird images (mirror)
quay.io/hummingbird-community/ Community-supported Hummingbird images
quay.io/hummingbird-rawhide/ All Rawhide-based images
quay.io/hummingbird-ci/ CI and tools images

Production Registry: Red Hat supported Hummingbird images are published to registry.access.redhat.com/hi/ via the rh-advisories release pipeline and mirrored to quay.io/hummingbird/.

Community Registry (hummingbird-community): Publishes community-supported images (those with support_level: community in properties.yml). Examples include minio, minio-client, and os-minimal.

CI/tools Registry (hummingbird-ci): Publishes tooling and infrastructure images. This includes tools-repo images (built outside the containers repository) and experimental images from the containers repository (support_level: experimental), such as hummingbird-builder.

Release Process

  1. Merge request passes all tests
  2. Merge request is merged to main branch
  3. Konflux builds images from main
  4. ReleasePlanAdmission resources trigger the release pipeline
  5. Images are signed via Cosign for supply chain attestation
  6. Images are copied from the Konflux registry to target registries
  7. Tags are applied based on properties.yml configuration
  8. For production releases: images are registered in the Red Hat container catalog via Pyxis

Release Mechanism

Releases are configured via ReleasePlanAdmission (RPA) resources in the containers repo and ReleasePlan resources in the infrastructure repo:

  • ReleasePlanAdmission - Defines per-registry release configuration (target registry, tags, visibility settings)
  • ReleasePlan - Triggers the release pipeline for a specific application and registry

Each distro/registry combination has its own RPA.

Production release pipeline

The production release uses the rh-advisories pipeline from the release-service-catalog. The push-snapshot task pushes images from the development registry to:

  • Production: registry.access.redhat.com/hi/<image>:<tags>

Configuration is defined in releng/hummingbird-containers-prod.yaml.

Image signing

Images are signed using Cosign as part of the release pipeline. Signing provides supply chain attestation, allowing consumers to verify image provenance.

Pyxis catalog registration

The release pipeline registers images in the Red Hat container catalog via the Pyxis API:

  • Product: Red Hat Hardened Images (Product ID 1071)
  • Metadata: version, categories, layer information, SBOM references
  • Configuration: releng/pyxis-hummingbird.yaml

Pyxis registration makes images discoverable in the Red Hat Ecosystem Catalog and links them to security advisories.

Release Output

Released images are published to registries based on distro and support level:

registry.access.redhat.com/hi/<image-repository>:<image-tag>
quay.io/hummingbird/<image-repository>:<image-tag>
quay.io/hummingbird-community/<image-repository>:<image-tag>
quay.io/hummingbird-rawhide/<image-repository>:<image-tag>
quay.io/hummingbird-ci/<image-repository>:<image-tag>

Examples:

  • registry.access.redhat.com/hi/curl:8 - Production Red Hat supported image
  • quay.io/hummingbird/nodejs:20 - Red Hat supported Hummingbird image (mirror)
  • quay.io/hummingbird-community/minio:latest - Community-supported image
  • quay.io/hummingbird-ci/hummingbird-builder:latest - Hummingbird Builder image
  • quay.io/hummingbird-rawhide/curl:latest - Rawhide image

Release Tags

Tags are extracted from Containerfile labels as defined in properties.yml:

  • latest - Latest version of the image
  • <major> - Major version (e.g., 20 for Node.js 20.x)
  • <major>.<minor> - Major.minor version (e.g., 20.11)
  • <full-version> - Complete version with release (e.g., 20.11.1-1.fc42)
  • <YYYYMMDD> - UTC date when RPM lockfile content last changed, available via rpm_lockfile.content_date for any image that includes it in its tags. Only advances when the resolved RPM set changes.
  • <timestamp> - Build timestamp (production releases only)

Non-default variants receive a -<variant> suffix (e.g., latest-builder).

Advisory Creation

When the production release pipeline runs, advisories are created:

  1. The Release Service generates an Advisory YAML from ReleasePlan and ReleasePlanAdmission metadata
  2. The advisory is pushed to the advisories repo on CEE GitLab
  3. GitLab CI validates the advisory against the schema and enforces field-level permissions
  4. On merge to main, the advisory is published via UMB and Kafka

Advisory types:

Type Meaning
RHSA Security Advisory (contains CVE fixes)
RHBA Bug Fix Advisory
RHEA Enhancement Advisory

After publication, the Pyxis API links each image record to its advisory via image_advisory_id.

VEX Feed Update

After advisory publication, Red Hat SDEngine generates public vulnerability data:

  • CSAF VEX documents (per-CVE) published at https://security.access.redhat.com/data/csaf/v2/vex-feed/
  • CSAF Advisory documents (per-advisory) published at https://security.access.redhat.com/data/csaf/v2/advisories/

The hummingbird-vex-feed repository maintains CPE mappings that associate Hummingbird package repositories with cpe:/a:redhat:hummingbird:1.

Image Documentation

When a commit to main changes a README.md file, the content is automatically pushed to quay.io as the image description via the update_quay_description job.

References

4.3.2 - Global Variables Reference

Complete reference for global configuration in images/variables.yml

Complete reference for global configuration settings that apply to all container images.

Overview

The images/variables.yml file contains global configuration that applies to all container images in the repository. These settings define project-wide defaults, security parameters, and variant-specific behavior.

Location: images/variables.yml (in the containers repository root)

Scope: All images inherit these settings unless overridden in image-specific properties.yml

Configuration Reference

cpe

  • Type: String (CPE 2.2 formatted string)
  • Default: "cpe:/a:redhat:hummingbird:1"
  • Description: Common Platform Enumeration (CPE) identifier for all Hummingbird containers.
  • Usage: Automatically added as a label to all generated container images and passed to inject-source-info during the build process.
  • Format: CPE 2.2 URI format: cpe:/part:vendor:product:version

default_user

  • Type: String (numeric UID)
  • Default: "65532"
  • Description: The default unprivileged user ID that all containers run as.
  • Usage: Referenced by the {{ set_user() }} macro when user: default is specified (or when user: is omitted, as default is the default value).
  • Value: 65532 is chosen as a high, non-conflicting UID that:
    • Avoids conflicts with system users (typically < 1000)
    • Avoids conflicts with regular users (typically 1000-60000)

default_distros

  • Type: Array of strings
  • Default: ["rawhide", "hummingbird"]
  • Description: Defines the distribution variants (distros) for which images are built. Each distro represents a different base package source.
  • Directory Structure: Images are organized as images/<name>/<distro>/<variant>/
  • Common Values:
    • rawhide - Uses Fedora Rawhide repositories only
    • hummingbird - Uses both Fedora Rawhide and Hummingbird repositories
  • Usage: Scripts iterate over distros and variants to find all image variants to build. Each distro/variant combination produces a separate container image.
  • See Also: default_variant_repos for repository configuration per distro

default_variants

  • Type: Array of strings
  • Default: ["default", "builder"]
  • Description: Defines the default set of variants to generate for each image within each distro when not explicitly specified in the image’s properties.yml.
  • Directory Structure: Variants are subdirectories within each distro: images/<name>/<distro>/<variant>/
  • Common Values:
    • default - The minimal runtime variant
    • builder - Extended variant with build tools and package managers
  • Extending: Images can use additional_variants in properties.yml to add extra variants while keeping the defaults (preferred approach)
  • Override: Images can specify variants in properties.yml to replace these defaults entirely (use when you need to exclude default variants)
  • See Also: Image Configuration Reference - variants

default_rpm_packages

  • Type: Object with all or variant names as keys, arrays of package names as values

  • Default:

    default_rpm_packages:
      all:
        - system-release
      builder:
        - bash
        - dnf5
        - shadow-utils
    
  • Description: Adds packages to the packages from each image’s properties.yml. Both lockfile generation and Containerfile generation use these defaults.

  • Variants: all applies to every image and variant. builder also applies to variants with the -builder suffix and the hummingbird-builder image.

  • Usage: Use all for shared image requirements. The system-release package preserves distribution identity and repository configuration after setup removed its dependency on system-release. Use builder for package management tools.

default_variant_repos

  • Type: Object with distro names as keys, arrays of repository filenames as values

  • Default:

    default_variant_repos:
      rawhide:
        - fedora-44.repo
      hummingbird:
        - fedora-44.repo
        - hummingbird.repo
    
  • Description: Defines which yum repositories each distro uses by default for package installation and lockfile generation. The Rawhide distro uses the latest branched Fedora development repo, while Hummingbird uses a stable Fedora release plus the Hummingbird package repository.

  • Repository Files: References files in the yum-repos/ directory

  • Usage: This allows different distros to use different package sources. The hummingbird distro includes additional repositories that provide Hummingbird-specific packages.

  • Override: Image-specific additional_repos in properties.yml are appended to the distro-specific repos

  • See Also: Image Configuration Reference - additional_repos

oscap

  • Type: Object

  • Default:

    oscap:
      enabled: true
      profiles:
        cis: true
        stig:
          variants:
            - "*fips*"
      exclude_rules:
        - id: xccdf_org.ssgproject.content_rule_root_path_no_dot
          reason: "False positive: environmentvariable58 probe requires /proc
            which is unavailable in offline chroot scanning"
    
  • Description: Global defaults for OpenSCAP compliance scanning. Scanning is enabled by default (enabled: true). Images can opt out by setting enabled: false in their properties.yml.

  • Fields:

    • enabled - Whether compliance scanning is active (overridden per image)
    • profiles - Which compliance profiles run per variant. Each profile can be true (all variants), false (disabled), or {variants: [...]} with glob patterns. Default: CIS for all variants, STIG for FIPS variants.
    • exclude_rules - Global rule exclusions applied to all oscap-enabled images. Image-specific exclusions in properties.yml are concatenated with these (not replaced).
  • Merge Behavior: When an image overrides oscap fields:

    • enabled (scalar) is replaced by the image value
    • profiles (dict) is recursively merged, so an image can override individual profiles without affecting others
    • exclude_rules (list) is concatenated, so image rules are added after global rules
  • See Also: Image Configuration Reference - Compliance Scanning

readme_targets

  • Type: Object with target names as keys, configuration objects as values

  • Default: See the actual file

  • Description: Defines target-specific configuration for generating multiple README files from a single template. Each target represents a different build/distribution channel (e.g., Hummingbird project vs Red Hat product).

  • Target Configuration Fields:

    • filename - Output filename for this target’s README
    • registry - Container registry URL for image references
    • product_name - Full product name for documentation
    • product_name_short - Short product name for headings and titles
    • doc_base_url - Base URL for documentation links
  • Usage: Templates access these values via {{ readme_targets[target].registry }} and similar expressions. The build system generates one README file per target from the same README.md.j2 template.

  • See Also: Image Pipeline - README Generation

Next Steps

4.3.3 - Konflux Resource Deployment

How Konflux resources are defined and deployed across repositories

Konflux resources for container images are split between two repositories:

  • containers: Per-image resources generated from properties.yml
  • infrastructure: Application-level resources and deployment to the Konflux cluster

This separation allows independent iteration on each concern.

Deployment Matrix

Resource Type Source Containers MR Containers Push Infra MR Infra Push
Component containers manual manual - -
ImageRepository containers manual manual - -
ReleasePlanAdmission containers - automatic manual automatic
EnterpriseContractPolicy containers - automatic manual automatic
IntegrationTestScenario containers - automatic manual automatic
Application infrastructure - - manual automatic
ReleasePlan infrastructure - - manual automatic
ServiceAccount, Secret infrastructure - - manual automatic

Legend:

  • Containers MR: Downstream pipeline triggered from containers MR (deploys from MR commit)
  • Containers Push: Downstream pipeline triggered from containers push to main
  • Infra MR: Infrastructure MR pipeline (manual trigger)
  • Infra Push: Infrastructure push to main or web pipeline

Resource Locations and Rationale

Containers Repo

Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.

Component and ImageRepository:

  1. Ownership: Components are managed by the containers repo. The infrastructure pipeline uses ONLY_DOWNSTREAM so only explicit downstream triggers from containers deploy changes, giving the containers repo full control over the component lifecycle.
  2. Dynamic generation: Generated from images/*/properties.yml via ci/internal/generate_konflux_resources.sh, depending on per-image configuration (variants, tags, repository names).
  3. Timing: Must be deployed early during MR review so Konflux can build and test new images.

ReleasePlanAdmission:

  1. Dynamic generation: Includes per-image tag mappings extracted from properties.yml.
  2. Main branch only: Prevents race conditions—deploying from a feature branch risks undoing other images’ onboarding. If branch A adds image-foo and branch B (created earlier) adds image-bar, deploying B’s RPA would remove image-foo from the release config.

EnterpriseContractPolicy:

  1. Documentation co-location: Policy exclusions are documented in Image Pipeline. Keeping policy and documentation together eases maintenance.

IntegrationTestScenario:

  1. Ownership co-location: Scenarios live alongside the components, tests-k8s.yml inventory, and trigger manifests that describe them. Config is split between per-type CR templates (konflux-templates/scenarios/<type>.yml.j2, owning all params and resolver refs) and a pure application/trigger matrix (konflux-templates/scenarios.yml). Both the CR renderer (konflux-templates/rendered.yml) and the trigger-manifest generator read the same config, so deployed scenarios and manifest expectations cannot drift apart. See the trigger manifests reference for the manifest schema.
  2. Deployment: Deployed through the *-releases sync (fetches rendered.yml from the main branch on every merge to main), not the *-components sync. The components sync deploys from MR-head SHAs, so concurrent or stale MR pipelines would reset the live ITS CR to an older version; ITS is static config that is not supposed to change in MRs. The releases sync avoids this race and also means Renovate bundle-digest and IMAGE_TAG bumps auto-deploy on merge without any manual step.
  3. Renovate: Bundle digest and IMAGE_TAG pins are managed by regex managers in this repo’s renovate.json.

Infrastructure Repo

Application and ReleasePlan are defined in kubernetes/containers-*/ directories (one per Konflux application):

  1. Independent iteration: Changes are decoupled from containers repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the containers repo.
  2. Testable before merge: If defined in the containers repo, these would only deploy after merging to main (like RPAs), making iteration difficult.
  3. Static configuration: These resources don’t depend on per-image data from properties.yml.

ServiceAccount and Secret for releases are defined in kubernetes/release-quay-hummingbird*/ directories (one per Quay organization):

  1. Security: Secret specifications (names, structure, credential references) should not be exposed in the containers repo.
  2. Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the containers repo.

The service accounts are referenced by ReleasePlanAdmissions to authorize pushing images to each quay.io/hummingbird* organization.

See Also

4.3.4 - Security Labels and Metadata

Container labels, embedded metadata, and SBOMs for vulnerability scanning

Hummingbird container images include labels, embedded metadata, and SBOMs that enable security scanners to perform container-first vulnerability reporting.

Overview

Security scanners need a way to determine which vulnerabilities are applicable to a container image. Three mechanisms provide this information:

  1. Container Labels - OCI image labels (name, cpe) set in the Containerfile
  2. Embedded Metadata - A labels.json file written to the container filesystem during build
  3. Software Bill of Materials (SBOM) - An SPDX document listing all packages, attached as an OCI artifact alongside the image

Labels and embedded metadata identify the product; the SBOM identifies the packages within it. Labels and embedded metadata provide the same core information but serve different access paths: labels are accessible via container inspection tools, while embedded metadata is accessible to scanners with only filesystem access.

Applicability

The cpe label is only added to released Hummingbird distro images (no CPE identifier for non-product images). All other labels are added to all images.

The labels.json embedded metadata file contains all Containerfile LABEL values plus auto-computed fields.

SBOMs are attached to all released images (both Rawhide and Hummingbird).

Container Labels

name

  • Type: String
  • Format: <org>/<image> or <org>/<image>-<variant>
  • Examples: hummingbird/nodejs-24, hummingbird/nodejs-24-builder, hummingbird/caddy
  • Description: Canonical name for the container image. This is the name that appears in VEX (Vulnerability Exploitability eXchange) statements and is used by scanners to map vulnerabilities to specific images.

The canonical name is derived from the image directory name (which includes the version, e.g. nodejs-24) and the variant. For default variants, the name is hummingbird/<image> (e.g. hummingbird/nodejs-24). For non-default variants, the variant is appended with a hyphen (e.g. hummingbird/nodejs-24-builder). Note that the canonical name may differ from the registry repository path. For example, nodejs-24 and nodejs-20 are both pushed to the hummingbird/nodejs registry repository with different tags, but their canonical names are hummingbird/nodejs-24 and hummingbird/nodejs-20 respectively.

See container-image-labels.md for the full name label format across all image types.

cpe

  • Type: String (CPE 2.2 formatted)
  • Format: cpe:/a:redhat:hummingbird:1
  • Description: Common Platform Enumeration (CPE) identifier for the container. CPE is a standardized naming scheme for software products that enables correlation with vulnerability databases.

The CPE value is defined globally in images/variables.yml and applied to all Hummingbird images. Containers with the same CPE are considered the same software product for vulnerability reporting purposes.

org.opencontainers.image.created

  • Type: String (RFC3339 timestamp)
  • Format: 2025-01-15T12:00:00Z
  • Description: Creation timestamp of the container image. Used by scanners to determine if an image predates or postdates a vulnerability fix.

If SOURCE_DATE_EPOCH is set during the build, that timestamp is used instead of the actual build time, supporting reproducible builds.

Embedded Metadata (labels.json)

The labels.json file is written to /usr/share/buildinfo/labels.json inside the container filesystem by the inject-source-info script during build. It contains all Containerfile LABEL values plus auto-computed fields like architecture and org.opencontainers.image.created.

Schema

The file follows the embedded_metadata.v1 schema published by Red Hat Product Security.

Fields

The embedded_metadata.v1 schema defines the minimum fields required by security scanners:

Field Type Description
name string Canonical container name (e.g., hummingbird/nodejs-24)
cpe string CPE identifier (e.g., cpe:/a:redhat:hummingbird:1)
architecture string Target architecture (e.g., amd64, arm64)
org.opencontainers.image.created string RFC3339 creation timestamp

The file also includes all other Containerfile LABEL values (e.g., description, summary, vendor, version). See container-image-labels.md for the complete label reference.

Example

{
  "name": "hummingbird/caddy",
  "cpe": "cpe:/a:redhat:hummingbird:1",
  "architecture": "amd64",
  "org.opencontainers.image.created": "2025-01-15T12:00:00Z"
}

Software Bill of Materials (SBOM)

Each released image has a per-architecture SPDX 2.3 SBOM attached as an OCI artifact. See SBOM Generation for how the production SBOMs are built from Syft, Hermeto, and Mobster.

Accessing SBOMs

SBOMs are stored as OCI artifacts alongside each per-architecture image. To download one, resolve the per-arch digest and use cosign:

IMAGE="registry.access.redhat.com/hi/caddy"
TAG="latest"
ARCH="amd64"

# Get per-architecture digests from the image index
skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq '.manifests[] | {digest, platform}'

# Download the SBOM for the chosen architecture
ARCH_DIGEST="$(skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
  | jq -r --arg arch "${ARCH}" \
    '.manifests[] | select(.platform.architecture == $arch) | .digest')"
cosign download sbom "${IMAGE}@${ARCH_DIGEST}" > sbom.json

Entry Sources

The merged SBOM contains entries from two tools:

Source Content Scope
Syft Installed binary RPMs + non-RPM packages Image architecture only
Hermeto Build-time dependencies from lockfiles All architectures + source RPMs

A package installed in the final image may have entries from both Syft and Hermeto (with different metadata), since it was both a build dependency and is present at runtime.

Distinguishing Entry Sources

After the Mobster merge, Syft and Hermeto entries carry mutually exclusive markers:

Marker Syft Hermeto
sourceInfo field present ✓
licenseDeclared set ✓
CPE references ✓
upstream= in PURL ✓
Annotation containing “hermeto” ✓
repository_id= in PURL ✓

As a general rule, any entry with an annotation containing the word “hermeto” (case-insensitive) originates from Hermeto (build-time provenance). All other entries originate from Syft (runtime image scan).

PURL Format

Syft and Hermeto use different PURL qualifier sets for the same package. Examples from caddy:latest (Hummingbird, amd64):

Syft (runtime):

pkg:rpm/hummingbird/caddy@2.10.2-1.hum1?arch=x86_64&distro=hummingbird-20251124&upstream=caddy-2.10.2-1.hum1.src.rpm

Hermeto (build, binary):

pkg:rpm/caddy@2.10.2-1.hum1?arch=x86_64&checksum=sha256:ca02a0...&repository_id=public-hummingbird-x86_64-rpms

Hermeto (build, source):

pkg:rpm/caddy@2.10.2-1.hum1?arch=src&checksum=sha256:42912d...&repository_id=public-hummingbird-source-rpms

Key differences:

  • Distro namespace: Syft includes the distro in the PURL path (pkg:rpm/hummingbird/...). Hermeto omits it (pkg:rpm/...).
  • EVR: Syft populates versionInfo with the full epoch:version-release. Hermeto sets versionInfo to the bare upstream version; the full EVR is only in the PURL @version.
  • Architecture: Syft entries match the image architecture. Hermeto entries carry an arch= PURL qualifier that may be the image arch, a different arch (cross-arch), noarch, or src (source RPMs).
  • Source RPM: Syft entries carry upstream=<srpm> in the PURL. Hermeto entries have separate arch=src entries instead.

Hermeto Annotation Format

Hermeto entries carry annotations with JSON-encoded metadata:

{
  "annotationDate": "2026-03-02T11:58:12Z",
  "annotationType": "OTHER",
  "annotator": "Tool: hermeto:jsonencoded",
  "comment": "{\"name\": \"hermeto:found_by\", \"value\": \"hermeto\"}"
}

Entry Breakdown Example

Typical counts for caddy:latest (Hummingbird, amd64); exact counts vary as image dependencies change:

Source Typical count Content
Syft ~53 Binary RPMs (x86_64 + noarch)
Syft 1 Go module (stdlib)
Syft 2 OCI image metadata
Hermeto ~41 Binary RPMs (x86_64)
Hermeto ~41 Binary RPMs (aarch64, cross-arch)
Hermeto ~11 Binary RPMs (noarch)
Hermeto ~36 Source RPMs (arch=src)

How Scanners Use This Metadata

  1. CPE Matching: Scanners use the cpe value to look up applicable VEX statements for the product
  2. Name Matching: The name identifies which specific container the VEX statements apply to
  3. Version Comparison: The creation timestamp enables comparison between the scanned image and fixed versions reported in VEX statements
  4. Package Enumeration: Scanners use the SBOM to enumerate all packages in the image and correlate them with vulnerability databases via PURLs and CPEs
File Purpose
documentation/background/container-image-labels.md Complete reference for all image labels
documentation/background/image-pipeline.md SBOM generation pipeline (Stage 3)
images/variables.yml Defines the global cpe value
images/hummingbird-builder/inject-source-info.sh Script that creates labels.json
macros/inject_source_info_labels.yml.j2 Macro that adds LABEL to Containerfile
macros/install_newroot.yml.j2 Macro that invokes inject-source-info

See Also

References

4.3.5 - Container Image Labels

Complete reference for all container image labels

Hummingbird container images carry labels from multiple standards and namespaces. This page is the single reference for every label.

Standards: C = Conforma (rule dataset), O = OCI Image Spec, H = Hummingbird project, S = Security schema

Labels

Label Aliases Value C O H S
architecture Host architecture (e.g., x86_64) ✓
com.redhat.component hummingbird ✓
com.redhat.license_terms UBI EULA ¹
cpe CPE identifier (Hummingbird only) ² ✓ ✓
distribution-scope public ✓
io.hummingbird-project.containerfile Containerfile path relative to repo root ⁹ ✓
io.hummingbird-project.deprecated true for deprecated image streams ✓
io.hummingbird-project.major-minor-version Major.minor from tags (e.g., 2.10) ³ ✓
io.hummingbird-project.major-version Major from tags (e.g., 2) ³ ✓
io.hummingbird-project.repository Publishing name (e.g., caddy) ⁴ ✓
io.hummingbird-project.stream Version stream (e.g., 2) ⁴ ✓
io.hummingbird-project.variant Variant name (e.g., fpm-builder) ⁸ ✓
io.hummingbird-project.variant.base Base specialization (e.g., fpm) ⁸ ✓
io.hummingbird-project.variant.builder true when builder (absent otherwise) ⁸ ✓
io.hummingbird-project.variant.description Base variant description (no modifiers) ⁸ ✓
io.hummingbird-project.variant.fips true when FIPS (absent otherwise) ⁸ ✓
io.k8s.description Long description ⁵ ✓
maintainer Project Hummingbird / Red Hat ✓
name Canonical publishing repository ² ✓ ✓
org.opencontainers.image.created build-date RFC3339 build timestamp ✓ ✓ ✓
org.opencontainers.image.description description Long description ⁵ ✓ ✓
org.opencontainers.image.revision vcs-ref Full git commit SHA ✓ ✓
org.opencontainers.image.source GitLab repository URL ✓
org.opencontainers.image.title Image name (e.g., caddy) ✓
org.opencontainers.image.url url Upstream project URL ⁵ ✓ ✓
org.opencontainers.image.vendor vendor Red Hat, Inc. ⁶ ✓ ✓
org.opencontainers.image.version version Full version from tags (e.g., 2.10.2) ³ ✓ ✓
release SOURCE_DATE_EPOCH (commit timestamp) ⁷ ✓
summary Short one-liner ⁵ ✓
vcs-type git ✓

Notes

¹ Licenses

The com.redhat.license_terms label is not required by Conforma, OCI, or the security schema. It is a Red Hat convention present on all Red Hat container images (UBI, language runtimes, etc.), pointing to the UBI EULA.

The org.opencontainers.image.licenses label is intentionally not set. Per-package license data is derived from SBOMs stored as OCI artifacts alongside each image. A manually-set SPDX expression would be incomplete compared to the SBOM-derived data available in the image catalog.

² Name and identity

The name label identifies the canonical publishing repository. It uses the repository value from properties.yml, prefixed by the registry organization:

  • Red Hat supported Hummingbird images: hi/<repository>
  • Community images: hummingbird-community/<repository>
  • Experimental images: hummingbird-ci/<repository>
  • Rawhide images: hummingbird-rawhide/<repository>

All variants published to the same repository have the same name label. For example, the default and builder variants of Caddy both use hi/caddy.

The cpe label is only set for Hummingbird distro images (not Rawhide). See Security Labels and Metadata for details on the scanning workflow and labels.json.

³ Version labels

Three version granularities are derived from the image’s tags: version (full, e.g., 2.10.2), major-minor-version (e.g., 2.10), and major-version (e.g., 2). The full version is the OCI/Conforma org.opencontainers.image.version; the two coarser granularities are Hummingbird project labels used for tag aliasing. There is no io.hummingbird-project.version label — it would duplicate the OCI label.

⁴ Repository and stream

The io.hummingbird-project.repository and io.hummingbird-project.stream fields together form the release identity pair. Both are defined in properties.yml. See image-configuration-reference for field definitions.

The io.hummingbird-project.deprecated label is present with the value true on the final release of a deprecated stream. It is omitted for non-deprecated streams. The catalog uses the label to preserve the current deprecation state without changing older image digests.

⁵ Description, summary, and URL

The description, summary, and url fields are defined in properties.yml. Description and summary target different display contexts:

  • summary: One-liner (~40-80 chars) for table/list views
  • description: Short paragraph (~100-250 chars, 1-2 sentences) for card views and podman inspect

Style rules:

  • Do not start summary or description with the image name
  • Use >- YAML scalar for multi-line readability in properties.yml
  • Avoid embedded double quotes and backslashes (no escaping in templates)

⁶ Vendor

All images use Red Hat, Inc. as vendor (the distributing entity). The OCI org.opencontainers.image.vendor uses Red Hat (without “, Inc.”) per OCI convention. Both values are set on all distros.

⁷ Release

The release label uses the commit timestamp as a Unix epoch (matching Red Hat convention). In Konflux this is set via the generate-labels pipeline task.

⁸ Variant labels

Each variant name is decomposed into a base specialization and cross-cutting modifiers (builder, fips). The naming convention is <base>[-fips][-builder], where modifier order does not matter. A bare modifier like builder has base default.

Examples: fpm-builder → base=fpm, builder=yes; fips-builder → base=default, builder=yes, fips=yes.

The .description label stores only the base description (e.g., “PHP FastCGI process manager”). Modifier display is handled by consumers using the .builder and .fips boolean labels. Base descriptions come from variant_descriptions in images/variables.yml (for default) and image-specific properties.yml (for bases like fpm, runtime).

⁹ Containerfile path

The io.hummingbird-project.containerfile label contains the path to the Containerfile source relative to the repository root (e.g., generated/images/caddy/hummingbird/default/Containerfile). A full URL to the source can be constructed from org.opencontainers.image.source + org.opencontainers.image.revision + this path.

Embedded Metadata (labels.json)

All labels are written to /usr/share/buildinfo/labels.json inside the container filesystem, providing filesystem-level access for security scanners. See Security Labels and Metadata for schema details.

File Purpose
images/<image>/properties.yml Per-image metadata fields
ci/internal/generate_jinja2.py Builds inject_labels dict from properties
macros/image_metadata_labels.yml.j2 Emits LABEL instructions from inject_labels
macros/inject_source_info_labels.yml.j2 Emits name and cpe LABEL instructions
macros/install_newroot.yml.j2 Invokes inject-source-info.sh with labels
images/hummingbird-builder/inject-source-info.sh Writes labels.json to container filesystem
ci/build_images.sh Adds build-time labels
ci/check_release_fields.py Validates label field values in properties.yml

4.3.6 - CI Scripts

This section documents the CI scripts used in the Project Hummingbird container build and test pipeline.

These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline.

4.3.6.1 - build_images.sh

Build container images using buildah with support for multiple architectures and container engines

Purpose

Build container images using buildah with support for multiple architectures and container engines.

Usage

Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...]

OPTIONS:
    --verbose, -v        Enable verbose output during build
    --arch ARCHITECTURE  Specify target architecture (e.g., amd64, arm64, s390x)
    --engine ENGINE      Specify runtime engine for testing/export (podman or docker)
    --setup              Set up Docker-in-Docker environment before exporting
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --local-rpms-dir DIR Directory containing custom RPMs to use during build.
                         These have a higher priority and thus override the
                         standard repositories.
    --build-deps         Build all dependencies of specified groups, but NOT the
                         specified groups themselves. Collects forward, reverse,
                         and transitive dependencies automatically.
    --affected-tests     With --build-deps, select dependencies used by affected tests
    --pull               Pull images from registry instead of building locally.
                         Fast alternative for local development. Uses published
                         images from quay.io.
    --dryrun             Show what would be built (or pulled with --pull) without
                         actually building/pulling.
    --help, -h           Show this help message

Examples

# Build single image group (all distro/variants)
ci/build_images.sh nginx
ci/build_images.sh nodejs-20

# Build specific distro/variant only
ci/build_images.sh nginx/rawhide/builder
ci/build_images.sh nodejs-20/hummingbird/default

# Build multiple image groups at once
ci/build_images.sh nginx curl git
ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder

# Build with options
ci/build_images.sh --verbose curl
ci/build_images.sh --arch arm64 nginx
ci/build_images.sh --engine docker nginx
ci/build_images.sh --engine podman --verbose nginx

# Build multiple image groups with options
ci/build_images.sh --verbose nginx curl git
ci/build_images.sh --arch arm64 nginx postgresql

# Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies from registry instead of building (faster for local dev)
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Pull a specific image from registry
ci/build_images.sh --pull nginx/rawhide/default

# Build from CI component name format (used in CI environments)
ci/build_images.sh --component-name curl--rawhide--default

# Build with custom RPMs (for testing modified packages)
ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder

Note: Some images require git submodules initialized (use git init --recurse-submodules when cloning or git submodule update --init if already cloned). When building for foreign architectures, make sure qemu-user-static is available.

Dependency Building

The --build-deps flag builds all dependencies needed for testing an image, but not the image itself. This is used in CI to ensure all required images are available before running tests.

Pulling vs Building Dependencies

For local test development, use --pull with --build-deps to pull pre-built images from the registry instead of building them locally. This is much faster and uses the same dependency collection logic:

# Slow: Build all dependencies locally
ci/build_images.sh --build-deps nginx

# Fast: Pull all dependencies from registry
ci/build_images.sh --build-deps --pull nginx

The --pull flag works without --build-deps too:

# Pull a specific image instead of building it
ci/build_images.sh --pull caddy/rawhide/default

How Dependency Collection Works

When you run ci/build_images.sh --build-deps <image>, it performs a 2-level expansion:

  1. Level 1: Collects forward and reverse dependencies of the specified image

    • Forward dependencies: Images that the specified image’s tests depend on (detected by TEST_IMAGES[...] references in test files)
    • Reverse dependencies: Images that depend on the specified image (filtered by reverse_dependency_tests: true in properties.yml)
  2. Level 2: Collects forward dependencies of the reverse dependencies

  3. Stops: No further expansion (avoids infinite graph traversal)

All dependencies are automatically deduplicated to ensure each image is built exactly once.

Relationship to Testing

The reverse_dependency_tests property in properties.yml affects both building and testing:

  • Build phase (--build-deps): Filters which reverse dependencies to build
  • Test phase (ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps): Filters which reverse dependencies to test

Set reverse_dependency_tests: false for images like curl that are used pervasively but don’t need reverse dependency workflows.

Dependency Building Examples

# Build dependencies for dotnet-runtime-10-0
# Builds: dotnet-sdk-10-0 (forward dependency)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies for dotnet-runtime-10-0 (faster alternative)
# Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Build dependencies for core-runtime
# Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true)
# Plus their forward dependencies
ci/build_images.sh --build-deps core-runtime

# Use --dryrun to see what would be built/pulled without actually doing it
ci/build_images.sh --dryrun --build-deps core-runtime
ci/build_images.sh --dryrun --build-deps --pull core-runtime

Building with custom RPMs

The --local-rpms-dir option enables testing container images with custom-built RPM packages, for iterating on package changes before committing to the RPM repository.

Workflow

  1. Build custom RPMs in the rpms repository (see its documentation for the complete workflow):

    cd ../rpms
    # modify a package
    ci/build_rpms.sh packagename
    # Built RPMs will be in builds/packagename/RPMS/
    
  2. Build container image using the custom RPMs:

    cd ../containers
    ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder
    
  3. Verify the custom package was installed:

    podman run --rm --entrypoint '' registry.access.redhat.com/hi/imagename:latest-builder rpm -qa
    

Builds for affected container tests

Use --build-deps --affected-tests to select dependencies needed by affected container checks across both engines. Testing Farm adds --pull to fetch published dependencies. Supplied Konflux image URL overrides take precedence. Omit --pull to validate unpublished dependency changes from the checkout.

ci/build_images.sh --build-deps --affected-tests --pull core-runtime/hummingbird/default

--affected-tests requires --build-deps. Without it, dependency building retains the full reverse suite for callers that need broader coverage.

4.3.6.2 - run_tests_container.sh

Run tests for image groups using containerized test environment with support for both Docker and Podman engines

Purpose

Run tests for image groups using containerized test environment with support for both Docker and Podman engines.

Usage

Run ci/run_tests_container.sh --help for full usage information.

Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --hermetic           Use hermetic builds (--pull=never, only use prefetched/built images)
                         Default for CI. Without this, missing images are pulled on demand.
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --affected-tests     Run affected tests and checks that reference dependencies
    --dryrun             Print selected tests without invoking container engines
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Engine Selection: Use --engine to specify the container engine (podman or docker). When using Docker, add --setup for automatic Docker-in-Docker environment setup.

Hermetic vs Non-Hermetic Testing:

  • Local development (default): Without --hermetic, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first.
  • CI environment: Use --hermetic to enforce that tests only use prefetched or locally-built images (via --pull=never). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built.

Building and Testing

For local development, you’ll typically want to build the images first, then test them:

# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups

The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.

When working on base images (like core-runtime) that other images depend on, build the base image and its dependencies before running reverse-dependency tests:

# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime

The -p/--pause option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging.

Testing with Specific Image Builds

To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT> environment variables (uppercase, hyphens/dots → underscores, __ separates parts):

# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
  ci/run_tests_container.sh tomcat-10/hummingbird/builder

Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.

Retry Behavior:

  • Failed tests are checked against a list of retriable error patterns
  • If a match is found, the test is automatically retried
  • If the test still fails after all attempts, it’s reported as a normal failure

Examples

# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder

# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default

Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:

  • Find images that depend on it by scanning for TEST_IMAGES[group/distro/variant] references
  • Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting reverse_dependency_tests: false in properties.yml (e.g., curl, which is widely used but typically doesn’t need reverse dependency testing).

Affected dependency tests

Testing Farm uses --affected-tests for single-component jobs. Dependency pulls use ci/build_images.sh --build-deps --affected-tests --pull with the same selector. The requested component runs its complete suite. Dependent variants run only checks, including global checks, that reference the changed component variant or a selected dependency. Unrelated global checks remain covered by the requested component.

Use --dryrun to list runnable checks without invoking container engines. Use --include-reverse-deps without --affected-tests to run full dependent suites.

Run each suite with an isolated container engine. Cleanup preserves resources present at startup and removes resources created during the suite, including resources created concurrently by other processes.

4.3.6.3 - run_tests_k8s.sh

Run K8s tests for image groups using kubectl with support for local development and CI environments

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also run tests that use the specified image variants
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

The --push-image flag:

  • Compares local image digest with remote
  • Skips push if image already exists with same digest
  • Uses port-forward to registry-proxy for pushing
  • Sets TEST_IMAGE to the internal registry reference

Prerequisites:

  • oc login to the target cluster
  • oc project <namespace> to set the target namespace
  • Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Reverse dependency selection

--include-reverse-deps uses ci/internal/image_test_plan.py, shared with the container runner, to select dependent tests individually. It reads Kubernetes commands from tests-k8s.yml and follows shell scripts and sourced fixtures for TEST_IMAGES[group/variant] references. Container-only references do not select Kubernetes tests.

ci/run_tests_k8s.sh --context kind-kind --include-reverse-deps nginx/hummingbird/default

This selects all tests for the requested nginx variant and only dependent tests that use nginx/default in the same distro. For example, Python custom CA tests run, but standalone Python version checks do not. Explicit test selectors remain restricted to the requested test. reverse_dependency_tests: false disables selection of consumers of that image.

--include-reverse-deps runs the planner in single-component mode: reverse dependency tests that need a same-group sibling variant are excluded. Such a test references TEST_IMAGES[<same-group>/variant] for a variant other than the one under test (or is itself defined for a same-group variant), but a per-component run provides only the image under test. The sibling would fall back to a released image — silently testing the changed component against an older sibling — or fail to pull when no release exists. Those combinations run in the group pipelines (pr group), which provide every variant of the group at the exact commit. Cross-group reverse dependencies are still selected, and tests of the requested variant itself are unaffected.

Kubernetes variants lists match exact variant names. Tests without a list apply to all image variants except group. Tests with variants: [group] run as group tests. The planner does not apply container engine or host FIPS filters to Kubernetes definitions, or infer the cluster architecture from the local host.

Environment Variables

Tests have access to these environment variables:

Variable Description
TEST_IMAGE Container image under test (with digest)
TEST_IMAGES Associative array with group/variant image URLs (uses TEST_DISTRO context)
TEST_IMAGES_PATH Path to file containing serialized TEST_IMAGES array
TEST_GROUP The image group being tested
TEST_DISTRO The distro being tested (e.g., rawhide)
TEST_VARIANT The variant being tested (e.g., default)
TEST_VERBOSE Show test command output (true or false)
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (hum-k8s-test=<id>)

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Function Description
test_fail Fail the test with a custom error message

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

# Build locally and push to internal registry
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

  • --kubeconfig pointing to the ephemeral namespace kubeconfig
  • --component-name or --group-component-name for component identification
  • --output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.

4.3.6.4 - retrigger_failed_checks.py

Retrigger failed Konflux CI checks for a given GitLab merge request

Purpose

Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running.

Usage

Run ci/retrigger_failed_checks.py --help for full usage information.

Usage: ci/retrigger_failed_checks.py <MR_URL> [OPTIONS]

OPTIONS:
    -h, --help              show this help message and exit
    --dry-run               just print what comments would be posted
    --token-path, -t PATH   path to file containing GitLab API token

Authentication (in order of precedence):
  1. --token-path: Path to file containing GitLab API token (highest priority)
  2. GITLAB_TOKEN_PATH: Environment variable with path to token file
  3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback)

Behavior

The script will:

  1. Extract failed pipeline runs from the commit statuses on the MR
  2. Generate /retest {pipeline-name} commands for each failed run
  3. Post the retest commands as MR comments (unless –dry-run)
  4. Wait for the pipeline runs to start (unless –dry-run)

Examples

# Using --token-path option (highest priority)
ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN_PATH environment variable
GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN environment variable (legacy)
GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Dry-run mode (show what would be done, don't post or wait)
ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234

4.3.6.5 - gitlab_sync.py

Sync a generated file to an external GitLab repository via merge request, with automatic merge wait, pipeline retry, and preemption support

Purpose

Sync a file from a public source URL to a target GitLab repository via merge request. The script always waits for the MR to be merged and for the post-merge pipeline to complete before returning. It is invoked by infrastructure CI jobs to keep external repos (pyxis-repo-configs, konflux-release-data) in sync with generated files from the source repos.

Usage

Run gitlab_sync.py --help for full usage information.

usage: gitlab_sync.py [-h] --source-url SOURCE_URL
                      --target-project TARGET_PROJECT
                      --target-file TARGET_FILE --sync-branch SYNC_BRANCH
                      --mr-title MR_TITLE --gitlab-url GITLAB_URL
                      [--merge-timeout MERGE_TIMEOUT]
                      [--post-merge-timeout POST_MERGE_TIMEOUT]
                      [--mr-description MR_DESCRIPTION] [--squash-on-merge]
                      [--dry-run]

Authentication:
  GITLAB_TOKEN    Environment variable with GitLab API token (required)

Monitoring:
  SENTRY_DSN      Sentry DSN for alerting on failures (optional)

How It Works

  1. Fetch source from the public --source-url (with HTTP retries)
  2. Determine comparison ref: use the sync branch if it exists, otherwise the project’s default branch
  3. Compare source content against the comparison ref (stripped whitespace)
  4. Early exit if content matches and the comparison ref is the default branch (nothing to deploy)
  5. Commit the source to the sync branch using the GitLab Commits API with force: true (creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits.
  6. Create or update MR targeting the default branch
  7. Self-approve the MR (best-effort; continues if already approved)
  8. Post takeover comment with CI_JOB_ID for preemption tracking
  9. Wait for MR merge (polling every 30s, up to --merge-timeout)
  10. Wait for post-merge pipeline to succeed (up to --post-merge-timeout)

Exit Codes

Code Meaning
0 Success (MR merged) or no changes needed
1 Failure (timeout, pipeline failure, unrecoverable error)
42 Preempted by a newer CI job managing the same MR

Preemption

When multiple CI jobs target the same sync branch, the script uses MR comments to coordinate. Each job posts a comment containing its CI_JOB_ID (monotonically increasing within a GitLab instance). Before each merge attempt, the script checks for comments with a higher job ID. If found, it yields by exiting with code 42.

The caller should handle exit 42 to stop processing (a newer job will handle all remaining syncs). When running outside CI (CI_JOB_ID not set), preemption is disabled.

Error Recovery

  • Pipeline failure: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted.
  • Merge conflict: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert.
  • Post-merge pipeline failure: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA).
  • Draft MR: Keeps polling without attempting to merge (allows manual intervention).

Examples

# Dry run: show diff without modifying anything
GITLAB_TOKEN=$TOKEN gitlab_sync.py \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --dry-run

# Sync Pyxis config (requires squash merge)
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --squash-on-merge

# Sync RPM RPA to konflux-release-data
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \
  --target-project "releng/konflux-release-data" \
  --target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \
  --sync-branch "hummingbird/sync-rpms-rpa" \
  --mr-title "Update hummingbird RPM ReleasePlanAdmission"

Preemption wrapper for sequential syncs in a CI job:

gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge
rc=$?
if [ "$rc" -eq 42 ]; then
  echo "Preempted by newer job, stopping"
  exit 0
elif [ "$rc" -ne 0 ]; then
  exit "$rc"
fi
# Proceed to next sync only on success
gitlab_sync.py --gitlab-url ... --source-url ...

Environment Variables

Variable Required Description
GITLAB_TOKEN Yes API token with Developer+ access on the target project
CI_JOB_ID No Set automatically in CI; enables preemption detection
SENTRY_DSN No Sentry DSN for failure alerting

Development

The script and gitlab-ci tool image are maintained in the Hummingbird tools repository, not this repository. Make implementation and test changes there.

4.4 - K8s Test Pipeline

Background documentation sourced from the K8s test pipeline repository, covering pipeline design, test format, and EaaS debugging for Kubernetes integration tests.

4.4.1 - Pipeline Design

Design guidelines and architecture of the K8s test pipeline.

Task/Step Overview

Task Type Steps Results Retries Timeout
check-for-tests Inline taskSpec write-snapshot, resolve-component, fetch-source, check-for-tests, write-results HAS_TESTS, TEST_OUTPUT 2 10m
provision-namespace External bundle (task-eaas-provision-space) (managed by task) secretRef 2 10m
run-tests Inline taskSpec write-snapshot, resolve-component, fetch-source, run-tests, write-results TEST_OUTPUT 2 24h
fail-pipeline-on-test-failure Inline taskSpec (finally) check-results — 0 10m

Both check-for-tests and run-tests independently fetch source — each task gets its own emptyDir volume, so no state carries across tasks.

Design Guidelines

1. Filesystem over results

Pass data between steps via files on a shared emptyDir, not Tekton results. Tekton results have a 4KB termination message limit (tektoncd/pipeline#4060) and large $(params.*) substitutions hit ARG_MAX.

Each task mounts an emptyDir at /workdir. Steps communicate via files: snapshot.json, component, image_names, src/, kubeconfig, result, error. Zero $(params.*) references remain inside script: blocks — all param values are passed via env vars or written to /workdir.

2. Distinguish retryable from non-retryable errors

Use exit 1 for transient failures (network, registry) that benefit from retry. Use exit 0 + a sentinel file for permanent failures (bad data) to prevent wasted retries.

Non-retryable errors write Konflux-format JSON to /workdir/error and exit 0. Subsequent steps check for this sentinel and skip. The write-results step copies it to TEST_OUTPUT. Tasks default to retries: 2, but fail-pipeline-on-test-failure has retries: 0 — test failures should not be retried.

3. Fail fast

Every shell script enables set -euo pipefail so unexpected command failures surface immediately rather than cascading silently.

4. Separate data from status

Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration.

Konflux reads TEST_OUTPUT results as data but does not fail the pipeline based on them. The fail-pipeline-on-test-failure finally-task reads both TEST_OUTPUT values and exit 1 on non-SUCCESS. This is needed because GitLab PAC integration requires the pipeline itself to fail for MR status reporting.

5. Gate expensive work behind cheap checks

Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do.

provision-namespace and run-tests are gated on check-for-tests.results.HAS_TESTS == "true" via Tekton when expressions. If no tests-k8s.yml exists, the pipeline completes with just check-for-tests, skipping EaaS provisioning. This is the common case for components without K8s tests.

6. Make failures reproducible

Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally.

Stderr from cosign/jq/oras is not swallowed. The full run_tests_k8s.sh command line is logged.

7. Set task timeouts deliberately

Tekton applies a global default TaskRun timeout (typically 1h from default-timeout-minutes in config-defaults) to every task that has no explicit timeout, regardless of pipeline-level timeouts. This means omitting a task timeout does NOT let the pipeline timeout control the bound — the task gets killed at 1h.

Every task in this pipeline has an explicit timeout. Bounded operations (check-for-tests, provision-namespace, fail-pipeline-on-test-failure) use 10m. The variable-duration run-tests task uses 24h so the pipeline-level timeout (controlled per-application via ITS annotations like test.appstudio.openshift.io/pipeline_timeout) is the effective bound, following the Testing Farm pipeline pattern.

8. Define shared logic once

When multiple tasks need identical steps, use YAML anchors to define them once and reuse via aliases. Tekton pipelines have no function/import mechanism, so anchors are the only DRY tool available within a single pipeline file. Without this, step logic drifts between tasks when one copy is updated but not the other.

write-snapshot, resolve-component, and fetch-source are defined as anchors (&write-snapshot-step, etc.) in check-for-tests and reused via *write-snapshot-step in run-tests. Both tasks independently execute the same steps to fetch and process source, staying in sync.

Trusted Artifacts Source Fetch

Source code is fetched via the Konflux Trusted Artifacts chain:

  1. cosign download attestation — retrieves build attestations (handles multiple JSONL attestations)
  2. Extract SOURCE_ARTIFACT from .predicate.buildConfig.tasks[].results[]
  3. oras blob fetch — downloads the source tarball
  4. tar -xzf — extracts to /workdir/src

The oci: prefix is stripped from the artifact URI. --no-same-owner avoids permission issues during extraction.

4.4.2 - Test Format

Pipeline-level test discovery, environment variables, parameters, and group snapshot handling.

Test Discovery

The pipeline delegates test discovery to each repository’s own test runner. It calls ci/run_tests_k8s.sh --has-tests with the component names from the snapshot and reads true/false from stdout to determine whether any component has K8s tests defined. This keeps test file location knowledge in the repository that owns the tests, not in the pipeline.

The actual tests-k8s.yml format is opaque to the pipeline — it is passed to the repository’s test runner script, which defines the structure and semantics. See the respective repository documentation for the YAML format.

Environment Variables

The test runner receives these variables from the pipeline:

Variable Description
TEST_IMAGE Container image under test (with digest)
TEST_GROUP Image group name (e.g., git, postgresql)
TEST_VARIANT Variant being tested (e.g., default, builder)
KUBECONFIG Path to kubeconfig for the ephemeral namespace
TEST_IMAGES Bash associative array of all images in the snapshot
TEST_IMAGES_PATH Path to source TEST_IMAGES in external scripts
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (e.g., hum-k8s-test=<id>)
test_fail Shell function — call test_fail "message" to fail a test

Pipeline Parameters

Parameter Description Default
SNAPSHOT Konflux snapshot JSON or snapshot name (with snapshot-param-as-name annotation) —
EXTRA_SINGLE_COMPONENT_ARGS Additional arguments appended to the test runner for single-component (PR) snapshots --include-reverse-deps

Group Snapshot Handling

For group snapshots (post-merge), the pipeline exports:

  • IMAGE_NAMES — space-separated list of all component names
  • IMAGE_URL_<COMPONENT> — image URL for each component (name uppercased, dashes replaced with underscores)

The test runner script is called once with multiple --group-component-name arguments. Test discovery checks all components for tests-k8s.yml. Source is fetched from the first component’s build attestation (shared source tree).

4.4.3 - EaaS and Debugging

How the pipeline uses Konflux Environment as a Service, and how to debug test failures.

Environment as a Service (EaaS)

EaaS is a Konflux-managed service that provisions ephemeral OpenShift namespaces on member clusters for integration testing.

Provisioning Flow

  1. The provision-namespace task creates an eaas.konflux-ci.dev/v1alpha1 Namespace claim, owned by the PipelineRun.
  2. The EaaS controller allocates a namespace on a member cluster (currently kflux-prd-es01) and writes a scoped kubeconfig into a Kubernetes Secret.
  3. The run-tests step reads the kubeconfig from the Secret to get cluster access.
  4. When the PipelineRun is deleted, the owner reference triggers garbage collection of the claim, namespace, and Secret.

The member cluster may change — it is dynamically assigned by the EaaS controller.

Debugging Test Failures

Using Kubearchive for Historical PLRs

Kubearchive stores historical PipelineRun, TaskRun, and Pod data. The containers repo includes a helper script at ci/internal/k8s_helper.py.

Example: find PLRs for a specific PR:

cd /path/to/rprm-containers
python3 -c "
from ci.internal import k8s_helper
h = k8s_helper.K8sHelper('https://konflux-ui.apps.kflux-prd-rh03.nnv1.p1.openshiftapps.com/ns/hummingbird-tenant/')
endpoint = h._tekton_endpoint('pipelineruns')
items = h._fetch_from_both_sources(endpoint, 'pac.test.appstudio.openshift.io/pull-request=421,test.appstudio.openshift.io/scenario=tools-k8s-test')
for plr in items:
    print(plr['metadata']['name'], plr['status']['conditions'][0]['reason'])
"

Useful label selectors:

  • test.appstudio.openshift.io/scenario=<scenario-name> — filter by ITS name
  • pac.test.appstudio.openshift.io/pull-request=<number> — filter by MR
  • appstudio.openshift.io/component=<component> — filter by component

Accessing the EaaS Namespace During a Live PLR

While a PipelineRun is running, you can access the ephemeral namespace:

  1. Find the provision secret name from the provision-namespace task results (visible in the PLR status or Konflux UI).

  2. Extract the kubeconfig:

    kubectl get secret <secret-name> -n hummingbird-tenant \
      -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/eaas-kubeconfig
    
  3. Use it to inspect the namespace:

    export KUBECONFIG=/tmp/eaas-kubeconfig
    kubectl get pods
    kubectl get events --sort-by='.lastTimestamp'
    kubectl logs <pod-name>
    

Finding Which Cluster EaaS Uses

To determine the current EaaS member cluster, extract the server URL from a kubeconfig provisioned by EaaS:

kubectl get secret <secret-name> -n hummingbird-tenant \
  -o jsonpath='{.data.kubeconfig}' | base64 -d | grep server

As of this writing, the EaaS member cluster is kflux-prd-es01.1ion.p1.openshiftapps.com. This may change as the EaaS controller dynamically assigns clusters.

4.5 - Tools repository

Background documentation sourced from the tools repository, covering infrastructure components like event forwarders, the message bus, and monitoring tools.

4.5.1 - Message Bus Architecture

The Hummingbird message bus is an event-driven architecture built on AWS SNS. Events from multiple sources flow through a central topic, enabling subscribers to filter and process only the events they need.

Architecture

flowchart TD
    subgraph Publishers
        CVE[CVE Analysis]
        GL[GitLab Webhooks]
        JIRA[Jira Automation]
        K8S[Kubernetes Clusters]
    end

    subgraph MessageBus [Message Bus]
        SNS[(SNS Topic)]
    end

    subgraph Archiver
        ARCH[SNS S3 Archiver]
        S3[(S3 Bucket)]
    end

    subgraph StatusDB [Status Database]
        SQS1[SQS Queue]
        WORKER[hummingbird-status]
        PG[(PostgreSQL)]
    end

    subgraph ConsumerQueue [Consumer Queue]
        SQS2[SQS Queue]
    end

    subgraph Catalog [Container Catalog]
        SYNC[SyncFunction]
        DDB[(DynamoDB)]
    end

    subgraph Consumers
        CONSUMER[Event Consumer]
    end

    CVE -->|CVE-agent sweep request| SNS
    GL -->|gitlab-event-forwarder| SNS
    JIRA -->|jira-event-forwarder| SNS
    K8S -->|kubernetes-event-forwarder| SNS
    SNS --> ARCH
    ARCH --> S3
    SNS --> SQS1
    SQS1 --> WORKER
    WORKER --> PG
    SNS -->|"FilterPolicy\nkind=Release"| SYNC
    SYNC -->|fetch manifests| Registry[(OCI Registry)]
    SYNC -->|read/write| DDB
    SNS --> SQS2
    SQS2 -->|new events| CONSUMER
    S3 -.->|historic events| CONSUMER
    PG -.->|aggregated status| CONSUMER

Components

Component Role Description Staging
hummingbird-events-topic Infrastructure Central SNS topic for all events Yes
hummingbird-cve-analysis Publisher Publishes requests for CVE-agent sweeps No
hummingbird-cve-agent Subscriber Consumes sweep requests and investigates eligible Jira tickets No
gitlab-event-forwarder Publisher Receives GitLab webhooks, publishes to SNS Yes
jira-event-forwarder Publisher Receives Jira Automation webhooks, publishes to SNS Yes
kubernetes-event-forwarder Publisher Watches K8s resources, publishes changes to SNS Yes
sns-s3-archiver Subscriber Archives all events to S3 for querying/replay Yes
hummingbird-status Subscriber Ingests events to PostgreSQL for structured queries Yes
pipeline-timing Subscriber Stores pipeline timing in its own schema and queue Yes
hummingbird-agent Subscriber Processes events to drive automated workflows Yes
workqueue-service Subscriber Queues and dispatches work items from events Yes
container-catalog Subscriber Incrementally syncs image metadata to DynamoDB on Release events No

For subscriber rows, the Staging column describes running consumers. Pipeline Timing queues and subscriptions can be provisioned in both environments before the workers run; their presence does not mark the service as deployed.

Message Format

All messages include standard attributes for filtering:

Attribute Description Examples
source Event origin cve-analysis, gitlab, jira, kubernetes
event_type Type of event cve_agent_sweep_requested, push, merge_request, pipeline, …

Additional attributes vary by source - see individual publisher docs for details.

GitLab Events

Published by gitlab-event-forwarder:

Attribute Description Example
project_path Full project path redhat/hummingbird/containers
group_path Full group path redhat/hummingbird

When configured, Pipeline Hooks use event_type: pipeline; Job Hooks use event_type: build.

CVE-agent Sweep Requests

Published by hummingbird-cve-analysis:

Attribute Value Description
source cve-analysis Identifies the publisher
event_type cve_agent_sweep_requested Requests a CVE-agent sweep (wake-up request)

The SNS Message is plain JSON, with no content_encoding attribute. The subscription uses SNS envelope delivery, so the SQS body contains the SNS notification envelope and its Message field contains the JSON above. Publish errors are logged but do not fail the completed analysis cycle; a later enabled cycle can publish another request.

Jira Events

Published by jira-event-forwarder:

Attribute Description Example
project_key Jira project key HUM
issue_type Issue type name Story, Bug

The event_type attribute is set from the X-Jira-Event-Type request header, which each Jira Automation rule hardcodes for its trigger type (e.g., issue_created, issue_updated).

Kubernetes Events

Published by kubernetes-event-forwarder:

Attribute Description Example
cluster Logical kubeconfig cluster entry name konflux-rh03
namespace Object namespace production
kind Resource kind PipelineRun, TaskRun, Snapshot, Release
api_version API version tekton.dev/v1, appstudio.redhat.com/v1alpha1
object_name Resource name nginx-7d8c4c9d6f

Published kinds depend on the deployed forwarder configuration. TaskRun source activation is separate rollout work. The forwarder obtains cluster from the selected kubeconfig context’s context.cluster value, not the API server URL.

ExternalSyncQueue

SQS queue fed by ExternalReleaseFunction (triggered by EventBridge schedule every 6 hours) and consumed by ExternalSyncFunction (one message per invocation, BatchSize: 1, ReportBatchItemFailures) for registry tag resolution and metadata sync.

Message body:

{ "registry": "docker.io/library/nginx" }

Same-account SQS with no resource policy — the sender’s Lambda execution role carries the sqs:SendMessage policy. Failed messages redrive to ExternalSyncDLQ after 3 receives.

ExternalIndexQueue

SQS queue fed by ExternalSyncFunction and consumed by ExternalIndexFunction (one message per invocation, BatchSize: 1, ReportBatchItemFailures) for Syft SBOM generation. One message per digest, emitted after each TAGS write for every non-superseded digest of the registry (registries with no changes at all emit nothing).

Message body:

{
  "registry": "docker.io/library/python",
  "canonicals": ["3.15"],
  "digest": "sha256:…",
  "architectures": ["amd64", "arm64"]
}

Same-account SQS with no resource policy — the sender’s Lambda execution role carries the sqs:SendMessage policy. Failed messages redrive to ExternalIndexDLQ after 3 receives; DLQ entries are not retried automatically (see the container-catalog documentation for recovery).

Subscription Filtering

SNS filter policies enable subscribers to receive only relevant events:

{
  "source": ["gitlab"],
  "event_type": ["push", "merge_request"]
}
{
  "source": ["jira"],
  "event_type": ["issue_created", "issue_updated"]
}
{
  "source": ["kubernetes"],
  "kind": ["Deployment"],
  "event_type": ["MODIFIED"]
}

Kubernetes Release events (used by container-catalog sync Lambda):

{
  "kind": ["Release"]
}

See hummingbird-events-topic for subscription setup instructions.

Event Flow Example

  1. Developer pushes to GitLab repository
  2. GitLab sends webhook to gitlab-event-forwarder Lambda
  3. Lambda validates token, extracts metadata, publishes to SNS
  4. SNS delivers to all matching subscribers:
    • sns-s3-archiver stores event in S3
    • Event consumers process new events in real-time

Archive keys

The S3 archiver writes one deterministic key for each SNS message. Duplicate delivery rewrites the same object. GitLab pipeline and job identities include the project and their numeric IDs; all Kubernetes identities include logical cluster, namespace, and name. URL-valued cluster attributes from older forwarders retain their name-only archive identities.

See SNS S3 Archiver for the exact key format and the machine-readable archive-key contract used by the archiver and GitLab forwarder tests.

Consuming Events

Consumers can receive events from multiple sources:

  • New events: Subscribe to SNS topic for real-time processing
  • Historic events: Download from S3 archive for replay or catch-up
  • Structured queries: Query PostgreSQL via hummingbird-status for pipeline status, component state, and cross-referenced data

The S3 archive enables consumers to bootstrap state, then switch to live SNS events. Hummingbird Status provides a queryable view of its current pipeline status data. Pipeline Timing consumes its own SQS queue and stores selected reporting fields in the pipeline_timing schema. Follow a domain row’s execution identity and parentage to query creation/start/completion times and overall status. After historical creation-timestamp recovery, the legacy event catalogue and pointer columns are removed. Timing ingestion maintains only reporting state and ordering metadata; the independent S3 archiver continues unchanged. Archived notifications can be searched using publication time and resource identity, subject to archive retention, without a permanent database row for every message.

Replaying Events

The sns-s3-archiver stores complete SNS records with decoded payloads, enabling easy replay:

# Download archived events
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/

# Browse events
zcat ./local-events/2025/12/13/12/34/*.json.gz | jq
import gzip
import json
from pathlib import Path

# Replay archived events to handler
for path in sorted(Path("./local-events").rglob("*.json.gz")):
    with gzip.open(path, "rt") as f:
        sns_record = json.load(f)
    event = {"Records": [{"Sns": sns_record}]}
    my_handler.lambda_handler(event, None)

See sns-s3-archiver documentation for details on storage format and the _decode_message pattern for handlers.

Staging Environment

Production and staging use independent SNS topics. Each environment has its own publishers, archiver, and consumer queues so that changes can be tested in staging before production deployment.

flowchart TD
    GL["GitLab Projects\n(dual webhook delivery)"]
    JIRA["Jira Automation\n(dual webhook delivery)"]
    CVE["CVE Analysis\n(CronJob)"]
    K8S["Konflux Cluster\n(K8s resources)"]

    subgraph production ["Production"]
        CVE_P["hummingbird-cve-analysis\n(CronJob)"]
        CVE_Q_P["CVE-agent SQS queue"]
        CVE_A_P["hummingbird-cve-agent"]
        GEF_P["gitlab-event-forwarder\n(Lambda)"]
        JEF_P["jira-event-forwarder\n(Lambda)"]
        KEF_P["kubernetes-event-forwarder\n(K8s pod)"]
        SNS_P["SNS: arr-hummingbird-prod-events"]
        prodConsumers["consumers\n(status, agent, workqueue,\narchiver, etc.)"]

        CVE_P --> SNS_P
        SNS_P -->|sweep request filter| CVE_Q_P
        CVE_Q_P --> CVE_A_P
        GEF_P --> SNS_P
        JEF_P --> SNS_P
        KEF_P --> SNS_P
        SNS_P --> prodConsumers
    end

    subgraph staging ["Staging"]
        CVE_S["hummingbird-cve-analysis\n(CronJob)"]
        CVE_Q_S["CVE-agent SQS queue"]
        CVE_A_S["hummingbird-cve-agent"]
        GEF_S["gitlab-event-forwarder\n(Lambda)"]
        JEF_S["jira-event-forwarder\n(Lambda)"]
        KEF_S["kubernetes-event-forwarder\n(K8s pod)"]
        SNS_S["SNS: arr-hummingbird-staging-events"]
        stagConsumers["consumers\n(status, agent, workqueue,\narchiver)"]

        CVE_S --> SNS_S
        SNS_S -->|sweep request filter| CVE_Q_S
        CVE_Q_S --> CVE_A_S
        GEF_S --> SNS_S
        JEF_S --> SNS_S
        KEF_S --> SNS_S
        SNS_S --> stagConsumers
    end

    GL --> GEF_P
    GL --> GEF_S
    CVE --> CVE_P
    CVE --> CVE_S
    JIRA --> JEF_P
    JIRA --> JEF_S
    K8S --> KEF_P
    K8S --> KEF_S

How it works

  • Duplicate webhook delivery: GitLab sends every event to both gitlab-event-forwarder.hummingbird-project.io (prod) and gitlab-event-forwarder.staging.hummingbird-project.io (staging). Each forwarder publishes to its own SNS topic.
  • Duplicate webhook delivery: Jira Automation sends every event to both jira-event-forwarder.hummingbird-project.io (prod) and jira-event-forwarder.staging.hummingbird-project.io (staging). Each forwarder publishes to its own SNS topic.
  • Dual Kubernetes watches: Both prod and staging kubernetes-event-forwarder pods watch the same Konflux cluster resources and publish to their respective topics.
  • CVE-agent sweep requests: Each analysis CronJob publishes to its environment’s topic only when its sweep switch is enabled. Attribute filters route the request to that environment’s dedicated agent queue.
  • Consumer isolation: Staging consumers (hummingbird-status, pipeline-timing, hummingbird-agent, workqueue-service) have separate SQS queues subscribed to the staging topic. Pipeline Timing shares Hummingbird Status’s staging events database and owns only the pipeline_timing schema.
  • Staging archiver: A separate sns-s3-archiver subscribes to the staging topic and archives to its own S3 bucket with a shorter retention period than production.

4.5.2 - Alloy CloudWatch

An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana Alloy’s prometheus.exporter.cloudwatch component. Grants access to CloudWatch Metrics APIs so selected AWS service metrics (Lambda, SQS, SNS, DynamoDB) can be ingested into Mimir for alerting.

Features

  • Read-Only IAM User: Dedicated user for Alloy CloudWatch metrics exporter
  • CloudWatch Metrics: GetMetricData, GetMetricStatistics, ListMetrics
  • Resource Discovery: tag:GetResources for filtering by app-code=RPRM-001
  • Account Discovery: sts:GetCallerIdentity and ec2:DescribeRegions for YACE auto-discovery

Architecture

Single SAM template that creates:

  1. IAM User - Read-only service account (${ResourcePrefix}-user)
  2. IAM Policy - CloudWatch Metrics read, tag and region discovery, STS caller identity

The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Alloy Hub pod on MPP via a Kubernetes Secret.

flowchart LR
  subgraph aws [AWS]
    iam["IAM User\n(read-only)"]
    cwMetrics["CloudWatch Metrics"]
  end
  subgraph mpp [MPP Hub]
    alloy["Alloy Hub"]
    mimir["Mimir"]
  end
  alloy -->|"GetMetricData"| cwMetrics
  alloy -->|"remote_write"| mimir
  alloy -.->|"credentials from Vault"| iam

IAM Permissions

All permissions are read-only with Resource: "*" (CloudWatch does not support resource-level restrictions for most read operations).

Category Actions
Metrics GetMetricData, GetMetricStatistics, ListMetrics
Discovery ec2:DescribeRegions, tag:GetResources, sts:GetCallerIdentity

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd alloy-cloudwatch
sam deploy --guided   # First deployment
sam deploy            # Subsequent deployments

After deployment, create an access key for the IAM user and store it in Vault using cki_tools.credentials.manager.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.3 - Deployment Bot

An AWS CloudFormation/SAM stack that provisions IAM resources for automated AWS deployments from GitLab CI. Creates a service account with permissions to deploy SAM applications including Lambda functions, API Gateway, SNS topics, and related resources.

Features

  • IAM Service Account: Dedicated user for CI/CD pipelines
  • Scoped Permissions: Least-privilege access for SAM deployments
  • Resource Naming: All managed resources use consistent prefix

Architecture

Single SAM template that creates:

  1. IAM User - Service account for deployments (${ResourcePrefix}-deployment-bot)
  2. IAM Policy - Permissions for SAM deployment operations

Prerequisites

  • AWS CLI configured with admin credentials
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Deploy using containerized AWS SAM CLI:

cd deployment-bot
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely in HashiCorp Vault.

SAM Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
AdditionalResourcePrefix Additional prefix this bot can manage (e.g. staging) ""

Resource naming: IAM resources follow {ResourcePrefix}-deployment-bot pattern (e.g., myapp-prod-deployment-bot). Use AdditionalResourcePrefix to allow the same bot to manage resources in another environment.

Permissions

The deployment bot has permissions to manage:

Service Scope
CloudFormation Stacks matching ${ResourcePrefix}-*, SAM bootstrap stack
S3 SAM CLI managed buckets
Lambda Functions matching ${ResourcePrefix}-*
API Gateway All REST APIs and domains
SNS Topics matching ${ResourcePrefix}-*
IAM Users/policies/roles matching ${ResourcePrefix}-*
CloudWatch Log groups for managed Lambda functions
Route53 Record changes (for custom domains)
ACM Certificate management

Usage

Bootstrap (One-Time)

The deployment bot must be deployed first using admin credentials:

# With admin AWS credentials
cd deployment-bot
sam build
sam deploy --guided

Create access keys for the new IAM user and store in HashiCorp Vault.

Automated Deployments

Once bootstrapped, the deployment bot credentials are used by aws_deploy.sh in the infrastructure repository for all subsequent SAM deployments:

export PROJECT_NAME=gitlab-event-forwarder
./aws_deploy.sh

GitLab CI runs this automatically via the aws matrix job.

Development

This project contains only a SAM template with no application code.

Security

  • Access keys should be stored in HashiCorp Vault
  • Permissions are scoped to resources with the configured prefix
  • Route53 and ACM have broader permissions due to API limitations

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.4 - Grafana CloudWatch

An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana’s CloudWatch datasource plugin. Grants access to CloudWatch Logs and Metrics APIs so Lambda execution logs and AWS service metrics are browsable directly in Grafana.

Features

  • Read-Only IAM User: Dedicated user for Grafana CloudWatch datasource
  • CloudWatch Logs: Browse Lambda and AWS service log groups via Logs Insights
  • CloudWatch Metrics: Browse metrics across all AWS namespaces (Lambda, SQS, SNS, DynamoDB, etc.)
  • Resource Discovery: tag:GetResources and ec2:DescribeRegions for Grafana plugin autodiscovery

Architecture

Single SAM template that creates:

  1. IAM User - Read-only service account (${ResourcePrefix}-user)
  2. IAM Policy - CloudWatch Logs read, Metrics read, tag and region discovery

The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Grafana pod on MPP via a Kubernetes Secret.

flowchart LR
  subgraph aws [AWS]
    iam["IAM User\n(read-only)"]
    cwLogs["CloudWatch Logs"]
    cwMetrics["CloudWatch Metrics"]
  end
  subgraph mpp [MPP Hub]
    grafana["Grafana\n(CloudWatch datasource)"]
  end
  grafana -->|"Logs Insights API"| cwLogs
  grafana -->|"Metrics API"| cwMetrics
  grafana -.->|"credentials from Vault"| iam

IAM Permissions

All permissions are read-only with Resource: "*" (CloudWatch does not support resource-level restrictions for most read operations).

Category Actions
Logs DescribeLogGroups, DescribeLogStreams, GetLogEvents, FilterLogEvents, StartQuery, StopQuery, GetQueryResults, DescribeQueries
Metrics GetMetricData, GetMetricStatistics, ListMetrics, DescribeAlarms, DescribeAlarmsForMetric
Discovery ec2:DescribeRegions, tag:GetResources

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd grafana-cloudwatch
sam deploy --guided   # First deployment
sam deploy            # Subsequent deployments

After deployment, create an access key for the IAM user and store it in Vault using cki_tools.credentials.manager.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.5 - Hummingbird AI

hummingbird-ai is a synchronous Python library used in-process by Hummingbird agents to call Gemini and Anthropic models through normalized interfaces.

Features

  • Shared Gemini API/Vertex AI and Anthropic Vertex AI adapters.
  • Model construction, regional selection, token pricing, and usage accounting.
  • Retryable model-call handling and Prometheus request/token/cost metrics.
  • HTTP sessions and Google Application Default Credentials authentication.

The library is not a service and has no deployment of its own. Model adapter instances are scoped to individual conversations. Callers provide bounded concurrency for independent work.

Model interface

hummingbird_ai.models defines the ModelAdapter protocol and normalized ToolDef, ToolCall, Usage, ModelResponse, and ModelError types. Adapters accept provider-neutral tool definitions and return normalized responses. Their make_user_content() and make_tool_responses() methods create provider-specific conversation content; callers keep contents opaque.

Adapters also implement to_canonical() and from_canonical() for persisted conversation history. The canonical message format lets an agent resume a session with a different provider without storing either provider’s native wire format.

Provider adapters

Gemini

GeminiModel supports the direct Gemini API with an API key and Vertex AI with Google credentials. It translates the normalized tool and response types to Gemini REST JSON. Gemini’s implicit prompt cache is reflected in Usage.cache_read_tokens from cachedContentTokenCount.

Anthropic on Vertex AI

AnthropicVertexModel uses Vertex AI rawPredict. It handles Anthropic’s assistant/user turn alternation and associates tool results with the conversation-local tool-use IDs. For prompt caching it adds sliding cache_control breakpoints to shallow copies of messages; the original conversation history is not mutated. Transient messages can use ephemeral so warning text is excluded from the cache write prefix.

When Anthropic web search is enabled, the adapter adds the server-side search tool. Search blocks remain opaque to the agent tool loop and survive canonical session serialization. The adapter also follows pause_turn continuations and merges content and usage from those provider-side calls into the returned response.

Provider wire formats

The agent uses normalized types; adapters translate them to provider payloads:

Normalized data Gemini Anthropic on Vertex AI
System instruction systemInstruction top-level system string
Tool definitions functionDeclarations with JSON Schema tools with input_schema
Tool calls functionCall parts tool_use content blocks
Tool results functionResponse parts tool_result content blocks in a user turn

Gemini API-key requests use the direct Gemini endpoint; Vertex requests use Google authentication and a configured region. Anthropic requests use Vertex AI rawPredict. The adapters keep provider-specific message normalization inside generate(); the agent appends the returned raw_content without interpreting it.

Canonical sessions

Adapters convert their native conversation content to and from the shared canonical message format at persistence boundaries. Anthropic thinking and server-tool blocks are preserved as opaque pass-through fields and restored in their original order. This allows sessions saved by one provider to resume on another while keeping the agent loop independent of provider payload shapes. Extended thinking blocks in assistant responses are part of the cached prefix. They do not break cache hits when the following user message contains only tool_result blocks.

Prompt caching

Gemini Vertex AI prompt caching is implicit. The adapter reads cachedContentTokenCount from usage metadata and reports it as cache_read_tokens; it adds no cache annotations to requests.

Anthropic Vertex caching uses explicit cache_control: {"type": "ephemeral"} annotations. The adapter uses two of the four available breakpoints on messages: B2 writes the latest prefix and B1 reads the previous prefix. The prefix hash includes prior tools and system instructions, so extra breakpoints on those components are redundant. Breakpoints are added to shallow copies so the agent-owned history is unchanged. Ephemeral turns keep warning text out of the cache: B1 still reads the last stable prefix, while B2 and the tracked write position advance only to the last stable message before the transient tail. If there is no stable message to annotate, no breakpoint is moved.

Vertex AI Anthropic prompt caches have a five-minute TTL refreshed on a hit. Cache writes use 1.25 times the base input-token rate and reads use 0.10 times that rate. Minimum cacheable prefixes are 1,024 tokens for Sonnet/Opus and 4,096 for Haiku. A newly constructed adapter starts without a prior breakpoint position, so the first resumed call writes a fresh prefix before later calls can read it.

T1:     U1·B2                                       prev=0
T2:     U1·B1  M1  U2·B2                            prev=2
T3(E):  U1  M1  U2·B1  M2·B2  U3+E                  prev=3
T4(E):  U1  M1  U2  M2·B1  U3  M3·B2  U4+E          prev=5
T5:     U1  M1  U2  M2  U3  M3·B1  U4  M4  U5·B2    prev=8

Calls, retries, and accounting

hummingbird_ai.calls.generate_with_retry() owns model-call retry and usage accounting. A caller selects either an elapsed-time retry budget or a maximum attempt count. It can also pass response validation into the same attempt loop, so validation failures do not multiply a caller’s transport retry limit. Provider-specific HTTP status handling remains in each adapter.

Each returned ModelResponse contributes its input, output, cache-read, and cache-creation tokens and estimated cost exactly once, even when caller-side validation rejects that response. The library emits shared Vertex request, token, and cost counters while callers retain workflow-specific error handling and summaries.

Pricing is kept in models.MODEL_PRICING as per-million-token rates for input, output, cache reads, and cache creation. models.estimate_cost() returns an estimated USD cost or None for an unpriced model.

HTTP and authentication

hummingbird_ai.http.new_session() creates requests sessions with configurable transport retries. Model adapters disable transport-level retries for 429 so that rate-limit failures use the model-call retry policy. VertexAuth uses Google Application Default Credentials and resolves the project from an explicit argument, GOOGLE_CLOUD_PROJECT, then ADC.

requests.Timeout is caught before requests.ConnectionError in adapters because ConnectTimeout is a subclass of both. The urllib3 retry adapter does not retry POST requests (non-idempotent), so all transport errors from model calls propagate to the generate_with_retry retry layer.

Prerequisites

  • Python 3.11 or later.
  • Google credentials and a project for Vertex AI models; Gemini API key mode is also supported.

Installation

The library is installed as an in-repository dependency of both agents:

make hummingbird-agent/setup
make hummingbird-cve-agent/setup

Usage

from hummingbird_ai import models

model = models.build_model(
    "gemini-2.5-flash",
    google_api_key="",
    google_cloud_project="my-project",
    model_regions={"gemini-2.5-": "us-central1"},
)

Shared calls and usage accounting are provided by hummingbird_ai.calls. Agent-specific prompts, tools, response parsing, and decisions remain in the consuming agent.

Configuration

Setting Use
GOOGLE_API_KEY Passed by the caller for direct Gemini API access
GOOGLE_CLOUD_PROJECT Vertex AI project fallback when credentials do not specify a project

Model regions and Vertex labels are passed by the consuming agent when it constructs an adapter.

Development

See the main README for development workflows.

make check                         # lint
make test                          # run tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.6 - Hummingbird Events Topic

An SNS topic for all Hummingbird project events. This topic serves as a central hub for distributing events from multiple sources (GitLab webhooks, Kubernetes, etc.) to multiple subscribers, enabling event-driven architectures and integrations.

Features

  • Central Event Hub: Dedicated SNS topic per environment (production and staging) for all project events from multiple sources
  • Multiple Subscribers: Supports Lambda, SQS, HTTP endpoints, email, SMS, and more
  • Event Filtering: Subscribers can filter events using SNS subscription filter policies

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for SNS, CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-events-topic
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs:

  • TopicArn - SNS topic ARN (for event publishers)
  • TopicName - SNS topic name

Parameters

Parameter Description Default
TopicName SNS topic name myapp-prod-events

Resource naming: The SNS topic uses the provided TopicName parameter.

Usage

Publishing Events

Event publishers need the topic ARN to publish messages:

# Get topic ARN from CloudFormation stack
aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --query 'Stacks[0].Outputs[?OutputKey==`TopicArn`].OutputValue' \
  --output text

Event publishers:

Subscribing to Events

Subscribe services to receive events:

Via AWS Console:

  1. Open SNS console → Topics
  2. Select the topic
  3. Create subscription (choose protocol: Lambda, SQS, HTTP, Email, etc.)
  4. Add subscription filter policy (optional)

Via AWS CLI:

aws sns subscribe \
  --topic-arn <topic-arn> \
  --protocol lambda \
  --notification-endpoint <lambda-arn>

Add filter policy:

aws sns set-subscription-attributes \
  --subscription-arn <subscription-arn> \
  --attribute-name FilterPolicy \
  --attribute-value '{"source": ["gitlab"], "event_type": ["push"]}'

Subscription Filter Examples

GitLab push events from specific project:

{
  "source": ["gitlab"],
  "event_type": ["push"],
  "project_path": ["redhat/hummingbird/containers"]
}

All merge request events:

{
  "source": ["gitlab"],
  "event_type": ["merge_request"]
}

All events from GitLab:

{
  "source": ["gitlab"]
}

Event metadata: See publisher documentation for available metadata:

Pipeline Timing queue and recovery

Pipeline Timing uses a separate standard SQS queue and dead-letter queue in each environment. The source queue subscribes to its environment’s SNS topic through pipeline-timing/template.yaml; the DLQ receives messages from the source queue after the receive limit. The Status queue and worker remain separate and unchanged. Provision queues and subscriptions in both environments before deploying the workers. Messages received before worker deployment may expire after four days; this pre-launch loss is accepted. Without a consumer, messages are not received and do not automatically move to the DLQ. After launch, monitor queue age during worker pauses to avoid losing events. Validate the staging worker before deploying the production worker or enabling new production event sources.

Setting Value
Queue arr-hummingbird-<env>-pipeline-timing-sqs
Dead-letter queue arr-hummingbird-<env>-pipeline-timing-sqs-dlq
Message retention 4 days on the source queue; 14 days on the DLQ
Redrive limit 3 receives
Visibility timeout 300 seconds
SNS raw message delivery false, preserving the SNS envelope, message ID, and timestamp
Subscription filtering None, the worker receives each topic event

When SQS automatically moves a standard-queue message to its DLQ, it preserves the original enqueue timestamp. The DLQ’s 14-day retention is measured from that timestamp, not added after the source queue’s four days. An operator redrive from the DLQ back to the source queue resets the enqueue timestamp.

The worker follows Status’s 20-second long polling and receives up to 10 messages per batch. Timing commits accepted messages individually, then acknowledges eligible receipts in one batch-delete request per receive. Partial or uncertain delete failures leave committed messages available for idempotent redelivery. The 300-second visibility timeout controls SQS redelivery; it does not interrupt message processing.

The queue names include sqs-dlq so the existing CloudWatch rules classify the DLQ separately. Grafana Alloy is Hummingbird’s monitoring collector. SAM stack tags let its CloudWatch exporter find the source queue and DLQ and forward their queue-depth and oldest-message-age metrics to Mimir. Alloy does not read or consume SQS messages.

The dedicated worker IAM user can call sqs:ReceiveMessage and sqs:DeleteMessage on the timing queue only. It has no access to the Status queue or the DLQ. The queue policy grants SNS delivery only from the matching environment topic and denies non-HTTPS requests. The DLQ redrive policy accepts only its matching source queue. The template uses the Status stack’s UserName and UserArn output names. Operator redrive uses separate AWS credentials.

Fix malformed events before redriving them. An operator can move messages from the DLQ back to the timing queue and check the move task. Set ENV_PREFIX to staging or prod, then read the queue ARNs from the stack outputs:

STACK_NAME="arr-hummingbird-${ENV_PREFIX}-pipeline-timing"
TIMING_QUEUE_ARN=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --query "Stacks[0].Outputs[?OutputKey=='QueueArn'].OutputValue" \
  --output text)
TIMING_DLQ_ARN=$(aws cloudformation describe-stacks \
  --stack-name "$STACK_NAME" \
  --query "Stacks[0].Outputs[?OutputKey=='DeadLetterQueueArn'].OutputValue" \
  --output text)

aws sqs start-message-move-task \
  --source-arn "$TIMING_DLQ_ARN" \
  --destination-arn "$TIMING_QUEUE_ARN"

aws sqs list-message-move-tasks --source-arn "$TIMING_DLQ_ARN"

Verify that the message returns to the timing queue and that the worker processes it. During the initial infrastructure test, if the worker is not deployed yet, remove only the identifiable test message after confirming the move. Do not purge the queue. SNS message IDs make worker redelivery idempotent.

Development

This is a pure infrastructure project (no application code). See the main README for SAM build/deploy commands.

Security & Limitations

Security:

  • SNS topic follows least privilege principle
  • Access controlled via IAM policies
  • Supports server-side encryption (optional)
  • Publishers require sns:Publish permission
  • Subscribers require appropriate protocol permissions

Limitations:

  • Message size: Up to 256 KB
  • Maximum subscriptions: 12,500,000 per topic
  • Message retention: Not supported (use SQS for durable queuing)
  • Delivery retries: Protocol-dependent

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.7 - Vertex AI Cost Metrics

This document defines the shared cost-observability contract for all Hummingbird services that call Vertex AI. It covers two obligations:

  1. Prometheus metrics — estimated cost counters scraped into Mimir.
  2. Vertex request labels — billing metadata attached to every API call.

All services that make Vertex AI calls MUST implement both.

hummingbird-ai emits shared model request, token, and cost metrics for its consumers. The dashboard continues to expose independent database-backed gauges with the same names and labels.

1. Prometheus metrics (estimated cost)

Every Vertex AI caller MUST emit the following counters on its scraped /metrics endpoint. Using identical names and label sets across services enables a single Grafana dashboard without per-app query logic.

Counters

Name Unit Labels Description
hummingbird_vertex_cost_dollars_total USD model, workflow Estimated cost based on MODEL_PRICING x tokens
hummingbird_vertex_tokens_total tokens model, workflow, direction Token counts per API call
hummingbird_vertex_requests_total requests model, workflow, status API call attempts

Labels

Applications MUST set these labels. Additional labels are permitted if cardinality remains bounded.

Label Values
model Model name as sent to Vertex (e.g. gemini-2.5-flash, claude-sonnet-4-6)
workflow Workflow or operation name (e.g. code-review, renovate-babysit)
direction input, output, cache_read, cache_creation
status success, error, retry, parse_error

Request statuses are mutually exclusive per model attempt: success means the response was accepted, error is a terminal model-call failure, retry marks an attempt that will be retried, and parse_error marks a terminal response-validation failure. Token and cost metrics still include every model response, including responses rejected by validation.

Applications MUST NOT set cluster, namespace, app, service, or pod — Alloy adds these automatically at scrape time via relabel rules.

Cardinality

  • model: bounded by deployed model set (currently 2-4)
  • workflow: bounded by workflow config entries (~10 for agent, 1 for dashboard)
  • direction: fixed set of 4
  • status: fixed set of 4

Total series per app: model x workflow x max(direction, status) = 4 x 10 x 4 = 160. Well within Mimir per-metric limits.

2. Vertex request labels (billed cost)

Every Vertex API call MUST include billing labels so that GCP Cloud Billing rows can be attributed to a specific application and workflow once BigQuery export is enabled.

Required labels

Key Value Example
app Application name (matches Kubernetes app) agent, dashboard
workflow Workflow or operation name within the app code-review, analyze-failures

Callers MUST set both keys on every generateContent, streamGenerateContent, or rawPredict request.

How to attach

Endpoint Mechanism
generateContent (Gemini) Top-level "labels" key in JSON body
rawPredict / streamRawPredict X-Vertex-AI-Labels HTTP header

For generateContent, add the labels object to the request body:

{
  "contents": [...],
  "labels": {"app": "agent", "workflow": "code-review"}
}

For rawPredict, base64-encode the labels JSON and pass as a header (shown decoded for clarity):

import base64
import json

labels = {"app": "dashboard", "workflow": "analyze-failures"}
headers["X-Vertex-AI-Labels"] = base64.b64encode(
    json.dumps(labels).encode(),
).decode()

Constraints (GCP requirements)

  • Keys and values: lowercase letters, numbers, underscores, dashes only
  • Max 63 characters each
  • Keys MUST start with a letter
  • Labels are only forwarded for PayGo consumption (Provisioned Throughput silently ignores them)

Reference

Example PromQL

Total estimated spend yesterday (all apps):

sum(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-app daily cost:

sum by (app)(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-workflow breakdown for the agent:

sum by (workflow, model)(
  increase(hummingbird_vertex_cost_dollars_total{app="hummingbird-agent"}[1d])
)

Token consumption by direction:

sum by (app, direction)(increase(hummingbird_vertex_tokens_total[1d]))

Implementing apps

Application Port Workflow values
hummingbird-agent 9090 from wf_cfg.name per workflow
hummingbird-dashboard 8080 analyze-failures

4.5.8 - Hummingbird Sandbox

Shared Python library for isolated command execution used by Hummingbird agents.

Features

  • Podman, Kubernetes, and KubeVirt sandbox backends.
  • Command execution, streamed input/output, file transfer, and sandbox cleanup.
  • Kubernetes and KubeVirt pools for reusing pre-warmed sandboxes.
  • Dedicated execution images for Hummingbird Agent and CVE preparation.

Prerequisites

  • Python 3.11 or later.
  • Podman for local containers, or kubectl, Kubernetes credentials, and namespace-scoped permissions for Kubernetes-backed containers.
  • An image appropriate for the selected backend; Kubernetes deployments must also provide the required network policy and resource controls.

Installation

Install the development package from the tools repository:

make hummingbird-sandbox/setup

Usage

Create a backend through the shared factory, then always clean it up:

from hummingbird_sandbox import config, sandbox

backend = sandbox.create_sandbox(config.SandboxConfig(image="example:latest"))
backend.start()
try:
    result = backend.exec("printf 'hello\\n'")
    if result.exit_code:
        raise RuntimeError(result.stderr)
    print(result.stdout)
finally:
    backend.cleanup()

create_sandbox accepts sandbox_type="podman", "k8s", or "kubevirt"; Kubernetes-backed modes also accept a context and namespace. Pool backends are created separately by callers and require pre-warmed pods or VMIs.

Execution Images

Image Source path Purpose
Hummingbird Agent sandbox (hummingbird-agent-sandbox) hummingbird-sandbox/images/agent/ Command tools for Podman, direct Kubernetes, and Kubernetes pod pools.
CVE preparation sandbox (hummingbird-cve-sandbox) hummingbird-sandbox/images/cve/ RPM preparation tools for the CVE agent; optional and not yet its default image.

The agent’s config example and pod-pool Deployment still use the digest-pinned gitlab-ci image. Switch them only after the agent sandbox image is published and verified. Direct Podman/K8s uses settings.sandbox.image; k8spool uses the image in its pre-warmed Deployment.

Configuration

SandboxConfig holds the backend image, namespace, active deadline, pool linger limits, pod metadata/resources, and KubeVirt image and SSH settings. Callers are responsible for loading configuration and keeping their credentials outside the sandbox.

Hummingbird Agent supplies these settings in settings.sandbox. This example is moved from the agent documentation; the library itself accepts a SandboxConfig object and does not load YAML or environment variables:

sandbox: # sandbox pod configuration
  image: quay.io/.../gitlab-ci:latest #   container image (Podman or direct K8s)
  namespace: default #   sandbox namespace (or CLI --namespace)
  active_deadline_seconds: 1800 #   pod hard timeout / reap interval
  linger_seconds: 300 #   keep pod alive after success (pool mode, 0=disable)
  max_lingering_pods: 2 #   max idle lingering pods before eviction (pool mode)
  vm_image: quay.io/.../vm-disk:latest #   containerDisk image (kubevirt mode only)
  vm_memory: 1Gi #   VM guest memory (kubevirt mode only)
  vm_ssh_private_key: /path/to/key #   SSH private key for pool VMs (kubevirtpool mode)
  metadata: #   pod/VMI metadata (k8s/kubevirt mode)
    labels:
      app.kubernetes.io/name: hummingbird-agent-sandbox
  resources: #   K8s resource requests/limits (k8s mode)
    requests:
      cpu: "100m"
      memory: "256Mi"
      ephemeral-storage: "256Mi"
    limits:
      cpu: "1"
      memory: "1Gi"
      ephemeral-storage: "2Gi"

Security and Design Constraints

hummingbird-sandbox supplies execution backends; it does not create the caller’s namespace RBAC or NetworkPolicy, load configuration, or make deployment decisions. Callers own configuration and credentials, and must provide the infrastructure controls described below. The following security and backend documentation is moved from Hummingbird Agent without changing its behavior claims.

The agent is designed to run in a shared OpenShift cluster without cluster-admin access, processing potentially untrusted merge requests. These constraints shaped the architecture:

Sandbox isolation. All arbitrary commands executed by the LLM run inside an ephemeral container, never on the host:

  • Podman (local): --network=none, --user 65532, no host mounts. Complete network isolation.
  • Kubernetes (production): Pods comply with OpenShift’s restricted-v2 Security Context Constraint: runAsNonRoot, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: ["ALL"], automountServiceAccountToken: false (no K8s API access from sandbox), activeDeadlineSeconds (configurable, default 1800). Security context fields are hardcoded in the pod manifest for portability to vanilla Kubernetes with Pod Security Admission (restricted level). Resource requests/limits, metadata, and activeDeadlineSeconds are configurable via settings.sandbox in the config file. Network access is denied by a NetworkPolicy on the sandbox namespace that blocks all egress from all pods (podSelector: {}).

No cluster-admin required. The agent operates with namespace-scoped permissions only. The orchestrator’s ServiceAccount needs only:

  • pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
  • pods/exec: create – command execution via kubectl exec
  • virtualmachineinstances.kubevirt.io: create, get, list, delete, patch – KubeVirt VMI lifecycle (kubevirt/kubevirtpool modes only)
  • secrets: create, get, delete – SSH key Secrets for VMI provisioning (kubevirt/kubevirtpool modes only)

These permissions are granted via a Role in the sandbox namespace, not the orchestrator’s own namespace. No CRDs, no custom runtimes, no cluster-scoped resources. Konflux data is fetched via bearer token from kubeconfig, not from inside the cluster.

Namespace separation. Sandbox pods are created in a dedicated namespace, separate from the orchestrator. This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens. The sandbox namespace is locked down with standard K8s resources:

  • RBAC: Role + RoleBinding scoped to the namespace, granting only the permissions above to the orchestrator’s ServiceAccount
  • NetworkPolicy: uses podSelector: {} to select all pods in the dedicated sandbox namespace, denying all egress (egress: []). The sandbox cannot reach the internet, the K8s API, or other pods.
  • activeDeadlineSeconds: sandbox pods self-terminate after the configured timeout (default 1800s / 30 minutes) even if the orchestrator crashes or is killed, preventing orphaned pods

Credential separation. Data source credentials (GitLab tokens, kubeconfig) live in the orchestrator process only, injected via K8s Secrets. The sandbox container has no credentials, no SA token (automountServiceAccountToken: false), and no network access. Data flows into the sandbox via stdin piping through write_file.

Command execution via kubectl exec. The K8s sandbox uses a hybrid approach: the Kubernetes Python client manages pod lifecycle (create, wait, delete), while kubectl exec handles command execution. This avoids the complexity and reliability issues of the websocket-based exec API.

KubeVirt VM isolation. KubeVirt sandbox VMs run as root inside the guest OS, but the VM itself is contained by the KubeVirt hypervisor (QEMU/KVM). The VM has no access to the Kubernetes API, no ServiceAccount token, and no credentials. SSH keys are ephemeral (generated per sandbox start for direct mode, or shared per pool for pool mode) and cleaned up with the VMI.

Sandbox Backends

Podman (local) K8s (direct) K8sPool (production) KubeVirt (direct) KubeVirtPool
Start podman run -d --network=none create_namespaced_pod Claim standby pod from Deployment Create VMI + SSH Secret Claim standby VMI from VMIRS
Exec podman exec kubectl exec kubectl exec ssh ssh
Auth Local Podman socket In-cluster SA or kubeconfig In-cluster SA or kubeconfig In-cluster SA or kubeconfig In-cluster SA or kubeconfig
Network None (--network=none) None (deny-all NetworkPolicy) None (deny-all NetworkPolicy) Cluster pod network (SSH) Cluster pod network (SSH)
User 65532 (fixed) Namespace UID range (SCC) Namespace UID range (SCC) root (inside VM) root (inside VM)
Cleanup podman rm -f delete_namespaced_pod delete_namespaced_pod Delete VMI + Secret + temp keys Delete VMI

All five implement the Sandbox protocol: start(), exec(), write_file(), read_file(), cleanup(), linger().

The pod pool backend (k8spool) eliminates pod startup latency by claiming pre-warmed pods from a Kubernetes Deployment. Claimed pods are detached from the ReplicaSet and the Deployment automatically creates replacements. After a successful workflow, pool pods linger for linger_seconds (default 300, configurable, 0 to disable) so that user replies can reuse the same pod without re-creating it or restoring from S3. The reaper runs once per workflow execution and deletes expired lingering pods. It also evicts excess lingering pods beyond max_lingering_pods (default 2), starting with those closest to their deadline. See the design doc (section 8.8) for details.

The KubeVirt backends (kubevirt, kubevirtpool) provide full VM isolation using KubeVirt VirtualMachineInstances. The VM boots from a containerDisk image and SSH keys are injected via a Kubernetes Secret volume (the VM image’s inject-ssh-keys.service reads from /dev/disk/by-id/virtio-ssh-pubkeys). Command execution uses SSH instead of kubectl exec. The kubevirtpool backend claims pre-warmed VMIs from a VirtualMachineInstanceReplicaSet, with the same claim/reap/linger semantics as the pod pool. The sandbox backend can be set per-workflow via the sandbox: field in the workflow config, with resolution order: workflow config > CLI --sandbox > default.

The Python API does not pass caller credentials into sandbox processes. Callers must continue to keep credentials on the trusted side of the API and provide the namespace network and resource controls required by their backend. For workflow and CLI integration details, see Hummingbird Agent.

Development

Build the Hummingbird Agent execution image from the tools repository root:

podman build -f hummingbird-sandbox/images/agent/Containerfile \
  -t localhost/hb-agent-sandbox:latest hummingbird-sandbox/

The CVE agent’s optional RPM-preparation image can be built locally with:

podman build -f hummingbird-sandbox/images/cve/Containerfile \
  -t localhost/hb-cve-sandbox:latest hummingbird-sandbox/

Run the sandbox library tests and lint from the tools repository root:

cd hummingbird-sandbox
python3 -m unittest discover -s tests -t .
ruff check .

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.9 - SNS S3 Archiver

A Lambda function that archives all messages from the Hummingbird Events SNS topic to S3. Events are stored as clean JSON with decoded payloads, enabling easy browsing with zcat | jq and replay to handlers.

Features

  • S3 Storage: Events stored with time-hierarchical key structure for efficient time-range queries
  • Clean JSON: Decodes gzip+base64 Message payload for human-readable storage
  • Replay Support: Preserves all MessageAttributes with updated content_encoding for replay fidelity
  • Easy Access: Download via aws s3 sync, browse with zcat | jq

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, S3, SNS, CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd sns-s3-archiver
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs:

  • BucketName - S3 bucket name for archived events
  • BucketArn - S3 bucket ARN
  • FunctionArn - Lambda function ARN

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
SnsTopicArn ARN of the SNS topic to subscribe to (required)
SentryDsn Optional Sentry DSN for error tracking ""
RetentionDays Days to retain events (0=indefinite) 0

Resource naming: Bucket and Lambda names are derived from ResourcePrefix:

  • S3 bucket: {ResourcePrefix}-s3-events
  • Lambda function: {ResourcePrefix}-lambda-s3-archiver

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

S3 Key Structure

Events are stored with a time-hierarchical key structure optimized for time-range queries:

sns/YYYY/MM/DD/HH/MM/TIMESTAMP#MSGID::source::kind::name.json.gz

Examples:

sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#abc12345::kubernetes::Snapshot::konflux-rh03--tenant--my-snapshot.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#def67890::gitlab::push::abc123def456.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#0123abcd::gitlab::pipeline::org-project-42.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#4567efab::gitlab::build::org-project-99.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#89abcdef::kubernetes::TaskRun::konflux-rh03--tenant--task-1.json.gz

The sns/ prefix separates the new format from legacy data, enabling incremental migration. The :: delimiter separates metadata fields (avoiding conflicts with K8s names that contain --). The resource name in the key enables efficient “latest state per resource” queries without downloading all files.

The minute-level subdirectory (/MM/) provides write distribution for high-throughput scenarios.

Resource identities

The timestamp and first eight characters of the SNS MessageId make each key deterministic for one SNS delivery. The archiver never uses an SQS message ID. Repeated archiver delivery of the same SNS record writes the same S3 object key.

Source Kind Resource identity
GitLab push First 12 characters of checkout_sha, falling back to after
GitLab pipeline Project path with / replaced by -, followed by object_attributes.id
GitLab build Project path with / replaced by -, followed by build_id
GitLab Other events Project path with / replaced by -
Kubernetes All kinds Logical cluster name, namespace, and name joined by --

The logical cluster name comes from the Kubernetes event forwarder’s selected kubeconfig context, not from the API server URL. URL-valued cluster attributes from an older forwarder retain the name-only identity used by existing archives.

When a Pipeline or Job Hook omits its numeric ID, the archiver retains the project-only GitLab identity.

sns-s3-archiver/tests/fixtures/archive-key-contract.json is the versioned, machine-readable contract for these identities. Archiver and GitLab forwarder tests load it directly. Hummingbird Status tests will load the same repository file as part of HUM-7529. It is test data, not a runtime dependency.

Storage Format

Each S3 object contains a gzip-compressed JSON record:

{
  "MessageId": "abc12345-1234-5678-9abc-def012345678",
  "Timestamp": "2025-12-13T12:34:56.789Z",
  "Message": {
    "apiVersion": "appstudio.redhat.com/v1alpha1",
    "kind": "Snapshot",
    "metadata": { ... },
    "spec": { ... }
  },
  "MessageAttributes": {
    "source": { "Type": "String", "Value": "kubernetes" },
    "kind": { "Type": "String", "Value": "Snapshot" },
    "content_encoding": { "Type": "String", "Value": "json" }
  }
}

Key points:

  • Message is a decoded JSON object (not the original base64+gzip string from SNS)
  • content_encoding is set to "json" to indicate the decoded format

Usage

Download Events Locally

# Sync entire bucket
aws s3 sync s3://bucket-name/ ./local-events/

# Sync specific time range
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/2025/12/13/

# Sync with filtering
aws s3 sync s3://bucket-name/ ./local-events/ --exclude "*" --include "*kubernetes*Snapshot*"

Browse Events

# List events for a specific hour
aws s3 ls s3://bucket-name/2025/12/13/12/ --recursive

# View a single event
aws s3 cp s3://bucket-name/2025/12/13/12/34/event.json.gz - | zcat | jq

# Local browsing
zcat ./local-events/2025/12/13/12/34/event.json.gz | jq

Replay Events

Consumer handlers should support both live SNS format and archived format:

def _decode_message(sns_record: dict) -> dict:
    """Decode Message - handles live SNS and archived formats."""
    message = sns_record.get("Message", "")
    encoding = sns_record.get("MessageAttributes", {}).get("content_encoding", {}).get("Value")

    # Archived format: Message is already decoded
    if encoding == "json" or isinstance(message, dict):
        return message if isinstance(message, dict) else json.loads(message)

    # Live SNS format: Message is gzip+base64
    if encoding == "gzip+base64":
        return json.loads(gzip.decompress(base64.b64decode(message)))

    # Fallback: plain JSON string
    return json.loads(message)

Compression Flow

Forwarder → SNS → Archiver → S3

1. Forwarder: gzip(payload) → base64 → SNS Message
2. SNS: Passes through with content_encoding="gzip+base64"
3. Archiver: Decodes Message, sets content_encoding="json", gzip(record) → S3
4. S3: Stores clean JSON record, gzip-compressed at object level

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

S3 Lifecycle Policies

The bucket has two lifecycle rules:

  • AbortIncompleteUploads: Automatically cleans up incomplete multipart uploads after 1 day (prevents storage cost from failed uploads)
  • ExpireOldEvents: When RetentionDays > 0, automatically deletes objects older than the specified retention period. Disabled when RetentionDays = 0 (indefinite retention, the default).

Security & Limitations

Security:

  • S3 bucket blocks all public access by default
  • S3 objects encrypted at rest (AES256)
  • Lambda follows least privilege principle (PutObject only)
  • CloudWatch logs capture all archive operations (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • S3 object key length: max 1024 bytes
  • Single-region deployment (follows SNS topic region)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.10 - Hummingbird Tools

Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup.

The image also contains Tekton Tasks for MR artifact promotion: they find the tested image and create a push-style Component Snapshot after merge.

Features

  • Streaming Backup: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases
  • Streaming Restore: Streams S3 through gzip and psql with bounded memory
  • Automatic Rotation: Keeps configurable number of backups per group (daily/weekly/monthly)
  • Separate IAM Users: Write user for production backups, read-only user for staging restore
  • MR Artifact Lookup: Derive the MR head from a merge commit and verify its tested Snapshot against the linked build PipelineRun
  • Promotion Component Snapshot: Create a single-Component Snapshot using the tested revision and digest, allowing Integration Service to release it

Prerequisites

  • AWS CLI configured with appropriate credentials
  • PostgreSQL client tools (pg_dump, psql)
  • Python 3.11 or later
  • Access to the hummingbird-status PostgreSQL database

The MR artifact lookup Task needs a Git checkout Trusted Artifact at the merge commit, a digest-pinned Hummingbird tools image, and a ServiceAccount that can list and get Snapshots and PipelineRuns in the Konflux namespace. KubeArchive must be reachable by HTTPS; it authorizes reads using the Task’s projected ServiceAccount token and the same namespace permissions.

The promotion Component Snapshot Task also needs permission to list IntegrationTestScenarios and create Snapshots in that namespace. Creating a Component Snapshot for an Application with automatic ReleasePlans can start a release.

Both Tasks unpack the source Trusted Artifact into /var/workdir/source and run their Python step from /var/workdir. Tekton creates a step’s working directory before any step in the Task runs; using the extraction path as the working directory makes it root-owned and prevents unpacking under the restricted SCC.

MR Artifact Lookup

The Task manifest is hummingbird-tools/tekton/find-tested-mr-artifact.yaml. Pass the merge commit SHA, target repository URL, application, Component, and namespace. Pass the clone Task’s SOURCE_ARTIFACT result as sourceArtifact. The clone must include full Git history. Supply the Hummingbird tools image by digest through toolImage. Set archiveUrl to the cluster’s KubeArchive API URL, for example https://kubearchive.apps.cluster.example.com.

The Task reads the merge commit’s second Git parent, then selects the newest passed MR Snapshot from the live API and KubeArchive. A passed /test or /retest build is eligible when its Snapshot and build carry the same PaC test or retest event. Multiple passed Snapshots for one commit are normal — a /test run creates another — and while RPM builds are not reproducible their image digests may differ (HUM-9785); the newest eligible Snapshot is selected. It reads a linked build PipelineRun from KubeArchive when the live object has been removed, then checks the source URL and revision, target branch, and image digest. The Task fails when KubeArchive cannot be queried. Its results are SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL. It does not create a Snapshot or Release.

Promotion Component Snapshot

The Task manifest is hummingbird-tools/tekton/create-promotion-component.yaml. Pass the same merge commit, target repository, Application, Component, and namespace as the lookup Task. Pass the same sourceArtifact and the lookup Task’s SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL results as sourceSnapshot, buildPipelineRun, image, sourceSha, and sourceUrl. Supply the tools image by digest through toolImage. Pass the same archiveUrl used by the lookup Task and the current PipelineRun name as promotionPipelineRun.

The Task rechecks Git parent B, the MR Snapshot, its linked build PipelineRun, and the selected digest. It reads removed MR evidence from KubeArchive while checking current IntegrationTestScenarios and creating the Component Snapshot through the live Kubernetes API. It fails if an IntegrationTestScenario for the Application could run on the push-style component Snapshot. It then creates a Snapshot named deterministically from the full Component name and merge SHA, containing only the selected Component entry. The Snapshot has Application and Component labels, type=component, and a push event label. Its annotations link the merge commit, MR Snapshot, build PipelineRun, and promotion PipelineRun. The test.appstudio.openshift.io/pipelinerunstarttime annotation is the promotion PipelineRun’s status.startTime in epoch milliseconds. The COMPONENT_SNAPSHOT result contains its name.

Before creation, the Task checks the target branch with Git. For RPM builds, it compares the build’s monorepo-subdir with the current branch when other merges have advanced it. An unrelated package merge does not block promotion; a newer change to the same package does. Builds without a monorepo-subdir require the merge commit to remain the branch head. A retry reuses a live Component Snapshot only when its Application, Component, source revision, image digest, provenance, and original promotion start time match; otherwise it reports the conflicting fields. It never reuses an archived Snapshot. Git branch access is required for this check.

The Task creates no Release or build attestation. Integration Service handles the automatic Release. See MR artifact promotion for the complete flow and Snapshot ordering. If a retry finds a matching Snapshot, the Task returns that name without creating another one.

Promotion failure and recovery

If a stale run stops, let promotion for the current target branch proceed. If the Component Snapshot conflicts, inspect it and its source MR evidence; do not delete or replace it to force another Release. A failed automatic Release needs an explicit operator-owned retry through the release workflow. If the tested MR Snapshot or build evidence is unavailable, recover through an explicit new build and required tests before release. Check Git access for branch lookup failures, ServiceAccount permissions for forbidden writes, and tenant quota for quota failures. A normal Task retry does not repair a failed Release.

Deployment

The SAM template creates:

  • S3 bucket for database backups
  • IAM user with write permissions (for production backup CronJobs)
  • IAM user with read-only permissions (for staging restore CronJob)

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-tools
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Note: Only deployed in production - staging restore reads from the production backup bucket.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod

Resource naming:

  • S3 bucket: {ResourcePrefix}
  • Write IAM user: {ResourcePrefix}-write-user
  • Read IAM user: {ResourcePrefix}-read-user

Usage

Backup

Run a backup with rotation:

python3 -m hummingbird_tools.backup <group> <rotation>

Arguments:

  • group: Backup group name (e.g., daily, weekly, monthly)
  • rotation: Number of backups to keep for this group

Environment variables:

Variable Description
S3_BUCKET S3 bucket name
S3_PREFIX Optional prefix for S3 keys
POSTGRESQL_HOST Database hostname
POSTGRESQL_USER Database username
POSTGRESQL_PASSWORD Database password
POSTGRESQL_DATABASE Database name

Example:

export S3_BUCKET=myapp-prod-db-backups
export S3_PREFIX=mydb/
export POSTGRESQL_HOST=myapp-postgres
export POSTGRESQL_USER=postgres
export POSTGRESQL_PASSWORD=secret
export POSTGRESQL_DATABASE=mydb

python3 -m hummingbird_tools.backup daily 7

This creates a backup like mydb/2026-01-16-02-30.daily.sql.gz and removes any daily backups beyond 7.

Restore

Restore from the latest backup:

python3 -m hummingbird_tools.restore

Environment variables are the same as for backup.

The restore process:

  1. Lists all backups matching *.{daily,weekly,monthly}.sql.gz
  2. Selects the latest by filename sort (most recent timestamp)
  3. Opens the backup stream and starts a database transaction
  4. Removes the rpm_tracker schema and existing public tables
  5. Streams the decompressed SQL through psql with ON_ERROR_STOP=1
  6. Commits only after the complete gzip stream validates and all SQL succeeds

SQL errors and damaged backups fail the restore and roll back both cleanup and import. This replaces tracker data rather than merging it. An older backup without tracker tables removes the target tracker schema rather than retaining stale data. Other schemas are not removed automatically. If the dump conflicts with an existing schema outside this cleanup scope, the restore fails and rolls back instead of merging it.

Pause all database writers and wait for active Jobs to finish before restoring. SCALE_DEPLOYMENT pauses only its configured Deployment, not tracker CronJobs or other services. Ensure database roles referenced by the dump already exist. After restoring, run service initializers to apply migrations and refresh environment specific permissions before resuming workloads.

CronJob Schedule

Deployed via Kubernetes CronJobs in kubernetes/hummingbird-status/:

CronJob Schedule Environment Command
daily 30 2 * * * production python3 -m hummingbird_tools.backup daily 7
weekly 30 6 * * 0 production python3 -m hummingbird_tools.backup weekly 4
monthly 30 10 1 * * production python3 -m hummingbird_tools.backup monthly 12
restore (suspended) staging python3 -m hummingbird_tools.restore

To manually trigger a staging restore:

kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s)

S3 Key Structure

Backups are stored with a flat key structure per database:

{database}/{timestamp}.{group}.sql.gz

Examples:

mydb/2026-01-16-02-30.daily.sql.gz
mydb/2026-01-12-06-30.weekly.sql.gz
mydb/2026-01-01-10-30.monthly.sql.gz

Multiple databases can share the same bucket using different prefixes.

Development

See the main README for development workflows.

make hummingbird-tools/setup  # Install dependencies
make check                     # Lint code (ruff)
make fmt                       # Format code
make test                      # Run unit tests
make coverage                  # Run tests with coverage

Restore integration tests require PostgreSQL client tools and an empty disposable database whose name starts with test_restore_. They refuse an existing populated database and use fake S3 input, without contacting AWS or Kubernetes:

cd hummingbird-tools
export PGHOST=localhost PGPORT=5432 PGUSER=postgres
export RESTORE_TEST_DATABASE=test_restore_local
python3 -m unittest tests.test_restore_db

Create the empty test database first. Supply PGPASSWORD through your local test environment if required. Never point these tests at a deployed database.

Security

  • S3 bucket blocks all public access
  • S3 objects encrypted at rest (AES256)
  • Separate IAM users with least privilege:
    • Write user: s3:PutObject, s3:DeleteObject, s3:ListBucket
    • Read user: s3:GetObject, s3:ListBucket
  • Database passwords passed via environment variables (Kubernetes secrets)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.11 - ProdSec RPM Catalog

Builds an RPM catalog from each Hummingbird Pulp domain and publishes it to that domain’s rpm-catalog file repository. The public catalog reads RPM metadata through DNF. Private catalogs read repository versions through the authenticated Pulp API so content guards on downloads do not block catalog generation. A monitoring companion prints the last public publication time.

Features

  • DNF-Based Public Query: Reads package metadata from all 5 public RPM repos (source, x86_64, aarch64, ppc64le, s390x)
  • Authenticated Private Query: Reads the same repos from a pinned Pulp repository version, requesting only the fields used in the catalog
  • Pulp REST API Upload: Creates file content in the selected repository; autopublish updates its distribution
  • Artifact Deduplication: Skips upload when content SHA-256 matches an existing artifact
  • Async Task Polling: Handles Pulp’s asynchronous tasks with configurable timeout (10 minutes)
  • Sentry Error Reporting: Errors are reported to Sentry when SENTRY_DSN is set
  • Monitoring Check: Companion module prints the publication timestamp for automated freshness monitoring

Prerequisites

  • Python 3.11 or later
  • dnf system library for the public catalog (available on Fedora/RHEL)
  • Network access to the Pulp API and public repo metadata
  • Pulp API access with client certificate or username/password auth
  • An autopublishing file repository and distribution named rpm-catalog in each target domain
  • Writer access to view and modify the file repository in each target domain

Usage

Build and Publish Catalog

python3 -m hummingbird_tools.prodsec_catalog rpm-catalog

For a private domain, use its restricted-reader config for package listing and its writer config for catalog publication:

PULP_READER_CONFIG=./reader-cli.toml PULP_CONFIG=./writer-cli.toml \
  python3 -m hummingbird_tools.prodsec_catalog rpm-catalog \
    --domain private-hummingbird-example --source pulp-api

The private CronJob mounts a ConfigMap with one domain per line and processes each domain with the same reader and writer credentials:

PULP_READER_CONFIG=./reader-cli.toml PULP_CONFIG=./writer-cli.toml \
  python3 -m hummingbird_tools.prodsec_catalog rpm-catalog \
    --domains-file ./domains.txt --source pulp-api

Blank lines and lines beginning with # are ignored. Only private-* domains are accepted. If one domain fails, the updater continues with the others and exits unsuccessfully after processing the full list.

The updater refuses to publish an empty catalog. A missing RPM repository or an incomplete Pulp API page also fails the run instead of publishing partial data.

Check Last Publication Time

python3 -m hummingbird_tools.prodsec_catalog_check

Prints the timestamp of the most recent catalog publication in local timezone. Uses the same authentication as the catalog builder.

Configuration

Authentication

Client certificate auth (when configured):

Variable Description
HUMMINGBIRD_PULP_BOT_CERTIFICATE Path to client certificate PEM file
HUMMINGBIRD_PULP_BOT_KEY Path to private key PEM file (optional)
HUMMINGBIRD_PULP_BOT_PASSWORD Optional passphrase for the key

Basic auth (alternative for local use):

Variable Description
PULP_USERNAME Pulp API username
PULP_PASSWORD Pulp API password

Credentials can also be configured in ~/.config/pulp/cli.toml (same format as the pulp CLI). Set PULP_CONFIG to override the config file path (useful in containers where the home directory may vary).

Pulp API

Variable Description
PULP_BASE_URL Pulp API base URL (or read from cli.toml)
PULP_CONFIG Path to pulp CLI config file (default ~/.config/pulp/cli.toml)
PULP_READER_CONFIG Optional separate read-only CLI config for API enumeration

--domain selects one Pulp domain (default public-hummingbird), while --domains-file reads private domains from a mounted ConfigMap. --source selects dnf (public default) or pulp-api (private domains). PULP_BASE_URL, when set, must already include the selected domain’s /api/v3 path. When PULP_READER_CONFIG is set, its credentials are used only for package listing; PULP_CONFIG remains the writer used to upload the catalog. Without a separate reader config, the writer performs both operations.

Error Reporting

Variable Description
SENTRY_DSN Optional Sentry DSN for error tracking

CronJob Schedule

Deployed via Kubernetes CronJobs in the infrastructure repository’s kubernetes/hummingbird-tools/ templates:

CronJob Schedule Package source
hummingbird-tools-prodsec-catalog @daily Public DNF metadata
hummingbird-tools-private-catalog @daily Pulp API for configured private domains

The private CronJob has one container and mounts a ConfigMap containing the private domain list. To onboard a domain, provision its RPM repositories and an autopublishing rpm-catalog file repository and distribution, grant the shared restricted-reader and writer accounts access to that domain, then add its name to the ConfigMap in the infrastructure repository. Regenerate and commit the infrastructure manifests. The CronJob mounts generic reader and writer Kubernetes Secrets; one domain’s failure does not stop updates for the remaining domains, but the Job still fails overall.

Catalog Format

The catalog is a tab-separated file with one RPM per line:

{name}-{version}-{release}.{arch}.rpm\t{repo}\t{build_timestamp}

Example:

nginx-1.28.0-1.hum1.x86_64.rpm  x86_64  2026-01-15 14:30:00
kernel-6.12.5-1.hum1.src.rpm    source  2026-01-10 08:00:00

For example, a catalog can be served at https://packages.example.com/api/pulp-content/private-hummingbird-example/rpm-catalog/hummingbird-rpm-catalog.txt. The distribution’s content guard still applies when downloading it.

Development

See the main README for development workflows.

make hummingbird-tools/setup  # Install dependencies
make check                     # Lint code (ruff)
make test                      # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.12 - Chainguard Comparison

Collects CVE posture data comparing Hummingbird and Chainguard container images and surfaces the results in the Hummingbird dashboard under the Competitive tab.

Features

  • Fetches Hummingbird CVE data from the catalog API — the same source used by images.redhat.com, updated hourly by the scan pipeline
  • Scans Chainguard images with Grype (the only available source for Chainguard CVE data)
  • Stores results, including the individual CVE IDs found by each side, in PostgreSQL for trend tracking and historical comparison
  • Dashboard shows per-image CVE counts, severity breakdowns (C/H/M/L), delta (HB minus CG), 30-day trend charts, aggregate summary stats, and a per-image CVE ID diff drawer
  • Runs every 6 hours as a Kubernetes CronJob

Prerequisites

  • Access to the Hummingbird cluster (PostgreSQL, image pull credentials)
  • The chainguard_comparison package from competitive-tools installed in the container image

Deployment

The tool is deployed as a Kubernetes CronJob. The container image is built by Konflux from chainguard-comparison/Containerfile.

Environment variables

Variable Required Description
DATABASE_URL Yes PostgreSQL connection string, e.g. postgresql://user:pass@host/db
GRYPE_TIMEOUT No Per-image Grype scan timeout in seconds (default: 300)
SENTRY_DSN No Sentry DSN for error reporting

How it works

  1. Discovery — lists Hummingbird images from the catalog API and Chainguard images via the OCI distribution API
  2. Matching — pairs images by name using the alias map in competitive-tools (e.g. nodejs ↔ node, postgresql ↔ postgres). Each pair also captures the catalog’s own real image name and stream for the chosen Hummingbird tag (e.g. real name valkey, stream 9.0) alongside the “common” display name/parsed version used for comparison (e.g. redis, 9) — the two can disagree, either because the display name is a cross-vendor alias or because the parsed version doesn’t match the catalog’s own stream label. Both are stored (see schema below) so the dashboard can match HUM tickets against the catalog’s real identity instead of the display name (see hummingbird-dashboard.md)
  3. HB collection — calls GET /v1/images/{name}/vulnerabilities/{tag} on the catalog API; no image pull or local scan required. The catalog API’s own scan is Grype-based too, without --by-cve, so its vulnerabilities[].id can also be a raw GHSA/Go vulnerability DB ID
  4. CG collection — pulls each Chainguard image and runs grype locally with --by-cve; Chainguard has no publicly accessible pre-scanned CVE feed. --by-cve reports a CVE ID instead of the native GHSA/Go vulnerability DB ID wherever grype’s own local vulnerability DB has that cross-reference, with no added network call
  5. Alias resolution — every non-CVE ID collected from either side (cg_comparison.__main__._resolve_non_cve_ids) is looked up in one batched call to grype db search vuln against grype’s already-downloaded local database, which checks every data provider’s record for that ID, not just the one that produced a live scan match – this finds cross-references --by-cve’s own live match misses. An ID with no known CVE anywhere in the database is stored as-is
  6. Storage — writes both sides to the chainguard_scans table in PostgreSQL

Dashboard

The /competitive page in hummingbird-dashboard shows:

  • Summary cards: total images compared, count worse/better/same
  • Sortable table with per-image CVE counts and delta
  • Inline severity breakdown: Critical / High / Medium / Low
  • Per-image 30-day trend chart (click trend on any row)
  • Filter box for quick image lookup
  • A per-image CVE diff drawer (click the Delta value on any row) listing every CVE ID that differs between the two scans:
    • + — a CVE ID reported only by Hummingbird’s scan
    • - — a CVE ID reported only by Chainguard’s scan
    • This is a scan result difference, not a confirmed determination of whether either side is actually affected
    • CVE IDs both sides report are shared and excluded from the diff
    • Each CVE ID links to its Red Hat CVE page; when a HUM Jira ticket tracks it, the ticket is shown and linked alongside it (HUM-5678/CVE-2026-1234)
    • If a scan predates the cve_ids column, the drawer shows a notice instead of a misleading empty diff

Database schema

CREATE TABLE chainguard_scans (
    id          SERIAL PRIMARY KEY,
    collected_at TIMESTAMPTZ NOT NULL,
    source      TEXT NOT NULL,        -- 'hummingbird' or 'chainguard'
    image_name  TEXT NOT NULL,
    tag         TEXT NOT NULL DEFAULT 'latest',
    image_ref   TEXT,
    total_cves  INTEGER NOT NULL DEFAULT 0,
    critical    INTEGER NOT NULL DEFAULT 0,
    high        INTEGER NOT NULL DEFAULT 0,
    medium      INTEGER NOT NULL DEFAULT 0,
    low         INTEGER NOT NULL DEFAULT 0,
    negligible  INTEGER NOT NULL DEFAULT 0,
    unknown     INTEGER NOT NULL DEFAULT 0,
    cve_ids     JSONB NOT NULL DEFAULT '[]'::jsonb,  -- distinct CVE IDs found in this scan
    catalog_image_name TEXT,  -- real Hummingbird image name, e.g. "valkey" (image_name may be a display alias, e.g. "redis")
    catalog_stream     TEXT   -- real catalog stream, e.g. "9.0" (tag may be a parsed, less precise version)
);

The table is created by the Alembic migration b4e7f8a1c3d5_add_chainguard_scans.py; cve_ids was added later by 9c4f7e21ab36_add_cve_ids_to_chainguard_scans.py, and catalog_image_name / catalog_stream by d3f6a8b1c2e4_add_catalog_identity_to_chainguard_scans.py. Scans collected before the cve_ids migration shipped have an empty cve_ids array even though total_cves is nonzero — the dashboard detects this case and shows a “check back after the next collection run” notice instead of a false empty diff. Scans collected before catalog_image_name / catalog_stream existed have both as NULL; the dashboard’s HUM ticket matching falls back to image_name / tag (the display name/parsed version) in that case, matching its pre-migration behavior exactly.

Development

cd chainguard-comparison
pip install -e ".[dev]"
python -m pytest tests/

To run the collector locally against a development database:

export DATABASE_URL=postgresql://postgres:dev@localhost:5432/events
python -m cg_comparison

License

GPL-3.0-or-later

4.5.13 - CVE Analysis

Automated analysis of HUM Jira Security (CVE) tickets. For each ticket, the tool fetches vulnerability data from MITRE and NVD, compares it against the versions shipped in the Hummingbird package repository, searches for upstream and Fedora fixes, and computes whether the ticket should be closed, moved to In Progress, or flagged for manual investigation.

CVE Lifecycle & State Vocabulary

A CVE tracker moves through a shared lifecycle that both this tool and the CVE agent reason about. The deterministic analysis decides a target resolution for each ticket; the residual it cannot resolve (cve-needs-attention) is handed to the agent; closed tickets are later reconciled against the VEX feed.

stateDiagram-v2
    direction TB

    [*] --> New

    New --> InProgress: affected / needs investigation
    New --> DoneErrata: fixed by analysis
    New --> NotABug: component not present
    New --> WontDo: package EOL

    InProgress --> Agent: upstream-fix-available / cve-needs-attention

    Agent: CVE Agent
    Agent --> Review: fix MR opened
    Agent --> DoneErrata: already_fixed
    Agent --> NotABug: not_affected
    Agent --> InProgress: no_upstream_fix / needs_human / error

    Review --> DoneErrata: MR merged
    Review --> Review: MR conflicts / needs human

    state "Closed (JiraStatus)" as Closed {
        DoneErrata: Done-Errata
        NotABug: Not a Bug
        WontDo: Won't Do
    }

    DoneErrata --> AwaitingVEX: enqueue awaiting-vex
    NotABug --> AwaitingVEX: enqueue awaiting-vex
    WontDo --> [*]: no VEX reconciliation
    AwaitingVEX --> [*]: VEX matches resolution
    AwaitingVEX --> AwaitingVEX: pending / mismatch

    classDef jiraStatus fill:#e5e7e9,stroke:#27ae60,color:#1f2937
    classDef closedStatus fill:#e5e7e9,stroke:#1f5f8b,color:#fff
    classDef resolution fill:#d5f5e3,stroke:#27ae60,color:#1f2937
    classDef automation fill:#e5e7e9,stroke:#7f8c8d,color:#1f2937
    class New,InProgress,Review jiraStatus
    class Closed closedStatus
    class DoneErrata,NotABug,WontDo resolution
    class Agent,AwaitingVEX automation

Notes on the diagram:

  • Agent is the CVE agent. See that tool’s doc for the full internals and automated sweep deployment.
  • Blue states are Jira statuses, green states are resolutions, and gray states represent supporting automation.
  • Won’t Do (package EOL) closes without VEX reconciliation; Done-Errata and Not a Bug enqueue awaiting-vex and leave AwaitingVEX only once the VEX feed matches the resolution.

These state names live in exactly one place, hummingbird_cve_analysis/lib/states.py, imported by both tools:

  • JiraStatus — New, In Progress, Review, Closed.
  • Resolution — Done-Errata, Not a Bug, Won't Do (set when closing).
  • Disposition — the second field of a computed_resolution line: the three resolutions above plus the open-ticket pseudo-dispositions affected and needs investigation.
  • Verdict — the CVE agent’s per-ticket outcome (needs_change, not_affected, already_fixed, no_upstream_fix, needs_human, error), with VERDICT_OUTCOME mapping each verdict to the Jira status/resolution it implies.

How It Works

1. Ticket Selection

The tool queries Jira for HUM project tickets with component Security. Tickets can be filtered by creation period (--show-since "2 weeks"), limited to specific keys (HUM-796 HUM-518), or scanned in bulk. Closed tickets are skipped by default unless --include-closed is specified or specific ticket keys are given on the command line.

Embargoed tickets (security level “Embargoed Security Issue” or embargo status field set to true) are always skipped.

Trackers filed against a Hummingbird image rather than an RPM (a pscomponent:hi/<name> label, or CVE-YYYY-NNNN hi/<name>: in the Summary) are filed in error and are always skipped. Like other automation, this respects the human override: trackers assigned to someone other than the bot only get the skip-assignee notice. Otherwise each run logs a warning with the alert text and ticket link, and with --resolve open trackers get a tracker comment, the cve-needs-attention label, and a one-time Slack alert via --slack-webhook-url (":alert-siren: HUM CVE TRACKER FILED IN ERROR – TRACKER FILED AGAINST IMAGE AND NOT AN RPM", with the ticket link and an @-mention of prarit). A separate comment marker records successful Slack delivery. The notice, the delivery marker, and the label are checked independently, so a failed webhook call or Jira write is retried on the next run.

Tickets with no valid CVE-YYYY-NNNN in the CVE ID field or Summary (for example GHSA/OSV-only trackers) are postponed: analysis is skipped, an INFO message is logged, and with --resolve a one-time tracker comment is posted and the cve-needs-attention label is applied (open tickets only). Use --skip-no-cve to omit those tickets without posting the postpone notice or adding the label.

Jira mutations driven by ticket selection itself (the image tracker alert, skip-assignee notice, and no-CVE postponement above) run serially, before the per-ticket pool starts. Per-ticket analysis (sections 2-7) is read-only. With --resolve, each ticket’s decide step (analyze plus the Jira/advisory mutations in section 9) also runs on that same pool. Both use --max-workers threads (default: 8; also CVE_ANALYSIS_MAX_WORKERS). The advisory batch MR apply (section 8, push/create) stays serial after the pool drains. Results are collected as each ticket finishes but reported — both on stdout and in the --handoff-file payload — back in the same order Jira returned the tickets, regardless of which ticket finished first.

2. CVE Data Collection

For each ticket the tool:

  1. Extracts the CVE ID from the Jira CVE ID field (customfield_10667) when populated, otherwise from the Summary via a regex match for CVE-YYYY-NNNN patterns. If neither source has a CVE ID, analysis is postponed (see above).

  2. Loads the CVE record from a local clone of the cvelistV5 repository and parses the CVE 5.0 affected block, following the CVE 5.0 Product and Version Encodings specification. This includes:

    • Parsing lessThan and lessThanOrEqual version ranges
    • Handling lessThan: "*" (no upper bound) and lessThan: "4.*" (end-of-series, all versions in the 4.x series) wildcards per the spec
    • Processing changes lists that subdivide ranges into affected and unaffected segments
    • Respecting defaultStatus at the product level
    • Filtering out versionType: "git" entries that use commit hashes instead of numeric versions (these are preserved as informational data but not used for version comparison)
    • Detecting CNA data errors where git commit hashes are used in version fields without versionType: "git", including hashes wrapped in operator syntax (e.g., < 6374ae0bcdfe...)
    • Parsing inline comparison operators (< 12.3.0, >= 4.0, <= 2.5) that are not part of the CVE 5.0 schema but are widely used by CNAs in practice
    • Parsing compound range bands (>= 2.0, < 2.2.26 and > 1.32.3, < 1.34.6) used by some CNAs to express a closed-open range in a single version string
  3. Looks up version ranges from the NVD data feeds and merges them with the MITRE data. NVD data is obtained from the JSON 2.0 data feeds at nvd.nist.gov/feeds/json/cve/2.0/ (per-year .json.gz tarballs updated daily, plus a CVE-Modified overlay updated every 2 hours). Feeds are cached locally (--nvd-cache-dir) and only re-downloaded when the NVD .meta file SHA256 indicates newer data is available. When MITRE has no vendor-specific data (all vendors are “n/a”), NVD ranges replace the MITRE data entirely. NVD references tagged “Patch” or matching commit URL patterns are also extracted and used for upstream fix detection (see below).

  4. Looks up the Hummingbird package version by scraping the Pulp repository index at packages.redhat.com for the latest RPM matching the package name extracted from the ticket summary.

Pulp directory listings are parsed once and shared by version and SRPM lookups. The parsed filenames and upload times are cached by directory URL until clear_listing_cache() clears the cache between analysis cycles, so repository updates become visible after the cache is cleared.

3. Resolution Computation

The tool compares the shipped Hummingbird version against the affected version ranges to compute a recommended resolution:

  • Closed / Done-Errata: The repo version is explicitly unaffected (an exact version, an unaffected range, or product defaultStatus: unaffected), or NVD-only data establishes that it is past the fix/outside affected ranges. An exact unaffected version does not cover later versions. For CVE 5 data, when no entry matches, omitted defaultStatus means unknown, not safe. Overlapping affected ranges from multiple streams prevent a version-based close unless the version is explicitly unaffected.
  • In Progress / affected: The repo version falls within an affected range.
  • In Progress / affected (no version data): Neither MITRE nor NVD has affected version information. The ticket is moved to In Progress for manual review but does not receive the cve-needs-attention label.
  • In Progress / needs investigation: Version data was available but could not be compared (e.g., CVE only provides git commit hashes), or multiple distinct products with different versioning schemes are listed and no cve_product override is configured. These cases receive the cve-needs-attention label.

4. Product Mismatch Detection

The tool detects when a CVE was filed against the wrong Hummingbird package. It compares the CVE vendor/product names against the Hummingbird package name and upstream repo URL from the package map. For example, a CVE for isaacs/node-tar (an npm package) filed against the tar RPM (GNU tar) is flagged as a mismatch. This detection works even without a repo URL by comparing normalized product names.

Mismatch triage uses a two-phase evidence gate:

  1. Identity mismatch — CVE product/vendor does not match the package (cve_product override or heuristic name/repo matching)
  2. SBOM (+ binary) evidence — only then decide misfiled vs vendored vs needs investigation

Phase 2 fetches the latest package SBOM from the Hummingbird Pulp repository and records the artifact used (NVR/URL) plus match evidence. SBOM JSON is parsed incrementally; an invalid document is treated as unavailable. Go module matching accepts the module root and major-version paths (github.com/vendor/product or .../v5). Deeper import paths such as .../api or .../daemon only count as a hit when a CVE path hint (from malformed/version-path fields) aligns with that subpackage, so a client/API module does not satisfy a daemon-only CVE. If the CVE product is found as a vendored dependency, the tool runs binary confirmation in this order:

  1. SPDX SBOM evidence — sourceInfo pointing at shipped paths (e.g. Go buildinfo under /usr/bin/...) or CONTAINS from binary RPM roots (*.x86_64, *.aarch64, …) marks present_in_binary without Syft. Evidence only under .src / go.mod is treated as source-only for the next step.
  2. RPM Provides: bundled(...) — downloaded non-debug binary RPMs are queried with rpm -qp --provides. A matching Fedora bundled Provide (npm bundled(npm(name)), Go bundled(golang(IMPORT_PATH)), Node core bundled(nodejs-undici), etc.) marks present_in_binary with evidence_method=rpm_bundled_provides. Ecosystem wrappers such as npm(...) are unwrapped so a CVE/SBOM search for nanoid matches bundled(npm(nanoid)). This catches dependencies embedded into binaries (including minified frontend JS) that Syft cannot inventory as on-disk packages.
  3. Syft on binary RPMs — Syft scans the same RPMs, even after a Provides or SPDX hit, to find additional shipped copies. When OSIDB reports subpackage names that match RPMs for the NVR, only those RPMs are scanned for absence checks; vendored-version checks scan the full NVR. If OSIDB data is missing or the names do not match this package’s RPMs (for example a Redis CVE filed against boost), the full NVR set is used. Pulp download URLs use the arch from the RPM filename (aarch64 files under aarch64/, x86_64 under x86_64/; noarch stays on the x86_64 listing).

Those scans are reused. The process keeps provides tuples and Syft artifact lists in memory, and when RPM_SCAN_CACHE_S3_BUCKET is set it also stores that JSON in S3, keyed by RPM URL. An object is written only after both rpm --provides and Syft ran, so a cycle that could not run Syft is not later reused as a confirmed absence. --rpm-scan-cache-max-bytes (default 1.5 GiB) is a soft LRU budget on the bucket. Each analysis cycle clears the in-memory cache and leaves the S3 objects in place. The client is configured with RPM_SCAN_CACHE_S3_REGION, RPM_SCAN_CACHE_S3_ACCESS_KEY_ID, and RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY from the RPM scan cache ObjectBucketClaim.

Syft absence only becomes absent_in_binary_confirmed when SBOM evidence is source-only (or no SBOM document was available) and Syft actually ran. A miss with ambiguous SBOM evidence stays unknown so embedded Go/Rust modules are not closed as source-only when SPDX already ties them to a binary. A Provides-only scan (Syft unavailable) never confirms absence. Per-RPM download or scan failures are skipped so sibling RPMs can still confirm presence; a failed scan never proves absence or that all vendored copies are fixed. A Provides query failure falls through to Syft for that same RPM but leaves the version evidence incomplete for auto-closure.

  • present_in_binary (mismatch_vendored): legitimate vendored ticket; compare the confirmed binary/Provide version (falling back to the SBOM vendored dep version) against the CVE ranges (parent latest_repo_version / Hummingbird repo version stay the package), and apply cve-needs-attention so operators can review the binary hit. Matching versions across binary RPMs, Syft artifacts, SPDX binary packages, and bundled Provides are all collected. A binary scan must complete before the versions can justify auto-closure; a confirmed presence with an incomplete scan remains open for investigation. Resolution is worst-of / all-must-be-fixed: the ticket stays affected if any copy is still affected; Done-Errata only when every copy is not affected. The tool does not invent multi-major fix mappings from a single CVE fixed line — Fixed in Build remains the override when CVE version data only describes one release line. Placeholder dep versions such as cargo std@0.0.0 are not comparable: fall back to the parent package version when available; otherwise set non_comparable_vendored_version, recommend New / needs investigation, and apply cve-needs-attention — never Done-Errata from “0.0.0 is not in any affected range”
  • absent_in_binary_confirmed (mismatch_source_only): source-only; eligible to close as Not a Bug / Component not Present. Must never set Fixed in Build or take the advisory / Done-Errata path.
  • unknown (mismatch_binary_unknown): leave for investigation; do not auto-close or call the ticket misfiled

Stable reason codes are emitted in analysis output (Reason codes:) and on each CVE entry (reason_code / mismatch_gate) for automation:

Reason code Meaning
mismatch_sbom_miss Identity mismatch + SBOM miss → misfiled candidate
mismatch_sbom_unavailable Identity mismatch but SBOM could not be checked
mismatch_binary_unknown SBOM hit but binary presence unknown
mismatch_source_only SBOM hit confirmed absent from binaries
mismatch_vendored SBOM hit confirmed present in binaries → label for review
non_comparable_vendored_version Vendored version is a placeholder (e.g. 0.0.0) with no parent fallback → needs attention

A misfiled recommendation requires identity mismatch + SBOM miss (no vendored hit). Ambiguous checks and confirmed binary RPM matches keep cve-needs-attention with the reason code above. When present in binaries, normal resolution flow still proceeds (In Progress / Close) in addition to the attention label. The upstream fix search targets the parent package’s repo. Each analysis cycle re-fetches the SBOM and re-runs binary confirmation.

5. Upstream Fix Detection

Fix search uses a precedence model so tickets are not dual-labeled from both Fedora and upstream when a definitive path exists:

  1. CVE/NVD fix links (preferred): NVD references tagged “Patch” or matching commit URL patterns, plus CVE/NVD PR/MR references that match the package’s upstream repo. When these yield a fix label, Bodhi and forge search are skipped.
  2. Otherwise, repo-aware search from metadata/*.json upstream_repo:
    • Fedora-based (src.fedoraproject.org, pagure.io, or missing repo URL): Bodhi + DistGit only (see §6)
    • Non-Fedora forge (GitHub/GitLab/cgit): forge search only

Forge Search (non-Fedora fallback):

  • GitHub: Searches PRs via the GitHub API (/search/issues) for PRs mentioning the CVE ID. Fetches commit counts from the pulls API.
  • GitLab: Searches merge requests via the GitLab API on supported instances (gitlab.com, gitlab.gnome.org, gitlab.freedesktop.org, etc.). The --gitlab-token is only sent to gitlab.com to avoid 401 errors on other instances.
  • cgit: Scrapes commit log pages on cgit hosts (Savannah, Sourceware, kernel.org, busybox.net, etc.) for commits mentioning the CVE ID. Handles URL rewrites for Savannah hosts (e.g., git.savannah.gnu.org/git/ to cgit.git.savannah.gnu.org/cgit/).

Fix status is classified as:

  • upstream-fix-available: At least one merged/closed PR, committed fix, or NVD patch reference
  • upstream-fix-in-progress: At least one open PR with no merged fixes

6. Fedora Fix Detection

Used only for Fedora-based packages (or when no upstream repo is configured) and only when CVE/NVD fix links did not already produce a label. Hits are classified with the same upstream-fix-* labels as forge/NVD evidence (Bodhi update links are still shown in analysis output):

  • Bodhi: Queries the Fedora Bodhi API for updates matching the package name that reference the CVE ID (in the cves list or notes field). Supports multi-page results. Stable → upstream-fix-available; testing/pending → upstream-fix-in-progress.
  • DistGit Spec Scan: When Bodhi has no matches, fetches the Fedora DistGit spec file and scans for CVE references in patch filenames, changelog entries, and comments to detect backported fixes (upstream-fix-available).

7. Fixed Build Detection

The tool automatically detects whether the current Hummingbird build fixed a CVE by checking if all CVE IDs are mentioned anywhere in the package directory (patch names, changelogs, comments in the .spec file and so on). If found, it returns the current source RPM name. Assuming that we run the analysis frequently enough, this is precise enough, otherwise it errs on the side of caution (i.e. it possibly marks a higher version as fixed even if an earlier one already carried the fix).

The detected build is shown in the output as Detected Fix: package-version-release.hum1.src.rpm.

When --resolve is specified and a fixed build is detected, the tool automatically populates the “Fixed in Build” Jira field with the detected SRPM name, but only if:

  • The field is not already set (manual values take precedence)
  • The issue is not in Closed status (respects human closure decisions)
  • The assessment is not mismatch_source_only / source_only_confirmed (Component not Present closes as Not a Bug and clears any Fixed in Build)
  • The computed resolution is already Closed / Done-Errata or every ticket CVE ID appears on an uncommented PatchN: line (HUM-6182). A spec comment or leftover unapplied file that merely mentions the CVE is not enough. Product-mismatch and EOL tickets never take this PatchN override.

Version-range analysis can still say “affected” after a backport that does not bump Version (for example an unbounded range such as 1.1.1+ on popt 1.19). In that case an applied CVE-YYYY-NNNN.patch listed as PatchN: is allowed to set Fixed in Build. The SRPM must already be in Pulp (the existing fix_committed_not_built gate is unchanged). The ticket is not closed here; advisory review remains the backstop.

The RPMs repository can be provided via --rpms-repo, pointing to a local clone. This is preferable with multiple runs to avoid the git clone operation. When not given, the repository will be (shallow) cloned into a temporary directory.

Similarly, the cvelistV5 repository (used for CVE record lookups) can be provided via --cve-repo. When not given, a shallow clone is performed into a temporary directory. For production use, pre-clone and update the repo externally (e.g. via a CronJob) to avoid the clone overhead on each run. The tool does not attempt to update the provided repo itself.

8. Advisory Integration (with --resolve)

When a CVE is resolved as “Closed / Done-Errata”, the tool integrates with the CEE GitLab advisories repo to document the fix:

  1. Clone: The bot fork of the advisories repo (default upstream: releng/advisories) is cloned blobless (--filter=blob:none), synced to upstream main, and force-pushed to the fork. A blobless clone keeps the commit history that pushes to a stale fork need while downloading only the checked-out files; a --depth 1 clone would be rejected by the fork. The token is embedded in the git URL and redacted from logs.
  2. Modify: For each resolved ticket, the advisory YAML is updated: cves.fixed entries are added, the type is changed from RHBA to RHSA, and CVE references are appended.
  3. Batch MR: Per-ticket advisory YAML edits run inside the decide worker pool (serialized by an advisory write lock). Changes accumulate as individual commits on a single branch (cve-analysis/batch). One merge request is created after the pool drains, covering all tickets.
  4. Review state: Tickets are transitioned to “Review” (not directly closed) and the batch MR URL is posted to each affected ticket. The MR is approved and queued onto the GitLab merge train; the run does not wait for merge or the post-merge pipeline (HUM-7562).
  5. Auto-close: On subsequent runs, tickets in “Review” are checked: if the advisory MR has been merged, the ticket is closed as “Done-Errata”. The MR URL is taken from the ticket comment, or, if that comment is missing, from an advisories search by ticket key. If the MR has unresolvable rebase conflicts, a comment with manual resolution steps is posted and the advisory-mr-failed label is added.
  6. Slack notifications: When --slack-webhook-url (or the SLACK_WEBHOOK_URL env var) is set, the tool sends a Slack message when batch MR creation fails. Failure messages @-mention prarit and jstibran. It also alerts, with an @-mention of prarit, when a tracker is filed against an image instead of an RPM (see above).

The advisory flow is skipped when --advisories-project points to a non-production URL, allowing safe testing without modifying Jira.

The clone lives in a temporary directory and is removed at the end of the run. It is intentionally left on disk when the batch MR push fails or --keep-advisory-repo is passed, so an engineer can inspect it; a leftover clone from a prior run is automatically removed once it is a few hours old to prevent repeated failures from exhausting disk/inode space.

8.1 Recovery for failed advisory MRs

Troubleshoot an advisory MR

An advisory MR updates release metadata for a published fixed build.

What changes Advisory field Purpose
CVE associations spec.content.artifacts[].cves.fixed Records which CVEs the applicable artifacts fix.
Tracker links spec.issues.fixed Links the HUM trackers covered by the advisory.
Security references spec.references Adds links to CVE details and security information.
Advisory type spec.type Changes RHBA to RHSA when the update includes security fixes.
Advisory text, when needed spec.synopsis, spec.topic, spec.description Describes the security fixes and packages covered.

If a retry is not enough, the fix depends on the failure: resolve merge conflicts against the current advisory, or check that the CVE, tracker, Fixed in Build and artifact entries refer to the right package/build. Correct advisory text and references where needed, preserving unrelated fixes, artifact provenance, advisory IDs and ship dates. Permission failures need an authorized author/workflow rather than another retry.

  1. Open the Advisory MR linked in the Jira tracker comments or the failure notification. Check its pipeline, failed job logs and merge status.
  2. For a transient failure, retry the failed job or rerun the pipeline in GitLab. This is often enough; no advisory edits are needed.
  3. If retrying does not help, or the MR has conflicts, permission failures or needs an advisory correction, ask ProdSec Team for help. Include the MR URL, affected HUM tracker(s), failed job/error and what you retried. For manual changes, see the advisories repository’s contribution guide and current field permissions. Note: changes to spec.type require an allowlisted author and must be committed through the Hummingbird CVE Analysis <hummingbird-jira-bot-admins@redhat.com> bot.
  4. Confirm that the MR merges and its default-branch publication pipeline succeeds. Leave Done-Errata closure to advisory automation. If the tracker or dashboard remains out of sync, use the recovery steps below.

Recover lifecycle events after advisory delivery

In some failure cases (for example, advisory MR conflicts followed by manual ticket closure), Jira status can be corrected to Done-Errata while dashboard lifecycle metadata remains incomplete. This leaves R-Time rows PENDING (elapsed-to-now) until image, VEX, and close all exist.

Use the recovery script to backfill delivery events:

  • rpm_fix_published_to_pulp (hb_rpm_fix) from packages.redhat.com source RPM timestamps
  • image_rebuilt_on_quay (hb_image_fix) from catalog/quay image history
  • hum_ticket_closed from Jira resolutiondate

Only tickets that are Closed / Done-Errata are used for event derivation and dashboard import. Tickets that remain in Review (or any non-closed state) are reported and skipped for event backfill.

For tickets still in Review, the script also attempts advisory reconciliation before deriving events:

  • Read advisory MR URL from Jira comments (Advisory MR: https://...)
  • If no MR URL is present in comments, search merged advisories MRs for the HUM ticket key in MR description/title
  • When a merged MR is found and --apply is used, transition the ticket to Closed / Done-Errata and then derive hum_ticket_closed

Script location:

  • hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py

Run in dry-run mode first (default):

cd /path/to/tools
export JIRA_TOKEN=your_jira_token
export CEE_GITLAB_TOKEN=your_cee_gitlab_token
export CVE_REPORT_TOKEN=your_dashboard_token

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json

Apply the backfill to dashboard event storage:

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json \
  --apply

Target specific tickets (instead of default JQL):

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --apply \
  HUM-4884 HUM-4871 HUM-4844

Verification example:

# Replace with the actual dashboard URL (same as --dashboard-url default)
DASHBOARD_URL=https://hummingbird-dashboard-hummingbird--internal.apps.int.spoke.prod.us-east-1.aws.paas.redhat.com
curl -s "${DASHBOARD_URL}/api/cve/r-time?days=30" | jq -r '
  .entries[]
  | select(.key=="HUM-4884")
  | [.key, (.fix_delivered_at // "-"), (.stages.hb_rpm_fix // "-"), (.stages.hb_image_fix // "-")]
  | @tsv
'

If hb_rpm_fix and hb_image_fix remain empty in the derived JSON output, the issue is source-data availability (no resolvable RPM/image publish signal), not dashboard ingestion.

If a ticket is in Review and no advisory MR can be found in Jira comments or merged advisories MRs, it is skipped and reported for manual follow-up. By default, the script targets advisory-mr-failed tickets (for both Closed / Done-Errata and Review states); override with --jql when needed.

8.2 Dashboard collection ticket selection

Dashboard lifecycle collection (HUM-5603) uses a separate ticket-selection path from cve_analysis mutations. Helpers live in hummingbird_cve_analysis/dashboard_selection.py.

Watermark (start of previous run): read the newest entry from the dashboard Run Log (GET /api/cve/run-log?limit=1). Use run_started_at when available; older entries estimate the start as recorded_at - duration_seconds (recorded_at is ingest time). Subtract a small buffer (default 2 minutes) for clock skew. The overlap avoids missing tickets updated while the previous collector run was in progress and is safe because /api/cve-report event upserts are idempotent.

JQL selection:

  • All still-open HUM Security tickets (rescan for updates)
  • Closed tickets with Jira updated >= watermark
  • Bootstrap (empty Run Log): open tickets only
  • Explicit ticket keys: key in (...) (any status)

Use --start-time (ISO-8601, e.g. 2026-08-01T00:00:00Z) to override the Run Log watermark and recollect Closed tickets with updated >= that start, so dashboard data can be overwritten/backfilled.

Inspect watermark and JQL without modifying Jira or the dashboard. From the tools repo root, either install the package (make hummingbird-cve-analysis/setup) or set PYTHONPATH:

cd /path/to/tools
export PYTHONPATH=hummingbird-cve-analysis
export CVE_REPORT_TOKEN=your_dashboard_token

python3 -m hummingbird_cve_analysis.dashboard_selection --prod

# Optional: also search Jira and list matching keys
export JIRA_TOKEN=your_jira_token
python3 -m hummingbird_cve_analysis.dashboard_selection \
  --prod \
  --jira-user user@example.com \
  --fetch-issues -o json-pretty
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for /api/cve/run-log
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--watermark-buffer-minutes Minutes subtracted from previous-run start estimate (default: 2)
--fetch-issues Also search Jira with the constructed JQL
--jira-token / --jira-user JIRA_TOKEN Jira credentials (only with --fetch-issues)
--jira-url JIRA_URL Jira base URL (only with --fetch-issues)
--max-results Max issues to fetch with --fetch-issues (default: 2000)
--output, -o human, json, or json-pretty

8.3 Analysis → collector handoff file

cve_analysis writes a small JSON handoff file for collect_cve_dashboard (HUM-5799 / HUM-5800). This carries Run Log counters, captured log_output, and per-ticket fields that are expensive to recompute (upstream_fix, fedora_fix) plus human_text for Run Log details. Delivery timestamps (hb_rpm_fix / hb_image_fix) and hum_ticket_closed are gathered by the collector, not this file. Analysis stdout is human text only; it does not emit /api/cve-report JSON.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.cve_analysis \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --handoff-file /tmp/cve_analysis_handoff.json \
  ...

Handoff shape (schema_version: 1):

  • Top level: run_started_at, duration_seconds, mutation counters (labels_changed, advisories_created / advisories_failed, tickets_closed, comments_posted, attachments_uploaded, fatal_errors), log_output
  • tickets[]: key, upstream_fix, fedora_fix, human_text
  • vex_updates[] (HUM-5843): key, vex_status, vex_match_state, vex_resolved, labels from the awaiting-vex reconcile pass

8.4 collect_cve_dashboard

collect_cve_dashboard (HUM-5798 / HUM-5800) owns dashboard JSON emission. cve_analysis no longer prints /api/cve-report JSON on stdout; it prints human analysis text and optionally writes --handoff-file for the collector.

  1. Resolve Run Log watermark + open-ticket rescan (section 8.2). Selection always includes Closed tickets labeled awaiting-vex (HUM-5843), awaiting-cve-published, or awaiting-image-rebuild (HUM-9975), then any vex_updates[] keys missing from that JQL
  2. For each ticket, gather slim lifecycle fields from Jira / cvelistV5 / Fixed-in-Build → Pulp (cve_published, hum_ticket_created, hum_ticket_closed, hb_rpm_fix) and catalog image history (hb_image_fix), plus cve_ids – every CVE ID the ticket tracks, from its dedicated CVE ID Jira field (falling back to Summary text), not just the first found in labels; a ticket can legitimately track more than one CVE, and consumers keyed by CVE ID (e.g. the dashboard’s Competitive page HUM ticket matching) index it under every entry here, not just one (see hummingbird-dashboard.md), plus OSIDB osidb_flaw_created / osidb_affect_created from the public OSIDB API (omitted when OSIDB has no data), cve_embargo_lifted – the OSIDB flaw unembargo_dt, included only when it postdates the flaw’s created_dt (i.e. the CVE was genuinely embargoed; omitted otherwise) – and osidb_flaw_public – the same unembargo_dt, but included unconditionally whenever OSIDB reports one, regardless of whether the flaw was ever embargoed – computed_resolution from Jira status/resolution (R-Time gates on Done-Errata in that string), catalog_image_source from the catalog source map (R-Time delivery is image publish when true, RPM publish when false), and catalog_images – every {image}:{stream} the ticket’s package actually appears in, per the same catalog source map’s SBOM data (used by the dashboard’s Competitive page to match a ticket to a specific image precisely instead of guessing from the package name; see hummingbird-dashboard.md). Timestamp helpers live in lib/lifecycle.py and are shared with analyze_issue
  3. Merge --handoff-file counters, log_output, per-ticket upstream_fix / fedora_fix / human_text, and vex_updates
  4. Collect agent costs: every ticket carrying a cve-agent-* label is found with one paginated key-only Jira search, independent of this run’s watermark selection – an unchanged Closed ticket would otherwise never be retried after a failed download. Unrecorded keys (checked in batches against /api/cve/agent-cost-keys) have their <TICKET>-cve-agent-analysis.json attachment downloaded for its token_cost block, and the result travels as the report’s agent_costs list. A ticket without an attachment yet is retried on the next run; an attachment predating cost reporting is recorded once with a null estimated_cost_usd so it is not downloaded again.
  5. Dry-run prints /api/cve-report JSON; --apply POSTs it

The collector builds the catalog source map once per run for hb_image_fix, catalog_image_source, and catalog_images. When a delivery timestamp is omitted, the dashboard keeps any existing rpm_fix_published_to_pulp / image_rebuilt_on_quay value rather than clearing it. An empty or failed catalog map omits catalog_image_source and catalog_images so the dashboard keeps requiring an image / falls back to its own package-name heuristics.

Catalog fetches (an image’s tag listing, or a specific stream’s SBOM) are not retried; a failure is logged and that image/stream is silently skipped when building the source map, so catalog_images’s absence of an image/stream is not always confirmed ground truth – it can also mean that image/stream’s data simply failed to collect this run. build_catalog_source_map() returns the set of (image, stream) pairs it couldn’t verify alongside the map (stream is None when the image’s own tag listing failed, so every one of its streams is unknown). Every ticket in the run carries this set as catalog_unverified_images (rendered as {image}:{stream}, or {image}:* for a whole unverified image) so a downstream consumer – the dashboard’s _match_cve_tickets – can tell “confirmed absent” from “unknown” before excluding a ticket candidate based on catalog_images alone (see hummingbird-dashboard.md).

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json

# POST to preprod dashboard
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json \
  --apply
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for Run Log + /api/cve-report
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Analysis handoff JSON from cve_analysis --handoff-file
--apply POST report to /api/cve-report (default: dry-run JSON on stdout)
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--cve-repo Local cvelistV5 repo for cve_published
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--output, -o human, json, or json-pretty

8.5 verify_image_publish

verify_image_publish (HUM-721) catches CVE tickets that claim a fix via image rebuild (hb_image_fix in the whiteboard cve_cycle) but whose image was never actually published to the registry — hb_image_fix is set from catalog release-history metadata (oldest_created), which can lag or be wrong; this re-verifies with a live check.

  1. Search Jira for Closed HUM Security tickets updated within the last --window-minutes (default 60)
  2. For each, read hb_rpm_fix / hb_image_fix from the whiteboard; skip tickets where hb_image_fix falls outside the window
  3. Re-derive the source package name from the summary/labels (no SBOM / network calls) and re-run the same catalog-history lookup (find_image_publish_release) that originally produced hb_image_fix, recovering the specific image/stream/variant/digest
  4. HEAD the image manifest by digest directly against the registry (image_manifest_exists) — this is the part hb_image_fix itself never did
  5. Post a Slack alert for each unverified claim; log a summary either way

Read-only: never modifies Jira, the dashboard, or the CVE analysis pipeline.

PYTHONPATH=hummingbird-cve-analysis python3 -m hummingbird_cve_analysis.verify_image_publish \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --window-minutes 60
Option Environment Variable Description
--window-minutes Look back this many minutes for hb_image_fix (default: 60)
--slack-webhook-url SLACK_WEBHOOK_URL Webhook for unverified-claim alerts
--dry-run Check and log, but do not post to Slack
--catalog-api-base Catalog API base URL
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--jira-url JIRA_URL Jira base URL

9. Jira Actions (with --resolve)

When --resolve is specified, the tool modifies Jira tickets. Those per-ticket mutations run concurrently with analysis on the --max-workers pool (HUM-11537); only the advisory batch MR push/create (section 8) runs once after the pool finishes. Shared advisory-repo writes are serialized by an internal lock; Jira API GET/PUT retries cover transient conflicts.

The deployed analysis CronJob also uses --publish-cve-agent-sweep with --resolve. After a completed cycle it publishes a small SNS event (source=cve-analysis, event_type=cve_agent_sweep_requested) to wake the CVE-agent SQS worker. The event contains only the run timestamp; the agent searches Jira for eligible tickets itself. SNS publishing failures are logged without undoing or failing completed Jira analysis; the next scheduled run publishes another request. See the message bus contract for the exact message body and attributes.

  • Fixed in Build: When a fixed build is detected and the field is not already set, the tool populates it with the detected SRPM name (only for non-closed issues). Not-affected Closed / Done-Errata closes also set Fixed in Build from detected_fixed_build or latest_srpm when empty, so R-Time image timestamps can be resolved.
  • Closed / Done-Errata: Tickets where the shipped version is not affected are transitioned to Review with an advisory MR (see above). After the MR merges, they are closed as Done-Errata on the next run.
  • Move to In Progress: Tickets where the shipped version is affected are transitioned to In Progress with a comment including affected version ranges, upstream fix status, and Fedora update links.
  • Needs Investigation: Tickets with incomplete version data get a comment explaining why automatic resolution was not possible.
  • Product Mismatch: A comment is posted explaining the mismatch (with SBOM artifact evidence) and the cve-needs-attention label is applied. No transition is performed. Auto-detected Fixed-build / delivery timestamps (hb_rpm_fix, hb_image_fix) are cleared so R-Time does not treat misfiled tickets as done. Human Fixed in Build overrides are kept. When --resolve runs against the production advisories project, the analyzed package SBOM JSON is also attached to the ticket as {nvr}.sbom.json (skipped if that filename is already present). The same attachment is applied for other SBOM-backed review paths (vendored binary hits, source-only closes, binary-unknown).
  • Package Not Present: If the package is missing from the rpms repo, the tool still runs an SBOM-first check against the CVE product(s). An SBOM hit triggers Syft binary confirmation: absent_in_binary_confirmed closes as Not a Bug / Component not Present; presence or unknown leaves the ticket for investigation. An SBOM miss/unavailable closes as Not a Bug with VEX Component not Present. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Package EOL (fix_status: 0): Tickets for packages marked End-Of-Life in rpms metadata are closed as Won’t Do with no VEX Justification. This takes precedence over Done-Errata and Fixed-in-Build / advisory paths. Automation labels (upstream-fix-*, legacy fedora-fix-*, cve-needs-attention, cve-next-release, advisory-mr-failed) are removed; fedora-bz-filed is kept as an audit record. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Label Management: Labels are applied and updated:
    • upstream-fix-available / upstream-fix-in-progress (including Bodhi/DistGit evidence for Fedora-based packages)
    • cve-needs-attention (applied when human review is needed; removed when resolved)
    • Legacy fedora-fix-* labels are stripped on subsequent runs
    • Labels are upgraded (in-progress to available) and stale labels are cleaned up on ticket closure.

Every Jira action (label add/remove, status transition, product mismatch) includes a single comment with a bold action summary heading followed by the full analysis output in a preformatted code block, giving the reader the same detail they would see on the CLI.

10. Assignee-Based Automation Control

When --resolve is active, the tool checks each ticket’s assignee. If the assignee is not the bot account (--jira-user), the tool skips all automated actions (comments, labels, transitions) for that ticket. A one-time comment is posted explaining that automation is skipped. This allows humans to take ownership of a ticket by assigning it to themselves, preventing the bot from interfering with manual work. Reassigning back to the bot account re-enables automation.

For testing purposes, use --skip-assignee-check to process all tickets regardless of assignee.

11. Human Closure Protection

When the tool encounters a Closed ticket whose analysis recommends a non-Closed resolution (e.g., In Progress), it checks the Jira changelog to determine who performed the last status transition. If a human (not the bot) closed the ticket, the tool skips the transition to respect the human’s decision. This prevents the tool from reopening tickets where a fix was backported without a version bump or where a human determined the CVE does not apply.

12. Stale Closure Detection (with --include-closed)

When --include-closed is used, the tool also analyzes Closed tickets. If the current analysis recommends a non-Closed resolution (e.g., the package version changed and is now affected), a warning is emitted flagging the ticket for review.

13. Sprint and Epic Assignment

When --resolve is active, the tool automatically adds CVE tickets to the current Hummingbird sprint and links them to a per-sprint CVE tracking epic when human interaction is detected. This ensures sprint metrics capture human work on CVE tickets.

The tool queries the Jira Agile API for the active sprint on the Hummingbird board (--board-id, default 1489) and filters by name prefix (--sprint-prefix, default “Hum S”) to identify the correct sprint among multiple teams sharing the HUM project.

Sprint and epic assignment is triggered in two cases:

  • Self-assigned tickets: When a ticket is assigned to someone other than the bot, the ticket is added to the current sprint and linked to the sprint’s CVE epic.
  • Human-set Fixed in Build: When the Fixed in Build field was set by a human (not the bot), as determined by the issue changelog, the ticket is added to the current sprint.

If no matching epic exists for the current sprint, one is created automatically with the summary “CVE tickets for sprint <name>”. The epic key is cached for the duration of the run. A Jira comment is posted on the ticket recording the sprint and epic assignment.

Tickets that already have a sprint assigned are skipped. Tickets closed manually without going through either automated path should be added to the sprint by hand (see the manual process guide).

14. Continuous Operation

When --time-between-runs N is set (N > 0), the tool runs in a loop, re-executing the full analysis every N minutes. The default is 0 (single run). Shutdown is graceful: SIGINT/SIGTERM finishes the current cycle before exiting.

15. Feature Flag

When --feature-flag-name is set (default: cve_analysis_enabled), the tool checks a GitLab feature flag on the project specified by --feature-flag-project-id (default: 73447720, i.e. redhat/hummingbird/rpms) at the start of each cycle. If the flag is inactive, the cycle is skipped and the tool sleeps for 60 seconds before checking again.

The check uses the --gitlab-token / GITLAB_TOKEN credential, which must have Developer role or higher on the target project. The check is fail-open: if GitLab is unreachable or the token lacks permissions, the tool assumes the flag is enabled and proceeds.

Set --feature-flag-name "" to disable the check entirely.

16. Timing Instrumentation

A @timed(label) decorator (hummingbird_cve_analysis.lib.timing) wraps the heavier per-ticket network calls – NVD (fetch_nvd_ranges, get_nvd_references), Pulp (fetch_hummingbird_versions, fetch_hummingbird_latest_srpm_with_time, fetch_hummingbird_latest_srpm_as_of, fetch_srpm_upload_time), forge search (search_forge), Fedora (fetch_bodhi_updates), and SBOM/Syft (fetch_latest_sbom, confirm_component_in_binary_rpms) – timing each call with time.perf_counter().

Two levels of output:

  • Per ticket: _analyze_one_issue() (read-only) logs <KEY> analysis timing: ...; with --resolve, _process_one_ticket() logs <KEY> process timing: ... covering analyze plus decide. Both include the per-label breakdown for that ticket and are logged via finally, so they still appear when the ticket raised.
  • Per cycle: after the ticket loop, one Cycle sub-step timing: ... line aggregates every label’s total seconds and call count across the whole cycle (all tickets, all workers).

The per-ticket breakdown is isolated per ticket even under the --max-workers thread pool (HUM-10139 / HUM-11537): it’s tracked with a contextvars.ContextVar, which each worker thread gets its own copy of, rather than a single shared dict two concurrent tickets could stomp on. The cycle-level totals are a lock-guarded dict shared across all workers, the same pattern as resolve.increment_run_stat() (HUM-10136).

Before/after: validating the HUM-10139 parallelization

Three rounds of paired runs (sequential baseline at the commit just before HUM-10139, vs. the parallelized version, same frozen ticket list and shared NVD/cvelistV5/rpms-repo caches for both runs in a pair, run back-to-back) measured the actual wall-clock effect of parallelizing Phase 1:

Round Pairs Tickets/run Mean speedup Aggregate speedup Worst single run
1: most-recent 150 tickets 10 150 1.72x 1.47x 0.43x
2: random sample of 200 tickets 10 200 2.77x 2.74x 1.84x
3: all currently-open tickets 10 309 1.43x 1.41x 0.95x

Across all three rounds (30 pairs, ~5,400 ticket-analyses total), resolution output matched between the sequential and parallel run of each pair with zero correctness regressions; every mismatch traced to a live external cause (a Hummingbird build publishing to Pulp between the two runs of a pair, or the real production bot mutating a ticket’s Jira fields concurrently), not to the code.

Run-to-run variance (both within and across rounds) is dominated by a small number of tickets needing large, multi-architecture RPM downloads for SBOM/Syft binary confirmation – real network/CDN time that varies independent of sequential vs. parallel execution, and that a bounded thread pool can’t finish faster than its slowest in-flight ticket. Round 2’s random sample, less weighted toward any one such cluster, shows both the highest and the most consistent speedup of the three.

See Also

  • CVE Analysis Reference — Package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output, logs, library layout, development, license

4.5.13.1 - CVE Analysis Reference

Reference and operational documentation for the CVE Analysis tool. This page covers package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output format, logs, library layout, development, and license. For conceptual documentation (CVE lifecycle, how the analysis pipeline works), see CVE Analysis.

Package Map

The package map is loaded from the rpms repo’s per-package metadata files (metadata/<package>.json). Each file contains an upstream_repo field pointing to the canonical upstream git repository, plus optional fields:

  • upstream_branch – upstream branch (for versioned packages sharing a repo)
  • cve_product – CVE vendor/product override
  • version_transform – version transform rule
  • fix_status – package fix policy (0 = End-Of-Life / will not fix CVEs; close as Won’t Do)

If fix_status is 0, analysis always resolves to Closed / Won’t Do (including when the shipped version is outside the CVE affected range or a product mismatch would otherwise need investigation). No VEX Justification is set. If fix_status is missing or set to 1, behavior is unchanged from the default analysis flow.

When an upstream fix is available, Jira backport guidance keeps the NEXT STEPS: Backport upstream patches to Hummingbird marker and reminds maintainers to keep independent packages’ modification_status as independent and omit modification_reason, mark Fedora-imported packages modified with a modification_reason, and leave metadata release unchanged. This guidance does not affect the CVE assessment or fix decision.

The cve_product field is an optional human-curated value that specifies which CVE vendor/product entry maps to this package. It supports these formats:

  • Vendor / Product (exact match): matches a specific vendor and product pair. Example: "F5 / NGINX Open Source" for the nginx package, which excludes NGINX Plus entries that use incompatible R-versioning.
  • Vendor (vendor prefix match): matches any product whose vendor starts with the given string. Example: "Go " for golang packages, where the vendor varies (Go standard library, Go toolchain, etc.) and the product varies by module (net/url, os, crypto/x509).
  • Multiple selectors: provide more than one acceptable value either as a JSON list in metadata (preferred) or as a semicolon-delimited string. Example: ["vda-linux / busybox_mirror", "BusyBox / BusyBox"] or "vda-linux / busybox_mirror; BusyBox / BusyBox". The tool treats this as “match any selector”.

When cve_product is set, the tool uses it for exact product matching in both resolution computation (filtering to the correct product in multi-product CVEs) and mismatch detection. When empty, the tool falls back to heuristic name matching.

The rpms repo is cloned automatically at startup (or provided via --rpms-repo). A CSV override can be passed via --package-map for backward compatibility.

Vendored Dependency Detection

Vendored dependency detection uses live SBOM lookups from the Hummingbird Pulp repository. When a CVE product mismatch is detected, the tool fetches the SPDX SBOM for the ticket’s package from packages.redhat.com/.../metadata/sboms/{package}-main/ (same public index used by the rpms CVE skill), selects the newest dated sha256-….sbom entry from that listing, and searches that document for the CVE product(s) only (via purl references), stopping on the first product hit. Binary evidence then collects all matching copies of that product. The analysis output always records the SBOM artifact used and whether the lookup hit, missed, or was unavailable.

The same SBOM-first check also gates Component not Present closures for packages missing from the rpms repo. An SBOM hit alone no longer blocks that closure: binary confirmation must reach absent_in_binary_confirmed (no bundled Provide match and Syft miss on source-only SPDX evidence) before Not a Bug / Component not Present is recommended. A binary hit or unknown result keeps the ticket open for investigation.

Go subpackages are matched by progressively stripping path components (e.g., github.com/jackc/pgx/v5/pgproto3 matches github.com/jackc/pgx/v5). When a match is found and confirmed in binaries, the confirmed binary/Provide version is preferred for resolution over the SBOM lockfile version. The SBOM and binary confirmation are re-run each analysis cycle.

Requires the rpm and/or syft CLI on PATH (both are available in the analysis container image via rpm-build and the Syft install). Without either tool, binary confirmation returns unknown. With only rpm, a bundled Provide can still confirm present_in_binary, but absence is never auto-confirmed without Syft.

The generate_vendored_map_sbom.py script in package_maps/ can still be used for auditing vendored dependencies across all packages, but is no longer required at runtime.

Prerequisites

  • Python 3.11 or later
  • Jira API token (Bearer or Basic auth)
  • Network access to redhat.atlassian.net, github.com, nvd.nist.gov, bodhi.fedoraproject.org, src.fedoraproject.org, and packages.redhat.com (note: github.com is only needed when --cve-repo is not provided and the tool must clone cvelistV5 itself)
  • rpm and syft CLIs on PATH (for binary RPM confirmation after SBOM hits: bundled Provides via rpm, component inventory via Syft)
  • GitHub API token (optional, for upstream PR search)
  • GitLab API token (optional, for gitlab.com MR search)
  • CEE GitLab API token (required with --resolve, for advisory repo)

Usage

# Basic usage with Jira token from environment
export JIRA_TOKEN=your_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Analyze specific tickets
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com HUM-796 HUM-518

# Show only tickets from the last 2 weeks
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --show-since "2 weeks"

# Analyze with upstream fix detection (report findings without modifying Jira)
export GITHUB_TOKEN=your_github_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Resolve tickets and apply labels
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --resolve

# Include closed tickets and check for stale closures
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --include-closed

# Include closed tickets but exclude specific ones
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --include-closed --exclude "HUM-555,HUM-552"

# Skip issues without CVE links; write collector handoff
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --skip-no-cve --handoff-file /tmp/cve_analysis_handoff.json

# Continuous mode: resolve tickets every 30 minutes
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --resolve --time-between-runs 30

Dashboard /api/cve-report JSON is produced by collect_cve_dashboard (section 8.4), not by cve_analysis stdout (HUM-5800).

Inspecting lifecycle timing with dump_lifecycle

scripts/dump_lifecycle reads lifecycle milestones and package data from the dashboard HTTP API and prints them for one or more HUM tickets. CVE_REPORT_TOKEN is needed.

# Text output for one ticket against prod
scripts/dump_lifecycle --prod HUM-1234

# JSON output for multiple tickets against preprod
scripts/dump_lifecycle --preprod --json HUM-1234 HUM-5678

# Custom dashboard URL
scripts/dump_lifecycle --dashboard-url https://... HUM-1234

Output includes all milestone timestamps (cve_published, osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, hum_ticket_created, …), computed duration legs, and rpm_first_published (package) sourced from package_lifecycle. The --json flag emits the output as a JSON array instead of human-readable text.

Backfilling OSIDB timestamps with backfill_osidb_timestamps.py

scripts/backfill_osidb_timestamps.py is a one-time helper that fills osidb_flaw_created and osidb_affect_created for tickets already stored in cve_ticket_events. It reads existing rows from /api/cve-export, looks up timestamps from the public OSIDB API, and POSTs new rows to /api/cve-import with reset=false (write-once; existing timestamps are kept). Tickets that already have both OSIDB events are skipped unless --force is set. CVE_REPORT_TOKEN is needed.

# Backfill prod
scripts/backfill_osidb_timestamps.py --prod

# Dry run against preprod
scripts/backfill_osidb_timestamps.py --preprod --dry-run

# Limit to specific tickets
scripts/backfill_osidb_timestamps.py --prod HUM-1234 HUM-5678

Ongoing collection is handled by collect_cve_dashboard. After a prod backfill, copy_prod_to_preprod_db copies the new event types with the rest of cve_ticket_events (they are on the dashboard import allowlist).

Syncing prod CVE data to preprod with copy_prod_to_preprod_db

scripts/copy_prod_to_preprod_db copies cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame from prod to preprod via a direct pg_dump/pg_restore over two kubectl port-forward tunnels, instead of paginating them through /api/cve-export -> /api/cve-import. cve_analysis_log rows can carry up to 512 KiB of log_output, so the HTTP path needs hundreds of small paginated requests and aborts entirely on the first 5xx; the direct-DB path does the whole sync in one dump/restore pair, typically well under a minute end to end.

The script manages its own port-forwards — killing anything already bound to the local ports first — and retries pg_dump/pg_restore if a tunnel goes stale mid-transfer (a kubectl port-forward can keep its local listener up while silently dropping all traffic). It also checks kubectl auth whoami on both contexts before doing anything, printing the OpenShift console login URL if a token has expired.

Prerequisites: kubectl, pg_dump, pg_restore, and pg_isready on PATH, and an active oc login to both the prod and preprod OpenShift clusters.

# Check auth, start tunnels, and preview the planned pg_dump/pg_restore
# commands; no data is transferred
scripts/copy_prod_to_preprod_db --dry-run

# Run the sync (destructive on preprod; --yes is required)
scripts/copy_prod_to_preprod_db --yes

# Sync a subset of tables only
scripts/copy_prod_to_preprod_db --yes --table cve_analysis_log

A stalled tunnel is detected by watching for no progress (growing dump file size on export, pg_isready on restore) for --stall-timeout seconds (default 300s — preprod’s Postgres pod runs under a 500m CPU limit and has been observed taking 150s+ to respond during a checkpoint, so the default gives 2x margin above that; a shorter timeout causes false-positive “dead tunnel” restarts). On a stall, the affected tunnel is restarted and the step retried up to --max-attempts times (default 3); pg_restore --clean --if-exists makes every retry idempotent. A timing summary (per phase and total) prints at the end.

Collecting rpm_first_published with collect_rpm_first_published

scripts/collect_rpm_first_published sweeps the Hummingbird Pulp repo for all packages, finds the earliest SRPM upload timestamp for each, and writes the results to the package_lifecycle table via POST /api/cve-export - no direct DB access.

# Sweep all packages from the rpms repo and write to prod
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

# Only process specific packages
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --package curl wget

# Dry run — log what would be written without posting
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --dry-run

Re-runs are safe: the dashboard upsert only updates occurred_at if the incoming timestamp is earlier than the stored one.

Analyzing CVE Response Delays with analyze_rpm_first_published

scripts/analyze_rpm_first_published inspects the relation between CVE publication, package onboarding (rpm_first_published), and HUM ticket creation:

# Analyze all tickets on prod
scripts/analyze_rpm_first_published --prod --token $CVE_REPORT_TOKEN

# Filter to new packages onboarded after CVE publication
scripts/analyze_rpm_first_published --prod --filter new-pkg

# Filter to pre-existing packages with at least 30 days actionable delay
scripts/analyze_rpm_first_published --prod --filter pre-existing --min-days 30 --sort actionable

CLI options:

  • --filter {all,new-pkg,pre-existing}: filter by onboarding timing relative to CVE.
  • --min-days N: minimum delay threshold in days (default: 0).
  • --sort {cve_to_hum,actionable,cve_to_rpm}: sort column (default: actionable).
  • --limit N: max rows to display (default: 50).

Configuration

Option Environment Variable Description
--jira-token JIRA_TOKEN Jira API or Bearer token
--jira-url JIRA_URL Jira base URL (default: https://redhat.atlassian.net)
--jira-user Username for Basic auth
--output, -o Stdout format: human only (json / json-pretty removed in HUM-5800)
--show-since Filter by creation period (e.g. 2 weeks, 3 hours)
--resolve Transition Jira tickets and apply labels
--skip-assignee-check Skip assignee validation (for testing)
--include-closed Include Closed tickets in analysis; warn on stale closures
--exclude Comma-separated ticket keys to skip (e.g. HUM-555,HUM-552)
--skip-no-cve Omit issues with no CVE ID (CVE ID field or Summary); no postpone comment/label
--max-results Max number of issues to fetch (default: 2000)
--max-workers CVE_ANALYSIS_MAX_WORKERS Worker threads for per-ticket analysis and (with --resolve) decide mutations (default: 8)
--github-token GITHUB_TOKEN GitHub API token for upstream PR search
--gitlab-token GITLAB_TOKEN GitLab API token for gitlab.com MR search
--nvd-cache-dir NVD_CACHE_DIR Directory for caching NVD data feed files; avoids re-downloading unchanged feeds
--rpm-scan-cache-s3-bucket RPM_SCAN_CACHE_S3_BUCKET S3 bucket for durable RPM provides and Syft artifact JSON; disabled if unset
--rpm-scan-cache-s3-region RPM_SCAN_CACHE_S3_REGION Region for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-s3-access-key-id RPM_SCAN_CACHE_S3_ACCESS_KEY_ID Access key for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-s3-secret-access-key RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY Secret key for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-max-bytes RPM_SCAN_CACHE_MAX_BYTES Soft LRU size budget in bytes for the RPM scan S3 cache (default 1.5 GiB)
--cee-gitlab-token CEE_GITLAB_TOKEN CEE GitLab token for advisory repo operations
--advisories-project ADVISORIES_REPO Advisories repo URL (default: releng/advisories)
--advisories-fork ADVISORIES_FORK Bot’s fork URL for advisory MR creation
--slack-webhook-url SLACK_WEBHOOK_URL Slack webhook URL for failure and image-tracker alerts; empty or unset disables Slack
--no-merge-request Skip advisory MR creation during --resolve
--keep-advisory-repo Do not delete the cloned advisory repo after the run (useful for debugging)
--test-advisory Create advisory MR then immediately close it (for testing)
--package-map Path to CSV override; by default uses rpms repo metadata
--rpms-repo Path to RPMs git repo for fixed build detection
--cve-repo Path to local cvelistV5 git repo for CVE record lookups
--time-between-runs Re-run every N minutes; 0 = single run (default)
--feature-flag-project-id GitLab project ID for feature flag lookup (default: 73447720, rpms project)
--feature-flag-name Feature flag name to check each cycle (default: cve_analysis_enabled; empty = disabled)
--board-id Jira Agile board ID for sprint lookup (default: 1489)
--sprint-prefix Sprint name prefix to identify Hummingbird sprints (default: Hum S)
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Write analysis→collector handoff JSON (counters, log, fix times, human_text)
SENTRY_DSN Optional Sentry DSN for error tracking

Managed Labels

The tool manages the following labels on Jira tickets. These are automatically applied, upgraded, and cleaned up:

Label Meaning
upstream-fix-available A fix exists (forge PR/commit, NVD/CVE link, or Fedora Bodhi/DistGit)
upstream-fix-in-progress A fix is in progress (open forge PR, or testing/pending Bodhi update)
fedora-bz-filed A Fedora Bugzilla has been filed for this CVE
cve-needs-attention Analysis needs human attention (see below)
advisory-mr-failed Advisory MR has unresolvable rebase conflicts
cve-next-release Fix will arrive via the next upstream release (set manually; see below)
awaiting-vex Closed ticket awaiting VEX agreement with Jira resolution (HUM-5843 work queue)
awaiting-cve-published Ticket (open or closed) awaiting a cve_published date from cvelistV5
awaiting-image-rebuild Closed Done-Errata ticket awaiting its image rebuild on Quay (HUM-9975)

Legacy fedora-fix-available / fedora-fix-in-progress labels are no longer applied; the bot removes them on subsequent runs (Bodhi/DistGit evidence now uses upstream-fix-*).

Labels are upgraded automatically (e.g., upstream-fix-in-progress is replaced by upstream-fix-available when a fix is merged). Stale labels are removed when tickets are closed, except fedora-bz-filed which is preserved as an audit record and awaiting-vex which is managed by the VEX reconcile pass. The cve-needs-attention label is removed automatically when the warning condition no longer applies.

awaiting-vex (HUM-5843)

When a ticket is closed as Done-Errata or Not a Bug in production, analysis adds awaiting-vex. Each --resolve cycle then:

  1. Queries Closed tickets labeled awaiting-vex, plus Closed Done-Errata / Not a Bug tickets resolved in the last 7 days (so human/agent closes that skipped the close-time enqueue still enter the queue)
  2. Fetches Hummingbird status from the Red Hat CSAF VEX feed for every CVE ID in the ticket summary, scoped to that ticket’s package (not a CVE-wide worst-case across other Hummingbird products)
  3. Compares to the Jira resolution for the awaiting-vex work queue (Done-Errata → fixed, Not a Bug → known_not_affected, package_not_listed when CSAF omits the package, or fixed – CSAF’s known_not_affected/fixed split tracks dependency-version presence, not code-path reachability, so a Not a Bug verdict is compatible with either, e.g. a downstream package swept into an unrelated dependency’s rebuild-and-fix); match requires all CVE IDs to agree
  4. On match: records vex_status + vex_resolved (scan time) in the collector handoff and removes the label if present
  5. On pending/mismatch: adds awaiting-vex if missing, otherwise keeps it, and still emits current vex_status. Catch-up closes with no CVE ID in the summary are not labeled: MATCH is impossible, and adding the label would re-select the ticket forever.

The Closed tab computes MATCH from stored vex_status plus Jira resolution; it does not use stored vex_match_state as source of truth. Dashboard reads merged event metadata (later rows overlay earlier ones) so a ticket closed after cve_published does not keep leftover New / In Progress analysis text as Resolution (HUM-6091).

The collector also fetches any vex_updates[] keys that the watermark JQL missed, so a same-cycle catch-up match still lands on the Closed tab.

Won’t Do closures do not enqueue awaiting-vex. Open-ticket VEX mismatches remain a vex-checker reconcile concern.

awaiting-image-rebuild (HUM-9975)

A ticket closes Done-Errata the moment the RPM-level fix ships – resolution is a version comparison with no dependency on whether the Hummingbird image has rebuilt with that fix yet, and image rebuilds routinely lag RPM publish by hours. collect_cve_dashboard’s watermark JQL only re-examines a Closed ticket when it was updated within the current watermark window or it carries an awaiting-* workflow label, so without a dedicated label a ticket that closes before its image finishes rebuilding falls out of the watermark window on the very next run and image_rebuilt_on_quay is never captured.

_sync_awaiting_image_rebuild_labels (collect_cve_dashboard.py) adds awaiting-image-rebuild to a Closed Done-Errata ticket once it has an hb_rpm_fix but no hb_image_fix yet, so the JQL keeps re-selecting it every run; the label is removed once hb_image_fix resolves or the ticket is no longer Done-Errata. This mirrors the awaiting-vex / awaiting-cve-published pattern.

cve-next-release

The cve-next-release label is used for CVEs where no immediate action is possible and the fix will arrive via the next upstream release. It is set manually. Common scenarios include:

  • Vendored dependencies: A fix exists in a vendored package (e.g. ws inside dotnet) but cannot be consumed until the parent package updates its vendored copy.
  • No backport path: The fix cannot be backported to the current release and must wait for a future upstream version.
  • Upstream fix pending: A fix is expected upstream but has not landed yet, and no interim mitigation is available.

Package metadata fix_status: 0 (EOL) does not use this label; those tickets are closed as Won’t Do instead.

When this label is present on a ticket:

  • The automation skips applying upstream-fix-available and upstream-fix-in-progress labels, since they would be misleading, and removes them if already present
  • The SBOM version check continues each analysis cycle, so when a new upstream release containing the fix is consumed, the ticket is closed normally
  • Fedora labels and cve-needs-attention are still managed normally
  • The label is not removed automatically; it must be removed manually when no longer applicable

cve-next-release tickets are not closed just because Hummingbird shipped a new build. They close when analysis concludes the CVE is no longer affected; for vendored dependencies, that means the SBOM shows the fixed upstream version, not merely a new parent NVR.

cve-needs-attention conditions

The cve-needs-attention label is applied when any of these conditions are detected:

  • Tracker filed against an image: the tracker component is a Hummingbird image (hi/<name>) instead of an RPM; a Slack alert is also sent
  • Product mismatch: CVE vendor/product does not match the Hummingbird package (e.g. node-tar CVE filed against GNU tar) after an SBOM-first check finds no vendored dependency hit
  • Package missing from rpms but present in SBOM/binaries: package directory is absent from the rpms repo, yet the CVE product appears in the package SBOM and binary confirmation is present or unknown
  • Multiple products: CVE lists multiple distinct products with different versioning schemes and no cve_product override is configured (e.g. NGINX Open Source + NGINX Plus)
  • Git-only version data: CVE version data uses commit hashes instead of numeric versions (e.g. libsodium with lessThan: ad3004ec...)
  • CNA data error: A git commit hash is used where a version number is expected without setting versionType: "git", including hashes embedded in operator syntax (e.g., "version": "< 6374ae0bcdfe..."). The warning includes a link to the CVE 5.0 source control versions spec
  • Malformed version field: The version field contains syntax that could not be parsed (e.g., compound operator+hash ranges like >= hash1, < hash2)
  • CVE record not found: the CVE record file was not found in the cvelistV5 repo
  • No repo version: the Hummingbird package was not found in the Pulp repository (package name mismatch or missing SRPM)
  • Stale closure: ticket is Closed in Jira but current analysis recommends a different resolution

Human override workflow

When a human wants to take over a ticket from the bot:

  1. Assign the ticket to yourself – the bot skips all automation on tickets not assigned to the bot account. Reassign to the bot to re-enable automation.

  2. Set “Fixed in Build” – if you know the fix is in a specific SRPM, set the “Fixed in Build” field to the SRPM name (e.g. libarchive-3.8.7-1.hum1.src.rpm). The bot will use this to create the advisory MR and close the ticket, bypassing its own analysis.

Migration note: The cve-analysis-okay label is deprecated and no longer suppresses automation. Existing tickets with this label will be re-processed by the bot on its next run. To keep the bot from touching a specific ticket, reassign it to yourself before the next run. The cve-analysis-okay label can then be removed manually.

Output

Human-Readable

Project: HUM  Component: Security

  HUM-796  CVE-2026-2673 openssl: buffer overflow [hummingbird-1]
    Open since:       7 days (Mar 24 2026)
    Labels resolution: upstream-fix-available
    OpenSSL / OpenSSL:
      Affected versions: 3.5.0 < 3.5.6
      Fixed in:          3.5.6
      Hummingbird repo (latest):  3.5.5 / openssl-3.5.5-1.hum1.src.rpm
    Repo:             https://github.com/openssl/openssl
    CVE-2026-2673: upstream-fix-available (3 PRs)
      [CLOSED, 3 commits] Fix group tuple handling in DEFAULT expansion (3.5)
              https://github.com/openssl/openssl/pull/30110
    Fedora update: FEDORA-2026-abc123 openssl-3.5.6-1.fc44 (stable, security)
              https://bodhi.fedoraproject.org/updates/FEDORA-2026-abc123
    Jira current:     In Progress / (none)
    Jira resolution:  In Progress / affected (repo 3.5.5 is in affected range 3.5.0 < 3.5.6)

JSON

Each issue includes: key, summary, labels, open_since, cve_ids, cves (with version and resolution data), jira_current (status/resolution), computed_resolution, and upstream (with PR/MR search results, Bodhi updates, and Fedora version data).

Viewing Logs

The CVE analysis tool runs as a pod in the hummingbird--internal namespace on mpp-prod. See the OpenShift MP+ internal docs page for general log access instructions.

For CVE analysis specifically:

  • Live: Open the mpp-prod pods list in the OpenShift console, filter for cve-analysis, and open the Logs tab.
  • Grafana/Loki: Open the Hummingbird Grafana logs dashboard, select cluster mpp-prod and namespace hummingbird--internal, then filter for hummingbird-cve-analysis pods.

Library layout

Reusable library code lives under hummingbird_cve_analysis/lib/. The CLI entry point remains cve_analysis.py at the package root.

Module Responsibility
lib/github_client.py GitHub PR and release API reads
lib/gitlab_client.py gitlab.com merge request, release, and feature-flag reads
lib/fedora.py Fedora Bodhi, DistGit spec parsing, and Bugzilla helpers
lib/upstream.py Upstream forge search, cgit commits, and fix-status analysis
lib/net.py Shared network error tuple used by library HTTP callers
lib/nvd.py NVD JSON 2.0 feed cache, download, and reference parsing
lib/cvelist.py Local cvelistV5 repository access and CVE 5.0 affected parsing
lib/jira_client.py Jira REST read/write primitives and ADF comment helpers
lib/versions.py Pure version comparison and range helpers
lib/analysis.py CVE decision pipeline, product matching, and lifecycle analysis
lib/formatting.py Human and Jira comment/output formatting
lib/resolve.py Jira/advisory mutation helpers (labels, close, attach SBOM)
lib/ticket.py Shared one-ticket analyze/resolve API
lib/advisory_handler.py CEE advisories repo clone/edit/MR helpers
lib/osidb_client.py OSIDB subpackage PURL lookups and flaw/affect created timestamps
lib/pulp.py Hummingbird Pulp repo RPM/SRPM listings and repodata lookups
lib/catalog.py Container catalog API for image publish times
lib/rpms_repo.py Local rpms git repo, package map loading, fixed-build detection
lib/rpm_scan_cache.py S3 LRU cache of RPM provides and Syft artifact JSON
lib/sbom.py SBOM fetch, vendored dependency lookup, binary RPM confirmation
lib/slack.py Slack webhook helper
lib/version_transforms.py Named version-transform helpers

cve_analysis.py is the CLI entry point only (argument parsing, JQL building, signal handling, and the main orchestration loop). Library code lives under lib/; import and patch those modules directly. New cron jobs, webhooks, and other integrations should prefer ticket.process_ticket for analyzing (and optionally resolving) a single Jira issue:

from hummingbird_cve_analysis.lib import ticket

result = ticket.process_ticket(
    issue,
    pkg_map,
    catalog_source_map,
    github_token,
    gitlab_token,
    base_url=base_url,
    token=token,
    resolve=False,
)

Development

See the main README for development workflows.

make hummingbird-cve-analysis/setup  # Install dependencies
make check                            # Lint code (ruff)
make test                             # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.14 - Dashboard MR Linker

An AWS Lambda function that posts internal notes on newly opened GitLab merge requests linking to the Hummingbird dashboard status page. This provides an easy way for developers to check Konflux build status directly from their MRs.

Features

  • Internal Notes: Posts an internal note with dashboard link when MRs are opened (visible to project members only)
  • Configurable Projects: Only monitors specified projects via space-delimited list
  • SNS Integration: Subscribes to GitLab events via SNS with filter policy

Architecture

The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for merge_request events:

GitLab Webhook → gitlab-event-forwarder → SNS Topic → dashboard-mr-linker → GitLab API

When a new MR is opened on a configured repository, the Lambda posts an internal note:

:robot: **Hummingbird Status**

View Konflux build status for this MR: [org/group/repo!123](https://dashboard.example.com/mr/org/group/repo/123)

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)
  • GitLab API token with api scope (Reporter role or higher on monitored projects)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd dashboard-mr-linker
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabToken GitLab API token (api scope, Reporter+) (required)
GitLabUrl GitLab instance URL https://gitlab.com
DashboardBaseUrl Base URL of the dashboard (required)
GitLabProjects Space-delimited list of GitLab project paths (required)
SentryDsn Optional Sentry DSN ``

Resource naming: Lambda follows {ResourcePrefix}-lambda-linker pattern.

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Usage

Configure the GITLAB_PROJECTS parameter with space-delimited project paths:

org/group/containers org/group/rpms org/other/project

Only MRs opened on these projects will receive dashboard link notes.

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

Configuration

Lambda function receives configuration via environment variables (automatically set by CloudFormation):

Variable Description
GITLAB_TOKEN GitLab API token (api scope, Reporter+)
GITLAB_URL GitLab instance URL
DASHBOARD_BASE_URL Base URL of the dashboard
GITLAB_PROJECTS Space-delimited list of GitLab project paths
SENTRY_DSN Optional Sentry DSN
AWS_REGION AWS region (auto-set)

Security & Limitations

Security:

  • GitLab API token should have minimal required scopes (api)
  • Token stored as CloudFormation parameter with NoEcho: true
  • SNS subscription uses filter policy to only receive relevant events
  • Notes are posted as internal (visible to project members only)
  • CloudWatch logs capture all note posts (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • Only processes action: open events (not updates or other actions)
  • Skips MRs with the managed-by::workqueue label (created by workqueue-service). The workqueue-service manages its own dashboard integration for these MRs, so the Lambda backs off to avoid duplicate notes

Integration

The workqueue-service creates the dashboard note itself for MRs it manages. The managed-by::workqueue scoped label, applied by the create_mr executor, causes this Lambda to skip those MRs. The workqueue-service’s ensure_dashboard_note executor then posts the note and stores the dashboard_link dimension with the note ID.

For non-managed MRs (those not created by the workqueue-service), the Lambda remains the sole creator of the dashboard note.

Both systems use the same marker in the note body: :robot: **Hummingbird Status**. The Lambda’s COMMENT_MARKER and the workqueue-service’s DASHBOARD_LINK_MARKER constant must stay in sync — changing one without the other will break identification and may cause issues with future reconciliation or status rendering.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.15 - GitLab Event Forwarder

An AWS Lambda function that receives GitLab webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates webhook signatures and extracts minimal metadata (source, event type, project/group paths) as SNS message attributes, enabling downstream subscribers to filter events precisely.

The original GitLab JSON payload is forwarded unchanged as the SNS message body.

Features

  • Webhook Signature Validation: Verifies authenticity using HMAC-SHA256 signing tokens (Standard Webhooks), with legacy X-Gitlab-Token fallback
  • Structured Metadata: Extracts event type, project/group paths as SNS message attributes
  • SNS Subscription Filtering: Enables precise event routing to subscribers
  • Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Single Lambda function handles the webhook processing:

  1. Handler - API Gateway endpoint (/webhook) that validates GitLab webhook signatures, extracts metadata, and publishes to SNS

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd gitlab-event-forwarder
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs: ApiEndpoint - API Gateway endpoint URL (<api-endpoint>)

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabWebhookTokens GitLab webhook tokens (JSON) (required)
GitLabSigningTokens Signing tokens (JSON array) []
SentryDsn Optional Sentry DSN ``
CustomDomainName Custom domain name ``
HostedZoneId Route53 hosted zone ``

Resource naming: Lambda and API resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-lambda-handler, myapp-prod-api).

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Usage

Configure GitLab projects or groups to send webhooks to <api-endpoint>:

webhooks:
  <api-endpoint>:
    token: <secret-token>
    signing_token: <whsec-signing-token>
    push_events: true
    merge_requests_events: true
    pipeline_events: true
    job_events: true

signing_token (recommended) uses HMAC-SHA256 verification with replay protection (±5 min tolerance) per the Standard Webhooks spec. token uses a static X-Gitlab-Token header. When signing tokens are configured on the forwarder, they take priority and webhook-signature is required on all requests.

SNS Subscription Filter Examples:

Push events from a specific project:

{
  "source": ["gitlab"],
  "event_type": ["push"],
  "project_path": ["redhat/hummingbird/containers"]
}

All merge request events:

{
  "source": ["gitlab"],
  "event_type": ["merge_request"]
}

Member events from a group:

{
  "source": ["gitlab"],
  "event_type": ["member"],
  "group_path": ["redhat/hummingbird"]
}

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

Configuration

Lambda function receives configuration via environment variables (automatically set by CloudFormation):

Variable Description
SNS_TOPIC_ARN SNS topic ARN
GITLAB_WEBHOOK_TOKENS GitLab webhook tokens (JSON array)
GITLAB_SIGNING_TOKENS Signing tokens (JSON array)
SENTRY_DSN Optional Sentry DSN
AWS_REGION AWS region (auto-set)

Event Metadata

The Lambda function extracts minimal metadata from GitLab webhook events and adds them as SNS message attributes:

Attribute Description Example
source Always "gitlab" gitlab
event_type From object_kind push, merge_request, pipeline, build
project_path Full project path redhat/hummingbird/containers
group_path Full group path redhat/hummingbird

Security & Limitations

Security:

  • HMAC-SHA256 signing token verification via webhook-signature header (Standard Webhooks spec) with ±5 min replay protection
  • When signing tokens are configured, static X-Gitlab-Token is not accepted
  • Legacy X-Gitlab-Token validation when no signing tokens are configured
  • Supports multiple active tokens for zero-downtime rotation
  • Invalid/missing tokens return HTTP 401
  • SNS topic follows least privilege principle
  • CloudWatch logs capture all webhook deliveries (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • Webhook payload size: Up to 6 MB (API Gateway limit)
  • SNS limits the compressed and base64-encoded message plus attributes to 256 KiB

Pipeline Hooks use object_kind: pipeline; Job Hooks use object_kind: build. The forwarder compresses the original JSON for SNS transport. Consumers that decode gzip+base64 receive the unchanged webhook payload.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.16 - Hummingbird MR Human Tracker

A CLI tool that detects human intervention on automation-opened GitLab merge requests (Renovate, Dependabot, etc.) and links them to a per-sprint Jira epic for tracking.

Features

  • Bot MR Detection: Identifies MRs authored by bot accounts across multiple GitLab projects using configurable username patterns
  • Human Activity Detection: Finds MRs where humans have commented or pushed commits (via GitLab system notes)
  • Per-Sprint Jira Epic: Automatically finds or creates an epic per sprint to track automation fix work (e.g. “Automation MR fixes for sprint Hum S16”)
  • Remote Link Tracking: Adds affected MR URLs as remote links on the Jira epic, with deduplication to avoid duplicates on repeated runs
  • Merged/Closed MR Scanning: --since flag catches MRs that were merged or closed before the next scheduled run
  • Dry Run: Preview what would be linked without making any changes

Prerequisites

  • Python 3.11 or later
  • GitLab API token with read access to target projects (including MR notes)
  • Jira API token with permission to create epics and remote links in the HUM project

Installation

pip install -e hummingbird-mr-human-tracker

Usage

mr-human-tracker --jira-user user@redhat.com --dry-run -v

Common invocations

# Preview what would be linked (no changes made)
mr-human-tracker --jira-user user@redhat.com --dry-run -v

# Run for real, including MRs merged in the last 2 days
mr-human-tracker --jira-user user@redhat.com --since 2d

# Scan only specific projects
mr-human-tracker --jira-user user@redhat.com --projects redhat/hummingbird/tools

# Verbose output for debugging
mr-human-tracker --jira-user user@redhat.com --since 1w -v

Output

On completion, the tool prints a summary of linked MRs:

INFO hummingbird_mr_human_tracker.tracker: Active sprint: Hum S16 [June 11 - June 25] (id=68851)
INFO hummingbird_mr_human_tracker.tracker: Using epic HUM-2602
INFO hummingbird_mr_human_tracker.tracker: Linked 2 MR(s) to HUM-2602:
INFO hummingbird_mr_human_tracker.tracker:   https://gitlab.com/redhat/hummingbird/rpms/-/merge_requests/2395
INFO hummingbird_mr_human_tracker.tracker:   https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/8476

Configuration

All configuration is via CLI arguments and environment variables.

CLI arguments

Argument Description Default
--gitlab-url GitLab instance URL GITLAB_URL env
--gitlab-token GitLab private token GITLAB_TOKEN env
--jira-url Jira base URL JIRA_URL env
--jira-token Jira API token JIRA_TOKEN env
--jira-user Jira email for Basic auth (omit for Bearer) None
--projects GitLab project paths to scan tools, rpms, containers
--board-id Jira board ID for sprint lookup 1489
--sprint-prefix Sprint name prefix to match Hum S
--since Also scan merged/closed MRs updated within window None (open MRs only)
--dry-run Report without making changes False
-v, --verbose Enable debug logging False

Environment variables

Variable Description
GITLAB_URL GitLab instance URL
GITLAB_TOKEN GitLab private token
JIRA_URL Jira base URL
JIRA_TOKEN Jira API token

Duration format for --since

The --since flag accepts a number followed by a unit:

Unit Meaning
h Hours
d Days
w Weeks

Examples: 12h, 2d, 1w

How it works

  1. Fetch active sprint from the Jira Agile API (board 1489, prefix “Hum S”)
  2. Find or create a Jira epic named “Automation MR fixes for sprint {sprint_name}” in the HUM project, assigned to the active sprint
  3. Scan GitLab projects for open MRs authored by bot accounts; if --since is set, also scan recently merged/closed MRs
  4. Detect human activity on each bot MR by checking:
    • Non-bot, non-system comments
    • System notes indicating a human pushed commits (“added N commit”)
    • Reopen events by non-bot users
  5. Add remote links on the Jira epic for each human-touched MR (skipping URLs already linked)

Bot detection

A username is considered a bot if it matches any of:

  • Contains bot_ or bot- (e.g. renovate_bot)
  • Ends with _bot (e.g. some_bot)
  • Ends with [bot] (e.g. renovate[bot], dependabot[bot])
  • Starts with project_<digits>_bot or group_<digits>_bot (GitLab service accounts)

Development

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.17 - MR Auto-Approver

An AWS Lambda function that auto-approves GitLab merge requests based on configurable per-project rules. Replaces CI/CD-based approval jobs and moves approval tokens out of GitLab CI/CD variables.

Features

  • Rules-Based Config: YAML config with per-project ordered rules matching on user IDs, username patterns, branch patterns, and Konflux pipeline status
  • First-Match-Wins: Rules evaluated in order; first match determines action (approve or deny)
  • Konflux Integration: Optional check that Konflux pipelines have posted commit statuses before approving
  • Fork Rejection: Unconditionally rejects MRs from forked projects
  • User Verification: Matches against the authenticated webhook event user, not forgeable Git commit metadata
  • Stateless & Idempotent: No queues or stored state; re-approving is a no-op

Architecture

The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for merge_request and pipeline events:

GitLab Webhook → gitlab-event-forwarder → SNS Topic → mr-auto-approver → GitLab API

Merge request events (open, update): evaluate rules using the authenticated pusher from the webhook payload. If matched and no Konflux check needed, approve immediately. If Konflux check needed, check statuses now and approve only if present and not failed.

Pipeline events (success, failed): look up the MR from the pipeline event via the GitLab API, evaluate rules, and approve if Konflux statuses are ready. This handles the case where Konflux posts commit statuses after the initial MR event.

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)
  • GitLab API token(s) with api scope and permission to approve MRs on target projects
  • Target project webhooks must send merge_requests_events and pipeline_events to the gitlab-event-forwarder endpoint

Deployment

Build and deploy using containerized AWS SAM CLI:

cd mr-auto-approver
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
GitLabUrl GitLab instance URL https://gitlab.com
ConfigPath Path to YAML config (bundled with Lambda) config.yaml
ApprovalTokens JSON map of env var names to tokens {}
SentryDsn Optional Sentry DSN ``

Resource naming: Lambda follows {ResourcePrefix}-lambda-approver pattern.

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Configuration

The Lambda uses a YAML config file bundled at deploy time. The config defines per-project rules:

settings:
  gitlab_url: https://gitlab.com

projects:
  org/group/repo:
    token_env: APPROVAL_GITLAB_TOKEN_REPO
    rules:
      - branch_regexes: ["renovate/skip/.*"]
        action: deny
      - user_ids: [12345678]
        branch_regexes: ["renovate/.*"]
      - user_ids: [87654321]
        check_konflux: true

Rule fields

Field Type Default Description
user_ids list[int] [] GitLab user IDs (immutable, preferred)
user_regexes list[str] [] Fullmatch regexes for username
branch_regexes list[str] [] Fullmatch regexes for source branch
check_konflux bool false Require Konflux statuses before approval
action str approve approve or deny

Match logic: AND across field types, OR within a field. Empty fields match anything. First matching rule wins. user_ids and user_regexes are both user identity constraints – if either is set, the user must match at least one entry from across both lists. Prefer user_ids over user_regexes to prevent username confusion attacks.

Adding a new project

  1. Create a project-scoped GitLab token (Developer, api scope) in Vault
  2. Add the token to vars.sh and the jq command that builds APPROVAL_TOKENS
  3. Add a project entry to config.yaml in the infrastructure repo with token_env and rules
  4. Deploy the Lambda
  5. Ensure the project’s GitLab webhook sends events to the gitlab-event-forwarder endpoint

Environment variables

Variable Description
CONFIG_PATH Path to YAML config file
GITLAB_URL GitLab instance URL
APPROVAL_TOKENS JSON object mapping env var names to GitLab tokens
SENTRY_DSN Optional Sentry DSN

Security

  • Fork MRs are unconditionally rejected before any rule evaluation (source_project_id != target_project_id)
  • User matching supports immutable user.id (preferred) in addition to user.username from the webhook payload, preventing username confusion attacks. Never uses Git commit author/committer metadata
  • For pipeline events, the Lambda looks up head_pipeline.user (both id and username) via the GitLab API to verify the last pusher
  • Approval tokens stored as CloudFormation parameters with NoEcho: true
  • SNS subscription filter policy limits events to merge_request and pipeline
  • CloudWatch logs capture all approval decisions (7-day retention)

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.18 - Jira Event Forwarder

An AWS Lambda function that receives Jira Automation webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates Bearer token authentication and restricts access to Atlassian egress IP ranges via an API Gateway resource policy.

The original Jira JSON payload is forwarded unchanged as the SNS message body.

Features

  • Bearer Token Authentication: Validates Authorization: Bearer <token> headers with constant-time comparison, supporting multiple tokens for zero-downtime rotation
  • Atlassian IP Restriction: API Gateway resource policy restricts access to Atlassian egress IP ranges (IPv4 and IPv6), fetched dynamically at deploy time
  • Structured Metadata: Extracts event type, project key, and issue type as SNS message attributes
  • SNS Subscription Filtering: Enables precise event routing to subscribers
  • Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Single Lambda function handles the webhook processing:

  1. Handler - API Gateway endpoint (/webhook) that validates the Bearer token, extracts metadata from the Jira payload and request headers, and publishes to SNS

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd jira-event-forwarder
make build     # Build Lambda package
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs: ApiEndpoint - API Gateway endpoint URL (<api-endpoint>)

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)
JiraWebhookTokens Jira webhook tokens (JSON array) []
SentryDsn Optional Sentry DSN ``
AtlassianIpv4Cidrs Atlassian egress IPv4 CIDRs (comma-list) ``
AtlassianIpv6Cidrs Atlassian egress IPv6 CIDRs (comma-list) ``
CustomDomainName Custom domain name ``
HostedZoneId Route53 hosted zone ``

Resource naming: Lambda and API resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-lambda-handler, myapp-prod-api).

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first to create the topic, then use its ARN for the SnsTopicArn parameter.

Usage

Jira Automation allows only one trigger per rule, and the “Multiple issue events” trigger does not expose which event fired. Since each event needs a distinct X-Jira-Event-Type header, configure 6 separate automation rules — one per event type:

Trigger X-Jira-Event-Type value
Created issue_created
Updated issue_updated
Deleted issue_deleted
Assigned issue_assigned
Transitioned issue_transitioned
Moved issue_moved

Each rule’s “Send web request” action:

  • URL: <api-endpoint>
  • Method: POST
  • Headers:
    • Authorization: Bearer <token> — enter the token value and click “Hidden” to mask it in the Jira Automation UI
    • X-Jira-Event-Type: <value from table above>
  • Body: “Issue data (Jira format)”

SNS Subscription Filter Examples:

All Jira issue events:

{
  "source": ["jira"]
}

Specific Jira event types:

{
  "source": ["jira"],
  "event_type": ["issue_created", "issue_updated"]
}

Events from a specific project:

{
  "source": ["jira"],
  "project_key": ["HUM"]
}

Development

See the main README for development workflows.

make setup     # Install dependencies
make check     # Lint code (ruff)
make fmt       # Format code
make test      # Run unit tests
make coverage  # Run tests with coverage

Configuration

Lambda function receives configuration via environment variables (automatically set by CloudFormation):

Variable Description
SNS_TOPIC_ARN SNS topic ARN
JIRA_WEBHOOK_TOKENS Jira webhook tokens (JSON array)
SENTRY_DSN Optional Sentry DSN
AWS_REGION AWS region (auto-set)

Event Metadata

The Lambda function extracts metadata from Jira webhook events and request headers, adding them as SNS message attributes:

Attribute Description Example
source Always "jira" jira
event_type From X-Jira-Event-Type header issue_created
project_key Jira project key HUM
issue_type Issue type name Story, Bug
content_encoding Compression format gzip+base64

The event_type attribute is set from the X-Jira-Event-Type request header. Each Jira Automation rule hardcodes the appropriate value for its trigger type in this header (see Usage). If the header is absent, the event_type attribute is omitted from the SNS message.

Security & Limitations

Security:

  • Bearer token validation via Authorization header with constant-time comparison (hmac.compare_digest)
  • API Gateway resource policy restricting source IPs to Atlassian egress ranges (IPv4 and IPv6 CIDRs both via aws:SourceIp), fetched dynamically at deploy time from ip-ranges.atlassian.com
  • Supports multiple active tokens for zero-downtime rotation
  • Missing or invalid tokens return HTTP 401; no configured tokens returns HTTP 500
  • SNS topic follows least privilege principle
  • CloudWatch logs capture all webhook deliveries (7-day retention)
  • Sentry integration for error tracking

Limitations:

  • Lambda timeout: 30 seconds
  • Lambda memory: 256 MB
  • Webhook payload size: Up to 6 MB (API Gateway limit)
  • SNS limits the compressed and base64-encoded message plus attributes to 256 KiB

The forwarder compresses the original JSON for SNS transport. Consumers that decode gzip+base64 receive the unchanged webhook payload.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.19 - Container Catalog

Per-distro serverless catalog API for Hummingbird container images.

Features

  • Image Directory - Browse all container images with metadata
  • Tag Browser - View tags, digests, architectures per image
  • Specifications - Per-architecture OCI config details (env, cmd, user, labels)
  • SBOM - Per-architecture package lists from syft scans with a Hermeto build-only merge
  • Vulnerabilities - CVE scanning results via Grype
  • CVE Metrics - Catalog-wide vulnerability aggregate
  • Provenance - Source traceability from SLSA attestations
  • Release History - Timeline of past builds with drill-down
  • Deprecation status - Stream and release status from OCI labels
  • OpenAPI Spec - Machine-readable API documentation at /v1/openapi.json
  • Swagger UI - Interactive API explorer at /v1/docs/
  • External Images - CVE comparison with external image sources
  • External Comparison UI - Side-by-side CVE and SBOM comparison between Hummingbird and external counterparts

Architecture

Rust serverless stack deployed as two isolated per-distro stacks:

  • API Lambda - DynamoDB pass-through (~10ms response)
  • Sync Lambda - Incremental DynamoDB sync from SNS Release events (~22 registry calls per release), then enqueues one index event per release
  • Index Lambda (index-lambda) - Queue-driven Syft runs per architecture, writes the SBOM items to DynamoDB, uploads native Syft JSON to S3 for scanner consumption (same function shape as the external index — the future unified function)
  • Scan Lambda (scan-lambda) - SQS-triggered CVE scanning via Grype from native Syft JSON in S3, per-digest processing with first_seen tracking and partial batch failure reporting
  • Enqueue Lambda (enqueue-lambda) - Hourly EventBridge-triggered fan-out that checks Grype DB for updates and enqueues non-superseded digests to SQS
  • Metrics Lambda (metrics-lambda) - DynamoDB Stream-triggered vulnerability aggregate
  • External Release Lambda (external-release-lambda) - Scheduled trigger feeding the external sync queue
  • External Sync Lambda (external-sync-lambda) - Registry tag resolution and DynamoDB metadata writes for external images
  • External Index Lambda (external-index-lambda) - Syft SBOM generation for external images, uploads to S3 for Grype scanning
  • DynamoDB - Pre-computed JSON items (single-table PK/SK design, Streams: KEYS_ONLY)
  • S3 ScanDataBucket - Native Syft JSON storage for scanner data (gzipped, keyed by grype/{image}/{digest_hex}.json.gz for Hummingbird and grype-external/{registry}/{digest_hex}.json.gz for external images; SPDX SBOMs are stored at sbom/{image}/{digest_hex}.json.gz for Hummingbird and sbom-external/{registry}/{digest_hex}.json.gz for external images — the sbom* prefixes match no scan notification filter)
  • SQS IndexQueue - Decouples the index (syft) path from sync: the Sync Lambda enqueues one tag-release event per release after its DynamoDB writes; failed index records are retried individually via SQS batch-item-failure semantics, redriving to the IndexDLQ after 3 attempts
  • SQS ScanQueue - Central queue for scan tasks (fed by both grype/ and grype-external/ S3 event notifications and Enqueue Lambda)
  • CloudFront - CDN with per-endpoint cache TTLs
  • catalog sync - Full DynamoDB population from GitLab + Quay.io OCI v2 registry
  • catalog scan - CLI CVE scanning via Grype; reads native Syft JSON from S3 (--bucket required); historic releases read the same S3 objects, and digests whose object is missing are skipped with a warning
  • catalog index - CLI Syft JSON generation for backfilling S3
  • Catalog SPA - Lit 3 web app served from S3 via CloudFront

Scanner Architecture

CVE vulnerability data must match direct grype <image> scans exactly for all images. Synthetic Syft JSON (built from stored SBOM packages) cannot reliably reproduce the native output because Grype relies on metadata fields, artifact relationships, and deduplication logic that are lost in the SPDX-to-API roundtrip. Additionally, raw SPDX from the build system contains package bloat (e.g. Go sub-modules, empty-version entries) that native Syft filters out when scanning a compiled binary directly.

To guarantee exact-match results, the scanner chain stores native Syft JSON in S3 rather than DynamoDB: a single image’s Syft JSON is typically 1-10 MB (gzipped to 100 KB - 1 MB), which exceeds DynamoDB’s 400 KB item limit. S3 has no per-object size constraint, avoids provisioned throughput costs for large blobs, and supports event notifications for triggering downstream scanners. The Index Lambda runs syft <image> on each new release and uploads the full output to s3://{bucket}/grype/{image}/{hex}.json.gz. The S3 upload fires an event notification to SQS, which triggers the Scan Lambda to read the native JSON, run Grype, and write per-canonical vulnerability data to DynamoDB.

Each syft run extracts image layers into its own temp directory, passed to the syft child via TMPDIR and removed when the run finishes; concurrent runs never share temp data. Dirs left behind by killed runs are swept before each new run (age-gated to never touch a possibly-active run).

When running inside a Lambda, the syft child process inherits GOMEMLIMIT derived as 90% of AWS_LAMBDA_FUNCTION_MEMORY_SIZE (injected by Lambda, in MB; e.g. 3072 MB → 2764 MiB). This makes the cgroup limit and the GC soft limit share a single source of truth — a memory bump auto-rescales both with no code change. Outside Lambda (CLI catalog index backfill, local runs) the variable is absent and Go’s defaults apply. Without GOMEMLIMIT, Go’s default GOGC=100 lets syft’s heap grow to 2× its live size (~2.8 GB for large multi-layer images → ~5.7 GB peak), exceeding the Lambda cgroup and causing OOM-kills. If a future image’s live heap grows beyond the limit, the scan degrades to slower GC thrash rather than an instant kill.

The scanner chain is decoupled from the catalog data chain by a queue: the SNS Release event triggers the Sync Lambda (catalog data to DynamoDB), which then enqueues one index event per release; the Index Lambda consumes those events (Syft JSON to S3). The event carries everything the index needs, so the slow syft path never blocks the sync invocation and a failed record is retried individually by the queue.

Grype runs with --by-cve: some ecosystems (e.g. Go modules) are otherwise reported using their native GHSA or Go vulnerability DB ID rather than a CVE. --by-cve reports the CVE ID instead wherever grype’s own local vulnerability database has that cross-reference, with no added network call. An ID with no known CVE anywhere in that database is returned as-is.

Vulnerability components include the source RPM filename from the syft scan when available. Consumers can group affected binary RPMs by this value.

CVE Data Flow

graph TD
    subgraph Hummingbird
        Attestation[(Registry .sbom attestations)] -->|"fetch + merge"| Index
        Konflux([Konflux Release])
        Konflux -->|SNS| Sync[SyncFunction]
        Sync -->|"index events via SQS"| Index[IndexFunction]
        Sync -->|update| Tags[(Tags)]
        Index -->|upload| S3Syft[(S3 grype/)]
        Index -->|write| SbomItems[(SBOM items)]
    end

    subgraph External Images
        ExtSchedule([Schedule 6h]) -->|invoke| ExtRelease[ExternalReleaseFunction]
        ExtRelease -->|"per registry via SQS"| ExtSync[ExternalSyncFunction]
        ExtSync -->|"per-tag release events via SQS"| ExtIndex[ExternalIndexFunction]
        ExtIndex -->|upload| S3Ext[(S3 grype-external/)]
        ExtIndex -->|write| ExtSbomItems[(External SBOM items)]
        ExtSync -->|update| ExtTags[(External Tags)]
    end

    Tags -->|read| Scan[ScanFunction]
    ExtTags -->|read| Scan
    S3Syft -->|"S3 event via SQS"| Scan
    S3Ext -->|"S3 event via SQS"| Scan
    Hourly([Schedule 1h]) -->|invoke| Enqueue[EnqueueFunction]
    Enqueue --> Gate[check Grype DB]
    Gate -->|"fan out via SQS"| Scan
    Scan -->|update| Vulns[(Vulnerabilities)]
    Vulns -->|"stream via ESM"| Metrics[MetricsFunction]
    Tags -->|"stream via ESM"| Metrics
    Metrics -->|"write aggregate"| CatalogVulns[(Catalog Vulnerabilities)]

    classDef lambda fill:#d4e6f1,stroke:#2980b9
    classDef dynamo fill:#fdebd0,stroke:#e67e22
    classDef trigger fill:#d5f5e3,stroke:#27ae60
    classDef gate fill:#e5e7e9,stroke:#7f8c8d
    class Sync,Scan,Enqueue,Metrics,Index,ExtRelease,ExtSync,ExtIndex lambda
    class Vulns,Tags,ExtTags,CatalogVulns,S3Syft,S3Ext,S3Spdx,S3ExtSpdx,SbomItems,ExtSbomItems,Attestation dynamo
    class Konflux,Hourly,ExtSchedule trigger
    class Gate gate

The sbom/ and sbom-external/ records are the recovery inputs for the already-indexed path: each index function persists its primary-arch SPDX before the grype* blob, which is deliberately the last write — it is the dedup key and the scan trigger, so it never arms before its recovery input exists. Both planes share the same ordering guarantee: the sync function’s DynamoDB writes complete before its index events are enqueued — the queue is the ordering barrier. The attestation edge is internal-only: the Hummingbird index fetches the per-arch Hermeto attestations from the registry and merges build-only entries into each arch’s SBOM; external images have no attestation input.

Prerequisites

  • Rust 1.75+ (for building backend)
  • Node.js 22+ (for building frontend)
  • AWS credentials (for DynamoDB access and deployment)
  • SAM CLI (for deployment)

API Endpoints

Both namespaces share the same response types and URL structure. The Hummingbird prefix is /v1/images/{name}, the external prefix is /v1/external-images/{name} where {name} is a percent-encoded registry reference (e.g. docker.io%2Flibrary%2Fnginx). DynamoDB partition keys are IMAGE#{name} and EXTERNAL#{registry} respectively.

Suffix Response type HB Ext
(directory) DirectoryResponse ✅ ✅
(config) ExternalImagesConfig ✅
/{…} ImageResponse ✅ ✅
/{…}/tags TagsResponse ✅ ✅
/{…}/details/{canonical} DetailsResponse ✅ ✅
/{…}/sbom/{canonical} SbomResponse ✅ ✅
/{…}/vulnerabilities/{canonical} VulnerabilitiesResponse ✅ ✅
/{…}/history/{stream}/{variant} HistoryResponse ✅ ✅
/{…}/releases/details/{digest} DetailsResponse ✅ ✅
/{…}/releases/sbom/{digest} SbomResponse ✅ ✅
/{…}/releases/vulnerabilities/{digest} VulnerabilitiesResponse ✅ ✅
/v1/vulnerabilities CatalogVulnerabilitiesResponse ✅

GET /v1/openapi.json and GET /v1/docs/ serve the OpenAPI 3.1 spec and Swagger UI for all endpoints.

Response types

DirectoryResponse

Catalog-level image listing with aggregate statistics. Used for both Hummingbird (CATALOG/DIRECTORY) and external (CATALOG/EXTERNAL_DIRECTORY) image directories.

Field Type Description
images ImageSummary[] All images in the catalog
image_count u32 Total number of images
variant_count u32 Total unique variant names across all images
stream_variant_count u32 Total published (stream, variant) pairs
stream_count u32 Total version streams across all images
tag_count u32 Total tags across all images
release_count u32 Total releases across all images

ImageSummary

Per-image summary used in both directory listings and as the base of ImageResponse.

Field Type Hummingbird External
name String Image name (e.g. python) Registry reference (e.g. docker.io/library/nginx)
summary String From summary OCI label Omitted — no OCI labels available from upstream registries
application_category String From io.hummingbird-project.application-category label Omitted — Hummingbird-specific label
description String From org.opencontainers.image.description label or README Omitted — no README available from upstream registries
latest_tag String Most recent canonical tag Same
pull_url String quay.io/…/{name}:latest {registry}:{latest_tag}
oldest_created String? Earliest OCI created timestamp Same
architectures String[] From OCI manifest list Same
variants VariantDescription[] From containers repo From config
stream_variant_count u32 Computed from tags Same
streams String[] From containers repo directory names From config
deprecated_streams String[] Streams with deprecated=true label Always [] — external images have no deprecation mechanism
tag_count u32 Number of tags Same
release_count u32 Sum of history entries Same

ImageResponse

Full image detail — flattens ImageSummary into the top level with additional fields.

Field Type Hummingbird External
(all ImageSummary fields) (see above) (see above)
readme_html String README rendered as HTML Omitted — no README available from upstream
licenses String[] SPDX identifiers from the attestation SBOM’s runtime packages Omitted — no license labels on upstream images

TagsResponse

Field Type Description
tags Tag[] Tags ordered by oldest_created desc
variants VariantDescription[] Available variant descriptions

Tag

Field Type Hummingbird External
name String Tag name (e.g. 3.14.2) Same
canonical String Most specific tag for this release Same (by semver specificity)
variant String From containers repo From config
pull_url String quay.io/…:{tag} {registry}:{tag}
digest String Manifest list digest Same
sizes {arch: bytes} Compressed layer sizes from OCI manifest Same
architectures String[] From OCI manifest list Same
oldest_created String? Earliest OCI created timestamp Same
stream String From containers repo directory name From config
deprecated bool From deprecated=true OCI label Always false — no deprecation mechanism for external images
superseded bool Computed by mark_superseded Same

DetailsResponse / ArchDetails

Per-architecture OCI container configuration keyed by architecture name (e.g. {"amd64": {...}, "arm64": {...}}).

Field Type Hummingbird External
user String From OCI config Same
env {key: value} From OCI config Same
entrypoint String[] From OCI config Same
cmd String[] From OCI config Same
exposed_ports String[] From OCI config Same
volumes String[] From OCI config Same
working_dir String From OCI config Same
stop_signal String From OCI config Same
labels {key: value} From OCI config Same
created String? From OCI config root Same
source_repo String From SLSA attestation Omitted — no signed attestations on upstream images
source_commit String From SLSA attestation Omitted — no signed attestations on upstream images
containerfile String From SLSA attestation Omitted — no signed attestations on upstream images
slsa_level String From SLSA attestation Omitted — no signed attestations on upstream images
build_system String From SLSA attestation Omitted — no signed attestations on upstream images

SbomResponse / ArchSbom

Per-architecture package list keyed by architecture name.

Field Type Hummingbird Rawhide External
runtime_count u32 From the syft scan From the syft scan (Hermeto merge disabled — see below) Same as Hummingbird
build_count u32 From the Hermeto merge Always 0 Always 0 — Hermeto is a Konflux-specific build tool
source_count u32 Always 0 — arch=src entries are skipped Always 0 Always 0 — Hermeto is a Konflux-specific build tool
packages SbomPackage[] All known architectures — one ArchSbom per arch from the manifest list Same All known architectures — one ArchSbom per arch from the manifest list

SbomPackage

Field Type Notes
name String Package name (e.g. python3-libs, github.com/caddyserver/caddy/v2).
evr String Version. RPM: version-release (e.g. 1.33-5.1.hum1); other types: upstream version (e.g. v2.9.1).
type String Package ecosystem, carried verbatim from the Syft artifact type: rpm, go-module, python, npm, gem, dotnet, java-archive. Items written before the syft-native switch may carry purl-derived names (golang, pypi, nuget, maven).
license String SPDX license expression, NOASSERTION when unknown.
sourcerpm String Full source RPM filename (e.g. python3-3.14.2-1.fc42.src.rpm). Only set for RPM runtime packages.
category String runtime (Syft) or build (Hermeto binary with no syft twin). source appears only on legacy items — arch=src entries are skipped.
purl String Full purl string (e.g. pkg:golang/golang.org/x/crypto@0.35.0).
cpes String[] CPE 2.3 identifiers.

Schema revision 1.1.0: the arch map of SbomResponse serializes its keys sorted (byte-stable writes of identical data; semantically identical JSON for consumers). As SBOM content regeneration from the native syft scans rolls out, the type vocabulary switches to the syft-native names above — until an image’s items are regenerated they keep the purl-derived names, so the type value itself distinguishes regenerated from legacy items (go-module and python only appear on regenerated items; legacy items carry golang and pypi).

SBOM content sources: the Index Lambda builds items from the native syft scan of each architecture — all runtime packages, with the type vocabulary carried verbatim from syft — and merges the per-arch Hermeto attestation on top: attestation entries with a syft runtime twin (same name and version-release; a leading RPM epoch prefix on the syft side, e.g. 1:3.5.5-1.hum1, is excluded from the comparison) are dropped, and no-twin entries with a matching arch (or noarch) become build rows carrying their attestation purl verbatim. arch=src entries are skipped — source rows are not emitted, the SRPM identity is derivable from any binary row’s sourcerpm. A version skew keeps both entries — visible, never a silent drop. The merge is gated by the distro switch (classify_by_hermeto): Hummingbird merges; Rawhide serves syft-only items. The attestation SPDX itself remains load-bearing only for derived fields in sync (OVERVIEW licenses, the variant-label fallback) and the conditionally-seeded RELEASE_SBOM for digests that were never scanned while current (an atomic write-if-absent write) — SBOM item content no longer depends on the mobster-merged attestation.

Historical items: items written before the cutover keep their attestation-derived content and purl-derived type vocabulary until the SBOM backfill regenerates them from the stored scan blobs (see the schema revision note above for the interim marker). Digests with no scan blob at all keep their attestation-derived RELEASE_SBOM seed indefinitely.

HistoryResponse / HistorySummary

Release timeline for a specific (stream, variant) combination.

Field Type Hummingbird External
digest String Manifest list digest Same
oldest_created String Earliest OCI created Same
architectures String[] From OCI manifest list Same
source_commit String From SLSA attestation Omitted — no signed attestations on upstream images
source_repo String From SLSA attestation Omitted — no signed attestations on upstream images

VulnerabilitiesResponse

Vulnerability scan results are identical in structure. The only behavioral difference: ignore_list (CPE-based false-positive suppression) is populated for Hummingbird images but empty for external images, since suppression rules are maintained per Hummingbird image.

Timestamp Fields

The oldest_created field on ImageSummary, Tag, and HistorySummary is the earliest OCI created timestamp across all architectures in the release. All architectures were built at or after this date, making it useful for conservative staleness detection.

The specifications endpoint returns per-architecture data keyed by architecture name. Each architecture’s created field is the direct OCI config root timestamp for that specific architecture.

Deprecation status

The containers repository can mark an image stream with the io.hummingbird-project.deprecated=true OCI label. The catalog exposes this status in two places:

  • Tag.deprecated identifies a release carrying the label.
  • ImageSummary.deprecated_streams lists streams whose current release is deprecated.

Both fields are additive API fields. Older DynamoDB items without these fields are read as active releases with no deprecated streams.

Usage

All tools are built as a single catalog binary with subcommands (api, sync, sync-lambda, scan, scan-lambda, index, index-lambda, enqueue-lambda, metrics, metrics-lambda). The binary is built in the Rust container and CLI subcommands are run in the gitlab-ci container (which provides grype, syft, and other tools). Only make and podman are required.

Sync

# Dry run (print items to stdout)
make container-catalog/sync ARGS="--distro rawhide --dry-run"

# Populate DynamoDB
make container-catalog/sync ARGS="--distro rawhide --table-name <table>"

Sync Lambda

The sync-lambda subcommand runs as an AWS Lambda function triggered by SNS Release events from kubernetes-event-forwarder. It incrementally syncs a single image release to DynamoDB (~22 registry API calls per release vs ~104 for a full sync).

The Lambda:

  1. Decodes gzip+base64 SNS messages
  2. Filters for Succeeded releases targeting the configured Quay.io namespace. Ignores DELETED events and releases with a non-null deletionTimestamp.
  3. Fetches OCI manifest data for the new digest
  4. Writes per-digest items (DETAILS, RELEASE_DETAILS, and a RELEASE_SBOM seed for digests that were never scanned while current — an atomic write-if-absent conditional write, so a sweep or a retried invocation can never overwrite index-normalized content); SBOM#{canonical} items are written by the Index Lambda
  5. Merges into aggregate items (TAGS, HISTORY, OVERVIEW, DIRECTORY)
  6. Fetches README from GitLab for OVERVIEW content (uses README.md for Hummingbird, rawhide/README.md for Rawhide)
  7. Enqueues one index event per release on the IndexQueue — the message carries the image, the canonical just written, the digest, and the manifest-list architectures, everything the Index Lambda needs (it never reads TAGS). A redelivered release re-emits (the known-digest early exit is cheap and the index dedups); historical releases do not emit.

The attestation SBOM fetch remains load-bearing for OVERVIEW licenses and the variant-label fallback — moving licenses off the attestation is follow-up polish.

Registry fetch errors (manifest, SBOM, attestation) propagate as hard failures so the Lambda retries automatically (up to 2 retries with backoff) before sending to the DLQ. GitLab README failures are non-fatal – the existing README is preserved if the fetch fails.

Environment Variable Description
TABLE_NAME DynamoDB table name
INDEX_QUEUE_URL SQS queue URL for index events
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Index

The index subcommand generates native Syft JSON for all non-superseded image digests and uploads them to S3 for scanner consumption. It reads the image directory and TAGS from DynamoDB, checks S3 for existing objects (dedup via HeadObject), runs syft <image> --platform linux/amd64, gzips the output, and uploads to s3://{bucket}/grype/{image}/{hex}.json.gz.

# Dry run (show what would be uploaded)
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --dry-run"

# Backfill S3 for all images
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket>"

# Backfill a single image
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --image caddy"

Index Lambda

The index-lambda subcommand runs as an AWS Lambda function triggered by the IndexQueue (SQS, BatchSize: 1, batch-item-failure reporting). The Sync Lambda enqueues one TagReleaseEvent per release after its writes; it is the same event shape and per-event processing structure as the external index — the function is the future unified index, with the blob-key prefixes and the Hermeto-merge gating as per-deployment configuration. It generates the native Syft JSON for scanner consumption and owns the SBOM items.

The Lambda:

  1. Parses TagReleaseEvent records (image, canonicals, digest, architectures); a record that fails to parse or process is reported as an SQS batch-item failure and retried individually by the queue, redriving to the IndexDLQ after 3 attempts.
  2. Checks S3 for existing objects (HeadObject dedup by digest). On a hit, it heals any canonical whose SBOM# item is missing (canonical rename, DLQ replay) by rebuilding it from the persisted SPDX record — RELEASE_SBOM#{hex} is only written when absent, so a recovered primary-arch-only item cannot downgrade a multi-arch item. Blobs written by the catalog index CLI carry no persisted SPDX; the recovery skips those with a warning (the SBOM backfill is their designated repair).
  3. Resolves the architecture list and the per-arch Hermeto attestations from the manifest list over the public registry (no IAM). A missing per-arch attestation degrades that arch to syft-only; a manifest-list failure falls back to the event’s recorded architectures.
  4. Runs syft <image>@<digest> --platform linux/{arch} for every architecture, bounded by the invocation deadline: the primary arch (amd64 when present, otherwise the first) runs with dual output (syft-json + spdx-json — its spdx is the persisted recovery input) and is a hard failure; the remaining arches run native-only, and their failures degrade the SBOM to primary-arch-only.
  5. Merges Hermeto build-only entries per arch (see SBOM content sources).
  6. Re-verifies every canonical against the registry after all arch scans, immediately before the writes: canonicals that moved to a newer digest are dropped, a digest no canonical references anymore is discarded entirely (no items, no upload), and a failed verification fails the record after the writes so the retry heals the skipped canonical.
  7. Writes SBOM#{canonical} per still-current canonical plus RELEASE_SBOM#{hex} to DynamoDB, then the primary-arch SPDX to s3://{bucket}/sbom/{image}/{hex}.json.gz with primary-arch object metadata (the recovery input for step 2), then the gzipped syft JSON to s3://{bucket}/grype/{image}/{hex}.json.gz — the last write and the completion marker that triggers the Scan Lambda via S3→SQS and arms the dedup key.

TAGS is never read — the event carries everything the function needs, and canonical identity is re-resolved against the registry.

Failures are retried by the queue (up to 3 attempts) before the IndexDLQ. S3 gaps can be backfilled with the catalog index CLI (it writes S3 objects only); SBOM item gaps are healed by the already-indexed recovery on the next event or DLQ redrive, or by the SBOM backfill.

Environment Variable Description
SCAN_DATA_BUCKET S3 bucket for scanner data
TABLE_NAME DynamoDB table name
SCAN_KEY_PREFIX Scan-blob key prefix (default grype/)
SBOM_KEY_PREFIX SPDX-record key prefix (default sbom/)
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Scan

The scan subcommand reads image listings and tags from DynamoDB and runs Grype against each image using native Syft JSON from S3 (--bucket is required). Results include a first_seen timestamp per CVE, tracked at the group+variant+stream level and carried across releases for SLI computation.

Tagged scans read native Syft JSON from S3 (grype/{image}/{hex}.json.gz), matching the lambda pipeline. Historic releases (--scope all) read the same S3 objects; digests whose object is missing are skipped with a warning.

# Dry run (print items to stdout)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --dry-run"

# Scan and write to DynamoDB (purge stale vuln data first, implies --scope=all)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --purge"

# Scan only non-superseded (current) tags
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --scope non-superseded"

# Scan all releases including historic (tagless) releases
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --scope all"

# Scan a single image
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --image caddy --dry-run"

Scan Lambda

The scan-lambda subcommand runs as an AWS Lambda function triggered by SQS messages. Two paths feed the ScanQueue for Hummingbird images, and two parallel paths for external images:

  1. Real-time (S3 event notification): When the Index Lambda uploads a new Syft JSON to S3, an s3:ObjectCreated event (filtered on grype/ prefix) is sent directly to SQS
  2. Hourly (Enqueue Lambda): Fans out all non-superseded digests when the Grype vulnerability database has been updated, sending {"bucket": "...", "key": "grype/{image}/{hex}.json.gz"} messages
  3. External real-time: When the External Index Lambda uploads Syft JSON to grype-external/{registry}/{hex}.json.gz, the same ScanQueue receives the event
  4. External hourly: The Enqueue Lambda also fans out external re-scans using grype-external/{registry}/{hex}.json.gz keys

The Lambda accepts both S3 event notification JSON and direct {"bucket", "key"} messages. It detects the S3 key prefix to dispatch between two code paths:

  • grype/ prefix → Hummingbird path: writes IMAGE#{name}/VULNERABILITIES#{canonical} and IMAGE#{name}/RELEASE_VULNERABILITIES#{hex}
  • grype-external/ prefix → external path: parses (registry, hex) from the key, then writes EXTERNAL#{registry}/VULNERABILITIES#{canonical} and EXTERNAL#{registry}/RELEASE_VULNERABILITIES#{hex} using the same Grype scan and first_seen tracking logic as Hummingbird

For each message the Lambda:

  1. Downloads and decompresses the native Syft JSON from S3
  2. Runs Grype on the raw bytes
  3. Looks up all canonical tags matching the digest from the TAGS item
  4. Writes VULNERABILITIES#{canonical} for each matching tag (with first_seen tracking) and RELEASE_VULNERABILITIES#{hex} once per digest

Processing is per-digest: a single Syft JSON serves all canonicals sharing that digest, avoiding redundant Grype invocations.

The Lambda downloads and hydrates the Grype DB on the first handler invocation of each cold start (cached in /tmp for warm invocations), processes one message per batch, and reports partial batch failures so failed records return to the queue.

Concurrency is capped at 100 concurrent invocations (ScalingConfig on the SQS event source). Every Grype DB update triggers a full-catalog rescan fan-out of several hundred scans, and without a cap the event source scales with that backlog — measured at ~490 concurrent executions, about half the account’s shared 1000-execution limit, with each cold container downloading the multi-GB Grype DB. The cap keeps one function at ~10% of the account budget and amortizes the DB setup across warm invocations. The template deploys to both the hummingbird and rawhide stacks, so the theoretical simultaneous maximum is 2 × 100 from the scan Lambdas plus 2 × 5 from the external index Lambdas — 210 of the account limit.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Enqueue Lambda

The enqueue-lambda subcommand runs hourly via EventBridge Schedule. On each invocation it performs a lightweight HTTP GET of the public Grype DB listing (latest.json, ~200 bytes) and compares the built timestamp against the stored CATALOG/LAST_FULL_SCAN_DB item in DynamoDB. If the DB hasn’t changed, it returns early (~23 of 24 hourly invocations short-circuit). When an update is detected it runs two fan-out loops:

  1. Hummingbird images: reads the image directory and all TAGS items, deduplicates by digest, and sends one SQS message per unique digest. Messages use {"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}.
  2. External images: reads the external-images config from DynamoDB (CATALOG/EXTERNAL_CONFIG), then for each configured registry reads EXTERNAL#{registry}/TAGS and fans out non-superseded digests with grype-external/{registry}/{hex}.json.gz keys. If no TAGS item exists (first run, before External Sync Lambda), the registry is skipped. A malformed or missing EXTERNAL_CONFIG degrades gracefully — a warning is logged and the Hummingbird fan-out already sent is unaffected.

Both loops use the same SendMessageBatch batching and !superseded digest filter.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SCAN_QUEUE_URL SQS queue URL for scan messages
SCAN_DATA_BUCKET S3 bucket for scanner data
GRYPE_DB_LATEST_URL Grype DB listing URL (has sensible default)
SENTRY_DSN Optional Sentry DSN for error tracking

Metrics

The metrics subcommand performs a one-shot read of all non-superseded vulnerability data from DynamoDB and writes the catalog-wide vulnerability aggregate to DynamoDB (PK=CATALOG, SK=VULNERABILITIES).

make container-catalog/metrics ARGS="--table-name <table>"

Metrics Lambda

The metrics-lambda subcommand runs as a DynamoDB Stream-triggered Lambda that writes a catalog-wide vulnerability aggregate to DynamoDB (served by GET /v1/vulnerabilities).

How It Works

The Lambda is triggered by DynamoDB Stream events filtered on VULNERABILITIES# and TAGS changes, with a 60-second batching window and reserved concurrency of 1 (single instance). It maintains an in-memory active CVE table across warm invocations:

  • Cold start: Reads CATALOG/DIRECTORY, all TAGS, and all non-superseded VULNERABILITIES# items from DynamoDB to build the full table (~2-3s at 10000 tags)
  • Warm invocations: Incrementally updates the table from stream event keys (~50-100 GetItem calls per batch)
  • After each invocation: Recomputes the aggregate

Stream events for EXTERNAL# PKs (external images) are intentionally excluded from the aggregate. The incremental_update path filters on pk.starts_with("IMAGE#") so that external-image VULNERABILITIES# stream events do not pollute the Hummingbird-only CATALOG/VULNERABILITIES item. The GET /v1/vulnerabilities endpoint reflects only Hummingbird CVE data.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN

Deployment

make container-catalog/build
make container-catalog/deploy

External Images

The external images feature enables CVE comparison between Hummingbird and external image sources (e.g. standard upstream registries). External image data is exposed under the /v1/external-images/ namespace, which is fully additive — existing /v1/images/ clients are unaffected.

Overview

External images share the same DynamoDB table as Hummingbird images but use an EXTERNAL#{registry} PK prefix instead of IMAGE#{name}, where {registry} is the full registry pull reference (e.g. docker.io/library/nginx). The same response types, SK patterns, and API conventions apply: tags, details, SBOM, vulnerabilities, history, and release snapshots all work identically. Provenance fields (source_repo, source_commit, containerfile, slsa_level, build_system) are empty strings for external images — they are not available from upstream registries.

The /v1/external-images/config endpoint returns the config that maps Hummingbird images to their external sources. The join key for a given image detail page (IMAGE#{repository}) is the repository on each tag entry (resolved from the source-level default when not set per-tag). Clients filter tag entries by repository, then match stream and variant values against the page’s stream/variant selection to construct the /v1/external-images/{percent-encoded-registry}/… URLs (registry refs containing / must be percent-encoded in the URL path).

Tag-based discovery and re-verification (resolve_digest — HEAD manifest by tag, used by both External Sync and External Index) goes directly to registry-1.docker.io with anonymous tokens from auth.docker.io (service=registry.docker.io). HEAD manifest requests are free against Docker Hub’s pull limit (100 manifest GETs per hour per IP, verified empirically), return docker-content-digest, and always resolve to Docker Hub’s current view of the tag. ratelimit-remaining from each HEAD response is logged as a numeric budget gauge.

Digest-pinned architecture fetches (fetch_details — manifest GETs by digest) and syft image pulls both route through mirror.gcr.io first, with a direct-docker.io fallback on any failure. Digest-pinned requests are content-addressed and cannot be stale, so the mirror is a safe first attempt; the fallback covers mirror ingestion lag on a freshly discovered digest. In Lambda each syft attempt runs under a per-call budget derived from the invocation deadline; in CLI mode syft runs until it exits or the operator interrupts it. Stored data uses the canonical docker.io/... registry name; other registries are accessed directly.

Configuration Format

The external-images configuration is stored in DynamoDB at CATALOG/EXTERNAL_CONFIG and served at GET /v1/external-images/config. The JSON config is maintained outside this repo (e.g. in the infrastructure repo) and applied to DynamoDB via the catalog config set CLI command.

The config is a flat list of sources. Each source maps registry tags to (repository, stream, variant) triples — the same concepts Hummingbird uses for canonical determination and mark_superseded. Stream and repository values are whatever the corresponding Hummingbird image uses. Variant names use Hummingbird names where an equivalent exists (e.g. builder for upstream dev images); upstream-only variants like slim or alpine keep their upstream names.

Each tag carries its own repository — the Hummingbird repository the tag’s comparison belongs to. This allows tags from the same upstream registry to join different Hummingbird repositories (e.g. docker.io/library/caddy tags 2 → repository caddy and 2-builder → repository xcaddy):

{
  "sources": [
    {
      "registry": "docker.io/library/caddy",
      "tags": [
        {
          "tag": "2",
          "repository": "caddy",
          "stream": "2",
          "variant": "default"
        },
        {
          "tag": "2-builder",
          "repository": "xcaddy",
          "stream": "2",
          "variant": "builder"
        }
      ],
      "variants": [
        {
          "name": "default",
          "description": "Caddy server",
          "base": "default",
          "builder": false,
          "fips": false
        },
        {
          "name": "builder",
          "description": "Build image with xcaddy",
          "base": "default",
          "builder": true,
          "fips": false
        }
      ]
    }
  ]
}

Every tag must carry a repository — the per-tag form is the only shape. The older source-level repository field (a default inherited by all tags) was transitional and has been removed: configs are parsed with deny_unknown_fields, so a source-level repository is now a parse error. The containers aggregation emits per-tag repository on every tag.

Stream Assignment

External image stream values follow the same rules as Hummingbird images (see Choosing a Stream Value in the containers repo). Key rules for external image config:

  • If the Hummingbird image uses numbered streams (e.g. "1.27", "3.14"), external image tags must map to the same stream values. A Docker Hub latest tag maps to the current numbered stream — it is never a separate "latest" stream alongside numbered ones.
  • Use "latest" as a stream value only for sources where no meaningful version series exists (rolling releases, date-based versions).
  • The assign_canonicals function groups tags by (stream, variant, digest). Within each group, the most semver-specific tag becomes canonical. When latest and 1.27 both map to stream "1.27" and resolve to the same digest, 1.27 wins by specificity.

Variant values share the Hummingbird vocabulary where equivalents exist: use "builder" for upstream dev images that add a shell and package manager (e.g. Chainguard’s latest-dev), and "default" for plain runtime images. Upstream-only variants with no Hummingbird equivalent (e.g. "slim", "alpine" on Docker Hub) keep their upstream names. CVE data for all tracked variants is served the same way, so clients can present them however fits.

To add a new image or source, update the config JSON and run catalog config set --file <path> --table-name <table> to apply the change to DynamoDB (no Lambda redeploy needed).

External Release Lambda

Triggered by EventBridge on a schedule (every 6 hours). Reads the external-images config from DynamoDB and sends one SQS message per configured registry to ExternalSyncQueue via send_batch. Sends unconditionally — change detection is handled downstream by External Sync Lambda.

The Lambda is lightweight: one DynamoDB GetItem and one or two SQS SendMessageBatch calls per invocation. No registry calls, no S3 access.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO hummingbird (no-op on rawhide)
EXTERNAL_SYNC_QUEUE_URL SQS queue URL for per-registry messages
SENTRY_DSN Optional Sentry DSN for error tracking

External Sync Lambda

Triggered by ExternalSyncQueue (SQS, BatchSize: 1). Each invocation processes one registry reference — identical scope to SyncFunction processing one Konflux release.

  1. Resolves all configured tags to digests via registry HEAD requests. Digests are validated (sha256:[0-9a-f]{64}) before use. Attestation manifests (platform.architecture = "unknown") are filtered out. All configured tags must resolve successfully; partial resolution aborts the write to prevent silently shrinking the TAGS item.
  2. Compares with current EXTERNAL#{registry}/TAGS in DynamoDB; early-returns if no digests changed (no-op on most invocations).
  3. Groups tags by digest within each (stream, variant) pair; picks canonical by semver specificity (1.27.2 beats 1.27).
  4. Writes DETAILS, RELEASE_DETAILS, HISTORY, OVERVIEW, and EXTERNAL_DIRECTORY to DynamoDB.
  5. Emits one SQS message per non-superseded digest to ExternalIndexQueue. Emission is unconditional rather than change-detection based: the index Lambda’s S3 existence check makes an event for an already-indexed digest a cheap no-op, and unconditional emission restores the self-healing of the old per-registry topology — canonical renames with unchanged digests get their SBOM records backfilled on the next sync write. When several canonicals resolve to the same digest, they share one event and one scan. Each message carries everything External Index Lambda needs: registry, canonicals, digest, architectures.
  6. Writes TAGS last so it serves as a completion marker — a crashed invocation will be retried in full, and an SQS send failure in step 5 is retried without the early-exit firing (TAGS still holds the old state).

The moment sync observes a changed digest is the release event for an external image — sync stands where the Hummingbird build system stands.

Environment Variable Description
TABLE_NAME DynamoDB table name
DISTRO rawhide or hummingbird
EXTERNAL_INDEX_QUEUE_URL SQS queue URL for per-tag index events
SENTRY_DSN Optional Sentry DSN for error tracking

External Index Lambda

Triggered by ExternalIndexQueue (SQS, BatchSize: 1). Each invocation processes one digest from a release event emitted by External Sync Lambda. The event carries everything needed — registry, canonicals, digest, architectures — so TAGS is not read, eliminating the first-run race between the two lambdas.

  1. Validates the event at the queue boundary (registry, digest, canonicals, architectures). Checks S3 for an existing grype-external/{registry}/{hex}.json.gz object; skips if already present (idempotent, mirrors IndexFunction). If the S3 object exists but a canonical’s DynamoDB SBOM item is missing (e.g. a canonical rename after the last syft run), the SBOM is backfilled from the persisted SPDX at sbom-external/{registry}/{hex}.json.gz — one scan serves every canonical of the digest.
  2. Runs Syft with dual output for the primary architecture (amd64 when present, otherwise the first architecture of the event — some upstream images are single-arch non-amd64): syft-json for Grype, spdx-json for the SBOM. Then one spdx-json run per remaining architecture; one ArchSbom per arch is stored in the SbomResponse.
  3. Re-verifies the registry digest has not changed during the Syft run. If the digest changed, the output is discarded to avoid indexing a stale digest. SBOM writes happen only after this verification.
  4. Writes one SBOM#{canonical} per canonical of the digest (all from this single scan) plus RELEASE_SBOM#{hex} to DynamoDB.
  5. Uploads the gzipped Syft JSON to S3; the S3 event notification triggers the Scan Lambda via SQS. SPDX is stored separately at sbom-external/{registry}/{hex}.json.gz (different prefix to avoid triggering a spurious scan).

Each invocation is bounded to one digest’s architecture matrix (minutes, not the whole registry), so timeout and OOM blast radius are per-digest rather than per-registry. DLQ entries are single-digest releases — replayable and meaningful. DLQ entries are not retried automatically: recover via SQS DLQ redrive, or wait for the registry’s next change (any change re-emits every non-superseded digest).

Environment Variable Description
TABLE_NAME DynamoDB table name
SCAN_DATA_BUCKET S3 bucket for scanner data
DISTRO rawhide or hummingbird
SENTRY_DSN Optional Sentry DSN for error tracking

Data Freshness and Known Asymmetries

  • External images are polled every 6 hours. Grype DB re-scans run hourly, same as Hummingbird images.
  • The CATALOG/VULNERABILITIES aggregate (GET /v1/vulnerabilities) reflects only Hummingbird CVE data. There is no external-images catalog-wide CVE aggregate.
  • Sources that offer only a latest tag and have no numbered version series will always have "latest" as their canonical.

For per-field availability differences between the two namespaces, see the response type tables in the API Endpoints section.

Docker Hub Limits and mirror.gcr.io Behavior

Reference data for the routing decisions in the External Sync and External Index Lambdas (measured 2026-11-10).

Docker Hub metering

Only manifest GETs count against the pull limit — one unit each (index and platform manifests separately). HEAD manifest requests, blob HEAD/GETs, auth token requests, and tag listing GETs are unmetered.

Access type Limit (anonymous) Notes
Manifest GETs 100 / hour / IPv4 (or IPv6 /64) ratelimit-limit: 100;w=3600 in headers
Manifest HEADs Unmetered Still returns docker-content-digest
Blob GETs / HEADs Unmetered Measured; Docker docs do not confirm
Auth token fetches Unmetered
Tag listing GETs Unmetered Full listing in one response; paginated via ?n=&last=; mirror.gcr.io serves the same endpoint as its cache inventory

The registry API endpoint is registry-1.docker.io (the bare docker.io host redirects to the Docker website). Anonymous tokens are issued by auth.docker.io with service=registry.docker.io and per-repository scope; they are valid for approximately 5 minutes. ratelimit-remaining is present on HEAD responses; resolve_digest logs it as a budget gauge.

Routing policy

Request Route Reason
resolve_digest (HEAD by tag) registry-1.docker.io direct Tags are mutable; mirror can serve stale. HEADs are unmetered.
fetch_details (GET by digest) mirror.gcr.io → docker.io fallback Content-addressed — cannot be stale. Mirror-first; falls back direct on any failure.
Syft image pulls mirror.gcr.io → docker.io fallback Mirror-first; retries direct on any failure.

mirror.gcr.io receives all digest-addressed traffic from this service as the first attempt.

Routing requirements

The routing design is governed by the following requirements:

  • R1 — tag resolution is authoritative: resolve_digest HEADs go directly to registry-1.docker.io with anonymous auth.docker.io tokens. Tags are mutable; HEAD manifest requests are unmetered and always return Docker Hub’s current view of the tag.
  • R2 — digest-addressed fetches go mirror-first with direct fallback: fetch_details (manifest list, per-arch manifests, config blobs) and the @digest-pinned syft pulls both try mirror.gcr.io first. Content-addressed fetches cannot be stale — only missing — so the mirror is safe; the direct fallback covers mirror ingestion lag on a freshly discovered digest (the window between direct tag resolution and the mirror ingesting the new object). The fallback costs ~1 + archs metered manifest GETs, only on a mirror miss; when the mirror serves the request the metered budget stays at zero.
  • R3 — mirror failures fall back to docker.io: any failure on a mirror GET retries the same request directly against docker.io — both for syft pulls (via run_syft_outputs in scanner.rs) and for fetch_details metadata GETs (via routed_get in external_sync_lambda.rs). Reliability is the reason; error-classification by response body would be brittle. A fallback costs approximately 2–3 metered manifest GETs for a syft pull (blobs are unmetered) or ~1 + archs GETs for a fetch_details miss; the fallback granularity for a pull is the whole ref since syft fetches from a single reference. In Lambda each attempt runs under a per-call budget derived from the invocation deadline (deadline − now − 60 s headroom for post-scan work); a zero remaining budget returns the error before spawning. In CLI mode syft runs until it exits or the operator interrupts it — Ctrl-C causes kill_on_drop to terminate the child and TempDirGuard to remove its temp dir. The stale sweep uses the owner PID as the garbage authority: a syft temp dir is removed when its owner process has exited.
  • R4 — Accept-set consistency: resolve_digest HEADs include both index and platform manifest media types (application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.index.v1+json, plus the single-manifest types) so a tag resolves to the index digest rather than a platform-specific one. The same Accept set is used for fetch_details GETs via the mirror.
  • R5 — bounded concurrency against the mirror: the mirror’s anonymous limiter escalates to persistent per-IP 429s under sustained load, so pull fan-out must stay bounded. Partially holds via External Index Lambda MaximumConcurrency: 5.
  • R6 — ratelimit-remaining budget gauge: resolve_digest logs the ratelimit-remaining header from each Docker Hub HEAD response as a numeric structured field (remaining), including from error responses (e.g. 429) where remaining=0 is most actionable. The header is returned on HEAD responses at no additional cost. The header value may carry a window suffix (100;w=3600) — only the numeric prefix is parsed.

mirror.gcr.io behavior

  • Demand-driven cache: objects ingest only when requested; each object (tag mapping, manifest, blob) warms independently.
  • Digest-addressed content: served cold (fetched from Docker Hub on miss). Whole-image cold pulls work end-to-end once warmed.
  • Blob ingestion is async and flaky: a pull can fail with BLOB_UNKNOWN for layers that ingest over subsequent requests (observed taking days). Retries serve as cache warming.
  • Eviction: images not requested for approximately 5 days are removed. Cached does not mean stable.
  • Rate limiting: burst 429s after approximately 10 rapid manifest requests, escalating to a persistent per-IP throttle under sustained load. Bounds pull fan-out concurrency.
  • HEAD is unreliable as a cache probe: can 404 for content that is cached and servable. GET /v2/{repo}/tags/list lists cached tag mappings only — digest-addressed content can be served while its tag is unlisted.

Configuration

catalog sync

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name
--purge Delete all items before writing
--cache-dir Cache directory (auto-detected)
--image Sync only a specific repo
--legacy-discovery Use GitLab-based repo discovery

catalog index

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name (for image/tag discovery)
--bucket S3 bucket for scanner data
--dry-run Print what would be uploaded without uploading
--image Index only a specific image
--parallel Number of concurrent operations (default: 2)

catalog scan

Argument Description
--distro rawhide or hummingbird
--table-name DynamoDB table name (required)
--bucket S3 bucket with native Syft JSON (required)
--scope non-superseded, tags (default), or all
--dry-run Print items without writing
--purge Purge vuln data before writing (implies --scope all)
--parallel Number of concurrent scans (default: 4)
--image Scan only a specific image
--tag Scan only a specific tag (requires --image)

catalog config

catalog config set --file <path> --table-name <table> reads the external-images JSON config and writes it to the DynamoDB item CATALOG/EXTERNAL_CONFIG. Run once on initial setup and after each config change. No Lambda redeploy is needed — all External* Lambdas read config from DynamoDB at invocation time.

Argument Description
--table-name DynamoDB table name
--file Path to external-images JSON file

SAM Parameters

Parameter Description
Distro rawhide or hummingbird
CacheEnabled Enable CloudFront caching
CatalogDomainName Catalog web UI domain
ApiDomainName API domain
HostedZoneId Route53 hosted zone
CorsOrigins Comma-separated CORS origins (default *)
SnsTopicArn SNS topic ARN for Release events (enables sync/index/external Lambdas)

The External Release, Sync, and Index Lambdas are all conditioned on HasSnsTopicArn (a feature flag historically tied to the Release-events topic) and are only deployed for the hummingbird stack. The external chain itself is queue-based: the EXTERNAL_SYNC_QUEUE_URL and EXTERNAL_INDEX_QUEUE_URL environment variables are set automatically from the queue resources defined in the same SAM template.

Frontend

The catalog web UI is a Lit 3 SPA (Web Components) with Tailwind CSS, built per-distro with Vite. Source is in container-catalog/frontend/.

The image detail view includes an External tab (gated behind VITE_VULNERABILITIES_ENABLED) that compares Hummingbird images against their external counterparts. It shows a summary card with CVE and runtime package count deltas, side-by-side severity breakdowns, a categorized CVE table (only-in-upstream / only-in-Hummingbird / shared), and an SBOM package diff.

Only make and podman are required (no local Node.js needed). Defaults from .envrc.defaults are applied automatically.

# Install dependencies
make container-catalog/frontend/setup

# Development server at http://localhost:5173
make container-catalog/frontend/dev

# Production build
make container-catalog/frontend/build

Host variants (*-host) run without podman (for CI or local Node.js).

Frontend Build Variables

Variable Description
VITE_API_URL API base URL for the distro
VITE_DISTRO rawhide or hummingbird
VITE_DISTRO_LABEL Display label for current distro
VITE_OTHER_CATALOG_URL URL of the other distro’s catalog (optional, hides link if unset)
VITE_OTHER_DISTRO_LABEL Display label for other distro (optional)
VITE_VULNERABILITIES_ENABLED Show vulnerabilities and external comparison tabs

Development

# Backend
cargo test                    # Run tests
cargo clippy --all-targets   # Lint
cargo fmt                     # Format

# Frontend (host variants, requires local Node.js)
cd container-catalog/frontend
npm run typecheck             # Type check
npm run build                 # Production build

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.19.1 - Container Catalog Design Decisions

Design Decision Registry (DDR) for container-catalog — every major design choice, the alternatives considered, and why the chosen approach won. For operational usage, see Container Catalog.

DDR-1: Historical RELEASE_SBOM writer survives the SBOM cutover as a write-if-absent seed

Decision: When the index function takes over SBOM-item writes, sync’s build_release_sbom_item keeps its RELEASE_SBOM#{hex} construction, written via an atomic write-if-absent conditional write (put_content_versioned with attribute_not_exists(ver) — every post-cutover writer stamps ver, so the condition blocks any rewrite of normalized content); the historical branch of merge_release writes only digest-keyed items; SBOM#{canonical} becomes index-exclusive.

Context: The SBOM cutover (W1 MR2) moves SBOM-item writes from sync to index. Index only writes for digests current in TAGS; the SBOM backfill enumerates scan blobs. Digests that were never scanned while current therefore lose every writer. Production census (2026-11-11, 10 images, 10,697 history digests, vuln-item presence as the scan/blob proxy): 228 digests (2.1%) have an SBOM item but were never scanned — 225 from 2026-04, before the scan pipeline wrote blobs, plus ~1/month of new misses (events raced out of TAGS, dropped events). The registry tagging model (unique sha256-{hex}/timestamp tags per build; ~a dozen moving plain version tags) means every rebuild supersedes the previous digest out of TAGS, so this population accrues slowly but indefinitely. The only code path that can serve these digests is the catalog sync CLI’s historical sweep — the production sync Lambda processes only each event’s release and never sweeps. The sweep is an on-demand repair/repopulation tool (“Full DynamoDB population”, --purge).

Alternatives: (a) Accept the gap — delete the RELEASE_SBOM construction outright; SBOM items become exclusively the index/backfill domain; the frozen 225 keep their existing items, new misses get history and details but no SBOM, documented as a known gap. (b) Keep an unconditional retained write — preserve today’s behavior for the historical branch unchanged. (c) Extend the backfill to synthesize RELEASE_SBOM from attestations for no-blob digests.

Rationale: (a) rejected because a repair tool with a measured blind spot is an operational footgun: after an incident or data loss, a catalog sync run would leave SBOM holes nothing else can fill (no blob → the backfill enumerates blobs and skips them; no event → index never sees them). (b) rejected because the sweep population is dominated by ex-current, blob-backed digests that the SBOM backfill normalizes to scan-derived content; sync’s writes are unconditional (batch_write; put_content_versioned has no content comparison), so every CLI run would overwrite blob-derived items back to attestation-derived content, silently undoing the backfill. (c) rejected because the backfill is one-off by design — it cannot cover future misses, only the current cohort. The write-if-absent seed keeps the repair tool complete for exactly the population only it can reach, while the conditional write makes it converge with the backfill instead of fighting it: sync seeds once (attestation-derived, today’s field spellings — consistent with the accepted mixed-vocabulary story for no-blob digests), index and the backfill own the final content for blob-backed digests, and a CLI run never rewrites an item that already exists. Zero new IAM (one DDB read); ~20 lines retained plus the guard, confined to a CLI-only code path.

DDR-2: Index invocation converges on the queue-decoupled pattern

Decision: The internal index function drops its SNS Release subscription and consumes a queue of tag-release events; sync performs its DynamoDB writes and then enqueues one event per release canonical, using the shared TagReleaseEvent shape (the promoted ExternalTagReleaseEvent). The function is shaped as the future unified index — the same per-event processing structure as the external index, with the event shape and blob-key prefixes already shared — so the external plane’s unification (retiring external-index-lambda into this function) remains a small, well-bounded change.

Context: The internal index was SNS-triggered like sync, which forced it to read TAGS to learn a release’s canonicals — introducing the sync-fast/index-slow race (TAGS-absent → retry → DLQ) and requiring per-image error aggregation inside one invocation. The external plane had already solved invocation decoupling: its sync resolves tags, writes TAGS, then enqueues per-digest events to an index queue whose batch-item-failure semantics provide per-record retry for free. Downstream of sync’s writes the two planes are the same pattern; only the chain’s head differs (the internal producer is the near-realtime bus event, the external one a 6-hour poll).

Alternatives: (a) Keep the SNS trigger and heal the TAGS race with retries — the race class stays, one slow image can block the record, and the two index functions keep divergent shapes. (b) Unify into a single shared deployment rather than two deployments of one function — rejected for blast-radius isolation: a bad deployment would take out both planes at once. (c) Defer the convergence to the external-plane MR — rejected: every week of divergence adds code that both index functions must carry, and the queue convergence was already decided for the internal plane on its own merits.

Rationale: Queue decoupling deletes the TAGS-race class by construction: the event is emitted after the writes it depends on and carries everything the index needs (canonicals, digest, architectures), so the index never reads TAGS and canonical identity is re-resolved against the registry. SQS batch-item failures replace hand-rolled per-image aggregation with the platform’s own per-record retry. Sync redelivery is harmless — a duplicate event hits the index’s dedup skip and flows into the recovery path, which is the recovery’s designed purpose. The canonical re-verification stays regardless of ordering: the moved-digest mid-scan race is real in any invocation shape. Shaping the internal index as the unified function now (shared event type, external’s processing structure, env-configured prefixes) keeps MR4 small: point the external queue at the function and delete external-index-lambda, plus four parameterizations the planes still differ on — the DDB partition key (IMAGE# vs EXTERNAL#), the pull-reference source (config.pull_registry vs the raw registry reference), the arch source (manifest list via fetch_arch_attestations vs the event’s architectures), and the Hermeto-merge gating (classify_by_hermeto from DISTRO vs none). The event shape and the blob-key prefixes are already zero-code.

4.5.20 - Hummingbird Agent

An event-driven LLM agent that investigates CI/CD failures and posts findings as GitLab merge request notes. The agent executes markdown-defined workflows using tool calling, with all data processing running inside an isolated sandbox container.

Model adapters, provider calls, and shared AI usage metrics come from the in-process hummingbird-ai library.

For the architectural design rationale, module boundaries, security invariants, and design decision registry, see Agent Design. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.

flowchart TD
    Pipeline["Pipeline Fails"]
    MREvent["MR Created / Updated"]
    Slash["/hummingbird command"]

    Pipeline -->|"event"| Agent
    MREvent -->|"event"| Agent
    Slash -->|"command"| Agent

    subgraph Agent ["Hummingbird Agent"]
        APIs["GitLab + Konflux +<br/>Testing Farm"]
        Model["LLM<br/>(Gemini / Claude)"]
        subgraph Sandbox ["Isolated Sandbox (no network)"]
            Tools["jq / python3 / yq<br/>data processing"]
        end
        APIs <-->|"data"| Model
        Model <-->|"commands"| Tools
    end

    Note["MR Note<br/>(analysis or review)"]

    Agent -->|"posts"| Note
    Note -->|"reply to continue"| Agent

Features

  • Workflow-driven analysis - Investigation logic lives in .md files, not in code; easy to iterate without redeployment
  • Centralized YAML config - A single config file defines operational settings, workflows, enabled data sources with token env var names, project allowlists, and per-project limits
  • Sandboxed execution - All untrusted commands (jq, python3, shell) run in an isolated container, never on the host
  • Five sandbox backends - Podman for local development (network-isolated), direct K8s pod creation, Deployment-backed pod pool for low-latency production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or VMIRS-backed VM pool for heavier workloads requiring full OS isolation
  • Data source abstraction - GitLab, Konflux, and Testing Farm are registered as tool-calling functions the model invokes directly
  • Auto-spill for large outputs - Stdout/stderr and data source responses exceeding 4 KB are automatically saved to sandbox files with a compact preview returned to the model, keeping context usage bounded
  • Prompt caching - Gemini uses implicit server-side caching automatically; Claude uses explicit sliding-window cache breakpoints that reduce input token costs by ~80% on multi-turn agent runs
  • Token budget management - Per-call context ceiling and iteration-based soft/hard limits prevent runaway sessions
  • Session persistence - Conversation history, transcript, and sandbox files saved to S3 (production) or local directory (development) for debugging and future session resumption. Sessions are stored in a provider-neutral format, enabling model switching between conversations.

Architecture

flowchart LR
    subgraph input [Input]
        SQS["SQS Queue"]
        CLI["CLI --event"]
    end

    subgraph agentLoop [Agent Loop]
        WF["Workflow .md<br/>(system prompt)"]
        LLM["LLM&lt;br/&gt;(Gemini / Claude)"]
        TR["Tool Registry"]
    end

    subgraph tools [Tools]
        SE["sandbox_exec"]
        FTS["fetch_to_sandbox"]
        DS["Data Sources"]
    end

    subgraph sandbox [Sandbox Container]
        JQ["jq / python3 / yq"]
        Files["Spilled files"]
    end

    subgraph external [External APIs]
        GL["GitLab API"]
        KX["Konflux K8s"]
        TF["Testing Farm"]
    end

    subgraph output [Output]
        Note["GitLab MR Note"]
        Session["Session State"]
    end

    SQS --> WF
    CLI --> WF
    WF --> LLM
    LLM -->|"tool calls"| TR
    TR --> SE --> sandbox
    TR --> FTS --> sandbox
    TR --> DS
    DS --> GL
    DS --> KX
    DS --> TF
    DS -->|"auto-spill"| sandbox
    LLM -->|"final text"| Note
    LLM --> Session

An event (CLI --event JSON or SQS message) identifies a GitLab project and MR IID. The config file maps project paths to workflows and provides action, max_iterations, enabled data sources, and token env var names. Both run and serve use this config. The markdown body of the prompt file becomes the LLM system prompt. The agent loop iterates until the model produces a final text response or hits the iteration/context limit. Tool calls are dispatched through the ToolRegistry: sandbox_exec runs shell commands, fetch_to_sandbox pipes data source output into the sandbox, and direct data source calls return results inline (or auto-spill large responses to files).

For a detailed walkthrough of the model loop – what gets sent to the model each iteration, how tool calls flow, the exact wire format, and how user replies integrate for session resumption – see Agent Model Loop.

Event-Driven Triggers

In production, the agent consumes events from an SQS queue subscribed to the central SNS topic. The SNS filter policy delivers three event types:

  • gitlab::pipeline – Fires when a GitLab CI pipeline completes. The agent triggers on status=failed pipelines from merge_request_event sources where the triggering user has at least Developer access on the project. Only workflows with trigger: pipeline are executed. Since the pipeline stays open until all Konflux external stages resolve, this naturally waits for all builds and tests to finish before triggering.

  • gitlab::merge_request – Fires on MR open and update events. Only workflows with trigger: merge_request are executed. The agent triggers in three cases: a new MR is opened (action=open), code is pushed to an existing MR (action=update with oldrev), or a draft MR is marked as ready (action=update with a changes.draft transition from true to false). All other update events (labels, assignees, description changes) are skipped – they carry no new code. Draft MRs are skipped; SHA-based deduplication prevents reviewing the same code revision twice. Access checks and trigger_rules filtering evaluate the push author: for open and push events the webhook user is the pusher; for undraft transitions the agent resolves the actual pusher from MR system notes so that the person undrafting cannot bypass the access check on code pushed by an untrusted user.

  • gitlab::note – MR comment events. Two sub-flows:

    • Slash command (/hummingbird <workflow-name>): triggers a specific workflow. Prefix matching is supported (e.g. /hummingbird analyze matches analyze-failures). /hummingbird or /hummingbird help lists available workflows. The note author must have Developer+ access on the project. Optional runtime overrides can be appended: /hummingbird code-review model=claude-sonnet-4-6 max_iterations=25.
    • Reply to agent note: when a user replies to an existing agent note (which contains a session marker), the agent loads the previous session from S3 (conversation history + sandbox files) and continues the conversation with the user’s reply as input. The system prompt includes a CONTINUATION_PROMPT that prevents the model from re-running the full workflow. If the session is not found (expired/deleted), the agent falls back to a cold start. Replies may include overrides on a separate line (e.g. /hummingbird model=claude-opus-4-6); override lines are stripped from the user message. Overrides persist in the session until explicitly changed.

    Notes generated by the agent itself (containing session markers) are skipped to prevent infinite loops. Reply threading respects the internal_notes config: if the project requires internal notes but the original thread was public, the reply is posted as a new top-level internal note instead.

Both pipeline and merge_request triggers apply per-workflow trigger rules filtering. Each workflow may define an ordered trigger_rules array; the first matching rule decides whether the event is allowed or denied (implicit deny on fallthrough). Rules can match on pipeline status, user_regex/user_regexes, branch_regex/branch_regexes, and title_regex/title_regexes. All conditions within a rule are ANDed; multiple patterns within a regex field are ORed. The ! prefix negates a pattern. If no trigger_rules are specified, merge_request triggers allow all events and pipeline triggers default to allowing only failed status (preserving backward compatibility).

The legacy ignore_users and ignore_branches fields are still accepted and automatically desugared into equivalent trigger_rules (deny + catch-all allow). They cannot be mixed with explicit trigger_rules.

Rate limiting is per-workflow: each workflow’s thread count is tracked independently via JSON session markers that embed the workflow name and commit SHA. The max_runs_per_mr limit applies separately to each workflow on a given MR.

Events flow through a two-stage SQS pipeline. A slim ingress router forwards webhooks from the standard queue to an SQS FIFO queue, grouped by discussion_id for note events (ensuring same-discussion ordering) and by SQS MessageId for pipeline/MR events (no serialization needed). Phase 1 handlers validate the event and resolve the session, then post a continuation message back to the FIFO grouped by session_id. Phase 2 picks up the continuation and runs the workflow. This ensures all work for a given session is serialized even across multiple discussion threads.

The SQS infrastructure is defined in template.yaml (SAM/CloudFormation): a standard ingress queue (60s visibility timeout for the router hop) and a FIFO work queue (30-minute visibility timeout for workflow execution).

Model Configuration

The agent supports multiple LLM providers via a unified adapter interface.

Gemini

  • API key (local dev) – Set GOOGLE_API_KEY. Calls generativelanguage.googleapis.com directly. No region configuration needed.
  • Vertex AI (production) – Set GOOGLE_CLOUD_PROJECT and configure model_regions in the YAML settings. Uses Application Default Credentials (ADC) via google-auth. For OpenShift, mount a service account key JSON file and set GOOGLE_APPLICATION_CREDENTIALS, or use workload identity.

Gemini uses implicit server-side caching – repeated prefixes are automatically cached by the Vertex AI backend with no opt-in required. Cached input tokens are billed at 25% of the base input rate. The agent tracks cached token counts from API responses for cost estimation.

Anthropic Claude

Claude models are accessed via Vertex AI using the same GCP project (GOOGLE_CLOUD_PROJECT) with the region resolved from model_regions. Enable the desired model in the GCP Model Garden. Use model: claude-sonnet-4-20250514 in workflow config.

Prompt caching is enabled automatically for Claude models. The agent places explicit cache breakpoints on messages so that the full conversation prefix is served from cache on every turn after the first. Cache writes cost 1.25x the base input rate; cache reads cost 0.10x (90% discount). For a typical 20-iteration agent run, this reduces input token costs by approximately 80%. Ephemeral messages (iteration warnings, nudges) are excluded from cache writes to avoid polluting the cache with transient content.

Region configuration

GCP regions are configured via settings.model_regions in the config file – a map of model name prefixes to GCP regions. The agent resolves the region by longest-prefix match on the effective model name (same algorithm as cost estimation). Example:

settings:
  model_regions:
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global

At least one of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT must be set. If both are set, the API key takes precedence for Gemini models. Claude models always require Vertex AI mode (GOOGLE_CLOUD_PROJECT + model_regions); GOOGLE_API_KEY direct mode is for Gemini only.

Prerequisites

  • Python 3.11+
  • Podman for local sandboxes, or kubectl and Kubernetes access to a dedicated sandbox namespace for K8s sandboxes.
  • Authentication: GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT (see above)
  • For Claude models: enable the desired model in GCP Model Garden
  • GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens)
  • Kubeconfig with access to the Konflux cluster (if using Konflux data sources)

Installation

cd hummingbird-agent
pip install -e .

Usage

Local development (run)

The run command uses the config file for workflow lookup, data source registration, and token resolution – the same code path as serve. By default, results are printed to stdout (dry-run). Use --execute to post the result as a GitLab MR note.

There are two ways to select what to run:

Direct workflow selection (--workflow + --project + --event):

# Run a specific workflow on an MR, Podman sandbox, print to stdout
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}'

# Same but with K8s sandbox
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --context my-cluster/my-namespace

# Post the result as a GitLab MR note (also saves session to S3 if configured)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --execute

# Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --save-session /tmp/my-session

# Resume from a saved session with a follow-up question
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --resume-session /tmp/my-session \
    --message "Can you look at the clair-scan timeout more closely?"

# Chain: resume and save the new session for another round
hummingbird-agent run \
    --event-file event.json \
    --resume-session /tmp/my-session \
    --message "What layer hash failed?" \
    --save-session /tmp/my-session-2

Event-file replay (--event-file): routes the event through the same project-index lookup as serve, but skips rate limiting and status filtering:

# Replay a real webhook event, dry-run with Podman
hummingbird-agent run \
    --event-file event.json

# Replay with K8s sandbox and post notes
hummingbird-agent run \
    --event-file event.json \
    --context my-cluster/my-namespace \
    --execute

Production (serve)

# Poll SQS queue for events (pool sandbox by default, posts results as MR notes)
CONFIG_PATH=config.yml hummingbird-agent serve

Requires CONFIG_PATH pointing to a config file with settings.sqs_queue_url and settings.sqs_fifo_queue_url set. The config file defines which workflows run on which projects, with per-project limits and data source token mappings. Handles SIGTERM/SIGINT for graceful shutdown. A background router thread forwards events from the standard queue to the FIFO; the main thread consumes the FIFO with a semaphore-gated thread pool (controlled by settings.max_concurrent_agents) so excess messages stay in SQS for other instances.

Config hot-reload: In serve mode a background thread polls the config file for changes (every 5 seconds by default). When the file changes, the new config is validated and atomically swapped in – subsequent event dispatches use the updated config. If the new config is invalid, the previous config is kept and a warning is logged. No restart required for config changes.

The serve command also accepts --sandbox, --context, and --namespace for local development with a different sandbox backend (e.g. Podman).

CLI Options

run subcommand:

Option Description Default
--event Inline event JSON string (mutually exclusive with --event-file) -
--event-file Read event from a JSON file (mutually exclusive with --event) -
--workflow Workflow name from config file (requires --project) -
--project GitLab project path (requires --workflow) -
--execute Post result as GitLab MR note and save session to S3 -
--save-session Save session artifacts to this directory -
--resume-session Resume from a saved session directory (requires --message) -
--message Follow-up message for session resumption (requires --resume-session) -
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) podman
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

serve subcommand:

Option Description Default
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) k8spool
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

Configuration

Config file

Both run and serve use a single YAML config file. Set the path via CONFIG_PATH (default: config.example.yaml). The config has two sections: settings for operational parameters, and workflows for workflow definitions. Most settings have defaults; configure the sandbox image and namespace as needed for the selected backend. Sandbox options for settings.sandbox are documented by the shared Hummingbird Sandbox library.

The dedicated agent execution image is not yet published or selected in runtime configuration. See sandbox images for build and rollout details.

settings:
  gitlab_url: https://gitlab.com # GitLab instance URL
  sandbox: {}
  max_concurrent_agents: 4 # max concurrent workflows (serve)
  sqs_queue_url: "" # standard SQS ingress queue URL (serve)
  sqs_fifo_queue_url: "" # FIFO work queue URL (serve)
  s3_session_bucket: "" # S3 bucket for session persistence
  model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
  model_regions: # model prefix -> GCP region
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global
  max_iterations: 30 # default iteration limit
  max_runs_per_mr: 5 # default per-MR rate limit
  internal_notes: true # default note visibility
  docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md
  source_url: https://gitlab.com/org/group/project
  slack_url: https://slack.example.com/archives/C0123456789
  slack_label: "#my-channel"

workflows:
  code-review:
    trigger: merge_request # auto-trigger on MR events
    description: Performs AI-powered code review
    sandbox: k8spool # per-workflow sandbox override (optional)
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md
    trigger_rules: # ordered rule chain, first match wins
      - user_regex: "renovate\\[bot\\]" # deny bot MRs (regex fullmatch)
        action: deny
      - branch_regex: "chore/.*" # deny maintenance branches
        action: deny
      - action: allow # allow everything else
    prompt: workflows/code-review.md
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
    max_iterations: 15
    max_inline_size: 200000 # keep full diffs in context
    context_limit: 500000 # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows
    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO
    projects:
      org/group/project: {}

  analyze-failures:
    trigger: pipeline # auto-trigger on failed pipelines
    description: Investigates CI/CD pipeline failures
    sandbox: kubevirtpool # use VM sandbox for heavier workloads
    auto_resolve_on_push: true # resolve threads when a new SHA is pushed
    auto_resolve_on_success: true # resolve threads when pipeline succeeds
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md
    prompt: workflows/analyze-failures.md # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
    max_iterations: 50 # per-workflow iteration override

    data_sources: # model tool tokens (read-only)
      gitlab:
        token_env: GITLAB_TOKEN_RO # env var name, not the token
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens: # per-project model token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO
        action_tokens: # per-workflow write tokens
          gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS

With --workflow/--project, the workflow and project are looked up directly in the config. With --event-file, the project is extracted from the event body and matched against the project index to find applicable workflows. Pool overrides in this example require the corresponding pool to be provisioned and selected with --sandbox at startup. Only the CLI-selected pool is initialized, so these two pool overrides cannot both be used in one service. Without a matching pool, a workflow can fall back to a direct sandbox instead.

Discussion threads

All workflow results are posted as discussion threads: a placeholder note starts the discussion and the full result is posted as a reply. The placeholder is never edited, so email notifications include the actual result text. For slash commands, the result is posted as a reply in the triggering discussion.

Auto-resolve

Workflows can opt in to automatic resolution of their discussion threads:

  • auto_resolve_on_push (default false): When a new commit is pushed to the MR (i.e. a merge_request event with action: update), all agent discussion threads for the workflow whose SHA differs from the new HEAD are resolved. This clears stale failure analyses when the developer pushes a fix.
  • auto_resolve_on_success (default false): When the head pipeline succeeds, all agent discussion threads for the workflow on that MR are resolved (regardless of SHA). This handles pipeline reruns on the same SHA where a transient failure is now green.

Both flags are independent and can be combined. Resolution runs before rate limit checks, so threads are resolved even if the workflow’s per-MR run limit has been reached.

Workflows can enable Anthropic’s built-in web search server tool by listing web_search as a data source:

workflows:
  renovate-babysit:
    model: claude-sonnet-4-20250514
    data_sources:
      gitlab: {}
      web_search: {}
    # ...

When web_search is present in data_sources, the web_search_20250305 server tool is included in API requests to Claude. The model decides autonomously when to search. Search execution happens server-side (no client-side tool dispatch), and results appear as server_tool_use / web_search_tool_result content blocks in the response. These blocks are preserved through session save/resume.

Unlike other data sources, web_search has no configuration options and does not register any orchestrator-side tools – it is handled entirely by the model provider.

If the API returns a pause_turn stop reason (server-side search loop hit its iteration limit), the adapter automatically re-sends the conversation to continue, up to 5 continuations per generate() call.

Web search is only supported with Claude models. The setting is ignored for Gemini.

The first agent-authored note (placeholder) includes a footer with links to documentation, source code, the Slack channel, the workflow prompt, and a continuation prompt. These links are configured via global settings:

Setting Description
settings.docs_url Link to agent documentation
settings.source_url Link to agent source repository
settings.slack_url Link to support Slack channel
settings.slack_label Display text for Slack link (default: “Slack”)

Per-workflow, set workflow_url to link to the workflow’s prompt file. The footer stays on the placeholder and the result is a separate reply.

Token separation

Tokens are split into three tiers that never mix:

  • Model tool tokens (in YAML data_sources / tokens): read-only tokens passed to the LLM’s tool calls. Declared in the config file as env var names. These are user-defined and resolved at runtime from the referenced env vars. Create as project access tokens with Reporter role and read_api scope. Reporter is the minimum role required to see internal (confidential) notes in the discussions tool.
  • Workflow action tokens (in YAML action_tokens, per-project): write tokens used by the orchestrator for per-workflow GitLab writes (notes, thread resolution). Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the project config as env var names. Create as project access tokens with Developer role and api scope. These tokens are never exposed to the LLM and never enter the ToolRegistry. Naming convention: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
  • Orchestrator tokens (ORCHESTRATOR_* env vars, NOT in YAML): tokens used by the runner for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Create as project access tokens with Developer role and read_api scope (or api for infrastructure notes). Developer role is required because the discussions tool trust-filters notes by author access level (>= Developer); if the orchestrator bot has only Reporter access, its own notes are redacted. Resolved by convention: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (fallback). The ORCHESTRATOR_ prefix makes these impossible to confuse with model tokens.

Environment Variables

The agent reads only secrets and authentication from environment variables. All operational settings come from the config file’s settings section.

Variable Required Default Description
CONFIG_PATH no config.example.yaml Path to config YAML
GOOGLE_API_KEY yes* - Gemini API key; Gemini direct mode only (not for Claude)
GOOGLE_CLOUD_PROJECT yes* - GCP project ID (Vertex AI mode); required for Claude models
ORCHESTRATOR_GITLAB_TOKEN serve - Orchestrator GitLab token (global fallback)
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> no - Per-project orchestrator token
SENTRY_DSN no - Sentry DSN for error tracking

*One of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT is required. Claude models require Vertex AI (GOOGLE_CLOUD_PROJECT); GOOGLE_API_KEY is for Gemini direct mode only.

Model tool tokens (e.g. GITLAB_TOKEN_RO, GITLAB_TOKEN_CONTAINERS_RO) and data source credentials (e.g. KONFLUX_CLUSTER_URL, KUBECONFIG) are referenced by name in the config file’s data_sources and tokens sections. They are not listed in the table above because their names are user-defined.

Sandbox Security

Sandbox isolation, security constraints, permissions, and backend behavior are documented in the shared Hummingbird Sandbox library. The agent continues to own workflow selection, CLI options, session persistence, and the decision to acquire or reuse a sandbox.

Data Sources

Data sources are registered as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (or auto-spills large responses to the sandbox).

GitLab

Tool Description
gitlab_get_mr_details MR metadata (title, author, state, SHA, labels)
gitlab_get_mr_unified_diff Complete unified diff in patch format
gitlab_get_mr_diff Per-file structured change data
gitlab_get_mr_commits List of commits in a merge request
gitlab_get_mr_discussions Discussion threads with redacted agent transcripts and trust-filtered comments
gitlab_get_commit_statuses CI/CD pipeline statuses for a commit
gitlab_get_file_at_ref Raw file content at a git ref
gitlab_get_repo_archive Repository tar.gz (binary, auto-spilled)
gitlab_get_job_log CI job trace output (ANSI codes stripped)

Konflux

Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and Kubearchive (for completed resources), with deduplication by UID.

Tool Description
konflux_list_pipelineruns PipelineRuns for a commit SHA; optional status allowlist filter
konflux_list_taskruns TaskRuns by commit SHA or parent PipelineRun name; optional status filter
konflux_list_pods Pods for a commit SHA; optional status allowlist filter
konflux_get_pod Full pod resource (spec, status, conditions, container statuses)
konflux_get_pod_log Pod logs; optional container param, fetches all containers when omitted

Status filtering: All three list tools accept an optional status parameter — an allowlist of status.conditions[0].status values ("True" = success, "False" = failure, "Unknown" = in-progress). Only items matching the allowlist are written to the sandbox file. Response metadata always includes counts (all items seen, by reason), included, and excluded — the agent knows the full reason breakdown even when fetching only failures. Items without conditions always pass.

Per-PLR TaskRun fetching: konflux_list_taskruns accepts a pipelinerun parameter. When set, it uses tekton.dev/pipelineRun=<name> as a server-side label selector, returning only the ~15-21 TaskRuns for that PipelineRun (~600 KB) instead of all TaskRuns for the commit (~530 MB for 340 components).

Recommended pattern for analyze-failures: Fetch failing PLRs and TaskRuns together in one batch with status=["False"]. This captures every failure reason (Failed, Cancelled, PipelineRunTimeout, StepFailed, etc.) without enumerating them. Use response metadata counts for the summary. Use per-PLR fetch only when deeper investigation is needed.

Response metadata includes konflux_ui base URL for building reviewer-facing links.

Testing Farm

Tool Description
tf_get_results JUnit XML results for a request ID
tf_get_test_log Individual test log by URL (restricted to Testing Farm artifact URLs)
tf_get_request_status Request state, queue/run times

Response metadata includes artifacts_base URL for building artifact links.

Workflow System

Workflows are .md files whose content becomes the LLM system prompt verbatim. Workflow metadata (action, model, max_iterations, enabled data sources, project allowlists) is defined in the config file. The .md file is pure system prompt text.

Available workflows:

  • analyze-failures.md - Investigates CI/CD pipeline failures by fetching MR details, identifying failed pipelines via commit statuses, retrieving PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm, and producing a grouped root-cause report with reviewer-facing URLs.
  • code-review.md - Performs AI-powered code review by fetching MR details, the unified diff, and prior discussion threads in parallel, then attempting to load per-project rules from workflows/repo-rules/ (a mandatory step – rules take precedence over generic standards when present) and producing structured feedback with severity ratings, code examples, and actionable suggestions. On follow-up reviews (after SHA updates), the agent sees its own previous findings, developer responses, and resolved threads – avoiding duplicate findings and respecting developer explanations. Uses elevated max_inline_size (200 KB) and context_limit (500K tokens) to keep the full diff in context.
  • renovate-babysit.md - Triages a single Renovate-authored MR triggered by a successful pipeline. Classifies the MR as safe or risky based on diff scope, upstream dependency changes (from MR description, web search), and local codebase impact (via gitlab_get_repo_archive + grep). Posts a structured note with verdict, upstream change summary, and suggested actions for risky MRs. Requires a Claude model with web_search data source.

Token Budget Management

Prompt caching, cached token accounting, and provider-specific cost estimates are implemented in hummingbird-ai.

The agent uses a dual-limit approach instead of a cumulative token budget:

  • Iteration limit (settings.max_iterations, default 30, overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up prompt is injected at 80% (SOFT_ITERATION_RATIO).
  • Context limit (CONTEXT_LIMIT, default 60,000 tokens, overridable per-workflow via context_limit) - Per-call input token ceiling. When exceeded, a wrap-up prompt forces the model to finalize.

Large outputs are automatically redirected to sandbox files to keep the LLM context small. The spill threshold defaults to 4 KB (MAX_INLINE_SIZE) but can be overridden per-workflow via max_inline_size in the config:

  • sandbox_exec - stdout/stderr exceeding the threshold saved to /tmp/_out/{N}.txt; model receives a preview (head + tail) with file path
  • Data sources - text exceeding the threshold saved to /tmp/_out/{name}_{N}.txt with preview; binary data saved to .bin
  • fetch_to_sandbox - always writes to the caller-specified path; returns metadata only

All tuning constants are centralized in config.py:

Constant Default Purpose
DEFAULT_MAX_ITERATIONS 30 Hard iteration cap
SOFT_ITERATION_RATIO 0.8 Inject wrap-up at this fraction
CONTEXT_LIMIT 60,000 Per-call input token ceiling
OUTPUT_PREVIEW_BYTES 4,096 Preview size for spilled outputs
OUTPUT_TAIL_BYTES 512 Extra tail appended to previews
MAX_INLINE_SIZE 4,096 Max inline size for data source responses

Evaluation Framework

The agent includes an evaluation framework for scientific measurement of prompt and model quality. See Hummingbird Agent Evals for the full guide covering the shared hummingbird_agent.eval library, existing evaluations, and how to write new ones.

Development

See the main README for development workflows.

make hummingbird-agent/setup  # Install dependencies
make check                    # Lint code (ruff)
make fmt                      # Format code
make test                     # Run unit tests
make coverage                 # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.20.1 - Hummingbird Agent Model Loop

How the Hummingbird agent runs conversation turns, calls tools, and integrates user replies.

For the architectural design rationale behind the loop (why full history replay, why iteration-based budgets, why nudge via system prompt), see Agent Design – Section 6. For operational usage documentation, see Hummingbird Agent. Provider adapters, wire formats, canonical message conversion, and caching are documented in Hummingbird AI.

What gets sent to the model

Every call to the model API sends three things:

1. System instruction

A single text string, rebuilt every iteration. The adapter maps it to the active provider’s request format. It is composed of:

BASE_SYSTEM_PROMPT            # agent.py: sandbox rules, tool usage tips
+ tool_notes                  # per-data-source notes from ToolRegistry
+ workflow_prompt             # full content of e.g. workflows/analyze-failures.md

Near the iteration or context limit, a warning suffix is appended to the system prompt for that call. The suffix escalates through three levels:

  • ITERATION_WARNING / CONTEXT_WARNING – soft “start wrapping up” at 80% of the iteration or context limit. Tools remain available.
  • FINAL_TURN_WARNING – hard stop on the absolute last iteration or after the context limit is exceeded. Combined with tool-calling disabled for that request to mechanically prevent further tool calls.
  • EMPTY_RESPONSE_NUDGE – appended when the model returns an empty response (no text, no tool calls). Tools remain available.

2. contents

A list of messages representing the full conversation history. It grows every iteration. Provider-specific content conversion is handled by the adapter; see Hummingbird AI.

3. tools

Tool definitions, static across all iterations. Adapters translate the shared tool definition into the provider’s schema.

4. Tool availability (conditional)

On the final turn (last iteration or after context limit exceeded), the request disables tools to mechanically prevent function calls. The active adapter applies the provider-specific setting.

This is only sent when FINAL_TURN_WARNING is active. All other iterations omit this constraint, allowing the model to choose freely between text and tools.

The agent loop, turn by turn

The loop in run_agent_loop runs up to max_iterations times. Each iteration is one round-trip to the model.

Initialization

system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt
tool_defs     = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, <data sources...>]
contents      = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}]

Iteration 1

-> Send: system_instruction + contents (1 item) + tool_defs
<- Response: tool_call(gitlab_get_mr_details, {project: "org/repo", iid: 42})

contents.append(model: response.raw_content)  # opaque provider content
  execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"}
contents.append(tool: model.make_tool_responses([(name, result)]))

Iteration 2

-> Send: system_instruction + contents (3 items) + tool_defs
<- Response: tool_call(fetch_batch_to_sandbox, {requests: [...]})

contents.append(model: ...)
  execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]}
contents.append(tool: ...)

Iteration 3

-> Send: system_instruction + contents (5 items) + tool_defs
<- Response: tool_call(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"})

contents.append(model: ...)
  execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"}
contents.append(tool: ...)

Iteration N (final)

-> Send: system_instruction + contents (2N-1 items) + tool_defs
<- Response: text("## Konflux Failure Analysis\n\n...")    // text + no tool calls = DONE

contents.append(model: {text: "## Konflux Failure Analysis..."})
BREAK -- return (text, usage, transcript, contents)

Response handling and termination

Each iteration, the model’s response can contain text, tool calls, both, or neither. The agent handles each case:

Response Action
Text only Final response. Save text, break. (Happy path.)
Text + tool calls Save text as last_text fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting.
Tool calls only Execute the tool calls, continue loop.
Neither Empty response – retry up to MAX_EMPTY_RETRIES (2) times. Each retry nudges the model via EMPTY_RESPONSE_NUDGE appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries.

The text + tool calls case is worth noting: the model’s text is not returned to the user immediately. It is stored as last_text (a fallback in case the loop terminates later without a clean final text, e.g. by hitting max iterations). The provider response is appended to contents unchanged; its wire format is documented in Hummingbird AI.

The loop terminates on:

  1. Text only – the model produced a final response.
  2. Max iterations reached – the final iteration uses FINAL_TURN_WARNING and disables tools through the adapter to force text output. Falls back to whatever last_text was seen, or "[Agent did not produce a final response]".
  3. Context limit exceeded – input_tokens >= CONTEXT_LIMIT sets context_exceeded, making the next iteration final (same as #2). A soft warning (CONTEXT_WARNING) fires earlier at 80% of the limit.
  4. Fatal model error – a provider error after shared retry handling is exhausted. Error classification and retries are described in Hummingbird AI.
  5. Empty responses exhausted – MAX_EMPTY_RETRIES (2) nudge retries failed.
  6. Unexpected error – any exception not handled by the retry mechanism (e.g. malformed API response) breaks the loop and returns partial results. Accumulated contents, transcript, and sandbox are preserved and the session is saved normally.

Budget escalation

Both iterations and context size use the same two-tier pattern:

Soft warning (80%) Hard stop (100%)
Iterations ITERATION_WARNING at 80% of max_iterations FINAL_TURN_WARNING on last iteration
Context CONTEXT_WARNING at 80% of CONTEXT_LIMIT FINAL_TURN_WARNING on next iteration after exceeding CONTEXT_LIMIT

Soft warnings are advisory (“start wrapping up”) and keep tools available. The hard stop disables tools through the active adapter to mechanically prevent further tool calls, ensuring the model produces text.

The contents list in detail

The agent treats each entry in contents as opaque adapter content during execution. Provider representations and canonical conversions are documented in Hummingbird AI.

model.make_user_content() constructs the initial event turn and model.make_tool_responses() appends results for dispatched calls. The agent stores the provider response as raw_content and appends it unchanged.

Typical 8-iteration conversation shape

contents[0]  = user:  {"project":"org/repo","iid":42,...}          # initial event
contents[1]  = model: tool_call(gitlab_get_mr_details)             # iter 1
contents[2]  = tool:  result(gitlab_get_mr_details)                # iter 1
contents[3]  = model: tool_call(fetch_batch_to_sandbox)            # iter 2
contents[4]  = tool:  result(fetch_batch_to_sandbox)               # iter 2
contents[5]  = model: tool_call(sandbox_exec)                      # iter 3
contents[6]  = tool:  result(sandbox_exec)                         # iter 3
...
contents[13] = model: tool_call(sandbox_exec)                      # iter 7
contents[14] = tool:  result(sandbox_exec)                         # iter 7
contents[15] = model: text("## Konflux Failure Analysis...")       # iter 8 (final)

This entire list is persisted as canonical messages in context.json; the adapter converts to and from its provider format at the persistence boundary. See Hummingbird AI for details.

How tools are called

When the model’s response contains tool calls, _execute_tool_calls iterates them sequentially:

tool_registry.execute(ToolCall(name, args))
  match name:
     "sandbox_exec"           -> runs sh -c in container -> {exit_code, stdout, stderr}
     "fetch_to_sandbox"       -> calls data source func -> writes to sandbox file -> {saved_to, bytes}
     "fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call
     any data source name     -> calls func directly -> returns inline or auto-spills large output

Large output handling

When stdout or a data source response exceeds 4KB, it is automatically saved to a sandbox file (/tmp/_out/N.txt) and only a preview (head + tail) is returned to the model. The model gets the file path and can use sandbox_exec with jq/grep/head to process it. This keeps the context window manageable.

How a user reply integrates (session resumption)

When a user replies to an agent note on GitLab, the orchestrator resumes the conversation by restoring the previous state and appending the reply.

What gets restored from S3

  • context.json – the persisted conversation in canonical form; the active adapter converts it to the in-memory representation before the loop runs
  • sandbox.tar.gz – /tmp/_out/ files (PipelineRun JSONs, test logs, jq output, etc.) restored into the new sandbox

Canonical storage allows sessions to resume with another provider; conversion details are in Hummingbird AI.

The resumed contents list

// Restored from context.json (previous run)
contents[0]  = user:  {"project":"org/repo","iid":42,...}           # original event
contents[1]  = model: tool_call(gitlab_get_mr_details)              # iter 1
contents[2]  = tool:  result(gitlab_get_mr_details)                 # iter 1
...                                                                  # all prior turns
contents[15] = model: text("## Konflux Failure Analysis...")        # previous final text

// NEW: user reply appended
contents[16] = user:  "Can you look at the clair-scan timeout more closely?"

The agent loop continues

ITERATION 1 (resumed):
  -> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs
  <- Response: tool_call(sandbox_exec, {command: "grep -i timeout /tmp/data/..."})
                                                       ^ using restored sandbox files
  contents[17] = model: tool_call(sandbox_exec)
  contents[18] = tool:  result(sandbox_exec)

ITERATION 2 (resumed):
  <- Response: text("The clair-scan timeout is caused by...")
  contents[19] = model: text("The clair-scan timeout is caused by...")
  BREAK

Cold start vs resumed – key differences

Aspect Cold start Resumed
System prompt BASE + tool_notes + workflow BASE + tool_notes + workflow + CONTINUATION_PROMPT
First user message {"project":..., "iid":...} {"project":..., "iid":...} (restored)
Conversation history Empty (just the event) Full prior conversation
Latest user message – "Can you look at the clair-scan timeout?"
Sandbox files Empty Restored from archive
Tool definitions Same Same
Session format Adapter content in memory Canonical on disk; adapter content after load

The CONTINUATION_PROMPT

Without this, the workflow prompt (e.g. analyze-failures.md) tells the model to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1… The model might try to re-run the entire analysis. The continuation prompt overrides this:

## Continuation

This is a follow-up to a previous conversation. The conversation history
contains your prior analysis and tool calls. The user is replying with a
question or request about your previous work.

**Do NOT re-run the full workflow from scratch.** Instead:
- Respond directly to the user's question
- Use your tools to investigate further if needed (logs, data are still
  available in the sandbox)
- Reference your previous findings where relevant
- Keep your response focused on what the user asked

The model now understands: “I already did the analysis (it’s all in the conversation history). The user has a specific question. Let me answer it.”

Iteration and token budget for resumed sessions

The resumed session gets a full fresh iteration budget. run_workflow sets max_iterations from the workflow config (e.g. 50 for analyze-failures), and the loop counter starts at 0 regardless of how many iterations the previous session used. The model can make as many tool calls as it needs to answer the follow-up.

The practical constraint is the context window, not iterations. The restored contents can be large – a typical 8-iteration cold start uses ~50-100K input tokens. Since the model re-counts the full history every iteration, the resumed session starts at roughly the token count where the previous session ended, plus the new user message. Each new tool call adds more. Token counts logged by the agent are billed tokens (cumulative across API calls; each call re-sends the full history, so these overlap).

The existing CONTEXT_LIMIT check still applies: at 80% of the limit, a soft CONTEXT_WARNING tells the model to start wrapping up. If the limit is exceeded, the next iteration becomes final (FINAL_TURN_WARNING with provider tools disabled by the adapter).

For the typical case (user asks one focused follow-up, agent does 1-3 tool calls), this is fine. Multi-turn deep dives will eventually hit the context limit, at which point the agent wraps up – and the user can start a fresh conversation if needed.

Provider behavior

Provider wire formats, canonical message conversion, prompt caching, server-side tools, and adapter continuations are owned by the hummingbird-ai library. The Hummingbird agent loop remains responsible for tool dispatch, iteration/context budgets, and session resumption.

4.5.20.2 - Hummingbird Agent Design

Architectural design document for the hummingbird-agent. Covers the reasoning behind every major design choice so that future changes can be made safely, without accidentally violating invariants that hold the system together.

For operational usage (CLI, config reference, deployment), see Hummingbird Agent. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.

1. Design Philosophy

Five principles shaped the agent’s architecture. Every component traces back to at least one of these.

Security by default. The agent processes untrusted merge requests. All LLM-driven commands run inside isolated sandbox containers with no network and no credentials. Orchestrator write tokens and model read tokens live in separate code paths that never cross. No cluster-admin is required.

Config over code. Investigation logic lives in markdown workflow files that become the LLM system prompt verbatim – changing the analysis strategy requires no code change and no redeployment. Operational settings (data sources, project allowlists, iteration limits) live in a YAML config file that is auditable and committable. Secrets are the only thing in environment variables.

Bounded cost. Every output path has a size cap. Large tool outputs are auto-spilled to sandbox files with only a preview returned to the model. Each session has both an iteration limit and a per-call context token limit with two-tier escalation (soft warning, then hard stop with tool disabling). The relationship between these constants is documented and centralized.

Partial over nothing. When some data is unavailable (expired pod logs, unreachable Konflux cluster, Testing Farm 404), the agent continues with whatever data it has and notes the gap in its output. A partial report is more valuable than a crash.

Model-agnostic agent loop. The agent loop (agent.py) does not inspect the internal structure of the contents list. It appends raw_content from model responses and make_tool_responses() output without looking inside. Each model backend owns its wire format. This makes it possible to add new model backends (Claude, GPT) without touching the agent loop.

2. System Architecture

2.1. Component overview

flowchart TB
    subgraph input [Event Sources]
        SQS["SQS Queue<br/>(gitlab::pipeline, gitlab::note)"]
        CLI["CLI<br/>(--event / --event-file)"]
    end

    subgraph orchestrator [Orchestrator Process]
        Events["events.py<br/>SQS consumer"]
        Runner["runner.py<br/>Event routing"]
        Agent["agent.py<br/>Model loop"]
        Tools["tools.py<br/>Tool registry"]
        Actions["actions.py<br/>GitLab notes"]
        Sessions["sessions.py<br/>Persistence"]
        WfConfig["workflow_config.py<br/>YAML config"]
    end

    subgraph ai [Shared AI Library]
        Models["hummingbird-ai<br/>Gemini / Anthropic adapters"]
    end

    subgraph sandbox [Sandbox Container]
        SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox<br/>jq, python3, yq"]
        SpillFiles["/tmp/data/_out/ spill files"]
        DataFiles["/tmp/data/ working files"]
    end

    subgraph dataSources [Data Source Modules]
        GL["gitlab.py"]
        KX["konflux.py"]
        TF["testing_farm.py"]
    end

    subgraph external [External APIs]
        GitLabAPI["GitLab API"]
        KonfluxAPI["K8s + Kubearchive"]
        TFAPI["Testing Farm API"]
    end

    subgraph storage [Storage]
        S3["S3 Sessions"]
        MRNote["GitLab MR Notes"]
    end

    SQS --> Events --> Runner
    CLI --> Runner
    Runner --> Agent
    Agent --> Tools
    Tools --> SB
    Tools --> dataSources
    dataSources --> external
    Agent --> Models
    Runner --> Actions --> MRNote
    Runner --> Sessions --> S3
    Runner --> WfConfig

2.2. Request lifecycle

A complete run proceeds through these stages:

  1. Event arrival. An SQS message (production) or CLI invocation (dev) provides a GitLab project path and MR IID.

  2. Config lookup. workflow_config.py maps the project to one or more workflows via the project_index. Each match yields a WorkflowConfig and ProjectEntry with data source declarations and per-project settings.

  3. Rate-limit check and SHA dedup. actions.scan_agent_threads() scans MR discussions in a single pass, parsing JSON session markers to determine the per-workflow thread count and whether the current SHA has already been reviewed. If the SHA was already reviewed, the workflow is skipped. If the thread count meets or exceeds max_runs_per_mr, the workflow is skipped. Slash commands and replies bypass this check.

  4. Placeholder note. actions.create_placeholder_note() posts a placeholder so the reviewer knows analysis is in progress. The note contains a JSON session marker (<!-- hummingbird-session: {"id":"UUID","wf":"name","sha":"abc"} -->) for rate limiting, SHA dedup, and session resumption.

  5. Sandbox start. sandbox.create_sandbox() starts a Podman container or K8s pod. /tmp/data/ is pre-created for the model’s use.

  6. Data source registration. data_sources.register_selected() resolves tokens and URLs from the config and registers tool functions on the ToolRegistry. Only data sources declared in the workflow config are registered.

  7. Agent loop. agent.run_agent_loop() runs the model loop: the workflow markdown becomes the system prompt, tool definitions are provided, and the model iterates calling tools and producing text until it emits a final response or hits a budget limit.

  8. Note update. The final text, wrapped with a session marker and reply prompt, replaces the placeholder note.

  9. Session save. Conversation history (contents), transcript, and sandbox archive are saved to S3 (production) or a local directory (dev).

  10. Sandbox cleanup. The container/pod is deleted. K8s pods also have a configurable activeDeadlineSeconds backstop (default 1800s) in case the orchestrator dies.

2.3. Architectural boundaries

The codebase is organized around four strict boundaries:

Runner (runner.py) is the orchestration layer. It owns event routing, the placeholder/update note lifecycle, session save/load, and sandbox lifecycle. It calls agent.run_agent_loop() but never reaches into the agent’s internals.

Agent (agent.py) is the model loop. It knows about the model interface, tool definitions, and the contents list, but nothing about GitLab, SQS, sessions, or actions. It returns (text, usage, transcript, contents) and is completely unaware of what happens with those values.

Tools (tools.py) bridge the agent and the sandbox/data-sources. The agent calls tool_registry.execute(tool_call) and gets a dict back. It never calls sandbox methods directly. This indirection is what enables auto-spill: the tool registry can transparently save large outputs to sandbox files and return previews.

Model providers are supplied by the shared hummingbird-ai library, which owns adapter construction, provider formats, retries, and AI usage accounting. The agent loop uses the library’s normalized adapter interface and treats conversation content as opaque.

3. Module Architecture

3.1. Dependency graph

flowchart TD
    main["__main__.py"]
    config_mod["config.py"]
    wf_config["workflow_config.py"]
    events_mod["events.py"]
    runner_mod["runner.py"]
    agent_mod["agent.py"]
    tools_mod["tools.py"]
    sandbox_mod["sandbox.py"]
    actions_mod["actions.py"]
    sessions_mod["sessions.py"]
    transcript_mod["transcript.py"]
    ds_init["data_sources/__init__.py"]
    ds_gitlab["data_sources/gitlab.py"]
    ds_konflux["data_sources/konflux.py"]
    ds_tf["data_sources/testing_farm.py"]
    config_watch_mod["config_watch.py"]
    ai_pkg["hummingbird-ai library"]

    main --> config_mod
    main --> config_watch_mod
    main --> wf_config
    main --> events_mod
    main --> runner_mod
    main --> sessions_mod
    main --> transcript_mod
    main --> actions_mod

    config_watch_mod --> config_mod
    config_watch_mod --> wf_config

    runner_mod --> actions_mod
    runner_mod --> agent_mod
    runner_mod --> config_mod
    runner_mod --> ds_init
    runner_mod --> sandbox_mod
    runner_mod --> sessions_mod
    runner_mod --> tools_mod
    runner_mod --> ai_pkg
    runner_mod --> wf_config
    runner_mod --> transcript_mod

    agent_mod --> config_mod
    agent_mod --> ai_pkg
    agent_mod --> tools_mod

    tools_mod --> config_mod
    tools_mod --> ai_pkg
    tools_mod --> sandbox_mod

    ds_init --> tools_mod
    ds_init --> wf_config
    ds_init --> ds_gitlab
    ds_init --> ds_konflux
    ds_init --> ds_tf

    ds_gitlab --> ai_pkg
    ds_konflux --> ai_pkg
    ds_tf --> ai_pkg

    sessions_mod --> sandbox_mod

3.2. Module responsibilities

Each module has a single, well-defined responsibility. The boundary rules below are invariants – violating them breaks the security model or the separation of concerns.

config.py – Runtime configuration and constants

  • Owns: Config dataclass and agent tuning constants (DEFAULT_MAX_ITERATIONS, CONTEXT_LIMIT, OUTPUT_PREVIEW_BYTES, etc.)
  • Boundary: Pure data. No I/O except reading env vars in Config.load(). No imports from other Hummingbird agent modules; model setup and pricing are owned by hummingbird-ai.
  • Invariant: All interdependent budget/spill constants must be defined here with their relationship documented in the comment block.

workflow_config.py – YAML config loader and token resolution

  • Owns: AgentConfig, WorkflowConfig (with trigger, description, trigger_rules, ignore_users, ignore_branches fields), TriggerRule, ProjectEntry, DataSourceConfig dataclasses; load(), validate_env(), evaluate_trigger_rules(), resolve_tool_token(), resolve_cluster_url(), get_prompt().
  • Boundary: Reads YAML from disk and env vars. Never instantiates network clients or model objects.
  • Invariant: resolve_tool_token() resolves model tool tokens only. It must never return orchestrator tokens. The resolution chain is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> "" (empty).

events.py – SQS consumer

  • Owns: decode_sns_message() (SNS envelope decoding), poll_loop() (blocking SQS consumer with concurrency control).
  • Boundary: Knows about SQS/SNS wire formats. Calls a generic EventHandler callback. Does not know about GitLab, workflows, or the agent.
  • Invariant: Failed messages are not deleted from SQS (they go to the DLQ after visibility timeout expires). Successful messages are deleted after handler returns.

config_watch.py – Background config hot-reload

  • Owns: ConfigHolder (thread-safe config pair with atomic swap), start_watcher() (daemon thread that polls config file mtime), _watch_loop().
  • Boundary: Knows about config.Config.load() and workflow_config.load(). Does not know about events, the agent, or any runtime state.
  • Invariant: On reload failure (parse error, missing env var), the previous config is kept and a warning is logged. The watcher never crashes the serve loop.

runner.py – Event routing and workflow execution

  • Owns: handle_event(), handle_pipeline(), handle_merge_request(), handle_note(), _execute_workflow(), _handle_reply(), run_workflow(), _acquire_sandbox(), _resolve_slash_workflow(), _format_help(), _is_ignored_user(), WorkflowRequest, WorkflowResult, SandboxOpts.
  • Boundary: Orchestrates everything: config lookup, rate limiting, placeholder notes, sandbox lifecycle, agent invocation, session save. This is the only module that touches both actions and agent.
  • Invariant: run_workflow() either lingers the sandbox (success) or cleans it up (exception).

agent.py – Model-agnostic tool-calling loop

  • Owns: run_agent_loop(), system prompt constants (BASE_SYSTEM_PROMPT, CONTINUATION_PROMPT, warning/nudge strings), budget logic.
  • Boundary: Uses the ModelAdapter interface from hummingbird-ai and tools (for execute()). Does not know about GitLab, SQS, sessions, or actions, or which backend is in use.
  • Invariant: The contents list is treated as opaque. Items are appended via response.raw_content and model.make_tool_responses(). The agent never inspects, modifies, or deletes items in the list (except for empty response retries, which pop() the last appended item before the model has seen any tool results for it).

tools.py – Tool registry

  • Owns: ToolRegistry class with sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox built-in tools; data source registration and dispatch; auto-spill logic.
  • Boundary: Owns the sandbox reference and all tool execution. The agent never touches the sandbox directly.
  • Invariant: _spill_counter is shared across all spill paths (sandbox exec stdout, sandbox exec stderr, data source auto-spill) to prevent filename collisions in /tmp/data/_out/.

sandbox.py – Sandbox backends

  • Owns: Sandbox protocol, PodmanSandbox, K8sSandbox, K8sPoolSandbox, SandboxPool, KubeVirtSandbox, VmiPool, KubeVirtPoolSandbox, create_sandbox(), ExecResult.
  • Boundary: Knows about container/pod/VM lifecycle and command execution. Does not know about tools, models, or the agent.
  • Invariant: All backends must implement the same 8-method protocol (start, exec, write_file, stream_to_file, stream_exec, read_file_iter, cleanup, linger). exec() accepts optional stdin_data for piping raw bytes. All must pre-create /tmp/data/ in start(). All must use stdin piping for write_file() (never host volume mounts).

actions.py – GitLab note lifecycle

  • Owns: Orchestrator token resolution (resolve_orchestrator_token()), workflow action token client creation (_make_client()), bot user ID collection (collect_workflow_bot_ids()), note CRUD, JSON marker parsing (marker_tag, parse_marker), scan_agent_threads (per-workflow rate limiting + SHA dedup), post_simple_reply, member access checks, session-for-reply lookup, AgentThreadInfo, SessionRef.
  • Boundary: Functions are split by token tier:
    • Orchestrator token (operational reads + infrastructure notes, use _get_client): check_member_access, get_latest_push_author, get_head_pipeline_id, is_first_note_in_discussion, reply_access_denied, notify_rate_limit, post_simple_reply.
    • Workflow action token (per-workflow-attributable writes, take explicit token: str): create_placeholder_discussion, post_discussion_note, resolve_agent_threads, scan_agent_threads. Never touches model tool tokens, the tool registry, or the agent.
  • Invariant: Orchestrator tokens are resolved from ORCHESTRATOR_* env vars only. Workflow action tokens are resolved from action_tokens in YAML and passed as explicit parameters. Neither tier is exposed to the model.

sessions.py – Session persistence

  • Owns: archive_sandbox(), save_local(), save_s3(), load_s3(), load_local(), restore_sandbox(), SessionData.
  • Boundary: Knows about sandbox (for archiving) and S3/filesystem (for storage). Does not know about the agent, tools, or models. Load/save uses canonical envelopes with format_version: 1; unversioned legacy sessions are not supported.
  • Invariant: The sandbox archive is never extracted on the orchestrator. It is created inside the sandbox (tar czf), streamed out via read_file_iter(), and restored inside a new sandbox via stream_exec("tar xzf -").

transcript.py – Transcript rendering

  • Owns: render_markdown(), per-tool rendering functions, truncation.
  • Boundary: Pure transformation from TranscriptEntry list to markdown. No I/O, no side effects.

data_sources/__init__.py – Data source registration

  • Owns: register_selected() – resolves tokens/URLs from config and calls each data source’s register() function.
  • Boundary: Bridges workflow_config (for token/URL resolution) and tools (for registration). Only registers data sources declared in the workflow config.

data_sources/gitlab.py, konflux.py, testing_farm.py

  • Own: register() function that creates tool definitions and closures over credentials, and registers them on the ToolRegistry.
  • Boundary: Each module talks to one external API. Credentials are captured at registration time via closure, never stored globally.

4. Security Model

The agent runs in a shared OpenShift cluster without cluster-admin access, processing merge requests from repositories where external contributors can submit code. The security model addresses two threat vectors: (1) the LLM executing arbitrary commands chosen by an attacker-controlled MR, and (2) credential leakage between the model, the sandbox, and orchestrator actions.

4.1. Sandbox isolation

All commands generated by the LLM run inside an ephemeral container, never on the orchestrator host. The sandbox has no credentials, no network, and no visibility into the orchestrator.

flowchart LR
    subgraph orchestrator [Orchestrator]
        Agent["Agent Loop"]
        Creds["Secrets<br/>(tokens, kubeconfig)"]
    end
    subgraph sandbox [Sandbox Container]
        Shell["sh -c commands"]
        Files["/tmp/data/<br/>(incl. _out/ spill dir)"]
    end
    Agent -->|"stdin pipe<br/>(write_file)"| sandbox
    Agent -->|"exec command"| sandbox
    sandbox -->|"stdout/stderr"| Agent
    sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
    sandbox -.-x|"NO access"| Creds

Podman (local development):

  • --network=none – complete network isolation; curl, wget, pip install all fail
  • --user 65532 – fixed non-root UID; no privilege escalation
  • No host volume mounts – data enters only via stdin piping through write_file()

Kubernetes (production):

Every field in the pod manifest is set explicitly for portability to vanilla Kubernetes with Pod Security Admission (restricted level), not just reliance on OpenShift’s restricted-v2 SCC admission:

  • automountServiceAccountToken: false – no K8s API access from sandbox
  • runAsNonRoot: true – enforced at pod level
  • seccompProfile: RuntimeDefault – required by restricted level
  • allowPrivilegeEscalation: false – container level
  • capabilities.drop: ["ALL"] – container level
  • activeDeadlineSeconds: 1800 – pod self-terminates after 30 minutes even if the orchestrator crashes; prevents orphaned pods
  • restartPolicy: Never – pod does not restart on failure
  • NetworkPolicy on the sandbox namespace blocks all egress from all pods in the namespace (podSelector: {}, egress: [])

The pod does not set runAsUser explicitly. On OpenShift, the SCC assigns a UID from the namespace range. On vanilla K8s, the image’s USER directive is used.

4.2. Credential separation (four-tier token model)

Tokens are split into four tiers with strict code-path separation:

flowchart TB
    subgraph tier1 [Tier 1: Orchestrator Tokens]
        OT["ORCHESTRATOR_GITLAB_TOKEN_*<br/>Operational reads<br/>Env vars only, never in YAML"]
    end
    subgraph tier2 [Tier 2: Workflow Action Tokens]
        AT["action_tokens in YAML<br/>Per-workflow bot identity<br/>Write scope (api)"]
    end
    subgraph tier3 [Tier 3: Model Tool Tokens]
        MT["GITLAB_TOKEN_RO, etc.<br/>Read scope (read_api)<br/>Env var NAMES in YAML"]
    end
    subgraph tier4 [Tier 4: Sandbox]
        SB["Zero credentials<br/>Zero network<br/>Zero SA token"]
    end

    OT -->|"used by"| OpsReads["actions.py<br/>(member access, push author,<br/>rate-limit notices)"]
    AT -->|"used by"| WfWrites["actions.py<br/>(notes, thread resolution)"]
    MT -->|"used by"| DataSources["data_sources/<br/>(GitLab, Konflux, TF)"]
    SB -->|"used by"| SandboxExec["sandbox_exec<br/>(sh -c commands)"]

    OpsReads -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| SandboxExec
    DataSources -.-x|"NEVER"| OpsReads
    DataSources -.-x|"NEVER"| WfWrites

Tier 1 – Orchestrator tokens. Tokens used by actions.py for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Resolved by convention from env vars: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (global fallback). The ORCHESTRATOR_ prefix is a structural safeguard – these env var names can never appear in the YAML config’s data_sources, tokens, or action_tokens sections. These tokens can be scoped to read_api since all per-workflow writes moved to Tier 2.

Tier 2 – Workflow action tokens. Per-workflow bot identity tokens (api scope) used by actions.py for all workflow-attributable GitLab writes: placeholder notes, discussion notes, and thread resolution. Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the YAML config as env var names at the project level:

projects:
  redhat/hummingbird/containers:
    action_tokens:
      gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS

Resolved by resolve_action_token(project_entry, ds_name) which reads the env var name from action_tokens and resolves it from os.environ. These tokens are passed as explicit token: str parameters to write functions in actions.py. They NEVER enter the ToolRegistry or model tool calls.

At startup, collect_workflow_bot_ids() authenticates each unique workflow token to build a set of all workflow bot user IDs. This set is used by handle_note for self-filtering (skipping notes from any workflow bot) and by find_session_for_reply for session marker scanning.

The naming convention groups tokens alphabetically: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.

Tier 3 – Model tool tokens. Read-only tokens (read_api scope for GitLab) used by data source modules during tool execution. Declared in the YAML config as env var names (not values):

data_sources:
  gitlab:
    token_env: GITLAB_TOKEN_RO # name of the env var

Per-project overrides are possible:

projects:
  redhat/hummingbird/containers:
    tokens:
      gitlab: GITLAB_TOKEN_CONTAINERS_RO # overrides token_env for this project

The resolution chain in resolve_tool_token() is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""

Storing env var names (not values) in YAML means the config file is safe to commit and audit. Actual secret values live in environment variables, injected via K8s Secrets at deployment time.

Tier 4 – Sandbox. The sandbox container has zero credentials, zero network access, and automountServiceAccountToken: false (no K8s API access). Data enters the sandbox only via write_file() (stdin piping). The model cannot instruct the sandbox to reach external APIs – it must use the orchestrator’s data source tools.

4.3. Namespace separation

Sandbox pods run in a dedicated namespace (hummingbird--agent-sandbox), separate from the orchestrator namespace (hummingbird--internal). This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens.

RBAC setup:

The orchestrator’s ServiceAccount gets a Role in the sandbox namespace (not its own namespace) granting only:

  • pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
  • pods/exec: create – command execution via kubectl exec

No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator needs only namespace-scoped permissions, so it works with standard OpenShift RBAC without requesting cluster-admin.

Konflux data is fetched via bearer token from kubeconfig credentials, not from inside the cluster. The orchestrator’s ServiceAccount does not need access to Konflux namespaces.

NetworkPolicy:

A deny-all-egress NetworkPolicy in the sandbox namespace uses podSelector: {} to match all pods and sets egress: []. The sandbox cannot reach the internet, the K8s API, or other pods in the cluster.

4.4. Security invariants

These must hold for the security model to be effective. Any change that violates one of these is a security regression:

  1. Orchestrator tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are resolved in actions.py only.

  2. Workflow action tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are passed as explicit token: str parameters within actions.py and runner.py only.

  3. Model tool tokens must NEVER flow into actions.py. They are resolved in workflow_config.py and consumed in data_sources/.

  4. The sandbox must NEVER have network access or credentials. No host mounts, no SA token, no env vars with secrets.

  5. The sandbox archive is NEVER extracted on the orchestrator. It is created inside one sandbox and restored inside another. The orchestrator only transports the bytes.

  6. No cluster-admin required. Only namespace-scoped resources (Role, RoleBinding, Pod, NetworkPolicy) are used.

  7. Agent-generated notes are skipped on re-processing. Notes containing SESSION_MARKER_PREFIX are filtered out in handle_note() to prevent infinite loops. Self-filtering checks against all workflow bot user IDs (collected at startup) plus the orchestrator bot ID.

  8. All auto-triggers require Developer access. handle_pipeline(), handle_merge_request(), and handle_note() each check check_member_access() on the event’s user before executing a workflow. Events from non-developers are silently skipped (or replied to with an access-denied message for slash commands). This ensures that the target MR’s work products – its diff, description, CI logs, and commit messages – originate from a trusted author. Fork MRs from external contributors are blocked because the MR author lacks Developer+ on the target project.

    Scope limitation: this gate only covers the target MR. Once the agent is running, the model controls tool arguments and can direct tools at content beyond the target MR – other MRs, other refs, other job IDs, even other projects reachable by the read-only token. The current mitigations are: (a) workflow prompts instruct the model to use the event’s {project, iid, sha}, (b) the model would need to be manipulated via prompt injection from already-trusted content to deviate, (c) the sandbox prevents the model from acting on manipulated reasoning beyond producing text output, and (d) the token is read-only with minimal scope. However, if a data source tool is added that returns user-authored prose from arbitrary resources (e.g. issue bodies, wiki pages), it should apply per-author trust filtering (invariant #10).

  9. Data source tools that accept URLs must validate them against an allowlist. tf_get_test_log restricts URLs to the Testing Farm artifacts prefix to prevent the LLM from directing the orchestrator to fetch arbitrary URLs (SSRF).

  10. Third-party commentary entering the model prompt must be trust-filtered. Data sources that feed text from users other than the event trigger into contents (discussion comments, issue bodies) must gate on project membership at Developer+ level per author. Content from untrusted authors must be redacted to a fixed placeholder, never sanitized or escaped – there is no reliable way to escape adversarial text for an LLM prompt. Discussions where all notes are untrusted must be dropped entirely. See §9.1 for the reference implementation in gitlab_get_mr_discussions.

This invariant covers commentary (what other people said about the MR), not work products (the diff, CI logs, commit messages). Work products are inherently the input to the agent’s analysis and cannot be content-filtered without defeating the agent’s purpose. For the target MR, their trust comes from invariant #8 (the event trigger is Developer+). For content the model fetches beyond the target MR, trust depends on the tool: discussion tools must filter per-author (#10), while code/log/metadata tools rely on the mitigations described in #7.

5. Configuration System

5.1. Two sources, strict separation

Configuration comes from exactly two sources with no overlap:

  • YAML config file (CONFIG_PATH): operational settings, workflow definitions, project allowlists, data source declarations, and token env var names. This file is safe to commit, review, and audit.
  • Environment variables: secrets only (API keys, GitLab tokens, kubeconfig paths). These are injected at deployment time via K8s Secrets.

This split exists because YAML provides structure, validation, and audit trails, while secrets must stay out of version control.

5.2. Config loading

Two config objects are built at startup:

Config (from config.py): loaded by Config.load(settings), where settings is the settings: section from the YAML file. Auth/bootstrap fields come from env vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, etc.). Operational fields come from the settings dict with sensible defaults.

AgentConfig (from workflow_config.py): loaded by workflow_config.load(path). Contains all workflow definitions, project entries, and a pre-built project_index.

The two are kept separate because Config is needed everywhere (model construction, sandbox creation, session storage), while AgentConfig is only needed for event routing and data source registration.

5.3. YAML config structure

settings:
  gitlab_url: https://gitlab.com # operational, not a secret
  sandbox:
    image: quay.io/.../image:tag
    namespace: my-namespace # K8s only
  model: gemini-3.1-pro-preview
  # model: claude-sonnet-4-5@20250929   # Anthropic via Vertex AI (alternative)
  max_iterations: 30
  max_runs_per_mr: 5
  internal_notes: true
  max_concurrent_agents: 4
  sqs_queue_url: "" # empty = no SQS (local dev)
  s3_session_bucket: "" # empty = no S3 (local dev)
  trigger_prefix: /hummingbird # slash command prefix
  session_marker_prefix: hummingbird-session # HTML comment marker ID
  auto_trigger: true # auto-run on failed pipelines

workflows:
  analyze-failures:
    prompt: workflows/analyze-failures.md # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # per-workflow override
    max_iterations: 50 # per-workflow override

    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO # env var name, not the value
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens: # per-project token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO

Design choices in this structure:

Workflow-first organization. Each workflow owns its project list, not the other way around. This scopes data source permissions per workflow-project pair. A future code-review workflow can have different GitLab tokens (with different scopes) than the analyze-failures workflow, with no ambiguity.

Token env var names in YAML (not values). The YAML file is committed to the repo. It contains token_env: GITLAB_TOKEN_RO (a name), not the actual token. The actual secret value is resolved at runtime via os.environ.get(env_var). This allows the config to be reviewed and audited without exposing secrets.

Inline cluster_url only. The Konflux cluster URL is operational configuration (it identifies which cluster to talk to), not a secret. Putting it inline in YAML makes it visible and auditable. The URL must include the namespace path (e.g. /ns/my-tenant).

5.4. Project index

At load time, _build_project_index() constructs a reverse lookup:

project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}

This provides O(1) lookup when a pipeline event arrives with a project path. A single project can appear in multiple workflows (e.g. both analyze-failures and a future code-review), and each matching workflow will be triggered independently.

5.5. Token resolution chains

Model tool tokens (for data source API calls):

resolve_tool_token(project_entry, ds_name, workflow_cfg):
  1. project_entry.tokens[ds_name]           -> per-project override
  2. workflow_cfg.data_sources[ds_name].token_env  -> workflow default
  3. "" (empty string)                        -> no token
  Each step resolves the env var NAME, then reads os.environ[name].

Workflow action tokens (for per-workflow GitLab writes):

resolve_action_token(project_entry, ds_name):
  1. project_entry.action_tokens[ds_name]  -> env var NAME
  2. os.environ[name]                      -> resolved token value
  3. "" (empty)                            -> not configured
  No fallback chain. Missing tokens cause validate_env to fail at startup.

Orchestrator tokens (for operational reads and infrastructure notes):

resolve_orchestrator_token(project_path):
  1. ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT>  -> per-project
  2. ORCHESTRATOR_GITLAB_TOKEN                     -> global fallback
  Mangling: "/" -> "_", "-" -> "_", uppercase.
  e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS

Cluster URL (for Konflux):

resolve_cluster_url(ds_cfg):
  -> ds_cfg.cluster_url (inline value in YAML)

5.6. Environment validation

validate_env(agent_cfg) checks at startup that all referenced env vars exist. It walks every workflow’s data sources, project token overrides, and project action token entries, collecting missing vars into a single error message. This catches configuration errors early instead of failing mid-run when a specific data source or workflow action is first used. Missing action_tokens env vars are treated the same as missing model tool tokens: startup fails unconditionally.

See Also

4.5.20.3 - Hummingbird Agent Design — Engine

Agent loop, tool system, and sandbox backends for the Hummingbird Agent. This page covers the execution engine: how the agent iterates, dispatches tools, and isolates execution. For foundations (design philosophy, system architecture, modules, security, config), see Hummingbird Agent Design.

6. Agent Loop Design

For the wire-format walkthrough (what bytes go to the model API, what comes back), see Agent Model Loop. This section covers the design rationale behind the loop.

6.1. Full history replay

The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list from the beginning of the conversation. This means every large tool output sitting in history inflates every subsequent API call.

This property is fundamental to why auto-spill exists (section 7). Without auto-spill, a single cat of a 32KB file early in the conversation adds ~8K tokens to every remaining API call. Over a 15-iteration run, that is ~120K wasted tokens.

The alternative – conversation compaction (replacing old tool results with summaries) – was considered and deferred. Each provider has strict requirements about content structure, and modifying history risks confusing the model or violating API constraints. Provider-specific format requirements are documented in Hummingbird AI. Auto-spill solves 90% of the problem with none of the risk.

6.2. Budget model: iterations + context limit

The agent uses a dual-limit approach rather than a cumulative token budget:

Iteration limit (max_iterations, default 30 per workflow config). Hard cap on the number of model round-trips. This is the primary cost control lever. With auto-spill keeping per-call context bounded, iteration count is roughly proportional to cost.

Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens). Checked after each API call using response.usage.total_input_tokens. This is a safety net for cases where auto-spill is not sufficient (e.g., many small tool results that individually stay under the spill threshold but cumulatively fill the context).

Why not a cumulative token budget? Because with full history replay, each API call re-sends everything. “Cumulative billed tokens” double-counts: call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed total is 15K but actual new content is only 10K. Iteration count is a simpler and more predictable proxy for cost.

6.3. Two-tier budget escalation

Both limits use the same escalation pattern:

flowchart LR
    Normal["Normal<br/>tools available"]
    Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
    Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]

    Normal -->|"80% reached"| Soft
    Soft -->|"100% reached"| Hard

Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is injected into contents for that turn only, then popped before the response is persisted. Tools remain available so the model can finish in-progress work.

Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral user message AND tool_defs is set to [] (empty list) to physically prevent further tool calls. The model must produce text. This is more reliable than relying on provider-side tool disabling, which models may ignore.

6.3.1 Ephemeral messages and provider normalization

All per-turn warnings use ephemeral user messages: they are appended to contents before the API call and popped immediately after. This keeps the system prompt stable across all iterations and prevents warnings from polluting the conversation history saved in sessions.

Provider adapters normalize provider-specific conversation requirements while preserving the agent’s saved history; details are in Hummingbird AI.

6.4. Empty response handling

Models occasionally return unusable responses (no text and no valid tool calls). The agent retries up to MAX_EMPTY_RETRIES (2) times:

  1. Pop the empty response from contents. It adds nothing and may confuse the model on the next call.
  2. Inject EMPTY_RESPONSE_NUDGE as an ephemeral user message on the next turn: “Your previous response was empty. Please continue…”
  3. Continue the loop (consuming an iteration).

The nudge is delivered as an ephemeral user message (injected before the API call and popped after). This avoids mutating the system prompt and keeps the conversation history clean for session persistence.

The empty_retries counter resets to 0 after any successful iteration (one where tool calls were executed). This means the model gets fresh retries if it produces empty responses at different points in the conversation.

MALFORMED_FUNCTION_CALL handling. Models sometimes return a finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls. This is treated as a special case of empty response: the retry mechanism kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which tells the model to retry with simpler arguments and avoid large text payloads in tool call arguments.

6.5. Error recovery

Model API and transport errors. The hummingbird-ai library classifies provider and transport failures, applies the workflow’s retry budget, and records AI usage and request metrics. See Hummingbird AI for provider-specific behavior. The agent loop retains partial results when a terminal error reaches it.

Unexpected loop errors. run_agent_loop wraps the _generate_with_retry call in a try/except Exception that logs and breaks instead of propagating. This ensures the function always returns partial results (accumulated contents, transcript, sandbox state) even when an unexpected exception occurs (e.g. JSONDecodeError from a truncated API response). The caller saves the session normally – conversation history and sandbox archive are preserved for resumption. This follows the “partial over nothing” principle: 32 iterations of work are more valuable than a crash.

No failure-path session save. _execute_workflow does not save a session when run_workflow raises. With the agent loop catching unexpected errors, the failure path only fires for infrastructure errors (sandbox start, config) where there is no useful state. Not saving avoids overwriting a previous good session when a reply attempt fails.

Tool execution errors. _execute_tool_calls() wraps each tool_registry.execute() in a try/except. Unhandled exceptions are caught and returned to the model as {"error": "Tool X failed: ..."}. This prevents a single broken tool from crashing the entire session – the model sees the error and can adapt.

Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the TimeoutExpired exception is caught in the tool registry and returned as a structured error dict. The model can retry with a different command or proceed without the result.

6.6. Session resumption

When resuming from a previous session:

  1. initial_contents (the full contents list from the previous run) is prepended, followed by the user’s reply as a new user turn.
  2. CONTINUATION_PROMPT is appended to the system prompt.
  3. The sandbox is restored from the archived tarball.
  4. A fresh iteration budget starts from 0.

Saved sessions use a versioned JSON envelope (format_version: 1) whose contents are in a canonical message format independent of the active model. Unversioned legacy sessions are not supported. On resume, run_workflow asks the active adapter to load canonical history; after a successful run, it converts native contents back to canonical form before persistence. The library documents the canonical format and provider conversions in Hummingbird AI.

CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g., analyze-failures.md) tells the model to follow a rigid multi-phase workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the entire analysis. The continuation prompt overrides this: “Do NOT re-run the full workflow. Respond directly to the user’s question.”

The practical constraint on resumed sessions is the context window, not iterations. The restored history from an 8-iteration cold start uses ~50-100K input tokens. Each new tool call adds more. The existing CONTEXT_LIMIT check still applies and will force wrap-up if the context grows too large.

Canonical session format (rationale)

Storing sessions in canonical form decouples persisted history from any one provider’s JSON shape. The agent loop treats native contents as opaque; the library owns provider conversion at load and save boundaries.

6.7. Provider caching and pricing

Provider-specific prompt caching and token-cost estimation are owned by hummingbird-ai.

7. Tool System and Auto-Spill

7.1. Tool registry design

ToolRegistry is the single point of dispatch for all tool calls. The agent calls registry.execute(tool_call) and gets a dict back. It never calls sandbox methods or data source functions directly.

Three built-in tools are always available:

  • sandbox_exec – runs sh -c <command> in the sandbox container. Returns {exit_code, stdout, stderr}, with auto-spill for large outputs.
  • fetch_to_sandbox – calls a data source function and writes the result to a specified path in the sandbox. Returns metadata only (path, byte count, line count). Used when the model wants explicit path control.
  • fetch_batch_to_sandbox – calls multiple fetch_to_sandbox in one tool call. Saves iterations vs. sequential calls (e.g., fetching both pipelineruns and taskruns in one round-trip).

Data source tools are registered dynamically per workflow config. Each data source module provides a register() function that creates ToolDef objects and closures over credentials, then calls registry.register_data_source(tool_def, func, response_metadata).

7.2. Auto-spill architecture

Auto-spill is the key mechanism for keeping the LLM context bounded. Without it, large outputs accumulate in the contents list and inflate every subsequent API call (because both APIs replay the full history).

flowchart TD
    Output["Tool produces output"]
    SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
    Inline["Return full output inline"]
    Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
    Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]

    Output --> SizeCheck
    SizeCheck -->|"<= 4 KB"| Inline
    SizeCheck -->|"> 4 KB"| Spill --> Preview

Three spill paths share the same _spill_counter to prevent filename collisions:

sandbox_exec spill. When stdout or stderr exceeds OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to /tmp/data/_out/{counter}.txt via _spill_field(). The model receives:

{
  "exit_code": 0,
  "stdout": "<first 4KB>",
  "stdout_truncated": true,
  "stdout_file": "/tmp/data/_out/0.txt",
  "stdout_bytes": 32768,
  "stdout_lines": 1024,
  "stdout_tail": "<last 512 bytes>"
}

The preview (head + tail) gives the model enough context to decide whether to process the full file with jq/grep/head.

Data source inline spill. When a direct data source call returns text larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes it to /tmp/data/_out/{name}_{counter}.txt and returns:

{
  "saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
  "bytes": 85432,
  "lines": 2100,
  "preview": "<first 4KB>"
}

Streaming responses (e.g., gitlab_get_repo_archive which returns a StreamingResponse with suffix .tar.gz) are written via _spill_streaming(). Text streams (binary=False, the default) capture a UTF-8 preview from the head while piping chunks to the sandbox. Binary streams (binary=True) skip preview capture entirely and return only {saved_to, bytes}, avoiding meaningless decoded output for formats like tar.gz. The StreamingResponse.suffix field controls the file extension (.jsonl, .tar.gz, .log).

Non-streaming binary responses are written to .bin files with no preview via _spill_binary().

Sandbox-optional mode. ToolRegistry accepts sb=None for use cases that do not require a sandbox (e.g. evaluation harnesses that only use native model tools like Google Search grounding). When no sandbox is available:

  • Sandbox tools (sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox) are omitted from get_tool_defs() so the model never sees them.
  • Text data sources (inline and streaming) degrade to truncated previews (OUTPUT_PREVIEW_BYTES). The model receives {preview, truncated: true} instead of {saved_to, preview}. Streaming text is consumed into a memory buffer up to the preview limit and the rest is discarded.
  • Binary data (streaming and non-streaming) returns an error with the byte count ({error, bytes}). Binary content cannot be meaningfully previewed as text.

fetch_to_sandbox spill. Always writes to the caller-specified path (not /tmp/data/_out/). Returns metadata only (no preview). The model uses this when it wants a specific filename for later processing.

7.3. Why auto-spill instead of rejecting large outputs

The earlier design rejected large data source responses with “use fetch_to_sandbox instead.” This caused two wasted iterations per rejection: the model makes the call, gets rejected, then has to repeat with fetch_to_sandbox. With auto-spill, the data flows to a sandbox file transparently. Validation showed ~10% fewer iterations with auto-spill.

7.4. Why fetch_to_sandbox still exists alongside auto-spill

Auto-spill handles the common case, but fetch_to_sandbox provides:

  • Explicit path control. The model can choose meaningful filenames (/tmp/data/pipelineruns.json) instead of getting auto-generated names (/tmp/data/_out/konflux_list_pipelineruns_1.txt).
  • Batch fetching. fetch_batch_to_sandbox combines multiple fetches in one tool call, saving iterations.
  • No preview overhead. fetch_to_sandbox returns metadata only, which is useful when the model knows it will process the file with sandbox_exec anyway.

7.5. ToolDef notes

ToolDef has an optional notes field for domain knowledge that belongs in the system prompt but not in the tool’s JSON schema. Examples:

  • Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and the two label selectors (BUILD vs. TEST).
  • Testing Farm notes explain XML result structure and usage patterns.

ToolRegistry.get_tool_notes() collects all non-None notes into a ## Data Source Notes section that is prepended to the workflow body in the system prompt:

BASE_SYSTEM_PROMPT + tool_notes + workflow_body

This keeps domain knowledge close to the tool definitions (in the data source module) rather than duplicated in every workflow .md file.

7.6. Response metadata

register_data_source() accepts optional response_metadata – a dict that is merged into every response from that tool (inline, auto-spill, and fetch_to_sandbox). Used for:

  • Konflux: {"konflux_ui": "https://..."} so the model can build reviewer-facing PipelineRun links.
  • Testing Farm: {"artifacts_base": "https://..."} so the model can build artifact links.

This avoids having the model ask “what is the Konflux UI URL?” – the information arrives with every tool response.

8. Sandbox Architecture

8.1. The Sandbox protocol

All backends implement an 8-method typing.Protocol:

class Sandbox(Protocol):
    def start(self) -> None                                                  # create container/pod, pre-create /tmp/data/
    def exec(command, *, stdin_data: bytes | None) -> ExecResult             # sh -c in sandbox
    def write_file(path, data: bytes) -> None                               # stdin pipe: cat > path
    def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int]    # Popen stdin pipe, returns (bytes, lines)
    def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult         # Popen with streaming stdin
    def read_file_iter(path) -> Iterator[bytes]                             # Popen stdout pipe in chunks
    def cleanup(self) -> None                                               # rm -f container / delete pod
    def linger(session_id) -> None                                          # keep alive for reuse, or fall back to cleanup

ExecResult is (exit_code: int, stdout: str, stderr: str).

stream_to_file() and stream_exec() use subprocess.Popen with a stdin pipe, writing chunks incrementally. This avoids buffering large payloads in orchestrator memory (e.g., streaming a tar.gz archive into the sandbox). read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.

write_file() uses stdin piping (cat > path), never host volume mounts. This is critical: it means data flows through the orchestrator process, not through a shared filesystem. The sandbox has no host mounts.

8.2. Backend selection

Five sandbox backends are available:

Backend --sandbox When to use
PodmanSandbox podman Local development (default for run)
K8sSandbox k8s Direct pod creation on a K8s cluster
K8sPoolSandbox k8spool Pre-warmed pool via Deployment (default for serve)
KubeVirtSandbox kubevirt Direct VMI creation with SSH exec
KubeVirtPoolSandbox kubevirtpool Pre-warmed VMI pool via VMIRS

create_sandbox() factory handles podman, k8s, and kubevirt. Pool backends (k8spool, kubevirtpool) are handled in __main__.py, which creates a SandboxPool or VmiPool and passes it to run_workflow().

The sandbox backend can be set per-workflow via the sandbox: field in the workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox

default. This allows different workflows to use different backends (e.g., lightweight code review uses pod pool, heavier analysis uses VM pool).

K8s is never auto-detected from the environment. It requires explicit CLI flags. This prevents accidental use of a K8s sandbox when developing locally.

For K8s namespace resolution (k8s and k8spool):

  1. If --namespace is provided, use it.
  2. If --context is provided, extract the namespace from the kubeconfig context. If the context has no default namespace, raise an error.
  3. If neither is provided (in-cluster), use sandbox.namespace from the config file.

8.3. PodmanSandbox

podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
    --workdir /tmp {image} sleep infinity
  • Unique container name with UUID suffix prevents collisions
  • sleep infinity keeps the container alive for repeated exec calls
  • --network=none provides complete network isolation
  • --user 65532 is a fixed non-root UID (matches nonroot in distroless)
  • Cleanup: podman rm -f (force, in case exec is still running)

8.4. K8sSandbox: the hybrid approach

The K8s sandbox uses a hybrid of two tools:

  • kubernetes Python library for pod lifecycle: create_namespaced_pod, read_namespaced_pod (poll for Running), delete_namespaced_pod.
  • kubectl exec subprocess for command execution.

Why not use the kubernetes Python library for exec too? Three problems discovered during development:

  1. No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions 1-4 have no mechanism to signal “stdin is done.” Commands like cat > /file hang forever waiting for more input. Python client v5 support does not exist.

  2. BrokenPipeError on large stdin. Sending more than ~1MB through the WebSocket stream() API causes pipe errors, breaking write_file() for large data source responses.

  3. Unbounded memory from stream(). The stream() function accumulates all stdout/stderr data in memory with no streaming control. A command producing megabytes of output would consume unbounded memory.

kubectl exec as a subprocess avoids all three problems and provides the same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr captured, exit code from return code. The implementation in exec() is nearly identical between PodmanSandbox and K8sSandbox.

8.5. Pod manifest design

The K8s pod manifest is built by _build_pod_manifest():

metadata:
  labels:
    app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
  automountServiceAccountToken: false # no K8s API from sandbox
  activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    seccompProfile: RuntimeDefault
  containers:
    - name: sandbox
      command: ["sleep", "infinity"]
      workingDir: /tmp
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
        limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }

runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC assigns a UID from the namespace UID range. On vanilla K8s, the image’s USER directive is used. Setting an explicit UID would conflict with OpenShift’s SCC admission.

8.6. Auth modes

  • Local development: --context flag passes the kubeconfig context to config.new_client_from_config(context=...), creating a per-instance ApiClient.
  • Production (in-cluster): context=None triggers config.load_incluster_config(), using the pod’s ServiceAccount token.

The kubectl exec commands include --context when running locally but omit it when in-cluster (kubectl uses the default in-cluster config).

8.7. Pre-created directories

Both backends run mkdir -p /tmp/data in start() after the container/pod is up. This provides a working directory for the model without consuming an iteration. /tmp/data/_out (the spill directory) is created on demand by mkdir -p $(dirname ...) in write_file().

8.8. K8sPoolSandbox: Deployment-backed pool

On-demand pod creation adds 5-30 seconds of latency per workflow (image pull, scheduling, container start). For interactive use (slash commands, reply-based resumption), this delay is user-facing. The pool eliminates it.

Mechanism. A Kubernetes Deployment maintains a set of pods labelled hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim() finds a Running standby pod, patches it to role: active, clears its ownerReferences (detaching it from the ReplicaSet), and annotates it with hummingbird/reap-by (an absolute UTC deadline). The Deployment controller sees the ReplicaSet is under the desired replica count and creates a replacement.

Pod lifecycle.

  1. Deployment creates pod -> role: standby (managed by ReplicaSet)
  2. claim() patches -> role: active, ownerReferences: [], reap-by: now + max_active_seconds
  3. Agent uses the pod via inherited K8sSandbox.exec()
  4. On success: linger() patches -> reap-by: now + linger_seconds, session-id: <id> (pod stays alive)
  5. On reply within linger window: try_reclaim() patches -> reap-by: now + max_active_seconds, clears session-id
  6. On failure or linger expiry: pod is deleted (by cleanup() or the reaper)

Pod lingering. After a successful workflow, pool sandboxes are kept alive for linger_seconds (default 300) instead of being deleted immediately. The pod is annotated with hummingbird/session-id to link it back to the session. If a user reply arrives within the linger window, try_reclaim() finds the pod by session-id, clears the annotation (marking it as in-use), and resets reap-by. This skips pod creation and S3 archive restoration. If no reply arrives, the reaper deletes the pod when its reap-by deadline passes.

Concurrency safety. session-id is only present while lingering. Its absence during active execution prevents concurrent replies from adopting the same pod. try_reclaim() is serialized by a threading lock; the first caller wins, others fall back to S3 restore with a fresh pod.

Reaping. reap_expired() is called once per workflow execution (after sandbox acquisition, before the model loop). It performs a single sorted pass over all active pods:

  1. Non-lingering pods (no session-id) past their reap-by deadline are deleted immediately.
  2. Lingering pods (with session-id) are sorted by reap-by ascending. The loop deletes pods that are either expired (now > reap-by) or exceed max_lingering_pods (default 2, configurable), starting with those closest to their deadline. The loop stops when both conditions are satisfied (now <= reap-by and remaining count <= limit).

This replaces the previous design where _reap_expired() ran inside claim() on every poll iteration. Moving it to a once-per-workflow call reduces API load and centralizes cleanup. The max_lingering_pods cap prevents unbounded pod accumulation from many short-lived workflows.

Indefinite wait. claim() polls until a pod is available or the shutdown event fires. It logs at DEBUG level on each poll, escalating to WARNING after ~2 minutes. This is normal behavior when the pool (Deployment replicas) is smaller than max_concurrent_agents – the SQS semaphore caps concurrency, so demand will not permanently exceed supply.

Config simplification. In pool mode, the pod image, resources, metadata, and security context are defined solely in the Deployment template. The agent config only needs sandbox.namespace and sandbox.active_deadline_seconds. This eliminates duplication between the agent configmap and the Deployment.

Class design. K8sPoolSandbox inherits from K8sSandbox to reuse exec(), write_file(), read_file_iter(), and cleanup(). It overrides __init__() (does not call super().__init__() since that resolves kubeconfig), start() (claims from pool instead of creating a pod), and linger() (patches annotations instead of deleting). It adds start_from_existing() for the reclaim path. All sandbox backends implement linger(session_id): non-pool backends fall back to cleanup().

8.9. KubeVirtSandbox: SSH-based VM execution

KubeVirtSandbox provides full VM isolation using KubeVirt VirtualMachineInstances. It parallels K8sSandbox but uses CustomObjectsApi for VMI lifecycle and ssh for command execution instead of kubectl exec.

Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides virtctl console (serial) and VNC, but these are not suitable for programmatic command execution. SSH provides a reliable, well-understood exec channel with stdin/stdout/stderr piping.

SSH key injection. The VM sandbox image includes a custom inject-ssh-keys.service that reads SSH public keys from a KubeVirt Secret volume attached as a virtio disk with serial ssh-pubkeys (visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to /root/.ssh/authorized_keys before sshd starts. This avoids the need for cloudInitNoCloud or qemu-guest-agent.

Single-VM lifecycle:

  1. Generate an ephemeral ed25519 key pair via ssh-keygen
  2. Create a K8s Secret with the public key
  3. Create the VMI referencing the containerDisk image and the SSH Secret
  4. Poll VMI status until Running with a pod network IP
  5. Poll SSH readiness (ssh sandbox@ip true)
  6. Execute workflow commands via SSH
  7. Cleanup: delete VMI, delete Secret, remove temp key files

Polling strategy. Two-phase wait: first poll status.phase via get_namespaced_custom_object until Running (same interval as pod polling), then poll SSH readiness with ConnectTimeout=5. Total timeout matches POD_START_TIMEOUT (300s) for VMI startup plus _SSH_READY_TIMEOUT (120s) for SSH.

8.10. KubeVirtPoolSandbox: VMIRS-backed pool

KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox + SandboxPool, operating on VMI custom resources instead of Pods.

VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native controller maintains a desired number of identical VMIs. When a VMI is claimed (label flipped to active, ownerReferences cleared), the VMIRS automatically creates a replacement – the same pattern as a Deployment replenishing claimed pods.

Shared SSH key. Pool mode uses a single key pair for all standby VMIs. The public key is stored in a K8s Secret referenced by all VMIs in the VMIRS template. The private key path is provided via sandbox.vm_ssh_private_key in the config. On pool restart, a new key pair can be generated; existing VMIs drain naturally with the old key.

Claim/reap/linger semantics. Identical to the pod pool: claim() polls for Running VMIs with hummingbird/role=standby, patches to active, clears ownerReferences. reap_expired() deletes VMIs past their hummingbird/reap-by deadline. try_reclaim() finds lingering VMIs by hummingbird/session-id. The only difference is the API: all operations go through CustomObjectsApi with kubevirt.io/v1 virtualmachineinstances.

Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox to reuse exec(), write_file(), read_file_iter() (all SSH-based). It overrides __init__() (takes pool, skips key generation), start() (claims from pool), cleanup() (only deletes VMI, not Secret/keys which are pool-owned), and linger() (patches annotations).

4.5.20.4 - Hummingbird Agent Evals

Evaluation framework for scientific measurement of LLM output quality. For the agent itself, see Hummingbird Agent. For architecture details, see Hummingbird Agent Design.

What evals are

Evals measure the quality of LLM outputs under controlled conditions. They answer questions like “does prompt v5 produce better code reviews than v4?” or “can an LLM answer this question correctly using only public documentation?”

Evals are not unit tests. Unit tests verify deterministic behavior; evals quantify stochastic output quality through structured scoring.

General pattern

Evals generally follow a pattern of:

  1. Inputs – a set of test cases: golden dataset with scoring criteria, real MRs to review, or other structured inputs.
  2. Model execution – run one or more LLM configurations against the inputs, optionally with tools (web search, sandbox).
  3. Scoring – evaluate outputs via LLM-as-judge, comparison against reference answers, or human review.
  4. Iteration – adjust prompts or config, re-run, check for improvement.

Not all evals use every step. Baseline measurements may stop at step 3; comparison evals may skip the golden dataset entirely.

Shared infrastructure

hummingbird_agent.eval

The hummingbird_agent.eval module provides generic evaluation primitives used across all evals. Key components:

  • EvalJudge – LLM-as-judge wrapper over Vertex AI with explicit temperature and token limit control. Supports both strict judging (low temperature) and open-ended analysis (higher temperature).
  • Snapshot management – saves timestamped evaluation results with metadata, maintains a latest symlink.
  • Variant tracking – persists the best-performing prompt or config variant across iterations.
  • Convergence detection – checks whether scores have plateaued across consecutive iterations.
  • Holdout validation – compares holdout scores against dev scores using standard error of the mean.

See the module docstrings for the full API.

hummingbird_agent.evals.mr_helpers

MR-experiment-specific helpers built on top of hummingbird_agent.eval. Provides GitLab auth, MR data I/O, review generation orchestration, and CLI scaffolding. Used by the code-review and renovate-triage evals.

Running an eval

Prerequisites

  1. GCP Application Default Credentials:

    gcloud auth application-default login
    export GOOGLE_CLOUD_PROJECT=<your-gcp-project>
    
  2. The agent package installed in editable mode:

    cd hummingbird-agent
    pip install -e .
    

Execution

Each eval is a Python module under hummingbird_agent/evals/. Run from hummingbird-agent/:

python -m hummingbird_agent.evals.<name>.<script> [options]

Refer to each eval’s own README for specific options and configuration.

Existing evals

Writing a new eval

Create a new directory under hummingbird_agent/evals/ with:

  • README.md – what the eval measures and how to run it, linking back to this document for the general framework.
  • Golden dataset – inputs and scoring criteria. Can live outside the eval directory (e.g. in the documentation repo alongside the content being measured) and be passed via CLI.
  • run_*.py – orchestration script using hummingbird_agent.eval primitives (or hummingbird_agent.evals.mr_helpers for MR-based evals).

The eval’s run script should be importable as a module (python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed. Do not hardcode GCP project IDs; VertexAuth resolves them at runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.

Authentication

All evals use VertexAuth from hummingbird_ai.http, which resolves credentials and project in this order:

  1. Explicit project= parameter (if passed).
  2. GOOGLE_CLOUD_PROJECT environment variable.
  3. ADC default project (from gcloud auth application-default login).

Service account keys work automatically when GOOGLE_APPLICATION_CREDENTIALS is set.

4.5.20.5 - Documentation Quality Evaluation

Reference for the documentation quality evaluation, which measures whether public documentation is complete enough for LLMs with web search to answer questions correctly. For the general evaluation framework, see Hummingbird Agent Evals.

Config file

A single YAML file combines evaluation settings and the golden dataset. It is passed to the run script via -c:

python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml

The config file lives in the documentation repo alongside the content being measured.

Evaluation settings

Key Required Default Description
models yes – List of model configurations
models[].id yes – Vertex AI model identifier
models[].region yes – Vertex AI region
models[].google_search no false Enable Google Search grounding (Gemini)
models[].web_search no false Enable web search tool (Claude)
judge yes – Judge model configuration
judge.model yes – Judge model identifier
judge.region no global Judge model region
repetitions no 3 Runs per (question, model) pair
max_iterations no 5 Agent loop iteration cap per question

Golden dataset

The questions list defines what the evaluation measures:

Field Required Default Description
id yes – Unique identifier (used in output paths)
question yes – Question sent to each model
facts yes – List of fact checks
facts[].question yes – Yes/no question about the response
facts[].weight no 1 Relative importance
coherence no – Criterion for narrative quality scoring
coherence_weight no 1.5 Weight of coherence in the final score
threshold no 0.4 Minimum weighted score to pass

CLI options

Option Default Description
-c / --config required Path to config YAML
--phase all ask, judge, or all
--results-dir results/ next to config Output directory
--run-id UTC timestamp Subdirectory under results

Scoring

Each response is scored independently:

  • Fact scoring – the judge answers each fact question with yes or no. Passing facts contribute their weight to the numerator.
  • Coherence scoring – the judge rates narrative quality from 0.0 to 1.0. The coherence score is multiplied by its weight and added to the total.
  • Weighted score – sum(passed_fact_weights + coherence_contribution) / total_weight. A response passes if weighted_score >= threshold.

The summary aggregates per-model averages across all repetitions.

4.5.20.6 - Hummingbird Agent Design — Runtime

Data sources, event pipeline, sessions, workflows, and deployment for the Hummingbird Agent. This page covers the runtime: how the agent connects to external systems, processes events, manages sessions, and is deployed. For the execution engine (agent loop, tool system, sandbox), see Hummingbird Agent Design — Engine.

9. Data Sources

Data sources are external APIs wrapped as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (inline or auto-spilled). Each data source module follows the same pattern:

  1. Define TOOL_DEFS – a list of ToolDef objects with names, descriptions, parameter schemas, and optional notes.
  2. Define implementation functions that take a pre-configured client as the first argument.
  3. Provide a register() function that creates the client, wraps implementations with functools.partial, and calls registry.register_data_source() for each tool.

Credentials are captured at registration time via functools.partial closures. They are never stored as global state and never leak into tool definitions or model contents.

9.1. GitLab

Uses python-gitlab library with retry_transient_errors=True for automatic retry on transient HTTP errors.

Tools:

  • gitlab_get_mr_details – MR metadata (title, author, SHA, labels, URLs)
  • gitlab_get_commit_statuses – all CI/CD statuses for a commit SHA (paginated automatically). Covers both Konflux external statuses and native GitLab CI job statuses. Each status includes target_url (job page link containing the job ID) and allow_failure.
  • gitlab_get_mr_diff – changed files in the MR
  • gitlab_get_file_at_ref – raw file content at a git ref
  • gitlab_get_repo_archive – repository tar.gz via repository_archive(iterator=True). Returns a StreamingResponse with chunked binary data (suffix .tar.gz), so the archive streams to the sandbox without buffering in orchestrator memory.
  • gitlab_get_job_log – job trace (log output) for a GitLab CI job. Uses lazy=True on the job object and trace(iterator=True) for streaming. Each chunk is decoded with errors='replace' and ANSI escape codes are stripped per-chunk before re-encoding. Returns a StreamingResponse (suffix .log). Per-chunk ANSI stripping is safe because escape sequences are <20 bytes and chunks are 1024+ bytes.

No response_metadata is set because GitLab commit statuses already contain target_url fields that the model uses for linking.

gitlab_get_mr_discussions – trust filtering and redaction.

MR discussions are the first data source that feeds user-generated free text into the model prompt. Unlike diffs and CI logs (which are code or machine output), discussion comments can contain arbitrary prose written by anyone who can post on the MR – including external contributors on public projects. This creates a prompt injection vector: an attacker posts a comment containing instructions that manipulate the model’s review output.

The discussions tool addresses this with a three-layer defence:

  1. Author trust gate. Each note’s author is checked for Developer+ access (level >= 30) on the project via members_all.get(author_id). Results are cached per author_id within a single call to avoid repeated API lookups. System notes (merge events, label changes) bypass the author check. Agent-authored notes are trusted via their author’s Developer+ access (the bot account holds Developer access on configured projects). Note: agent note detection for redaction purposes (stripping transcripts and footers) uses session marker presence, but trust is always based on the author’s access level, not the marker. This prevents marker injection from granting trust to non-member notes.

  2. Redaction, not sanitization. Untrusted notes in a discussion that also contains trusted notes are replaced with a fixed placeholder ([redacted: non-member comment]). The placeholder preserves the conversation structure (the model sees that someone replied, but not what they said). Discussions where all notes are untrusted are dropped entirely. No attempt is made to sanitize or escape untrusted content – there is no reliable escaping mechanism for LLM prompts, so the only safe option is to withhold the content entirely.

  3. Agent note stripping. Agent-authored notes contain large <details><summary>Agent transcript</summary>...</details> blocks (15K-90K chars of raw tool calls), metadata footers, and session markers. These are stripped before the note enters the model prompt, leaving only the review text. This serves dual purposes: token efficiency and avoiding feeding the model its own raw tool call history (which could cause degenerate self-referential loops).

Residual risks and scope limitations:

  • A compromised Developer+ account can inject adversarial content. This is accepted as equivalent to the existing risk of a compromised developer pushing malicious code (which the agent would also process).
  • The trust threshold is project-level (Developer role on the project), not MR-level. A developer on the project can influence any MR’s review.
  • Comments posted after the discussions are fetched but before the review is posted are not seen. This is a TOCTOU gap but has no security impact (the model simply misses late comments).

9.2. Konflux

Uses raw requests against K8s and Kubearchive APIs (not the kubernetes Python SDK). This avoids the heavy kubernetes client dependency for what is essentially bearer-token HTTP with label selectors.

Dual-fetch architecture:

flowchart LR
    subgraph fetch [Fetch Phase]
        KA["Kubearchive<br/>(historical)"]
        K8s["K8s API<br/>(live)"]
    end
    Combine["Combine items"]
    Dedup["Deduplicate<br/>by metadata.uid"]

    KA --> Combine
    K8s --> Combine
    Combine --> Dedup

For each resource type (PipelineRuns, TaskRuns), the client fetches from both Kubearchive (completed/historical resources) and the live K8s API, combines the results, and deduplicates by metadata.uid. This ensures no resources are missed regardless of whether they have been archived yet.

Two label selectors. Konflux uses different labels for BUILD and TEST PipelineRuns:

  • BUILD: pipelinesascode.tekton.dev/sha=<commit_sha>
  • TEST: pac.test.appstudio.openshift.io/sha=<commit_sha>

Each iter_pipelineruns()/iter_taskruns() call queries both selectors and combines the results. _iter_resources(endpoint, label_selectors) is the generic implementation: it creates one session, iterates each selector through _iter_both_sources(), and deduplicates across selectors by UID.

Per-PLR TaskRun fetching. iter_taskruns() accepts either commit_sha (two-selector path) or pipelinerun (single selector: tekton.dev/pipelineRun=<name>). When pipelinerun is set, only the ~15-21 TaskRuns for that PipelineRun are fetched (~600 KB) instead of all TaskRuns for the commit (~530 MB for large commits). This is server-side filtering — the K8s API and Kubearchive both support the tekton.dev/pipelineRun label selector.

Status filtering and streaming counts. _items_to_jsonl(items, status, stats) accepts an optional status: set[str] | None allowlist and a _StreamStats accumulator. For each item it increments stats.by_reason (keyed by reason) regardless of the filter (so counts reflect all items seen), then skips items whose status.conditions[0].status is not in the allowlist. The allowlist values are "True" (success), "False" (failure), or "Unknown" (in-progress) — only 3 possible values, covering every Tekton reason without enumeration. Items without conditions always pass. The _finalize() generator wrapper exhausts the chunks iterator, then writes counts, included, and excluded into the StreamingResponse.metadata dict — after the sandbox file is fully written. The filtering happens O(1) per item with no buffering.

StreamingResponse.metadata. StreamingResponse (in tools.py) has a mutable metadata: dict[str, Any] field (default empty dict). Both _exec_fetch_to_sandbox() and _spill_streaming() merge it into the response dict after the iterator is exhausted. The Konflux code populates it with counts, included, and excluded; other data sources can use it for their own post-streaming summaries without touching tools.py.

Pod log fetching. get_pod_log() accepts an optional container parameter. When specified, it fetches logs for that single container. When omitted, it discovers all containers from the pod spec and concatenates their logs with === container_name === headers. Each individual log fetch tries Kubearchive first, then falls back to the live K8s API. Logs may be unavailable from both sources if the pod has expired.

Streaming pagination. K8s list endpoints can return pages of 80+ MB when a commit touches many components (e.g. 500 PipelineRuns per page). iter_paginated() uses session.get(stream=True) and ijson.parse() to stream-parse each page: items are yielded one at a time via ObjectBuilder, and the metadata.continue pagination token is captured from the same parse pass. resp.raw.decode_content = True is set so urllib3 transparently decompresses gzip/deflate Content-Encoding inline (Kubearchive returns gzip-compressed responses). Only one item is in memory at a time – O(single_item) regardless of page size. Truncated or malformed streams raise IncompleteJSONError, which is caught and treated like a network error (log a warning, stop iterating that source).

Credential resolution. KonfluxClient.__init__() parses the kubeconfig file to extract the API server URL and bearer token for the cluster. The cluster domain is extracted from the cluster_url config value. This approach avoids depending on kubectl or the kubernetes Python library for API authentication.

Response metadata: Static metadata {"konflux_ui": "...", "konflux_pipelinerun_url_pattern": "..."} is merged into every response so the model can build reviewer-facing links. Dynamic per-call metadata (counts, included, excluded) is populated into StreamingResponse.metadata by _finalize() during streaming and merged by tools.py after the iterator is exhausted.

9.3. Testing Farm

Uses requests.Session with a module-level retry adapter (429, 5xx) for resilience against transient errors.

Tools:

  • tf_get_results – JUnit XML results for a request ID
  • tf_get_test_log – individual test log by URL (from results.xml); restricted to the ARTIFACTS_BASE prefix to prevent SSRF
  • tf_get_request_status – request state, queue/run times

ToolDef notes on tf_get_results document the XML structure: //testsuites/@overall-result, //testcase/@result, //testcase/logs/log with @name and @href. This domain knowledge goes into the system prompt so the model knows how to parse the XML with sandbox_exec using python3 xml.etree.ElementTree.

Response metadata: {"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"} is merged into every response so the model can build artifact links like [{request_id}]({artifacts_base}/{request_id}/).

9.4. Registration flow

data_sources.register_selected() is the entry point called by runner.py. It iterates the workflow config’s data_sources dict and registers only the declared sources:

for ds_name, ds_cfg in wf_cfg.data_sources.items():
    if ds_name == "gitlab":
        token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
        gitlab.register(registry, gitlab_url, token)
    elif ds_name == "konflux":
        cluster_url = resolve_cluster_url(ds_cfg)
        kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
        konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
    elif ds_name == "testing_farm":
        testing_farm.register(registry)

This selective registration means a workflow with data_sources: {gitlab: {...}} only exposes GitLab tools to the model. Konflux and Testing Farm tools do not appear in the tool definitions, preventing the model from attempting to use unconfigured data sources.

9.5. Data flow: streaming vs buffered

Data moves from external APIs through the orchestrator to the model (inline or auto-spilled to sandbox). The memory profile of each tool depends on whether the HTTP response is consumed incrementally or buffered entirely.

Streaming (preferred for large/unbounded responses). The HTTP response is consumed incrementally – via ijson stream parsing, iterator=True in python-gitlab, or lazy pagination. Only one chunk or item is in memory at a time. Used by:

  • iter_paginated() (Konflux) – session.get(stream=True) + ijson
  • get_commit_statuses() (GitLab) – statuses.list(iterator=True) via _iter_commit_statuses(), yielding JSONL bytes one status at a time
  • get_repo_archive() (GitLab) – repository_archive(iterator=True), returns chunked tar.gz binary via StreamingResponse(suffix=".tar.gz", binary=True) – no text preview is generated
  • get_job_log() (GitLab) – trace(iterator=True) with per-chunk ANSI stripping, returns StreamingResponse(suffix=".log")

All produce StreamingResponse objects that flow through stream_to_file() into the sandbox without accumulating in orchestrator memory. StreamingResponse.suffix controls the auto-spill filename extension (.jsonl, .tar.gz, .log). StreamingResponse.binary controls whether a head preview is captured (False for text, True to skip for opaque binary formats).

Buffered (acceptable for small/bounded responses). The full response is loaded into memory as a string or dict. This is fine when the response size is bounded and small (typically < 1 MB). Used by:

  • get_mr_details() (GitLab) – single MR object, < 10 KB
  • get_file_at_ref() (GitLab) – single file, bounded by repo constraints
  • tf_get_results(), tf_get_test_log(), tf_get_request_status() (Testing Farm) – XML/text/JSON, typically < 1 MB

Buffered with risk (candidates for future streaming). Same buffered pattern but the response size is not bounded by design. Auto-spill mitigates the context growth problem (large outputs are written to sandbox files), but the orchestrator still spikes RSS during the fetch:

  • get_pod_log() (Konflux) – resp.text per container, concatenated. Multi-container pods accumulate all logs.
  • get_mr_diff() (GitLab) – full diff JSON, scales with MR size. GitLab truncates server-side but the result can still be large.

Invariant: paginated K8s list endpoints must always use streaming. Page sizes scale with the number of components in the commit – a single page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON). Buffering these responses risks OOM under normal production workloads, especially with concurrent workflows.

10. Event Pipeline and Production Operations

10.1. Two-stage SQS pipeline (ingress router + FIFO worker)

Events flow through a two-stage pipeline to serialize per-session work:

flowchart LR
  STD["Standard SQS<br/>(ingress)"]
  R["Router"]
  FIFO["SQS FIFO<br/>(work queue)"]
  W["Worker<br/>(poll_loop)"]

  STD -->|ReceiveMessage| R
  R -->|"SendMessage<br/>MessageGroupId"| FIFO
  W -->|"continuation<br/>group=session_id"| FIFO
  FIFO -->|ReceiveMessage| W

Ingress router. events.router_loop() is a single-threaded loop that long-polls the standard (ingress) queue, peeks into each SNS envelope to assign a MessageGroupId, forwards the raw message body to the FIFO queue, then deletes from the standard queue.

  • Note events: MessageGroupId = discussion_id from the webhook body. This serializes replies to the same agent session – FIFO delivers at most one in-flight message per group.
  • Pipeline / MR events: MessageGroupId = SQS MessageId (unique per message). No serialization – each event is its own group.
  • MessageDeduplicationId: Always the SQS MessageId from the standard queue. Absorbs duplicate deliveries from standard SQS’s at-least-once semantics (5-minute dedup window).

If SendMessage to FIFO fails, the message is not deleted from the standard queue and retries via visibility timeout.

FIFO worker. events.poll_loop() long-polls the FIFO queue, decodes SNS envelopes, and dispatches to handle_event. The max_concurrent_agents semaphore caps parallel groups being processed. FIFO guarantees that within a group, only one message is in-flight.

Two-phase processing. When the FIFO worker picks up a webhook (Phase 1), the handler validates the event, resolves the session_id, then posts a continuation message back to the same FIFO with MessageGroupId = session_id. The Phase 1 message is deleted quickly (~1-5s). Phase 2 processes the continuation: loading session state, building WorkflowRequest, and calling _execute_workflow. Because Phase 2 continuations share a MessageGroupId per session, all work for a given session is serialized – even if it spans multiple GitLab discussion threads.

Uniform message format. Both webhook messages (from SNS) and continuation messages use the same SNS-style envelope with gzip+base64 compression. encode_envelope() is the inverse of decode_sns_message(). The consumer doesn’t need to distinguish between webhook and continuation messages at the transport layer.

Graceful shutdown. SIGTERM/SIGINT set a shutdown_event. Both loops exit. Messages in SQS become visible again after the visibility timeout for other consumers.

Why two queues. The standard queue receives events from SNS (via subscription). The FIFO queue serializes per-session work. The gitlab-event-forwarder and SNS subscription are unchanged – the routing logic lives entirely in the agent codebase.

10.2. SNS envelope decoding

Messages arrive as SNS notifications with:

  • MessageAttributes.source – event source (e.g., "gitlab")
  • MessageAttributes.event_type – event type (e.g., "pipeline", "merge_request", "note")
  • MessageAttributes.content_encoding – "gzip+base64" for gitlab-event-forwarder, absent for kubernetes-event-forwarder
  • Message – the actual event body (JSON string, or gzip+base64 encoded)

decode_sns_message() handles both encoding formats transparently.

10.3. Event routing

runner.handle_event() dispatches on (source, event_type):

flowchart TD
    Event["Event arrives"]
    Check{"source/type?"}
    Cont["handle_continuation()"]
    Pipeline["handle_pipeline()"]
    MRHandler["handle_merge_request()"]
    Note["handle_note()"]
    Ignore["Ignore"]

    Event --> Check
    Check -->|"agent/continuation"| Cont
    Check -->|"gitlab/pipeline"| Pipeline
    Check -->|"gitlab/merge_request"| MRHandler
    Check -->|"gitlab/note"| Note
    Check -->|"other"| Ignore

    Pipeline --> FilterSource{"source = merge_request_event?"}
    FilterSource -->|"yes"| AutoResolve["auto-resolve on success<br/>(unconditional)"]
    FilterSource -->|"no"| Skip1["Skip"]
    AutoResolve --> PipeRules["evaluate trigger_rules<br/>(status + user + branch + title)"]
    PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
    PipeRules -->|"deny"| Skip1a["Skip"]

    MRHandler --> FilterMRAction{"action in open/update?<br/>draft = false?"}
    FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules<br/>(user + branch + title)"]
    FilterMRAction -->|"no"| Skip1b["Skip"]
    MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
    MRRules -->|"deny"| Skip1c["Skip"]

    Note --> FilterMRNote{"MR note?<br/>action = create?"}
    FilterMRNote -->|"yes"| NoteType
    FilterMRNote -->|"no"| Skip3["Skip"]

    NoteType{"Note type?"}
    NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
    NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
    NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()<br/>→ _enqueue_continuation()"]
    NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
    NoteType -->|"other"| Skip5["Skip"]

    FindSession --> Found{"session found?"}
    Found -->|"yes"| Enqueue3["_enqueue_continuation()<br/>(resume_session)"]
    Found -->|"no"| Skip6["Skip"]

    Cont --> ContType{"work_type?"}
    ContType -->|"new_session"| ExecNew["_execute_workflow()"]
    ContType -->|"resume_session"| LoadS3["load session from S3<br/>→ _execute_workflow()"]

Phase 1 handlers (handle_pipeline, handle_merge_request, handle_note) perform validation, auth checks, and session resolution, then post a WorkOrder continuation message via _enqueue_continuation(). They do not call _execute_workflow directly.

Phase 2 (handle_continuation) receives the WorkOrder, resolves the workflow config, builds a WorkflowRequest, and calls _execute_workflow. For resume_session orders, it loads the previous session from S3. _execute_workflow handles SHA dedup and per-workflow rate limiting via scan_agent_threads.

Slash commands dispatch to a single named workflow via _resolve_slash_workflow (exact match, then prefix match). Help and error replies are posted via post_simple_reply (no session marker, no rate limit impact).

10.4. Pipeline trigger design

Why gitlab::pipeline instead of kubernetes::PipelineRun: A GitLab pipeline event fires once when the pipeline completes. Since Konflux external stages are attached to the pipeline, the event naturally waits for all builds and tests to finish before triggering. This means the agent sees the full picture in one event, without needing to deduplicate or wait for stragglers.

Trigger filters:

  • status in {failed} – only failed pipelines trigger analysis
  • source == "merge_request_event" – only MR pipelines, not branch/tag

10.4a. Merge request trigger design

The handle_merge_request handler fires on MR open, reopen, and update events (action in {"open", "reopen", "update"}). Draft MRs are skipped (object_attributes.draft == True); marking a draft as ready triggers a review since the event arrives with draft: false.

Event classification is deliberately simple: the handler does not inspect oldrev or changes.draft fields. Instead, SHA-based deduplication in _execute_workflow (via JSON session markers) ensures each code revision is reviewed at most once. Metadata-only updates (title, label changes) on an already-reviewed SHA are silently skipped.

10.5. Note trigger design

Two sub-flows:

Slash command (/hummingbird <workflow-name>): Triggers a specific named workflow, bypassing rate limiting. _resolve_slash_workflow performs exact-match lookup first, then falls back to unique prefix matching. If the subcommand is missing or is “help”, _format_help returns a list of available workflows with descriptions. Ambiguous or unknown subcommands produce an error message via post_simple_reply. The note author must have Developer+ access (level >= 30) on the project, checked via check_member_access(). If denied, the agent replies in the same discussion thread with a short access-denied message.

Reply to agent note: When a user replies to an existing agent note (which contains a JSON session marker):

  1. find_session_for_reply() walks the discussion thread, filtering by bot author ID (get_bot_user_id()), and returns the first matching session marker. All bot markers in a thread share the same session ID.
  2. handle_note posts a resume_session continuation to the FIFO (grouped by session_id).
  3. handle_continuation loads the session from S3 (conversation history and sandbox archive), builds a WorkflowRequest, and calls _execute_workflow() to resume with the user’s reply.
  4. If the session is not found (expired/deleted), the agent replies with a message explaining the session has expired and suggesting to start a new run.

Reply authors are also subject to the Developer+ access check. If denied, the agent replies in the discussion thread with the same access-denied message.

Non-agent-directed notes: Regular comments that are neither slash commands nor replies to agent threads are silently ignored (debug-level log). No access check is performed for these.

Self-note filtering: Notes where author_id matches the bot’s own user ID (resolved via get_bot_user_id()) are skipped immediately. This prevents infinite loops where the agent’s own output triggers another agent run. The author-ID check replaces the previous substring check (marker_prefix_str in note_body), which was vulnerable to denial-of-service: a user including the marker prefix in their reply would cause the bot to silently ignore it.

Threading logic for replies: If the project requires internal notes (internal_notes: true) but the original discussion was public, the reply is posted as a new top-level internal note instead of replying in the public thread. This prevents leaking internal analysis into public threads.

10.6. Note lifecycle

1. create_placeholder_note()   -> "Running the <workflow> workflow..."
                                   + JSON session marker (id, wf, sha)
2. run_workflow()              -> agent loop
3. update_note()               -> replace placeholder with result
                                   + reply prompt + JSON session marker

Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.

On failure, the placeholder is updated to “Hummingbird analysis failed.” The JSON session marker is always present so the note can be identified as agent-generated (for rate limiting, SHA dedup, and self-note filtering). The marker embeds the workflow name and commit SHA, enabling per-workflow rate limiting and SHA-based deduplication.

10.7. Rate limiting and SHA deduplication

scan_agent_threads() performs a single pass over MR discussions, parsing JSON session markers (<!-- hummingbird-session: {"id":...,"wf":...,"sha":...} -->) to determine:

  1. Per-workflow thread count – how many threads the given workflow has created on this MR. If the count meets or exceeds max_runs_per_mr, the workflow is skipped.
  2. SHA dedup – whether the current commit SHA has already been reviewed by this workflow. If so, the workflow is skipped (prevents re-reviewing the same code on metadata-only MR updates).

Slash commands and replies bypass both checks entirely.

The rate limit is per-workflow per-MR: different workflows maintain independent thread counts on the same MR. JSON markers are backward compatible – old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} with no workflow or SHA information, so they are not counted toward any specific workflow’s limit.

11. Session System

Sessions enable conversation continuity: a user can reply to an agent note and the agent picks up where it left off, with full context and sandbox files restored.

11.1. What gets saved

Three artifacts are saved after each run:

  • context.json – the full contents list from the agent loop in the canonical provider-independent format. It is the minimum state needed to resume the conversation; format conversion is handled by the AI library.
  • transcript.md – a human-readable markdown rendering of the run for debugging and auditing. Not used for resumption.
  • sandbox.tar.gz – an archive of /tmp/data/ (the sandbox working directory, which includes the _out/ spill subdirectory). Contains PipelineRun JSONs, test logs, jq output, and any other files the model created during the run. Restored into the new sandbox on resumption so the model can reference its previous work.

11.2. Storage backends

S3 (production):

s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz

Local directory (development, --save-session):

{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz

Both backends have the same interface. S3 save is best-effort: wrapped in try/except so a transient S3 error does not prevent the MR note from being posted. The note always gets delivered first.

11.3. Sandbox archive transport

The sandbox archive requires special handling because the orchestrator cannot directly access the sandbox filesystem (no volume mounts):

Archive (inside sandbox):
  sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")

Stream out (sandbox -> host temp file):
  for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
      tmp.write(chunk)                     # 64 KB chunks, no base64

Restore (host temp file -> new sandbox):
  sb.stream_exec("tar xzf - -C /tmp", file_chunks())

Both directions use streaming binary I/O via Popen stdin/stdout pipes. No base64 encoding is needed: read_file_iter() yields raw bytes from the sandbox via a Popen stdout pipe, and stream_exec() feeds raw bytes into a Popen stdin pipe. The archive is never extracted on the orchestrator – it exists only as opaque bytes being transported between sandboxes. This is a security invariant: the orchestrator never parses or inspects the archive contents.

11.4. Resumption flow

flowchart TD
    Reply["User replies to agent note"]
    FindSession["find_session_for_reply()<br/>walks discussion thread"]
    LoadS3["load_s3(session_id)"]
    NotFound{"session found?"}
    Expired["Post session-expired reply"]
    NewSandbox["Start new sandbox"]
    Restore["restore_sandbox(archive)"]
    BuildContents["contents = old_contents + user_reply"]
    RunLoop["run_agent_loop(<br/>initial_contents, CONTINUATION_PROMPT)"]

    Reply --> FindSession --> LoadS3
    LoadS3 --> NotFound
    NotFound -->|"no"| Expired
    NotFound -->|"yes"| NewSandbox
    NewSandbox --> Restore --> BuildContents --> RunLoop

Key aspects:

  • Fresh iteration budget. The resumed session starts iteration 0 with the full max_iterations budget, regardless of how many iterations the previous session used.
  • Context window is the real limit. A typical 8-iteration cold start uses ~50-100K input tokens. The restored history is sent in full on every API call. The CONTEXT_LIMIT check applies and will force wrap-up if needed.
  • Session ID reuse. The resumed session keeps the original session ID. S3 state is overwritten in place with the updated conversation history. Since GitLab discussions are linear (not branching), there is no need for a tree of sessions – the conversation is a single sequential thread.
  • Graceful degradation. If the S3 session is not found (expired, deleted), the agent posts a reply explaining the session has expired and suggesting to start a new run. It does not fall back to a cold start.

11.5. Session markers

Every agent note contains a hidden HTML comment with a JSON payload:

<!-- hummingbird-session: {"id":"UUID","wf":"code-review","sha":"abc123"} -->

The JSON payload contains:

  • id – session UUID (always present)
  • wf – workflow name (present for new-format markers)
  • sha – commit SHA at the time of review (present for auto-triggered runs)

This marker serves four purposes:

  1. Resumption: find_session_for_reply() searches discussion threads for this marker to find the session ID. Only bot-authored notes are considered; the first matching marker wins.
  2. Per-workflow rate limiting: scan_agent_threads() counts discussion threads for a specific workflow using the wf field. Only bot-authored notes are scanned.
  3. SHA deduplication: scan_agent_threads() checks whether the current SHA has already been reviewed by the workflow using the sha field.
  4. Self-filtering: handle_note() compares the webhook event’s author_id against the bot’s own user ID (via get_bot_user_id()) to prevent infinite loops.

All marker-scanning functions resolve the bot user ID via gl.auth() (GET /user) on the orchestrator token. This ensures markers in non-bot notes (user replies, external contributors) are never parsed, preventing session hijacking via injected markers.

Old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} for backward compatibility. They are counted for resumption but not for per-workflow rate limiting or SHA dedup (no wf/sha information).

12. Workflow System

12.1. Separation of prompt and metadata

A workflow has two parts that live in different places:

  • Prompt (.md file): the LLM system prompt, verbatim. This is the investigation strategy, tool usage guidance, data patterns, and output format. Pure text, no code.
  • Metadata (YAML config): operational settings – action, model, max_iterations, data_sources, projects. This controls what the orchestrator does with the workflow, not what the model does.

This separation means changing the analysis strategy (e.g., adding a new investigation step) requires editing a markdown file. Changing operational parameters (e.g., which projects use this workflow, which model to use) requires editing the YAML config. Neither requires a code change.

12.2. System prompt layering

The system prompt sent to the model is built from three layers:

BASE_SYSTEM_PROMPT        # agent.py: sandbox rules, tool usage tips
+ tool_notes              # from ToolRegistry: per-data-source domain knowledge
+ workflow_body           # full content of e.g. workflows/analyze-failures.md

On session resumption, a fourth layer is appended:

+ CONTINUATION_PROMPT     # agent.py: "do not re-run the full workflow"

The system prompt is rebuilt every iteration (to allow warning suffixes to be appended), but the base content is stable. Warning suffixes are appended at the end so they override earlier instructions.

12.3. Design choice: strategy, not procedure

The workflow .md file describes strategy and guidance, not a rigid script. The model decides when and how to use each tool based on what it sees.

This matters because merge request failures are diverse. A fixed procedure would either miss edge cases (e.g., build failures before tests ran) or waste iterations on steps that don’t apply. By giving the model a strategy (“identify failing tests, fetch details, analyze root causes, group by similarity”), it can adapt to whatever it encounters.

The workflow does structure the investigation into phases (Data Collection, Analysis, Output) for clarity, but these are guidelines, not enforced checkpoints.

12.4. Workflow file anatomy (analyze-failures.md)

The primary workflow is structured as:

  1. Scope and approach – what this workflow analyzes and what it ignores
  2. Input – event JSON format (project, iid)
  3. Data Collection Phase:
    • Data.1: Fetch MR details (direct call, small response)
    • Data.2: Fetch commit statuses, identify failures
    • Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
    • Data.4: Process with jq, extract Testing Farm data in bulk
  4. Analysis Phase:
    • Analysis.1a: Investigate each failed PipelineRun individually
    • Analysis.1b: Summarize and group by root cause
  5. Output – markdown template with root causes, collapsible details, clickable links (PipelineRun, Testing Farm, test logs)
  6. Error Handling – partial report philosophy

Design considerations embedded in the workflow:

  • Efficient bulk fetching: PipelineRuns and TaskRuns are fetched in one fetch_batch_to_sandbox call, not individually.
  • Testing Farm data extracted in Data.4, analyzed in Analysis.1: All TF results.xml are fetched in bulk before analysis starts, enabling cross-failure pattern detection.
  • Log fetching is selective: Only 1-2 representative logs per failure pattern, not all logs. This keeps iteration count bounded.
  • Reviewer-facing URLs: The output template instructs the model to include clickable links using konflux_ui and artifacts_base from response metadata.

12.5. Prompt file resolution

get_prompt(workflow_cfg, config_dir) resolves the prompt: field relative to the config file’s directory:

prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md

In the container image, workflows are baked in at /app/workflows/. A ConfigMap can override them at deploy time by mounting at the same path.

13. Deployment and Container

13.1. Container build strategy

The Containerfile uses an all-RPM builder+installroot+scratch pattern (no pip, no venv):

  1. Builder stage: Uses a Fedora-based builder image with dnf. Installs all dependencies as RPMs into a clean --installroot.
  2. Application code: Copies hummingbird_agent/ and workflows/ into the installroot at /app/.
  3. Final stage: FROM scratch, copies the entire installroot. No package manager, no shell beyond what RPMs provide.

RPM dependencies: python3-boto3, python3-google-auth+requests, python3-gitlab, python3-kubernetes, python3-pyyaml, python3-requests, python3-sentry-sdk, kubernetes1.35-client (for kubectl).

This approach was chosen over pip because:

  • All deps come from Fedora’s package repository – no PyPI supply chain risk
  • Smaller image (~22 MB saved vs. google-genai SDK alone)
  • Reproducible builds from known RPM versions
  • No compilation step (no gcc/python3-devel in the image)

13.2. Container runtime properties

CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532

The default CMD runs serve mode for production. Local development uses run mode via explicit command override. USER 65532 matches the standard nonroot UID used by distroless images and the Podman sandbox.

13.3. K8s deployment manifests

Located in hummingbird-agent/kubernetes/:

deployment.yaml:

  • Single replica with rolling update (maxSurge: 1, maxUnavailable: 0)
  • terminationGracePeriodSeconds: 900 (15 minutes) to allow in-flight agent runs to complete on shutdown
  • ServiceAccount: hummingbird-agent
  • Secrets mounted from K8s Secret hummingbird-agent
  • Commented-out mounts for workflow ConfigMap and custom CA trust

rbac.yaml (applied in the sandbox namespace):

  • ServiceAccount hummingbird-agent
  • Role with pods: create, get, list, delete, patch and pods/exec: create
  • RoleBinding linking the SA to the Role
  • patch is required for pool mode (relabeling pods during claim)

sandbox-pool.yaml (applied in the sandbox namespace):

  • Deployment with replicas: 3 (tuned to balance latency vs cost)
  • Pods labelled hummingbird/role: standby for pool discovery
  • Same security context and resource limits as direct-creation pods
  • revisionHistoryLimit: 2, rolling update with maxSurge: 1, maxUnavailable: 0

networkpolicy.yaml:

  • podSelector: {} – applies to all pods in the namespace
  • egress: [] – deny all outbound traffic

secret.yaml:

  • Template with all required env vars (API keys, tokens, URLs)
  • Values must be populated per deployment

13.4. Workflow mounting

Workflows are baked into the image at /app/workflows/. To update workflows without rebuilding the image:

  1. Create a ConfigMap from the workflows directory
  2. Uncomment the volume mount in the deployment YAML
  3. The ConfigMap mount replaces the baked-in directory (all-or-nothing)

This is useful for rapid iteration in staging without waiting for a new image build.

13.5. Custom CA trust

For clusters with internal CA certificates (common in enterprise environments), the deployment supports OpenShift’s CA injection:

  1. Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle label
  2. Mount it at /etc/pki/custom
  3. Set REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt

OpenShift automatically injects the cluster’s CA bundle into the ConfigMap.

4.5.20.7 - Hummingbird Agent Design — Decisions

Design Decision Registry (DDR) for the Hummingbird Agent — every major architectural choice, the alternatives considered, why the chosen approach won, and what would break if reversed. For the full design context, see Hummingbird Agent Design. Provider adapter decisions are in Hummingbird AI Design.

Design Decisions

kubectl exec over kubernetes Python exec API

  • Chosen: kubectl exec as subprocess for sandbox command execution.
  • Alternative: kubernetes Python client stream() API.
  • Why: Three showstopper bugs in the Python client: (1) WebSocket v1-v4 has no stdin EOF signal, so cat > /file hangs forever; (2) BrokenPipeError on stdin larger than ~1MB; (3) stream() accumulates all stdout in memory with no control. kubectl avoids all three and provides the same subprocess interface as Podman.
  • If reversed: write_file() would hang or fail on large data. Pod log retrieval with large outputs would OOM. Sandbox reliability would drop significantly.

YAML config over env vars for operational settings

  • Chosen: Single YAML file for workflows, projects, limits, data source declarations.
  • Alternative: Everything in env vars (original design).
  • Why: Env vars cannot express structured data (workflow-project mappings, per-project token overrides, data source config with multiple fields). YAML provides structure, validation, and audit trails.
  • If reversed: Token scoping would be lost (no per-workflow-project token overrides). Project allowlists would be impossible. The config would be unauditable.

Token env var names in YAML, not values

  • Chosen: YAML contains token_env: GITLAB_TOKEN_RO (the env var name), not the actual token value.
  • Alternative: Inline secrets in YAML, or env vars for everything.
  • Why: The YAML file can be committed, reviewed, and audited. Actual secrets stay in env vars (injected via K8s Secrets). Inline secrets would make the config file a secret itself.
  • If reversed: The config file would become a secret, breaking audit trails and code review workflows.

Orchestrator token prefix convention (ORCHESTRATOR_*)

  • Chosen: Orchestrator tokens use ORCHESTRATOR_GITLAB_TOKEN_* env var names by convention.
  • Alternative: Same env var namespace as model tokens, distinguished by context.
  • Why: Structural separation makes it impossible to accidentally pass an orchestrator token to a model tool (or vice versa). The ORCHESTRATOR_ prefix is never used in YAML token_env fields.
  • If reversed: A misconfiguration could leak write-capable tokens to the model, which could then expose them via tool calls.

Auto-spill over conversation compaction

  • Chosen: Large outputs are saved to sandbox files with previews returned to the model.
  • Alternative: Replace old tool results in contents with compact summaries (conversation compaction).
  • Why: Compaction requires modifying the contents list, which risks violating provider conversation constraints. Auto-spill achieves ~90% of the token reduction without modifying model history. See Hummingbird AI for adapter format requirements.
  • If reversed: Token usage would increase ~15-30%. Per-call context would grow unbounded. Sessions would hit the context limit much sooner. Conversation compaction could be added on top of auto-spill in the future, but is not needed with current workloads.

Iteration count over cumulative token budget

  • Chosen: max_iterations as the primary cost control lever.
  • Alternative: Cumulative token budget (stop when total billed tokens exceed a threshold).
  • Why: With full history replay, each API call re-sends everything. Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K (including 5K again), total = 15K billed but only 10K new content. With auto-spill keeping per-call size bounded, iteration count is a much simpler and more predictable proxy for actual cost.
  • If reversed: The budget model would be confusing and inaccurate. Cost estimates would be wrong. The cumulative metric is still logged for observability, but it does not drive termination.

ToolDef notes in system prompt, not tool schema

  • Chosen: Domain knowledge (Konflux dual-fetch, TF XML structure) goes in ToolDef.notes, injected into the system prompt.
  • Alternative: Put everything in the tool schema description.
  • Why: Tool schemas have character limits and are sent in the tools field of every API call. Long descriptions waste tokens on tool defs. System prompt notes are sent once and can be arbitrarily detailed.
  • If reversed: Tool descriptions would be bloated. Domain knowledge would need to be duplicated in every workflow .md file.

Full history replay (no compaction)

  • Chosen: The contents list grows monotonically. Items are never removed or modified (except empty response retries).
  • Alternative: Compact old turns to reduce context size.
  • Why: Provider APIs impose strict conversation structure. Modifying or removing items risks invalid history. Auto-spill handles the growth problem at the source (preventing large items from entering history).
  • If reversed: Risk of provider API errors from malformed conversation structure and of confusing the model with summarized results.

fetch_to_sandbox kept alongside auto-spill

  • Chosen: Both fetch_to_sandbox and auto-spill coexist.
  • Alternative: Remove fetch_to_sandbox since auto-spill handles large outputs automatically.
  • Why: fetch_to_sandbox provides explicit path control (model can choose meaningful filenames), batch fetching (one tool call for multiple fetches), and no-preview responses (useful when the model will process with jq anyway).
  • If reversed: The model would lose path control and batch fetching would require multiple auto-spilled calls. The workflow would need more iterations to achieve the same result.

K8s library for lifecycle + kubectl for exec (hybrid)

  • Chosen: Use the kubernetes Python library for pod create/read/delete, kubectl subprocess for exec.
  • Alternative: All-kubectl (subprocess for everything) or all-library.
  • Why: The library provides typed pod status polling and clean error handling for lifecycle. kubectl provides reliable exec (see “kubectl exec over kubernetes Python exec API” above). Using the library for exec would require solving the three WebSocket bugs.
  • If reversed: Either lifecycle management would be fragile (parsing kubectl JSON output for pod status) or exec would be unreliable.

Single-file YAML config over multiple config files

  • Chosen: Everything in one config.yaml with settings and workflows sections.
  • Alternative: Separate files per workflow, or settings.yaml + workflows.yaml.
  • Why: Single source of truth. All project-workflow mappings visible in one place. Settings-level defaults flow down to all projects. No file discovery logic needed.
  • If reversed: Token resolution would need cross-file lookups. Project index would need multi-file aggregation. Config validation would be more complex.

Workflow .md as pure prompt, metadata in YAML

  • Chosen: The workflow .md file is pure system prompt text. Metadata (action, model, max_iterations, data_sources, projects) lives in the YAML config.
  • Alternative: YAML frontmatter in the .md file (original design).
  • Why: Separation of concerns. The .md file is the model’s instructions – it should be readable and editable by anyone writing prompts. The YAML config is the orchestrator’s instructions – it controls routing, limits, and credentials. Mixing them in one file conflates two audiences.
  • If reversed: Prompt authors would need to understand YAML config structure. Token/project config would be scattered across .md files instead of centralized.

Nudge via system prompt suffix, not user message

  • Chosen: Empty response nudge and budget warnings are appended to the system prompt as suffixes.
  • Alternative: Inject synthetic user messages into contents.
  • Why: User messages in contents must come from the actual user (the initial event, or a reply). Injecting synthetic messages pollutes the conversation history that is saved in sessions. System prompt suffixes are transient – they affect one API call without permanently modifying the conversation state.
  • If reversed: Saved sessions would contain synthetic user messages. Resumed conversations would be confusing. The model might respond to the synthetic messages instead of the user’s actual input.

Session ID reuse over new-ID-per-resume

  • Chosen: Resumed sessions keep the original session ID. S3 state is overwritten in place.
  • Alternative: Each resume generates a new UUID, saving alongside the original (append-only tree of sessions).
  • Why: GitLab discussions are linear, not branching. There is no scenario where two different sessions from the same thread are both valid. A new UUID per resume creates orphaned S3 snapshots that are never referenced again, since find_session_for_reply() always picks the latest marker. Reusing the ID is simpler, uses less storage, and matches the linear conversation model.
  • If reversed: S3 would accumulate orphaned session snapshots. Each resume would need a new note marker, but the old markers would still be in the thread, creating confusion about which session is current.

Reply to unauthorized agent-directed notes instead of silent skip

  • Chosen: When an unauthorized user sends a slash command or replies to an agent thread, the agent replies in the same discussion with a short access-denied message. Non-agent-directed notes are silently ignored (debug log, no access check).
  • Alternative: Log a warning and skip silently for all unauthorized notes (the original behavior).
  • Why: Silent skip gives no feedback to someone who intentionally tried to engage the agent, which is confusing. On the other hand, checking access and logging a warning for every random comment on a public project is noisy and pointless. Moving the access check after intent detection cleanly separates the two cases. The reply uses the discussions API so it automatically inherits confidentiality from the parent note/thread.
  • If reversed: Users without sufficient access who try /hummingbird would get no feedback. The agent log would be noisy with warnings for every comment on public MRs.

Deployment-backed pod pool over Python-thread pool

  • Chosen: A Kubernetes Deployment maintains pre-warmed standby pods. The agent claims a pod by relabeling it; the Deployment replaces it.
  • Alternative: A Python-side thread pool that pre-creates pods and queues them for use.
  • Why: The Deployment provides self-healing (restart crashed pods), native scaling (kubectl scale), rolling updates (image changes), and monitoring via standard K8s tooling. A Python pool would need to reimplement all of these.
  • If reversed: Pod replenishment, crash recovery, and image updates would all need custom code. Scaling would require agent redeployment.

Pod isolation: never reuse sandbox pods

  • Chosen: Each workflow run gets its own pod. After cleanup, the pod is deleted. Claimed pods are detached from the ReplicaSet.
  • Alternative: Return used pods to the pool and reset them.
  • Why: Residual state from a previous run (files, environment, running processes) could leak between MR investigations, creating security and correctness risks. Deletion is simple and foolproof.
  • If reversed: Would need a reliable pod-reset mechanism and auditing that no state survives between runs.

Absolute reap-by deadline over relative claimed-at age

  • Chosen: Active pods are annotated with hummingbird/reap-by (an absolute UTC timestamp). The reaper deletes pods where now > reap-by.
  • Alternative: Annotate with claimed-at and compute age relative to max_active_seconds; or use activeDeadlineSeconds on the pod spec.
  • Why: An absolute deadline simplifies the reaper to a single comparison. It also supports varying deadlines: claim sets reap-by = now + max_active_seconds, while linger sets reap-by = now + linger_seconds. With a relative timestamp, the reaper would need to know which mode the pod is in. activeDeadlineSeconds applies from pod creation, not from claim – standby pods would expire before being used.
  • If reversed: The reaper would need mode-aware age calculations. Lingering pods would require a separate annotation or reaper path.

Indefinite claim wait over timeout

  • Chosen: SandboxPool.claim() polls indefinitely (governed by shutdown event), logging at DEBUG then WARNING.
  • Alternative: Timeout after N seconds and raise an error.
  • Why: The pool is typically smaller than max_concurrent_agents for cost reasons. Waiting for a replacement pod is normal operational behavior, not an error. A timeout would cause spurious failures during burst traffic. The SQS semaphore already bounds concurrency.
  • If reversed: Burst traffic would cause avoidable failures instead of brief delays.

Pool config in Deployment only, not in agent configmap

  • Chosen: In pool mode (k8spool), the pod image, resources, labels, and security context are defined solely in the Deployment template. The agent config only needs namespace and active_deadline_seconds.
  • Alternative: Keep image/resources/metadata in the agent configmap too (as in k8s mode).
  • Why: Eliminates config duplication. The Deployment template is the single source of truth. Changes to pod resources or image only require updating one place and re-rolling the Deployment.
  • If reversed: Config drift between the Deployment and the agent configmap would be a constant risk.

Pod lingering over immediate cleanup for reply latency

  • Chosen: After a successful workflow, pool sandbox pods linger for linger_seconds (default 300) with a session-id annotation. Replies within the window reclaim the pod via try_reclaim(), skipping pod creation and S3 archive restoration.
  • Alternative: Always delete the pod immediately and restore from S3 on every reply.
  • Why: Reply latency drops from seconds (pod claim + S3 restore) to near-zero. The S3 archive is still saved as a fallback if the pod is gone. The reap-by annotation ensures lingering pods are cleaned up if no reply arrives. The session-id annotation doubles as a concurrency guard: it is only present while lingering, preventing active pods from being reclaimed.
  • If reversed: Every reply would pay full pod + restore latency, even for immediate follow-ups. User experience for conversational interactions would degrade noticeably.

linger() on Sandbox protocol over isinstance checks in runner

  • Chosen: All sandbox backends implement linger(session_id). Non-pool backends fall back to cleanup(). The runner calls sb.linger() without type checks.
  • Alternative: isinstance(sb, K8sPoolSandbox) in run_workflow().
  • Why: Keeps the runner backend-agnostic. Adding a new backend requires only implementing the protocol, not touching the runner. The fallback behavior is co-located with each backend.
  • If reversed: The runner would need to know about every backend type and their linger capabilities.

GCP SA key over Workload Identity Federation

  • Chosen: GCP service account key stored in Vault, rotated via cki-tools credential manager (prepare/switch/clean cycle).
  • Alternative: Workload Identity Federation (WIF) with projected SA token. Two sub-options: (a) automatic OIDC discovery if the cluster issuer is public, (b) manual JWKS upload for internal issuers.
  • Why: mpp-prod’s OIDC issuer is https://kubernetes.default.svc (internal, not publicly reachable), so GCP STS cannot discover it automatically. Manual JWKS upload works but requires re-upload after SRE-triggered signing key rotations. SA key integrates with the existing credential manager rotation infrastructure (same pattern as AWS keys and GitLab tokens), needs no OIDC reachability, and enables automated validate/update via google-auth in CI. The application code (hummingbird_ai.http.VertexAuth / google.auth.default()) works identically with both approaches – switching to WIF later requires only infrastructure changes.
  • If reversed: Replace SA key with WIF projected token + credential config ConfigMap. The gcp_service_account_key token type in cki-tools would no longer be needed for this use case.

Developer+ trust filtering over unfiltered or sanitized discussion comments

  • Chosen: gitlab_get_mr_discussions checks each note author’s project access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed placeholder in mixed-trust discussions, or the entire discussion is dropped if all notes are untrusted. Agent notes are always trusted (detected by session marker) but have transcripts and metadata stripped.
  • Alternatives considered:
    • (a) Include all comments unfiltered. Simplest, but any external contributor can craft comments that manipulate the model’s review output (prompt injection).
    • (b) Sanitize/escape untrusted content (strip markdown, quote as code blocks, prefix with “[external]”). There is no reliable escaping mechanism for LLM prompts – the model interprets natural language regardless of formatting. Escaping gives a false sense of security.
    • (c) Only include agent-authored notes (skip all human comments). Safe, but defeats the purpose: the model would never see developer responses to its own findings.
    • (d) Include only discussions where the agent participated. Better, but still misses developer-initiated review threads that provide relevant context.
  • Why: Developer+ is the same threshold used for pipeline trigger authorization (invariant #8) and slash command access. It matches the trust boundary already established: people who can push code and approve MRs are trusted to provide review context. The placeholder approach preserves discussion structure (the model sees that someone replied) without exposing the content. Full-drop for all-untrusted discussions avoids noise from discussions that contain zero useful context.
  • If reversed: (a) opens a prompt injection vector on any project that accepts external MRs. (b) provides no actual protection. (c) makes follow-up reviews unable to see developer explanations, causing repeated false positives. (d) misses developer-initiated context.

Bot-author filtering for session marker parsing over unfiltered note scanning

  • Chosen: find_session_for_reply(), scan_agent_threads(), and the handle_note() self-filter all resolve the bot user ID via get_bot_user_id() (which calls gl.auth() on the orchestrator token) and only consider notes where author.id matches. In find_session_for_reply(), the first matching marker wins.
  • Alternative: Parse markers from any note and keep the last match; substring self-filter in handle_note (previous behavior).
  • Why: The previous unfiltered approach allowed session hijacking: a user reply or agent review prose containing an example marker (<!-- hummingbird-session: ... -->) was parsed as a real session, causing “session expired” errors in production. For handle_note, the substring check caused the reverse problem: a user embedding the marker prefix in a reply silently suppressed the event (denial of service). The gl.auth() call is one GET /user per invocation – negligible cost. _get_client_with_bot_id() returns both the authenticated client and the bot user ID, avoiding redundant client construction.
  • If reversed: Any MR participant can inject a marker to hijack the session ID or inflate rate-limit counts. The agent’s own review prose containing example markers causes spurious “session expired” messages.

ijson streaming over buffered JSON for K8s list responses

  • Chosen: session.get(stream=True) + ijson.parse(resp.raw) for iter_paginated(). Items are yielded one at a time via ObjectBuilder; the metadata.continue pagination token is captured in the same parse pass. resp.raw.decode_content = True is required because Kubearchive returns gzip Content-Encoding; without it ijson sees compressed bytes.
  • Alternative: session.get().json() loads the full page into memory (the original implementation).
  • Why: A single K8s list page can contain 500 PipelineRuns (80+ MB JSON). Parsing this with .json() creates a +247 MB RSS spike (raw bytes + decoded string + parsed dict coexist). With 4 concurrent workflows, this exceeds any reasonable pod memory limit. ijson stream-parsing yields items one at a time, reducing the peak to +12 MB for the same data – a 95% reduction. The continue token appears in the metadata object (before or after items depending on the server); the event-driven parser captures it regardless of order.
  • If reversed: Large MRs (100+ components) would OOM the agent pod. Concurrent workflows would multiply the problem. The pod memory limit would need to scale with the largest possible page size, which is unbounded.

Action token tier for per-workflow identity

  • Chosen: A dedicated Tier 2 of workflow action tokens, separate from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets a dedicated GitLab bot user per project for all write operations (notes, thread resolution).
  • Alternative: Extend orchestrator tokens with a per-workflow fallback chain (ORCHESTRATOR_<WORKFLOW>_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN).
  • Why: Orchestrator tokens serve a fundamentally different purpose (operational reads + infrastructure notes) than workflow output (analysis results, review comments). Mixing them in one fallback chain risks a single bot identity posting notes on behalf of multiple workflows, eliminating the audit trail. A separate tier with explicit token: str parameters makes the code path unambiguous: callers must choose which tier they use. The naming convention (HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>) groups tokens alphabetically by workflow, making env var auditing straightforward.
  • If reversed: All workflow notes would appear from the same bot user, making it impossible to distinguish which agent (code-review vs analyze-failures vs renovate-babysit) produced a given note in the MR timeline. GitLab’s audit log would attribute all actions to one identity.

Per-project-only action token config (no workflow-level default)

  • Chosen: action_tokens is defined at the project level only in YAML config. No workflow-level action_tokens default is supported.
  • Alternative: Allow workflow-level action_tokens that projects inherit by default, with per-project overrides.
  • Why: GitLab project access tokens are project-scoped. A workflow-level default would be misleading – a single token cannot write to multiple projects. Forcing per-project declaration makes the mapping explicit and prevents configuration errors where a cross-project default silently fails.
  • If reversed: Operators could set a workflow-level token that only works for one project, causing silent 403 errors for other projects in the same workflow.

SSH over kubectl exec for KubeVirt VMs

  • Chosen: SSH (ssh -T sandbox@ip command) for VM command execution.
  • Alternative: virtctl console (serial console), KubeVirt websocket API, or qemu-guest-agent exec.
  • Why: VMs have no kubectl exec equivalent. virtctl console provides a serial terminal, not a programmable exec channel. The websocket API requires a VNC or serial connection. qemu-guest-agent is not installed in the VM image. SSH provides clean stdin/stdout/stderr separation, timeout control, and is already well-tested in subprocess pipelines.
  • If reversed: Would need qemu-guest-agent in the image and the exec endpoint, which has poorer error reporting and no streaming stdin.

Secret volume for SSH key injection

  • Chosen: Kubernetes Secret mounted as a virtio disk (serial ssh-pubkeys) read by the VM’s inject-ssh-keys.service at boot.
  • Alternative: cloudInitNoCloud or accessCredentials with qemu-guest-agent.
  • Why: The VM image already has inject-ssh-keys.service which reads from the virtio disk. cloudInitNoCloud requires cloud-init in the image. accessCredentials requires qemu-guest-agent. The Secret volume approach requires no additional software in the VM image.
  • If reversed: Would need cloud-init or guest agent in the bootc image, adding complexity and attack surface.

VMIRS for VM pool replenishment

  • Chosen: VirtualMachineInstanceReplicaSet (VMIRS) as the replenishment controller for the VM pool.
  • Alternative: Custom controller, or VirtualMachine with RunStrategy.
  • Why: VMIRS is KubeVirt’s native equivalent of a Deployment’s ReplicaSet – it maintains a desired count of identical VMIs. When a VMI is claimed (ownerReferences cleared), the VMIRS creates a replacement automatically. Same pattern as the pod pool with Deployment.
  • If reversed: Would need a custom controller to maintain the VMI pool, adding operational complexity.

Per-workflow sandbox backend

  • Chosen: sandbox: field in workflow config, resolved per-workflow with fallback to CLI --sandbox flag.
  • Alternative: Global-only sandbox selection via CLI.
  • Why: Different workflows have different isolation requirements. Code review needs only a lightweight container, while failure analysis may need a full VM with root access. Per-workflow selection avoids running all workflows in VMs (wasteful) or all in containers (insufficient isolation).
  • If reversed: All workflows would share the same sandbox backend, requiring either over-provisioning (VMs for everything) or accepting weaker isolation for some workflows.

4.5.21 - Red Hat Catalog

Web UI for browsing Hummingbird container images, security advisories, and API documentation.

Features

  • Image Catalog - Browse container images with tags, architectures, and specifications
  • Security Feed - Security advisory feed with CVE details
  • API Documentation - Interactive API documentation browser
  • Image Details - SBOM, provenance, and release history per image
  • My image requests - Signed-in users can see current and historical Jira requests associated with their email, including newly submitted items in triage, with their current status, latest note, and last-updated date
  • Analytics identity - After Red Hat SSO login, Amplitude receives a hashed user_id (issuer + subject) plus is_authenticated and email_domain. Logout resets identity. Raw email is not sent.

Architecture

React 18 + PatternFly v6 SPA. Data fetching via TanStack Query against the container-catalog API (proxied through webpack-dev-server in development, direct in production). Webpack 5 build. Hosted on CloudFront + S3 (production) with MR previews on GitLab Pages.

Prerequisites

  • Node.js 22+ (for host targets)
  • make + podman (for container targets)

Usage

Only make and podman are required for container targets. Host variants (*-host) run without podman.

Target Description
redhat-catalog/setup Install dependencies (container)
redhat-catalog/check Type-check, lint, and test (container)
redhat-catalog/build Production build (container)
redhat-catalog/dev Dev server on port 9000 (container)
redhat-catalog/setup-host Install dependencies (host)
redhat-catalog/check-host Type-check, lint, and test (host)
redhat-catalog/build-host Production build (host)
redhat-catalog/dev-host Dev server (host)

Set ASSET_PATH to control the base path for non-root deployments (used by webpack output.publicPath and React Router basename).

Configuration

Build Variables

Variable Description
ASSET_PATH Webpack public path / React Router basename

API base URLs are configured in app-config.ts.

SAM Parameters

Parameter Description
ResourcePrefix Prefix for all resource names
CatalogDomainName Custom domain (optional, leave empty for CloudFront default)

Deployment

Production deployment uses a CloudFront + S3 stack defined in template.yaml. The infrastructure post-deploy script builds the SPA in a Node.js container, syncs it to S3, and creates a CloudFront cache invalidation. Custom domain support is optional via the CatalogDomainName SAM parameter. When set, an ACM certificate is created in-template with DNS validation – the validation CNAME must already exist in the external DNS zone before deployment.

MR previews are deployed to GitLab Pages with path_prefix per MR, auto-cleaned on merge/close with a 1-week expiry.

Environments and CI promotion

Experimental, staging, and production URLs, MR vs main pipelines, deploy triggers, and UAT sign-off are documented in Red Hat Catalog Environment Promotion (monorepo documentation/) and Red Hat Catalog UAT Program.

Development

Detailed contributor documentation lives in-tree:

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.22 - Hummingbird Data Flow

Overview of all data sources, what is stored where, and how data moves between systems.


Data sources

Source What it provides
Jira REST API CVE tracker ticket fields (status, timestamps, labels, custom fields)
Jira whiteboard (customfield_10841) Per-ticket timestamp cache written back by the analysis tool
Red Hat Pulp (packages.redhat.com) SRPM/RPM publish timestamps for Hummingbird repos
Hummingbird container catalog API Image rebuild history (when a package first appeared in a rebuilt image)
Red Hat CSAF VEX feed (security.access.redhat.com) Advisory fix/not-affected status per CVE
OSIDB (osidb.prodsec.redhat.com) Subpackage-level affectedness; flaw created_dt and hummingbird-1 affect created_dt
NVD / CVE list CVE publish dates, product/version data
GitHub / GitLab Upstream fix commit / PR / release dates
Fedora updates Fedora update availability timestamps
Konflux SNS/SQS events Pipeline run, snapshot, release, MR, push events

PostgreSQL databases

There are two Postgres databases.

1. hummingbird-status — Konflux pipeline events

Fed by SNS/SQS events via the hummingbird-status ingestor.

Table Primary key What is stored
gitlab_pushes (sha, repo, ref) Git push events — sha, branch, commit list
components name Konflux component definitions and git context
pipelineruns name Build/test PLR outcomes, status, start/completion times
snapshots name Multi-component snapshots, source PLR, sha
releases name Release outcomes, images, LLM analysis text
gitlab_merge_requests (project, iid) Current MR state, merge commit sha
gitlab_mr_versions (project, iid, sha) Per-commit history of each MR head

The legacy tables above remain in public. The independent pipeline-timing service consumes a second SQS copy of each SNS envelope and writes only to its Alembic-managed pipeline_timing schema in the same database. The pipeline-timing Pod’s init container creates that schema and runs migrations before its worker container starts. The single-replica Deployment uses Recreate for upgrades. Scaling it to zero leaves the queue untouched without affecting legacy ingestion.

Timeline table Primary key What is stored
events SNS message ID Source, kind, timestamp, exact S3 bucket and key
merge_requests, commits, merge_request_commits GitLab project ID and source identities MR lifecycle and head/merge commits
gitlab_pipelines, gitlab_jobs GitLab project ID and pipeline/job ID Execution status and timing per attempt
pipeline_runs, task_runs, snapshots, releases Cluster and Kubernetes UID Konflux execution, bridge lineage, and Release completion

These tables store selected fields and event references, not raw event JSON. The archive retains the raw SNS envelope. Canonical HTTPS repository URL plus SHA links MR commits, GitLab pipelines, and Konflux PipelineRuns; project and pipeline IDs link GitLab jobs. Kubernetes owner UIDs link TaskRuns to PipelineRuns, while the Snapshot label and source run name connect build to test, and a Release names its Snapshot and managed PipelineRun. Distinct run UIDs and job IDs preserve retries. Consumers calculate elapsed intervals from the stored timestamps. The Dashboard continues to read only the public tables.

2. hummingbird-dashboard — CVE lifecycle + dashboard overlays

Table Primary key / unique What is stored
cve_ticket_events (ticket_key, event_type) One row per lifecycle milestone per ticket. occurred_at = timestamp; metadata = JSONB (see below). The source of all R-Time computations.
cve_analysis_log id Each analysis tool run: timestamp, ticket count, log output
cve_ticket_claims ticket_key Claim/lock for deduplicating concurrent analysis runs
cve_ticket_blame (ticket_key, category) unique Human-assigned R-Time blame categories (hummingbird, prodsec, konflux, testing, pulp, other); a ticket may have multiple
dashboard_settings key Runtime flags: auto-rerun enabled, analysis enabled, etc.
auto_rerun_log id History of automatic retrigger attempts
blocked_error_patterns id Regex patterns that suppress auto-rerun
blocked_snapshots snapshot_name Manually blocked snapshots
analysis_log / analysis_costs id LLM failure analysis runs and token costs
blocked_push_builds / push_build_rerun_log id Push build blocking and retry history
package_lifecycle (package, event_type) Package-scoped lifecycle milestones (e.g. rpm_first_published). Distinct from cve_ticket_events — one row per package/milestone, not per ticket.

cve_ticket_events lifecycle milestone event_type values

These are the canonical names after the HUM-5918 migration:

event_type Meaning
cve_published CVE published date (NVD / CVE list)
osidb_flaw_created OSIDB flaw created_dt (when ProdSec ingested the CVE)
osidb_affect_created hummingbird-1 affect created_dt for this ticket’s pscomponent (when a HUM tracker could be filed)
cve_embargo_lifted OSIDB flaw unembargo_dt, only when it postdates created_dt (i.e. the flaw was genuinely embargoed; omitted otherwise)
osidb_flaw_public OSIDB flaw unembargo_dt, unconditionally – when OSIDB considers the flaw public, whether genuinely embargoed or not
hum_ticket_created HUM Jira ticket creation date
hum_ticket_closed HUM Jira ticket resolution date
upstream_fix_merged Upstream fix commit / release merged
fedora_update_available Fedora update containing the fix became available
rpm_fix_published_to_pulp Fix RPM published to Hummingbird Pulp repo
image_rebuilt_on_quay Hummingbird container image rebuilt with the fix
vex_resolved Red Hat CSAF VEX advisory confirmed the resolution (HUM-5843)

rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable — they are updated when a newer delivery event supersedes the previous one. All other event types are write-once: once set, occurred_at is never overwritten.

cve_ticket_events.metadata JSONB fields

Each row carries a metadata blob that reflects the ticket’s state at the time of the most recent analysis run:

Field Source Description
computed_resolution analysis tool Hummingbird’s computed fix status
jira_status Jira status field Current Jira issue status
labels Jira labels field All labels on the ticket
fixed_in_build Jira customfield_10578 SRPM set manually in the “Fixed in Build” field
detected_fixed_build Pulp / SRPM detection SRPM filename auto-detected from Pulp repodata
vex_status Red Hat VEX feed fixed / known_not_affected / …
vex_match_state reconciliation logic matched / pending / mismatch
vex_resolved VEX reconciliation Timestamp when VEX first agreed with Jira resolution
catalog_image_source collector catalog map true when the package is a catalog image SBOM source (R-Time delivery is image publish); false means RPM publish. Missing on old rows: dashboard still requires image

Jira whiteboard (customfield_10841)

The CVE analysis tool uses the Jira whiteboard field as a per-ticket timestamp cache. This is being phased out in favour of Postgres as the durable store (HUM-5917), but is still the source for a subset of fields.

The whiteboard holds compact JSON. The relevant sub-key is cve_cycle:

cve_cycle key Meaning Mutable?
cve_published CVE publish date No — write-once
jira_created HUM ticket creation date No — write-once
jira_closed HUM ticket close date No — disabled (255-char limit)
upstream_fix Upstream fix timestamp No — write-once
fedora_fix Fedora update timestamp No — write-once
hb_rpm_fix RPM published to Pulp Yes — re-evaluated each run
hb_image_fix Image rebuilt on Quay Yes — re-evaluated each run

The whiteboard is read and written only by the CVE analysis tool. The dashboard never reads it directly; its authoritative source is always cve_ticket_events.


Jira fields read by the CVE analysis tool

On each run the analysis tool fetches every open HUM CVE tracker ticket and reads the following fields from the Jira REST API:

Jira field / custom field Purpose
summary Ticket title / package detection
status, resolution, resolutiondate Ticket lifecycle state
created Jira ticket creation date → hum_ticket_created
description, comment CVE ID extraction, fix evidence
labels cve-next-release, fix-in-progress, etc.
security Embargo level
assignee Ticket owner
customfield_10578 (Fixed in Build) Manually set SRPM identifying the fix build
customfield_10841 (Whiteboard) Cached cve_cycle timestamps (read + write)
customfield_10667 (CVE ID) Structured CVE identifiers
customfield_10860 (Embargo Status) Embargo flag
customfield_10020 (Sprint) Sprint membership
customfield_10014 (Epic Link) Parent epic

Data flow

┌─────────────────────────────────────────────────────────────────┐
│  External data sources                                          │
│                                                                 │
│  Jira REST API ─────────────────────────────────────┐           │
│  NVD / CVE list ────────────────────────────────────┤           │
│  GitHub / GitLab (upstream fix commits) ────────────┤           │
│  Fedora updates ────────────────────────────────────┤           │
│  Red Hat Pulp (RPM publish times) ─────────────────►│           │
│  Hummingbird catalog API (image rebuild times) ─────┤           │
│  Red Hat CSAF VEX feed ─────────────────────────────┤           │
│  OSIDB (subpackage affectedness) ───────────────────┘           │
└──────────────────────────────┬──────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│  CVE analysis tool (cron, hummingbird-cve-analysis)              │
│                                                                  │
│  • Reads Jira ticket fields + whiteboard cache                   │
│  • Resolves fix SRPM from Pulp repodata                          │
│  • Resolves image rebuild timestamp from catalog API             │
│  • Checks upstream PR/release dates (GitHub/GitLab)              │
│  • Fetches CVE publish date (NVD/CVE list)                       │
│  • Reconciles VEX status (CSAF feed)                             │
│  • Checks subpackage affectedness (OSIDB)                        │
│                                                                  │
│  Writes back to:                                                 │
│  ├── Jira whiteboard (cve_cycle timestamp cache)                 │
│  ├── Jira labels / Fixed-in-Build field                          │
│  ├── Jira comments                                               │
│  └── dashboard DB → cve_ticket_events (one row per milestone)    │
│                      cve_analysis_log (run record)               │
└──────────────────────────────┬───────────────────────────────────┘
                               │
           ┌───────────────────┴───────────────────┐
           │                                       │
           ▼                                       ▼
┌──────────────────────────┐      ┌───────────────────────────────┐
│  Jira whiteboard         │      │  dashboard DB                 │
│  (per-ticket JSON cache) │      │  cve_ticket_events table      │
│  cve_cycle timestamps    │      │  one row per (ticket,         │
│  ← being phased out      │      │    event_type) milestone      │
│    (HUM-5917)            │      └───────────────┬───────────────┘
└──────────────────────────┘                      │
                                                  ▼
                               ┌──────────────────────────────────┐
                               │  hummingbird-dashboard API       │
                               │                                  │
                               │  Pivots cve_ticket_events rows   │
                               │  into per-ticket dicts, computes:│
                               │  • entry-level R-Time fields     │
                               │    (HUM-5920)                    │
                               │  • duration legs inside stages   │
                               │    (HUM-5921)                    │
                               │                                  │
                               │  Serves JSON + Jinja templates   │
                               └──────────────────────────────────┘
flowchart LR
    EVENTS[GitLab pushes, pipelines, jobs, MRs<br/>Kubernetes PipelineRuns, TaskRuns, Snapshots, Releases]
    SNS[(SNS topic)]
    ARCHIVER[S3 archiver]
    S3[(Deterministic archive)]
    STATUS[hummingbird-status ingestor]
    STATUS_DB[(hummingbird-status DB)]
    DASHBOARD[Dashboard API]
    TIMELINE[Pipeline timing queue and worker]
    TIMELINE_DB[(pipeline_timing schema)]

    EVENTS --> SNS
    SNS --> ARCHIVER --> S3
    SNS --> STATUS --> STATUS_DB --> DASHBOARD
    SNS --> TIMELINE --> TIMELINE_DB

Kubernetes TaskRun transport

The Kubernetes event forwarder applies optional resource labelSelector and fieldSelector values to both its initial LIST and subsequent Watch requests. Field-selector support depends on the Kubernetes resource and API server; TaskRun collection should use a confirmed stable RPM application label instead of a status field selector.

The staging and production forwarders watch the configured resource kinds in both hummingbird-tenant and rhtap-releng-tenant, including TaskRuns in each namespace. Events go through each environment’s SNS topic and existing SQS copies; its Pipeline Timing worker consumes its dedicated queue. Because the resource list applies to each kubeconfig context, the managed context also watches the configured PipelineRun, Snapshot, Release, and Component kinds there. The TaskRun entry has no selector, so both forwarders list and watch all TaskRuns in both namespaces until a stable RPM application label is confirmed or namespace-wide volume is approved.

The forwarder publishes the logical kubeconfig context.cluster value, rather than the API server URL, with each Kubernetes event. For TaskRuns, it removes only metadata.managedFields before gzip/base64 encoding; identity, owner references, labels, annotations, task specification, conditions, results, pod name, and timing remain in the published and archived payload.


API field names

The dashboard API exposes these computed fields per R-Time entry:

Entry-level fields

API field Meaning
cve_published_at Start of R-Time: CVE published, falling back to HUM created when missing (Include CVE-HUM); the later of HUM created and CVE published (Exclude, so an embargoed ticket also starts at disclosure); either way, then floored at Upstream Fix Merged so the clock never starts before a fix exists to backport
fix_delivered_at Done timestamp: latest of delivery and Jira close (Exclude VEX), plus VEX when Include VEX is on; unset until the required gates exist. Delivery is image publish for catalog image sources, RPM publish otherwise
cve_to_delivery_hours Completed R-Time only (cve_published_at → fix_delivered_at); None while still accumulating
cve_to_hum_created_hours Filing lag after NVD: max(HUM created − NVD, 0); 0.0 when HUM is first
advisory_to_vex_hours Completed ADV-VEX only (hum_ticket_closed → vex_resolved); None until VEX exists
display_advisory_to_vex_hours ADV-VEX hours, or elapsed-to-now when the VEX feed has not updated yet
fix_before_cve_published True when the delivery timestamp predates notification (informational)
hum_ticket_open True until the selected done gates exist
deferred_to_next_release True for cve-next-release tickets
display_duration_hours R-Time hours, or elapsed-to-now when the ticket is not yet done

Duration legs (stages dict)

API field Interval
cve_published_to_osidb_flaw CVE publish → OSIDB flaw created
osidb_flaw_to_osidb_affect OSIDB flaw created → hummingbird-1 affect created
osidb_affect_to_hum_created hummingbird-1 affect created → HUM ticket opened
cve_published_to_hum_created CVE publish → HUM ticket opened
cve_published_to_upstream_fix CVE publish → upstream fix merged
hum_created_to_upstream_fix HUM ticket → upstream fix merged
upstream_fix_to_fedora_update Upstream fix → Fedora update available
fedora_update_to_rpm_fix_published Fedora update → fix RPM in Pulp
upstream_fix_to_rpm_fix_published Upstream fix → fix RPM in Pulp
hum_created_to_rpm_fix_published HUM ticket → fix RPM in Pulp
rpm_fix_published_to_image_rebuilt Fix RPM in Pulp → image rebuilt on Quay
image_rebuilt_to_hum_closed Image rebuild → HUM ticket closed
hum_closed_to_vex_resolved HUM ticket closed (advisory MR merge) → VEX feed update

Derived flags (stages dict)

osidb_flaw_unavailable is computed in _build_stages(), not persisted to cve_ticket_events: True only when osidb_affect_created is present but osidb_flaw_created is absent – OSIDB demonstrably knows the CVE (an affect was found) but has no public Flaw record for it. A ticket missing all three OSIDB fields is ambiguous (the collector may predate these fields, or OSIDB may have no data yet) and is not flagged. When true, _timeline_events() prepends a synthetic osidb_flaw_public entry with no real timestamp, displayed as “INTERNAL” and always first in the Stage Breakdown’s timestamp grid on the Open, Blame, and R-Time pages – a flaw must exist before any public OSIDB/CVE activity can reference it, so it precedes every other milestone even without a known date. It is intentionally excluded from the chronological bar graph (stages.timeline), which only plots real timestamps.

osidb_data_missing is the weaker, mutually exclusive counterpart: True when OSIDB has neither a Flaw nor an Affect record for the CVE – stating a fact (no OSIDB data found), not a cause, since this can simply mean OSIDB has not yet ingested a very recently published CVE. Rendered as a small “osidb-missing” badge next to the ticket key in the collapsed table row (the same spot and style as the R-Time page’s budget-excluded badge), not inside the Stage Breakdown drawer, on the Open, Blame, and R-Time pages.

Aggregate stats

API field Meaning
avg_hum_created_to_rpm_fix_hours Mean HUM ticket → RPM fix (skips missing legs)
avg_rpm_fix_to_image_rebuilt_hours Mean RPM publish → image rebuild
avg_advisory_to_vex_hours Mean Done-Errata close → VEX feed (completed ADV-VEX only)

Package lifecycle data

rpm_first_published is the earliest timestamp at which any SRPM for a given package appeared in the Hummingbird Pulp repo. It is package-scoped — one row per package — as opposed to cve_ticket_events which is ticket-scoped.

Collection

scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

Flow:

  1. Loads the package list from rpms_repo.load_package_map_from_metadata().
  2. For each package: calls pulp.fetch_earliest_srpm_time(), which browses packages.redhat.com/.../source/Packages/{letter}/, parses the HTML listing already used by the analysis tool, and returns the earliest upload timestamp across all SRPMs for that package.
  3. POSTs all results in one request to /api/cve-import (same endpoint used for cve_ticket_events sync). The dashboard upserts with COALESCE(LEAST(existing, incoming), existing, incoming) — re-runs only update occurred_at if the incoming timestamp is earlier, and NULLs are never stored over a real value.

Surfaced in dump_lifecycle

scripts/dump_lifecycle --prod HUM-1234

After fetching ticket milestones from cve_ticket_events, the script resolves the package name from those rows and makes a second call to /api/cve-export?section=package_lifecycle to fetch rpm_first_published for that package. It is shown at the bottom of the text output and under rpm_first_published in the JSON output.

4.5.23 - Kubernetes Event Forwarder

A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch.

The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded.

The staging and production forwarders watch the configured resources in both hummingbird-tenant and rhtap-releng-tenant on RH03. TaskRun collection is enabled in both namespaces and environments.

For TaskRun events only, Kubernetes metadata.managedFields is removed before compression. All other object data, including identity, owner references, labels, annotations, task specification, conditions, results, pod name, and timing is retained.

Features

  • Multi-Cluster Support: Watch resources across multiple Kubernetes clusters using kubeconfig contexts
  • Dynamic Resource Watching: Configure any namespaced resource type using standard Kubernetes apiVersion and kind
  • SNS Integration: Publish events with structured message attributes for precise filtering
  • Compression: Events are compressed (gzip+base64) to reduce SNS message size
  • Selectors: Resource-specific Kubernetes label and field selectors limit LIST and Watch requests
  • Automatic Reconnection: Handles watch connection failures and reconnects automatically

Architecture

Threading Model

The forwarder uses a multi-threaded architecture:

  • Watcher threads (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn’t affect others.

  • Publisher thread (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness.

Memory optimization: When SPOOL_DIR is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills).

LIST + Watch Pattern

The forwarder uses explicit LIST followed by Watch rather than resource_version="0":

  1. LIST retrieves all current objects and a snapshot resourceVersion
  2. Watch starts from the LIST’s resourceVersion (not object RVs)
  3. On graceful watch timeout (300s), restart watch from last known RV (no re-list)
  4. On errors, fresh LIST to ensure current state

This is necessary because synthetic ADDED events from resource_version="0" have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade.

Connection recovery

LIST and Watch requests use a 10-second client connection timeout. The client read timeout is 30 seconds longer than the server watch timeout. With the default 300-second watch timeout, a silent socket fails after 330 seconds without data. The server timeout alone cannot interrupt a client read on a broken connection. The transport does not retry requests internally. The LIST+Watch loop controls retries so that each request failure triggers resource-version recovery.

After a request error, the watcher closes the stream and starts a fresh LIST. This recovers the current state of resources that changed during the outage. Consumers must tolerate repeated events from this LIST. Objects deleted before the LIST cannot be recovered through this process.

The hummingbird_event_forwarder_watch_cycles_total metric records completed and failed cycles. Labels identify the cluster, namespace, resource kind, and outcome, which is completed or error. Normal watch completion also produces an INFO log with the resource version. An idle watch completes normally and does not require a resource change to show progress.

Prerequisites

  • Target cluster: ServiceAccount with watch permissions on the resources you want to monitor
  • Deployment cluster: Kubernetes cluster to run the forwarder
  • AWS credentials: SNS publish permissions (optional - logs events if not configured)

Deployment

The container image is built via Konflux CI/CD and published to quay.io/hummingbird-ci/kubernetes-event-forwarder:latest.

Example Kubernetes manifests are provided in the kubernetes/ directory:

  • rbac.yaml - ServiceAccount, Role, RoleBinding for the target cluster
  • secret.yaml - Kubeconfig and AWS credentials
  • configmap.yaml - Resource watch configuration
  • deployment.yaml - Forwarder deployment

Quick Start

  1. On the target cluster (the one you want to watch), apply RBAC and create a token:

    kubectl apply -f kubernetes/rbac.yaml
    kubectl create token kubernetes-event-forwarder --duration=8760h
    
  2. Update the example manifests with your values:

    • secret.yaml: cluster URL, token, AWS credentials
    • configmap.yaml: resources to watch
    • deployment.yaml: SNS topic ARN, AWS region
  3. Apply the manifests to your deployment cluster:

    kubectl apply -f kubernetes/secret.yaml
    kubectl apply -f kubernetes/configmap.yaml
    kubectl apply -f kubernetes/deployment.yaml
    

Prerequisites: Deploy hummingbird-events-topic first to create the SNS topic, then deploy the AWS resources (see below).

AWS Resources

The SAM template (template.yaml) provisions IAM resources for SNS publishing:

  • IAM User (${ResourcePrefix}-user) - Service account for the forwarder
  • IAM Policy (${ResourcePrefix}-policy) - Grants sns:Publish to the SNS topic

Deploy using containerized AWS SAM CLI:

cd kubernetes-event-forwarder
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely.

SAM Parameters

Parameter Description Default
ResourcePrefix Prefix for resources myapp-prod
SnsTopicArn SNS topic ARN (required)

Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern (e.g., myapp-prod-user, myapp-prod-policy).

Usage

Configure resources to watch in config.yaml:

resources:
  - apiVersion: v1
    kind: Pod
    labelSelector: app=example
    fieldSelector: status.phase=Running
  - apiVersion: apps/v1
    kind: Deployment
  - apiVersion: v1
    kind: ConfigMap

labelSelector and fieldSelector are optional per-resource settings and use standard Kubernetes selector syntax. Field-selector support is resource and API server dependent; do not use a TaskRun status field selector. Use a stable RPM application label for TaskRuns only after confirming its propagation in the target environment. The deployed resource list applies to every kubeconfig context, so both forwarders watch all configured kinds in both namespaces. The TaskRun entry has no selector and therefore watches all TaskRuns in both namespaces. Confirm a stable RPM application label and representative payload sizes; otherwise review and approve the namespace-wide event rate. HUM-7532 provides the managed namespace access, and HUM-7535 owns environment-specific infrastructure.

SNS Subscription Filter Examples:

Pod events in a specific namespace:

{
  "source": ["kubernetes"],
  "kind": ["Pod"],
  "namespace": ["production"]
}

All deployment changes:

{
  "source": ["kubernetes"],
  "kind": ["Deployment"]
}

Deleted resources across all clusters:

{
  "source": ["kubernetes"],
  "event_type": ["DELETED"]
}

Development

See the main README for development workflows.

make kubernetes-event-forwarder/setup  # Install dependencies
make check                             # Lint code (ruff)
make fmt                               # Format code
make test                              # Run unit tests
make coverage                          # Run tests with coverage

Configuration

The deployment is configured via environment variables:

Variable Description
CONFIG_PATH Path to config YAML file
CONFIG Inline config YAML (alternative)
SNS_TOPIC_ARN SNS topic ARN (optional - logs if unset)
SPOOL_DIR Optional spool directory for file-based message queue
AWS_ACCESS_KEY_ID AWS access key ID
AWS_SECRET_ACCESS_KEY AWS secret access key
AWS_DEFAULT_REGION AWS region
METRICS_PORT Prometheus metrics port (default: 9090)
SENTRY_DSN Optional Sentry DSN

Event Metadata

The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes:

Attribute Description Example
source Always "kubernetes" kubernetes
cluster Logical kubeconfig cluster entry name konflux-rh03
namespace Object namespace production
api_version Resource API version v1, apps/v1
kind Resource kind Pod, Deployment
event_type Event type ADDED, MODIFIED, DELETED
object_name Name of the object nginx-7d8c4c9d6f
content_encoding Message encoding gzip+base64

cluster is the context.cluster value for the kubeconfig context that the forwarder uses to watch the resource. It is not the API server URL.

Security & Limitations

Security:

  • AWS credentials stored in Kubernetes Secret
  • Kubeconfig credentials stored in Kubernetes Secret
  • SNS topic follows least privilege principle (publish-only)
  • Sentry integration for error tracking

Limitations:

  • Only namespaced resources supported
  • One thread per context + resource type combination
  • SNS message size limit: 256 KB (after compression)
  • SNS measures the UTF-8 gzip/base64 message plus attribute names, types, and values against its 262,144-byte limit. The forwarder logs each computed size and remaining headroom and exports hummingbird_event_forwarder_sns_publish_size_bytes. Review representative RPM build, test, and tenant-release TaskRun measurements before enabling the source; do not remove further fields without an approved projection.
  • The source-tree fixtures in kubernetes-event-forwarder/tests/fixtures/ are synthetic, redacted RPM build, test, and tenant-release TaskRun regression fixtures. Their unit test measures the complete publish body and attributes and requires at least 32 KiB of headroom. HUM-7533 owns source-activation measurement review.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.24 - MR Artifact Promotion

How Hummingbird releases an MR-tested artifact after merge

Hummingbird builds and tests an RPM artifact during a merge request (MR). After merge, the push pipeline promotes that same artifact to release without building it again. The tested image digest is therefore the one consumed by the release pipeline. This flow is in use for RPMs; container promotion is separate work.

RPM flow

flowchart LR
    A["MR build and tests<br/>source B, digest D"] --> B["Merge to main<br/>merge commit M"]
    B --> C["Promotion PipelineRun<br/>finds passed MR Snapshot"]
    C --> D["Component Snapshot<br/>same B and D"]
    D --> E["Integration Service<br/>creates Release"]
    E --> F["Release Service<br/>checks and publishes RPMs"]
  1. The promotion PipelineRun reads the two parents of merge commit M from Git. The second parent, B, is the tested MR head. The pipeline requires a two-parent merge commit; it does not need an MR number or GitLab API lookup.
  2. The artifact lookup Task finds a passed MR Component Snapshot for B in live Konflux or KubeArchive. It verifies the linked build PipelineRun, target branch, source repository (including forks), test result, and image digest D. A passed /test or /retest build is eligible when the Snapshot and build record the same PaC event. Multiple passed Snapshots for one commit are normal — a /test or /retest run creates another — and while RPM builds are non-reproducible (HUM-9785) their image digests may differ. Promotion uses the newest eligible Snapshot.
  3. The promotion Task rechecks that evidence and creates a single-Component, push-event Snapshot with the same source revision B and image digest D. It records M, the source MR Snapshot and build, and the promotion PipelineRun. The Snapshot’s test.appstudio.openshift.io/pipelinerunstarttime is the promotion PipelineRun’s start time in epoch milliseconds.
  4. Integration Service processes the Component Snapshot and creates the automatic Release. Release Service runs the existing Conforma, signing, and publication checks. The promotion PipelineRun creates no Release or new build attestation. The released RPMs come from the tested OCI artifact.

The promotion Task checks that no IntegrationTestScenario would run on the new Snapshot, so this handoff does not repeat the MR tests. RPM promotion does not create an override Snapshot or update the Global Candidate List: RPMs do not use it for group tests.

For RPM operations, including push pipeline overrides, verifying a promotion, and recovery, see the RPM MR artifact promotion guide.

Ordering and recovery

Before creating a Snapshot, promotion checks whether a newer merge changed the same package. An unrelated package merge does not block it. The Component Snapshot start-time annotation also lets Integration Service apply its existing push Snapshot ordering check. These are the same ordering controls used by push builds; they do not promise a global serial release queue.

A retry uses the same Snapshot name derived from the Component and M. It reuses an existing live Snapshot only when its identity, digest, provenance, and original promotion start time match. It never replaces an archived Snapshot or creates a second one to retry a failed Release.

Promotion stops if required MR evidence is unavailable in both live Konflux and KubeArchive, the source image has expired, a test failed, or a matching IntegrationTestScenario would run. If promotion cannot use the tested artifact, recovery requires an explicit new build and required tests. A failed automatic Release is retried through the release workflow. MR Snapshot retention and Quay image retention are independent; the source image must remain available through merge and release retries.

Relationship to RHEL on Konflux

RHEL on Konflux (RoK) also reuses an MR-tested RPM after merge. Its current MR artifact is an imported Koji draft build, and its release continues through Brew and Errata. Hummingbird creates its RPM artifact in Konflux and uses a Component Snapshot and automatic Release to publish to Pulp. The shared design point is release of the tested artifact without a post-merge rebuild; artifact storage and publication differ.

Container promotion

Container promotion remains in the container epic. Its group tests use the Global Candidate List, so it needs a separate, non-releasing override Snapshot as well as the Component Snapshot used for release. The container path must establish a no-retest handoff and prevent an older override from replacing a newer candidate before rollout.

MR container images also need tag-level expiry instead of the embedded quay.expires-after label: that label would follow a promoted container image into release. A set-tag-expiry task should expire the MR tag through Quay without changing the image; its retention window, Quay access, and Conforma trust remain open design questions. RPM releases publish the RPMs inside the OCI artifact, so that label does not appear on released RPMs. The MR source image must still survive until the RPM release completes.

For Task inputs, permissions, and failure handling, see the Hummingbird tools Task reference.

4.5.25 - Jira Image Requests

AWS Lambda proxy that serves Hummingbird Image Request issues from Jira (with DynamoDB caching) and creates Stories/Epics from the catalog request form.

Features

  • Public list: GET /image-requests returns all Ready Image Request issues, following Jira search pagination so requests are not hidden by its page limit
  • Personal request history: Authenticated GET /my-image-requests returns current and historical requests whose Jira contact email matches the signed-in user
  • Form submit: POST /image-requests creates a version Story. An Epic is created only when a second request uses the same image name; later requests reuse that Epic. A first-time name is Story-only. Authenticated submissions immediately appear in My requests as In Triage.
  • Customer scoring: Writes the Customer criterion on the Epic from paid email domains (placeholder until subscriber login)
  • Total scoring: Maintained by Jira Automation from manual triage fields

Prerequisites

  • AWS CLI with permissions for Lambda, API Gateway, DynamoDB, SSM, CloudFormation
  • Podman or Docker for SAM build
  • Jira bot token in SSM (SecureString)
  • HUM project custom fields for scoring (see below)

Deployment

make jira-image-requests/build
make jira-image-requests/deploy

Configuration

Parameter / env Description
ResourcePrefix Prefix for AWS resource names
JiraTokenParameter SSM parameter name for the Jira API token
CacheTtlSeconds Cache TTL for the public list (default 300)
CorsAllowOrigin Allowed CORS origin
ScoringCustomerField Jira custom field id for Customer (customfield_…); empty skips write
PaidCustomerDomains Comma-separated email domains scored as paid (e.g. acme.com,contoso.com)
OidcUserinfoUrl OIDC userinfo endpoint used to verify the bearer token and resolve email
SentryDsn Optional Sentry DSN

My requests ownership

The browser sends the signed-in user’s access token when it loads or submits a personal request. The Lambda resolves that token through OidcUserinfoUrl and uses a SHA-256 digest of the normalized email as the DynamoDB partition key. Raw email is not stored in the ownership records.

On the first authenticated load, the service searches every page of Image Request Stories in Jira, verifies the exact contact email from each description, and writes the matching records to DynamoDB. Later loads reconcile Jira again so status and notes remain current. Anonymous submissions remain supported and are associated with the contact email entered in the form.

Every request uses the same /image-requests?request=REQ-<number> URL throughout its lifecycle. While a request is In Triage, the drawer resolves it only from the authenticated owner’s personal history. After the team accepts it and Jira includes it in the public Ready list, the same URL resolves from the public list for signed-out visitors and other users as well.

Request status projection

The catalog projects Jira workflow states into the following customer-facing state machine. A newly submitted item starts in In Triage. Jira reopen or backlog changes provide the regression paths.

stateDiagram-v2
    [*] --> InTriage: form submitted
    InTriage --> Submitted: accepted for backlog
    InTriage --> Declined: triage declines
    Submitted --> InProgress: work starts
    Submitted --> Declined: request declined
    InProgress --> Delivered: work completes
    InProgress --> Declined: work stops
    InProgress --> Submitted: returned to backlog
    Submitted --> InTriage: Jira resets to new
    Delivered --> InProgress: Jira reopens
    Declined --> InTriage: Jira reopens for triage

Each DynamoDB ownership record stores the request key, summary, projected status, submitted and updated timestamps, resolution date, latest public note, and whether the latest visible change was a status change, note change, or both. The UI falls back to Request updated when the previous snapshot is not available or another Jira field changed. Its keys are OWNER#<email digest> and REQUEST#<Jira key>.

Image request scoring

Triage scores live on the Image Request Epic, not on customer-facing Stories.

Custom fields (HUM Epics)

Create (or map) these fields, then record each customfield_XXXXX id:

Field Type / values Who sets it
IR Legal Select: Y / N Manual
IR Competitive Number (typically 0–1) Manual
IR Marketable Number Manual
IR Supportability Number Manual
IR Upstream Health Number Manual
IR Level of Effort Number: 3 easy, 2 medium, 1 hard Manual
IR Technical Feasibility Number: 0 blocking, 1 feasible Manual
IR Portfolio Conflicts Number: 0 conflict, 1 good Manual
IR Customer Number: 0 or 1 Lambda (ScoringCustomerField)
IR Total Number Jira Automation only

Gate: Legal must be Y to continue scoring. Competitive, Marketable, and Supportability / Maturity are also expected to be 1 before a meaningful Total (team convention from the triage spreadsheet).

Lambda: Customer criterion

When a POST creates or reuses an Epic (second or later request for a name), the Lambda:

  1. Loads Stories under the Epic
  2. Sets Customer to 1 if any submitter email domain is in PaidCustomerDomains, otherwise 0
  3. Writes the value to ScoringCustomerField when configured

A first request for a name does not create an Epic, so Customer is not written until a second request for that name arrives.

When subscriber login lands, replace the domain list with real subscription status.

Configure two rules in Jira (Project settings → Automation). Use the real field names/ids from your instance.

  • Trigger: Field value changed for any of: Competitive, Marketable, Supportability, Upstream Health, Level of Effort, Technical Feasibility, Portfolio Conflicts, Customer (and optionally Legal)
  • Condition: Issue type = Epic AND component = Image Request AND IR Legal = Y
  • Action: Edit issue → set IR Total to the sum of the numeric criteria (exclude Legal). Example smart value shape:
{{#=}}
{{issue.IR Competitive}} + {{issue.IR Marketable}} + {{issue.IR Supportability}} +
{{issue.IR Upstream Health}} + {{issue.IR Level of Effort}} +
{{issue.IR Technical Feasibility}} + {{issue.IR Portfolio Conflicts}} +
{{issue.IR Customer}}
{{/}}

Use your instance’s smart-value field keys (often customfield_XXXXX).

  • Optional action: Remove label ir-legal-blocked when Legal is Y
  • Trigger: IR Legal changed
  • Condition: Issue type = Epic AND component = Image Request AND IR Legal is empty OR IR Legal = N
  • Actions:
    1. Clear IR Total (or set to empty)
    2. Add label ir-legal-blocked

Jira Automation recalculates Total whenever triage fields change — no webhook or Lambda rescoring path is required.

Development

make jira-image-requests/setup
make test

License

GPL-3.0-or-later

4.5.26 - PAC Trigger

A CLI tool to manually trigger Konflux PAC (Pipelines-as-Code) pipelines for a specific component and branch. Useful for debugging pipeline issues or re-running builds without pushing a new commit.

Usage

# Trigger pipeline for a component (uses branch HEAD)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --cluster-url https://konflux-ui.apps.cluster.example.com

# Trigger with specific commit
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --commit abc123def456 \
            --cluster-url https://konflux-ui.apps.cluster.example.com

# Dry run (preview without creating)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
            --component myimage--default--main \
            --cluster-url https://konflux-ui.apps.cluster.example.com \
            --dry-run

Options

Option Description Default
--gitlab-project-url GitLab project URL (required) -
--component Component name (required) -
--branch Branch name main
--commit Specific commit SHA HEAD
--cluster-url Konflux cluster URL (required) -
--dry-run Preview PipelineRun without creating false
-v, --verbose Enable debug logging false

Installation

cd pac-trigger
pip install -e .

Prerequisites

  • Kubeconfig: Context with access to the target Konflux cluster/namespace
  • Repository resource: PAC Repository must exist for the GitLab project
  • Push template: Component must have a push template in .tekton/*.yaml

How It Works

  1. Fetch commit: Gets HEAD commit SHA from GitLab (or uses provided commit)
  2. Fetch template: Downloads .tekton/*.yaml files via GitLab API, finds push template matching the component
  3. Get namespace: Extracts namespace from template metadata
  4. Find credentials: Looks up PAC Repository resource to find git secret
  5. Create git-auth secret: Creates ephemeral pac-trigger-gitauth-* secret with git credentials
  6. Create PipelineRun: Substitutes template variables and creates the PipelineRun
  7. Link secret: Sets ownerReference on secret for garbage collection
  8. Print URL: Outputs Konflux UI URL for the PipelineRun

Template Variables

The following PAC template variables are supported:

Variable Substituted with
{{revision}} Commit SHA
{{target_branch}} Branch name
{{repo_url}} GitLab project URL
{{git_auth_secret}} Created secret name

Unsupported template variables will cause an error.

Labels Added

Label Value Purpose
pac-trigger/manual true Identifies manual triggers

Features

  • Template-based: Fetches PipelineRun templates from .tekton/*.yaml files via anonymous GitLab API
  • Push-only: Only triggers push templates (filters by CEL expression)
  • Automatic credentials: Creates ephemeral git-auth secrets from existing PAC Repository secrets
  • Garbage collection: Secrets are linked to PipelineRun via ownerReference
  • Kubeconfig-based: Uses local kubeconfig for cluster authentication

Development

See the main README for development workflows.

make pac-trigger/setup  # Install dependencies
make check              # Lint code (ruff)
make fmt                # Format code
make test               # Run unit tests
make coverage           # Run tests with coverage

Comparison with PAC-triggered Runs

Manually triggered PipelineRuns differ from PAC-triggered ones:

Present in both:

  • appstudio.openshift.io/application
  • appstudio.openshift.io/component
  • pipelines.appstudio.openshift.io/type: build
  • build.appstudio.redhat.com/commit_sha
  • build.appstudio.redhat.com/target_branch

Unique to pac-trigger:

  • pac-trigger/manual: true label

Missing (PAC internal metadata):

  • pipelinesascode.tekton.dev/event-type
  • pipelinesascode.tekton.dev/sha
  • GitLab status reporting annotations

These differences don’t affect pipeline execution—they’re used for PAC’s internal tracking and GitLab commit status updates.

Limitations

  • Only push templates supported (not pull-request)
  • Only GitLab repositories supported
  • Requires existing PAC Repository resource
  • No GitLab commit status reporting

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.27 - Hummingbird Status

SQS worker and database management for ingesting Konflux pipeline events.

Features

  • SQS Worker - Real-time database updates from SNS pipeline events
  • GitLab Event Ingestion - Push events and merge request events with version tracking
  • Database Init - Populate from pg_dump, S3 archive, or local filesystem mirror
  • Prometheus Metrics - Built-in metrics endpoint for monitoring
  • MR Reconciliation - Fix stale MR state by checking GitLab API
  • Schema Management - Idempotent database schema creation

Pipeline timing and lineage are ingested by the separate Pipeline Timing service into a dedicated schema in the same events database. The Status worker ignores pipeline, job, and TaskRun events it does not store in public.

Prerequisites

  • Python 3.11+
  • PostgreSQL 16
  • AWS credentials (for SQS worker and S3 initialization)

Installation

cd hummingbird-status
pip install -e .

Usage

Local Development

cd hummingbird-status

# Database management
./dev.sh db-start              # Start PostgreSQL container
./dev.sh db-init               # Initialize from available source
./dev.sh db-shell              # PostgreSQL interactive shell
./dev.sh db-dump               # Create pg_dump file
./dev.sh db-stop               # Stop PostgreSQL
./dev.sh db-reset              # Stop and delete volume

# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init   # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init        # From S3

# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker

Container Deployment

The container runs the SQS worker by default:

podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status

For database initialization:

podman run -e DATABASE_URL=... -e S3_BUCKET=... \
    hummingbird-status python3 -m hummingbird_status.worker.init_db

MR Reconciliation

MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear as “opened” in the dashboard when it is actually merged or closed. The reconcile command checks all open MRs against the GitLab API and corrects stale entries:

# Using environment variables
GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile

# Using CLI arguments
python -m hummingbird_status.reconcile \
    --database-url postgresql://... \
    --gitlab-token glpat-...

In a container:

podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \
    hummingbird-status python3 -m hummingbird_status.reconcile

Configuration

Environment Variables

Variable Default Description
DATABASE_URL - PostgreSQL connection URL
SQS_QUEUE_URL - SQS queue URL (worker)
S3_BUCKET - S3 bucket for init
S3_PREFIX sns/ S3 key prefix
LOCAL_MIRROR_PATH /data/sns-mirror Local S3 mirror path
INIT_DUMP_PATH /data/init/dump.sql pg_dump file path
GITLAB_URL https://gitlab.com GitLab instance URL (reconcile)
GITLAB_TOKEN - GitLab API token (reconcile)
METRICS_PORT 9090 Prometheus metrics port

Database Initialization Priority

  1. pg_dump file - $INIT_DUMP_PATH if exists
  2. Local S3 mirror - $LOCAL_MIRROR_PATH/sns/*.json.gz if exists
  3. S3 bucket - $S3_BUCKET/$S3_PREFIX with AWS credentials

Database Schema

The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).

flowchart TD
    push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
    mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
    mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
    comp[["<b>components</b><br/>git_context → component mapping"]]
    build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
    promotion[["<b>pipelineruns</b> (type=promotion)<br/>reuses the tested MR image"]]
    snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
    promoted["<b>promotion Snapshot</b><br/>source_plr from Hummingbird annotation"]
    test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
    rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
    relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]

    push -- "+ affected" --> build
    push -- "promotion-enabled RPM" --> promotion
    mr -- "sha" --> mrv
    mrv -- "sha joins" --> build
    comp -- "components" --> build
    build -- "success<br/>creates" --> snap
    snap -. "tested digest" .-> promotion
    promotion -- "creates" --> promoted
    snap -- "triggers" --> test
    test -- "success<br/>creates" --> rel
    promoted -- "auto-release<br/>without retest" --> rel
    rel -- "managed<br/>by" --> relplr

RPM promotion Snapshots identify their creating PipelineRun with the hummingbird-project.io/promotion-pipelinerun annotation. The ingest stores that name as source_plr. Historical Snapshot records without source_plr are linked through the promotion PipelineRun’s COMPONENT_SNAPSHOT result.

Tables

Table Primary Key Description
gitlab_pushes sha,repo,ref GitLab push events to main branch
gitlab_merge_requests project,iid Current state of merge requests
gitlab_mr_versions project,iid,sha Head commit history for each MR
components name Konflux Component resources
pipelineruns name Build, test, and release pipelines
snapshots name Image snapshots after successful builds
releases name Published releases to target registry

Merge Request Tracking

The MR tables enable tracking build status across MR versions:

  • gitlab_merge_requests - Stores current MR metadata (title, state, branches, author, latest head SHA). Updated via ON CONFLICT ... WHERE updated_at < to keep the most recent state. author_name captures the webhook user.name alongside author (the username) so the dashboard can show a bot/access-token account’s real GitLab name (e.g. “chore-mr”) instead of its generated username – like author, it’s set once and preserved via COALESCE on conflict, so it isn’t backfilled for MRs that predate this column.

  • gitlab_mr_versions - Records each unique head commit SHA for an MR. When force-pushing the same SHA, created_at is updated to the latest event timestamp via GREATEST(), ensuring correct ordering even after force pushes.

Development

Running Tests

cd hummingbird-status
pip install -e ".[dev]"
pytest

Project Structure

hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml         # SAM template for AWS resources
├── hummingbird_status/
│   ├── db.py             # Database schema and utilities
│   ├── ingest.py         # SNS event parsing and ingestion
│   ├── reconcile.py      # MR state reconciliation with GitLab
│   └── worker/
│       ├── sqs.py        # SQS consumer
│       └── init_db.py    # Database initializer
└── tests/

AWS Resources

Deploy the SQS queue using SAM:

cd hummingbird-status
make build     # Build SAM application
make deploy    # First deployment (guided)
make redeploy  # Subsequent deployments

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
SnsTopicArn ARN of the SNS topic to subscribe (required)

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first.

See the main README for development workflows.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.28 - Lambda S3 Cache

An AWS Lambda setup that caches files from public URLs in S3, supporting both HTTP file caching (RPMs, repository metadata) and OCI Distribution API v2 (container image manifests and blobs). When a URL is requested, the service returns a 302 redirect to either the cached S3 copy (via presigned URL) or the original source. Cache misses trigger asynchronous downloads to S3, ensuring future requests are served from the cache.

Caching uses the original URL’s host and path as the S3 key for HTTP content, and a flat content-addressed layout (cache/oci/<digest>) for OCI artifacts. Frequently accessed content automatically extends its cache lifetime on each access.

Note: Cached content behavior depends on file type:

  • Immutable content (matching immutable extension filter, e.g., .rpm): Changes at origin won’t be reflected until cache expires
  • Mutable content (non-matching extension, e.g., repomd.xml): Cache is revalidated on each request via HEAD check; stale behavior is configurable

Cache Revalidation: Mutable content uses HTTP ETags for efficient validation. The uploader stores the origin’s ETag in S3 metadata when caching content. On subsequent requests, the handler sends a HEAD request with If-None-Match header containing the stored ETag. If the origin responds with 304 Not Modified, the cache is fresh and served directly. If the ETag differs, the cache is stale and an async refresh is triggered; response behavior depends on StaleCacheBehavior.

Note: If the origin doesn’t provide ETags, the cache cannot validate freshness. In this case, mutable content always redirects to origin without triggering cache updates (caching would be ineffective since every request would redirect anyway).

Features

  • Streaming Upload: Handles files efficiently by streaming directly from source to S3 without loading into memory
  • Presigned S3 URLs: Returns short-lived signed URLs for cached content
  • Automatic Expiration: Cached content expires after a certain time, with lifetime extended on each access
  • URL Prefix Allowlist: Only caches content matching explicitly allowed host+path prefixes
  • Immutable Extension Filter: Identifies immutable content that doesn’t require revalidation
  • Cache Revalidation: Mutable content (non-matching extensions) is validated on each request; stale cache behavior is configurable (origin or cache)
  • OCI Image Caching: Caches OCI container image manifests and blobs via the OCI Distribution API v2, protecting against upstream digest garbage collection
  • Multi-Registry Support: OCI caching supports multiple upstream registries by encoding the registry hostname in the request path
  • Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Three Lambda functions handle the caching workflow:

  1. Handler - API Gateway endpoint that checks cache, returns 302 redirects, and triggers async operations. Implements touch cooldown to prevent S3 throttling by only touching objects after a configurable time period has elapsed since last modification.
  2. Uploader - Downloads from origin and streams to S3 on cache misses (invoked asynchronously)
  3. Touch - Updates S3 object timestamps to extend cache lifetime on cache hits (invoked asynchronously only when cooldown period has elapsed)

Request Flow

flowchart TD
    Start([Request]) --> CheckURL{URL matches<br/>AllowedPrefixes?}

    CheckURL -->|No| RedirectOrigin
    CheckURL -->|Yes| HeadS3[HEAD S3]:::network
    HeadS3 --> CheckCache{Cache exists?}

    CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async
    CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?}

    CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin<br/>If-None-Match: stored ETag]:::network
    HeadOrigin --> CheckChanged{Origin changed?}

    CheckChanged -->|No ETag from origin| RedirectOrigin
    CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async
    CheckChanged -->|ETag matches| CheckCooldown
    CheckChanged -->|304 Not Modified| CheckCooldown
    CheckChanged -->|"Error/timeout"| CheckCooldown

    CheckImmutable -->|Yes| CheckCooldown

    CheckCooldown{Touch cooldown<br/>elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async

    InvokeUploaderMiss --> RedirectOrigin
    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin

    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache
    CheckCooldown -->|No| RedirectCache
    InvokeTouch --> RedirectCache

    subgraph redirectGroup [ ]
        RedirectOrigin[302 to Origin URL]:::redirect
        RedirectCache[302 to S3 Presigned URL]:::redirect
    end
    style redirectGroup fill:none,stroke:none

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff

Legend: Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
  • Podman or Docker (for containerized SAM build/deploy)
  • Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

make lambda-s3-cache/build     # Build Lambda package
make lambda-s3-cache/deploy    # First deployment (interactive/guided)
make lambda-s3-cache/redeploy  # Subsequent deployments (non-interactive)

Deployment output: ApiEndpoint - the API Gateway URL to use for requests

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
PresignedUrlExpiration Presigned URL expiration (seconds) 3600
AllowedPrefixes Whitespace-separated list of allowed URL prefixes example.com/path/
ImmutableExtensions File extensions for immutable content (no revalidation) .rpm
CacheExpirationDays Days to keep cached content (minimum: 1) 14
TouchCooldownMinutes Minimum minutes between touch operations 60
StaleCacheBehavior When cache is stale: origin or cache origin
OciAllowedPrefixes Whitespace-separated OCI image prefixes (registry/name) ``
OciOriginHeadTimeout Timeout (seconds) for HEAD requests to OCI registries 5
CustomDomainName Optional custom domain name ``
HostedZoneId Route53 hosted zone ID (required if custom domain) ``
SentryDsn Optional Sentry DSN for error tracking ``

Resource naming: All AWS resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-bucket, myapp-prod-lambda-handler).

Usage

Make GET requests to the API endpoint (or custom domain if configured). The URL to cache is encoded in the path (without the https:// prefix):

curl -L "https://<api-endpoint>/example.com/path/to/file.rpm"

Behavior:

  • First request (cache miss): Redirects to original URL while triggering async S3 upload
  • Subsequent requests (cache hit):
    • Immutable content (matching immutable extension filter): Redirects to presigned S3 URL and resets cache expiration
    • Mutable content (non-matching extension): HEAD request validates cache freshness (10s timeout). If stale, behavior depends on StaleCacheBehavior; if fresh, serves from cache
  • Disallowed prefixes: URLs not matching allowed prefixes are transparently redirected to the original URL (302 pass-through)

Immutable Extension Filter Behavior

The ImmutableExtensions parameter determines caching behavior:

  • Matching extension (e.g., .rpm): Treated as immutable content. Cached without revalidation - changes at origin won’t be reflected until cache expires. Served directly from cache on all requests.

  • Non-matching extension (e.g., .xml, .gz): Treated as mutable content. Cached with revalidation - HEAD request on each cache hit validates freshness using ETags. If stale, behavior depends on StaleCacheBehavior:

    • origin (default): Redirects to origin for fresh content, async refresh
    • cache: Serves stale cache immediately (faster), async refresh in background

Important: All files matching AllowedPrefixes are cached, regardless of extension. The extension filter only determines whether revalidation is performed.

Usage as Repository Proxy

The cache can be used as a DNF/yum baseurl for RPM repositories. Configure ImmutableExtensions to include .rpm:

  • .rpm files (matching extension): Cached as immutable - fast, no revalidation
  • Metadata files (non-matching extension, e.g., repomd.xml, primary.xml.gz): Cached with revalidation - ensures fresh metadata while benefiting from cache
[myrepo]
name=My Repository
baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/
enabled=1

This provides caching benefits for RPM downloads while ensuring repository metadata stays fresh.

OCI Image Caching

The cache also supports the OCI Distribution API v2 for caching container image manifests and blobs. This prevents deployment failures caused by upstream registries garbage-collecting digest-pinned images when tags move.

Requests with a v2/ path prefix are handled as OCI requests. The upstream registry is extracted from the first path segment after v2/, enabling multi-registry support through a single cache endpoint.

OCI request format:

GET /v2/<registry>/<image-name>/manifests/<digest>
GET /v2/<registry>/<image-name>/blobs/<digest>

Example:

# Health check
curl https://cache.example.com/v2/

# Manifest request (quay.io/openshift/origin-oauth-proxy)
curl -L https://cache.example.com/v2/quay.io/openshift/origin-oauth-proxy/manifests/sha256:504c036d...

OCI cache strategy (origin-first):

The OCI cache uses an origin-first strategy, matching the proven CKI dependency-archive pattern. This minimizes S3 egress — the cache is only consulted when the upstream registry is unavailable:

  1. HEAD upstream registry — check if the digest is still available
  2. Origin available + not cached → redirect to upstream, trigger async S3 upload
  3. Origin available + cached → redirect to upstream (touch if cooldown elapsed)
  4. Origin unavailable + cached manifest → serve inline from S3 (200 with body, Content-Type from metadata, Docker-Content-Digest header)
  5. Origin unavailable + cached blob → 302 to presigned S3 URL
  6. Origin unavailable + not cached → redirect to upstream as last resort
  7. Any error → redirect to upstream (fail-open)

S3 key layout: cache/oci/sha256:<digest> — flat, content-addressed. The same blob shared by different images is stored only once.

OCI request flow:

flowchart TD
    Start([OCI Request]) --> CheckPath{Path starts<br/>with v2/?}
    CheckPath -->|No| RPMHandler[RPM cache handler]
    CheckPath -->|Yes| V2Health{v2 or v2/?}
    V2Health -->|Yes| Return200[200 OK<br/>registry/2.0]:::redirect
    V2Health -->|No| ParsePath[Parse path:<br/>registry/name/kind/digest]
    ParsePath --> ValidDigest{Valid digest?<br/>manifests or blobs?}
    ValidDigest -->|No| ForwardUpstream[302 to upstream]:::redirect
    ValidDigest -->|Yes| CheckAllow{In OCI<br/>AllowedPrefixes?}
    CheckAllow -->|No| ForwardUpstream
    CheckAllow -->|Yes| HeadOrigin[HEAD upstream registry]:::network
    HeadOrigin --> OriginUp{Origin available?}
    OriginUp -->|Yes| CheckOciCache{Cached in S3?}
    CheckOciCache -->|No| InvokeUploader[Invoke Uploader async]:::async
    CheckOciCache -->|Yes| TouchCheck{Touch cooldown?}
    TouchCheck -->|Elapsed| InvokeTouchOci[Invoke Touch async]:::async
    TouchCheck -->|Not elapsed| RedirectUpstream
    InvokeTouchOci --> RedirectUpstream[302 to upstream]:::redirect
    InvokeUploader --> RedirectUpstream
    OriginUp -->|No| CheckFallback{Cached in S3?}
    CheckFallback -->|Yes, manifest| ServeInline[200 inline<br/>from S3]:::redirect
    CheckFallback -->|Yes, blob| PresignedRedirect[302 to<br/>presigned S3]:::redirect
    CheckFallback -->|No| ForwardUpstream

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff

Note: Tag-based pulls (/manifests/latest), tag listings (/tags/list), and images not in OciAllowedPrefixes are transparently forwarded to the upstream registry without caching.

Development

See the main README for development workflows.

make lambda-s3-cache/setup  # Install dependencies
make check                  # Lint code
make fmt                    # Format code
make test                   # Run unit tests
make coverage               # Run tests with coverage

Configuration

Lambda functions receive configuration via environment variables (automatically set by CloudFormation):

Variable Handler Uploader Touch Description
S3_BUCKET_NAME ✅ ✅ ✅ S3 bucket name
PRESIGNED_URL_EXPIRATION ✅ Presigned URL expiration (sec)
UPLOADER_LAMBDA_ARN ✅ Uploader Lambda ARN
TOUCH_LAMBDA_ARN ✅ Touch Lambda ARN
TOUCH_COOLDOWN_MINUTES ✅ Min minutes between touch ops
ALLOWED_PREFIXES ✅ Allowed URL prefixes
IMMUTABLE_EXTENSIONS ✅ Extensions for immutable content
STALE_CACHE_BEHAVIOR ✅ Stale behavior: origin or cache
OCI_ALLOWED_PREFIXES ✅ OCI image prefixes to cache
ORIGIN_HEAD_TIMEOUT ✅ OCI origin HEAD timeout (sec)
SENTRY_DSN ✅ ✅ ✅ Optional Sentry DSN

Security & Limitations

Security:

  • S3 bucket has public access blocked; all objects encrypted at rest (AES256)
  • Presigned URLs expire after a certain time
  • IAM policies follow least privilege principle
  • URL prefix allowlist prevents caching arbitrary URLs
  • OCI image allowlist (OCI_ALLOWED_PREFIXES) controls which images are cached
  • Immutable extension filter distinguishes immutable vs mutable content for revalidation
  • OCI caching only handles public images (no auth forwarding)

Limitations:

  • Lambda timeout: 15 min (uploader), 30 sec (handler, touch)
  • Lambda memory: 1024 MB (uploader), 256 MB (handler, touch)
  • S3 object size: Up to 5 TB (AWS limit)
  • API Gateway 6 MB response limit for inline manifest responses (manifests are typically under 100 KB)
  • OCI caching only supports digest-based pulls (sha256:...); tag-based pulls are forwarded to the upstream registry

Design Decisions Record

DDR-1: S3 Lambda OCI cache vs ECR pull-through cache

Decision: Build the OCI cache as a new code path in the existing lambda-s3-cache Lambda, rather than using AWS ECR pull-through cache.

Context: Third-party container images pinned by digest vanish when upstream registries garbage-collect old manifests after tag re-pushes (e.g., quay.io/openshift/origin-oauth-proxy:5.1 digests disappear within hours). This causes ImagePullBackOff failures on pod restarts.

Alternatives: (a) ECR pull-through cache — managed service, no custom code, retains cached digests when upstream is unavailable. (b) images.paas.redhat.com — Red Hat internal registry with existing pull-through proxy orgs for some registries. (c) Artifactory — supports any upstream registry as a remote repository.

Rationale: ECR pull-through cache requires private ECR registries, which need IAM authentication — tokens expire every 12 hours, requiring a CronJob for credential rotation on OCP pods. The S3 Lambda approach serves content via a public HTTPS URL with zero authentication overhead. images.paas.redhat.com was ruled out due to dependency on Red Hat IT (rug-pulling risk if they decommission the proxy orgs). Artifactory was not in use and would add a new service dependency. The S3 Lambda approach reuses existing infrastructure (same Lambda, API Gateway, S3 bucket, custom domain) and adds ~100 lines of OCI handler code ported from the proven CKI dependency-archive.

DDR-2: Origin-first (HEAD-first) strategy vs cache-first

Decision: Use an origin-first strategy: always check the upstream registry first via HEAD request, redirect to upstream when available, and serve from S3 only when upstream returns 404 or errors.

Context: The existing RPM cache uses a cache-first strategy (check S3 first, revalidate with origin). For OCI artifacts, the design choice affects S3 egress costs and cache freshness.

Alternatives: (a) Cache-first — check S3 first, serve from cache if present. (b) Origin-first — check upstream first, fall back to S3.

Rationale: OCI digests are content-addressed and immutable — there is no staleness concern (unlike RPM repo metadata). The origin-first approach minimizes S3 egress costs: when upstream is healthy (the common case), every pull redirects directly to the upstream registry with zero S3 data transfer. S3 is only read when upstream has garbage-collected the digest. This matches the CKI dependency-archive pattern, which is proven in production. The cache populates asynchronously on first access and serves as a safety net, not a performance layer.

DDR-3: Registry hostname in URL path vs per-registry subdomains

Decision: Encode the upstream registry hostname as the first path segment after v2/ (e.g., koji-s3-cache.example.com/quay.io/openshift/origin-oauth-proxy@sha256:...).

Context: The CKI dependency-archive assumes a single upstream registry (OCI_REGISTRY env var). Hummingbird pulls from 4+ registries (quay.io, docker.io, ghcr.io, registry.gitlab.com).

Alternatives: (a) Per-registry subdomains (quay-io.cache.example.com/...) — standard OCI URL format but requires wildcard TLS certificates and DNS records, potentially separate Lambda deployments. (b) Registry hostname in the URL path — single endpoint, handler extracts the registry from the first path segment.

Rationale: Encoding the registry in the path allows a single Lambda, single API Gateway, single custom domain, and single TLS certificate to serve all registries. The /{path+} catch-all route already captures everything. The v2/ prefix naturally separates OCI traffic from RPM cache traffic. The kubelet constructs the correct path automatically from the image reference. This approach requires zero new infrastructure beyond the existing lambda-s3-cache stack.

DDR-4: Jinja macro rewrite vs registries.conf

Decision: Rewrite image URLs at Jinja template render time using a cached_image() macro, rather than configuring registries.conf on OCP nodes.

Context: OCP pods need to pull from the cache instead of directly from upstream registries. The standard approach is registries.conf mirror configuration on the cluster nodes.

Alternatives: (a) Node-level registries.conf via MachineConfig. (b) Jinja macro rewriting image refs at deploy time.

Rationale: Hummingbird has namespace-admin access only on OCP clusters — no MachineConfig, no node-level changes. The Jinja macro approach works entirely at the deployment manifest level: vars.d/*.yml files continue to contain upstream refs (Renovate keeps working unchanged), and the macro rewrites them to cache URLs at render time. This is transparent to Renovate, requires no cluster-admin privileges, and can be disabled with a single variable change.

DDR-5: Scope limited to third-party images

Decision: Only cache third-party images. Hummingbird-owned images (quay.io/hummingbird-ci/* and registry.access.redhat.com/hi/*) are excluded.

Context: The cache exists to protect against upstream registries garbage-collecting digest-pinned images.

Alternatives: Cache all images uniformly.

Rationale: Hummingbird-owned images have additional sha-based tags (commit SHA, build ID) that persist independently of the floating :latest tag. Old digests are not garbage-collected because these immutable tags keep them reachable. Caching them would add S3 storage and Lambda invocations with no benefit. The cached_image() Jinja macro skips rewriting for quay.io/hummingbird-ci/ prefixes.

DDR-6: Separate OCI handler function vs extending RPM cache logic

Decision: Implement _handle_oci_cacheable() as a separate function from the existing RPM cache logic (lambda_handler → _handle_rpm).

Context: Both OCI and RPM caching share the same S3 bucket, uploader Lambda, and touch mechanism. The code could be unified.

Alternatives: Extend the existing RPM handler to handle OCI requests by adding OCI-specific branches.

Rationale: The RPM cache has ETag-based revalidation and configurable stale-cache behavior (origin vs cache) that do not apply to OCI — digests are immutable by definition. Keeping the handlers separate avoids accidentally changing RPM behavior and keeps each code path focused. The OCI handler was ported from the CKI dependency-archive, which is proven in production; staying close to the CKI code structure reduces the risk of introducing untested edge cases.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.29 - RPM CVE Count

A CLI tool to count the number of known CVEs for a given list of RPM packages by querying the Red Hat OSIDB database.

Features

  • Batch processing: Query multiple packages from a file
  • Impact filtering: Filter by CVE severity (CRITICAL, IMPORTANT, MODERATE, LOW)
  • Date filtering: Count only CVEs created after a specific date
  • CSV output: Easy to import into spreadsheets or process with other tools

Prerequisites

  • Red Hat VPN: Must be connected to access the OSIDB database
  • Go 1.21+: For building from source

Installation

Install the latest version:

go install gitlab.com/redhat/hummingbird/tools/rpm-cve-count@latest

Or build from source:

git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/rpm-cve-count
go build

Usage

rpm-cve-count -file <package-file> [-after <date>] [-impact <level>]

Options

Option Description Required
-file Read packages from file (one per line) Yes
-after Count CVEs created after date (YYYY-MM-DD) No
-impact Filter by impact: CRITICAL, IMPORTANT, MODERATE, LOW No

Examples

Create a file with package names (one per line):

# packages.txt
kernel
systemd
openssl
glibc

Count all CVEs for the packages:

$ rpm-cve-count -file packages.txt
kernel,342
systemd,87
openssl,156
glibc,234

Count only CRITICAL CVEs:

$ rpm-cve-count -file packages.txt -impact CRITICAL
kernel,23
systemd,5
openssl,18
glibc,12

Count CVEs created after a specific date:

$ rpm-cve-count -file packages.txt -after 2024-01-01
kernel,45
systemd,12
openssl,28
glibc,31

Combine filters to count CRITICAL CVEs from the last year:

$ rpm-cve-count -file packages.txt -impact CRITICAL -after 2024-01-01
kernel,8
systemd,2
openssl,5
glibc,3

Save results to CSV:

rpm-cve-count -file packages.txt > results.csv

Output Format

CSV format with two columns:

  • Package name
  • CVE count

Development

# Build
go build

# Run tests
go test ./...

# Install locally
go install

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.30 - Hummingbird Dashboard

Web dashboard and CLI for monitoring Konflux build pipeline status.

Features

  • Web Dashboard - Real-time view of build, test, and release status
  • Commits View - Detailed pipeline status per commit with expandable releases
  • Components View - Latest status per component grouped by state
  • Merge Requests View - Track build status across MR versions
  • CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
  • Failed Releases - View and manage failed releases with LLM-powered analysis
  • Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
  • Auto-Rerun - Automatically retry transient release and push build failures
  • Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
  • Smart Triggering - Application-aware rules for determining affected components
  • Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
  • Blame Page - /cve/blame view listing late unblamed tickets with per-category breakdown and per-ticket stage detail
  • ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
  • Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks
  • RPM Tracker API - Read-only snapshot of tracker-owned tables in the shared database
  • RPM Tracker Views - Package currency, upstream staleness, merge based rollout lag, and stuck update MRs

Prerequisites

Installation

cd hummingbird-dashboard
pip install -e .

Usage

Local Development

cd hummingbird-dashboard

# Option 1: Port-forward to production database
./dev.sh port-forward  # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start  # In terminal 2

# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start

The dashboard runs at http://localhost:8080 with live reload.

The dashboard’s Alembic migrations do not create the components table: it belongs to hummingbird-status. With a fresh local database created by dev.sh db-start alone, every page that lists applications fails with relation "components" does not exist, so run hummingbird-status/dev.sh db-init first (or hummingbird_status.db.create_schema() when the venv lacks boto3).

Web UI

Endpoint Description
/ Dashboard with all applications
/apps/{app}/commits Commits with build or promotion PipelineRuns and releases
/apps/{app}/components Component status from builds, promotions, and releases
/mrs Merge requests overview (filterable)
/mr/{project}/{iid} MR builds and the post-merge build or promotion
/releases/failed Failed releases with analysis
/releases/blocked Blocked snapshots management
/releases/analysis-log LLM analysis run history
/cve/status CVE ticket status dashboard
/cve/run-log CVE analysis run history
/cve/open Open CVE trackers with SLO status
/cve/closed Closed tickets with VEX reconciliation
/cve/r-time CVE R-Time (notified to done) metrics
/cve/blame Late unblamed tickets and per-category blame breakdown
/rpm/rollout RPM currency, staleness, lifecycle, and update MRs
/rpm/stuck-mrs Stuck RPM update MRs, including unknown ages
/availability Estimated image pull availability (SLI-UE1-B)
/health Health check endpoint
/metrics Prometheus metrics

For RPMs configured for MR artifact promotion, the post-merge PipelineRun promotes the tested MR image. The commits and MR detail views label that run as a promotion and follow its Component Snapshot to the automatic Release. Component status counts a successful promotion and Release as current. If a retry reuses the Component Snapshot, the dashboard also links it through the retry’s COMPONENT_SNAPSHOT result. Failed promotions offer the manual retrigger button: the clone keeps the merge SHA, gets a fresh git-auth secret, and the promotion Task reuses the deterministic Component Snapshot when its evidence still matches. Build throughput and failed push-build auto-rerun continue to count actual builds only.

Snapshot lookups use the indexed source_plr column or the Snapshot name from the promotion PipelineRun result. This also covers historical promotion Snapshots that were ingested before their annotation was stored in source_plr. The dashboard summary validates the newest successful run per component first, then checks older runs in one batch when a required test failed or a promotion has no successful Release.

Each CVE tracker page (/cve/status, /cve/open, /cve/closed, /cve/r-time) has a “Download CSV” button at the bottom of the page that downloads the current view (same filters as the page) as a CSV file, via a matching /cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID” toggle in the header reveals the CVE ID next to each HUM ticket key (HUM-1234/CVE-2026-56789); the preference persists via localStorage.

On /cve/status, a “Show no-CVE” toggle next to it reveals HUM tickets that have no assigned CVE ID in the category drawers. They are hidden by default, and the preference persists via localStorage.

/cve/r-time/csv appends one column per Stage Breakdown milestone (HUM Ticket Created, CVE Published, OSIDB Flaw Created, OSIDB Flaw Public, OSIDB Affect Created, Upstream Fix Merged, Fedora Update Available, RPM Published to Pulp, Image Rebuilt on Quay, HUM Ticket Closed, VEX Updated, CVE Embargo Lifted) after the summary columns, in the same chronological order used in the per-ticket Stage Breakdown panel. Each cell is an ISO 8601 timestamp, or blank when that milestone has no data for the ticket.

On the CVE Tracker Status, Open, and Closed pages, a compact outline icon beside a HUM ticket indicates the CVE Agent’s final Jira verdict label, not a confirmed Jira resolution. Hover or focus the ticket for the recommendation; the “Agent verdicts” help control explains the icons. One policy table per verdict drives the icon, the expected closure, the automatic review and the failure handling. It mirrors hummingbird_cve_agent/labels.py and hummingbird_cve_analysis/lib/states.py without adding a build dependency.

Closed-tab scoring

Verdict Expected resolution Treatment
needs_change, already_fixed Done-Errata Matching closure agrees; contradictory closure disagrees
not_affected Not a Bug Matching closure agrees; contradictory closure disagrees
needs_human, error — Failure
no_upstream_fix — Reviewable, excluded from closure scoring

Accuracy is matches divided by comparable closures plus failures, within the selected period. A current saved review overrides Jira closure for both the row label and accuracy, so a reviewed no_upstream_fix counts while it stays current; stale reviews fall back to Jira closure. Tickets without a verdict and incomparable resolutions without a review (such as Won't Do or a missing resolution) are excluded; no score is shown without scorable results. Closure agreement establishes neither decision correctness nor MR quality.

Reviews

Review Behavior
Automatic Derived without database writes, for failures and for not_affected closed as Not a Bug with no linked agent MR and no VEX mismatch
Saved Takes precedence over an automatic review; a non-stale decision overrules the closure comparison
No upstream fix Can confirm the historical judgment; recorded upstream_fix_merged evidence marks the verdict outdated and its review stale

The agent never re-investigates a recorded verdict, and saving another review does not clear the no-fix stale flag. Automatic re-investigation is not implemented.

RPM Tracker Views

The Rollout view supports package search, status and packaged stream lifecycle filters, and sortable columns. Summary counts describe the full inventory before filters. By default, All packages lists current and behind packages first, with unmonitored and unknown packages last. Each group is sorted alphabetically. Explicit column sorting overrides this default grouping. Unavailable sort values remain last in either direction.

Each row shows the packaged and eligible upstream versions, currency, staleness, merge based rollout lag, packaged stream lifecycle, and a selected open update MR. Staleness starts at the first newer eligible upstream release, not the most recent one. Rollout lag is available only for current packages whose version equals the selected upstream version and has a resolved merge time. Missing or inconsistent dates remain unavailable. The merge timestamp is a proxy, not proof of successful publication. A fallback MR is labeled Package candidate only, not a verified update to the selected upstream version. Unsupported streams flag support risk without changing downstream CVE policy.

The Stuck MRs view lists every open update MR older than 24 hours, plus MRs with unknown creation based ages. It includes multiple MRs for the same package and MRs whose packages are absent from the inventory. The view also supports listing all open update MRs and searching by package or MR title. Unknown ages are not shown as zero or silently excluded.

Both views read the rpm_tracker schema through this dashboard deployment’s existing DATABASE_URL configuration. The preprod dashboard reads the preprod Status database, and the production dashboard reads the production Status database. No separate tracker URL, data copying, or environment selector is required. The dashboard does not migrate or write tracker tables.

Rollout reads are limited to rollouts for currently packaged versions. The MR view and listing API do not read package inventory or rollout history. Retained historical rollouts remain available through GET /api/rpm-upstream-state.

The pages show the last successful collection and independent MR reconciliation time. Collection older than eight hours displays a warning while retaining the last successful inventory. Package rows show their check time, upstream monitor scan state, last successful scan, and scan errors separately. A missing or incompatible schema produces a readable HTTP 503 page, rather than an empty inventory. Database connection failures and pool timeouts also return HTTP 503, with a temporary unavailability message rather than a schema warning. These failures are logged without exposing connection details on the page. A schema with no successful collection or MR reconciliation shows the corresponding pending state. See RPM Tracker for collection, reconciliation, and migration details.

Examples:

/rpm/rollout?status=behind&search=openssl
/rpm/rollout?status=current&lifecycle=active&sort=rollout_lag_hours&order=desc
/rpm/stuck-mrs?stuck_only=false&search=openssl

On /competitive, click trend for an image to compare Hummingbird and Chainguard CVE counts over the past 30 days. Distributor series share a chronological time axis and value scale, with distinct colors and legend labels. Hover over a point to see its distributor, collection date and time, and CVE count. Missing observations are not filled with zero, and explicit null counts leave gaps in the line. Points with invalid collection timestamps are omitted. Opening another image clears the previous chart immediately, so a failed request cannot leave another image’s data displayed. Data collection and CVE calculations are unchanged.

Competitive CVE Diff

On /competitive, click the Delta value on any row to expand a drawer listing the CVE IDs that differ between the Hummingbird and Chainguard scans for that image. A + marks a CVE ID reported only by Hummingbird; a - marks a CVE ID reported only by Chainguard. This is a scan result difference between the two tools, not a confirmed affectedness determination — reasons a CVE shows up on only one side include differing scanner databases, matching heuristics, and false positives/negatives, not necessarily that the other side is truly unaffected. CVE IDs both sides report are shared and excluded from the diff. Each CVE ID links to its Red Hat CVE page; when a HUM Jira ticket tracks it, the ticket is shown and linked alongside it (HUM-5678/CVE-2026-1234). If a scan predates CVE ID collection, the drawer shows a notice instead of a misleading empty diff.

GET /api/competitive/comparison (the page’s main listing) returns only per-image counts and delta — not the CVE ID arrays, HUM ticket links, or even whether per-CVE detail is available — with hundreds of tracked images, computing and shipping that detail for every row up front would be wasted work for the rows no one expands. A row’s drawer content is fetched from GET /api/competitive/diff?image_name=...&tag=... the first time it’s opened, then cached client-side so reopening it doesn’t refetch; that response’s own cve_detail_available decides whether to render the diff or a “check back later” notice.

Both sides of a scan can report a native GHSA or Go vulnerability DB ID instead of a CVE for some ecosystems (e.g. Go modules in an image like cosign): Chainguard’s own Grype scan and the Hummingbird catalog API’s Grype-based scan alike. The collector resolves these against grype’s local database before storage (see chainguard-comparison.md), so by the time data reaches the dashboard, only IDs with no CVE anywhere in that database remain unresolved. These are shown as plain text with no link and no HUM ticket match attempted, rather than resolved here: that would require an OSV.dev network call per unresolved ID, which is unreliable (many environments, including this one, can’t reach it) and, done sequentially, slow enough to hang the drawer open for tens of seconds on an image with several unresolved IDs.

A ticket is indexed under every CVE ID in its cve_ids field (from hummingbird_cve_analysis’s dedicated CVE ID Jira field — see hummingbird-cve-analysis.md), since a single ticket can legitimately track more than one CVE; tickets collected before that field existed fall back to the single first CVE-format label found. The Competitive page compares whole images, but HUM tickets are filed per affected RPM. A CVE in a single bundled dependency (e.g. openssh) can affect several unrelated images (e.g. both git and curl) with no per-image ticket at all. Tickets resolved known_not_affected (VEX “Not a Bug”) are excluded from matching entirely — a ticket confirming its own package isn’t actually impacted must never be suggested as if it were tracking the CVE for some other image. Of the remaining candidates, each one’s catalog_images field (every {image}:{stream} its package actually appears in, from hummingbird_cve_analysis’s SBOM-derived catalog source map — see hummingbird-cve-analysis.md) is checked first: an exact match there is ground truth, so it’s used directly. Older tickets collected before that field existed fall back to a package-name heuristic (an exact {image}{stream} match, the Red Hat packaging convention for versioned streams like mariadb11.8, then substring overlap). When nothing disambiguates to a single ticket either way, every remaining candidate is linked rather than guessing one or hiding all of them.

The image name/tag checked against catalog_images is the catalog’s own identity (chainguard_scans.catalog_image_name / catalog_stream, populated by chainguard-comparison.md’s collector — see below), not the Competitive page’s “common” display name or parsed version: a few images are shown under a cross-vendor alias (Hummingbird’s valkey image is displayed as redis, to pair with Chainguard’s image of that name) and the page’s parsed version does not always match the catalog’s own stream label (e.g. a canonical tag 8.23.0 parses to 8.23 here while the catalog calls that stream 8). Comparing the display name/version against catalog_images would spuriously reject tickets that actually apply. Scan rows collected before catalog_image_name / catalog_stream existed fall back to the display name/tag, matching the pre-migration behavior exactly.

A candidate is only excluded by a non-matching catalog_images when that image/stream was actually checked during the ticket’s collection run. hummingbird_cve_analysis’s catalog source map silently skips an image/stream whose tag listing or SBOM fetch failed (see hummingbird-cve-analysis.md), so a nonempty catalog_images list is not necessarily complete; each ticket’s catalog_unverified_images field lists exactly which image/stream pairs that run couldn’t verify, and a candidate matching one of those is kept as plausible (falling through to the package-name heuristics) rather than excluded.

Only the most-recently-collected tag is shown per image; if a tracked image’s version changes between collection runs (e.g. curl 8.22 -> 8.23), the superseded tag’s row is dropped rather than lingering alongside the current one.

CVE Search API

GET /api/cve-runs/search — search CVE analysis run history.

Parameter Type Default Description
package string — Filter by package name (substring match)
ticket string — Filter by ticket key (substring match)
q string — Free-text search across ticket details
log string — Search CronJob log output
days int 30 Time window in days (1–365)
limit int 500 Max results (1–2000)

At least one of package, ticket, q, or log is required.

The response includes two result sets:

  • results — per-ticket matches from the JSONB details column
  • log_matches — per-run matches from CronJob log output, each with a log_excerpt showing the matching lines in context

Example:

# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {recorded_at, log_excerpt}'

# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
  | jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'

# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'

CLI

# View latest commit status
hummingbird-dashboard --application myapp --format table commit

# View specific commit
hummingbird-dashboard --application myapp commit abc1234

# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10

# Component status overview
hummingbird-dashboard --application myapp component

# JSON output
hummingbird-dashboard --format json commit | jq .

# Auto-rerun failed releases
hummingbird-dashboard auto-rerun

# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures

# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force

REST API Reference

All endpoints return JSON. Read-only GET endpoints are unauthenticated. Interactive OpenAPI/Swagger documentation is available at /docs.

RPM Tracker

GET /api/rpm-upstream-state returns the collection generation and checkout SHA with upstream, package status, rollout reasons, and all open update MRs from the independently migrated rpm_tracker schema. It also exposes the MR reconciliation scope and freshness and one selected blocking_mr per package row. GET /api/rpm-update-mrs lists every open update MR independently of that selection, including multiple MRs for one package. Add ?stuck_only=true for MRs older than 24 hours and MRs with unknown creation-based ages; unknown ages remain null rather than disappearing. The dashboard does not create or write these tables. Run the tracker initializer before deploying this API; until a compatible tracker schema is present, this endpoint returns HTTP 503. See RPM Tracker for migration and storage details.

Tier 1 — Core Pipeline Status

GET /api/apps/{app_name}/commits

Pipeline status for commits in an application.

Parameter Type Default Description
sha string — Filter by commit SHA (prefix)
component string — Filter by component name
limit int 10 Max commits to return (max 100)
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .

GET /api/apps/{app_name}/components

Latest build status per component, grouped by state.

Parameter Type Default Description
search string — Filter components by name (substring)
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .

Tier 2 — Operational Visibility

GET /api/dashboard

Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.

curl -s 'http://localhost:8080/api/dashboard' | jq .

GET /api/releases/failed

List of currently failed releases with analysis results.

Parameter Type Default Description
application string — Filter by application name
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .

GET /api/releases/{name}/analysis

LLM failure analysis for a specific release (root cause, classification, recommendation).

curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .

GET /api/releases/auto-rerun-log

History of automatic rerun attempts.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .

GET /api/releases/analysis-log

History of LLM analysis runs.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .

GET /api/releases/analysis-costs

Cumulative token usage and estimated cost for LLM analysis runs.

curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .

GET /api/mrs

Merge requests across monitored repositories.

Parameter Type Default Description
state string opened MR state: opened, merged, closed, all
project string — Filter by project (repository) name
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .

GET /api/mrs/projects

List of all projects (repositories) with tracked merge requests.

curl -s 'http://localhost:8080/api/mrs/projects' | jq .

GET /api/mr/{project}/{iid}

Detail for a single merge request, including all versions and per-version build status.

curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .

GET /api/cve/status

Current CVE ticket status across all tracked packages.

curl -s 'http://localhost:8080/api/cve/status' | jq .

GET /api/cve/open

Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).

Parameter Type Default Description
slo float 24 SLO threshold hours (24, 72, or 168)
include_next_release bool true Include cve-next-release tickets
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .

GET /api/cve/closed

Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation (HUM-5843). Analysis stores package-scoped vex_status and Jira resolution; the dashboard computes MATCH from those facts (Done-Errata → fixed, Not a Bug → known_not_affected, package_not_listed, or fixed – a downstream package rebuilt by an unrelated dependency’s fix is not a contradiction of a Not a Bug verdict). Shows Hummingbird vex_status, match state (matched / pending / mismatch), and vex_resolved (first scan time where VEX agreed with the Jira resolution). Event metadata is merged across a ticket’s rows in occurred_at order (HUM-6091): later non-empty fields overlay earlier ones, so Close/VEX facts are not stuck on the first cve_published row. The Resolution column uses Jira Closed / {resolution} when stored analysis text is not already Closed. The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat CSAF VEX document when the package is named in CSAF. package_not_listed (Not a Bug, package absent from the document) shows N/A with no link: this ticket did not produce a VEX change. Tickets without a VEX match status are counted below the summary cards. The awaiting-vex label count overlaps the matched, pending, mismatch, and unknown groups; it is not an additional category to add to the total.

The HTML page renders 100 tickets per page (page query, 1-based) while the summary cards, CVE Agent performance and CSV export cover the whole period.

Parameter Type Default Description
days int 28 Time window (0 = all time)
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .

Entries carry the agent verdict, closure agreement, linked MRs, any recorded ticket cost (agent_cost, with a null estimated_cost_usd when the analysis predates cost reporting, and unpriced_calls when some calls had no pricing) and any saved review; agent_stats aggregates them as described above, including cost_total, cost_tickets, cost_unknown and cost_partial.

PUT /api/retrigger/cve-agent-review/{ticket_key}

Save a reviewer’s needed and mr_outcome, plus an optional note, for a closed CVE ticket. Requires login; the reviewed verdict, resolution and MR identities/head SHAs are snapshotted server-side. Returns 422 when the ticket has no agent verdict. The response returns status, ticket_key, the ticket’s updated agent_outcome_label, agent_outcome_tone and agent_outcome_note, and the reviewed_text byline, so the page updates the row and review panel in place without reloading. The save reads only the reviewed ticket, not the whole Closed tab; period statistics refresh on the next page load.

curl -s -X PUT 'http://localhost:8080/api/retrigger/cve-agent-review/HUM-123' \
  -H 'Content-Type: application/json' \
  -d '{"needed": "backport", "mr_outcome": "edits", "note": "test needed fixing"}'

GET /api/cve/r-time

CVE R-Time (notified to done) for Done-Errata tickets measures time spent on a fixable CVE, so its start never predates the upstream fix in either mode. Include CVE-HUM starts at CVE publication (NVD datePublished / cve_published), falling back to HUM ticket creation only when CVE publication is missing – CVE publication wins even when the HUM ticket predates it, so an embargoed ticket’s R-Time starts at public disclosure rather than at ticket creation (which can predate disclosure by days while the CVE is still privately embargoed). Exclude CVE-HUM starts at the later of HUM created and CVE publication – same as HUM created for a normal ticket (Exclude’s purpose is to ignore filing lag after disclosure), but an embargoed ticket’s clock must never start before public disclosure in either mode, so it also starts at CVE publication there. Whichever of the two the mode picks, Upstream Fix Merged is then applied as a floor on top: the final notified timestamp is the later of that value and Upstream Fix Merged, so neither mode ever counts time before a fix existed to backport. Delivery is the catalog image publish when the package is a catalog image source, otherwise the RPM publish. Include VEX requires delivery, VEX updated, and Closed / Done-Errata; Exclude VEX requires delivery and close only. The done timestamp is the latest of the required gates, and only when all of them exist. Incomplete tickets stay on the table with elapsed time until now; the days window filters by completion (fallback: Done-Errata close, then notified), so a period means completed in that window – the Closed tab filters by Jira close date instead. CVE-HUM is filing lag after NVD (max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is Done-Errata close to VEX feed update (max(VEX − close, 0)); close is recorded when the advisory MR merges. Fedora is not start or end; Upstream Fix Merged only ever gates R-Time’s start, never its end. Pre-built (delivery before notification) is informational only. Rows without catalog_image_source in metadata still require an image until the collector restamps them. Each entry includes package onboarding timestamps and delay analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours). Default excludes cve-next-release, CVE-HUM time, and VEX time.

Parameter Type Default Description
days int 28 Time window (0 = all time)
slo float 168 SLO threshold hours (24, 72, or 168)
include_next_release bool false Include cve-next-release tickets
include_hum_cve_time bool false Include NVD-to-HUM filing lag in R-Time
include_vex_time bool false Require VEX feed update before R-Time ends
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .

GET /api/cve/run-log

History of CVE analysis CronJob runs.

Parameter Type Default Description
limit int 100 Max entries
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .

POST /api/cve/agent-cost-keys

Report which of the posted tickets already have a recorded CVE agent cost, so the collector downloads only the remaining analysis attachments. The check returns keys only and uses the unique ticket-key index. Requires the same bearer token as /api/cve-report, which accepts the costs as an agent_costs list (ticket_key, input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens, unpriced_calls, estimated_cost_usd, where a null cost marks a pre-cost attachment) and rejects the whole report when a row is malformed. The cache counters are subsets of input_tokens, and a non-zero unpriced_calls makes the cost a lower bound.

curl -s -X POST 'http://localhost:8080/api/cve/agent-cost-keys' \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"tickets": ["HUM-123", "HUM-124"]}' | jq .

GET /api/cve-export

Export CVE sync data as JSON for prod-to-preprod replication. Valid section values are cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame.

Optional pagination parameters:

Parameter Type Default Description
section string — Section to export
limit int 2000 Rows per page
offset int 0 Pagination offset
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'

curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
  | jq '.count,.has_more'

# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
  | jq '.rows[] | select(.event_type == "rpm_first_published")'

Event type validation. The import path (POST /api/cve-import) validates every event_type value against a canonical allowlist defined in hummingbird_dashboard/sources.py:

  • CANONICAL_TICKET_EVENT_TYPES — values accepted for cve_ticket_events: cve_published, osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, osidb_flaw_public, hum_ticket_created, hum_ticket_closed, upstream_fix_merged, fedora_update_available, rpm_fix_published_to_pulp, image_rebuilt_on_quay, vex_resolved.
  • CANONICAL_PACKAGE_EVENT_TYPES — values accepted for package_lifecycle: rpm_first_published.

Legacy names (e.g. jira_created, delivered_in_rpm, …) are mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP. Unknown names are rejected with HTTP 400.

POST /api/cve-import

Replace local CVE sync data from a payload previously returned by /api/cve-export.

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data @cve-export.json \
  'http://localhost:8080/api/cve-import' | jq .

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
  'http://localhost:8080/api/cve-import' | jq .

Ticket-event types such as osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, and osidb_flaw_public import with the rest of cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.

Analysis-log exports include run_started_at when available. Paginated imports ignore rows whose non-null identity is already present, making retries safe; legacy rows without an identity remain append-only. The ingest timestamp is exported as recorded_at; imports still accept the legacy run_at key, but a sync target older than this release cannot read the new key.

occurred_at conflict resolution on import matches the live collector path (HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable delivery timestamps where a non-null incoming value wins, so a sync can update them, without letting a null incoming value erase an already-recorded delivery timestamp; every other event_type is write-once, keeping the existing timestamp once set so a sync or backfill re-run cannot overwrite an established immutable milestone.

Bulk prod-to-preprod replication of cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame no longer goes through this API — cve_analysis_log rows can carry up to 512 KiB of log_output, so paginating that table meant hundreds of small requests that aborted entirely on the first 5xx. See copy_prod_to_preprod_db in CVE Analysis Reference for the direct pg_dump/pg_restore sync tool that replaced it. These endpoints remain in use for the smaller, targeted writes collect_rpm_first_published and backfill_osidb_timestamps.py make directly against prod or preprod.

Tier 3 — Analytics & SRE Metrics

GET /api/stats/build-throughput

Build counts over time (successful, failed, total).

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .

GET /api/stats/release-success-rate

Release success/failure ratios over time.

Parameter Type Default Description
days int 30 Lookback window in days
application string — Filter by application
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .

GET /api/stats/auto-rerun-effectiveness

Success rate and time-to-resolution for automatic reruns.

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .

GET /api/components/staleness

Components ranked by time since last successful build.

curl -s 'http://localhost:8080/api/components/staleness' | jq .

GET /api/releases/similar-failures

Find releases with error messages similar to a given string.

Parameter Type Default Description
error string — Error text to match against
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .

GET /api/apps/{app_name}/last-successful

Timestamp and SHA of the last fully successful pipeline per component.

curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .

Tier 4 — System

GET /api/service-status/json

Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).

curl -s 'http://localhost:8080/api/service-status/json' | jq .

GET /api/settings

Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).

curl -s 'http://localhost:8080/api/settings' | jq .

GET /health

Enriched health check returning service version, uptime, database connectivity, and dependency status.

curl -s 'http://localhost:8080/health' | jq .

Configuration

Environment Variables

Variable Default Description
DATABASE_URL postgresql://...localhost:5432/ PostgreSQL connection URL
PORT 8080 Web server port

Authentication Variables

These control the retrigger functionality (requires OAuth proxy in production):

Variable Default Description
TRIGGER_AUTH_MODE oauth oauth (production) or local
TRIGGER_AUTH_GROUP konflux-hummingbird-admin-access OpenShift group required to retrigger
TRIGGER_LOCAL_USER local-dev Username when TRIGGER_AUTH_MODE=local
CVE_REPORT_TOKEN Bearer token for CVE API auth

The dashboard checks TRIGGER_AUTH_GROUP against the user’s groups on the OpenShift cluster hosting the dashboard. Access on the Konflux cluster alone does not grant retrigger permission. Membership in the default konflux-hummingbird-admin-access group is managed in Rover and synchronized to the dashboard cluster by the infrastructure configuration. Claiming a CVE ticket requires only a dashboard login, so the user menu can show Claim ✅ and Retrigger ❌. Successful group lookups are cached for five minutes; failed lookups are retried on the next request.

For local development with retrigger enabled:

TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start

Auto-Rerun Variables

These control the auto-rerun cronjob and failure analysis:

Variable Default Description
AUTO_RERUN_MIN_AGE_MINUTES 30 Minimum failure age before retrying
AUTO_RERUN_MAX_RETRIES 3 Max rerun attempts per snapshot+plan
KONFLUX_KUBECONFIG_PATH Path to Konflux kubeconfig file
RELEASE_NAMESPACE Namespace where releases are created
MANAGED_NAMESPACE Namespace for fetching release PLRs
KUBEARCHIVE_URL KubeArchive API URL for archived resources
GOOGLE_APPLICATION_CREDENTIALS Path to GCP SA key for Vertex AI
GOOGLE_CLOUD_PROJECT GCP project ID for Vertex AI
ANALYSIS_MODEL_API_KEY Gemini API key (fallback if no GCP creds)
ANALYSIS_MODEL gemini-2.5-flash LLM model for failure analysis
ANALYSIS_MODEL_REGION global Vertex AI region
MAX_ANALYSES_PER_CYCLE 5 Max releases to analyze per cycle
SLACK_WEBHOOK_URL Slack webhook for rerun/analysis notifications
DASHBOARD_URL Dashboard base URL for Slack links
GITLAB_SLACK_MAP_PATH /etc/hummingbird/gitlab-slack-map.yaml GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention)
PUSH_RERUN_MIN_AGE_MINUTES 10 Min push build failure age before retrying
PUSH_RERUN_MAX_RETRIES 3 Max retry attempts per component+sha

Error Budget Alert Variables

The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:

Remaining budget Slack alert
≤20% Warning
≤5% Critical
0% Exhausted (sprint-stop)

Incoming webhooks always post to the channel they were created for; the payload cannot select #team-hummingbird. Create the webhook for that channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared SLACK_WEBHOOK_URL at that webhook).

State is stored in dashboard_settings (error_budget_rtime_alert_state) so the same threshold is not re-alerted. The stored state is claimed with a compare-and-set before Slack is called, and rolled back if the post fails, so a transient Slack error is retried on the next interval and concurrent evaluators cannot double-post the same transition. Recovery updates the stored state silently so a later re-worsening can alert again. No message is sent when the webhook URL is unset, or when alerts are disabled via the R-Time page toggle (error_budget_alerts_enabled in dashboard_settings; default enabled).

Variable Default Description
ERROR_BUDGET_SLACK_WEBHOOK_URL SLACK_WEBHOOK_URL Incoming webhook created for the alert channel (typically #team-hummingbird)
SLACK_WEBHOOK_URL Fallback webhook if the error-budget URL is unset
DASHBOARD_URL Base URL for the R-Time deep link in alerts
ERROR_BUDGET_ALERT_INTERVAL_SECONDS 900 Seconds between background checks

Failures

The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.

Failed Releases

The /failures/releases page shows all failed releases with:

  • LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
  • Failure Classification — Transient, Configuration, Code, External Service, or Unknown
  • Rerun History — Past rerun attempts and outcomes per snapshot
  • Error Pattern Matching — Auto-block snapshots matching known error patterns
  • Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
  • Analysis Toggle — Enable/disable LLM analysis from the dashboard

CLI Subcommands

The auto-rerun subcommand retries eligible failed releases:

hummingbird-dashboard auto-rerun --application myapp

The auto-rerun-push subcommand retries eligible failed push builds:

hummingbird-dashboard auto-rerun-push --application myapp

Before creating a retry PipelineRun, the command records an atomic claim for its component and commit in push_build_rerun_claims. Later CronJob runs skip that pair until the claimed PipelineRun is recorded as failed in the dashboard database. A successful or still-running retry prevents another attempt, and PUSH_RERUN_MAX_RETRIES also applies to claims that are not yet ingested. If the create request has an uncertain outcome, the claim is retained to avoid accidentally creating a duplicate. Check the claimed plr_name in Konflux before manually clearing such a claim.

The analyze-failures subcommand runs LLM analysis on unanalyzed failures:

hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant

All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. Release success messages follow reruns tracked by the previous auto-rerun run, so delayed status ingestion does not skip a success. Failed Slack posts are recorded as pending in the auto-rerun log and retried on the next run.

Image Pull Availability

The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually pull the images we publish. Continuously pulling every catalog image on a tight schedule is not feasible (terabytes of registry traffic), so the check is split into three tiers, run by check-image-pulls:

  • Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes transferred, against every latest* image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours.
  • Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
  • Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.

Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.

  • SLO: 95% target, 28-day rolling window, based on the manifest tier
  • Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
Option Description
--dry-run Probe without writing results to the database
--force Force live manifest HEAD checks (ignore digest cache)
--catalog-api-base Catalog API base URL (default: CATALOG_API_BASE or built-in)

Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be disabled from the dashboard via the image_pull_checks_enabled setting.

Merge Requests

The /mrs page shows merge requests across all monitored repositories with:

  • State filter - Open, merged, closed, or all MRs
  • Project filter - Filter by specific repository
  • Build status - Aggregate status across all components

The /mr/{project}/{iid} detail page shows:

  • All MR versions - Each head commit SHA that was pushed to the MR
  • Build status per version - Full pipeline status (build, snapshot, test, release)
  • Links to Konflux UI - Direct links to PipelineRuns and Snapshots

Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.

Component Status

The dashboard tracks component build status with these states:

Status Icon Description
Success ✅ Build passed for expected commit
Superseded 🔄 Expected commit not built, but newer succeeded
Failed ❌ Build failed for expected commit
Stale ⚠️ Build not triggered for expected commit
Running ⏳ Build in progress
Missing ❓ No build found

Components are grouped by status: Failed/Stale → Running → OK → Missing.

Trigger Rules

The dashboard uses application-specific rules to determine which components are affected by a push:

  • containers: Changes in images/{component} trigger builds
  • rpms: Changes in rpms/{component} or mock/mock.cfg trigger builds
  • tools: Changes in {component} directory trigger builds

Certain files are excluded from triggering (README, templates, etc.).

Development

See the main README for development workflows.

Reusable text filtering

hummingbird_dashboard/web/static/text-filter.mjs exports initTextFilter(root). It is independent of RPM data and can enhance other dashboard tables.

Mark the input with data-text-filter-input and each searchable row with data-filter-text. The row attribute contains the text to search, such as a package name or a combination of ticket key, package, and CVE ID. The optional count, table wrapper, and empty message use the attributes shown below. Matching ignores case and surrounding whitespace, and leaves table order intact.

<div data-text-filter>
  <form method="get">
    <label for="package-search">Package</label>
    <input
      id="package-search"
      name="search"
      type="search"
      data-text-filter-input
    />
    <button type="submit">Apply filters</button>
  </form>
  <p aria-live="polite">Showing <span data-text-filter-count>1</span></p>
  <div data-text-filter-table>
    <table>
      <tbody>
        <tr data-filter-text="example-package">
          <td>example-package</td>
        </tr>
      </tbody>
    </table>
  </div>
  <p data-text-filter-empty hidden>No matches.</p>
</div>
<script type="module">
  import { initTextFilter } from "/static/text-filter.mjs";
  document.querySelectorAll("[data-text-filter]").forEach(initTextFilter);
</script>

The input’s name is the URL query parameter. Mark sort or filter links with data-text-filter-link to preserve the typed text when navigating. An unnamed input filters rows without changing URLs. Components are scoped to their root, and a form is optional. Use a dedicated GET filter form when server controls are present. Text alone does not submit a request, but changing another form field still submits normally.

For expandable tickets, put data-filter-text on a shared row group, such as a tbody, rather than only the main row. The group hides together while its details keep their expansion state.

Render all rows allowed by the other filters, not just the initial text matches. Use hidden on initial nonmatching rows and set the initial count and empty message on the server. Paginated tables should keep server filtering instead. RPM Tracker uses this module now; adding it to CVE Tracker is separate work.

Running Tests

cd hummingbird-dashboard
pip install -e ".[dev]"
pytest
node --test tests/*.mjs

Browser logic tests use the Node.js built in test runner and are also included in make node-test-host from the repository root.

The PostgreSQL migration regression test uses the CI PostgreSQL sidecar via DATABASE_URL. Locally, set DATABASE_URL to a disposable PostgreSQL database to run it; without one, the test is skipped.

Building Container Image

cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .

Database Migrations

The dashboard uses Alembic to manage its schema. Migrations run automatically on every startup via alembic upgrade head.

On first startup against an existing environment, the dashboard detects that the schema is already present (via the blocked_snapshots sentinel table) and automatically stamps the baseline migration as applied before upgrading. No manual intervention is required.

Local development — migrations run automatically via ./dev.sh db-seed:

./dev.sh db-start
./dev.sh db-seed   # runs alembic upgrade head then seeds test data
./dev.sh start

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.30.1 - Hummingbird Dashboard: RPM upstream rollout tracker

Problem

We lack a clear view of which RPMs are behind upstream, how long they have been behind, and which update MRs are stuck. Staleness is found reactively.

Background

The upstream monitor service discovers, normalizes and classifies upstream releases. It exposes a released_at timestamp per release, groups versions into streams, and carries support lifecycle state sourced from endoflife.date. The timestamp’s meaning depends on the upstream source. It currently monitors about 200 projects.

The rpms repo package metadata carries upstream_repo and sometimes track_upstream, upstream_branch, upstream_version_transform, or version_suffix_strip. The packaged version comes from the spec file. The existing ci/check_upstream_versions.py reads it with rpmspec, which resolves macros. A scheduled upstream_version_update CI job bumps specs, fetches sources, and opens update MRs.

hummingbird-status initializes the shared events PostgreSQL database and ingests GitLab MRs into its public.gitlab_merge_requests table. The dashboard reads that table and owns additional tables for its own features. The CVE Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM tracker instead follows Pipeline Timing: an independent schema owner writes directly to its schema in the shared database, and the dashboard reads the resulting state.

Findings

  1. All 557 packages carry upstream_repo in metadata. Read their packaged versions from the specs with the existing parse_spec_version helper rather than treating the metadata version field as authoritative.
  2. The upstream monitor reports released_at and the release source. The tracker does not need to fetch release dates from multiple upstream APIs, although a reported date is not always the instant a source became public.
  3. The monitor has about 200 projects, not necessarily 200 of our packages. Package coverage must be counted after matching packages to projects. Its published API already exposes project definitions, paginated release history, stream lifecycle, and project scan status.
  4. Stream lifecycle (active, security, extended, unsupported) is exposed per stream. An unsupported upstream stream is a support-risk signal. It does not change our downstream CVE policy on its own.
  5. V1 can use an update MR’s actual merge time as a proxy for rollout when it is available. The planned Phase 2 definition is successful Release completion, confirmed with Benjamin Sherman (HUM-5820). The timeline data model and its resolution path are not yet implemented.
  6. A pipeline timeline alone cannot identify the target package and version of an update MR. That identity must be resolved from the rpms repo’s update MRs.

Proposal

Mirror upstream state from the monitor into the shared events database, compare it against our packaged versions, and render an RPM Tracker tab. A collector maintains the upstream and version rows through tracker-owned transactional storage. The dashboard reads these rows and resolves open update MRs from the existing Status data.

Data

RPM tracker owns and migrates its tables in a dedicated rpm_tracker schema within the shared events database. Table names below refer to this schema unless qualified otherwise. The original event tables in public, including public.gitlab_merge_requests, are owned by hummingbird-status. Keeping these use cases in distinct schemas preserves table ownership while allowing the dashboard to join tracker and MR data across schemas in one query. Run the tracker initializer before exposing the dashboard read API and before each collector rollout; the collector checks the schema revision before collecting or writing. A failed migration or revision check stops collection without changing Status ingestion. Postgres can filter and sort computed staleness alongside the existing MR data.

rpm_upstream_state is a tracker-owned read model. The monitor remains the authority for upstream releases. Anything else that needs upstream state consumes the monitor directly rather than reading our mirror.

rpm_upstream_state: Latest eligible upstream state and packaged stream lifecycle per mapped package, computed from the monitor on every collector run. Unmonitored packages have no upstream row and appear through a left join with rpm_version_status.

Column Meaning
package rpms repo package name (primary key)
project_id Verified monitor project identifier
upstream_version Latest eligible upstream version
upstream_released_at Monitor timestamp for that release
upstream_source Source reported for the upstream release
packaged_stream Stream containing the packaged version, if known
lifecycle_status Lifecycle of packaged_stream, or unknown
monitor_last_success_at Last successful upstream scan for this project
monitor_state Current upstream monitor scan state
monitor_last_error Latest upstream monitor scan error, if any
updated_at When we last wrote this row

Map each package’s upstream_repo to a monitor project using its project definition’s source and a verified repository mapping. If that mapping is not unique, require an explicit upstream_monitor_project_id in package metadata. A project ID is not derivable from upstream_repo alone.

Use track_upstream to select eligible releases as the existing update job does. A version prefix restricts candidates to that line. latest considers all streams. For a package without track_upstream, use an explicit upstream_stream metadata override or the unique stream matching its packaged version. An ambiguous choice yields unknown, not the project’s global latest. When track_upstream is set, an upstream_stream override can identify the packaged stream for lifecycle but cannot silently change the update target. Do not treat upstream_branch as a stream name without a verified mapping. Resolve packaged_stream independently of the update target so lifecycle describes the version we package even when we track a newer stream. Respect the monitor’s prerelease policy and the package’s version_suffix_strip and upstream_version_transform when comparing with RPM versions. The unrelated version_transform metadata field is for CVE version matching. If the monitor already normalized a version in the same way, do not apply a transform twice. Source artifact availability may delay update automation, but does not change whether an upstream release is visible in this tracker.

rpm_version_status: One row per package, holding our side of the comparison and the derived state. Upstream version and release date are read from rpm_upstream_state via JOIN, and rollout timing from rpm_version_rollouts, so neither is duplicated here.

Column Meaning
package Package name (primary key)
current_version Version currently packaged, from the spec file
status current, behind, unmonitored, unknown
first_behind_at Earliest newer upstream release date, if behind
checked_at When the collector last successfully refreshed this row

The dashboard resolves the blocking MR from public.gitlab_merge_requests for display; MR identity and URL are not part of the replaceable inventory. MR creation time is not stored in that table today, so the Stuck MRs view must obtain it from GitLab or the MR-owning feed.

current means the packaged spec version is equal to or ahead of the latest eligible upstream version. behind means an eligible upstream version is newer. unmonitored means there is no verified monitor project mapping. unknown means a mapped project exists, but its update target, release history, spec version, or comparison is unusable.

rpm_sync_state: One row holding the last successful collection time, the rpms checkout SHA, and a generation number. A successful full inventory replaces current tracker rows and increments the generation in the same database transaction. A failed run leaves the previous rows and generation intact.

Column Meaning
id Primary key fixed to 1 for the singleton row
generation Integer incremented after each successful ingest
rpms_checkout_sha Default branch commit inspected by the collector
last_successful_at Completion time of the last successful collection

rpm_version_rollouts: Resolved rollout times, one row per (package, version). A later tracker-owned reconciliation step can populate them from the status-owned public.gitlab_merge_requests table. Keeping these rows separate from the replaceable inventory preserves historical rollouts; the dashboard reads them but does not write them.

Column Meaning
package Package name (composite primary key)
version The version that was rolled out (composite primary key)
rolled_out_at Actual merge time of the matched MR (v1 proxy for publication)
mr_iid IID of the matched MR
mr_project Project of the matched MR
resolved_at When this entry was resolved

Match merged update MRs in public.gitlab_merge_requests by rpms project, package, target version, and target branch. The update job uses a source branch containing the package name and an MR title containing both package and version. A branch match alone does not identify a version. Use GitLab’s actual merged_at from the webhook payload when present. Obtain it from GitLab for older MRs that lack it. Do not substitute the mutable MR updated_at. If the version or merge time cannot be established, or multiple MRs still match, leave rolled_out_at null and show rollout lag as unavailable. Manual bumps without an identifiable package and version remain unresolved rather than silently matching a different MR. Re-resolve unresolved rows as MR data arrives.

Collection and reconciliation

Every four hours, collect a fresh inventory from the rpms checkout and the monitor’s existing REST API. GET /v1/projects lists projects. GET /v1/projects/{projectID}/definition supplies the source configuration used to verify repository mappings. For each distinct mapped project, GET /v1/projects/{projectID}/status reports scan health, GET /v1/projects/{projectID}/streams provides lifecycle, and GET /v1/projects/{projectID}/releases provides dated release history with X-Next-Cursor pagination. Fetch every release page and GET /v1/projects/{projectID} for snapshot releases and observations, including undated versions omitted from pagination. Merge by source and release ID, with snapshot records taking precedence, including dates corrected to unknown. Treat the monitor’s JSON null empty-stream representation as an empty list. Group releases by project and reuse the results for packages sharing a project. Choose the latest eligible version with RPM comparison rather than relying on API page order. The combined releases give the first newer upstream release date; an eligible newer release with no date makes staleness unavailable. This full read also sees changed dates, stream lifecycle changes, and new project mappings without an event cursor or a second database of historical releases. The monitor’s historical catalog preserves observed releases after they disappear upstream; a full read alone cannot identify withdrawals. Excluding withdrawn releases requires an authoritative monitor eligibility signal.

The collector is implemented in rpm-tracker/rpm_tracker/collect.py, with monitor reads in monitor.py and package selection in policy.py. It reads the expected generation before cloning the default branch, reuses the checkout’s spec parser and named transforms, and writes only through store.ingest. See RPM Tracker for runtime prerequisites and usage. Deployment as an OpenShift CronJob remains a subsequent step.

The monitor’s per-project endpoints do not promise one snapshot across all pages or projects. If any required request or page fails, do not replace the last successful inventory. If releases change during collection, the next full run reconciles them. Do not claim that each four-hour run is an atomic view of all upstream projects. Expose the monitor’s project scan state and last_success_at alongside the collector’s own freshness so a successful API read does not conceal failed upstream polling. Prevent overlapping scheduled runs and check the expected rpm_sync_state generation on ingest, including runs with no release changes.

Metrics

  1. Currency (behind / current): compare the spec version against the latest eligible upstream version with RPM semantics. behind means upstream is newer. An equal or ahead packaged version is current. If versions cannot be compared or the update target cannot be identified, use unknown rather than assuming current. These are source repository statuses, not proof of publication.
  2. Staleness (behind packages): Time since first_behind_at, the earliest monitor released_at among eligible releases newer than the spec version. A newer upstream release does not reset the clock. This is based on the monitor’s reported dates and does not measure security exposure. If history or its date is missing, show unavailable, not zero.
  3. Rollout lag (current packages whose version equals upstream_version): Time between upstream_released_at and rolled_out_at. An ahead packaged version is still current, but its lag is unavailable. In v1, rolled_out_at is the matched update MR’s merge time, not proof of successful publication. Historical rollout rows remain stored, but the v1 Rollout view stops showing their lag when a newer upstream release makes the package behind again.
  4. Lifecycle: flag an unsupported upstream stream as support risk. Whether to fix or close CVEs follows the downstream fix_status policy in package metadata, not this upstream flag.
  5. Stuck update MRs: all open update MRs older than 24 hours by GitLab MR creation time, independently of the single MR shown on a package row.

Approach

  1. Deploy the collector as a Kubernetes CronJob on OpenShift. Its batch/v1 schedule starts a Job every four hours, independently of upstream_version_update. Set concurrencyPolicy: Forbid to prevent overlapping scheduled runs. Run the tracker schema initializer before collection starts. Clone the rpms repo’s default branch read-only at the start of each run and record the checked commit SHA.
  2. Read metadata/*.json from that checkout and parse each package’s spec version with the existing parse_spec_version helper. If a spec cannot be parsed, mark that package unknown rather than trusting a possibly stale metadata version. Map packages to projects using verified monitor definitions, fetch all needed project status, streams, and release history, and compute upstream state and first_behind_at with rpm.labelCompare (provided by the collector image). Create a SQLAlchemy engine and read the generation with store.get_generation(engine) before cloning or collecting external data.
  3. Pass the complete package inventory, checked checkout SHA, and expected generation to store.ingest(engine, request). It rejects an outdated generation with GenerationConflictError, atomically replaces upstream and status rows, and increments the generation. Packages removed from a successfully read default branch checkout disappear. An incomplete fetch must not call ingest; a failed database write leaves the previous inventory unchanged. A conflict fails the Job visibly, and the next run collects a fresh inventory. Grant the CronJob database access limited to tracker-owned tables; use a separate migration role for schema changes. The dashboard remains a read-only consumer of tracker data.
  4. A later tracker-owned reconciliation step resolves rolled_out_at for a package/version from matching merged update MRs. The dashboard may show one open update MR on a package row, but the Stuck MRs view queries all open rpms update MRs independently. Obtain MR creation time from the MR-owning feed or GitLab when missing; MRs without a known creation time appear as unknown age, not silently filtered out.
  5. Render an RPM Tracker tab with a Rollout view (version, status, staleness or rollout lag, lifecycle, blocking MR) and a Stuck MRs view, mirroring the CVE Tracker, plus JSON API endpoints. Show checked_at, the rpms checkout SHA, and the last successful collection time. Warn when collection exceeds eight hours without success. Show monitor scan failures and each project’s last successful upstream check separately. unmonitored means no verified monitor project. unknown means a mapped project without a usable update target, release, or comparison. On collection failure, retain the last successful state and show its age.
  6. Keep the existing Slack and failure-file output for update automation. Monitor failed or missed collector Jobs in OpenShift and show the last successful collection time in the dashboard.

MR reconciliation refinements

  • Add rpm_update_mrs in the tracker schema, keyed by (project, iid), as the dashboard’s MR read model. Reconciliation reads the Status table and stores package/version identity, MR state and links, explicit creation/merge times, and uncertainty reasons. Preserve merged identity evidence across later edits and retain known timestamps when later payloads omit them. Only currently qualifying MRs are rollout candidates.
  • Extend rpm_sync_state with mr_project, mr_target_branch, and last_mr_reconciled_at. Fix project/branch scope on the first successful reconciliation and track MR freshness independently of inventory generation and collection freshness. Commit MR and rollout changes under the sync-row lock without changing the inventory generation.
  • Add unavailable_reason to rpm_version_rollouts and define resolved_at as the time the resolution result last changed. Revisit historical keys after inventory advances and invalidate results when new ambiguity appears.
  • The initial reconciliation step uses only explicit timestamps in stored Status webhook payloads. The planned GitLab backfill remains follow-up work; missing timestamps stay unavailable until reliable evidence is supplied.
  • Expose the independent open-MR listing through GET /api/rpm-update-mrs, including its stuck_only filter, and include MR information in GET /api/rpm-upstream-state. Select the single package-row MR from the read model without restricting the independent listing.

Dependencies

  1. Package registration and mapping. Count actual package coverage by verifying metadata against monitor definitions. Provision missing projects separately. POST /v1/projects requires a project ID, name, homepage, source provider configuration, version policy, and stream policy. An upstream_repo URL alone is not sufficient to register a project. A package without a verified project mapping remains unmonitored.
  2. Monitor load. The existing API provides paginated release history, project streams, and project scan status. Confirm that a complete collection of mapped projects fits within the four-hour schedule at a bounded request rate. No new monitor endpoint or cursor protocol is required for v1.
  3. Collector runtime. The CronJob image needs Git, rpmspec, the Python RPM bindings, and the dependencies needed to use the version parsing helper from the rpms repo. Install the tracker package with its bundled migrations, run the initializer before the collector, and give the CronJob schema-scoped database credentials. Provide a read-only GitLab credential if the repo requires one.
  4. MR timestamps. MR reconciliation needs actual merged_at and creation times from the MR-owning feed or GitLab for missing historical data. It cannot infer these times from mutable updated_at.

Acceptance Criteria

  1. The Rollout view lists every package with its status. Behind packages show time since the first newer upstream release and the blocking update MR. Current packages show merge-time rollout lag only when their version equals the selected upstream version and has a matching MR. Unsupported upstream streams are flagged without changing CVE policy.
  2. Packages not registered with the monitor are shown as unmonitored, not omitted or assumed current. Ambiguous stream mappings are unknown. A packaged version equal to or ahead of the latest eligible upstream version is current. A package pinned with track_upstream ignores releases outside its pinned line. A package tracking latest considers eligible releases in all streams while lifecycle reflects the packaged stream.
  3. The Stuck MRs view lists every open update MR older than 24 hours with links and creation-based ages, including older MRs hidden by a newer MR on the same package. MRs with unknown creation time remain visible as such.
  4. The collector runs as a separate CronJob on OpenShift every four hours. It recomputes every package from the current specs and complete upstream history without slowing upstream_version_update. Failed and missed runs are visible through OpenShift and dashboard freshness.
  5. A release arriving during a run appears in that or the next complete run. Failed pagination or ingest retains the last successful inventory. Later runs pick up corrected release dates, lifecycle changes, and newly registered projects without replaying a release feed. Withdrawals require a monitor eligibility signal rather than disappearance from upstream. An outdated ingest generation is rejected even when upstream releases did not change.
  6. A newer upstream release does not reset staleness. An MR for a different version or with no reliable merge time does not acquire a rollout timestamp.
  7. Failed collection retains the last successful data with visible age. Monitor scan failures remain visible even after a successful collector run. Removed packages disappear only after a complete inventory run.
  8. Existing dashboard behavior, the update loop, and existing reporting are unchanged.

Phase 2

Future work, contingent on the independent pipeline-timing service from !1261 providing tested lineage and sufficient data in its pipeline_timing schema. The joins needed by this tracker remain to be verified. !1261 preserves the existing public tables and Dashboard reads. There is no planned removal of public.gitlab_merge_requests that requires this phase.

What changes

Rollout timestamp: Replace the v1 merge-time proxy with completion time of the first successful publication for the matched package and version. The future model must support an exact component match, MR merge commit → build PipelineRun → Snapshot → Release lineage, and identification of the intended ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient. If several builds or releases qualify, choose the earliest successful completion for that destination. Failed releases and releases to other destinations must not count. If the lineage or destination is ambiguous, leave rolled_out_at null. Retry unresolved rows as new timeline events arrive. A version shipped by a later repair commit needs a documented version-to-release association rather than being attributed to the original MR by SHA alone. Use pipeline_timing completion times only where the lineage is verified.

Backfill existing rollout rows and compare results with the v1 merge-time proxy before switching the timestamp. Retain the matched run and Release identifiers for traceability. Settle their column names against the implemented schema.

What stays the same

The collector CronJob, upstream mirror, and public.gitlab_merge_requests as the source for versioned MR matching and the Stuck MRs view. The bot-convention matching, rpm_version_status columns, and UI also stay the same. Phase 2 is a rollout timestamp upgrade with no planned UI restructure.

4.5.31 - Pipeline Timing

Independent SQS worker for RPM and container execution timing and cross-system lineage. It shares the Status events database but owns only the pipeline_timing schema.

Features

  • Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
  • Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
  • Source-derived execution creation, start, and completion times
  • Alembic-managed schema, dedicated queue, and worker metrics

Prerequisites

  • Python 3.11+ and PostgreSQL 16
  • A dedicated timing SQS queue subscribed to the environment’s SNS topic with RawMessageDelivery: false

The existing Status worker and event forwarders keep their own images and queues. Infrastructure provisions the timeline queues, IAM, and alerts separately from the Status worker. Each environment has a separate queue named arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue. The source queue retains messages for 4 days; its DLQ retains them for 14 days from the original enqueue time when SQS automatically moves a message to the DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue moves a message to its DLQ after 3 receives and uses a 300-second visibility timeout, matching Status. The SNS subscription preserves the SNS envelope (RawMessageDelivery: false) and does not filter events. See Hummingbird Events Topic for queue policy and redrive details. Provision both environments’ queues and SNS subscriptions before deploying the workers. Messages collected before a worker starts may expire after four days; this pre-launch loss is accepted. A queue without a consumer does not move messages to its DLQ because no receives occur. Validate the staging worker before deploying the production worker or enabling new production event sources.

Installation

cd pipeline-timing
pip install -e ".[dev]"

The pipeline-timing Containerfile builds the separate quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker run from this image.

Usage

Schema initialization

Run the initializer as the init container of the single-replica timing worker Deployment. Use a Recreate strategy so Kubernetes stops the old worker before starting a Pod whose init container may upgrade the schema:

DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db

The initializer creates the pipeline_timing schema and runs alembic upgrade head each time a Pod starts. This is a no-op at the expected revision; a new image can apply a forward migration during rollout. If migration fails, the initializer exits nonzero and Kubernetes does not start the worker. The worker checks the revision again before polling. Fix migration errors with a forward revision; do not downgrade or drop the schema during rollback.

Staging deployment from a merge request

Wait for the tools MR’s tests and Konflux image build to pass, then run the manual deploy_pipeline_timing_staging job. The job does not wait for those checks automatically. It triggers the infrastructure pipeline with ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using the MR commit’s image for both the migration init container and worker. Production is not deployed by this job.

Verify the init container completed, the worker accepts the new Alembic revision, and its queue is catching up before promoting the published image to production. Normal infrastructure pipelines use a shared image pin for both environments. Coordinate those deployments during MR validation so an older pinned image does not replace the MR image. The worker requires an exact Alembic revision match; an older image will refuse a newer revision even when the table layout is unchanged.

Execution-only schema and catalogue removal

Under HUM-10942, revision 003_execution_records added nullable created_at to TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery pointers temporarily. Historical creation-timestamp recovery completed in staging and production using a one-time local tool. The tool was intentionally not retained.

Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign keys and columns from the reporting tables, then drops pipeline_timing.events and its remaining indexes. It does not change reporting rows, their lifecycle fields, parentage, or ordering metadata. The database named events and legacy Status tables in public remain unchanged. Original migrations remain intact so fresh initialization follows the same revision chain.

Before deployment, confirm recovery is complete or its gaps accepted in both environments, no external consumer needs the catalogue, and the normal backup is available. Record the catalogue allocation and PVC free space, then verify the space recovery and continued ingestion in staging before production promotion. The table/index allocation is released when the migration commits without a bulk delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small amounts of physical row space; that does not justify a rewrite.

The migration limits lock waits to five seconds. A conflicting reader or unexpected dependency makes initialization fail and the transactional DDL roll back. Resolve contention or the dependency and retry with the compatible image. Downgrade is explicitly unsupported: recreating empty history would not restore the deleted catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.

Kubernetes creation time comes from metadata.creationTimestamp; start and completion come from status.startTime and status.completionTime. Missing source timestamps remain null, not inferred from SNS publication time. Pending conditions are distinct from running, and failure reasons retain cancellation details. GitLab jobs and pipelines retain their source lifecycle fields and status.

Only execution state and lineage are maintained. There is no event-history array, per-step timing, new archive-location table, or permanent S3 pointer on each row. Resource deletion preserves the execution’s result, timings, and parentage.

Timing worker

Deploy one worker replica with the environment’s dedicated queue. Kubernetes starts it only after the init container succeeds:

SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker

The worker polls when it starts. Accepted notifications update reporting rows in one committed transaction before SQS acknowledgement, without event-table writes. Malformed events remain for retry and eventual dead-letter-queue redrive; database or revision failures stop polling without affecting legacy Status. Scale the timing worker to zero replicas to pause ingestion. After launch, monitor queue age against the four-day source retention limit during a pause to avoid losing events. SQS preserves the original enqueue time for automatic DLQ transfer, and an operator redrive resets it. A new rollout uses Recreate: the old worker stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its own SNS copy and remains independent of timing ingestion.

The worker follows Status’s polling pattern: a 20-second long poll receives up to 10 messages, processed sequentially with boto3’s default request retries. Each accepted event commits in its own transaction. At the end of each receive, one SQS batch-delete request acknowledges committed events and ignored messages. Short receives flush immediately; failed processing never adds a receipt to the delete request. Receive errors are logged and polling continues. Partial delete failures and uncertain request outcomes leave committed events available for idempotent redelivery, and polling continues. If a database failure interrupts a receive, the worker first attempts to acknowledge earlier confirmed commits. Database connection or schema failures stop the worker for Kubernetes to restart it. SQS visibility controls when an unacknowledged message becomes available again.

The worker caches up to 128 UPSERT statement shapes by table and ordered columns. Each execution binds fresh event values; the cache does not retain payloads, connections, or cursors. Source update timestamps order messages when both the incoming and existing row have them; SNS publication timestamps break ties or provide the fallback. Kubernetes condition transition times supply source ordering. Late pending/running snapshots cannot regress a completed Kubernetes execution. Sparse payloads preserve existing values, and creation time stays immutable once known.

Queue age and DLQ depth are monitored independently from the Status queue. Fix malformed messages before using the documented redrive procedure. If the worker is paused or not yet deployed, remove only a known test message after confirming that redrive returned it to the timing queue.

Configuration

Variable Default Description
DATABASE_URL - Existing Status events database URL
SQS_QUEUE_URL - Environment-specific timing queue URL
SQS_WAIT_TIME 20 SQS long-poll interval in seconds
METRICS_PORT 9090 Prometheus port when the worker is enabled

Infrastructure supplies queue values for each environment:

Environment Queue URL variable
Staging ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL
Production ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL

S3_BUCKET is no longer required or used by the worker. Remove the old deployment variable only after the compatible worker is deployed in both environments. The independent S3 archiver and its configuration remain unchanged.

POSTGRES_URL is also accepted when DATABASE_URL is not set. For local development only, the database URL defaults to postgresql://postgres:dev@localhost:5432/events.

Schema and lineage

Table Identity Purpose
merge_requests GitLab project ID and MR IID Current state and lifecycle times
commits GitLab project ID and SHA Commit metadata
merge_request_commits Project ID, IID, SHA, relation Head and merge-commit links
gitlab_pipelines Project ID and pipeline ID Pipeline status and timing
gitlab_jobs Project ID and job ID Individual job attempts
pipeline_runs Cluster and Kubernetes UID Build, test, and Release attempts
task_runs Cluster and Kubernetes UID Task timing and parent UID
snapshots Cluster and Kubernetes UID Build-to-test bridge
releases Cluster and Kubernetes UID Snapshot-to-Release bridge

Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.

latest_observed_at and source_updated_at order updates without a notification catalogue or event foreign keys. The schema stores selected reporting fields, not raw event JSON. Join MR head and merge commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join jobs by project and pipeline ID. Konflux PipelineRuns use the same repository URL and SHA, then fan out by component and execution UID. TaskRuns link through PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels, and Release references connect the remaining phases. The Release completion time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain each retry’s status and timing.

PipelineRun SHAs come from supported labels and annotations, including pac.test.appstudio.openshift.io/sha, before falling back to the revision parameter.

Promotion Snapshots populate source_pipeline_run_name from hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the promotion run rather than the original MR build. Other Snapshots keep the log-URL behavior. This connects the promotion run to its Snapshot and Releases without changing the schema. References to the original MR build and source Snapshot in promotion annotations are not stored separately.

Existing promotion Snapshots need a separate backfill if no newer notification arrives after this extraction change. Deploying the worker does not rescan stored Snapshots or automatically repair historical links.

Finding archived notifications

The independent S3 archiver still receives its own SNS copy. The timing database does not store a direct archive pointer or a complete notification history.

For Kubernetes executions, use the environment’s archive bucket and latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source, kind, cluster, namespace, and resource name in candidate filenames, then verify the UID in the downloaded payload. For GitLab, verify the project and execution/commit identity in candidate payloads.

This is candidate discovery, not a guaranteed unique or per-field provenance link. Several notifications may share a timestamp, and sparse updates can retain facts from earlier payloads. Lookup depends on archive delivery, naming, and retention; it does not recreate the removed database catalogue.

Metrics include hummingbird_pipeline_timing_records_total{kind,result}, hummingbird_pipeline_timing_failures_total{reason}, hummingbird_pipeline_timing_missing_lineage_total{kind}, and hummingbird_pipeline_timing_worker_stopped. success and ignored record results count only confirmed SQS acknowledgements, including duplicates. Each failed or uncertain batch entry increments the sqs failure counter. These counters do not measure unique event rows. Monitor queue age and dead-letter-queue depth separately from the legacy Status queue. Fix malformed events before redriving them. Identity-based upserts and ordering guards keep redelivery idempotent; there is no database row for every SNS message.

Development

Run tests only against the disposable PostgreSQL service provided in CI or a disposable local database. Tests truncate timing data and exercise schema upgrades and historical downgrades; never use a staging or production database URL. Migration tests create uniquely named disposable databases and need CREATEDB on that test server. Older-revision tests stop at revision 003; they do not downgrade through irreversible catalogue removal:

DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
  python3 -m unittest discover -s tests -t .

Tests also load shared publisher payload fixtures from sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no longer calculates or persists archive keys. CI provides DATABASE_URL for the PostgreSQL integration tests. The timing image has a Kubernetes import smoke test in tests-k8s.yml.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.

4.5.31.1 - RPM pipeline timing design

Problem

We lack a clear way to see where time is spent in the RPM build pipeline.

Background

References: HUM-5819 - Create a diagram that describes all phases and steps of various pipelines from start to finish. Then review with team. HUM-5820 - Time breakdown from MR creation to completion of Release pipeline.

During the F44 rebuild, RPM builds had problems whose causes were not immediately clear. The sprint retrospective and a later F44-specific retrospective created HUM-5819 and HUM-5820 to improve visibility into pipeline behavior and time spent in each phase.

An initial pass through HUM-5819 identified key areas for time tracking. HUM-5820 records those areas.

Findings

The team initially considered a small addition to the tools/hummingbird-status database. A draft MR explored that approach.

Findings from the draft MR:

  1. The existing Status events database is a suitable home because the system forwards GitLab and Konflux events through SNS, archives them in S3, and updates PostgreSQL. The timing worker runs as an independent service with its own image and deployment lifecycle.
  2. Some hummingbird-dashboard functions do directly query the raw JSON.
  3. The original purpose of hummingbird-status was to diagnose Konflux event issues, not long-term system diagnosis.
  4. The raw event JSON for Konflux PipelineRuns consumes significant space, about 97% of the 47 GB database, despite redundant event storage in S3.
  5. hummingbird-status PVC monitoring is now in place; the draft MR highlighted its absence.
  6. The solution to our problem statement is not to mimic the existing approach.

Proposal

Track required event data in new tables in the existing database and update event processing to populate them without changing existing tables or Dashboard behavior.

Events Already Agreed Upon for Tracking

HUM-5820 outlines the events already agreed upon for tracking.

  1. GitLab MRs and commits
  2. GitLab pipelines
  3. GitLab pipeline jobs
  4. Konflux PipelineRuns
  5. Konflux Tekton TaskRuns

The model must support reporting of:

  1. Status
  2. Relevant start and end times, including MR creation and merge times
  3. Event timestamps
  4. Event lineage across GitLab and Konflux, sufficient to reconstruct a single pipeline timeline

This is not a database design. It identifies the reporting requirements that detailed design must meet.

Approach

  1. Preserve existing event ingestion, database tables, and Dashboard behavior. Run timing ingestion in a separate pipeline-timing service and image while retaining the existing publishers, archiver, and events database.
  2. Create a new schema in the existing PostgreSQL database.
    • The schema provides logical isolation so existing consumers do not accidentally use the new tables.
  3. Create tables that hold the data needed to meet the agreed reporting requirements without duplicating raw event JSON.
  4. Treat the S3 archive as the authoritative raw-event record for the new schema.
    • Store a reference that lets consumers retrieve the archived event for detail, display, or later data-model improvements.
  5. Configure GitLab webhooks and the Kubernetes event forwarder to include the agreed event types, then route them through the existing SNS event pipeline.
  6. Ensure the S3 archiver can archive and identify every agreed event type.

Acceptance Criteria

  1. Existing Status database tables and Dashboard behavior remain unchanged.
  2. The new PostgreSQL schema provides the agreed event coverage.
  3. Representative data demonstrates an RPM pipeline timeline from MR creation through completion of the Release pipeline.
  4. Consumers can calculate time spent between tracked pipeline phases without querying raw event JSON.
  5. Consumers can trace lineage across all agreed event types and retrieve the corresponding authoritative S3 event when needed.
  6. Tests verify the new data model, lineage, and S3 references.

Out of Scope

  1. Creating a UI for visualization is out of scope.
    • The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
  2. Tracking the containers repository pipeline is out of scope.
    • That pipeline builds OCI images from RPMs produced by the RPM pipeline and is a potential follow-up effort.

4.5.31.2 - Pipeline timing implementation plan

Purpose

The original RPM pipeline timing design records the HUM-5820 design and remains unchanged. This implementation plan records the HUM-10942 revisions to retain execution timing and lineage for both RPM and container pipelines, not a permanent notification catalogue or permanent S3 references on reporting rows. Its execution-only requirements supersede the original design’s archive-reference and RPM-only scope decisions.

The Status events database remains the storage location, but pipeline-timing is a separate project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent publishers. This service split follows HUM-7529 MR 1261 review without changing the event contracts or lineage model.

Prior Exploration

Closed tools MR 1165 and closed infrastructure MR 1381 informed this plan. They are not implementation vehicles for the Stories above. New focused merge requests should reference the Story that owns their work.

Current execution-only transition

The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised execution-only requirements and supersedes the old archive-reference and RPM-only scope decisions. Existing Status tables, publishers, the archiver, and queue contracts remain unchanged.

  1. Release one removes the unused composite S3 lookup index.
  2. Release two, revision 003_execution_records, adds nullable creation times to TaskRuns and PipelineRuns and makes the reporting tables’ latest_event_id nullable. The worker stops inserting events and never changes old pointers. New rows use null pointers; the old foreign keys remain valid temporarily.
  3. Operator-run local recovery filled historical creation timestamps through those frozen pointers without replaying state. Both environments were verified with zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies. The one-time tooling was intentionally removed rather than retained in the repo.
  4. Release three, revision 004_drop_event_catalogue, explicitly removes the legacy foreign keys and latest_event_id columns, then drops events and its remaining indexes. All reporting values and ordering metadata are preserved.

Catalogue removal is transactional, uses five-second lock waits, and has no supported downgrade. Confirm recovery/backup readiness and no remaining external consumer, validate staging ingestion and physical space recovery, then use the existing image promotion path for production. Do not recreate empty history as a rollback. The events database and legacy public schema are not removed.

The worker does not require S3_BUCKET or calculate archive keys. Remove unused infrastructure bucket configuration only after the compatible app is deployed in both environments. Do not change the worker’s IAM permissions for local recovery.

Before each promotion, verify migration revision, execution timing/lineage, commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR validation, prevent ordinary infrastructure pipelines from restoring the older shared image pin. Merge can lead to automatic Konflux publication, Renovate image pin updates, and deployment to both environments; it is not a recovery hold.

Test fresh initialization and populated-schema upgrades, preserving all reporting values and relation files while removing only the catalogue and legacy references. Keep revision-001/002/003 regression tests in isolated disposable databases that stop before irreversible removal. Verify a forced cleanup failure rolls back all constraint/column drops, and an attempted downgrade cannot fabricate history. Continue testing lifecycle times, states, lineage, redelivery, and ingestion with the catalogue absent. No change to the worker’s ingestion logic is required.

Implementation Plan

Scope Decisions

  1. The timeline tracks through completion of the Konflux Release resource.
    • The Release completion timestamp is the final tracked milestone.
    • PipelineRun and TaskRun detail in rhtap-releng-tenant requires separate provisioning and is required before HUM-5820 is complete.
    • PipelineRuns and TaskRuns in hummingbird-tenant remain in scope.
  2. Snapshot and Release events are required as lineage bridges even though they were omitted from the initial agreed-event list.
    • A Snapshot links a build PipelineRun to test PipelineRuns.
    • A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
    • The new schema stores only their reporting and lineage fields, not raw JSON.
  3. The new model stores the latest known state of each execution and its relevant timestamps. It does not duplicate every status transition as a domain row.
    • A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time. It does not replace the failed PipelineRun.
    • For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes remain two separate records. A report must not show only the ten-minute retry as the time for both attempts.
  4. Both RPM and container pipeline executions are in scope. Retain the existing source mix rather than introducing repository filters during this transition.
  5. Full event replay and per-step timing are out of scope. Historical recovery is limited to best-effort creation timestamps and runs outside startup migrations.
  6. Each environment needs a dedicated timing SQS queue, dead-letter queue, and pipeline-timing worker deployment using its own image. The existing Status queue, image, and worker continue legacy ingestion unchanged.

PostgreSQL Schema

Create a pipeline_timing schema in the existing Status events database. Do not add columns, indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues to query only the existing tables.

Domain tables contain selected scalar fields, source timestamps, lineage, and ordering metadata, not a raw payload, history array, or catalogue foreign key.

Table Identity columns Purpose
pipeline_timing.merge_requests gitlab_project_id, iid Current MR state and lifecycle timestamps
pipeline_timing.commits gitlab_project_id, sha Commit identity and available commit metadata
pipeline_timing.merge_request_commits gitlab_project_id, iid, sha, relationship MR head and merge-commit lineage
pipeline_timing.gitlab_pipelines gitlab_project_id, pipeline_id Pipeline status, source, ref, SHA, and execution timestamps
pipeline_timing.gitlab_jobs gitlab_project_id, job_id Job attempt status, stage, timing, failure reason, and retry information
pipeline_timing.pipeline_runs cluster, uid Konflux PipelineRun state, source identity, timing, and Snapshot lineage
pipeline_timing.task_runs cluster, uid Tekton task state, timing, and parent PipelineRun UID
pipeline_timing.snapshots cluster, uid Build PipelineRun to Snapshot lineage
pipeline_timing.releases cluster, uid Snapshot to Release completion and PipelineRun lineage

Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID is the stable GitLab identity. The repository URL is needed because Konflux resources contain source URLs rather than GitLab project IDs.

004_drop_event_catalogue removes the temporary recovery catalogue and pointers after recovery completion. latest_observed_at and source_updated_at remain on all reporting tables for ordering, independently of notification-history storage. The S3 archiver remains an independent subscriber; execution fields require no archive lookup during normal ingestion or reporting.

Schema Management

Use Alembic (a database schema migration tool already used by Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for those tables is separate work.

  1. Add Alembic configuration and revisions to the pipeline-timing project and image.
  2. Add pipeline_timing.worker.init_db for the worker Deployment’s init container. It executes CREATE SCHEMA IF NOT EXISTS pipeline_timing directly and commits before invoking Alembic. Do not rely on Alembic to create the schema. Alembic manages every object inside the schema after that bootstrap.
  3. Set Alembic’s version_table_schema="pipeline_timing" and qualify every migration object with the schema. The version table and all managed objects must remain outside public.
  4. Create the tables, constraints, and required indexes in the initial revision.
  5. Run alembic upgrade head unconditionally from pipeline_timing.worker.init_db in the single-replica Deployment’s init container. Use Recreate so the old worker stops before a new image migrates the schema. Alembic is a no-op when the schema is already at head.
  6. On migration failure, log the exception and expected revision at ERROR level and exit nonzero. Kubernetes must not start the worker container after a failed init container; the separate Hummingbird Status Deployment continues unchanged.
  7. Have the timing worker independently verify that the database is at the expected Alembic revision before polling its queue. If the revision check fails, log the expected and observed revision at ERROR level and stop the timing worker. The legacy worker continues unchanged.
  8. Deploy one replica with the timing image and init container once the queue and archiver are ready. To pause polling, scale the Deployment to zero and monitor queue age against the source queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the DLQ resets the enqueue timestamp.
  9. Do not automatically downgrade or drop the schema during application rollback. Stop the timeline worker and use a forward migration to correct schema problems.

Selected Fields

The detailed schema should include at least the following reporting fields. These are proposed database column names, not source-payload field names.

Record type Required columns
Merge request gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url
Commit gitlab_project_id, repository_url, sha, title, message, committed_at, web_url
MR commit link gitlab_project_id, repository_url, iid, sha, relationship
GitLab pipeline gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url
GitLab job gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url
PipelineRun cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at
TaskRun cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at
Snapshot cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at
Release cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at

TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and completion are the resource’s own status timestamps. Missing values remain null. Unknown conditions with a pending reason report Pending, not Running. Cancellation remains distinguishable through the failure reason. GitLab execution timestamps and status remain source-derived. Non-execution entities retain their meaningful lifecycle milestones rather than fabricated start/completion fields.

GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by project, pipeline, stage, and job name.

PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID. Keep the timing and result for every attempt. This plan does not define a retry-group column or a single combined retry duration. Consumers that need an end-to-end duration across retries must keep the duration of each attempt visible and calculate the full elapsed time from the first attempt’s start to the final attempt’s completion.

GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request by appending /-/jobs/{build_id} to project.web_url.

merged_at and closed_at are new nullable TIMESTAMPTZ columns on pipeline_timing.merge_requests; they are not additions to the existing public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the webhook supplies them. Otherwise, derive them from the first event that reports the corresponding action or state, preferring the source updated_at timestamp and falling back to observed_at. Preserve the first derived terminal timestamp across later sparse events. This makes MR creation-to-merge timing available without requiring a GitLab API lookup.

Repository URL Normalization

Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style Git, and GitLab tree URLs from Konflux metadata.

These source values:

  • https://gitlab.com/redhat/hummingbird/rpms.git
  • git@gitlab.com:redhat/hummingbird/rpms.git
  • https://gitlab.com/redhat/hummingbird/rpms/-/tree/main

all normalize to:

https://gitlab.com/redhat/hummingbird/rpms

Use repository_url consistently in the new schema. Do not use a bare project path or source-specific Git URL as the cross-system identity.

Identity and Update Ordering

  1. Use (cluster, uid) as the primary execution identity for Kubernetes resources.
    • Names are not sufficient because Kubernetes can recreate a resource with the same name.
    • Store namespace and name for display and fallback joins.
  2. Use source IDs for GitLab records and retain repository_url for cross-system joins.
  3. Store the latest accepted SNS timestamp as latest_observed_at on reporting rows.
  4. Use source update timestamps from GitLab and Kubernetes condition transition times.
  5. Prevent an older event delivered later by SQS from replacing newer state.
    • Compare source timestamps only when both records have them; use SNS publication timestamps as the fallback and tie-breaker, including migration of legacy rows.
    • Late non-terminal snapshots must not regress completed Kubernetes executions.
  6. Preserve source fields absent from sparse updates and keep known creation time immutable. Ordering metadata advances independently of notification history.
  7. Do not require foreign keys for cross-source lineage that may arrive out of order. Use indexed identity columns and late joins. There are no legacy event foreign keys or pointer columns after catalogue removal.

Required Indexes

The schema must provide indexes for the known lineage and timeline queries. Exact index names and partial predicates are implementation details, but the following column order is required as a baseline:

Table Columns Supports
merge_request_commits gitlab_project_id, iid commits for one MR
merge_request_commits repository_url, sha MR lookup from a commit
gitlab_pipelines repository_url, sha pipelines for a commit
gitlab_jobs gitlab_project_id, pipeline_id jobs for a pipeline
pipeline_runs repository_url, sha Konflux runs for a commit
pipeline_runs cluster, namespace, name name-based fallback lineage
pipeline_runs cluster, namespace, snapshot test PipelineRuns for a Snapshot
task_runs cluster, pipeline_run_uid tasks for a PipelineRun
snapshots cluster, namespace, source_pipeline_run_name Snapshot produced by a build PipelineRun
releases cluster, namespace, snapshot Releases for a Snapshot

Primary keys provide the identity indexes listed in the schema table and do not need duplicate secondary indexes. Validate the final indexes with the representative acceptance query and PostgreSQL EXPLAIN before source enablement.

The initial design required a reverse archive lookup index on s3_bucket, s3_key. The staging storage investigation on October 22, 2026 found that this index occupied about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes, with zero recorded scans. No application query using the reverse lookup was found. The temporary legacy recovery used frozen latest_event_id values to read the archived creation metadata. Runtime ingestion never needed that reverse index.

Revision 002_drop_events_s3_index removes this unused application index to reclaim disk space and avoid maintaining it for every new event. It preserves event rows, columns, and archive references. Scan statistics do not prove that no external consumer needs the index; confirm that before deployment. Deploy and verify index space recovery in staging and production before the execution-only transition. Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the table. No execution-reporting index is removed or rebuilt.

Lineage Model

Consumers reconstruct a timeline with the following relationships:

  1. A merge request links to its head commits and merge commit through merge_request_commits.
  2. A commit links to GitLab pipelines by repository_url and SHA. A direct pipeline MR IID is used when the webhook supplies it.
  3. GitLab jobs link to their pipeline by project and pipeline ID.
  4. Konflux PipelineRuns link to a commit by repository_url and SHA.
    • Extract the repository from Pipelines as Code annotations first, then the git-url parameter.
    • Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
  5. TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or older resources without an owner UID.
  6. A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
  7. Test PipelineRuns link to a Snapshot through the Snapshot label.
  8. A Release links to its Snapshot and to any referenced Release PipelineRun.
  9. In the current scope, the Release completion timestamp closes the timeline even when its managed PipelineRun is in an inaccessible namespace.

The primary cross-system correlation key within one source pipeline is canonical repository_url plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the resulting timeline fans out by component and execution rather than assuming one build per commit.

Event Processing and Failure Isolation

Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.

For each timeline message:

  1. Start polling only after the Pod’s migration init container succeeds and the worker verifies the expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
  2. Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
  3. For an accepted notification, parse and upsert the reporting records in one transaction. Do not append a notification-history row or require a catalogue link.
  4. Commit the transaction, then delete the SQS message.
  5. On malformed data for an accepted event, roll back and leave the message on the queue. SQS retries it and moves it to the timeline dead-letter queue after the configured receive limit.
  6. On a database connection, schema, or migration failure, roll back, stop polling, and leave the queue intact for recovery. Do not send valid events to the dead-letter queue during an infrastructure outage.

A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed event or parser error, then redrive messages from the timeline dead-letter queue. They fix an infrastructure failure, then restart the worker to resume polling. Configure the source queue with 4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it. Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.

Completed recovery and archive inspection

S3 archival remains independent and unchanged. The timing worker neither computes archive keys nor needs bucket configuration or S3 read permissions. The operator-run recovery read immutable creation timestamps from archived snapshots, updated only null creation fields, and finished before catalogue removal. It was not part of a migration, and its tooling is not retained.

latest_observed_at and resource identity can narrow an archive search while objects are retained. Validate candidate payload identities; this is not guaranteed unique lookup or per-field provenance because sparse updates can merge older facts. See finding archived notifications. No permanent per-execution S3 pointer or replacement history table is added.

Original HUM-5820 delivery and source rollout

The remaining delivery-story sections record the original rollout. Their permanent archive-reference, bucket-configuration, and RPM-only assumptions are historical and superseded by HUM-10942 and the execution-only transition above. The publishers and independent archiver retain their existing event contracts.

Tools Repository: Publisher and Archiver Changes

This section describes code, tests, and documentation in the tools repository. Environment-specific source infrastructure is owned by HUM-7535 and activated under HUM-7533.

GitLab Event Forwarder

The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from object_kind. No event-specific forwarding implementation is required.

Add fixtures and tests for:

  • Pipeline Hooks with object_kind: pipeline
  • Job Hooks with object_kind: build
  • project_path and event_type SNS message attributes
  • compressed payload round trips

Kubernetes Event Forwarder

  1. Add TaskRun to the configured resources only after the deployed image understands its payload requirements.
  2. Extend the resource configuration schema with optional labelSelector and fieldSelector keys.
  3. Pass configured selectors to the existing ResourceWatcher label_selector and field_selector parameters.
  4. Remove metadata.managedFields from TaskRun messages before SNS publication because it is transport metadata and can dominate payload size.
  5. Keep status, results, owner references, task identity, and timing data needed for extraction and future diagnosis.
  6. Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and tenant Release PipelineRuns.
  7. Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and review any additional projection before rollout rather than silently dropping fields.
  8. Preserve spooled events on publish failure so a transient SNS error does not discard them.

Use a TaskRun label selector for the RPM application if production samples confirm that Tekton propagates a stable application label. If no reliable selector exists, estimate and approve the namespace-wide event rate before enabling the watch because every matching TaskRun will also be archived to S3.

S3 Archiver

The archiver is already generic, but its resource identity logic must change for the new event types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys include their numeric pipeline or job ID and Kubernetes keys include logical cluster name, namespace, and name. URL-valued cluster attributes from older forwarders retain name-only identities. Cover those identities with the canonical contract fixture.

Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun, TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.

Pipeline Timing and Hummingbird Status Changes

  1. Add a pipeline-timing/ project, package, image, and Alembic configuration with an initial pipeline_timing revision. Generate its independent Konflux build resources.
  2. Add pipeline_timing/timeline.py for extraction, ordering rules, and upserts.
  3. Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image responsible only for the public schema.
  4. Add timing dispatch for GitLab push, merge_request, pipeline, and build events and Kubernetes PipelineRun, TaskRun, Snapshot, and Release events.
  5. Mark the newly handled event types as known to the legacy Status parser so expected timing events do not generate unknown-kind warnings; do not route them to legacy tables.
  6. Add pipeline_timing.worker.init_db to the timing Deployment as an init container. Leave hummingbird_status.worker.init_db and legacy worker/sqs.py responsible only for public.
  7. Deploy the independent timing image to poll only the timing queue and write only the pipeline_timing schema in the shared events database.
  8. Add archive bucket configuration to the timing worker deployment.
  9. Start worker polling only after its init container succeeds; retain the worker’s own startup revision check before queue access.
  10. Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot prevent legacy worker startup.
  11. Provide separate pipeline_timing database counts without changing existing Status statistics queries.

Observability

Add metrics and structured logs for:

  • timeline records processed by kind and result
  • timeline extraction and database failures
  • events missing expected lineage fields
  • oversized Kubernetes payloads and SNS publish failures
  • timeline queue age and dead-letter-queue depth
  • timeline worker stopped because of a migration or revision-check failure

Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver failures, PostgreSQL availability, and PVC usage.

Documentation Updates

Update the existing documentation for every component changed by implementation. Do not create a separate documentation Story. Each delivery Story owns the pages for the behavior it changes:

Story Required documentation updates
HUM-7529 New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md
HUM-7530 gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md
HUM-7531 staging worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7532 managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md
HUM-7533 final documentation audit across all affected pages before HUM-5820 closes
HUM-7534 production worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7535 both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation

HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535 clarifies that timing queue subscriptions can precede worker deployment.

Testing

Unit Tests

Cover:

  • every accepted GitLab and Kubernetes event type
  • status and timestamp extraction
  • explicit and derived MR merge and close timestamps
  • repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
  • GitLab job URL derivation from project.web_url and build_id
  • MR head and merge-commit lineage
  • GitLab job retries as distinct job IDs
  • a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
  • PipelineRun-to-TaskRun owner UID lineage
  • Snapshot and Release bridge extraction
  • sparse and out-of-order updates
  • deterministic archive keys
  • missing optional lineage without rejecting otherwise valid records
  • TaskRun serialization without managedFields
  • init-container migration gating and zero-replica pause behavior
  • Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only the timing worker
  • timeline acknowledgement only after a committed timeline transaction
  • timeline retry and dead-letter-queue handling for parsing, data, and database failures
  • an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
  • ignored acknowledgement of valid events outside the accepted timeline event types
  • stopped polling on database, schema, or migration failure

PostgreSQL Integration Tests

Use the PostgreSQL service already available in repository CI to:

  1. Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
  2. Verify Alembic reports the expected head revision and that its version table exists in pipeline_timing, not public.
  3. Confirm the migration does not alter existing public table definitions or representative legacy queries.
  4. Ingest representative events in and out of order.
  5. Verify idempotent current-state upserts.
  6. Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
  7. Query one RPM timeline from MR creation through Release completion and calculate each available interval without raw JSON.
  8. Verify all domain rows resolve to event records with the expected S3 references.
  9. Inspect the new schema to confirm it has no raw JSON columns.
  10. Verify that timeline redrive produces idempotent rows and retains the same S3 reference.

End-to-End Acceptance Test

After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and test PipelineRuns, and a Release. Record the MR and component selected for the test.

The acceptance query must demonstrate:

  • MR creation and merge timestamps
  • GitLab pipeline and job timing
  • derived GitLab job URLs that match project.web_url and build_id
  • pre-merge and post-merge commit lineage
  • component build PipelineRun and TaskRun timing
  • a failed PipelineRun and successful retry shown as separate attempts, without losing the failed attempt’s timing
  • Snapshot and test PipelineRun lineage
  • Release completion
  • calculable durations between phases
  • retrievable S3 objects for every row used by the timeline
  • paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status

Run make test, make coverage, make check, and SAM template validation for the affected projects.

Infrastructure Implementation

HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue, dead-letter queue, subscription, IAM, and alerts in staging and production.

Staging Event Path

Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:

  1. Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status consumer are deployed and that message-bus.md describes them.
  2. Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective topics with RawMessageDelivery: false. Configure each source queue with a finite receive limit, a visibility timeout at least twice the worker’s per-message processing budget, its own dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not depend on staging worker acceptance.
  3. Configure 4-day retention for both source queues and 14-day retention for their dead-letter queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker deployment may expire after four days; this pre-launch loss is accepted. Without receives, messages do not automatically move to the DLQ. Verify redrive before source enablement.
  4. Under HUM-7531, deploy one staging pipeline-timing replica with Recreate strategy, its migration init container, timing queue, resource limits, and explicit staging S3_BUCKET; do not use the production bucket. Verify the init container reached the expected revision and the worker is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent loss during pauses.
  5. Keep new production event sources disabled until staging acceptance succeeds.

The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes only to the staging topic. Staging and production forwarders must not publish the same watch stream to the same topic.

Production Event Path

  1. Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
  2. Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
  3. Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
  4. Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned under HUM-7535. After staging worker acceptance, deploy one pipeline-timing replica with Recreate strategy, a migration init container, resource limits, and an explicit production S3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required. Verify the revision and worker before activating new event sources, and monitor queue age after launch.
  5. Verify the production redrive procedure before enabling new sources.
  6. Enable new production event sources only through HUM-7533 after staging acceptance; the timing worker starts under HUM-7534.

Managed Release Pipeline Access

HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.

  1. Obtain read-only get, list, and watch permissions for PipelineRuns and TaskRuns in the managed namespace.
  2. Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
  3. Verify each event-forwarder service account with auth can-i before enabling its watches.
  4. Link the managed PipelineRun to the Release record through its namespaced reference.
  5. Include signing and Pulp publication TaskRuns in the final acceptance timeline.

Infrastructure Verification Gates

  1. Render and resolve each final RPM webhook configuration before deployment.
    • The production endpoint must retain its token and existing push, merge request, note, confidential note, and pipeline events while adding job events.
    • If same-URL inheritance does not deep-merge those settings, use an explicit webhook block containing the complete configuration.
  2. Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images before enabling Job Hook or TaskRun events in that environment.
  3. Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
  4. Verify representative TaskRun payload size after removing managedFields.
  5. Use an RPM application label selector if production samples confirm Tekton propagates a stable label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
  6. Use a single replica and Recreate strategy; verify the migration init container succeeded and the worker checked the expected Alembic revision before it polls.
  7. Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with alerts and a tested redrive procedure.

Rollout

  1. Complete the independent Pipeline Timing image and the legacy Status parser compatibility change in HUM-7529, and event-bus support in HUM-7530.
  2. Verify the HUM-8015 staging message bus and provision both environments’ timing queues, subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch messages may expire without reaching the DLQ; this loss is accepted.
  3. Under HUM-7531, deploy the single-replica staging timing worker with Recreate strategy and its migration init container. Verify the new schema and legacy isolation before enabling new sources.
  4. Complete rhtap-releng-tenant provisioning in HUM-7532 and add it to the staging event forwarder.
  5. Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the running timing worker.
  6. Run a controlled RPM merge request and verify staging lineage, duration calculations, managed Release tasks, and S3 references.
  7. Under HUM-7534, deploy the single-replica production timing worker with Recreate strategy and its migration init container after staging worker validation.
  8. Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
  9. Add the managed namespace to the production event forwarder, then activate new production sources under HUM-7533 after verifying the timing worker is running.
  10. Expect each Kubernetes watcher to emit an initial ADDED event for every existing matching resource. Schedule source activation during a monitored window.
  11. Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors, forwarder spool growth, database size, and PVC free space.
  12. Run the final controlled RPM acceptance merge request in production before closing HUM-5820.

Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion continues. After launch, monitor queue age while paused: messages expire after four days unless the worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven after repair.

Epic Completion Criteria

The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:

  1. HUM-7529 through HUM-7535 are complete.
  2. The design’s acceptance criteria pass in both staging and production.
  3. Representative timelines include managed PipelineRun and TaskRun timing from rhtap-releng-tenant.
  4. The staging-first source rollout and final production acceptance run are documented under HUM-7533.
  5. HUM-7533 has verified that each Story’s required documentation update is complete.

Out of Scope

  1. Creating a UI for visualization is out of scope.
    • The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
  2. Tracking the containers repository pipeline is out of scope.
    • The pipeline-neutral schema, event contracts, and execution identities are intended to support its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
    • The RPM and containers pipelines do not connect directly. A separate scheduled process discovers updated RPMs and commits changes to rpms.lock.yaml files in the containers repository.
  3. Timeline replay and historical backfill are out of scope.
    • Existing Hummingbird Status S3 ingestion bootstraps the public schema and collapses Kubernetes history, so it is not suitable for timeline recovery.
    • Future work must define scoped S3 read permissions and chronological ingestion into the Alembic-managed pipeline_timing schema. Events that were never collected cannot be reconstructed.

Delivery Stories

Story Responsibility
HUM-7529 Make RPM pipeline timing data queryable Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility
HUM-7530 Carry RPM pipeline timing events through the event bus GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests
HUM-7531 Validate RPM pipeline timeline storage in staging Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries
HUM-7534 Make RPM pipeline timeline storage available in production Production Pipeline Timing deployment with event sources still disabled
HUM-7535 Support environment-specific RPM pipeline event collection Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout
HUM-7532 Expose managed Release pipeline timing Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection
HUM-7533 Roll out RPM pipeline event collection Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation

Dependency Order

  1. HUM-7529 and HUM-7530 can proceed in parallel.
  2. HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event contracts and provisions timeline-specific infrastructure in both environments without waiting for staging worker acceptance.
  3. HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
  4. HUM-7534 depends on successful staging validation in HUM-7531.
  5. HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments, but completion gates final acceptance under HUM-7533.
  6. HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the final rollout is verified.

4.5.32 - RPM Tracker

RPM Tracker owns the rpm_tracker schema in the shared events PostgreSQL database. It stores upstream state and packaged version status for the dashboard to read. See the system proposal for the collection and UI design.

Features

  1. Independent Alembic migration chain with an initializer and revision gate.
  2. Transactional replacement of the current inventory with a generation check.
  3. Read-only dashboard access to tracker data in the shared database.
  4. Full RPM spec and upstream monitor collection with currency, staleness, packaged-stream lifecycle, and independent upstream scan health.
  5. Database-only update MR reconciliation with merge-time rollouts, independent open-MR listing, and creation-based ages with explicit uncertainty.
  6. A standalone Konflux image with native RPM support and independent collection and MR-reconciliation CronJob templates for staging and production.

Prerequisites

Python 3.11 or newer and access to the shared PostgreSQL events database. The migration user needs permission to create the rpm_tracker schema. Collection also requires Git, rpmspec, the system Python RPM bindings, and read access to the rpms repository and upstream monitor. On Fedora, install git-core, rpm-build, and python3-rpm. The Python dependencies needed to load the rpms version helper are included in the tracker package. Do not run collection in a virtual environment that hides the system RPM bindings. The helper also needs the packaged specfile dependency to include sudo’s upstream patchlevel from its extraver macro. MR reconciliation requires read access to public.gitlab_merge_requests and write access to tracker tables. It needs neither GitLab API credentials nor native RPM bindings.

Installation

cd rpm-tracker
pip install -e .
DATABASE_URL=postgresql://postgres@localhost:5432/events python -m rpm_tracker.init_db

The Alembic chain uses rpm_tracker.alembic_version, separate from other services’ migration histories and outside public. The migration files are bundled in the installed rpm_tracker package. The initializer creates the schema and applies pending migrations on each invocation; it is a no-op at the current revision. Run it before deploying the dashboard API, and run it as an init step before the collector starts. A failed migration must prevent collection. The dashboard read endpoint returns HTTP 503 until a migrated schema compatible with its response model is available. Later tracker revisions remain readable when they preserve the fields used by the dashboard.

Online initialization takes a PostgreSQL transaction advisory lock, so collection and reconciliation init containers can start together without racing migrations. The initializer delegates this database work to rpm_tracker.db.initialize(). That helper creates the schema, opens the transaction, and passes its connection to Alembic before applying runtime grants. Alembic’s migrations/env.py only configures offline or online migration execution and the tracker ORM metadata. When RPM_TRACKER_RUNTIME_PASSWORD is supplied, initialization also creates or updates the rpm_tracker login role. This requires the existing Status database administrator credential. The role receives tracker-table reads and writes, read-only access to the tracker revision and public.gitlab_merge_requests, and no schema-creation or administrator privileges. Role/password updates and migrations commit together. Omit this variable for migration-only initialization.

Standalone Alembic commands do not create the schema or configure runtime roles. Run the initializer first. Offline SQL likewise assumes that the tracker schema has already been provisioned.

Container image

The rpm-tracker/Containerfile installs Git, rpm-build, python3-rpm, CA certificates, and the tracker package from pyproject.toml. The installed package includes its migrations; the image runs as the base image’s non-root user. Konflux discovers the Containerfile through the existing tools renderer and builds both supported architectures. The release must be pinned by digest in the infrastructure repository’s vars.d/rpm-tracker.yml before deployment.

podman build -t localhost/rpm-tracker:dev rpm-tracker
podman run --rm localhost/rpm-tracker:dev rpm-tracker-collect --help

The infrastructure kubernetes/rpm-tracker/ templates initialize the database before each run, then use the restricted runtime role. Collection is scheduled at 0 */4 * * * UTC; reconciliation is scheduled independently every ten minutes. The staging and production variants use their own Status database and runtime password. The infrastructure operating runbook documents rollout and manual checks.

Usage

Collect an inventory

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m rpm_tracker.collect \
  --rpms-repository https://gitlab.com/org/group/rpms.git \
  --monitor-url https://upstream-monitor.example.org

The installed rpm-tracker-collect command accepts the same arguments. Each run checks the migrated schema and reads the expected generation before cloning the repository’s default branch into a temporary directory. It reads every package directory, queries spec versions through that checkout’s parse_spec_version helper, and records the checked commit SHA. Metadata version is not a fallback. Missing or malformed metadata and failed checkout enumeration stop collection; an unparseable spec does not omit a package. Clone failures log Git’s captured stderr before re-raising the original CalledProcessError, which retains the diagnostics in its stderr attribute.

The collector reads every monitor project-catalog and release-history page, following X-Next-Cursor even on an empty page. History pagination omits releases without dates, so collection also reads GET /v1/projects/{id} and merges its releases and observed_releases into the history by source and release ID. Snapshot records take precedence for duplicates, including corrected or newly unknown dates. Undated newer releases therefore affect currency and make staleness unavailable instead of disappearing from the comparison. Anitya’s latest-only date coverage is supported. A JSON null stream list is treated as empty; malformed non-list responses remain errors. A project without eligible releases produces unknown without preventing other packages from refreshing.

Requests have a 30-second timeout and are paced sequentially with a 0.2-second delay. All monitor redirects are rejected, including same-origin redirects, so bearer tokens are never forwarded to a redirect destination. Configure the canonical monitor API base URL; a redirect fails collection and retains the previous inventory. Definitions verify repository matches; shared projects are fetched only once. HTTP, response validation, or pagination failures, including a failed snapshot request, prevent any inventory write. Monitor APIs do not promise a single snapshot across requests; a later complete run reconciles changes.

Currency uses native RPM version comparison with no packaging or lexical fallback. A version equal to or ahead of the eligible upstream maximum is current; an eligible newer release makes it behind. No verified project mapping means unmonitored. A mapped package with an unusable spec, update target, or comparison is unknown; its reason is logged and monitor health is retained. Pinned lines use component-boundary matching. With no track_upstream, use upstream_stream or a uniquely matched packaged stream, never global latest. An explicit stream does not change a pinned or latest update target.

Monitor scheme names are trimmed and lowercased during definition validation, matching the monitor registry. Mixed-case or whitespace-padded names therefore use the same normalization and pinned-target filtering as their canonical names.

first_behind_at is recomputed as the earliest date of an eligible release newer than the packaged version. Later releases do not reset it. If any eligible newer release lacks a date, staleness is unavailable. Lifecycle always describes the packaged stream, independently of the update target. Non-numeric/custom stream names require an exact release match or an explicit upstream_stream override. Package suffix stripping and named transforms reuse the rpms helper; already normalized OpenJDK versions are not transformed twice. After explicit package transforms, semantic +build metadata is removed and prerelease hyphens are represented as RPM tildes. Build-only differences do not create upgrades or staleness, while OpenJDK’s build number is retained by its explicit transform. For monitor schemes pep440 and python, PEP 440 alpha, beta, and release-candidate versions are normalized to RPM tilde notation (3.15.0rc2 becomes 3.15.0~rc2), so equivalent packaged prereleases compare equal and final releases sort later. The standard packaging parser handles PEP 440 aliases and casing. Nonzero epochs, development releases, post-releases, local versions, and invalid PEP 440 values produce unknown rather than an unsupported comparison. CVE version_transform and source-artifact availability checks do not affect tracker currency.

Repository URLs are matched across HTTPS, SSH, and Git syntax with .git and trailing slashes removed. Multiple matching projects require the metadata field upstream_monitor_project_id. This is a monitor identifier, not an Anitya release_monitoring_project_id. Providers whose repository cannot be recovered from their definition, such as Anitya or numeric GitLab IDs, require that explicit verified mapping. A resolvable source repository must agree with upstream_repo.

A monitor status reporting a failed upstream scan is preserved even if readable history permits successful collection. Dashboard consumers can distinguish old collection data from failed scanning using sync timestamps and the separate monitor_state, monitor_last_error, and monitor_last_success_at fields. Write failures and generation conflicts propagate and leave the last successful inventory unchanged; they require a fresh collection rather than retrying an old payload. The collector does not migrate the database; the CronJob’s blocking init container runs the initializer first.

Reconcile update MRs

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m rpm_tracker.reconcile \
  --rpms-project org/group/rpms --target-branch main

The installed rpm-tracker-reconcile command accepts the same arguments. Run the initializer before reconciliation and schedule reconciliation periodically, independently of monitor collection. The first successful reconciliation stores the exact project and target branch in rpm_sync_state; subsequent runs reject a different scope to prevent historical rollout records being reassigned.

The reconciler reads the existing Status MR rows in the shared database. It recognizes chore(rpms): Update <package> to <version> and Update <package> to <version>, including Draft:, WIP:, and STALE PATCH: prefixes. Both chore/upstream-update-<package> and chore/upstream-update-failure-<package> corroborate package identity, including diagnostic MRs retitled to valid update titles, but never establish a version. Package names and versions are exact, not substring or prefix matches. Unsupported RPM version syntax and conflicting branch/title packages remain uncertain. Diagnostic failure MRs with title Draft: investigate failed upstream update for <package> remain visible as package-level candidates without a target version.

rpm_update_mrs preserves every qualifying MR by (project, iid), including multiple open MRs for one package and packages absent from the current inventory. Each successful run refreshes the candidate snapshot from the Status rows; closed and no-longer-qualifying MRs disappear from the open list. Previously observed merged identities survive unrecognized titles, changes of target branch, and temporarily missing Status rows. This retained evidence can reject a later version change, but only currently qualifying MRs can resolve a rollout. The tracker stores creation and merge timestamps from the raw webhook’s object_attributes.created_at and object_attributes.merged_at. Timestamps must include an offset, Z, or GitLab’s space-separated UTC suffix and are normalized to UTC. Neither mutable updated_at, observation time, nor MR-head-version dates are substitutes. Previously reliable timestamps survive later omitted fields. Conflicting timestamps remain unavailable, with their reason retained for investigation.

Rollouts are keyed by (package, version). Exactly one verified merged MR in the configured scope with an explicit reliable merge timestamp resolves a rollout. Missing timestamps, conflicting identities, and multiple matching merged MRs leave rolled_out_at null with an unavailable_reason. Merge times before creation or in the future are unavailable. Changing a verified version identity after merge cannot move its rollout to another version. The timestamp is a v1 merge-time proxy, not proof of successful publication.

Every run revisits current packaged versions, verified merged update targets, and historical rollout keys. Later MR evidence can therefore resolve an unavailable historical version after the inventory has advanced. New ambiguity invalidates an earlier resolution. A rollout’s resolved_at changes only when its resolution result changes. The MR snapshot, rollout changes, and last_mr_reconciled_at commit atomically under the existing sync-row lock; failure retains the previous snapshot and freshness timestamp. MR reconciliation does not change the inventory generation or collection freshness.

Read MR information

The dashboard exposes:

  • GET /api/rpm-upstream-state: inventory, rollout reasons, reconciliation scope/freshness, all open_update_mrs, and one blocking_mr per package row.
  • GET /api/rpm-update-mrs: all qualifying open MRs independently of package rows, plus last_mr_reconciled_at.
  • GET /api/rpm-update-mrs?stuck_only=true: MRs older than 24 hours and MRs with unknown ages. Exactly 24 hours is not older than the threshold.

MR ages are computed from creation time at read time. Missing, conflicting, or future creation times produce null age_seconds and null is_stuck, never zero or omission. created_at_issue explains invalid or conflicting evidence. The package-row MR prefers a verified match to the selected upstream version, then the newest known creation time, source update time, project, and IID for deterministic selection. A fallback MR link is a package-level candidate, not evidence of a target-version match. Selecting one package-row MR never removes other MRs from either listing. A null reconciliation timestamp means no MR snapshot has been collected yet.

Transactional storage

Create a SQLAlchemy engine with rpm_tracker.db.get_engine() for the storage module, and dispose of it when collection finishes. The tracker-owned ORM models mirror the Alembic tables; only the initializer migrates them. Call store.get_generation(engine) before collecting; both it and store.ingest(engine, request) require the bundled schema revision. Fetch the complete RPM checkout, every required monitor release page, and each mapped project’s snapshot before calling ingest() with one status per package and the checked checkout SHA. A failed fetch must not submit a partial inventory. ingest() locks the singleton sync row, checks the expected generation, and replaces upstream and status rows in one transaction. A failed write or conflicting generation leaves the previous inventory unchanged. Stored rollout times and MR information are separate from inventory replacement. The dashboard reads the sync, upstream, status, rollout, and MR tables in one read-only snapshot; it does not migrate or write tracker tables.

Configuration

Variable Description
DATABASE_URL Shared events PostgreSQL connection URL
POSTGRES_URL Connection URL fallback
RPMS_REPOSITORY_URL Repository URL, alternatively supplied by --rpms-repository
UPSTREAM_MONITOR_URL Monitor base URL, alternatively supplied by --monitor-url
UPSTREAM_MONITOR_TOKEN Optional bearer token for monitor API reads
RPM_TRACKER_RUNTIME_PASSWORD Initializer-only password for the restricted runtime role
PGPASSWORD PostgreSQL password supplied separately from the connection URL
SSL_CERT_FILE Trusted CA bundle for the monitor client’s stdlib HTTPS requests
GIT_SSL_CAINFO Trusted CA bundle for Git HTTPS requests
REQUESTS_CA_BUNDLE Trusted CA bundle for the rpms helper’s Requests dependency
Reconciliation option Description
--rpms-project Required exact GitLab project path, e.g. org/group/rpms
--target-branch Exact inventory target branch; defaults to main

Development

cd rpm-tracker
pip install -e '.[dev]'
pytest

To run the PostgreSQL migration, reconciliation, and atomic replacement tests, set DATABASE_URL to a disposable PostgreSQL database before running the tracker and dashboard tests. CI supplies test_events; the database-backed tests are skipped when DATABASE_URL is unset. The tracker tests refuse to replace an existing nonempty tracker inventory.

The initializer privilege test requires no existing rpm_tracker login role and uses a disposable database. It verifies concurrent initialization, quoted passwords, allowed tracker writes, and rejected public/revision/schema writes. Image smoke checks in rpm-tracker/tests-k8s.yml verify executable CLIs, native RPM ordering, spec parsing, specfile macro expansion, and bundled migrations through the existing Konflux K8s integration scenario.

Native RPM ordering and collector database integration checks require the system Python RPM bindings. Policy and collection-failure unit tests can run without them, but production collection fails immediately when the bindings are unavailable.

Check model and migration drift

After initializing a disposable database, compare its migrated schema with the tracker ORM models from the rpm-tracker/ directory:

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m alembic -c rpm_tracker/alembic.ini check

Alembic receives models.Base.metadata and reflects only the rpm_tracker schema. Supported schema differences, including unexpected tracker columns and tables, fail the check. Status tables in public and other service schemas are excluded. Checking does not run role provisioning or change the database.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.

4.5.33 - Work Queue Service

Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation and updates; designed to support CVE remediation, advisory tracking, and other work item types.

For design decisions and architectural rationale, see Work Queue Service Design.

Architecture

The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).

flowchart TD
    clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
    gl_events["GitLab\n(events)"] -- webhook --> gl_fwd["gitlab-event-forwarder"]
    gl_fwd --> sns["SNS"]
    k8s_events["Konflux K8s\n(PipelineRun events)"] -- watch --> k8s_fwd["k8s-event-forwarder"]
    k8s_fwd --> sns
    sns -->|"filter: source=gitlab,\nevent_type=[merge_request, build]"| sqs["SQS Queue"]
    sns -->|"filter: source=kubernetes,\nkind=PipelineRun"| sqs
    sqs --> svc
    svc -- "Commits API\n(create + update)\nJobs API\n(trace + retry)" --> gl["GitLab\n(API)"]
    svc -- "PLR fallback fetch\n(label selector on SHA)" --> k8s_api["Konflux K8s API"]
    svc -- "archived PLR fetch" --> ka["Kubearchive"]
    svc -- "shared DB" --> pg[("PostgreSQL")]

The SQS consumer thread receives GitLab merge_request and pipeline events and Kubernetes PipelineRun events from the SNS topic via one filtered SQS subscription with a two-branch $or filter policy covering both source=gitlab and source=kubernetes. Events are stored in the pending_events table and processed by existing work-item threads, which map MR state changes to phase transitions, track Konflux build status, and trigger CI job retries on pipeline failure.

Design

Generic + detail tables

The schema separates queue machinery from type-specific data:

  • work_items — generic: id, type, phase, actionable, attempts, error, timestamps. Shared by all work item types.
  • mr_details — MR-specific: package, project, file_actions, commit_message, content_hash, mr_iid, etc. Joined to work_items via foreign key.
  • work_item_statuses — independent readiness dimensions with composite key (work_item_id, dimension, variant). Includes pending_until for deferred re-evaluation.

New work item types add their own detail table (e.g. cve_details) — no changes to the generic layer. SQLAlchemy joined table inheritance loads the correct subclass automatically via polymorphic_load="selectin".

Phases

Four generic phases track coarse lifecycle status:

Phase Description
active Being processed — the only mutable phase
completed All lifecycle steps done (terminal)
failed Error occurred (terminal, retryable)
cancelled Cancelled (terminal, retryable)

Items start as active. Only active items accept event-driven phase transitions and executor actions. The three terminal phases (completed, failed, cancelled) are immutable to events — recovery requires explicit action via the retry API. GitLab merge_request events drive transitions from active: merge -> completed, close -> cancelled.

Dimensions

Independent readiness signals stored in work_item_statuses. All dimensions follow a uniform needs_* trigger model: status is "true" (action needed) or "false" (handled). Absent dimensions are treated as actionable by the decider (!= "false"). Each dimension has an optional variant for multi-instance signals (e.g. per-component Konflux builds) and an optional pending_until timestamp for deferred re-evaluation (see Deferred dimensions below).

Dimensions are set by executors or event handlers via EventResult.dimensions; the generic handle_event layer applies event-driven dimensions and emits audit entries.

Exception — needs_plr: Unlike other dimensions, needs_plr requires explicit arming to "true" to be actionable. Absent or "false" means inactive (not awaiting PLR events). The decider checks dims.get("needs_plr") == "true", so a missing row is treated as inactive, not actionable.

Dimension schema

work_item_statuses:
  work_item_id: UUID  (PK, FK)
  dimension:    Text  (PK)
  variant:      Text  (PK, default="")
  status:       Text  (NOT NULL)       -- "true" or "false"
  updated_at:   DateTime(tz)
  detail:       JSONB (nullable)
  pending_until: DateTime(tz, nullable)

Unified state diagram

All dimensions share the same state machine:

stateDiagram-v2
    [*] --> absent : item created
    absent --> true : decider treats absent as actionable
    true --> true_deferred : executor (defer for later)
    true --> false : executor (done, no further checks)
    true_deferred --> true : sweep notifies (pending_until expired)
    true_deferred --> true : webhook/handler (clears pending_until)
    false --> true : webhook/handler (re-activates)
    note right of absent : no dimension row exists
    note right of true : status="true", pending_until=NULL
    note right of true_deferred : status="true", pending_until=future
    note right of false : status="false", decider ignores

Per-dimension transitions

Dimension On success (→ "false") On error / re-trigger (→ "true") Deferred?
needs_mr MR created or already exists — No
needs_expected_plrs Expected components fetched GitLab API error, unexpected error (malformed manifest permanently completes with error: "parse_error") Yes — 5 min error backoff
needs_plr All PLRs terminal or fetched PLR event or PLR fallback fetch (non-terminal) Yes — 30 min fallback poll
needs_dashboard Dashboard note posted or updated PLR event, PLR fallback fetch, transient GitLab 5xx No — always set “false”; re-armed by PLR handler/executor
needs_reconcile Reconciler completed Webhook (approved, pipeline, update), sweep expiry Yes — 30 min cycle / 5 min error backoff / 15 s→5 min recheck backoff for unchecked/checking/unknown
needs_empty_check Empty-diff check completed Sweep expiry Yes — 30 min cycle / 5 min error backoff
needs_ci_retry All jobs processed Build event (job failed), sweep expiry Yes — per-job not_before delay

Dimension detail schemas

The detail JSONB column carries executor-specific state. Dimensions without detail (needs_mr, needs_empty_check) store null; needs_reconcile stores null except during a merge-recheck streak. Frozen dataclass definitions live in workqueue_service/mr/dimension_types.py; typed accessor functions (get_plr_detail, get_dashboard_detail, get_expected_plrs_detail) deserialize the stored JSONB into the dataclass, or return None when absent. JSON arrays round-trip as tuples — NeedsCiRetryDetail.jobs and NeedsExpectedPlrsDetail.components are immutable.

Detail preservation. DimensionUpdate.detail defaults to a _PRESERVE sentinel. Event handlers that return a DimensionUpdate without explicit detail automatically keep whatever is stored in the database (via SQL COALESCE). Pass detail=None explicitly to clear, or a detail dataclass to replace. See Dimension update helpers below.

Updating detail. Event handlers update the existing detail instead of rebuilding it: read it via the typed accessor, then apply a mutation helper — NeedsPlrDetail.with_component() adds or replaces one component entry, NeedsCiRetryDetail.with_job() appends one job. Both return a new frozen instance; the original is untouched. Handlers pass the dataclass to DimensionUpdate, and executors that snapshot authoritative state (ensure_plr_status, ensure_expected_plrs, retry_ci_jobs) construct a fresh dataclass. crud.upsert_dimension serializes dataclasses via dataclasses.asdict() before the DB write, so callers never handle raw dicts.

needs_expected_plrs detail

Stored after fetching the Konflux trigger manifest for a commit SHA. Set by ensure_expected_plrs; read by ensure_plr_status, ensure_dashboard_note, and the PLR event handler.

Field Type Required Description
sha str | null yes Commit SHA the manifest was fetched for. null for items without a SHA.
components list[str] yes Expected component names (e.g. "caddy--hummingbird--default"). Empty when the repo has no trigger manifest (404).
error str no Set on parse failure; empty components list accompanies it.
needs_plr detail

Tracks observed PipelineRun state per component. Updated by the PLR SNS handler (merge per event) and ensure_plr_status (full snapshot from K8s/Kubearchive). Read by ensure_dashboard_note.

Field Type Required Description
components dict[str, PlrComponentInfo] yes Keyed by component name. Empty dict on init.

Each PlrComponentInfo value:

Field Type Required Description
status str yes PLR condition: "Running", "Succeeded", "Failed", "Unknown"
plr_name str yes Kubernetes PipelineRun resource name
start_time str yes ISO 8601 timestamp from status.startTime (pipeline start, not resource creation)
needs_dashboard detail

Stored after posting or updating the MR dashboard note. Set by ensure_dashboard_note; read on subsequent runs to decide whether to update or post. The PLR handler and ensure_plr_status re-arm needs_dashboard to "true" without passing detail, which preserves this detail.

Field Type Required Description
note_id int yes GitLab note ID for in-place updates
url str yes Dashboard URL included in the note body
note_body str no Last rendered note body; compared to skip no-op GitLab API calls
needs_reconcile detail

Optional — only present during a merge-recheck streak. When reconcile_mr observes detailed_merge_status in the fast family (unchecked, checking, unknown — recheck in flight), it re-arms the dimension at now + MERGE_RECHECK_SCHEDULE.delay(retry_step) (15 s doubling to a 5-minute cap) and bumps the step via advance_retry. The converge outcomes (merge attempted, real blocker, idle) clear the detail, so a fresh streak starts at 15 s again; the early exits (no MR yet, GitLab fetch error) leave the stored step untouched. Webhook re-arms preserve the stored step through the _PRESERVE sentinel.

Field Type Required Description
retry_step int yes Consecutive fast-family recheck count (starts at 1)
needs_ci_retry detail

Accumulates failed CI jobs eligible for retry. Set by the build event handler (appends per job) and retry_ci_jobs executor (removes processed jobs). Each entry represents one failed GitLab CI job.

Field Type Required Description
jobs list[CiRetryJobEntry] yes List of job entries (see below)

Each job entry:

Field Type Required Description
build_id int yes GitLab job ID
build_name str yes Job name (e.g. "test", "check")
build_failure_reason str yes GitLab failure reason (e.g. "script_failure")
build_finished_at str yes ISO 8601 timestamp when the job finished
retries_count int yes Number of retries already attempted
pipeline_id int yes GitLab pipeline ID the job belongs to
matcher_name str no Name of the retry matcher rule that matched this job
not_before str no ISO 8601 timestamp — earliest time the retry may be attempted

Deferred dimensions

When an executor or handler sets pending_until to a future timestamp alongside status="true", the decider treats the dimension as "false" (not actionable) until that time. The periodic sweep finds items with expired pending_until and notifies them — no synthetic events are inserted. The sweep wakes exactly at the earliest armed deadline — or after ~1 s when a deferred timer is already due, letting the next pass’s expired scan pick it up (see Processing loop) — so even short timers are honored precisely rather than rounded up to a sweep tick.

status pending_until Decider sees Meaning
(absent) — actionable No row — needs action
"true" NULL actionable Needs action now
"true" past actionable Sweep notified, ready to act
"true" future not actionable Deferred — sweep will notify
"false" any not actionable Done

Dimension update helpers

crud.upsert_dimension provides safe-by-default mutation patterns via its detail parameter:

  • Omitted — keep existing detail (SQL COALESCE on conflict); e.g. re-triggering needs_dashboard after a PLR event without clobbering note_id
  • Detail dataclass — replace with a full snapshot; e.g. initialising a dimension or completing an executor action
  • None — clear the stored detail explicitly

Partial detail updates do not use a merge helper — event handlers apply the frozen dataclass mutation helpers (with_component(), with_job()) to the existing detail instead, and executors construct a fresh dataclass for a full snapshot. crud.upsert_dimension takes a detail dataclass (or None) and serializes it via dataclasses.asdict() internally; callers never handle raw dicts.

See Dimension detail schemas for the _PRESERVE sentinel, frozen dataclass definitions, and typed accessors.

Dimension state (status plus optional deferred re-arm) is expressed via the DimensionState enum in workqueue_service/mr/constants.py. Passing a bare status string or timedelta to the state parameter is a type error; always use an enum member:

Member Status Delay Used for
DONE false — Dimension fulfilled, no action needed
PROCESS_IMMEDIATELY true — Needs action now
PROCESS_AFTER_TIMEOUT true 30 min PLR silence fallback, reconcile/empty-check cycle
PROCESS_AFTER_ERROR true 5 min Transient API error backoff, CI job not_before delay

For computed absolute times (e.g. the CI retry not_before), use DimensionState.process_at(when). It returns a ProcessAt wrapper that the CRUD layer resolves to status="true" with pending_until set to the given time.

Decider

A pure function registered per type that derives the next action from the work item’s current state. The decider reads the selectin-loaded item.statuses list (in-memory, no DB call) to check dimensions, pre-filtering deferred dimensions (where pending_until is in the future). See workqueue_service/mr/decider.py for the MR decider implementation.

The decider scans dimensions in priority order (DDR-45, DDR-48). Absent dimensions are treated as actionable (!= "false"). Deferred dimensions (pending_until in the future) are excluded from the scan. Adding a new concern = new dimension + handler, no decider changes.

The decider determines the next action to execute — it does not set phases or modify state.

MR upsert semantics

The MR submission endpoint uses upsert logic keyed on (project, source_branch). Callers always submit the current desired state; the service decides what to do based on existing work items:

stateDiagram-v2
    [*] --> created : no active item
    [*] --> unchanged : active, same hash
    [*] --> updated : active, different hash
    [*] --> retried : failed item

    created --> active : new work item
    unchanged --> active : no-op
    updated --> active : refresh fields
    retried --> active : reset phase
Upsert action Condition HTTP status Effect
created No active/failed item for (project, branch) 201 New work item, actionable = true
unchanged Active item exists, same content_hash 200 No-op, return existing item
updated Active item exists, different content_hash 200 Refresh file_actions/title/description, set actionable = true
retried Failed item exists for (project, branch) 200 Reset to active, clear error, update content, set actionable = true

Content hash. A SHA-256 of the normalized file_actions JSON (sorted by file_path + action, compact separators, sorted keys). Enables cheap same-content detection without comparing the full payload.

Idempotency key. Derived server-side as mr:{project}:{source_branch}:{generation} where generation is a counter per (project, source_branch) pair. A partial unique index ensures uniqueness only among non-terminal phases — completed and cancelled items can share a key prefix without blocking new items for the same branch.

Race handling. If two concurrent inserts race for the same idempotency key, the loser catches the IntegrityError, rolls back, and retries the full upsert — the winner is now visible and the correct action (unchanged/updated) is taken.

MR action chain

The MR work item lifecycle chains executor actions. After upsert, the processing loop calls the decider to select the first action; subsequent actions are also selected by the decider after each executor completes:

  • ensure_mr — creates or updates the MR. If mr_iid is not set, creates the branch, commit, and MR via the GitLab Commits API and applies the managed-by::workqueue scoped label. If mr_iid is already set, force-pushes updated content to the existing MR branch and syncs MR metadata (title, description). Detects empty diffs (before creation) or closes an empty MR after force-push. Captures the commit SHA as current_sha, resets the needs_plr dimension (clearing stale component state), and re-arms needs_expected_plrs. Sets needs_mr to "false" on success.
  • ensure_expected_plrs — fetches the Konflux trigger manifest (.tekton/trigger-manifest-pull-request.json) from the target project via the GitLab repository files API, parses the component-to-path mappings, and derives the set of affected components from the MR’s changed file paths. Stores the component list in the needs_expected_plrs dimension detail. If the manifest is missing (404), stores an empty list. Retries on transient API errors with a 5-minute backoff.
  • ensure_plr_status — fallback executor that fetches current PLR state from the Konflux K8s API (with Kubearchive fallback) using label selectors on the commit SHA. Fires on 30-minute pending_until re-arm when PipelineRun events have been silent. Writes the complete component snapshot into the needs_plr dimension detail and re-arms needs_dashboard to trigger a note update. Per component, the newest PipelineRun by status.startTime wins — after a retrigger, the older archived run does not overwrite the newer live one. If any component is still non-terminal, re-arms needs_plr with another 30-minute delay. Requires KONFLUX_KUBECONFIG_PATH and optionally KUBEARCHIVE_URL environment variables. The KubeArchive client reuses the bearer token from the kubeconfig’s auth settings.
  • ensure_dashboard_note — posts or updates an internal note on the MR with the Hummingbird dashboard link and a per-component Konflux build status table. The note uses the :robot: **Hummingbird Status** header shared with the dashboard-mr-linker Lambda. On first run, it creates the note and stores note_id in the dimension detail. On subsequent runs (triggered by PipelineRun events or the PLR fallback fetch), updates the existing note in place if the rendered body differs from the stored note_body. Always sets needs_dashboard to "false" on completion — re-arming is handled by the PLR SNS handler and the ensure_plr_status executor, which set needs_dashboard = "true" whenever PLR state changes.
  • check_empty_diff — checks whether an active MR’s diff became empty (target branch moved). Triggered by the needs_empty_check dimension via pending_until expiry (~every 30 minutes per item). If the diff is empty, closes the MR with an explanatory note, deletes the branch, and cancels the item. Sets needs_empty_check to "false" on completion. Transient GitLab API errors are logged but do not fail the item.
  • reconcile_mr — reconciles desired vs actual MR state. Triggered by the needs_reconcile dimension, set by webhooks (approved, unapproved, update, pipeline success/failure) and pending_until expiry. Merges directly when detailed_merge_status is mergeable and auto_merge is enabled. Syncs title and description if drifted. Re-arms needs_reconcile according to the merge outcome: a failed merge PUT retries immediately; unchecked/checking/unknown (a mergeability recheck in flight, completing seconds later) retry on an exponential schedule — 15 s doubling to a 5-minute cap, with the step tracked in the dimension detail; every other status keeps the 30-minute cycle.

Draft state. The draft submission option controls initial MR creation only. After creation, GitLab owns Ready/Draft state; update_mr and reconcile_mr preserve the current GitLab state while synchronizing title text. The details.draft response field reports the initial creation setting, not the live GitLab status.

After all actions complete, the decider returns None and the item sits in active phase waiting for event-driven lifecycle tracking (merge → completed, close → cancelled).

Label coordination with dashboard-mr-linker. The managed-by::workqueue label causes the dashboard-mr-linker Lambda to skip the MR, so the workqueue-service creates the dashboard note for its MRs. When dashboard_base_url is not configured, ensure_dashboard_note returns immediately without posting a note.

head_pipeline_id on mr_details

The current head pipeline ID is stored as a typed integer column (head_pipeline_id) on the mr_details table rather than as a dimension. Build events and the reconciler update it directly. The CI retry executor uses it to filter stale jobs from previous pipelines.

Processing loop

The ProcessingManager uses a bounded ThreadPoolExecutor (10 workers):

  • One loop per item — a loop takes a session-level advisory lock pg_try_advisory_lock(0x5751, item_id & 0x7FFFFFFF) on a pinned connection and holds it for its whole lifetime. Another loop (another pod, the sweep, or the local rerun path) returns immediately when the lock is held; duplicate submissions are cheap. Losers increment hummingbird_workqueue_processing_lock_skipped_total, labelled by item type. The key is the item UUID’s low 31 bits, so two unrelated items can rarely collide; the skipped item is re-spawned by the sweep once the winner finishes — delayed, never lost.
  • Serial per item — one action at a time per work item, across pods.
  • Parallel across items — independent items processed concurrently.
  • Per-item deadline — processing loops exit after 60 seconds to prevent stuck items from holding pool workers.
  • Executors apply state updates directly (phase, MR fields) and must not call session.commit() or session.rollback().
  • After execution, the loop re-derives the next action via the decider.
  • Crash recovery — the lock lives on a pinned database connection; if the pod or connection dies, PostgreSQL releases it. A death at any loop step (inside an executor or at the post-execution state write) aborts the loop; it never continues on a reconnected connection without the lock. The item stays actionable, so the periodic sweep re-spawns it. There is no lease, heartbeat, or TTL.
  • The periodic sweep submits actionable items and items with pending events or due pending_until dimensions (no synthetic events). It is deadline-driven: after each pass it sleeps until the earliest armed pending_until deadline (plus ~1 s so the deadline is already expired on wake), or for ~1 s when a deferred timer is already due (a single query returns the earliest future deadline and a past-due flag over one snapshot, so a timer expiring between the sweep’s scans cannot be missed by both), keeping the former 5-minute interval as a backstop cap for the un-timed legs — actionable crash recovery, pending events, and deadlines armed by other pods. A wake event cuts a long wait short whenever an enqueue or an executor commit arms an earlier deadline in-process; sweep-initiated submissions do not wake it, as they arm no timer themselves.

Connection pinning. Each loop checks out one connection and binds every session it opens to that connection, so the session lock survives the per-action commits. The engine pool is sized to the worker count with overflow headroom, and libpq keepalives plus tcp_user_timeout detect a dead server instead of leaving a lock on a half-open socket. The loop releases the lock explicitly before returning the connection to the pool; if the release fails, it invalidates the DBAPI connection, because a pooled close() alone would keep the session lock alive.

MR labeling

The executor always adds the managed-by::workqueue scoped label to every MR it creates, merged with any caller-provided labels. This label is a coordination signal: other services (e.g. dashboard-mr-linker) check for this label and skip managed MRs, since the workqueue-service handles dashboard note creation itself via the ensure_dashboard_note action.

Event processing

The SQS consumer thread receives events from the SNS topic — GitLab webhooks (merge_request, build) and Kubernetes resource events (PipelineRun) — and stores them in the pending_events table. Work-item threads drain and process these events as part of their main loop.

Consumer thread. A single daemon thread (poll_loop) long-polls SQS with MaxNumberOfMessages=10 and WaitTimeSeconds=20. For each message it decodes the SNS envelope (handles gzip+base64), resolves matching work items via the resolver registry, INSERTs into pending_events, commits, deletes the SQS message, and calls manager.spawn_loop() for each affected work item. The consumer is started only when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is set.

State mapping. MR state changes are mapped to phases:

GitLab state Phase Notes
merged completed Terminal
closed cancelled Terminal (recover via retry API)
opened — Ignored (only active items accept events)
locked — Ignored (transient during merge)

Only items in active phase accept event-driven transitions. Events for terminal items (completed, failed, cancelled) are silently discarded.

PipelineRun events. Kubernetes PipelineRun events (forwarded via the event forwarder → SNS) are resolved to MR work items by matching the commit SHA in the PipelineRun’s labels against mr_details.current_sha. Only type=build PipelineRuns are processed; test PipelineRuns are ignored. The handler also rejects events whose SHA label no longer matches the item’s current_sha (stale SNS delivery after a force-push). Each accepted event merges the component’s status into the needs_plr dimension’s JSONB detail and re-arms needs_dashboard to trigger a note update. Events are attributed to a PipelineRun by metadata.name: every event for the same PipelineRun carries the same status.startTime, so matching names always apply the newer status (a terminal run never regresses to a non-terminal one). For a different PipelineRun — a retrigger — the event is accepted only when its status.startTime is strictly newer (compared as parsed instants, so fractional-second formats order correctly). An event without a startTime cannot overwrite a component already recorded as Succeeded or Failed, nor an in-progress component that already has a stored startTime.

Konflux SHA labels. Build and test PipelineRuns carry different SHA labels — using the wrong label returns zero results:

PLR type SHA label Type label value
Build pipelinesascode.tekton.dev/sha build
Test pac.test.appstudio.openshift.io/sha test

The event resolver checks pipelinesascode.tekton.dev/sha first (build PLRs), falling back to pac.test.appstudio.openshift.io/sha for edge cases. The K8s/Kubearchive fallback fetcher (list_build_plrs_for_sha) queries only pipelinesascode.tekton.dev/sha with type=build, matching the label that build PLRs actually carry.

Out-of-order guard. The last_event_at column on work_items rejects events with updated_at older than the last processed event. Same-timestamp events pass the <= guard (processing is idempotent).

Exit coordination. The work-item exit decision runs under a SELECT ... FOR UPDATE row lock on work_items, so an enqueue that updates the item in the same transaction is either visible to the exit or blocks until the exit commits and then leaves actionable = true. The former pg_advisory_xact_lock insert/exit handshake was removed; the row lock covers every writer, including the API enqueue paths the handshake never covered. See DDR-51.

After a no-work exit commits, the loop releases the session lock and re-checks pending_events and the decider on the pinned connection. If a writer committed in that gap, the loop re-acquires the session lock and continues instead of returning. A writer that arrives after the re-check is caught by the sweep; the item remains actionable.

Sweep extension. The deadline-driven periodic sweep submits actionable items and items with unprocessed pending_events, providing startup recovery and a safety net behind the row-lock exit and the unlock/re-check. It wakes exactly at the earliest armed pending_until deadline; the 5-minute backstop interval covers everything un-timed (see DDR-52).

Concurrent drain tolerance. In steady state the per-item session lock prevents two pods from draining the same item. During the rolling update that introduced the session lock, an old pod still used leased_until while a new pod used the session lock, so both could briefly drain the same item. The drain therefore reads each listed event’s (id, event_type, event_body) before processing the first one and deletes by id — a row another pod already deleted is a no-op. An event may be applied by both pods; application is idempotent (dimension upserts, last_event_at guard), matching the at-least-once semantics of crash recovery. See DDR-50.

The same coexistence applied to executor actions, not just the drain: the two-key session lock deliberately could not conflict with the old single-key/lease path, so an old pod’s leased loop and a new pod’s locked loop could both run the same action for one item during the overlap. Read-and-upsert actions (reconcile_mr, ensure_plr_status, ensure_expected_plrs) converge on the same state. Creation actions (ensure_mr, ensure_dashboard_note) and retry_ci_jobs check their result field (e.g. mr_iid) before acting, so a duplicate requires both loops to pass that check before either commits — possible in the worst case, producing one duplicate external mutation (a second MR or note, or a second CI retry) for an item that was mid-action during the pod replacement. New pods deliberately ignored leased_until rather than skipping rows with an unexpired value: a lease written by a pod that died mid-rollout must not hide the item from the sweep. That coexistence window ended when the locking redesign finished rolling out; the contract migration drops the leased_until column, so the lease mechanism no longer exists. See DDR-51.

Adding a new work item type

  1. Create a detail table and SQLAlchemy model (e.g. CveWorkItem)
  2. Register a decider function: decider.register("cve", _cve_decide)
  3. Register executor functions for each action
  4. Add a submission endpoint: POST /api/work-items/cve

No changes needed to the processing loop, claiming, or generic API.

API

Endpoints

Method Path Description
POST /api/work-items/mr Upsert an MR work item (create, update, or skip)
GET /api/work-items/mr List MR items (source, project, etc.)
GET /api/work-items List all items (type, phase filters)
GET /api/work-items/{id} Get item with nested details
POST /api/work-items/{id}/retry Retry a failed item
GET /api/whoami Introspect caller identity and permissions
GET /oauth/start Initiate OAuth login (handled by oauth-proxy)
GET /oauth/callback OAuth provider callback (handled by oauth-proxy)
POST /oauth/token Exchange OAuth session cookie for Bearer token
GET /healthz Health check
GET /metrics Prometheus metrics

Responses include a nested details dict with type-specific fields:

id: "..."
type: mr
phase: active
details:
  source: rpms-ci
  mr_iid: 42
  mr_url: https://gitlab.com/org/group/project/-/merge_requests/42

Authentication

All callers authenticate via Authorization: Bearer tokens. The service validates each token in one of two ways:

Caller Token type Validation
K8s SA JWT (projected) JWKS signature verification
GitLab CI JWT (OIDC) JWKS signature verification
Human (web/CLI) sha256~... (OCP) OCP users/~ API
Local dev none WORKQUEUE_SERVICE_LOCAL_AUTH_USER env var

X-Forwarded-* headers are never trusted on API paths. The proxy protects only the OAuth login flow (/oauth/start, /oauth/callback) and the POST /oauth/token endpoint, which bridges cookie sessions to Bearer tokens for browser and CLI users. The /oauth/token endpoint uses POST (not GET) to prevent CSRF — browser-initiated GETs via <img> or <script> tags cannot trigger it.

OAuth proxy deployment requirements

The oauth-proxy sidecar handles only the OAuth login flow. All /api/ paths bypass the proxy — the app handles auth itself. The proxy must be configured with these flags:

Flag Purpose
--bypass-auth-except-for=^/oauth/ Proxy only protects the OAuth flow (login + token exchange)
--pass-access-token Sets X-Forwarded-Access-Token on cookie-authenticated requests (required for /oauth/token). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size)
--pass-user-headers Sets X-Forwarded-User on protected paths
--cookie-refresh=1h Refreshes the OCP token inside the cookie before it expires (see below)
--cookie-expire=24h Hard session limit, forces re-login
--cookie-samesite=lax CSRF defense-in-depth for /oauth/token (POST already blocks the main vectors)

The proxy manages two independent lifetimes that are easy to confuse:

  • Session cookie — controlled by the proxy (--cookie-expire, defaults to browser session). Contains the encrypted OCP access token.
  • OCP access token — controlled by OCP (default 24h). The actual credential the app uses to call users/~.

The proxy does not know when the OCP token inside its cookie expires. It only checks whether the cookie is valid (signature, expiry). Without --cookie-refresh, the cookie outlives the token:

Time What happens
T+0h User logs in. Proxy creates cookie with fresh OCP token (expires T+24h)
T+24h OCP token expires. Cookie still valid.
T+25h User makes request. Proxy sets X-Forwarded-User (valid cookie) and X-Forwarded-Access-Token (expired token). App calls users/~ with expired token → 401 → anonymous → “not authorized”.

--cookie-refresh=1h tells the proxy to silently refresh the OCP token inside the cookie every hour, keeping it well within the 24h expiry. The workqueue-service also uses a TTL-bounded cache (5 minutes) for users/~ results, so expired-token failures are never permanent.

Authorization

Access rules are defined in the config file. Each rule maps a remote identity to permissions (read, retry, submit), optionally scoped to work item types. Rules use either groups (OCP group membership) or issuer+claims (JWT bearer). Claim matching follows Vault’s bound_claims pattern: all conditions AND, list values OR.

All permissions (including read) must be granted by a matching rule. Authenticated callers with no matching rules get 403. Multiple matching rules are merged (union of permissions).

See Configuration for the config file format.

Whoami introspection

GET /api/whoami returns the caller’s identity and effective permissions. Example response:

authentication:
  tier: ocp-user
  identity: admin-user
  groups:
    - konflux-hummingbird-admin-access
authorization:
  effective_permissions:
    mr:
      - read
      - retry
      - submit
  matching_rules:
    - groups:
        - konflux-hummingbird-admin-access
      permissions:
        - read
        - submit
        - retry
      item_types: []

API Documentation

Interactive API documentation available without authentication: /docs (Swagger UI), /redoc (ReDoc), /openapi.json.

Audit Log

State-changing operations (processing, API calls, auth decisions) emit audit entries. Every entry is logged at INFO level. When WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set, entries are additionally archived to S3 as gzipped JSON objects for long-term querying via Athena.

Audited events

Event type Source Description
action_started Worker Action execution began
action_completed Worker Action finished (success or failure)
action_decided Decider Decider assigned next action
action_no_op Decider Decider returned no next action
phase_change Worker / API / Events Phase transition (max attempts, submit, event)
item_updated API Work item content updated via upsert
dimension_change Events / Executor Dimension upsert (needs_dashboard posted, etc.)
retry_requested API Retry endpoint reset a failed item
auth_failed Auth Anonymous request rejected (401)
auth_denied Auth Authenticated caller lacks permission (403)

S3 key format

When S3 archival is enabled, entries are stored as gzipped JSON with an Athena-partitionable key layout:

{prefix}/YYYY/MM/DD/HH/MM/{timestamp}#{work_item_id}::{event_type}.json.gz

The ContentEncoding: gzip header is set so S3 API consumers get transparent decompression.

Field conventions

Follow these conventions when adding new audit calls:

  • actor: caller identity for API/auth events; thread name for worker events; "system" for decider/system-triggered transitions.
  • old_state: state before the change. Include phase and action when both are relevant. Omit on creation (no prior state) and on auth events (no work item state).
  • new_state: state after the change. Same keys as old_state. Omit fields that are None (filtered by to_dict()).
  • trigger: what caused this event. Always include source ("api", "worker", or "decider"). Add action for deliberate operations, reason for error/system transitions, error for failure details.
  • metadata: context orthogonal to the state transition — request info on auth events, type-specific data on submit. Not for duplicating state already in old_state/new_state.

CI Job Retry

When a CI pipeline on a managed MR fails, the service can automatically retry jobs that match known transient failure patterns. This is ported from the CKI pipeline-herder’s proven regex-based job log matching engine.

How it works

  1. GitLab sends a build (job) webhook when a CI job finishes
  2. The event handler appends the failed job to the needs_ci_retry dimension’s jobs list and sets status to "true"
  3. The decider selects the retry_ci_jobs executor
  4. The executor processes all jobs in the list, matching each against the config tree and retrying matching jobs via the GitLab Jobs API
  5. Jobs with non-zero delay set pending_until to the earliest per-job not_before timestamp — the sweep notifies when it expires
  6. Multiple failed jobs in the same pipeline accumulate in the list

needs_ci_retry dimension

The needs_ci_retry dimension follows the unified state machine (see Unified state diagram). Jobs with non-zero delay use the deferred path: status="true" with pending_until set to the earliest per-job not_before timestamp. The sweep notifies when any job’s delay elapses.

The true → true self-loop fires when another job in the same pipeline fails — the build event handler appends the new job to the existing list (clearing pending_until so the decider picks it up immediately).

The detail JSONB stores a jobs list. Each entry carries build_id, build_name, build_failure_reason, build_finished_at, retries_count, and pipeline_id — all from the build webhook payload. The executor deduplicates by build_id and skips jobs whose pipeline_id does not match the MR’s current head_pipeline_id. When a job has a non-zero delay, the executor stamps a per-job not_before ISO timestamp based on build_finished_at + retry_delays[retries_count]. The dimension’s pending_until is set to the earliest per-job not_before so the sweep fires as soon as any job is ready.

Matcher configuration

CI job retry is configured per type under handled_types.mr.job_retry. The config uses a recursive matcher tree where parent fields are inherited by children:

handled_types:
  mr:
    job_retry:
      retry_delays: ["0", "5m", "10m"]
      retry_limit: 5
      matchers:
        - name: runner-system-failure
          description: Job failed with runner_system_failure
          failure_reason: runner_system_failure
        - name: git-clone
          description: Git clone failure
          messages:
            - "fatal: unable to access"
            - '/error: RPC failed; curl 92 HTTP\/2 stream \d+/'

Matcher fields

Field Type Default Description
name string "" Matcher name (used in metrics and audit)
description string "" Human-readable description
failure_reason string "" Exact match on GitLab job failure_reason
job_name string "" Prefix match on job name
messages list[str] [] Literal substring or /regex/ on job trace
tail_lines int 300 Number of trace lines to scan (from the end)
retry_delays list[str] ["5m"] Escalating delays indexed by attempt ("0", "5m")
retry_limit int 3 Maximum retry count per job name
matchers list [] Nested child matchers (recursive)

Message matching

  • Literal: "fatal: unable to access" → substring match per line
  • Regex: '/error: RPC failed; curl \d+/' → re.search on joined lines (slash-delimited)

Retry safety

Before retrying a job, the executor checks:

  1. No newer job with the same name exists (prevents double retries)
  2. Retry limit not exceeded (count of jobs with the same name)

Retry timing

The retry_delays list is indexed by the job’s attempt count (0-based). When the index exceeds the list length, the last entry repeats. A delay of "0" means immediate retry; non-zero delays set pending_until to the computed not_before timestamp. The sweep notifies the item when pending_until expires, waking exactly at the earliest armed deadline.

Prometheus Metrics

The service exposes Prometheus metrics at /metrics on the same port as the API (no sidecar). Metrics follow the repo naming convention: generic operational metrics use the shared hummingbird_ prefix, service-specific metrics use hummingbird_workqueue_.

Generic operational metrics

Metric Type Labels Description
hummingbird_http_requests_total Counter method, path, status Total HTTP requests
hummingbird_http_request_duration_seconds Histogram method, path HTTP request latency
hummingbird_sqs_messages_received_total Counter — SQS messages pulled
hummingbird_sqs_messages_processed_total Counter status SQS messages processed

Service-specific metrics

Metric Type Labels Description
hummingbird_workqueue_work_items_created_total Counter item_type Work items submitted via API
hummingbird_workqueue_work_items_completed_total Counter item_type, result Action executions completed (success/failed)
hummingbird_workqueue_work_items_retried_total Counter item_type Work items retried via API
hummingbird_workqueue_action_duration_seconds Histogram action Executor action duration
hummingbird_workqueue_active_processing_threads Gauge — Running work item threads
hummingbird_workqueue_processing_lock_skipped_total Counter item_type Loops that lost the per-item session lock
hummingbird_workqueue_pending_events_inserted_total Counter — Pending events inserted into DB
hummingbird_workqueue_ci_jobs_retried_total Counter project, matcher_name CI jobs retried by the herder
hummingbird_workqueue_ci_retry_skipped_total Counter project, matcher_name, reason CI job retries skipped
hummingbird_workqueue_merge_rechecks_total Counter merge_status Fast-family merge recheck cycles by status

UUID path segments in HTTP metrics are normalized to /{id} to limit label cardinality. The /healthz and /metrics paths are excluded from HTTP metrics.

CLI

export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)

# Update an existing file (local_path:repo_path)
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --file /path/to/sources.spec:sources.spec \
  --commit-message "Sync sources from upstream" \
  --source-branch mr-service/sync-1 --title "Sync sources"

# Create a new file
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --create /path/to/new-file.txt:new-file.txt \
  --commit-message "Add new file" \
  --source-branch mr-service/add-1 --title "Add file"

# Check status
workqueue-service status --id <uuid>

# List active MR items
workqueue-service status --phase active

# Show identity and permissions
workqueue-service whoami

# Raw JSON output
workqueue-service whoami --json

--url overrides WORKQUEUE_SERVICE_URL. --no-auth skips all credential loading and sends unauthenticated requests (like aws --no-sign-request). Useful with whoami to verify anonymous access.

CLI authentication

The CLI resolves credentials in this order:

  1. WORKQUEUE_SERVICE_TOKEN env var — used directly as a Bearer token. Recommended for CI (OIDC/projected SA tokens).
  2. Cached token — if a valid token exists in the per-server cache, it is reused automatically.
  3. OAuth/Kerberos login — when neither env var nor cache is available, the CLI performs a synchronous headless SPNEGO login against the oauth-proxy, exchanges the session cookie for an OCP Bearer token via POST /oauth/token, and caches it for subsequent calls. Requires requests-gssapi and a valid Kerberos ticket (kinit).

Tokens are cached per-server under $XDG_CACHE_HOME/workqueue-service/tokens/ (defaults to ~/.cache/workqueue-service/tokens/) with 0600 permissions and a 20-hour TTL. If a cached token is rejected (HTTP 401), the CLI invalidates the cache and retries once with a fresh OAuth login.

Configuration

Config file

The service reads its configuration from a YAML file at startup. The path defaults to /etc/workqueue/config.yml and can be overridden with WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing or invalid (fail-closed).

authorization:
  rules:
    # Any in-cluster SA with the right audience can read
    - issuer: k8s
      claims:
        aud: workqueue-service
      permissions: [read]

    # Human users -- OCP group membership
    - groups:
        - konflux-hummingbird-admin-access
      permissions: [read, submit, retry]

    # Specific K8s SA for CI automation
    - issuer: k8s
      claims:
        aud: workqueue-service
        sub: system:serviceaccount:hummingbird--internal:rpms-ci
      permissions: [submit, retry]
      item_types: [mr]

    # GitLab CI OIDC
    - issuer: https://gitlab.com
      claims:
        aud: workqueue-service
        project_path: redhat/hummingbird/rpms
      permissions: [read, submit, retry]
      item_types: [mr]

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

Authorization rules

Each rule has either groups (OCP group membership from users/~ response) or issuer+claims (JWT bearer), not both.

  • issuer determines which JWKS keys validate the token signature. Use k8s as shorthand for the auto-detected cluster OIDC issuer.
  • claims matches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numeric project_id, boolean ref_protected).
  • permissions grants read, retry, and/or submit.
  • item_types optionally restricts the rule to specific work item types. Omit to apply to all types.
  • Every issuer rule must include aud in claims to prevent cross-service token reuse.

Handled types

handled_types declares which work item types this instance serves. Only listed types get their routers and executors registered. The processing sweep only picks up items matching these types. Empty or omitted = handle nothing.

Each type can have per-type operational config:

Type Field Description
mr gitlab_url GitLab instance URL (default: https://gitlab.com)
mr dashboard_base_url Dashboard URL for note creation (when unset, notes are not created)
mr job_retry CI job retry matcher config (when unset, retries are disabled)
mr projects Per-project/group token and URL configuration

Per-project token configuration

The projects map under handled_types.mr controls which GitLab API token the executor uses for each target project. Keys are GitLab project paths or group prefixes. Every project submitted to the service must match an entry – there is no default fallback token.

Matching order:

  1. Exact match on the full project path
  2. Longest group-prefix match on a / boundary (e.g. redhat/rhel/src matches redhat/rhel/src/kernel)

Each entry requires token_env (the name of the environment variable holding the GitLab API token) and optionally overrides gitlab_url:

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
        gitlab_url: https://gitlab.example.com
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

At startup, the service validates that all referenced token_env variables exist in the process environment and refuses to start if any are missing.

Adding a new issuer

Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:

- issuer: https://token.actions.githubusercontent.com
  claims:
    aud: workqueue-service
    repository: org/repo
  permissions: [submit]
  types: [mr]

Environment variables

Variable Default Description
DATABASE_URL — PostgreSQL connection URL (required)
SENTRY_DSN — Sentry DSN for error reporting
WORKQUEUE_SERVICE_CONFIG /etc/workqueue/config.yml Config file path
WORKQUEUE_SERVICE_LOCAL_AUTH_USER — Dev-only auth bypass (warns at startup)
WORKQUEUE_SERVICE_URL — CLI: service URL
WORKQUEUE_SERVICE_TOKEN — CLI: Bearer token
(per-project token_env) — GitLab API token(s) for MR executor (set in config projects)
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL — SQS queue URL for events (when unset, event consumer is disabled)
KONFLUX_KUBECONFIG_PATH — Path to Konflux kubeconfig for PLR fallback fetch
KUBEARCHIVE_URL — Kubearchive API URL for historical PLR lookups
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET — S3 bucket for audit log (production)
WORKQUEUE_SERVICE_AUDIT_S3_PREFIX workqueue-service-audit Key prefix within the audit S3 bucket
AWS_ACCESS_KEY_ID — AWS credentials for S3 audit archival
AWS_SECRET_ACCESS_KEY — AWS credentials for S3 audit archival
AWS_DEFAULT_REGION — AWS region for S3 audit archival

Development

Local Development

cd workqueue-service
./dev.sh db-start    # Start local PostgreSQL
./dev.sh db-migrate  # Run Alembic migrations
./dev.sh start       # Start the service
./dev.sh db-shell    # PostgreSQL shell
./dev.sh db-stop     # Stop PostgreSQL
./dev.sh db-reset    # Stop and delete volume

Running Tests

cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests

Tests use testcontainers[postgres] locally (auto-detects Podman) or a CI-provided PostgreSQL sidecar via DATABASE_URL.

License

This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.

4.5.33.1 - Work Queue Service Design

Architectural design document for workqueue-service. Covers the reasoning behind every major design choice so that future changes can be made safely. For operational usage, see Work Queue Service.

1. Design Philosophy

Four principles shaped the service’s architecture:

Security boundary. The service sits between untrusted callers (AI agents, CI jobs, automation scripts) and trusted write operations (GitLab pushes, MR creation). All changes are inspectable structured data, not opaque blobs. Path blocklists and action validation run before any GitLab write.

Type-agnostic queue, type-specific behaviour. The processing loop and per-item locking (a loop-lifetime session advisory lock) are generic. Type-specific logic (MR creation, CVE remediation) lives in pluggable subpackages that register deciders and executors at import time. Adding a new work item type requires no changes to the generic layer.

Single pending action. Only one action per work item at a time. After each action, the decider re-evaluates with current state. This avoids queuing actions that assume a future state which may not hold.

Phase + dimensions, not a linear state machine. The coarse lifecycle (active/completed/failed/cancelled) is separate from independent readiness signals (CI, approval, Konflux build). Dimensions can regress (force-push resets CI), which a linear state machine cannot express.

2. Architecture

Callers submit work items via REST API. The ProcessingManager drains pending actions in per-item threads, dispatching to type-specific executors. Executors create and update GitLab MRs via the Commits API.

flowchart LR
    callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
    API --> DB[("PostgreSQL")]
    Processing["ProcessingManager"] --> DB
    Processing --> GL["GitLab\nCommits API"]
    API -.-> Processing

3. Design Decisions Record

See Work Queue Service Design Decisions for all DDR entries covering data model, auth, processing, event pipeline, and MR lifecycle decisions.

4. Future Work

Event-driven dimension updates

The dimension mechanism is implemented with all MR dimensions following the uniform needs_* model (DDR-48). MR state events drive phase transitions (merge → completed, close → cancelled). Konflux PipelineRun events merge per-component state into the needs_plr dimension’s JSONB detail and re-arm needs_dashboard to trigger dashboard note updates. A 30-minute fallback executor (ensure_plr_status) fetches current PLR state from K8s/Kubearchive to catch missed events.

Non-managed MR tracking

Design how the service handles MRs it did not create (MRs without the managed-by::workqueue label). Requires subscribing to note events so the service can discover dashboard notes posted by the Lambda and adopt them as needs_dashboard dimensions. Prerequisite for retiring the Lambda.

Status rendering

The ensure_dashboard_note executor now renders a per-component Konflux build status table (using needs_plr dimension JSONB detail and the expected components from needs_expected_plrs) and updates the existing note in place via note_id.

CVE work item type

Add cve/ subpackage with CveWorkItem, CVE-specific decider (analyze → label → create advisory MR → check VEX → close), and CVE-specific executors. No changes to the generic layer.

Content validation and size limits

Add content-level inspection to the pre-commit checks: maximum file size, binary detection, and content pattern matching. Currently only path blocklists and action validation are enforced.

Reconciliation sweep

Note: The reconcile_mr executor (DDR-46) now provides per-item reconciliation. A full sweep (querying GitLab for all active items in bulk) remains a future optimization for catching items that fall through individual reconciliation.

Service rename

The service was renamed from hummingbird-mr-service to workqueue-service to reflect its generalized purpose. A further rename to hummingbird-work-queue or similar is deferred until a second work item type is implemented.

4.5.33.2 - Work Queue Service Design Decisions

Design Decision Registry (DDR) for workqueue-service — every major design choice, the alternatives considered, and why the chosen approach won. For the design overview, see Work Queue Service Design. For operational usage, see Work Queue Service.

DDR-1: Phase + dimensions vs linear state machine

Decision: Replace the 12-state linear state machine with a phase + dimensions model.

Context: The initial design used a single state column: pending → in_progress → created → mr_ci → approved → merged → post_merge_ci → building → releasing → completed. This broke because post-creation concerns (CI, approval, Konflux build) are independent — CI can pass before approval, approval can be revoked, force-push resets CI but not approval.

Alternatives: (a) Composite states (ci_passed_approval_pending) — exponential explosion. (b) Multiple boolean columns — rigid, schema migration for each new signal.

Rationale: Phase tracks coarse lifecycle; independent dimensions in a separate table track readiness signals with variant support for multi-instance cases (per-component Konflux builds). Dimensions can regress naturally. New dimension types need no schema migration.

DDR-2: Four generic phases (active/completed/failed/cancelled)

Decision: Narrow from 6 MR-specific phases to 4 generic phases.

Context: The intermediate design had pending, active, merged, completed, failed, closed — mixing generic concepts with MR-specific milestones.

Alternatives: Keep MR-specific phases and add more for each type.

Rationale: Phase answers “is this item done, and how?” Type-specific milestones (MR merged, CI passed) are tracked via dimensions and detail-table columns. Items start as active — there is no pending phase because the decider immediately assigns an action at creation time.

DDR-3: Dimensions in a separate table with variant support

Decision: Store dimensions in work_item_statuses table, not as columns on work_items.

Context: Initially considered a column per dimension (ci_status, approval_status, etc.).

Alternatives: (a) Column per dimension — schema migration for each new one. (b) JSONB column — flexible but loses CHECK constraints and complicates queries. (c) Separate table with composite PK (work_item_id, dimension, variant).

Rationale: Option (c). New dimensions need no migration. Variants handle multi-instance signals (multiple Konflux components). Each work item only has rows for dimensions that apply to it. The “are all dimensions satisfied?” query is a natural GROUP BY/HAVING.

DDR-4: Decider as pure function, not state machine transitions

Decision: The decider is a pure function decide(item) → action | None registered per type. It only determines the next action to execute.

Context: The original design had a TRANSITIONS dict mapping state → frozenset[state] with validate_transition().

Alternatives: (a) Transition table — rigid, can’t express “depends on which dimensions passed”. (b) Decider function — flexible, testable.

Rationale: The decider is trivially testable (given state, assert action). State transitions come from executors (they apply phase changes directly) and events (they update dimensions). The decider just picks the next action based on current state.

DDR-5: Single pending action, not a queue

Decision: One pending action per work item at a time.

Context: Considered queuing multiple actions (“merge, then trigger Konflux build”).

Alternatives: Action queue table per work item.

Rationale: Queued actions assume future state that may not hold. If “merge” fails, the queued “trigger Konflux build” is nonsense. After each action, the decider re-evaluates with current state. The attempts counter tracks consecutive failures, not total actions.

DDR-6: Claiming via leased_until (reusable)

Superseded by DDR-51. Work items are owned by a loop-lifetime session advisory lock, and the contract migration drops the leased_until column. See DDR-51 for the design and its motivation.

DDR-7: Generic + detail tables (joined table inheritance)

Decision: Split work_items into generic queue table + per-type detail tables (mr_details, future cve_details).

Context: The initial design had all MR-specific columns on the work_items table.

Alternatives: (a) Single table with nullable type-specific columns — sparse, doesn’t enforce type constraints. (b) JSONB payload column — flexible but loses type safety. (c) Joined table inheritance.

Rationale: Option (c). SQLAlchemy’s polymorphic_load="selectin" automatically loads the correct subclass. New types add their own detail table — no changes to the generic layer. Type-specific columns have proper types and constraints.

DDR-8: Type-dispatched decider/executor registries

Decision: Registry pattern for deciders and executors. Each type registers at import time via decider.register("mr", _mr_decide).

Context: Needed a way to dispatch to type-specific logic without the generic layer knowing about types.

Alternatives: (a) if/elif chains in the processing loop. (b) Class-based dispatch (strategy pattern). (c) Registry functions.

Rationale: Option (c). Simple, explicit, no class hierarchy needed. The mr/__init__.py import triggers registration. New types just add their own register() calls.

DDR-9: Structured file actions vs git patch_data

Decision: Replace opaque patch_data (BYTEA, git format-patch output) with structured file_actions (JSONB) and commit_message.

Context: The MR service sits on the security boundary — it writes to GitLab repos on behalf of untrusted callers. HUM-851 and the AI Investigation design rules require pre-commit inspection (path blocklists, content validation).

Alternatives: (a) Keep patch_data, clone repo, apply patch, then inspect diff — inspection happens after creating a working tree with credentials. (b) Parse patches with unidiff library — gives diffs not full content, still needs original files. (c) Structured file actions.

Rationale: Option (c). Inspection is a pure function over data the service already has — no clone, no git binary, no temp dirs. Path blocklists and action validation run before any API call. The file_actions JSONB column is also a queryable audit record.

DDR-10: GitLab Commits API vs git subprocess

Decision: Use the GitLab Commits API for branch creation and file commits. Delete git_ops.py.

Context: The original executor shelled out to git (clone, am, push), requiring: git binary in the container, subprocess timeout handling, GIT_ASKPASS credential management, temp directory lifecycle.

Alternatives: (a) Keep git subprocess with GIT_ASKPASS and timeout. (b) GitLab Commits API.

Rationale: Option (b). gitlab_sync.py in the same repo already proves the pattern. The Commits API’s start_branch parameter creates the branch automatically. No subprocess, no credentials in URLs, no temp dirs. Callers have full file contents (not patches), so git am semantics aren’t needed.

DDR-11: OIDC issuer from discovery, not hardcoded

Decision: Read the OIDC issuer from the /.well-known/openid-configuration discovery response. Rewrite the JWKS URI to use kubernetes.default.svc.

Context: The initial implementation hardcoded issuer = "https://kubernetes.default.svc". This failed on EaaS clusters where --service-account-issuer is https://oidc.op1.openshiftapps.com/.... The JWKS URI from discovery pointed to an IP:6443 that was unreachable from pods.

Alternatives: (a) Environment variable override (KUBE_OIDC_ISSUER). (b) Read from discovery.

Rationale: Option (b). No configuration needed — works on any cluster automatically. The JWKS fetch uses the path from discovery but rewrites the host to kubernetes.default.svc (always reachable on port 443). SA bearer token authenticates both requests (OpenShift blocks anonymous JWKS access).

DDR-12: item_type not type

Decision: Name the discriminator column item_type everywhere (DB, ORM, API, schemas).

Context: type is a Python builtin. Using it as a function parameter triggers ruff A002.

Alternatives: (a) Use type with alias in different contexts. (b) Rename to item_type everywhere.

Rationale: Option (b). Clean rename, no aliases, no special casing. Consistent at every layer.

DDR-13: source field removed

Decision: Remove the source column from mr_details. The caller_identity field (set automatically from authentication) serves the same purpose.

Context: source was a user-declared label (“rpms-ci”, “dependency-updater”) while caller_identity is the authenticated identity. Having both was redundant.

Alternatives: Keep both for cases where a shared SA submits on behalf of different logical sources.

Rationale: Removed until needed. caller_identity captures who submitted. If per-source filtering is needed later, it can be re-introduced as a field on MRWorkItemCreate.

DDR-14: attempts counts consecutive failures, not total actions

Decision: attempts increments only on executor failure and resets to 0 on success.

Context: The initial implementation incremented attempts before every action execution. A work item chaining 4 actions (ensure_mr → merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the third action.

Alternatives: (a) Separate total_actions counter. (b) Only count failures.

Rationale: Option (b). attempts answers “how many consecutive times has this action failed?” — the signal for escalation. Total actions executed is an audit concern, not a processing concern.

DDR-15: Executor applies state directly; decider only picks next action

Decision: Executors directly modify item.phase, item.mr_iid, etc. The decider only returns the next action to execute.

Context: Initially the processing loop set the phase based on ExecutorResult.success. This mixed two concerns.

Alternatives: (a) Processing loop sets phase. (b) Decider sets phase. (c) Executor sets phase.

Rationale: Option (c). The executor knows what happened (“I created an MR, so phase stays active”). Events that update dimensions also directly set state. The pattern is the same: something happens → state is updated → decider evaluates for the next action. Symmetry with event-driven dimension updates.

DDR-16: Items stay active after MR creation

Decision: After the ensure_mr executor succeeds, the item remains in active phase with no further actions pending.

Context: After MR creation, the decider returns None (no action). The item sits in active indefinitely.

Alternatives: (a) Set phase to completed after MR creation. (b) Keep active for future event-driven lifecycle tracking.

Rationale: Option (b). The active phase covers the entire lifecycle — from submission through CI, approval, merge, and post-merge tracking. The decider returns no actions until event-driven dimension updates (future SQS consumer) trigger the next step.

DDR-17: Audit log to S3, not database

Decision: Service-level audit (state transitions, decider decisions, actions taken) goes to S3 as immutable objects.

Context: Raw webhook events are already archived by sns-s3-archiver. The service needs its own audit trail for state transitions and decider evaluations.

Alternatives: (a) Database table (work_item_events). (b) S3 immutable objects. (c) Both.

Rationale: Option (b). S3 is immutable and cheap. The operational DB stays lean. Raw events are already in S3 via sns-s3-archiver; service-level audit captures the service’s reaction to those events.

DDR-18: Agent direction — dumber, not smarter

Decision: The hummingbird-agent should become a pure compute engine. The work queue service is the MR lifecycle orchestrator. The agent never calls the work queue service; the work queue service calls the agent.

Context: HUM-857 proposed adding create_merge_request as a new action in the agent’s actions.py. This was the wrong direction.

Alternatives: (a) Agent gains more actions (push, merge, MR management). (b) Agent becomes stateless compute; work queue service orchestrates.

Rationale: Option (b). The agent keeps LLM loop, tool registry, and sandbox — nothing else. Event routing, rate limiting, placeholder notes, result posting, session management, and all GitLab writes move to the work queue service. HUM-857 closed as Obsolete.

DDR-19: Shared database

Decision: The work queue service uses the same events PostgreSQL database as hummingbird-status and hummingbird-dashboard.

Context: Cross-service joins (e.g. work_items JOIN gitlab_merge_requests JOIN pipelineruns) are useful for the dashboard.

Alternatives: Separate database per service.

Rationale: Shared DB with service-specific Alembic migration chains (alembic_version_mr_service). Each service manages its own tables independently. The migration chain is isolated so one service can evolve its schema without affecting others.

DDR-20: Pydantic config over cki-lib/jsonschema

Decision: Use Pydantic for config validation. Load config from a YAML file at startup.

Context: Needed structured config for authorization rules and per-type operational settings. Alternatives: cki-lib config helpers, jsonschema, or plain dicts.

Alternatives: (a) cki-lib — adds a dependency, the service stays self-contained without it. (b) jsonschema — separate schema file, no type-safe access. (c) Pydantic — already a dependency (API schemas).

Rationale: Option (c). Pydantic provides type-safe attribute access, defaults, cross-field validators (e.g. mutual exclusivity of groups vs issuer+claims), and optional JSON Schema export. No new dependencies.

DDR-21: Claim-based rules (Vault bound_claims pattern)

Decision: Authorization rules match on arbitrary JWT claims, not just sub. All claim conditions AND; list values OR.

Context: Studied JWT trust/authz config from AWS API Gateway, HashiCorp Vault, Kubernetes Structured Auth, Istio, Traefik Hub, and oauth2-proxy. All support matching on arbitrary claims.

Alternatives: (a) Subject allowlist — rigid, requires exact sub values. (b) Claim-based matching — flexible, supports project_path, ref, groups, etc.

Rationale: Option (b). Vault’s bound_claims pattern is proven and simple. Values are coerced to lowercase strings before comparison to handle int/string mismatches (GitLab’s numeric project_id) and bool/string mismatches (ref_protected: true).

DDR-22: Issuer per rule, audience as claim

Decision: issuer is the only special field on a token rule — it determines which JWKS keys to use. All other JWT conditions (including aud) are standard claim matches in the claims dict.

Context: Considered a separate jwt.issuers section and/or treating aud as a first-class field alongside issuer.

Alternatives: (a) Global jwt section with trusted issuers + separate rules section. (b) aud as a dedicated field on each rule. (c) Issuer per rule, audience as a claim.

Rationale: Option (c). Keeps each rule self-contained and self-documenting. The Pydantic validator enforces that every issuer rule includes aud in claims to prevent cross-service token reuse.

DDR-23: Unified rule model for groups and tokens

Decision: A single rules list where every rule maps a remote identity (groups or issuer+claims) to local permissions.

Context: The original design had admin_groups/retry_groups (local permission → remote groups) alongside rules (remote identity → local permissions) — opposite mapping directions.

Alternatives: Keep separate group and token rule formats.

Rationale: Unified model. Every rule follows the same structure: remote identity → local permissions. Pydantic cross-field validators enforce mutual exclusivity.

DDR-24: Bearer-only auth with users/~ self-authentication

Decision: All API auth uses Authorization: Bearer tokens. Opaque OCP tokens (sha256~...) are validated by calling the OCP users/~ API with the user’s own token (self-authenticating).

Context: The oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. The TokenReview API would be cleaner but requires system:auth-delegator ClusterRoleBinding — a cluster-scoped permission not available on managed OCP clusters.

Alternatives: (a) Trust X-Forwarded-User from the proxy on all paths — injection risk on bypassed paths. (b) TokenReview API — requires cluster RBAC. (c) users/~ self-authentication.

Rationale: Option (c). No special RBAC needed. The users/~ endpoint returns username and groups. Results are cached with a thread-safe TTL cache (5 minutes). The /me endpoint bridges OAuth cookie sessions to Bearer tokens.

DDR-25: handled_types as explicit opt-in

Decision: handled_types declares which work item types this instance serves. Empty means handle nothing.

Context: Different deployments may serve different types from the same container image. Types carry per-type config (e.g. gitlab_url for MR).

Alternatives: (a) Handle all types by default. (b) Explicit opt-in.

Rationale: Option (b). Prevents accidental catch-all deployments. The sweep returns early when handled_types is empty — no items stolen from other instances in a shared-database deployment.

DDR-26: k8s shorthand skipped when not in-cluster

Decision: Rules with issuer: k8s are silently skipped when the in-cluster CA bundle is absent.

Context: The K8s OIDC issuer URL varies by cluster and is auto-detected at runtime. Local development doesn’t have a cluster.

Alternatives: (a) Error on startup if k8s issuer can’t be resolved. (b) Skip silently.

Rationale: Option (b). Allows using a production config file for local development and testing without startup errors.

Decision: The oauth-proxy sidecar handles only the OAuth login flow. API endpoints bypass the proxy. A proxy-protected /me endpoint bridges cookie sessions to Bearer tokens. The proxy must be configured with --cookie-refresh=1h to prevent stale-token failures.

Context: The OCP oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. Protected paths validate Bearer tokens via TokenReview (requires cluster RBAC we cannot obtain) or reject non-OCP tokens. Bypassed paths pass all Bearer tokens through but ignore cookies. Additionally, the OCP access token embedded in the proxy’s session cookie has its own expiry (default 24h), independent of the cookie lifetime. Without --cookie-refresh, the cookie outlives the token, causing the proxy to forward expired credentials while still reporting the user as authenticated — the “logged in but not authorized” failure observed in the hummingbird-dashboard deployment.

Alternatives: (a) All paths protected — blocks non-OCP Bearer tokens. (b) --openshift-delegate-urls — requires system:auth-delegator ClusterRoleBinding. (c) App reads proxy cookies directly — tight coupling to proxy internals. (d) Login-only proxy with /me bridge.

Rationale: Option (d). The /me endpoint is the only proxy-protected API path. It reads X-Forwarded-Access-Token (set by the proxy from the cookie session) and returns the OCP access token to the caller. Subsequent API calls use Authorization: Bearer sha256~... on bypassed paths, where the app validates via the OCP users/~ API. --cookie-refresh=1h keeps the embedded token fresh. TTL-bounded caching (5 minutes) on users/~ results ensures expired-token failures are never permanent (unlike @functools.cache which caches failures indefinitely).

DDR-28: Audit log with logging + optional S3 archival

Decision: Always log audit entries via Python logging at INFO level, and optionally archive to S3 as gzipped JSON objects when WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set.

Context: Needed an immutable record of state transitions, action outcomes, and auth decisions for compliance and debugging.

Alternatives: (a) Database table — bloats the operational DB with a different concern. (b) S3-only — no visibility without querying Athena; loses events if S3 is unreachable.

Why logging + S3: Logging to stdout is zero-config and integrates with existing container log collection for real-time visibility. S3 provides the durable, queryable archive (Athena) for compliance.

Key design rules:

  • Audit failures never propagate — write_audit_entry() catches all exceptions and logs them, but never raises.
  • S3 keys use YYYY/MM/DD/HH/MM/ prefix for Athena partitioning.
  • ContentEncoding: gzip set so S3 API consumers get transparent decompression.

Field conventions: See Audit Log Field Conventions in the operational documentation.

DDR-29: Events processed in work-item threads, not consumer threads

Decision: The SQS consumer thread stores events in the pending_events table. Existing per-work-item threads drain and process them.

Context: The ProcessingManager already runs one thread per work item with per-item ownership (DDR-51). Processing events in the same thread means events and actions are naturally serialized per work item.

Alternatives: (a) Consumer threads handle events directly — concurrency conflicts with action threads. (b) FIFO queue + in-memory dispatch — receipt handle management, visibility timeout extensions, message loss on crash.

Rationale: Option (c). Events join the existing thread structure. The decider runs after each event, determining follow-up actions. The single-thread-per-item invariant is preserved.

DDR-30: pending_events table as durable event store

Decision: Use a PostgreSQL table (pending_events) instead of FIFO queues or in-memory queues for event delivery to work-item threads.

Context: Since work-item threads do the processing, the SQS consumer just needs to deliver events to them durably.

Alternatives: (a) FIFO queue + in-memory dispatch — complex EventBroker, receipt handle management, message loss on crash. (b) FIFO queue + direct handling — concurrency conflicts.

Rationale: The DB provides durability (crash recovery), serialization (single thread per work item), and ordering (last_event_at guard). No FIFO queue, router thread, MessageGroupId computation, in-memory queues, or visibility timeout management needed.

DDR-31: SQS delete after INSERT, not after full processing

Decision: Delete the SQS message after INSERT+COMMIT into pending_events, not after full event processing.

Context: The hummingbird-agent deletes after handling (no DB intermediary). The workqueue-service can delete after INSERT+COMMIT because the event is crash-recoverable in PostgreSQL.

Alternatives: Delete after full processing — requires visibility timeout extensions, receipt handle passing, heartbeat management.

Rationale: After INSERT+COMMIT the event is durable in PostgreSQL. Duplicate SQS delivery may INSERT the same event twice; the last_event_at guard makes processing idempotent.

DDR-32: Resolver filters by phase, consumer stays read-only

Decision: The resolver filters by phase = 'active' on the MRWorkItem query. SQLAlchemy’s joined table inheritance handles the work_items table access implicitly. The consumer does SELECT + INSERT only, no UPDATE, no row lock contention.

Context: Only active items accept event-driven transitions (see DDR-37). Filtering at resolve time avoids unnecessary INSERTs into pending_events, thread spawns, and no-op processing for terminal items.

Alternatives: (a) Resolve without phase filter, guard only at processing time — wastes work for terminal items. (b) Consumer checks phase and applies transitions directly — contention with work-item threads.

Rationale: The phase filter adds negligible cost (PK lookup on work_items, typically one row). The consumer remains read-only. Business logic (phase transitions, decider, audit) is applied by the generic handle_event layer in the work-item thread.

DDR-33: Raw events stored, interpretation separated from application

Decision: The pending_events table stores the raw event JSON body and event type. Type-specific handlers interpret events (read-only on the work item), the generic layer applies the result.

Context: Event processing has three concerns: interpretation (what does this event mean?), application (phase transition, decider, audit), and routing (which work items?). Mixing them makes handlers hard to test and inconsistent across types.

Alternatives: (a) Handlers apply all changes directly — mixes interpretation with mutation, each handler reimplements decider/audit boilerplate. (b) Pre-process events in the consumer — consumer needs type-specific knowledge.

Rationale: Handlers return an EventResult dataclass (phase, event_ts, trigger metadata). The generic handle_event applies the phase transition via update_work_item_from_event, runs the decider, and emits audit entries. Handlers have read-only access to the work item and session (e.g. to check existing dimensions or detail fields) but EventResult is their only output — they never mutate the work item directly. This ensures decider/audit logic is consistent across all item types. See DDR-38 for the dimension delivery extension.

DDR-34: Per-work-item advisory lock for insert/drain coordination

Superseded by DDR-51. The insert/exit handshake was replaced by a SELECT ... FOR UPDATE row lock on work_items around the exit decision, which serializes every enqueue writer. See DDR-51 for the design and its motivation.

DDR-35: Sweep for startup recovery

Decision: Extend the periodic sweep to also check for work items with unprocessed pending_events rows.

Context: Events that arrive while the service is stopped, or that survive a mid-processing crash, must be processed on restart.

Alternatives: Separate startup-only scan.

Rationale: Unifying startup recovery with the periodic sweep keeps a single code path. The sweep runs immediately on manager.start(), catching events from downtime. It also serves as a safety net behind the advisory lock.

DDR-36: Consumer lifecycle and shutdown ordering

Decision: The SQS consumer is a single daemon thread started from the FastAPI lifespan. Shutdown stops the consumer before work-item threads.

Context: If work-item threads stop while the consumer is still INSERTing, events could be orphaned. The consumer must stop first.

Alternatives: Non-daemon thread with explicit join.

Rationale: Single thread is sufficient for low event volume (MR state changes). Daemon thread ensures process exit even if the thread hangs. Shutdown ordering: (1) set shutdown_event, (2) join consumer thread (timeout=30s), (3) call manager.stop(). No consumer when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is unset.

DDR-37: Terminal phase semantics — only active items accept events

Decision: Only items in phase = 'active' accept event-driven phase transitions. completed, failed, and cancelled are terminal phases — recovery requires explicit action via the retry API.

Context: Initially only failed was guarded (phase != 'failed'). completed and cancelled items could be modified by events: a closed event could transition a completed item to cancelled, and an opened event could reactivate a cancelled item.

Alternatives: (a) Guard only failed — inconsistent, completed items are accidentally mutable. (b) Growing exclusion list (NOT IN ('failed', 'completed')) — fragile if new phases are added. (c) Positive match phase = 'active'.

Rationale: Option (c). A single positive match is clearer and future-proof. If a work item was cancelled (MR closed) or completed (MR merged), only deliberate human action (retry API) should reactivate it — not an automated event. The retry API accepts both failed and cancelled items. The resolver also filters by phase = 'active' at query time (DDR-32) to avoid unnecessary INSERT/processing for terminal items.

DDR-38: Dimension updates via EventResult

Decision: Event handlers return dimension upserts as part of EventResult. The generic handle_event layer applies them.

Context: DDR-33 established that handlers return structured results and the generic layer applies them. Dimensions (DDR-1, DDR-3) need the same pattern — handlers identify which dimensions changed, the generic layer upserts them and emits audit entries.

Alternatives: (a) Handlers call crud.upsert_dimension() directly — breaks the read-only handler contract from DDR-33. (b) Separate dimension-specific consumer — duplicates the resolver/handler infrastructure.

Rationale: EventResult.dimensions is a list of DimensionUpdate dataclasses. After phase transition logic, handle_event iterates over dimensions and calls crud.upsert_dimension() for each, emitting an AuditEntry with event_type="dimension_change" using the existing dimensions field. Dimension upserts run even when the phase transition is stale (the early return is refactored to skip only the phase/decider block). This ensures dimension-only events (e.g. note events with no phase change) are still applied.

DDR-39: Server-derived idempotency keys with phase-aware uniqueness

Decision: Replace the client-supplied idempotency_key field with a server-derived key mr:{project}:{source_branch}:{generation}. Enforce uniqueness only among non-terminal phases via a partial unique index.

Context: The original API required callers to supply a unique idempotency_key per submission. Recurring callers (dist-git sync, upstream sync) had to invent unique keys for each invocation and track them across runs. The global unique constraint also blocked new submissions for a branch that had a previous completed or cancelled item with the same key.

Alternatives: (a) Client-supplied key with a prefix convention — callers still need to generate and track keys. (b) Server-derived key from the natural deduplication identity (project, source_branch) with a generation counter for lifecycle restarts.

Rationale: Option (b). The idempotency key is a DB-level safety net, not a caller concern. The natural identity is (project, source_branch); the generation counter increments when a new item is created after the previous one reached a terminal phase. The partial unique index (WHERE phase NOT IN ('completed', 'cancelled')) ensures at most one active/failed item per idempotency key while allowing completed items to coexist with their successors.

DDR-40: Upsert semantics for recurring MR management

Decision: Replace one-shot insert_mr_work_item with upsert_mr_work_item that looks up an existing active/failed item by (project, source_branch) and determines the correct action from a content hash comparison.

Context: Recurring callers submit MRs for the same branch repeatedly. Under the old insert model, each submission created a new work item — even when the previous one was still active, leading to orphaned branches and duplicate MRs.

Alternatives: (a) Client manages deduplication — check before submit, handle conflicts. (b) Server-side upsert — callers always submit current desired state, the service deduplicates transparently.

Rationale: Option (b). The upsert does a (project, source_branch) lookup among non-terminal items and branches on the result:

  • No match → create a new item (actionable = true)
  • Active, same content → no-op (return existing, "unchanged")
  • Active, different content → update fields, set actionable = true, reset needs_mr = "true"
  • Failed → reset to active, update content, clear error ("retried"), set actionable = true
  • Terminal (completed/cancelled) → create a new generation (DDR-39)

Concurrent insert races are handled via DDR-39’s partial unique index — the loser catches the IntegrityError, rolls back, and retries.

DDR-41: ThreadPoolExecutor with bounded workers

Decision: Replace unbounded per-item threading.Thread spawning with a fixed-size ThreadPoolExecutor (10 workers).

Context: The periodic tick mechanism (DDR-42) can wake many idle items simultaneously. Unbounded thread creation would cause resource spikes (~109 threads in the worst case).

Alternatives: (a) Keep per-item threads with a semaphore. (b) Bounded thread pool.

Rationale: Option (b). ThreadPoolExecutor provides built-in queuing when all workers are busy, deduplication via _submitted set, and clean shutdown via pool.shutdown(wait=True). The 10-worker limit caps concurrent GitLab API calls.

DDR-42: Hash-bucketed tick events for periodic re-evaluation

Superseded by DDR-48. The synthetic tick_30m and tick_5m events were replaced by the pending_until column on work_item_statuses. See DDR-48 for the design and its motivation.

DDR-43: Empty diff detection via repository_compare

Decision: Detect and handle empty MR diffs at two call sites: ensure_mr (prevention on create and force-push) and check_empty_diff (periodic via sweep).

Context: The detection-execution race causes empty MRs when the target branch moves between change detection and MR creation. Also, existing MRs can become empty after creation.

Alternatives: (a) Compare file content hashes before commit. (b) GitLab repository_compare API after commit/push.

Rationale: Option (b). The repository_compare endpoint returns the diff between two branches. Empty diffs list means the branches are identical. This is authoritative and handles edge cases that hash comparison might miss (e.g. whitespace normalization). Cost: one API call per check (~4 calls/minute across all items for periodic checks).

DDR-44: Per-item processing deadline

Decision: Exit the work item loop after 60 seconds of wall-clock time.

Context: The tick mechanism (DDR-42) and check_stale_mr action could cause extended processing loops for items that repeatedly generate new actions. A deadline prevents any single item from holding a pool worker indefinitely.

Alternatives: (a) Limit loop iterations. (b) Wall-clock deadline.

Rationale: Option (b). Wall-clock time is predictable regardless of action duration. The deadline is between loop iterations, not mid-executor, so executors have their own HTTP timeouts. The item is re-picked by the next sweep if work remains.

DDR-45: Dimension-driven decider with needs_* flags

Decision: The decider scans needs_* dimensions in priority order instead of encoding per-concern conditionals.

Context: The original decider checked item.mr_iid is None for create, dashboard_link dimension absence for dashboard notes, and would need more checks for auto-merge, stale detection, metadata sync. Each new concern added another conditional.

Alternatives: (a) Per-concern conditionals in decider. (b) Dimension-driven: events/executors set needs_* flags, decider scans them.

Rationale: Option (b). The decider becomes a simple priority scan. Executors own their trigger-to-next-step logic by setting the appropriate needs_* dimension. Events set needs_reconcile to request re-evaluation. Adding a new concern = new dimension + handler in reconciler, no decider changes needed.

Update (DDR-48): The original three-category model (has_*, needs_*, observed state) was unified into a single needs_* trigger model with "true"/"false" status. See DDR-48.

DDR-46: reconcile_mr pattern

Updated by DDR-52. The “merge latency to seconds” trade-off below did not hold on busy target branches: the branch churn (largely the workqueue’s own fleet) resets mergeability to unchecked faster than the 30-minute cadence re-samples it, starving managed MRs for days. DDR-52 adds the deadline-driven sweep and an adaptive recheck backoff that remove this failure mode; the direct-merge decision itself stands.

Decision: Single reconcile_mr executor handles all MR state convergence, with cheap (always) and expensive (flagged) tiers. Uses direct mr.merge() instead of GitLab auto-merge.

Context: Separate executors per concern (check_stale_mr, enable_auto_merge, sync_metadata) each needed their own registration, decider rule, and dimension tracking. GitLab auto-merge (auto_merge=True) introduced state-management complexity: merge_when_pipeline_succeeds tracking, stale SHA detection after force-push, and recurring 422 noise for non-mergeable MRs.

Rationale: One mergerequests.get() call serves all cheap checks. The reconciler checks detailed_merge_status == "mergeable" and calls mr.merge() directly — no GitLab auto-merge state to track, no skip-lists to maintain, no stale SHA detection. Pipeline completion and approval webhooks trigger reconcile via the needs_reconcile dimension, keeping merge latency to seconds. Expensive operations (empty diff via repository_compare) only run when needs_empty_check is actionable (periodic via pending_until, DDR-48). The reconciler is idempotent — all corrections are conditional on actual vs desired state.

DDR-47: CI job retry herder

Updated by DDR-48. The needs_ci_retry="pending_tick_5m" status and tick_5m promotion mechanism have been replaced by pending_until on the dimension row.

Problem: Hummingbird CI pipelines frequently fail due to transient infrastructure errors (runner system failures, git clone timeouts, OOM kills). Manual retries waste developer time and delay merges.

Options considered:

  • A. Executor action in workqueue-service (selected): Shares existing GitLab tokens, project config, DB-persisted state, and audit trail.
  • B. Standalone herder service: Maximum CKI fidelity but requires a new deployment, duplicates token management, and has no MR lifecycle integration.
  • C. Reconciliation polling: No new webhooks, but 30-minute detection latency and wastes API calls polling healthy pipelines.

Decision: Option A. The matcher engine is ported from the CKI pipeline-herder with a reduced field set. Build events are received via the existing SNS→SQS pipeline and resolved to MR work items by matching pipeline_id against head_pipeline_id on mr_details.

DDR-48: Uniform needs_* dimension model with pending_until

Decision: Unify all dimensions under a single needs_* trigger model with "true"/"false" status and an optional pending_until column for deferred re-evaluation. Drop synthetic tick events.

Context: The dimension system had grown three distinct categories (DDR-45): has_* positive completion markers, needs_* action triggers, and observed state dimensions (approval, pipeline, merge_status, has_conflicts). The has_* and needs_* split created asymmetric decider logic. Observed state dimensions were write-only — the reconciler set them but no code read them. The tick_5m/tick_30m synthetic event mechanism (DDR-42) required hash bucketing, NOT EXISTS guards, and special "pending_tick_5m" status values.

Alternatives: (a) Keep the three-category model and add pending_until alongside tick events — increases complexity with two deferral mechanisms. (b) Unify to needs_* with pending_until.

Rationale: Option (b).

  • Uniform semantics: every dimension is a needs_* trigger. Status is "true" (action needed) or "false" (handled). The decider uses a single check pattern: != "false" means actionable.
  • Absent = actionable: missing dimension rows are treated as needing action, so new items don’t require explicit initialization of every dimension.
  • pending_until replaces ticks: executors set pending_until to schedule future re-evaluation. The sweep finds expired pending_until and notifies — no synthetic events, no hash bucketing, no "pending_tick_5m" status values.
  • Dropped dimensions: approval, merge_status, has_conflicts were write-only. The pipeline dimension carried a single integer and is better modeled as a typed column on mr_details.
  • Deferred dimensions: the decider pre-filters dimensions where pending_until is in the future, treating them as "false".

DDR-49: Replace pending_action TEXT with actionable BOOLEAN

Decision: Drop the pending_action TEXT column. Replace it with actionable BOOLEAN DEFAULT FALSE. The decider is called in exactly one place: the processing loop’s derive-and-dispatch step.

Context: pending_action served three roles: (1) sweep discoverability (WHERE pending_action IS NOT NULL), (2) processing loop continue/exit signal, (3) executor dispatch key. Roles 2 and 3 made the column authoritative for which action to run, creating a sync burden with the dimension-driven decider.

Alternatives:

  • Remove entirely — rejected: the decider treats absent dimensions as actionable, so a SQL query over dimension rows cannot find items whose next action derives from a missing row. A wake-up signal persisted to the DB is required.
  • next_action TEXT (renamed cache) — rejected: requires decider.decide() calls in the CRUD layer. The authoritative-looking name invites future dispatch reads.
  • actionable BOOLEAN — selected.

Rationale: A boolean is sufficient to bridge the absent-dimension discoverability gap. Stale True is harmless (false-positive sweep → loop starts, decider returns None, exits immediately). actionable is written in exactly two places: True when work is enqueued (creation, update, retry, event insertion), False only on the loop’s exit branch (committed atomically when both decider.decide() returns None and no pending events exist). This invariant ensures the item stays sweep-visible throughout processing — a crash at any point leaves actionable = True, preventing items from becoming permanently stuck.

DDR-50: Event drain tolerates concurrent deletion

Decision: The event drain (_process_pending_events) reads the (id, event_type, event_body) of all listed pending events before processing the first one, and deletes rows by id. A row deleted by another pod mid-drain makes the delete a no-op instead of an error.

Context: The lease is acquired once per loop run and renewed only while executor actions run; it is neither renewed nor re-checked during the drain, and the per-item deadline is only checked between drains (DDR-44). A slow drain can therefore outlive the lease — most notably during rolling deployments, where the terminating pod’s in-flight drain overlaps the new pod’s startup sweep. Committing per event expires the remaining ORM instances, so the next iteration’s attribute access lazily refreshes against the database; when the other pod already deleted that row, SQLAlchemy raises ObjectDeletedError and the whole work-item loop dies (observed as Sentry issue 7829339838, 35 seconds after a rollout).

Alternatives:

  • Hold the per-item advisory lock across the drain — rejected: the lock is transaction-scoped (DDR-34) and the drain commits per event for crash isolation; spanning all events would require a single transaction and lose per-event progress.
  • Renew or re-check the lease inside the drain — rejected: shrinks but does not close the takeover window (the lease can still expire mid-iteration), and does not fix the stale-instance crash itself.
  • Re-query each event before processing it — rejected: adds a query per event, and the row can still vanish between the check and the delete.

Rationale: Reading the attributes while the instances are fresh — before the first commit — means no lazy refresh can fire mid-drain, and delete-by-id is naturally idempotent. An event may then be applied by both pods, but application is already at-least-once for crash recovery (DDR-31) and idempotent (dimension upserts, last_event_at guard). The drain becomes robust no matter which actor deletes a row.

DDR-51: Per-item processing ownership and exit safety

Decision: Loop ownership is a loop-lifetime session advisory lock — pg_try_advisory_lock(0x5751, work_item_id.int & 0x7FFFFFFF) taken on a pinned connection at loop entry — and the loop’s exit decision runs under a SELECT ... FOR UPDATE row lock on work_items. The leased_until lease with its heartbeat (DDR-6) and the insert/exit advisory-lock handshake (DDR-34) are removed; the column is dropped by the contract migration after the redesigned version has fully rolled out.

Context: DDR-29/DDR-30 require at most one processing loop per work item across pods, but no mechanism delivered that invariant. The lease was acquired once per loop run, renewed only while an action ran, and had no owner, so it excluded only the next action, not the loop. The exit wrote actionable = false blindly, so an API enqueue committing between the loop’s state read and that write was overwritten — permanently stalling items whose only wake-up signal was a PROCESS_IMMEDIATELY dimension (no pending_events row, invisible to every sweep leg). The transaction-scoped advisory lock covered only the consumer’s INSERT and the exit, not the API writers. The 2026-11-04 ObjectDeletedError (Sentry 7829339838) was one symptom: a new pod’s startup sweep took an item whose lease the old loop had cleared between actions. !1494 (DDR-50) fixed the crash, not the ownership model.

The two-32-bit-key form (classid=0x5751, objsubid=2 in pg_locks) deliberately does not overlap the legacy single-64-bit-key space (objsubid=1), so during a mixed-version rolling update an old pod’s handshake and a new pod’s session lock cannot block each other.

The key is the item UUID’s low 31 bits — the positive range of the int4 second argument. Two unrelated items whose UUIDs share those bits collide: the loser returns immediately (counted in hummingbird_workqueue_processing_lock_skipped_total) and the sweep re-spawns it once the winner’s loop finishes. A collision delays one item; it never allows two loops for the same item and never loses work.

Every enqueue path already updates the work_items row in the same transaction as its durable write (dimension upsert, pending_events insert, retry mark). The row lock serializes the exit against all of them without any new protocol; this contract is now explicit and must hold for future enqueue paths.

Alternatives:

  • Compare-and-set on a new revision column — correct and non-blocking, but adds a migration and a permanent column, and onupdate is ORM-level, so old-pod writers would not bump it during the rollout.
  • CAS on updated_at or xmin — no migration, but updated_at is transaction-start time (uniqueness only by microsecond) and xmin is an obscure system column with freeze/wrap edge cases.
  • Durable enqueue token (a wake-up row per enqueue, consumed only on a clean exit, enumerated by the sweep) — the structural fix that removes the shared-boolean race by construction; deferred as a larger change (a table, an insert on every enqueue path, a drain lifecycle, and a sweep leg).
  • Extend the advisory lock to API writers — couples synchronous HTTP handlers to the processing lock for a whole loop and reintroduces the bug whenever a writer forgets.
  • Fix the timestamp lease in place (owner token, renewal for the whole loop, re-assert before each action) — time-based expiry still cannot exclude a paused or partitioned worker holding the lock past its TTL, and keeps the heartbeat machinery.
  • SERIALIZABLE exit transaction — PostgreSQL SSI only guarantees serialization among serializable transactions; mixing isolation levels and retrying under contention is more machinery than the row lock.
  • Leader election / single-writer pod — removes rollout overlap but defeats replica throughput and distinct-item parallelism.

Rationale: The session lock makes “one loop per item” true by construction for the whole loop lifetime, not per action: a loser returns immediately without holding a worker slot, crashes recover through connection death plus the sweep, and no TTL or heartbeat is needed. The row lock makes the exit decision atomic with respect to every writer without a migration, and old-pod writers block on it too. A spawn request that arrives while the loop is still exiting is captured by a per-item rerun flag, and after a clean exit the loop unlocks, re-checks for work, and re-acquires if a writer slipped in.

Each running loop pins one pooled connection (pool sized to the worker count with overflow headroom, pool_pre_ping, libpq keepalives and tcp_user_timeout), and every loop session binds to it. A dead connection therefore fails the work — the exception propagates and the loop aborts — instead of leaving a loop running under a lock the server already released; the still-actionable item is recovered by the sweep. The API request path can block for the duration of the exit decision (a few DB-only statements), so the exit transaction must stay DB-only.

During a mixed-version rolling update, an old pod’s leased loop and a new pod’s locked loop can briefly both run the same executor action for one item: the two-key lock cannot conflict with the old lease path. Read-and-upsert actions (reconcile_mr, ensure_plr_status, ensure_expected_plrs) converge on the same state. Creation actions and retry_ci_jobs check their result field (e.g. mr_iid) before acting, so a duplicate requires both loops to pass that check before either commits — possible in the worst case, producing one duplicate external mutation (a second MR or note, or a second CI retry) for an item mid-action during the pod replacement. New pods deliberately do not skip rows with an unexpired leased_until: a lease written by a pod that died mid-rollout must not hide the item. The window is bounded by the rollout and ends when the contract migration drops the column.

DDR-52: Deadline-driven sweep and adaptive merge-recheck

Decision: The sweep wakes at deadlines instead of polling: after each pass it sleeps until the earliest armed pending_until (plus ~1 s epsilon), capped at the former 5-minute interval as a backstop — or for just the epsilon when a deferred timer is already due, letting the next pass’s expired scan pick it up. unchecked/checking/unknown merge statuses re-arm needs_reconcile at now + RetrySchedule.delay(step) on an exponential schedule (15 s, doubling, 5-minute cap) with the step persisted in the dimension detail, and a partial index on work_item_statuses (pending_until) covers the sweep’s hot path.

Context: The direct-merge design (DDR-46 lineage; !1327) merges when a reconcile GET observes detailed_merge_status == "mergeable", with a 30-minute retry cadence and the sweep (fixed 5 minutes) as the only timer. On busy target branches this starves: every push to the target branch resets stored mergeability to unchecked, GitLab recomputes it only lazily on a single-MR GET (seconds), the service samples once per 30 minutes, and GitLab emits no webhook when a recheck completes. Evidence: containers!22697 unmerged 6 days while green; 2026-11-10 logs show 15/16 reconcile GETs observing unchecked and the two mergeable observations losing the merge PUT race to concurrent fleet merges (422 “Branch cannot be merged”). The branch churn is largely the workqueue’s own lockfile fleet (59 merges in two hours on 2026-11-10), so the service invalidates its own merge candidates. The branch churn also means any fixed short interval is a compromise: real blockers need minutes, rechecks need seconds.

now is passed to the deadline query as a bound parameter rather than SQL now() because pending_until values are written with the Python clock; comparing in one clock domain avoids app-vs-DB skew shifting wakeups. The MIN query mirrors find_expired_pending’s item filters (active phase, accepted types) so completed items — which keep their last dimension rows — cannot pin the sweep to a stale deadline. The deadline lookup computes the earliest future deadline and a past-due flag in one filtered-aggregate query — one snapshot — so a timer expiring between the sweep’s expired scan and the lookup cannot be missed by both; the past-due flag excludes actionable items, mirroring the expired scan, so a running loop’s timer cannot spin the sweep. A wake event interrupts the sleep when an earlier deadline is armed in-process (set from spawn_loop()’s enqueue paths and after executor action commits; sweep-initiated submissions deliberately do not poke — they arm no timer, and poking would wake the sweep into a pass that re-submits, spinning cross-pod while another pod’s loop holds the item lock during a rollout); spurious wakes cost one indexed query and a no-op sweep. Multi-pod: each pod computes its own MIN; the arming pod wakes exactly, others see the work within the backstop cap, and duplicate spawns are absorbed by the per-item advisory lock (DDR-51).

Alternatives:

  • Shorter fixed sweep interval — wasted cycles and still granular; a +30 s timer would still round up to the interval.
  • Short dimension timers without a sweep change — the 5-minute sweep granularity silently becomes the effective interval; the timer is a lie without deadline-driven wakeups.
  • Sleeping and re-GETting inside the executor — holds a worker slot (10 max) and the pinned DB connection, brushes the 60-second item deadline (DDR-44), and violates the “reconcile_mr: cheap operations only” contract (DDR-46).
  • Fixed fast interval without backoff — a permanently-unchecked MR would poll ~2880×/day.
  • Backoff step derived from row timestamps (updated_at deltas) — fragile, implicit, breaks on any other write to the dimension row.
  • RetryAfter state merged into the crud state union with a reserved detail key — more machinery and magic than needed for one consumer; revisit on a second non-MR consumer.
  • Re-enabling GitLab server-side auto-merge — removes service ownership of the merge decision (DDR-46); the adaptive recheck achieves convergence without it.

Rationale: A mergeable MR converges within one or two recheck probes (~15–45 s) instead of winning a 30-minute lottery, and concurrent-merge race losers recover within seconds, so the fleet race self-heals — the 422 → immediate retry → unchecked → +15 s → mergeable → merged sequence replaces the 30-minute re-sample. unknown shares the fast family (GitLab documents it alongside unchecked); a persistent unknown settles at the 5-minute cap, and the mr.state != "opened" guard keeps closed/merged MRs from being polled. Permanently-unchecked MRs poll at the 5-minute cap (the same cadence as error backoff), visible via the hummingbird_workqueue_merge_rechecks_total counter labeled by merge status. Real blockers keep the 30-minute cadence and reset the backoff step (detail=None, the documented clear path); webhook re-arms preserve the step (the _PRESERVE sentinel) and consume one — each immediate reconcile observing unchecked advances the backoff, deliberately throttling poll storms on chatty MRs. The GitLab-error path (the 5-minute error re-arm without a detail write) also preserves a matured step, so the first unchecked probe after an outage waits at the cap — deliberate. The sweep runs as often as the earliest deadline (potentially every ~15 s under recheck storms); each pass is one indexed MIN query, one expired-pending query, and the actionable-or-pending-events union query, and only items with due work spawn loops — hence the partial index. The direct-merge decision stands; this record removes its starvation failure mode on busy branches. Rollout: the deadline-driven sweep and its supporting partial index land first, then the reusable RetrySchedule type, then the merge-recheck application (schedule constant, detail step, outcome enum, metric, and docs); the migration adds the partial index only.

4.5.34 - Service metrics

Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes Metrics API CPU/memory gauges, OpenShift cluster resource quota usage, PVC filesystem usage, GitLab CI/CD schedule status, and GitLab webhook alert status.

Features

  • Explicit collectors - nothing runs unless listed in METRICS_CONFIG
  • Kubernetes Metrics API - per-container CPU and memory gauges
  • Cluster resource quotas - AppliedClusterResourceQuota usage and limits
  • AWS Cost Explorer - yesterday’s NetAmortizedCost by team and service
  • Volume filesystem usage - PVC disk usage via shutil.disk_usage()
  • GitLab CI/CD schedules - pipeline schedule health via GraphQL
  • GitLab webhooks - webhook alert status via the REST API
  • Prometheus - HTTP metrics on port 9090

Configuration

Collectors are off unless listed in METRICS_CONFIG:

enabled: [kubernetes]

Unknown names fail startup.

Variable Purpose
METRICS_CONFIG YAML with enabled collector list (required)
KUBERNETES_CONFIG YAML with namespaces for the kubernetes collector
VOLUMES_CONFIG YAML with volumes list for the volume collector
GITLAB_URL GitLab instance URL (default https://gitlab.com)
GITLAB_TOKEN GitLab API token with read_api scope (required for gitlab collectors)
GITLAB_CONFIG YAML with namespaces (schedules) and webhooks (webhooks) lists
METRICS_PORT HTTP port (default 9090)
SENTRY_DSN Optional Sentry DSN
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY Cost Explorer IAM user (hub)
AWS_DEFAULT_REGION Must be us-east-1 for Cost Explorer

Kubernetes Metrics API

hummingbird_k8s_resource_usage{namespace,pod,container,resource,app} with resource=cpu|memory and app sourced from the pod’s app.kubernetes.io/name label (empty string if absent). Instant gauges from /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods. This is not cAdvisor: managed OpenShift does not grant nodes/metrics or cluster-monitoring-view, so kubelet /metrics/cadvisor and platform Prometheus federation are not available.

If KUBERNETES_CONFIG.namespaces is unset, the collector uses the pod’s ServiceAccount namespace.

Cluster resource quotas

The kubernetes collector also reads namespace-scoped AppliedClusterResourceQuota objects from the quota.openshift.io/v1 API and exposes two gauges that mirror the openshift-state-metrics schema with a hummingbird_ prefix:

  • hummingbird_clusterresourcequota_usage{name,resource,type} – cluster-wide totals from status.total (type=hard|used)
  • hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type} – per-namespace breakdown from status.namespaces (type=hard|used)

resource values match Kubernetes quantity names: cpu, memory, requests.cpu, limits.memory, pods, etc. Quantities are converted to base units (cores, bytes, count).

The same ACRQ may appear in multiple namespaces; the global metric is idempotent across duplicates. Stale series are removed when ACRQs disappear.

Requires list on appliedclusterresourcequotas in quota.openshift.io (namespace-scoped Role – no ClusterRole needed).

Volume filesystem usage

The volume collector reports disk usage for PVC-backed volumes. It is designed to run as a sidecar on pods that mount PVCs, since kubelet_volume_stats_* metrics are inaccessible on managed clusters (no ClusterRole for nodes/metrics) and RWO volumes cannot be mounted cross-namespace.

Configure via VOLUMES_CONFIG:

VOLUMES_CONFIG: |
  volumes:
    - {name: my-postgres, path: /data}

Metrics (collected every 5 minutes):

  • hummingbird_volume_usage_bytes{name} – used bytes
  • hummingbird_volume_total_bytes{name} – total capacity
  • hummingbird_volume_available_bytes{name} – available bytes

Deploy with enabled: [volume] and a non-default METRICS_PORT (e.g. 9091) to avoid port conflicts when running as a sidecar alongside other containers. See the hummingbird-status StatefulSet in the infrastructure repo for an example.

GitLab CI/CD schedules

The gitlab collector monitors the health of pipeline schedules across all projects in configured GitLab namespaces. It queries the GitLab GraphQL API once per hour, paginating through projects to fetch each schedule’s last pipeline status.

Configure via GITLAB_CONFIG:

GITLAB_CONFIG: |
  namespaces:
    - redhat/hummingbird

The GITLAB_TOKEN environment variable must contain a token with read_api scope that has access to the monitored namespaces. Keep it in an OCP Secret (not the ConfigMap).

Metric (collected every hour):

  • hummingbird_gitlab_schedule_status{instance,project,active,description,cron} – Enum reporting the last pipeline status of each CI/CD schedule

States: created, waiting_for_resource, preparing, pending, running, success, failed, canceled, skipped, manual, scheduled, unknown. Schedules with no last pipeline report unknown.

Stale series are removed automatically when schedules disappear.

GitLab webhooks

The gitlab_webhooks collector monitors the alert status of the webhooks configured on specific GitLab projects. GitLab disables webhooks that fail repeatedly (for example after a signing-token rotation), which silently stops all event-driven automation behind them; this collector makes that state visible. It queries the GitLab REST API once every 2 minutes.

Configure via GITLAB_CONFIG:

GITLAB_CONFIG: |
  webhooks:
    - redhat/hummingbird/tools

The GITLAB_TOKEN environment variable must contain a token with read_api scope that has access to the monitored projects. Keep it in an OCP Secret (not the ConfigMap).

Metric (collected every 2 minutes):

  • hummingbird_gitlab_webhook_alert_status{instance,project,url} – Enum reporting the alert status of each project webhook

States: executable, disabled, temporarily_disabled, unknown. An unrecognized status reported by GitLab maps to unknown. Stale series are removed automatically when hooks disappear from the configured projects.

The infrastructure repo consumes this metric with the GitLabWebhookDisabled Mimir alert (severity critical) and a Grafana webhook-status dashboard.

AWS costs

hummingbird_aws_cost{group,type,key} is yesterday’s NetAmortizedCost. Queries run at process start and every 24 hours.

Why NetAmortizedCost

AWS Cost Explorer offers several cost metrics. This collector uses NetAmortizedCost — the amortized cost after enterprise discounts.

This account is part of a consolidated billing organization with Compute Savings Plans shared across accounts. The cost metrics differ due to two independent effects:

  1. Enterprise discounts (affects all services): This account receives EDP (Enterprise Discount Program) and Private Rate Card discounts. In UnblendedCost and AmortizedCost these appear as separate negative line items; in the Net variants (NetAmortizedCost, NetUnblendedCost) they are folded into each line item directly.
  2. Savings Plans amortization (affects SP-covered services): In UnblendedCost, SP costs appear in two record types: SavingsPlanRecurringFee carries the commitment cost, while SavingsPlanCoveredUsage shows the on-demand equivalent with an equal SavingsPlanNegation that cancels it — net zero. In AmortizedCost, this flips: the recurring fee goes to zero and the covered usage is valued at the SP effective cost instead. If only this account’s own SPs covered its usage, the amortized covered usage would equal the unblended recurring fee. The gap between them reveals how much org-wide SPs from other accounts contribute. To isolate the SP effect from enterprise discounts, compare Unblended against Amortized (SP amortization only) or Unblended against NetUnblended (discounts only).

NetAmortizedCost shows the true cost of this account’s resource consumption to the organization, including compute covered by Savings Plans owned by other accounts. For non-compute services, amortized and unblended costs are identical.

Run service-metrics/scripts/cost_metric_comparison.py [YYYY-MM] to see how these effects compose. It queries Cost Explorer for a single month, groups by record type, and shows all five metrics side by side — making visible how enterprise discounts fold into Net variants, how SavingsPlanNegation cancels covered usage in unblended, and how amortization redistributes SP costs. A summary section breaks down what this account actually pays versus what other accounts’ Savings Plans cover.

Groupings

group=all is the shared account by app-code and SERVICE. group=hummingbird filters app-code=RPRM-001 and groups by SERVICE, OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION.

Untagged Cost Explorer keys such as app-code$ become key=untagged. Well-known SERVICE names are shortened (S3, EC2). Dual grouping uses S3|PutObject.

The SAM template creates an IAM user with ce:GetCostAndUsage only. Access keys are created after deploy and stored in Vault. Deployment lives in the infrastructure repo.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.35 - VEX Checker

A CLI tool that works with the Red Hat CSAF VEX feed and the Hummingbird catalog API to check CVE statuses and track coverage across Jira, VEX advisories, and container image scan results.

Subcommands

Subcommand Purpose
check Look up a single CVE and show Hummingbird product statuses
reconcile Cross-reference Jira CVE tickets with the VEX feed
catalog Cross-reference catalog API CVEs with Jira tickets

Prerequisites

  • Python 3.11+
  • Internet access to security.access.redhat.com (no VPN required) for check/reconcile; to api-hummingbird.hummingbird-project.io (no VPN required) for catalog
  • JIRA_EMAIL Atlassian account email — required for reconcile and catalog
  • JIRA_TOKEN Jira API token — required for reconcile and catalog

check — Single CVE lookup

Fetches the CSAF VEX document for a given CVE and prints the status of all Hummingbird products, filtering out all other RHEL/OpenShift/etc. entries.

check: usage

./check_vex.py check CVE-YYYY-NNNNN [--json]

check: options

Option Description
CVE-ID CVE to look up
--json, -j Output results as JSON

check: examples

$ ./check_vex.py check CVE-2014-8090

CVE-2014-8090 — CVE-2014-8090 ruby: REXML billion laughs attack (Moderate)
  hummingbird-1:ruby.src      known_affected  none_available
  hummingbird-1:ruby3.3.src   known_affected  none_available
  hummingbird-1:ruby3.4.src   known_affected  none_available
  hummingbird-1:ruby4.0.src   known_affected  none_available
$ ./check_vex.py check CVE-2014-8090 --json
{
  "cve": "CVE-2014-8090",
  "title": "...",
  "severity": "Moderate",
  "hummingbird": [
    { "product_id": "hummingbird-1:ruby.src", "status": "known_affected", "remediation": "none_available" },
    ...
  ]
}

check: exit codes

Code Meaning
0 No VEX document found, no Hummingbird products listed, or no known_affected status
1 One or more Hummingbird products are known_affected
2 Invalid CVE ID format

reconcile — Jira/VEX sync check

Queries the Jira HUM project (Security component) for all CVE tickets, then checks each CVE in the VEX feed. Reports two types of mismatches:

  • Open in Jira but not known_affected in VEX — VEX says the issue is resolved but the Jira ticket is still open.
  • Closed in Jira but known_affected in VEX — the Jira ticket was closed but Red Hat’s advisory still marks Hummingbird as affected.

Continuous closed-ticket VEX reconciliation (Done-Errata → fixed, Not a Bug → known_not_affected or package_not_listed, scoped to the ticket’s package, with awaiting-vex work queue and dashboard persistence) lives in hummingbird-cve-analysis / the dashboard Closed tab (HUM-5843). reconcile remains the one-shot CLI audit.

Jira tickets are considered closed when their status is one of: Done, Closed, Won't Fix, Not a Bug.

VEX documents are fetched in parallel (--workers, default 20) so the command is fast even with hundreds of CVE tickets.

reconcile: usage

export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py reconcile [--workers N] [--json]

Credentials are read exclusively from environment variables to avoid exposing them in process listings.

reconcile: options

Option Description
--jira-url Jira base URL (default: https://redhat.atlassian.net)
--workers Parallel VEX fetch workers (default: 20)
--json, -j Output results as JSON

reconcile: example

$ ./check_vex.py reconcile

Fetching Jira tickets... 142 ticket(s)
Fetching VEX statuses for 89 unique CVE(s)...
  89/89
2 mismatch(es) found:

  HUM-1234  CVE-2025-1234  closed → known_affected  (VEX still open)
    Jira: https://redhat.atlassian.net/browse/HUM-1234
    VEX:  https://security.access.redhat.com/data/csaf/v2/vex-feed/2025/cve-2025-1234.json

  HUM-5678  CVE-2024-5678  open → fixed  (VEX not affected)
    Jira: https://redhat.atlassian.net/browse/HUM-5678
    VEX:  https://security.access.redhat.com/data/csaf/v2/vex-feed/2024/cve-2024-5678.json

reconcile: exit codes

Code Meaning
0 No mismatches found
1 One or more mismatches detected
2 Missing JIRA_EMAIL or JIRA_TOKEN

catalog — Catalog API / Jira coverage check

Fetches all CVEs currently detected in Hummingbird container images (via Grype scan results stored in the catalog API) and cross-references them with Jira HUM Security tickets. Reports CVEs that have no corresponding Jira ticket.

CVEs with a closed Jira ticket are reported separately as expected propagation delay — the typical flow is CVE fix → RPM → image rebuild → Grype DB update, and Jira tickets are closed early while the catalog API reflects the latest Grype scan (updated ~once daily).

Non-CVE vulnerability identifiers (e.g. GHSA-*) from Grype are filtered out since they are not tracked in Jira.

catalog: usage

export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py catalog [--api-url URL] [--workers N] [--json]

Each CVE line shows severity, age since first detection, affected images, and for tracked CVEs the Jira ticket key. Components with known fix versions are shown in brackets when available.

Closed Jira tickets are cross-referenced with the VEX feed to distinguish between different root causes:

  • Done-Errata, VEX still affected – fix shipped in RPM but not yet propagated to container images; needs rebuild or lockfile refresh.
  • Done-Errata, VEX resolved – fix fully propagated, Grype DB just needs its daily update to stop flagging.
  • Won’t Do / Not a Bug, VEX resolved – VEX already marks CVE as not affected; Grype DB delay, will self-resolve.
  • Won’t Do / Not a Bug, VEX still affected – genuine known issue that will persist in scans (accepted risk, upstream won’t fix, etc.).

catalog: options

Option Description
--api-url Catalog API base URL (default: https://api-hummingbird.hummingbird-project.io/v1)
--jira-url Jira base URL (default: https://redhat.atlassian.net)
--workers Parallel VEX fetch workers (default: 20)
--json, -j Output results as JSON

catalog: example

$ ./check_vex.py catalog

Catalog: 72 CVEs across 29 images (scanned 2026-04-28T10:45:38Z)
Jira: 748 tickets (437 unique CVEs)

11 untracked CVE(s) (no Jira ticket):

  CVE-2008-2662  High     8d  ruby (12 tags)  [ruby3.3@3.3.10, +36 more]
  CVE-2026-2950  Medium   8d  aspnet-runtime, ... (30 tags)  [lodash@4.17.21]
  ...

45 tracked CVE(s) open in Jira:

  CVE-2025-68114  High  8d  php (3 tags)  [capstone@5.0.6]  HUM-925
  ...

8 closed in Jira (Done-Errata) but VEX still affected -- verify fix propagated:

  CVE-2026-27143  Critical  8d  caddy, go-fdo-client, ... (10 tags)  HUM-969

4 closed in Jira (Done-Errata), VEX resolved -- Grype DB delay:

  CVE-2025-61732  High  8d  go, xcaddy (6 tags)  HUM-1119

3 closed in Jira (Won't Do / Not a Bug), VEX resolved -- Grype DB delay:

  CVE-2026-27140  High  8d  caddy, go, ... (10 tags)  HUM-967

1 closed in Jira (Won't Do / Not a Bug), VEX still affected -- genuine:

  CVE-2025-12781  Medium  8d  postgresql, python, ... (17 tags)  HUM-1387

catalog: exit codes

Code Meaning
0 All catalog CVEs are tracked (have a Jira ticket)
1 One or more untracked CVEs found
2 Missing JIRA_EMAIL or JIRA_TOKEN

Development

# Run tests
cd vex-checker && python3 -m unittest discover tests -v

# Run linter (from repo root)
ruff check vex-checker

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.36 - Hummingbird CVE agent

CLI and SQS worker that investigate open HUM Security tickets labeled cve-needs-attention or upstream-fix-available, using AI where deterministic analysis could not resolve the CVE. Fixes are handed to workqueue-service as mr work items.

System overview

flowchart TB

  TICKET["run HUM-XXXX"] --> COL
  SNS["Successful cve_analysis --resolve<br/>SNS → SQS wake-up"] --> SWEEP["Worker: fresh Jira sweep"]
  CLI["sweep [--limit N] [--workers N]"] --> SEARCH
  SWEEP --> SEARCH["Open HUM Security tickets<br/>in-scope labels · no cve-next-release<br/>not embargoed · no prev analysis"]
  SEARCH --> PREFLIGHT["Start and preflight all worker backends<br/>before ticket collection · skip if no tickets"]
  PREFLIGHT --> THREAD

  subgraph WORKER["Trusted worker · network and credentials"]
      direction TB
      THREAD["Sweep worker thread"] --> COL
      COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Deterministic candidate?<br/>affected + backport + fix label/reference<br/>not stale"}
      CAND -->|yes| ID{"Exact current version<br/>and SRPM NVR identified?"}
      ID -->|no| HUMAN["needs_human · no actions"]
      CAND -->|"no / stale"| AIINV["AI investigation<br/>bounded read-only tools"]
      ID -->|yes| FETCH
      AIINV -->|"source tools / needs_change + fix"| FETCH["Fetch pinned inputs<br/>hardened no-checkout Git transport<br/>checksummed sources · upstream patches"]
      AIINV -->|verdict| RESULT["InvestigationResult"]
      HUMAN --> RESULT
      EVIDENCE["Bounded evidence + verified file_actions"] --> RESULT
      RESULT --> OUT["Write result JSON<br/>stdout / --output before remote writes"]
      OUT --> SUBMIT{"--submit-mr<br/>and prepared actions?"}
       SUBMIT -->|yes| MR["Submit draft mr work item<br/>to workqueue-service"]
      SUBMIT -->|no| POST
      MR --> POST{"--post-to-jira?"}
       POST -->|yes| COMMENT["Post Jira comment → verdict label<br/>→ analysis attachment"]
  end

  subgraph SANDBOX["Offline execution · no network or credentials"]
      direction TB
      BACKEND["WorkerSandbox owns backend + package tools<br/>one per worker thread"] --> FILES["per-ticket WORK workspace<br/>reset state/files between tickets"]
      FILES --> PREP["RPM %prep<br/>existing downstream patches"]
      PREP --> INSPECT["Source inspection<br/>+ disposable candidate checks"]
       INSPECT -->|"backport path"| STAGE["Patch staging + spec edits"]
      STAGE --> REPLAY["Final-spec replay from clean inputs"]
  end
  THREAD -->|owns| BACKEND
  FETCH -->|"transfer each input once"| FILES
  INSPECT -->|"AI tools: bounded evidence"| AIINV
  REPLAY --> EVIDENCE
  MR --> REVIEW["Konflux build + human review"]

  classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
  classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
  class THREAD,COL,CAND,ID,HUMAN,AIINV,FETCH,EVIDENCE,RESULT,OUT,SUBMIT,POST,COMMENT agent
  class BACKEND,FILES,PREP,INSPECT,STAGE,REPLAY sandbox

The agent runs as a CLI and is stateless: run handles one ticket, while sweep pages through all eligible tickets and investigates them concurrently. The deployed worker consumes sweep requests from SQS and invokes that same bounded sweep; the event is only a wake-up signal, not a ticket snapshot. CLI writes are opt-in; the deployed worker enables both Jira posting and MR submission (see Writes).

Critical evidence network, download, or parse failures abort the investigation; their local diagnostic report leaves the ticket eligible for a later retry. Unsupported or ambiguous evidence becomes needs_human. The ticket-level state machine lives in the CVE Lifecycle & State Vocabulary.

Tl;Dr

  • Scoped to CVE Security tickets - investigates open HUM Security tickets with either cve-needs-attention or upstream-fix-available; these are the deterministic-analysis residual and upstream-fix populations respectively. Sweeps skip cve-next-release, embargoed tickets, and tickets already carrying their <ticket>-cve-agent-analysis.json attachment.
  • Backporting - verifies upstream patches against pinned package source and prepares RPM changes; see Backporting.
  • AI in one place - every LLM call goes through ai.AgentAiClient, which uses the shared hummingbird_ai.models library directly, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own; see What the model sees for the exact context. The model can use bounded read-only tools to search the SBOM, inspect package files, inspect checksum-pinned package sources, and look up exact npm release metadata. Before the first model turn, the agent prefetches predictable evidence through those same tools: ticket CVE advisories, matching SBOM components, and the package file listing and spec. Evidence is reused only within one sweep, keyed by its exact source identity and bounded counts: package artifacts, advisories, identical package-file reads and searches, and deterministic patch checks. Transient failures, corrupt artifacts, and final backport verification are never cached. Transient HTTP and transport failures are retried by one sweep-scoped policy: at most three attempts and 30 seconds of cumulative wait, honoring Retry-After. Redirects are still refused. A retryable external read failure reuses the worker sandbox only when it resets cleanly; a failed reset or any local corruption discards it. Each ticket has at most five tool-enabled turns plus one final-answer turn, whose request carries no tool definitions and explicitly asks for a JSON verdict using the available evidence. Invalid final replies receive a corrective request containing the result schema within the existing bounded retry budget. Exhausted retries fail the ticket without posting a result; no verdict is inferred from malformed output. Diagnostics log response length, finish reason, each collection step’s duration, and the model call duration, not response contents. Retries can make multiple model requests within a turn; six turns is not a six-request cap. SBOM searches return at most 20 components with binary-presence evidence (present, source_only, or unknown) from the shared SBOM classifier; a name/version match or source-only hit does not prove the component ships. When a matching SBOM result is unknown, the agent can scan published RPMs for that component. SBOM files are limited to 256 MiB. SBOM JSON is parsed incrementally; the sweep-scoped LRU retains at most eight full documents, so its entry limit is not a byte or process memory limit.
  • Structured results - every investigation produces a validated result contract (there is no schema-version field). The result status is a Verdict from the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent and cve_analysis speak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine. Every result also carries the ticket’s collection records: each evidence step’s outcome (success, empty, partial, failure, or limitation), its retryability, and its provenance. Initial collection records can appear in the model’s input facts; the caller owns the final collection records, including later tool outcomes.

Prerequisites

  • Python 3.11+
  • Podman with a running Linux VM for local source preparation, or Kubernetes sandbox access in deployment.
  • A sandbox image with RPM build tools, Python RPM bindings, python3-specfile, autoconf, Git, GNU patch, and the RPM macros/helpers required by the package.
  • GITHUB_TOKEN for GitHub backport lookup when unauthenticated API requests hit GitHub’s rate limit.
  • Jira API credentials (JIRA_TOKEN, and JIRA_USER where basic auth applies)
  • Vertex AI access for the AI path: GOOGLE_CLOUD_PROJECT, or GOOGLE_API_KEY for the direct Gemini API. The deterministic fast path needs neither.
  • For --submit-mr only: WORKQUEUE_SERVICE_URL and either WORKQUEUE_SERVICE_TOKEN_FILE or WORKQUEUE_SERVICE_TOKEN, with submit permission on the mr work type. The deployed worker uses a projected token file; local development can use the environment token described in Writes: Jira comment and MR submission.

Installation

make hummingbird-cve-agent/setup

The target installs hummingbird-agent, hummingbird-ai, and hummingbird-cve-analysis editable alongside this package. Equivalent by hand:

cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"

For the environment, copy hummingbird-cve-agent/.env.example to .env and source it for a quick way to set up all necessary variables. For local CA setup, activate the Python environment used to run the CLI before sourcing .env, so python -m certifi resolves that environment’s public CA bundle. If the workqueue endpoint uses an internal CA, export REQUESTS_CA_BUNDLE in the same shell as --submit-mr. Its PEM file must contain both the public roots (for Google OAuth) and the internal root (for the workqueue endpoint).

Configuration

Operational settings come from the YAML config file; secrets and model credentials come from the environment.

Variable Purpose
HUMMINGBIRD_CVE_AGENT_CONFIG Path to a YAML config file (optional)
HUMMINGBIRD_CVE_AGENT_WORKERS Worker concurrency default (2; must be positive; --workers overrides)
JIRA_TOKEN Jira API token
JIRA_USER Jira basic-auth user (optional)
SENTRY_DSN Sentry error reporting for the deployed worker
GITHUB_TOKEN Read-only GitHub token for PR commit and advisory lookups (optional)
GITLAB_TOKEN Read-only GitLab.com token for MR commit lookups (optional)
GOOGLE_API_KEY Gemini API key (direct API mode)
GOOGLE_CLOUD_PROJECT GCP project for Vertex AI mode
WORKQUEUE_SERVICE_URL Base URL of a running workqueue-service instance (for --submit-mr)
WORKQUEUE_SERVICE_TOKEN_FILE Path to projected bearer token (preferred; re-read on each submission)
WORKQUEUE_SERVICE_TOKEN Bearer token fallback for local --submit-mr use
SQS_QUEUE_URL Dedicated queue consumed by the deployed worker command

YAML settings:

Key Default Purpose
jira_url Red Hat Jira Jira base URL
sandbox_type podman Sandbox backend; deployments use k8s
sandbox Published, digest-pinned CI image Override image locally to use the CVE RPM-preparation image
model claude-sonnet-4-6 Model name passed to build_model
model_regions Gemini and Claude region prefixes Model-name prefix to Vertex region

Neither forge token grants write access: they only authenticate the read-only API calls that resolve a pull/merge request to its commits, and are needed only for private or rate-limited repositories. Resolving one pull request costs two API requests and unauthenticated GitHub access allows 60 per hour, so set GITHUB_TOKEN before looking up many backports. The sandbox never holds a push token.

Usage

hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-to-jira
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-to-jira --submit-mr

This gathers facts for the ticket, runs the investigation, and prints the result as JSON or persists it to --output. See Backporting for how upstream fixes become package changes and what happens when preparation fails.

CLI logs include ticket and package context, e.g. - INFO (HUM-1234 openssl): collected facts for HUM-1234; the package is - until known, and unscoped logs show - -.

sweep defaults to all eligible tickets and two concurrent workers. --limit caps attempted investigations, including failed attempts; analyzed, embargoed, and out-of-scope search results do not consume the limit. A sweep searches once and then investigates for a long time, so a ticket can also lose scope after it was selected, by human re-triage to cve-next-release or by an overlapping sweep closing it. That is reported as out of scope, not as a failure: it does not fail the run or change the exit code. Standard output shows counts, verdicts, token usage, estimated cost, and compact per-ticket summaries. --output FILE also saves the complete JSON report. The command exits nonzero if any ticket fails, while continuing after handled ticket-level failures. A source-initialization TypeError propagates out of investigate, so that ticket produces no investigation-result JSON and makes no Jira/MR writes. The sweep includes TypeError in its existing ticket-error handling, records the failed ticket in its report, and continues other tickets normally; this is not a catch-all for unexpected exceptions. Without --post-to-jira, the sweep does not add the attachment used to mark a ticket as investigated, so a later sweep can retry it. Critical collection failures never post a recommendation or analysis attachment, even when the write flags are enabled. The CLI and sweep retain bounded failure records locally; transient failures are marked retryable and are not cached.

After finding eligible tickets, the sweep starts and preflights all min(workers, eligible tickets) execution backends before any ticket collection, investigation, or model calls. No eligible tickets means no backends. A thread pool runs that many worker loops consuming a ticket queue, each with its own backend; it does not create a backend per ticket or package. Preflight failure aborts the sweep before any ticket starts or remote writes occur and cleans up started backends. Backends are cleaned up when the sweep exits; a handled ticket failure affects only that ticket, not other workers. Transport/collection failures discard its backend so timed-out child processes cannot reach the next ticket; the next ticket starts a fresh backend. Other handled failures reset ticket state and workspace for reuse.

Backporting

The agent separates deciding that a package needs a fix from preparing the RPM changes. AI may judge affectedness or identify an upstream commit; the backporting code fetches and verifies the upstream patch, then prepares RPM changes. It does not rewrite or generate patch content (yet, see Planned work).

flowchart TD
    C{"Affected + backport + fix reference<br/>+ upstream-fix-available?"} -->|Yes| S{"Analysis stale?"}
    C -->|No| I["AI investigates"]
    S -->|Yes| I
    S -->|No| ID{"Exact current version<br/>and SRPM NVR identified?"}
    ID -->|No| H["needs_human · no file_actions"]
    ID -->|Yes| U["Resolve fix and select release patches"]
    I -->|Needs change + fix| U
    I -->|Otherwise| R["Return investigation result"]
    U --> V["Per-ticket workspace on worker-owned backend<br/>RPM prep → disposable candidate checks"]
    V --> W["Stage patches · edit spec/metadata<br/>replay final spec from clean inputs"]
    V -->|Unsupported / rejected| F["Deterministic: needs_human<br/>AI: retain finding/confidence<br/>both: no file_actions"]
    W -->|Unsupported / rejected| F
    V -->|Critical collection failure| A["Abort · local diagnostics<br/>no remote writes"]
    W -->|Verified| R
    F --> R
    H --> R
    R --> J["Agent's result processing"]

1. Identify and select the upstream fix

The fast path skips AI only when the existing CVE analysis says the package is affected, names a backport as the next step, the ticket has the upstream-fix-available label, and the analysis still matches the package version collected now and has not already reached a recorded fixed version on its release line. Stale analysis goes to AI. A candidate that is not stale but does not explicitly identify both the current version and exact SRPM NVR returns needs_human, not an AI fallback. A needs_change result with a usable fix reference enters the same preparation path; otherwise, the agent returns the investigation result only.

Commit URLs are fetched directly. A merged PR URL is first resolved to its commits in application order. A commit is accepted only when it is reachable from the package’s declared upstream repository’s default branch or from its exact release branch named in metadata/<package>.json. Commits from another repository, including forks, are refused even if reachable. Open and closed-but-unmerged PRs are rejected before their commits are fetched. An unknown release branch stays unverified. A failed fetch is retried instead: it says nothing about the commit, so it is never reported as unmerged. A squash-merged change has no landed SHA in common with the branch commit, so such a URL stays unverified rather than being assumed merged. Every patch fetched from a commit URL has its full SHA checked against the cited commit before it can reach package applicability checks or be saved for backport. The merged-backport PR diff found by title is instead matched on its merged state and release-line base. Branch-tagged patches must target the release line of the shipped package; a mismatch requires human review. Mixed tagged and untagged commits are rejected as ambiguous because untagged commits might be prerequisites rather than alternative backports. After selecting the line, only identical diff bodies are removed, retaining hunk positions, context, and file metadata in the comparison so distinct fixes are not silently discarded. The series has no commit-count limit: discovery work is bounded by an aggregate 10 MiB patch budget (and 2 MiB per patch), because patch count alone is not a correctness limit.

2. Prepare and verify against package source

Each sweep worker thread owns one WorkerSandbox, which owns the execution backend and package tools: a Podman container locally or an ephemeral Kubernetes pod in deployment. Its .ticket(package) scope resets per-ticket state and files in the WORK workspace. Sequential tickets reuse the physical container/pod, but each gets fresh Preparation state, checksum pins, and candidates; none carry over to the next ticket, except the sweep-scoped evidence cache. This avoids startup per investigation without sharing ticket state. .open() lazily imports the package into the existing worker backend. Standalone run uses its own WorkerSandbox and may avoid backend startup entirely for an SBOM-only investigation; any backend it starts is cleaned up when the run ends.

Within a ticket, the offline package workspace is shared by file/source inspection, disposable candidate checks, patch staging, spec edits, and final replay. Preparation performs RPM operations on that backend; it does not own the backend or its lifecycle. Verified base trees and the disposable candidate copy are released before the edited-spec replay, so a ticket never holds both tree sets at once. RPM macro expansion and rpmbuild -bp --nodeps run only inside that sandbox. RPM owns architecture conditions, source selection, macros, patch ordering, archive extraction, and shell commands.

The worker has no package checkout. A temporary, hardened no-checkout Git object transport pins one repository commit and exports only rpms/<package>/, metadata/<package>.json and mock/dist-git-client.ini from that same commit. The worker brokers source/lookaside downloads, validates checksums, and fetches upstream patches; the selected inputs are validated and then transferred once as a single archive stream, preserving executable package scripts. Every RPM preparation uses fresh sandbox-local copies of those inputs, so a script changing its sources cannot contaminate the next architecture or final replay. Downloads are limited to 128 MiB each, with a shared 256 MiB input budget. The execution sandbox has no network or worker credentials, so missing inputs, macros, signatures, or preparation helpers block preparation instead of triggering dependency installation or guessed behavior.

The worker retains immutable metadata, checksum pins, and lookaside routing facts from that Git snapshot. It never uses sandbox-modified copies to choose download destinations or checksums; these acquisition facts are not a second working tree.

Inspection reads and searches the real prepared tree, cached per architecture. Reads reject escaping paths and symlinks and are limited to 8 MiB per file; searches are limited to 48 MiB and 50,000 filesystem entries. Commands have timeouts, bounded diagnostics, and a 256 MiB individual-file limit. Podman keeps its original shared startup behavior and writable filesystem. CPU, memory, storage, process-count, and container-lifetime policy belong to deployment infrastructure; the CVE agent does not impose them on shared Podman users. Infrastructure enforcement is deferred, so these application-level checks do not provide a whole-sandbox resource bound for local runs.

Candidates are checked on disposable copies using GNU patch with fuzz zero. Before each candidate is applied, a reverse dry-run on the clean copy separates source that already carries the fix from source that needs a rebased patch: the first blocks backporting as CVE already applied, the second as a patch that does not apply. Mismatched context is rejected without retrying with greater fuzz. Exact-context hunks that land at a line offset still pass, but they are recorded and reported to the MR reviewer. Original source paths take precedence. When they are missing, deterministic adaptation can remove a nested monorepo prefix, add a downstream prefix such as src/ or vendor/<package>/, or replace one shared directory prefix with another. It preserves relative paths and requires a unique mapping supported by every initially existing file touched by the ordered series. Added files must not overwrite existing destinations. A bounded search ignores symlink directories; ambiguous or incomplete searches are rejected, and addition-only patches do not guess a target directory. Only diff headers change, never source hunks or commit messages; individual file renames and source-level rewrites are not inferred. Automatic backporting must use the same adapted patch on every applicable supported target (x86_64, aarch64, and s390x). Source tools accept an explicit arch; when multiple targets exist, they require it rather than guessing which published binary the model is investigating. A selected architecture remains in effect for later source-tool calls on the same ticket unless explicitly changed.

The installed specfile library performs structural edits. The edited spec is then prepared again from clean inputs, using RPM’s own patch backend and options. RPM’s default patch fuzz is explicitly set to zero for every preparation. Every candidate-touched file must match its verified final content, including deletions; a later source overlay cannot silently erase the fix. If necessary, explicit architecture-guarded patch applications are appended and verified again.

The CVE-agent sandbox image supplies RPM configuration, macro packages, and prep helpers. Preparation does not establish buildroot equivalence, native execution on another architecture, compilation, runtime behavior, or that a fix resolves the CVE. Konflux builds and human review remain required.

3. Review the generated changes

Only verified patch text is staged. One file is created for each selected upstream patch; the spec and package metadata are updated as follows:

Output Change
rpms/<package>/<NNNNN>-<cve>-<slug>.patch New numbered patch file; the slug uses up to five words from the ticket summary, or the short commit ID if no usable summary exists.
Package .spec Adds patch declarations and arranges their application. Bumps Release by a trailing micro version (4.1 → 4.2), including %{gcc_release}.1 → %{gcc_release}.2 and values from a %global macro, so local changes stay out of Fedora’s integer namespace; date-shaped releases require human review.
metadata/<package>.json Marks imported packages modified and adds the ticket/CVE to modification_reason. Independent packages remain independent; unrelated metadata is preserved.
Why preparation stopped Result
Supported %prep, path, or patch checks reject the fast-path fix needs_human, no file_actions.
AI investigation says needs_change, but automatic preparation fails Keep the finding and confidence; add the preparation failure to the summary; no file_actions.
Critical source fetch, checksum, or other evidence collection fails Abort with a local diagnostic report; retry behavior follows the failure classification.

4. Hand off the prepared changes

Use --submit-mr to hand off prepared actions; see Writes for submission and review details.

Planned work

HUM-8512 tracks one bounded AI attempt to adapt a patch after a hunk conflict. The planned flow replays the existing package patch stack, then verifies the adapted candidate from a clean tree with zero fuzz. It also proposes regenerating the candidate as a clean downstream diff and verifying that diff before staging. Source-level AI adaptation and patch regeneration are not implemented today; deterministic directory-prefix mapping is tried automatically during source checks and backport verification.

Writes: Jira comment and MR submission

Both writes are off by default in CLI use; each is enabled solely by its own flag, so a bare run never touches Jira or GitLab. The two flags are independent — a comment can be posted without submitting an MR and vice versa — and a failed write exits non-zero. The result writer runs first, before either remote write. When both are enabled, the MR is submitted before the Jira comment and analysis attachment, so a failed MR submission leaves the ticket eligible for a later sweep.

  • --post-to-jira posts a deterministic RECOMMENDATION: ... comment rendered from the validated InvestigationResult — no AI call at comment time — through cve_analysis’s jira_client, which handles ADF formatting and duplicate suppression, so re-running a command is idempotent. It also synchronizes one cve-agent-* Jira label to the current verdict, replacing any prior CVE Agent verdict label and preserving unrelated labels. A label update failure stops the write before the analysis attachment is uploaded. The comment carries the verdict, summary, cited evidence, and the next review step when human review is needed, including when a necessary package fix has no automated backport. The full result JSON is attached as <TICKET>-cve-agent-analysis.json with the supporting evidence, file actions, commit message, and a token_cost block (input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens, unpriced_calls, estimated_cost_usd) measured from every model response of the investigation. The cost is a subtotal of the calls with known pricing; unpriced_calls counts the rest, so a non-zero count makes estimated_cost_usd a lower bound. A duplicate comment does not prevent the agent from synchronizing the label or uploading a missing analysis attachment. Existing tickets are not backfilled; a later --post-to-jira run synchronizes their verdict label.

    Verdict Jira label
    needs_change cve-agent-needs-change
    not_affected cve-agent-not-affected
    already_fixed cve-agent-already-fixed
    no_upstream_fix cve-agent-no-upstream-fix
    needs_human cve-agent-needs-human
    error cve-agent-error

    As tickets are currently evaluated only once, no_upstream_fix therefore would need a TTL (for example 24 hours) after which the ticket is investigated again, since a fix may have landed since the verdict.

  • --submit-mr submits a draft MR work item to a running workqueue-service instance (work items default to auto_merge=false). It is a no-op unless the result carries a prepared backport (file_actions + commit_message). Submission POSTs to WORKQUEUE_SERVICE_URL with the current token from WORKQUEUE_SERVICE_TOKEN_FILE when configured, otherwise WORKQUEUE_SERVICE_TOKEN; the file is re-read for each submission so projected token rotation is honored. The agent is headless, so there is no interactive login fallback. The description follows the rpms CVE workflow’s Scope/Fix/Verification structure: the Jira tracker, package, CVEs, and built NVR; the summary, selected upstream fixes, and investigation steps; and the checks performed versus those left to Konflux and human review. Currently, workqueue-service only applies this description when updating an existing MR.

Automated sweep deployment

After a successful cve_analysis --resolve cycle, the analysis CronJob publishes a small SNS event requesting a sweep. A dedicated SQS queue filters for that event, and one long-running agent Deployment runs the existing Jira search and investigates eligible tickets. The message is a wake-up signal; it contains no ticket data. Every request starts a fresh Jira search. The deployed child sweep uses two workers by default; HUMMINGBIRD_CVE_AGENT_WORKERS sets that default, and the worker’s --workers option overrides it. The value must be positive. The sweep attempts at most forty tickets and enables both --post-to-jira and --submit-mr.

The consumer renews message visibility while the sweep runs, enforces a one-hour deadline, and acknowledges only a completed sweep report. A completed pass is acknowledged even if some individual tickets failed; those tickets stay eligible for a later sweep. Failed or interrupted sweeps are retried by SQS. Repeated requests are handled sequentially by the single worker replica.

Before rollout, infrastructure must budget for the configured number of concurrent sandbox pods plus the controller pod, enforce per-pod resource limits, and set sandbox deadlines to cover the full one-hour sweep (the shared 1,800-second default is insufficient). Reap orphan sandbox pods before deployment. These are infrastructure requirements; this change adds no resource-policy flags or shared backend changes.

The analysis publisher has permission to publish to its environment’s SNS topic; the agent has permission to consume only its dedicated SQS queue. Neither needs cluster-wide Kubernetes API access; the agent needs pod management only in the sandbox namespace. The worker uses a projected short-lived token for the workqueue-service audience, where its identity can only submit mr work items.

Both environment switches start disabled in infrastructure/vars.d/hummingbird-cve-agent.yml. Deploy the SAM stack and provision each environment’s publisher and worker IAM keys in Vault before enabling its switch:

Variable Purpose
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_STAGING Enable staging event publishing and the worker replica.
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_PRODUCTION Enable production event publishing and the worker replica.

Eval harness

eval scores investigations against completed human /cve sessions. The eval set, eval-tickets.txt, holds Jira ticket keys and nothing else — ticket content is internal data and is never committed — so facts are fetched live and running the eval needs Jira access.

hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json

Read-only: every Jira request is a GET (issue, comments, changelog); test_evaluate asserts it.

A resolved ticket gives the answer away — fixed_in_build set, labels moved to awaiting-vex, status Closed, newest analysis comment post-fix — so each ticket is rewound through its changelog, dropping later comments.

The rewind cutoff is the last scope-label handoff plus five minutes, allowing the assessment comment from the same triage burst to remain visible. Resolution time is the fallback when no handoff was recorded or it occurred after resolution; the fix-build timestamp is not the cutoff. fixed_in_build is also blanked so the recorded answer cannot leak through that field.

What is scored

Two axes, because one is not enough:

  • Disposition — where each side says the ticket ends up. The verdict is mapped to its eventual disposition, not through states.VERDICT_OUTCOME: that shared table answers “what does the agent do next”, so needs_change is In Progress — but a ticket the human backported still ends Done-Errata, and scoring against the immediate state would mark a correct needs_change as a miss and reward a wrong already_fixed.
  • Route — which branch the orchestrator would take, since already_fixed (Fixed in Build) and needs_change (MR creation) share the Done-Errata ending and are indistinguishable on the first axis. The human’s branch is derived from Pulp upload times: a fix build published after the handoff means the package had to change (needs_change), an earlier one means the shipped package already carried the fix (already_fixed); a Not a Bug closure is not_affected. A build Pulp no longer lists is reported as n/a and excluded from the route denominator rather than guessed.

Verdicts that reach no conclusion (needs_human, error) and conclusions no verdict can produce (unknown resolutions) never match. Package EOL Won't Do closures are deterministic cve_analysis outcomes and excluded before the AI eval runs. A ticket that cannot be rewound or investigated scores error without aborting the run.

HUM-7554     agree     verdict=already_fixed  expected=Closed / Done-Errata  actual=Closed / Done-Errata  route=!=needs_change
HUM-7537     DISAGREE  verdict=needs_human    expected=Closed / Won't Do     actual=                      route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective

Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM, and the pinned rpms repository inputs — are read live, so an old ticket is re-investigated against today’s package state rather than the state that existed when the human decided. Where the fix has since shipped, already_fixed is partly handed to the agent, so the score is retrospective current-state agreement, not a historical replay; every report prints that caveat. True replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms revision, and is deliberately left to a follow-up.

What the model sees

Only non-candidates reach the model. The prompt combines static /cve instructions with validated TicketFacts: ticket, package and CVE details, Jira content, analysis/discussion, built NVR, and SBOM identity.

The full SBOM is not included. Evidence is collected through bounded tools:

Evidence Tool What it establishes
SBOM contents search_sbom Searches SPDX metadata, including npm PURLs. Only present confirms shipping; ambiguous versions or unknown require investigation.
Packaging files search_package_files, read_package_file Finds build settings, patches and packaging evidence. Searches can be scoped to a relative file or directory; truncated results are incomplete, and empty results or scans stopped by file, byte or member limits are inconclusive.
Other packages lookup_shipped_package Returns the latest source-package SBOM, NVR, component version and binary presence. source_only or missing data is inconclusive.
Prepared source read_package_source and source searches Inspects sandboxed, spec-pinned source and records provenance. built_nvr_match: unverified means it is not proven to match the ticket build. Preparation failures are not bypassed.
npm metadata Registry lookup Provides publication time, gitHead, repository and integrity. Metadata alone proves neither shipping nor fixedness.
CVE advisories lookup_cve_advisory Returns affected packages, vulnerable ranges and patched versions from GitHub advisories. Missing, truncated or mismatched results are inconclusive.
Known upstream fix check_upstream_fix Checks a commit from the package’s upstream repository against prepared source. It must be merged into the upstream default branch or its declared release branch. Clean application means that exact diff is absent; already_applied means it reverse-applies cleanly. Other failures are inconclusive.
Upstream history Upstream source probe Searches an official repository once per ticket within a 20-candidate and 10 MiB discovery budget. incomplete: true means the search was truncated, so a missing candidate is inconclusive. Ancestry and applicability require a resolved exact base commit; unresolved candidates are not evidence. No candidate found means needs_human, not no_upstream_fix.

Evidence rules

  • Keep conclusions narrow. Patch applicability, %files entries, missing disable flags and failed searches alone do not prove affectedness, shipping or fix inclusion. Unresolved conclusions require needs_human.
  • Respect bounds. Tools enforce request, response and shared read budgets. Reads support line windows; excerpts may be truncated. Upstream access uses approved HTTPS hosts without redirect fallback.
  • Do not repeat permanent failures. retryable: false means move on and record the gap. Source tools permit retrying with arch when architecture selection is required.
  • Preserve partial evidence. Upstream collection failures abort the run; unresolved refs remain partial evidence. A bounded upstream search that hits its candidate or byte limit returns incomplete and is recorded as partial.

Model output

The model returns only status, confidence, summary, evidence and vex_justification. vex_justification is accepted only with not_affected and only as Component not Present or Vulnerable Code not Present.

The caller supplies file_actions, commit_message, ai_generated, ticket, package and cves, and replaces collection records with complete evidence.

Security

The agent processes untrusted input, including executable RPM macros and prep commands. The credential-bearing worker never executes or imports RPM specs.

  • Isolated source execution. Spec parsing, structural edits, preparation, and candidate application run in an ephemeral, uncredentialed sandbox. There are no host mounts or worker secrets, and command environments are cleared. Sandbox backends come from the shared Hummingbird Sandbox library. Podman disables networking; Kubernetes requires deny-all egress policy in the dedicated sandbox namespace. Privileged mock, full builds, and runtime dependency installation are not used. Reusing a container with a reset workspace is not a fresh security boundary: use controlled images and packaging, not mutually hostile tenants.
  • Deployment isolation. Repository Git operations use private, temporary no-checkout object transport; package files exist only in the offline sandbox. The worker may manage execution pods only in the sandbox namespace once explicit API credentials are available. Those pods run non-root, drop capabilities, disable privilege escalation and token mounts, and use infrastructure-managed pod deadlines and resource limits sufficient for the sweep. Normal context-manager exit deletes sandbox pods; forced termination can leave pods until their active deadline, so operators must remove leftovers. Standalone investigations clean up at run end. Sandbox-produced source trees are never extracted into the worker.
  • Secret-free repository subprocesses. Worker Git transport operations use a minimal, allowlisted environment and do not inherit the Jira or model tokens used by the trusted worker and sweep CLI.
  • Uncredentialed, hardened git. Repository transport is public and read-only with no push token. Credential helpers and interactive prompts are disabled, system/global git config is neutralized (an attacker-controlled GIT_CONFIG_GLOBAL cannot change behaviour), and submodule recursion is off.
  • Input validation. Package names are restricted to a safe charset (no path traversal or absolute paths), file reads are confined to the package directory, saved patch names must be *.patch and cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.

Development

See the main README for development workflows.

Build the local CVE sandbox image for explicit local configuration overrides:

podman build -f hummingbird-sandbox/images/cve/Containerfile \
  -t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
make hummingbird-cve-agent/setup
cd hummingbird-cve-agent
PYTHONPATH=.. python3 -m unittest discover -s tests -t .
# Opt into real offline RPM tests using an image with the required macros:
PYTHONPATH=.. HUMMINGBIRD_RPM_TEST_IMAGE=registry.example.com/rpm-prep:dev \
  python3 -m unittest tests.test_rpm_integration

The integration suite uses Podman without network access. Only source downloads are mocked; RPM parsing, Go preparation, architecture conditions, source overlays, patch checks, and final-spec replay execute for real. Standard unit discovery skips these tests unless the image variable is set.

The Containerfile pins the RPM repository commit supplying hum in HUM_COMMIT and records it in the io.hummingbird-project.hum-revision image label. Keep that pin aligned with the CVE analysis Containerfile and .gitlab-ci.yml, so test and coverage jobs exercise the same library as both images. The CI revision test in ci/internal/tests/test_hum_revision.py checks that the pins match. The agent’s fetch layer comes before Python dependencies, so dependency-only changes reuse the pinned checkout.

Rebuild and deploy both CVE images to refresh their bundled library. Updating a runtime RPMs checkout does not replace an already-imported hum.meta.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.37 - Upstream Monitor

Upstream release discovery and catalog API for Hummingbird. Application code lives in upstream-monitor/. Project definitions belong to the separate upstream-monitor-projects repository.

Features

  • Release discovery from Git, GitHub, GitLab, Anitya, and HTTP sources
  • Version filtering, release streams, lifecycle dates, and Git provenance
  • Catalog API and web interface
  • File, PostgreSQL, or S3 storage
  • Git catalog reconciliation and optional NATS JetStream events

Prerequisites

  • Go 1.27 or newer and a C compiler for local builds and tests
  • Podman for container builds
  • A project catalog and writable file storage, PostgreSQL, or S3 for results
  • Deployment managed credentials for authenticated upstreams and API operations

Build and Deployment

From the tools repository root:

podman build -t localhost/upstream-monitor:dev upstream-monitor/

Konflux builds ARM64 and AMD64 images as upstream-monitor--tools and releases them to quay.io/hummingbird-ci/upstream-monitor. The image runs as UID 65532 and contains these commands:

Command Purpose
upstream-monitor Poll upstreams and persist results, the default entry point
upstream-monitor-api Serve the catalog API and web interface
upstream-monitor-enricher Resolve optional Git provenance
upstream-monitor-project-reconciler Reconcile Git definitions into PostgreSQL

The infrastructure repository owns deployments, storage, messaging, credentials, and image pins. Register the Konflux component before building, and pin the released image digest in infrastructure and the catalog validator. Tools does not deploy the service or bundle a project catalog.

Usage

Validate a catalog

From the tools repository root:

podman run --rm --pull=never --network=none --read-only \
  -e PROJECTS_DIR=/projects \
  -v "$PWD/upstream-monitor/examples/file-layout/config/projects:/projects:ro" \
  localhost/upstream-monitor:dev validate

Replace the mounted directory with a checkout of the deployment catalog to validate it. Validation does not contact upstreams or write to a database. An empty or missing directory validates zero projects, so check the reported count before treating validation as successful.

Run locally

From the application directory, start the monitor:

cd upstream-monitor
PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor

In another terminal, from the same directory, start the API:

PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor-api

Both processes must use the same project and results storage. File results default to observations/, with provider cache data under cache/. Without project definitions, the monitor has nothing to check.

For an upstream check without persisting results:

go run ./cmd/upstream-monitor dry-run \
  --projects-dir /path/to/projects --project example-widget

dry-run contacts the upstream and can consume API quota. It resolves credentials through source.credential_ref, not the monitor’s GitHub App configuration.

Configuration

Variable Default Purpose
PROJECT_STORAGE_BACKEND file Definitions: file, postgres, or s3
RESULTS_STORAGE_BACKEND file Results: file, postgres, or s3
PROJECT_MUTATIONS auto Definition writes: auto, enabled, or disabled
PROJECTS_DIR config/projects External project JSON directory
POSTGRES_DSN empty Required when either backend is PostgreSQL
S3_BUCKET empty Required when either backend is S3
S3_PREFIX empty Unique prefix for each installation sharing a bucket
S3_REGION us-east-1 AWS region
S3_ENDPOINT empty Optional S3 compatible endpoint
S3_USE_PATH_STYLE false Enable path style S3 addressing
OBSERVATIONS_DIR observations File results directory
CACHE_DIR cache File backend provider cache
LISTEN_ADDR :8080 API listen address
POLL_INTERVAL 5m Scheduler interval and default project check interval
UPSTREAM_TIMEOUT 5m Maximum duration of an upstream check
MAX_CONCURRENT_CHECKS 4 Concurrent checks per worker
MAX_CONCURRENT_GIT_FETCHES 1 Concurrent uncached Git fetches per worker
ENRICHMENT_ENABLED false Include provenance in API responses

File results support a single writer. Use PostgreSQL or S3 for shared worker coordination. S3 must support conditional writes. Do not expire observation or coordination objects through bucket lifecycle policies.

Use GITHUB_TOKEN for token based GitHub access. For a GitHub App, set GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and GITHUB_APP_PRIVATE_KEY_FILE together. A project’s source.credential_ref names an environment variable containing a credential, never the credential itself. Keep secrets outside Git and container images.

See configuration defaults and environment overrides for remaining cache, lease, retry, and concurrency settings.

Project Definitions

Each JSON filename must match its id. IDs start with a lowercase letter, contain only lowercase letters, digits, and hyphens, and have at most 128 characters. Do not supply uid, which the service assigns.

{
  "id": "example-widget",
  "name": "Example widget",
  "homepage": "https://example.org/widget",
  "source": {
    "type": "github-releases",
    "location": "example/widget"
  },
  "version_policy": {
    "scheme": "semantic",
    "include_prereleases": false
  },
  "stream_policy": {
    "granularity": "minor"
  },
  "check_policy": {
    "mode": "fixed",
    "interval": "15m"
  }
}
Source type Location
git-tags Git remote URL
github-releases GitHub owner/repository
gitlab-releases, gitlab-tags GitLab numeric project ID or namespace path
anitya Numeric Anitya project ID
http-regex HTTP(S) URL with a source.extract regular expression

Set source.base_url for a separate GitLab or Anitya instance, and source.tag_pattern to select a tag family. Upstream destinations are not host allowlisted, so infrastructure must enforce the required network egress policy.

The default version scheme is semantic. Other choices are calendar, rpm, pep440 (python), and lexical (generic). Use transforms to normalize versions, and version_filters and prerelease_filters to control release selection. Streams group releases by major or minor, with optional semantic numeric component buckets.

Optional lifecycle modes are latest-release, latest-n-lines, date-based, endoflife-date, and manual, with per stream overrides. Fixed check policies use interval. Adaptive policies back off between min_interval and max_interval, resetting after a new release. The maximum interval is also the worst polling detection delay.

Catalog Reconciliation

The project reconciler validates an atomic git-sync checkout and replaces the PostgreSQL definition set. Supply POSTGRES_DSN through the environment:

PROJECT_STORAGE_BACKEND=postgres \
PROJECT_MUTATIONS=disabled \
PROJECT_CHECKOUT_LINK=/git-sync/current \
PROJECT_CHECKOUT_SUBPATH=projects \
  go run ./cmd/upstream-monitor-project-reconciler

Run from upstream-monitor/. The projects subpath matches the separate catalog repository. Set PROJECT_MUTATIONS=disabled on every process using that catalog. Workers and API replicas read its database copy.

Invalid revisions leave the last valid catalog active. Reconciliation removes definitions absent from Git. Verify completeness and back up the database before enabling it. Git cannot restore deleted observation history.

The monitor initializes database schemas. The API does not. Schema and image rollouts belong to infrastructure, not API startup.

API and Authentication

Open / for the catalog browser. The generated API schema is available at /openapi.json and /openapi.yaml. /docs redirects to the JSON schema.

Endpoint Purpose
/healthz Process health
/readyz Project store readability, not proof of successful polling
/v1/status Aggregate and per project check status
/v1/project-reconciliation Attempted and active Git revisions and reconciliation errors
/v1/projects Project search and pagination

Use last_success_at and project errors to assess freshness. Follow X-Next-Cursor for paginated reads. Definition updates and deletes require the current ETag in If-Match. A queued scan retains observations, while rescan clears that project’s snapshot and release history before checking.

A registered project may have no stored snapshot before its first successful scan. When project status has no last_success_at and does not report succeeded, GET /v1/projects/{project_id} returns 200 OK with the project identity, empty releases and streams arrays, and observed_at set to the zero timestamp 0001-01-01T00:00:00Z. The /streams and project /releases endpoints also return empty arrays. These reads do not persist a snapshot or mark a scan successful. Read /v1/projects/{project_id}/status for scan state and failure details.

Unknown projects and streams still return 404 Not Found. A missing snapshot after a successful scan also remains an error. Unexpected storage failures, including an inability to read project status, remain server errors rather than empty results. Without a configured status reader, a missing snapshot remains 404 Not Found because the API cannot verify that no scan has succeeded.

Catalog reads are public. Definition reads, writes, inline checks, and scan requests require authentication and Cedar authorization. PROJECT_MUTATIONS=auto disables file definition writes but permits supported PostgreSQL and S3 writes.

API_AUTH_MODE Required configuration
disabled Default. Authenticated operations are unavailable.
static-token API_AUTH_TOKEN, API_AUTH_PRINCIPAL, API_AUTH_ROLES
kubernetes-tokenreview API_KUBERNETES_AUDIENCE and a role mapping file
oidc API_OIDC_ISSUER, API_OIDC_AUDIENCE, and a role mapping file
trusted-proxy API_TRUSTED_PROXY_CIDRS and a role mapping file
insecure Local development only. Never use in production.

API_AUTH_ROLE_MAPPINGS_FILE maps subjects and groups to Cedar roles:

{
  "subjects": {
    "system:serviceaccount:ci:release-bot": ["ProjectChecker"]
  },
  "groups": {
    "editors": ["ProjectEditor"]
  }
}

The default policy grants definition reads to ProjectReader, inline checks to ProjectChecker, editing and scan operations to ProjectEditor, and all actions to ProjectAdmin. Use API_CEDAR_POLICY_FILE to replace that policy.

TokenReview requires a mounted reviewer token and permission to create tokenreviews.authentication.k8s.io. Trust proxy identity headers only from an isolated authenticating proxy that callers cannot bypass. Remove inbound identity headers at the proxy, and prefer loopback CIDRs for a sidecar rather than trusting a whole cluster network.

Project status

Each project fetch runs independently. A fetch failure or timeout records a project error and schedules a retry with backoff. Other project checks continue.

Repository lease errors do not stop the monitor. The scheduler reports the error and defers unfinished projects until the next poll interval. Completed checks retain their results. Deferred checks do not count as upstream failures. A batch lease claim error defers the due projects for the same interval. Cancellation still stops the monitor.

Workers report progress and results through the scheduler’s event channel. Only the scheduler changes the in-memory project schedules and statuses.

Successful checks return to scheduled. waiting and succeeded are accepted for compatibility with older stored records:

stateDiagram-v2
    [*] --> queued : definition due now
    [*] --> scheduled : definition with future check
    [*] --> waiting : load legacy record
    [*] --> succeeded : load older successful record
    waiting --> queued : restart with due check
    waiting --> scheduled : restart with future check
    succeeded --> queued : next check claimed
    scheduled --> queued : check due, scan, or definition changed
    queued --> processing : worker starts
    queued --> scheduled : repository lease unavailable
    processing --> scheduled : snapshot persisted
    processing --> failed : check or persistence error
    failed --> queued : retry or definition changed
    processing --> queued : recover interrupted file worker

Records include the project identity, definition fingerprint, pending scan request, queue/start/finish/next check times, last_success_at, failure count, last error, and release/stream counts.

Events and Monitoring

Set NATS_URL to enable JetStream publishing and NATS_CREDS_FILE for mounted credentials. Default subjects are hummingbird.upstream.version.v1 and hummingbird.upstream.stream.v1. Set NATS_MANAGE_STREAM=false if infrastructure provisions the stream. Delivery is at least once, so consumers must deduplicate using event_key. Publication failures retain events for retry.

Logs are JSON on stdout by default, configured through LOG_LEVEL and LOG_FORMAT. Set OTEL_METRICS_EXPORTER=otlp with standard OpenTelemetry endpoint settings to export metrics. Unset exporter configuration does not connect to a local collector.

Monitor time since the last successful check, check failures, GitHub quota, and outbox depth and oldest pending age. Quiet projects remain healthy when checks succeed and adaptive intervals grow.

Development

Run from upstream-monitor/:

go vet ./...
go test ./...
go build ./cmd/...
go run ./cmd/upstream-monitor validate --projects-dir examples/file-layout/config/projects

The GitLab job also rejects empty fixture directories. Container smoke tests mount the examples and check API health through the K8s test runner.

Integration tests require disposable services. Set TEST_POSTGRES_DSN for PostgreSQL storage, TEST_POSTGRES_RECONCILE_DSN for a separate catalog replacement database, TEST_S3_ENDPOINT and TEST_S3_BUCKET for S3, or NATS_CLUSTER_TEST_URL for a JetStream cluster with three stream replicas. Never point these tests at production services.

License

See the repository LICENSE.

4.5.38 - Hummingbird OSV Generator

A CLI tool that converts Red Hat vulnerability data into OSV records for Hummingbird RPMs, so that vulnerability scanners can match them against the packages installed in an image.

For how a run works end to end, see the workflow.

Features

  • Two sources: shipped fixes from the advisories repository and all known affects, including unfixed CVEs, from OSIDB
  • Binary RPM names: scanners match installed binary packages, so source RPMs are expanded to their binary RPMs
  • Fixed and unfixed states: records carry fixed, affected, will_not_fix or under_investigation in database_specific.status
  • RPM epochs: fixed versions are written as epoch:version-release
  • Incremental sync: content hashing means only changed inputs are parsed and only changed records are written
  • Embargo safe: embargoed or non-public OSIDB data is never published

Output format

The output follows OSV schema 1.9.0 with one record per source RPM per CVE and one affected entry per binary RPM.

Field Content
id Opaque, stable identifier: x_HBA prefix, CVE year and a hash
upstream The CVE the record covers
affected[].package.ecosystem Ecosystem scoped to one product stream by its CPE
affected[].package.purl Versionless package URL
affected[].ranges introduced: 0 and, once shipped, fixed: <epoch:version-release>
affected[].ecosystem_specific.advisory_id Advisory that supplied the fixed version, absent on unfixed entries
database_specific.status fixed, affected, under_investigation or will_not_fix
modified Time of the last content change

The x_ in the identifier prefix marks a local database. It is dropped once HBA is registered with the OSV schema project. The ecosystem name is also a placeholder until it is registered.

Prerequisites

  • Go: the version in hummingbird-osv-generator/go.mod
  • Advisories checkout: a clone of the advisories repository, passed as its data/ directory
  • OSIDB access: the OSIDB base URL, reachable from where the tool runs
  • Git: only when using -git or -git-remote

Installation

git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/hummingbird-osv-generator
go build -o rhcve-osv-sync ./cmd/rhcve-osv-sync/

Usage

Full sync for one product stream, committing the output to git:

./rhcve-osv-sync \
  -advisories /path/to/advisories/data \
  -osidb-url https://osidb.example.com \
  -streams hummingbird-1 \
  -products hummingbird \
  -output /path/to/osv-output \
  -state /path/to/state.db \
  -full -git

Later runs without -full use an OSIDB delta query. Run a full reconciliation daily so that missed changes are corrected.

Binary RPM expansion

OSIDB tracks CVEs against source RPM names, while scanners need binary RPM names. -expansion selects how a source RPM is expanded:

Mode Source
osidb OSIDB subpackage_purls
advisory Binary RPM names derived from advisory purls
chain OSIDB first, then the advisory mapping

The default is chain.

Incremental sync

State lives in a SQLite database. Each run goes through these phases: advisory diff, OSIDB diff, convert, write output and commit state. The workflow describes each phase.

  • Advisory files are content-hashed and only changed files are parsed
  • OSIDB uses a delta query, plus a full reconciliation on demand
  • Output files are compared with a manifest and only changed files are written
  • Output is written before state is committed, so a crash is safe
  • If the manifest is lost, the next full run rebuilds it from the output and withdraws stale records
  • If OSIDB fails and the convert phase does not need it, advisory output is still published and the process exits non-zero so a scheduler can retry or alert

Unchanged input produces unchanged output, so duplicate scheduling wastes work but does not corrupt the output.

Configuration

Flag Required Description
-advisories Yes Path to the advisories repository data/ directory
-osidb-url Yes OSIDB base URL
-streams Yes Comma-separated ps_update_stream values, also filters advisories by product_stream
-output Yes Output directory for the OSV JSON files
-state Yes Path to the SQLite state database
-products No Filter advisory conversion by CPE product name
-full No Force a full OSIDB reconciliation
-expansion No Binary RPM expansion mode: osidb, advisory or chain
-git No Run git init, add and commit on the output after the sync
-git-remote No Git remote to push to after the commit (implies -git)
-log-json No Write structured JSON logs

Development

cd hummingbird-osv-generator

# Vet and test, including the OSV schema validation test
go vet ./...
go test ./...

The hummingbird-osv-generator CI job runs the same commands on merge requests.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

4.5.38.1 - Hummingbird OSV Generator Workflow

How a run of the Hummingbird OSV Generator turns its inputs into OSV records: what it reads, the phases of a sync, and what it writes.

Overview

flowchart LR
    adv[Advisories data directory] --> p1[Advisory diff]
    osidb[OSIDB] --> p2[OSIDB diff]
    db[(State database)] --> p1
    db --> p2
    p1 --> p3[Convert]
    p2 --> p3
    osidb --> p3
    p3 --> p4[Write output]
    p4 --> out[OSV JSON files]
    p4 --> p5[Commit state]
    p5 --> db
    out --> git[Optional git commit and push]

One run reads the advisories and OSIDB, converts what changed into OSV records, writes only the records whose content changed, and records what it did in the state database. Each run is a complete pass: nothing is kept in memory between runs.

Inputs

Input Flag What the generator reads
Advisories directory -advisories Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists
OSIDB -osidb-url Affects for the given update streams, and the flaws for their CVEs
Update streams -streams Which OSIDB update streams to read. Also filters advisories by product_stream
Products -products Optional filter on the CPE product name of an advisory
State database -state SQLite file from earlier runs, created on the first run
Output directory -output Read only when the record ledger was lost and has to be rebuilt from the published files

Advisories

  • Only security advisories (RHSA) are read. Bug fix and enhancement advisories are skipped, even if they mention CVEs.
  • Each advisory gives the product CPE, and each artifact gives a package URL and the CVEs it fixes. The package URL carries the binary RPM name, the version and release, and the RPM epoch.
  • From the advisories the generator also builds two lookups: the update stream to CPE map, and the source RPM to binary RPM mapping.

OSIDB

An affect is ignored when it has no CVE ID, when it is embargoed or not public, or when it is marked not affected (including a delegated not affected). An affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also ignored, because the advisory supplies that fix.

Flaws supply the CVE title and description, severity, CWE and Red Hat statement. A flaw that is embargoed or not public is ignored, and any text cached from it is dropped. A rejected flaw marks its records as withdrawn.

State database

Table Content
input_hashes Content hashes of advisory files and of OSIDB affects, used to detect changes
sync_meta Cursors: the last sync, the last full reconciliation, and the last flaw change window
cve_meta Cached CVE text and severity, refreshed from the OSIDB flaw change feed
output_manifest One row per published record: path, content hash, time written and time last modified

The input hashes and the cursors are a cache that can be rebuilt. The manifest also holds each record’s last modification time, so it is durable state.

Phases

  1. Advisory diff. Hash every advisory file and compare with the stored hashes. The result is the list of added, changed and removed files.
  2. OSIDB diff. Decide between a delta and a full reconciliation. A delta asks OSIDB for affects updated since the last sync, minus a clock skew allowance, and compares their hashes. A full reconciliation fetches every affect and also detects deletions. By default a full reconciliation runs on the first run, when -full is given, and again once a day. If OSIDB fails here, the run goes on with advisory data when nothing needs OSIDB, and ends with an error (see Failure handling).
  3. Convert. Build records from both sources:
    • Fixed records come from the advisories.
    • Records for every other status come from OSIDB affects. Each affect is expanded from its source RPM to binary RPM names with the mode chosen by -expansion.
    • CVE text and severity come from the flaw cache, refreshed from the OSIDB flaw change feed. If the refresh fails, cached values are used.
    • When an advisory changed and output already exists, the complete OSIDB data is fetched, so that a record is never published from partial input.
  4. Write output. Compare each record’s content hash with the manifest and write only the records that changed. Remove records that are no longer produced (see Failure handling for when this is safe).
  5. Commit state. Store the new input hashes, the manifest and the cursors. The state is committed after the output is written, so a crash between the two repeats work and does not lose changes.

If -git or -git-remote is given, the output directory is then committed, and pushed when a remote is given.

Records

A record covers one source RPM in one update stream for one CVE. It has one affected entry for each binary RPM, and each entry keeps the fixed version of its own binary because subpackages can have different epochs.

When one source RPM has entries in different states, the record takes the status with the highest precedence:

Precedence Status Meaning
1 fixed An advisory shipped the fix
2 affected OSIDB says the product is affected and no fix has shipped
3 under_investigation OSIDB has a new affect with no resolution yet
4 will_not_fix OSIDB resolution is won’t fix or defer

A shipped fix always wins over an unfixed affect for the same record.

published is the earliest of the dates known for the record: when the flaw became public, when the affect was created, and when an advisory shipped. modified changes only when the content of the record changes: a new record takes its source update time, an unchanged record keeps its value, and a changed record gets the time of the write.

Status over time

The status is recomputed from the inputs on every run and is not stored. A typical record moves as follows.

stateDiagram-v2
    [*] --> UnderInvestigation: OSIDB affect created, no resolution
    [*] --> Affected: OSIDB affect with a resolution
    [*] --> WillNotFix: OSIDB resolution won't fix or defer

    UnderInvestigation --> Affected: triaged as affected
    UnderInvestigation --> WillNotFix: resolution won't fix or defer
    WillNotFix --> Affected: resolution changed back
    Affected --> Fixed: advisory ships the fix
    UnderInvestigation --> Fixed: advisory ships the fix
    WillNotFix --> Fixed: advisory ships the fix
    Affected --> UnderInvestigation: OSIDB affect reset to new

    UnderInvestigation --> Removed: marked not affected, embargoed or not public
    Affected --> Removed: marked not affected, embargoed or not public
    WillNotFix --> Removed: marked not affected, embargoed or not public
    Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
    Removed --> [*]

    state "under_investigation" as UnderInvestigation
    state "affected" as Affected
    state "will_not_fix" as WillNotFix
    state "fixed" as Fixed
    state "file removed from the output" as Removed

A removed record is deleted from the output directory and from the manifest. A record whose flaw was rejected is different: it stays in the output with a withdrawn timestamp.

Outputs

Output Location Content
OSV records <output>/<product path>/<ID>.json One JSON file per record. The product path comes from the CPE, for example redhat/hummingbird/1
State database The -state file Updated hashes, manifest, cursors and CVE cache (see State database)
Git commit (optional) The -output directory All changes committed, with a message that summarizes the run
Run summary standard error Counts of records created, updated, withdrawn and unchanged, and files written and removed

The git step initializes the repository if needed, skips the commit when nothing changed, and pushes the current branch when -git-remote is given. It runs only after a sync that ended without an error.

The process exits with 0 on success, 1 when the sync or the git step fails, and 2 for invalid flags.

Failure handling

  • OSIDB failure during the diff. If the convert phase does not need OSIDB data, advisory output is still written and the state is committed, and the process exits with 1 so a scheduler can retry or alert. The git step does not run in that case, so the written files are committed by the next successful run. Records that depend on OSIDB are not removed, because the record set is incomplete.
  • OSIDB unavailable when convert needs it. This happens when an advisory changed and output already exists, because the complete OSIDB data is needed to avoid a partial record. The run fails before writing anything.
  • No OSIDB changes in a delta run. The advisory-only result is not compared with existing records, and no record is removed, because OSIDB data is missing from it.
  • Lost state database. The next run behaves like a first run.
  • Lost manifest, output kept. A full run rebuilds the manifest from the published files and rewrites each record once, so stale records can still be removed.
  • A corrupt published file fails the run when the manifest is rebuilt, so it cannot hide stale state.
  • Repeated or duplicate runs. The same input produces the same output, so a duplicate run wastes work and does not corrupt anything.

4.5.39 - S3 Lookaside Cache

Infrastructure for caching dist-git lookaside content, providing access to source artifacts for the Hummingbird build pipeline.

Overview

This stack provides S3-based infrastructure for dist-git artifacts with CloudFront CDN. It caches content from upstream sources like Git tarballs.

A companion stack (s3-lookaside-cache-upload-role) creates a GitLab OIDC identity provider and an IAM role that GitLab CI jobs can assume using short-lived JWT tokens, eliminating the need for long-lived access keys.

Architecture

Cache Infrastructure

flowchart LR
    clients["Build Clients"]
    cf["CloudFront\nDistribution"]
    s3["S3 Bucket\n(dist-git-cache)"]
    logs["S3 Logs\nBucket"]
    backup["AWS Backup\nVault"]
    headers["Security\nHeaders"]

    clients --> cf --> s3
    cf --> headers
    s3 --> logs --> backup

GitLab CI Upload Role

flowchart LR
    gitlab["GitLab CI\n(.gitlab-ci.yml)"]
    sts["AWS STS\n(validates JWT\nvia OIDC/JWKS)"]
    role["IAM Role\n(scoped to\nS3 PutObject)"]
    s3["S3 Bucket\n(dist-git cache)"]

    gitlab -- "id_token" --> sts
    sts -- "AssumeRoleWithWebIdentity" --> role
    role -- "temporary credentials" --> sts
    sts -- "credentials" --> gitlab
    gitlab -- "s3:PutObject" --> s3

Components

Cache Stack (s3-lookaside-cache)

Resource Type Description
DistGitCacheBucket S3 Bucket Main cache storage with versioning and object lock
DistGitLogBucket S3 Bucket Access logs with tiered storage lifecycle
DistGitCacheDistribution CloudFront CDN with HTTP/2+3, IPv6, TLS 1.2+
DistGitCacheOAC Origin Access Ctrl Secure S3 access from CloudFront
DistGitCacheCachePolicy Cache Policy 1-day default TTL, 1-year max, Gzip/Brotli
DistGitCacheResponseHeadersPolicy Response Headers HSTS, X-Frame-Options, XSS protection
DistGitBackupVault Backup Vault AWS Backup vault for data protection
DistGitBackupPlan Backup Plan Daily (35-day) and weekly (365-day) backups
DistGitUploadPolicy IAM Managed Policy Grants s3:PutObject to the cache bucket

Upload Role Stack (s3-lookaside-cache-upload-role)

Resource Type Description
GitLabOIDCProvider IAM OIDC Provider Registers gitlab.com as a trusted identity provider
DistGitUploadRole IAM Role Web identity role assumable by the configured GitLab project

Parameters

Cache Stack Parameters

Parameter Description
ResourcePrefix Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-cache)
BucketName Globally unique name for the cache bucket (logs bucket appends -logs)
TrustedKeyGroupId CloudFront key group ID whose keys may validate signed URLs — created and managed by the CKI credential manager; empty = the distribution serves unsigned/public

Upload Role Parameters

Parameter Description
ResourcePrefix Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-upload)
CacheStackName Name of the deployed s3-lookaside-cache CloudFormation stack
AdditionalCacheStackName Optional second cache stack whose upload policy is attached to the role
GitLabProjectPath GitLab project path allowed to assume the role (e.g., redhat/hummingbird/rpms)

S3 Key Structure

Files are stored using the dist-git lookaside path convention:

{namespace}/{package}/{filename}/{hashType}/{hash}/{filename}

Example:

rpms/tar/tar-1.35.tar.xz/sha512/abc123.../tar-1.35.tar.xz

Trust Policy

The upload role’s trust policy restricts access using three conditions (all StringEquals):

  • Audience (gitlab.com:aud): Must be https://gitlab.com
  • Subject (gitlab.com:sub): Must match project_path:<GitLabProjectPath>:ref_type:branch:ref:main
  • Protected ref (gitlab.com:ref_protected): Must be "true"

Only the main branch of the configured project can assume the role.

GitLab CI Usage

Configure your .gitlab-ci.yml to assume the role using id_tokens. The AWS CLI automatically calls AssumeRoleWithWebIdentity when AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN are set:

upload to cache:
  image:
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  script:
    - set +x
    - printenv GITLAB_OIDC_TOKEN > /tmp/oidc-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/oidc-token
    - export AWS_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME}"
    - export AWS_ROLE_SESSION_NAME="gitlab-ci-${CI_JOB_ID}"
    - aws s3 cp "$FILE" "s3://${BUCKET}/${S3_KEY}"
    - rm -f /tmp/oidc-token

The aud value in id_tokens must match the audience configured in the OIDC identity provider (https://gitlab.com). The token is written to a file because the AWS SDK reads it from AWS_WEB_IDENTITY_TOKEN_FILE rather than accepting it inline.

Security Features

  • Encryption: AES256 server-side encryption with S3 bucket keys
  • Public Access: All public access blocked on both buckets
  • Transport Security: HTTPS enforced via bucket policy
  • Object Lock: GOVERNANCE mode with 1-day default retention (will be increased in the future)
  • CloudFront OAC: Modern Origin Access Control (not legacy OAI)
  • Security Headers: HSTS, X-Frame-Options (DENY), X-Content-Type-Options, X-XSS-Protection, strict referrer policy
  • TLS 1.2 Minimum: For all CloudFront connections
  • OIDC Federation: Short-lived tokens instead of long-lived access keys

Signing Key Rotation

CloudFront public-key material is immutable — a rotation always creates a new key object. The signing key is a managed credential — one token group per key pair with token_type: cloudfront_signing_key. The CKI credentials documentation is the reference for the type: fields, selectors, CloudFront resource naming, and the key-group ownership (the manager creates the key group, registers every public key in it, and maintains the membership).

This stack references the key group by ID: the TrustedKeyGroupId parameter carries the CloudFront key group ID, and the distribution trusts exactly the keys that group holds. An empty parameter (the default) leaves the distribution unsigned/public — the public-cache sibling stack deploys that way. Key-group membership changes take effect without redeploying this stack: the distribution follows the live group.

Rotate from an infrastructure repository worktree with direnv set up (Vault tooling), following the standard prepare → switch → clean cycle; the managed-credentials runbook (documentation/operating/secrets.md in the infrastructure repository) covers the merge, bake and rollback pattern:

python3 -m cki_tools.credentials.manager prepare --token-name <KEY>
python3 -m cki_tools.credentials.manager switch --token-name <KEY>
python3 -m cki_tools.credentials.manager clean --token-name <KEY>

Merge the secrets.yml change between steps. On switch, the signer (kubernetes/setup-konflux/48-…, reading the [deployed] private key and [deployed]#key_pair_id) moves to the new key. Clean merges after a bake period; the key-group drop is the revocation.

Fresh deployments

manager create bootstraps the credential and the key group — the founding version deploys immediately. Bring up a signed deployment in this order:

  1. Add the signing-key entry to secrets.yml with token_type: cloudfront_signing_key and the profile_name and aws_name_prefix meta.
  2. Run python3 -m cki_tools.credentials.manager create --token-name <KEY>.
  3. Set the stack’s TrustedKeyGroupId parameter to the recorded key_group_id.
  4. Deploy the stack.

The deployment bot holds the CloudFront public-key and key-group read actions (GetPublicKey, GetPublicKeyConfig, GetKeyGroup, GetKeyGroupConfig) on Resource: "*": these actions authorize without a resource ARN, so only Resource: "*" grants match — resource-scoped grants silently never apply (verified live during the 2026-11 signing-key rotation). The CKI credential manager performs all key and key-group changes with operator credentials. The deployment-bot template reflects this.

Backup Strategy

Schedule Retention Cold Storage
Daily (5:00 AM UTC) 35 days -
Weekly (Sunday 5:00 AM UTC) 365 days After 30 days

Continuous backup is enabled for point-in-time recovery.

Log Lifecycle

Access logs transition through storage tiers:

Age Storage Class
0-30 days Standard
30-60 days Standard-IA
60-90 days Glacier IR
90+ days Expired

Outputs

Cache Stack Outputs

Output Description
BucketName Name of the cache S3 bucket
BucketArn ARN of the cache S3 bucket
LogBucketName Name of the logs S3 bucket
LogBucketArn ARN of the logs S3 bucket
DistributionDomainName CloudFront domain name
DistributionId CloudFront distribution ID
DistributionArn CloudFront distribution ARN
BackupVaultName AWS Backup vault name
BackupVaultArn AWS Backup vault ARN
BackupPlanId AWS Backup plan ID
UploadPolicyArn ARN of the managed policy granting upload access

Upload Role Outputs

Output Description
RoleArn ARN of the upload role (pass to GitLab CI as ROLE_ARN)
RoleName Name of the upload role
OIDCProviderArn ARN of the GitLab OIDC identity provider

References

4.5.40 - Docs Content Negotiation

CloudFront distribution with HTTP content negotiation for the Hummingbird public documentation site (hummingbird-project.io). When a client sends Accept: text/markdown, a CloudFront Function rewrites the request URI to the pre-built .md equivalent so GitLab Pages serves Hugo’s markdown output instead of HTML.

Overview

Hugo generates .md files alongside .html for every page. GitLab Pages is a static file server — it cannot inspect Accept headers and select a response type. This stack interposes a CloudFront distribution between DNS and GitLab Pages, adding the header-based routing that GitLab Pages cannot provide.

The stack is deployed by the infrastructure repository. After each deploy, Route53 A and AAAA alias records for hummingbird-project.io are updated to point at the CloudFront distribution.

Architecture

flowchart LR
    client["Client\n(browser / agent)"]
    dns["Route53\nhummingbird-project.io"]
    cf["CloudFront\nDistribution"]
    fn_req["CloudFront Function\n(viewer-request)"]
    fn_resp["CloudFront Function\n(viewer-response)"]
    pages["GitLab Pages\nredhat.gitlab.io"]

    client -->|"HTTPS"| dns
    dns -->|"A/AAAA alias"| cf
    cf --> fn_req
    fn_req -->|"rewrite URI if\nAccept: text/markdown"| cf
    cf -->|"forwards viewer Host header"| pages
    pages --> fn_resp
    fn_resp -->|"auth redirects → 404"| client

Content Negotiation Logic

The cloudfront-js-2.0 viewer-request function runs at every edge location before the cache is consulted:

  1. If the URI starts with /l/ → normalize and pass through (regardless of headers). These are stable alias paths that GitLab Pages handles via a single _redirects splat rule per alias (/l/foo/*). The function normalizes variant request forms so the splat rule matches: /l/foo gets a trailing slash, /l/foo.md becomes /l/foo/index.md. Requests already in /l/foo/ or /l/foo/index.md form pass through unchanged.
  2. If Accept does not include text/markdown → pass through unchanged (HTML response served as before).
  3. Rewrite only known HTML-like patterns:
    • /docs/ (trailing slash) → /docs/index.md
    • /docs/page.html → /docs/page.md
    • /docs/page (no extension) → /docs/page/index.md
  4. Everything else (.css, .js, .png, .md, .txt, etc.) → pass through unchanged.

Because the rewrite happens before the cache key is computed (viewer-request event), HTML and markdown responses land in separate cache entries. No custom cache policy is needed — the standard CachingOptimized managed policy is used.

Auth-Redirect Guard

A second CloudFront Function (AuthRedirectGuardFunction) runs on viewer-response. It detects 302 responses whose Location header points to projects.gitlab.io/auth — the GitLab Pages access-control gate — and converts them into a 404 Not Found before they reach the viewer.

GitLab Pages access control causes the origin to return a 302 auth redirect for nonexistent paths instead of a proper 404. CloudFront caches origin responses per Accept-Encoding variant (gzip vs brotli), so these redirects can persist for up to 24 hours. Without the guard, clients follow a redirect chain to gitlab.com/users/sign_in and land on a Cloudflare challenge page.

Origin

The origin is redhat.gitlab.io (the GitLab Pages default domain). GitLab Pages routes requests based on the Host header. The AllViewer origin request policy forwards the viewer’s Host: hummingbird-project.io header to the origin, so the correct project is served regardless of what DNS hummingbird-project.io resolves to.

Components

Resource Type Description
ContentNegotiationFunction CloudFront Function viewer-request; rewrites URI for Accept: text/markdown
AuthRedirectGuardFunction CloudFront Function viewer-response; converts auth redirects to 404
DocsCertificate ACM Certificate TLS for the custom domain; DNS-validated via Route53
DocsDistribution CloudFront Distribution CDN proxy in front of GitLab Pages

Parameters

Parameter Description
ResourcePrefix Prefix for resource names (e.g., arr-hummingbird-prod-docs-cdn)
DocsDomainName Custom domain for the documentation site (e.g., hummingbird-project.io)
HostedZoneId Route53 hosted zone ID — used for ACM DNS validation CNAME auto-creation

Outputs

Output Description
DistributionDomainName CloudFront domain name
DistributionId CloudFront distribution ID

Deployment

Deployed by the infrastructure repository. Stack name and region are configured in samconfig.toml.j2 in the infrastructure repo.

On the first deploy, CloudFormation requests an ACM certificate and auto-creates the DNS validation CNAME in Route53 (via HostedZoneId). Once validated (~2–5 minutes), the distribution is created and Route53 alias records are updated to point at it.

Verification

# Content negotiation: should return text/markdown
curl -H "Accept: text/markdown" https://hummingbird-project.io/docs/ -sI

# Normal browsing: should return text/html (unchanged)
curl https://hummingbird-project.io/docs/ -sI

# /l/ alias paths are not rewritten — GitLab Pages issues HTTP 301 instead
curl -H "Accept: text/markdown" https://hummingbird-project.io/l/adding-images -sI

# Nonexistent paths return 404 (auth-redirect guard converts the origin's 302)
curl https://hummingbird-project.io/this-does-not-exist -sI

# Static assets are excluded from rewriting
curl -H "Accept: text/markdown" https://hummingbird-project.io/favicon.ico -sI

# Alternate domains still redirect to primary (unchanged)
curl https://project-hummingbird.io/ -sI   # expect 308
curl https://hummingbird-project.org/ -sI  # expect 308

# DNS resolves to CloudFront, not the GitLab Pages IP
dig hummingbird-project.io A +short

Scope

Only the primary domain (hummingbird-project.io) is proxied through CloudFront. The alternate domains (project-hummingbird.io, hummingbird-project.org) continue pointing directly to GitLab Pages, which issues 308 redirects to the primary domain.

4.5.41 - Hummingbird MR Collaboration

How to open merge requests and help on someone else’s MR in the redhat/hummingbird/tools monorepo. Applies to all components (including redhat-catalog/).

Features

  • Same-repo branches — contributors work in the shared GitLab project, not personal forks for day-to-day changes
  • One MR per change — seniors push fixes to the author’s source branch instead of opening a nested MR
  • Automatic CI — each push to the MR branch re-runs the pipeline, GitLab Pages preview, and dashboard status links

Prerequisites

  • GitLab account with Developer (or higher) access on redhat/hummingbird/tools
  • Git clone of the monorepo
  • Optional: glab CLI authenticated to GitLab

Git remotes

Many developers use two remotes:

Remote Typical URL Use for
upstream git@gitlab.com:redhat/hummingbird/tools.git Fetch MRs, push to shared branches
origin Personal fork (if configured) Optional; not used for team MRs

Add upstream if missing:

git remote add upstream git@gitlab.com:redhat/hummingbird/tools.git
git fetch upstream

Do not open fork MRs for routine team work. mr-auto-approver unconditionally rejects MRs where source_project_id != target_project_id.

Open an MR (author)

git clone git@gitlab.com:redhat/hummingbird/tools.git
cd tools
git checkout -b hum-XXXX-short-description
# edit, commit
git push -u upstream hum-XXXX-short-description

Open an MR targeting main in the GitLab UI.

Branch naming: prefer hum-XXXX-* or a descriptive feat/*, fix/*, or experiment/* prefix (see Environment Promotion for experiment/* deploy behavior).

Help on someone else’s MR (reviewer)

Use the MR source branch name from the GitLab MR page (not necessarily your local checkout name).

1. Check out the MR locally

Option A — fetch by MR number (works without glab):

git fetch upstream merge-requests/<IID>/head:mr-<IID>
git checkout mr-<IID>

Option B — checkout the source branch:

git fetch upstream <source-branch>
git checkout -B <source-branch> upstream/<source-branch>

Option C — glab:

glab mr checkout <IID> --repo redhat/hummingbird/tools

2. Edit and verify locally

Component-specific checks (example for Red Hat Catalog):

cd redhat-catalog
npm run ci-checks
npm run start:dev   # optional UI smoke test

3. Push to the author’s branch

If your local branch name matches the MR source branch:

git push upstream <source-branch>

If you checked out via mr-<IID> (local name differs from remote), push explicitly:

git push upstream HEAD:<source-branch>

Example: local branch mr-748, remote source branch feat/image-request-form:

git push upstream HEAD:feat/image-request-form

Optional — rename locally so future pushes are simpler:

git branch -m mr-<IID> <source-branch>
git push -u upstream <source-branch>

The existing MR updates; CI re-runs; no second MR is needed.

What runs on each MR push

For redhat-catalog/** changes, see Red Hat Catalog Environment Promotion — Pipeline Flow. Summary:

Stage Result
Test Build, lint, type-check, unit tests, audit, SAST
Visual Mock visual diff (MR head vs merge-base)
Preview GitLab Pages at /mr-{IID}
Status Internal MR note with Hummingbird dashboard link
Agent Code review on open/update; /hummingbird analyze-failures on CI fail

After merge to main, staging deploys automatically; production requires manual gate and UAT sign-off.

Troubleshooting

error: src refspec <branch> does not match any

Git has no local branch with that name. You are probably on mr-<IID> while pushing git push upstream feat/....

Fix: push the current branch to the remote source branch:

git push upstream HEAD:<source-branch>

Push rejected / permission denied

  • Confirm Developer+ access on redhat/hummingbird/tools
  • Confirm you are pushing to upstream, not a personal fork
  • Check whether the MR author enabled “Prevent pushing to source branch” (uncommon)

fatal: couldn't find remote ref merge-requests/.../head

Fetch from upstream, not origin:

git fetch upstream merge-requests/<IID>/head:mr-<IID>

Team tips

Tip Why
Assign yourself when taking over pushes Clear ownership on the MR
Use MR Pages preview Review UI without local dev
Use dashboard link in MR internal note Per-commit Konflux/build status
Use GitLab suggestions Small fixes without a local checkout
Use experiment/<name> for long exploration Live host before opening an MR

4.5.42 - K8s Integration Tests

Smoke tests that verify tools container images work in a real Kubernetes cluster. Each test creates short-lived pods, checks expected behavior, and cleans up. Tests run automatically on every MR via the Konflux K8s test pipeline.

Test Runner

Tests are executed by ci/run_tests_k8s.sh. For each component it:

  1. Reads {component}/tests-k8s.yml
  2. Parses each test entry (YAML -> JSON via Python)
  3. Runs the command block with kubectl wired to the target cluster
  4. Cleans up labeled resources (hum-k8s-test=<run-id>) after each test

Running Locally

Test against a local cluster (kind, minikube, or remote):

# Single component with a published image
IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \
IMAGE_NAME=gitlab-ci--tools \
  ci/run_tests_k8s.sh --context kind-kind gitlab-ci--tools

# Single component with a locally-built image
podman build -t localhost/hummingbird-dashboard:dev hummingbird-dashboard/
kind load docker-image localhost/hummingbird-dashboard:dev
IMAGE_URL=localhost/hummingbird-dashboard:dev \
IMAGE_NAME=hummingbird-dashboard--tools \
  ci/run_tests_k8s.sh --context kind-kind hummingbird-dashboard--tools

CI Integration

In Konflux, the tools-k8s-test IntegrationTestScenario triggers on every MR. The scenario is config-driven: it is declared in konflux-templates/scenarios.yml and rendered into konflux-templates/rendered.yml. The tools-k8s-test and tools-conforma IntegrationTestScenario CRs are deployed to the Konflux cluster by the infrastructure tools-releases sync, which fetches konflux-templates/rendered.yml from main and runs on every merge; they are intentionally not deployed from MR pipelines, because ITS is static repo-level config that must not change between MRs (concurrent MR pipelines would otherwise reset the live CR to stale versions). The pipeline:

  1. Checks whether tests-k8s.yml exists for the changed component
  2. Provisions an ephemeral EaaS namespace
  3. Runs ci/run_tests_k8s.sh with the built image

Components without tests-k8s.yml skip the test (the pipeline exits early after the check step).

When an MR touches multiple components of the same application, Konflux runs per-component PipelineRuns (one per component snapshot) and a group PipelineRun (one for the PR-group snapshot, created after all per-component builds in the application complete). The group run invokes the script once with --group-component-name (repeated) and per-component IMAGE_URL_<COMPONENT> env vars; per-component runs use the single-component IMAGE_URL/IMAGE_NAME pair. --group-component-name is handled identically to --component-name. An MR touching fewer than two components of the same application gets no group run (Konflux creates a group snapshot only when ≥2 components of the same application are on the same source branch).

Adding Tests for a New Component

  1. Create {component}/tests-k8s.yml with one or more named tests:

    ---
    smoke-test:
      command: |
        name="${TEST_GROUP}-smoke-${TEST_RUN_ID}"
        kubectl run "${name}" --image="${TEST_IMAGE:?}" --restart=Never \
          --labels="${TEST_RUN_LABEL}" \
          --command -- echo "hello"
        kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \
          "pod/${name}" --timeout=120s || test_fail "Pod did not succeed"
        kubectl logs "${name}"
    
  2. Use ${TEST_IMAGE} for the image under test, ${TEST_RUN_ID} and ${TEST_RUN_LABEL} for unique naming and cleanup.

  3. Call test_fail "message" to fail a test with a clear message.

  4. Keep tests fast (under 2 minutes) — these are smoke tests, not integration suites.

Common pitfalls

  • All lines in command: | block scalars must be indented relative to the command key. Unindented lines break YAML parsing.
  • Use command -v instead of which to check for commands — which is not available in all container images.
  • For multi-statement Python one-liners, use semicolons on a single line: python3 -c 'import foo; import bar; print("ok")'

Debugging Failures

See the EaaS and Debugging guide for accessing ephemeral namespaces and using Kubearchive for historical PipelineRun data.

4.5.43 - Trigger Manifests

Trigger manifests are declarative JSON files committed alongside each Hummingbird source repo. They map changed file paths to the Konflux build PipelineRuns expected to fire, and list the IntegrationTestScenario CRs expected to run after a successful build. The workqueue service reads them to compute build and test expectations for open MRs; any other tooling that needs expectations can read the same files.

The same manifest concept is generated by the containers, tools, and rpms repos.

Location

Each repo carries two manifests:

  • .tekton/trigger-manifest-pull-request.json — PR (pre-merge) event
  • .tekton/trigger-manifest-push.json — push (post-merge) event

Both are kept committed and regenerated by the repo’s own generator:

  • tools — ci/internal/render_template.py (make all)
  • containers — ci/internal/generate_pac_resources.sh (make global-resources, which is also a prerequisite of make all-host)
  • rpms — ci/generate_resources.py (make generate-host, or make generate to run inside the build container)

The tree_status CI job enforces that the committed files match the generator output on every pipeline, so the files are always fresh.

Schema

Both manifests share the same top-level structure:

{
  "type": "push",
  "excludes": [],
  "applications": { ... },
  "integration_test_scenarios": { ... }
}

type

String. Either "push" or "pull_request". Identifies which event class the manifest covers.

excludes

Array of path regex strings. Any changed path matching one of these patterns suppresses triggering entirely, regardless of the applications section. Tools and rpms carry an empty array; containers carries exclusion patterns for image-documentation files (e.g. README.md.j2, report.yml, Containerfile.j2, properties.yml) that do not affect builds.

applications

Object keyed by Konflux application name. Each value is an object keyed by component name, whose value is an array of path regex strings:

"applications": {
  "tools": {
    "workqueue-service--tools": ["^workqueue-service/"],
    "hummingbird-agent--tools": [
      "^hummingbird-agent/",
      "^hummingbird-ai/",
      "^hummingbird-sandbox/"
    ]
  }
}

A component is considered affected by a commit when at least one of its trigger regexes matches at least one changed path. An application with no affected components produces no build expectations for that commit.

integration_test_scenarios

Object keyed by application name. Present in a manifest only when the repo has integration-test scenarios applicable to that event type; an absent key is treated as “no test expectations” by consumers (backward-compatible with old SHAs before scenarios were added).

Concretely: the push manifest carries integration_test_scenarios only when the repo has component-context scenarios (they fire on individual component snapshots post-merge). Tools defines one such scenario (conforma), so its push manifest includes the section. Rpms defines scenarios but all are pull_request-context, so its push manifest has no integration_test_scenarios key. Containers currently defines no scenarios, so neither manifest includes the section.

The shape differs between the two manifests:

PR manifest (.tekton/trigger-manifest-pull-request.json):

"integration_test_scenarios": {
  "tools": [
    { "name": "tools-conforma", "group": false },
    { "name": "tools-k8s-test", "group": true }
  ]
}

Push manifest (.tekton/trigger-manifest-push.json):

"integration_test_scenarios": {
  "tools": [{ "name": "tools-conforma" }]
}

Each entry:

Field Type Present in Meaning
name string both manifests The IntegrationTestScenario CR name. Matches the test.appstudio.openshift.io/scenario label on test PipelineRuns.
group boolean PR only true for pull_request-context scenarios, which also trigger a group-snapshot run when ≥2 components are affected. false for component-context scenarios.

Scenarios marked optional: true in konflux-templates/scenarios.yml are excluded from both manifests. component-context scenarios appear in both manifests (they fire on individual component snapshots). pull_request-context scenarios appear only in the PR manifest (no group snapshots exist post-merge).

Consumer semantics

For each application with ≥1 affected component, every listed scenario must have ≥1 observed test PipelineRun and all observed runs must be green. For group: true entries, an additional group-snapshot run is required when the application has ≥2 affected components. Scenario names are matched against the test.appstudio.openshift.io/scenario labels on test PipelineRuns.

See the workqueue service documentation for the full consumer-side implementation.

Generation

scenarios.yml schema

Each repo that owns IntegrationTestScenario CRs carries a konflux-templates/scenarios.yml matrix file. This is the canonical schema documentation — the file itself carries no inline comments (it is machine-managed via PyYAML round-trip; this page is the schema reference).

applications:
  <application>:
    - template: <type>
      suffix: <name>
      triggers:
        - pull_request
        - component
      optional: true
      vars:
        KEY: value

Entry fields:

Field Required Description
template yes Scenario type. Maps to scenarios/<template>.yml.j2 and becomes the CR name suffix: <application>-<template>.
suffix no CR-name override. When set, CR name = <application>-<suffix> instead of <application>-<template>.
triggers yes* Konflux trigger context names (pull_request, component). Unknown or empty values on non-optional entries fail the generator. *Optional entries may omit it.
optional no If true, the scenario is excluded from both trigger manifests.
vars no Per-instance param overrides passed to the template, rendered after the template’s default params.

All CR content — params, resolverRef, Renovate markers, timeout annotations — lives in the per-type template file (konflux-templates/scenarios/<template>.yml.j2). The matrix holds only the instantiation relationships: application × template × triggers × overrides. Unknown or empty triggers on non-optional entries cause the generator to fail hard, so the manifests are always consistent with the deployed IntegrationTestScenario CRs.

Component-to-path mappings

The component-to-path mappings in the applications section are derived differently per repo:

  • tools — each component’s Containerfile location, with an optional .build-context override file for extra trigger paths.
  • containers — images/*/properties.yml files aggregated with shared configuration from images/variables.yml (distros, variants, hermetic settings).
  • rpms — rpms/<package>/ directories, with PR-time additions from test/rpms/<package>.yml per-package test specs.

To regenerate both manifests after editing scenarios.yml or adding a component:

make all

5 - Agentic SDLC

How Project Hummingbird uses agentic AI to drive velocity across the software development lifecycle.

Project Hummingbird’s core mission is velocity: shipping RPM updates, CVE fixes, and new container images as fast as possible. The pipeline already delivers at a scale unprecedented within Red Hat — consistently over 1,000 automated commits per week across the containers and rpms repositories, a throughput that would be impossible to sustain manually — work that in most Linux distributions is still performed manually by dedicated teams. Managing 400+ packages and 1,750+ image builds with a 24-hour security SLO, that pipeline is not a nice-to-have — it is what makes the project viable with a small team.

The pipeline solves throughput. What it does not solve is the full software development lifecycle: assessing what to build, implementing it, documenting it, testing it, reviewing UX, coordinating releases. For a small team maintaining hundreds of containers and building an entire OS, staffing all of those functions manually does not scale — and agentic AI is what changes that equation. Where rule-based automation handles the deterministic, agents take on everything that has historically required a human to pick up, think through, and act on. Hummingbird is building toward becoming an open source reference implementation for responsible agentic AI in software engineering: demonstrating through production use, not promises, that this kind of automation can be adopted incrementally and grounded in engineering discipline.

The Agentic SDLC Vision

The target state is a development pipeline driven by agents at every stage — with humans engaged where their judgment is needed, not where their time is consumed by routine work.

Every change flows through the same lifecycle:

1. Request. A customer, product team, or automated trigger submits a change: a new container image, an RPM update, a feature request, a bug fix.

2. Analysis. A product management agent assesses the request — evaluating feasibility, priority, and fit against existing components and conventions. For straightforward requests it proceeds automatically; for complex or ambiguous ones it surfaces the relevant context and flags the case for human input before continuing. This is the work that normally sits in a product manager’s queue.

3. Implementation. The request splits based on its nature:

  • Deterministic changes — dependency updates, lockfile syncs, rebuilds triggered by upstream changes — are implemented by rule-based automation and merged automatically once CI passes. No ticket, no approval, no delay.
  • Everything else is handled by an engineering agent: a developer co-programming with an agent, or an agent working autonomously to write the code or configuration, run validation, and open a merge request. Agentic AI is the default answer for everything that has historically required a human to pick it up, think through it, and act on it.
  • Specialist agents contribute alongside: a tech writer agent drafts or updates the documentation and long-form Red Hat product content; a testing agent generates or extends tests; and a user design agent applies UX patterns and practices to any interface-related changes.

4. Review. Automated agents check for policy compliance and code quality, including the production code review agent. The same agent capabilities that contribute during implementation also apply here: a tech writer agent reviews documentation changes for clarity and correctness; a testing agent reviews test coverage and flags gaps; and a user design agent reviews UX-related changes against established patterns. A human engineer makes the final call on all of it. No agent-produced work is merged without human approval — the human is not eliminated, but elevated to the role where judgment actually matters.

5. Merge. The change ships.

The goal is maximum automation, pursued pragmatically rather than dogmatically — the right tool for each kind of work. Where agentic AI excels is in the space just before human judgment is required: gathering context, surfacing options, resolving the routine, so that when a human does engage, the decision in front of them is sharp, informed, and fast.

Responsible AI

Agent-produced code and decisions require human approval before merge. This is not a temporary constraint — it is a design principle. Humans may stay in the loop to steer or closely supervise complex work when that helps, but the standing requirement is approval — not constant supervision of every agent run. The feedback between human judgment and agent behavior is what keeps the system trustworthy over time.

The high-velocity automated pipeline — over 1,000 commits per week — operates through rule-based automation and does not involve agents; no human review is required there. The failure analysis agent does run at high volume — triggering on every pipeline, build, or test failure — but it produces analysis and comments, not code changes requiring approval. That alone saves hundreds of minutes per week of manual log triage. Agent work that requires human approval is scoped to SDLC tasks: feature requests, documentation, code review — a much lower volume where the requirement is entirely manageable.

Security

Agents run in containers — locally via Podman, in production on OpenShift. This is not a novel security model: it is the same approach the industry has long used for untrusted code, applied consistently. The Hummingbird agent platform is architecturally designed to ensure agents have no access to secrets or tokens. That single constraint effectively neutralizes prompt injection as a threat: an agent that cannot access credentials or act on external systems is, in practice, just another contributor — and the review process treats it as such. The same gatekeeping that applies to any third-party contribution applies to agent-produced work. The worst-case outcome is a low-quality merge request — and that is precisely what the review process, including the review agents themselves, is designed to catch.

Learning from Feedback

Agents learn from feedback — and this is already working in production. When a reviewer comments on an agent-produced merge request, the agent reads those conversations, incorporates the feedback, and updates its work. It avoids re-raising issues already addressed and respects when a developer has explained an intentional design choice. That feedback is also preserved — stored so that future agent runs can draw on it, improving with each iteration rather than repeating the same mistakes — an approach grounded in the frameworks established in Pascal Bornet’s Agentic Artificial Intelligence.

From Pair Programmer to Autonomous Agent

Deploying production agents requires a foundation. The approach that works is not to automate first and document later — it is the reverse.

Phase 1 — Agent as pair programmer. Agents assist individual developers: drafting code, explaining unfamiliar systems, proposing changes for review. This phase builds understanding of what agents need: context, conventions, constraints.

Phase 2 — Documentation as fuel. Agent output quality is directly proportional to documentation quality. Every repository carries an AGENTS.md — a file with instructions targeting agents specifically, but one that mostly points to other documentation rather than duplicating it. That underlying documentation is written for humans and agents alike: clear, specific, and rich with rationale. The AGENTS.md is the entry point; the broader docs are the substance. An agent with good documentation reasons from the same foundation as the team; an agent without it guesses.

Phase 3 — Agents in production. With the foundation in place, agents move into the critical path: first as reviewers proposing concrete changes, then as autonomous contributors on well-defined tasks. The next section describes what that looks like in practice for Hummingbird today.

An agent running autonomously in production is not fundamentally different from a developer running an agent during pair programming. The same agent, the same context, the same documentation — the difference is who initiates the run and how much human supervision follows. If an agent consistently does good work when a developer is in the loop, that is a strong signal the project is ready to let it operate more independently. The quality of pair programming performance is the leading indicator for autonomous deployment readiness. Trust is built incrementally, one well-handled task at a time.

See From Pair Programmer to Autonomous Agent by Valentin Rothberg for the full treatment.

Documentation as Infrastructure

Writing Documentation with Agents

In an agentic system, documentation is infrastructure — the primary interface between the team and its agents, and the quality of what an agent produces is bounded by the quality of what it can read. That makes writing documentation with agent assistance not a separate task but a natural extension of the work: the same agent that helps an engineer write and review code can help write the documentation for it — capturing decisions, explaining rationale, drafting contribution guides. This is not a shortcut; it is the same pair-programming model applied to a different kind of output. This is how Hummingbird built its documentation from the start: engineers worked alongside agents to draft and improve it in the same way they wrote code — iteratively, collaboratively, with humans in the lead. The result is a virtuous cycle: better docs make agents more capable, and more capable agents help produce better docs.

Tech writers are not a prerequisite — engineers writing clear, intentional documentation with agent assistance is sufficient to start. But their skills become more valuable in this model, not less: precision over generality, rationale alongside rules, explicit anti-patterns. A tech writer contributing to an agentic codebase is directly shaping the behavior of the agents that work within it.

In Hummingbird, this is not theoretical. The containers repository is developed in active collaboration with Red Hat tech writers, whose customer-facing product documentation lives in the same repository as the code — keeping internal conventions, agent context, and customer-facing content in one place, shared across engineers, tech writers, and agents alike.

Knowledge Transfer Across Teams

The same principle — knowledge encoded in markdown, readable by humans and agents alike — also answers a broader question: how does one team’s expertise become available to agents maintained by another? The answer is the same as everything else in this system — markdown. A UX team encodes their patterns and design practices in a skills file; the user design agent picks it up. The documentation team maintains their style guide, standards, and best practices — including Red Hat’s documentation guidelines — encoded once and available to every agent that produces or reviews written content. No code changes, no integrations, no handoff meetings. Any team can contribute their domain knowledge to the agentic system simply by writing it down in a form that both humans and agents can read. The markdown file is the interface between teams.

Agents in Production Today

Two agents are running in production today, built on the Hummingbird Agent platform. They operate on every merge request across the containers and rpms repositories, which together see over 1,000 automated commits per week — a throughput that would be impossible to review and maintain manually without agent assistance.

More agents are under active development in parallel, with production deployment planned after Red Hat Summit 2026 — likely by end of Q2. One example is the merge conflict agent — which will automatically resolve merge conflicts that arise during automated RPM updates. The full roadmap is tracked in the HUM-687 epic.

Failure Analysis Agent

When a merge request pipeline fails, this agent automatically investigates and posts its findings directly on the merge request. It pulls data from multiple sources — GitLab CI job logs, Konflux build and test pipeline runs, Testing Farm test results — and correlates them to identify root causes. Where the same underlying issue causes multiple failures across different images or architectures, it groups them so engineers see patterns rather than an overwhelming list of individual failures. Every finding includes links to the specific logs and artifacts so engineers can dig deeper without having to hunt.

What previously required clicking through multiple levels of web UI, downloading logs, and working through thousands of lines of output to find a single root cause now happens automatically. It is not rocket science — but it is some of the most time-consuming and draining work an engineer can be asked to do repeatedly.

Workflow definition: analyze-failures.md

Code Review Agent

When a merge request is opened, this agent performs a technical code review with the priorities of a senior engineer: security first, then correctness, then performance, then maintainability. It provides specific line references and concrete code examples for every issue it raises — not vague suggestions, but actual fixes. It reads prior review discussions before posting, avoids re-raising issues already addressed, and respects when a developer has explained an intentional design choice.

The result is consistent, high-quality review coverage on every merge request, with human reviewers free to focus on the issues that genuinely need their judgment.

The team is already pleased with the quality of the reviews — and this is only the beginning. A false positive costs nothing: a human reviewer reads it, disagrees, and moves on. The design ensures that bad agent output has no blast radius. And as human reviewers comment on agent-produced work, the agent learns and incorporates that feedback — improving with every iteration.

Workflow definition: code-review.md

The near-term impact is measured in cycle time. A feature request that previously took weeks to move from intake through product assessment, implementation, documentation, and review can be compressed to days, sometimes hours. That changes what a small team can realistically take on. The longer-term picture is equally straightforward: the system already delivers value today. If agents continue to improve, Hummingbird benefits more. If they plateau, the team continues to operate at a level that would otherwise require far more people. The infrastructure built now is not a bet on a technology that might not materialize — it is an investment that is already paying off, with optionality for more.

6 - Architecture Decision Records

Architecture Decision Records (ADRs) capture significant technical decisions made across the Hummingbird project, along with their context and consequences.

Records

Implemented decisions are also accepted. An acceptance date is listed only when recorded in the ADR.

ID Title Status Proposed Accepted
ADR-0001 Unified Observability Stack Implemented 2026-07-19 Not recorded
ADR-0002 AI-Assisted SDLC Workqueue Proposed 2026-08-24 Pending
ADR-0003 RPM metadata and versioning Accepted 2026-08-31 2026-11-17
ADR-0004 Move private_product into package metadata Accepted 2026-11-05 2026-11-18

6.1 - ADR-0001: Unified Observability Stack

  • Status: Implemented
  • Date: 2026-07-19 (updated 2026-08-19)
  • Author: Robert Sturla
  • Jira: HUM-4790 (Epic), HUM-4791 (Design task)

Context

Hummingbird had no single place to view the health of builds, infrastructure, and services. Multiple disconnected monitoring and alerting implementations existed — CloudWatch dashboards, a custom cloudwatch-log-forwarder Lambda, an SNS AlertsTopic for consumer CloudWatch alarms, and per-service custom dashboards. Workloads span multiple clusters with no cross-cluster visibility. This created blind spots, duplicated effort, and inconsistent alerting behaviour.

One standardised observability stack is needed. No duplication. One toolchain, used everywhere, federated across clusters from day one.

Operational runbooks live in documentation/monitoring.md in the infrastructure repository. This ADR records the architecture decision.

Cluster Constraints

Hummingbird has no cluster-admin access on any cluster. All resources must be deployed within tenant-provisioned namespaces. Operator CRDs and RBAC grants are controlled by the cluster platform team — what is available differs between clusters.

This constraint drives a hub-spoke split architecture: the hub (MPP) runs the backends and Grafana, while spoke clusters run lightweight collectors that forward all telemetry to the hub.

Decision

Stack

Hub (MPP, hummingbird--monitoring-hub):

Signal Backend Storage
Metrics Mimir (monolithic) S3 via ObjectBucketClaim
Logs Loki (monolithic) S3 via ObjectBucketClaim
Collection Alloy Hub (YACE + textfile) N/A
Visualisation Grafana N/A
Alerting Mimir ruler + Alertmanager N/A

Spoke (every cluster with hummingbird--monitoring):

Signal Backend Storage
Metrics Alloy prometheus.scrape → hub Mimir remote-write N/A
Logs Alloy loki.source.kubernetes → hub Loki N/A
Collection Alloy + kube-state-metrics N/A
Visualisation None (use hub Grafana) N/A

Spoke clusters have no local backends. All storage, alerting, and visualisation is centralised on the hub. Each spoke forwards to both the staging and production hubs.

Distributed tracing (Tempo) is not deployed. See Future Work.

Alerting Architecture

Mimir evaluates PromQL alerting rules (ruler) and delivers notifications through its built-in Alertmanager. Grafana is not in the evaluation path.

Mimir ruler (PromQL) → Mimir Alertmanager → Slack / email
  • Rules are Git-provisioned in the infrastructure repository and mounted into Mimir as a ConfigMap.
  • All alerting runs on the hub. Spoke clusters do not run Alertmanager. Spoke alerting depends on network connectivity to the hub — if a spoke loses connectivity, alerts for that cluster are delayed until the connection is restored. This is an accepted tradeoff given the operator constraints on spoke clusters (see Cluster Constraints).
  • Staging Alertmanager receivers are empty (a blackhole) so staging does not page Slack or email.

Alert Routing

Severity Channel
critical Slack + email
info Slack
everything else Slack + email

Production only; staging is a blackhole.

Deployment Model

The hub runs the backends, Grafana, and Alloy Hub. Spoke clusters run Alloy and kube-state-metrics.

Hub (MPP):

  • Namespace: dedicated hummingbird--monitoring-hub namespace (tenant-provisioned)
  • Tenant-scoped: all resources deployed in namespaces owned by Hummingbird, no cluster-admin dependency
  • Mimir (monolithic): accepts Prometheus remote-write from spoke Alloy and Alloy Hub, stores TSDB blocks in S3, serves PromQL, and runs the ruler and Alertmanager. Grafana queries Mimir in-cluster (no nginx basic auth on the Service port).
  • Loki (monolithic): receives logs from spoke Alloy via authenticated Routes. Grafana queries Loki in-cluster.
  • Alloy Hub: a dedicated Alloy instance in the hub namespace. CloudWatch metrics are ingested with prometheus.exporter.cloudwatch (YACE), filtered to Hummingbird resources (app-code=RPRM-001). Credential-expiry metrics are ingested with the textfile collector from a CI-generated ConfigMap. Alloy Hub remote-writes to in-cluster Mimir (Service DNS; no Route or CA bundle).
  • Grafana: exposed via an internal Route. Authentication is Generic OAuth against gitlab.com (redhat/hummingbird group). GitLab Owner and Maintainer map to Grafana Admin, Developer to Editor, everyone else in the allowed group to Viewer. Datasources and dashboards live in grafana_data/ and are synced with grafana_data/deploy.sh. Grafana also has a CloudWatch datasource for ad-hoc browsing of AWS logs and metrics; that path is query-only and is not used for alerting or pod-log forwarding.
  • External access: Loki and Mimir Routes sit behind an nginx sidecar with HTTP basic auth. In-cluster Service access (Grafana, Alloy Hub) does not require auth.

Spoke (every cluster):

  • Namespace: hummingbird--monitoring
  • Alloy: collects container logs via loki.source.kubernetes and scrapes prometheus.io/scrape Services, including kube-state-metrics. Forwards logs to hub Loki and metrics to hub Mimir. Dual-writes to staging and production hubs.
  • kube-state-metrics: Kubernetes object metrics (deployments, pods, jobs, PVCs, resource quotas) with a curated allowlist.
  • No local Mimir, Loki, Grafana, or Alertmanager.

AWS Lambda Services

Some Hummingbird services (e.g. container-catalog) run as AWS Lambda functions. Observability for those services uses the existing AWS signals rather than an OpenTelemetry Lambda extension:

  • Metrics: Alloy Hub YACE scrapes curated AWS/Lambda metrics (Invocations, Errors, Duration, Throttles) into Mimir. Mimir alerting rules cover error rate and throttling.
  • Logs: Grafana queries CloudWatch Logs directly via the CloudWatch datasource. Pod logs are not involved; Lambda logs stay in CloudWatch.
  • Low-traffic utility Lambdas are covered by the same YACE allowlist when tagged app-code=RPRM-001.

Log Collection

Container stdout/stderr is tailed by spoke Alloy using loki.source.kubernetes (Kubernetes API; no DaemonSet, no sidecar, no ClusterLogForwarder). Logs are labeled with cluster, namespace, pod, container, and app, then dual-written to hub Loki.

This requires no application code changes. ClusterLogForwarder is available on some clusters but is not used.

High Availability

Hub:

Component Replicas Rationale
Mimir 1 Monolithic; brief downtime acceptable
Loki 1 Monolithic; brief downtime acceptable
Grafana 1 View-only; brief downtime acceptable
Alloy Hub 1 Stateless scrape; gap until next scrape is OK

Spoke:

Component Replicas Rationale
Alloy 1 Stateless forwarder; brief gap OK
kube-state-metrics 1 Stateless; brief gap OK

Spoke clusters have no stateful observability components. HA concerns are concentrated on the hub. Single-replica monolithic Mimir and Loki are an accepted tradeoff against the proposed Prometheus + Thanos HA pair: there is no existing Prometheus fleet to retrofit, and Alloy already remote-writes.

Retention

Environment Mimir Loki
Production 90d 90d
Staging 14d 14d

S3 is provisioned with ObjectBucketClaim. There is no separate warm/cold tier or multi-year archive.

Operator Availability

CRD availability was audited on 2026-08-09. Hummingbird has no cluster-admin access — only operator CRDs explicitly granted to tenant namespaces are usable. That audit is why this stack does not use Cluster Observability Operator (MonitoringStack / Thanos), LokiStack, TempoStack, ClusterLogForwarder, or an OpenTelemetry Collector as the live path:

  • LokiStack and TempoStack are not available to tenant namespaces on any cluster. Loki is a standalone Deployment. Tempo is deferred.
  • COO MonitoringStack is available on MPP but not on spoke, and would have required Prometheus + Thanos for a fleet we do not have. Mimir accepts remote-write directly from Alloy.
  • ClusterLogForwarder is available, but Alloy loki.source.kubernetes collects the same pod logs without a second operator.
  • OpenTelemetry Collector was deployed earlier and removed once Alloy wrote directly to Loki and Mimir.

Relationship to Existing Dashboard

The existing hummingbird-dashboard app serves two distinct roles:

  1. Data visualisation — CVE status charts, failed release counts, build metrics
  2. Operational workflows — click to rerun a failed release, interactive triage, actionable controls

This observability stack takes over data collection, storage, and visualisation (role 1). Grafana is the primary place for charts, time-series dashboards, and alert-driven visualisations. The dashboard app retains operational workflow features (role 2) — interactive actions that Grafana cannot provide.

The two systems complement each other:

  • The dashboard app can embed Grafana panels for richer visualisation without reimplementing charting
  • The dashboard app can query the same backends (Mimir, Loki) directly for data it needs to drive workflows
  • Over time, the dashboard app consumes data from the unified backends instead of maintaining parallel collection

Alerting (e.g. failed release Slack notifications) moves to Alertmanager. The dashboard app no longer needs its own alerting path.

What This Replaces

Previous Replaced by
cloudwatch-log-forwarder Lambda Spoke Alloy → hub Loki
SNS AlertsTopic + consumer CloudWatch alarms Mimir ruler + Alertmanager
Per-service custom dashboards Grafana dashboards
Dashboard app data collection Unified backends (Mimir, Loki)
Dashboard app alerting (Slack) Alertmanager
Dashboard app visualisations Grafana (embeddable in the dashboard app)

The dashboard app itself is not decommissioned — its operational workflow features (release reruns, interactive triage) remain. Only its data collection and visualisation responsibilities shift to the unified stack.

hummingbird-events-topic is the live event bus. It was not an alerting channel and is not replaced by this stack.

Consumer CloudWatch alarms and the log-forwarder were removed after the Mimir/Loki path was in place. Remaining CloudWatch use is intentional:

  • Grafana CloudWatch datasource for ad-hoc AWS log and metric browsing
  • CVE error-budget SLO dashboards and hummingbird-slo-alerts (no Grafana port yet)

Operations

Day-to-day endpoints, datasources, query examples, and onboarding a new cluster are documented in documentation/monitoring.md in the infrastructure repository.

Future Work

  • Distributed tracing: Tempo (or equivalent) for request and pipeline traces. Not deployed; traces are not a current signal.
  • CVE lifecycle tracing: instrument CVE analysis, RPM build, and container image pipelines with spans. Trace a CVE from detection → RPM fix → image rebuild → publish. Requires a trace backend and a schema (separate story).
  • Spoke local alerting: if operator access on spoke clusters changes, a local evaluation path could remove the hub dependency for spoke alerts. Not required today.

Alternatives Considered

Alerting: Grafana-only vs Alertmanager-only vs hybrid

  • Grafana-only: less mature routing, dedup, and silencing.
  • Hybrid Grafana + Alertmanager (original proposal): Grafana would evaluate rules across metrics, logs, and traces, then forward to Alertmanager. That is not what shipped: there is no Tempo, and log alerting is not required yet.
  • Mimir ruler + Alertmanager (chosen): PromQL rules and delivery in one component. Matches the metrics-first rollout. Applications can still POST to the Alertmanager API later if needed.

Metrics backend: Prometheus + Thanos vs Mimir

Prometheus + Thanos (MonitoringStack) was the original hub design. It fits retrofitting an existing Prometheus fleet. Hummingbird had no such fleet — Alloy already remote-writes. Mimir is one component for receive, storage, PromQL, ruler, and Alertmanager, instead of Prometheus + Thanos Sidecar + Store Gateway + Querier + a separate Alertmanager.

Logs: Sumo Logic vs Loki

Red Hat has a company-wide Sumo Logic license. However, the Sumo Logic Grafana plugin requires Grafana Enterprise or Grafana Cloud — it is not available in open-source Grafana. Loki integrates natively with open-source Grafana and keeps logs on-cluster.

Log collection: ClusterLogForwarder vs Alloy

ClusterLogForwarder is available on hub and spoke. Alloy loki.source.kubernetes collects the same pod logs and already runs on every spoke for metrics, so a second operator was not justified.

Spoke architecture: full stack vs collector-only

  • Full stack per cluster: each cluster runs its own backends and Grafana. Local alerting survives network partitions. Requires operator CRDs Hummingbird does not have on spoke.
  • Collector-only on spoke (chosen): Alloy + kube-state-metrics forwarding to the hub. Simpler spoke footprint. Tradeoff: spoke alerting depends on hub connectivity.

Lambda: OTel extension vs CloudWatch ingest

An OpenTelemetry Collector Lambda extension would export OTLP to the hub. That needs a network path from Lambda to MPP and a collector that no longer exists. YACE metrics into Mimir plus Grafana’s CloudWatch datasource for logs reuse AWS signals already produced by the functions.

HA: all components vs selective vs single replica

Full HA doubles cost with little benefit for view-only tools. The original proposal kept HA only on Prometheus and Alertmanager. The as-built stack uses single-replica monolithic Mimir and Loki; Alertmanager is in-process with Mimir. Brief downtime is accepted.

Consequences

  • Team must learn Grafana dashboarding and PromQL/LogQL basics
  • New hub services should expose Prometheus metrics (scrape annotation) and may add a Grafana dashboard under grafana_data/
  • New spoke services must be scraped by Alloy (scrape annotation) and emit logs to stdout/stderr
  • Spoke alerting depends on hub connectivity — no local alerting on spoke
  • Deploying a new hub cluster requires Mimir, Loki, Grafana, and Alloy Hub, plus ObjectBucketClaims
  • Deploying a new spoke cluster requires a hummingbird--monitoring namespace, Alloy, and kube-state-metrics pointing at both hubs
  • CloudWatch remains for Lambda logs (Grafana datasource) and CVE SLO error-budget dashboards; it is not used for pod-log forwarding or consumer alarms

6.2 - ADR-0002: AI-Assisted SDLC Workqueue

  • Status: Proposed
  • Date: 2026-08-24
  • Author: Michael Hofmann
  • Jira: HUM-6342 (spike)

Context

The containers and rpms repositories produce over 1,000 automated commits per week. The automation that drives this throughput is spread across multiple independent mechanisms: GitLab CI scheduled jobs running shell scripts, Renovate CronJobs in the infrastructure repo, AWS Lambda functions for MR approval, and manual developer workflows run from dev machines. Each mechanism creates and manages MRs with its own lifecycle, credentials, and error handling.

There is no unified system that tracks an MR from creation through approval to merge. The Hummingbird Agent already runs AI-powered failure analysis and code review on every MR, but cannot create or modify MRs. The security foundation for agent-authored MRs is designed in HUM-851 but not yet implemented.

This ADR documents the current state of MR automation across all repositories and entrypoints as discovery for a spike to design a unified workqueue-based system.

Scope

This document inventories what exists today. It does not prescribe the target architecture — that is the output of the spike. Design questions and patterns are included to inform the spike investigation.

Current State

Repositories

rpms (redhat/hummingbird/rpms): RPM package definitions. Each package lives in its own directory with a .spec file and metadata/<package>.json for upstream repo URL, CVE product mapping, and modification status. Fedora dist-git imports and upstream version bumps are the primary automated flows.

containers (redhat/hummingbird/containers): Container image definitions. Each image lives in images/<name>/ with Containerfiles and RPM lockfiles (rpms.lock.yaml). Lockfile refresh and Renovate dependency updates are the primary automated flows.

tools (redhat/hummingbird/tools): Lambdas, CLI tools, and the Hummingbird Agent framework. Contains mr-auto-approver, gitlab-event-forwarder, hummingbird-agent, and the CVE analysis pipeline.

infrastructure (redhat/hummingbird/infrastructure): Kubernetes manifests, AWS SAM templates, and deployment configuration. Hosts Renovate CronJob definitions and agent deployment manifests.

MR Creation Entrypoints

Entrypoint Repo Trigger What it creates
dist_git_update_multi_mr.sh rpms GitLab schedule (every 4h) One MR per package needing Fedora dist-git sync
upstream_update_multi_mr.sh rpms GitLab schedule One MR per package with a newer upstream release
rebuild_multi_mr.sh rpms Manual (operator-triggered) One MR per package needing rebuild
create_lockfile_update_mrs.sh containers GitLab schedule One MR per image with changed RPM lockfiles
ci/create_mr.sh (metrics) containers GitLab schedule (daily) Single chore MR for metrics report
Renovate (rpms) rpms K8s CronJob (hourly) Tekton pipeline digest and Konflux release-catalog updates only
Renovate (containers) containers K8s CronJob (hourly) Dependency updates, lockfile maintenance, pipeline migration
Renovate (tools/infra/k8s-test) tools/infra/k8s-test K8s CronJob (hourly) Dependency updates
CVE analysis (advisory_handler) CEE advisories K8s CronJob (30 min) Advisory MRs; handles rebase on conflicts
/cve skill rpms Manual (Cursor/Claude) Fix MRs from staged CVE analysis data in Jira
gitlab_sync tools Various File sync MRs with poll, rebase, and best-effort approve
Manual dev workflows rpms/containers Human on dev machine Package updates, CVE fixes, spec changes

MR Creation Mechanism: Git Push Options

All shell-script-based MR creation uses git push options, not the GitLab Merge Requests REST API. The push options control MR creation, title, description, labels, draft status, and merge-when-pipeline-succeeds (MWPS):

git push \
  --push-option merge_request.create \
  --push-option "merge_request.title=chore(rpms): Update foo to 1.2.3" \
  --push-option merge_request.remove_source_branch \
  --push-option merge_request.merge_when_pipeline_succeeds \
  origin HEAD:refs/heads/chore/upstream-update-foo

Approval is always a separate step — never part of the creation scripts.

MR Lifecycle and Approval Flows

GitLab Project Settings (containers and rpms)

Setting Value
Approvals required 1 (any_approver); author cannot self-approve
Pipeline must succeed Yes
Discussions resolved Required
Jira required Yes (prevent_merge_without_jira_issue)
Merge trains / merge pipelines Off (Konflux external statuses must stay blocking)
Merge method Merge commit
Auto-merge MWPS via merge_request.merge_when_pipeline_succeeds push option

Bot Identities

Role Identity Token type
Create chore MRs chore-mr Project access token
Approve chore MRs (CI) chore-mr-approval Separate project access token
Create lockfile MRs lockfile-update Project access token
Approve lockfile MRs (CI) lockfile-update-approval Separate project access token
Renovate + Konflux PAC pipelines-as-code MintMaker token
Lambda approvals Per-project tokens Separate project access tokens

Approval Paths

Three parallel approval mechanisms exist, with overlapping scope:

1. CI approval jobs — delayed jobs in each repo’s .gitlab-ci.yml:

  • rpms chore_mr_approval: gates on check_konflux_statuses.py (polls every 60s up to 2 hours for Konflux statuses to appear and succeed), then approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. Triggered for author chore-mr or pipelines-as-code on chore/* branches.

  • containers chore_mr_approval: same author/branch gate, approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. No Konflux polling. allow_failure: true.

  • containers lockfile_update_approval: triggered for pipelines-as-code on lock-file-maintenance(-vulnerability)? branches. Real when: delayed / start_in: 10 minutes. No Konflux polling.

2. mr-auto-approver Lambda — event-driven via GitLab webhooks:

GitLab webhook → gitlab-event-forwarder → SNS → mr-auto-approver Lambda

Approves only (does not set MWPS). Rejects forks. Checks Konflux statuses where configured.

Project Allowed authors Branch patterns Konflux check
containers PAC + chore-mr renovate/.*, chore/.* Yes
rpms PAC + chore-mr chore/.* only Yes
tools PAC renovate/.* (deny renovate/redhat-catalog/.*) No
infrastructure renovate token renovate/.* No

The Lambda is intended to replace CI approval jobs; both currently run in parallel for chore/lockfile paths.

3. Renovate automerge — Renovate sets automerge: true in its MR configuration, but GitLab still requires a non-author approval before merge. For rpms Renovate MRs, neither CI approval jobs nor the Lambda cover renovate/.* branches — these MRs are stuck until a human approves.

Konflux Status Gate

External commit statuses prefixed Konflux kflux-prd-rh03 / are set by Konflux PipelineRuns triggered via Pipelines-as-Code. The check_konflux_statuses.py script (rpms) polls these statuses:

  • Missing → wait (poll continues)
  • Failed → do not approve
  • Invoked/running/pending/success → approve (MWPS then waits for final green, including Testing Farm via IntegrationTestScenarios)

Lifecycle by MR Type

RPMs dist-git clean (chore/dist-git-update-<pkg>): schedule creates → MWPS at creation → GitLab CI + Konflux builds + Testing Farm → CI approval job (poll Konflux, up to 2h) + Lambda → merge. Typical: tens of minutes to hours.

RPMs dist-git conflict: same branch, but MR is created as draft with no-test label and no MWPS. Requires human conflict resolution (or claude_resolve_conflict.sh), then remove draft/no-test and re-trigger.

RPMs upstream (chore/upstream-update-<pkg>): same as dist-git clean. Uses OIDC AWS lookaside upload for tarballs.

RPMs rebuild (chore/rebuild-<pkg>): same approval path. Typically operator-triggered, not scheduled.

RPMs Renovate (renovate/…): Renovate sets automerge, but neither CI approval job nor Lambda covers renovate/.* branches on rpms → stuck until human approves. The renovate-babysit agent workflow posts a triage note but does not approve or merge.

Containers Renovate lockfile (lock-file-maintenance): lockfile_update_approval (10 min delay) + Lambda (renovate/.* with Konflux check) → Renovate automerge / MWPS. Typical: 30–90+ minutes.

Containers other Renovate: Lambda covers renovate/.*. No lockfile_update_approval unless branch matches lockfile pattern.

Containers metrics report (chore/metrics-report): daily schedule → MWPS → chore_mr_approval + Lambda.

Containers CI lockfile (rpm-lockfile-updates/<image>): schedule is currently active: false. Approval path is incomplete — CI rule matches lock-file-maintenance, Lambda does not match rpm-lockfile-updates/.*.

Shell Script Automation Detail

rpms create_mr.sh (shared helper)

Used by all rpms multi-MR scripts. Expects commits already on HEAD.

Inputs: --branch NAME (required), --title TEXT (required), optional --description, --auto-merge, --draft, --label NAME (repeatable).

Algorithm:

  1. Count commits ahead of target branch; if 0 → exit 0.
  2. Dedup: git ls-remote --heads for branch name. If present → exit 2 (“MR already exists”).
  3. Create branch, push with MR push options.
  4. Parse push stdout for MR URL; if missing → exit 1.

Exit code contract: 0 = MR created, 2 = remote branch exists (skip), 1 = failure.

Limitation: stale remote branch from a closed MR still triggers exit 2. Does not update existing MR/branch content. No --force-with-lease.

rpms dist_git_update_multi_mr.sh

Creates one MR per package needing Fedora dist-git sync.

Modes: production (cwd = repo, always create MRs) vs clone dry-run vs clone + --create-mrs.

Algorithm:

  1. Select metadata/*.json files; apply --clean-only / --modified-only filter; skip native packages.
  2. Record START_COMMIT on target branch.
  3. For each package: run ./ci/dist_git.py update PACKAGE. Exit 0 + new commit = clean update. Exit 2 + new commit = conflict update.
  4. Collect all new commits, hard-reset target to START_COMMIT.
  5. Per commit: parse subject (^(Update|Sync) ([^ ]+) from ... to ...), create branch chore/dist-git-update-<pkg>, cherry-pick. Conflict → draft + no-test, no auto-merge. Clean → --auto-merge.
  6. Call create_mr.sh.

Branch convention: chore/dist-git-update-<pkg>. Title: chore(rpms): <commit subject> or CONFLICT: chore(rpms): <subject>.

rpms upstream_update_multi_mr.sh

Same multi-MR / cherry-pick pattern as dist-git, for upstream version bumps.

Algorithm: runs ./ci/check_upstream_versions.py check --update --sign-off, collects commits, splits into per-package MRs. Always --auto-merge. Branch: chore/upstream-update-<pkg>.

rpms rebuild_multi_mr.sh

Does not generate rebuilds. Expects local commits from dist_git.py rebuild or rebuild-rev-deps, then splits them into per-package MRs.

Unique feature: --force flag deletes existing remote branch before create, working around stale branches from closed MRs. Branch: chore/rebuild-<pkg>.

containers create_lockfile_update_mrs.sh

Creates or updates one MR per image group when RPM lockfiles changed. Prerequisite: lockfiles already refreshed (make all-host FORCE_REFRESH=true).

Algorithm (process_single_image):

  1. Branch: rpm-lockfile-updates/<image>.
  2. For each distro/variant lockfile: compare to HEAD after stripping .arches[].packages[].url and .arches[].source[].url via yq (URL churn ignored).
  3. If no meaningful changes → skip.
  4. If push-mode ≠ force and origin/<branch> exists: compare to remote; if identical → skip.
  5. git switch --force-create branch, commit lockfiles.
  6. Push --force-with-lease + MR push options.

Key difference from rpms: uses --force-with-lease to update existing branches in place. Dedup compares lockfile content, not just branch existence.

containers create_mr.sh (simpler helper)

Used by metrics report. Creates from working tree (not existing commits). git add --all, commit, --force-with-lease push. No dedup — overwrites existing branch.

Deduplication Summary

Script Existing branch Content refresh
rpms create_mr.sh Skip (exit 2) Never
rpms multi-MR scripts Via create_mr.sh Never (unless rebuild --force)
containers lockfile Skip if remote content matches Yes (--force-with-lease)
containers create_mr.sh Overwrites Yes

Hummingbird Agent Framework

The Hummingbird Agent is an event-driven LLM agent deployed on OpenShift that processes GitLab events via SQS.

Architecture

  • Event-driven: consumes SQS events (pipeline failures, MR events, slash commands). No REST API — triggered only by SQS or CLI.
  • Workflow-driven: investigation logic defined in markdown files as LLM system prompts. New behaviors do not require code changes.
  • Sandboxed execution: all LLM commands run in isolated containers (Podman/K8s/KubeVirt) with no network access and no credentials.
  • Multi-model: supports Gemini and Claude via Vertex AI.
  • Token separation: model tokens (read-only, Reporter-level) are distinct from orchestrator tokens (write-capable, used only by deterministic code outside the sandbox).
  • Single output action: post_gitlab_note is the only way the agent affects the outside world. It cannot create, modify, or merge MRs.

Current Workflows

analyze-failures: triggered on pipeline failure. Bulk-fetches GitLab CI job logs, Konflux PipelineRun/TaskRun metadata and logs, and Testing Farm results. Groups failures by root cause. Posts a structured MR note with collapsible forensic evidence and links.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_diff, gitlab_get_commit_statuses, konflux_list_pipelineruns, konflux_get_pod_log, tf_get_results, tf_get_test_log, sandbox_exec
  • Pattern: bulk metadata fetch → selective deep log pulls → grouped analysis → structured note
  • Flexibility: medium-low (tightly coupled to Konflux/Tekton/TF pipelines)

code-review: triggered on MR open/update. Reads the unified diff, prior review discussions, and per-repo rules. Posts review findings with severity, category, and concrete code fix examples.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, gitlab_get_mr_discussions, gitlab_get_file_at_ref, sandbox_exec
  • Pattern: diff-first analysis, discussion-aware follow-up (avoids re-raising resolved issues), per-project rules overlay
  • Per-repo rules: workflows/repo-rules/<group>-<repo>.md files provide project-specific review guidance (e.g., hummingbird-rpms.md documents package metadata conventions)
  • Flexibility: high (generic for any GitLab MR; domain knowledge plugs in via repo-rules/)

renovate-babysit: triggered on successful Renovate MR pipeline. Classifies the update as safe or risky based on diff scope, upstream changelogs (via web search), and local API usage. Posts a triage verdict.

  • Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, web_search, gitlab_get_repo_archive, sandbox_exec
  • Pattern: diff scope classification, embedded changelog first then web search, archive + grep for call-site impact
  • Flexibility: medium-high (ecosystem-agnostic risk matrix; does not approve/merge — triage note only)

What the Agent Has That Maps to a Workqueue

  • SQS FIFO queue with message grouping (serializes per session)
  • Session persistence (S3 context.json)
  • Config hot-reload in serve mode
  • Per-project token separation (model / action / orchestrator)
  • Rate limiting (per-workflow per-MR)
  • Data source abstraction (GitLab, Konflux, Testing Farm)
  • Auto-resolve on push/success

What the Agent Lacks for MR Lifecycle Management

  • No MR creation capability (no git write operations, no push access)
  • Read-only GitLab tokens for the model
  • No workqueue for MR lifecycle state (events are fire-and-forget)
  • Single output action (post_gitlab_note only)
  • No safe-outputs pattern (no validation of proposed writes)
  • No deterministic fallback paths for well-understood operations
  • No request/response API (SQS consumption and CLI only; external services cannot invoke a workflow and retrieve structured results)

Event Bus

The existing event pipeline could serve as the backbone for a workqueue:

GitLab MR / note / pipeline webhooks
  → gitlab-event-forwarder (API Gateway → Lambda)
  → SNS hummingbird-events-topic
  → mr-auto-approver Lambda (approve safe bot MRs)
  → hummingbird-agent SQS (analyze / review / babysit)
  → hummingbird-status SQS (pipeline tracking)

mr-auto-approver is a pure rule-based approver — no LLM involvement. hummingbird-mr-human-tracker detects human fixes on Renovate/bot MRs and tracks them in Jira epics.

Security Foundation: HUM-851

HUM-851 (“Enable Secure Agent MR Creation”) is an existing epic (status: New) that designs the security posture for agents creating MRs.

Core Principle

Instead of protecting CI secrets from agent-authored MR pipelines, eliminate all secrets from containers/rpms/tools CI entirely. Move all secret-dependent jobs to the infrastructure repo’s CI. With zero secrets, a compromised agent MR pipeline can only access CI_JOB_TOKEN (single-project-scoped for bot users) and the runner’s network.

What Moves to Infrastructure Repo CI

Job Current project Current trigger
deploy_hugo_tag containers, rpms, tools On merge to main
lockfile_update containers Scheduled
Lockfile/chore approval containers, rpms After MR CI passes
Chore MR creation containers, rpms Scheduled
dist_git_update rpms Scheduled
upstream_version_update rpms Scheduled
metrics_report containers Scheduled
update_quay_description containers Scheduled/on merge

This migration directly enables centralized MR creation.

Target Security Posture

  • Zero CI secrets in target projects
  • Restricted runners with egress filtered to package registries only
  • Agent never holds write tokens — writes happen in deterministic orchestrator code
  • Diff safety validation — blocklist prevents changes to CI config, .tekton/, dependency files
  • Human review required for every agent MR, auto-labeled agent-generated
  • Rate limiting per-project and global; auto-close stale MRs
  • Monitoring: every agent MR logged and Slack-notified

Task Breakdown

Task Status What it does
HUM-852 New Threat model and security design
HUM-853 New Move ops jobs to infra CI
HUM-854 Closed K8s CronJob/Job manifests (done)
HUM-855 New Validate and remove CI variables
HUM-856 New Restrict runner egress
HUM-857 New MR creation orchestrator action
HUM-858 New Rate limiting and abuse prevention
HUM-859 New Monitoring and incident response

Renovate Deployment

Deployed as Kubernetes CronJobs in the infrastructure repo (kubernetes/renovate/20-cronjobs.yml.j2):

  • 5 repos: containers, infrastructure, k8s-test-pipeline, rpms, tools
  • Each gets its own CronJob running hourly with concurrencyPolicy: Replace
  • Uses MintMaker image (MINTMAKER_IMAGE)
  • containers/rpms use bash wrappers and custom config files
  • Custom env variables for RPM lockfile cache, Rust/Cargo paths

CVE Analysis Pipeline

The CVE analysis pipeline is a complex, mature system. The relevant interface points for a workqueue are:

  1. hummingbird-cve-analysis CronJob (K8s, every 30 min prod) — runs cve_analysis --resolve which triages Jira Security tickets, transitions statuses, creates advisory MRs, manages labels.
  2. collect_cve_dashboard — runs after analysis, posts lifecycle data to the dashboard API.
  3. /cve skill — reads staged CVE analysis data from Jira tickets and creates fix MRs in the rpms repo.

The workqueue integration point: when /cve or CVE analysis determines an MR should be created, that request becomes a work item.

RPM-to-Container Pipeline Flow

A single RPM spec change triggers a cascade that spans both repos and multiple build systems. The full flow is documented in the RPM Pipeline and Image Pipeline docs.

RPM spec change (rpms repo)
  → MR validation (GitLab CI + Konflux build + Testing Farm)
  → Merge to main
  → Konflux RPM build (Tekton PipelineRun per package, mock hermetic)
  → RPM signing (Kerberos-based)
  → Publish to Pulp (packages.redhat.com)
  → Lockfile update detects new RPM versions (containers repo)
  → Lockfile MR created (Renovate or create_lockfile_update_mrs.sh)
  → Container build (Konflux, multi-arch)
  → Container testing (Testing Farm + K8s tests)
  → Enterprise Contract validation (Conforma)
  → Release to Quay.io

Key observations for the workqueue:

  • The cascade is implicit — there is no explicit trigger from RPM publish to container lockfile update. Lockfile updates discover new RPMs by polling Pulp (via make all-host FORCE_REFRESH=true or Renovate’s RPM lockfile maintenance).
  • A single RPM change can fan out to many container images (any image that includes that package in its lockfile).
  • The RPM pipeline has 5 stages (spec change, MR validation, build, signing, publishing). The container pipeline has 6 stages (source templates, generation, build, testing, Enterprise Contract, release).
  • Testing Farm runs integration tests on merge requests only, not on main branch builds. Container tests include reverse dependency testing (rebuilding dependent images locally in the test environment).
  • Enterprise Contract (Conforma) validates supply chain security, hermetic builds, and policy compliance before release.

Decision

To be determined by the spike investigation. This section will be updated with the chosen architecture.

The spike should investigate a unified workqueue-based system where:

  • Queue feeders replace or wrap the current scheduled scripts and Renovate outputs
  • Queue processors handle MR creation, approval, rebasing, and merge through deterministic code
  • Agentic capabilities (review, patch generation, triage) are invoked as composable services with structured output
  • The security model follows HUM-851’s design: agents reason read-only, orchestrators execute writes

Design Patterns and Prior Art

GitHub Agentic Workflows (gh-aw)

gh-aw provides a reference architecture for AI-assisted repository automation with security guardrails:

  • Read-only agent execution: agents run with no write permissions, no secrets
  • Safe Outputs: agents buffer intended write operations as structured JSON artifacts; separate permission-controlled jobs validate and execute them
  • Threat detection: a separate AI agent scans agent outputs before write execution
  • Plan-level trust: trust is embedded in the workflow definition (what the agent is allowed to do), not in the agent’s runtime decisions

Key pattern: separate reasoning (agentic, sandboxed, read-only) from execution (deterministic, permission-controlled, auditable).

Existing Hummingbird Patterns

  • SQS FIFO queue with message grouping: already serializes work per session; can be extended to serialize per-MR
  • Workflow-as-prompt: markdown files define behavior; new behaviors do not need code changes
  • Data source abstraction: GitLab API access is modular
  • Token tier separation: model read tokens, action write tokens, orchestrator tokens — maps directly to the safe-outputs pattern
  • Sandbox isolation: reusable for agentic review/analysis tasks

Agent as Composable AI Service

One approach to bridge the agent framework and a workqueue is to evolve the agent from a self-contained event processor into a building block that non-AI services can call:

  • Request/response API: HTTP endpoint accepts workflow requests, returns a UUID; caller polls for structured JSON results. The caller (e.g., a workqueue herder) applies deterministic logic to decide what to do with the output.
  • Structured output: agent produces typed JSON (not just a GitLab note), enabling callers to process results programmatically.
  • Safe-output actions: new action types beyond post_gitlab_note (e.g., create_mr, update_mr) with per-workflow path allowlists, diff size limits, and branch naming constraints. The orchestrator validates each action against the workflow’s configured constraints before execution.

Open Questions

  1. Queue technology: extend SQS FIFO (already in use by the agent) or use a different queue (K8s-native, Redis, PostgreSQL)?
  2. Processor deployment: K8s Deployments polling SQS, GitLab CI jobs triggered by webhooks, Lambda functions, or a combination?
  3. Renovate strategy: keep Renovate as a separate MR creator and add a herder, or replace Renovate’s MR creation with the workqueue?
  4. Write credentials: per-repo project access tokens (current model) or a single bot account? How does HUM-851’s “eliminate CI secrets” model interact with workqueue processors that need write access?
  5. Safe outputs validation: simple schema validation, deterministic allowlist checks, or full threat detection (as in gh-aw)?
  6. Review scope: which MRs need AI review before merging vs which can be auto-merged with deterministic checks only?
  7. State tracking: queue, GitLab labels, a database, or the existing hummingbird-status PostgreSQL?
  8. Agent evolution: request/response API, safe-output actions, or hybrid? How does this relate to HUM-857?
  9. HUM-851 relationship: is the workqueue spike a superset of HUM-851, or a separate epic that depends on HUM-851 Phase 1?

Consequences

To be updated after the spike investigation determines the architecture.

Known consequences of any workqueue approach:

  • Centralizing MR creation provides a single point for audit, rate limiting, and monitoring
  • Moving secret-dependent jobs to the infrastructure repo (per HUM-851) is a prerequisite regardless of workqueue design
  • The existing CI approval jobs, Lambda approver, and Renovate automerge have overlapping scope that must be reconciled
  • RPMs Renovate MRs currently have no automated approval path — any design must address this gap
  • The agent’s current workflows (failure analysis, code review, Renovate babysit) complement the workqueue and should not be disrupted

File and System Inventory

Infrastructure repo

Path Contents
kubernetes/renovate/ Renovate CronJob definitions (5 repos)
kubernetes/hummingbird-agent/ Agent deployment manifests (8 files)
kubernetes/hummingbird-cve-analysis/ CVE analysis CronJob
aws/hummingbird-agent/ Agent SQS/SNS SAM templates

Tools repo

Path Contents
hummingbird-agent/ Agent framework (loop, sandbox, models, data sources, workflows)
hummingbird-agent/workflows/ Workflow definitions (analyze-failures, code-review, renovate-babysit)
hummingbird-agent/workflows/repo-rules/ Per-project review rules (hummingbird-rpms.md)
hummingbird-cve-analysis/ CVE analysis pipeline (preserve)
hummingbird-dashboard/ Dashboard web app (preserve)
mr-auto-approver/ Lambda: rule-based MR approval
hummingbird-mr-human-tracker/ Lambda: tracks human fixes on bot MRs
gitlab-ci/gitlab_sync/ File sync MR creation with rebase
gitlab-event-forwarder/ Webhook → SNS bridge
hummingbird-events-topic/ SNS topic infrastructure

RPMs repo

Path Contents
ci/create_mr.sh Shared MR creation helper (push options, dedup by branch)
ci/dist_git_update_multi_mr.sh Dist-git sync: commit-all → reset → cherry-pick per package
ci/upstream_update_multi_mr.sh Upstream version bumps: same cherry-pick pattern
ci/rebuild_multi_mr.sh Rebuild MR splitter (with --force for stale branches)
ci/dist_git.py CLI for import/update/sync/rebuild operations
ci/check_konflux_statuses.py Polls Konflux external statuses before approval
ci/claude_resolve_conflict.sh AI-assisted conflict resolution for dist-git MRs
metadata/<package>.json Per-package metadata (upstream URL, CVE mapping, status)

Containers repo

Path Contents
ci/create_lockfile_update_mrs.sh Per-image lockfile MR creation (force-with-lease, content dedup)
ci/create_mr.sh Generic chore MR helper (force-with-lease, no dedup)
ci/internal/shared_lib.sh Shared utilities (get_distro_variants, build helpers)
images/<name>/ Per-image Containerfiles and RPM lockfiles

6.3 - ADR-0003: RPM metadata and versioning

  • Status: Accepted
  • Date: 2026-08-31
  • Accepted: 2026-11-17
  • Author: Brent Baude
  • Jira: HUM-6212

Context

Each RPM package has build inputs in rpms/<package>/ and a machine-readable record in metadata/<package>.json. Automation uses metadata to decide whether Fedora updates may overwrite local build inputs, which upstream project to track, and how CVE analysis maps a package to its upstream identity.

Package metadata needs consistent definitions for the relationship to Fedora, required fields, and versioning behavior. In particular, storing a release base that can disagree with the spec’s current Release: value caused incorrect rebuild increments in HUM-5182.

This ADR originally contained the full metadata specification. Subsequent proposals to change the metadata contract need a single current specification, rather than definitions spread across the original ADR and later decisions.

Decision

Adopt the RPM metadata and versioning specification as the authoritative contract for metadata/<package>.json and its relationship to RPM versioning. The specification was extracted from this ADR without changing the metadata contract.

The adopted model records a package’s relationship to Fedora through modification_status and Fedora provenance. Metadata version matches the spec’s resolved Version:. The spec’s Release: line is the sole authoritative release value for what ships. fedora.release, when present, records only the Fedora base from which that value is built.

No metadata field records a release base for packages ahead of Fedora or packages without a Fedora origin. Rebuilds increment the spec’s current release directly in those cases, removing the stored value that could disagree with the spec and cause the bug class behind HUM-5182.

Specification changes

Changes to the metadata contract or its semantics require an ADR and a corresponding specification update. The ADR records the proposal, rationale, and compatibility implications. The specification defines the resulting contract and remains the source of truth.

Review the ADR and specification update together, using linked merge requests when necessary. Mark the ADR accepted when the specification change is approved, and record the change in the specification’s change history with a reference to that ADR. Editorial corrections and example clarifications that do not change the contract do not require an ADR.

Consequences

  • metadata/*.json across the rpms repository must be brought in line with the specification. That migration remains separate, tracked work. Accepted status records the decision, not completion of the migration.
  • dist_git.py, check_upstream_versions.py, and CVE analysis tooling can implement the specification’s validation rules directly.
  • The spec’s Release: line remains the sole authoritative value for what ships; no metadata field may be treated as a substitute for it, including fedora.release.
  • Future contract changes update the standalone specification, without rewriting this ADR’s original metadata decision or requiring readers to assemble the contract from multiple ADRs.
  • RPM operating documentation explains workflows and links to the specification for definitions. It does not define a competing metadata contract.

Specification sections

The following headings preserve links to sections originally embedded in this ADR.

Package states

See Package states in the specification.

Field reference

See Field reference in the specification.

Computing the built NVR

See Computing the built NVR in the specification.

Validation rules

See Validation rules in the specification.

Examples

See Examples in the specification.

6.4 - ADR-0004: Move private_product into package metadata

  • Status: Accepted
  • Authors: Prarit Bhargava
  • Jira: HUM-11758
  • Date: 2026-11-05
  • Accepted: 2026-11-18
  • Last updated: 2026-11-19

This record relocates the design approved in infrastructure!1979. The decision is unchanged.

Summary

This ADR moves the per-package private_product setting out of ci/package-overrides.yaml in the rpms repository and into the package metadata JSON (metadata/<package>.json). It also requires that every read, write, and validation of private_product is implemented in the core library, hum.meta, and that the CI scripts that use the value go through that library instead of parsing YAML or JSON themselves.

The RPM metadata and versioning specification defines the authoritative metadata contract. This ADR records the decision and rationale for extending it.

Motivation

private_product routes a package to a private, per-product Pulp repository (for example nodejs20) instead of the public one. A private product groups packages for publication to access-restricted repositories. Today the setting is stored in ci/package-overrides.yaml, which is a build-configuration file, not package metadata. This causes the following problems:

  • Implicit classification. Because the setting is not part of the package metadata, other tooling tends to infer private or long-term-support (LTS) status from the package name, for example a -lts suffix. A name is not a reliable signal: an upstream project can legitimately have a name that ends in -lts without being an LTS package. Routing and classification must be driven by explicit data, never by naming conventions.
  • Split source of truth. Everything else about a package (origin, version, modification status, upstream tracking) lives in metadata/<package>.json and is handled by hum.meta. private_product is the exception, so a reader has to consult two files to understand one package.
  • Duplicated file handling. ci/generate_resources.py, ci/check_nevr_conflicts.py, and ci/dist_git.py each read or edit package-overrides.yaml directly. dist_git.add_private_product() edits the file line by line to preserve formatting. This is the kind of direct file manipulation that hum.meta exists to replace, and it is not covered by the library’s validation.

Depends on

Impacts

User facing

None for external users. Packages continue to be built and published to the same repositories. Maintainers set private_product in the package’s metadata JSON instead of package-overrides.yaml.

Repositories

  • documentation/
  • gorget/
  • rpms/
  • containers/
  • infrastructure/
  • skills/
  • tools/ (CI, Website)

rpms/ holds the metadata, the library, and all affected scripts. documentation/ holds this ADR and the metadata specification. infrastructure/ removes the original design document in favor of this ADR.

Decision

1. Metadata field

Add an optional private_product field to metadata/<package>.json:

  • Type: string. When absent, the package is public.
  • The value is the product name and must match the private_product of an entry in ci/konflux_rpa_config.yml.
  • The field is valid in every metadata state; it is orthogonal to the clean, modified, and independent states.
  • The private_product key in ci/konflux_rpa_config.yml is a property of the RPA entry, not of a package. It is unchanged by this ADR.

2. Core library changes (hum.meta)

All handling of the field is implemented in hum.meta. No script may read or write private_product by parsing a metadata or overrides file directly.

  • hum.meta.model: add private_product: str | None to PackageMetadata and the concrete subclasses, with a validated setter and JSON round-tripping.
  • hum.meta.validation: validate the value (non-empty string, allowed character set). Check that it matches an RPA entry in ci/konflux_rpa_config.yml; this check moves out of generate_resources.py.
  • hum.meta.repository: provide the read and write paths. Setting the field is a load, set, save operation on MetadataRepository, which replaces the line-based YAML edit.
  • Queries: provide library helpers to enumerate packages by product ({package: private_product} and the inverse) so consumers do not scan files.
  • Tests in hum/meta/tests/ cover the field, its validation, and the repository round trip.

3. Consumers

Each consumer switches to the library:

File Change
ci/generate_resources.py Use the library for per-package product lookup and the RPA cross-check. Public RPAs still exclude packages that have a product.
ci/check_nevr_conflicts.py Replace load_private_products() with a library query. Behavior is unchanged.
ci/dist_git.py add_private_product() sets the field through MetadataRepository instead of editing package-overrides.yaml. It still adds the RPA entry in konflux_rpa_config.yml.

The private_product documentation block in ci/package-overrides.yaml is removed once no entries remain.

4. Migration

Existing entries (nodejs20 and the example entry) are moved into the matching metadata/<package>.json files in the same MR that ports the consumers, so the data and the code change together. This follows the pattern used for HUM-6232. A one-time script is acceptable, but it is not kept in the repository afterwards. During the migration, no package may have the field in both places.

Risks and open questions

  • Schema change process. Metadata changes require an ADR and a corresponding specification update. Review and approve this ADR with the specification update before merging any implementation MRs.
  • Two-file consistency. A package’s product and the RPA entry live in different files. The library validation in section 2 is the guard against drift. The open question is whether the RPA entry should also move into metadata in a later ADR.
  • Allowed characters. The allowed character set for product names still needs to be defined.
  • Other readers of the overrides file. Scripts outside rpms/ci and documentation may refer to private_product in package-overrides.yaml. The implementation MR must search for these before the key is removed.

Implementation notes

  • Implementation is tracked in HUM-13038.
  • Land the library change (hum.meta, with tests) first, then the consumer and migration MR, then the operating documentation updates.
  • Documentation to update in rpms/: documentation/operating/private-rpm-repositories.md, documentation/operating/package-metadata-fields.md, and documentation/background/rpm-pipeline.md.
  • Update the tests in test/test_generate_resources.py, test/test_check_nevr_conflicts.py, and test/test_dist_git.py to build their fixtures from metadata JSON instead of package-overrides.yaml.
  • When the implementation is complete, a follow-up MR changes this ADR’s status to Implemented.

7 - Specifications

Specifications define current technical contracts. Architecture decision records capture the decisions and rationale behind those contracts.

7.1 - RPM metadata and versioning specification

This is the authoritative specification for RPM package metadata and versioning. ADR-0003 records the adopting decision and rationale. Contract changes require an ADR and a corresponding update to this specification; editorial corrections do not require an ADR.

Scope

The metadata file at metadata/<package>.json and the package directory at rpms/<package>/ form a pair, one per package. The package directory holds the build input: the spec file, patches, and the sources checksum manifest that mock consumes to produce RPMs. This specification does not change anything about that directory.

The metadata file is the machine-readable record of a package’s relationship to Fedora and upstream. Automation reads it to decide what it may do: whether dist_git.py update may overwrite the package directory, which upstream project check_upstream_versions.py tracks, and how CVE analysis maps the package to its upstream identity. The optional private_product field records a product assignment for Hummingbird’s internal use.

The metadata file does not duplicate build inputs, and it does not record the release currently shipped: the spec’s Release: line is the sole authoritative value for what ships, and is what NVR computation uses. See Computing the built NVR. For packages on Fedora’s version, fedora.release also appears in metadata, but only as the base value the spec’s Release: is built from, never as a substitute for it.

The two locations are updated independently; neither is derived from the other. They must stay consistent by convention, not by construction. modification_status is correct only if it accurately reflects whether the package directory actually diverges from Fedora’s import.

This document normatively defines the target structure of metadata/<package>.json and how release and versioning behave as a function of modification_status. It describes the target metadata shape, not what is present on disk in metadata/*.json today. Bringing existing metadata in line with this specification is separate, tracked work.

Package states

Every package occupies exactly one of three states, describing its relationship to Fedora at a given moment. A state is not itself a metadata field. It is what modification_status means in combination with which version the package currently ships. The rest of this specification refers to these states by name, so they are defined here first.

A package’s state follows from its modification_status, which is itself defined by the package’s relationship to a Fedora origin. modification_status is one of three values:

  • clean: the package has a Fedora origin, and currently matches that Fedora import.
  • modified: the package has a Fedora origin, but currently diverges from it, either through local changes while still on Fedora’s version, or by having moved ahead to a newer upstream version.
  • independent: the package has no Fedora origin at all.

For a modified package, the state further depends on whether it currently ships Fedora’s exact version or has moved ahead to a newer upstream version. The fedora object’s complete or partial shape records which of those is the case; it does not itself define the state.

The table below names all three states by the version they currently ship, not just by modification_status, since modified alone is ambiguous between the two Fedora-origin cases:

Version currently shipped modification_status
Fedora’s version clean or modified
An upstream version ahead of Fedora modified only
No Fedora origin (independent) independent

Field reference

This is the complete field-by-field reference for metadata/<package>.json, covering every field consumed by tooling in the rpms repository, not only the fields that determine modification_status or the Fedora relationship. Fields are listed in four groups, in order: package identity and state (version through modification_reason), the Fedora-origin group (fedora and its sub-fields), fields that configure upstream-version tracking, and fields unrelated to either Fedora or upstream tracking (private_product, cve_product, version_transform, fix_status). The “Presence” column states the condition under which a field appears in the JSON; fields are omitted entirely rather than set to null or an empty value when their condition does not hold.

Field Type Presence Description
version string Always The version currently shipped.
modification_status enum: clean, modified, independent Always Whether the package matches its Fedora import unmodified (clean), carries local changes (modified), or has no Fedora origin at all (independent).
modification_reason string Only in the modified state Why the package carries local changes. Absent for clean and independent packages.
fedora object In the clean state or either sub-case of the modified state Everything about the Fedora build this package tracks or last tracked. Absent entirely in the independent state.
fedora.git_url string Whenever fedora is present The Fedora dist-git repository URL.
fedora.branch string Whenever fedora is present The Fedora dist-git branch (e.g. rawhide).
fedora.sha string Whenever fedora is present The exact Fedora commit imported or last synced.
fedora.release string Only in the on-Fedora-version state (clean, or modified while still on Fedora’s version) The release number of the Fedora build tracked, dist tag stripped. Absent once the package moves to an upstream version ahead of Fedora.
upstream_repo string (git URL) Always The canonical upstream project repository. Falls back to the Fedora dist-git URL when no independent upstream repository exists.
upstream_branch string Optional; independent of state Pins one package’s upstream line when several packages share a single upstream_repo (e.g. an nodejs2x-style family).
version_from_ref object: {"type": "commit-date"} Optional; independent of state; requires upstream_repo Configures dist_git.py update’s fixed-ref pin-refresh logic to compute a gorget source-pipeline’s --version string automatically from a newly-pinned commit, instead of requiring manual resolution. commit-date is currently the only supported type.
track_upstream string: "latest" or a version prefix (e.g. "1.26") Optional; independent of state Enables check_upstream_versions.py tracking of this package; a version prefix constrains which upstream releases are accepted.
release_monitoring_project_id integer or string Optional; independent of state; only meaningful paired with track_upstream The release-monitoring.org (Anitya) project ID or name check_upstream_versions.py queries.
version_source string: currently only "gitlab_tags" Optional; independent of state Selects an alternate upstream-version source for check_upstream_versions.py instead of Anitya.
tag_strip_prefix string, default "v" Optional; independent of state Used with version_source: "gitlab_tags" to strip a tag-name prefix before comparing versions.
version_suffix_strip string Optional; independent of state A suffix check_upstream_versions.py strips from an Anitya-reported version before comparison and update (e.g. "-RELEASE").
upstream_version_transform string Optional; independent of state A named transform check_upstream_versions.py applies to normalize an Anitya-reported version into RPM version syntax (e.g. openjdk_to_rpm). A distinct field from version_transform below, with a different consumer. Do not conflate the two.
source_availability_check string Optional; independent of state A named checker (registered in SOURCE_AVAILABILITY_CHECKERS) that HEAD-probes a candidate source URL before check_upstream_versions.py accepts that version.
private_product string Optional; independent of state An internal product identifier used for private package distribution. Absent for public packages.
cve_product string, or list of strings Optional; independent of state A CVE vendor/product override, independent of Fedora or upstream tracking.
version_transform string Optional; independent of state A version-mapping rule consumed by CVE analysis tooling in the tools repository (e.g. dotnet_sdk_to_runtime). Not consumed by anything in the rpms repository’s own ci/ scripts.
fix_status integer Optional; independent of state Not consumed by any tooling in the rpms repository today. Carried over from history and likely stale. Not a field to use for new work.

Private product assignment

private_product is an optional, non-empty string identifying a private product, such as nodejs20, for Hummingbird’s internal use. When absent, the package is public. The field is valid in every package state and does not change the package’s relationship to Fedora or its versioning behavior.

ADR-0004 describes the tooling behavior and implementation requirements.

Computing the built NVR

The built RPM is identified by its NVR: Name, Version, and Release. Name is the spec/package name and is unaffected by any of this. Only Version: and Release: vary with state.

Metadata version and the spec Version: line always match. Metadata stores it separately not because the two can diverge, but because reading Version: reliably from every spec (which may build it from macros or indirection) isn’t practical, so metadata is the field automation trusts instead. Release: works the other way: the spec Release: line is the sole authoritative value for what ships, in every state. fedora.release, when present, is only the base value Release: is built from, never a substitute for it.

The table below shows, for each state, where the spec’s Version: and Release: lines come from. The prose beneath it gives the precise rebuild-increment rules that don’t compress into a table cell.

Version currently shipped Spec Version: Spec Release:
Fedora’s version (clean or modified) Matches metadata version fedora.release plus %{?dist}, with an optional trailing .N for local rebuilds
An upstream version ahead of Fedora (modified only) Matches metadata version Starts at 0.1%{?dist} the first time the package moves ahead of Fedora; local rebuilds increment the trailing .N directly from the spec’s own current value
No Fedora origin (independent) Matches metadata version A one-time human-chosen base (typically 1 or 0.1) plus %{?dist}; local rebuilds increment the trailing .N directly

On Fedora version (clean or modified): fedora.release holds Fedora’s confirmed release number, dist tag stripped. The spec Release: is that value plus %{?dist}, with an optional trailing .N if the package has been locally rebuilt without a source change since the last import or sync. dist_git.py rebuild computes the next .N by comparing the spec’s current Release: against fedora.release: if they match exactly, this is the first local rebuild since import or sync, and .1 is appended; if the spec already carries a trailing .N ahead of fedora.release, that trailing number is incremented.

On upstream version (always modified): No fedora.release, and no other metadata field records a base value. The spec uses Release: 0.1%{?dist} the first time a package moves ahead of Fedora. Every subsequent local rebuild increments the trailing dot-number directly from the spec’s own current Release: value (0.1 → 0.2 → 0.3, and so on), never by re-deriving it from a stored metadata value, because none exists. This is what makes the bug class behind HUM-5182 structurally impossible: that bug occurred because a stored placeholder value could disagree with the spec’s actual current Release:; with nothing stored to disagree with, the mis-increment cannot happen.

Independent: The same rule as the on-upstream-version case applies. No metadata field records a base value, and the spec Release: is authoritative. Rebuilds increment its trailing .N directly. The only difference is the very first value chosen when the package is added (typically 1 or 0.1) is a one-time human decision, not derived from anything.

Validation rules

Every rule below is a single, unconditional statement: given the field values present, exactly one outcome is correct. None of them carry a qualifier like “should,” “generally,” or “in most cases,” and none fold an exception into the wording. A rule that needs an exception is really two rules, stated separately. The rationale for why each rule holds lives elsewhere in this document (Package states, Field reference, Computing the built NVR); this section exists so a validator can implement each line as a direct check with nothing left to interpret.

  1. version is present.
  2. modification_status is present.
  3. upstream_repo is present.
  4. modification_status is clean, modified, or independent.
  5. version_source, when present, is gitlab_tags.
  6. source_availability_check, when present, is a name registered in SOURCE_AVAILABILITY_CHECKERS.
  7. The top-level field release does not appear.
  8. If modification_status is clean, fedora is present.
  9. If modification_status is modified, fedora is present.
  10. If modification_status is independent, fedora is absent.
  11. If fedora is present, it includes git_url, branch, and sha.
  12. If modification_status is clean, fedora.release is present.
  13. If modification_status is modified, fedora.release is present if and only if version equals the version Fedora ships at fedora.sha.
  14. fedora.release, when present, does not include a dist-tag suffix (e.g. .fc42, .el9).
  15. If modification_status is modified, modification_reason is present.
  16. If modification_status is clean or independent, modification_reason is absent.
  17. If track_upstream is present and version_source is absent, release_monitoring_project_id is present.
  18. If release_monitoring_project_id is present, track_upstream is present.
  19. If version_source is gitlab_tags, release_monitoring_project_id is absent.
  20. If tag_strip_prefix is present, version_source is gitlab_tags.
  21. The spec’s Version: line, resolved through macro expansion, equals metadata version.
  22. If version_from_ref is present, upstream_repo is present.
  23. version_from_ref.type, when present, is commit-date.
  24. private_product, when present, is a non-empty string.

Examples

Metadata for a package on Fedora version, clean:

{
  "version": "5.4.3.0",
  "modification_status": "clean",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/dnf5.git",
    "branch": "rawhide",
    "sha": "f174d0fcedc78a34d74b800697862b93683ad5e0"
  },
  "upstream_repo": "https://github.com/rpm-software-management/dnf5"
}

Metadata for a package on Fedora version, modified:

{
  "version": "5.3.15",
  "modification_status": "modified",
  "modification_reason": "Added gorget source-pipeline.yaml (HUM-5841; pattern from HUM-4622): a fresh fetch of Source0's bash-5.3.tar.gz no longer matches the recorded checksum -- upstream re-published the base tarball (translation file + build stamp only, GPG signature re-verified against the already-trusted key) -- so Fedora's cache no longer serves bytes matching the corrected checksum",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/bash.git",
    "branch": "rawhide",
    "sha": "2768211b5135c7169f513965d79e9f89e0ca6124"
  },
  "upstream_repo": "https://git.savannah.gnu.org/git/bash.git"
}

Metadata for a package on an upstream version ahead of Fedora, modified:

{
  "version": "0.9~rc4",
  "modification_status": "modified",
  "modification_reason": "update to 0.9rc4",
  "fedora": {
    "git_url": "https://src.fedoraproject.org/rpms/avahi.git",
    "branch": "rawhide",
    "sha": "195919c9f9a8dff0e996921afd8736df8f12715b"
  },
  "upstream_repo": "https://github.com/avahi/avahi"
}

Note the absence of fedora.release and of any top-level release field. This package is ahead of whatever Fedora currently ships, so there is no confirmed Fedora release to record.

Metadata for an independent package:

{
  "version": "1.3.3",
  "modification_status": "independent",
  "upstream_repo": "https://github.com/oras-project/oras",
  "track_upstream": "latest",
  "release_monitoring_project_id": 205787
}

No fedora object and no release-related field. The release currently shipped lives only in that package’s spec Release: line.

For a package assigned to the private nodejs20 product, its metadata includes this field:

{
  "private_product": "nodejs20"
}

This is a partial record; all fields required by the package’s state remain required.

Change history

Record contract changes here with a reference to the ADR that approved each change.

Date Change Decision
2026-11-17 Extracted the specification from ADR-0003 without changing the metadata contract. ADR-0003
2026-11-18 Added optional private_product metadata for internal product classification. ADR-0004